Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1208 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.8182. Min Range: -1.8182

Max Act: 1.7621. Min Act: -0.0001

Data Index: 965592 (C4 (Web Text))

Max Activating Token Index: 814

Text #1

Max Range: 1.8182. Min Range: -1.8182

Max Act: 1.8182. Min Act: -0.0001

Data Index: 101330 (C4 (Web Text))

Max Activating Token Index: 732

Text #2

Max Range: 1.8182. Min Range: -1.8182

Max Act: 1.4097. Min Act: -0.0001

Data Index: 317386 (C4 (Web Text))

Max Activating Token Index: 183

Text #3

Max Range: 1.8182. Min Range: -1.8182

Max Act: 1.4846. Min Act: -0.0001

Data Index: 221191 (C4 (Web Text))

Max Activating Token Index: 557

Text #4

Max Range: 1.8182. Min Range: -1.8182

Max Act: 1.5113. Min Act: -0.0001

Data Index: 32302 (C4 (Web Text))

Max Activating Token Index: 731

Text #5

Max Range: 1.8182. Min Range: -1.8182

Max Act: 1.3413. Min Act: -0.0001

Data Index: 199518 (C4 (Web Text))

Max Activating Token Index: 306

Text #6

Max Range: 1.8182. Min Range: -1.8182

Max Act: 1.5761. Min Act: -0.0001

Data Index: 1060932 (C4 (Web Text))

Max Activating Token Index: 110

Text #7

Max Range: 1.8182. Min Range: -1.8182

Max Act: 1.3581. Min Act: -0.0001

Data Index: 474841 (C4 (Web Text))

Max Activating Token Index: 191

Text #8

Max Range: 1.8182. Min Range: -1.8182

Max Act: 1.3424. Min Act: -0.0001

Data Index: 89027 (C4 (Web Text))

Max Activating Token Index: 95

Text #9

Max Range: 1.8182. Min Range: -1.8182

Max Act: 1.4026. Min Act: -0.0001

Data Index: 457391 (C4 (Web Text))

Max Activating Token Index: 273

Text #10

Max Range: 1.8182. Min Range: -1.8182

Max Act: 1.4408. Min Act: -0.0001

Data Index: 1035922 (C4 (Web Text))

Max Activating Token Index: 7

Text #11

Max Range: 1.8182. Min Range: -1.8182

Max Act: 1.3546. Min Act: -0.0001

Data Index: 98939 (C4 (Web Text))

Max Activating Token Index: 458

Text #12

Max Range: 1.8182. Min Range: -1.8182

Max Act: 1.3981. Min Act: -0.0001

Data Index: 472332 (C4 (Web Text))

Max Activating Token Index: 166

Text #13

Max Range: 1.8182. Min Range: -1.8182

Max Act: 1.3258. Min Act: -0.0001

Data Index: 1334999 (C4 (Web Text))

Max Activating Token Index: 835

Text #14

Max Range: 1.8182. Min Range: -1.8182

Max Act: 1.3315. Min Act: -0.0001

Data Index: 1637 (C4 (Web Text))

Max Activating Token Index: 840

Text #15

Max Range: 1.8182. Min Range: -1.8182

Max Act: 1.0785. Min Act: -0.0001

Data Index: 78925 (C4 (Web Text))

Max Activating Token Index: 371

Text #16

Max Range: 1.8182. Min Range: -1.8182

Max Act: 1.1649. Min Act: -0.0001

Data Index: 684489 (C4 (Web Text))

Max Activating Token Index: 62

Text #17

Max Range: 1.8182. Min Range: -1.8182

Max Act: 1.2014. Min Act: -0.0001

Data Index: 666706 (C4 (Web Text))

Max Activating Token Index: 627

Text #18

Max Range: 1.8182. Min Range: -1.8182

Max Act: 1.1855. Min Act: -0.0001

Data Index: 999698 (C4 (Web Text))

Max Activating Token Index: 395

Text #19

Max Range: 1.8182. Min Range: -1.8182

Max Act: 1.2147. Min Act: -0.0001

Data Index: 710370 (C4 (Web Text))

Max Activating Token Index: 278