Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4049 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.7717. Min Range: -2.7717

Max Act: 2.6936. Min Act: -0.0000

Data Index: 474735 (C4 (Web Text))

Max Activating Token Index: 55

Text #1

Max Range: 2.7717. Min Range: -2.7717

Max Act: 2.7717. Min Act: -0.0000

Data Index: 627113 (C4 (Web Text))

Max Activating Token Index: 50

Text #2

Max Range: 2.7717. Min Range: -2.7717

Max Act: 2.6711. Min Act: -0.0000

Data Index: 1164472 (C4 (Web Text))

Max Activating Token Index: 44

Text #3

Max Range: 2.7717. Min Range: -2.7717

Max Act: 2.4948. Min Act: -0.0000

Data Index: 534736 (C4 (Web Text))

Max Activating Token Index: 150

Text #4

Max Range: 2.7717. Min Range: -2.7717

Max Act: 2.6313. Min Act: -0.0000

Data Index: 839134 (C4 (Web Text))

Max Activating Token Index: 180

Text #5

Max Range: 2.7717. Min Range: -2.7717

Max Act: 2.4184. Min Act: -0.0000

Data Index: 172630 (C4 (Web Text))

Max Activating Token Index: 82

Text #6

Max Range: 2.7717. Min Range: -2.7717

Max Act: 2.2445. Min Act: -0.0000

Data Index: 1061955 (C4 (Web Text))

Max Activating Token Index: 518

Text #7

Max Range: 2.7717. Min Range: -2.7717

Max Act: 2.5000. Min Act: -0.0000

Data Index: 986613 (C4 (Web Text))

Max Activating Token Index: 213

Text #8

Max Range: 2.7717. Min Range: -2.7717

Max Act: 2.2302. Min Act: -0.0000

Data Index: 649273 (C4 (Web Text))

Max Activating Token Index: 378

Text #9

Max Range: 2.7717. Min Range: -2.7717

Max Act: 2.6069. Min Act: -0.0000

Data Index: 637018 (C4 (Web Text))

Max Activating Token Index: 449

Text #10

Max Range: 2.7717. Min Range: -2.7717

Max Act: 2.2034. Min Act: -0.0000

Data Index: 1081265 (C4 (Web Text))

Max Activating Token Index: 631

Text #11

Max Range: 2.7717. Min Range: -2.7717

Max Act: 2.2340. Min Act: -0.0000

Data Index: 101951 (C4 (Web Text))

Max Activating Token Index: 558

Text #12

Max Range: 2.7717. Min Range: -2.7717

Max Act: 2.2380. Min Act: 0.0000

Data Index: 542409 (C4 (Web Text))

Max Activating Token Index: 111

Text #13

Max Range: 2.7717. Min Range: -2.7717

Max Act: 2.3541. Min Act: -0.0000

Data Index: 860466 (C4 (Web Text))

Max Activating Token Index: 408

Text #14

Max Range: 2.7717. Min Range: -2.7717

Max Act: 2.4841. Min Act: -0.0000

Data Index: 335719 (C4 (Web Text))

Max Activating Token Index: 300

Text #15

Max Range: 2.7717. Min Range: -2.7717

Max Act: 2.3367. Min Act: -0.0000

Data Index: 1041063 (C4 (Web Text))

Max Activating Token Index: 571

Text #16

Max Range: 2.7717. Min Range: -2.7717

Max Act: 2.0380. Min Act: -0.0000

Data Index: 722364 (C4 (Web Text))

Max Activating Token Index: 765

Text #17

Max Range: 2.7717. Min Range: -2.7717

Max Act: 2.0247. Min Act: -0.0000

Data Index: 1199719 (C4 (Web Text))

Max Activating Token Index: 928

Text #18

Max Range: 2.7717. Min Range: -2.7717

Max Act: 2.3600. Min Act: -0.0000

Data Index: 152264 (C4 (Web Text))

Max Activating Token Index: 429

Text #19

Max Range: 2.7717. Min Range: -2.7717

Max Act: 2.1938. Min Act: -0.0000

Data Index: 806200 (C4 (Web Text))

Max Activating Token Index: 601