Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1262 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 0.8621. Min Range: -0.8621

Max Act: 0.8204. Min Act: -0.0000

Data Index: 310635 (C4 (Web Text))

Max Activating Token Index: 798

Text #1

Max Range: 0.8621. Min Range: -0.8621

Max Act: 0.8357. Min Act: -0.0000

Data Index: 984196 (C4 (Web Text))

Max Activating Token Index: 936

Text #2

Max Range: 0.8621. Min Range: -0.8621

Max Act: 0.8621. Min Act: -0.0000

Data Index: 477285 (C4 (Web Text))

Max Activating Token Index: 901

Text #3

Max Range: 0.8621. Min Range: -0.8621

Max Act: 0.8210. Min Act: -0.0000

Data Index: 715633 (C4 (Web Text))

Max Activating Token Index: 519

Text #4

Max Range: 0.8621. Min Range: -0.8621

Max Act: 0.7912. Min Act: -0.0001

Data Index: 115757 (C4 (Web Text))

Max Activating Token Index: 352

Text #5

Max Range: 0.8621. Min Range: -0.8621

Max Act: 0.7661. Min Act: -0.0000

Data Index: 1187615 (C4 (Web Text))

Max Activating Token Index: 25

Text #6

Max Range: 0.8621. Min Range: -0.8621

Max Act: 0.7866. Min Act: -0.0000

Data Index: 436117 (C4 (Web Text))

Max Activating Token Index: 552

Text #7

Max Range: 0.8621. Min Range: -0.8621

Max Act: 0.7802. Min Act: -0.0000

Data Index: 330479 (C4 (Web Text))

Max Activating Token Index: 29

Text #8

Max Range: 0.8621. Min Range: -0.8621

Max Act: 0.7700. Min Act: -0.0001

Data Index: 19210 (C4 (Web Text))

Max Activating Token Index: 981

Text #9

Max Range: 0.8621. Min Range: -0.8621

Max Act: 0.7652. Min Act: -0.0001

Data Index: 18 (C4 (Web Text))

Max Activating Token Index: 240

Text #10

Max Range: 0.8621. Min Range: -0.8621

Max Act: 0.7790. Min Act: -0.0001

Data Index: 281627 (C4 (Web Text))

Max Activating Token Index: 829

Text #11

Max Range: 0.8621. Min Range: -0.8621

Max Act: 0.7812. Min Act: -0.0000

Data Index: 690031 (C4 (Web Text))

Max Activating Token Index: 128

Text #12

Max Range: 0.8621. Min Range: -0.8621

Max Act: 0.7570. Min Act: -0.0000

Data Index: 64075 (C4 (Web Text))

Max Activating Token Index: 552

Text #13

Max Range: 0.8621. Min Range: -0.8621

Max Act: 0.7482. Min Act: -0.0000

Data Index: 1017963 (C4 (Web Text))

Max Activating Token Index: 627

Text #14

Max Range: 0.8621. Min Range: -0.8621

Max Act: 0.7237. Min Act: -0.0001

Data Index: 296058 (C4 (Web Text))

Max Activating Token Index: 512

Text #15

Max Range: 0.8621. Min Range: -0.8621

Max Act: 0.7492. Min Act: -0.0001

Data Index: 153944 (C4 (Web Text))

Max Activating Token Index: 60

Text #16

Max Range: 0.8621. Min Range: -0.8621

Max Act: 0.7977. Min Act: -0.0001

Data Index: 848145 (C4 (Web Text))

Max Activating Token Index: 581

Text #17

Max Range: 0.8621. Min Range: -0.8621

Max Act: 0.7495. Min Act: -0.0000

Data Index: 1282815 (C4 (Web Text))

Max Activating Token Index: 325

Text #18

Max Range: 0.8621. Min Range: -0.8621

Max Act: 0.7187. Min Act: -0.0001

Data Index: 419629 (C4 (Web Text))

Max Activating Token Index: 716

Text #19

Max Range: 0.8621. Min Range: -0.8621

Max Act: 0.7397. Min Act: -0.0001

Data Index: 1041969 (C4 (Web Text))

Max Activating Token Index: 667