Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3473 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.7011. Min Range: -0.7011

Max Act: 0.7011. Min Act: -0.0000

Data Index: 971982 (C4 (Web Text))

Max Activating Token Index: 393

Text #1

Max Range: 0.7011. Min Range: -0.7011

Max Act: 0.6660. Min Act: -0.0000

Data Index: 6534 (C4 (Web Text))

Max Activating Token Index: 324

Text #2

Max Range: 0.7011. Min Range: -0.7011

Max Act: 0.6969. Min Act: -0.0000

Data Index: 221808 (C4 (Web Text))

Max Activating Token Index: 73

Text #3

Max Range: 0.7011. Min Range: -0.7011

Max Act: 0.6196. Min Act: -0.0000

Data Index: 795888 (C4 (Web Text))

Max Activating Token Index: 41

Text #4

Max Range: 0.7011. Min Range: -0.7011

Max Act: 0.6155. Min Act: -0.0000

Data Index: 867608 (C4 (Web Text))

Max Activating Token Index: 586

Text #5

Max Range: 0.7011. Min Range: -0.7011

Max Act: 0.6209. Min Act: -0.0000

Data Index: 162900 (C4 (Web Text))

Max Activating Token Index: 443

Text #6

Max Range: 0.7011. Min Range: -0.7011

Max Act: 0.6434. Min Act: -0.0000

Data Index: 360220 (C4 (Web Text))

Max Activating Token Index: 757

Text #7

Max Range: 0.7011. Min Range: -0.7011

Max Act: 0.5505. Min Act: -0.0000

Data Index: 102075 (C4 (Web Text))

Max Activating Token Index: 663

Text #8

Max Range: 0.7011. Min Range: -0.7011

Max Act: 0.5922. Min Act: -0.0000

Data Index: 143171 (C4 (Web Text))

Max Activating Token Index: 42

Text #9

Max Range: 0.7011. Min Range: -0.7011

Max Act: 0.5511. Min Act: -0.0000

Data Index: 1327799 (C4 (Web Text))

Max Activating Token Index: 476

Text #10

Max Range: 0.7011. Min Range: -0.7011

Max Act: 0.5245. Min Act: -0.0000

Data Index: 1140639 (C4 (Web Text))

Max Activating Token Index: 1019

Text #11

Max Range: 0.7011. Min Range: -0.7011

Max Act: 0.5370. Min Act: -0.0000

Data Index: 606145 (C4 (Web Text))

Max Activating Token Index: 296

Text #12

Max Range: 0.7011. Min Range: -0.7011

Max Act: 0.5373. Min Act: -0.0000

Data Index: 1351106 (C4 (Web Text))

Max Activating Token Index: 228

Text #13

Max Range: 0.7011. Min Range: -0.7011

Max Act: 0.5409. Min Act: -0.0000

Data Index: 1208058 (C4 (Web Text))

Max Activating Token Index: 285

Text #14

Max Range: 0.7011. Min Range: -0.7011

Max Act: 0.5394. Min Act: -0.0000

Data Index: 60135 (C4 (Web Text))

Max Activating Token Index: 908

Text #15

Max Range: 0.7011. Min Range: -0.7011

Max Act: 0.5525. Min Act: -0.0000

Data Index: 452927 (C4 (Web Text))

Max Activating Token Index: 443

Text #16

Max Range: 0.7011. Min Range: -0.7011

Max Act: 0.4494. Min Act: -0.0000

Data Index: 613494 (C4 (Web Text))

Max Activating Token Index: 810

Text #17

Max Range: 0.7011. Min Range: -0.7011

Max Act: 0.4920. Min Act: -0.0000

Data Index: 739171 (C4 (Web Text))

Max Activating Token Index: 428

Text #18

Max Range: 0.7011. Min Range: -0.7011

Max Act: 0.5196. Min Act: -0.0000

Data Index: 143208 (C4 (Web Text))

Max Activating Token Index: 935

Text #19

Max Range: 0.7011. Min Range: -0.7011

Max Act: 0.5309. Min Act: -0.0000

Data Index: 383563 (C4 (Web Text))

Max Activating Token Index: 142