Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2632 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.9223. Min Range: -2.9223

Max Act: 2.6629. Min Act: -0.0000

Data Index: 393449 (C4 (Web Text))

Max Activating Token Index: 943

Text #1

Max Range: 2.9223. Min Range: -2.9223

Max Act: 2.9223. Min Act: -0.0000

Data Index: 456408 (C4 (Web Text))

Max Activating Token Index: 807

Text #2

Max Range: 2.9223. Min Range: -2.9223

Max Act: 2.4786. Min Act: -0.0000

Data Index: 312267 (C4 (Web Text))

Max Activating Token Index: 762

Text #3

Max Range: 2.9223. Min Range: -2.9223

Max Act: 2.4646. Min Act: -0.0001

Data Index: 84782 (C4 (Web Text))

Max Activating Token Index: 331

Text #4

Max Range: 2.9223. Min Range: -2.9223

Max Act: 2.3157. Min Act: -0.0000

Data Index: 73476 (C4 (Web Text))

Max Activating Token Index: 356

Text #5

Max Range: 2.9223. Min Range: -2.9223

Max Act: 2.3733. Min Act: -0.0000

Data Index: 372091 (C4 (Web Text))

Max Activating Token Index: 872

Text #6

Max Range: 2.9223. Min Range: -2.9223

Max Act: 2.1531. Min Act: -0.0000

Data Index: 779574 (C4 (Web Text))

Max Activating Token Index: 645

Text #7

Max Range: 2.9223. Min Range: -2.9223

Max Act: 2.2292. Min Act: -0.0001

Data Index: 913999 (C4 (Web Text))

Max Activating Token Index: 745

Text #8

Max Range: 2.9223. Min Range: -2.9223

Max Act: 2.2056. Min Act: -0.0000

Data Index: 821290 (C4 (Web Text))

Max Activating Token Index: 450

Text #9

Max Range: 2.9223. Min Range: -2.9223

Max Act: 2.3080. Min Act: -0.0001

Data Index: 701853 (C4 (Web Text))

Max Activating Token Index: 54

Text #10

Max Range: 2.9223. Min Range: -2.9223

Max Act: 2.0035. Min Act: -0.0000

Data Index: 62198 (C4 (Web Text))

Max Activating Token Index: 508

Text #11

Max Range: 2.9223. Min Range: -2.9223

Max Act: 2.1869. Min Act: -0.0000

Data Index: 1227381 (C4 (Web Text))

Max Activating Token Index: 620

Text #12

Max Range: 2.9223. Min Range: -2.9223

Max Act: 2.0709. Min Act: -0.0000

Data Index: 773735 (C4 (Web Text))

Max Activating Token Index: 46

Text #13

Max Range: 2.9223. Min Range: -2.9223

Max Act: 2.0265. Min Act: -0.0000

Data Index: 512997 (C4 (Web Text))

Max Activating Token Index: 146

Text #14

Max Range: 2.9223. Min Range: -2.9223

Max Act: 2.1242. Min Act: -0.0000

Data Index: 419209 (C4 (Web Text))

Max Activating Token Index: 577

Text #15

Max Range: 2.9223. Min Range: -2.9223

Max Act: 1.9966. Min Act: -0.0000

Data Index: 978032 (C4 (Web Text))

Max Activating Token Index: 20

Text #16

Max Range: 2.9223. Min Range: -2.9223

Max Act: 1.9461. Min Act: -0.0000

Data Index: 1075934 (C4 (Web Text))

Max Activating Token Index: 446

Text #17

Max Range: 2.9223. Min Range: -2.9223

Max Act: 1.9313. Min Act: -0.0000

Data Index: 1226799 (C4 (Web Text))

Max Activating Token Index: 68

Text #18

Max Range: 2.9223. Min Range: -2.9223

Max Act: 1.7897. Min Act: -0.0000

Data Index: 1225340 (C4 (Web Text))

Max Activating Token Index: 432

Text #19

Max Range: 2.9223. Min Range: -2.9223

Max Act: 1.9665. Min Act: -0.0000

Data Index: 254275 (C4 (Web Text))

Max Activating Token Index: 32