Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2464 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.5292. Min Range: -2.5292

Max Act: 2.5292. Min Act: -0.0000

Data Index: 1210446 (C4 (Web Text))

Max Activating Token Index: 290

Text #1

Max Range: 2.5292. Min Range: -2.5292

Max Act: 2.2474. Min Act: -0.0000

Data Index: 1319701 (C4 (Web Text))

Max Activating Token Index: 119

Text #2

Max Range: 2.5292. Min Range: -2.5292

Max Act: 2.4629. Min Act: -0.0000

Data Index: 1171281 (C4 (Web Text))

Max Activating Token Index: 241

Text #3

Max Range: 2.5292. Min Range: -2.5292

Max Act: 2.2190. Min Act: -0.0000

Data Index: 158444 (C4 (Web Text))

Max Activating Token Index: 211

Text #4

Max Range: 2.5292. Min Range: -2.5292

Max Act: 2.4277. Min Act: -0.0000

Data Index: 163852 (C4 (Web Text))

Max Activating Token Index: 657

Text #5

Max Range: 2.5292. Min Range: -2.5292

Max Act: 2.4273. Min Act: -0.0000

Data Index: 328368 (C4 (Web Text))

Max Activating Token Index: 708

Text #6

Max Range: 2.5292. Min Range: -2.5292

Max Act: 2.1940. Min Act: -0.0000

Data Index: 581729 (C4 (Web Text))

Max Activating Token Index: 563

Text #7

Max Range: 2.5292. Min Range: -2.5292

Max Act: 2.1781. Min Act: -0.0000

Data Index: 657848 (C4 (Web Text))

Max Activating Token Index: 759

Text #8

Max Range: 2.5292. Min Range: -2.5292

Max Act: 2.2885. Min Act: -0.0000

Data Index: 444311 (C4 (Web Text))

Max Activating Token Index: 765

Text #9

Max Range: 2.5292. Min Range: -2.5292

Max Act: 2.0420. Min Act: -0.0000

Data Index: 171858 (C4 (Web Text))

Max Activating Token Index: 614

Text #10

Max Range: 2.5292. Min Range: -2.5292

Max Act: 2.2644. Min Act: -0.0000

Data Index: 721898 (C4 (Web Text))

Max Activating Token Index: 807

Text #11

Max Range: 2.5292. Min Range: -2.5292

Max Act: 2.2203. Min Act: -0.0000

Data Index: 234366 (C4 (Web Text))

Max Activating Token Index: 346

Text #12

Max Range: 2.5292. Min Range: -2.5292

Max Act: 2.2208. Min Act: -0.0000

Data Index: 878463 (C4 (Web Text))

Max Activating Token Index: 826

Text #13

Max Range: 2.5292. Min Range: -2.5292

Max Act: 2.2046. Min Act: -0.0000

Data Index: 631575 (C4 (Web Text))

Max Activating Token Index: 841

Text #14

Max Range: 2.5292. Min Range: -2.5292

Max Act: 2.1423. Min Act: -0.0000

Data Index: 557066 (C4 (Web Text))

Max Activating Token Index: 491

Text #15

Max Range: 2.5292. Min Range: -2.5292

Max Act: 2.0116. Min Act: -0.0000

Data Index: 783751 (C4 (Web Text))

Max Activating Token Index: 52

Text #16

Max Range: 2.5292. Min Range: -2.5292

Max Act: 2.0403. Min Act: -0.0000

Data Index: 574441 (C4 (Web Text))

Max Activating Token Index: 810

Text #17

Max Range: 2.5292. Min Range: -2.5292

Max Act: 1.9826. Min Act: -0.0000

Data Index: 635940 (C4 (Web Text))

Max Activating Token Index: 1012

Text #18

Max Range: 2.5292. Min Range: -2.5292

Max Act: 2.0490. Min Act: -0.0000

Data Index: 87731 (C4 (Web Text))

Max Activating Token Index: 797

Text #19

Max Range: 2.5292. Min Range: -2.5292

Max Act: 2.1829. Min Act: -0.0000

Data Index: 1262798 (C4 (Web Text))

Max Activating Token Index: 701