Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2754 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 3.6451. Min Range: -3.6451

Max Act: 3.6451. Min Act: -0.0000

Data Index: 1258754 (C4 (Web Text))

Max Activating Token Index: 3

Text #1

Max Range: 3.6451. Min Range: -3.6451

Max Act: 3.5631. Min Act: -0.0000

Data Index: 801176 (C4 (Web Text))

Max Activating Token Index: 2

Text #2

Max Range: 3.6451. Min Range: -3.6451

Max Act: 3.2454. Min Act: -0.0000

Data Index: 390838 (C4 (Web Text))

Max Activating Token Index: 859

Text #3

Max Range: 3.6451. Min Range: -3.6451

Max Act: 3.0134. Min Act: -0.0000

Data Index: 641773 (C4 (Web Text))

Max Activating Token Index: 213

Text #4

Max Range: 3.6451. Min Range: -3.6451

Max Act: 2.8669. Min Act: -0.0000

Data Index: 1277363 (C4 (Web Text))

Max Activating Token Index: 770

Text #5

Max Range: 3.6451. Min Range: -3.6451

Max Act: 2.8412. Min Act: -0.0000

Data Index: 71388 (C4 (Web Text))

Max Activating Token Index: 248

Text #6

Max Range: 3.6451. Min Range: -3.6451

Max Act: 2.9014. Min Act: -0.0000

Data Index: 52293 (C4 (Web Text))

Max Activating Token Index: 950

Text #7

Max Range: 3.6451. Min Range: -3.6451

Max Act: 2.8694. Min Act: -0.0000

Data Index: 316895 (C4 (Web Text))

Max Activating Token Index: 5

Text #8

Max Range: 3.6451. Min Range: -3.6451

Max Act: 2.7282. Min Act: -0.0000

Data Index: 629184 (C4 (Web Text))

Max Activating Token Index: 483

Text #9

Max Range: 3.6451. Min Range: -3.6451

Max Act: 2.7285. Min Act: -0.0000

Data Index: 801909 (C4 (Web Text))

Max Activating Token Index: 781

Text #10

Max Range: 3.6451. Min Range: -3.6451

Max Act: 2.7169. Min Act: -0.0000

Data Index: 132805 (C4 (Web Text))

Max Activating Token Index: 694

Text #11

Max Range: 3.6451. Min Range: -3.6451

Max Act: 2.7973. Min Act: -0.0000

Data Index: 956603 (C4 (Web Text))

Max Activating Token Index: 907

Text #12

Max Range: 3.6451. Min Range: -3.6451

Max Act: 2.6267. Min Act: -0.0000

Data Index: 1330941 (C4 (Web Text))

Max Activating Token Index: 658

Text #13

Max Range: 3.6451. Min Range: -3.6451

Max Act: 2.5595. Min Act: -0.0000

Data Index: 601715 (C4 (Web Text))

Max Activating Token Index: 938

Text #14

Max Range: 3.6451. Min Range: -3.6451

Max Act: 2.4513. Min Act: -0.0000

Data Index: 273889 (C4 (Web Text))

Max Activating Token Index: 531

Text #15

Max Range: 3.6451. Min Range: -3.6451

Max Act: 2.4360. Min Act: -0.0000

Data Index: 691213 (C4 (Web Text))

Max Activating Token Index: 380

Text #16

Max Range: 3.6451. Min Range: -3.6451

Max Act: 2.5587. Min Act: -0.0000

Data Index: 533314 (C4 (Web Text))

Max Activating Token Index: 699

Text #17

Max Range: 3.6451. Min Range: -3.6451

Max Act: 2.5716. Min Act: -0.0000

Data Index: 461529 (C4 (Web Text))

Max Activating Token Index: 891

Text #18

Max Range: 3.6451. Min Range: -3.6451

Max Act: 2.7260. Min Act: -0.0000

Data Index: 1318494 (C4 (Web Text))

Max Activating Token Index: 50

Text #19

Max Range: 3.6451. Min Range: -3.6451

Max Act: 2.5701. Min Act: -0.0000

Data Index: 322208 (C4 (Web Text))

Max Activating Token Index: 185