Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 21 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 4.4232. Min Range: -4.4232

Max Act: 4.4232. Min Act: -0.0000

Data Index: 958920 (C4 (Web Text))

Max Activating Token Index: 785

Text #1

Max Range: 4.4232. Min Range: -4.4232

Max Act: 4.2058. Min Act: -0.0000

Data Index: 1302531 (C4 (Web Text))

Max Activating Token Index: 606

Text #2

Max Range: 4.4232. Min Range: -4.4232

Max Act: 4.0387. Min Act: -0.0000

Data Index: 1239202 (C4 (Web Text))

Max Activating Token Index: 423

Text #3

Max Range: 4.4232. Min Range: -4.4232

Max Act: 3.9977. Min Act: -0.0000

Data Index: 177639 (C4 (Web Text))

Max Activating Token Index: 216

Text #4

Max Range: 4.4232. Min Range: -4.4232

Max Act: 3.9307. Min Act: -0.0000

Data Index: 837111 (C4 (Web Text))

Max Activating Token Index: 895

Text #5

Max Range: 4.4232. Min Range: -4.4232

Max Act: 3.7858. Min Act: -0.0000

Data Index: 998835 (C4 (Web Text))

Max Activating Token Index: 243

Text #6

Max Range: 4.4232. Min Range: -4.4232

Max Act: 3.7457. Min Act: -0.0000

Data Index: 268911 (C4 (Web Text))

Max Activating Token Index: 195

Text #7

Max Range: 4.4232. Min Range: -4.4232

Max Act: 3.7901. Min Act: -0.0000

Data Index: 478703 (C4 (Web Text))

Max Activating Token Index: 1019

Text #8

Max Range: 4.4232. Min Range: -4.4232

Max Act: 3.8125. Min Act: -0.0000

Data Index: 959740 (C4 (Web Text))

Max Activating Token Index: 646

Text #9

Max Range: 4.4232. Min Range: -4.4232

Max Act: 3.8876. Min Act: -0.0000

Data Index: 675982 (C4 (Web Text))

Max Activating Token Index: 809

Text #10

Max Range: 4.4232. Min Range: -4.4232

Max Act: 3.7030. Min Act: -0.0000

Data Index: 1169123 (C4 (Web Text))

Max Activating Token Index: 663

Text #11

Max Range: 4.4232. Min Range: -4.4232

Max Act: 3.6433. Min Act: -0.0000

Data Index: 632894 (C4 (Web Text))

Max Activating Token Index: 681

Text #12

Max Range: 4.4232. Min Range: -4.4232

Max Act: 3.4206. Min Act: -0.0000

Data Index: 472195 (C4 (Web Text))

Max Activating Token Index: 668

Text #13

Max Range: 4.4232. Min Range: -4.4232

Max Act: 3.5795. Min Act: -0.0000

Data Index: 1069999 (C4 (Web Text))

Max Activating Token Index: 909

Text #14

Max Range: 4.4232. Min Range: -4.4232

Max Act: 3.5548. Min Act: -0.0000

Data Index: 689419 (C4 (Web Text))

Max Activating Token Index: 981

Text #15

Max Range: 4.4232. Min Range: -4.4232

Max Act: 3.5232. Min Act: -0.0000

Data Index: 686819 (C4 (Web Text))

Max Activating Token Index: 621

Text #16

Max Range: 4.4232. Min Range: -4.4232

Max Act: 3.5421. Min Act: -0.0000

Data Index: 689899 (C4 (Web Text))

Max Activating Token Index: 347

Text #17

Max Range: 4.4232. Min Range: -4.4232

Max Act: 3.6828. Min Act: -0.0000

Data Index: 755856 (C4 (Web Text))

Max Activating Token Index: 879

Text #18

Max Range: 4.4232. Min Range: -4.4232

Max Act: 3.4709. Min Act: -0.0000

Data Index: 410336 (C4 (Web Text))

Max Activating Token Index: 79

Text #19

Max Range: 4.4232. Min Range: -4.4232

Max Act: 3.2610. Min Act: -0.0000

Data Index: 516808 (C4 (Web Text))

Max Activating Token Index: 642