Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3000 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.3437. Min Range: -0.3437

Max Act: 0.3437. Min Act: -0.0001

Data Index: 828973 (C4 (Web Text))

Max Activating Token Index: 37

Text #1

Max Range: 0.3437. Min Range: -0.3437

Max Act: 0.2545. Min Act: -0.0001

Data Index: 815479 (C4 (Web Text))

Max Activating Token Index: 83

Text #2

Max Range: 0.3437. Min Range: -0.3437

Max Act: 0.2465. Min Act: -0.0001

Data Index: 1575336 (Python Code)

Max Activating Token Index: 192

Text #3

Max Range: 0.3437. Min Range: -0.3437

Max Act: 0.2257. Min Act: -0.0001

Data Index: 448642 (C4 (Web Text))

Max Activating Token Index: 253

Text #4

Max Range: 0.3437. Min Range: -0.3437

Max Act: 0.2580. Min Act: -0.0001

Data Index: 1585261 (Python Code)

Max Activating Token Index: 270

Text #5

Max Range: 0.3437. Min Range: -0.3437

Max Act: 0.2533. Min Act: -0.0001

Data Index: 461056 (C4 (Web Text))

Max Activating Token Index: 76

Text #6

Max Range: 0.3437. Min Range: -0.3437

Max Act: 0.2307. Min Act: -0.0001

Data Index: 102594 (C4 (Web Text))

Max Activating Token Index: 393

Text #7

Max Range: 0.3437. Min Range: -0.3437

Max Act: 0.2380. Min Act: -0.0001

Data Index: 64392 (C4 (Web Text))

Max Activating Token Index: 157

Text #8

Max Range: 0.3437. Min Range: -0.3437

Max Act: 0.2334. Min Act: -0.0001

Data Index: 1302008 (C4 (Web Text))

Max Activating Token Index: 912

Text #9

Max Range: 0.3437. Min Range: -0.3437

Max Act: 0.2251. Min Act: -0.0001

Data Index: 839438 (C4 (Web Text))

Max Activating Token Index: 663

Text #10

Max Range: 0.3437. Min Range: -0.3437

Max Act: 0.2151. Min Act: -0.0001

Data Index: 742235 (C4 (Web Text))

Max Activating Token Index: 363

Text #11

Max Range: 0.3437. Min Range: -0.3437

Max Act: 0.2134. Min Act: -0.0001

Data Index: 1030190 (C4 (Web Text))

Max Activating Token Index: 760

Text #12

Max Range: 0.3437. Min Range: -0.3437

Max Act: 0.2050. Min Act: -0.0001

Data Index: 331175 (C4 (Web Text))

Max Activating Token Index: 560

Text #13

Max Range: 0.3437. Min Range: -0.3437

Max Act: 0.2122. Min Act: -0.0001

Data Index: 45343 (C4 (Web Text))

Max Activating Token Index: 598

Text #14

Max Range: 0.3437. Min Range: -0.3437

Max Act: 0.2030. Min Act: -0.0001

Data Index: 1368649 (Python Code)

Max Activating Token Index: 565

Text #15

Max Range: 0.3437. Min Range: -0.3437

Max Act: 0.1824. Min Act: -0.0001

Data Index: 405945 (C4 (Web Text))

Max Activating Token Index: 136

Text #16

Max Range: 0.3437. Min Range: -0.3437

Max Act: 0.2262. Min Act: -0.0001

Data Index: 1219764 (C4 (Web Text))

Max Activating Token Index: 570

Text #17

Max Range: 0.3437. Min Range: -0.3437

Max Act: 0.2167. Min Act: -0.0001

Data Index: 1085759 (C4 (Web Text))

Max Activating Token Index: 911

Text #18

Max Range: 0.3437. Min Range: -0.3437

Max Act: 0.2076. Min Act: -0.0001

Data Index: 123181 (C4 (Web Text))

Max Activating Token Index: 229

Text #19

Max Range: 0.3437. Min Range: -0.3437

Max Act: 0.2111. Min Act: -0.0001

Data Index: 79053 (C4 (Web Text))

Max Activating Token Index: 183