Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 239 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 4.6229. Min Range: -4.6229

Max Act: 4.5624. Min Act: -0.0000

Data Index: 636587 (C4 (Web Text))

Max Activating Token Index: 132

Text #1

Max Range: 4.6229. Min Range: -4.6229

Max Act: 4.6229. Min Act: -0.0000

Data Index: 790672 (C4 (Web Text))

Max Activating Token Index: 230

Text #2

Max Range: 4.6229. Min Range: -4.6229

Max Act: 4.2604. Min Act: -0.0000

Data Index: 1299645 (C4 (Web Text))

Max Activating Token Index: 215

Text #3

Max Range: 4.6229. Min Range: -4.6229

Max Act: 4.1912. Min Act: -0.0000

Data Index: 660186 (C4 (Web Text))

Max Activating Token Index: 802

Text #4

Max Range: 4.6229. Min Range: -4.6229

Max Act: 4.0531. Min Act: -0.0000

Data Index: 747255 (C4 (Web Text))

Max Activating Token Index: 192

Text #5

Max Range: 4.6229. Min Range: -4.6229

Max Act: 3.8513. Min Act: -0.0000

Data Index: 906778 (C4 (Web Text))

Max Activating Token Index: 391

Text #6

Max Range: 4.6229. Min Range: -4.6229

Max Act: 3.9017. Min Act: -0.0000

Data Index: 682937 (C4 (Web Text))

Max Activating Token Index: 440

Text #7

Max Range: 4.6229. Min Range: -4.6229

Max Act: 3.5910. Min Act: -0.0000

Data Index: 161797 (C4 (Web Text))

Max Activating Token Index: 239

Text #8

Max Range: 4.6229. Min Range: -4.6229

Max Act: 3.4015. Min Act: -0.0000

Data Index: 297314 (C4 (Web Text))

Max Activating Token Index: 384

Text #9

Max Range: 4.6229. Min Range: -4.6229

Max Act: 3.6440. Min Act: -0.0000

Data Index: 277688 (C4 (Web Text))

Max Activating Token Index: 401

Text #10

Max Range: 4.6229. Min Range: -4.6229

Max Act: 3.4805. Min Act: -0.0000

Data Index: 1292405 (C4 (Web Text))

Max Activating Token Index: 583

Text #11

Max Range: 4.6229. Min Range: -4.6229

Max Act: 3.4433. Min Act: -0.0000

Data Index: 69935 (C4 (Web Text))

Max Activating Token Index: 886

Text #12

Max Range: 4.6229. Min Range: -4.6229

Max Act: 3.3961. Min Act: -0.0000

Data Index: 987280 (C4 (Web Text))

Max Activating Token Index: 400

Text #13

Max Range: 4.6229. Min Range: -4.6229

Max Act: 3.2325. Min Act: -0.0000

Data Index: 217791 (C4 (Web Text))

Max Activating Token Index: 185

Text #14

Max Range: 4.6229. Min Range: -4.6229

Max Act: 3.1936. Min Act: -0.0000

Data Index: 1030907 (C4 (Web Text))

Max Activating Token Index: 328

Text #15

Max Range: 4.6229. Min Range: -4.6229

Max Act: 3.3411. Min Act: -0.0000

Data Index: 1325433 (C4 (Web Text))

Max Activating Token Index: 724

Text #16

Max Range: 4.6229. Min Range: -4.6229

Max Act: 2.9611. Min Act: -0.0000

Data Index: 721918 (C4 (Web Text))

Max Activating Token Index: 387

Text #17

Max Range: 4.6229. Min Range: -4.6229

Max Act: 3.4279. Min Act: -0.0000

Data Index: 752085 (C4 (Web Text))

Max Activating Token Index: 761

Text #18

Max Range: 4.6229. Min Range: -4.6229

Max Act: 2.8054. Min Act: -0.0000

Data Index: 22986 (C4 (Web Text))

Max Activating Token Index: 954

Text #19

Max Range: 4.6229. Min Range: -4.6229

Max Act: 3.0852. Min Act: -0.0000

Data Index: 913646 (C4 (Web Text))

Max Activating Token Index: 445