Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 375 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.0186. Min Range: -2.0186

Max Act: 2.0186. Min Act: -0.0000

Data Index: 693671 (C4 (Web Text))

Max Activating Token Index: 622

Text #1

Max Range: 2.0186. Min Range: -2.0186

Max Act: 1.8767. Min Act: -0.0000

Data Index: 1241307 (C4 (Web Text))

Max Activating Token Index: 142

Text #2

Max Range: 2.0186. Min Range: -2.0186

Max Act: 1.9130. Min Act: -0.0000

Data Index: 530614 (C4 (Web Text))

Max Activating Token Index: 442

Text #3

Max Range: 2.0186. Min Range: -2.0186

Max Act: 1.8859. Min Act: -0.0000

Data Index: 364783 (C4 (Web Text))

Max Activating Token Index: 858

Text #4

Max Range: 2.0186. Min Range: -2.0186

Max Act: 1.6952. Min Act: -0.0000

Data Index: 408704 (C4 (Web Text))

Max Activating Token Index: 247

Text #5

Max Range: 2.0186. Min Range: -2.0186

Max Act: 1.7432. Min Act: -0.0000

Data Index: 1332206 (C4 (Web Text))

Max Activating Token Index: 150

Text #6

Max Range: 2.0186. Min Range: -2.0186

Max Act: 1.7261. Min Act: -0.0000

Data Index: 146795 (C4 (Web Text))

Max Activating Token Index: 617

Text #7

Max Range: 2.0186. Min Range: -2.0186

Max Act: 1.7303. Min Act: -0.0000

Data Index: 258228 (C4 (Web Text))

Max Activating Token Index: 628

Text #8

Max Range: 2.0186. Min Range: -2.0186

Max Act: 1.6283. Min Act: -0.0000

Data Index: 180045 (C4 (Web Text))

Max Activating Token Index: 583

Text #9

Max Range: 2.0186. Min Range: -2.0186

Max Act: 1.6870. Min Act: -0.0000

Data Index: 518467 (C4 (Web Text))

Max Activating Token Index: 883

Text #10

Max Range: 2.0186. Min Range: -2.0186

Max Act: 1.5898. Min Act: -0.0000

Data Index: 1243320 (C4 (Web Text))

Max Activating Token Index: 440

Text #11

Max Range: 2.0186. Min Range: -2.0186

Max Act: 1.5640. Min Act: -0.0000

Data Index: 438564 (C4 (Web Text))

Max Activating Token Index: 265

Text #12

Max Range: 2.0186. Min Range: -2.0186

Max Act: 1.5340. Min Act: -0.0000

Data Index: 648821 (C4 (Web Text))

Max Activating Token Index: 16

Text #13

Max Range: 2.0186. Min Range: -2.0186

Max Act: 1.6167. Min Act: -0.0000

Data Index: 561816 (C4 (Web Text))

Max Activating Token Index: 790

Text #14

Max Range: 2.0186. Min Range: -2.0186

Max Act: 1.5197. Min Act: -0.0000

Data Index: 427662 (C4 (Web Text))

Max Activating Token Index: 126

Text #15

Max Range: 2.0186. Min Range: -2.0186

Max Act: 1.5921. Min Act: -0.0000

Data Index: 659591 (C4 (Web Text))

Max Activating Token Index: 112

Text #16

Max Range: 2.0186. Min Range: -2.0186

Max Act: 1.4791. Min Act: -0.0000

Data Index: 1194531 (C4 (Web Text))

Max Activating Token Index: 565

Text #17

Max Range: 2.0186. Min Range: -2.0186

Max Act: 1.5501. Min Act: -0.0000

Data Index: 574731 (C4 (Web Text))

Max Activating Token Index: 855

Text #18

Max Range: 2.0186. Min Range: -2.0186

Max Act: 1.4531. Min Act: -0.0000

Data Index: 887598 (C4 (Web Text))

Max Activating Token Index: 558

Text #19

Max Range: 2.0186. Min Range: -2.0186

Max Act: 1.3648. Min Act: -0.0000

Data Index: 376639 (C4 (Web Text))

Max Activating Token Index: 928