Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3061 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.0065. Min Range: -2.0065

Max Act: 2.0065. Min Act: -0.0000

Data Index: 335867 (C4 (Web Text))

Max Activating Token Index: 731

Text #1

Max Range: 2.0065. Min Range: -2.0065

Max Act: 1.5946. Min Act: -0.0000

Data Index: 353636 (C4 (Web Text))

Max Activating Token Index: 406

Text #2

Max Range: 2.0065. Min Range: -2.0065

Max Act: 1.5070. Min Act: -0.0000

Data Index: 1204740 (C4 (Web Text))

Max Activating Token Index: 166

Text #3

Max Range: 2.0065. Min Range: -2.0065

Max Act: 1.4081. Min Act: -0.0000

Data Index: 1009924 (C4 (Web Text))

Max Activating Token Index: 118

Text #4

Max Range: 2.0065. Min Range: -2.0065

Max Act: 1.4660. Min Act: -0.0000

Data Index: 129873 (C4 (Web Text))

Max Activating Token Index: 427

Text #5

Max Range: 2.0065. Min Range: -2.0065

Max Act: 1.4293. Min Act: -0.0000

Data Index: 952798 (C4 (Web Text))

Max Activating Token Index: 120

Text #6

Max Range: 2.0065. Min Range: -2.0065

Max Act: 1.3973. Min Act: -0.0000

Data Index: 775703 (C4 (Web Text))

Max Activating Token Index: 881

Text #7

Max Range: 2.0065. Min Range: -2.0065

Max Act: 1.3896. Min Act: -0.0000

Data Index: 1147301 (C4 (Web Text))

Max Activating Token Index: 411

Text #8

Max Range: 2.0065. Min Range: -2.0065

Max Act: 1.3657. Min Act: -0.0000

Data Index: 1312833 (C4 (Web Text))

Max Activating Token Index: 16

Text #9

Max Range: 2.0065. Min Range: -2.0065

Max Act: 1.4222. Min Act: -0.0000

Data Index: 257796 (C4 (Web Text))

Max Activating Token Index: 475

Text #10

Max Range: 2.0065. Min Range: -2.0065

Max Act: 1.3390. Min Act: -0.0000

Data Index: 349146 (C4 (Web Text))

Max Activating Token Index: 359

Text #11

Max Range: 2.0065. Min Range: -2.0065

Max Act: 1.3247. Min Act: -0.0000

Data Index: 1224184 (C4 (Web Text))

Max Activating Token Index: 419

Text #12

Max Range: 2.0065. Min Range: -2.0065

Max Act: 1.2312. Min Act: -0.0000

Data Index: 1301310 (C4 (Web Text))

Max Activating Token Index: 641

Text #13

Max Range: 2.0065. Min Range: -2.0065

Max Act: 1.3854. Min Act: -0.0000

Data Index: 214281 (C4 (Web Text))

Max Activating Token Index: 303

Text #14

Max Range: 2.0065. Min Range: -2.0065

Max Act: 1.4467. Min Act: -0.0000

Data Index: 757675 (C4 (Web Text))

Max Activating Token Index: 671

Text #15

Max Range: 2.0065. Min Range: -2.0065

Max Act: 1.0772. Min Act: -0.0000

Data Index: 884012 (C4 (Web Text))

Max Activating Token Index: 187

Text #16

Max Range: 2.0065. Min Range: -2.0065

Max Act: 1.2950. Min Act: -0.0000

Data Index: 16692 (C4 (Web Text))

Max Activating Token Index: 329

Text #17

Max Range: 2.0065. Min Range: -2.0065

Max Act: 1.2916. Min Act: -0.0000

Data Index: 743478 (C4 (Web Text))

Max Activating Token Index: 282

Text #18

Max Range: 2.0065. Min Range: -2.0065

Max Act: 1.2985. Min Act: -0.0000

Data Index: 253281 (C4 (Web Text))

Max Activating Token Index: 659

Text #19

Max Range: 2.0065. Min Range: -2.0065

Max Act: 1.1496. Min Act: -0.0000

Data Index: 171586 (C4 (Web Text))

Max Activating Token Index: 99