Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 6125 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.4713. Min Range: -1.4713

Max Act: 1.4713. Min Act: -0.0000

Data Index: 166678 (C4 (Web Text))

Max Activating Token Index: 404

Text #1

Max Range: 1.4713. Min Range: -1.4713

Max Act: 1.4398. Min Act: -0.0000

Data Index: 367308 (C4 (Web Text))

Max Activating Token Index: 948

Text #2

Max Range: 1.4713. Min Range: -1.4713

Max Act: 1.2644. Min Act: -0.0000

Data Index: 232384 (C4 (Web Text))

Max Activating Token Index: 296

Text #3

Max Range: 1.4713. Min Range: -1.4713

Max Act: 1.1479. Min Act: -0.0000

Data Index: 473543 (C4 (Web Text))

Max Activating Token Index: 232

Text #4

Max Range: 1.4713. Min Range: -1.4713

Max Act: 1.1134. Min Act: -0.0000

Data Index: 652952 (C4 (Web Text))

Max Activating Token Index: 900

Text #5

Max Range: 1.4713. Min Range: -1.4713

Max Act: 1.1724. Min Act: -0.0000

Data Index: 1140769 (C4 (Web Text))

Max Activating Token Index: 107

Text #6

Max Range: 1.4713. Min Range: -1.4713

Max Act: 0.9919. Min Act: -0.0000

Data Index: 166400 (C4 (Web Text))

Max Activating Token Index: 843

Text #7

Max Range: 1.4713. Min Range: -1.4713

Max Act: 1.0468. Min Act: -0.0000

Data Index: 456461 (C4 (Web Text))

Max Activating Token Index: 165

Text #8

Max Range: 1.4713. Min Range: -1.4713

Max Act: 1.0096. Min Act: -0.0000

Data Index: 645118 (C4 (Web Text))

Max Activating Token Index: 135

Text #9

Max Range: 1.4713. Min Range: -1.4713

Max Act: 1.0325. Min Act: -0.0000

Data Index: 185959 (C4 (Web Text))

Max Activating Token Index: 824

Text #10

Max Range: 1.4713. Min Range: -1.4713

Max Act: 0.8715. Min Act: -0.0000

Data Index: 1135306 (C4 (Web Text))

Max Activating Token Index: 325

Text #11

Max Range: 1.4713. Min Range: -1.4713

Max Act: 0.8635. Min Act: -0.0000

Data Index: 976869 (C4 (Web Text))

Max Activating Token Index: 668

Text #12

Max Range: 1.4713. Min Range: -1.4713

Max Act: 0.9108. Min Act: -0.0000

Data Index: 1156307 (C4 (Web Text))

Max Activating Token Index: 135

Text #13

Max Range: 1.4713. Min Range: -1.4713

Max Act: 0.8959. Min Act: -0.0000

Data Index: 1141725 (C4 (Web Text))

Max Activating Token Index: 737

Text #14

Max Range: 1.4713. Min Range: -1.4713

Max Act: 0.8850. Min Act: -0.0000

Data Index: 1310105 (C4 (Web Text))

Max Activating Token Index: 552

Text #15

Max Range: 1.4713. Min Range: -1.4713

Max Act: 0.8826. Min Act: -0.0000

Data Index: 895159 (C4 (Web Text))

Max Activating Token Index: 549

Text #16

Max Range: 1.4713. Min Range: -1.4713

Max Act: 0.8787. Min Act: -0.0000

Data Index: 812669 (C4 (Web Text))

Max Activating Token Index: 957

Text #17

Max Range: 1.4713. Min Range: -1.4713

Max Act: 0.8655. Min Act: -0.0000

Data Index: 274474 (C4 (Web Text))

Max Activating Token Index: 291

Text #18

Max Range: 1.4713. Min Range: -1.4713

Max Act: 0.9120. Min Act: -0.0000

Data Index: 247440 (C4 (Web Text))

Max Activating Token Index: 241

Text #19

Max Range: 1.4713. Min Range: -1.4713

Max Act: 0.8330. Min Act: -0.0000

Data Index: 1228811 (C4 (Web Text))

Max Activating Token Index: 515