Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2298 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.0123. Min Range: -1.0123

Max Act: 1.0123. Min Act: -0.0000

Data Index: 272945 (C4 (Web Text))

Max Activating Token Index: 576

Text #1

Max Range: 1.0123. Min Range: -1.0123

Max Act: 0.9580. Min Act: -0.0000

Data Index: 317242 (C4 (Web Text))

Max Activating Token Index: 658

Text #2

Max Range: 1.0123. Min Range: -1.0123

Max Act: 0.8316. Min Act: 0.0000

Data Index: 554926 (C4 (Web Text))

Max Activating Token Index: 422

Text #3

Max Range: 1.0123. Min Range: -1.0123

Max Act: 0.9173. Min Act: -0.0000

Data Index: 310481 (C4 (Web Text))

Max Activating Token Index: 646

Text #4

Max Range: 1.0123. Min Range: -1.0123

Max Act: 0.8377. Min Act: -0.0000

Data Index: 167388 (C4 (Web Text))

Max Activating Token Index: 321

Text #5

Max Range: 1.0123. Min Range: -1.0123

Max Act: 0.8867. Min Act: -0.0000

Data Index: 1207060 (C4 (Web Text))

Max Activating Token Index: 758

Text #6

Max Range: 1.0123. Min Range: -1.0123

Max Act: 0.8785. Min Act: -0.0000

Data Index: 1153894 (C4 (Web Text))

Max Activating Token Index: 747

Text #7

Max Range: 1.0123. Min Range: -1.0123

Max Act: 0.8193. Min Act: -0.0000

Data Index: 607962 (C4 (Web Text))

Max Activating Token Index: 543

Text #8

Max Range: 1.0123. Min Range: -1.0123

Max Act: 0.7179. Min Act: -0.0000

Data Index: 1240519 (C4 (Web Text))

Max Activating Token Index: 792

Text #9

Max Range: 1.0123. Min Range: -1.0123

Max Act: 0.6949. Min Act: -0.0000

Data Index: 626815 (C4 (Web Text))

Max Activating Token Index: 786

Text #10

Max Range: 1.0123. Min Range: -1.0123

Max Act: 0.7414. Min Act: -0.0000

Data Index: 131415 (C4 (Web Text))

Max Activating Token Index: 979

Text #11

Max Range: 1.0123. Min Range: -1.0123

Max Act: 0.7128. Min Act: -0.0000

Data Index: 119697 (C4 (Web Text))

Max Activating Token Index: 275

Text #12

Max Range: 1.0123. Min Range: -1.0123

Max Act: 0.7445. Min Act: -0.0000

Data Index: 323970 (C4 (Web Text))

Max Activating Token Index: 557

Text #13

Max Range: 1.0123. Min Range: -1.0123

Max Act: 0.7204. Min Act: -0.0000

Data Index: 1166583 (C4 (Web Text))

Max Activating Token Index: 744

Text #14

Max Range: 1.0123. Min Range: -1.0123

Max Act: 0.7216. Min Act: -0.0000

Data Index: 975278 (C4 (Web Text))

Max Activating Token Index: 550

Text #15

Max Range: 1.0123. Min Range: -1.0123

Max Act: 0.7533. Min Act: -0.0000

Data Index: 143786 (C4 (Web Text))

Max Activating Token Index: 473

Text #16

Max Range: 1.0123. Min Range: -1.0123

Max Act: 0.7118. Min Act: -0.0000

Data Index: 646777 (C4 (Web Text))

Max Activating Token Index: 186

Text #17

Max Range: 1.0123. Min Range: -1.0123

Max Act: 0.6927. Min Act: -0.0000

Data Index: 968291 (C4 (Web Text))

Max Activating Token Index: 404

Text #18

Max Range: 1.0123. Min Range: -1.0123

Max Act: 0.7088. Min Act: -0.0000

Data Index: 146768 (C4 (Web Text))

Max Activating Token Index: 682

Text #19

Max Range: 1.0123. Min Range: -1.0123

Max Act: 0.6870. Min Act: -0.0000

Data Index: 1353331 (C4 (Web Text))

Max Activating Token Index: 383