Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2607 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 0.4601. Min Range: -0.4601

Max Act: 0.4601. Min Act: -0.0001

Data Index: 131765 (C4 (Web Text))

Max Activating Token Index: 857

Text #1

Max Range: 0.4601. Min Range: -0.4601

Max Act: 0.4130. Min Act: -0.0001

Data Index: 1029128 (C4 (Web Text))

Max Activating Token Index: 387

Text #2

Max Range: 0.4601. Min Range: -0.4601

Max Act: 0.4214. Min Act: -0.0000

Data Index: 359198 (C4 (Web Text))

Max Activating Token Index: 578

Text #3

Max Range: 0.4601. Min Range: -0.4601

Max Act: 0.4142. Min Act: -0.0001

Data Index: 1236044 (C4 (Web Text))

Max Activating Token Index: 364

Text #4

Max Range: 0.4601. Min Range: -0.4601

Max Act: 0.3592. Min Act: -0.0000

Data Index: 1045933 (C4 (Web Text))

Max Activating Token Index: 863

Text #5

Max Range: 0.4601. Min Range: -0.4601

Max Act: 0.3920. Min Act: -0.0001

Data Index: 294401 (C4 (Web Text))

Max Activating Token Index: 972

Text #6

Max Range: 0.4601. Min Range: -0.4601

Max Act: 0.3687. Min Act: -0.0001

Data Index: 673865 (C4 (Web Text))

Max Activating Token Index: 416

Text #7

Max Range: 0.4601. Min Range: -0.4601

Max Act: 0.3759. Min Act: -0.0001

Data Index: 1266382 (C4 (Web Text))

Max Activating Token Index: 217

Text #8

Max Range: 0.4601. Min Range: -0.4601

Max Act: 0.3729. Min Act: -0.0001

Data Index: 1109862 (C4 (Web Text))

Max Activating Token Index: 262

Text #9

Max Range: 0.4601. Min Range: -0.4601

Max Act: 0.3591. Min Act: -0.0001

Data Index: 802939 (C4 (Web Text))

Max Activating Token Index: 693

Text #10

Max Range: 0.4601. Min Range: -0.4601

Max Act: 0.3453. Min Act: -0.0001

Data Index: 290803 (C4 (Web Text))

Max Activating Token Index: 1016

Text #11

Max Range: 0.4601. Min Range: -0.4601

Max Act: 0.3538. Min Act: -0.0001

Data Index: 607283 (C4 (Web Text))

Max Activating Token Index: 940

Text #12

Max Range: 0.4601. Min Range: -0.4601

Max Act: 0.3689. Min Act: -0.0001

Data Index: 162865 (C4 (Web Text))

Max Activating Token Index: 325

Text #13

Max Range: 0.4601. Min Range: -0.4601

Max Act: 0.3661. Min Act: -0.0001

Data Index: 43252 (C4 (Web Text))

Max Activating Token Index: 516

Text #14

Max Range: 0.4601. Min Range: -0.4601

Max Act: 0.3676. Min Act: -0.0001

Data Index: 439908 (C4 (Web Text))

Max Activating Token Index: 753

Text #15

Max Range: 0.4601. Min Range: -0.4601

Max Act: 0.3490. Min Act: -0.0001

Data Index: 397033 (C4 (Web Text))

Max Activating Token Index: 860

Text #16

Max Range: 0.4601. Min Range: -0.4601

Max Act: 0.3468. Min Act: -0.0001

Data Index: 1309958 (C4 (Web Text))

Max Activating Token Index: 197

Text #17

Max Range: 0.4601. Min Range: -0.4601

Max Act: 0.3285. Min Act: -0.0001

Data Index: 1256096 (C4 (Web Text))

Max Activating Token Index: 178

Text #18

Max Range: 0.4601. Min Range: -0.4601

Max Act: 0.3520. Min Act: -0.0001

Data Index: 363161 (C4 (Web Text))

Max Activating Token Index: 275

Text #19

Max Range: 0.4601. Min Range: -0.4601

Max Act: 0.3500. Min Act: -0.0001

Data Index: 953977 (C4 (Web Text))

Max Activating Token Index: 38