Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3404 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.9733. Min Range: -0.9733

Max Act: 0.9733. Min Act: -0.0000

Data Index: 190247 (C4 (Web Text))

Max Activating Token Index: 357

Text #1

Max Range: 0.9733. Min Range: -0.9733

Max Act: 0.9608. Min Act: -0.0000

Data Index: 1117369 (C4 (Web Text))

Max Activating Token Index: 610

Text #2

Max Range: 0.9733. Min Range: -0.9733

Max Act: 0.9012. Min Act: -0.0000

Data Index: 373230 (C4 (Web Text))

Max Activating Token Index: 295

Text #3

Max Range: 0.9733. Min Range: -0.9733

Max Act: 0.8155. Min Act: -0.0000

Data Index: 1355382 (C4 (Web Text))

Max Activating Token Index: 298

Text #4

Max Range: 0.9733. Min Range: -0.9733

Max Act: 0.7376. Min Act: -0.0000

Data Index: 5397 (C4 (Web Text))

Max Activating Token Index: 776

Text #5

Max Range: 0.9733. Min Range: -0.9733

Max Act: 0.7469. Min Act: -0.0000

Data Index: 680839 (C4 (Web Text))

Max Activating Token Index: 207

Text #6

Max Range: 0.9733. Min Range: -0.9733

Max Act: 0.7032. Min Act: -0.0000

Data Index: 467779 (C4 (Web Text))

Max Activating Token Index: 786

Text #7

Max Range: 0.9733. Min Range: -0.9733

Max Act: 0.7360. Min Act: -0.0000

Data Index: 672082 (C4 (Web Text))

Max Activating Token Index: 750

Text #8

Max Range: 0.9733. Min Range: -0.9733

Max Act: 0.6449. Min Act: -0.0000

Data Index: 909838 (C4 (Web Text))

Max Activating Token Index: 613

Text #9

Max Range: 0.9733. Min Range: -0.9733

Max Act: 0.6953. Min Act: -0.0000

Data Index: 1129098 (C4 (Web Text))

Max Activating Token Index: 711

Text #10

Max Range: 0.9733. Min Range: -0.9733

Max Act: 0.6368. Min Act: -0.0000

Data Index: 1190162 (C4 (Web Text))

Max Activating Token Index: 1002

Text #11

Max Range: 0.9733. Min Range: -0.9733

Max Act: 0.6668. Min Act: -0.0000

Data Index: 1133797 (C4 (Web Text))

Max Activating Token Index: 355

Text #12

Max Range: 0.9733. Min Range: -0.9733

Max Act: 0.6477. Min Act: -0.0000

Data Index: 1240294 (C4 (Web Text))

Max Activating Token Index: 964

Text #13

Max Range: 0.9733. Min Range: -0.9733

Max Act: 0.6655. Min Act: -0.0000

Data Index: 980720 (C4 (Web Text))

Max Activating Token Index: 649

Text #14

Max Range: 0.9733. Min Range: -0.9733

Max Act: 0.6627. Min Act: -0.0000

Data Index: 814017 (C4 (Web Text))

Max Activating Token Index: 9

Text #15

Max Range: 0.9733. Min Range: -0.9733

Max Act: 0.6352. Min Act: -0.0000

Data Index: 955438 (C4 (Web Text))

Max Activating Token Index: 859

Text #16

Max Range: 0.9733. Min Range: -0.9733

Max Act: 0.6214. Min Act: -0.0000

Data Index: 238003 (C4 (Web Text))

Max Activating Token Index: 942

Text #17

Max Range: 0.9733. Min Range: -0.9733

Max Act: 0.6004. Min Act: -0.0000

Data Index: 1055036 (C4 (Web Text))

Max Activating Token Index: 387

Text #18

Max Range: 0.9733. Min Range: -0.9733

Max Act: 0.5962. Min Act: -0.0000

Data Index: 607154 (C4 (Web Text))

Max Activating Token Index: 914

Text #19

Max Range: 0.9733. Min Range: -0.9733

Max Act: 0.6135. Min Act: -0.0000

Data Index: 1194325 (C4 (Web Text))

Max Activating Token Index: 502