Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2326 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 4.1705. Min Range: -4.1705

Max Act: 4.1705. Min Act: -0.0002

Data Index: 158570 (C4 (Web Text))

Max Activating Token Index: 11

Text #1

Max Range: 4.1705. Min Range: -4.1705

Max Act: 3.5844. Min Act: -0.0001

Data Index: 1112255 (C4 (Web Text))

Max Activating Token Index: 684

Text #2

Max Range: 4.1705. Min Range: -4.1705

Max Act: 3.5400. Min Act: -0.0001

Data Index: 552706 (C4 (Web Text))

Max Activating Token Index: 46

Text #3

Max Range: 4.1705. Min Range: -4.1705

Max Act: 3.5644. Min Act: -0.0001

Data Index: 1062728 (C4 (Web Text))

Max Activating Token Index: 59

Text #4

Max Range: 4.1705. Min Range: -4.1705

Max Act: 3.5486. Min Act: -0.0002

Data Index: 417848 (C4 (Web Text))

Max Activating Token Index: 26

Text #5

Max Range: 4.1705. Min Range: -4.1705

Max Act: 3.3713. Min Act: -0.0001

Data Index: 658647 (C4 (Web Text))

Max Activating Token Index: 34

Text #6

Max Range: 4.1705. Min Range: -4.1705

Max Act: 3.5156. Min Act: -0.0001

Data Index: 1020494 (C4 (Web Text))

Max Activating Token Index: 17

Text #7

Max Range: 4.1705. Min Range: -4.1705

Max Act: 3.4784. Min Act: -0.0001

Data Index: 56916 (C4 (Web Text))

Max Activating Token Index: 305

Text #8

Max Range: 4.1705. Min Range: -4.1705

Max Act: 3.5076. Min Act: -0.0002

Data Index: 1359113 (C4 (Web Text))

Max Activating Token Index: 6

Text #9

Max Range: 4.1705. Min Range: -4.1705

Max Act: 3.4852. Min Act: -0.0001

Data Index: 165464 (C4 (Web Text))

Max Activating Token Index: 27

Text #10

Max Range: 4.1705. Min Range: -4.1705

Max Act: 3.4946. Min Act: -0.0002

Data Index: 885518 (C4 (Web Text))

Max Activating Token Index: 15

Text #11

Max Range: 4.1705. Min Range: -4.1705

Max Act: 3.4093. Min Act: -0.0001

Data Index: 376871 (C4 (Web Text))

Max Activating Token Index: 15

Text #12

Max Range: 4.1705. Min Range: -4.1705

Max Act: 3.3887. Min Act: -0.0001

Data Index: 549463 (C4 (Web Text))

Max Activating Token Index: 9

Text #13

Max Range: 4.1705. Min Range: -4.1705

Max Act: 3.3885. Min Act: -0.0001

Data Index: 192795 (C4 (Web Text))

Max Activating Token Index: 7

Text #14

Max Range: 4.1705. Min Range: -4.1705

Max Act: 3.3469. Min Act: -0.0001

Data Index: 1266606 (C4 (Web Text))

Max Activating Token Index: 256

Text #15

Max Range: 4.1705. Min Range: -4.1705

Max Act: 3.1727. Min Act: -0.0001

Data Index: 591479 (C4 (Web Text))

Max Activating Token Index: 25

Text #16

Max Range: 4.1705. Min Range: -4.1705

Max Act: 3.4546. Min Act: -0.0001

Data Index: 1158757 (C4 (Web Text))

Max Activating Token Index: 586

Text #17

Max Range: 4.1705. Min Range: -4.1705

Max Act: 3.3021. Min Act: -0.0001

Data Index: 525168 (C4 (Web Text))

Max Activating Token Index: 466

Text #18

Max Range: 4.1705. Min Range: -4.1705

Max Act: 3.2983. Min Act: -0.0001

Data Index: 1226122 (C4 (Web Text))

Max Activating Token Index: 263

Text #19

Max Range: 4.1705. Min Range: -4.1705

Max Act: 3.1339. Min Act: -0.0001

Data Index: 27188 (C4 (Web Text))

Max Activating Token Index: 39