Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3536 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.2859. Min Range: -3.2859

Max Act: 3.2859. Min Act: -0.0000

Data Index: 1147328 (C4 (Web Text))

Max Activating Token Index: 60

Text #1

Max Range: 3.2859. Min Range: -3.2859

Max Act: 2.9495. Min Act: -0.0000

Data Index: 836708 (C4 (Web Text))

Max Activating Token Index: 278

Text #2

Max Range: 3.2859. Min Range: -3.2859

Max Act: 2.8596. Min Act: -0.0000

Data Index: 22727 (C4 (Web Text))

Max Activating Token Index: 100

Text #3

Max Range: 3.2859. Min Range: -3.2859

Max Act: 2.7747. Min Act: -0.0000

Data Index: 687162 (C4 (Web Text))

Max Activating Token Index: 997

Text #4

Max Range: 3.2859. Min Range: -3.2859

Max Act: 2.7173. Min Act: -0.0000

Data Index: 425657 (C4 (Web Text))

Max Activating Token Index: 1016

Text #5

Max Range: 3.2859. Min Range: -3.2859

Max Act: 2.6890. Min Act: -0.0000

Data Index: 1286163 (C4 (Web Text))

Max Activating Token Index: 373

Text #6

Max Range: 3.2859. Min Range: -3.2859

Max Act: 2.4382. Min Act: -0.0000

Data Index: 514859 (C4 (Web Text))

Max Activating Token Index: 230

Text #7

Max Range: 3.2859. Min Range: -3.2859

Max Act: 2.5242. Min Act: -0.0000

Data Index: 1050219 (C4 (Web Text))

Max Activating Token Index: 924

Text #8

Max Range: 3.2859. Min Range: -3.2859

Max Act: 2.6137. Min Act: -0.0000

Data Index: 166793 (C4 (Web Text))

Max Activating Token Index: 828

Text #9

Max Range: 3.2859. Min Range: -3.2859

Max Act: 2.3816. Min Act: -0.0000

Data Index: 837537 (C4 (Web Text))

Max Activating Token Index: 696

Text #10

Max Range: 3.2859. Min Range: -3.2859

Max Act: 2.0668. Min Act: -0.0000

Data Index: 473756 (C4 (Web Text))

Max Activating Token Index: 277

Text #11

Max Range: 3.2859. Min Range: -3.2859

Max Act: 2.2729. Min Act: -0.0000

Data Index: 745472 (C4 (Web Text))

Max Activating Token Index: 950

Text #12

Max Range: 3.2859. Min Range: -3.2859

Max Act: 2.2344. Min Act: -0.0000

Data Index: 500898 (C4 (Web Text))

Max Activating Token Index: 61

Text #13

Max Range: 3.2859. Min Range: -3.2859

Max Act: 2.4433. Min Act: -0.0000

Data Index: 107147 (C4 (Web Text))

Max Activating Token Index: 144

Text #14

Max Range: 3.2859. Min Range: -3.2859

Max Act: 2.1873. Min Act: -0.0000

Data Index: 152290 (C4 (Web Text))

Max Activating Token Index: 111

Text #15

Max Range: 3.2859. Min Range: -3.2859

Max Act: 2.1803. Min Act: -0.0000

Data Index: 156523 (C4 (Web Text))

Max Activating Token Index: 409

Text #16

Max Range: 3.2859. Min Range: -3.2859

Max Act: 2.1746. Min Act: -0.0000

Data Index: 647012 (C4 (Web Text))

Max Activating Token Index: 238

Text #17

Max Range: 3.2859. Min Range: -3.2859

Max Act: 2.2564. Min Act: -0.0000

Data Index: 129873 (C4 (Web Text))

Max Activating Token Index: 840

Text #18

Max Range: 3.2859. Min Range: -3.2859

Max Act: 2.0022. Min Act: -0.0000

Data Index: 992740 (C4 (Web Text))

Max Activating Token Index: 115

Text #19

Max Range: 3.2859. Min Range: -3.2859

Max Act: 2.0979. Min Act: -0.0000

Data Index: 883067 (C4 (Web Text))

Max Activating Token Index: 257