Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 381 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.7207. Min Range: -2.7207

Max Act: 2.7207. Min Act: -0.0000

Data Index: 1136074 (C4 (Web Text))

Max Activating Token Index: 585

Text #1

Max Range: 2.7207. Min Range: -2.7207

Max Act: 2.1472. Min Act: -0.0000

Data Index: 383745 (C4 (Web Text))

Max Activating Token Index: 68

Text #2

Max Range: 2.7207. Min Range: -2.7207

Max Act: 1.8920. Min Act: -0.0000

Data Index: 291337 (C4 (Web Text))

Max Activating Token Index: 859

Text #3

Max Range: 2.7207. Min Range: -2.7207

Max Act: 1.8893. Min Act: -0.0000

Data Index: 1200692 (C4 (Web Text))

Max Activating Token Index: 651

Text #4

Max Range: 2.7207. Min Range: -2.7207

Max Act: 1.7553. Min Act: -0.0000

Data Index: 242329 (C4 (Web Text))

Max Activating Token Index: 154

Text #5

Max Range: 2.7207. Min Range: -2.7207

Max Act: 2.0305. Min Act: -0.0000

Data Index: 1033889 (C4 (Web Text))

Max Activating Token Index: 425

Text #6

Max Range: 2.7207. Min Range: -2.7207

Max Act: 1.9292. Min Act: -0.0000

Data Index: 292308 (C4 (Web Text))

Max Activating Token Index: 946

Text #7

Max Range: 2.7207. Min Range: -2.7207

Max Act: 1.7915. Min Act: -0.0000

Data Index: 429555 (C4 (Web Text))

Max Activating Token Index: 490

Text #8

Max Range: 2.7207. Min Range: -2.7207

Max Act: 1.5475. Min Act: -0.0000

Data Index: 260417 (C4 (Web Text))

Max Activating Token Index: 165

Text #9

Max Range: 2.7207. Min Range: -2.7207

Max Act: 1.6303. Min Act: -0.0000

Data Index: 862146 (C4 (Web Text))

Max Activating Token Index: 146

Text #10

Max Range: 2.7207. Min Range: -2.7207

Max Act: 1.6807. Min Act: -0.0000

Data Index: 606145 (C4 (Web Text))

Max Activating Token Index: 950

Text #11

Max Range: 2.7207. Min Range: -2.7207

Max Act: 1.4780. Min Act: -0.0000

Data Index: 185393 (C4 (Web Text))

Max Activating Token Index: 964

Text #12

Max Range: 2.7207. Min Range: -2.7207

Max Act: 1.4071. Min Act: -0.0000

Data Index: 731759 (C4 (Web Text))

Max Activating Token Index: 853

Text #13

Max Range: 2.7207. Min Range: -2.7207

Max Act: 1.2787. Min Act: -0.0000

Data Index: 47307 (C4 (Web Text))

Max Activating Token Index: 702

Text #14

Max Range: 2.7207. Min Range: -2.7207

Max Act: 1.2596. Min Act: -0.0000

Data Index: 700996 (C4 (Web Text))

Max Activating Token Index: 528

Text #15

Max Range: 2.7207. Min Range: -2.7207

Max Act: 1.2602. Min Act: -0.0000

Data Index: 1328823 (C4 (Web Text))

Max Activating Token Index: 645

Text #16

Max Range: 2.7207. Min Range: -2.7207

Max Act: 1.1459. Min Act: -0.0000

Data Index: 51628 (C4 (Web Text))

Max Activating Token Index: 644

Text #17

Max Range: 2.7207. Min Range: -2.7207

Max Act: 1.0717. Min Act: -0.0000

Data Index: 1238337 (C4 (Web Text))

Max Activating Token Index: 1015

Text #18

Max Range: 2.7207. Min Range: -2.7207

Max Act: 1.1711. Min Act: -0.0000

Data Index: 1266466 (C4 (Web Text))

Max Activating Token Index: 1005

Text #19

Max Range: 2.7207. Min Range: -2.7207

Max Act: 1.1184. Min Act: -0.0000

Data Index: 56762 (C4 (Web Text))

Max Activating Token Index: 868