Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4629 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.1747. Min Range: -2.1747

Max Act: 2.1747. Min Act: -0.0000

Data Index: 845805 (C4 (Web Text))

Max Activating Token Index: 165

Text #1

Max Range: 2.1747. Min Range: -2.1747

Max Act: 2.1282. Min Act: -0.0000

Data Index: 834215 (C4 (Web Text))

Max Activating Token Index: 533

Text #2

Max Range: 2.1747. Min Range: -2.1747

Max Act: 2.0223. Min Act: -0.0000

Data Index: 797597 (C4 (Web Text))

Max Activating Token Index: 771

Text #3

Max Range: 2.1747. Min Range: -2.1747

Max Act: 1.9930. Min Act: -0.0000

Data Index: 1249193 (C4 (Web Text))

Max Activating Token Index: 995

Text #4

Max Range: 2.1747. Min Range: -2.1747

Max Act: 1.7913. Min Act: -0.0000

Data Index: 1351109 (C4 (Web Text))

Max Activating Token Index: 20

Text #5

Max Range: 2.1747. Min Range: -2.1747

Max Act: 1.8435. Min Act: -0.0000

Data Index: 1147220 (C4 (Web Text))

Max Activating Token Index: 14

Text #6

Max Range: 2.1747. Min Range: -2.1747

Max Act: 1.7861. Min Act: -0.0000

Data Index: 730500 (C4 (Web Text))

Max Activating Token Index: 991

Text #7

Max Range: 2.1747. Min Range: -2.1747

Max Act: 1.6887. Min Act: -0.0000

Data Index: 517713 (C4 (Web Text))

Max Activating Token Index: 273

Text #8

Max Range: 2.1747. Min Range: -2.1747

Max Act: 1.8972. Min Act: -0.0000

Data Index: 1267811 (C4 (Web Text))

Max Activating Token Index: 966

Text #9

Max Range: 2.1747. Min Range: -2.1747

Max Act: 1.6329. Min Act: -0.0000

Data Index: 939619 (C4 (Web Text))

Max Activating Token Index: 721

Text #10

Max Range: 2.1747. Min Range: -2.1747

Max Act: 1.7122. Min Act: -0.0000

Data Index: 23189 (C4 (Web Text))

Max Activating Token Index: 621

Text #11

Max Range: 2.1747. Min Range: -2.1747

Max Act: 1.7050. Min Act: -0.0000

Data Index: 782861 (C4 (Web Text))

Max Activating Token Index: 642

Text #12

Max Range: 2.1747. Min Range: -2.1747

Max Act: 1.6882. Min Act: -0.0000

Data Index: 1180923 (C4 (Web Text))

Max Activating Token Index: 69

Text #13

Max Range: 2.1747. Min Range: -2.1747

Max Act: 1.7586. Min Act: -0.0000

Data Index: 758483 (C4 (Web Text))

Max Activating Token Index: 989

Text #14

Max Range: 2.1747. Min Range: -2.1747

Max Act: 1.6535. Min Act: -0.0000

Data Index: 71644 (C4 (Web Text))

Max Activating Token Index: 540

Text #15

Max Range: 2.1747. Min Range: -2.1747

Max Act: 1.7029. Min Act: -0.0000

Data Index: 763048 (C4 (Web Text))

Max Activating Token Index: 837

Text #16

Max Range: 2.1747. Min Range: -2.1747

Max Act: 1.5828. Min Act: -0.0000

Data Index: 184169 (C4 (Web Text))

Max Activating Token Index: 671

Text #17

Max Range: 2.1747. Min Range: -2.1747

Max Act: 1.5829. Min Act: -0.0000

Data Index: 249766 (C4 (Web Text))

Max Activating Token Index: 818

Text #18

Max Range: 2.1747. Min Range: -2.1747

Max Act: 1.4755. Min Act: -0.0000

Data Index: 1348387 (C4 (Web Text))

Max Activating Token Index: 830

Text #19

Max Range: 2.1747. Min Range: -2.1747

Max Act: 1.5217. Min Act: -0.0000

Data Index: 533822 (C4 (Web Text))

Max Activating Token Index: 163