Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4194 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.3114. Min Range: -1.3114

Max Act: 1.3114. Min Act: -0.0000

Data Index: 524428 (C4 (Web Text))

Max Activating Token Index: 561

Text #1

Max Range: 1.3114. Min Range: -1.3114

Max Act: 1.1644. Min Act: -0.0000

Data Index: 564091 (C4 (Web Text))

Max Activating Token Index: 428

Text #2

Max Range: 1.3114. Min Range: -1.3114

Max Act: 1.0559. Min Act: -0.0000

Data Index: 294127 (C4 (Web Text))

Max Activating Token Index: 370

Text #3

Max Range: 1.3114. Min Range: -1.3114

Max Act: 1.0272. Min Act: -0.0000

Data Index: 233385 (C4 (Web Text))

Max Activating Token Index: 874

Text #4

Max Range: 1.3114. Min Range: -1.3114

Max Act: 0.9456. Min Act: -0.0000

Data Index: 136875 (C4 (Web Text))

Max Activating Token Index: 744

Text #5

Max Range: 1.3114. Min Range: -1.3114

Max Act: 0.9699. Min Act: -0.0000

Data Index: 1197756 (C4 (Web Text))

Max Activating Token Index: 784

Text #6

Max Range: 1.3114. Min Range: -1.3114

Max Act: 0.8825. Min Act: -0.0000

Data Index: 271941 (C4 (Web Text))

Max Activating Token Index: 421

Text #7

Max Range: 1.3114. Min Range: -1.3114

Max Act: 0.8996. Min Act: -0.0000

Data Index: 564291 (C4 (Web Text))

Max Activating Token Index: 170

Text #8

Max Range: 1.3114. Min Range: -1.3114

Max Act: 0.8875. Min Act: -0.0000

Data Index: 1173759 (C4 (Web Text))

Max Activating Token Index: 549

Text #9

Max Range: 1.3114. Min Range: -1.3114

Max Act: 0.9358. Min Act: -0.0000

Data Index: 611212 (C4 (Web Text))

Max Activating Token Index: 603

Text #10

Max Range: 1.3114. Min Range: -1.3114

Max Act: 0.7284. Min Act: -0.0000

Data Index: 806558 (C4 (Web Text))

Max Activating Token Index: 659

Text #11

Max Range: 1.3114. Min Range: -1.3114

Max Act: 0.8387. Min Act: -0.0000

Data Index: 820225 (C4 (Web Text))

Max Activating Token Index: 654

Text #12

Max Range: 1.3114. Min Range: -1.3114

Max Act: 0.7220. Min Act: -0.0000

Data Index: 1141879 (C4 (Web Text))

Max Activating Token Index: 592

Text #13

Max Range: 1.3114. Min Range: -1.3114

Max Act: 0.7864. Min Act: -0.0000

Data Index: 995004 (C4 (Web Text))

Max Activating Token Index: 54

Text #14

Max Range: 1.3114. Min Range: -1.3114

Max Act: 0.8022. Min Act: -0.0000

Data Index: 687375 (C4 (Web Text))

Max Activating Token Index: 318

Text #15

Max Range: 1.3114. Min Range: -1.3114

Max Act: 0.8434. Min Act: -0.0000

Data Index: 1228602 (C4 (Web Text))

Max Activating Token Index: 75

Text #16

Max Range: 1.3114. Min Range: -1.3114

Max Act: 0.7514. Min Act: -0.0000

Data Index: 808017 (C4 (Web Text))

Max Activating Token Index: 278

Text #17

Max Range: 1.3114. Min Range: -1.3114

Max Act: 0.7275. Min Act: -0.0000

Data Index: 1344340 (C4 (Web Text))

Max Activating Token Index: 228

Text #18

Max Range: 1.3114. Min Range: -1.3114

Max Act: 0.7689. Min Act: -0.0000

Data Index: 875611 (C4 (Web Text))

Max Activating Token Index: 42

Text #19

Max Range: 1.3114. Min Range: -1.3114

Max Act: 0.7576. Min Act: -0.0000

Data Index: 1255340 (C4 (Web Text))

Max Activating Token Index: 431