Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1676 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.4136. Min Range: -0.4136

Max Act: 0.4040. Min Act: -0.0000

Data Index: 1234541 (C4 (Web Text))

Max Activating Token Index: 273

Text #1

Max Range: 0.4136. Min Range: -0.4136

Max Act: 0.4076. Min Act: -0.0000

Data Index: 1191821 (C4 (Web Text))

Max Activating Token Index: 645

Text #2

Max Range: 0.4136. Min Range: -0.4136

Max Act: 0.4136. Min Act: -0.0000

Data Index: 1582563 (Python Code)

Max Activating Token Index: 655

Text #3

Max Range: 0.4136. Min Range: -0.4136

Max Act: 0.3732. Min Act: 0.0000

Data Index: 1511935 (Python Code)

Max Activating Token Index: 774

Text #4

Max Range: 0.4136. Min Range: -0.4136

Max Act: 0.3420. Min Act: -0.0000

Data Index: 365486 (C4 (Web Text))

Max Activating Token Index: 223

Text #5

Max Range: 0.4136. Min Range: -0.4136

Max Act: 0.3622. Min Act: -0.0000

Data Index: 1275971 (C4 (Web Text))

Max Activating Token Index: 572

Text #6

Max Range: 0.4136. Min Range: -0.4136

Max Act: 0.3705. Min Act: -0.0000

Data Index: 392060 (C4 (Web Text))

Max Activating Token Index: 702

Text #7

Max Range: 0.4136. Min Range: -0.4136

Max Act: 0.3575. Min Act: -0.0000

Data Index: 1337129 (C4 (Web Text))

Max Activating Token Index: 1013

Text #8

Max Range: 0.4136. Min Range: -0.4136

Max Act: 0.3537. Min Act: -0.0000

Data Index: 1234014 (C4 (Web Text))

Max Activating Token Index: 651

Text #9

Max Range: 0.4136. Min Range: -0.4136

Max Act: 0.3589. Min Act: -0.0000

Data Index: 851598 (C4 (Web Text))

Max Activating Token Index: 490

Text #10

Max Range: 0.4136. Min Range: -0.4136

Max Act: 0.3480. Min Act: -0.0000

Data Index: 424195 (C4 (Web Text))

Max Activating Token Index: 839

Text #11

Max Range: 0.4136. Min Range: -0.4136

Max Act: 0.3327. Min Act: -0.0000

Data Index: 373614 (C4 (Web Text))

Max Activating Token Index: 221

Text #12

Max Range: 0.4136. Min Range: -0.4136

Max Act: 0.3413. Min Act: -0.0000

Data Index: 384148 (C4 (Web Text))

Max Activating Token Index: 115

Text #13

Max Range: 0.4136. Min Range: -0.4136

Max Act: 0.3251. Min Act: -0.0000

Data Index: 989704 (C4 (Web Text))

Max Activating Token Index: 597

Text #14

Max Range: 0.4136. Min Range: -0.4136

Max Act: 0.3472. Min Act: -0.0000

Data Index: 1077218 (C4 (Web Text))

Max Activating Token Index: 309

Text #15

Max Range: 0.4136. Min Range: -0.4136

Max Act: 0.3182. Min Act: -0.0000

Data Index: 368430 (C4 (Web Text))

Max Activating Token Index: 336

Text #16

Max Range: 0.4136. Min Range: -0.4136

Max Act: 0.3320. Min Act: -0.0000

Data Index: 62047 (C4 (Web Text))

Max Activating Token Index: 578

Text #17

Max Range: 0.4136. Min Range: -0.4136

Max Act: 0.3292. Min Act: -0.0000

Data Index: 175314 (C4 (Web Text))

Max Activating Token Index: 711

Text #18

Max Range: 0.4136. Min Range: -0.4136

Max Act: 0.3149. Min Act: -0.0000

Data Index: 1014059 (C4 (Web Text))

Max Activating Token Index: 949

Text #19

Max Range: 0.4136. Min Range: -0.4136

Max Act: 0.3065. Min Act: -0.0000

Data Index: 1145225 (C4 (Web Text))

Max Activating Token Index: 532