Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3293 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.2638. Min Range: -2.2638

Max Act: 2.2501. Min Act: -0.0000

Data Index: 244183 (C4 (Web Text))

Max Activating Token Index: 827

Text #1

Max Range: 2.2638. Min Range: -2.2638

Max Act: 2.2638. Min Act: -0.0000

Data Index: 392648 (C4 (Web Text))

Max Activating Token Index: 641

Text #2

Max Range: 2.2638. Min Range: -2.2638

Max Act: 2.0375. Min Act: -0.0000

Data Index: 1089134 (C4 (Web Text))

Max Activating Token Index: 143

Text #3

Max Range: 2.2638. Min Range: -2.2638

Max Act: 2.1276. Min Act: -0.0000

Data Index: 1878 (C4 (Web Text))

Max Activating Token Index: 621

Text #4

Max Range: 2.2638. Min Range: -2.2638

Max Act: 1.9878. Min Act: -0.0000

Data Index: 126421 (C4 (Web Text))

Max Activating Token Index: 393

Text #5

Max Range: 2.2638. Min Range: -2.2638

Max Act: 1.8911. Min Act: -0.0000

Data Index: 69436 (C4 (Web Text))

Max Activating Token Index: 435

Text #6

Max Range: 2.2638. Min Range: -2.2638

Max Act: 1.8084. Min Act: -0.0000

Data Index: 1183423 (C4 (Web Text))

Max Activating Token Index: 98

Text #7

Max Range: 2.2638. Min Range: -2.2638

Max Act: 1.6489. Min Act: -0.0000

Data Index: 1256908 (C4 (Web Text))

Max Activating Token Index: 248

Text #8

Max Range: 2.2638. Min Range: -2.2638

Max Act: 1.1659. Min Act: -0.0000

Data Index: 390381 (C4 (Web Text))

Max Activating Token Index: 611

Text #9

Max Range: 2.2638. Min Range: -2.2638

Max Act: 1.0399. Min Act: -0.0000

Data Index: 1163085 (C4 (Web Text))

Max Activating Token Index: 547

Text #10

Max Range: 2.2638. Min Range: -2.2638

Max Act: 1.0423. Min Act: -0.0000

Data Index: 1330529 (C4 (Web Text))

Max Activating Token Index: 244

Text #11

Max Range: 2.2638. Min Range: -2.2638

Max Act: 1.0319. Min Act: -0.0000

Data Index: 981049 (C4 (Web Text))

Max Activating Token Index: 522

Text #12

Max Range: 2.2638. Min Range: -2.2638

Max Act: 1.0134. Min Act: -0.0000

Data Index: 907527 (C4 (Web Text))

Max Activating Token Index: 344

Text #13

Max Range: 2.2638. Min Range: -2.2638

Max Act: 1.0517. Min Act: -0.0000

Data Index: 834202 (C4 (Web Text))

Max Activating Token Index: 504

Text #14

Max Range: 2.2638. Min Range: -2.2638

Max Act: 0.9499. Min Act: -0.0000

Data Index: 589320 (C4 (Web Text))

Max Activating Token Index: 247

Text #15

Max Range: 2.2638. Min Range: -2.2638

Max Act: 0.9411. Min Act: -0.0000

Data Index: 649781 (C4 (Web Text))

Max Activating Token Index: 442

Text #16

Max Range: 2.2638. Min Range: -2.2638

Max Act: 0.9419. Min Act: -0.0000

Data Index: 595965 (C4 (Web Text))

Max Activating Token Index: 653

Text #17

Max Range: 2.2638. Min Range: -2.2638

Max Act: 0.9291. Min Act: -0.0000

Data Index: 832304 (C4 (Web Text))

Max Activating Token Index: 545

Text #18

Max Range: 2.2638. Min Range: -2.2638

Max Act: 0.9222. Min Act: -0.0000

Data Index: 715562 (C4 (Web Text))

Max Activating Token Index: 243

Text #19

Max Range: 2.2638. Min Range: -2.2638

Max Act: 0.8750. Min Act: -0.0000

Data Index: 1059036 (C4 (Web Text))

Max Activating Token Index: 693