Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4405 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 4.5837. Min Range: -4.5837

Max Act: 4.5814. Min Act: -0.0000

Data Index: 393750 (C4 (Web Text))

Max Activating Token Index: 1023

Text #1

Max Range: 4.5837. Min Range: -4.5837

Max Act: 4.4076. Min Act: -0.0000

Data Index: 975580 (C4 (Web Text))

Max Activating Token Index: 83

Text #2

Max Range: 4.5837. Min Range: -4.5837

Max Act: 4.3253. Min Act: -0.0000

Data Index: 454976 (C4 (Web Text))

Max Activating Token Index: 108

Text #3

Max Range: 4.5837. Min Range: -4.5837

Max Act: 4.5837. Min Act: -0.0000

Data Index: 366006 (C4 (Web Text))

Max Activating Token Index: 143

Text #4

Max Range: 4.5837. Min Range: -4.5837

Max Act: 4.5582. Min Act: -0.0000

Data Index: 185720 (C4 (Web Text))

Max Activating Token Index: 610

Text #5

Max Range: 4.5837. Min Range: -4.5837

Max Act: 4.3440. Min Act: -0.0000

Data Index: 1153825 (C4 (Web Text))

Max Activating Token Index: 784

Text #6

Max Range: 4.5837. Min Range: -4.5837

Max Act: 4.2873. Min Act: -0.0000

Data Index: 948978 (C4 (Web Text))

Max Activating Token Index: 579

Text #7

Max Range: 4.5837. Min Range: -4.5837

Max Act: 4.2461. Min Act: -0.0000

Data Index: 383329 (C4 (Web Text))

Max Activating Token Index: 478

Text #8

Max Range: 4.5837. Min Range: -4.5837

Max Act: 4.5205. Min Act: -0.0000

Data Index: 987863 (C4 (Web Text))

Max Activating Token Index: 873

Text #9

Max Range: 4.5837. Min Range: -4.5837

Max Act: 3.7474. Min Act: -0.0000

Data Index: 203481 (C4 (Web Text))

Max Activating Token Index: 842

Text #10

Max Range: 4.5837. Min Range: -4.5837

Max Act: 3.9957. Min Act: -0.0000

Data Index: 1104278 (C4 (Web Text))

Max Activating Token Index: 904

Text #11

Max Range: 4.5837. Min Range: -4.5837

Max Act: 4.2882. Min Act: -0.0000

Data Index: 693001 (C4 (Web Text))

Max Activating Token Index: 833

Text #12

Max Range: 4.5837. Min Range: -4.5837

Max Act: 4.2794. Min Act: -0.0000

Data Index: 620634 (C4 (Web Text))

Max Activating Token Index: 760

Text #13

Max Range: 4.5837. Min Range: -4.5837

Max Act: 4.1337. Min Act: -0.0000

Data Index: 1330096 (C4 (Web Text))

Max Activating Token Index: 852

Text #14

Max Range: 4.5837. Min Range: -4.5837

Max Act: 3.9582. Min Act: -0.0000

Data Index: 722117 (C4 (Web Text))

Max Activating Token Index: 912

Text #15

Max Range: 4.5837. Min Range: -4.5837

Max Act: 4.0856. Min Act: -0.0000

Data Index: 316499 (C4 (Web Text))

Max Activating Token Index: 146

Text #16

Max Range: 4.5837. Min Range: -4.5837

Max Act: 3.8696. Min Act: -0.0000

Data Index: 472937 (C4 (Web Text))

Max Activating Token Index: 71

Text #17

Max Range: 4.5837. Min Range: -4.5837

Max Act: 3.7344. Min Act: -0.0000

Data Index: 1252026 (C4 (Web Text))

Max Activating Token Index: 901

Text #18

Max Range: 4.5837. Min Range: -4.5837

Max Act: 3.8579. Min Act: -0.0000

Data Index: 548695 (C4 (Web Text))

Max Activating Token Index: 537

Text #19

Max Range: 4.5837. Min Range: -4.5837

Max Act: 4.0043. Min Act: -0.0000

Data Index: 1166742 (C4 (Web Text))

Max Activating Token Index: 842