Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3919 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.8950. Min Range: -1.8950

Max Act: 1.8950. Min Act: -0.0000

Data Index: 223990 (C4 (Web Text))

Max Activating Token Index: 642

Text #1

Max Range: 1.8950. Min Range: -1.8950

Max Act: 1.7540. Min Act: -0.0000

Data Index: 143685 (C4 (Web Text))

Max Activating Token Index: 161

Text #2

Max Range: 1.8950. Min Range: -1.8950

Max Act: 1.7324. Min Act: -0.0000

Data Index: 1322125 (C4 (Web Text))

Max Activating Token Index: 112

Text #3

Max Range: 1.8950. Min Range: -1.8950

Max Act: 1.7180. Min Act: -0.0000

Data Index: 721571 (C4 (Web Text))

Max Activating Token Index: 194

Text #4

Max Range: 1.8950. Min Range: -1.8950

Max Act: 1.7673. Min Act: -0.0000

Data Index: 920898 (C4 (Web Text))

Max Activating Token Index: 984

Text #5

Max Range: 1.8950. Min Range: -1.8950

Max Act: 1.6592. Min Act: -0.0000

Data Index: 227646 (C4 (Web Text))

Max Activating Token Index: 380

Text #6

Max Range: 1.8950. Min Range: -1.8950

Max Act: 1.6912. Min Act: -0.0000

Data Index: 362160 (C4 (Web Text))

Max Activating Token Index: 64

Text #7

Max Range: 1.8950. Min Range: -1.8950

Max Act: 1.6855. Min Act: -0.0000

Data Index: 341895 (C4 (Web Text))

Max Activating Token Index: 742

Text #8

Max Range: 1.8950. Min Range: -1.8950

Max Act: 1.6229. Min Act: -0.0000

Data Index: 194369 (C4 (Web Text))

Max Activating Token Index: 192

Text #9

Max Range: 1.8950. Min Range: -1.8950

Max Act: 1.7409. Min Act: -0.0000

Data Index: 428852 (C4 (Web Text))

Max Activating Token Index: 997

Text #10

Max Range: 1.8950. Min Range: -1.8950

Max Act: 1.6321. Min Act: -0.0000

Data Index: 707515 (C4 (Web Text))

Max Activating Token Index: 63

Text #11

Max Range: 1.8950. Min Range: -1.8950

Max Act: 1.6136. Min Act: -0.0000

Data Index: 1282745 (C4 (Web Text))

Max Activating Token Index: 853

Text #12

Max Range: 1.8950. Min Range: -1.8950

Max Act: 1.6523. Min Act: -0.0000

Data Index: 925850 (C4 (Web Text))

Max Activating Token Index: 196

Text #13

Max Range: 1.8950. Min Range: -1.8950

Max Act: 1.7004. Min Act: -0.0000

Data Index: 1258283 (C4 (Web Text))

Max Activating Token Index: 252

Text #14

Max Range: 1.8950. Min Range: -1.8950

Max Act: 1.5931. Min Act: -0.0000

Data Index: 1272815 (C4 (Web Text))

Max Activating Token Index: 799

Text #15

Max Range: 1.8950. Min Range: -1.8950

Max Act: 1.5386. Min Act: -0.0000

Data Index: 220974 (C4 (Web Text))

Max Activating Token Index: 138

Text #16

Max Range: 1.8950. Min Range: -1.8950

Max Act: 1.5349. Min Act: -0.0000

Data Index: 882539 (C4 (Web Text))

Max Activating Token Index: 276

Text #17

Max Range: 1.8950. Min Range: -1.8950

Max Act: 1.5170. Min Act: -0.0000

Data Index: 367408 (C4 (Web Text))

Max Activating Token Index: 746

Text #18

Max Range: 1.8950. Min Range: -1.8950

Max Act: 1.5169. Min Act: -0.0000

Data Index: 180666 (C4 (Web Text))

Max Activating Token Index: 689

Text #19

Max Range: 1.8950. Min Range: -1.8950

Max Act: 1.5651. Min Act: -0.0000

Data Index: 1295633 (C4 (Web Text))

Max Activating Token Index: 271