Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5853 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.0123. Min Range: -3.0123

Max Act: 3.0103. Min Act: -0.0000

Data Index: 499081 (C4 (Web Text))

Max Activating Token Index: 269

Text #1

Max Range: 3.0123. Min Range: -3.0123

Max Act: 2.6985. Min Act: -0.0000

Data Index: 10131 (C4 (Web Text))

Max Activating Token Index: 819

Text #2

Max Range: 3.0123. Min Range: -3.0123

Max Act: 2.9115. Min Act: -0.0000

Data Index: 279459 (C4 (Web Text))

Max Activating Token Index: 395

Text #3

Max Range: 3.0123. Min Range: -3.0123

Max Act: 3.0123. Min Act: -0.0000

Data Index: 1025408 (C4 (Web Text))

Max Activating Token Index: 389

Text #4

Max Range: 3.0123. Min Range: -3.0123

Max Act: 2.8746. Min Act: -0.0000

Data Index: 1071628 (C4 (Web Text))

Max Activating Token Index: 947

Text #5

Max Range: 3.0123. Min Range: -3.0123

Max Act: 2.8550. Min Act: -0.0000

Data Index: 463087 (C4 (Web Text))

Max Activating Token Index: 3

Text #6

Max Range: 3.0123. Min Range: -3.0123

Max Act: 2.9661. Min Act: -0.0000

Data Index: 1069669 (C4 (Web Text))

Max Activating Token Index: 408

Text #7

Max Range: 3.0123. Min Range: -3.0123

Max Act: 2.7602. Min Act: -0.0000

Data Index: 189981 (C4 (Web Text))

Max Activating Token Index: 562

Text #8

Max Range: 3.0123. Min Range: -3.0123

Max Act: 2.5685. Min Act: -0.0000

Data Index: 769508 (C4 (Web Text))

Max Activating Token Index: 280

Text #9

Max Range: 3.0123. Min Range: -3.0123

Max Act: 2.8331. Min Act: -0.0000

Data Index: 392380 (C4 (Web Text))

Max Activating Token Index: 332

Text #10

Max Range: 3.0123. Min Range: -3.0123

Max Act: 2.4958. Min Act: -0.0000

Data Index: 766123 (C4 (Web Text))

Max Activating Token Index: 237

Text #11

Max Range: 3.0123. Min Range: -3.0123

Max Act: 2.5053. Min Act: -0.0000

Data Index: 331616 (C4 (Web Text))

Max Activating Token Index: 482

Text #12

Max Range: 3.0123. Min Range: -3.0123

Max Act: 2.3422. Min Act: -0.0000

Data Index: 999071 (C4 (Web Text))

Max Activating Token Index: 197

Text #13

Max Range: 3.0123. Min Range: -3.0123

Max Act: 2.4843. Min Act: -0.0000

Data Index: 961773 (C4 (Web Text))

Max Activating Token Index: 145

Text #14

Max Range: 3.0123. Min Range: -3.0123

Max Act: 2.6209. Min Act: -0.0000

Data Index: 651102 (C4 (Web Text))

Max Activating Token Index: 34

Text #15

Max Range: 3.0123. Min Range: -3.0123

Max Act: 2.6076. Min Act: -0.0000

Data Index: 258192 (C4 (Web Text))

Max Activating Token Index: 198

Text #16

Max Range: 3.0123. Min Range: -3.0123

Max Act: 2.8715. Min Act: -0.0000

Data Index: 778283 (C4 (Web Text))

Max Activating Token Index: 673

Text #17

Max Range: 3.0123. Min Range: -3.0123

Max Act: 2.4383. Min Act: -0.0000

Data Index: 548082 (C4 (Web Text))

Max Activating Token Index: 757

Text #18

Max Range: 3.0123. Min Range: -3.0123

Max Act: 2.5567. Min Act: -0.0000

Data Index: 333311 (C4 (Web Text))

Max Activating Token Index: 213

Text #19

Max Range: 3.0123. Min Range: -3.0123

Max Act: 2.5566. Min Act: -0.0000

Data Index: 757675 (C4 (Web Text))

Max Activating Token Index: 175