Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4801 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.3311. Min Range: -3.3311

Max Act: 3.3157. Min Act: -0.0000

Data Index: 1007190 (C4 (Web Text))

Max Activating Token Index: 610

Text #1

Max Range: 3.3311. Min Range: -3.3311

Max Act: 3.1773. Min Act: -0.0000

Data Index: 93958 (C4 (Web Text))

Max Activating Token Index: 357

Text #2

Max Range: 3.3311. Min Range: -3.3311

Max Act: 2.9950. Min Act: -0.0000

Data Index: 980979 (C4 (Web Text))

Max Activating Token Index: 528

Text #3

Max Range: 3.3311. Min Range: -3.3311

Max Act: 3.3311. Min Act: -0.0000

Data Index: 1108476 (C4 (Web Text))

Max Activating Token Index: 253

Text #4

Max Range: 3.3311. Min Range: -3.3311

Max Act: 3.1002. Min Act: 0.0000

Data Index: 584106 (C4 (Web Text))

Max Activating Token Index: 627

Text #5

Max Range: 3.3311. Min Range: -3.3311

Max Act: 3.2415. Min Act: -0.0000

Data Index: 570494 (C4 (Web Text))

Max Activating Token Index: 400

Text #6

Max Range: 3.3311. Min Range: -3.3311

Max Act: 3.0655. Min Act: -0.0000

Data Index: 506256 (C4 (Web Text))

Max Activating Token Index: 494

Text #7

Max Range: 3.3311. Min Range: -3.3311

Max Act: 3.0698. Min Act: -0.0000

Data Index: 512670 (C4 (Web Text))

Max Activating Token Index: 590

Text #8

Max Range: 3.3311. Min Range: -3.3311

Max Act: 3.1280. Min Act: -0.0000

Data Index: 984966 (C4 (Web Text))

Max Activating Token Index: 622

Text #9

Max Range: 3.3311. Min Range: -3.3311

Max Act: 2.8448. Min Act: -0.0000

Data Index: 420236 (C4 (Web Text))

Max Activating Token Index: 596

Text #10

Max Range: 3.3311. Min Range: -3.3311

Max Act: 2.9370. Min Act: -0.0000

Data Index: 484562 (C4 (Web Text))

Max Activating Token Index: 628

Text #11

Max Range: 3.3311. Min Range: -3.3311

Max Act: 3.0704. Min Act: -0.0000

Data Index: 804504 (C4 (Web Text))

Max Activating Token Index: 763

Text #12

Max Range: 3.3311. Min Range: -3.3311

Max Act: 2.9085. Min Act: -0.0000

Data Index: 639352 (C4 (Web Text))

Max Activating Token Index: 60

Text #13

Max Range: 3.3311. Min Range: -3.3311

Max Act: 2.9182. Min Act: -0.0000

Data Index: 1356516 (C4 (Web Text))

Max Activating Token Index: 273

Text #14

Max Range: 3.3311. Min Range: -3.3311

Max Act: 2.9033. Min Act: -0.0000

Data Index: 249295 (C4 (Web Text))

Max Activating Token Index: 319

Text #15

Max Range: 3.3311. Min Range: -3.3311

Max Act: 2.8654. Min Act: -0.0000

Data Index: 1103802 (C4 (Web Text))

Max Activating Token Index: 791

Text #16

Max Range: 3.3311. Min Range: -3.3311

Max Act: 3.1590. Min Act: -0.0000

Data Index: 981388 (C4 (Web Text))

Max Activating Token Index: 590

Text #17

Max Range: 3.3311. Min Range: -3.3311

Max Act: 2.6516. Min Act: -0.0000

Data Index: 256271 (C4 (Web Text))

Max Activating Token Index: 361

Text #18

Max Range: 3.3311. Min Range: -3.3311

Max Act: 2.8532. Min Act: -0.0000

Data Index: 188526 (C4 (Web Text))

Max Activating Token Index: 527

Text #19

Max Range: 3.3311. Min Range: -3.3311

Max Act: 2.9954. Min Act: -0.0000

Data Index: 1074184 (C4 (Web Text))

Max Activating Token Index: 670