Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1910 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.7677. Min Range: -3.7677

Max Act: 3.7677. Min Act: -0.0000

Data Index: 154510 (C4 (Web Text))

Max Activating Token Index: 641

Text #1

Max Range: 3.7677. Min Range: -3.7677

Max Act: 3.6246. Min Act: 0.0000

Data Index: 1305496 (C4 (Web Text))

Max Activating Token Index: 178

Text #2

Max Range: 3.7677. Min Range: -3.7677

Max Act: 3.6070. Min Act: -0.0000

Data Index: 1205776 (C4 (Web Text))

Max Activating Token Index: 543

Text #3

Max Range: 3.7677. Min Range: -3.7677

Max Act: 3.5774. Min Act: -0.0000

Data Index: 584675 (C4 (Web Text))

Max Activating Token Index: 899

Text #4

Max Range: 3.7677. Min Range: -3.7677

Max Act: 3.5709. Min Act: -0.0000

Data Index: 1271666 (C4 (Web Text))

Max Activating Token Index: 618

Text #5

Max Range: 3.7677. Min Range: -3.7677

Max Act: 3.5692. Min Act: 0.0000

Data Index: 356518 (C4 (Web Text))

Max Activating Token Index: 520

Text #6

Max Range: 3.7677. Min Range: -3.7677

Max Act: 3.3876. Min Act: -0.0000

Data Index: 1323776 (C4 (Web Text))

Max Activating Token Index: 156

Text #7

Max Range: 3.7677. Min Range: -3.7677

Max Act: 3.5333. Min Act: -0.0000

Data Index: 217890 (C4 (Web Text))

Max Activating Token Index: 357

Text #8

Max Range: 3.7677. Min Range: -3.7677

Max Act: 3.4783. Min Act: -0.0000

Data Index: 772946 (C4 (Web Text))

Max Activating Token Index: 560

Text #9

Max Range: 3.7677. Min Range: -3.7677

Max Act: 3.3082. Min Act: -0.0000

Data Index: 159362 (C4 (Web Text))

Max Activating Token Index: 674

Text #10

Max Range: 3.7677. Min Range: -3.7677

Max Act: 3.4169. Min Act: -0.0000

Data Index: 880591 (C4 (Web Text))

Max Activating Token Index: 181

Text #11

Max Range: 3.7677. Min Range: -3.7677

Max Act: 3.4085. Min Act: -0.0000

Data Index: 969474 (C4 (Web Text))

Max Activating Token Index: 201

Text #12

Max Range: 3.7677. Min Range: -3.7677

Max Act: 3.3598. Min Act: 0.0000

Data Index: 611063 (C4 (Web Text))

Max Activating Token Index: 723

Text #13

Max Range: 3.7677. Min Range: -3.7677

Max Act: 3.3535. Min Act: -0.0000

Data Index: 310279 (C4 (Web Text))

Max Activating Token Index: 991

Text #14

Max Range: 3.7677. Min Range: -3.7677

Max Act: 3.3456. Min Act: -0.0000

Data Index: 293133 (C4 (Web Text))

Max Activating Token Index: 308

Text #15

Max Range: 3.7677. Min Range: -3.7677

Max Act: 3.3384. Min Act: -0.0000

Data Index: 1269797 (C4 (Web Text))

Max Activating Token Index: 586

Text #16

Max Range: 3.7677. Min Range: -3.7677

Max Act: 3.1469. Min Act: -0.0000

Data Index: 1237677 (C4 (Web Text))

Max Activating Token Index: 190

Text #17

Max Range: 3.7677. Min Range: -3.7677

Max Act: 3.2705. Min Act: 0.0000

Data Index: 1316529 (C4 (Web Text))

Max Activating Token Index: 530

Text #18

Max Range: 3.7677. Min Range: -3.7677

Max Act: 3.1107. Min Act: -0.0000

Data Index: 779926 (C4 (Web Text))

Max Activating Token Index: 434

Text #19

Max Range: 3.7677. Min Range: -3.7677

Max Act: 3.2383. Min Act: 0.0000

Data Index: 1053595 (C4 (Web Text))

Max Activating Token Index: 332