Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 6117 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 4.2303. Min Range: -4.2303

Max Act: 3.9194. Min Act: -0.0000

Data Index: 151478 (C4 (Web Text))

Max Activating Token Index: 373

Text #1

Max Range: 4.2303. Min Range: -4.2303

Max Act: 4.2303. Min Act: -0.0000

Data Index: 1150346 (C4 (Web Text))

Max Activating Token Index: 571

Text #2

Max Range: 4.2303. Min Range: -4.2303

Max Act: 4.0378. Min Act: -0.0000

Data Index: 1183141 (C4 (Web Text))

Max Activating Token Index: 575

Text #3

Max Range: 4.2303. Min Range: -4.2303

Max Act: 3.9864. Min Act: -0.0000

Data Index: 709528 (C4 (Web Text))

Max Activating Token Index: 735

Text #4

Max Range: 4.2303. Min Range: -4.2303

Max Act: 3.2151. Min Act: -0.0000

Data Index: 774999 (C4 (Web Text))

Max Activating Token Index: 355

Text #5

Max Range: 4.2303. Min Range: -4.2303

Max Act: 3.3403. Min Act: -0.0000

Data Index: 593953 (C4 (Web Text))

Max Activating Token Index: 18

Text #6

Max Range: 4.2303. Min Range: -4.2303

Max Act: 3.2899. Min Act: -0.0000

Data Index: 1303988 (C4 (Web Text))

Max Activating Token Index: 275

Text #7

Max Range: 4.2303. Min Range: -4.2303

Max Act: 3.1232. Min Act: -0.0000

Data Index: 1082298 (C4 (Web Text))

Max Activating Token Index: 223

Text #8

Max Range: 4.2303. Min Range: -4.2303

Max Act: 3.1465. Min Act: -0.0000

Data Index: 1014429 (C4 (Web Text))

Max Activating Token Index: 388

Text #9

Max Range: 4.2303. Min Range: -4.2303

Max Act: 3.3082. Min Act: -0.0000

Data Index: 707469 (C4 (Web Text))

Max Activating Token Index: 769

Text #10

Max Range: 4.2303. Min Range: -4.2303

Max Act: 3.1869. Min Act: -0.0000

Data Index: 11282 (C4 (Web Text))

Max Activating Token Index: 16

Text #11

Max Range: 4.2303. Min Range: -4.2303

Max Act: 3.1806. Min Act: -0.0000

Data Index: 1052377 (C4 (Web Text))

Max Activating Token Index: 36

Text #12

Max Range: 4.2303. Min Range: -4.2303

Max Act: 2.8103. Min Act: -0.0000

Data Index: 371889 (C4 (Web Text))

Max Activating Token Index: 868

Text #13

Max Range: 4.2303. Min Range: -4.2303

Max Act: 2.9149. Min Act: -0.0000

Data Index: 535386 (C4 (Web Text))

Max Activating Token Index: 538

Text #14

Max Range: 4.2303. Min Range: -4.2303

Max Act: 2.8881. Min Act: -0.0000

Data Index: 1116461 (C4 (Web Text))

Max Activating Token Index: 389

Text #15

Max Range: 4.2303. Min Range: -4.2303

Max Act: 2.9000. Min Act: -0.0000

Data Index: 618422 (C4 (Web Text))

Max Activating Token Index: 238

Text #16

Max Range: 4.2303. Min Range: -4.2303

Max Act: 2.8702. Min Act: -0.0000

Data Index: 1092009 (C4 (Web Text))

Max Activating Token Index: 229

Text #17

Max Range: 4.2303. Min Range: -4.2303

Max Act: 2.7068. Min Act: -0.0000

Data Index: 1471436 (Python Code)

Max Activating Token Index: 777

Text #18

Max Range: 4.2303. Min Range: -4.2303

Max Act: 2.6213. Min Act: -0.0000

Data Index: 765770 (C4 (Web Text))

Max Activating Token Index: 141

Text #19

Max Range: 4.2303. Min Range: -4.2303

Max Act: 2.6059. Min Act: -0.0000

Data Index: 773931 (C4 (Web Text))

Max Activating Token Index: 55