Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3717 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 5.7403. Min Range: -5.7403

Max Act: 5.6689. Min Act: -0.0000

Data Index: 208698 (C4 (Web Text))

Max Activating Token Index: 103

Text #1

Max Range: 5.7403. Min Range: -5.7403

Max Act: 5.6987. Min Act: -0.0000

Data Index: 1195143 (C4 (Web Text))

Max Activating Token Index: 185

Text #2

Max Range: 5.7403. Min Range: -5.7403

Max Act: 5.6928. Min Act: -0.0000

Data Index: 700336 (C4 (Web Text))

Max Activating Token Index: 186

Text #3

Max Range: 5.7403. Min Range: -5.7403

Max Act: 5.7403. Min Act: -0.0000

Data Index: 255400 (C4 (Web Text))

Max Activating Token Index: 220

Text #4

Max Range: 5.7403. Min Range: -5.7403

Max Act: 5.2177. Min Act: -0.0000

Data Index: 872269 (C4 (Web Text))

Max Activating Token Index: 480

Text #5

Max Range: 5.7403. Min Range: -5.7403

Max Act: 5.2357. Min Act: -0.0000

Data Index: 1039015 (C4 (Web Text))

Max Activating Token Index: 466

Text #6

Max Range: 5.7403. Min Range: -5.7403

Max Act: 5.3880. Min Act: -0.0000

Data Index: 966977 (C4 (Web Text))

Max Activating Token Index: 370

Text #7

Max Range: 5.7403. Min Range: -5.7403

Max Act: 5.4074. Min Act: -0.0000

Data Index: 385744 (C4 (Web Text))

Max Activating Token Index: 460

Text #8

Max Range: 5.7403. Min Range: -5.7403

Max Act: 5.2096. Min Act: -0.0000

Data Index: 49712 (C4 (Web Text))

Max Activating Token Index: 132

Text #9

Max Range: 5.7403. Min Range: -5.7403

Max Act: 5.2189. Min Act: -0.0000

Data Index: 489149 (C4 (Web Text))

Max Activating Token Index: 335

Text #10

Max Range: 5.7403. Min Range: -5.7403

Max Act: 5.3929. Min Act: -0.0000

Data Index: 296135 (C4 (Web Text))

Max Activating Token Index: 383

Text #11

Max Range: 5.7403. Min Range: -5.7403

Max Act: 5.3161. Min Act: -0.0000

Data Index: 34114 (C4 (Web Text))

Max Activating Token Index: 688

Text #12

Max Range: 5.7403. Min Range: -5.7403

Max Act: 5.2769. Min Act: -0.0000

Data Index: 489918 (C4 (Web Text))

Max Activating Token Index: 112

Text #13

Max Range: 5.7403. Min Range: -5.7403

Max Act: 5.2337. Min Act: -0.0000

Data Index: 1135500 (C4 (Web Text))

Max Activating Token Index: 408

Text #14

Max Range: 5.7403. Min Range: -5.7403

Max Act: 5.2324. Min Act: -0.0000

Data Index: 1083444 (C4 (Web Text))

Max Activating Token Index: 539

Text #15

Max Range: 5.7403. Min Range: -5.7403

Max Act: 5.0478. Min Act: -0.0000

Data Index: 1348691 (C4 (Web Text))

Max Activating Token Index: 346

Text #16

Max Range: 5.7403. Min Range: -5.7403

Max Act: 5.2077. Min Act: -0.0000

Data Index: 1130307 (C4 (Web Text))

Max Activating Token Index: 396

Text #17

Max Range: 5.7403. Min Range: -5.7403

Max Act: 5.1983. Min Act: -0.0000

Data Index: 857370 (C4 (Web Text))

Max Activating Token Index: 325

Text #18

Max Range: 5.7403. Min Range: -5.7403

Max Act: 5.1940. Min Act: -0.0000

Data Index: 409270 (C4 (Web Text))

Max Activating Token Index: 673

Text #19

Max Range: 5.7403. Min Range: -5.7403

Max Act: 5.1896. Min Act: -0.0000

Data Index: 1023780 (C4 (Web Text))

Max Activating Token Index: 631