Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3722 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.8318. Min Range: -1.8318

Max Act: 1.8318. Min Act: -0.0000

Data Index: 38297 (C4 (Web Text))

Max Activating Token Index: 500

Text #1

Max Range: 1.8318. Min Range: -1.8318

Max Act: 1.6716. Min Act: -0.0000

Data Index: 1278136 (C4 (Web Text))

Max Activating Token Index: 258

Text #2

Max Range: 1.8318. Min Range: -1.8318

Max Act: 1.2575. Min Act: -0.0000

Data Index: 826821 (C4 (Web Text))

Max Activating Token Index: 329

Text #3

Max Range: 1.8318. Min Range: -1.8318

Max Act: 1.1149. Min Act: -0.0000

Data Index: 1228352 (C4 (Web Text))

Max Activating Token Index: 296

Text #4

Max Range: 1.8318. Min Range: -1.8318

Max Act: 1.1206. Min Act: -0.0000

Data Index: 1280642 (C4 (Web Text))

Max Activating Token Index: 315

Text #5

Max Range: 1.8318. Min Range: -1.8318

Max Act: 1.0402. Min Act: -0.0000

Data Index: 917653 (C4 (Web Text))

Max Activating Token Index: 381

Text #6

Max Range: 1.8318. Min Range: -1.8318

Max Act: 0.9545. Min Act: -0.0000

Data Index: 955966 (C4 (Web Text))

Max Activating Token Index: 501

Text #7

Max Range: 1.8318. Min Range: -1.8318

Max Act: 0.9300. Min Act: -0.0000

Data Index: 745898 (C4 (Web Text))

Max Activating Token Index: 569

Text #8

Max Range: 1.8318. Min Range: -1.8318

Max Act: 0.9392. Min Act: -0.0000

Data Index: 595631 (C4 (Web Text))

Max Activating Token Index: 111

Text #9

Max Range: 1.8318. Min Range: -1.8318

Max Act: 0.9250. Min Act: -0.0000

Data Index: 738570 (C4 (Web Text))

Max Activating Token Index: 372

Text #10

Max Range: 1.8318. Min Range: -1.8318

Max Act: 0.8880. Min Act: -0.0000

Data Index: 234809 (C4 (Web Text))

Max Activating Token Index: 431

Text #11

Max Range: 1.8318. Min Range: -1.8318

Max Act: 0.8437. Min Act: -0.0000

Data Index: 897574 (C4 (Web Text))

Max Activating Token Index: 409

Text #12

Max Range: 1.8318. Min Range: -1.8318

Max Act: 0.8699. Min Act: -0.0000

Data Index: 251938 (C4 (Web Text))

Max Activating Token Index: 392

Text #13

Max Range: 1.8318. Min Range: -1.8318

Max Act: 0.8657. Min Act: -0.0000

Data Index: 1030529 (C4 (Web Text))

Max Activating Token Index: 101

Text #14

Max Range: 1.8318. Min Range: -1.8318

Max Act: 0.8358. Min Act: -0.0000

Data Index: 33676 (C4 (Web Text))

Max Activating Token Index: 56

Text #15

Max Range: 1.8318. Min Range: -1.8318

Max Act: 0.8377. Min Act: -0.0000

Data Index: 1359519 (C4 (Web Text))

Max Activating Token Index: 738

Text #16

Max Range: 1.8318. Min Range: -1.8318

Max Act: 0.7878. Min Act: -0.0000

Data Index: 871257 (C4 (Web Text))

Max Activating Token Index: 129

Text #17

Max Range: 1.8318. Min Range: -1.8318

Max Act: 0.8812. Min Act: -0.0000

Data Index: 884380 (C4 (Web Text))

Max Activating Token Index: 718

Text #18

Max Range: 1.8318. Min Range: -1.8318

Max Act: 0.8125. Min Act: -0.0000

Data Index: 235438 (C4 (Web Text))

Max Activating Token Index: 159

Text #19

Max Range: 1.8318. Min Range: -1.8318

Max Act: 0.7910. Min Act: -0.0000

Data Index: 115445 (C4 (Web Text))

Max Activating Token Index: 735