Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1385 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.8362. Min Range: -0.8362

Max Act: 0.8362. Min Act: -0.0000

Data Index: 347857 (C4 (Web Text))

Max Activating Token Index: 576

Text #1

Max Range: 0.8362. Min Range: -0.8362

Max Act: 0.8074. Min Act: -0.0000

Data Index: 792437 (C4 (Web Text))

Max Activating Token Index: 676

Text #2

Max Range: 0.8362. Min Range: -0.8362

Max Act: 0.8086. Min Act: -0.0000

Data Index: 856896 (C4 (Web Text))

Max Activating Token Index: 766

Text #3

Max Range: 0.8362. Min Range: -0.8362

Max Act: 0.6544. Min Act: -0.0000

Data Index: 572810 (C4 (Web Text))

Max Activating Token Index: 759

Text #4

Max Range: 0.8362. Min Range: -0.8362

Max Act: 0.6577. Min Act: -0.0000

Data Index: 564657 (C4 (Web Text))

Max Activating Token Index: 180

Text #5

Max Range: 0.8362. Min Range: -0.8362

Max Act: 0.7021. Min Act: -0.0000

Data Index: 61777 (C4 (Web Text))

Max Activating Token Index: 467

Text #6

Max Range: 0.8362. Min Range: -0.8362

Max Act: 0.6651. Min Act: -0.0000

Data Index: 804380 (C4 (Web Text))

Max Activating Token Index: 993

Text #7

Max Range: 0.8362. Min Range: -0.8362

Max Act: 0.6783. Min Act: -0.0000

Data Index: 385743 (C4 (Web Text))

Max Activating Token Index: 514

Text #8

Max Range: 0.8362. Min Range: -0.8362

Max Act: 0.6250. Min Act: -0.0000

Data Index: 1121642 (C4 (Web Text))

Max Activating Token Index: 423

Text #9

Max Range: 0.8362. Min Range: -0.8362

Max Act: 0.5905. Min Act: -0.0000

Data Index: 1311237 (C4 (Web Text))

Max Activating Token Index: 441

Text #10

Max Range: 0.8362. Min Range: -0.8362

Max Act: 0.6022. Min Act: -0.0000

Data Index: 478329 (C4 (Web Text))

Max Activating Token Index: 179

Text #11

Max Range: 0.8362. Min Range: -0.8362

Max Act: 0.5858. Min Act: -0.0000

Data Index: 41821 (C4 (Web Text))

Max Activating Token Index: 991

Text #12

Max Range: 0.8362. Min Range: -0.8362

Max Act: 0.5907. Min Act: -0.0000

Data Index: 1244553 (C4 (Web Text))

Max Activating Token Index: 177

Text #13

Max Range: 0.8362. Min Range: -0.8362

Max Act: 0.5784. Min Act: -0.0000

Data Index: 1356443 (C4 (Web Text))

Max Activating Token Index: 869

Text #14

Max Range: 0.8362. Min Range: -0.8362

Max Act: 0.5597. Min Act: -0.0000

Data Index: 308803 (C4 (Web Text))

Max Activating Token Index: 474

Text #15

Max Range: 0.8362. Min Range: -0.8362

Max Act: 0.5272. Min Act: -0.0000

Data Index: 632527 (C4 (Web Text))

Max Activating Token Index: 155

Text #16

Max Range: 0.8362. Min Range: -0.8362

Max Act: 0.5450. Min Act: -0.0000

Data Index: 1241550 (C4 (Web Text))

Max Activating Token Index: 923

Text #17

Max Range: 0.8362. Min Range: -0.8362

Max Act: 0.5420. Min Act: -0.0000

Data Index: 388437 (C4 (Web Text))

Max Activating Token Index: 207

Text #18

Max Range: 0.8362. Min Range: -0.8362

Max Act: 0.5415. Min Act: -0.0000

Data Index: 1187580 (C4 (Web Text))

Max Activating Token Index: 683

Text #19

Max Range: 0.8362. Min Range: -0.8362

Max Act: 0.5515. Min Act: -0.0000

Data Index: 914215 (C4 (Web Text))

Max Activating Token Index: 767