Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2442 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.5047. Min Range: -2.5047

Max Act: 2.5047. Min Act: -0.0000

Data Index: 375335 (C4 (Web Text))

Max Activating Token Index: 182

Text #1

Max Range: 2.5047. Min Range: -2.5047

Max Act: 2.1193. Min Act: -0.0000

Data Index: 1177394 (C4 (Web Text))

Max Activating Token Index: 993

Text #2

Max Range: 2.5047. Min Range: -2.5047

Max Act: 1.9865. Min Act: -0.0000

Data Index: 343017 (C4 (Web Text))

Max Activating Token Index: 601

Text #3

Max Range: 2.5047. Min Range: -2.5047

Max Act: 2.0096. Min Act: -0.0000

Data Index: 1345355 (C4 (Web Text))

Max Activating Token Index: 655

Text #4

Max Range: 2.5047. Min Range: -2.5047

Max Act: 1.8242. Min Act: -0.0000

Data Index: 855590 (C4 (Web Text))

Max Activating Token Index: 785

Text #5

Max Range: 2.5047. Min Range: -2.5047

Max Act: 1.9136. Min Act: -0.0000

Data Index: 584438 (C4 (Web Text))

Max Activating Token Index: 488

Text #6

Max Range: 2.5047. Min Range: -2.5047

Max Act: 1.9098. Min Act: -0.0000

Data Index: 418419 (C4 (Web Text))

Max Activating Token Index: 408

Text #7

Max Range: 2.5047. Min Range: -2.5047

Max Act: 1.7860. Min Act: -0.0000

Data Index: 424046 (C4 (Web Text))

Max Activating Token Index: 575

Text #8

Max Range: 2.5047. Min Range: -2.5047

Max Act: 1.8763. Min Act: -0.0000

Data Index: 1014931 (C4 (Web Text))

Max Activating Token Index: 95

Text #9

Max Range: 2.5047. Min Range: -2.5047

Max Act: 1.8533. Min Act: -0.0000

Data Index: 688996 (C4 (Web Text))

Max Activating Token Index: 582

Text #10

Max Range: 2.5047. Min Range: -2.5047

Max Act: 1.8562. Min Act: -0.0000

Data Index: 896894 (C4 (Web Text))

Max Activating Token Index: 577

Text #11

Max Range: 2.5047. Min Range: -2.5047

Max Act: 1.8320. Min Act: -0.0000

Data Index: 1279096 (C4 (Web Text))

Max Activating Token Index: 235

Text #12

Max Range: 2.5047. Min Range: -2.5047

Max Act: 1.8287. Min Act: -0.0000

Data Index: 3402 (C4 (Web Text))

Max Activating Token Index: 441

Text #13

Max Range: 2.5047. Min Range: -2.5047

Max Act: 1.8177. Min Act: 0.0000

Data Index: 667801 (C4 (Web Text))

Max Activating Token Index: 769

Text #14

Max Range: 2.5047. Min Range: -2.5047

Max Act: 1.7947. Min Act: -0.0000

Data Index: 334381 (C4 (Web Text))

Max Activating Token Index: 632

Text #15

Max Range: 2.5047. Min Range: -2.5047

Max Act: 1.7886. Min Act: -0.0000

Data Index: 71322 (C4 (Web Text))

Max Activating Token Index: 215

Text #16

Max Range: 2.5047. Min Range: -2.5047

Max Act: 1.7924. Min Act: -0.0000

Data Index: 563169 (C4 (Web Text))

Max Activating Token Index: 668

Text #17

Max Range: 2.5047. Min Range: -2.5047

Max Act: 1.7816. Min Act: -0.0000

Data Index: 645749 (C4 (Web Text))

Max Activating Token Index: 67

Text #18

Max Range: 2.5047. Min Range: -2.5047

Max Act: 1.7486. Min Act: -0.0000

Data Index: 1194094 (C4 (Web Text))

Max Activating Token Index: 498

Text #19

Max Range: 2.5047. Min Range: -2.5047

Max Act: 1.7256. Min Act: -0.0000

Data Index: 1239362 (C4 (Web Text))

Max Activating Token Index: 111