Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2942 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.7348. Min Range: -0.7348

Max Act: 0.7208. Min Act: -0.0003

Data Index: 971338 (C4 (Web Text))

Max Activating Token Index: 627

Text #1

Max Range: 0.7348. Min Range: -0.7348

Max Act: 0.7348. Min Act: -0.0002

Data Index: 890445 (C4 (Web Text))

Max Activating Token Index: 348

Text #2

Max Range: 0.7348. Min Range: -0.7348

Max Act: 0.7278. Min Act: -0.0002

Data Index: 622147 (C4 (Web Text))

Max Activating Token Index: 561

Text #3

Max Range: 0.7348. Min Range: -0.7348

Max Act: 0.6837. Min Act: -0.0003

Data Index: 492609 (C4 (Web Text))

Max Activating Token Index: 571

Text #4

Max Range: 0.7348. Min Range: -0.7348

Max Act: 0.6708. Min Act: -0.0003

Data Index: 696648 (C4 (Web Text))

Max Activating Token Index: 375

Text #5

Max Range: 0.7348. Min Range: -0.7348

Max Act: 0.6986. Min Act: -0.0003

Data Index: 1245314 (C4 (Web Text))

Max Activating Token Index: 264

Text #6

Max Range: 0.7348. Min Range: -0.7348

Max Act: 0.6957. Min Act: -0.0002

Data Index: 1048686 (C4 (Web Text))

Max Activating Token Index: 544

Text #7

Max Range: 0.7348. Min Range: -0.7348

Max Act: 0.6925. Min Act: -0.0002

Data Index: 439130 (C4 (Web Text))

Max Activating Token Index: 524

Text #8

Max Range: 0.7348. Min Range: -0.7348

Max Act: 0.6687. Min Act: -0.0003

Data Index: 801573 (C4 (Web Text))

Max Activating Token Index: 332

Text #9

Max Range: 0.7348. Min Range: -0.7348

Max Act: 0.6632. Min Act: -0.0003

Data Index: 258680 (C4 (Web Text))

Max Activating Token Index: 491

Text #10

Max Range: 0.7348. Min Range: -0.7348

Max Act: 0.6615. Min Act: -0.0002

Data Index: 589225 (C4 (Web Text))

Max Activating Token Index: 849

Text #11

Max Range: 0.7348. Min Range: -0.7348

Max Act: 0.6601. Min Act: -0.0003

Data Index: 1318637 (C4 (Web Text))

Max Activating Token Index: 213

Text #12

Max Range: 0.7348. Min Range: -0.7348

Max Act: 0.6495. Min Act: -0.0003

Data Index: 270380 (C4 (Web Text))

Max Activating Token Index: 800

Text #13

Max Range: 0.7348. Min Range: -0.7348

Max Act: 0.6459. Min Act: -0.0003

Data Index: 523020 (C4 (Web Text))

Max Activating Token Index: 160

Text #14

Max Range: 0.7348. Min Range: -0.7348

Max Act: 0.6420. Min Act: -0.0003

Data Index: 289809 (C4 (Web Text))

Max Activating Token Index: 376

Text #15

Max Range: 0.7348. Min Range: -0.7348

Max Act: 0.6182. Min Act: -0.0002

Data Index: 1171677 (C4 (Web Text))

Max Activating Token Index: 275

Text #16

Max Range: 0.7348. Min Range: -0.7348

Max Act: 0.6401. Min Act: -0.0003

Data Index: 1334281 (C4 (Web Text))

Max Activating Token Index: 327

Text #17

Max Range: 0.7348. Min Range: -0.7348

Max Act: 0.6548. Min Act: -0.0003

Data Index: 839394 (C4 (Web Text))

Max Activating Token Index: 605

Text #18

Max Range: 0.7348. Min Range: -0.7348

Max Act: 0.6354. Min Act: -0.0003

Data Index: 1349130 (C4 (Web Text))

Max Activating Token Index: 764

Text #19

Max Range: 0.7348. Min Range: -0.7348

Max Act: 0.6377. Min Act: -0.0003

Data Index: 216887 (C4 (Web Text))

Max Activating Token Index: 281