Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3648 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.4670. Min Range: -0.4670

Max Act: 0.4670. Min Act: -0.0000

Data Index: 799844 (C4 (Web Text))

Max Activating Token Index: 648

Text #1

Max Range: 0.4670. Min Range: -0.4670

Max Act: 0.4325. Min Act: -0.0000

Data Index: 709250 (C4 (Web Text))

Max Activating Token Index: 147

Text #2

Max Range: 0.4670. Min Range: -0.4670

Max Act: 0.4121. Min Act: -0.0000

Data Index: 556695 (C4 (Web Text))

Max Activating Token Index: 639

Text #3

Max Range: 0.4670. Min Range: -0.4670

Max Act: 0.4234. Min Act: 0.0001

Data Index: 345959 (C4 (Web Text))

Max Activating Token Index: 288

Text #4

Max Range: 0.4670. Min Range: -0.4670

Max Act: 0.3243. Min Act: -0.0000

Data Index: 917931 (C4 (Web Text))

Max Activating Token Index: 272

Text #5

Max Range: 0.4670. Min Range: -0.4670

Max Act: 0.3734. Min Act: 0.0000

Data Index: 537688 (C4 (Web Text))

Max Activating Token Index: 312

Text #6

Max Range: 0.4670. Min Range: -0.4670

Max Act: 0.3726. Min Act: -0.0000

Data Index: 143580 (C4 (Web Text))

Max Activating Token Index: 531

Text #7

Max Range: 0.4670. Min Range: -0.4670

Max Act: 0.3443. Min Act: -0.0000

Data Index: 1316902 (C4 (Web Text))

Max Activating Token Index: 959

Text #8

Max Range: 0.4670. Min Range: -0.4670

Max Act: 0.3396. Min Act: -0.0000

Data Index: 154915 (C4 (Web Text))

Max Activating Token Index: 787

Text #9

Max Range: 0.4670. Min Range: -0.4670

Max Act: 0.3303. Min Act: 0.0001

Data Index: 695353 (C4 (Web Text))

Max Activating Token Index: 335

Text #10

Max Range: 0.4670. Min Range: -0.4670

Max Act: 0.3417. Min Act: -0.0000

Data Index: 228111 (C4 (Web Text))

Max Activating Token Index: 698

Text #11

Max Range: 0.4670. Min Range: -0.4670

Max Act: 0.3428. Min Act: -0.0000

Data Index: 975223 (C4 (Web Text))

Max Activating Token Index: 784

Text #12

Max Range: 0.4670. Min Range: -0.4670

Max Act: 0.3280. Min Act: -0.0000

Data Index: 1014904 (C4 (Web Text))

Max Activating Token Index: 780

Text #13

Max Range: 0.4670. Min Range: -0.4670

Max Act: 0.3478. Min Act: -0.0000

Data Index: 813966 (C4 (Web Text))

Max Activating Token Index: 789

Text #14

Max Range: 0.4670. Min Range: -0.4670

Max Act: 0.3373. Min Act: -0.0000

Data Index: 1199478 (C4 (Web Text))

Max Activating Token Index: 471

Text #15

Max Range: 0.4670. Min Range: -0.4670

Max Act: 0.3221. Min Act: -0.0000

Data Index: 635963 (C4 (Web Text))

Max Activating Token Index: 419

Text #16

Max Range: 0.4670. Min Range: -0.4670

Max Act: 0.3318. Min Act: 0.0001

Data Index: 13060 (C4 (Web Text))

Max Activating Token Index: 999

Text #17

Max Range: 0.4670. Min Range: -0.4670

Max Act: 0.3419. Min Act: -0.0000

Data Index: 580823 (C4 (Web Text))

Max Activating Token Index: 267

Text #18

Max Range: 0.4670. Min Range: -0.4670

Max Act: 0.3048. Min Act: -0.0000

Data Index: 1060815 (C4 (Web Text))

Max Activating Token Index: 859

Text #19

Max Range: 0.4670. Min Range: -0.4670

Max Act: 0.3269. Min Act: 0.0000

Data Index: 1071257 (C4 (Web Text))

Max Activating Token Index: 959