Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3603 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.0454. Min Range: -0.0454

Max Act: 0.0454. Min Act: -0.0003

Data Index: 1221341 (C4 (Web Text))

Max Activating Token Index: 475

Text #1

Max Range: 0.0454. Min Range: -0.0454

Max Act: 0.0362. Min Act: -0.0003

Data Index: 653716 (C4 (Web Text))

Max Activating Token Index: 43

Text #2

Max Range: 0.0454. Min Range: -0.0454

Max Act: 0.0354. Min Act: -0.0003

Data Index: 549539 (C4 (Web Text))

Max Activating Token Index: 437

Text #3

Max Range: 0.0454. Min Range: -0.0454

Max Act: 0.0374. Min Act: -0.0003

Data Index: 673235 (C4 (Web Text))

Max Activating Token Index: 516

Text #4

Max Range: 0.0454. Min Range: -0.0454

Max Act: 0.0340. Min Act: -0.0003

Data Index: 301694 (C4 (Web Text))

Max Activating Token Index: 895

Text #5

Max Range: 0.0454. Min Range: -0.0454

Max Act: 0.0331. Min Act: -0.0003

Data Index: 75426 (C4 (Web Text))

Max Activating Token Index: 680

Text #6

Max Range: 0.0454. Min Range: -0.0454

Max Act: 0.0321. Min Act: -0.0003

Data Index: 979902 (C4 (Web Text))

Max Activating Token Index: 698

Text #7

Max Range: 0.0454. Min Range: -0.0454

Max Act: 0.0315. Min Act: -0.0003

Data Index: 1121281 (C4 (Web Text))

Max Activating Token Index: 180

Text #8

Max Range: 0.0454. Min Range: -0.0454

Max Act: 0.0301. Min Act: -0.0003

Data Index: 7289 (C4 (Web Text))

Max Activating Token Index: 545

Text #9

Max Range: 0.0454. Min Range: -0.0454

Max Act: 0.0300. Min Act: -0.0003

Data Index: 308278 (C4 (Web Text))

Max Activating Token Index: 340

Text #10

Max Range: 0.0454. Min Range: -0.0454

Max Act: 0.0295. Min Act: -0.0002

Data Index: 52270 (C4 (Web Text))

Max Activating Token Index: 263

Text #11

Max Range: 0.0454. Min Range: -0.0454

Max Act: 0.0289. Min Act: -0.0003

Data Index: 979233 (C4 (Web Text))

Max Activating Token Index: 550

Text #12

Max Range: 0.0454. Min Range: -0.0454

Max Act: 0.0286. Min Act: -0.0003

Data Index: 1004614 (C4 (Web Text))

Max Activating Token Index: 402

Text #13

Max Range: 0.0454. Min Range: -0.0454

Max Act: 0.0276. Min Act: -0.0003

Data Index: 1008079 (C4 (Web Text))

Max Activating Token Index: 132

Text #14

Max Range: 0.0454. Min Range: -0.0454

Max Act: 0.0280. Min Act: -0.0003

Data Index: 495956 (C4 (Web Text))

Max Activating Token Index: 210

Text #15

Max Range: 0.0454. Min Range: -0.0454

Max Act: 0.0273. Min Act: -0.0003

Data Index: 1019365 (C4 (Web Text))

Max Activating Token Index: 12

Text #16

Max Range: 0.0454. Min Range: -0.0454

Max Act: 0.0272. Min Act: -0.0003

Data Index: 606053 (C4 (Web Text))

Max Activating Token Index: 440

Text #17

Max Range: 0.0454. Min Range: -0.0454

Max Act: 0.0282. Min Act: -0.0003

Data Index: 629966 (C4 (Web Text))

Max Activating Token Index: 520

Text #18

Max Range: 0.0454. Min Range: -0.0454

Max Act: 0.0276. Min Act: -0.0003

Data Index: 1261174 (C4 (Web Text))

Max Activating Token Index: 667

Text #19

Max Range: 0.0454. Min Range: -0.0454

Max Act: 0.0274. Min Act: -0.0003

Data Index: 85661 (C4 (Web Text))

Max Activating Token Index: 63