Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4667 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.4684. Min Range: -0.4684

Max Act: 0.4475. Min Act: -0.0000

Data Index: 199515 (C4 (Web Text))

Max Activating Token Index: 304

Text #1

Max Range: 0.4684. Min Range: -0.4684

Max Act: 0.4382. Min Act: -0.0000

Data Index: 6326 (C4 (Web Text))

Max Activating Token Index: 586

Text #2

Max Range: 0.4684. Min Range: -0.4684

Max Act: 0.4193. Min Act: -0.0000

Data Index: 701124 (C4 (Web Text))

Max Activating Token Index: 816

Text #3

Max Range: 0.4684. Min Range: -0.4684

Max Act: 0.4684. Min Act: -0.0000

Data Index: 543778 (C4 (Web Text))

Max Activating Token Index: 876

Text #4

Max Range: 0.4684. Min Range: -0.4684

Max Act: 0.4209. Min Act: -0.0000

Data Index: 317233 (C4 (Web Text))

Max Activating Token Index: 33

Text #5

Max Range: 0.4684. Min Range: -0.4684

Max Act: 0.4181. Min Act: -0.0000

Data Index: 796011 (C4 (Web Text))

Max Activating Token Index: 309

Text #6

Max Range: 0.4684. Min Range: -0.4684

Max Act: 0.4282. Min Act: -0.0000

Data Index: 185002 (C4 (Web Text))

Max Activating Token Index: 381

Text #7

Max Range: 0.4684. Min Range: -0.4684

Max Act: 0.4056. Min Act: -0.0000

Data Index: 510919 (C4 (Web Text))

Max Activating Token Index: 327

Text #8

Max Range: 0.4684. Min Range: -0.4684

Max Act: 0.3834. Min Act: -0.0000

Data Index: 1092346 (C4 (Web Text))

Max Activating Token Index: 181

Text #9

Max Range: 0.4684. Min Range: -0.4684

Max Act: 0.4205. Min Act: -0.0000

Data Index: 479678 (C4 (Web Text))

Max Activating Token Index: 90

Text #10

Max Range: 0.4684. Min Range: -0.4684

Max Act: 0.3881. Min Act: -0.0000

Data Index: 857119 (C4 (Web Text))

Max Activating Token Index: 318

Text #11

Max Range: 0.4684. Min Range: -0.4684

Max Act: 0.3723. Min Act: -0.0000

Data Index: 1293431 (C4 (Web Text))

Max Activating Token Index: 253

Text #12

Max Range: 0.4684. Min Range: -0.4684

Max Act: 0.3809. Min Act: -0.0000

Data Index: 779634 (C4 (Web Text))

Max Activating Token Index: 1009

Text #13

Max Range: 0.4684. Min Range: -0.4684

Max Act: 0.3593. Min Act: -0.0000

Data Index: 747452 (C4 (Web Text))

Max Activating Token Index: 47

Text #14

Max Range: 0.4684. Min Range: -0.4684

Max Act: 0.3694. Min Act: -0.0000

Data Index: 533466 (C4 (Web Text))

Max Activating Token Index: 993

Text #15

Max Range: 0.4684. Min Range: -0.4684

Max Act: 0.3790. Min Act: -0.0000

Data Index: 1344372 (C4 (Web Text))

Max Activating Token Index: 714

Text #16

Max Range: 0.4684. Min Range: -0.4684

Max Act: 0.3797. Min Act: -0.0000

Data Index: 648513 (C4 (Web Text))

Max Activating Token Index: 89

Text #17

Max Range: 0.4684. Min Range: -0.4684

Max Act: 0.3781. Min Act: -0.0000

Data Index: 403315 (C4 (Web Text))

Max Activating Token Index: 421

Text #18

Max Range: 0.4684. Min Range: -0.4684

Max Act: 0.3866. Min Act: -0.0000

Data Index: 1298270 (C4 (Web Text))

Max Activating Token Index: 545

Text #19

Max Range: 0.4684. Min Range: -0.4684

Max Act: 0.3747. Min Act: -0.0000

Data Index: 130287 (C4 (Web Text))

Max Activating Token Index: 1002