Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1853 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.3452. Min Range: -0.3452

Max Act: 0.3278. Min Act: -0.0000

Data Index: 49667 (C4 (Web Text))

Max Activating Token Index: 118

Text #1

Max Range: 0.3452. Min Range: -0.3452

Max Act: 0.3452. Min Act: -0.0000

Data Index: 11279 (C4 (Web Text))

Max Activating Token Index: 438

Text #2

Max Range: 0.3452. Min Range: -0.3452

Max Act: 0.3353. Min Act: -0.0000

Data Index: 543896 (C4 (Web Text))

Max Activating Token Index: 935

Text #3

Max Range: 0.3452. Min Range: -0.3452

Max Act: 0.3323. Min Act: -0.0000

Data Index: 454549 (C4 (Web Text))

Max Activating Token Index: 382

Text #4

Max Range: 0.3452. Min Range: -0.3452

Max Act: 0.3354. Min Act: -0.0000

Data Index: 798820 (C4 (Web Text))

Max Activating Token Index: 21

Text #5

Max Range: 0.3452. Min Range: -0.3452

Max Act: 0.3061. Min Act: -0.0000

Data Index: 819391 (C4 (Web Text))

Max Activating Token Index: 478

Text #6

Max Range: 0.3452. Min Range: -0.3452

Max Act: 0.3046. Min Act: -0.0000

Data Index: 649665 (C4 (Web Text))

Max Activating Token Index: 255

Text #7

Max Range: 0.3452. Min Range: -0.3452

Max Act: 0.2892. Min Act: -0.0000

Data Index: 258148 (C4 (Web Text))

Max Activating Token Index: 234

Text #8

Max Range: 0.3452. Min Range: -0.3452

Max Act: 0.2829. Min Act: -0.0000

Data Index: 43415 (C4 (Web Text))

Max Activating Token Index: 971

Text #9

Max Range: 0.3452. Min Range: -0.3452

Max Act: 0.2749. Min Act: -0.0000

Data Index: 507705 (C4 (Web Text))

Max Activating Token Index: 213

Text #10

Max Range: 0.3452. Min Range: -0.3452

Max Act: 0.2731. Min Act: -0.0000

Data Index: 857431 (C4 (Web Text))

Max Activating Token Index: 868

Text #11

Max Range: 0.3452. Min Range: -0.3452

Max Act: 0.2739. Min Act: -0.0000

Data Index: 507774 (C4 (Web Text))

Max Activating Token Index: 79

Text #12

Max Range: 0.3452. Min Range: -0.3452

Max Act: 0.3235. Min Act: -0.0000

Data Index: 851509 (C4 (Web Text))

Max Activating Token Index: 144

Text #13

Max Range: 0.3452. Min Range: -0.3452

Max Act: 0.2528. Min Act: -0.0000

Data Index: 775871 (C4 (Web Text))

Max Activating Token Index: 243

Text #14

Max Range: 0.3452. Min Range: -0.3452

Max Act: 0.2764. Min Act: -0.0000

Data Index: 457047 (C4 (Web Text))

Max Activating Token Index: 268

Text #15

Max Range: 0.3452. Min Range: -0.3452

Max Act: 0.2659. Min Act: -0.0000

Data Index: 961380 (C4 (Web Text))

Max Activating Token Index: 583

Text #16

Max Range: 0.3452. Min Range: -0.3452

Max Act: 0.2660. Min Act: -0.0000

Data Index: 490855 (C4 (Web Text))

Max Activating Token Index: 50

Text #17

Max Range: 0.3452. Min Range: -0.3452

Max Act: 0.2739. Min Act: -0.0000

Data Index: 1325798 (C4 (Web Text))

Max Activating Token Index: 110

Text #18

Max Range: 0.3452. Min Range: -0.3452

Max Act: 0.2549. Min Act: -0.0000

Data Index: 268431 (C4 (Web Text))

Max Activating Token Index: 726

Text #19

Max Range: 0.3452. Min Range: -0.3452

Max Act: 0.2524. Min Act: -0.0000

Data Index: 156719 (C4 (Web Text))

Max Activating Token Index: 26