Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4478 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.3333. Min Range: -0.3333

Max Act: 0.3333. Min Act: -0.0002

Data Index: 679262 (C4 (Web Text))

Max Activating Token Index: 26

Text #1

Max Range: 0.3333. Min Range: -0.3333

Max Act: 0.3117. Min Act: -0.0002

Data Index: 778625 (C4 (Web Text))

Max Activating Token Index: 25

Text #2

Max Range: 0.3333. Min Range: -0.3333

Max Act: 0.2984. Min Act: -0.0002

Data Index: 490561 (C4 (Web Text))

Max Activating Token Index: 34

Text #3

Max Range: 0.3333. Min Range: -0.3333

Max Act: 0.2539. Min Act: -0.0002

Data Index: 74184 (C4 (Web Text))

Max Activating Token Index: 816

Text #4

Max Range: 0.3333. Min Range: -0.3333

Max Act: 0.2905. Min Act: -0.0002

Data Index: 1376250 (Python Code)

Max Activating Token Index: 595

Text #5

Max Range: 0.3333. Min Range: -0.3333

Max Act: 0.2823. Min Act: -0.0002

Data Index: 1447650 (Python Code)

Max Activating Token Index: 62

Text #6

Max Range: 0.3333. Min Range: -0.3333

Max Act: 0.2791. Min Act: -0.0002

Data Index: 778868 (C4 (Web Text))

Max Activating Token Index: 25

Text #7

Max Range: 0.3333. Min Range: -0.3333

Max Act: 0.2432. Min Act: -0.0002

Data Index: 1496964 (Python Code)

Max Activating Token Index: 690

Text #8

Max Range: 0.3333. Min Range: -0.3333

Max Act: 0.2423. Min Act: -0.0002

Data Index: 1209249 (C4 (Web Text))

Max Activating Token Index: 38

Text #9

Max Range: 0.3333. Min Range: -0.3333

Max Act: 0.2225. Min Act: -0.0002

Data Index: 555112 (C4 (Web Text))

Max Activating Token Index: 181

Text #10

Max Range: 0.3333. Min Range: -0.3333

Max Act: 0.2641. Min Act: -0.0002

Data Index: 1399848 (Python Code)

Max Activating Token Index: 618

Text #11

Max Range: 0.3333. Min Range: -0.3333

Max Act: 0.3117. Min Act: -0.0002

Data Index: 1569897 (Python Code)

Max Activating Token Index: 133

Text #12

Max Range: 0.3333. Min Range: -0.3333

Max Act: 0.2407. Min Act: -0.0002

Data Index: 1448533 (Python Code)

Max Activating Token Index: 187

Text #13

Max Range: 0.3333. Min Range: -0.3333

Max Act: 0.2516. Min Act: -0.0002

Data Index: 981515 (C4 (Web Text))

Max Activating Token Index: 34

Text #14

Max Range: 0.3333. Min Range: -0.3333

Max Act: 0.2588. Min Act: -0.0002

Data Index: 1496167 (Python Code)

Max Activating Token Index: 177

Text #15

Max Range: 0.3333. Min Range: -0.3333

Max Act: 0.2523. Min Act: -0.0002

Data Index: 351273 (C4 (Web Text))

Max Activating Token Index: 88

Text #16

Max Range: 0.3333. Min Range: -0.3333

Max Act: 0.3054. Min Act: -0.0002

Data Index: 1114419 (C4 (Web Text))

Max Activating Token Index: 39

Text #17

Max Range: 0.3333. Min Range: -0.3333

Max Act: 0.2353. Min Act: -0.0002

Data Index: 1237119 (C4 (Web Text))

Max Activating Token Index: 734

Text #18

Max Range: 0.3333. Min Range: -0.3333

Max Act: 0.2391. Min Act: -0.0002

Data Index: 483883 (C4 (Web Text))

Max Activating Token Index: 59

Text #19

Max Range: 0.3333. Min Range: -0.3333

Max Act: 0.2491. Min Act: -0.0002

Data Index: 1620579 (Python Code)

Max Activating Token Index: 397