Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 148 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.0104. Min Range: -1.0104

Max Act: 1.0104. Min Act: -0.0002

Data Index: 408154 (C4 (Web Text))

Max Activating Token Index: 20

Text #1

Max Range: 1.0104. Min Range: -1.0104

Max Act: 0.9583. Min Act: -0.0002

Data Index: 504046 (C4 (Web Text))

Max Activating Token Index: 748

Text #2

Max Range: 1.0104. Min Range: -1.0104

Max Act: 0.9907. Min Act: -0.0002

Data Index: 683465 (C4 (Web Text))

Max Activating Token Index: 46

Text #3

Max Range: 1.0104. Min Range: -1.0104

Max Act: 0.9355. Min Act: -0.0002

Data Index: 1072878 (C4 (Web Text))

Max Activating Token Index: 41

Text #4

Max Range: 1.0104. Min Range: -1.0104

Max Act: 0.9351. Min Act: -0.0002

Data Index: 1055966 (C4 (Web Text))

Max Activating Token Index: 13

Text #5

Max Range: 1.0104. Min Range: -1.0104

Max Act: 0.9562. Min Act: -0.0002

Data Index: 731226 (C4 (Web Text))

Max Activating Token Index: 10

Text #6

Max Range: 1.0104. Min Range: -1.0104

Max Act: 0.9030. Min Act: -0.0002

Data Index: 153026 (C4 (Web Text))

Max Activating Token Index: 338

Text #7

Max Range: 1.0104. Min Range: -1.0104

Max Act: 0.9641. Min Act: -0.0002

Data Index: 485491 (C4 (Web Text))

Max Activating Token Index: 17

Text #8

Max Range: 1.0104. Min Range: -1.0104

Max Act: 0.9324. Min Act: -0.0002

Data Index: 1237160 (C4 (Web Text))

Max Activating Token Index: 960

Text #9

Max Range: 1.0104. Min Range: -1.0104

Max Act: 0.8974. Min Act: -0.0002

Data Index: 89118 (C4 (Web Text))

Max Activating Token Index: 61

Text #10

Max Range: 1.0104. Min Range: -1.0104

Max Act: 0.8301. Min Act: -0.0002

Data Index: 907251 (C4 (Web Text))

Max Activating Token Index: 172

Text #11

Max Range: 1.0104. Min Range: -1.0104

Max Act: 0.8291. Min Act: -0.0002

Data Index: 1189229 (C4 (Web Text))

Max Activating Token Index: 120

Text #12

Max Range: 1.0104. Min Range: -1.0104

Max Act: 0.8778. Min Act: -0.0002

Data Index: 863868 (C4 (Web Text))

Max Activating Token Index: 44

Text #13

Max Range: 1.0104. Min Range: -1.0104

Max Act: 0.9645. Min Act: -0.0002

Data Index: 325971 (C4 (Web Text))

Max Activating Token Index: 300

Text #14

Max Range: 1.0104. Min Range: -1.0104

Max Act: 0.8042. Min Act: -0.0002

Data Index: 1312233 (C4 (Web Text))

Max Activating Token Index: 143

Text #15

Max Range: 1.0104. Min Range: -1.0104

Max Act: 0.8234. Min Act: -0.0002

Data Index: 134698 (C4 (Web Text))

Max Activating Token Index: 743

Text #16

Max Range: 1.0104. Min Range: -1.0104

Max Act: 0.8753. Min Act: -0.0002

Data Index: 1233064 (C4 (Web Text))

Max Activating Token Index: 49

Text #17

Max Range: 1.0104. Min Range: -1.0104

Max Act: 0.8078. Min Act: -0.0002

Data Index: 99914 (C4 (Web Text))

Max Activating Token Index: 252

Text #18

Max Range: 1.0104. Min Range: -1.0104

Max Act: 0.8150. Min Act: -0.0002

Data Index: 694074 (C4 (Web Text))

Max Activating Token Index: 710

Text #19

Max Range: 1.0104. Min Range: -1.0104

Max Act: 0.8137. Min Act: -0.0002

Data Index: 1066014 (C4 (Web Text))

Max Activating Token Index: 188