Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 418 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 3.1288. Min Range: -3.1288

Max Act: 3.1288. Min Act: -0.0002

Data Index: 746553 (C4 (Web Text))

Max Activating Token Index: 37

Text #1

Max Range: 3.1288. Min Range: -3.1288

Max Act: 2.8276. Min Act: -0.0002

Data Index: 275662 (C4 (Web Text))

Max Activating Token Index: 402

Text #2

Max Range: 3.1288. Min Range: -3.1288

Max Act: 2.7332. Min Act: -0.0002

Data Index: 472480 (C4 (Web Text))

Max Activating Token Index: 929

Text #3

Max Range: 3.1288. Min Range: -3.1288

Max Act: 2.6703. Min Act: -0.0002

Data Index: 564814 (C4 (Web Text))

Max Activating Token Index: 41

Text #4

Max Range: 3.1288. Min Range: -3.1288

Max Act: 2.4415. Min Act: -0.0002

Data Index: 455067 (C4 (Web Text))

Max Activating Token Index: 112

Text #5

Max Range: 3.1288. Min Range: -3.1288

Max Act: 2.3997. Min Act: -0.0002

Data Index: 587172 (C4 (Web Text))

Max Activating Token Index: 77

Text #6

Max Range: 3.1288. Min Range: -3.1288

Max Act: 2.2786. Min Act: -0.0002

Data Index: 1223293 (C4 (Web Text))

Max Activating Token Index: 331

Text #7

Max Range: 3.1288. Min Range: -3.1288

Max Act: 2.3219. Min Act: -0.0002

Data Index: 1187283 (C4 (Web Text))

Max Activating Token Index: 550

Text #8

Max Range: 3.1288. Min Range: -3.1288

Max Act: 2.2584. Min Act: -0.0002

Data Index: 222941 (C4 (Web Text))

Max Activating Token Index: 861

Text #9

Max Range: 3.1288. Min Range: -3.1288

Max Act: 2.2179. Min Act: -0.0003

Data Index: 1077864 (C4 (Web Text))

Max Activating Token Index: 848

Text #10

Max Range: 3.1288. Min Range: -3.1288

Max Act: 2.1604. Min Act: -0.0002

Data Index: 504872 (C4 (Web Text))

Max Activating Token Index: 319

Text #11

Max Range: 3.1288. Min Range: -3.1288

Max Act: 2.2216. Min Act: -0.0002

Data Index: 268513 (C4 (Web Text))

Max Activating Token Index: 61

Text #12

Max Range: 3.1288. Min Range: -3.1288

Max Act: 2.1386. Min Act: -0.0002

Data Index: 48804 (C4 (Web Text))

Max Activating Token Index: 259

Text #13

Max Range: 3.1288. Min Range: -3.1288

Max Act: 2.1185. Min Act: -0.0002

Data Index: 1190797 (C4 (Web Text))

Max Activating Token Index: 266

Text #14

Max Range: 3.1288. Min Range: -3.1288

Max Act: 2.1493. Min Act: -0.0002

Data Index: 190286 (C4 (Web Text))

Max Activating Token Index: 123

Text #15

Max Range: 3.1288. Min Range: -3.1288

Max Act: 2.1317. Min Act: -0.0002

Data Index: 515044 (C4 (Web Text))

Max Activating Token Index: 252

Text #16

Max Range: 3.1288. Min Range: -3.1288

Max Act: 2.1268. Min Act: -0.0003

Data Index: 411045 (C4 (Web Text))

Max Activating Token Index: 47

Text #17

Max Range: 3.1288. Min Range: -3.1288

Max Act: 2.1069. Min Act: -0.0002

Data Index: 466511 (C4 (Web Text))

Max Activating Token Index: 770

Text #18

Max Range: 3.1288. Min Range: -3.1288

Max Act: 2.0750. Min Act: -0.0002

Data Index: 487859 (C4 (Web Text))

Max Activating Token Index: 214

Text #19

Max Range: 3.1288. Min Range: -3.1288

Max Act: 2.0867. Min Act: -0.0002

Data Index: 301832 (C4 (Web Text))

Max Activating Token Index: 51