Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 544 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.9191. Min Range: -0.9191

Max Act: 0.9191. Min Act: -0.0001

Data Index: 459483 (C4 (Web Text))

Max Activating Token Index: 677

Text #1

Max Range: 0.9191. Min Range: -0.9191

Max Act: 0.7289. Min Act: -0.0001

Data Index: 430198 (C4 (Web Text))

Max Activating Token Index: 218

Text #2

Max Range: 0.9191. Min Range: -0.9191

Max Act: 0.8814. Min Act: -0.0001

Data Index: 1165001 (C4 (Web Text))

Max Activating Token Index: 155

Text #3

Max Range: 0.9191. Min Range: -0.9191

Max Act: 0.8016. Min Act: -0.0000

Data Index: 617074 (C4 (Web Text))

Max Activating Token Index: 885

Text #4

Max Range: 0.9191. Min Range: -0.9191

Max Act: 0.8171. Min Act: -0.0000

Data Index: 861799 (C4 (Web Text))

Max Activating Token Index: 346

Text #5

Max Range: 0.9191. Min Range: -0.9191

Max Act: 0.8254. Min Act: -0.0001

Data Index: 291415 (C4 (Web Text))

Max Activating Token Index: 859

Text #6

Max Range: 0.9191. Min Range: -0.9191

Max Act: 0.7888. Min Act: -0.0001

Data Index: 118287 (C4 (Web Text))

Max Activating Token Index: 728

Text #7

Max Range: 0.9191. Min Range: -0.9191

Max Act: 0.7506. Min Act: -0.0000

Data Index: 1214127 (C4 (Web Text))

Max Activating Token Index: 732

Text #8

Max Range: 0.9191. Min Range: -0.9191

Max Act: 0.7714. Min Act: -0.0000

Data Index: 1274915 (C4 (Web Text))

Max Activating Token Index: 1023

Text #9

Max Range: 0.9191. Min Range: -0.9191

Max Act: 0.7416. Min Act: -0.0000

Data Index: 1049203 (C4 (Web Text))

Max Activating Token Index: 562

Text #10

Max Range: 0.9191. Min Range: -0.9191

Max Act: 0.7131. Min Act: -0.0001

Data Index: 706502 (C4 (Web Text))

Max Activating Token Index: 130

Text #11

Max Range: 0.9191. Min Range: -0.9191

Max Act: 0.7309. Min Act: -0.0001

Data Index: 833724 (C4 (Web Text))

Max Activating Token Index: 318

Text #12

Max Range: 0.9191. Min Range: -0.9191

Max Act: 0.6872. Min Act: -0.0001

Data Index: 835943 (C4 (Web Text))

Max Activating Token Index: 336

Text #13

Max Range: 0.9191. Min Range: -0.9191

Max Act: 0.6814. Min Act: -0.0001

Data Index: 377935 (C4 (Web Text))

Max Activating Token Index: 801

Text #14

Max Range: 0.9191. Min Range: -0.9191

Max Act: 0.6792. Min Act: -0.0001

Data Index: 501873 (C4 (Web Text))

Max Activating Token Index: 362

Text #15

Max Range: 0.9191. Min Range: -0.9191

Max Act: 0.6702. Min Act: -0.0001

Data Index: 55836 (C4 (Web Text))

Max Activating Token Index: 661

Text #16

Max Range: 0.9191. Min Range: -0.9191

Max Act: 0.6522. Min Act: -0.0000

Data Index: 255684 (C4 (Web Text))

Max Activating Token Index: 899

Text #17

Max Range: 0.9191. Min Range: -0.9191

Max Act: 0.5895. Min Act: -0.0001

Data Index: 408092 (C4 (Web Text))

Max Activating Token Index: 932

Text #18

Max Range: 0.9191. Min Range: -0.9191

Max Act: 0.6306. Min Act: -0.0001

Data Index: 1141117 (C4 (Web Text))

Max Activating Token Index: 939

Text #19

Max Range: 0.9191. Min Range: -0.9191

Max Act: 0.6177. Min Act: -0.0000

Data Index: 1383919 (Python Code)

Max Activating Token Index: 918