Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 375 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.2813. Min Range: -1.2813

Max Act: 1.2222. Min Act: -0.0000

Data Index: 581119 (C4 (Web Text))

Max Activating Token Index: 493

Text #1

Max Range: 1.2813. Min Range: -1.2813

Max Act: 1.1909. Min Act: -0.0000

Data Index: 1028938 (C4 (Web Text))

Max Activating Token Index: 145

Text #2

Max Range: 1.2813. Min Range: -1.2813

Max Act: 1.2346. Min Act: -0.0000

Data Index: 649792 (C4 (Web Text))

Max Activating Token Index: 641

Text #3

Max Range: 1.2813. Min Range: -1.2813

Max Act: 1.2241. Min Act: -0.0000

Data Index: 314644 (C4 (Web Text))

Max Activating Token Index: 365

Text #4

Max Range: 1.2813. Min Range: -1.2813

Max Act: 1.2813. Min Act: -0.0000

Data Index: 696421 (C4 (Web Text))

Max Activating Token Index: 15

Text #5

Max Range: 1.2813. Min Range: -1.2813

Max Act: 1.1747. Min Act: -0.0000

Data Index: 987571 (C4 (Web Text))

Max Activating Token Index: 636

Text #6

Max Range: 1.2813. Min Range: -1.2813

Max Act: 1.1622. Min Act: -0.0000

Data Index: 1167038 (C4 (Web Text))

Max Activating Token Index: 19

Text #7

Max Range: 1.2813. Min Range: -1.2813

Max Act: 1.0534. Min Act: -0.0000

Data Index: 1346309 (C4 (Web Text))

Max Activating Token Index: 69

Text #8

Max Range: 1.2813. Min Range: -1.2813

Max Act: 1.1588. Min Act: -0.0000

Data Index: 851431 (C4 (Web Text))

Max Activating Token Index: 483

Text #9

Max Range: 1.2813. Min Range: -1.2813

Max Act: 1.0835. Min Act: 0.0000

Data Index: 996893 (C4 (Web Text))

Max Activating Token Index: 763

Text #10

Max Range: 1.2813. Min Range: -1.2813

Max Act: 1.0740. Min Act: -0.0000

Data Index: 130570 (C4 (Web Text))

Max Activating Token Index: 552

Text #11

Max Range: 1.2813. Min Range: -1.2813

Max Act: 0.9777. Min Act: 0.0000

Data Index: 145845 (C4 (Web Text))

Max Activating Token Index: 1021

Text #12

Max Range: 1.2813. Min Range: -1.2813

Max Act: 1.1031. Min Act: -0.0000

Data Index: 1316666 (C4 (Web Text))

Max Activating Token Index: 869

Text #13

Max Range: 1.2813. Min Range: -1.2813

Max Act: 1.0356. Min Act: -0.0000

Data Index: 590368 (C4 (Web Text))

Max Activating Token Index: 987

Text #14

Max Range: 1.2813. Min Range: -1.2813

Max Act: 1.1037. Min Act: -0.0000

Data Index: 232803 (C4 (Web Text))

Max Activating Token Index: 984

Text #15

Max Range: 1.2813. Min Range: -1.2813

Max Act: 1.0317. Min Act: -0.0000

Data Index: 29710 (C4 (Web Text))

Max Activating Token Index: 383

Text #16

Max Range: 1.2813. Min Range: -1.2813

Max Act: 1.0249. Min Act: -0.0000

Data Index: 368844 (C4 (Web Text))

Max Activating Token Index: 695

Text #17

Max Range: 1.2813. Min Range: -1.2813

Max Act: 1.0200. Min Act: -0.0000

Data Index: 360717 (C4 (Web Text))

Max Activating Token Index: 896

Text #18

Max Range: 1.2813. Min Range: -1.2813

Max Act: 0.9920. Min Act: -0.0000

Data Index: 214432 (C4 (Web Text))

Max Activating Token Index: 40

Text #19

Max Range: 1.2813. Min Range: -1.2813

Max Act: 1.1908. Min Act: -0.0000

Data Index: 614736 (C4 (Web Text))

Max Activating Token Index: 332