Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 810 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.9413. Min Range: -2.9413

Max Act: 2.9413. Min Act: -0.0000

Data Index: 7715 (C4 (Web Text))

Max Activating Token Index: 715

Text #1

Max Range: 2.9413. Min Range: -2.9413

Max Act: 2.8738. Min Act: -0.0000

Data Index: 1193824 (C4 (Web Text))

Max Activating Token Index: 44

Text #2

Max Range: 2.9413. Min Range: -2.9413

Max Act: 2.8974. Min Act: -0.0000

Data Index: 827550 (C4 (Web Text))

Max Activating Token Index: 1014

Text #3

Max Range: 2.9413. Min Range: -2.9413

Max Act: 2.6620. Min Act: -0.0000

Data Index: 334888 (C4 (Web Text))

Max Activating Token Index: 901

Text #4

Max Range: 2.9413. Min Range: -2.9413

Max Act: 2.5714. Min Act: -0.0000

Data Index: 490559 (C4 (Web Text))

Max Activating Token Index: 516

Text #5

Max Range: 2.9413. Min Range: -2.9413

Max Act: 2.5078. Min Act: -0.0000

Data Index: 930605 (C4 (Web Text))

Max Activating Token Index: 478

Text #6

Max Range: 2.9413. Min Range: -2.9413

Max Act: 2.3277. Min Act: -0.0000

Data Index: 624062 (C4 (Web Text))

Max Activating Token Index: 639

Text #7

Max Range: 2.9413. Min Range: -2.9413

Max Act: 2.4935. Min Act: -0.0000

Data Index: 266411 (C4 (Web Text))

Max Activating Token Index: 109

Text #8

Max Range: 2.9413. Min Range: -2.9413

Max Act: 2.3737. Min Act: -0.0000

Data Index: 139156 (C4 (Web Text))

Max Activating Token Index: 183

Text #9

Max Range: 2.9413. Min Range: -2.9413

Max Act: 2.2362. Min Act: -0.0000

Data Index: 70272 (C4 (Web Text))

Max Activating Token Index: 325

Text #10

Max Range: 2.9413. Min Range: -2.9413

Max Act: 2.3433. Min Act: -0.0000

Data Index: 1119023 (C4 (Web Text))

Max Activating Token Index: 898

Text #11

Max Range: 2.9413. Min Range: -2.9413

Max Act: 2.3360. Min Act: -0.0000

Data Index: 436173 (C4 (Web Text))

Max Activating Token Index: 990

Text #12

Max Range: 2.9413. Min Range: -2.9413

Max Act: 2.2808. Min Act: -0.0000

Data Index: 1139991 (C4 (Web Text))

Max Activating Token Index: 205

Text #13

Max Range: 2.9413. Min Range: -2.9413

Max Act: 2.2277. Min Act: -0.0000

Data Index: 415889 (C4 (Web Text))

Max Activating Token Index: 972

Text #14

Max Range: 2.9413. Min Range: -2.9413

Max Act: 2.2739. Min Act: -0.0000

Data Index: 673121 (C4 (Web Text))

Max Activating Token Index: 365

Text #15

Max Range: 2.9413. Min Range: -2.9413

Max Act: 2.1520. Min Act: -0.0000

Data Index: 10542 (C4 (Web Text))

Max Activating Token Index: 640

Text #16

Max Range: 2.9413. Min Range: -2.9413

Max Act: 2.1542. Min Act: -0.0000

Data Index: 38251 (C4 (Web Text))

Max Activating Token Index: 274

Text #17

Max Range: 2.9413. Min Range: -2.9413

Max Act: 2.0105. Min Act: -0.0000

Data Index: 647434 (C4 (Web Text))

Max Activating Token Index: 83

Text #18

Max Range: 2.9413. Min Range: -2.9413

Max Act: 2.0233. Min Act: -0.0000

Data Index: 314027 (C4 (Web Text))

Max Activating Token Index: 962

Text #19

Max Range: 2.9413. Min Range: -2.9413

Max Act: 1.9733. Min Act: -0.0000

Data Index: 1270197 (C4 (Web Text))

Max Activating Token Index: 18