Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 188 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.6382. Min Range: -0.6382

Max Act: 0.6382. Min Act: -0.0001

Data Index: 736365 (C4 (Web Text))

Max Activating Token Index: 119

Text #1

Max Range: 0.6382. Min Range: -0.6382

Max Act: 0.3041. Min Act: -0.0001

Data Index: 1191178 (C4 (Web Text))

Max Activating Token Index: 847

Text #2

Max Range: 0.6382. Min Range: -0.6382

Max Act: 0.2912. Min Act: -0.0001

Data Index: 623725 (C4 (Web Text))

Max Activating Token Index: 987

Text #3

Max Range: 0.6382. Min Range: -0.6382

Max Act: 0.2948. Min Act: -0.0001

Data Index: 521590 (C4 (Web Text))

Max Activating Token Index: 606

Text #4

Max Range: 0.6382. Min Range: -0.6382

Max Act: 0.3017. Min Act: -0.0001

Data Index: 313398 (C4 (Web Text))

Max Activating Token Index: 349

Text #5

Max Range: 0.6382. Min Range: -0.6382

Max Act: 0.3137. Min Act: -0.0001

Data Index: 419219 (C4 (Web Text))

Max Activating Token Index: 717

Text #6

Max Range: 0.6382. Min Range: -0.6382

Max Act: 0.2942. Min Act: -0.0001

Data Index: 823264 (C4 (Web Text))

Max Activating Token Index: 639

Text #7

Max Range: 0.6382. Min Range: -0.6382

Max Act: 0.2745. Min Act: -0.0001

Data Index: 991344 (C4 (Web Text))

Max Activating Token Index: 127

Text #8

Max Range: 0.6382. Min Range: -0.6382

Max Act: 0.2580. Min Act: -0.0001

Data Index: 407865 (C4 (Web Text))

Max Activating Token Index: 979

Text #9

Max Range: 0.6382. Min Range: -0.6382

Max Act: 0.2575. Min Act: -0.0001

Data Index: 986493 (C4 (Web Text))

Max Activating Token Index: 308

Text #10

Max Range: 0.6382. Min Range: -0.6382

Max Act: 0.2622. Min Act: -0.0001

Data Index: 1188799 (C4 (Web Text))

Max Activating Token Index: 58

Text #11

Max Range: 0.6382. Min Range: -0.6382

Max Act: 0.2409. Min Act: -0.0001

Data Index: 962177 (C4 (Web Text))

Max Activating Token Index: 325

Text #12

Max Range: 0.6382. Min Range: -0.6382

Max Act: 0.2224. Min Act: -0.0001

Data Index: 588380 (C4 (Web Text))

Max Activating Token Index: 243

Text #13

Max Range: 0.6382. Min Range: -0.6382

Max Act: 0.2429. Min Act: -0.0001

Data Index: 194127 (C4 (Web Text))

Max Activating Token Index: 563

Text #14

Max Range: 0.6382. Min Range: -0.6382

Max Act: 0.2433. Min Act: -0.0001

Data Index: 1041371 (C4 (Web Text))

Max Activating Token Index: 135

Text #15

Max Range: 0.6382. Min Range: -0.6382

Max Act: 0.2335. Min Act: -0.0001

Data Index: 1326835 (C4 (Web Text))

Max Activating Token Index: 82

Text #16

Max Range: 0.6382. Min Range: -0.6382

Max Act: 0.2190. Min Act: -0.0001

Data Index: 249965 (C4 (Web Text))

Max Activating Token Index: 843

Text #17

Max Range: 0.6382. Min Range: -0.6382

Max Act: 0.2418. Min Act: -0.0001

Data Index: 752137 (C4 (Web Text))

Max Activating Token Index: 329

Text #18

Max Range: 0.6382. Min Range: -0.6382

Max Act: 0.2389. Min Act: -0.0001

Data Index: 1444674 (Python Code)

Max Activating Token Index: 158

Text #19

Max Range: 0.6382. Min Range: -0.6382

Max Act: 0.2382. Min Act: -0.0001

Data Index: 1611835 (Python Code)

Max Activating Token Index: 144