Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1554 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 0.9135. Min Range: -0.9135

Max Act: 0.9135. Min Act: -0.0001

Data Index: 463869 (C4 (Web Text))

Max Activating Token Index: 273

Text #1

Max Range: 0.9135. Min Range: -0.9135

Max Act: 0.8889. Min Act: -0.0001

Data Index: 108778 (C4 (Web Text))

Max Activating Token Index: 979

Text #2

Max Range: 0.9135. Min Range: -0.9135

Max Act: 0.8577. Min Act: -0.0001

Data Index: 1155746 (C4 (Web Text))

Max Activating Token Index: 963

Text #3

Max Range: 0.9135. Min Range: -0.9135

Max Act: 0.9030. Min Act: -0.0001

Data Index: 367005 (C4 (Web Text))

Max Activating Token Index: 771

Text #4

Max Range: 0.9135. Min Range: -0.9135

Max Act: 0.8503. Min Act: -0.0001

Data Index: 1137409 (C4 (Web Text))

Max Activating Token Index: 982

Text #5

Max Range: 0.9135. Min Range: -0.9135

Max Act: 0.8691. Min Act: -0.0001

Data Index: 829541 (C4 (Web Text))

Max Activating Token Index: 183

Text #6

Max Range: 0.9135. Min Range: -0.9135

Max Act: 0.8474. Min Act: -0.0001

Data Index: 1123027 (C4 (Web Text))

Max Activating Token Index: 275

Text #7

Max Range: 0.9135. Min Range: -0.9135

Max Act: 0.8520. Min Act: -0.0001

Data Index: 975804 (C4 (Web Text))

Max Activating Token Index: 540

Text #8

Max Range: 0.9135. Min Range: -0.9135

Max Act: 0.8486. Min Act: -0.0001

Data Index: 984542 (C4 (Web Text))

Max Activating Token Index: 680

Text #9

Max Range: 0.9135. Min Range: -0.9135

Max Act: 0.7906. Min Act: -0.0001

Data Index: 701061 (C4 (Web Text))

Max Activating Token Index: 77

Text #10

Max Range: 0.9135. Min Range: -0.9135

Max Act: 0.8081. Min Act: -0.0001

Data Index: 1251441 (C4 (Web Text))

Max Activating Token Index: 70

Text #11

Max Range: 0.9135. Min Range: -0.9135

Max Act: 0.8293. Min Act: -0.0001

Data Index: 1258030 (C4 (Web Text))

Max Activating Token Index: 761

Text #12

Max Range: 0.9135. Min Range: -0.9135

Max Act: 0.8055. Min Act: -0.0001

Data Index: 130186 (C4 (Web Text))

Max Activating Token Index: 869

Text #13

Max Range: 0.9135. Min Range: -0.9135

Max Act: 0.7960. Min Act: -0.0001

Data Index: 922549 (C4 (Web Text))

Max Activating Token Index: 768

Text #14

Max Range: 0.9135. Min Range: -0.9135

Max Act: 0.7385. Min Act: -0.0001

Data Index: 1087698 (C4 (Web Text))

Max Activating Token Index: 618

Text #15

Max Range: 0.9135. Min Range: -0.9135

Max Act: 0.8545. Min Act: -0.0001

Data Index: 627837 (C4 (Web Text))

Max Activating Token Index: 705

Text #16

Max Range: 0.9135. Min Range: -0.9135

Max Act: 0.7652. Min Act: -0.0001

Data Index: 143919 (C4 (Web Text))

Max Activating Token Index: 891

Text #17

Max Range: 0.9135. Min Range: -0.9135

Max Act: 0.7605. Min Act: -0.0001

Data Index: 183517 (C4 (Web Text))

Max Activating Token Index: 219

Text #18

Max Range: 0.9135. Min Range: -0.9135

Max Act: 0.7343. Min Act: -0.0001

Data Index: 633209 (C4 (Web Text))

Max Activating Token Index: 784

Text #19

Max Range: 0.9135. Min Range: -0.9135

Max Act: 0.7391. Min Act: -0.0001

Data Index: 220394 (C4 (Web Text))

Max Activating Token Index: 233