Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2696 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.4950. Min Range: -2.4950

Max Act: 2.4950. Min Act: -0.0000

Data Index: 487102 (C4 (Web Text))

Max Activating Token Index: 19

Text #1

Max Range: 2.4950. Min Range: -2.4950

Max Act: 2.2256. Min Act: -0.0000

Data Index: 248410 (C4 (Web Text))

Max Activating Token Index: 112

Text #2

Max Range: 2.4950. Min Range: -2.4950

Max Act: 2.2144. Min Act: -0.0000

Data Index: 137384 (C4 (Web Text))

Max Activating Token Index: 111

Text #3

Max Range: 2.4950. Min Range: -2.4950

Max Act: 2.0264. Min Act: -0.0000

Data Index: 827830 (C4 (Web Text))

Max Activating Token Index: 67

Text #4

Max Range: 2.4950. Min Range: -2.4950

Max Act: 1.8813. Min Act: -0.0000

Data Index: 1297327 (C4 (Web Text))

Max Activating Token Index: 30

Text #5

Max Range: 2.4950. Min Range: -2.4950

Max Act: 1.9847. Min Act: -0.0000

Data Index: 631080 (C4 (Web Text))

Max Activating Token Index: 32

Text #6

Max Range: 2.4950. Min Range: -2.4950

Max Act: 1.8484. Min Act: -0.0000

Data Index: 206355 (C4 (Web Text))

Max Activating Token Index: 419

Text #7

Max Range: 2.4950. Min Range: -2.4950

Max Act: 1.9446. Min Act: -0.0000

Data Index: 207864 (C4 (Web Text))

Max Activating Token Index: 22

Text #8

Max Range: 2.4950. Min Range: -2.4950

Max Act: 1.9334. Min Act: -0.0000

Data Index: 597551 (C4 (Web Text))

Max Activating Token Index: 48

Text #9

Max Range: 2.4950. Min Range: -2.4950

Max Act: 2.0191. Min Act: -0.0000

Data Index: 1063745 (C4 (Web Text))

Max Activating Token Index: 900

Text #10

Max Range: 2.4950. Min Range: -2.4950

Max Act: 1.9107. Min Act: -0.0000

Data Index: 1325778 (C4 (Web Text))

Max Activating Token Index: 6

Text #11

Max Range: 2.4950. Min Range: -2.4950

Max Act: 1.7862. Min Act: -0.0000

Data Index: 565203 (C4 (Web Text))

Max Activating Token Index: 61

Text #12

Max Range: 2.4950. Min Range: -2.4950

Max Act: 1.8497. Min Act: -0.0000

Data Index: 109939 (C4 (Web Text))

Max Activating Token Index: 972

Text #13

Max Range: 2.4950. Min Range: -2.4950

Max Act: 1.8313. Min Act: -0.0000

Data Index: 669690 (C4 (Web Text))

Max Activating Token Index: 52

Text #14

Max Range: 2.4950. Min Range: -2.4950

Max Act: 1.9090. Min Act: -0.0000

Data Index: 650516 (C4 (Web Text))

Max Activating Token Index: 31

Text #15

Max Range: 2.4950. Min Range: -2.4950

Max Act: 1.8918. Min Act: -0.0000

Data Index: 389025 (C4 (Web Text))

Max Activating Token Index: 98

Text #16

Max Range: 2.4950. Min Range: -2.4950

Max Act: 1.7890. Min Act: -0.0000

Data Index: 310136 (C4 (Web Text))

Max Activating Token Index: 21

Text #17

Max Range: 2.4950. Min Range: -2.4950

Max Act: 1.7739. Min Act: -0.0000

Data Index: 187880 (C4 (Web Text))

Max Activating Token Index: 7

Text #18

Max Range: 2.4950. Min Range: -2.4950

Max Act: 1.8449. Min Act: -0.0000

Data Index: 1275307 (C4 (Web Text))

Max Activating Token Index: 262

Text #19

Max Range: 2.4950. Min Range: -2.4950

Max Act: 1.6823. Min Act: -0.0000

Data Index: 17344 (C4 (Web Text))

Max Activating Token Index: 924