Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1676 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.8131. Min Range: -0.8131

Max Act: 0.8131. Min Act: -0.0000

Data Index: 1345025 (C4 (Web Text))

Max Activating Token Index: 987

Text #1

Max Range: 0.8131. Min Range: -0.8131

Max Act: 0.6421. Min Act: -0.0000

Data Index: 1043177 (C4 (Web Text))

Max Activating Token Index: 93

Text #2

Max Range: 0.8131. Min Range: -0.8131

Max Act: 0.5991. Min Act: -0.0000

Data Index: 964758 (C4 (Web Text))

Max Activating Token Index: 1015

Text #3

Max Range: 0.8131. Min Range: -0.8131

Max Act: 0.6153. Min Act: -0.0000

Data Index: 631096 (C4 (Web Text))

Max Activating Token Index: 518

Text #4

Max Range: 0.8131. Min Range: -0.8131

Max Act: 0.6040. Min Act: -0.0000

Data Index: 272104 (C4 (Web Text))

Max Activating Token Index: 804

Text #5

Max Range: 0.8131. Min Range: -0.8131

Max Act: 0.5659. Min Act: -0.0000

Data Index: 969315 (C4 (Web Text))

Max Activating Token Index: 969

Text #6

Max Range: 0.8131. Min Range: -0.8131

Max Act: 0.6175. Min Act: -0.0000

Data Index: 422199 (C4 (Web Text))

Max Activating Token Index: 730

Text #7

Max Range: 0.8131. Min Range: -0.8131

Max Act: 0.5174. Min Act: -0.0000

Data Index: 767014 (C4 (Web Text))

Max Activating Token Index: 260

Text #8

Max Range: 0.8131. Min Range: -0.8131

Max Act: 0.5422. Min Act: -0.0000

Data Index: 322481 (C4 (Web Text))

Max Activating Token Index: 260

Text #9

Max Range: 0.8131. Min Range: -0.8131

Max Act: 0.5592. Min Act: -0.0000

Data Index: 1213544 (C4 (Web Text))

Max Activating Token Index: 795

Text #10

Max Range: 0.8131. Min Range: -0.8131

Max Act: 0.5208. Min Act: -0.0000

Data Index: 200302 (C4 (Web Text))

Max Activating Token Index: 927

Text #11

Max Range: 0.8131. Min Range: -0.8131

Max Act: 0.5960. Min Act: -0.0000

Data Index: 1023901 (C4 (Web Text))

Max Activating Token Index: 785

Text #12

Max Range: 0.8131. Min Range: -0.8131

Max Act: 0.5464. Min Act: -0.0000

Data Index: 91516 (C4 (Web Text))

Max Activating Token Index: 991

Text #13

Max Range: 0.8131. Min Range: -0.8131

Max Act: 0.5398. Min Act: -0.0000

Data Index: 483819 (C4 (Web Text))

Max Activating Token Index: 619

Text #14

Max Range: 0.8131. Min Range: -0.8131

Max Act: 0.5211. Min Act: -0.0000

Data Index: 1193519 (C4 (Web Text))

Max Activating Token Index: 324

Text #15

Max Range: 0.8131. Min Range: -0.8131

Max Act: 0.5313. Min Act: -0.0000

Data Index: 254020 (C4 (Web Text))

Max Activating Token Index: 958

Text #16

Max Range: 0.8131. Min Range: -0.8131

Max Act: 0.5226. Min Act: -0.0000

Data Index: 575795 (C4 (Web Text))

Max Activating Token Index: 491

Text #17

Max Range: 0.8131. Min Range: -0.8131

Max Act: 0.5207. Min Act: -0.0000

Data Index: 990622 (C4 (Web Text))

Max Activating Token Index: 235

Text #18

Max Range: 0.8131. Min Range: -0.8131

Max Act: 0.5008. Min Act: -0.0000

Data Index: 854123 (C4 (Web Text))

Max Activating Token Index: 40

Text #19

Max Range: 0.8131. Min Range: -0.8131

Max Act: 0.5274. Min Act: -0.0000

Data Index: 1166986 (C4 (Web Text))

Max Activating Token Index: 842