Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3336 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.0193. Min Range: -1.0193

Max Act: 1.0193. Min Act: -0.0001

Data Index: 1156011 (C4 (Web Text))

Max Activating Token Index: 837

Text #1

Max Range: 1.0193. Min Range: -1.0193

Max Act: 0.9217. Min Act: -0.0001

Data Index: 1339149 (C4 (Web Text))

Max Activating Token Index: 527

Text #2

Max Range: 1.0193. Min Range: -1.0193

Max Act: 0.7726. Min Act: -0.0001

Data Index: 903710 (C4 (Web Text))

Max Activating Token Index: 867

Text #3

Max Range: 1.0193. Min Range: -1.0193

Max Act: 0.6908. Min Act: -0.0001

Data Index: 378682 (C4 (Web Text))

Max Activating Token Index: 607

Text #4

Max Range: 1.0193. Min Range: -1.0193

Max Act: 0.8934. Min Act: -0.0001

Data Index: 774632 (C4 (Web Text))

Max Activating Token Index: 955

Text #5

Max Range: 1.0193. Min Range: -1.0193

Max Act: 0.8125. Min Act: -0.0001

Data Index: 1294851 (C4 (Web Text))

Max Activating Token Index: 945

Text #6

Max Range: 1.0193. Min Range: -1.0193

Max Act: 0.7494. Min Act: -0.0001

Data Index: 407320 (C4 (Web Text))

Max Activating Token Index: 302

Text #7

Max Range: 1.0193. Min Range: -1.0193

Max Act: 0.7346. Min Act: -0.0001

Data Index: 128730 (C4 (Web Text))

Max Activating Token Index: 743

Text #8

Max Range: 1.0193. Min Range: -1.0193

Max Act: 0.7713. Min Act: -0.0001

Data Index: 371096 (C4 (Web Text))

Max Activating Token Index: 682

Text #9

Max Range: 1.0193. Min Range: -1.0193

Max Act: 0.7126. Min Act: -0.0001

Data Index: 1123808 (C4 (Web Text))

Max Activating Token Index: 127

Text #10

Max Range: 1.0193. Min Range: -1.0193

Max Act: 0.7323. Min Act: -0.0001

Data Index: 1220318 (C4 (Web Text))

Max Activating Token Index: 711

Text #11

Max Range: 1.0193. Min Range: -1.0193

Max Act: 0.6166. Min Act: -0.0001

Data Index: 1318813 (C4 (Web Text))

Max Activating Token Index: 276

Text #12

Max Range: 1.0193. Min Range: -1.0193

Max Act: 0.7416. Min Act: -0.0001

Data Index: 832835 (C4 (Web Text))

Max Activating Token Index: 845

Text #13

Max Range: 1.0193. Min Range: -1.0193

Max Act: 0.7585. Min Act: -0.0001

Data Index: 471533 (C4 (Web Text))

Max Activating Token Index: 705

Text #14

Max Range: 1.0193. Min Range: -1.0193

Max Act: 0.6153. Min Act: -0.0001

Data Index: 792990 (C4 (Web Text))

Max Activating Token Index: 5

Text #15

Max Range: 1.0193. Min Range: -1.0193

Max Act: 0.6741. Min Act: -0.0001

Data Index: 712436 (C4 (Web Text))

Max Activating Token Index: 101

Text #16

Max Range: 1.0193. Min Range: -1.0193

Max Act: 0.4657. Min Act: -0.0001

Data Index: 216238 (C4 (Web Text))

Max Activating Token Index: 1012

Text #17

Max Range: 1.0193. Min Range: -1.0193

Max Act: 0.5870. Min Act: -0.0001

Data Index: 980343 (C4 (Web Text))

Max Activating Token Index: 300

Text #18

Max Range: 1.0193. Min Range: -1.0193

Max Act: 0.6003. Min Act: -0.0001

Data Index: 1060867 (C4 (Web Text))

Max Activating Token Index: 785

Text #19

Max Range: 1.0193. Min Range: -1.0193

Max Act: 0.7702. Min Act: -0.0001

Data Index: 950512 (C4 (Web Text))

Max Activating Token Index: 956