Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 144 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.8933. Min Range: -2.8933

Max Act: 2.8933. Min Act: -0.0000

Data Index: 888123 (C4 (Web Text))

Max Activating Token Index: 791

Text #1

Max Range: 2.8933. Min Range: -2.8933

Max Act: 2.5153. Min Act: -0.0000

Data Index: 966132 (C4 (Web Text))

Max Activating Token Index: 196

Text #2

Max Range: 2.8933. Min Range: -2.8933

Max Act: 2.3172. Min Act: -0.0000

Data Index: 181293 (C4 (Web Text))

Max Activating Token Index: 845

Text #3

Max Range: 2.8933. Min Range: -2.8933

Max Act: 2.1301. Min Act: -0.0000

Data Index: 351923 (C4 (Web Text))

Max Activating Token Index: 977

Text #4

Max Range: 2.8933. Min Range: -2.8933

Max Act: 1.9941. Min Act: -0.0000

Data Index: 856241 (C4 (Web Text))

Max Activating Token Index: 170

Text #5

Max Range: 2.8933. Min Range: -2.8933

Max Act: 2.2204. Min Act: -0.0000

Data Index: 888398 (C4 (Web Text))

Max Activating Token Index: 837

Text #6

Max Range: 2.8933. Min Range: -2.8933

Max Act: 2.2070. Min Act: -0.0000

Data Index: 1350926 (C4 (Web Text))

Max Activating Token Index: 195

Text #7

Max Range: 2.8933. Min Range: -2.8933

Max Act: 1.9034. Min Act: -0.0000

Data Index: 1216233 (C4 (Web Text))

Max Activating Token Index: 41

Text #8

Max Range: 2.8933. Min Range: -2.8933

Max Act: 1.8521. Min Act: -0.0000

Data Index: 88564 (C4 (Web Text))

Max Activating Token Index: 160

Text #9

Max Range: 2.8933. Min Range: -2.8933

Max Act: 2.0243. Min Act: -0.0000

Data Index: 1116629 (C4 (Web Text))

Max Activating Token Index: 876

Text #10

Max Range: 2.8933. Min Range: -2.8933

Max Act: 2.1181. Min Act: -0.0000

Data Index: 16271 (C4 (Web Text))

Max Activating Token Index: 24

Text #11

Max Range: 2.8933. Min Range: -2.8933

Max Act: 1.8924. Min Act: -0.0000

Data Index: 1295190 (C4 (Web Text))

Max Activating Token Index: 465

Text #12

Max Range: 2.8933. Min Range: -2.8933

Max Act: 1.8695. Min Act: -0.0000

Data Index: 1260128 (C4 (Web Text))

Max Activating Token Index: 670

Text #13

Max Range: 2.8933. Min Range: -2.8933

Max Act: 1.8631. Min Act: -0.0000

Data Index: 799806 (C4 (Web Text))

Max Activating Token Index: 996

Text #14

Max Range: 2.8933. Min Range: -2.8933

Max Act: 1.8290. Min Act: -0.0000

Data Index: 303749 (C4 (Web Text))

Max Activating Token Index: 289

Text #15

Max Range: 2.8933. Min Range: -2.8933

Max Act: 1.7223. Min Act: -0.0000

Data Index: 534611 (C4 (Web Text))

Max Activating Token Index: 656

Text #16

Max Range: 2.8933. Min Range: -2.8933

Max Act: 1.8079. Min Act: -0.0000

Data Index: 798528 (C4 (Web Text))

Max Activating Token Index: 373

Text #17

Max Range: 2.8933. Min Range: -2.8933

Max Act: 1.8734. Min Act: -0.0000

Data Index: 1269766 (C4 (Web Text))

Max Activating Token Index: 707

Text #18

Max Range: 2.8933. Min Range: -2.8933

Max Act: 1.7185. Min Act: -0.0000

Data Index: 665895 (C4 (Web Text))

Max Activating Token Index: 512

Text #19

Max Range: 2.8933. Min Range: -2.8933

Max Act: 1.6810. Min Act: -0.0000

Data Index: 1285071 (C4 (Web Text))

Max Activating Token Index: 697