Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2045 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.7210. Min Range: -0.7210

Max Act: 0.7210. Min Act: -0.0000

Data Index: 598348 (C4 (Web Text))

Max Activating Token Index: 359

Text #1

Max Range: 0.7210. Min Range: -0.7210

Max Act: 0.6059. Min Act: -0.0000

Data Index: 282704 (C4 (Web Text))

Max Activating Token Index: 166

Text #2

Max Range: 0.7210. Min Range: -0.7210

Max Act: 0.6435. Min Act: -0.0000

Data Index: 1292361 (C4 (Web Text))

Max Activating Token Index: 992

Text #3

Max Range: 0.7210. Min Range: -0.7210

Max Act: 0.5698. Min Act: -0.0000

Data Index: 1006783 (C4 (Web Text))

Max Activating Token Index: 183

Text #4

Max Range: 0.7210. Min Range: -0.7210

Max Act: 0.5470. Min Act: -0.0000

Data Index: 261186 (C4 (Web Text))

Max Activating Token Index: 130

Text #5

Max Range: 0.7210. Min Range: -0.7210

Max Act: 0.5620. Min Act: -0.0000

Data Index: 1031311 (C4 (Web Text))

Max Activating Token Index: 92

Text #6

Max Range: 0.7210. Min Range: -0.7210

Max Act: 0.5596. Min Act: -0.0000

Data Index: 364937 (C4 (Web Text))

Max Activating Token Index: 937

Text #7

Max Range: 0.7210. Min Range: -0.7210

Max Act: 0.5502. Min Act: -0.0000

Data Index: 120085 (C4 (Web Text))

Max Activating Token Index: 389

Text #8

Max Range: 0.7210. Min Range: -0.7210

Max Act: 0.5274. Min Act: -0.0000

Data Index: 33656 (C4 (Web Text))

Max Activating Token Index: 112

Text #9

Max Range: 0.7210. Min Range: -0.7210

Max Act: 0.5235. Min Act: -0.0000

Data Index: 662659 (C4 (Web Text))

Max Activating Token Index: 15

Text #10

Max Range: 0.7210. Min Range: -0.7210

Max Act: 0.5350. Min Act: -0.0000

Data Index: 983789 (C4 (Web Text))

Max Activating Token Index: 847

Text #11

Max Range: 0.7210. Min Range: -0.7210

Max Act: 0.5354. Min Act: -0.0000

Data Index: 659899 (C4 (Web Text))

Max Activating Token Index: 147

Text #12

Max Range: 0.7210. Min Range: -0.7210

Max Act: 0.4763. Min Act: -0.0000

Data Index: 103944 (C4 (Web Text))

Max Activating Token Index: 315

Text #13

Max Range: 0.7210. Min Range: -0.7210

Max Act: 0.4960. Min Act: -0.0000

Data Index: 589902 (C4 (Web Text))

Max Activating Token Index: 624

Text #14

Max Range: 0.7210. Min Range: -0.7210

Max Act: 0.5088. Min Act: -0.0000

Data Index: 897638 (C4 (Web Text))

Max Activating Token Index: 283

Text #15

Max Range: 0.7210. Min Range: -0.7210

Max Act: 0.4438. Min Act: -0.0000

Data Index: 1149251 (C4 (Web Text))

Max Activating Token Index: 143

Text #16

Max Range: 0.7210. Min Range: -0.7210

Max Act: 0.4974. Min Act: -0.0000

Data Index: 707034 (C4 (Web Text))

Max Activating Token Index: 71

Text #17

Max Range: 0.7210. Min Range: -0.7210

Max Act: 0.5107. Min Act: -0.0000

Data Index: 1317876 (C4 (Web Text))

Max Activating Token Index: 258

Text #18

Max Range: 0.7210. Min Range: -0.7210

Max Act: 0.4646. Min Act: -0.0000

Data Index: 517806 (C4 (Web Text))

Max Activating Token Index: 889

Text #19

Max Range: 0.7210. Min Range: -0.7210

Max Act: 0.4695. Min Act: -0.0000

Data Index: 50396 (C4 (Web Text))

Max Activating Token Index: 482