Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2272 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.1930. Min Range: -1.1930

Max Act: 1.1930. Min Act: -0.0000

Data Index: 1359704 (C4 (Web Text))

Max Activating Token Index: 926

Text #1

Max Range: 1.1930. Min Range: -1.1930

Max Act: 1.0494. Min Act: -0.0000

Data Index: 48305 (C4 (Web Text))

Max Activating Token Index: 380

Text #2

Max Range: 1.1930. Min Range: -1.1930

Max Act: 1.0150. Min Act: -0.0000

Data Index: 306959 (C4 (Web Text))

Max Activating Token Index: 650

Text #3

Max Range: 1.1930. Min Range: -1.1930

Max Act: 1.0313. Min Act: -0.0000

Data Index: 1236768 (C4 (Web Text))

Max Activating Token Index: 76

Text #4

Max Range: 1.1930. Min Range: -1.1930

Max Act: 1.0267. Min Act: -0.0000

Data Index: 460573 (C4 (Web Text))

Max Activating Token Index: 95

Text #5

Max Range: 1.1930. Min Range: -1.1930

Max Act: 0.9504. Min Act: -0.0000

Data Index: 609235 (C4 (Web Text))

Max Activating Token Index: 184

Text #6

Max Range: 1.1930. Min Range: -1.1930

Max Act: 0.9583. Min Act: -0.0000

Data Index: 78937 (C4 (Web Text))

Max Activating Token Index: 766

Text #7

Max Range: 1.1930. Min Range: -1.1930

Max Act: 1.0109. Min Act: -0.0000

Data Index: 470487 (C4 (Web Text))

Max Activating Token Index: 976

Text #8

Max Range: 1.1930. Min Range: -1.1930

Max Act: 0.9396. Min Act: -0.0000

Data Index: 476174 (C4 (Web Text))

Max Activating Token Index: 645

Text #9

Max Range: 1.1930. Min Range: -1.1930

Max Act: 0.9227. Min Act: -0.0000

Data Index: 267401 (C4 (Web Text))

Max Activating Token Index: 796

Text #10

Max Range: 1.1930. Min Range: -1.1930

Max Act: 0.8226. Min Act: -0.0000

Data Index: 451547 (C4 (Web Text))

Max Activating Token Index: 827

Text #11

Max Range: 1.1930. Min Range: -1.1930

Max Act: 0.8352. Min Act: -0.0000

Data Index: 1122207 (C4 (Web Text))

Max Activating Token Index: 532

Text #12

Max Range: 1.1930. Min Range: -1.1930

Max Act: 0.8192. Min Act: -0.0000

Data Index: 61179 (C4 (Web Text))

Max Activating Token Index: 697

Text #13

Max Range: 1.1930. Min Range: -1.1930

Max Act: 0.8406. Min Act: -0.0000

Data Index: 837095 (C4 (Web Text))

Max Activating Token Index: 973

Text #14

Max Range: 1.1930. Min Range: -1.1930

Max Act: 0.8090. Min Act: -0.0000

Data Index: 18615 (C4 (Web Text))

Max Activating Token Index: 87

Text #15

Max Range: 1.1930. Min Range: -1.1930

Max Act: 0.7717. Min Act: -0.0000

Data Index: 259181 (C4 (Web Text))

Max Activating Token Index: 904

Text #16

Max Range: 1.1930. Min Range: -1.1930

Max Act: 0.7898. Min Act: -0.0000

Data Index: 1146234 (C4 (Web Text))

Max Activating Token Index: 840

Text #17

Max Range: 1.1930. Min Range: -1.1930

Max Act: 0.7532. Min Act: -0.0000

Data Index: 14330 (C4 (Web Text))

Max Activating Token Index: 123

Text #18

Max Range: 1.1930. Min Range: -1.1930

Max Act: 0.7698. Min Act: -0.0000

Data Index: 1034980 (C4 (Web Text))

Max Activating Token Index: 200

Text #19

Max Range: 1.1930. Min Range: -1.1930

Max Act: 0.6950. Min Act: -0.0000

Data Index: 1235353 (C4 (Web Text))

Max Activating Token Index: 176