Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1706 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.9128. Min Range: -1.9128

Max Act: 1.8006. Min Act: -0.0000

Data Index: 1130097 (C4 (Web Text))

Max Activating Token Index: 94

Text #1

Max Range: 1.9128. Min Range: -1.9128

Max Act: 1.9128. Min Act: -0.0000

Data Index: 1085439 (C4 (Web Text))

Max Activating Token Index: 811

Text #2

Max Range: 1.9128. Min Range: -1.9128

Max Act: 1.7329. Min Act: -0.0000

Data Index: 1358318 (C4 (Web Text))

Max Activating Token Index: 639

Text #3

Max Range: 1.9128. Min Range: -1.9128

Max Act: 1.7269. Min Act: -0.0000

Data Index: 109655 (C4 (Web Text))

Max Activating Token Index: 781

Text #4

Max Range: 1.9128. Min Range: -1.9128

Max Act: 1.5800. Min Act: -0.0000

Data Index: 158524 (C4 (Web Text))

Max Activating Token Index: 372

Text #5

Max Range: 1.9128. Min Range: -1.9128

Max Act: 1.6623. Min Act: -0.0000

Data Index: 46116 (C4 (Web Text))

Max Activating Token Index: 605

Text #6

Max Range: 1.9128. Min Range: -1.9128

Max Act: 1.6728. Min Act: -0.0000

Data Index: 746211 (C4 (Web Text))

Max Activating Token Index: 622

Text #7

Max Range: 1.9128. Min Range: -1.9128

Max Act: 1.7378. Min Act: -0.0000

Data Index: 800899 (C4 (Web Text))

Max Activating Token Index: 640

Text #8

Max Range: 1.9128. Min Range: -1.9128

Max Act: 1.7111. Min Act: -0.0000

Data Index: 496502 (C4 (Web Text))

Max Activating Token Index: 39

Text #9

Max Range: 1.9128. Min Range: -1.9128

Max Act: 1.5671. Min Act: -0.0000

Data Index: 537914 (C4 (Web Text))

Max Activating Token Index: 72

Text #10

Max Range: 1.9128. Min Range: -1.9128

Max Act: 1.5725. Min Act: -0.0000

Data Index: 1239860 (C4 (Web Text))

Max Activating Token Index: 559

Text #11

Max Range: 1.9128. Min Range: -1.9128

Max Act: 1.6525. Min Act: -0.0000

Data Index: 356287 (C4 (Web Text))

Max Activating Token Index: 419

Text #12

Max Range: 1.9128. Min Range: -1.9128

Max Act: 1.6496. Min Act: -0.0000

Data Index: 848903 (C4 (Web Text))

Max Activating Token Index: 290

Text #13

Max Range: 1.9128. Min Range: -1.9128

Max Act: 1.6128. Min Act: -0.0000

Data Index: 1356684 (C4 (Web Text))

Max Activating Token Index: 228

Text #14

Max Range: 1.9128. Min Range: -1.9128

Max Act: 1.5535. Min Act: -0.0000

Data Index: 470058 (C4 (Web Text))

Max Activating Token Index: 359

Text #15

Max Range: 1.9128. Min Range: -1.9128

Max Act: 1.5318. Min Act: -0.0000

Data Index: 1234137 (C4 (Web Text))

Max Activating Token Index: 601

Text #16

Max Range: 1.9128. Min Range: -1.9128

Max Act: 1.6330. Min Act: -0.0000

Data Index: 757659 (C4 (Web Text))

Max Activating Token Index: 822

Text #17

Max Range: 1.9128. Min Range: -1.9128

Max Act: 1.6252. Min Act: -0.0000

Data Index: 453112 (C4 (Web Text))

Max Activating Token Index: 310

Text #18

Max Range: 1.9128. Min Range: -1.9128

Max Act: 1.4278. Min Act: -0.0000

Data Index: 1086091 (C4 (Web Text))

Max Activating Token Index: 328

Text #19

Max Range: 1.9128. Min Range: -1.9128

Max Act: 1.4358. Min Act: -0.0000

Data Index: 1044236 (C4 (Web Text))

Max Activating Token Index: 843