Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 6034 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.1041. Min Range: -1.1041

Max Act: 1.1041. Min Act: -0.0000

Data Index: 574315 (C4 (Web Text))

Max Activating Token Index: 846

Text #1

Max Range: 1.1041. Min Range: -1.1041

Max Act: 1.0329. Min Act: -0.0000

Data Index: 1248640 (C4 (Web Text))

Max Activating Token Index: 380

Text #2

Max Range: 1.1041. Min Range: -1.1041

Max Act: 0.9339. Min Act: -0.0000

Data Index: 249471 (C4 (Web Text))

Max Activating Token Index: 348

Text #3

Max Range: 1.1041. Min Range: -1.1041

Max Act: 0.9344. Min Act: -0.0000

Data Index: 1014688 (C4 (Web Text))

Max Activating Token Index: 273

Text #4

Max Range: 1.1041. Min Range: -1.1041

Max Act: 0.9293. Min Act: -0.0000

Data Index: 546930 (C4 (Web Text))

Max Activating Token Index: 387

Text #5

Max Range: 1.1041. Min Range: -1.1041

Max Act: 0.8936. Min Act: -0.0000

Data Index: 1183532 (C4 (Web Text))

Max Activating Token Index: 599

Text #6

Max Range: 1.1041. Min Range: -1.1041

Max Act: 0.9125. Min Act: -0.0000

Data Index: 172850 (C4 (Web Text))

Max Activating Token Index: 584

Text #7

Max Range: 1.1041. Min Range: -1.1041

Max Act: 0.8545. Min Act: -0.0000

Data Index: 1282288 (C4 (Web Text))

Max Activating Token Index: 228

Text #8

Max Range: 1.1041. Min Range: -1.1041

Max Act: 0.8184. Min Act: -0.0000

Data Index: 82231 (C4 (Web Text))

Max Activating Token Index: 796

Text #9

Max Range: 1.1041. Min Range: -1.1041

Max Act: 0.8267. Min Act: -0.0000

Data Index: 1017743 (C4 (Web Text))

Max Activating Token Index: 1007

Text #10

Max Range: 1.1041. Min Range: -1.1041

Max Act: 0.8093. Min Act: -0.0000

Data Index: 554562 (C4 (Web Text))

Max Activating Token Index: 157

Text #11

Max Range: 1.1041. Min Range: -1.1041

Max Act: 0.8032. Min Act: -0.0000

Data Index: 305702 (C4 (Web Text))

Max Activating Token Index: 157

Text #12

Max Range: 1.1041. Min Range: -1.1041

Max Act: 0.7370. Min Act: -0.0000

Data Index: 626054 (C4 (Web Text))

Max Activating Token Index: 862

Text #13

Max Range: 1.1041. Min Range: -1.1041

Max Act: 0.7629. Min Act: -0.0000

Data Index: 1063156 (C4 (Web Text))

Max Activating Token Index: 838

Text #14

Max Range: 1.1041. Min Range: -1.1041

Max Act: 0.7630. Min Act: -0.0000

Data Index: 579197 (C4 (Web Text))

Max Activating Token Index: 659

Text #15

Max Range: 1.1041. Min Range: -1.1041

Max Act: 0.7596. Min Act: -0.0000

Data Index: 713204 (C4 (Web Text))

Max Activating Token Index: 1016

Text #16

Max Range: 1.1041. Min Range: -1.1041

Max Act: 0.7639. Min Act: -0.0000

Data Index: 627965 (C4 (Web Text))

Max Activating Token Index: 649

Text #17

Max Range: 1.1041. Min Range: -1.1041

Max Act: 0.7807. Min Act: -0.0000

Data Index: 132472 (C4 (Web Text))

Max Activating Token Index: 38

Text #18

Max Range: 1.1041. Min Range: -1.1041

Max Act: 0.7774. Min Act: -0.0000

Data Index: 491160 (C4 (Web Text))

Max Activating Token Index: 727

Text #19

Max Range: 1.1041. Min Range: -1.1041

Max Act: 0.7528. Min Act: -0.0000

Data Index: 1281366 (C4 (Web Text))

Max Activating Token Index: 532