Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5069 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 9.6374. Min Range: -9.6374

Max Act: 9.6374. Min Act: -0.0001

Data Index: 376703 (C4 (Web Text))

Max Activating Token Index: 656

Text #1

Max Range: 9.6374. Min Range: -9.6374

Max Act: 8.7904. Min Act: -0.0001

Data Index: 689985 (C4 (Web Text))

Max Activating Token Index: 193

Text #2

Max Range: 9.6374. Min Range: -9.6374

Max Act: 8.4131. Min Act: -0.0001

Data Index: 1278991 (C4 (Web Text))

Max Activating Token Index: 443

Text #3

Max Range: 9.6374. Min Range: -9.6374

Max Act: 8.4590. Min Act: -0.0001

Data Index: 1182037 (C4 (Web Text))

Max Activating Token Index: 863

Text #4

Max Range: 9.6374. Min Range: -9.6374

Max Act: 7.7380. Min Act: -0.0001

Data Index: 655750 (C4 (Web Text))

Max Activating Token Index: 124

Text #5

Max Range: 9.6374. Min Range: -9.6374

Max Act: 8.0185. Min Act: -0.0001

Data Index: 177260 (C4 (Web Text))

Max Activating Token Index: 630

Text #6

Max Range: 9.6374. Min Range: -9.6374

Max Act: 8.0082. Min Act: -0.0001

Data Index: 481493 (C4 (Web Text))

Max Activating Token Index: 465

Text #7

Max Range: 9.6374. Min Range: -9.6374

Max Act: 7.9831. Min Act: -0.0001

Data Index: 1259157 (C4 (Web Text))

Max Activating Token Index: 930

Text #8

Max Range: 9.6374. Min Range: -9.6374

Max Act: 7.9770. Min Act: -0.0001

Data Index: 477920 (C4 (Web Text))

Max Activating Token Index: 914

Text #9

Max Range: 9.6374. Min Range: -9.6374

Max Act: 7.6027. Min Act: -0.0001

Data Index: 523036 (C4 (Web Text))

Max Activating Token Index: 876

Text #10

Max Range: 9.6374. Min Range: -9.6374

Max Act: 7.8707. Min Act: -0.0001

Data Index: 1024898 (C4 (Web Text))

Max Activating Token Index: 560

Text #11

Max Range: 9.6374. Min Range: -9.6374

Max Act: 8.0127. Min Act: -0.0001

Data Index: 645696 (C4 (Web Text))

Max Activating Token Index: 902

Text #12

Max Range: 9.6374. Min Range: -9.6374

Max Act: 7.8319. Min Act: -0.0001

Data Index: 1194580 (C4 (Web Text))

Max Activating Token Index: 959

Text #13

Max Range: 9.6374. Min Range: -9.6374

Max Act: 7.9593. Min Act: -0.0001

Data Index: 827950 (C4 (Web Text))

Max Activating Token Index: 874

Text #14

Max Range: 9.6374. Min Range: -9.6374

Max Act: 7.7339. Min Act: -0.0001

Data Index: 1275939 (C4 (Web Text))

Max Activating Token Index: 230

Text #15

Max Range: 9.6374. Min Range: -9.6374

Max Act: 7.7639. Min Act: -0.0001

Data Index: 637214 (C4 (Web Text))

Max Activating Token Index: 505

Text #16

Max Range: 9.6374. Min Range: -9.6374

Max Act: 6.7725. Min Act: -0.0001

Data Index: 656683 (C4 (Web Text))

Max Activating Token Index: 242

Text #17

Max Range: 9.6374. Min Range: -9.6374

Max Act: 7.4592. Min Act: -0.0001

Data Index: 59380 (C4 (Web Text))

Max Activating Token Index: 538

Text #18

Max Range: 9.6374. Min Range: -9.6374

Max Act: 7.9650. Min Act: -0.0001

Data Index: 1140622 (C4 (Web Text))

Max Activating Token Index: 185

Text #19

Max Range: 9.6374. Min Range: -9.6374

Max Act: 7.7594. Min Act: -0.0001

Data Index: 240646 (C4 (Web Text))

Max Activating Token Index: 737