Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3203 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.3431. Min Range: -1.3431

Max Act: 1.3431. Min Act: -0.0000

Data Index: 965445 (C4 (Web Text))

Max Activating Token Index: 252

Text #1

Max Range: 1.3431. Min Range: -1.3431

Max Act: 1.2145. Min Act: -0.0000

Data Index: 954291 (C4 (Web Text))

Max Activating Token Index: 406

Text #2

Max Range: 1.3431. Min Range: -1.3431

Max Act: 1.1774. Min Act: -0.0000

Data Index: 1110928 (C4 (Web Text))

Max Activating Token Index: 254

Text #3

Max Range: 1.3431. Min Range: -1.3431

Max Act: 1.2042. Min Act: 0.0000

Data Index: 1101216 (C4 (Web Text))

Max Activating Token Index: 884

Text #4

Max Range: 1.3431. Min Range: -1.3431

Max Act: 1.1786. Min Act: -0.0000

Data Index: 1269406 (C4 (Web Text))

Max Activating Token Index: 76

Text #5

Max Range: 1.3431. Min Range: -1.3431

Max Act: 1.1647. Min Act: -0.0000

Data Index: 959995 (C4 (Web Text))

Max Activating Token Index: 565

Text #6

Max Range: 1.3431. Min Range: -1.3431

Max Act: 1.0788. Min Act: -0.0000

Data Index: 358598 (C4 (Web Text))

Max Activating Token Index: 656

Text #7

Max Range: 1.3431. Min Range: -1.3431

Max Act: 1.0699. Min Act: -0.0000

Data Index: 978533 (C4 (Web Text))

Max Activating Token Index: 236

Text #8

Max Range: 1.3431. Min Range: -1.3431

Max Act: 1.1368. Min Act: -0.0000

Data Index: 1055345 (C4 (Web Text))

Max Activating Token Index: 654

Text #9

Max Range: 1.3431. Min Range: -1.3431

Max Act: 1.1851. Min Act: -0.0000

Data Index: 1121415 (C4 (Web Text))

Max Activating Token Index: 642

Text #10

Max Range: 1.3431. Min Range: -1.3431

Max Act: 1.1477. Min Act: -0.0000

Data Index: 1213383 (C4 (Web Text))

Max Activating Token Index: 111

Text #11

Max Range: 1.3431. Min Range: -1.3431

Max Act: 1.1071. Min Act: -0.0000

Data Index: 1231209 (C4 (Web Text))

Max Activating Token Index: 327

Text #12

Max Range: 1.3431. Min Range: -1.3431

Max Act: 1.1341. Min Act: 0.0000

Data Index: 1067388 (C4 (Web Text))

Max Activating Token Index: 304

Text #13

Max Range: 1.3431. Min Range: -1.3431

Max Act: 0.9906. Min Act: -0.0000

Data Index: 68851 (C4 (Web Text))

Max Activating Token Index: 178

Text #14

Max Range: 1.3431. Min Range: -1.3431

Max Act: 1.0009. Min Act: -0.0000

Data Index: 293716 (C4 (Web Text))

Max Activating Token Index: 86

Text #15

Max Range: 1.3431. Min Range: -1.3431

Max Act: 1.0129. Min Act: -0.0000

Data Index: 562175 (C4 (Web Text))

Max Activating Token Index: 609

Text #16

Max Range: 1.3431. Min Range: -1.3431

Max Act: 1.0713. Min Act: -0.0000

Data Index: 106456 (C4 (Web Text))

Max Activating Token Index: 50

Text #17

Max Range: 1.3431. Min Range: -1.3431

Max Act: 1.0629. Min Act: -0.0000

Data Index: 284419 (C4 (Web Text))

Max Activating Token Index: 756

Text #18

Max Range: 1.3431. Min Range: -1.3431

Max Act: 0.9642. Min Act: -0.0000

Data Index: 736281 (C4 (Web Text))

Max Activating Token Index: 959

Text #19

Max Range: 1.3431. Min Range: -1.3431

Max Act: 1.0278. Min Act: -0.0000

Data Index: 686964 (C4 (Web Text))

Max Activating Token Index: 986