Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2493 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.8583. Min Range: -0.8583

Max Act: 0.8572. Min Act: -0.0001

Data Index: 127747 (C4 (Web Text))

Max Activating Token Index: 334

Text #1

Max Range: 0.8583. Min Range: -0.8583

Max Act: 0.8583. Min Act: -0.0001

Data Index: 79151 (C4 (Web Text))

Max Activating Token Index: 210

Text #2

Max Range: 0.8583. Min Range: -0.8583

Max Act: 0.8330. Min Act: -0.0001

Data Index: 1258474 (C4 (Web Text))

Max Activating Token Index: 442

Text #3

Max Range: 0.8583. Min Range: -0.8583

Max Act: 0.8093. Min Act: -0.0001

Data Index: 395150 (C4 (Web Text))

Max Activating Token Index: 767

Text #4

Max Range: 0.8583. Min Range: -0.8583

Max Act: 0.7798. Min Act: -0.0001

Data Index: 626483 (C4 (Web Text))

Max Activating Token Index: 696

Text #5

Max Range: 0.8583. Min Range: -0.8583

Max Act: 0.7965. Min Act: -0.0001

Data Index: 1030650 (C4 (Web Text))

Max Activating Token Index: 744

Text #6

Max Range: 0.8583. Min Range: -0.8583

Max Act: 0.7140. Min Act: -0.0001

Data Index: 38492 (C4 (Web Text))

Max Activating Token Index: 178

Text #7

Max Range: 0.8583. Min Range: -0.8583

Max Act: 0.7609. Min Act: -0.0001

Data Index: 1190028 (C4 (Web Text))

Max Activating Token Index: 540

Text #8

Max Range: 0.8583. Min Range: -0.8583

Max Act: 0.7859. Min Act: -0.0001

Data Index: 1186508 (C4 (Web Text))

Max Activating Token Index: 645

Text #9

Max Range: 0.8583. Min Range: -0.8583

Max Act: 0.7280. Min Act: -0.0001

Data Index: 507601 (C4 (Web Text))

Max Activating Token Index: 816

Text #10

Max Range: 0.8583. Min Range: -0.8583

Max Act: 0.7301. Min Act: -0.0001

Data Index: 113089 (C4 (Web Text))

Max Activating Token Index: 556

Text #11

Max Range: 0.8583. Min Range: -0.8583

Max Act: 0.7186. Min Act: -0.0001

Data Index: 617591 (C4 (Web Text))

Max Activating Token Index: 48

Text #12

Max Range: 0.8583. Min Range: -0.8583

Max Act: 0.6918. Min Act: -0.0001

Data Index: 708207 (C4 (Web Text))

Max Activating Token Index: 835

Text #13

Max Range: 0.8583. Min Range: -0.8583

Max Act: 0.7143. Min Act: -0.0001

Data Index: 964231 (C4 (Web Text))

Max Activating Token Index: 621

Text #14

Max Range: 0.8583. Min Range: -0.8583

Max Act: 0.7498. Min Act: -0.0001

Data Index: 1011000 (C4 (Web Text))

Max Activating Token Index: 899

Text #15

Max Range: 0.8583. Min Range: -0.8583

Max Act: 0.7261. Min Act: -0.0001

Data Index: 921235 (C4 (Web Text))

Max Activating Token Index: 771

Text #16

Max Range: 0.8583. Min Range: -0.8583

Max Act: 0.6931. Min Act: -0.0001

Data Index: 613848 (C4 (Web Text))

Max Activating Token Index: 718

Text #17

Max Range: 0.8583. Min Range: -0.8583

Max Act: 0.6986. Min Act: -0.0001

Data Index: 24452 (C4 (Web Text))

Max Activating Token Index: 908

Text #18

Max Range: 0.8583. Min Range: -0.8583

Max Act: 0.6645. Min Act: -0.0001

Data Index: 606540 (C4 (Web Text))

Max Activating Token Index: 358

Text #19

Max Range: 0.8583. Min Range: -0.8583

Max Act: 0.6772. Min Act: -0.0001

Data Index: 539274 (C4 (Web Text))

Max Activating Token Index: 888