Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4548 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.1688. Min Range: -2.1688

Max Act: 2.1688. Min Act: -0.0000

Data Index: 1101082 (C4 (Web Text))

Max Activating Token Index: 829

Text #1

Max Range: 2.1688. Min Range: -2.1688

Max Act: 2.0820. Min Act: -0.0000

Data Index: 824602 (C4 (Web Text))

Max Activating Token Index: 675

Text #2

Max Range: 2.1688. Min Range: -2.1688

Max Act: 1.9866. Min Act: -0.0000

Data Index: 1321151 (C4 (Web Text))

Max Activating Token Index: 324

Text #3

Max Range: 2.1688. Min Range: -2.1688

Max Act: 1.9853. Min Act: -0.0000

Data Index: 1340502 (C4 (Web Text))

Max Activating Token Index: 780

Text #4

Max Range: 2.1688. Min Range: -2.1688

Max Act: 2.0836. Min Act: -0.0000

Data Index: 868825 (C4 (Web Text))

Max Activating Token Index: 26

Text #5

Max Range: 2.1688. Min Range: -2.1688

Max Act: 1.9784. Min Act: -0.0000

Data Index: 1218142 (C4 (Web Text))

Max Activating Token Index: 439

Text #6

Max Range: 2.1688. Min Range: -2.1688

Max Act: 1.9450. Min Act: -0.0000

Data Index: 1034984 (C4 (Web Text))

Max Activating Token Index: 929

Text #7

Max Range: 2.1688. Min Range: -2.1688

Max Act: 1.9100. Min Act: -0.0000

Data Index: 546976 (C4 (Web Text))

Max Activating Token Index: 140

Text #8

Max Range: 2.1688. Min Range: -2.1688

Max Act: 2.0144. Min Act: -0.0000

Data Index: 348240 (C4 (Web Text))

Max Activating Token Index: 374

Text #9

Max Range: 2.1688. Min Range: -2.1688

Max Act: 1.8866. Min Act: -0.0000

Data Index: 431303 (C4 (Web Text))

Max Activating Token Index: 885

Text #10

Max Range: 2.1688. Min Range: -2.1688

Max Act: 1.8464. Min Act: -0.0000

Data Index: 1207782 (C4 (Web Text))

Max Activating Token Index: 537

Text #11

Max Range: 2.1688. Min Range: -2.1688

Max Act: 1.8383. Min Act: -0.0000

Data Index: 1283859 (C4 (Web Text))

Max Activating Token Index: 263

Text #12

Max Range: 2.1688. Min Range: -2.1688

Max Act: 1.8407. Min Act: -0.0000

Data Index: 6137 (C4 (Web Text))

Max Activating Token Index: 501

Text #13

Max Range: 2.1688. Min Range: -2.1688

Max Act: 1.6390. Min Act: -0.0000

Data Index: 244233 (C4 (Web Text))

Max Activating Token Index: 824

Text #14

Max Range: 2.1688. Min Range: -2.1688

Max Act: 1.6937. Min Act: -0.0000

Data Index: 638823 (C4 (Web Text))

Max Activating Token Index: 601

Text #15

Max Range: 2.1688. Min Range: -2.1688

Max Act: 1.7909. Min Act: -0.0000

Data Index: 983464 (C4 (Web Text))

Max Activating Token Index: 554

Text #16

Max Range: 2.1688. Min Range: -2.1688

Max Act: 1.7804. Min Act: -0.0000

Data Index: 382236 (C4 (Web Text))

Max Activating Token Index: 77

Text #17

Max Range: 2.1688. Min Range: -2.1688

Max Act: 1.7696. Min Act: -0.0000

Data Index: 1344052 (C4 (Web Text))

Max Activating Token Index: 485

Text #18

Max Range: 2.1688. Min Range: -2.1688

Max Act: 1.6691. Min Act: -0.0000

Data Index: 1197287 (C4 (Web Text))

Max Activating Token Index: 347

Text #19

Max Range: 2.1688. Min Range: -2.1688

Max Act: 1.7706. Min Act: -0.0000

Data Index: 611664 (C4 (Web Text))

Max Activating Token Index: 139