Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2766 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.8880. Min Range: -3.8880

Max Act: 3.8880. Min Act: -0.0000

Data Index: 793966 (C4 (Web Text))

Max Activating Token Index: 809

Text #1

Max Range: 3.8880. Min Range: -3.8880

Max Act: 3.6907. Min Act: -0.0000

Data Index: 751670 (C4 (Web Text))

Max Activating Token Index: 280

Text #2

Max Range: 3.8880. Min Range: -3.8880

Max Act: 3.4664. Min Act: -0.0000

Data Index: 707996 (C4 (Web Text))

Max Activating Token Index: 1014

Text #3

Max Range: 3.8880. Min Range: -3.8880

Max Act: 3.3846. Min Act: -0.0000

Data Index: 648763 (C4 (Web Text))

Max Activating Token Index: 675

Text #4

Max Range: 3.8880. Min Range: -3.8880

Max Act: 3.3740. Min Act: -0.0000

Data Index: 535150 (C4 (Web Text))

Max Activating Token Index: 1018

Text #5

Max Range: 3.8880. Min Range: -3.8880

Max Act: 3.3420. Min Act: -0.0000

Data Index: 1160639 (C4 (Web Text))

Max Activating Token Index: 39

Text #6

Max Range: 3.8880. Min Range: -3.8880

Max Act: 3.2416. Min Act: -0.0000

Data Index: 5538 (C4 (Web Text))

Max Activating Token Index: 973

Text #7

Max Range: 3.8880. Min Range: -3.8880

Max Act: 3.5498. Min Act: -0.0000

Data Index: 588810 (C4 (Web Text))

Max Activating Token Index: 471

Text #8

Max Range: 3.8880. Min Range: -3.8880

Max Act: 3.4793. Min Act: -0.0000

Data Index: 1327739 (C4 (Web Text))

Max Activating Token Index: 203

Text #9

Max Range: 3.8880. Min Range: -3.8880

Max Act: 3.1288. Min Act: -0.0000

Data Index: 494125 (C4 (Web Text))

Max Activating Token Index: 332

Text #10

Max Range: 3.8880. Min Range: -3.8880

Max Act: 3.1077. Min Act: -0.0000

Data Index: 1279111 (C4 (Web Text))

Max Activating Token Index: 617

Text #11

Max Range: 3.8880. Min Range: -3.8880

Max Act: 3.2804. Min Act: -0.0000

Data Index: 390831 (C4 (Web Text))

Max Activating Token Index: 766

Text #12

Max Range: 3.8880. Min Range: -3.8880

Max Act: 3.1216. Min Act: -0.0000

Data Index: 514692 (C4 (Web Text))

Max Activating Token Index: 728

Text #13

Max Range: 3.8880. Min Range: -3.8880

Max Act: 3.2352. Min Act: -0.0000

Data Index: 133393 (C4 (Web Text))

Max Activating Token Index: 249

Text #14

Max Range: 3.8880. Min Range: -3.8880

Max Act: 3.2570. Min Act: -0.0000

Data Index: 217843 (C4 (Web Text))

Max Activating Token Index: 921

Text #15

Max Range: 3.8880. Min Range: -3.8880

Max Act: 3.0937. Min Act: -0.0000

Data Index: 521148 (C4 (Web Text))

Max Activating Token Index: 129

Text #16

Max Range: 3.8880. Min Range: -3.8880

Max Act: 3.2346. Min Act: -0.0000

Data Index: 442945 (C4 (Web Text))

Max Activating Token Index: 885

Text #17

Max Range: 3.8880. Min Range: -3.8880

Max Act: 2.9030. Min Act: -0.0000

Data Index: 1128732 (C4 (Web Text))

Max Activating Token Index: 349

Text #18

Max Range: 3.8880. Min Range: -3.8880

Max Act: 3.0250. Min Act: -0.0000

Data Index: 74181 (C4 (Web Text))

Max Activating Token Index: 81

Text #19

Max Range: 3.8880. Min Range: -3.8880

Max Act: 3.1210. Min Act: -0.0000

Data Index: 964372 (C4 (Web Text))

Max Activating Token Index: 238