Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2529 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 1.8439. Min Range: -1.8439

Max Act: 1.8439. Min Act: -0.0003

Data Index: 24850 (C4 (Web Text))

Max Activating Token Index: 365

Text #1

Max Range: 1.8439. Min Range: -1.8439

Max Act: 1.7539. Min Act: -0.0003

Data Index: 936790 (C4 (Web Text))

Max Activating Token Index: 133

Text #2

Max Range: 1.8439. Min Range: -1.8439

Max Act: 1.5096. Min Act: -0.0003

Data Index: 432534 (C4 (Web Text))

Max Activating Token Index: 250

Text #3

Max Range: 1.8439. Min Range: -1.8439

Max Act: 1.6901. Min Act: -0.0003

Data Index: 1251492 (C4 (Web Text))

Max Activating Token Index: 634

Text #4

Max Range: 1.8439. Min Range: -1.8439

Max Act: 1.5362. Min Act: -0.0003

Data Index: 1059376 (C4 (Web Text))

Max Activating Token Index: 176

Text #5

Max Range: 1.8439. Min Range: -1.8439

Max Act: 1.5232. Min Act: -0.0003

Data Index: 468162 (C4 (Web Text))

Max Activating Token Index: 108

Text #6

Max Range: 1.8439. Min Range: -1.8439

Max Act: 1.6114. Min Act: -0.0003

Data Index: 1250096 (C4 (Web Text))

Max Activating Token Index: 561

Text #7

Max Range: 1.8439. Min Range: -1.8439

Max Act: 1.5957. Min Act: -0.0003

Data Index: 580760 (C4 (Web Text))

Max Activating Token Index: 391

Text #8

Max Range: 1.8439. Min Range: -1.8439

Max Act: 1.4000. Min Act: -0.0003

Data Index: 170866 (C4 (Web Text))

Max Activating Token Index: 126

Text #9

Max Range: 1.8439. Min Range: -1.8439

Max Act: 1.4669. Min Act: -0.0003

Data Index: 533318 (C4 (Web Text))

Max Activating Token Index: 36

Text #10

Max Range: 1.8439. Min Range: -1.8439

Max Act: 1.3604. Min Act: -0.0003

Data Index: 149389 (C4 (Web Text))

Max Activating Token Index: 244

Text #11

Max Range: 1.8439. Min Range: -1.8439

Max Act: 1.2232. Min Act: -0.0003

Data Index: 84754 (C4 (Web Text))

Max Activating Token Index: 883

Text #12

Max Range: 1.8439. Min Range: -1.8439

Max Act: 1.3554. Min Act: -0.0003

Data Index: 628612 (C4 (Web Text))

Max Activating Token Index: 413

Text #13

Max Range: 1.8439. Min Range: -1.8439

Max Act: 1.3983. Min Act: -0.0003

Data Index: 832138 (C4 (Web Text))

Max Activating Token Index: 63

Text #14

Max Range: 1.8439. Min Range: -1.8439

Max Act: 1.2730. Min Act: -0.0003

Data Index: 545600 (C4 (Web Text))

Max Activating Token Index: 678

Text #15

Max Range: 1.8439. Min Range: -1.8439

Max Act: 1.3345. Min Act: -0.0003

Data Index: 316581 (C4 (Web Text))

Max Activating Token Index: 962

Text #16

Max Range: 1.8439. Min Range: -1.8439

Max Act: 1.2902. Min Act: -0.0003

Data Index: 1108356 (C4 (Web Text))

Max Activating Token Index: 136

Text #17

Max Range: 1.8439. Min Range: -1.8439

Max Act: 1.1682. Min Act: -0.0003

Data Index: 507212 (C4 (Web Text))

Max Activating Token Index: 939

Text #18

Max Range: 1.8439. Min Range: -1.8439

Max Act: 1.2126. Min Act: -0.0003

Data Index: 1003371 (C4 (Web Text))

Max Activating Token Index: 185

Text #19

Max Range: 1.8439. Min Range: -1.8439

Max Act: 1.3919. Min Act: -0.0003

Data Index: 1283800 (C4 (Web Text))

Max Activating Token Index: 300