Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4546 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.0147. Min Range: -3.0147

Max Act: 3.0147. Min Act: -0.0000

Data Index: 1197068 (C4 (Web Text))

Max Activating Token Index: 399

Text #1

Max Range: 3.0147. Min Range: -3.0147

Max Act: 2.8162. Min Act: -0.0000

Data Index: 940227 (C4 (Web Text))

Max Activating Token Index: 989

Text #2

Max Range: 3.0147. Min Range: -3.0147

Max Act: 2.6797. Min Act: -0.0000

Data Index: 846943 (C4 (Web Text))

Max Activating Token Index: 257

Text #3

Max Range: 3.0147. Min Range: -3.0147

Max Act: 2.6596. Min Act: -0.0000

Data Index: 504235 (C4 (Web Text))

Max Activating Token Index: 619

Text #4

Max Range: 3.0147. Min Range: -3.0147

Max Act: 2.2972. Min Act: -0.0000

Data Index: 536480 (C4 (Web Text))

Max Activating Token Index: 896

Text #5

Max Range: 3.0147. Min Range: -3.0147

Max Act: 2.4075. Min Act: -0.0000

Data Index: 694569 (C4 (Web Text))

Max Activating Token Index: 837

Text #6

Max Range: 3.0147. Min Range: -3.0147

Max Act: 2.8129. Min Act: -0.0000

Data Index: 358589 (C4 (Web Text))

Max Activating Token Index: 315

Text #7

Max Range: 3.0147. Min Range: -3.0147

Max Act: 2.2495. Min Act: -0.0000

Data Index: 833468 (C4 (Web Text))

Max Activating Token Index: 204

Text #8

Max Range: 3.0147. Min Range: -3.0147

Max Act: 2.6240. Min Act: -0.0000

Data Index: 945307 (C4 (Web Text))

Max Activating Token Index: 663

Text #9

Max Range: 3.0147. Min Range: -3.0147

Max Act: 2.4970. Min Act: -0.0000

Data Index: 639599 (C4 (Web Text))

Max Activating Token Index: 715

Text #10

Max Range: 3.0147. Min Range: -3.0147

Max Act: 2.3026. Min Act: -0.0000

Data Index: 1352854 (C4 (Web Text))

Max Activating Token Index: 509

Text #11

Max Range: 3.0147. Min Range: -3.0147

Max Act: 2.1540. Min Act: -0.0000

Data Index: 947940 (C4 (Web Text))

Max Activating Token Index: 812

Text #12

Max Range: 3.0147. Min Range: -3.0147

Max Act: 2.1822. Min Act: -0.0000

Data Index: 25862 (C4 (Web Text))

Max Activating Token Index: 861

Text #13

Max Range: 3.0147. Min Range: -3.0147

Max Act: 2.2913. Min Act: -0.0000

Data Index: 1002343 (C4 (Web Text))

Max Activating Token Index: 951

Text #14

Max Range: 3.0147. Min Range: -3.0147

Max Act: 2.1508. Min Act: -0.0000

Data Index: 305190 (C4 (Web Text))

Max Activating Token Index: 758

Text #15

Max Range: 3.0147. Min Range: -3.0147

Max Act: 2.1567. Min Act: -0.0000

Data Index: 579649 (C4 (Web Text))

Max Activating Token Index: 39

Text #16

Max Range: 3.0147. Min Range: -3.0147

Max Act: 2.0301. Min Act: -0.0000

Data Index: 450727 (C4 (Web Text))

Max Activating Token Index: 265

Text #17

Max Range: 3.0147. Min Range: -3.0147

Max Act: 2.0135. Min Act: -0.0000

Data Index: 1248606 (C4 (Web Text))

Max Activating Token Index: 153

Text #18

Max Range: 3.0147. Min Range: -3.0147

Max Act: 2.1273. Min Act: -0.0000

Data Index: 1234371 (C4 (Web Text))

Max Activating Token Index: 92

Text #19

Max Range: 3.0147. Min Range: -3.0147

Max Act: 2.1359. Min Act: -0.0000

Data Index: 649547 (C4 (Web Text))

Max Activating Token Index: 18