Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 176 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.1038. Min Range: -2.1038

Max Act: 2.1038. Min Act: -0.0000

Data Index: 1342942 (C4 (Web Text))

Max Activating Token Index: 482

Text #1

Max Range: 2.1038. Min Range: -2.1038

Max Act: 1.9031. Min Act: -0.0000

Data Index: 148122 (C4 (Web Text))

Max Activating Token Index: 566

Text #2

Max Range: 2.1038. Min Range: -2.1038

Max Act: 1.6976. Min Act: -0.0000

Data Index: 700204 (C4 (Web Text))

Max Activating Token Index: 798

Text #3

Max Range: 2.1038. Min Range: -2.1038

Max Act: 1.6198. Min Act: -0.0000

Data Index: 1107601 (C4 (Web Text))

Max Activating Token Index: 147

Text #4

Max Range: 2.1038. Min Range: -2.1038

Max Act: 1.6709. Min Act: -0.0000

Data Index: 273294 (C4 (Web Text))

Max Activating Token Index: 856

Text #5

Max Range: 2.1038. Min Range: -2.1038

Max Act: 1.6015. Min Act: -0.0000

Data Index: 116291 (C4 (Web Text))

Max Activating Token Index: 319

Text #6

Max Range: 2.1038. Min Range: -2.1038

Max Act: 1.5645. Min Act: -0.0000

Data Index: 514277 (C4 (Web Text))

Max Activating Token Index: 572

Text #7

Max Range: 2.1038. Min Range: -2.1038

Max Act: 1.6095. Min Act: -0.0000

Data Index: 608442 (C4 (Web Text))

Max Activating Token Index: 840

Text #8

Max Range: 2.1038. Min Range: -2.1038

Max Act: 1.6396. Min Act: -0.0000

Data Index: 861568 (C4 (Web Text))

Max Activating Token Index: 175

Text #9

Max Range: 2.1038. Min Range: -2.1038

Max Act: 1.6419. Min Act: -0.0000

Data Index: 672933 (C4 (Web Text))

Max Activating Token Index: 750

Text #10

Max Range: 2.1038. Min Range: -2.1038

Max Act: 1.5584. Min Act: -0.0000

Data Index: 983470 (C4 (Web Text))

Max Activating Token Index: 503

Text #11

Max Range: 2.1038. Min Range: -2.1038

Max Act: 1.6642. Min Act: -0.0000

Data Index: 28210 (C4 (Web Text))

Max Activating Token Index: 236

Text #12

Max Range: 2.1038. Min Range: -2.1038

Max Act: 1.5652. Min Act: -0.0000

Data Index: 1082016 (C4 (Web Text))

Max Activating Token Index: 481

Text #13

Max Range: 2.1038. Min Range: -2.1038

Max Act: 1.5491. Min Act: -0.0000

Data Index: 461245 (C4 (Web Text))

Max Activating Token Index: 481

Text #14

Max Range: 2.1038. Min Range: -2.1038

Max Act: 1.5468. Min Act: -0.0000

Data Index: 11699 (C4 (Web Text))

Max Activating Token Index: 478

Text #15

Max Range: 2.1038. Min Range: -2.1038

Max Act: 1.5032. Min Act: -0.0000

Data Index: 430275 (C4 (Web Text))

Max Activating Token Index: 931

Text #16

Max Range: 2.1038. Min Range: -2.1038

Max Act: 1.5650. Min Act: -0.0000

Data Index: 418648 (C4 (Web Text))

Max Activating Token Index: 568

Text #17

Max Range: 2.1038. Min Range: -2.1038

Max Act: 1.5872. Min Act: -0.0000

Data Index: 554449 (C4 (Web Text))

Max Activating Token Index: 825

Text #18

Max Range: 2.1038. Min Range: -2.1038

Max Act: 1.5415. Min Act: -0.0000

Data Index: 1339315 (C4 (Web Text))

Max Activating Token Index: 348

Text #19

Max Range: 2.1038. Min Range: -2.1038

Max Act: 1.4935. Min Act: -0.0000

Data Index: 448613 (C4 (Web Text))

Max Activating Token Index: 418