Model: SoLU Model: 3 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1858 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-3l')

Text #0

Max Range: 0.5753. Min Range: -0.5753

Max Act: 0.5753. Min Act: -0.0003

Data Index: 1232217 (C4 (Web Text))

Max Activating Token Index: 693

Text #1

Max Range: 0.5753. Min Range: -0.5753

Max Act: 0.5553. Min Act: -0.0003

Data Index: 1070345 (C4 (Web Text))

Max Activating Token Index: 327

Text #2

Max Range: 0.5753. Min Range: -0.5753

Max Act: 0.4568. Min Act: -0.0003

Data Index: 562898 (C4 (Web Text))

Max Activating Token Index: 607

Text #3

Max Range: 0.5753. Min Range: -0.5753

Max Act: 0.5163. Min Act: -0.0003

Data Index: 950955 (C4 (Web Text))

Max Activating Token Index: 370

Text #4

Max Range: 0.5753. Min Range: -0.5753

Max Act: 0.5330. Min Act: -0.0003

Data Index: 1329354 (C4 (Web Text))

Max Activating Token Index: 545

Text #5

Max Range: 0.5753. Min Range: -0.5753

Max Act: 0.4589. Min Act: -0.0003

Data Index: 967922 (C4 (Web Text))

Max Activating Token Index: 711

Text #6

Max Range: 0.5753. Min Range: -0.5753

Max Act: 0.4918. Min Act: -0.0003

Data Index: 20238 (C4 (Web Text))

Max Activating Token Index: 486

Text #7

Max Range: 0.5753. Min Range: -0.5753

Max Act: 0.4806. Min Act: -0.0003

Data Index: 847829 (C4 (Web Text))

Max Activating Token Index: 566

Text #8

Max Range: 0.5753. Min Range: -0.5753

Max Act: 0.4986. Min Act: -0.0003

Data Index: 559766 (C4 (Web Text))

Max Activating Token Index: 658

Text #9

Max Range: 0.5753. Min Range: -0.5753

Max Act: 0.4607. Min Act: -0.0003

Data Index: 1282581 (C4 (Web Text))

Max Activating Token Index: 392

Text #10

Max Range: 0.5753. Min Range: -0.5753

Max Act: 0.4367. Min Act: -0.0003

Data Index: 1262078 (C4 (Web Text))

Max Activating Token Index: 76

Text #11

Max Range: 0.5753. Min Range: -0.5753

Max Act: 0.4757. Min Act: -0.0003

Data Index: 1061667 (C4 (Web Text))

Max Activating Token Index: 676

Text #12

Max Range: 0.5753. Min Range: -0.5753

Max Act: 0.4752. Min Act: -0.0004

Data Index: 426732 (C4 (Web Text))

Max Activating Token Index: 859

Text #13

Max Range: 0.5753. Min Range: -0.5753

Max Act: 0.4401. Min Act: -0.0003

Data Index: 357861 (C4 (Web Text))

Max Activating Token Index: 727

Text #14

Max Range: 0.5753. Min Range: -0.5753

Max Act: 0.4489. Min Act: -0.0004

Data Index: 116922 (C4 (Web Text))

Max Activating Token Index: 723

Text #15

Max Range: 0.5753. Min Range: -0.5753

Max Act: 0.4345. Min Act: -0.0003

Data Index: 233691 (C4 (Web Text))

Max Activating Token Index: 312

Text #16

Max Range: 0.5753. Min Range: -0.5753

Max Act: 0.4632. Min Act: -0.0003

Data Index: 215657 (C4 (Web Text))

Max Activating Token Index: 311

Text #17

Max Range: 0.5753. Min Range: -0.5753

Max Act: 0.4419. Min Act: -0.0004

Data Index: 1242391 (C4 (Web Text))

Max Activating Token Index: 747

Text #18

Max Range: 0.5753. Min Range: -0.5753

Max Act: 0.4187. Min Act: -0.0003

Data Index: 833308 (C4 (Web Text))

Max Activating Token Index: 50

Text #19

Max Range: 0.5753. Min Range: -0.5753

Max Act: 0.4701. Min Act: -0.0003

Data Index: 441598 (C4 (Web Text))

Max Activating Token Index: 360