Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 957 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.8207. Min Range: -0.8207

Max Act: 0.8207. Min Act: -0.0000

Data Index: 1253867 (C4 (Web Text))

Max Activating Token Index: 414

Text #1

Max Range: 0.8207. Min Range: -0.8207

Max Act: 0.7893. Min Act: -0.0000

Data Index: 974407 (C4 (Web Text))

Max Activating Token Index: 710

Text #2

Max Range: 0.8207. Min Range: -0.8207

Max Act: 0.7448. Min Act: -0.0001

Data Index: 1113144 (C4 (Web Text))

Max Activating Token Index: 332

Text #3

Max Range: 0.8207. Min Range: -0.8207

Max Act: 0.7554. Min Act: -0.0000

Data Index: 703113 (C4 (Web Text))

Max Activating Token Index: 992

Text #4

Max Range: 0.8207. Min Range: -0.8207

Max Act: 0.6999. Min Act: -0.0000

Data Index: 502494 (C4 (Web Text))

Max Activating Token Index: 506

Text #5

Max Range: 0.8207. Min Range: -0.8207

Max Act: 0.7185. Min Act: -0.0000

Data Index: 545371 (C4 (Web Text))

Max Activating Token Index: 541

Text #6

Max Range: 0.8207. Min Range: -0.8207

Max Act: 0.6850. Min Act: -0.0000

Data Index: 549847 (C4 (Web Text))

Max Activating Token Index: 320

Text #7

Max Range: 0.8207. Min Range: -0.8207

Max Act: 0.7256. Min Act: -0.0000

Data Index: 690216 (C4 (Web Text))

Max Activating Token Index: 383

Text #8

Max Range: 0.8207. Min Range: -0.8207

Max Act: 0.7211. Min Act: -0.0000

Data Index: 1019643 (C4 (Web Text))

Max Activating Token Index: 633

Text #9

Max Range: 0.8207. Min Range: -0.8207

Max Act: 0.7140. Min Act: -0.0000

Data Index: 1115196 (C4 (Web Text))

Max Activating Token Index: 450

Text #10

Max Range: 0.8207. Min Range: -0.8207

Max Act: 0.6880. Min Act: -0.0001

Data Index: 892546 (C4 (Web Text))

Max Activating Token Index: 291

Text #11

Max Range: 0.8207. Min Range: -0.8207

Max Act: 0.6893. Min Act: -0.0000

Data Index: 682810 (C4 (Web Text))

Max Activating Token Index: 690

Text #12

Max Range: 0.8207. Min Range: -0.8207

Max Act: 0.6666. Min Act: -0.0000

Data Index: 655979 (C4 (Web Text))

Max Activating Token Index: 225

Text #13

Max Range: 0.8207. Min Range: -0.8207

Max Act: 0.6435. Min Act: -0.0000

Data Index: 1122105 (C4 (Web Text))

Max Activating Token Index: 193

Text #14

Max Range: 0.8207. Min Range: -0.8207

Max Act: 0.6953. Min Act: -0.0000

Data Index: 239075 (C4 (Web Text))

Max Activating Token Index: 364

Text #15

Max Range: 0.8207. Min Range: -0.8207

Max Act: 0.6924. Min Act: -0.0000

Data Index: 1345608 (C4 (Web Text))

Max Activating Token Index: 775

Text #16

Max Range: 0.8207. Min Range: -0.8207

Max Act: 0.6850. Min Act: -0.0000

Data Index: 1310208 (C4 (Web Text))

Max Activating Token Index: 111

Text #17

Max Range: 0.8207. Min Range: -0.8207

Max Act: 0.6308. Min Act: -0.0000

Data Index: 574993 (C4 (Web Text))

Max Activating Token Index: 224

Text #18

Max Range: 0.8207. Min Range: -0.8207

Max Act: 0.6624. Min Act: -0.0000

Data Index: 1194602 (C4 (Web Text))

Max Activating Token Index: 485

Text #19

Max Range: 0.8207. Min Range: -0.8207

Max Act: 0.6405. Min Act: -0.0000

Data Index: 240866 (C4 (Web Text))

Max Activating Token Index: 197