Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2092 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 5.4549. Min Range: -5.4549

Max Act: 5.3093. Min Act: -0.0001

Data Index: 534485 (C4 (Web Text))

Max Activating Token Index: 841

Text #1

Max Range: 5.4549. Min Range: -5.4549

Max Act: 5.4549. Min Act: -0.0000

Data Index: 146655 (C4 (Web Text))

Max Activating Token Index: 341

Text #2

Max Range: 5.4549. Min Range: -5.4549

Max Act: 4.3419. Min Act: -0.0001

Data Index: 258912 (C4 (Web Text))

Max Activating Token Index: 601

Text #3

Max Range: 5.4549. Min Range: -5.4549

Max Act: 5.1998. Min Act: -0.0000

Data Index: 542130 (C4 (Web Text))

Max Activating Token Index: 923

Text #4

Max Range: 5.4549. Min Range: -5.4549

Max Act: 5.1524. Min Act: -0.0001

Data Index: 690388 (C4 (Web Text))

Max Activating Token Index: 765

Text #5

Max Range: 5.4549. Min Range: -5.4549

Max Act: 5.4538. Min Act: -0.0001

Data Index: 110792 (C4 (Web Text))

Max Activating Token Index: 396

Text #6

Max Range: 5.4549. Min Range: -5.4549

Max Act: 4.5269. Min Act: -0.0000

Data Index: 1099206 (C4 (Web Text))

Max Activating Token Index: 134

Text #7

Max Range: 5.4549. Min Range: -5.4549

Max Act: 4.7639. Min Act: -0.0001

Data Index: 562009 (C4 (Web Text))

Max Activating Token Index: 891

Text #8

Max Range: 5.4549. Min Range: -5.4549

Max Act: 4.5895. Min Act: -0.0000

Data Index: 351287 (C4 (Web Text))

Max Activating Token Index: 602

Text #9

Max Range: 5.4549. Min Range: -5.4549

Max Act: 4.5103. Min Act: -0.0000

Data Index: 944083 (C4 (Web Text))

Max Activating Token Index: 200

Text #10

Max Range: 5.4549. Min Range: -5.4549

Max Act: 4.4913. Min Act: -0.0001

Data Index: 624728 (C4 (Web Text))

Max Activating Token Index: 927

Text #11

Max Range: 5.4549. Min Range: -5.4549

Max Act: 4.0449. Min Act: -0.0000

Data Index: 486734 (C4 (Web Text))

Max Activating Token Index: 681

Text #12

Max Range: 5.4549. Min Range: -5.4549

Max Act: 4.8998. Min Act: -0.0001

Data Index: 685889 (C4 (Web Text))

Max Activating Token Index: 941

Text #13

Max Range: 5.4549. Min Range: -5.4549

Max Act: 4.5266. Min Act: -0.0001

Data Index: 280826 (C4 (Web Text))

Max Activating Token Index: 527

Text #14

Max Range: 5.4549. Min Range: -5.4549

Max Act: 4.6880. Min Act: -0.0001

Data Index: 615231 (C4 (Web Text))

Max Activating Token Index: 595

Text #15

Max Range: 5.4549. Min Range: -5.4549

Max Act: 4.2994. Min Act: -0.0001

Data Index: 698902 (C4 (Web Text))

Max Activating Token Index: 707

Text #16

Max Range: 5.4549. Min Range: -5.4549

Max Act: 4.6611. Min Act: -0.0001

Data Index: 1350307 (C4 (Web Text))

Max Activating Token Index: 270

Text #17

Max Range: 5.4549. Min Range: -5.4549

Max Act: 4.7974. Min Act: -0.0001

Data Index: 1148273 (C4 (Web Text))

Max Activating Token Index: 802

Text #18

Max Range: 5.4549. Min Range: -5.4549

Max Act: 4.4290. Min Act: -0.0001

Data Index: 593457 (C4 (Web Text))

Max Activating Token Index: 588

Text #19

Max Range: 5.4549. Min Range: -5.4549

Max Act: 4.0870. Min Act: -0.0001

Data Index: 978703 (C4 (Web Text))

Max Activating Token Index: 709