Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 175 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.3389. Min Range: -0.3389

Max Act: 0.3365. Min Act: -0.0000

Data Index: 781565 (C4 (Web Text))

Max Activating Token Index: 377

Text #1

Max Range: 0.3389. Min Range: -0.3389

Max Act: 0.3389. Min Act: -0.0000

Data Index: 1041894 (C4 (Web Text))

Max Activating Token Index: 533

Text #2

Max Range: 0.3389. Min Range: -0.3389

Max Act: 0.3137. Min Act: -0.0000

Data Index: 1093118 (C4 (Web Text))

Max Activating Token Index: 513

Text #3

Max Range: 0.3389. Min Range: -0.3389

Max Act: 0.3199. Min Act: 0.0000

Data Index: 120619 (C4 (Web Text))

Max Activating Token Index: 270

Text #4

Max Range: 0.3389. Min Range: -0.3389

Max Act: 0.3103. Min Act: -0.0000

Data Index: 780698 (C4 (Web Text))

Max Activating Token Index: 99

Text #5

Max Range: 0.3389. Min Range: -0.3389

Max Act: 0.2987. Min Act: -0.0000

Data Index: 345138 (C4 (Web Text))

Max Activating Token Index: 83

Text #6

Max Range: 0.3389. Min Range: -0.3389

Max Act: 0.3091. Min Act: -0.0000

Data Index: 976549 (C4 (Web Text))

Max Activating Token Index: 116

Text #7

Max Range: 0.3389. Min Range: -0.3389

Max Act: 0.3041. Min Act: -0.0000

Data Index: 198432 (C4 (Web Text))

Max Activating Token Index: 611

Text #8

Max Range: 0.3389. Min Range: -0.3389

Max Act: 0.2795. Min Act: -0.0000

Data Index: 1176708 (C4 (Web Text))

Max Activating Token Index: 1011

Text #9

Max Range: 0.3389. Min Range: -0.3389

Max Act: 0.3010. Min Act: -0.0000

Data Index: 478989 (C4 (Web Text))

Max Activating Token Index: 439

Text #10

Max Range: 0.3389. Min Range: -0.3389

Max Act: 0.2761. Min Act: -0.0000

Data Index: 1289739 (C4 (Web Text))

Max Activating Token Index: 940

Text #11

Max Range: 0.3389. Min Range: -0.3389

Max Act: 0.2482. Min Act: -0.0000

Data Index: 199913 (C4 (Web Text))

Max Activating Token Index: 719

Text #12

Max Range: 0.3389. Min Range: -0.3389

Max Act: 0.2622. Min Act: -0.0000

Data Index: 1289006 (C4 (Web Text))

Max Activating Token Index: 431

Text #13

Max Range: 0.3389. Min Range: -0.3389

Max Act: 0.2618. Min Act: -0.0000

Data Index: 1358615 (C4 (Web Text))

Max Activating Token Index: 265

Text #14

Max Range: 0.3389. Min Range: -0.3389

Max Act: 0.2583. Min Act: -0.0000

Data Index: 799577 (C4 (Web Text))

Max Activating Token Index: 19

Text #15

Max Range: 0.3389. Min Range: -0.3389

Max Act: 0.2463. Min Act: -0.0000

Data Index: 1242884 (C4 (Web Text))

Max Activating Token Index: 157

Text #16

Max Range: 0.3389. Min Range: -0.3389

Max Act: 0.2523. Min Act: -0.0000

Data Index: 553524 (C4 (Web Text))

Max Activating Token Index: 28

Text #17

Max Range: 0.3389. Min Range: -0.3389

Max Act: 0.2610. Min Act: -0.0000

Data Index: 276440 (C4 (Web Text))

Max Activating Token Index: 280

Text #18

Max Range: 0.3389. Min Range: -0.3389

Max Act: 0.2486. Min Act: -0.0000

Data Index: 18216 (C4 (Web Text))

Max Activating Token Index: 313

Text #19

Max Range: 0.3389. Min Range: -0.3389

Max Act: 0.2594. Min Act: -0.0000

Data Index: 1111897 (C4 (Web Text))

Max Activating Token Index: 359