Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2107 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.5725. Min Range: -2.5725

Max Act: 2.1109. Min Act: -0.0000

Data Index: 1208874 (C4 (Web Text))

Max Activating Token Index: 677

Text #1

Max Range: 2.5725. Min Range: -2.5725

Max Act: 2.4993. Min Act: -0.0000

Data Index: 601147 (C4 (Web Text))

Max Activating Token Index: 662

Text #2

Max Range: 2.5725. Min Range: -2.5725

Max Act: 2.5725. Min Act: -0.0000

Data Index: 1006132 (C4 (Web Text))

Max Activating Token Index: 713

Text #3

Max Range: 2.5725. Min Range: -2.5725

Max Act: 2.3176. Min Act: -0.0000

Data Index: 674421 (C4 (Web Text))

Max Activating Token Index: 954

Text #4

Max Range: 2.5725. Min Range: -2.5725

Max Act: 2.3955. Min Act: -0.0000

Data Index: 30491 (C4 (Web Text))

Max Activating Token Index: 941

Text #5

Max Range: 2.5725. Min Range: -2.5725

Max Act: 2.4282. Min Act: -0.0000

Data Index: 1143521 (C4 (Web Text))

Max Activating Token Index: 78

Text #6

Max Range: 2.5725. Min Range: -2.5725

Max Act: 2.4294. Min Act: -0.0000

Data Index: 309571 (C4 (Web Text))

Max Activating Token Index: 353

Text #7

Max Range: 2.5725. Min Range: -2.5725

Max Act: 2.5182. Min Act: -0.0000

Data Index: 640221 (C4 (Web Text))

Max Activating Token Index: 947

Text #8

Max Range: 2.5725. Min Range: -2.5725

Max Act: 2.1698. Min Act: -0.0000

Data Index: 1127822 (C4 (Web Text))

Max Activating Token Index: 759

Text #9

Max Range: 2.5725. Min Range: -2.5725

Max Act: 2.4844. Min Act: -0.0000

Data Index: 1101426 (C4 (Web Text))

Max Activating Token Index: 46

Text #10

Max Range: 2.5725. Min Range: -2.5725

Max Act: 2.1246. Min Act: -0.0000

Data Index: 444916 (C4 (Web Text))

Max Activating Token Index: 355

Text #11

Max Range: 2.5725. Min Range: -2.5725

Max Act: 2.2169. Min Act: -0.0000

Data Index: 155818 (C4 (Web Text))

Max Activating Token Index: 948

Text #12

Max Range: 2.5725. Min Range: -2.5725

Max Act: 2.3367. Min Act: -0.0000

Data Index: 1000705 (C4 (Web Text))

Max Activating Token Index: 287

Text #13

Max Range: 2.5725. Min Range: -2.5725

Max Act: 2.2358. Min Act: -0.0000

Data Index: 794415 (C4 (Web Text))

Max Activating Token Index: 749

Text #14

Max Range: 2.5725. Min Range: -2.5725

Max Act: 2.3499. Min Act: -0.0000

Data Index: 790868 (C4 (Web Text))

Max Activating Token Index: 537

Text #15

Max Range: 2.5725. Min Range: -2.5725

Max Act: 2.0963. Min Act: -0.0000

Data Index: 499775 (C4 (Web Text))

Max Activating Token Index: 182

Text #16

Max Range: 2.5725. Min Range: -2.5725

Max Act: 2.4789. Min Act: -0.0000

Data Index: 1193553 (C4 (Web Text))

Max Activating Token Index: 825

Text #17

Max Range: 2.5725. Min Range: -2.5725

Max Act: 2.1957. Min Act: -0.0000

Data Index: 1126110 (C4 (Web Text))

Max Activating Token Index: 725

Text #18

Max Range: 2.5725. Min Range: -2.5725

Max Act: 2.0765. Min Act: -0.0000

Data Index: 580724 (C4 (Web Text))

Max Activating Token Index: 810

Text #19

Max Range: 2.5725. Min Range: -2.5725

Max Act: 1.8722. Min Act: -0.0000

Data Index: 1009587 (C4 (Web Text))

Max Activating Token Index: 731