Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 567 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.3984. Min Range: -2.3984

Max Act: 2.3984. Min Act: -0.0000

Data Index: 1200144 (C4 (Web Text))

Max Activating Token Index: 1019

Text #1

Max Range: 2.3984. Min Range: -2.3984

Max Act: 2.1886. Min Act: -0.0000

Data Index: 421342 (C4 (Web Text))

Max Activating Token Index: 945

Text #2

Max Range: 2.3984. Min Range: -2.3984

Max Act: 2.1614. Min Act: -0.0000

Data Index: 900050 (C4 (Web Text))

Max Activating Token Index: 711

Text #3

Max Range: 2.3984. Min Range: -2.3984

Max Act: 1.9295. Min Act: -0.0000

Data Index: 900488 (C4 (Web Text))

Max Activating Token Index: 179

Text #4

Max Range: 2.3984. Min Range: -2.3984

Max Act: 1.9813. Min Act: -0.0000

Data Index: 1207014 (C4 (Web Text))

Max Activating Token Index: 16

Text #5

Max Range: 2.3984. Min Range: -2.3984

Max Act: 1.9475. Min Act: -0.0000

Data Index: 509030 (C4 (Web Text))

Max Activating Token Index: 339

Text #6

Max Range: 2.3984. Min Range: -2.3984

Max Act: 1.9333. Min Act: -0.0000

Data Index: 748561 (C4 (Web Text))

Max Activating Token Index: 289

Text #7

Max Range: 2.3984. Min Range: -2.3984

Max Act: 2.0356. Min Act: -0.0000

Data Index: 980717 (C4 (Web Text))

Max Activating Token Index: 141

Text #8

Max Range: 2.3984. Min Range: -2.3984

Max Act: 1.8014. Min Act: -0.0000

Data Index: 1233172 (C4 (Web Text))

Max Activating Token Index: 153

Text #9

Max Range: 2.3984. Min Range: -2.3984

Max Act: 1.7638. Min Act: -0.0000

Data Index: 1097743 (C4 (Web Text))

Max Activating Token Index: 784

Text #10

Max Range: 2.3984. Min Range: -2.3984

Max Act: 1.8534. Min Act: -0.0000

Data Index: 355875 (C4 (Web Text))

Max Activating Token Index: 752

Text #11

Max Range: 2.3984. Min Range: -2.3984

Max Act: 1.8234. Min Act: -0.0000

Data Index: 1034798 (C4 (Web Text))

Max Activating Token Index: 934

Text #12

Max Range: 2.3984. Min Range: -2.3984

Max Act: 1.7295. Min Act: -0.0000

Data Index: 628696 (C4 (Web Text))

Max Activating Token Index: 992

Text #13

Max Range: 2.3984. Min Range: -2.3984

Max Act: 1.7876. Min Act: -0.0000

Data Index: 1209254 (C4 (Web Text))

Max Activating Token Index: 574

Text #14

Max Range: 2.3984. Min Range: -2.3984

Max Act: 1.7891. Min Act: -0.0000

Data Index: 1288170 (C4 (Web Text))

Max Activating Token Index: 361

Text #15

Max Range: 2.3984. Min Range: -2.3984

Max Act: 1.8937. Min Act: -0.0000

Data Index: 882248 (C4 (Web Text))

Max Activating Token Index: 100

Text #16

Max Range: 2.3984. Min Range: -2.3984

Max Act: 1.8698. Min Act: -0.0000

Data Index: 303737 (C4 (Web Text))

Max Activating Token Index: 691

Text #17

Max Range: 2.3984. Min Range: -2.3984

Max Act: 1.6652. Min Act: -0.0000

Data Index: 762042 (C4 (Web Text))

Max Activating Token Index: 889

Text #18

Max Range: 2.3984. Min Range: -2.3984

Max Act: 1.8717. Min Act: -0.0000

Data Index: 491005 (C4 (Web Text))

Max Activating Token Index: 110

Text #19

Max Range: 2.3984. Min Range: -2.3984

Max Act: 1.6502. Min Act: -0.0000

Data Index: 333683 (C4 (Web Text))

Max Activating Token Index: 285