Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3840 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.7055. Min Range: -1.7055

Max Act: 1.7055. Min Act: -0.0000

Data Index: 1242736 (C4 (Web Text))

Max Activating Token Index: 219

Text #1

Max Range: 1.7055. Min Range: -1.7055

Max Act: 1.5883. Min Act: -0.0000

Data Index: 483892 (C4 (Web Text))

Max Activating Token Index: 615

Text #2

Max Range: 1.7055. Min Range: -1.7055

Max Act: 1.5622. Min Act: -0.0000

Data Index: 888867 (C4 (Web Text))

Max Activating Token Index: 215

Text #3

Max Range: 1.7055. Min Range: -1.7055

Max Act: 1.5484. Min Act: -0.0000

Data Index: 405379 (C4 (Web Text))

Max Activating Token Index: 310

Text #4

Max Range: 1.7055. Min Range: -1.7055

Max Act: 1.3967. Min Act: -0.0000

Data Index: 989098 (C4 (Web Text))

Max Activating Token Index: 734

Text #5

Max Range: 1.7055. Min Range: -1.7055

Max Act: 1.3894. Min Act: -0.0000

Data Index: 915353 (C4 (Web Text))

Max Activating Token Index: 707

Text #6

Max Range: 1.7055. Min Range: -1.7055

Max Act: 1.3219. Min Act: -0.0000

Data Index: 743896 (C4 (Web Text))

Max Activating Token Index: 274

Text #7

Max Range: 1.7055. Min Range: -1.7055

Max Act: 1.2313. Min Act: -0.0000

Data Index: 963285 (C4 (Web Text))

Max Activating Token Index: 798

Text #8

Max Range: 1.7055. Min Range: -1.7055

Max Act: 1.1593. Min Act: -0.0000

Data Index: 117796 (C4 (Web Text))

Max Activating Token Index: 221

Text #9

Max Range: 1.7055. Min Range: -1.7055

Max Act: 1.0264. Min Act: -0.0000

Data Index: 608994 (C4 (Web Text))

Max Activating Token Index: 820

Text #10

Max Range: 1.7055. Min Range: -1.7055

Max Act: 1.1125. Min Act: -0.0000

Data Index: 392503 (C4 (Web Text))

Max Activating Token Index: 887

Text #11

Max Range: 1.7055. Min Range: -1.7055

Max Act: 1.0318. Min Act: -0.0000

Data Index: 469406 (C4 (Web Text))

Max Activating Token Index: 743

Text #12

Max Range: 1.7055. Min Range: -1.7055

Max Act: 1.0678. Min Act: -0.0000

Data Index: 18508 (C4 (Web Text))

Max Activating Token Index: 93

Text #13

Max Range: 1.7055. Min Range: -1.7055

Max Act: 1.0671. Min Act: -0.0000

Data Index: 139930 (C4 (Web Text))

Max Activating Token Index: 710

Text #14

Max Range: 1.7055. Min Range: -1.7055

Max Act: 1.0094. Min Act: -0.0000

Data Index: 1337038 (C4 (Web Text))

Max Activating Token Index: 127

Text #15

Max Range: 1.7055. Min Range: -1.7055

Max Act: 0.9646. Min Act: -0.0000

Data Index: 137173 (C4 (Web Text))

Max Activating Token Index: 984

Text #16

Max Range: 1.7055. Min Range: -1.7055

Max Act: 1.0008. Min Act: -0.0000

Data Index: 1259031 (C4 (Web Text))

Max Activating Token Index: 768

Text #17

Max Range: 1.7055. Min Range: -1.7055

Max Act: 1.0220. Min Act: -0.0000

Data Index: 287446 (C4 (Web Text))

Max Activating Token Index: 648

Text #18

Max Range: 1.7055. Min Range: -1.7055

Max Act: 0.9224. Min Act: -0.0000

Data Index: 1136316 (C4 (Web Text))

Max Activating Token Index: 453

Text #19

Max Range: 1.7055. Min Range: -1.7055

Max Act: 1.0086. Min Act: -0.0000

Data Index: 880055 (C4 (Web Text))

Max Activating Token Index: 100