Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.1620. Min Range: -1.1620

Max Act: 1.1620. Min Act: -0.0000

Data Index: 441660 (C4 (Web Text))

Max Activating Token Index: 716

Text #1

Max Range: 1.1620. Min Range: -1.1620

Max Act: 1.0776. Min Act: -0.0000

Data Index: 1107810 (C4 (Web Text))

Max Activating Token Index: 476

Text #2

Max Range: 1.1620. Min Range: -1.1620

Max Act: 1.0395. Min Act: -0.0000

Data Index: 778927 (C4 (Web Text))

Max Activating Token Index: 943

Text #3

Max Range: 1.1620. Min Range: -1.1620

Max Act: 1.0131. Min Act: -0.0000

Data Index: 188032 (C4 (Web Text))

Max Activating Token Index: 252

Text #4

Max Range: 1.1620. Min Range: -1.1620

Max Act: 0.9000. Min Act: -0.0000

Data Index: 599253 (C4 (Web Text))

Max Activating Token Index: 574

Text #5

Max Range: 1.1620. Min Range: -1.1620

Max Act: 0.8102. Min Act: -0.0000

Data Index: 1009151 (C4 (Web Text))

Max Activating Token Index: 76

Text #6

Max Range: 1.1620. Min Range: -1.1620

Max Act: 0.8380. Min Act: -0.0000

Data Index: 235790 (C4 (Web Text))

Max Activating Token Index: 359

Text #7

Max Range: 1.1620. Min Range: -1.1620

Max Act: 0.8251. Min Act: -0.0000

Data Index: 357519 (C4 (Web Text))

Max Activating Token Index: 254

Text #8

Max Range: 1.1620. Min Range: -1.1620

Max Act: 0.8496. Min Act: -0.0000

Data Index: 108542 (C4 (Web Text))

Max Activating Token Index: 411

Text #9

Max Range: 1.1620. Min Range: -1.1620

Max Act: 0.8806. Min Act: -0.0000

Data Index: 1341986 (C4 (Web Text))

Max Activating Token Index: 821

Text #10

Max Range: 1.1620. Min Range: -1.1620

Max Act: 0.7224. Min Act: -0.0000

Data Index: 1017624 (C4 (Web Text))

Max Activating Token Index: 273

Text #11

Max Range: 1.1620. Min Range: -1.1620

Max Act: 0.8470. Min Act: -0.0000

Data Index: 396552 (C4 (Web Text))

Max Activating Token Index: 235

Text #12

Max Range: 1.1620. Min Range: -1.1620

Max Act: 0.7242. Min Act: -0.0000

Data Index: 161089 (C4 (Web Text))

Max Activating Token Index: 528

Text #13

Max Range: 1.1620. Min Range: -1.1620

Max Act: 0.7286. Min Act: -0.0000

Data Index: 590036 (C4 (Web Text))

Max Activating Token Index: 470

Text #14

Max Range: 1.1620. Min Range: -1.1620

Max Act: 0.6872. Min Act: -0.0000

Data Index: 322077 (C4 (Web Text))

Max Activating Token Index: 966

Text #15

Max Range: 1.1620. Min Range: -1.1620

Max Act: 0.7125. Min Act: -0.0000

Data Index: 423479 (C4 (Web Text))

Max Activating Token Index: 576

Text #16

Max Range: 1.1620. Min Range: -1.1620

Max Act: 0.7623. Min Act: -0.0000

Data Index: 843043 (C4 (Web Text))

Max Activating Token Index: 174

Text #17

Max Range: 1.1620. Min Range: -1.1620

Max Act: 0.7302. Min Act: -0.0000

Data Index: 416072 (C4 (Web Text))

Max Activating Token Index: 868

Text #18

Max Range: 1.1620. Min Range: -1.1620

Max Act: 0.6334. Min Act: -0.0000

Data Index: 1254612 (C4 (Web Text))

Max Activating Token Index: 840

Text #19

Max Range: 1.1620. Min Range: -1.1620

Max Act: 0.6924. Min Act: -0.0000

Data Index: 1289745 (C4 (Web Text))

Max Activating Token Index: 823