Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 757 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.4847. Min Range: -2.4847

Max Act: 2.4847. Min Act: -0.0000

Data Index: 540698 (C4 (Web Text))

Max Activating Token Index: 93

Text #1

Max Range: 2.4847. Min Range: -2.4847

Max Act: 2.1518. Min Act: -0.0000

Data Index: 877930 (C4 (Web Text))

Max Activating Token Index: 104

Text #2

Max Range: 2.4847. Min Range: -2.4847

Max Act: 2.0892. Min Act: -0.0000

Data Index: 1212637 (C4 (Web Text))

Max Activating Token Index: 248

Text #3

Max Range: 2.4847. Min Range: -2.4847

Max Act: 2.2009. Min Act: -0.0000

Data Index: 725677 (C4 (Web Text))

Max Activating Token Index: 180

Text #4

Max Range: 2.4847. Min Range: -2.4847

Max Act: 2.1204. Min Act: -0.0000

Data Index: 1059926 (C4 (Web Text))

Max Activating Token Index: 208

Text #5

Max Range: 2.4847. Min Range: -2.4847

Max Act: 2.0106. Min Act: -0.0000

Data Index: 1320056 (C4 (Web Text))

Max Activating Token Index: 51

Text #6

Max Range: 2.4847. Min Range: -2.4847

Max Act: 1.9485. Min Act: -0.0000

Data Index: 1158175 (C4 (Web Text))

Max Activating Token Index: 126

Text #7

Max Range: 2.4847. Min Range: -2.4847

Max Act: 1.7820. Min Act: -0.0000

Data Index: 539389 (C4 (Web Text))

Max Activating Token Index: 106

Text #8

Max Range: 2.4847. Min Range: -2.4847

Max Act: 1.7795. Min Act: -0.0000

Data Index: 1258666 (C4 (Web Text))

Max Activating Token Index: 211

Text #9

Max Range: 2.4847. Min Range: -2.4847

Max Act: 1.8841. Min Act: -0.0000

Data Index: 1057316 (C4 (Web Text))

Max Activating Token Index: 84

Text #10

Max Range: 2.4847. Min Range: -2.4847

Max Act: 1.8255. Min Act: -0.0000

Data Index: 350382 (C4 (Web Text))

Max Activating Token Index: 462

Text #11

Max Range: 2.4847. Min Range: -2.4847

Max Act: 1.7357. Min Act: -0.0000

Data Index: 1087294 (C4 (Web Text))

Max Activating Token Index: 491

Text #12

Max Range: 2.4847. Min Range: -2.4847

Max Act: 1.6420. Min Act: -0.0000

Data Index: 141555 (C4 (Web Text))

Max Activating Token Index: 518

Text #13

Max Range: 2.4847. Min Range: -2.4847

Max Act: 1.6419. Min Act: -0.0000

Data Index: 321554 (C4 (Web Text))

Max Activating Token Index: 701

Text #14

Max Range: 2.4847. Min Range: -2.4847

Max Act: 1.8073. Min Act: -0.0000

Data Index: 912616 (C4 (Web Text))

Max Activating Token Index: 319

Text #15

Max Range: 2.4847. Min Range: -2.4847

Max Act: 1.7449. Min Act: -0.0000

Data Index: 861269 (C4 (Web Text))

Max Activating Token Index: 534

Text #16

Max Range: 2.4847. Min Range: -2.4847

Max Act: 1.5523. Min Act: -0.0000

Data Index: 189415 (C4 (Web Text))

Max Activating Token Index: 566

Text #17

Max Range: 2.4847. Min Range: -2.4847

Max Act: 1.6378. Min Act: -0.0000

Data Index: 453442 (C4 (Web Text))

Max Activating Token Index: 76

Text #18

Max Range: 2.4847. Min Range: -2.4847

Max Act: 1.6876. Min Act: -0.0000

Data Index: 283485 (C4 (Web Text))

Max Activating Token Index: 942

Text #19

Max Range: 2.4847. Min Range: -2.4847

Max Act: 1.5873. Min Act: -0.0000

Data Index: 63637 (C4 (Web Text))

Max Activating Token Index: 335