Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4580 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.3451. Min Range: -0.3451

Max Act: 0.3451. Min Act: -0.0000

Data Index: 396341 (C4 (Web Text))

Max Activating Token Index: 668

Text #1

Max Range: 0.3451. Min Range: -0.3451

Max Act: 0.2987. Min Act: 0.0001

Data Index: 47490 (C4 (Web Text))

Max Activating Token Index: 199

Text #2

Max Range: 0.3451. Min Range: -0.3451

Max Act: 0.3026. Min Act: 0.0001

Data Index: 1292435 (C4 (Web Text))

Max Activating Token Index: 236

Text #3

Max Range: 0.3451. Min Range: -0.3451

Max Act: 0.2699. Min Act: 0.0001

Data Index: 509902 (C4 (Web Text))

Max Activating Token Index: 815

Text #4

Max Range: 0.3451. Min Range: -0.3451

Max Act: 0.2534. Min Act: -0.0000

Data Index: 650644 (C4 (Web Text))

Max Activating Token Index: 544

Text #5

Max Range: 0.3451. Min Range: -0.3451

Max Act: 0.2858. Min Act: -0.0000

Data Index: 272295 (C4 (Web Text))

Max Activating Token Index: 472

Text #6

Max Range: 0.3451. Min Range: -0.3451

Max Act: 0.2872. Min Act: -0.0000

Data Index: 975120 (C4 (Web Text))

Max Activating Token Index: 280

Text #7

Max Range: 0.3451. Min Range: -0.3451

Max Act: 0.2425. Min Act: 0.0000

Data Index: 110110 (C4 (Web Text))

Max Activating Token Index: 493

Text #8

Max Range: 0.3451. Min Range: -0.3451

Max Act: 0.2478. Min Act: 0.0001

Data Index: 1141836 (C4 (Web Text))

Max Activating Token Index: 125

Text #9

Max Range: 0.3451. Min Range: -0.3451

Max Act: 0.2430. Min Act: -0.0000

Data Index: 18072 (C4 (Web Text))

Max Activating Token Index: 645

Text #10

Max Range: 0.3451. Min Range: -0.3451

Max Act: 0.2261. Min Act: -0.0000

Data Index: 1289502 (C4 (Web Text))

Max Activating Token Index: 608

Text #11

Max Range: 0.3451. Min Range: -0.3451

Max Act: 0.2435. Min Act: 0.0000

Data Index: 324009 (C4 (Web Text))

Max Activating Token Index: 333

Text #12

Max Range: 0.3451. Min Range: -0.3451

Max Act: 0.2435. Min Act: -0.0000

Data Index: 1290729 (C4 (Web Text))

Max Activating Token Index: 458

Text #13

Max Range: 0.3451. Min Range: -0.3451

Max Act: 0.2443. Min Act: -0.0000

Data Index: 501954 (C4 (Web Text))

Max Activating Token Index: 368

Text #14

Max Range: 0.3451. Min Range: -0.3451

Max Act: 0.2466. Min Act: 0.0001

Data Index: 874324 (C4 (Web Text))

Max Activating Token Index: 430

Text #15

Max Range: 0.3451. Min Range: -0.3451

Max Act: 0.2343. Min Act: 0.0000

Data Index: 99347 (C4 (Web Text))

Max Activating Token Index: 632

Text #16

Max Range: 0.3451. Min Range: -0.3451

Max Act: 0.2305. Min Act: -0.0000

Data Index: 71229 (C4 (Web Text))

Max Activating Token Index: 945

Text #17

Max Range: 0.3451. Min Range: -0.3451

Max Act: 0.2318. Min Act: -0.0000

Data Index: 1325399 (C4 (Web Text))

Max Activating Token Index: 844

Text #18

Max Range: 0.3451. Min Range: -0.3451

Max Act: 0.2284. Min Act: 0.0000

Data Index: 510553 (C4 (Web Text))

Max Activating Token Index: 90

Text #19

Max Range: 0.3451. Min Range: -0.3451

Max Act: 0.2254. Min Act: -0.0000

Data Index: 1085174 (C4 (Web Text))

Max Activating Token Index: 236