Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3952 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.8036. Min Range: -0.8036

Max Act: 0.8036. Min Act: -0.0000

Data Index: 1518525 (Python Code)

Max Activating Token Index: 862

Text #1

Max Range: 0.8036. Min Range: -0.8036

Max Act: 0.6416. Min Act: -0.0000

Data Index: 1553984 (Python Code)

Max Activating Token Index: 45

Text #2

Max Range: 0.8036. Min Range: -0.8036

Max Act: 0.6250. Min Act: -0.0000

Data Index: 1580521 (Python Code)

Max Activating Token Index: 137

Text #3

Max Range: 0.8036. Min Range: -0.8036

Max Act: 0.6355. Min Act: -0.0000

Data Index: 1574520 (Python Code)

Max Activating Token Index: 284

Text #4

Max Range: 0.8036. Min Range: -0.8036

Max Act: 0.6208. Min Act: -0.0000

Data Index: 1585381 (Python Code)

Max Activating Token Index: 14

Text #5

Max Range: 0.8036. Min Range: -0.8036

Max Act: 0.5730. Min Act: -0.0000

Data Index: 1545136 (Python Code)

Max Activating Token Index: 596

Text #6

Max Range: 0.8036. Min Range: -0.8036

Max Act: 0.5446. Min Act: -0.0000

Data Index: 452500 (C4 (Web Text))

Max Activating Token Index: 495

Text #7

Max Range: 0.8036. Min Range: -0.8036

Max Act: 0.5711. Min Act: -0.0000

Data Index: 1587263 (Python Code)

Max Activating Token Index: 91

Text #8

Max Range: 0.8036. Min Range: -0.8036

Max Act: 0.5494. Min Act: -0.0000

Data Index: 1505010 (Python Code)

Max Activating Token Index: 1017

Text #9

Max Range: 0.8036. Min Range: -0.8036

Max Act: 0.5458. Min Act: -0.0000

Data Index: 1467714 (Python Code)

Max Activating Token Index: 605

Text #10

Max Range: 0.8036. Min Range: -0.8036

Max Act: 0.5571. Min Act: -0.0000

Data Index: 1466277 (Python Code)

Max Activating Token Index: 770

Text #11

Max Range: 0.8036. Min Range: -0.8036

Max Act: 0.5159. Min Act: -0.0000

Data Index: 1110696 (C4 (Web Text))

Max Activating Token Index: 319

Text #12

Max Range: 0.8036. Min Range: -0.8036

Max Act: 0.5286. Min Act: -0.0000

Data Index: 1506147 (Python Code)

Max Activating Token Index: 744

Text #13

Max Range: 0.8036. Min Range: -0.8036

Max Act: 0.5014. Min Act: -0.0000

Data Index: 1382347 (Python Code)

Max Activating Token Index: 339

Text #14

Max Range: 0.8036. Min Range: -0.8036

Max Act: 0.4988. Min Act: -0.0000

Data Index: 1572141 (Python Code)

Max Activating Token Index: 399

Text #15

Max Range: 0.8036. Min Range: -0.8036

Max Act: 0.4891. Min Act: -0.0000

Data Index: 1499562 (Python Code)

Max Activating Token Index: 529

Text #16

Max Range: 0.8036. Min Range: -0.8036

Max Act: 0.5055. Min Act: -0.0000

Data Index: 1476370 (Python Code)

Max Activating Token Index: 15

Text #17

Max Range: 0.8036. Min Range: -0.8036

Max Act: 0.4849. Min Act: -0.0000

Data Index: 447265 (C4 (Web Text))

Max Activating Token Index: 997

Text #18

Max Range: 0.8036. Min Range: -0.8036

Max Act: 0.4629. Min Act: -0.0000

Data Index: 504240 (C4 (Web Text))

Max Activating Token Index: 102

Text #19

Max Range: 0.8036. Min Range: -0.8036

Max Act: 0.4426. Min Act: -0.0000

Data Index: 1512379 (Python Code)

Max Activating Token Index: 830