Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2708 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.4408. Min Range: -1.4408

Max Act: 1.4408. Min Act: -0.0000

Data Index: 1539755 (Python Code)

Max Activating Token Index: 657

Text #1

Max Range: 1.4408. Min Range: -1.4408

Max Act: 1.1509. Min Act: -0.0000

Data Index: 1504289 (Python Code)

Max Activating Token Index: 1022

Text #2

Max Range: 1.4408. Min Range: -1.4408

Max Act: 1.2727. Min Act: -0.0000

Data Index: 124073 (C4 (Web Text))

Max Activating Token Index: 115

Text #3

Max Range: 1.4408. Min Range: -1.4408

Max Act: 1.0503. Min Act: -0.0000

Data Index: 1340071 (C4 (Web Text))

Max Activating Token Index: 646

Text #4

Max Range: 1.4408. Min Range: -1.4408

Max Act: 1.1342. Min Act: -0.0000

Data Index: 1260624 (C4 (Web Text))

Max Activating Token Index: 329

Text #5

Max Range: 1.4408. Min Range: -1.4408

Max Act: 0.9816. Min Act: -0.0000

Data Index: 1256753 (C4 (Web Text))

Max Activating Token Index: 363

Text #6

Max Range: 1.4408. Min Range: -1.4408

Max Act: 1.0427. Min Act: -0.0000

Data Index: 884731 (C4 (Web Text))

Max Activating Token Index: 108

Text #7

Max Range: 1.4408. Min Range: -1.4408

Max Act: 1.0937. Min Act: -0.0000

Data Index: 1383392 (Python Code)

Max Activating Token Index: 623

Text #8

Max Range: 1.4408. Min Range: -1.4408

Max Act: 1.0971. Min Act: -0.0000

Data Index: 1610041 (Python Code)

Max Activating Token Index: 804

Text #9

Max Range: 1.4408. Min Range: -1.4408

Max Act: 0.9956. Min Act: -0.0000

Data Index: 1508149 (Python Code)

Max Activating Token Index: 249

Text #10

Max Range: 1.4408. Min Range: -1.4408

Max Act: 1.0034. Min Act: -0.0000

Data Index: 844328 (C4 (Web Text))

Max Activating Token Index: 131

Text #11

Max Range: 1.4408. Min Range: -1.4408

Max Act: 0.9021. Min Act: -0.0000

Data Index: 1625231 (Python Code)

Max Activating Token Index: 874

Text #12

Max Range: 1.4408. Min Range: -1.4408

Max Act: 0.9374. Min Act: -0.0000

Data Index: 43853 (C4 (Web Text))

Max Activating Token Index: 451

Text #13

Max Range: 1.4408. Min Range: -1.4408

Max Act: 1.0082. Min Act: -0.0000

Data Index: 1613646 (Python Code)

Max Activating Token Index: 173

Text #14

Max Range: 1.4408. Min Range: -1.4408

Max Act: 0.9509. Min Act: -0.0000

Data Index: 1498547 (Python Code)

Max Activating Token Index: 496

Text #15

Max Range: 1.4408. Min Range: -1.4408

Max Act: 0.9523. Min Act: -0.0000

Data Index: 1466800 (Python Code)

Max Activating Token Index: 651

Text #16

Max Range: 1.4408. Min Range: -1.4408

Max Act: 0.8927. Min Act: -0.0000

Data Index: 905731 (C4 (Web Text))

Max Activating Token Index: 648

Text #17

Max Range: 1.4408. Min Range: -1.4408

Max Act: 0.9153. Min Act: -0.0000

Data Index: 990452 (C4 (Web Text))

Max Activating Token Index: 788

Text #18

Max Range: 1.4408. Min Range: -1.4408

Max Act: 0.8904. Min Act: -0.0000

Data Index: 1466678 (Python Code)

Max Activating Token Index: 521

Text #19

Max Range: 1.4408. Min Range: -1.4408

Max Act: 0.8851. Min Act: -0.0000

Data Index: 628972 (C4 (Web Text))

Max Activating Token Index: 71