Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2400 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.4447. Min Range: -1.4447

Max Act: 1.4447. Min Act: -0.0000

Data Index: 956972 (C4 (Web Text))

Max Activating Token Index: 110

Text #1

Max Range: 1.4447. Min Range: -1.4447

Max Act: 1.1296. Min Act: -0.0000

Data Index: 835574 (C4 (Web Text))

Max Activating Token Index: 921

Text #2

Max Range: 1.4447. Min Range: -1.4447

Max Act: 1.0645. Min Act: -0.0000

Data Index: 840036 (C4 (Web Text))

Max Activating Token Index: 645

Text #3

Max Range: 1.4447. Min Range: -1.4447

Max Act: 1.0930. Min Act: -0.0000

Data Index: 1622148 (Python Code)

Max Activating Token Index: 265

Text #4

Max Range: 1.4447. Min Range: -1.4447

Max Act: 1.0209. Min Act: -0.0000

Data Index: 1483237 (Python Code)

Max Activating Token Index: 635

Text #5

Max Range: 1.4447. Min Range: -1.4447

Max Act: 0.9539. Min Act: -0.0000

Data Index: 739597 (C4 (Web Text))

Max Activating Token Index: 175

Text #6

Max Range: 1.4447. Min Range: -1.4447

Max Act: 1.0234. Min Act: -0.0000

Data Index: 1437604 (Python Code)

Max Activating Token Index: 280

Text #7

Max Range: 1.4447. Min Range: -1.4447

Max Act: 1.1695. Min Act: -0.0000

Data Index: 543597 (C4 (Web Text))

Max Activating Token Index: 784

Text #8

Max Range: 1.4447. Min Range: -1.4447

Max Act: 1.0229. Min Act: -0.0000

Data Index: 477700 (C4 (Web Text))

Max Activating Token Index: 374

Text #9

Max Range: 1.4447. Min Range: -1.4447

Max Act: 1.0568. Min Act: -0.0000

Data Index: 381521 (C4 (Web Text))

Max Activating Token Index: 723

Text #10

Max Range: 1.4447. Min Range: -1.4447

Max Act: 1.1816. Min Act: -0.0000

Data Index: 1514942 (Python Code)

Max Activating Token Index: 279

Text #11

Max Range: 1.4447. Min Range: -1.4447

Max Act: 1.0561. Min Act: -0.0000

Data Index: 1033458 (C4 (Web Text))

Max Activating Token Index: 56

Text #12

Max Range: 1.4447. Min Range: -1.4447

Max Act: 0.9662. Min Act: -0.0000

Data Index: 1613820 (Python Code)

Max Activating Token Index: 254

Text #13

Max Range: 1.4447. Min Range: -1.4447

Max Act: 0.8886. Min Act: -0.0000

Data Index: 36084 (C4 (Web Text))

Max Activating Token Index: 729

Text #14

Max Range: 1.4447. Min Range: -1.4447

Max Act: 1.0027. Min Act: -0.0000

Data Index: 667064 (C4 (Web Text))

Max Activating Token Index: 987

Text #15

Max Range: 1.4447. Min Range: -1.4447

Max Act: 0.9957. Min Act: -0.0000

Data Index: 331037 (C4 (Web Text))

Max Activating Token Index: 465

Text #16

Max Range: 1.4447. Min Range: -1.4447

Max Act: 0.9671. Min Act: -0.0000

Data Index: 109548 (C4 (Web Text))

Max Activating Token Index: 912

Text #17

Max Range: 1.4447. Min Range: -1.4447

Max Act: 1.0179. Min Act: -0.0000

Data Index: 401149 (C4 (Web Text))

Max Activating Token Index: 909

Text #18

Max Range: 1.4447. Min Range: -1.4447

Max Act: 0.9782. Min Act: -0.0000

Data Index: 1471150 (Python Code)

Max Activating Token Index: 987

Text #19

Max Range: 1.4447. Min Range: -1.4447

Max Act: 0.9255. Min Act: -0.0000

Data Index: 1402492 (Python Code)

Max Activating Token Index: 837