Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1968 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 4.7816. Min Range: -4.7816

Max Act: 4.4365. Min Act: -0.0000

Data Index: 1066508 (C4 (Web Text))

Max Activating Token Index: 110

Text #1

Max Range: 4.7816. Min Range: -4.7816

Max Act: 4.7816. Min Act: -0.0000

Data Index: 100194 (C4 (Web Text))

Max Activating Token Index: 842

Text #2

Max Range: 4.7816. Min Range: -4.7816

Max Act: 4.4386. Min Act: -0.0000

Data Index: 609756 (C4 (Web Text))

Max Activating Token Index: 196

Text #3

Max Range: 4.7816. Min Range: -4.7816

Max Act: 4.4426. Min Act: -0.0000

Data Index: 1139868 (C4 (Web Text))

Max Activating Token Index: 13

Text #4

Max Range: 4.7816. Min Range: -4.7816

Max Act: 4.3949. Min Act: -0.0000

Data Index: 1005239 (C4 (Web Text))

Max Activating Token Index: 815

Text #5

Max Range: 4.7816. Min Range: -4.7816

Max Act: 4.2123. Min Act: -0.0000

Data Index: 855115 (C4 (Web Text))

Max Activating Token Index: 816

Text #6

Max Range: 4.7816. Min Range: -4.7816

Max Act: 4.2194. Min Act: -0.0000

Data Index: 1060908 (C4 (Web Text))

Max Activating Token Index: 534

Text #7

Max Range: 4.7816. Min Range: -4.7816

Max Act: 3.9685. Min Act: -0.0000

Data Index: 926682 (C4 (Web Text))

Max Activating Token Index: 406

Text #8

Max Range: 4.7816. Min Range: -4.7816

Max Act: 3.7665. Min Act: -0.0000

Data Index: 110261 (C4 (Web Text))

Max Activating Token Index: 324

Text #9

Max Range: 4.7816. Min Range: -4.7816

Max Act: 3.9073. Min Act: -0.0000

Data Index: 389984 (C4 (Web Text))

Max Activating Token Index: 418

Text #10

Max Range: 4.7816. Min Range: -4.7816

Max Act: 3.7998. Min Act: -0.0000

Data Index: 550138 (C4 (Web Text))

Max Activating Token Index: 685

Text #11

Max Range: 4.7816. Min Range: -4.7816

Max Act: 3.7397. Min Act: -0.0000

Data Index: 1181780 (C4 (Web Text))

Max Activating Token Index: 81

Text #12

Max Range: 4.7816. Min Range: -4.7816

Max Act: 3.7332. Min Act: -0.0000

Data Index: 975054 (C4 (Web Text))

Max Activating Token Index: 32

Text #13

Max Range: 4.7816. Min Range: -4.7816

Max Act: 3.7260. Min Act: -0.0000

Data Index: 867848 (C4 (Web Text))

Max Activating Token Index: 686

Text #14

Max Range: 4.7816. Min Range: -4.7816

Max Act: 3.6354. Min Act: -0.0000

Data Index: 255360 (C4 (Web Text))

Max Activating Token Index: 414

Text #15

Max Range: 4.7816. Min Range: -4.7816

Max Act: 3.6125. Min Act: -0.0000

Data Index: 152488 (C4 (Web Text))

Max Activating Token Index: 19

Text #16

Max Range: 4.7816. Min Range: -4.7816

Max Act: 3.7843. Min Act: -0.0000

Data Index: 570421 (C4 (Web Text))

Max Activating Token Index: 299

Text #17

Max Range: 4.7816. Min Range: -4.7816

Max Act: 3.9892. Min Act: -0.0000

Data Index: 157025 (C4 (Web Text))

Max Activating Token Index: 268

Text #18

Max Range: 4.7816. Min Range: -4.7816

Max Act: 3.5926. Min Act: -0.0000

Data Index: 1106116 (C4 (Web Text))

Max Activating Token Index: 358

Text #19

Max Range: 4.7816. Min Range: -4.7816

Max Act: 3.5860. Min Act: -0.0000

Data Index: 131046 (C4 (Web Text))

Max Activating Token Index: 929