Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3474 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.4211. Min Range: -1.4211

Max Act: 1.4211. Min Act: -0.0001

Data Index: 728257 (C4 (Web Text))

Max Activating Token Index: 273

Text #1

Max Range: 1.4211. Min Range: -1.4211

Max Act: 1.2131. Min Act: -0.0001

Data Index: 1207265 (C4 (Web Text))

Max Activating Token Index: 827

Text #2

Max Range: 1.4211. Min Range: -1.4211

Max Act: 1.2041. Min Act: -0.0001

Data Index: 1126540 (C4 (Web Text))

Max Activating Token Index: 839

Text #3

Max Range: 1.4211. Min Range: -1.4211

Max Act: 1.2107. Min Act: -0.0001

Data Index: 631402 (C4 (Web Text))

Max Activating Token Index: 342

Text #4

Max Range: 1.4211. Min Range: -1.4211

Max Act: 1.1368. Min Act: -0.0001

Data Index: 1631536 (Python Code)

Max Activating Token Index: 832

Text #5

Max Range: 1.4211. Min Range: -1.4211

Max Act: 1.1578. Min Act: -0.0001

Data Index: 1655650 (Python Code)

Max Activating Token Index: 349

Text #6

Max Range: 1.4211. Min Range: -1.4211

Max Act: 0.9665. Min Act: -0.0001

Data Index: 1145716 (C4 (Web Text))

Max Activating Token Index: 752

Text #7

Max Range: 1.4211. Min Range: -1.4211

Max Act: 1.2227. Min Act: -0.0001

Data Index: 303463 (C4 (Web Text))

Max Activating Token Index: 474

Text #8

Max Range: 1.4211. Min Range: -1.4211

Max Act: 1.0779. Min Act: -0.0001

Data Index: 1621191 (Python Code)

Max Activating Token Index: 288

Text #9

Max Range: 1.4211. Min Range: -1.4211

Max Act: 1.0294. Min Act: -0.0001

Data Index: 1176184 (C4 (Web Text))

Max Activating Token Index: 343

Text #10

Max Range: 1.4211. Min Range: -1.4211

Max Act: 1.0964. Min Act: -0.0001

Data Index: 654385 (C4 (Web Text))

Max Activating Token Index: 162

Text #11

Max Range: 1.4211. Min Range: -1.4211

Max Act: 1.0064. Min Act: -0.0001

Data Index: 215172 (C4 (Web Text))

Max Activating Token Index: 182

Text #12

Max Range: 1.4211. Min Range: -1.4211

Max Act: 0.9205. Min Act: -0.0001

Data Index: 832212 (C4 (Web Text))

Max Activating Token Index: 254

Text #13

Max Range: 1.4211. Min Range: -1.4211

Max Act: 1.0025. Min Act: -0.0001

Data Index: 1069773 (C4 (Web Text))

Max Activating Token Index: 804

Text #14

Max Range: 1.4211. Min Range: -1.4211

Max Act: 0.9571. Min Act: -0.0001

Data Index: 1261481 (C4 (Web Text))

Max Activating Token Index: 480

Text #15

Max Range: 1.4211. Min Range: -1.4211

Max Act: 0.9580. Min Act: -0.0001

Data Index: 228783 (C4 (Web Text))

Max Activating Token Index: 898

Text #16

Max Range: 1.4211. Min Range: -1.4211

Max Act: 0.9832. Min Act: -0.0001

Data Index: 6645 (C4 (Web Text))

Max Activating Token Index: 988

Text #17

Max Range: 1.4211. Min Range: -1.4211

Max Act: 1.0337. Min Act: -0.0001

Data Index: 160397 (C4 (Web Text))

Max Activating Token Index: 679

Text #18

Max Range: 1.4211. Min Range: -1.4211

Max Act: 1.0133. Min Act: -0.0001

Data Index: 1253106 (C4 (Web Text))

Max Activating Token Index: 624

Text #19

Max Range: 1.4211. Min Range: -1.4211

Max Act: 0.8089. Min Act: -0.0001

Data Index: 1167366 (C4 (Web Text))

Max Activating Token Index: 607