Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2126 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.2684. Min Range: -1.2684

Max Act: 1.1359. Min Act: -0.0000

Data Index: 656824 (C4 (Web Text))

Max Activating Token Index: 802

Text #1

Max Range: 1.2684. Min Range: -1.2684

Max Act: 1.2684. Min Act: -0.0000

Data Index: 666889 (C4 (Web Text))

Max Activating Token Index: 734

Text #2

Max Range: 1.2684. Min Range: -1.2684

Max Act: 1.1194. Min Act: -0.0000

Data Index: 14265 (C4 (Web Text))

Max Activating Token Index: 52

Text #3

Max Range: 1.2684. Min Range: -1.2684

Max Act: 1.0361. Min Act: -0.0000

Data Index: 1264177 (C4 (Web Text))

Max Activating Token Index: 102

Text #4

Max Range: 1.2684. Min Range: -1.2684

Max Act: 1.1607. Min Act: -0.0000

Data Index: 248657 (C4 (Web Text))

Max Activating Token Index: 750

Text #5

Max Range: 1.2684. Min Range: -1.2684

Max Act: 1.0791. Min Act: -0.0000

Data Index: 922329 (C4 (Web Text))

Max Activating Token Index: 319

Text #6

Max Range: 1.2684. Min Range: -1.2684

Max Act: 1.0758. Min Act: -0.0000

Data Index: 1358376 (C4 (Web Text))

Max Activating Token Index: 27

Text #7

Max Range: 1.2684. Min Range: -1.2684

Max Act: 1.0633. Min Act: -0.0000

Data Index: 742719 (C4 (Web Text))

Max Activating Token Index: 138

Text #8

Max Range: 1.2684. Min Range: -1.2684

Max Act: 0.9520. Min Act: -0.0000

Data Index: 135084 (C4 (Web Text))

Max Activating Token Index: 68

Text #9

Max Range: 1.2684. Min Range: -1.2684

Max Act: 0.8826. Min Act: -0.0000

Data Index: 1180832 (C4 (Web Text))

Max Activating Token Index: 166

Text #10

Max Range: 1.2684. Min Range: -1.2684

Max Act: 0.9862. Min Act: -0.0000

Data Index: 779164 (C4 (Web Text))

Max Activating Token Index: 404

Text #11

Max Range: 1.2684. Min Range: -1.2684

Max Act: 0.9117. Min Act: -0.0000

Data Index: 247454 (C4 (Web Text))

Max Activating Token Index: 759

Text #12

Max Range: 1.2684. Min Range: -1.2684

Max Act: 0.9160. Min Act: -0.0000

Data Index: 17149 (C4 (Web Text))

Max Activating Token Index: 700

Text #13

Max Range: 1.2684. Min Range: -1.2684

Max Act: 0.8373. Min Act: -0.0000

Data Index: 1018222 (C4 (Web Text))

Max Activating Token Index: 201

Text #14

Max Range: 1.2684. Min Range: -1.2684

Max Act: 0.8483. Min Act: -0.0000

Data Index: 484474 (C4 (Web Text))

Max Activating Token Index: 654

Text #15

Max Range: 1.2684. Min Range: -1.2684

Max Act: 0.9337. Min Act: -0.0000

Data Index: 779512 (C4 (Web Text))

Max Activating Token Index: 213

Text #16

Max Range: 1.2684. Min Range: -1.2684

Max Act: 0.8165. Min Act: -0.0000

Data Index: 50831 (C4 (Web Text))

Max Activating Token Index: 936

Text #17

Max Range: 1.2684. Min Range: -1.2684

Max Act: 0.7980. Min Act: -0.0000

Data Index: 836386 (C4 (Web Text))

Max Activating Token Index: 186

Text #18

Max Range: 1.2684. Min Range: -1.2684

Max Act: 0.8559. Min Act: -0.0000

Data Index: 836862 (C4 (Web Text))

Max Activating Token Index: 873

Text #19

Max Range: 1.2684. Min Range: -1.2684

Max Act: 0.9048. Min Act: -0.0000

Data Index: 313620 (C4 (Web Text))

Max Activating Token Index: 646