Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2761 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.8680. Min Range: -1.8680

Max Act: 1.8680. Min Act: -0.0000

Data Index: 1230486 (C4 (Web Text))

Max Activating Token Index: 518

Text #1

Max Range: 1.8680. Min Range: -1.8680

Max Act: 1.7511. Min Act: -0.0000

Data Index: 1244862 (C4 (Web Text))

Max Activating Token Index: 47

Text #2

Max Range: 1.8680. Min Range: -1.8680

Max Act: 1.7031. Min Act: -0.0000

Data Index: 286569 (C4 (Web Text))

Max Activating Token Index: 214

Text #3

Max Range: 1.8680. Min Range: -1.8680

Max Act: 1.7647. Min Act: -0.0000

Data Index: 4427 (C4 (Web Text))

Max Activating Token Index: 816

Text #4

Max Range: 1.8680. Min Range: -1.8680

Max Act: 1.7276. Min Act: -0.0000

Data Index: 732343 (C4 (Web Text))

Max Activating Token Index: 854

Text #5

Max Range: 1.8680. Min Range: -1.8680

Max Act: 1.6485. Min Act: -0.0000

Data Index: 1234607 (C4 (Web Text))

Max Activating Token Index: 204

Text #6

Max Range: 1.8680. Min Range: -1.8680

Max Act: 1.7433. Min Act: -0.0000

Data Index: 942620 (C4 (Web Text))

Max Activating Token Index: 955

Text #7

Max Range: 1.8680. Min Range: -1.8680

Max Act: 1.6279. Min Act: -0.0000

Data Index: 228389 (C4 (Web Text))

Max Activating Token Index: 950

Text #8

Max Range: 1.8680. Min Range: -1.8680

Max Act: 1.6802. Min Act: -0.0000

Data Index: 1154207 (C4 (Web Text))

Max Activating Token Index: 574

Text #9

Max Range: 1.8680. Min Range: -1.8680

Max Act: 1.4830. Min Act: -0.0000

Data Index: 241556 (C4 (Web Text))

Max Activating Token Index: 778

Text #10

Max Range: 1.8680. Min Range: -1.8680

Max Act: 1.5637. Min Act: -0.0000

Data Index: 335912 (C4 (Web Text))

Max Activating Token Index: 464

Text #11

Max Range: 1.8680. Min Range: -1.8680

Max Act: 1.5625. Min Act: -0.0000

Data Index: 1070782 (C4 (Web Text))

Max Activating Token Index: 246

Text #12

Max Range: 1.8680. Min Range: -1.8680

Max Act: 1.6224. Min Act: -0.0000

Data Index: 770285 (C4 (Web Text))

Max Activating Token Index: 999

Text #13

Max Range: 1.8680. Min Range: -1.8680

Max Act: 1.5141. Min Act: -0.0000

Data Index: 1192177 (C4 (Web Text))

Max Activating Token Index: 974

Text #14

Max Range: 1.8680. Min Range: -1.8680

Max Act: 1.5007. Min Act: -0.0000

Data Index: 1223075 (C4 (Web Text))

Max Activating Token Index: 53

Text #15

Max Range: 1.8680. Min Range: -1.8680

Max Act: 1.4802. Min Act: -0.0000

Data Index: 1166705 (C4 (Web Text))

Max Activating Token Index: 212

Text #16

Max Range: 1.8680. Min Range: -1.8680

Max Act: 1.4451. Min Act: -0.0000

Data Index: 524494 (C4 (Web Text))

Max Activating Token Index: 883

Text #17

Max Range: 1.8680. Min Range: -1.8680

Max Act: 1.4469. Min Act: -0.0000

Data Index: 211644 (C4 (Web Text))

Max Activating Token Index: 254

Text #18

Max Range: 1.8680. Min Range: -1.8680

Max Act: 1.5120. Min Act: -0.0000

Data Index: 1117806 (C4 (Web Text))

Max Activating Token Index: 772

Text #19

Max Range: 1.8680. Min Range: -1.8680

Max Act: 1.4530. Min Act: -0.0000

Data Index: 1071470 (C4 (Web Text))

Max Activating Token Index: 951