Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2840 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.7910. Min Range: -0.7910

Max Act: 0.7910. Min Act: -0.0000

Data Index: 97492 (C4 (Web Text))

Max Activating Token Index: 63

Text #1

Max Range: 0.7910. Min Range: -0.7910

Max Act: 0.7277. Min Act: -0.0000

Data Index: 569371 (C4 (Web Text))

Max Activating Token Index: 482

Text #2

Max Range: 0.7910. Min Range: -0.7910

Max Act: 0.7470. Min Act: -0.0000

Data Index: 525619 (C4 (Web Text))

Max Activating Token Index: 74

Text #3

Max Range: 0.7910. Min Range: -0.7910

Max Act: 0.6633. Min Act: -0.0000

Data Index: 1345576 (C4 (Web Text))

Max Activating Token Index: 494

Text #4

Max Range: 0.7910. Min Range: -0.7910

Max Act: 0.6562. Min Act: -0.0000

Data Index: 344395 (C4 (Web Text))

Max Activating Token Index: 127

Text #5

Max Range: 0.7910. Min Range: -0.7910

Max Act: 0.6737. Min Act: -0.0000

Data Index: 906681 (C4 (Web Text))

Max Activating Token Index: 745

Text #6

Max Range: 0.7910. Min Range: -0.7910

Max Act: 0.6407. Min Act: -0.0000

Data Index: 869567 (C4 (Web Text))

Max Activating Token Index: 469

Text #7

Max Range: 0.7910. Min Range: -0.7910

Max Act: 0.5578. Min Act: -0.0000

Data Index: 254617 (C4 (Web Text))

Max Activating Token Index: 105

Text #8

Max Range: 0.7910. Min Range: -0.7910

Max Act: 0.5888. Min Act: -0.0000

Data Index: 960074 (C4 (Web Text))

Max Activating Token Index: 997

Text #9

Max Range: 0.7910. Min Range: -0.7910

Max Act: 0.5898. Min Act: -0.0000

Data Index: 530155 (C4 (Web Text))

Max Activating Token Index: 761

Text #10

Max Range: 0.7910. Min Range: -0.7910

Max Act: 0.5820. Min Act: -0.0000

Data Index: 965187 (C4 (Web Text))

Max Activating Token Index: 318

Text #11

Max Range: 0.7910. Min Range: -0.7910

Max Act: 0.5845. Min Act: -0.0000

Data Index: 767431 (C4 (Web Text))

Max Activating Token Index: 466

Text #12

Max Range: 0.7910. Min Range: -0.7910

Max Act: 0.6022. Min Act: -0.0000

Data Index: 1144955 (C4 (Web Text))

Max Activating Token Index: 104

Text #13

Max Range: 0.7910. Min Range: -0.7910

Max Act: 0.5435. Min Act: -0.0000

Data Index: 1295122 (C4 (Web Text))

Max Activating Token Index: 37

Text #14

Max Range: 0.7910. Min Range: -0.7910

Max Act: 0.4943. Min Act: -0.0000

Data Index: 862393 (C4 (Web Text))

Max Activating Token Index: 127

Text #15

Max Range: 0.7910. Min Range: -0.7910

Max Act: 0.5522. Min Act: -0.0000

Data Index: 732266 (C4 (Web Text))

Max Activating Token Index: 41

Text #16

Max Range: 0.7910. Min Range: -0.7910

Max Act: 0.4854. Min Act: -0.0000

Data Index: 40383 (C4 (Web Text))

Max Activating Token Index: 780

Text #17

Max Range: 0.7910. Min Range: -0.7910

Max Act: 0.5363. Min Act: -0.0000

Data Index: 1079075 (C4 (Web Text))

Max Activating Token Index: 320

Text #18

Max Range: 0.7910. Min Range: -0.7910

Max Act: 0.6169. Min Act: -0.0000

Data Index: 416299 (C4 (Web Text))

Max Activating Token Index: 142

Text #19

Max Range: 0.7910. Min Range: -0.7910

Max Act: 0.5222. Min Act: -0.0000

Data Index: 445789 (C4 (Web Text))

Max Activating Token Index: 491