Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 627 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.5278. Min Range: -1.5278

Max Act: 1.5278. Min Act: -0.0000

Data Index: 773595 (C4 (Web Text))

Max Activating Token Index: 141

Text #1

Max Range: 1.5278. Min Range: -1.5278

Max Act: 1.2155. Min Act: -0.0000

Data Index: 1164101 (C4 (Web Text))

Max Activating Token Index: 118

Text #2

Max Range: 1.5278. Min Range: -1.5278

Max Act: 1.1571. Min Act: -0.0000

Data Index: 115695 (C4 (Web Text))

Max Activating Token Index: 101

Text #3

Max Range: 1.5278. Min Range: -1.5278

Max Act: 1.1446. Min Act: -0.0000

Data Index: 684168 (C4 (Web Text))

Max Activating Token Index: 261

Text #4

Max Range: 1.5278. Min Range: -1.5278

Max Act: 1.1909. Min Act: -0.0000

Data Index: 556721 (C4 (Web Text))

Max Activating Token Index: 162

Text #5

Max Range: 1.5278. Min Range: -1.5278

Max Act: 0.9762. Min Act: -0.0000

Data Index: 526049 (C4 (Web Text))

Max Activating Token Index: 806

Text #6

Max Range: 1.5278. Min Range: -1.5278

Max Act: 1.0002. Min Act: -0.0000

Data Index: 325696 (C4 (Web Text))

Max Activating Token Index: 506

Text #7

Max Range: 1.5278. Min Range: -1.5278

Max Act: 0.9819. Min Act: -0.0000

Data Index: 626558 (C4 (Web Text))

Max Activating Token Index: 978

Text #8

Max Range: 1.5278. Min Range: -1.5278

Max Act: 0.8698. Min Act: -0.0000

Data Index: 279449 (C4 (Web Text))

Max Activating Token Index: 974

Text #9

Max Range: 1.5278. Min Range: -1.5278

Max Act: 0.9174. Min Act: -0.0000

Data Index: 1032473 (C4 (Web Text))

Max Activating Token Index: 260

Text #10

Max Range: 1.5278. Min Range: -1.5278

Max Act: 0.8740. Min Act: -0.0000

Data Index: 1597651 (Python Code)

Max Activating Token Index: 702

Text #11

Max Range: 1.5278. Min Range: -1.5278

Max Act: 0.8612. Min Act: -0.0000

Data Index: 677575 (C4 (Web Text))

Max Activating Token Index: 484

Text #12

Max Range: 1.5278. Min Range: -1.5278

Max Act: 0.8660. Min Act: -0.0000

Data Index: 262064 (C4 (Web Text))

Max Activating Token Index: 1010

Text #13

Max Range: 1.5278. Min Range: -1.5278

Max Act: 0.8878. Min Act: -0.0000

Data Index: 93076 (C4 (Web Text))

Max Activating Token Index: 949

Text #14

Max Range: 1.5278. Min Range: -1.5278

Max Act: 0.8377. Min Act: -0.0000

Data Index: 54843 (C4 (Web Text))

Max Activating Token Index: 364

Text #15

Max Range: 1.5278. Min Range: -1.5278

Max Act: 0.7593. Min Act: -0.0000

Data Index: 381394 (C4 (Web Text))

Max Activating Token Index: 874

Text #16

Max Range: 1.5278. Min Range: -1.5278

Max Act: 0.8020. Min Act: -0.0000

Data Index: 536917 (C4 (Web Text))

Max Activating Token Index: 917

Text #17

Max Range: 1.5278. Min Range: -1.5278

Max Act: 0.8387. Min Act: -0.0000

Data Index: 1588167 (Python Code)

Max Activating Token Index: 117

Text #18

Max Range: 1.5278. Min Range: -1.5278

Max Act: 0.8495. Min Act: -0.0000

Data Index: 422921 (C4 (Web Text))

Max Activating Token Index: 345

Text #19

Max Range: 1.5278. Min Range: -1.5278

Max Act: 0.8264. Min Act: -0.0000

Data Index: 1410109 (Python Code)

Max Activating Token Index: 724