Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4748 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.4708. Min Range: -0.4708

Max Act: 0.4708. Min Act: -0.0000

Data Index: 273073 (C4 (Web Text))

Max Activating Token Index: 919

Text #1

Max Range: 0.4708. Min Range: -0.4708

Max Act: 0.4319. Min Act: -0.0000

Data Index: 176772 (C4 (Web Text))

Max Activating Token Index: 897

Text #2

Max Range: 0.4708. Min Range: -0.4708

Max Act: 0.4221. Min Act: -0.0000

Data Index: 1044129 (C4 (Web Text))

Max Activating Token Index: 968

Text #3

Max Range: 0.4708. Min Range: -0.4708

Max Act: 0.4202. Min Act: -0.0000

Data Index: 541227 (C4 (Web Text))

Max Activating Token Index: 361

Text #4

Max Range: 0.4708. Min Range: -0.4708

Max Act: 0.4119. Min Act: -0.0000

Data Index: 918947 (C4 (Web Text))

Max Activating Token Index: 477

Text #5

Max Range: 0.4708. Min Range: -0.4708

Max Act: 0.4111. Min Act: -0.0000

Data Index: 1192883 (C4 (Web Text))

Max Activating Token Index: 611

Text #6

Max Range: 0.4708. Min Range: -0.4708

Max Act: 0.4262. Min Act: -0.0000

Data Index: 1130822 (C4 (Web Text))

Max Activating Token Index: 190

Text #7

Max Range: 0.4708. Min Range: -0.4708

Max Act: 0.3936. Min Act: -0.0000

Data Index: 1210862 (C4 (Web Text))

Max Activating Token Index: 888

Text #8

Max Range: 0.4708. Min Range: -0.4708

Max Act: 0.4171. Min Act: -0.0000

Data Index: 1123353 (C4 (Web Text))

Max Activating Token Index: 124

Text #9

Max Range: 0.4708. Min Range: -0.4708

Max Act: 0.4199. Min Act: -0.0000

Data Index: 1174801 (C4 (Web Text))

Max Activating Token Index: 545

Text #10

Max Range: 0.4708. Min Range: -0.4708

Max Act: 0.3790. Min Act: -0.0000

Data Index: 1004489 (C4 (Web Text))

Max Activating Token Index: 964

Text #11

Max Range: 0.4708. Min Range: -0.4708

Max Act: 0.3829. Min Act: -0.0000

Data Index: 935721 (C4 (Web Text))

Max Activating Token Index: 439

Text #12

Max Range: 0.4708. Min Range: -0.4708

Max Act: 0.3669. Min Act: -0.0000

Data Index: 762582 (C4 (Web Text))

Max Activating Token Index: 426

Text #13

Max Range: 0.4708. Min Range: -0.4708

Max Act: 0.3674. Min Act: -0.0000

Data Index: 99617 (C4 (Web Text))

Max Activating Token Index: 836

Text #14

Max Range: 0.4708. Min Range: -0.4708

Max Act: 0.3866. Min Act: -0.0000

Data Index: 64784 (C4 (Web Text))

Max Activating Token Index: 255

Text #15

Max Range: 0.4708. Min Range: -0.4708

Max Act: 0.3585. Min Act: -0.0000

Data Index: 1198942 (C4 (Web Text))

Max Activating Token Index: 919

Text #16

Max Range: 0.4708. Min Range: -0.4708

Max Act: 0.3594. Min Act: -0.0000

Data Index: 1216299 (C4 (Web Text))

Max Activating Token Index: 962

Text #17

Max Range: 0.4708. Min Range: -0.4708

Max Act: 0.3687. Min Act: -0.0000

Data Index: 967892 (C4 (Web Text))

Max Activating Token Index: 244

Text #18

Max Range: 0.4708. Min Range: -0.4708

Max Act: 0.3426. Min Act: -0.0000

Data Index: 377610 (C4 (Web Text))

Max Activating Token Index: 282

Text #19

Max Range: 0.4708. Min Range: -0.4708

Max Act: 0.3544. Min Act: -0.0000

Data Index: 39654 (C4 (Web Text))

Max Activating Token Index: 315