Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4379 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.4884. Min Range: -2.4884

Max Act: 2.4884. Min Act: -0.0000

Data Index: 35048 (C4 (Web Text))

Max Activating Token Index: 193

Text #1

Max Range: 2.4884. Min Range: -2.4884

Max Act: 2.2708. Min Act: -0.0000

Data Index: 204924 (C4 (Web Text))

Max Activating Token Index: 896

Text #2

Max Range: 2.4884. Min Range: -2.4884

Max Act: 2.3058. Min Act: -0.0000

Data Index: 500600 (C4 (Web Text))

Max Activating Token Index: 840

Text #3

Max Range: 2.4884. Min Range: -2.4884

Max Act: 1.8847. Min Act: -0.0000

Data Index: 265678 (C4 (Web Text))

Max Activating Token Index: 374

Text #4

Max Range: 2.4884. Min Range: -2.4884

Max Act: 2.0847. Min Act: -0.0000

Data Index: 986510 (C4 (Web Text))

Max Activating Token Index: 201

Text #5

Max Range: 2.4884. Min Range: -2.4884

Max Act: 1.9013. Min Act: -0.0000

Data Index: 555489 (C4 (Web Text))

Max Activating Token Index: 659

Text #6

Max Range: 2.4884. Min Range: -2.4884

Max Act: 1.7145. Min Act: -0.0000

Data Index: 1251974 (C4 (Web Text))

Max Activating Token Index: 922

Text #7

Max Range: 2.4884. Min Range: -2.4884

Max Act: 1.7370. Min Act: -0.0000

Data Index: 810584 (C4 (Web Text))

Max Activating Token Index: 622

Text #8

Max Range: 2.4884. Min Range: -2.4884

Max Act: 1.6780. Min Act: -0.0000

Data Index: 980406 (C4 (Web Text))

Max Activating Token Index: 364

Text #9

Max Range: 2.4884. Min Range: -2.4884

Max Act: 1.6572. Min Act: -0.0000

Data Index: 434443 (C4 (Web Text))

Max Activating Token Index: 512

Text #10

Max Range: 2.4884. Min Range: -2.4884

Max Act: 1.7149. Min Act: -0.0000

Data Index: 711358 (C4 (Web Text))

Max Activating Token Index: 839

Text #11

Max Range: 2.4884. Min Range: -2.4884

Max Act: 1.6362. Min Act: -0.0000

Data Index: 711748 (C4 (Web Text))

Max Activating Token Index: 417

Text #12

Max Range: 2.4884. Min Range: -2.4884

Max Act: 1.7257. Min Act: -0.0000

Data Index: 652940 (C4 (Web Text))

Max Activating Token Index: 456

Text #13

Max Range: 2.4884. Min Range: -2.4884

Max Act: 1.7079. Min Act: -0.0000

Data Index: 949914 (C4 (Web Text))

Max Activating Token Index: 314

Text #14

Max Range: 2.4884. Min Range: -2.4884

Max Act: 1.7568. Min Act: -0.0000

Data Index: 66038 (C4 (Web Text))

Max Activating Token Index: 571

Text #15

Max Range: 2.4884. Min Range: -2.4884

Max Act: 1.7510. Min Act: -0.0000

Data Index: 914789 (C4 (Web Text))

Max Activating Token Index: 72

Text #16

Max Range: 2.4884. Min Range: -2.4884

Max Act: 1.7356. Min Act: -0.0000

Data Index: 280818 (C4 (Web Text))

Max Activating Token Index: 632

Text #17

Max Range: 2.4884. Min Range: -2.4884

Max Act: 1.6982. Min Act: -0.0000

Data Index: 1150568 (C4 (Web Text))

Max Activating Token Index: 363

Text #18

Max Range: 2.4884. Min Range: -2.4884

Max Act: 1.7212. Min Act: -0.0000

Data Index: 1259552 (C4 (Web Text))

Max Activating Token Index: 472

Text #19

Max Range: 2.4884. Min Range: -2.4884

Max Act: 1.6298. Min Act: -0.0000

Data Index: 1319428 (C4 (Web Text))

Max Activating Token Index: 935