Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3969 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 2.1339. Min Range: -2.1339

Max Act: 1.9395. Min Act: -0.0000

Data Index: 321580 (C4 (Web Text))

Max Activating Token Index: 952

Text #1

Max Range: 2.1339. Min Range: -2.1339

Max Act: 2.1019. Min Act: -0.0000

Data Index: 1341900 (C4 (Web Text))

Max Activating Token Index: 15

Text #2

Max Range: 2.1339. Min Range: -2.1339

Max Act: 2.1339. Min Act: -0.0001

Data Index: 1290248 (C4 (Web Text))

Max Activating Token Index: 948

Text #3

Max Range: 2.1339. Min Range: -2.1339

Max Act: 2.0193. Min Act: -0.0000

Data Index: 849236 (C4 (Web Text))

Max Activating Token Index: 983

Text #4

Max Range: 2.1339. Min Range: -2.1339

Max Act: 2.1070. Min Act: -0.0000

Data Index: 1298426 (C4 (Web Text))

Max Activating Token Index: 1005

Text #5

Max Range: 2.1339. Min Range: -2.1339

Max Act: 2.0296. Min Act: -0.0001

Data Index: 708095 (C4 (Web Text))

Max Activating Token Index: 855

Text #6

Max Range: 2.1339. Min Range: -2.1339

Max Act: 1.7647. Min Act: -0.0001

Data Index: 1292582 (C4 (Web Text))

Max Activating Token Index: 776

Text #7

Max Range: 2.1339. Min Range: -2.1339

Max Act: 1.8376. Min Act: -0.0001

Data Index: 539781 (C4 (Web Text))

Max Activating Token Index: 749

Text #8

Max Range: 2.1339. Min Range: -2.1339

Max Act: 1.9757. Min Act: -0.0001

Data Index: 485473 (C4 (Web Text))

Max Activating Token Index: 927

Text #9

Max Range: 2.1339. Min Range: -2.1339

Max Act: 1.7829. Min Act: -0.0001

Data Index: 474979 (C4 (Web Text))

Max Activating Token Index: 762

Text #10

Max Range: 2.1339. Min Range: -2.1339

Max Act: 1.8826. Min Act: -0.0001

Data Index: 924368 (C4 (Web Text))

Max Activating Token Index: 784

Text #11

Max Range: 2.1339. Min Range: -2.1339

Max Act: 1.7065. Min Act: -0.0001

Data Index: 289660 (C4 (Web Text))

Max Activating Token Index: 914

Text #12

Max Range: 2.1339. Min Range: -2.1339

Max Act: 1.7515. Min Act: -0.0000

Data Index: 412043 (C4 (Web Text))

Max Activating Token Index: 692

Text #13

Max Range: 2.1339. Min Range: -2.1339

Max Act: 1.6860. Min Act: -0.0000

Data Index: 604035 (C4 (Web Text))

Max Activating Token Index: 660

Text #14

Max Range: 2.1339. Min Range: -2.1339

Max Act: 1.8822. Min Act: -0.0000

Data Index: 1259865 (C4 (Web Text))

Max Activating Token Index: 797

Text #15

Max Range: 2.1339. Min Range: -2.1339

Max Act: 1.8285. Min Act: -0.0001

Data Index: 1208566 (C4 (Web Text))

Max Activating Token Index: 831

Text #16

Max Range: 2.1339. Min Range: -2.1339

Max Act: 1.6653. Min Act: -0.0001

Data Index: 1316278 (C4 (Web Text))

Max Activating Token Index: 937

Text #17

Max Range: 2.1339. Min Range: -2.1339

Max Act: 1.8513. Min Act: -0.0001

Data Index: 764287 (C4 (Web Text))

Max Activating Token Index: 16

Text #18

Max Range: 2.1339. Min Range: -2.1339

Max Act: 1.8726. Min Act: -0.0000

Data Index: 1246825 (C4 (Web Text))

Max Activating Token Index: 855

Text #19

Max Range: 2.1339. Min Range: -2.1339

Max Act: 1.6141. Min Act: -0.0000

Data Index: 670253 (C4 (Web Text))

Max Activating Token Index: 757