Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1022 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.0425. Min Range: -1.0425

Max Act: 1.0425. Min Act: -0.0000

Data Index: 495170 (C4 (Web Text))

Max Activating Token Index: 305

Text #1

Max Range: 1.0425. Min Range: -1.0425

Max Act: 0.9536. Min Act: -0.0000

Data Index: 797514 (C4 (Web Text))

Max Activating Token Index: 411

Text #2

Max Range: 1.0425. Min Range: -1.0425

Max Act: 0.8738. Min Act: -0.0000

Data Index: 305060 (C4 (Web Text))

Max Activating Token Index: 876

Text #3

Max Range: 1.0425. Min Range: -1.0425

Max Act: 0.8457. Min Act: -0.0000

Data Index: 1290623 (C4 (Web Text))

Max Activating Token Index: 413

Text #4

Max Range: 1.0425. Min Range: -1.0425

Max Act: 0.8157. Min Act: -0.0000

Data Index: 300427 (C4 (Web Text))

Max Activating Token Index: 509

Text #5

Max Range: 1.0425. Min Range: -1.0425

Max Act: 0.8359. Min Act: -0.0000

Data Index: 58720 (C4 (Web Text))

Max Activating Token Index: 5

Text #6

Max Range: 1.0425. Min Range: -1.0425

Max Act: 0.7895. Min Act: -0.0000

Data Index: 1162577 (C4 (Web Text))

Max Activating Token Index: 151

Text #7

Max Range: 1.0425. Min Range: -1.0425

Max Act: 0.7846. Min Act: -0.0000

Data Index: 807394 (C4 (Web Text))

Max Activating Token Index: 997

Text #8

Max Range: 1.0425. Min Range: -1.0425

Max Act: 0.7633. Min Act: -0.0000

Data Index: 812067 (C4 (Web Text))

Max Activating Token Index: 329

Text #9

Max Range: 1.0425. Min Range: -1.0425

Max Act: 0.7226. Min Act: -0.0000

Data Index: 102560 (C4 (Web Text))

Max Activating Token Index: 549

Text #10

Max Range: 1.0425. Min Range: -1.0425

Max Act: 0.7456. Min Act: -0.0000

Data Index: 977061 (C4 (Web Text))

Max Activating Token Index: 387

Text #11

Max Range: 1.0425. Min Range: -1.0425

Max Act: 0.7738. Min Act: -0.0000

Data Index: 955547 (C4 (Web Text))

Max Activating Token Index: 1000

Text #12

Max Range: 1.0425. Min Range: -1.0425

Max Act: 0.7496. Min Act: -0.0000

Data Index: 831386 (C4 (Web Text))

Max Activating Token Index: 470

Text #13

Max Range: 1.0425. Min Range: -1.0425

Max Act: 0.7548. Min Act: -0.0000

Data Index: 793096 (C4 (Web Text))

Max Activating Token Index: 310

Text #14

Max Range: 1.0425. Min Range: -1.0425

Max Act: 0.7754. Min Act: -0.0000

Data Index: 1133750 (C4 (Web Text))

Max Activating Token Index: 319

Text #15

Max Range: 1.0425. Min Range: -1.0425

Max Act: 0.6960. Min Act: -0.0000

Data Index: 1011345 (C4 (Web Text))

Max Activating Token Index: 89

Text #16

Max Range: 1.0425. Min Range: -1.0425

Max Act: 0.7182. Min Act: -0.0000

Data Index: 511067 (C4 (Web Text))

Max Activating Token Index: 778

Text #17

Max Range: 1.0425. Min Range: -1.0425

Max Act: 0.7323. Min Act: -0.0000

Data Index: 946486 (C4 (Web Text))

Max Activating Token Index: 26

Text #18

Max Range: 1.0425. Min Range: -1.0425

Max Act: 0.6837. Min Act: -0.0000

Data Index: 698600 (C4 (Web Text))

Max Activating Token Index: 312

Text #19

Max Range: 1.0425. Min Range: -1.0425

Max Act: 0.6829. Min Act: -0.0000

Data Index: 414693 (C4 (Web Text))

Max Activating Token Index: 354