Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2710 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.0487. Min Range: -0.0487

Max Act: 0.0467. Min Act: -0.0002

Data Index: 565499 (C4 (Web Text))

Max Activating Token Index: 790

Text #1

Max Range: 0.0487. Min Range: -0.0487

Max Act: 0.0487. Min Act: -0.0002

Data Index: 958359 (C4 (Web Text))

Max Activating Token Index: 1018

Text #2

Max Range: 0.0487. Min Range: -0.0487

Max Act: 0.0465. Min Act: -0.0002

Data Index: 702941 (C4 (Web Text))

Max Activating Token Index: 460

Text #3

Max Range: 0.0487. Min Range: -0.0487

Max Act: 0.0462. Min Act: -0.0002

Data Index: 704948 (C4 (Web Text))

Max Activating Token Index: 621

Text #4

Max Range: 0.0487. Min Range: -0.0487

Max Act: 0.0438. Min Act: -0.0002

Data Index: 147295 (C4 (Web Text))

Max Activating Token Index: 861

Text #5

Max Range: 0.0487. Min Range: -0.0487

Max Act: 0.0447. Min Act: -0.0002

Data Index: 401439 (C4 (Web Text))

Max Activating Token Index: 377

Text #6

Max Range: 0.0487. Min Range: -0.0487

Max Act: 0.0444. Min Act: -0.0002

Data Index: 16026 (C4 (Web Text))

Max Activating Token Index: 499

Text #7

Max Range: 0.0487. Min Range: -0.0487

Max Act: 0.0414. Min Act: -0.0002

Data Index: 487654 (C4 (Web Text))

Max Activating Token Index: 604

Text #8

Max Range: 0.0487. Min Range: -0.0487

Max Act: 0.0419. Min Act: -0.0002

Data Index: 86777 (C4 (Web Text))

Max Activating Token Index: 905

Text #9

Max Range: 0.0487. Min Range: -0.0487

Max Act: 0.0430. Min Act: -0.0002

Data Index: 72096 (C4 (Web Text))

Max Activating Token Index: 813

Text #10

Max Range: 0.0487. Min Range: -0.0487

Max Act: 0.0426. Min Act: -0.0002

Data Index: 537950 (C4 (Web Text))

Max Activating Token Index: 430

Text #11

Max Range: 0.0487. Min Range: -0.0487

Max Act: 0.0417. Min Act: -0.0002

Data Index: 1214949 (C4 (Web Text))

Max Activating Token Index: 178

Text #12

Max Range: 0.0487. Min Range: -0.0487

Max Act: 0.0427. Min Act: -0.0002

Data Index: 517636 (C4 (Web Text))

Max Activating Token Index: 15

Text #13

Max Range: 0.0487. Min Range: -0.0487

Max Act: 0.0429. Min Act: -0.0002

Data Index: 1116708 (C4 (Web Text))

Max Activating Token Index: 341

Text #14

Max Range: 0.0487. Min Range: -0.0487

Max Act: 0.0389. Min Act: -0.0002

Data Index: 373952 (C4 (Web Text))

Max Activating Token Index: 190

Text #15

Max Range: 0.0487. Min Range: -0.0487

Max Act: 0.0417. Min Act: -0.0002

Data Index: 918563 (C4 (Web Text))

Max Activating Token Index: 750

Text #16

Max Range: 0.0487. Min Range: -0.0487

Max Act: 0.0364. Min Act: -0.0002

Data Index: 215409 (C4 (Web Text))

Max Activating Token Index: 763

Text #17

Max Range: 0.0487. Min Range: -0.0487

Max Act: 0.0403. Min Act: -0.0002

Data Index: 927762 (C4 (Web Text))

Max Activating Token Index: 920

Text #18

Max Range: 0.0487. Min Range: -0.0487

Max Act: 0.0347. Min Act: -0.0002

Data Index: 1267931 (C4 (Web Text))

Max Activating Token Index: 541

Text #19

Max Range: 0.0487. Min Range: -0.0487

Max Act: 0.0329. Min Act: -0.0002

Data Index: 6117 (C4 (Web Text))

Max Activating Token Index: 816