Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1112 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.9134. Min Range: -0.9134

Max Act: 0.9134. Min Act: -0.0001

Data Index: 1327944 (C4 (Web Text))

Max Activating Token Index: 304

Text #1

Max Range: 0.9134. Min Range: -0.9134

Max Act: 0.8367. Min Act: -0.0001

Data Index: 728125 (C4 (Web Text))

Max Activating Token Index: 346

Text #2

Max Range: 0.9134. Min Range: -0.9134

Max Act: 0.8594. Min Act: -0.0001

Data Index: 596929 (C4 (Web Text))

Max Activating Token Index: 191

Text #3

Max Range: 0.9134. Min Range: -0.9134

Max Act: 0.8118. Min Act: -0.0001

Data Index: 1166068 (C4 (Web Text))

Max Activating Token Index: 737

Text #4

Max Range: 0.9134. Min Range: -0.9134

Max Act: 0.7568. Min Act: -0.0001

Data Index: 147751 (C4 (Web Text))

Max Activating Token Index: 641

Text #5

Max Range: 0.9134. Min Range: -0.9134

Max Act: 0.7987. Min Act: -0.0001

Data Index: 566643 (C4 (Web Text))

Max Activating Token Index: 238

Text #6

Max Range: 0.9134. Min Range: -0.9134

Max Act: 0.7729. Min Act: -0.0001

Data Index: 1091751 (C4 (Web Text))

Max Activating Token Index: 894

Text #7

Max Range: 0.9134. Min Range: -0.9134

Max Act: 0.7466. Min Act: -0.0001

Data Index: 135824 (C4 (Web Text))

Max Activating Token Index: 312

Text #8

Max Range: 0.9134. Min Range: -0.9134

Max Act: 0.7647. Min Act: -0.0001

Data Index: 1264891 (C4 (Web Text))

Max Activating Token Index: 401

Text #9

Max Range: 0.9134. Min Range: -0.9134

Max Act: 0.7525. Min Act: -0.0001

Data Index: 108008 (C4 (Web Text))

Max Activating Token Index: 434

Text #10

Max Range: 0.9134. Min Range: -0.9134

Max Act: 0.6907. Min Act: -0.0001

Data Index: 653640 (C4 (Web Text))

Max Activating Token Index: 105

Text #11

Max Range: 0.9134. Min Range: -0.9134

Max Act: 0.7508. Min Act: -0.0001

Data Index: 384988 (C4 (Web Text))

Max Activating Token Index: 470

Text #12

Max Range: 0.9134. Min Range: -0.9134

Max Act: 0.7453. Min Act: -0.0001

Data Index: 1302041 (C4 (Web Text))

Max Activating Token Index: 58

Text #13

Max Range: 0.9134. Min Range: -0.9134

Max Act: 0.7431. Min Act: -0.0001

Data Index: 25932 (C4 (Web Text))

Max Activating Token Index: 252

Text #14

Max Range: 0.9134. Min Range: -0.9134

Max Act: 0.7629. Min Act: -0.0001

Data Index: 128724 (C4 (Web Text))

Max Activating Token Index: 823

Text #15

Max Range: 0.9134. Min Range: -0.9134

Max Act: 0.7326. Min Act: -0.0001

Data Index: 921292 (C4 (Web Text))

Max Activating Token Index: 359

Text #16

Max Range: 0.9134. Min Range: -0.9134

Max Act: 0.6804. Min Act: -0.0001

Data Index: 1242779 (C4 (Web Text))

Max Activating Token Index: 396

Text #17

Max Range: 0.9134. Min Range: -0.9134

Max Act: 0.6997. Min Act: -0.0001

Data Index: 1345511 (C4 (Web Text))

Max Activating Token Index: 496

Text #18

Max Range: 0.9134. Min Range: -0.9134

Max Act: 0.7506. Min Act: -0.0001

Data Index: 426033 (C4 (Web Text))

Max Activating Token Index: 499

Text #19

Max Range: 0.9134. Min Range: -0.9134

Max Act: 0.7181. Min Act: -0.0001

Data Index: 808509 (C4 (Web Text))

Max Activating Token Index: 584