Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 728 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.5928. Min Range: -1.5928

Max Act: 1.5928. Min Act: -0.0000

Data Index: 475551 (C4 (Web Text))

Max Activating Token Index: 540

Text #1

Max Range: 1.5928. Min Range: -1.5928

Max Act: 1.5529. Min Act: -0.0000

Data Index: 219851 (C4 (Web Text))

Max Activating Token Index: 27

Text #2

Max Range: 1.5928. Min Range: -1.5928

Max Act: 1.5526. Min Act: -0.0000

Data Index: 696800 (C4 (Web Text))

Max Activating Token Index: 846

Text #3

Max Range: 1.5928. Min Range: -1.5928

Max Act: 1.5315. Min Act: -0.0000

Data Index: 243734 (C4 (Web Text))

Max Activating Token Index: 127

Text #4

Max Range: 1.5928. Min Range: -1.5928

Max Act: 1.3489. Min Act: -0.0000

Data Index: 205401 (C4 (Web Text))

Max Activating Token Index: 516

Text #5

Max Range: 1.5928. Min Range: -1.5928

Max Act: 1.4261. Min Act: -0.0000

Data Index: 237731 (C4 (Web Text))

Max Activating Token Index: 908

Text #6

Max Range: 1.5928. Min Range: -1.5928

Max Act: 1.3833. Min Act: -0.0000

Data Index: 1097956 (C4 (Web Text))

Max Activating Token Index: 411

Text #7

Max Range: 1.5928. Min Range: -1.5928

Max Act: 1.4526. Min Act: -0.0000

Data Index: 411182 (C4 (Web Text))

Max Activating Token Index: 701

Text #8

Max Range: 1.5928. Min Range: -1.5928

Max Act: 1.3458. Min Act: -0.0000

Data Index: 1275069 (C4 (Web Text))

Max Activating Token Index: 384

Text #9

Max Range: 1.5928. Min Range: -1.5928

Max Act: 1.3570. Min Act: -0.0000

Data Index: 310912 (C4 (Web Text))

Max Activating Token Index: 1006

Text #10

Max Range: 1.5928. Min Range: -1.5928

Max Act: 1.3466. Min Act: -0.0000

Data Index: 132305 (C4 (Web Text))

Max Activating Token Index: 692

Text #11

Max Range: 1.5928. Min Range: -1.5928

Max Act: 1.4526. Min Act: -0.0000

Data Index: 1078590 (C4 (Web Text))

Max Activating Token Index: 341

Text #12

Max Range: 1.5928. Min Range: -1.5928

Max Act: 1.3656. Min Act: -0.0000

Data Index: 402572 (C4 (Web Text))

Max Activating Token Index: 140

Text #13

Max Range: 1.5928. Min Range: -1.5928

Max Act: 1.2850. Min Act: -0.0000

Data Index: 889793 (C4 (Web Text))

Max Activating Token Index: 557

Text #14

Max Range: 1.5928. Min Range: -1.5928

Max Act: 1.1981. Min Act: -0.0000

Data Index: 750020 (C4 (Web Text))

Max Activating Token Index: 167

Text #15

Max Range: 1.5928. Min Range: -1.5928

Max Act: 1.2448. Min Act: -0.0000

Data Index: 799988 (C4 (Web Text))

Max Activating Token Index: 672

Text #16

Max Range: 1.5928. Min Range: -1.5928

Max Act: 1.2283. Min Act: -0.0000

Data Index: 73533 (C4 (Web Text))

Max Activating Token Index: 315

Text #17

Max Range: 1.5928. Min Range: -1.5928

Max Act: 1.2930. Min Act: -0.0000

Data Index: 358636 (C4 (Web Text))

Max Activating Token Index: 402

Text #18

Max Range: 1.5928. Min Range: -1.5928

Max Act: 1.2906. Min Act: -0.0000

Data Index: 1328904 (C4 (Web Text))

Max Activating Token Index: 19

Text #19

Max Range: 1.5928. Min Range: -1.5928

Max Act: 1.2954. Min Act: -0.0000

Data Index: 473242 (C4 (Web Text))

Max Activating Token Index: 609