Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 444 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.6990. Min Range: -0.6990

Max Act: 0.6990. Min Act: -0.0000

Data Index: 984201 (C4 (Web Text))

Max Activating Token Index: 814

Text #1

Max Range: 0.6990. Min Range: -0.6990

Max Act: 0.6788. Min Act: -0.0000

Data Index: 41468 (C4 (Web Text))

Max Activating Token Index: 355

Text #2

Max Range: 0.6990. Min Range: -0.6990

Max Act: 0.6973. Min Act: 0.0001

Data Index: 719891 (C4 (Web Text))

Max Activating Token Index: 408

Text #3

Max Range: 0.6990. Min Range: -0.6990

Max Act: 0.6616. Min Act: -0.0000

Data Index: 40163 (C4 (Web Text))

Max Activating Token Index: 327

Text #4

Max Range: 0.6990. Min Range: -0.6990

Max Act: 0.5776. Min Act: -0.0000

Data Index: 29688 (C4 (Web Text))

Max Activating Token Index: 493

Text #5

Max Range: 0.6990. Min Range: -0.6990

Max Act: 0.5898. Min Act: -0.0000

Data Index: 786051 (C4 (Web Text))

Max Activating Token Index: 482

Text #6

Max Range: 0.6990. Min Range: -0.6990

Max Act: 0.5707. Min Act: -0.0000

Data Index: 447834 (C4 (Web Text))

Max Activating Token Index: 101

Text #7

Max Range: 0.6990. Min Range: -0.6990

Max Act: 0.5506. Min Act: -0.0000

Data Index: 1168735 (C4 (Web Text))

Max Activating Token Index: 975

Text #8

Max Range: 0.6990. Min Range: -0.6990

Max Act: 0.5848. Min Act: -0.0000

Data Index: 1109162 (C4 (Web Text))

Max Activating Token Index: 428

Text #9

Max Range: 0.6990. Min Range: -0.6990

Max Act: 0.5631. Min Act: 0.0000

Data Index: 1353382 (C4 (Web Text))

Max Activating Token Index: 987

Text #10

Max Range: 0.6990. Min Range: -0.6990

Max Act: 0.5728. Min Act: -0.0000

Data Index: 467148 (C4 (Web Text))

Max Activating Token Index: 768

Text #11

Max Range: 0.6990. Min Range: -0.6990

Max Act: 0.5884. Min Act: -0.0000

Data Index: 194253 (C4 (Web Text))

Max Activating Token Index: 219

Text #12

Max Range: 0.6990. Min Range: -0.6990

Max Act: 0.5721. Min Act: -0.0000

Data Index: 134183 (C4 (Web Text))

Max Activating Token Index: 703

Text #13

Max Range: 0.6990. Min Range: -0.6990

Max Act: 0.5367. Min Act: -0.0000

Data Index: 291274 (C4 (Web Text))

Max Activating Token Index: 748

Text #14

Max Range: 0.6990. Min Range: -0.6990

Max Act: 0.5617. Min Act: -0.0000

Data Index: 1016483 (C4 (Web Text))

Max Activating Token Index: 85

Text #15

Max Range: 0.6990. Min Range: -0.6990

Max Act: 0.5806. Min Act: -0.0000

Data Index: 536632 (C4 (Web Text))

Max Activating Token Index: 877

Text #16

Max Range: 0.6990. Min Range: -0.6990

Max Act: 0.5379. Min Act: -0.0000

Data Index: 1263772 (C4 (Web Text))

Max Activating Token Index: 873

Text #17

Max Range: 0.6990. Min Range: -0.6990

Max Act: 0.5575. Min Act: -0.0000

Data Index: 500673 (C4 (Web Text))

Max Activating Token Index: 673

Text #18

Max Range: 0.6990. Min Range: -0.6990

Max Act: 0.5274. Min Act: -0.0000

Data Index: 860424 (C4 (Web Text))

Max Activating Token Index: 233

Text #19

Max Range: 0.6990. Min Range: -0.6990

Max Act: 0.5466. Min Act: -0.0000

Data Index: 1258651 (C4 (Web Text))

Max Activating Token Index: 181