Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4750 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.9444. Min Range: -1.9444

Max Act: 1.9444. Min Act: -0.0001

Data Index: 37330 (C4 (Web Text))

Max Activating Token Index: 128

Text #1

Max Range: 1.9444. Min Range: -1.9444

Max Act: 1.7053. Min Act: -0.0001

Data Index: 542004 (C4 (Web Text))

Max Activating Token Index: 312

Text #2

Max Range: 1.9444. Min Range: -1.9444

Max Act: 1.5536. Min Act: -0.0001

Data Index: 638085 (C4 (Web Text))

Max Activating Token Index: 527

Text #3

Max Range: 1.9444. Min Range: -1.9444

Max Act: 1.3920. Min Act: -0.0000

Data Index: 1259382 (C4 (Web Text))

Max Activating Token Index: 32

Text #4

Max Range: 1.9444. Min Range: -1.9444

Max Act: 1.3774. Min Act: -0.0001

Data Index: 164536 (C4 (Web Text))

Max Activating Token Index: 321

Text #5

Max Range: 1.9444. Min Range: -1.9444

Max Act: 1.4673. Min Act: -0.0000

Data Index: 204449 (C4 (Web Text))

Max Activating Token Index: 959

Text #6

Max Range: 1.9444. Min Range: -1.9444

Max Act: 1.4338. Min Act: -0.0001

Data Index: 269606 (C4 (Web Text))

Max Activating Token Index: 709

Text #7

Max Range: 1.9444. Min Range: -1.9444

Max Act: 1.5203. Min Act: -0.0000

Data Index: 230873 (C4 (Web Text))

Max Activating Token Index: 895

Text #8

Max Range: 1.9444. Min Range: -1.9444

Max Act: 1.3960. Min Act: -0.0001

Data Index: 258783 (C4 (Web Text))

Max Activating Token Index: 808

Text #9

Max Range: 1.9444. Min Range: -1.9444

Max Act: 1.3821. Min Act: -0.0001

Data Index: 545681 (C4 (Web Text))

Max Activating Token Index: 927

Text #10

Max Range: 1.9444. Min Range: -1.9444

Max Act: 1.3657. Min Act: -0.0001

Data Index: 406580 (C4 (Web Text))

Max Activating Token Index: 827

Text #11

Max Range: 1.9444. Min Range: -1.9444

Max Act: 1.3523. Min Act: -0.0000

Data Index: 557999 (C4 (Web Text))

Max Activating Token Index: 991

Text #12

Max Range: 1.9444. Min Range: -1.9444

Max Act: 1.3580. Min Act: -0.0000

Data Index: 490003 (C4 (Web Text))

Max Activating Token Index: 30

Text #13

Max Range: 1.9444. Min Range: -1.9444

Max Act: 1.3304. Min Act: -0.0001

Data Index: 1017725 (C4 (Web Text))

Max Activating Token Index: 542

Text #14

Max Range: 1.9444. Min Range: -1.9444

Max Act: 1.3151. Min Act: -0.0001

Data Index: 1209408 (C4 (Web Text))

Max Activating Token Index: 69

Text #15

Max Range: 1.9444. Min Range: -1.9444

Max Act: 1.2979. Min Act: -0.0001

Data Index: 373344 (C4 (Web Text))

Max Activating Token Index: 751

Text #16

Max Range: 1.9444. Min Range: -1.9444

Max Act: 1.3838. Min Act: -0.0001

Data Index: 943546 (C4 (Web Text))

Max Activating Token Index: 632

Text #17

Max Range: 1.9444. Min Range: -1.9444

Max Act: 1.3758. Min Act: -0.0001

Data Index: 294742 (C4 (Web Text))

Max Activating Token Index: 356

Text #18

Max Range: 1.9444. Min Range: -1.9444

Max Act: 1.2783. Min Act: -0.0001

Data Index: 840632 (C4 (Web Text))

Max Activating Token Index: 465

Text #19

Max Range: 1.9444. Min Range: -1.9444

Max Act: 1.3315. Min Act: -0.0001

Data Index: 417656 (C4 (Web Text))

Max Activating Token Index: 1007