Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1847 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.9608. Min Range: -1.9608

Max Act: 1.9608. Min Act: -0.0000

Data Index: 991698 (C4 (Web Text))

Max Activating Token Index: 844

Text #1

Max Range: 1.9608. Min Range: -1.9608

Max Act: 1.8559. Min Act: -0.0000

Data Index: 384250 (C4 (Web Text))

Max Activating Token Index: 95

Text #2

Max Range: 1.9608. Min Range: -1.9608

Max Act: 1.8379. Min Act: -0.0000

Data Index: 181655 (C4 (Web Text))

Max Activating Token Index: 488

Text #3

Max Range: 1.9608. Min Range: -1.9608

Max Act: 1.6909. Min Act: -0.0000

Data Index: 157175 (C4 (Web Text))

Max Activating Token Index: 364

Text #4

Max Range: 1.9608. Min Range: -1.9608

Max Act: 1.6562. Min Act: -0.0000

Data Index: 670653 (C4 (Web Text))

Max Activating Token Index: 413

Text #5

Max Range: 1.9608. Min Range: -1.9608

Max Act: 1.5494. Min Act: -0.0000

Data Index: 120009 (C4 (Web Text))

Max Activating Token Index: 545

Text #6

Max Range: 1.9608. Min Range: -1.9608

Max Act: 1.6029. Min Act: -0.0000

Data Index: 798057 (C4 (Web Text))

Max Activating Token Index: 731

Text #7

Max Range: 1.9608. Min Range: -1.9608

Max Act: 1.5888. Min Act: -0.0000

Data Index: 757375 (C4 (Web Text))

Max Activating Token Index: 446

Text #8

Max Range: 1.9608. Min Range: -1.9608

Max Act: 1.5827. Min Act: -0.0000

Data Index: 821058 (C4 (Web Text))

Max Activating Token Index: 52

Text #9

Max Range: 1.9608. Min Range: -1.9608

Max Act: 1.5445. Min Act: -0.0000

Data Index: 1094479 (C4 (Web Text))

Max Activating Token Index: 818

Text #10

Max Range: 1.9608. Min Range: -1.9608

Max Act: 1.5253. Min Act: -0.0000

Data Index: 195055 (C4 (Web Text))

Max Activating Token Index: 800

Text #11

Max Range: 1.9608. Min Range: -1.9608

Max Act: 1.5103. Min Act: -0.0000

Data Index: 110939 (C4 (Web Text))

Max Activating Token Index: 100

Text #12

Max Range: 1.9608. Min Range: -1.9608

Max Act: 1.4227. Min Act: -0.0000

Data Index: 1113494 (C4 (Web Text))

Max Activating Token Index: 767

Text #13

Max Range: 1.9608. Min Range: -1.9608

Max Act: 1.5181. Min Act: -0.0000

Data Index: 744535 (C4 (Web Text))

Max Activating Token Index: 775

Text #14

Max Range: 1.9608. Min Range: -1.9608

Max Act: 1.5394. Min Act: -0.0000

Data Index: 1227331 (C4 (Web Text))

Max Activating Token Index: 97

Text #15

Max Range: 1.9608. Min Range: -1.9608

Max Act: 1.6941. Min Act: -0.0000

Data Index: 1029035 (C4 (Web Text))

Max Activating Token Index: 996

Text #16

Max Range: 1.9608. Min Range: -1.9608

Max Act: 1.5822. Min Act: -0.0000

Data Index: 1010123 (C4 (Web Text))

Max Activating Token Index: 857

Text #17

Max Range: 1.9608. Min Range: -1.9608

Max Act: 1.5666. Min Act: -0.0000

Data Index: 923132 (C4 (Web Text))

Max Activating Token Index: 87

Text #18

Max Range: 1.9608. Min Range: -1.9608

Max Act: 1.3195. Min Act: -0.0000

Data Index: 1029502 (C4 (Web Text))

Max Activating Token Index: 722

Text #19

Max Range: 1.9608. Min Range: -1.9608

Max Act: 1.4775. Min Act: -0.0000

Data Index: 1261781 (C4 (Web Text))

Max Activating Token Index: 108