Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1651 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.0792. Min Range: -2.0792

Max Act: 2.0792. Min Act: -0.0000

Data Index: 454180 (C4 (Web Text))

Max Activating Token Index: 54

Text #1

Max Range: 2.0792. Min Range: -2.0792

Max Act: 1.9614. Min Act: -0.0000

Data Index: 23687 (C4 (Web Text))

Max Activating Token Index: 545

Text #2

Max Range: 2.0792. Min Range: -2.0792

Max Act: 1.8328. Min Act: -0.0000

Data Index: 282515 (C4 (Web Text))

Max Activating Token Index: 541

Text #3

Max Range: 2.0792. Min Range: -2.0792

Max Act: 1.9396. Min Act: -0.0000

Data Index: 59146 (C4 (Web Text))

Max Activating Token Index: 244

Text #4

Max Range: 2.0792. Min Range: -2.0792

Max Act: 1.9066. Min Act: -0.0000

Data Index: 252798 (C4 (Web Text))

Max Activating Token Index: 247

Text #5

Max Range: 2.0792. Min Range: -2.0792

Max Act: 2.0064. Min Act: -0.0000

Data Index: 429214 (C4 (Web Text))

Max Activating Token Index: 718

Text #6

Max Range: 2.0792. Min Range: -2.0792

Max Act: 1.8666. Min Act: -0.0000

Data Index: 721168 (C4 (Web Text))

Max Activating Token Index: 579

Text #7

Max Range: 2.0792. Min Range: -2.0792

Max Act: 1.9666. Min Act: -0.0000

Data Index: 1331829 (C4 (Web Text))

Max Activating Token Index: 706

Text #8

Max Range: 2.0792. Min Range: -2.0792

Max Act: 1.8638. Min Act: -0.0000

Data Index: 1329189 (C4 (Web Text))

Max Activating Token Index: 995

Text #9

Max Range: 2.0792. Min Range: -2.0792

Max Act: 1.7886. Min Act: -0.0000

Data Index: 518016 (C4 (Web Text))

Max Activating Token Index: 354

Text #10

Max Range: 2.0792. Min Range: -2.0792

Max Act: 1.7585. Min Act: -0.0000

Data Index: 159115 (C4 (Web Text))

Max Activating Token Index: 731

Text #11

Max Range: 2.0792. Min Range: -2.0792

Max Act: 1.7592. Min Act: -0.0000

Data Index: 524960 (C4 (Web Text))

Max Activating Token Index: 894

Text #12

Max Range: 2.0792. Min Range: -2.0792

Max Act: 1.7696. Min Act: -0.0000

Data Index: 330585 (C4 (Web Text))

Max Activating Token Index: 530

Text #13

Max Range: 2.0792. Min Range: -2.0792

Max Act: 1.7424. Min Act: -0.0000

Data Index: 394225 (C4 (Web Text))

Max Activating Token Index: 940

Text #14

Max Range: 2.0792. Min Range: -2.0792

Max Act: 1.7357. Min Act: -0.0000

Data Index: 1280036 (C4 (Web Text))

Max Activating Token Index: 938

Text #15

Max Range: 2.0792. Min Range: -2.0792

Max Act: 1.8303. Min Act: -0.0000

Data Index: 528755 (C4 (Web Text))

Max Activating Token Index: 584

Text #16

Max Range: 2.0792. Min Range: -2.0792

Max Act: 1.5372. Min Act: -0.0000

Data Index: 542384 (C4 (Web Text))

Max Activating Token Index: 235

Text #17

Max Range: 2.0792. Min Range: -2.0792

Max Act: 1.6889. Min Act: -0.0000

Data Index: 846841 (C4 (Web Text))

Max Activating Token Index: 718

Text #18

Max Range: 2.0792. Min Range: -2.0792

Max Act: 1.6944. Min Act: -0.0000

Data Index: 835137 (C4 (Web Text))

Max Activating Token Index: 279

Text #19

Max Range: 2.0792. Min Range: -2.0792

Max Act: 1.7728. Min Act: -0.0000

Data Index: 1177878 (C4 (Web Text))

Max Activating Token Index: 740