Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2859 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 1.0521. Min Range: -1.0521

Max Act: 1.0521. Min Act: -0.0002

Data Index: 1228790 (C4 (Web Text))

Max Activating Token Index: 758

Text #1

Max Range: 1.0521. Min Range: -1.0521

Max Act: 0.9922. Min Act: -0.0001

Data Index: 232880 (C4 (Web Text))

Max Activating Token Index: 873

Text #2

Max Range: 1.0521. Min Range: -1.0521

Max Act: 1.0209. Min Act: -0.0002

Data Index: 933610 (C4 (Web Text))

Max Activating Token Index: 744

Text #3

Max Range: 1.0521. Min Range: -1.0521

Max Act: 0.9790. Min Act: -0.0001

Data Index: 1251242 (C4 (Web Text))

Max Activating Token Index: 557

Text #4

Max Range: 1.0521. Min Range: -1.0521

Max Act: 1.0520. Min Act: -0.0002

Data Index: 1276026 (C4 (Web Text))

Max Activating Token Index: 924

Text #5

Max Range: 1.0521. Min Range: -1.0521

Max Act: 0.8880. Min Act: -0.0002

Data Index: 625740 (C4 (Web Text))

Max Activating Token Index: 765

Text #6

Max Range: 1.0521. Min Range: -1.0521

Max Act: 0.8702. Min Act: -0.0001

Data Index: 443500 (C4 (Web Text))

Max Activating Token Index: 989

Text #7

Max Range: 1.0521. Min Range: -1.0521

Max Act: 0.8459. Min Act: -0.0002

Data Index: 650168 (C4 (Web Text))

Max Activating Token Index: 748

Text #8

Max Range: 1.0521. Min Range: -1.0521

Max Act: 0.8787. Min Act: -0.0002

Data Index: 1238395 (C4 (Web Text))

Max Activating Token Index: 456

Text #9

Max Range: 1.0521. Min Range: -1.0521

Max Act: 0.8655. Min Act: -0.0001

Data Index: 206679 (C4 (Web Text))

Max Activating Token Index: 786

Text #10

Max Range: 1.0521. Min Range: -1.0521

Max Act: 0.9484. Min Act: -0.0001

Data Index: 518642 (C4 (Web Text))

Max Activating Token Index: 499

Text #11

Max Range: 1.0521. Min Range: -1.0521

Max Act: 0.7862. Min Act: -0.0002

Data Index: 664403 (C4 (Web Text))

Max Activating Token Index: 987

Text #12

Max Range: 1.0521. Min Range: -1.0521

Max Act: 0.7854. Min Act: -0.0002

Data Index: 299593 (C4 (Web Text))

Max Activating Token Index: 487

Text #13

Max Range: 1.0521. Min Range: -1.0521

Max Act: 0.8901. Min Act: -0.0002

Data Index: 653138 (C4 (Web Text))

Max Activating Token Index: 721

Text #14

Max Range: 1.0521. Min Range: -1.0521

Max Act: 0.8515. Min Act: -0.0002

Data Index: 265673 (C4 (Web Text))

Max Activating Token Index: 806

Text #15

Max Range: 1.0521. Min Range: -1.0521

Max Act: 0.9082. Min Act: -0.0001

Data Index: 917463 (C4 (Web Text))

Max Activating Token Index: 538

Text #16

Max Range: 1.0521. Min Range: -1.0521

Max Act: 0.8784. Min Act: -0.0001

Data Index: 616190 (C4 (Web Text))

Max Activating Token Index: 854

Text #17

Max Range: 1.0521. Min Range: -1.0521

Max Act: 0.8723. Min Act: -0.0002

Data Index: 735295 (C4 (Web Text))

Max Activating Token Index: 270

Text #18

Max Range: 1.0521. Min Range: -1.0521

Max Act: 0.8941. Min Act: -0.0002

Data Index: 622777 (C4 (Web Text))

Max Activating Token Index: 685

Text #19

Max Range: 1.0521. Min Range: -1.0521

Max Act: 0.7635. Min Act: -0.0002

Data Index: 597906 (C4 (Web Text))

Max Activating Token Index: 272