Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2712 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.4788. Min Range: -0.4788

Max Act: 0.4788. Min Act: -0.0000

Data Index: 738005 (C4 (Web Text))

Max Activating Token Index: 433

Text #1

Max Range: 0.4788. Min Range: -0.4788

Max Act: 0.4168. Min Act: -0.0000

Data Index: 1349213 (C4 (Web Text))

Max Activating Token Index: 46

Text #2

Max Range: 0.4788. Min Range: -0.4788

Max Act: 0.4233. Min Act: -0.0000

Data Index: 620704 (C4 (Web Text))

Max Activating Token Index: 86

Text #3

Max Range: 0.4788. Min Range: -0.4788

Max Act: 0.4043. Min Act: -0.0000

Data Index: 447867 (C4 (Web Text))

Max Activating Token Index: 1007

Text #4

Max Range: 0.4788. Min Range: -0.4788

Max Act: 0.4095. Min Act: -0.0000

Data Index: 656216 (C4 (Web Text))

Max Activating Token Index: 459

Text #5

Max Range: 0.4788. Min Range: -0.4788

Max Act: 0.4010. Min Act: -0.0000

Data Index: 333898 (C4 (Web Text))

Max Activating Token Index: 177

Text #6

Max Range: 0.4788. Min Range: -0.4788

Max Act: 0.3856. Min Act: -0.0000

Data Index: 487654 (C4 (Web Text))

Max Activating Token Index: 447

Text #7

Max Range: 0.4788. Min Range: -0.4788

Max Act: 0.4189. Min Act: -0.0000

Data Index: 227651 (C4 (Web Text))

Max Activating Token Index: 661

Text #8

Max Range: 0.4788. Min Range: -0.4788

Max Act: 0.3760. Min Act: -0.0000

Data Index: 594137 (C4 (Web Text))

Max Activating Token Index: 223

Text #9

Max Range: 0.4788. Min Range: -0.4788

Max Act: 0.3697. Min Act: -0.0000

Data Index: 766510 (C4 (Web Text))

Max Activating Token Index: 86

Text #10

Max Range: 0.4788. Min Range: -0.4788

Max Act: 0.3792. Min Act: -0.0000

Data Index: 188171 (C4 (Web Text))

Max Activating Token Index: 434

Text #11

Max Range: 0.4788. Min Range: -0.4788

Max Act: 0.3764. Min Act: -0.0000

Data Index: 1198784 (C4 (Web Text))

Max Activating Token Index: 42

Text #12

Max Range: 0.4788. Min Range: -0.4788

Max Act: 0.3723. Min Act: -0.0000

Data Index: 871975 (C4 (Web Text))

Max Activating Token Index: 386

Text #13

Max Range: 0.4788. Min Range: -0.4788

Max Act: 0.3568. Min Act: -0.0000

Data Index: 27181 (C4 (Web Text))

Max Activating Token Index: 882

Text #14

Max Range: 0.4788. Min Range: -0.4788

Max Act: 0.3974. Min Act: -0.0000

Data Index: 814238 (C4 (Web Text))

Max Activating Token Index: 151

Text #15

Max Range: 0.4788. Min Range: -0.4788

Max Act: 0.3569. Min Act: -0.0000

Data Index: 1197547 (C4 (Web Text))

Max Activating Token Index: 633

Text #16

Max Range: 0.4788. Min Range: -0.4788

Max Act: 0.3785. Min Act: -0.0000

Data Index: 293669 (C4 (Web Text))

Max Activating Token Index: 991

Text #17

Max Range: 0.4788. Min Range: -0.4788

Max Act: 0.3684. Min Act: -0.0000

Data Index: 344486 (C4 (Web Text))

Max Activating Token Index: 493

Text #18

Max Range: 0.4788. Min Range: -0.4788

Max Act: 0.3804. Min Act: -0.0000

Data Index: 1269989 (C4 (Web Text))

Max Activating Token Index: 199

Text #19

Max Range: 0.4788. Min Range: -0.4788

Max Act: 0.3765. Min Act: -0.0000

Data Index: 431017 (C4 (Web Text))

Max Activating Token Index: 53