Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1777 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.5061. Min Range: -1.5061

Max Act: 1.4022. Min Act: -0.0000

Data Index: 123478 (C4 (Web Text))

Max Activating Token Index: 730

Text #1

Max Range: 1.5061. Min Range: -1.5061

Max Act: 1.5061. Min Act: -0.0000

Data Index: 1249251 (C4 (Web Text))

Max Activating Token Index: 97

Text #2

Max Range: 1.5061. Min Range: -1.5061

Max Act: 1.3323. Min Act: -0.0000

Data Index: 314928 (C4 (Web Text))

Max Activating Token Index: 351

Text #3

Max Range: 1.5061. Min Range: -1.5061

Max Act: 1.2071. Min Act: -0.0000

Data Index: 97183 (C4 (Web Text))

Max Activating Token Index: 313

Text #4

Max Range: 1.5061. Min Range: -1.5061

Max Act: 1.3770. Min Act: -0.0000

Data Index: 505498 (C4 (Web Text))

Max Activating Token Index: 56

Text #5

Max Range: 1.5061. Min Range: -1.5061

Max Act: 1.1584. Min Act: -0.0000

Data Index: 770066 (C4 (Web Text))

Max Activating Token Index: 416

Text #6

Max Range: 1.5061. Min Range: -1.5061

Max Act: 0.9340. Min Act: -0.0000

Data Index: 166831 (C4 (Web Text))

Max Activating Token Index: 713

Text #7

Max Range: 1.5061. Min Range: -1.5061

Max Act: 1.0170. Min Act: -0.0000

Data Index: 101246 (C4 (Web Text))

Max Activating Token Index: 385

Text #8

Max Range: 1.5061. Min Range: -1.5061

Max Act: 1.0273. Min Act: -0.0000

Data Index: 845607 (C4 (Web Text))

Max Activating Token Index: 515

Text #9

Max Range: 1.5061. Min Range: -1.5061

Max Act: 0.9905. Min Act: -0.0000

Data Index: 282733 (C4 (Web Text))

Max Activating Token Index: 469

Text #10

Max Range: 1.5061. Min Range: -1.5061

Max Act: 0.9772. Min Act: -0.0000

Data Index: 794699 (C4 (Web Text))

Max Activating Token Index: 239

Text #11

Max Range: 1.5061. Min Range: -1.5061

Max Act: 0.8686. Min Act: -0.0000

Data Index: 678509 (C4 (Web Text))

Max Activating Token Index: 931

Text #12

Max Range: 1.5061. Min Range: -1.5061

Max Act: 0.8723. Min Act: -0.0000

Data Index: 338232 (C4 (Web Text))

Max Activating Token Index: 749

Text #13

Max Range: 1.5061. Min Range: -1.5061

Max Act: 0.8845. Min Act: -0.0000

Data Index: 122249 (C4 (Web Text))

Max Activating Token Index: 390

Text #14

Max Range: 1.5061. Min Range: -1.5061

Max Act: 0.8064. Min Act: -0.0000

Data Index: 1047719 (C4 (Web Text))

Max Activating Token Index: 778

Text #15

Max Range: 1.5061. Min Range: -1.5061

Max Act: 0.8736. Min Act: -0.0000

Data Index: 331561 (C4 (Web Text))

Max Activating Token Index: 459

Text #16

Max Range: 1.5061. Min Range: -1.5061

Max Act: 0.7918. Min Act: -0.0000

Data Index: 844145 (C4 (Web Text))

Max Activating Token Index: 298

Text #17

Max Range: 1.5061. Min Range: -1.5061

Max Act: 0.9105. Min Act: -0.0000

Data Index: 13652 (C4 (Web Text))

Max Activating Token Index: 833

Text #18

Max Range: 1.5061. Min Range: -1.5061

Max Act: 0.8801. Min Act: -0.0000

Data Index: 440819 (C4 (Web Text))

Max Activating Token Index: 811

Text #19

Max Range: 1.5061. Min Range: -1.5061

Max Act: 0.8265. Min Act: -0.0000

Data Index: 759888 (C4 (Web Text))

Max Activating Token Index: 417