Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 238 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.4210. Min Range: -0.4210

Max Act: 0.4210. Min Act: -0.0000

Data Index: 790329 (C4 (Web Text))

Max Activating Token Index: 625

Text #1

Max Range: 0.4210. Min Range: -0.4210

Max Act: 0.3498. Min Act: -0.0000

Data Index: 343285 (C4 (Web Text))

Max Activating Token Index: 911

Text #2

Max Range: 0.4210. Min Range: -0.4210

Max Act: 0.3438. Min Act: -0.0000

Data Index: 693216 (C4 (Web Text))

Max Activating Token Index: 65

Text #3

Max Range: 0.4210. Min Range: -0.4210

Max Act: 0.3363. Min Act: -0.0000

Data Index: 193547 (C4 (Web Text))

Max Activating Token Index: 189

Text #4

Max Range: 0.4210. Min Range: -0.4210

Max Act: 0.3318. Min Act: -0.0000

Data Index: 1167055 (C4 (Web Text))

Max Activating Token Index: 995

Text #5

Max Range: 0.4210. Min Range: -0.4210

Max Act: 0.3313. Min Act: -0.0000

Data Index: 53473 (C4 (Web Text))

Max Activating Token Index: 617

Text #6

Max Range: 0.4210. Min Range: -0.4210

Max Act: 0.3113. Min Act: -0.0000

Data Index: 8809 (C4 (Web Text))

Max Activating Token Index: 22

Text #7

Max Range: 0.4210. Min Range: -0.4210

Max Act: 0.3103. Min Act: -0.0000

Data Index: 950235 (C4 (Web Text))

Max Activating Token Index: 949

Text #8

Max Range: 0.4210. Min Range: -0.4210

Max Act: 0.2806. Min Act: -0.0000

Data Index: 1016935 (C4 (Web Text))

Max Activating Token Index: 130

Text #9

Max Range: 0.4210. Min Range: -0.4210

Max Act: 0.2796. Min Act: -0.0000

Data Index: 519394 (C4 (Web Text))

Max Activating Token Index: 232

Text #10

Max Range: 0.4210. Min Range: -0.4210

Max Act: 0.2797. Min Act: -0.0000

Data Index: 31257 (C4 (Web Text))

Max Activating Token Index: 496

Text #11

Max Range: 0.4210. Min Range: -0.4210

Max Act: 0.2757. Min Act: -0.0000

Data Index: 37277 (C4 (Web Text))

Max Activating Token Index: 902

Text #12

Max Range: 0.4210. Min Range: -0.4210

Max Act: 0.2654. Min Act: -0.0000

Data Index: 587709 (C4 (Web Text))

Max Activating Token Index: 410

Text #13

Max Range: 0.4210. Min Range: -0.4210

Max Act: 0.2734. Min Act: -0.0000

Data Index: 1174035 (C4 (Web Text))

Max Activating Token Index: 354

Text #14

Max Range: 0.4210. Min Range: -0.4210

Max Act: 0.2713. Min Act: -0.0000

Data Index: 1055816 (C4 (Web Text))

Max Activating Token Index: 705

Text #15

Max Range: 0.4210. Min Range: -0.4210

Max Act: 0.2544. Min Act: -0.0000

Data Index: 743837 (C4 (Web Text))

Max Activating Token Index: 760

Text #16

Max Range: 0.4210. Min Range: -0.4210

Max Act: 0.2597. Min Act: -0.0000

Data Index: 626456 (C4 (Web Text))

Max Activating Token Index: 791

Text #17

Max Range: 0.4210. Min Range: -0.4210

Max Act: 0.2348. Min Act: -0.0000

Data Index: 424223 (C4 (Web Text))

Max Activating Token Index: 812

Text #18

Max Range: 0.4210. Min Range: -0.4210

Max Act: 0.2401. Min Act: -0.0000

Data Index: 441922 (C4 (Web Text))

Max Activating Token Index: 890

Text #19

Max Range: 0.4210. Min Range: -0.4210

Max Act: 0.2650. Min Act: -0.0000

Data Index: 913407 (C4 (Web Text))

Max Activating Token Index: 853