Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1523 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.8230. Min Range: -0.8230

Max Act: 0.8230. Min Act: -0.0000

Data Index: 239614 (C4 (Web Text))

Max Activating Token Index: 379

Text #1

Max Range: 0.8230. Min Range: -0.8230

Max Act: 0.7696. Min Act: -0.0000

Data Index: 870323 (C4 (Web Text))

Max Activating Token Index: 776

Text #2

Max Range: 0.8230. Min Range: -0.8230

Max Act: 0.7031. Min Act: -0.0000

Data Index: 1256320 (C4 (Web Text))

Max Activating Token Index: 1021

Text #3

Max Range: 0.8230. Min Range: -0.8230

Max Act: 0.7142. Min Act: -0.0000

Data Index: 977159 (C4 (Web Text))

Max Activating Token Index: 289

Text #4

Max Range: 0.8230. Min Range: -0.8230

Max Act: 0.7062. Min Act: -0.0000

Data Index: 1159024 (C4 (Web Text))

Max Activating Token Index: 541

Text #5

Max Range: 0.8230. Min Range: -0.8230

Max Act: 0.7254. Min Act: -0.0000

Data Index: 452767 (C4 (Web Text))

Max Activating Token Index: 993

Text #6

Max Range: 0.8230. Min Range: -0.8230

Max Act: 0.6716. Min Act: -0.0000

Data Index: 607228 (C4 (Web Text))

Max Activating Token Index: 389

Text #7

Max Range: 0.8230. Min Range: -0.8230

Max Act: 0.6374. Min Act: -0.0000

Data Index: 599952 (C4 (Web Text))

Max Activating Token Index: 57

Text #8

Max Range: 0.8230. Min Range: -0.8230

Max Act: 0.6265. Min Act: -0.0000

Data Index: 443255 (C4 (Web Text))

Max Activating Token Index: 876

Text #9

Max Range: 0.8230. Min Range: -0.8230

Max Act: 0.6096. Min Act: -0.0000

Data Index: 994442 (C4 (Web Text))

Max Activating Token Index: 454

Text #10

Max Range: 0.8230. Min Range: -0.8230

Max Act: 0.6425. Min Act: -0.0000

Data Index: 245931 (C4 (Web Text))

Max Activating Token Index: 489

Text #11

Max Range: 0.8230. Min Range: -0.8230

Max Act: 0.5795. Min Act: -0.0000

Data Index: 858166 (C4 (Web Text))

Max Activating Token Index: 721

Text #12

Max Range: 0.8230. Min Range: -0.8230

Max Act: 0.6131. Min Act: -0.0000

Data Index: 279818 (C4 (Web Text))

Max Activating Token Index: 375

Text #13

Max Range: 0.8230. Min Range: -0.8230

Max Act: 0.6289. Min Act: -0.0000

Data Index: 1036449 (C4 (Web Text))

Max Activating Token Index: 652

Text #14

Max Range: 0.8230. Min Range: -0.8230

Max Act: 0.6315. Min Act: -0.0000

Data Index: 184677 (C4 (Web Text))

Max Activating Token Index: 989

Text #15

Max Range: 0.8230. Min Range: -0.8230

Max Act: 0.6061. Min Act: -0.0000

Data Index: 27092 (C4 (Web Text))

Max Activating Token Index: 742

Text #16

Max Range: 0.8230. Min Range: -0.8230

Max Act: 0.5815. Min Act: -0.0000

Data Index: 183479 (C4 (Web Text))

Max Activating Token Index: 80

Text #17

Max Range: 0.8230. Min Range: -0.8230

Max Act: 0.5552. Min Act: -0.0000

Data Index: 1356210 (C4 (Web Text))

Max Activating Token Index: 295

Text #18

Max Range: 0.8230. Min Range: -0.8230

Max Act: 0.5741. Min Act: -0.0000

Data Index: 219089 (C4 (Web Text))

Max Activating Token Index: 595

Text #19

Max Range: 0.8230. Min Range: -0.8230

Max Act: 0.5482. Min Act: -0.0000

Data Index: 1115792 (C4 (Web Text))

Max Activating Token Index: 59