Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5656 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 4.0423. Min Range: -4.0423

Max Act: 3.6795. Min Act: -0.0000

Data Index: 23886 (C4 (Web Text))

Max Activating Token Index: 800

Text #1

Max Range: 4.0423. Min Range: -4.0423

Max Act: 4.0423. Min Act: -0.0000

Data Index: 221778 (C4 (Web Text))

Max Activating Token Index: 421

Text #2

Max Range: 4.0423. Min Range: -4.0423

Max Act: 3.3399. Min Act: -0.0000

Data Index: 971130 (C4 (Web Text))

Max Activating Token Index: 859

Text #3

Max Range: 4.0423. Min Range: -4.0423

Max Act: 3.4728. Min Act: -0.0000

Data Index: 1007816 (C4 (Web Text))

Max Activating Token Index: 508

Text #4

Max Range: 4.0423. Min Range: -4.0423

Max Act: 3.6305. Min Act: -0.0000

Data Index: 1097563 (C4 (Web Text))

Max Activating Token Index: 43

Text #5

Max Range: 4.0423. Min Range: -4.0423

Max Act: 3.5935. Min Act: -0.0000

Data Index: 982320 (C4 (Web Text))

Max Activating Token Index: 11

Text #6

Max Range: 4.0423. Min Range: -4.0423

Max Act: 3.9261. Min Act: -0.0000

Data Index: 925955 (C4 (Web Text))

Max Activating Token Index: 193

Text #7

Max Range: 4.0423. Min Range: -4.0423

Max Act: 3.7440. Min Act: -0.0000

Data Index: 6389 (C4 (Web Text))

Max Activating Token Index: 724

Text #8

Max Range: 4.0423. Min Range: -4.0423

Max Act: 3.5732. Min Act: -0.0000

Data Index: 1017159 (C4 (Web Text))

Max Activating Token Index: 15

Text #9

Max Range: 4.0423. Min Range: -4.0423

Max Act: 3.6139. Min Act: -0.0000

Data Index: 916810 (C4 (Web Text))

Max Activating Token Index: 617

Text #10

Max Range: 4.0423. Min Range: -4.0423

Max Act: 3.4735. Min Act: -0.0000

Data Index: 54211 (C4 (Web Text))

Max Activating Token Index: 553

Text #11

Max Range: 4.0423. Min Range: -4.0423

Max Act: 3.6009. Min Act: -0.0000

Data Index: 477936 (C4 (Web Text))

Max Activating Token Index: 616

Text #12

Max Range: 4.0423. Min Range: -4.0423

Max Act: 3.5723. Min Act: -0.0000

Data Index: 154437 (C4 (Web Text))

Max Activating Token Index: 826

Text #13

Max Range: 4.0423. Min Range: -4.0423

Max Act: 3.3639. Min Act: -0.0000

Data Index: 433279 (C4 (Web Text))

Max Activating Token Index: 867

Text #14

Max Range: 4.0423. Min Range: -4.0423

Max Act: 3.3403. Min Act: -0.0000

Data Index: 143605 (C4 (Web Text))

Max Activating Token Index: 332

Text #15

Max Range: 4.0423. Min Range: -4.0423

Max Act: 3.2749. Min Act: -0.0000

Data Index: 621312 (C4 (Web Text))

Max Activating Token Index: 1006

Text #16

Max Range: 4.0423. Min Range: -4.0423

Max Act: 3.0666. Min Act: -0.0000

Data Index: 1500322 (Python Code)

Max Activating Token Index: 16

Text #17

Max Range: 4.0423. Min Range: -4.0423

Max Act: 3.2171. Min Act: -0.0000

Data Index: 517717 (C4 (Web Text))

Max Activating Token Index: 185

Text #18

Max Range: 4.0423. Min Range: -4.0423

Max Act: 3.3453. Min Act: -0.0000

Data Index: 1195526 (C4 (Web Text))

Max Activating Token Index: 225

Text #19

Max Range: 4.0423. Min Range: -4.0423

Max Act: 3.2011. Min Act: -0.0000

Data Index: 558893 (C4 (Web Text))

Max Activating Token Index: 123