Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4007 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.5203. Min Range: -2.5203

Max Act: 2.5203. Min Act: -0.0000

Data Index: 793231 (C4 (Web Text))

Max Activating Token Index: 646

Text #1

Max Range: 2.5203. Min Range: -2.5203

Max Act: 2.1439. Min Act: -0.0000

Data Index: 478826 (C4 (Web Text))

Max Activating Token Index: 757

Text #2

Max Range: 2.5203. Min Range: -2.5203

Max Act: 1.6493. Min Act: -0.0000

Data Index: 102959 (C4 (Web Text))

Max Activating Token Index: 657

Text #3

Max Range: 2.5203. Min Range: -2.5203

Max Act: 1.5327. Min Act: -0.0000

Data Index: 72404 (C4 (Web Text))

Max Activating Token Index: 307

Text #4

Max Range: 2.5203. Min Range: -2.5203

Max Act: 1.6885. Min Act: -0.0000

Data Index: 1265052 (C4 (Web Text))

Max Activating Token Index: 504

Text #5

Max Range: 2.5203. Min Range: -2.5203

Max Act: 1.6155. Min Act: -0.0000

Data Index: 702970 (C4 (Web Text))

Max Activating Token Index: 512

Text #6

Max Range: 2.5203. Min Range: -2.5203

Max Act: 1.5356. Min Act: -0.0000

Data Index: 535955 (C4 (Web Text))

Max Activating Token Index: 484

Text #7

Max Range: 2.5203. Min Range: -2.5203

Max Act: 1.4751. Min Act: -0.0000

Data Index: 955656 (C4 (Web Text))

Max Activating Token Index: 729

Text #8

Max Range: 2.5203. Min Range: -2.5203

Max Act: 1.5012. Min Act: -0.0000

Data Index: 523241 (C4 (Web Text))

Max Activating Token Index: 480

Text #9

Max Range: 2.5203. Min Range: -2.5203

Max Act: 1.4421. Min Act: -0.0000

Data Index: 936138 (C4 (Web Text))

Max Activating Token Index: 422

Text #10

Max Range: 2.5203. Min Range: -2.5203

Max Act: 1.4003. Min Act: -0.0000

Data Index: 1325962 (C4 (Web Text))

Max Activating Token Index: 188

Text #11

Max Range: 2.5203. Min Range: -2.5203

Max Act: 1.2887. Min Act: -0.0000

Data Index: 435866 (C4 (Web Text))

Max Activating Token Index: 697

Text #12

Max Range: 2.5203. Min Range: -2.5203

Max Act: 1.0880. Min Act: -0.0000

Data Index: 826186 (C4 (Web Text))

Max Activating Token Index: 684

Text #13

Max Range: 2.5203. Min Range: -2.5203

Max Act: 1.2285. Min Act: -0.0000

Data Index: 431059 (C4 (Web Text))

Max Activating Token Index: 494

Text #14

Max Range: 2.5203. Min Range: -2.5203

Max Act: 1.1700. Min Act: -0.0000

Data Index: 165955 (C4 (Web Text))

Max Activating Token Index: 813

Text #15

Max Range: 2.5203. Min Range: -2.5203

Max Act: 1.2620. Min Act: -0.0000

Data Index: 1250202 (C4 (Web Text))

Max Activating Token Index: 274

Text #16

Max Range: 2.5203. Min Range: -2.5203

Max Act: 1.2022. Min Act: -0.0000

Data Index: 833749 (C4 (Web Text))

Max Activating Token Index: 858

Text #17

Max Range: 2.5203. Min Range: -2.5203

Max Act: 1.2050. Min Act: -0.0000

Data Index: 1037425 (C4 (Web Text))

Max Activating Token Index: 640

Text #18

Max Range: 2.5203. Min Range: -2.5203

Max Act: 1.2828. Min Act: -0.0000

Data Index: 827114 (C4 (Web Text))

Max Activating Token Index: 949

Text #19

Max Range: 2.5203. Min Range: -2.5203

Max Act: 1.0327. Min Act: -0.0000

Data Index: 325545 (C4 (Web Text))

Max Activating Token Index: 728