Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5221 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 4.4709. Min Range: -4.4709

Max Act: 4.4709. Min Act: -0.0000

Data Index: 513240 (C4 (Web Text))

Max Activating Token Index: 188

Text #1

Max Range: 4.4709. Min Range: -4.4709

Max Act: 3.7537. Min Act: -0.0000

Data Index: 1214612 (C4 (Web Text))

Max Activating Token Index: 963

Text #2

Max Range: 4.4709. Min Range: -4.4709

Max Act: 3.4746. Min Act: -0.0000

Data Index: 1265361 (C4 (Web Text))

Max Activating Token Index: 153

Text #3

Max Range: 4.4709. Min Range: -4.4709

Max Act: 3.2437. Min Act: -0.0000

Data Index: 648925 (C4 (Web Text))

Max Activating Token Index: 171

Text #4

Max Range: 4.4709. Min Range: -4.4709

Max Act: 2.8426. Min Act: -0.0000

Data Index: 721979 (C4 (Web Text))

Max Activating Token Index: 219

Text #5

Max Range: 4.4709. Min Range: -4.4709

Max Act: 2.7914. Min Act: -0.0000

Data Index: 748390 (C4 (Web Text))

Max Activating Token Index: 28

Text #6

Max Range: 4.4709. Min Range: -4.4709

Max Act: 2.8687. Min Act: -0.0000

Data Index: 792237 (C4 (Web Text))

Max Activating Token Index: 510

Text #7

Max Range: 4.4709. Min Range: -4.4709

Max Act: 2.8320. Min Act: -0.0000

Data Index: 1353678 (C4 (Web Text))

Max Activating Token Index: 20

Text #8

Max Range: 4.4709. Min Range: -4.4709

Max Act: 2.9243. Min Act: -0.0000

Data Index: 831428 (C4 (Web Text))

Max Activating Token Index: 38

Text #9

Max Range: 4.4709. Min Range: -4.4709

Max Act: 2.5894. Min Act: -0.0000

Data Index: 558496 (C4 (Web Text))

Max Activating Token Index: 715

Text #10

Max Range: 4.4709. Min Range: -4.4709

Max Act: 2.7008. Min Act: -0.0000

Data Index: 221414 (C4 (Web Text))

Max Activating Token Index: 697

Text #11

Max Range: 4.4709. Min Range: -4.4709

Max Act: 2.5222. Min Act: -0.0000

Data Index: 916280 (C4 (Web Text))

Max Activating Token Index: 811

Text #12

Max Range: 4.4709. Min Range: -4.4709

Max Act: 2.5855. Min Act: -0.0000

Data Index: 1317075 (C4 (Web Text))

Max Activating Token Index: 138

Text #13

Max Range: 4.4709. Min Range: -4.4709

Max Act: 2.5998. Min Act: -0.0000

Data Index: 674875 (C4 (Web Text))

Max Activating Token Index: 831

Text #14

Max Range: 4.4709. Min Range: -4.4709

Max Act: 2.4533. Min Act: -0.0000

Data Index: 342122 (C4 (Web Text))

Max Activating Token Index: 957

Text #15

Max Range: 4.4709. Min Range: -4.4709

Max Act: 2.4939. Min Act: -0.0000

Data Index: 33973 (C4 (Web Text))

Max Activating Token Index: 990

Text #16

Max Range: 4.4709. Min Range: -4.4709

Max Act: 2.4372. Min Act: -0.0000

Data Index: 1021870 (C4 (Web Text))

Max Activating Token Index: 1017

Text #17

Max Range: 4.4709. Min Range: -4.4709

Max Act: 2.4911. Min Act: -0.0000

Data Index: 1201561 (C4 (Web Text))

Max Activating Token Index: 745

Text #18

Max Range: 4.4709. Min Range: -4.4709

Max Act: 2.0944. Min Act: -0.0000

Data Index: 258626 (C4 (Web Text))

Max Activating Token Index: 716

Text #19

Max Range: 4.4709. Min Range: -4.4709

Max Act: 2.1756. Min Act: -0.0000

Data Index: 1294942 (C4 (Web Text))

Max Activating Token Index: 56