Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1041 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 1.3563. Min Range: -1.3563

Max Act: 1.2857. Min Act: -0.0003

Data Index: 257108 (C4 (Web Text))

Max Activating Token Index: 62

Text #1

Max Range: 1.3563. Min Range: -1.3563

Max Act: 1.3563. Min Act: -0.0003

Data Index: 107243 (C4 (Web Text))

Max Activating Token Index: 65

Text #2

Max Range: 1.3563. Min Range: -1.3563

Max Act: 1.2946. Min Act: -0.0003

Data Index: 357168 (C4 (Web Text))

Max Activating Token Index: 105

Text #3

Max Range: 1.3563. Min Range: -1.3563

Max Act: 1.1863. Min Act: -0.0003

Data Index: 1266461 (C4 (Web Text))

Max Activating Token Index: 271

Text #4

Max Range: 1.3563. Min Range: -1.3563

Max Act: 1.2855. Min Act: -0.0003

Data Index: 436933 (C4 (Web Text))

Max Activating Token Index: 98

Text #5

Max Range: 1.3563. Min Range: -1.3563

Max Act: 1.2016. Min Act: -0.0003

Data Index: 1238440 (C4 (Web Text))

Max Activating Token Index: 436

Text #6

Max Range: 1.3563. Min Range: -1.3563

Max Act: 1.1009. Min Act: -0.0003

Data Index: 822735 (C4 (Web Text))

Max Activating Token Index: 228

Text #7

Max Range: 1.3563. Min Range: -1.3563

Max Act: 1.1758. Min Act: -0.0003

Data Index: 1254448 (C4 (Web Text))

Max Activating Token Index: 423

Text #8

Max Range: 1.3563. Min Range: -1.3563

Max Act: 1.0059. Min Act: -0.0003

Data Index: 108975 (C4 (Web Text))

Max Activating Token Index: 828

Text #9

Max Range: 1.3563. Min Range: -1.3563

Max Act: 1.1315. Min Act: -0.0003

Data Index: 947932 (C4 (Web Text))

Max Activating Token Index: 98

Text #10

Max Range: 1.3563. Min Range: -1.3563

Max Act: 0.9247. Min Act: -0.0003

Data Index: 1291121 (C4 (Web Text))

Max Activating Token Index: 634

Text #11

Max Range: 1.3563. Min Range: -1.3563

Max Act: 0.9941. Min Act: -0.0003

Data Index: 906349 (C4 (Web Text))

Max Activating Token Index: 860

Text #12

Max Range: 1.3563. Min Range: -1.3563

Max Act: 1.1539. Min Act: -0.0003

Data Index: 1265541 (C4 (Web Text))

Max Activating Token Index: 986

Text #13

Max Range: 1.3563. Min Range: -1.3563

Max Act: 1.2099. Min Act: -0.0003

Data Index: 762237 (C4 (Web Text))

Max Activating Token Index: 900

Text #14

Max Range: 1.3563. Min Range: -1.3563

Max Act: 1.0854. Min Act: -0.0003

Data Index: 930464 (C4 (Web Text))

Max Activating Token Index: 187

Text #15

Max Range: 1.3563. Min Range: -1.3563

Max Act: 1.1390. Min Act: -0.0003

Data Index: 1103583 (C4 (Web Text))

Max Activating Token Index: 171

Text #16

Max Range: 1.3563. Min Range: -1.3563

Max Act: 1.2006. Min Act: -0.0003

Data Index: 798712 (C4 (Web Text))

Max Activating Token Index: 940

Text #17

Max Range: 1.3563. Min Range: -1.3563

Max Act: 1.0526. Min Act: -0.0003

Data Index: 454091 (C4 (Web Text))

Max Activating Token Index: 592

Text #18

Max Range: 1.3563. Min Range: -1.3563

Max Act: 1.0680. Min Act: -0.0003

Data Index: 1032527 (C4 (Web Text))

Max Activating Token Index: 143

Text #19

Max Range: 1.3563. Min Range: -1.3563

Max Act: 1.2264. Min Act: -0.0003

Data Index: 1133513 (C4 (Web Text))

Max Activating Token Index: 212