Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5587 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.5601. Min Range: -1.5601

Max Act: 1.5601. Min Act: -0.0000

Data Index: 1339283 (C4 (Web Text))

Max Activating Token Index: 470

Text #1

Max Range: 1.5601. Min Range: -1.5601

Max Act: 1.4576. Min Act: -0.0000

Data Index: 622993 (C4 (Web Text))

Max Activating Token Index: 446

Text #2

Max Range: 1.5601. Min Range: -1.5601

Max Act: 1.2849. Min Act: -0.0000

Data Index: 504927 (C4 (Web Text))

Max Activating Token Index: 419

Text #3

Max Range: 1.5601. Min Range: -1.5601

Max Act: 1.2442. Min Act: -0.0000

Data Index: 265995 (C4 (Web Text))

Max Activating Token Index: 269

Text #4

Max Range: 1.5601. Min Range: -1.5601

Max Act: 1.2819. Min Act: 0.0001

Data Index: 667050 (C4 (Web Text))

Max Activating Token Index: 538

Text #5

Max Range: 1.5601. Min Range: -1.5601

Max Act: 1.2028. Min Act: -0.0000

Data Index: 1059719 (C4 (Web Text))

Max Activating Token Index: 876

Text #6

Max Range: 1.5601. Min Range: -1.5601

Max Act: 1.1739. Min Act: -0.0000

Data Index: 887600 (C4 (Web Text))

Max Activating Token Index: 207

Text #7

Max Range: 1.5601. Min Range: -1.5601

Max Act: 1.2160. Min Act: -0.0000

Data Index: 641655 (C4 (Web Text))

Max Activating Token Index: 1022

Text #8

Max Range: 1.5601. Min Range: -1.5601

Max Act: 1.1496. Min Act: -0.0000

Data Index: 1008674 (C4 (Web Text))

Max Activating Token Index: 694

Text #9

Max Range: 1.5601. Min Range: -1.5601

Max Act: 1.1164. Min Act: 0.0000

Data Index: 778534 (C4 (Web Text))

Max Activating Token Index: 583

Text #10

Max Range: 1.5601. Min Range: -1.5601

Max Act: 1.1099. Min Act: -0.0000

Data Index: 503610 (C4 (Web Text))

Max Activating Token Index: 107

Text #11

Max Range: 1.5601. Min Range: -1.5601

Max Act: 1.1131. Min Act: 0.0001

Data Index: 878240 (C4 (Web Text))

Max Activating Token Index: 78

Text #12

Max Range: 1.5601. Min Range: -1.5601

Max Act: 1.0787. Min Act: 0.0000

Data Index: 1008815 (C4 (Web Text))

Max Activating Token Index: 303

Text #13

Max Range: 1.5601. Min Range: -1.5601

Max Act: 1.0132. Min Act: 0.0000

Data Index: 650738 (C4 (Web Text))

Max Activating Token Index: 249

Text #14

Max Range: 1.5601. Min Range: -1.5601

Max Act: 1.0514. Min Act: 0.0001

Data Index: 780644 (C4 (Web Text))

Max Activating Token Index: 43

Text #15

Max Range: 1.5601. Min Range: -1.5601

Max Act: 1.0354. Min Act: 0.0000

Data Index: 902251 (C4 (Web Text))

Max Activating Token Index: 106

Text #16

Max Range: 1.5601. Min Range: -1.5601

Max Act: 1.0353. Min Act: -0.0000

Data Index: 1186363 (C4 (Web Text))

Max Activating Token Index: 384

Text #17

Max Range: 1.5601. Min Range: -1.5601

Max Act: 1.1006. Min Act: -0.0000

Data Index: 1032474 (C4 (Web Text))

Max Activating Token Index: 334

Text #18

Max Range: 1.5601. Min Range: -1.5601

Max Act: 1.0007. Min Act: -0.0000

Data Index: 870453 (C4 (Web Text))

Max Activating Token Index: 222

Text #19

Max Range: 1.5601. Min Range: -1.5601

Max Act: 1.0663. Min Act: -0.0000

Data Index: 53214 (C4 (Web Text))

Max Activating Token Index: 520