Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3146 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.5066. Min Range: -2.5066

Max Act: 2.5066. Min Act: -0.0000

Data Index: 515885 (C4 (Web Text))

Max Activating Token Index: 276

Text #1

Max Range: 2.5066. Min Range: -2.5066

Max Act: 2.0597. Min Act: -0.0000

Data Index: 122536 (C4 (Web Text))

Max Activating Token Index: 105

Text #2

Max Range: 2.5066. Min Range: -2.5066

Max Act: 1.8662. Min Act: -0.0000

Data Index: 1302577 (C4 (Web Text))

Max Activating Token Index: 80

Text #3

Max Range: 2.5066. Min Range: -2.5066

Max Act: 1.9206. Min Act: -0.0000

Data Index: 1133217 (C4 (Web Text))

Max Activating Token Index: 351

Text #4

Max Range: 2.5066. Min Range: -2.5066

Max Act: 1.7392. Min Act: -0.0000

Data Index: 856502 (C4 (Web Text))

Max Activating Token Index: 227

Text #5

Max Range: 2.5066. Min Range: -2.5066

Max Act: 1.8900. Min Act: -0.0000

Data Index: 303340 (C4 (Web Text))

Max Activating Token Index: 973

Text #6

Max Range: 2.5066. Min Range: -2.5066

Max Act: 1.6579. Min Act: -0.0000

Data Index: 220001 (C4 (Web Text))

Max Activating Token Index: 258

Text #7

Max Range: 2.5066. Min Range: -2.5066

Max Act: 1.7474. Min Act: -0.0000

Data Index: 415372 (C4 (Web Text))

Max Activating Token Index: 228

Text #8

Max Range: 2.5066. Min Range: -2.5066

Max Act: 1.7024. Min Act: -0.0000

Data Index: 1166203 (C4 (Web Text))

Max Activating Token Index: 700

Text #9

Max Range: 2.5066. Min Range: -2.5066

Max Act: 1.5169. Min Act: -0.0000

Data Index: 189673 (C4 (Web Text))

Max Activating Token Index: 242

Text #10

Max Range: 2.5066. Min Range: -2.5066

Max Act: 1.4969. Min Act: -0.0000

Data Index: 661988 (C4 (Web Text))

Max Activating Token Index: 515

Text #11

Max Range: 2.5066. Min Range: -2.5066

Max Act: 1.6430. Min Act: -0.0000

Data Index: 1335939 (C4 (Web Text))

Max Activating Token Index: 491

Text #12

Max Range: 2.5066. Min Range: -2.5066

Max Act: 1.5575. Min Act: -0.0000

Data Index: 1190614 (C4 (Web Text))

Max Activating Token Index: 606

Text #13

Max Range: 2.5066. Min Range: -2.5066

Max Act: 1.6324. Min Act: -0.0000

Data Index: 791421 (C4 (Web Text))

Max Activating Token Index: 157

Text #14

Max Range: 2.5066. Min Range: -2.5066

Max Act: 1.6285. Min Act: -0.0000

Data Index: 1341023 (C4 (Web Text))

Max Activating Token Index: 563

Text #15

Max Range: 2.5066. Min Range: -2.5066

Max Act: 1.4544. Min Act: -0.0000

Data Index: 1348947 (C4 (Web Text))

Max Activating Token Index: 309

Text #16

Max Range: 2.5066. Min Range: -2.5066

Max Act: 1.6984. Min Act: -0.0000

Data Index: 796416 (C4 (Web Text))

Max Activating Token Index: 821

Text #17

Max Range: 2.5066. Min Range: -2.5066

Max Act: 1.5029. Min Act: -0.0000

Data Index: 474418 (C4 (Web Text))

Max Activating Token Index: 233

Text #18

Max Range: 2.5066. Min Range: -2.5066

Max Act: 1.4773. Min Act: -0.0000

Data Index: 515961 (C4 (Web Text))

Max Activating Token Index: 12

Text #19

Max Range: 2.5066. Min Range: -2.5066

Max Act: 1.4628. Min Act: -0.0000

Data Index: 80196 (C4 (Web Text))

Max Activating Token Index: 607