Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1051 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.8685. Min Range: -0.8685

Max Act: 0.8534. Min Act: -0.0000

Data Index: 331868 (C4 (Web Text))

Max Activating Token Index: 439

Text #1

Max Range: 0.8685. Min Range: -0.8685

Max Act: 0.8659. Min Act: -0.0000

Data Index: 1033433 (C4 (Web Text))

Max Activating Token Index: 199

Text #2

Max Range: 0.8685. Min Range: -0.8685

Max Act: 0.8568. Min Act: -0.0000

Data Index: 403306 (C4 (Web Text))

Max Activating Token Index: 238

Text #3

Max Range: 0.8685. Min Range: -0.8685

Max Act: 0.8661. Min Act: -0.0000

Data Index: 268231 (C4 (Web Text))

Max Activating Token Index: 749

Text #4

Max Range: 0.8685. Min Range: -0.8685

Max Act: 0.8685. Min Act: -0.0000

Data Index: 797844 (C4 (Web Text))

Max Activating Token Index: 594

Text #5

Max Range: 0.8685. Min Range: -0.8685

Max Act: 0.8014. Min Act: -0.0000

Data Index: 459982 (C4 (Web Text))

Max Activating Token Index: 438

Text #6

Max Range: 0.8685. Min Range: -0.8685

Max Act: 0.7841. Min Act: -0.0000

Data Index: 1050611 (C4 (Web Text))

Max Activating Token Index: 95

Text #7

Max Range: 0.8685. Min Range: -0.8685

Max Act: 0.8127. Min Act: -0.0000

Data Index: 932002 (C4 (Web Text))

Max Activating Token Index: 244

Text #8

Max Range: 0.8685. Min Range: -0.8685

Max Act: 0.7955. Min Act: -0.0000

Data Index: 911575 (C4 (Web Text))

Max Activating Token Index: 232

Text #9

Max Range: 0.8685. Min Range: -0.8685

Max Act: 0.7343. Min Act: -0.0000

Data Index: 177282 (C4 (Web Text))

Max Activating Token Index: 228

Text #10

Max Range: 0.8685. Min Range: -0.8685

Max Act: 0.7066. Min Act: -0.0000

Data Index: 541988 (C4 (Web Text))

Max Activating Token Index: 273

Text #11

Max Range: 0.8685. Min Range: -0.8685

Max Act: 0.6568. Min Act: -0.0000

Data Index: 1123212 (C4 (Web Text))

Max Activating Token Index: 322

Text #12

Max Range: 0.8685. Min Range: -0.8685

Max Act: 0.6791. Min Act: -0.0000

Data Index: 477299 (C4 (Web Text))

Max Activating Token Index: 526

Text #13

Max Range: 0.8685. Min Range: -0.8685

Max Act: 0.6969. Min Act: -0.0000

Data Index: 1205011 (C4 (Web Text))

Max Activating Token Index: 232

Text #14

Max Range: 0.8685. Min Range: -0.8685

Max Act: 0.7111. Min Act: -0.0000

Data Index: 344697 (C4 (Web Text))

Max Activating Token Index: 646

Text #15

Max Range: 0.8685. Min Range: -0.8685

Max Act: 0.6583. Min Act: -0.0000

Data Index: 953692 (C4 (Web Text))

Max Activating Token Index: 152

Text #16

Max Range: 0.8685. Min Range: -0.8685

Max Act: 0.6794. Min Act: -0.0000

Data Index: 735219 (C4 (Web Text))

Max Activating Token Index: 854

Text #17

Max Range: 0.8685. Min Range: -0.8685

Max Act: 0.6554. Min Act: -0.0000

Data Index: 736613 (C4 (Web Text))

Max Activating Token Index: 476

Text #18

Max Range: 0.8685. Min Range: -0.8685

Max Act: 0.6978. Min Act: -0.0000

Data Index: 678927 (C4 (Web Text))

Max Activating Token Index: 122

Text #19

Max Range: 0.8685. Min Range: -0.8685

Max Act: 0.6562. Min Act: -0.0000

Data Index: 247515 (C4 (Web Text))

Max Activating Token Index: 930