Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 887 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.6841. Min Range: -0.6841

Max Act: 0.6841. Min Act: -0.0000

Data Index: 540359 (C4 (Web Text))

Max Activating Token Index: 448

Text #1

Max Range: 0.6841. Min Range: -0.6841

Max Act: 0.6250. Min Act: -0.0000

Data Index: 1325900 (C4 (Web Text))

Max Activating Token Index: 649

Text #2

Max Range: 0.6841. Min Range: -0.6841

Max Act: 0.6124. Min Act: -0.0000

Data Index: 153211 (C4 (Web Text))

Max Activating Token Index: 81

Text #3

Max Range: 0.6841. Min Range: -0.6841

Max Act: 0.5993. Min Act: -0.0000

Data Index: 425847 (C4 (Web Text))

Max Activating Token Index: 484

Text #4

Max Range: 0.6841. Min Range: -0.6841

Max Act: 0.5897. Min Act: -0.0000

Data Index: 10607 (C4 (Web Text))

Max Activating Token Index: 678

Text #5

Max Range: 0.6841. Min Range: -0.6841

Max Act: 0.6359. Min Act: -0.0000

Data Index: 617622 (C4 (Web Text))

Max Activating Token Index: 797

Text #6

Max Range: 0.6841. Min Range: -0.6841

Max Act: 0.6260. Min Act: -0.0000

Data Index: 1242494 (C4 (Web Text))

Max Activating Token Index: 639

Text #7

Max Range: 0.6841. Min Range: -0.6841

Max Act: 0.5734. Min Act: -0.0000

Data Index: 1258353 (C4 (Web Text))

Max Activating Token Index: 646

Text #8

Max Range: 0.6841. Min Range: -0.6841

Max Act: 0.5551. Min Act: -0.0000

Data Index: 608568 (C4 (Web Text))

Max Activating Token Index: 92

Text #9

Max Range: 0.6841. Min Range: -0.6841

Max Act: 0.6013. Min Act: -0.0000

Data Index: 223599 (C4 (Web Text))

Max Activating Token Index: 537

Text #10

Max Range: 0.6841. Min Range: -0.6841

Max Act: 0.5903. Min Act: -0.0000

Data Index: 1350976 (C4 (Web Text))

Max Activating Token Index: 126

Text #11

Max Range: 0.6841. Min Range: -0.6841

Max Act: 0.5455. Min Act: -0.0000

Data Index: 1110791 (C4 (Web Text))

Max Activating Token Index: 312

Text #12

Max Range: 0.6841. Min Range: -0.6841

Max Act: 0.5445. Min Act: -0.0000

Data Index: 1312257 (C4 (Web Text))

Max Activating Token Index: 179

Text #13

Max Range: 0.6841. Min Range: -0.6841

Max Act: 0.5271. Min Act: -0.0000

Data Index: 184877 (C4 (Web Text))

Max Activating Token Index: 773

Text #14

Max Range: 0.6841. Min Range: -0.6841

Max Act: 0.5047. Min Act: -0.0000

Data Index: 1357457 (C4 (Web Text))

Max Activating Token Index: 162

Text #15

Max Range: 0.6841. Min Range: -0.6841

Max Act: 0.5008. Min Act: -0.0000

Data Index: 254972 (C4 (Web Text))

Max Activating Token Index: 614

Text #16

Max Range: 0.6841. Min Range: -0.6841

Max Act: 0.5414. Min Act: -0.0000

Data Index: 1154643 (C4 (Web Text))

Max Activating Token Index: 178

Text #17

Max Range: 0.6841. Min Range: -0.6841

Max Act: 0.5386. Min Act: -0.0000

Data Index: 1214370 (C4 (Web Text))

Max Activating Token Index: 766

Text #18

Max Range: 0.6841. Min Range: -0.6841

Max Act: 0.4841. Min Act: -0.0000

Data Index: 103382 (C4 (Web Text))

Max Activating Token Index: 929

Text #19

Max Range: 0.6841. Min Range: -0.6841

Max Act: 0.4602. Min Act: -0.0000

Data Index: 1011133 (C4 (Web Text))

Max Activating Token Index: 459