Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 750 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.6956. Min Range: -2.6956

Max Act: 2.6956. Min Act: -0.0000

Data Index: 508325 (C4 (Web Text))

Max Activating Token Index: 251

Text #1

Max Range: 2.6956. Min Range: -2.6956

Max Act: 2.2522. Min Act: -0.0000

Data Index: 732419 (C4 (Web Text))

Max Activating Token Index: 957

Text #2

Max Range: 2.6956. Min Range: -2.6956

Max Act: 1.9876. Min Act: -0.0000

Data Index: 717295 (C4 (Web Text))

Max Activating Token Index: 437

Text #3

Max Range: 2.6956. Min Range: -2.6956

Max Act: 1.9905. Min Act: -0.0000

Data Index: 1189858 (C4 (Web Text))

Max Activating Token Index: 137

Text #4

Max Range: 2.6956. Min Range: -2.6956

Max Act: 1.8361. Min Act: -0.0000

Data Index: 45919 (C4 (Web Text))

Max Activating Token Index: 874

Text #5

Max Range: 2.6956. Min Range: -2.6956

Max Act: 1.7433. Min Act: -0.0000

Data Index: 1223097 (C4 (Web Text))

Max Activating Token Index: 734

Text #6

Max Range: 2.6956. Min Range: -2.6956

Max Act: 1.5288. Min Act: -0.0000

Data Index: 1133121 (C4 (Web Text))

Max Activating Token Index: 674

Text #7

Max Range: 2.6956. Min Range: -2.6956

Max Act: 1.1976. Min Act: -0.0000

Data Index: 1162072 (C4 (Web Text))

Max Activating Token Index: 387

Text #8

Max Range: 2.6956. Min Range: -2.6956

Max Act: 1.0202. Min Act: -0.0000

Data Index: 558735 (C4 (Web Text))

Max Activating Token Index: 110

Text #9

Max Range: 2.6956. Min Range: -2.6956

Max Act: 1.0538. Min Act: -0.0000

Data Index: 242618 (C4 (Web Text))

Max Activating Token Index: 143

Text #10

Max Range: 2.6956. Min Range: -2.6956

Max Act: 1.0647. Min Act: -0.0000

Data Index: 1332166 (C4 (Web Text))

Max Activating Token Index: 143

Text #11

Max Range: 2.6956. Min Range: -2.6956

Max Act: 1.0526. Min Act: -0.0000

Data Index: 823063 (C4 (Web Text))

Max Activating Token Index: 113

Text #12

Max Range: 2.6956. Min Range: -2.6956

Max Act: 1.0464. Min Act: -0.0000

Data Index: 446002 (C4 (Web Text))

Max Activating Token Index: 896

Text #13

Max Range: 2.6956. Min Range: -2.6956

Max Act: 1.0745. Min Act: -0.0000

Data Index: 494380 (C4 (Web Text))

Max Activating Token Index: 1020

Text #14

Max Range: 2.6956. Min Range: -2.6956

Max Act: 1.0826. Min Act: -0.0000

Data Index: 1063472 (C4 (Web Text))

Max Activating Token Index: 682

Text #15

Max Range: 2.6956. Min Range: -2.6956

Max Act: 0.9109. Min Act: -0.0000

Data Index: 125702 (C4 (Web Text))

Max Activating Token Index: 357

Text #16

Max Range: 2.6956. Min Range: -2.6956

Max Act: 0.9740. Min Act: -0.0000

Data Index: 215670 (C4 (Web Text))

Max Activating Token Index: 565

Text #17

Max Range: 2.6956. Min Range: -2.6956

Max Act: 1.0611. Min Act: -0.0000

Data Index: 97409 (C4 (Web Text))

Max Activating Token Index: 90

Text #18

Max Range: 2.6956. Min Range: -2.6956

Max Act: 0.9349. Min Act: -0.0000

Data Index: 275131 (C4 (Web Text))

Max Activating Token Index: 926

Text #19

Max Range: 2.6956. Min Range: -2.6956

Max Act: 0.9358. Min Act: -0.0000

Data Index: 117616 (C4 (Web Text))

Max Activating Token Index: 130