Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 508 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.2313. Min Range: -1.2313

Max Act: 1.2313. Min Act: -0.0000

Data Index: 375445 (C4 (Web Text))

Max Activating Token Index: 36

Text #1

Max Range: 1.2313. Min Range: -1.2313

Max Act: 1.1904. Min Act: -0.0000

Data Index: 851110 (C4 (Web Text))

Max Activating Token Index: 455

Text #2

Max Range: 1.2313. Min Range: -1.2313

Max Act: 0.9576. Min Act: -0.0000

Data Index: 7868 (C4 (Web Text))

Max Activating Token Index: 932

Text #3

Max Range: 1.2313. Min Range: -1.2313

Max Act: 0.8980. Min Act: -0.0000

Data Index: 1163943 (C4 (Web Text))

Max Activating Token Index: 135

Text #4

Max Range: 1.2313. Min Range: -1.2313

Max Act: 0.9453. Min Act: -0.0000

Data Index: 660480 (C4 (Web Text))

Max Activating Token Index: 505

Text #5

Max Range: 1.2313. Min Range: -1.2313

Max Act: 0.9448. Min Act: -0.0000

Data Index: 646864 (C4 (Web Text))

Max Activating Token Index: 420

Text #6

Max Range: 1.2313. Min Range: -1.2313

Max Act: 0.8773. Min Act: -0.0000

Data Index: 1639264 (Python Code)

Max Activating Token Index: 45

Text #7

Max Range: 1.2313. Min Range: -1.2313

Max Act: 0.8773. Min Act: -0.0000

Data Index: 1633814 (Python Code)

Max Activating Token Index: 45

Text #8

Max Range: 1.2313. Min Range: -1.2313

Max Act: 0.8773. Min Act: -0.0000

Data Index: 1628559 (Python Code)

Max Activating Token Index: 45

Text #9

Max Range: 1.2313. Min Range: -1.2313

Max Act: 0.8476. Min Act: -0.0000

Data Index: 1635923 (Python Code)

Max Activating Token Index: 45

Text #10

Max Range: 1.2313. Min Range: -1.2313

Max Act: 0.8175. Min Act: -0.0000

Data Index: 454610 (C4 (Web Text))

Max Activating Token Index: 366

Text #11

Max Range: 1.2313. Min Range: -1.2313

Max Act: 0.9344. Min Act: -0.0000

Data Index: 320604 (C4 (Web Text))

Max Activating Token Index: 705

Text #12

Max Range: 1.2313. Min Range: -1.2313

Max Act: 0.8151. Min Act: -0.0000

Data Index: 1070147 (C4 (Web Text))

Max Activating Token Index: 317

Text #13

Max Range: 1.2313. Min Range: -1.2313

Max Act: 0.7840. Min Act: -0.0000

Data Index: 634008 (C4 (Web Text))

Max Activating Token Index: 119

Text #14

Max Range: 1.2313. Min Range: -1.2313

Max Act: 0.8435. Min Act: -0.0000

Data Index: 60574 (C4 (Web Text))

Max Activating Token Index: 73

Text #15

Max Range: 1.2313. Min Range: -1.2313

Max Act: 0.8165. Min Act: -0.0000

Data Index: 162561 (C4 (Web Text))

Max Activating Token Index: 483

Text #16

Max Range: 1.2313. Min Range: -1.2313

Max Act: 0.7917. Min Act: -0.0000

Data Index: 1553116 (Python Code)

Max Activating Token Index: 44

Text #17

Max Range: 1.2313. Min Range: -1.2313

Max Act: 0.8127. Min Act: -0.0000

Data Index: 1295711 (C4 (Web Text))

Max Activating Token Index: 19

Text #18

Max Range: 1.2313. Min Range: -1.2313

Max Act: 0.7751. Min Act: -0.0000

Data Index: 1278368 (C4 (Web Text))

Max Activating Token Index: 1014

Text #19

Max Range: 1.2313. Min Range: -1.2313

Max Act: 0.7924. Min Act: -0.0000

Data Index: 950096 (C4 (Web Text))

Max Activating Token Index: 586