Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4524 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.8784. Min Range: -0.8784

Max Act: 0.8784. Min Act: -0.0000

Data Index: 47829 (C4 (Web Text))

Max Activating Token Index: 69

Text #1

Max Range: 0.8784. Min Range: -0.8784

Max Act: 0.7477. Min Act: -0.0000

Data Index: 95686 (C4 (Web Text))

Max Activating Token Index: 393

Text #2

Max Range: 0.8784. Min Range: -0.8784

Max Act: 0.7328. Min Act: -0.0000

Data Index: 136825 (C4 (Web Text))

Max Activating Token Index: 747

Text #3

Max Range: 0.8784. Min Range: -0.8784

Max Act: 0.6840. Min Act: -0.0000

Data Index: 367052 (C4 (Web Text))

Max Activating Token Index: 206

Text #4

Max Range: 0.8784. Min Range: -0.8784

Max Act: 0.7443. Min Act: -0.0000

Data Index: 452424 (C4 (Web Text))

Max Activating Token Index: 815

Text #5

Max Range: 0.8784. Min Range: -0.8784

Max Act: 0.6969. Min Act: -0.0000

Data Index: 93146 (C4 (Web Text))

Max Activating Token Index: 81

Text #6

Max Range: 0.8784. Min Range: -0.8784

Max Act: 0.6883. Min Act: -0.0000

Data Index: 539580 (C4 (Web Text))

Max Activating Token Index: 383

Text #7

Max Range: 0.8784. Min Range: -0.8784

Max Act: 0.7065. Min Act: -0.0000

Data Index: 1124842 (C4 (Web Text))

Max Activating Token Index: 691

Text #8

Max Range: 0.8784. Min Range: -0.8784

Max Act: 0.6796. Min Act: -0.0000

Data Index: 433305 (C4 (Web Text))

Max Activating Token Index: 468

Text #9

Max Range: 0.8784. Min Range: -0.8784

Max Act: 0.6959. Min Act: -0.0000

Data Index: 1366371 (Python Code)

Max Activating Token Index: 263

Text #10

Max Range: 0.8784. Min Range: -0.8784

Max Act: 0.6520. Min Act: -0.0000

Data Index: 1211743 (C4 (Web Text))

Max Activating Token Index: 603

Text #11

Max Range: 0.8784. Min Range: -0.8784

Max Act: 0.6713. Min Act: -0.0000

Data Index: 423344 (C4 (Web Text))

Max Activating Token Index: 681

Text #12

Max Range: 0.8784. Min Range: -0.8784

Max Act: 0.6895. Min Act: -0.0000

Data Index: 585528 (C4 (Web Text))

Max Activating Token Index: 95

Text #13

Max Range: 0.8784. Min Range: -0.8784

Max Act: 0.6603. Min Act: -0.0000

Data Index: 699141 (C4 (Web Text))

Max Activating Token Index: 259

Text #14

Max Range: 0.8784. Min Range: -0.8784

Max Act: 0.6193. Min Act: -0.0000

Data Index: 1288581 (C4 (Web Text))

Max Activating Token Index: 656

Text #15

Max Range: 0.8784. Min Range: -0.8784

Max Act: 0.6790. Min Act: -0.0000

Data Index: 862707 (C4 (Web Text))

Max Activating Token Index: 188

Text #16

Max Range: 0.8784. Min Range: -0.8784

Max Act: 0.6538. Min Act: -0.0000

Data Index: 419794 (C4 (Web Text))

Max Activating Token Index: 1016

Text #17

Max Range: 0.8784. Min Range: -0.8784

Max Act: 0.6257. Min Act: -0.0000

Data Index: 476949 (C4 (Web Text))

Max Activating Token Index: 825

Text #18

Max Range: 0.8784. Min Range: -0.8784

Max Act: 0.6598. Min Act: -0.0000

Data Index: 808631 (C4 (Web Text))

Max Activating Token Index: 271

Text #19

Max Range: 0.8784. Min Range: -0.8784

Max Act: 0.6106. Min Act: -0.0000

Data Index: 500216 (C4 (Web Text))

Max Activating Token Index: 109