Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1116 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.1403. Min Range: -1.1403

Max Act: 1.1403. Min Act: 0.0000

Data Index: 1100665 (C4 (Web Text))

Max Activating Token Index: 955

Text #1

Max Range: 1.1403. Min Range: -1.1403

Max Act: 0.8301. Min Act: -0.0000

Data Index: 983993 (C4 (Web Text))

Max Activating Token Index: 767

Text #2

Max Range: 1.1403. Min Range: -1.1403

Max Act: 0.8476. Min Act: -0.0000

Data Index: 616499 (C4 (Web Text))

Max Activating Token Index: 178

Text #3

Max Range: 1.1403. Min Range: -1.1403

Max Act: 0.8283. Min Act: -0.0000

Data Index: 175810 (C4 (Web Text))

Max Activating Token Index: 858

Text #4

Max Range: 1.1403. Min Range: -1.1403

Max Act: 0.8717. Min Act: -0.0000

Data Index: 695767 (C4 (Web Text))

Max Activating Token Index: 372

Text #5

Max Range: 1.1403. Min Range: -1.1403

Max Act: 0.8029. Min Act: -0.0000

Data Index: 975237 (C4 (Web Text))

Max Activating Token Index: 596

Text #6

Max Range: 1.1403. Min Range: -1.1403

Max Act: 0.8447. Min Act: -0.0000

Data Index: 1241367 (C4 (Web Text))

Max Activating Token Index: 227

Text #7

Max Range: 1.1403. Min Range: -1.1403

Max Act: 0.7675. Min Act: -0.0000

Data Index: 58606 (C4 (Web Text))

Max Activating Token Index: 864

Text #8

Max Range: 1.1403. Min Range: -1.1403

Max Act: 0.7642. Min Act: -0.0000

Data Index: 879912 (C4 (Web Text))

Max Activating Token Index: 957

Text #9

Max Range: 1.1403. Min Range: -1.1403

Max Act: 0.7860. Min Act: -0.0000

Data Index: 217817 (C4 (Web Text))

Max Activating Token Index: 879

Text #10

Max Range: 1.1403. Min Range: -1.1403

Max Act: 0.7478. Min Act: -0.0000

Data Index: 346300 (C4 (Web Text))

Max Activating Token Index: 546

Text #11

Max Range: 1.1403. Min Range: -1.1403

Max Act: 0.7398. Min Act: -0.0000

Data Index: 1159042 (C4 (Web Text))

Max Activating Token Index: 1005

Text #12

Max Range: 1.1403. Min Range: -1.1403

Max Act: 0.7342. Min Act: -0.0000

Data Index: 424708 (C4 (Web Text))

Max Activating Token Index: 888

Text #13

Max Range: 1.1403. Min Range: -1.1403

Max Act: 0.7337. Min Act: -0.0000

Data Index: 844692 (C4 (Web Text))

Max Activating Token Index: 777

Text #14

Max Range: 1.1403. Min Range: -1.1403

Max Act: 0.7346. Min Act: 0.0000

Data Index: 626460 (C4 (Web Text))

Max Activating Token Index: 612

Text #15

Max Range: 1.1403. Min Range: -1.1403

Max Act: 0.7299. Min Act: -0.0000

Data Index: 168505 (C4 (Web Text))

Max Activating Token Index: 618

Text #16

Max Range: 1.1403. Min Range: -1.1403

Max Act: 0.7332. Min Act: -0.0000

Data Index: 585593 (C4 (Web Text))

Max Activating Token Index: 422

Text #17

Max Range: 1.1403. Min Range: -1.1403

Max Act: 0.7229. Min Act: 0.0000

Data Index: 1040424 (C4 (Web Text))

Max Activating Token Index: 717

Text #18

Max Range: 1.1403. Min Range: -1.1403

Max Act: 0.6947. Min Act: -0.0000

Data Index: 1277429 (C4 (Web Text))

Max Activating Token Index: 418

Text #19

Max Range: 1.1403. Min Range: -1.1403

Max Act: 0.6586. Min Act: -0.0000

Data Index: 570446 (C4 (Web Text))

Max Activating Token Index: 33