Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4140 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.4667. Min Range: -1.4667

Max Act: 1.4667. Min Act: -0.0002

Data Index: 479016 (C4 (Web Text))

Max Activating Token Index: 27

Text #1

Max Range: 1.4667. Min Range: -1.4667

Max Act: 1.3416. Min Act: -0.0002

Data Index: 319827 (C4 (Web Text))

Max Activating Token Index: 272

Text #2

Max Range: 1.4667. Min Range: -1.4667

Max Act: 1.1298. Min Act: -0.0002

Data Index: 522886 (C4 (Web Text))

Max Activating Token Index: 16

Text #3

Max Range: 1.4667. Min Range: -1.4667

Max Act: 1.0657. Min Act: -0.0002

Data Index: 968492 (C4 (Web Text))

Max Activating Token Index: 334

Text #4

Max Range: 1.4667. Min Range: -1.4667

Max Act: 1.0013. Min Act: -0.0002

Data Index: 612749 (C4 (Web Text))

Max Activating Token Index: 59

Text #5

Max Range: 1.4667. Min Range: -1.4667

Max Act: 0.9738. Min Act: -0.0002

Data Index: 875466 (C4 (Web Text))

Max Activating Token Index: 290

Text #6

Max Range: 1.4667. Min Range: -1.4667

Max Act: 0.9491. Min Act: -0.0002

Data Index: 871236 (C4 (Web Text))

Max Activating Token Index: 137

Text #7

Max Range: 1.4667. Min Range: -1.4667

Max Act: 1.0947. Min Act: -0.0003

Data Index: 390334 (C4 (Web Text))

Max Activating Token Index: 766

Text #8

Max Range: 1.4667. Min Range: -1.4667

Max Act: 1.0580. Min Act: -0.0002

Data Index: 366626 (C4 (Web Text))

Max Activating Token Index: 100

Text #9

Max Range: 1.4667. Min Range: -1.4667

Max Act: 1.0222. Min Act: -0.0002

Data Index: 418059 (C4 (Web Text))

Max Activating Token Index: 927

Text #10

Max Range: 1.4667. Min Range: -1.4667

Max Act: 1.0194. Min Act: -0.0002

Data Index: 1348726 (C4 (Web Text))

Max Activating Token Index: 194

Text #11

Max Range: 1.4667. Min Range: -1.4667

Max Act: 1.1471. Min Act: -0.0002

Data Index: 1123502 (C4 (Web Text))

Max Activating Token Index: 25

Text #12

Max Range: 1.4667. Min Range: -1.4667

Max Act: 1.0755. Min Act: -0.0002

Data Index: 1102703 (C4 (Web Text))

Max Activating Token Index: 220

Text #13

Max Range: 1.4667. Min Range: -1.4667

Max Act: 0.9796. Min Act: -0.0002

Data Index: 1294656 (C4 (Web Text))

Max Activating Token Index: 160

Text #14

Max Range: 1.4667. Min Range: -1.4667

Max Act: 0.9931. Min Act: -0.0002

Data Index: 327367 (C4 (Web Text))

Max Activating Token Index: 15

Text #15

Max Range: 1.4667. Min Range: -1.4667

Max Act: 1.0100. Min Act: -0.0002

Data Index: 1142682 (C4 (Web Text))

Max Activating Token Index: 578

Text #16

Max Range: 1.4667. Min Range: -1.4667

Max Act: 0.8723. Min Act: -0.0002

Data Index: 444211 (C4 (Web Text))

Max Activating Token Index: 241

Text #17

Max Range: 1.4667. Min Range: -1.4667

Max Act: 0.9483. Min Act: -0.0002

Data Index: 1353350 (C4 (Web Text))

Max Activating Token Index: 376

Text #18

Max Range: 1.4667. Min Range: -1.4667

Max Act: 0.9796. Min Act: -0.0002

Data Index: 358741 (C4 (Web Text))

Max Activating Token Index: 168

Text #19

Max Range: 1.4667. Min Range: -1.4667

Max Act: 0.9760. Min Act: -0.0002

Data Index: 798756 (C4 (Web Text))

Max Activating Token Index: 16