Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5270 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.6230. Min Range: -1.6230

Max Act: 1.5682. Min Act: -0.0000

Data Index: 1394053 (Python Code)

Max Activating Token Index: 379

Text #1

Max Range: 1.6230. Min Range: -1.6230

Max Act: 1.5498. Min Act: -0.0000

Data Index: 1571401 (Python Code)

Max Activating Token Index: 618

Text #2

Max Range: 1.6230. Min Range: -1.6230

Max Act: 1.6230. Min Act: -0.0000

Data Index: 1652548 (Python Code)

Max Activating Token Index: 570

Text #3

Max Range: 1.6230. Min Range: -1.6230

Max Act: 1.5381. Min Act: -0.0000

Data Index: 1491484 (Python Code)

Max Activating Token Index: 156

Text #4

Max Range: 1.6230. Min Range: -1.6230

Max Act: 1.5121. Min Act: -0.0000

Data Index: 1560680 (Python Code)

Max Activating Token Index: 377

Text #5

Max Range: 1.6230. Min Range: -1.6230

Max Act: 1.4164. Min Act: -0.0000

Data Index: 1396962 (Python Code)

Max Activating Token Index: 227

Text #6

Max Range: 1.6230. Min Range: -1.6230

Max Act: 1.2782. Min Act: -0.0000

Data Index: 1635364 (Python Code)

Max Activating Token Index: 342

Text #7

Max Range: 1.6230. Min Range: -1.6230

Max Act: 1.2953. Min Act: -0.0000

Data Index: 1395812 (Python Code)

Max Activating Token Index: 730

Text #8

Max Range: 1.6230. Min Range: -1.6230

Max Act: 1.2408. Min Act: -0.0000

Data Index: 1541765 (Python Code)

Max Activating Token Index: 261

Text #9

Max Range: 1.6230. Min Range: -1.6230

Max Act: 1.2404. Min Act: -0.0000

Data Index: 1556708 (Python Code)

Max Activating Token Index: 963

Text #10

Max Range: 1.6230. Min Range: -1.6230

Max Act: 1.2281. Min Act: -0.0000

Data Index: 1363340 (Python Code)

Max Activating Token Index: 583

Text #11

Max Range: 1.6230. Min Range: -1.6230

Max Act: 1.2117. Min Act: -0.0000

Data Index: 1439859 (Python Code)

Max Activating Token Index: 135

Text #12

Max Range: 1.6230. Min Range: -1.6230

Max Act: 1.2050. Min Act: -0.0000

Data Index: 1617291 (Python Code)

Max Activating Token Index: 332

Text #13

Max Range: 1.6230. Min Range: -1.6230

Max Act: 1.1216. Min Act: -0.0000

Data Index: 1573570 (Python Code)

Max Activating Token Index: 128

Text #14

Max Range: 1.6230. Min Range: -1.6230

Max Act: 1.2672. Min Act: -0.0000

Data Index: 1547469 (Python Code)

Max Activating Token Index: 572

Text #15

Max Range: 1.6230. Min Range: -1.6230

Max Act: 1.1846. Min Act: -0.0000

Data Index: 1638106 (Python Code)

Max Activating Token Index: 888

Text #16

Max Range: 1.6230. Min Range: -1.6230

Max Act: 1.1719. Min Act: -0.0000

Data Index: 1614109 (Python Code)

Max Activating Token Index: 728

Text #17

Max Range: 1.6230. Min Range: -1.6230

Max Act: 1.0930. Min Act: -0.0000

Data Index: 1595297 (Python Code)

Max Activating Token Index: 286

Text #18

Max Range: 1.6230. Min Range: -1.6230

Max Act: 1.1567. Min Act: -0.0000

Data Index: 1490745 (Python Code)

Max Activating Token Index: 222

Text #19

Max Range: 1.6230. Min Range: -1.6230

Max Act: 1.1547. Min Act: -0.0000

Data Index: 1572917 (Python Code)

Max Activating Token Index: 553