Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3914 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.4185. Min Range: -1.4185

Max Act: 1.4185. Min Act: -0.0000

Data Index: 259202 (C4 (Web Text))

Max Activating Token Index: 382

Text #1

Max Range: 1.4185. Min Range: -1.4185

Max Act: 1.3221. Min Act: -0.0000

Data Index: 523528 (C4 (Web Text))

Max Activating Token Index: 636

Text #2

Max Range: 1.4185. Min Range: -1.4185

Max Act: 1.2442. Min Act: -0.0000

Data Index: 1342552 (C4 (Web Text))

Max Activating Token Index: 2

Text #3

Max Range: 1.4185. Min Range: -1.4185

Max Act: 1.2539. Min Act: -0.0000

Data Index: 141155 (C4 (Web Text))

Max Activating Token Index: 681

Text #4

Max Range: 1.4185. Min Range: -1.4185

Max Act: 1.1971. Min Act: -0.0000

Data Index: 1263844 (C4 (Web Text))

Max Activating Token Index: 574

Text #5

Max Range: 1.4185. Min Range: -1.4185

Max Act: 1.0874. Min Act: -0.0000

Data Index: 696585 (C4 (Web Text))

Max Activating Token Index: 289

Text #6

Max Range: 1.4185. Min Range: -1.4185

Max Act: 1.2153. Min Act: -0.0000

Data Index: 1326002 (C4 (Web Text))

Max Activating Token Index: 615

Text #7

Max Range: 1.4185. Min Range: -1.4185

Max Act: 1.1473. Min Act: -0.0000

Data Index: 1034820 (C4 (Web Text))

Max Activating Token Index: 2

Text #8

Max Range: 1.4185. Min Range: -1.4185

Max Act: 1.1746. Min Act: -0.0000

Data Index: 216274 (C4 (Web Text))

Max Activating Token Index: 230

Text #9

Max Range: 1.4185. Min Range: -1.4185

Max Act: 1.1062. Min Act: -0.0000

Data Index: 1338414 (C4 (Web Text))

Max Activating Token Index: 7

Text #10

Max Range: 1.4185. Min Range: -1.4185

Max Act: 1.1109. Min Act: -0.0000

Data Index: 563787 (C4 (Web Text))

Max Activating Token Index: 409

Text #11

Max Range: 1.4185. Min Range: -1.4185

Max Act: 1.1347. Min Act: -0.0000

Data Index: 483596 (C4 (Web Text))

Max Activating Token Index: 939

Text #12

Max Range: 1.4185. Min Range: -1.4185

Max Act: 1.1908. Min Act: -0.0000

Data Index: 782649 (C4 (Web Text))

Max Activating Token Index: 879

Text #13

Max Range: 1.4185. Min Range: -1.4185

Max Act: 1.1334. Min Act: -0.0000

Data Index: 1184760 (C4 (Web Text))

Max Activating Token Index: 97

Text #14

Max Range: 1.4185. Min Range: -1.4185

Max Act: 1.0142. Min Act: -0.0000

Data Index: 755644 (C4 (Web Text))

Max Activating Token Index: 278

Text #15

Max Range: 1.4185. Min Range: -1.4185

Max Act: 1.1268. Min Act: -0.0000

Data Index: 865429 (C4 (Web Text))

Max Activating Token Index: 891

Text #16

Max Range: 1.4185. Min Range: -1.4185

Max Act: 1.0831. Min Act: -0.0000

Data Index: 576748 (C4 (Web Text))

Max Activating Token Index: 37

Text #17

Max Range: 1.4185. Min Range: -1.4185

Max Act: 0.9726. Min Act: -0.0000

Data Index: 804510 (C4 (Web Text))

Max Activating Token Index: 897

Text #18

Max Range: 1.4185. Min Range: -1.4185

Max Act: 1.1483. Min Act: -0.0000

Data Index: 613598 (C4 (Web Text))

Max Activating Token Index: 190

Text #19

Max Range: 1.4185. Min Range: -1.4185

Max Act: 0.9852. Min Act: -0.0000

Data Index: 49328 (C4 (Web Text))

Max Activating Token Index: 987