Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1128 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 0.6065. Min Range: -0.6065

Max Act: 0.5369. Min Act: -0.0001

Data Index: 1364926 (Python Code)

Max Activating Token Index: 23

Text #1

Max Range: 0.6065. Min Range: -0.6065

Max Act: 0.6065. Min Act: -0.0001

Data Index: 1561185 (Python Code)

Max Activating Token Index: 467

Text #2

Max Range: 0.6065. Min Range: -0.6065

Max Act: 0.5428. Min Act: -0.0001

Data Index: 1494571 (Python Code)

Max Activating Token Index: 415

Text #3

Max Range: 0.6065. Min Range: -0.6065

Max Act: 0.5078. Min Act: -0.0001

Data Index: 1422387 (Python Code)

Max Activating Token Index: 68

Text #4

Max Range: 0.6065. Min Range: -0.6065

Max Act: 0.5047. Min Act: -0.0001

Data Index: 1450462 (Python Code)

Max Activating Token Index: 787

Text #5

Max Range: 0.6065. Min Range: -0.6065

Max Act: 0.5246. Min Act: -0.0001

Data Index: 1604457 (Python Code)

Max Activating Token Index: 488

Text #6

Max Range: 0.6065. Min Range: -0.6065

Max Act: 0.5200. Min Act: -0.0001

Data Index: 1632595 (Python Code)

Max Activating Token Index: 372

Text #7

Max Range: 0.6065. Min Range: -0.6065

Max Act: 0.5187. Min Act: -0.0001

Data Index: 1649174 (Python Code)

Max Activating Token Index: 38

Text #8

Max Range: 0.6065. Min Range: -0.6065

Max Act: 0.5198. Min Act: -0.0001

Data Index: 1467452 (Python Code)

Max Activating Token Index: 512

Text #9

Max Range: 0.6065. Min Range: -0.6065

Max Act: 0.4653. Min Act: -0.0001

Data Index: 1564554 (Python Code)

Max Activating Token Index: 164

Text #10

Max Range: 0.6065. Min Range: -0.6065

Max Act: 0.4910. Min Act: -0.0001

Data Index: 1581165 (Python Code)

Max Activating Token Index: 504

Text #11

Max Range: 0.6065. Min Range: -0.6065

Max Act: 0.5067. Min Act: -0.0001

Data Index: 1628961 (Python Code)

Max Activating Token Index: 449

Text #12

Max Range: 0.6065. Min Range: -0.6065

Max Act: 0.4951. Min Act: -0.0001

Data Index: 1493995 (Python Code)

Max Activating Token Index: 636

Text #13

Max Range: 0.6065. Min Range: -0.6065

Max Act: 0.4816. Min Act: -0.0001

Data Index: 1566178 (Python Code)

Max Activating Token Index: 51

Text #14

Max Range: 0.6065. Min Range: -0.6065

Max Act: 0.5013. Min Act: -0.0001

Data Index: 1575719 (Python Code)

Max Activating Token Index: 83

Text #15

Max Range: 0.6065. Min Range: -0.6065

Max Act: 0.4958. Min Act: -0.0001

Data Index: 1468377 (Python Code)

Max Activating Token Index: 970

Text #16

Max Range: 0.6065. Min Range: -0.6065

Max Act: 0.4842. Min Act: -0.0001

Data Index: 1608715 (Python Code)

Max Activating Token Index: 327

Text #17

Max Range: 0.6065. Min Range: -0.6065

Max Act: 0.4962. Min Act: -0.0001

Data Index: 1380045 (Python Code)

Max Activating Token Index: 778

Text #18

Max Range: 0.6065. Min Range: -0.6065

Max Act: 0.4949. Min Act: -0.0001

Data Index: 1579894 (Python Code)

Max Activating Token Index: 816

Text #19

Max Range: 0.6065. Min Range: -0.6065

Max Act: 0.4923. Min Act: -0.0001

Data Index: 1495615 (Python Code)

Max Activating Token Index: 925