Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4079 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.7191. Min Range: -0.7191

Max Act: 0.7191. Min Act: -0.0000

Data Index: 1590689 (Python Code)

Max Activating Token Index: 72

Text #1

Max Range: 0.7191. Min Range: -0.7191

Max Act: 0.6708. Min Act: -0.0000

Data Index: 1652410 (Python Code)

Max Activating Token Index: 770

Text #2

Max Range: 0.7191. Min Range: -0.7191

Max Act: 0.6266. Min Act: -0.0000

Data Index: 1499249 (Python Code)

Max Activating Token Index: 423

Text #3

Max Range: 0.7191. Min Range: -0.7191

Max Act: 0.6063. Min Act: -0.0000

Data Index: 1546249 (Python Code)

Max Activating Token Index: 86

Text #4

Max Range: 0.7191. Min Range: -0.7191

Max Act: 0.6130. Min Act: -0.0000

Data Index: 1647626 (Python Code)

Max Activating Token Index: 440

Text #5

Max Range: 0.7191. Min Range: -0.7191

Max Act: 0.5402. Min Act: -0.0000

Data Index: 1407381 (Python Code)

Max Activating Token Index: 271

Text #6

Max Range: 0.7191. Min Range: -0.7191

Max Act: 0.6382. Min Act: -0.0000

Data Index: 1512962 (Python Code)

Max Activating Token Index: 447

Text #7

Max Range: 0.7191. Min Range: -0.7191

Max Act: 0.4815. Min Act: -0.0000

Data Index: 1459471 (Python Code)

Max Activating Token Index: 633

Text #8

Max Range: 0.7191. Min Range: -0.7191

Max Act: 0.6418. Min Act: -0.0000

Data Index: 1363443 (Python Code)

Max Activating Token Index: 298

Text #9

Max Range: 0.7191. Min Range: -0.7191

Max Act: 0.5483. Min Act: -0.0000

Data Index: 1610786 (Python Code)

Max Activating Token Index: 583

Text #10

Max Range: 0.7191. Min Range: -0.7191

Max Act: 0.5640. Min Act: -0.0000

Data Index: 1425650 (Python Code)

Max Activating Token Index: 118

Text #11

Max Range: 0.7191. Min Range: -0.7191

Max Act: 0.5239. Min Act: -0.0000

Data Index: 1581549 (Python Code)

Max Activating Token Index: 834

Text #12

Max Range: 0.7191. Min Range: -0.7191

Max Act: 0.5269. Min Act: -0.0000

Data Index: 1513022 (Python Code)

Max Activating Token Index: 134

Text #13

Max Range: 0.7191. Min Range: -0.7191

Max Act: 0.5546. Min Act: -0.0000

Data Index: 1614892 (Python Code)

Max Activating Token Index: 545

Text #14

Max Range: 0.7191. Min Range: -0.7191

Max Act: 0.5247. Min Act: -0.0000

Data Index: 1484503 (Python Code)

Max Activating Token Index: 759

Text #15

Max Range: 0.7191. Min Range: -0.7191

Max Act: 0.5508. Min Act: -0.0000

Data Index: 1437585 (Python Code)

Max Activating Token Index: 663

Text #16

Max Range: 0.7191. Min Range: -0.7191

Max Act: 0.5119. Min Act: -0.0000

Data Index: 1511208 (Python Code)

Max Activating Token Index: 508

Text #17

Max Range: 0.7191. Min Range: -0.7191

Max Act: 0.5113. Min Act: -0.0000

Data Index: 1378904 (Python Code)

Max Activating Token Index: 381

Text #18

Max Range: 0.7191. Min Range: -0.7191

Max Act: 0.5040. Min Act: -0.0000

Data Index: 1620046 (Python Code)

Max Activating Token Index: 476

Text #19

Max Range: 0.7191. Min Range: -0.7191

Max Act: 0.4677. Min Act: -0.0000

Data Index: 1493738 (Python Code)

Max Activating Token Index: 919