Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 53 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.6815. Min Range: -0.6815

Max Act: 0.6750. Min Act: -0.0000

Data Index: 1074485 (C4 (Web Text))

Max Activating Token Index: 215

Text #1

Max Range: 0.6815. Min Range: -0.6815

Max Act: 0.6790. Min Act: -0.0000

Data Index: 1623488 (Python Code)

Max Activating Token Index: 953

Text #2

Max Range: 0.6815. Min Range: -0.6815

Max Act: 0.6294. Min Act: -0.0000

Data Index: 1467395 (Python Code)

Max Activating Token Index: 817

Text #3

Max Range: 0.6815. Min Range: -0.6815

Max Act: 0.6815. Min Act: -0.0000

Data Index: 1569374 (Python Code)

Max Activating Token Index: 860

Text #4

Max Range: 0.6815. Min Range: -0.6815

Max Act: 0.5979. Min Act: -0.0000

Data Index: 1578727 (Python Code)

Max Activating Token Index: 925

Text #5

Max Range: 0.6815. Min Range: -0.6815

Max Act: 0.6398. Min Act: -0.0000

Data Index: 1416543 (Python Code)

Max Activating Token Index: 934

Text #6

Max Range: 0.6815. Min Range: -0.6815

Max Act: 0.5946. Min Act: -0.0000

Data Index: 115731 (C4 (Web Text))

Max Activating Token Index: 621

Text #7

Max Range: 0.6815. Min Range: -0.6815

Max Act: 0.6240. Min Act: -0.0000

Data Index: 1479391 (Python Code)

Max Activating Token Index: 920

Text #8

Max Range: 0.6815. Min Range: -0.6815

Max Act: 0.5758. Min Act: -0.0000

Data Index: 1417841 (Python Code)

Max Activating Token Index: 158

Text #9

Max Range: 0.6815. Min Range: -0.6815

Max Act: 0.5461. Min Act: -0.0000

Data Index: 1572997 (Python Code)

Max Activating Token Index: 538

Text #10

Max Range: 0.6815. Min Range: -0.6815

Max Act: 0.5777. Min Act: -0.0000

Data Index: 554514 (C4 (Web Text))

Max Activating Token Index: 185

Text #11

Max Range: 0.6815. Min Range: -0.6815

Max Act: 0.4763. Min Act: -0.0000

Data Index: 1459872 (Python Code)

Max Activating Token Index: 609

Text #12

Max Range: 0.6815. Min Range: -0.6815

Max Act: 0.5675. Min Act: -0.0000

Data Index: 1479586 (Python Code)

Max Activating Token Index: 775

Text #13

Max Range: 0.6815. Min Range: -0.6815

Max Act: 0.5342. Min Act: -0.0000

Data Index: 1609993 (Python Code)

Max Activating Token Index: 352

Text #14

Max Range: 0.6815. Min Range: -0.6815

Max Act: 0.5730. Min Act: -0.0000

Data Index: 1476604 (Python Code)

Max Activating Token Index: 994

Text #15

Max Range: 0.6815. Min Range: -0.6815

Max Act: 0.5394. Min Act: -0.0000

Data Index: 1415042 (Python Code)

Max Activating Token Index: 485

Text #16

Max Range: 0.6815. Min Range: -0.6815

Max Act: 0.5394. Min Act: -0.0000

Data Index: 1396872 (Python Code)

Max Activating Token Index: 437

Text #17

Max Range: 0.6815. Min Range: -0.6815

Max Act: 0.5228. Min Act: -0.0000

Data Index: 1401141 (Python Code)

Max Activating Token Index: 838

Text #18

Max Range: 0.6815. Min Range: -0.6815

Max Act: 0.5078. Min Act: -0.0000

Data Index: 520509 (C4 (Web Text))

Max Activating Token Index: 140

Text #19

Max Range: 0.6815. Min Range: -0.6815

Max Act: 0.5205. Min Act: -0.0000

Data Index: 1416305 (Python Code)

Max Activating Token Index: 284