Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 67 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 3.3131. Min Range: -3.3131

Max Act: 3.1912. Min Act: -0.0000

Data Index: 1579520 (Python Code)

Max Activating Token Index: 998

Text #1

Max Range: 3.3131. Min Range: -3.3131

Max Act: 3.3131. Min Act: -0.0000

Data Index: 412401 (C4 (Web Text))

Max Activating Token Index: 502

Text #2

Max Range: 3.3131. Min Range: -3.3131

Max Act: 3.0995. Min Act: -0.0000

Data Index: 1466996 (Python Code)

Max Activating Token Index: 921

Text #3

Max Range: 3.3131. Min Range: -3.3131

Max Act: 3.2355. Min Act: -0.0000

Data Index: 1614902 (Python Code)

Max Activating Token Index: 885

Text #4

Max Range: 3.3131. Min Range: -3.3131

Max Act: 3.0515. Min Act: -0.0000

Data Index: 681686 (C4 (Web Text))

Max Activating Token Index: 397

Text #5

Max Range: 3.3131. Min Range: -3.3131

Max Act: 3.2009. Min Act: -0.0000

Data Index: 1610677 (Python Code)

Max Activating Token Index: 811

Text #6

Max Range: 3.3131. Min Range: -3.3131

Max Act: 2.9563. Min Act: -0.0000

Data Index: 1626717 (Python Code)

Max Activating Token Index: 823

Text #7

Max Range: 3.3131. Min Range: -3.3131

Max Act: 2.8704. Min Act: -0.0000

Data Index: 1644871 (Python Code)

Max Activating Token Index: 441

Text #8

Max Range: 3.3131. Min Range: -3.3131

Max Act: 2.8395. Min Act: -0.0000

Data Index: 1574492 (Python Code)

Max Activating Token Index: 538

Text #9

Max Range: 3.3131. Min Range: -3.3131

Max Act: 2.7799. Min Act: -0.0000

Data Index: 1649867 (Python Code)

Max Activating Token Index: 645

Text #10

Max Range: 3.3131. Min Range: -3.3131

Max Act: 2.8049. Min Act: -0.0000

Data Index: 1417563 (Python Code)

Max Activating Token Index: 679

Text #11

Max Range: 3.3131. Min Range: -3.3131

Max Act: 2.7202. Min Act: -0.0000

Data Index: 1642462 (Python Code)

Max Activating Token Index: 505

Text #12

Max Range: 3.3131. Min Range: -3.3131

Max Act: 2.5512. Min Act: -0.0000

Data Index: 1465741 (Python Code)

Max Activating Token Index: 1016

Text #13

Max Range: 3.3131. Min Range: -3.3131

Max Act: 2.6321. Min Act: -0.0000

Data Index: 1564766 (Python Code)

Max Activating Token Index: 898

Text #14

Max Range: 3.3131. Min Range: -3.3131

Max Act: 2.3039. Min Act: -0.0000

Data Index: 1413526 (Python Code)

Max Activating Token Index: 549

Text #15

Max Range: 3.3131. Min Range: -3.3131

Max Act: 2.3060. Min Act: -0.0000

Data Index: 1603495 (Python Code)

Max Activating Token Index: 692

Text #16

Max Range: 3.3131. Min Range: -3.3131

Max Act: 2.1226. Min Act: -0.0000

Data Index: 1488242 (Python Code)

Max Activating Token Index: 246

Text #17

Max Range: 3.3131. Min Range: -3.3131

Max Act: 2.2889. Min Act: -0.0000

Data Index: 1586092 (Python Code)

Max Activating Token Index: 302

Text #18

Max Range: 3.3131. Min Range: -3.3131

Max Act: 2.1640. Min Act: -0.0000

Data Index: 1584356 (Python Code)

Max Activating Token Index: 531

Text #19

Max Range: 3.3131. Min Range: -3.3131

Max Act: 2.2356. Min Act: -0.0000

Data Index: 1619777 (Python Code)

Max Activating Token Index: 339