Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3035 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 0.2317. Min Range: -0.2317

Max Act: 0.2317. Min Act: -0.0001

Data Index: 1560267 (Python Code)

Max Activating Token Index: 291

Text #1

Max Range: 0.2317. Min Range: -0.2317

Max Act: 0.2114. Min Act: -0.0001

Data Index: 1468637 (Python Code)

Max Activating Token Index: 895

Text #2

Max Range: 0.2317. Min Range: -0.2317

Max Act: 0.2134. Min Act: -0.0001

Data Index: 1608007 (Python Code)

Max Activating Token Index: 547

Text #3

Max Range: 0.2317. Min Range: -0.2317

Max Act: 0.2123. Min Act: -0.0001

Data Index: 1647904 (Python Code)

Max Activating Token Index: 423

Text #4

Max Range: 0.2317. Min Range: -0.2317

Max Act: 0.2100. Min Act: -0.0001

Data Index: 1409034 (Python Code)

Max Activating Token Index: 44

Text #5

Max Range: 0.2317. Min Range: -0.2317

Max Act: 0.1937. Min Act: -0.0001

Data Index: 1586347 (Python Code)

Max Activating Token Index: 842

Text #6

Max Range: 0.2317. Min Range: -0.2317

Max Act: 0.1958. Min Act: -0.0001

Data Index: 1597027 (Python Code)

Max Activating Token Index: 740

Text #7

Max Range: 0.2317. Min Range: -0.2317

Max Act: 0.1952. Min Act: -0.0001

Data Index: 1505119 (Python Code)

Max Activating Token Index: 675

Text #8

Max Range: 0.2317. Min Range: -0.2317

Max Act: 0.1930. Min Act: -0.0001

Data Index: 1504428 (Python Code)

Max Activating Token Index: 822

Text #9

Max Range: 0.2317. Min Range: -0.2317

Max Act: 0.1815. Min Act: -0.0001

Data Index: 1399235 (Python Code)

Max Activating Token Index: 639

Text #10

Max Range: 0.2317. Min Range: -0.2317

Max Act: 0.1816. Min Act: -0.0001

Data Index: 1389850 (Python Code)

Max Activating Token Index: 879

Text #11

Max Range: 0.2317. Min Range: -0.2317

Max Act: 0.1936. Min Act: -0.0001

Data Index: 1543009 (Python Code)

Max Activating Token Index: 784

Text #12

Max Range: 0.2317. Min Range: -0.2317

Max Act: 0.1866. Min Act: -0.0001

Data Index: 1449863 (Python Code)

Max Activating Token Index: 977

Text #13

Max Range: 0.2317. Min Range: -0.2317

Max Act: 0.1792. Min Act: -0.0001

Data Index: 1512514 (Python Code)

Max Activating Token Index: 978

Text #14

Max Range: 0.2317. Min Range: -0.2317

Max Act: 0.1693. Min Act: -0.0001

Data Index: 1596571 (Python Code)

Max Activating Token Index: 508

Text #15

Max Range: 0.2317. Min Range: -0.2317

Max Act: 0.1813. Min Act: -0.0001

Data Index: 1536654 (Python Code)

Max Activating Token Index: 942

Text #16

Max Range: 0.2317. Min Range: -0.2317

Max Act: 0.1830. Min Act: -0.0001

Data Index: 1453198 (Python Code)

Max Activating Token Index: 928

Text #17

Max Range: 0.2317. Min Range: -0.2317

Max Act: 0.1803. Min Act: -0.0001

Data Index: 1521034 (Python Code)

Max Activating Token Index: 745

Text #18

Max Range: 0.2317. Min Range: -0.2317

Max Act: 0.1748. Min Act: -0.0001

Data Index: 1544453 (Python Code)

Max Activating Token Index: 232

Text #19

Max Range: 0.2317. Min Range: -0.2317

Max Act: 0.1491. Min Act: -0.0001

Data Index: 1631356 (Python Code)

Max Activating Token Index: 771