Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1590 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 0.2033. Min Range: -0.2033

Max Act: 0.2033. Min Act: -0.0001

Data Index: 1626184 (Python Code)

Max Activating Token Index: 232

Text #1

Max Range: 0.2033. Min Range: -0.2033

Max Act: 0.1973. Min Act: -0.0001

Data Index: 1656843 (Python Code)

Max Activating Token Index: 235

Text #2

Max Range: 0.2033. Min Range: -0.2033

Max Act: 0.1941. Min Act: -0.0001

Data Index: 1380889 (Python Code)

Max Activating Token Index: 53

Text #3

Max Range: 0.2033. Min Range: -0.2033

Max Act: 0.1834. Min Act: -0.0001

Data Index: 1484992 (Python Code)

Max Activating Token Index: 38

Text #4

Max Range: 0.2033. Min Range: -0.2033

Max Act: 0.1956. Min Act: -0.0001

Data Index: 1369123 (Python Code)

Max Activating Token Index: 41

Text #5

Max Range: 0.2033. Min Range: -0.2033

Max Act: 0.1880. Min Act: -0.0001

Data Index: 1426180 (Python Code)

Max Activating Token Index: 775

Text #6

Max Range: 0.2033. Min Range: -0.2033

Max Act: 0.1735. Min Act: -0.0001

Data Index: 1590415 (Python Code)

Max Activating Token Index: 370

Text #7

Max Range: 0.2033. Min Range: -0.2033

Max Act: 0.1784. Min Act: -0.0000

Data Index: 1581840 (Python Code)

Max Activating Token Index: 841

Text #8

Max Range: 0.2033. Min Range: -0.2033

Max Act: 0.1714. Min Act: -0.0001

Data Index: 1560792 (Python Code)

Max Activating Token Index: 543

Text #9

Max Range: 0.2033. Min Range: -0.2033

Max Act: 0.1899. Min Act: -0.0001

Data Index: 1405201 (Python Code)

Max Activating Token Index: 668

Text #10

Max Range: 0.2033. Min Range: -0.2033

Max Act: 0.1759. Min Act: -0.0001

Data Index: 1651375 (Python Code)

Max Activating Token Index: 211

Text #11

Max Range: 0.2033. Min Range: -0.2033

Max Act: 0.1750. Min Act: -0.0001

Data Index: 1592882 (Python Code)

Max Activating Token Index: 63

Text #12

Max Range: 0.2033. Min Range: -0.2033

Max Act: 0.1816. Min Act: -0.0000

Data Index: 1607072 (Python Code)

Max Activating Token Index: 717

Text #13

Max Range: 0.2033. Min Range: -0.2033

Max Act: 0.1875. Min Act: -0.0001

Data Index: 1472771 (Python Code)

Max Activating Token Index: 349

Text #14

Max Range: 0.2033. Min Range: -0.2033

Max Act: 0.1680. Min Act: -0.0001

Data Index: 1604835 (Python Code)

Max Activating Token Index: 251

Text #15

Max Range: 0.2033. Min Range: -0.2033

Max Act: 0.1793. Min Act: 0.0000

Data Index: 1419930 (Python Code)

Max Activating Token Index: 780

Text #16

Max Range: 0.2033. Min Range: -0.2033

Max Act: 0.1595. Min Act: -0.0001

Data Index: 1622923 (Python Code)

Max Activating Token Index: 374

Text #17

Max Range: 0.2033. Min Range: -0.2033

Max Act: 0.1724. Min Act: -0.0000

Data Index: 1373845 (Python Code)

Max Activating Token Index: 52

Text #18

Max Range: 0.2033. Min Range: -0.2033

Max Act: 0.1855. Min Act: -0.0001

Data Index: 1625835 (Python Code)

Max Activating Token Index: 462

Text #19

Max Range: 0.2033. Min Range: -0.2033

Max Act: 0.1808. Min Act: -0.0001

Data Index: 1516276 (Python Code)

Max Activating Token Index: 377