Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1248 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.2562. Min Range: -0.2562

Max Act: 0.2562. Min Act: -0.0002

Data Index: 1433437 (Python Code)

Max Activating Token Index: 8

Text #1

Max Range: 0.2562. Min Range: -0.2562

Max Act: 0.2091. Min Act: -0.0000

Data Index: 1472370 (Python Code)

Max Activating Token Index: 10

Text #2

Max Range: 0.2562. Min Range: -0.2562

Max Act: 0.2162. Min Act: -0.0000

Data Index: 1557619 (Python Code)

Max Activating Token Index: 21

Text #3

Max Range: 0.2562. Min Range: -0.2562

Max Act: 0.2033. Min Act: -0.0001

Data Index: 1589675 (Python Code)

Max Activating Token Index: 424

Text #4

Max Range: 0.2562. Min Range: -0.2562

Max Act: 0.1973. Min Act: -0.0001

Data Index: 1539226 (Python Code)

Max Activating Token Index: 230

Text #5

Max Range: 0.2562. Min Range: -0.2562

Max Act: 0.1959. Min Act: -0.0001

Data Index: 1492469 (Python Code)

Max Activating Token Index: 37

Text #6

Max Range: 0.2562. Min Range: -0.2562

Max Act: 0.1956. Min Act: -0.0001

Data Index: 1648091 (Python Code)

Max Activating Token Index: 190

Text #7

Max Range: 0.2562. Min Range: -0.2562

Max Act: 0.1888. Min Act: -0.0002

Data Index: 1651042 (Python Code)

Max Activating Token Index: 30

Text #8

Max Range: 0.2562. Min Range: -0.2562

Max Act: 0.1914. Min Act: -0.0001

Data Index: 1444441 (Python Code)

Max Activating Token Index: 100

Text #9

Max Range: 0.2562. Min Range: -0.2562

Max Act: 0.1865. Min Act: -0.0001

Data Index: 1390497 (Python Code)

Max Activating Token Index: 34

Text #10

Max Range: 0.2562. Min Range: -0.2562

Max Act: 0.1851. Min Act: -0.0002

Data Index: 1458628 (Python Code)

Max Activating Token Index: 30

Text #11

Max Range: 0.2562. Min Range: -0.2562

Max Act: 0.1829. Min Act: -0.0001

Data Index: 1382978 (Python Code)

Max Activating Token Index: 269

Text #12

Max Range: 0.2562. Min Range: -0.2562

Max Act: 0.1817. Min Act: -0.0001

Data Index: 1496484 (Python Code)

Max Activating Token Index: 1023

Text #13

Max Range: 0.2562. Min Range: -0.2562

Max Act: 0.1872. Min Act: -0.0001

Data Index: 1644461 (Python Code)

Max Activating Token Index: 84

Text #14

Max Range: 0.2562. Min Range: -0.2562

Max Act: 0.1786. Min Act: -0.0001

Data Index: 1492348 (Python Code)

Max Activating Token Index: 158

Text #15

Max Range: 0.2562. Min Range: -0.2562

Max Act: 0.1753. Min Act: -0.0000

Data Index: 1477212 (Python Code)

Max Activating Token Index: 309

Text #16

Max Range: 0.2562. Min Range: -0.2562

Max Act: 0.1749. Min Act: -0.0001

Data Index: 1435414 (Python Code)

Max Activating Token Index: 96

Text #17

Max Range: 0.2562. Min Range: -0.2562

Max Act: 0.1678. Min Act: -0.0002

Data Index: 1506203 (Python Code)

Max Activating Token Index: 57

Text #18

Max Range: 0.2562. Min Range: -0.2562

Max Act: 0.1742. Min Act: -0.0001

Data Index: 1423153 (Python Code)

Max Activating Token Index: 33

Text #19

Max Range: 0.2562. Min Range: -0.2562

Max Act: 0.1722. Min Act: -0.0002

Data Index: 1444378 (Python Code)

Max Activating Token Index: 77