Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3340 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.1919. Min Range: -0.1919

Max Act: 0.1919. Min Act: -0.0001

Data Index: 1492220 (Python Code)

Max Activating Token Index: 389

Text #1

Max Range: 0.1919. Min Range: -0.1919

Max Act: 0.1703. Min Act: -0.0001

Data Index: 1627741 (Python Code)

Max Activating Token Index: 477

Text #2

Max Range: 0.1919. Min Range: -0.1919

Max Act: 0.1548. Min Act: -0.0001

Data Index: 1492434 (Python Code)

Max Activating Token Index: 418

Text #3

Max Range: 0.1919. Min Range: -0.1919

Max Act: 0.1437. Min Act: -0.0001

Data Index: 1402835 (Python Code)

Max Activating Token Index: 533

Text #4

Max Range: 0.1919. Min Range: -0.1919

Max Act: 0.1198. Min Act: -0.0001

Data Index: 1442962 (Python Code)

Max Activating Token Index: 887

Text #5

Max Range: 0.1919. Min Range: -0.1919

Max Act: 0.1275. Min Act: -0.0001

Data Index: 1409201 (Python Code)

Max Activating Token Index: 361

Text #6

Max Range: 0.1919. Min Range: -0.1919

Max Act: 0.1290. Min Act: -0.0001

Data Index: 1383418 (Python Code)

Max Activating Token Index: 749

Text #7

Max Range: 0.1919. Min Range: -0.1919

Max Act: 0.1242. Min Act: -0.0001

Data Index: 1589797 (Python Code)

Max Activating Token Index: 161

Text #8

Max Range: 0.1919. Min Range: -0.1919

Max Act: 0.1396. Min Act: -0.0001

Data Index: 1381143 (Python Code)

Max Activating Token Index: 841

Text #9

Max Range: 0.1919. Min Range: -0.1919

Max Act: 0.1405. Min Act: -0.0001

Data Index: 1400928 (Python Code)

Max Activating Token Index: 898

Text #10

Max Range: 0.1919. Min Range: -0.1919

Max Act: 0.1399. Min Act: -0.0001

Data Index: 1429593 (Python Code)

Max Activating Token Index: 539

Text #11

Max Range: 0.1919. Min Range: -0.1919

Max Act: 0.1188. Min Act: -0.0001

Data Index: 1600897 (Python Code)

Max Activating Token Index: 293

Text #12

Max Range: 0.1919. Min Range: -0.1919

Max Act: 0.1429. Min Act: -0.0001

Data Index: 1361504 (Python Code)

Max Activating Token Index: 689

Text #13

Max Range: 0.1919. Min Range: -0.1919

Max Act: 0.1141. Min Act: -0.0001

Data Index: 998997 (C4 (Web Text))

Max Activating Token Index: 660

Text #14

Max Range: 0.1919. Min Range: -0.1919

Max Act: 0.1250. Min Act: -0.0001

Data Index: 1361470 (Python Code)

Max Activating Token Index: 726

Text #15

Max Range: 0.1919. Min Range: -0.1919

Max Act: 0.1146. Min Act: -0.0001

Data Index: 1454141 (Python Code)

Max Activating Token Index: 318

Text #16

Max Range: 0.1919. Min Range: -0.1919

Max Act: 0.1435. Min Act: -0.0001

Data Index: 1577010 (Python Code)

Max Activating Token Index: 306

Text #17

Max Range: 0.1919. Min Range: -0.1919

Max Act: 0.1198. Min Act: -0.0001

Data Index: 1436575 (Python Code)

Max Activating Token Index: 1021

Text #18

Max Range: 0.1919. Min Range: -0.1919

Max Act: 0.1160. Min Act: -0.0001

Data Index: 1434553 (Python Code)

Max Activating Token Index: 13

Text #19

Max Range: 0.1919. Min Range: -0.1919

Max Act: 0.1290. Min Act: -0.0001

Data Index: 1503812 (Python Code)

Max Activating Token Index: 723