Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1928 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 1.2454. Min Range: -1.2454

Max Act: 1.2454. Min Act: -0.0000

Data Index: 1611131 (Python Code)

Max Activating Token Index: 778

Text #1

Max Range: 1.2454. Min Range: -1.2454

Max Act: 1.1977. Min Act: -0.0001

Data Index: 1492471 (Python Code)

Max Activating Token Index: 959

Text #2

Max Range: 1.2454. Min Range: -1.2454

Max Act: 0.8749. Min Act: -0.0001

Data Index: 1468743 (Python Code)

Max Activating Token Index: 791

Text #3

Max Range: 1.2454. Min Range: -1.2454

Max Act: 1.0022. Min Act: -0.0000

Data Index: 1601693 (Python Code)

Max Activating Token Index: 927

Text #4

Max Range: 1.2454. Min Range: -1.2454

Max Act: 1.0323. Min Act: -0.0001

Data Index: 1614414 (Python Code)

Max Activating Token Index: 917

Text #5

Max Range: 1.2454. Min Range: -1.2454

Max Act: 0.8652. Min Act: -0.0000

Data Index: 1584919 (Python Code)

Max Activating Token Index: 330

Text #6

Max Range: 1.2454. Min Range: -1.2454

Max Act: 0.9201. Min Act: -0.0000

Data Index: 1547632 (Python Code)

Max Activating Token Index: 769

Text #7

Max Range: 1.2454. Min Range: -1.2454

Max Act: 1.0236. Min Act: -0.0000

Data Index: 1648555 (Python Code)

Max Activating Token Index: 898

Text #8

Max Range: 1.2454. Min Range: -1.2454

Max Act: 0.8669. Min Act: -0.0001

Data Index: 1605852 (Python Code)

Max Activating Token Index: 343

Text #9

Max Range: 1.2454. Min Range: -1.2454

Max Act: 0.8896. Min Act: -0.0001

Data Index: 1597742 (Python Code)

Max Activating Token Index: 717

Text #10

Max Range: 1.2454. Min Range: -1.2454

Max Act: 0.8102. Min Act: -0.0001

Data Index: 1579618 (Python Code)

Max Activating Token Index: 930

Text #11

Max Range: 1.2454. Min Range: -1.2454

Max Act: 0.7749. Min Act: -0.0000

Data Index: 1437681 (Python Code)

Max Activating Token Index: 760

Text #12

Max Range: 1.2454. Min Range: -1.2454

Max Act: 0.8005. Min Act: -0.0001

Data Index: 1513341 (Python Code)

Max Activating Token Index: 159

Text #13

Max Range: 1.2454. Min Range: -1.2454

Max Act: 0.7394. Min Act: -0.0001

Data Index: 1382503 (Python Code)

Max Activating Token Index: 523

Text #14

Max Range: 1.2454. Min Range: -1.2454

Max Act: 0.7218. Min Act: -0.0001

Data Index: 1367396 (Python Code)

Max Activating Token Index: 999

Text #15

Max Range: 1.2454. Min Range: -1.2454

Max Act: 0.7128. Min Act: -0.0001

Data Index: 1480605 (Python Code)

Max Activating Token Index: 472

Text #16

Max Range: 1.2454. Min Range: -1.2454

Max Act: 0.7489. Min Act: -0.0001

Data Index: 1456272 (Python Code)

Max Activating Token Index: 461

Text #17

Max Range: 1.2454. Min Range: -1.2454

Max Act: 0.6633. Min Act: -0.0001

Data Index: 1383126 (Python Code)

Max Activating Token Index: 556

Text #18

Max Range: 1.2454. Min Range: -1.2454

Max Act: 0.6500. Min Act: -0.0001

Data Index: 1491813 (Python Code)

Max Activating Token Index: 658

Text #19

Max Range: 1.2454. Min Range: -1.2454

Max Act: 0.7109. Min Act: -0.0001

Data Index: 1480928 (Python Code)

Max Activating Token Index: 297