Model: SoLU Model: 3 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1538 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-3l')

Text #0

Max Range: 0.1742. Min Range: -0.1742

Max Act: 0.1742. Min Act: -0.0003

Data Index: 1572892 (Python Code)

Max Activating Token Index: 370

Text #1

Max Range: 0.1742. Min Range: -0.1742

Max Act: 0.1217. Min Act: -0.0003

Data Index: 1367698 (Python Code)

Max Activating Token Index: 707

Text #2

Max Range: 0.1742. Min Range: -0.1742

Max Act: 0.1101. Min Act: -0.0003

Data Index: 1525315 (Python Code)

Max Activating Token Index: 814

Text #3

Max Range: 0.1742. Min Range: -0.1742

Max Act: 0.0990. Min Act: -0.0003

Data Index: 1634021 (Python Code)

Max Activating Token Index: 778

Text #4

Max Range: 0.1742. Min Range: -0.1742

Max Act: 0.1022. Min Act: -0.0003

Data Index: 1472732 (Python Code)

Max Activating Token Index: 734

Text #5

Max Range: 0.1742. Min Range: -0.1742

Max Act: 0.1021. Min Act: -0.0003

Data Index: 1560600 (Python Code)

Max Activating Token Index: 482

Text #6

Max Range: 0.1742. Min Range: -0.1742

Max Act: 0.0990. Min Act: -0.0003

Data Index: 1436152 (Python Code)

Max Activating Token Index: 984

Text #7

Max Range: 0.1742. Min Range: -0.1742

Max Act: 0.0968. Min Act: -0.0003

Data Index: 1447420 (Python Code)

Max Activating Token Index: 1004

Text #8

Max Range: 0.1742. Min Range: -0.1742

Max Act: 0.1005. Min Act: -0.0004

Data Index: 1401552 (Python Code)

Max Activating Token Index: 750

Text #9

Max Range: 0.1742. Min Range: -0.1742

Max Act: 0.0952. Min Act: -0.0003

Data Index: 1467157 (Python Code)

Max Activating Token Index: 1017

Text #10

Max Range: 0.1742. Min Range: -0.1742

Max Act: 0.0976. Min Act: -0.0003

Data Index: 1432906 (Python Code)

Max Activating Token Index: 495

Text #11

Max Range: 0.1742. Min Range: -0.1742

Max Act: 0.0993. Min Act: -0.0003

Data Index: 1500142 (Python Code)

Max Activating Token Index: 926

Text #12

Max Range: 0.1742. Min Range: -0.1742

Max Act: 0.0952. Min Act: -0.0002

Data Index: 1401312 (Python Code)

Max Activating Token Index: 250

Text #13

Max Range: 0.1742. Min Range: -0.1742

Max Act: 0.0986. Min Act: -0.0003

Data Index: 1409765 (Python Code)

Max Activating Token Index: 626

Text #14

Max Range: 0.1742. Min Range: -0.1742

Max Act: 0.0959. Min Act: -0.0003

Data Index: 1494049 (Python Code)

Max Activating Token Index: 45

Text #15

Max Range: 0.1742. Min Range: -0.1742

Max Act: 0.0912. Min Act: -0.0003

Data Index: 1595295 (Python Code)

Max Activating Token Index: 935

Text #16

Max Range: 0.1742. Min Range: -0.1742

Max Act: 0.0919. Min Act: -0.0003

Data Index: 1606959 (Python Code)

Max Activating Token Index: 60

Text #17

Max Range: 0.1742. Min Range: -0.1742

Max Act: 0.0909. Min Act: -0.0003

Data Index: 1452496 (Python Code)

Max Activating Token Index: 112

Text #18

Max Range: 0.1742. Min Range: -0.1742

Max Act: 0.0898. Min Act: -0.0003

Data Index: 1608326 (Python Code)

Max Activating Token Index: 964

Text #19

Max Range: 0.1742. Min Range: -0.1742

Max Act: 0.0928. Min Act: -0.0003

Data Index: 1495432 (Python Code)

Max Activating Token Index: 22