Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2961 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 0.6001. Min Range: -0.6001

Max Act: 0.6001. Min Act: -0.0001

Data Index: 1437447 (Python Code)

Max Activating Token Index: 853

Text #1

Max Range: 0.6001. Min Range: -0.6001

Max Act: 0.5438. Min Act: -0.0001

Data Index: 1485196 (Python Code)

Max Activating Token Index: 872

Text #2

Max Range: 0.6001. Min Range: -0.6001

Max Act: 0.5284. Min Act: -0.0001

Data Index: 1460409 (Python Code)

Max Activating Token Index: 271

Text #3

Max Range: 0.6001. Min Range: -0.6001

Max Act: 0.5421. Min Act: -0.0000

Data Index: 1466993 (Python Code)

Max Activating Token Index: 330

Text #4

Max Range: 0.6001. Min Range: -0.6001

Max Act: 0.5183. Min Act: -0.0001

Data Index: 1414087 (Python Code)

Max Activating Token Index: 319

Text #5

Max Range: 0.6001. Min Range: -0.6001

Max Act: 0.5102. Min Act: -0.0001

Data Index: 1529426 (Python Code)

Max Activating Token Index: 163

Text #6

Max Range: 0.6001. Min Range: -0.6001

Max Act: 0.5028. Min Act: -0.0001

Data Index: 1593658 (Python Code)

Max Activating Token Index: 495

Text #7

Max Range: 0.6001. Min Range: -0.6001

Max Act: 0.4548. Min Act: -0.0000

Data Index: 1622391 (Python Code)

Max Activating Token Index: 589

Text #8

Max Range: 0.6001. Min Range: -0.6001

Max Act: 0.5087. Min Act: -0.0000

Data Index: 1459069 (Python Code)

Max Activating Token Index: 854

Text #9

Max Range: 0.6001. Min Range: -0.6001

Max Act: 0.4895. Min Act: -0.0000

Data Index: 1461654 (Python Code)

Max Activating Token Index: 357

Text #10

Max Range: 0.6001. Min Range: -0.6001

Max Act: 0.4653. Min Act: -0.0001

Data Index: 1512976 (Python Code)

Max Activating Token Index: 495

Text #11

Max Range: 0.6001. Min Range: -0.6001

Max Act: 0.4495. Min Act: -0.0000

Data Index: 1607286 (Python Code)

Max Activating Token Index: 307

Text #12

Max Range: 0.6001. Min Range: -0.6001

Max Act: 0.4895. Min Act: -0.0000

Data Index: 1610055 (Python Code)

Max Activating Token Index: 449

Text #13

Max Range: 0.6001. Min Range: -0.6001

Max Act: 0.4834. Min Act: -0.0001

Data Index: 1633331 (Python Code)

Max Activating Token Index: 561

Text #14

Max Range: 0.6001. Min Range: -0.6001

Max Act: 0.4332. Min Act: -0.0001

Data Index: 1359975 (Python Code)

Max Activating Token Index: 875

Text #15

Max Range: 0.6001. Min Range: -0.6001

Max Act: 0.4617. Min Act: -0.0001

Data Index: 1448781 (Python Code)

Max Activating Token Index: 269

Text #16

Max Range: 0.6001. Min Range: -0.6001

Max Act: 0.4252. Min Act: -0.0000

Data Index: 1493165 (Python Code)

Max Activating Token Index: 46

Text #17

Max Range: 0.6001. Min Range: -0.6001

Max Act: 0.4436. Min Act: -0.0001

Data Index: 1562914 (Python Code)

Max Activating Token Index: 989

Text #18

Max Range: 0.6001. Min Range: -0.6001

Max Act: 0.4273. Min Act: -0.0000

Data Index: 1470120 (Python Code)

Max Activating Token Index: 132

Text #19

Max Range: 0.6001. Min Range: -0.6001

Max Act: 0.4362. Min Act: -0.0000

Data Index: 1578300 (Python Code)

Max Activating Token Index: 800