Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1543 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.6392. Min Range: -0.6392

Max Act: 0.6392. Min Act: -0.0000

Data Index: 1594668 (Python Code)

Max Activating Token Index: 147

Text #1

Max Range: 0.6392. Min Range: -0.6392

Max Act: 0.5638. Min Act: -0.0000

Data Index: 1563236 (Python Code)

Max Activating Token Index: 298

Text #2

Max Range: 0.6392. Min Range: -0.6392

Max Act: 0.5597. Min Act: -0.0000

Data Index: 1469331 (Python Code)

Max Activating Token Index: 96

Text #3

Max Range: 0.6392. Min Range: -0.6392

Max Act: 0.5588. Min Act: -0.0000

Data Index: 1441454 (Python Code)

Max Activating Token Index: 73

Text #4

Max Range: 0.6392. Min Range: -0.6392

Max Act: 0.4973. Min Act: -0.0000

Data Index: 1500466 (Python Code)

Max Activating Token Index: 262

Text #5

Max Range: 0.6392. Min Range: -0.6392

Max Act: 0.4871. Min Act: -0.0000

Data Index: 1393063 (Python Code)

Max Activating Token Index: 89

Text #6

Max Range: 0.6392. Min Range: -0.6392

Max Act: 0.4916. Min Act: -0.0000

Data Index: 1652513 (Python Code)

Max Activating Token Index: 115

Text #7

Max Range: 0.6392. Min Range: -0.6392

Max Act: 0.4862. Min Act: -0.0000

Data Index: 1379798 (Python Code)

Max Activating Token Index: 401

Text #8

Max Range: 0.6392. Min Range: -0.6392

Max Act: 0.4849. Min Act: -0.0000

Data Index: 1578740 (Python Code)

Max Activating Token Index: 200

Text #9

Max Range: 0.6392. Min Range: -0.6392

Max Act: 0.4896. Min Act: -0.0000

Data Index: 1601746 (Python Code)

Max Activating Token Index: 9

Text #10

Max Range: 0.6392. Min Range: -0.6392

Max Act: 0.4699. Min Act: -0.0000

Data Index: 1612668 (Python Code)

Max Activating Token Index: 430

Text #11

Max Range: 0.6392. Min Range: -0.6392

Max Act: 0.4705. Min Act: -0.0000

Data Index: 1529035 (Python Code)

Max Activating Token Index: 606

Text #12

Max Range: 0.6392. Min Range: -0.6392

Max Act: 0.4482. Min Act: -0.0000

Data Index: 1546581 (Python Code)

Max Activating Token Index: 17

Text #13

Max Range: 0.6392. Min Range: -0.6392

Max Act: 0.4471. Min Act: -0.0000

Data Index: 1532491 (Python Code)

Max Activating Token Index: 64

Text #14

Max Range: 0.6392. Min Range: -0.6392

Max Act: 0.4156. Min Act: -0.0000

Data Index: 1448046 (Python Code)

Max Activating Token Index: 593

Text #15

Max Range: 0.6392. Min Range: -0.6392

Max Act: 0.4387. Min Act: -0.0000

Data Index: 1614740 (Python Code)

Max Activating Token Index: 117

Text #16

Max Range: 0.6392. Min Range: -0.6392

Max Act: 0.4687. Min Act: -0.0000

Data Index: 1635561 (Python Code)

Max Activating Token Index: 214

Text #17

Max Range: 0.6392. Min Range: -0.6392

Max Act: 0.4340. Min Act: -0.0000

Data Index: 1487156 (Python Code)

Max Activating Token Index: 60

Text #18

Max Range: 0.6392. Min Range: -0.6392

Max Act: 0.4306. Min Act: -0.0000

Data Index: 1366529 (Python Code)

Max Activating Token Index: 165

Text #19

Max Range: 0.6392. Min Range: -0.6392

Max Act: 0.4496. Min Act: -0.0000

Data Index: 1624190 (Python Code)

Max Activating Token Index: 63