Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5235 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.4637. Min Range: -0.4637

Max Act: 0.4532. Min Act: -0.0000

Data Index: 1567435 (Python Code)

Max Activating Token Index: 337

Text #1

Max Range: 0.4637. Min Range: -0.4637

Max Act: 0.4586. Min Act: -0.0000

Data Index: 1369142 (Python Code)

Max Activating Token Index: 640

Text #2

Max Range: 0.4637. Min Range: -0.4637

Max Act: 0.4637. Min Act: -0.0000

Data Index: 1556401 (Python Code)

Max Activating Token Index: 672

Text #3

Max Range: 0.4637. Min Range: -0.4637

Max Act: 0.4467. Min Act: -0.0000

Data Index: 1405460 (Python Code)

Max Activating Token Index: 119

Text #4

Max Range: 0.4637. Min Range: -0.4637

Max Act: 0.4037. Min Act: -0.0000

Data Index: 1402728 (Python Code)

Max Activating Token Index: 230

Text #5

Max Range: 0.4637. Min Range: -0.4637

Max Act: 0.4416. Min Act: -0.0000

Data Index: 1399951 (Python Code)

Max Activating Token Index: 639

Text #6

Max Range: 0.4637. Min Range: -0.4637

Max Act: 0.4452. Min Act: -0.0000

Data Index: 1512266 (Python Code)

Max Activating Token Index: 46

Text #7

Max Range: 0.4637. Min Range: -0.4637

Max Act: 0.4140. Min Act: -0.0000

Data Index: 1656858 (Python Code)

Max Activating Token Index: 96

Text #8

Max Range: 0.4637. Min Range: -0.4637

Max Act: 0.4159. Min Act: -0.0000

Data Index: 1523146 (Python Code)

Max Activating Token Index: 926

Text #9

Max Range: 0.4637. Min Range: -0.4637

Max Act: 0.3592. Min Act: -0.0000

Data Index: 1576055 (Python Code)

Max Activating Token Index: 290

Text #10

Max Range: 0.4637. Min Range: -0.4637

Max Act: 0.3799. Min Act: -0.0000

Data Index: 1591124 (Python Code)

Max Activating Token Index: 920

Text #11

Max Range: 0.4637. Min Range: -0.4637

Max Act: 0.3752. Min Act: -0.0000

Data Index: 1643806 (Python Code)

Max Activating Token Index: 335

Text #12

Max Range: 0.4637. Min Range: -0.4637

Max Act: 0.3865. Min Act: -0.0000

Data Index: 1481225 (Python Code)

Max Activating Token Index: 95

Text #13

Max Range: 0.4637. Min Range: -0.4637

Max Act: 0.3820. Min Act: -0.0000

Data Index: 1653432 (Python Code)

Max Activating Token Index: 312

Text #14

Max Range: 0.4637. Min Range: -0.4637

Max Act: 0.3651. Min Act: -0.0000

Data Index: 1405729 (Python Code)

Max Activating Token Index: 562

Text #15

Max Range: 0.4637. Min Range: -0.4637

Max Act: 0.3714. Min Act: -0.0000

Data Index: 1467327 (Python Code)

Max Activating Token Index: 699

Text #16

Max Range: 0.4637. Min Range: -0.4637

Max Act: 0.3675. Min Act: -0.0000

Data Index: 1566610 (Python Code)

Max Activating Token Index: 699

Text #17

Max Range: 0.4637. Min Range: -0.4637

Max Act: 0.3940. Min Act: -0.0000

Data Index: 1516560 (Python Code)

Max Activating Token Index: 675

Text #18

Max Range: 0.4637. Min Range: -0.4637

Max Act: 0.3824. Min Act: -0.0000

Data Index: 1478855 (Python Code)

Max Activating Token Index: 621

Text #19

Max Range: 0.4637. Min Range: -0.4637

Max Act: 0.3642. Min Act: -0.0000

Data Index: 1486865 (Python Code)

Max Activating Token Index: 628