Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1318 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.6566. Min Range: -1.6566

Max Act: 1.6566. Min Act: -0.0000

Data Index: 481481 (C4 (Web Text))

Max Activating Token Index: 105

Text #1

Max Range: 1.6566. Min Range: -1.6566

Max Act: 1.1843. Min Act: -0.0000

Data Index: 1412150 (Python Code)

Max Activating Token Index: 59

Text #2

Max Range: 1.6566. Min Range: -1.6566

Max Act: 1.0591. Min Act: -0.0000

Data Index: 9923 (C4 (Web Text))

Max Activating Token Index: 179

Text #3

Max Range: 1.6566. Min Range: -1.6566

Max Act: 1.1153. Min Act: -0.0000

Data Index: 308046 (C4 (Web Text))

Max Activating Token Index: 530

Text #4

Max Range: 1.6566. Min Range: -1.6566

Max Act: 1.0712. Min Act: -0.0000

Data Index: 1382904 (Python Code)

Max Activating Token Index: 296

Text #5

Max Range: 1.6566. Min Range: -1.6566

Max Act: 1.1076. Min Act: -0.0000

Data Index: 1481922 (Python Code)

Max Activating Token Index: 79

Text #6

Max Range: 1.6566. Min Range: -1.6566

Max Act: 0.7975. Min Act: -0.0000

Data Index: 1464246 (Python Code)

Max Activating Token Index: 727

Text #7

Max Range: 1.6566. Min Range: -1.6566

Max Act: 0.9708. Min Act: -0.0000

Data Index: 425649 (C4 (Web Text))

Max Activating Token Index: 110

Text #8

Max Range: 1.6566. Min Range: -1.6566

Max Act: 0.8925. Min Act: -0.0000

Data Index: 440177 (C4 (Web Text))

Max Activating Token Index: 820

Text #9

Max Range: 1.6566. Min Range: -1.6566

Max Act: 0.8563. Min Act: -0.0000

Data Index: 1538606 (Python Code)

Max Activating Token Index: 157

Text #10

Max Range: 1.6566. Min Range: -1.6566

Max Act: 0.9313. Min Act: -0.0000

Data Index: 1587899 (Python Code)

Max Activating Token Index: 881

Text #11

Max Range: 1.6566. Min Range: -1.6566

Max Act: 0.8044. Min Act: -0.0000

Data Index: 1582973 (Python Code)

Max Activating Token Index: 740

Text #12

Max Range: 1.6566. Min Range: -1.6566

Max Act: 0.8704. Min Act: -0.0000

Data Index: 249796 (C4 (Web Text))

Max Activating Token Index: 167

Text #13

Max Range: 1.6566. Min Range: -1.6566

Max Act: 0.8427. Min Act: -0.0000

Data Index: 607543 (C4 (Web Text))

Max Activating Token Index: 113

Text #14

Max Range: 1.6566. Min Range: -1.6566

Max Act: 0.8971. Min Act: -0.0000

Data Index: 1431621 (Python Code)

Max Activating Token Index: 461

Text #15

Max Range: 1.6566. Min Range: -1.6566

Max Act: 0.8949. Min Act: -0.0000

Data Index: 1477073 (Python Code)

Max Activating Token Index: 462

Text #16

Max Range: 1.6566. Min Range: -1.6566

Max Act: 0.8668. Min Act: -0.0000

Data Index: 1639731 (Python Code)

Max Activating Token Index: 808

Text #17

Max Range: 1.6566. Min Range: -1.6566

Max Act: 0.8228. Min Act: -0.0000

Data Index: 1288189 (C4 (Web Text))

Max Activating Token Index: 246

Text #18

Max Range: 1.6566. Min Range: -1.6566

Max Act: 0.8986. Min Act: -0.0000

Data Index: 1618453 (Python Code)

Max Activating Token Index: 976

Text #19

Max Range: 1.6566. Min Range: -1.6566

Max Act: 0.8531. Min Act: -0.0000

Data Index: 922715 (C4 (Web Text))

Max Activating Token Index: 980