Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4097 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.1812. Min Range: -1.1812

Max Act: 0.8876. Min Act: -0.0000

Data Index: 884483 (C4 (Web Text))

Max Activating Token Index: 989

Text #1

Max Range: 1.1812. Min Range: -1.1812

Max Act: 0.9945. Min Act: -0.0000

Data Index: 299386 (C4 (Web Text))

Max Activating Token Index: 783

Text #2

Max Range: 1.1812. Min Range: -1.1812

Max Act: 1.1812. Min Act: -0.0000

Data Index: 1107655 (C4 (Web Text))

Max Activating Token Index: 371

Text #3

Max Range: 1.1812. Min Range: -1.1812

Max Act: 0.8354. Min Act: -0.0000

Data Index: 719815 (C4 (Web Text))

Max Activating Token Index: 173

Text #4

Max Range: 1.1812. Min Range: -1.1812

Max Act: 0.8075. Min Act: -0.0000

Data Index: 534271 (C4 (Web Text))

Max Activating Token Index: 1005

Text #5

Max Range: 1.1812. Min Range: -1.1812

Max Act: 0.8010. Min Act: -0.0000

Data Index: 86266 (C4 (Web Text))

Max Activating Token Index: 686

Text #6

Max Range: 1.1812. Min Range: -1.1812

Max Act: 0.7370. Min Act: -0.0000

Data Index: 372967 (C4 (Web Text))

Max Activating Token Index: 54

Text #7

Max Range: 1.1812. Min Range: -1.1812

Max Act: 0.7323. Min Act: -0.0000

Data Index: 901991 (C4 (Web Text))

Max Activating Token Index: 264

Text #8

Max Range: 1.1812. Min Range: -1.1812

Max Act: 0.7738. Min Act: -0.0000

Data Index: 1028137 (C4 (Web Text))

Max Activating Token Index: 991

Text #9

Max Range: 1.1812. Min Range: -1.1812

Max Act: 0.7455. Min Act: -0.0000

Data Index: 165038 (C4 (Web Text))

Max Activating Token Index: 216

Text #10

Max Range: 1.1812. Min Range: -1.1812

Max Act: 0.7579. Min Act: -0.0000

Data Index: 631649 (C4 (Web Text))

Max Activating Token Index: 393

Text #11

Max Range: 1.1812. Min Range: -1.1812

Max Act: 0.6883. Min Act: -0.0000

Data Index: 942179 (C4 (Web Text))

Max Activating Token Index: 128

Text #12

Max Range: 1.1812. Min Range: -1.1812

Max Act: 0.6321. Min Act: -0.0000

Data Index: 283638 (C4 (Web Text))

Max Activating Token Index: 99

Text #13

Max Range: 1.1812. Min Range: -1.1812

Max Act: 0.6687. Min Act: -0.0000

Data Index: 471969 (C4 (Web Text))

Max Activating Token Index: 458

Text #14

Max Range: 1.1812. Min Range: -1.1812

Max Act: 0.5848. Min Act: -0.0000

Data Index: 437532 (C4 (Web Text))

Max Activating Token Index: 630

Text #15

Max Range: 1.1812. Min Range: -1.1812

Max Act: 0.6204. Min Act: -0.0000

Data Index: 15177 (C4 (Web Text))

Max Activating Token Index: 248

Text #16

Max Range: 1.1812. Min Range: -1.1812

Max Act: 0.5934. Min Act: -0.0000

Data Index: 1193944 (C4 (Web Text))

Max Activating Token Index: 330

Text #17

Max Range: 1.1812. Min Range: -1.1812

Max Act: 0.6389. Min Act: -0.0000

Data Index: 413779 (C4 (Web Text))

Max Activating Token Index: 890

Text #18

Max Range: 1.1812. Min Range: -1.1812

Max Act: 0.5984. Min Act: -0.0000

Data Index: 824484 (C4 (Web Text))

Max Activating Token Index: 330

Text #19

Max Range: 1.1812. Min Range: -1.1812

Max Act: 0.6072. Min Act: -0.0000

Data Index: 167986 (C4 (Web Text))

Max Activating Token Index: 796