Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2888 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.6275. Min Range: -2.6275

Max Act: 2.6275. Min Act: -0.0000

Data Index: 1296744 (C4 (Web Text))

Max Activating Token Index: 366

Text #1

Max Range: 2.6275. Min Range: -2.6275

Max Act: 2.3072. Min Act: 0.0000

Data Index: 288903 (C4 (Web Text))

Max Activating Token Index: 381

Text #2

Max Range: 2.6275. Min Range: -2.6275

Max Act: 2.4099. Min Act: -0.0000

Data Index: 496920 (C4 (Web Text))

Max Activating Token Index: 412

Text #3

Max Range: 2.6275. Min Range: -2.6275

Max Act: 2.1728. Min Act: 0.0001

Data Index: 876175 (C4 (Web Text))

Max Activating Token Index: 242

Text #4

Max Range: 2.6275. Min Range: -2.6275

Max Act: 2.0009. Min Act: 0.0002

Data Index: 106440 (C4 (Web Text))

Max Activating Token Index: 643

Text #5

Max Range: 2.6275. Min Range: -2.6275

Max Act: 1.9946. Min Act: -0.0000

Data Index: 1089966 (C4 (Web Text))

Max Activating Token Index: 693

Text #6

Max Range: 2.6275. Min Range: -2.6275

Max Act: 1.9912. Min Act: -0.0000

Data Index: 1136880 (C4 (Web Text))

Max Activating Token Index: 919

Text #7

Max Range: 2.6275. Min Range: -2.6275

Max Act: 1.9815. Min Act: 0.0000

Data Index: 113289 (C4 (Web Text))

Max Activating Token Index: 868

Text #8

Max Range: 2.6275. Min Range: -2.6275

Max Act: 1.9499. Min Act: -0.0000

Data Index: 1089256 (C4 (Web Text))

Max Activating Token Index: 492

Text #9

Max Range: 2.6275. Min Range: -2.6275

Max Act: 1.9505. Min Act: -0.0000

Data Index: 539476 (C4 (Web Text))

Max Activating Token Index: 198

Text #10

Max Range: 2.6275. Min Range: -2.6275

Max Act: 1.9172. Min Act: -0.0000

Data Index: 87761 (C4 (Web Text))

Max Activating Token Index: 714

Text #11

Max Range: 2.6275. Min Range: -2.6275

Max Act: 1.9996. Min Act: 0.0000

Data Index: 1306600 (C4 (Web Text))

Max Activating Token Index: 341

Text #12

Max Range: 2.6275. Min Range: -2.6275

Max Act: 1.8846. Min Act: 0.0001

Data Index: 1139111 (C4 (Web Text))

Max Activating Token Index: 439

Text #13

Max Range: 2.6275. Min Range: -2.6275

Max Act: 1.8623. Min Act: 0.0000

Data Index: 958860 (C4 (Web Text))

Max Activating Token Index: 175

Text #14

Max Range: 2.6275. Min Range: -2.6275

Max Act: 1.8452. Min Act: 0.0000

Data Index: 736345 (C4 (Web Text))

Max Activating Token Index: 554

Text #15

Max Range: 2.6275. Min Range: -2.6275

Max Act: 1.8186. Min Act: -0.0000

Data Index: 116572 (C4 (Web Text))

Max Activating Token Index: 879

Text #16

Max Range: 2.6275. Min Range: -2.6275

Max Act: 1.7954. Min Act: -0.0000

Data Index: 1218503 (C4 (Web Text))

Max Activating Token Index: 298

Text #17

Max Range: 2.6275. Min Range: -2.6275

Max Act: 1.7700. Min Act: 0.0001

Data Index: 1283863 (C4 (Web Text))

Max Activating Token Index: 622

Text #18

Max Range: 2.6275. Min Range: -2.6275

Max Act: 1.6621. Min Act: -0.0000

Data Index: 864349 (C4 (Web Text))

Max Activating Token Index: 239

Text #19

Max Range: 2.6275. Min Range: -2.6275

Max Act: 1.8592. Min Act: 0.0000

Data Index: 948705 (C4 (Web Text))

Max Activating Token Index: 616