Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 6072 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.8640. Min Range: -1.8640

Max Act: 1.8640. Min Act: -0.0000

Data Index: 272817 (C4 (Web Text))

Max Activating Token Index: 283

Text #1

Max Range: 1.8640. Min Range: -1.8640

Max Act: 1.7384. Min Act: -0.0000

Data Index: 1207854 (C4 (Web Text))

Max Activating Token Index: 518

Text #2

Max Range: 1.8640. Min Range: -1.8640

Max Act: 1.8589. Min Act: -0.0000

Data Index: 417628 (C4 (Web Text))

Max Activating Token Index: 780

Text #3

Max Range: 1.8640. Min Range: -1.8640

Max Act: 1.7156. Min Act: -0.0000

Data Index: 1083405 (C4 (Web Text))

Max Activating Token Index: 579

Text #4

Max Range: 1.8640. Min Range: -1.8640

Max Act: 1.5282. Min Act: -0.0000

Data Index: 582974 (C4 (Web Text))

Max Activating Token Index: 464

Text #5

Max Range: 1.8640. Min Range: -1.8640

Max Act: 1.3459. Min Act: -0.0000

Data Index: 1095266 (C4 (Web Text))

Max Activating Token Index: 716

Text #6

Max Range: 1.8640. Min Range: -1.8640

Max Act: 1.5223. Min Act: -0.0000

Data Index: 819017 (C4 (Web Text))

Max Activating Token Index: 294

Text #7

Max Range: 1.8640. Min Range: -1.8640

Max Act: 1.3200. Min Act: -0.0000

Data Index: 447047 (C4 (Web Text))

Max Activating Token Index: 268

Text #8

Max Range: 1.8640. Min Range: -1.8640

Max Act: 1.3907. Min Act: -0.0000

Data Index: 749618 (C4 (Web Text))

Max Activating Token Index: 826

Text #9

Max Range: 1.8640. Min Range: -1.8640

Max Act: 1.4170. Min Act: -0.0000

Data Index: 1276187 (C4 (Web Text))

Max Activating Token Index: 275

Text #10

Max Range: 1.8640. Min Range: -1.8640

Max Act: 1.2497. Min Act: -0.0000

Data Index: 611434 (C4 (Web Text))

Max Activating Token Index: 236

Text #11

Max Range: 1.8640. Min Range: -1.8640

Max Act: 1.3923. Min Act: -0.0000

Data Index: 521893 (C4 (Web Text))

Max Activating Token Index: 665

Text #12

Max Range: 1.8640. Min Range: -1.8640

Max Act: 1.2440. Min Act: -0.0000

Data Index: 668777 (C4 (Web Text))

Max Activating Token Index: 332

Text #13

Max Range: 1.8640. Min Range: -1.8640

Max Act: 1.3125. Min Act: -0.0000

Data Index: 216782 (C4 (Web Text))

Max Activating Token Index: 701

Text #14

Max Range: 1.8640. Min Range: -1.8640

Max Act: 1.2920. Min Act: -0.0000

Data Index: 206220 (C4 (Web Text))

Max Activating Token Index: 542

Text #15

Max Range: 1.8640. Min Range: -1.8640

Max Act: 1.2108. Min Act: -0.0000

Data Index: 158197 (C4 (Web Text))

Max Activating Token Index: 16

Text #16

Max Range: 1.8640. Min Range: -1.8640

Max Act: 1.2637. Min Act: -0.0000

Data Index: 49611 (C4 (Web Text))

Max Activating Token Index: 111

Text #17

Max Range: 1.8640. Min Range: -1.8640

Max Act: 1.2460. Min Act: -0.0000

Data Index: 616815 (C4 (Web Text))

Max Activating Token Index: 922

Text #18

Max Range: 1.8640. Min Range: -1.8640

Max Act: 1.2382. Min Act: -0.0000

Data Index: 262389 (C4 (Web Text))

Max Activating Token Index: 143

Text #19

Max Range: 1.8640. Min Range: -1.8640

Max Act: 1.2390. Min Act: -0.0000

Data Index: 1232699 (C4 (Web Text))

Max Activating Token Index: 485