Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1336 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.9542. Min Range: -0.9542

Max Act: 0.9542. Min Act: -0.0000

Data Index: 358538 (C4 (Web Text))

Max Activating Token Index: 500

Text #1

Max Range: 0.9542. Min Range: -0.9542

Max Act: 0.8502. Min Act: -0.0000

Data Index: 1253120 (C4 (Web Text))

Max Activating Token Index: 760

Text #2

Max Range: 0.9542. Min Range: -0.9542

Max Act: 0.7344. Min Act: -0.0000

Data Index: 258472 (C4 (Web Text))

Max Activating Token Index: 201

Text #3

Max Range: 0.9542. Min Range: -0.9542

Max Act: 0.7547. Min Act: -0.0000

Data Index: 417842 (C4 (Web Text))

Max Activating Token Index: 82

Text #4

Max Range: 0.9542. Min Range: -0.9542

Max Act: 0.6933. Min Act: -0.0000

Data Index: 827128 (C4 (Web Text))

Max Activating Token Index: 498

Text #5

Max Range: 0.9542. Min Range: -0.9542

Max Act: 0.6802. Min Act: -0.0000

Data Index: 687801 (C4 (Web Text))

Max Activating Token Index: 836

Text #6

Max Range: 0.9542. Min Range: -0.9542

Max Act: 0.6813. Min Act: -0.0000

Data Index: 292537 (C4 (Web Text))

Max Activating Token Index: 572

Text #7

Max Range: 0.9542. Min Range: -0.9542

Max Act: 0.6464. Min Act: -0.0000

Data Index: 674446 (C4 (Web Text))

Max Activating Token Index: 653

Text #8

Max Range: 0.9542. Min Range: -0.9542

Max Act: 0.6383. Min Act: -0.0000

Data Index: 1232034 (C4 (Web Text))

Max Activating Token Index: 255

Text #9

Max Range: 0.9542. Min Range: -0.9542

Max Act: 0.6068. Min Act: -0.0000

Data Index: 844599 (C4 (Web Text))

Max Activating Token Index: 883

Text #10

Max Range: 0.9542. Min Range: -0.9542

Max Act: 0.6183. Min Act: -0.0000

Data Index: 1136465 (C4 (Web Text))

Max Activating Token Index: 55

Text #11

Max Range: 0.9542. Min Range: -0.9542

Max Act: 0.6104. Min Act: -0.0000

Data Index: 473476 (C4 (Web Text))

Max Activating Token Index: 850

Text #12

Max Range: 0.9542. Min Range: -0.9542

Max Act: 0.6285. Min Act: -0.0000

Data Index: 808023 (C4 (Web Text))

Max Activating Token Index: 817

Text #13

Max Range: 0.9542. Min Range: -0.9542

Max Act: 0.6309. Min Act: -0.0000

Data Index: 1117557 (C4 (Web Text))

Max Activating Token Index: 431

Text #14

Max Range: 0.9542. Min Range: -0.9542

Max Act: 0.6045. Min Act: -0.0000

Data Index: 644456 (C4 (Web Text))

Max Activating Token Index: 880

Text #15

Max Range: 0.9542. Min Range: -0.9542

Max Act: 0.5997. Min Act: -0.0000

Data Index: 128501 (C4 (Web Text))

Max Activating Token Index: 628

Text #16

Max Range: 0.9542. Min Range: -0.9542

Max Act: 0.5899. Min Act: -0.0000

Data Index: 862372 (C4 (Web Text))

Max Activating Token Index: 168

Text #17

Max Range: 0.9542. Min Range: -0.9542

Max Act: 0.6029. Min Act: -0.0000

Data Index: 464838 (C4 (Web Text))

Max Activating Token Index: 150

Text #18

Max Range: 0.9542. Min Range: -0.9542

Max Act: 0.5895. Min Act: -0.0000

Data Index: 156156 (C4 (Web Text))

Max Activating Token Index: 471

Text #19

Max Range: 0.9542. Min Range: -0.9542

Max Act: 0.5831. Min Act: -0.0000

Data Index: 261016 (C4 (Web Text))

Max Activating Token Index: 812