Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4371 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.3472. Min Range: -3.3472

Max Act: 3.3472. Min Act: -0.0000

Data Index: 506029 (C4 (Web Text))

Max Activating Token Index: 418

Text #1

Max Range: 3.3472. Min Range: -3.3472

Max Act: 3.0478. Min Act: -0.0000

Data Index: 1257570 (C4 (Web Text))

Max Activating Token Index: 766

Text #2

Max Range: 3.3472. Min Range: -3.3472

Max Act: 3.0145. Min Act: -0.0000

Data Index: 544849 (C4 (Web Text))

Max Activating Token Index: 879

Text #3

Max Range: 3.3472. Min Range: -3.3472

Max Act: 2.9484. Min Act: -0.0000

Data Index: 252483 (C4 (Web Text))

Max Activating Token Index: 739

Text #4

Max Range: 3.3472. Min Range: -3.3472

Max Act: 2.6968. Min Act: -0.0000

Data Index: 166451 (C4 (Web Text))

Max Activating Token Index: 59

Text #5

Max Range: 3.3472. Min Range: -3.3472

Max Act: 2.8376. Min Act: -0.0000

Data Index: 1097795 (C4 (Web Text))

Max Activating Token Index: 49

Text #6

Max Range: 3.3472. Min Range: -3.3472

Max Act: 2.8551. Min Act: -0.0000

Data Index: 1067867 (C4 (Web Text))

Max Activating Token Index: 252

Text #7

Max Range: 3.3472. Min Range: -3.3472

Max Act: 2.8338. Min Act: -0.0000

Data Index: 51363 (C4 (Web Text))

Max Activating Token Index: 877

Text #8

Max Range: 3.3472. Min Range: -3.3472

Max Act: 2.6644. Min Act: -0.0000

Data Index: 1036301 (C4 (Web Text))

Max Activating Token Index: 714

Text #9

Max Range: 3.3472. Min Range: -3.3472

Max Act: 2.6929. Min Act: -0.0000

Data Index: 625152 (C4 (Web Text))

Max Activating Token Index: 623

Text #10

Max Range: 3.3472. Min Range: -3.3472

Max Act: 2.6939. Min Act: -0.0000

Data Index: 352894 (C4 (Web Text))

Max Activating Token Index: 169

Text #11

Max Range: 3.3472. Min Range: -3.3472

Max Act: 2.6650. Min Act: -0.0000

Data Index: 1270202 (C4 (Web Text))

Max Activating Token Index: 566

Text #12

Max Range: 3.3472. Min Range: -3.3472

Max Act: 2.6399. Min Act: -0.0000

Data Index: 1271925 (C4 (Web Text))

Max Activating Token Index: 562

Text #13

Max Range: 3.3472. Min Range: -3.3472

Max Act: 2.6403. Min Act: -0.0000

Data Index: 18158 (C4 (Web Text))

Max Activating Token Index: 849

Text #14

Max Range: 3.3472. Min Range: -3.3472

Max Act: 2.6304. Min Act: -0.0000

Data Index: 562429 (C4 (Web Text))

Max Activating Token Index: 322

Text #15

Max Range: 3.3472. Min Range: -3.3472

Max Act: 2.6008. Min Act: -0.0000

Data Index: 163966 (C4 (Web Text))

Max Activating Token Index: 887

Text #16

Max Range: 3.3472. Min Range: -3.3472

Max Act: 2.5966. Min Act: -0.0000

Data Index: 367514 (C4 (Web Text))

Max Activating Token Index: 294

Text #17

Max Range: 3.3472. Min Range: -3.3472

Max Act: 2.5910. Min Act: -0.0000

Data Index: 790022 (C4 (Web Text))

Max Activating Token Index: 385

Text #18

Max Range: 3.3472. Min Range: -3.3472

Max Act: 2.5438. Min Act: -0.0000

Data Index: 1054102 (C4 (Web Text))

Max Activating Token Index: 557

Text #19

Max Range: 3.3472. Min Range: -3.3472

Max Act: 2.3706. Min Act: -0.0000

Data Index: 12465 (C4 (Web Text))

Max Activating Token Index: 774