Model: SoLU Model: 3 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1323 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-3l')

Text #0

Max Range: 0.0945. Min Range: -0.0945

Max Act: 0.0945. Min Act: -0.0006

Data Index: 1314444 (C4 (Web Text))

Max Activating Token Index: 911

Text #1

Max Range: 0.0945. Min Range: -0.0945

Max Act: 0.0899. Min Act: -0.0006

Data Index: 265865 (C4 (Web Text))

Max Activating Token Index: 594

Text #2

Max Range: 0.0945. Min Range: -0.0945

Max Act: 0.0853. Min Act: -0.0006

Data Index: 1104391 (C4 (Web Text))

Max Activating Token Index: 740

Text #3

Max Range: 0.0945. Min Range: -0.0945

Max Act: 0.0800. Min Act: -0.0006

Data Index: 439402 (C4 (Web Text))

Max Activating Token Index: 870

Text #4

Max Range: 0.0945. Min Range: -0.0945

Max Act: 0.0791. Min Act: -0.0006

Data Index: 1109401 (C4 (Web Text))

Max Activating Token Index: 467

Text #5

Max Range: 0.0945. Min Range: -0.0945

Max Act: 0.0737. Min Act: -0.0006

Data Index: 805103 (C4 (Web Text))

Max Activating Token Index: 744

Text #6

Max Range: 0.0945. Min Range: -0.0945

Max Act: 0.0681. Min Act: -0.0006

Data Index: 216776 (C4 (Web Text))

Max Activating Token Index: 960

Text #7

Max Range: 0.0945. Min Range: -0.0945

Max Act: 0.0732. Min Act: -0.0006

Data Index: 446437 (C4 (Web Text))

Max Activating Token Index: 278

Text #8

Max Range: 0.0945. Min Range: -0.0945

Max Act: 0.0704. Min Act: -0.0006

Data Index: 827710 (C4 (Web Text))

Max Activating Token Index: 595

Text #9

Max Range: 0.0945. Min Range: -0.0945

Max Act: 0.0699. Min Act: -0.0006

Data Index: 445707 (C4 (Web Text))

Max Activating Token Index: 374

Text #10

Max Range: 0.0945. Min Range: -0.0945

Max Act: 0.0726. Min Act: -0.0006

Data Index: 575315 (C4 (Web Text))

Max Activating Token Index: 592

Text #11

Max Range: 0.0945. Min Range: -0.0945

Max Act: 0.0713. Min Act: -0.0006

Data Index: 1275434 (C4 (Web Text))

Max Activating Token Index: 552

Text #12

Max Range: 0.0945. Min Range: -0.0945

Max Act: 0.0710. Min Act: -0.0005

Data Index: 943293 (C4 (Web Text))

Max Activating Token Index: 811

Text #13

Max Range: 0.0945. Min Range: -0.0945

Max Act: 0.0735. Min Act: -0.0006

Data Index: 738209 (C4 (Web Text))

Max Activating Token Index: 791

Text #14

Max Range: 0.0945. Min Range: -0.0945

Max Act: 0.0682. Min Act: -0.0006

Data Index: 1135294 (C4 (Web Text))

Max Activating Token Index: 696

Text #15

Max Range: 0.0945. Min Range: -0.0945

Max Act: 0.0700. Min Act: -0.0006

Data Index: 927393 (C4 (Web Text))

Max Activating Token Index: 644

Text #16

Max Range: 0.0945. Min Range: -0.0945

Max Act: 0.0699. Min Act: -0.0006

Data Index: 353621 (C4 (Web Text))

Max Activating Token Index: 176

Text #17

Max Range: 0.0945. Min Range: -0.0945

Max Act: 0.0668. Min Act: -0.0006

Data Index: 1179401 (C4 (Web Text))

Max Activating Token Index: 695

Text #18

Max Range: 0.0945. Min Range: -0.0945

Max Act: 0.0695. Min Act: -0.0006

Data Index: 537683 (C4 (Web Text))

Max Activating Token Index: 305

Text #19

Max Range: 0.0945. Min Range: -0.0945

Max Act: 0.0720. Min Act: -0.0005

Data Index: 913572 (C4 (Web Text))

Max Activating Token Index: 499