Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 569 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.0781. Min Range: -3.0781

Max Act: 3.0781. Min Act: -0.0000

Data Index: 91368 (C4 (Web Text))

Max Activating Token Index: 922

Text #1

Max Range: 3.0781. Min Range: -3.0781

Max Act: 2.8287. Min Act: -0.0000

Data Index: 195381 (C4 (Web Text))

Max Activating Token Index: 702

Text #2

Max Range: 3.0781. Min Range: -3.0781

Max Act: 2.6361. Min Act: -0.0000

Data Index: 862327 (C4 (Web Text))

Max Activating Token Index: 381

Text #3

Max Range: 3.0781. Min Range: -3.0781

Max Act: 2.4966. Min Act: -0.0000

Data Index: 982170 (C4 (Web Text))

Max Activating Token Index: 981

Text #4

Max Range: 3.0781. Min Range: -3.0781

Max Act: 2.6413. Min Act: -0.0000

Data Index: 421557 (C4 (Web Text))

Max Activating Token Index: 861

Text #5

Max Range: 3.0781. Min Range: -3.0781

Max Act: 2.7560. Min Act: -0.0000

Data Index: 503203 (C4 (Web Text))

Max Activating Token Index: 300

Text #6

Max Range: 3.0781. Min Range: -3.0781

Max Act: 2.5791. Min Act: -0.0000

Data Index: 1273094 (C4 (Web Text))

Max Activating Token Index: 325

Text #7

Max Range: 3.0781. Min Range: -3.0781

Max Act: 2.6295. Min Act: -0.0000

Data Index: 924406 (C4 (Web Text))

Max Activating Token Index: 742

Text #8

Max Range: 3.0781. Min Range: -3.0781

Max Act: 2.4904. Min Act: -0.0000

Data Index: 1167654 (C4 (Web Text))

Max Activating Token Index: 829

Text #9

Max Range: 3.0781. Min Range: -3.0781

Max Act: 2.4763. Min Act: -0.0000

Data Index: 946461 (C4 (Web Text))

Max Activating Token Index: 575

Text #10

Max Range: 3.0781. Min Range: -3.0781

Max Act: 2.4710. Min Act: -0.0000

Data Index: 847150 (C4 (Web Text))

Max Activating Token Index: 955

Text #11

Max Range: 3.0781. Min Range: -3.0781

Max Act: 2.5543. Min Act: -0.0000

Data Index: 225901 (C4 (Web Text))

Max Activating Token Index: 794

Text #12

Max Range: 3.0781. Min Range: -3.0781

Max Act: 2.3990. Min Act: -0.0000

Data Index: 26237 (C4 (Web Text))

Max Activating Token Index: 682

Text #13

Max Range: 3.0781. Min Range: -3.0781

Max Act: 2.2772. Min Act: -0.0000

Data Index: 1059079 (C4 (Web Text))

Max Activating Token Index: 579

Text #14

Max Range: 3.0781. Min Range: -3.0781

Max Act: 2.3925. Min Act: -0.0000

Data Index: 89417 (C4 (Web Text))

Max Activating Token Index: 598

Text #15

Max Range: 3.0781. Min Range: -3.0781

Max Act: 2.2734. Min Act: -0.0000

Data Index: 1303822 (C4 (Web Text))

Max Activating Token Index: 597

Text #16

Max Range: 3.0781. Min Range: -3.0781

Max Act: 2.3771. Min Act: -0.0000

Data Index: 1072960 (C4 (Web Text))

Max Activating Token Index: 353

Text #17

Max Range: 3.0781. Min Range: -3.0781

Max Act: 2.2559. Min Act: -0.0000

Data Index: 1060128 (C4 (Web Text))

Max Activating Token Index: 979

Text #18

Max Range: 3.0781. Min Range: -3.0781

Max Act: 2.3547. Min Act: -0.0000

Data Index: 693862 (C4 (Web Text))

Max Activating Token Index: 865

Text #19

Max Range: 3.0781. Min Range: -3.0781

Max Act: 2.3766. Min Act: -0.0000

Data Index: 767335 (C4 (Web Text))

Max Activating Token Index: 1008