Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 150 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.1400. Min Range: -2.1400

Max Act: 2.1400. Min Act: -0.0000

Data Index: 380147 (C4 (Web Text))

Max Activating Token Index: 869

Text #1

Max Range: 2.1400. Min Range: -2.1400

Max Act: 2.0608. Min Act: -0.0000

Data Index: 458056 (C4 (Web Text))

Max Activating Token Index: 687

Text #2

Max Range: 2.1400. Min Range: -2.1400

Max Act: 2.1072. Min Act: -0.0000

Data Index: 782432 (C4 (Web Text))

Max Activating Token Index: 1002

Text #3

Max Range: 2.1400. Min Range: -2.1400

Max Act: 1.9877. Min Act: -0.0000

Data Index: 1012532 (C4 (Web Text))

Max Activating Token Index: 659

Text #4

Max Range: 2.1400. Min Range: -2.1400

Max Act: 1.9666. Min Act: -0.0000

Data Index: 234407 (C4 (Web Text))

Max Activating Token Index: 178

Text #5

Max Range: 2.1400. Min Range: -2.1400

Max Act: 1.8684. Min Act: -0.0000

Data Index: 534498 (C4 (Web Text))

Max Activating Token Index: 819

Text #6

Max Range: 2.1400. Min Range: -2.1400

Max Act: 1.7963. Min Act: -0.0000

Data Index: 591938 (C4 (Web Text))

Max Activating Token Index: 85

Text #7

Max Range: 2.1400. Min Range: -2.1400

Max Act: 1.9128. Min Act: -0.0000

Data Index: 1030965 (C4 (Web Text))

Max Activating Token Index: 908

Text #8

Max Range: 2.1400. Min Range: -2.1400

Max Act: 1.9082. Min Act: -0.0000

Data Index: 53435 (C4 (Web Text))

Max Activating Token Index: 605

Text #9

Max Range: 2.1400. Min Range: -2.1400

Max Act: 1.7811. Min Act: -0.0000

Data Index: 926248 (C4 (Web Text))

Max Activating Token Index: 421

Text #10

Max Range: 2.1400. Min Range: -2.1400

Max Act: 1.7799. Min Act: -0.0000

Data Index: 133317 (C4 (Web Text))

Max Activating Token Index: 739

Text #11

Max Range: 2.1400. Min Range: -2.1400

Max Act: 1.5774. Min Act: -0.0000

Data Index: 152236 (C4 (Web Text))

Max Activating Token Index: 72

Text #12

Max Range: 2.1400. Min Range: -2.1400

Max Act: 1.7671. Min Act: -0.0000

Data Index: 94593 (C4 (Web Text))

Max Activating Token Index: 879

Text #13

Max Range: 2.1400. Min Range: -2.1400

Max Act: 1.6638. Min Act: -0.0000

Data Index: 348844 (C4 (Web Text))

Max Activating Token Index: 130

Text #14

Max Range: 2.1400. Min Range: -2.1400

Max Act: 1.6339. Min Act: -0.0000

Data Index: 1346220 (C4 (Web Text))

Max Activating Token Index: 898

Text #15

Max Range: 2.1400. Min Range: -2.1400

Max Act: 1.5741. Min Act: -0.0000

Data Index: 976826 (C4 (Web Text))

Max Activating Token Index: 90

Text #16

Max Range: 2.1400. Min Range: -2.1400

Max Act: 1.6147. Min Act: -0.0000

Data Index: 129313 (C4 (Web Text))

Max Activating Token Index: 414

Text #17

Max Range: 2.1400. Min Range: -2.1400

Max Act: 1.5188. Min Act: -0.0000

Data Index: 825938 (C4 (Web Text))

Max Activating Token Index: 946

Text #18

Max Range: 2.1400. Min Range: -2.1400

Max Act: 1.5203. Min Act: -0.0000

Data Index: 1190681 (C4 (Web Text))

Max Activating Token Index: 237

Text #19

Max Range: 2.1400. Min Range: -2.1400

Max Act: 1.5886. Min Act: -0.0000

Data Index: 570125 (C4 (Web Text))

Max Activating Token Index: 573