Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1697 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.7735. Min Range: -2.7735

Max Act: 2.7735. Min Act: -0.0000

Data Index: 950664 (C4 (Web Text))

Max Activating Token Index: 713

Text #1

Max Range: 2.7735. Min Range: -2.7735

Max Act: 2.5827. Min Act: -0.0000

Data Index: 789798 (C4 (Web Text))

Max Activating Token Index: 420

Text #2

Max Range: 2.7735. Min Range: -2.7735

Max Act: 2.5182. Min Act: -0.0000

Data Index: 638646 (C4 (Web Text))

Max Activating Token Index: 222

Text #3

Max Range: 2.7735. Min Range: -2.7735

Max Act: 2.3818. Min Act: -0.0000

Data Index: 747584 (C4 (Web Text))

Max Activating Token Index: 275

Text #4

Max Range: 2.7735. Min Range: -2.7735

Max Act: 2.3537. Min Act: -0.0000

Data Index: 535530 (C4 (Web Text))

Max Activating Token Index: 121

Text #5

Max Range: 2.7735. Min Range: -2.7735

Max Act: 2.2737. Min Act: -0.0000

Data Index: 681305 (C4 (Web Text))

Max Activating Token Index: 970

Text #6

Max Range: 2.7735. Min Range: -2.7735

Max Act: 2.0951. Min Act: -0.0000

Data Index: 978131 (C4 (Web Text))

Max Activating Token Index: 471

Text #7

Max Range: 2.7735. Min Range: -2.7735

Max Act: 2.1902. Min Act: -0.0000

Data Index: 646883 (C4 (Web Text))

Max Activating Token Index: 450

Text #8

Max Range: 2.7735. Min Range: -2.7735

Max Act: 2.0907. Min Act: -0.0000

Data Index: 659137 (C4 (Web Text))

Max Activating Token Index: 915

Text #9

Max Range: 2.7735. Min Range: -2.7735

Max Act: 2.0020. Min Act: -0.0000

Data Index: 838226 (C4 (Web Text))

Max Activating Token Index: 878

Text #10

Max Range: 2.7735. Min Range: -2.7735

Max Act: 2.1653. Min Act: -0.0000

Data Index: 1230076 (C4 (Web Text))

Max Activating Token Index: 921

Text #11

Max Range: 2.7735. Min Range: -2.7735

Max Act: 2.1624. Min Act: -0.0000

Data Index: 746353 (C4 (Web Text))

Max Activating Token Index: 275

Text #12

Max Range: 2.7735. Min Range: -2.7735

Max Act: 2.2714. Min Act: -0.0000

Data Index: 791942 (C4 (Web Text))

Max Activating Token Index: 849

Text #13

Max Range: 2.7735. Min Range: -2.7735

Max Act: 1.8402. Min Act: -0.0000

Data Index: 751281 (C4 (Web Text))

Max Activating Token Index: 401

Text #14

Max Range: 2.7735. Min Range: -2.7735

Max Act: 1.8774. Min Act: -0.0000

Data Index: 322703 (C4 (Web Text))

Max Activating Token Index: 833

Text #15

Max Range: 2.7735. Min Range: -2.7735

Max Act: 1.9653. Min Act: -0.0000

Data Index: 14020 (C4 (Web Text))

Max Activating Token Index: 971

Text #16

Max Range: 2.7735. Min Range: -2.7735

Max Act: 1.9482. Min Act: -0.0000

Data Index: 79285 (C4 (Web Text))

Max Activating Token Index: 544

Text #17

Max Range: 2.7735. Min Range: -2.7735

Max Act: 1.8272. Min Act: -0.0000

Data Index: 342319 (C4 (Web Text))

Max Activating Token Index: 85

Text #18

Max Range: 2.7735. Min Range: -2.7735

Max Act: 1.8248. Min Act: -0.0000

Data Index: 255639 (C4 (Web Text))

Max Activating Token Index: 444

Text #19

Max Range: 2.7735. Min Range: -2.7735

Max Act: 2.0092. Min Act: -0.0000

Data Index: 1143570 (C4 (Web Text))

Max Activating Token Index: 578