Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3805 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.6337. Min Range: -1.6337

Max Act: 1.6337. Min Act: -0.0001

Data Index: 641521 (C4 (Web Text))

Max Activating Token Index: 789

Text #1

Max Range: 1.6337. Min Range: -1.6337

Max Act: 1.4088. Min Act: -0.0001

Data Index: 408483 (C4 (Web Text))

Max Activating Token Index: 105

Text #2

Max Range: 1.6337. Min Range: -1.6337

Max Act: 1.1006. Min Act: -0.0001

Data Index: 876164 (C4 (Web Text))

Max Activating Token Index: 789

Text #3

Max Range: 1.6337. Min Range: -1.6337

Max Act: 1.4792. Min Act: -0.0001

Data Index: 1173124 (C4 (Web Text))

Max Activating Token Index: 921

Text #4

Max Range: 1.6337. Min Range: -1.6337

Max Act: 1.1852. Min Act: -0.0001

Data Index: 693404 (C4 (Web Text))

Max Activating Token Index: 323

Text #5

Max Range: 1.6337. Min Range: -1.6337

Max Act: 1.2606. Min Act: -0.0001

Data Index: 877382 (C4 (Web Text))

Max Activating Token Index: 881

Text #6

Max Range: 1.6337. Min Range: -1.6337

Max Act: 1.2529. Min Act: -0.0001

Data Index: 1183942 (C4 (Web Text))

Max Activating Token Index: 601

Text #7

Max Range: 1.6337. Min Range: -1.6337

Max Act: 1.3831. Min Act: -0.0001

Data Index: 1224625 (C4 (Web Text))

Max Activating Token Index: 773

Text #8

Max Range: 1.6337. Min Range: -1.6337

Max Act: 1.1471. Min Act: -0.0001

Data Index: 990593 (C4 (Web Text))

Max Activating Token Index: 180

Text #9

Max Range: 1.6337. Min Range: -1.6337

Max Act: 1.1478. Min Act: -0.0001

Data Index: 343667 (C4 (Web Text))

Max Activating Token Index: 476

Text #10

Max Range: 1.6337. Min Range: -1.6337

Max Act: 1.1839. Min Act: -0.0001

Data Index: 997989 (C4 (Web Text))

Max Activating Token Index: 43

Text #11

Max Range: 1.6337. Min Range: -1.6337

Max Act: 1.1067. Min Act: -0.0001

Data Index: 388002 (C4 (Web Text))

Max Activating Token Index: 224

Text #12

Max Range: 1.6337. Min Range: -1.6337

Max Act: 1.1253. Min Act: -0.0001

Data Index: 428934 (C4 (Web Text))

Max Activating Token Index: 113

Text #13

Max Range: 1.6337. Min Range: -1.6337

Max Act: 1.0594. Min Act: -0.0001

Data Index: 259222 (C4 (Web Text))

Max Activating Token Index: 869

Text #14

Max Range: 1.6337. Min Range: -1.6337

Max Act: 1.1759. Min Act: -0.0001

Data Index: 771583 (C4 (Web Text))

Max Activating Token Index: 731

Text #15

Max Range: 1.6337. Min Range: -1.6337

Max Act: 1.0273. Min Act: -0.0001

Data Index: 884560 (C4 (Web Text))

Max Activating Token Index: 187

Text #16

Max Range: 1.6337. Min Range: -1.6337

Max Act: 1.0872. Min Act: -0.0001

Data Index: 27484 (C4 (Web Text))

Max Activating Token Index: 131

Text #17

Max Range: 1.6337. Min Range: -1.6337

Max Act: 1.2046. Min Act: -0.0001

Data Index: 437413 (C4 (Web Text))

Max Activating Token Index: 751

Text #18

Max Range: 1.6337. Min Range: -1.6337

Max Act: 1.0681. Min Act: -0.0001

Data Index: 1263069 (C4 (Web Text))

Max Activating Token Index: 488

Text #19

Max Range: 1.6337. Min Range: -1.6337

Max Act: 1.1641. Min Act: -0.0001

Data Index: 865333 (C4 (Web Text))

Max Activating Token Index: 448