Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 962 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.0181. Min Range: -1.0181

Max Act: 0.9802. Min Act: -0.0000

Data Index: 705011 (C4 (Web Text))

Max Activating Token Index: 816

Text #1

Max Range: 1.0181. Min Range: -1.0181

Max Act: 0.9521. Min Act: -0.0000

Data Index: 601904 (C4 (Web Text))

Max Activating Token Index: 485

Text #2

Max Range: 1.0181. Min Range: -1.0181

Max Act: 1.0181. Min Act: -0.0000

Data Index: 950139 (C4 (Web Text))

Max Activating Token Index: 428

Text #3

Max Range: 1.0181. Min Range: -1.0181

Max Act: 0.9154. Min Act: -0.0000

Data Index: 198337 (C4 (Web Text))

Max Activating Token Index: 672

Text #4

Max Range: 1.0181. Min Range: -1.0181

Max Act: 0.9252. Min Act: -0.0000

Data Index: 1166988 (C4 (Web Text))

Max Activating Token Index: 644

Text #5

Max Range: 1.0181. Min Range: -1.0181

Max Act: 0.8178. Min Act: -0.0000

Data Index: 663208 (C4 (Web Text))

Max Activating Token Index: 443

Text #6

Max Range: 1.0181. Min Range: -1.0181

Max Act: 0.7235. Min Act: -0.0000

Data Index: 699446 (C4 (Web Text))

Max Activating Token Index: 474

Text #7

Max Range: 1.0181. Min Range: -1.0181

Max Act: 0.7624. Min Act: -0.0000

Data Index: 650026 (C4 (Web Text))

Max Activating Token Index: 468

Text #8

Max Range: 1.0181. Min Range: -1.0181

Max Act: 0.6258. Min Act: -0.0000

Data Index: 819747 (C4 (Web Text))

Max Activating Token Index: 578

Text #9

Max Range: 1.0181. Min Range: -1.0181

Max Act: 0.5597. Min Act: -0.0000

Data Index: 48254 (C4 (Web Text))

Max Activating Token Index: 742

Text #10

Max Range: 1.0181. Min Range: -1.0181

Max Act: 0.6090. Min Act: -0.0000

Data Index: 1086767 (C4 (Web Text))

Max Activating Token Index: 55

Text #11

Max Range: 1.0181. Min Range: -1.0181

Max Act: 0.5094. Min Act: -0.0000

Data Index: 694508 (C4 (Web Text))

Max Activating Token Index: 995

Text #12

Max Range: 1.0181. Min Range: -1.0181

Max Act: 0.4904. Min Act: -0.0000

Data Index: 934736 (C4 (Web Text))

Max Activating Token Index: 714

Text #13

Max Range: 1.0181. Min Range: -1.0181

Max Act: 0.4509. Min Act: -0.0000

Data Index: 380043 (C4 (Web Text))

Max Activating Token Index: 422

Text #14

Max Range: 1.0181. Min Range: -1.0181

Max Act: 0.4483. Min Act: -0.0000

Data Index: 452556 (C4 (Web Text))

Max Activating Token Index: 992

Text #15

Max Range: 1.0181. Min Range: -1.0181

Max Act: 0.4284. Min Act: -0.0000

Data Index: 545847 (C4 (Web Text))

Max Activating Token Index: 645

Text #16

Max Range: 1.0181. Min Range: -1.0181

Max Act: 0.4654. Min Act: -0.0000

Data Index: 277154 (C4 (Web Text))

Max Activating Token Index: 312

Text #17

Max Range: 1.0181. Min Range: -1.0181

Max Act: 0.4740. Min Act: -0.0000

Data Index: 845799 (C4 (Web Text))

Max Activating Token Index: 874

Text #18

Max Range: 1.0181. Min Range: -1.0181

Max Act: 0.4344. Min Act: -0.0000

Data Index: 59268 (C4 (Web Text))

Max Activating Token Index: 1021

Text #19

Max Range: 1.0181. Min Range: -1.0181

Max Act: 0.4167. Min Act: -0.0000

Data Index: 771213 (C4 (Web Text))

Max Activating Token Index: 382