Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2205 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.3916. Min Range: -2.3916

Max Act: 2.3426. Min Act: -0.0000

Data Index: 138285 (C4 (Web Text))

Max Activating Token Index: 94

Text #1

Max Range: 2.3916. Min Range: -2.3916

Max Act: 2.2774. Min Act: -0.0000

Data Index: 488070 (C4 (Web Text))

Max Activating Token Index: 387

Text #2

Max Range: 2.3916. Min Range: -2.3916

Max Act: 2.2788. Min Act: -0.0000

Data Index: 871859 (C4 (Web Text))

Max Activating Token Index: 750

Text #3

Max Range: 2.3916. Min Range: -2.3916

Max Act: 2.3916. Min Act: -0.0000

Data Index: 1104173 (C4 (Web Text))

Max Activating Token Index: 330

Text #4

Max Range: 2.3916. Min Range: -2.3916

Max Act: 2.3399. Min Act: -0.0000

Data Index: 194279 (C4 (Web Text))

Max Activating Token Index: 679

Text #5

Max Range: 2.3916. Min Range: -2.3916

Max Act: 2.0434. Min Act: -0.0000

Data Index: 311927 (C4 (Web Text))

Max Activating Token Index: 988

Text #6

Max Range: 2.3916. Min Range: -2.3916

Max Act: 2.1134. Min Act: -0.0000

Data Index: 963606 (C4 (Web Text))

Max Activating Token Index: 548

Text #7

Max Range: 2.3916. Min Range: -2.3916

Max Act: 1.8847. Min Act: -0.0000

Data Index: 651730 (C4 (Web Text))

Max Activating Token Index: 863

Text #8

Max Range: 2.3916. Min Range: -2.3916

Max Act: 1.8778. Min Act: -0.0000

Data Index: 1006154 (C4 (Web Text))

Max Activating Token Index: 145

Text #9

Max Range: 2.3916. Min Range: -2.3916

Max Act: 1.9645. Min Act: -0.0000

Data Index: 408049 (C4 (Web Text))

Max Activating Token Index: 920

Text #10

Max Range: 2.3916. Min Range: -2.3916

Max Act: 1.9040. Min Act: -0.0000

Data Index: 342872 (C4 (Web Text))

Max Activating Token Index: 845

Text #11

Max Range: 2.3916. Min Range: -2.3916

Max Act: 2.1081. Min Act: -0.0000

Data Index: 238711 (C4 (Web Text))

Max Activating Token Index: 667

Text #12

Max Range: 2.3916. Min Range: -2.3916

Max Act: 1.9004. Min Act: -0.0000

Data Index: 1237829 (C4 (Web Text))

Max Activating Token Index: 419

Text #13

Max Range: 2.3916. Min Range: -2.3916

Max Act: 1.8090. Min Act: -0.0000

Data Index: 994036 (C4 (Web Text))

Max Activating Token Index: 983

Text #14

Max Range: 2.3916. Min Range: -2.3916

Max Act: 1.7741. Min Act: -0.0000

Data Index: 814569 (C4 (Web Text))

Max Activating Token Index: 58

Text #15

Max Range: 2.3916. Min Range: -2.3916

Max Act: 1.7541. Min Act: -0.0000

Data Index: 981446 (C4 (Web Text))

Max Activating Token Index: 189

Text #16

Max Range: 2.3916. Min Range: -2.3916

Max Act: 1.8629. Min Act: -0.0000

Data Index: 1195710 (C4 (Web Text))

Max Activating Token Index: 997

Text #17

Max Range: 2.3916. Min Range: -2.3916

Max Act: 1.8545. Min Act: -0.0000

Data Index: 1343570 (C4 (Web Text))

Max Activating Token Index: 584

Text #18

Max Range: 2.3916. Min Range: -2.3916

Max Act: 2.1432. Min Act: -0.0000

Data Index: 132552 (C4 (Web Text))

Max Activating Token Index: 99

Text #19

Max Range: 2.3916. Min Range: -2.3916

Max Act: 1.5210. Min Act: -0.0000

Data Index: 1258498 (C4 (Web Text))

Max Activating Token Index: 999