Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4795 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 4.0917. Min Range: -4.0917

Max Act: 4.0917. Min Act: -0.0000

Data Index: 1317599 (C4 (Web Text))

Max Activating Token Index: 237

Text #1

Max Range: 4.0917. Min Range: -4.0917

Max Act: 3.1398. Min Act: -0.0000

Data Index: 939063 (C4 (Web Text))

Max Activating Token Index: 461

Text #2

Max Range: 4.0917. Min Range: -4.0917

Max Act: 3.3115. Min Act: -0.0000

Data Index: 1027122 (C4 (Web Text))

Max Activating Token Index: 476

Text #3

Max Range: 4.0917. Min Range: -4.0917

Max Act: 3.1648. Min Act: -0.0000

Data Index: 323383 (C4 (Web Text))

Max Activating Token Index: 74

Text #4

Max Range: 4.0917. Min Range: -4.0917

Max Act: 3.0252. Min Act: -0.0000

Data Index: 217136 (C4 (Web Text))

Max Activating Token Index: 968

Text #5

Max Range: 4.0917. Min Range: -4.0917

Max Act: 2.9272. Min Act: -0.0000

Data Index: 79582 (C4 (Web Text))

Max Activating Token Index: 34

Text #6

Max Range: 4.0917. Min Range: -4.0917

Max Act: 2.9062. Min Act: -0.0000

Data Index: 121732 (C4 (Web Text))

Max Activating Token Index: 1014

Text #7

Max Range: 4.0917. Min Range: -4.0917

Max Act: 2.7032. Min Act: -0.0000

Data Index: 472482 (C4 (Web Text))

Max Activating Token Index: 171

Text #8

Max Range: 4.0917. Min Range: -4.0917

Max Act: 2.9357. Min Act: -0.0000

Data Index: 388777 (C4 (Web Text))

Max Activating Token Index: 33

Text #9

Max Range: 4.0917. Min Range: -4.0917

Max Act: 3.0999. Min Act: -0.0000

Data Index: 739261 (C4 (Web Text))

Max Activating Token Index: 78

Text #10

Max Range: 4.0917. Min Range: -4.0917

Max Act: 3.0418. Min Act: -0.0000

Data Index: 645010 (C4 (Web Text))

Max Activating Token Index: 592

Text #11

Max Range: 4.0917. Min Range: -4.0917

Max Act: 2.8640. Min Act: -0.0000

Data Index: 812314 (C4 (Web Text))

Max Activating Token Index: 118

Text #12

Max Range: 4.0917. Min Range: -4.0917

Max Act: 2.8342. Min Act: -0.0000

Data Index: 1265161 (C4 (Web Text))

Max Activating Token Index: 163

Text #13

Max Range: 4.0917. Min Range: -4.0917

Max Act: 2.7247. Min Act: -0.0000

Data Index: 1316021 (C4 (Web Text))

Max Activating Token Index: 73

Text #14

Max Range: 4.0917. Min Range: -4.0917

Max Act: 2.6858. Min Act: -0.0001

Data Index: 280739 (C4 (Web Text))

Max Activating Token Index: 158

Text #15

Max Range: 4.0917. Min Range: -4.0917

Max Act: 2.8014. Min Act: -0.0000

Data Index: 1021277 (C4 (Web Text))

Max Activating Token Index: 265

Text #16

Max Range: 4.0917. Min Range: -4.0917

Max Act: 2.6713. Min Act: -0.0000

Data Index: 718450 (C4 (Web Text))

Max Activating Token Index: 46

Text #17

Max Range: 4.0917. Min Range: -4.0917

Max Act: 2.5103. Min Act: -0.0000

Data Index: 780109 (C4 (Web Text))

Max Activating Token Index: 22

Text #18

Max Range: 4.0917. Min Range: -4.0917

Max Act: 2.4181. Min Act: -0.0001

Data Index: 908619 (C4 (Web Text))

Max Activating Token Index: 368

Text #19

Max Range: 4.0917. Min Range: -4.0917

Max Act: 2.6079. Min Act: -0.0000

Data Index: 1007302 (C4 (Web Text))

Max Activating Token Index: 611