Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3152 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.0634. Min Range: -1.0634

Max Act: 1.0412. Min Act: -0.0000

Data Index: 863405 (C4 (Web Text))

Max Activating Token Index: 127

Text #1

Max Range: 1.0634. Min Range: -1.0634

Max Act: 0.9852. Min Act: -0.0000

Data Index: 183979 (C4 (Web Text))

Max Activating Token Index: 553

Text #2

Max Range: 1.0634. Min Range: -1.0634

Max Act: 1.0166. Min Act: -0.0000

Data Index: 172656 (C4 (Web Text))

Max Activating Token Index: 387

Text #3

Max Range: 1.0634. Min Range: -1.0634

Max Act: 1.0148. Min Act: -0.0000

Data Index: 666328 (C4 (Web Text))

Max Activating Token Index: 384

Text #4

Max Range: 1.0634. Min Range: -1.0634

Max Act: 1.0634. Min Act: -0.0000

Data Index: 1065314 (C4 (Web Text))

Max Activating Token Index: 332

Text #5

Max Range: 1.0634. Min Range: -1.0634

Max Act: 0.9343. Min Act: -0.0000

Data Index: 1320336 (C4 (Web Text))

Max Activating Token Index: 79

Text #6

Max Range: 1.0634. Min Range: -1.0634

Max Act: 0.9323. Min Act: -0.0000

Data Index: 545962 (C4 (Web Text))

Max Activating Token Index: 129

Text #7

Max Range: 1.0634. Min Range: -1.0634

Max Act: 0.8410. Min Act: -0.0000

Data Index: 444437 (C4 (Web Text))

Max Activating Token Index: 146

Text #8

Max Range: 1.0634. Min Range: -1.0634

Max Act: 0.8421. Min Act: -0.0000

Data Index: 845524 (C4 (Web Text))

Max Activating Token Index: 264

Text #9

Max Range: 1.0634. Min Range: -1.0634

Max Act: 0.8192. Min Act: -0.0000

Data Index: 909720 (C4 (Web Text))

Max Activating Token Index: 844

Text #10

Max Range: 1.0634. Min Range: -1.0634

Max Act: 0.8244. Min Act: -0.0000

Data Index: 78247 (C4 (Web Text))

Max Activating Token Index: 1008

Text #11

Max Range: 1.0634. Min Range: -1.0634

Max Act: 0.8159. Min Act: -0.0000

Data Index: 1540385 (Python Code)

Max Activating Token Index: 860

Text #12

Max Range: 1.0634. Min Range: -1.0634

Max Act: 0.8576. Min Act: -0.0000

Data Index: 336291 (C4 (Web Text))

Max Activating Token Index: 215

Text #13

Max Range: 1.0634. Min Range: -1.0634

Max Act: 0.8230. Min Act: -0.0000

Data Index: 681618 (C4 (Web Text))

Max Activating Token Index: 1020

Text #14

Max Range: 1.0634. Min Range: -1.0634

Max Act: 0.7813. Min Act: -0.0000

Data Index: 444348 (C4 (Web Text))

Max Activating Token Index: 172

Text #15

Max Range: 1.0634. Min Range: -1.0634

Max Act: 0.8012. Min Act: -0.0000

Data Index: 628273 (C4 (Web Text))

Max Activating Token Index: 904

Text #16

Max Range: 1.0634. Min Range: -1.0634

Max Act: 0.8427. Min Act: -0.0000

Data Index: 153308 (C4 (Web Text))

Max Activating Token Index: 564

Text #17

Max Range: 1.0634. Min Range: -1.0634

Max Act: 0.8075. Min Act: -0.0000

Data Index: 1187698 (C4 (Web Text))

Max Activating Token Index: 135

Text #18

Max Range: 1.0634. Min Range: -1.0634

Max Act: 0.7258. Min Act: -0.0000

Data Index: 745502 (C4 (Web Text))

Max Activating Token Index: 87

Text #19

Max Range: 1.0634. Min Range: -1.0634

Max Act: 0.7667. Min Act: -0.0000

Data Index: 642601 (C4 (Web Text))

Max Activating Token Index: 536