Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 2 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 1.3442. Min Range: -1.3442

Max Act: 1.3442. Min Act: -0.0000

Data Index: 45521 (The Pile)

Max Activating Token Index: 489

Text #1

Max Range: 1.3442. Min Range: -1.3442

Max Act: 1.2086. Min Act: -0.0000

Data Index: 712212 (The Pile)

Max Activating Token Index: 420

Text #2

Max Range: 1.3442. Min Range: -1.3442

Max Act: 1.3191. Min Act: -0.0000

Data Index: 654811 (The Pile)

Max Activating Token Index: 32

Text #3

Max Range: 1.3442. Min Range: -1.3442

Max Act: 1.3085. Min Act: -0.0000

Data Index: 1961370 (The Pile)

Max Activating Token Index: 443

Text #4

Max Range: 1.3442. Min Range: -1.3442

Max Act: 1.1495. Min Act: -0.0000

Data Index: 399723 (The Pile)

Max Activating Token Index: 113

Text #5

Max Range: 1.3442. Min Range: -1.3442

Max Act: 1.1615. Min Act: -0.0000

Data Index: 1601717 (The Pile)

Max Activating Token Index: 923

Text #6

Max Range: 1.3442. Min Range: -1.3442

Max Act: 1.1428. Min Act: -0.0000

Data Index: 435415 (The Pile)

Max Activating Token Index: 75

Text #7

Max Range: 1.3442. Min Range: -1.3442

Max Act: 1.1382. Min Act: -0.0000

Data Index: 1065672 (The Pile)

Max Activating Token Index: 140

Text #8

Max Range: 1.3442. Min Range: -1.3442

Max Act: 1.2080. Min Act: -0.0000

Data Index: 1426682 (The Pile)

Max Activating Token Index: 1014

Text #9

Max Range: 1.3442. Min Range: -1.3442

Max Act: 1.1318. Min Act: -0.0000

Data Index: 1411853 (The Pile)

Max Activating Token Index: 358

Text #10

Max Range: 1.3442. Min Range: -1.3442

Max Act: 1.0898. Min Act: -0.0000

Data Index: 73777 (The Pile)

Max Activating Token Index: 969

Text #11

Max Range: 1.3442. Min Range: -1.3442

Max Act: 1.0570. Min Act: -0.0000

Data Index: 498821 (The Pile)

Max Activating Token Index: 1016

Text #12

Max Range: 1.3442. Min Range: -1.3442

Max Act: 1.0947. Min Act: -0.0000

Data Index: 1172853 (The Pile)

Max Activating Token Index: 57

Text #13

Max Range: 1.3442. Min Range: -1.3442

Max Act: 1.0180. Min Act: -0.0000

Data Index: 683192 (The Pile)

Max Activating Token Index: 577

Text #14

Max Range: 1.3442. Min Range: -1.3442

Max Act: 1.0153. Min Act: -0.0000

Data Index: 1253284 (The Pile)

Max Activating Token Index: 117

Text #15

Max Range: 1.3442. Min Range: -1.3442

Max Act: 1.0496. Min Act: -0.0000

Data Index: 1937320 (The Pile)

Max Activating Token Index: 235

Text #16

Max Range: 1.3442. Min Range: -1.3442

Max Act: 1.0331. Min Act: -0.0000

Data Index: 1305670 (The Pile)

Max Activating Token Index: 91

Text #17

Max Range: 1.3442. Min Range: -1.3442

Max Act: 1.1213. Min Act: -0.0000

Data Index: 1296478 (The Pile)

Max Activating Token Index: 896

Text #18

Max Range: 1.3442. Min Range: -1.3442

Max Act: 0.9906. Min Act: -0.0000

Data Index: 1400159 (The Pile)

Max Activating Token Index: 567

Text #19

Max Range: 1.3442. Min Range: -1.3442

Max Act: 1.0124. Min Act: -0.0000

Data Index: 663817 (The Pile)

Max Activating Token Index: 483