Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 5061 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 7.0711. Min Range: -7.0711

Max Act: 7.0711. Min Act: -0.0000

Data Index: 1652093 (The Pile)

Max Activating Token Index: 489

Text #1

Max Range: 7.0711. Min Range: -7.0711

Max Act: 6.6293. Min Act: -0.0000

Data Index: 1508649 (The Pile)

Max Activating Token Index: 589

Text #2

Max Range: 7.0711. Min Range: -7.0711

Max Act: 6.4750. Min Act: -0.0000

Data Index: 1546213 (The Pile)

Max Activating Token Index: 49

Text #3

Max Range: 7.0711. Min Range: -7.0711

Max Act: 6.0491. Min Act: -0.0000

Data Index: 312746 (The Pile)

Max Activating Token Index: 189

Text #4

Max Range: 7.0711. Min Range: -7.0711

Max Act: 6.3139. Min Act: -0.0000

Data Index: 466347 (The Pile)

Max Activating Token Index: 27

Text #5

Max Range: 7.0711. Min Range: -7.0711

Max Act: 6.0509. Min Act: -0.0000

Data Index: 1032377 (The Pile)

Max Activating Token Index: 128

Text #6

Max Range: 7.0711. Min Range: -7.0711

Max Act: 6.3121. Min Act: -0.0000

Data Index: 1965533 (The Pile)

Max Activating Token Index: 137

Text #7

Max Range: 7.0711. Min Range: -7.0711

Max Act: 5.9026. Min Act: -0.0000

Data Index: 1181478 (The Pile)

Max Activating Token Index: 148

Text #8

Max Range: 7.0711. Min Range: -7.0711

Max Act: 6.3024. Min Act: -0.0000

Data Index: 971225 (The Pile)

Max Activating Token Index: 578

Text #9

Max Range: 7.0711. Min Range: -7.0711

Max Act: 6.0015. Min Act: -0.0000

Data Index: 1851921 (The Pile)

Max Activating Token Index: 501

Text #10

Max Range: 7.0711. Min Range: -7.0711

Max Act: 5.7446. Min Act: -0.0000

Data Index: 1416511 (The Pile)

Max Activating Token Index: 271

Text #11

Max Range: 7.0711. Min Range: -7.0711

Max Act: 5.9173. Min Act: -0.0000

Data Index: 984839 (The Pile)

Max Activating Token Index: 423

Text #12

Max Range: 7.0711. Min Range: -7.0711

Max Act: 5.7365. Min Act: -0.0000

Data Index: 737998 (The Pile)

Max Activating Token Index: 146

Text #13

Max Range: 7.0711. Min Range: -7.0711

Max Act: 5.4619. Min Act: -0.0000

Data Index: 763580 (The Pile)

Max Activating Token Index: 196

Text #14

Max Range: 7.0711. Min Range: -7.0711

Max Act: 5.6705. Min Act: -0.0000

Data Index: 621044 (The Pile)

Max Activating Token Index: 578

Text #15

Max Range: 7.0711. Min Range: -7.0711

Max Act: 5.8082. Min Act: -0.0000

Data Index: 832811 (The Pile)

Max Activating Token Index: 199

Text #16

Max Range: 7.0711. Min Range: -7.0711

Max Act: 5.5946. Min Act: -0.0000

Data Index: 1683652 (The Pile)

Max Activating Token Index: 247

Text #17

Max Range: 7.0711. Min Range: -7.0711

Max Act: 5.6105. Min Act: -0.0000

Data Index: 546603 (The Pile)

Max Activating Token Index: 537

Text #18

Max Range: 7.0711. Min Range: -7.0711

Max Act: 5.3936. Min Act: -0.0000

Data Index: 1208230 (The Pile)

Max Activating Token Index: 146

Text #19

Max Range: 7.0711. Min Range: -7.0711

Max Act: 5.7912. Min Act: -0.0000

Data Index: 950077 (The Pile)

Max Activating Token Index: 11