Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 744 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 1.6052. Min Range: -1.6052

Max Act: 1.6024. Min Act: -0.0000

Data Index: 1304192 (The Pile)

Max Activating Token Index: 963

Text #1

Max Range: 1.6052. Min Range: -1.6052

Max Act: 1.6052. Min Act: -0.0000

Data Index: 1988373 (The Pile)

Max Activating Token Index: 570

Text #2

Max Range: 1.6052. Min Range: -1.6052

Max Act: 1.4515. Min Act: -0.0000

Data Index: 40307 (The Pile)

Max Activating Token Index: 582

Text #3

Max Range: 1.6052. Min Range: -1.6052

Max Act: 1.4305. Min Act: -0.0000

Data Index: 474366 (The Pile)

Max Activating Token Index: 544

Text #4

Max Range: 1.6052. Min Range: -1.6052

Max Act: 1.3708. Min Act: -0.0000

Data Index: 785706 (The Pile)

Max Activating Token Index: 691

Text #5

Max Range: 1.6052. Min Range: -1.6052

Max Act: 1.3373. Min Act: -0.0000

Data Index: 1599373 (The Pile)

Max Activating Token Index: 890

Text #6

Max Range: 1.6052. Min Range: -1.6052

Max Act: 1.3376. Min Act: -0.0000

Data Index: 353397 (The Pile)

Max Activating Token Index: 593

Text #7

Max Range: 1.6052. Min Range: -1.6052

Max Act: 1.3320. Min Act: -0.0000

Data Index: 1875196 (The Pile)

Max Activating Token Index: 791

Text #8

Max Range: 1.6052. Min Range: -1.6052

Max Act: 1.3043. Min Act: -0.0000

Data Index: 443124 (The Pile)

Max Activating Token Index: 266

Text #9

Max Range: 1.6052. Min Range: -1.6052

Max Act: 1.3856. Min Act: -0.0000

Data Index: 643150 (The Pile)

Max Activating Token Index: 449

Text #10

Max Range: 1.6052. Min Range: -1.6052

Max Act: 1.5716. Min Act: -0.0000

Data Index: 843466 (The Pile)

Max Activating Token Index: 953

Text #11

Max Range: 1.6052. Min Range: -1.6052

Max Act: 1.3563. Min Act: -0.0000

Data Index: 744086 (The Pile)

Max Activating Token Index: 762

Text #12

Max Range: 1.6052. Min Range: -1.6052

Max Act: 1.3051. Min Act: -0.0000

Data Index: 1939348 (The Pile)

Max Activating Token Index: 89

Text #13

Max Range: 1.6052. Min Range: -1.6052

Max Act: 1.3214. Min Act: -0.0000

Data Index: 1503166 (The Pile)

Max Activating Token Index: 89

Text #14

Max Range: 1.6052. Min Range: -1.6052

Max Act: 1.3124. Min Act: -0.0000

Data Index: 518315 (The Pile)

Max Activating Token Index: 25

Text #15

Max Range: 1.6052. Min Range: -1.6052

Max Act: 1.3090. Min Act: -0.0000

Data Index: 813850 (The Pile)

Max Activating Token Index: 614

Text #16

Max Range: 1.6052. Min Range: -1.6052

Max Act: 1.3586. Min Act: -0.0000

Data Index: 783274 (The Pile)

Max Activating Token Index: 49

Text #17

Max Range: 1.6052. Min Range: -1.6052

Max Act: 1.3869. Min Act: -0.0000

Data Index: 1291769 (The Pile)

Max Activating Token Index: 699

Text #18

Max Range: 1.6052. Min Range: -1.6052

Max Act: 1.2568. Min Act: -0.0000

Data Index: 1750962 (The Pile)

Max Activating Token Index: 370

Text #19

Max Range: 1.6052. Min Range: -1.6052

Max Act: 1.3095. Min Act: -0.0000

Data Index: 1286927 (The Pile)

Max Activating Token Index: 301