Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 3030 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 0.3868. Min Range: -0.3868

Max Act: 0.3868. Min Act: -0.0000

Data Index: 355303 (The Pile)

Max Activating Token Index: 358

Text #1

Max Range: 0.3868. Min Range: -0.3868

Max Act: 0.3461. Min Act: -0.0000

Data Index: 503233 (The Pile)

Max Activating Token Index: 683

Text #2

Max Range: 0.3868. Min Range: -0.3868

Max Act: 0.2913. Min Act: -0.0000

Data Index: 341249 (The Pile)

Max Activating Token Index: 707

Text #3

Max Range: 0.3868. Min Range: -0.3868

Max Act: 0.3770. Min Act: -0.0000

Data Index: 1560145 (The Pile)

Max Activating Token Index: 364

Text #4

Max Range: 0.3868. Min Range: -0.3868

Max Act: 0.3406. Min Act: -0.0000

Data Index: 1292899 (The Pile)

Max Activating Token Index: 577

Text #5

Max Range: 0.3868. Min Range: -0.3868

Max Act: 0.3722. Min Act: -0.0000

Data Index: 1693598 (The Pile)

Max Activating Token Index: 864

Text #6

Max Range: 0.3868. Min Range: -0.3868

Max Act: 0.3150. Min Act: -0.0000

Data Index: 325423 (The Pile)

Max Activating Token Index: 883

Text #7

Max Range: 0.3868. Min Range: -0.3868

Max Act: 0.3594. Min Act: -0.0000

Data Index: 1170211 (The Pile)

Max Activating Token Index: 512

Text #8

Max Range: 0.3868. Min Range: -0.3868

Max Act: 0.2980. Min Act: -0.0000

Data Index: 572577 (The Pile)

Max Activating Token Index: 921

Text #9

Max Range: 0.3868. Min Range: -0.3868

Max Act: 0.2970. Min Act: -0.0000

Data Index: 844076 (The Pile)

Max Activating Token Index: 931

Text #10

Max Range: 0.3868. Min Range: -0.3868

Max Act: 0.3050. Min Act: -0.0000

Data Index: 1713283 (The Pile)

Max Activating Token Index: 735

Text #11

Max Range: 0.3868. Min Range: -0.3868

Max Act: 0.3238. Min Act: -0.0000

Data Index: 1254894 (The Pile)

Max Activating Token Index: 42

Text #12

Max Range: 0.3868. Min Range: -0.3868

Max Act: 0.2947. Min Act: -0.0000

Data Index: 1469426 (The Pile)

Max Activating Token Index: 224

Text #13

Max Range: 0.3868. Min Range: -0.3868

Max Act: 0.3118. Min Act: -0.0000

Data Index: 1259923 (The Pile)

Max Activating Token Index: 284

Text #14

Max Range: 0.3868. Min Range: -0.3868

Max Act: 0.3071. Min Act: -0.0000

Data Index: 352390 (The Pile)

Max Activating Token Index: 306

Text #15

Max Range: 0.3868. Min Range: -0.3868

Max Act: 0.3070. Min Act: -0.0000

Data Index: 184643 (The Pile)

Max Activating Token Index: 425

Text #16

Max Range: 0.3868. Min Range: -0.3868

Max Act: 0.2950. Min Act: -0.0000

Data Index: 83045 (The Pile)

Max Activating Token Index: 342

Text #17

Max Range: 0.3868. Min Range: -0.3868

Max Act: 0.3130. Min Act: -0.0000

Data Index: 683552 (The Pile)

Max Activating Token Index: 180

Text #18

Max Range: 0.3868. Min Range: -0.3868

Max Act: 0.3246. Min Act: -0.0000

Data Index: 1900429 (The Pile)

Max Activating Token Index: 255

Text #19

Max Range: 0.3868. Min Range: -0.3868

Max Act: 0.2826. Min Act: -0.0000

Data Index: 1496486 (The Pile)

Max Activating Token Index: 625