Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 53 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 1.3936. Min Range: -1.3936

Max Act: 1.3936. Min Act: -0.0000

Data Index: 438240 (The Pile)

Max Activating Token Index: 580

Text #1

Max Range: 1.3936. Min Range: -1.3936

Max Act: 1.1260. Min Act: -0.0000

Data Index: 1644839 (The Pile)

Max Activating Token Index: 675

Text #2

Max Range: 1.3936. Min Range: -1.3936

Max Act: 1.0221. Min Act: -0.0000

Data Index: 1650416 (The Pile)

Max Activating Token Index: 740

Text #3

Max Range: 1.3936. Min Range: -1.3936

Max Act: 1.0257. Min Act: -0.0000

Data Index: 639955 (The Pile)

Max Activating Token Index: 942

Text #4

Max Range: 1.3936. Min Range: -1.3936

Max Act: 1.0054. Min Act: -0.0000

Data Index: 1270508 (The Pile)

Max Activating Token Index: 560

Text #5

Max Range: 1.3936. Min Range: -1.3936

Max Act: 1.0049. Min Act: -0.0000

Data Index: 1858024 (The Pile)

Max Activating Token Index: 384

Text #6

Max Range: 1.3936. Min Range: -1.3936

Max Act: 0.9237. Min Act: -0.0000

Data Index: 1878505 (The Pile)

Max Activating Token Index: 574

Text #7

Max Range: 1.3936. Min Range: -1.3936

Max Act: 0.9771. Min Act: -0.0000

Data Index: 1045155 (The Pile)

Max Activating Token Index: 881

Text #8

Max Range: 1.3936. Min Range: -1.3936

Max Act: 0.9649. Min Act: -0.0000

Data Index: 239896 (The Pile)

Max Activating Token Index: 336

Text #9

Max Range: 1.3936. Min Range: -1.3936

Max Act: 0.8889. Min Act: -0.0000

Data Index: 1007858 (The Pile)

Max Activating Token Index: 763

Text #10

Max Range: 1.3936. Min Range: -1.3936

Max Act: 0.9407. Min Act: -0.0000

Data Index: 462916 (The Pile)

Max Activating Token Index: 583

Text #11

Max Range: 1.3936. Min Range: -1.3936

Max Act: 0.9432. Min Act: -0.0000

Data Index: 1344470 (The Pile)

Max Activating Token Index: 665

Text #12

Max Range: 1.3936. Min Range: -1.3936

Max Act: 0.9263. Min Act: -0.0000

Data Index: 834529 (The Pile)

Max Activating Token Index: 884

Text #13

Max Range: 1.3936. Min Range: -1.3936

Max Act: 0.8683. Min Act: -0.0000

Data Index: 232275 (The Pile)

Max Activating Token Index: 447

Text #14

Max Range: 1.3936. Min Range: -1.3936

Max Act: 0.8943. Min Act: -0.0000

Data Index: 160569 (The Pile)

Max Activating Token Index: 490

Text #15

Max Range: 1.3936. Min Range: -1.3936

Max Act: 0.9176. Min Act: -0.0000

Data Index: 498253 (The Pile)

Max Activating Token Index: 551

Text #16

Max Range: 1.3936. Min Range: -1.3936

Max Act: 0.8530. Min Act: -0.0000

Data Index: 1407263 (The Pile)

Max Activating Token Index: 334

Text #17

Max Range: 1.3936. Min Range: -1.3936

Max Act: 0.8641. Min Act: -0.0000

Data Index: 1259148 (The Pile)

Max Activating Token Index: 304

Text #18

Max Range: 1.3936. Min Range: -1.3936

Max Act: 0.8464. Min Act: -0.0000

Data Index: 1332683 (The Pile)

Max Activating Token Index: 101

Text #19

Max Range: 1.3936. Min Range: -1.3936

Max Act: 0.7912. Min Act: -0.0000

Data Index: 410143 (The Pile)

Max Activating Token Index: 786