Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 186 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 0.6955. Min Range: -0.6955

Max Act: 0.6955. Min Act: -0.0000

Data Index: 1257528 (The Pile)

Max Activating Token Index: 936

Text #1

Max Range: 0.6955. Min Range: -0.6955

Max Act: 0.6517. Min Act: -0.0000

Data Index: 1321400 (The Pile)

Max Activating Token Index: 134

Text #2

Max Range: 0.6955. Min Range: -0.6955

Max Act: 0.6475. Min Act: -0.0000

Data Index: 1490223 (The Pile)

Max Activating Token Index: 197

Text #3

Max Range: 0.6955. Min Range: -0.6955

Max Act: 0.6001. Min Act: -0.0000

Data Index: 257100 (The Pile)

Max Activating Token Index: 456

Text #4

Max Range: 0.6955. Min Range: -0.6955

Max Act: 0.5972. Min Act: -0.0000

Data Index: 1548546 (The Pile)

Max Activating Token Index: 274

Text #5

Max Range: 0.6955. Min Range: -0.6955

Max Act: 0.5961. Min Act: -0.0000

Data Index: 980912 (The Pile)

Max Activating Token Index: 964

Text #6

Max Range: 0.6955. Min Range: -0.6955

Max Act: 0.5605. Min Act: -0.0000

Data Index: 1897403 (The Pile)

Max Activating Token Index: 381

Text #7

Max Range: 0.6955. Min Range: -0.6955

Max Act: 0.5584. Min Act: -0.0000

Data Index: 1334183 (The Pile)

Max Activating Token Index: 49

Text #8

Max Range: 0.6955. Min Range: -0.6955

Max Act: 0.5750. Min Act: -0.0000

Data Index: 924753 (The Pile)

Max Activating Token Index: 563

Text #9

Max Range: 0.6955. Min Range: -0.6955

Max Act: 0.5517. Min Act: -0.0000

Data Index: 1999985 (The Pile)

Max Activating Token Index: 499

Text #10

Max Range: 0.6955. Min Range: -0.6955

Max Act: 0.5300. Min Act: -0.0000

Data Index: 840045 (The Pile)

Max Activating Token Index: 501

Text #11

Max Range: 0.6955. Min Range: -0.6955

Max Act: 0.5430. Min Act: -0.0000

Data Index: 1428402 (The Pile)

Max Activating Token Index: 455

Text #12

Max Range: 0.6955. Min Range: -0.6955

Max Act: 0.5448. Min Act: -0.0000

Data Index: 1188911 (The Pile)

Max Activating Token Index: 516

Text #13

Max Range: 0.6955. Min Range: -0.6955

Max Act: 0.5422. Min Act: -0.0000

Data Index: 1003325 (The Pile)

Max Activating Token Index: 197

Text #14

Max Range: 0.6955. Min Range: -0.6955

Max Act: 0.5721. Min Act: -0.0000

Data Index: 1474858 (The Pile)

Max Activating Token Index: 239

Text #15

Max Range: 0.6955. Min Range: -0.6955

Max Act: 0.5289. Min Act: -0.0000

Data Index: 1273337 (The Pile)

Max Activating Token Index: 347

Text #16

Max Range: 0.6955. Min Range: -0.6955

Max Act: 0.4953. Min Act: -0.0000

Data Index: 649684 (The Pile)

Max Activating Token Index: 679

Text #17

Max Range: 0.6955. Min Range: -0.6955

Max Act: 0.5073. Min Act: -0.0000

Data Index: 1326144 (The Pile)

Max Activating Token Index: 109

Text #18

Max Range: 0.6955. Min Range: -0.6955

Max Act: 0.5333. Min Act: -0.0000

Data Index: 951901 (The Pile)

Max Activating Token Index: 485

Text #19

Max Range: 0.6955. Min Range: -0.6955

Max Act: 0.5304. Min Act: -0.0000

Data Index: 1400400 (The Pile)

Max Activating Token Index: 189