Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 5112 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 1.7760. Min Range: -1.7760

Max Act: 1.7760. Min Act: -0.0000

Data Index: 1708337 (The Pile)

Max Activating Token Index: 665

Text #1

Max Range: 1.7760. Min Range: -1.7760

Max Act: 1.6267. Min Act: -0.0000

Data Index: 1329027 (The Pile)

Max Activating Token Index: 914

Text #2

Max Range: 1.7760. Min Range: -1.7760

Max Act: 1.4207. Min Act: -0.0000

Data Index: 1698917 (The Pile)

Max Activating Token Index: 893

Text #3

Max Range: 1.7760. Min Range: -1.7760

Max Act: 1.5264. Min Act: -0.0000

Data Index: 1243108 (The Pile)

Max Activating Token Index: 992

Text #4

Max Range: 1.7760. Min Range: -1.7760

Max Act: 1.4037. Min Act: -0.0000

Data Index: 1467642 (The Pile)

Max Activating Token Index: 463

Text #5

Max Range: 1.7760. Min Range: -1.7760

Max Act: 1.2549. Min Act: -0.0000

Data Index: 939116 (The Pile)

Max Activating Token Index: 16

Text #6

Max Range: 1.7760. Min Range: -1.7760

Max Act: 1.2340. Min Act: -0.0000

Data Index: 1460949 (The Pile)

Max Activating Token Index: 270

Text #7

Max Range: 1.7760. Min Range: -1.7760

Max Act: 1.2788. Min Act: -0.0000

Data Index: 1427545 (The Pile)

Max Activating Token Index: 48

Text #8

Max Range: 1.7760. Min Range: -1.7760

Max Act: 1.2825. Min Act: -0.0000

Data Index: 1193975 (The Pile)

Max Activating Token Index: 927

Text #9

Max Range: 1.7760. Min Range: -1.7760

Max Act: 0.8785. Min Act: -0.0000

Data Index: 1439445 (The Pile)

Max Activating Token Index: 306

Text #10

Max Range: 1.7760. Min Range: -1.7760

Max Act: 0.6869. Min Act: -0.0000

Data Index: 58351 (The Pile)

Max Activating Token Index: 289

Text #11

Max Range: 1.7760. Min Range: -1.7760

Max Act: 0.8533. Min Act: -0.0000

Data Index: 1733275 (The Pile)

Max Activating Token Index: 904

Text #12

Max Range: 1.7760. Min Range: -1.7760

Max Act: 0.7284. Min Act: -0.0000

Data Index: 1311623 (The Pile)

Max Activating Token Index: 345

Text #13

Max Range: 1.7760. Min Range: -1.7760

Max Act: 0.7284. Min Act: -0.0000

Data Index: 1455633 (The Pile)

Max Activating Token Index: 345

Text #14

Max Range: 1.7760. Min Range: -1.7760

Max Act: 0.7024. Min Act: -0.0000

Data Index: 698970 (The Pile)

Max Activating Token Index: 164

Text #15

Max Range: 1.7760. Min Range: -1.7760

Max Act: 0.7622. Min Act: -0.0000

Data Index: 1894239 (The Pile)

Max Activating Token Index: 307

Text #16

Max Range: 1.7760. Min Range: -1.7760

Max Act: 0.7578. Min Act: -0.0000

Data Index: 1334806 (The Pile)

Max Activating Token Index: 366

Text #17

Max Range: 1.7760. Min Range: -1.7760

Max Act: 0.6643. Min Act: -0.0000

Data Index: 1167915 (The Pile)

Max Activating Token Index: 420

Text #18

Max Range: 1.7760. Min Range: -1.7760

Max Act: 0.6702. Min Act: -0.0000

Data Index: 1803226 (The Pile)

Max Activating Token Index: 968

Text #19

Max Range: 1.7760. Min Range: -1.7760

Max Act: 0.6436. Min Act: -0.0000

Data Index: 1682240 (The Pile)

Max Activating Token Index: 991