Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 1900 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 3.6951. Min Range: -3.6951

Max Act: 3.6951. Min Act: -0.0000

Data Index: 551506 (The Pile)

Max Activating Token Index: 520

Text #1

Max Range: 3.6951. Min Range: -3.6951

Max Act: 3.4095. Min Act: -0.0000

Data Index: 134343 (The Pile)

Max Activating Token Index: 284

Text #2

Max Range: 3.6951. Min Range: -3.6951

Max Act: 2.6527. Min Act: -0.0000

Data Index: 680197 (The Pile)

Max Activating Token Index: 671

Text #3

Max Range: 3.6951. Min Range: -3.6951

Max Act: 2.2596. Min Act: -0.0000

Data Index: 636587 (The Pile)

Max Activating Token Index: 1012

Text #4

Max Range: 3.6951. Min Range: -3.6951

Max Act: 2.6099. Min Act: -0.0000

Data Index: 1297096 (The Pile)

Max Activating Token Index: 383

Text #5

Max Range: 3.6951. Min Range: -3.6951

Max Act: 2.3974. Min Act: -0.0000

Data Index: 1972871 (The Pile)

Max Activating Token Index: 248

Text #6

Max Range: 3.6951. Min Range: -3.6951

Max Act: 2.5585. Min Act: -0.0000

Data Index: 1094841 (The Pile)

Max Activating Token Index: 552

Text #7

Max Range: 3.6951. Min Range: -3.6951

Max Act: 2.6436. Min Act: -0.0000

Data Index: 419267 (The Pile)

Max Activating Token Index: 211

Text #8

Max Range: 3.6951. Min Range: -3.6951

Max Act: 2.4666. Min Act: -0.0000

Data Index: 1454173 (The Pile)

Max Activating Token Index: 401

Text #9

Max Range: 3.6951. Min Range: -3.6951

Max Act: 2.4715. Min Act: -0.0000

Data Index: 1062264 (The Pile)

Max Activating Token Index: 152

Text #10

Max Range: 3.6951. Min Range: -3.6951

Max Act: 2.0984. Min Act: -0.0000

Data Index: 1208489 (The Pile)

Max Activating Token Index: 703

Text #11

Max Range: 3.6951. Min Range: -3.6951

Max Act: 2.2063. Min Act: -0.0000

Data Index: 222281 (The Pile)

Max Activating Token Index: 288

Text #12

Max Range: 3.6951. Min Range: -3.6951

Max Act: 2.2650. Min Act: -0.0000

Data Index: 1871963 (The Pile)

Max Activating Token Index: 323

Text #13

Max Range: 3.6951. Min Range: -3.6951

Max Act: 2.1032. Min Act: -0.0000

Data Index: 1626231 (The Pile)

Max Activating Token Index: 1009

Text #14

Max Range: 3.6951. Min Range: -3.6951

Max Act: 2.2115. Min Act: -0.0000

Data Index: 937505 (The Pile)

Max Activating Token Index: 40

Text #15

Max Range: 3.6951. Min Range: -3.6951

Max Act: 1.9647. Min Act: -0.0000

Data Index: 752654 (The Pile)

Max Activating Token Index: 559

Text #16

Max Range: 3.6951. Min Range: -3.6951

Max Act: 2.1714. Min Act: -0.0000

Data Index: 221099 (The Pile)

Max Activating Token Index: 607

Text #17

Max Range: 3.6951. Min Range: -3.6951

Max Act: 2.1519. Min Act: -0.0000

Data Index: 1691524 (The Pile)

Max Activating Token Index: 381

Text #18

Max Range: 3.6951. Min Range: -3.6951

Max Act: 2.0244. Min Act: -0.0000

Data Index: 410097 (The Pile)

Max Activating Token Index: 569

Text #19

Max Range: 3.6951. Min Range: -3.6951

Max Act: 2.0723. Min Act: -0.0000

Data Index: 361051 (The Pile)

Max Activating Token Index: 238