Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 4271 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 4.8801. Min Range: -4.8801

Max Act: 4.8801. Min Act: -0.0000

Data Index: 1460948 (The Pile)

Max Activating Token Index: 267

Text #1

Max Range: 4.8801. Min Range: -4.8801

Max Act: 4.4186. Min Act: -0.0000

Data Index: 1022121 (The Pile)

Max Activating Token Index: 706

Text #2

Max Range: 4.8801. Min Range: -4.8801

Max Act: 3.6339. Min Act: -0.0000

Data Index: 1556325 (The Pile)

Max Activating Token Index: 630

Text #3

Max Range: 4.8801. Min Range: -4.8801

Max Act: 3.9489. Min Act: -0.0000

Data Index: 1798021 (The Pile)

Max Activating Token Index: 65

Text #4

Max Range: 4.8801. Min Range: -4.8801

Max Act: 4.5241. Min Act: -0.0000

Data Index: 878537 (The Pile)

Max Activating Token Index: 972

Text #5

Max Range: 4.8801. Min Range: -4.8801

Max Act: 4.0247. Min Act: -0.0000

Data Index: 1220876 (The Pile)

Max Activating Token Index: 535

Text #6

Max Range: 4.8801. Min Range: -4.8801

Max Act: 3.7860. Min Act: -0.0000

Data Index: 520086 (The Pile)

Max Activating Token Index: 878

Text #7

Max Range: 4.8801. Min Range: -4.8801

Max Act: 3.9140. Min Act: -0.0000

Data Index: 781011 (The Pile)

Max Activating Token Index: 1007

Text #8

Max Range: 4.8801. Min Range: -4.8801

Max Act: 4.1834. Min Act: -0.0000

Data Index: 701748 (The Pile)

Max Activating Token Index: 969

Text #9

Max Range: 4.8801. Min Range: -4.8801

Max Act: 3.7328. Min Act: -0.0000

Data Index: 388356 (The Pile)

Max Activating Token Index: 74

Text #10

Max Range: 4.8801. Min Range: -4.8801

Max Act: 3.7279. Min Act: -0.0000

Data Index: 1831687 (The Pile)

Max Activating Token Index: 709

Text #11

Max Range: 4.8801. Min Range: -4.8801

Max Act: 3.6735. Min Act: -0.0000

Data Index: 1717284 (The Pile)

Max Activating Token Index: 590

Text #12

Max Range: 4.8801. Min Range: -4.8801

Max Act: 3.7629. Min Act: -0.0000

Data Index: 1473807 (The Pile)

Max Activating Token Index: 660

Text #13

Max Range: 4.8801. Min Range: -4.8801

Max Act: 3.5583. Min Act: -0.0000

Data Index: 1505536 (The Pile)

Max Activating Token Index: 973

Text #14

Max Range: 4.8801. Min Range: -4.8801

Max Act: 3.5382. Min Act: -0.0000

Data Index: 368257 (The Pile)

Max Activating Token Index: 358

Text #15

Max Range: 4.8801. Min Range: -4.8801

Max Act: 3.5536. Min Act: -0.0000

Data Index: 417192 (The Pile)

Max Activating Token Index: 655

Text #16

Max Range: 4.8801. Min Range: -4.8801

Max Act: 3.2043. Min Act: -0.0000

Data Index: 1372877 (The Pile)

Max Activating Token Index: 978

Text #17

Max Range: 4.8801. Min Range: -4.8801

Max Act: 3.9363. Min Act: -0.0000

Data Index: 582273 (The Pile)

Max Activating Token Index: 1011

Text #18

Max Range: 4.8801. Min Range: -4.8801

Max Act: 3.4588. Min Act: -0.0000

Data Index: 1115462 (The Pile)

Max Activating Token Index: 481

Text #19

Max Range: 4.8801. Min Range: -4.8801

Max Act: 3.4482. Min Act: -0.0000

Data Index: 1723479 (The Pile)

Max Activating Token Index: 659