Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 3525 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 5.2004. Min Range: -5.2004

Max Act: 5.2004. Min Act: -0.0000

Data Index: 990818 (The Pile)

Max Activating Token Index: 986

Text #1

Max Range: 5.2004. Min Range: -5.2004

Max Act: 4.3150. Min Act: -0.0000

Data Index: 1168401 (The Pile)

Max Activating Token Index: 116

Text #2

Max Range: 5.2004. Min Range: -5.2004

Max Act: 4.4521. Min Act: -0.0000

Data Index: 1937721 (The Pile)

Max Activating Token Index: 869

Text #3

Max Range: 5.2004. Min Range: -5.2004

Max Act: 4.2084. Min Act: -0.0000

Data Index: 467981 (The Pile)

Max Activating Token Index: 52

Text #4

Max Range: 5.2004. Min Range: -5.2004

Max Act: 4.0694. Min Act: -0.0000

Data Index: 915628 (The Pile)

Max Activating Token Index: 777

Text #5

Max Range: 5.2004. Min Range: -5.2004

Max Act: 4.0472. Min Act: -0.0000

Data Index: 350439 (The Pile)

Max Activating Token Index: 630

Text #6

Max Range: 5.2004. Min Range: -5.2004

Max Act: 4.1325. Min Act: -0.0000

Data Index: 1652647 (The Pile)

Max Activating Token Index: 32

Text #7

Max Range: 5.2004. Min Range: -5.2004

Max Act: 3.6159. Min Act: -0.0000

Data Index: 815786 (The Pile)

Max Activating Token Index: 965

Text #8

Max Range: 5.2004. Min Range: -5.2004

Max Act: 3.6909. Min Act: -0.0000

Data Index: 860117 (The Pile)

Max Activating Token Index: 964

Text #9

Max Range: 5.2004. Min Range: -5.2004

Max Act: 3.6589. Min Act: -0.0000

Data Index: 1635295 (The Pile)

Max Activating Token Index: 941

Text #10

Max Range: 5.2004. Min Range: -5.2004

Max Act: 3.6257. Min Act: -0.0000

Data Index: 1310934 (The Pile)

Max Activating Token Index: 710

Text #11

Max Range: 5.2004. Min Range: -5.2004

Max Act: 4.0527. Min Act: -0.0000

Data Index: 1084568 (The Pile)

Max Activating Token Index: 887

Text #12

Max Range: 5.2004. Min Range: -5.2004

Max Act: 3.5471. Min Act: -0.0000

Data Index: 1154104 (The Pile)

Max Activating Token Index: 679

Text #13

Max Range: 5.2004. Min Range: -5.2004

Max Act: 3.8553. Min Act: -0.0000

Data Index: 1668265 (The Pile)

Max Activating Token Index: 892

Text #14

Max Range: 5.2004. Min Range: -5.2004

Max Act: 3.6497. Min Act: -0.0000

Data Index: 1679678 (The Pile)

Max Activating Token Index: 238

Text #15

Max Range: 5.2004. Min Range: -5.2004

Max Act: 3.4685. Min Act: -0.0000

Data Index: 1078802 (The Pile)

Max Activating Token Index: 405

Text #16

Max Range: 5.2004. Min Range: -5.2004

Max Act: 4.2796. Min Act: -0.0000

Data Index: 1411284 (The Pile)

Max Activating Token Index: 832

Text #17

Max Range: 5.2004. Min Range: -5.2004

Max Act: 3.7828. Min Act: -0.0000

Data Index: 1331893 (The Pile)

Max Activating Token Index: 809

Text #18

Max Range: 5.2004. Min Range: -5.2004

Max Act: 3.5550. Min Act: -0.0000

Data Index: 1586434 (The Pile)

Max Activating Token Index: 672

Text #19

Max Range: 5.2004. Min Range: -5.2004

Max Act: 3.7073. Min Act: -0.0000

Data Index: 1751419 (The Pile)

Max Activating Token Index: 591