Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 5074 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 3.5882. Min Range: -3.5882

Max Act: 3.2567. Min Act: -0.0000

Data Index: 641108 (The Pile)

Max Activating Token Index: 464

Text #1

Max Range: 3.5882. Min Range: -3.5882

Max Act: 3.4056. Min Act: -0.0000

Data Index: 266502 (The Pile)

Max Activating Token Index: 393

Text #2

Max Range: 3.5882. Min Range: -3.5882

Max Act: 3.2338. Min Act: -0.0000

Data Index: 1617546 (The Pile)

Max Activating Token Index: 286

Text #3

Max Range: 3.5882. Min Range: -3.5882

Max Act: 3.5640. Min Act: -0.0000

Data Index: 850008 (The Pile)

Max Activating Token Index: 223

Text #4

Max Range: 3.5882. Min Range: -3.5882

Max Act: 3.3710. Min Act: -0.0000

Data Index: 75177 (The Pile)

Max Activating Token Index: 621

Text #5

Max Range: 3.5882. Min Range: -3.5882

Max Act: 3.1314. Min Act: -0.0000

Data Index: 869323 (The Pile)

Max Activating Token Index: 608

Text #6

Max Range: 3.5882. Min Range: -3.5882

Max Act: 3.5882. Min Act: -0.0000

Data Index: 439539 (The Pile)

Max Activating Token Index: 765

Text #7

Max Range: 3.5882. Min Range: -3.5882

Max Act: 3.0730. Min Act: -0.0000

Data Index: 424564 (The Pile)

Max Activating Token Index: 75

Text #8

Max Range: 3.5882. Min Range: -3.5882

Max Act: 3.1690. Min Act: -0.0000

Data Index: 482598 (The Pile)

Max Activating Token Index: 314

Text #9

Max Range: 3.5882. Min Range: -3.5882

Max Act: 3.3363. Min Act: -0.0000

Data Index: 1079299 (The Pile)

Max Activating Token Index: 881

Text #10

Max Range: 3.5882. Min Range: -3.5882

Max Act: 3.1707. Min Act: -0.0000

Data Index: 1339283 (The Pile)

Max Activating Token Index: 797

Text #11

Max Range: 3.5882. Min Range: -3.5882

Max Act: 3.1690. Min Act: -0.0000

Data Index: 91012 (The Pile)

Max Activating Token Index: 518

Text #12

Max Range: 3.5882. Min Range: -3.5882

Max Act: 3.1526. Min Act: -0.0000

Data Index: 707844 (The Pile)

Max Activating Token Index: 374

Text #13

Max Range: 3.5882. Min Range: -3.5882

Max Act: 2.9879. Min Act: -0.0000

Data Index: 1222200 (The Pile)

Max Activating Token Index: 196

Text #14

Max Range: 3.5882. Min Range: -3.5882

Max Act: 2.9185. Min Act: -0.0000

Data Index: 480558 (The Pile)

Max Activating Token Index: 388

Text #15

Max Range: 3.5882. Min Range: -3.5882

Max Act: 2.9159. Min Act: -0.0000

Data Index: 864600 (The Pile)

Max Activating Token Index: 904

Text #16

Max Range: 3.5882. Min Range: -3.5882

Max Act: 2.8963. Min Act: -0.0000

Data Index: 1757957 (The Pile)

Max Activating Token Index: 976

Text #17

Max Range: 3.5882. Min Range: -3.5882

Max Act: 3.0489. Min Act: -0.0000

Data Index: 795779 (The Pile)

Max Activating Token Index: 721

Text #18

Max Range: 3.5882. Min Range: -3.5882

Max Act: 3.0349. Min Act: -0.0000

Data Index: 690640 (The Pile)

Max Activating Token Index: 474

Text #19

Max Range: 3.5882. Min Range: -3.5882

Max Act: 2.8697. Min Act: -0.0000

Data Index: 637258 (The Pile)

Max Activating Token Index: 853