Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 637 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 3.2612. Min Range: -3.2612

Max Act: 3.2612. Min Act: -0.0000

Data Index: 1075882 (The Pile)

Max Activating Token Index: 452

Text #1

Max Range: 3.2612. Min Range: -3.2612

Max Act: 3.2331. Min Act: -0.0000

Data Index: 552656 (The Pile)

Max Activating Token Index: 342

Text #2

Max Range: 3.2612. Min Range: -3.2612

Max Act: 2.9933. Min Act: -0.0000

Data Index: 820771 (The Pile)

Max Activating Token Index: 933

Text #3

Max Range: 3.2612. Min Range: -3.2612

Max Act: 2.9302. Min Act: -0.0000

Data Index: 1608207 (The Pile)

Max Activating Token Index: 567

Text #4

Max Range: 3.2612. Min Range: -3.2612

Max Act: 3.0033. Min Act: -0.0000

Data Index: 351950 (The Pile)

Max Activating Token Index: 566

Text #5

Max Range: 3.2612. Min Range: -3.2612

Max Act: 2.6929. Min Act: -0.0000

Data Index: 1348288 (The Pile)

Max Activating Token Index: 845

Text #6

Max Range: 3.2612. Min Range: -3.2612

Max Act: 2.8358. Min Act: -0.0000

Data Index: 714324 (The Pile)

Max Activating Token Index: 589

Text #7

Max Range: 3.2612. Min Range: -3.2612

Max Act: 2.8453. Min Act: -0.0000

Data Index: 1259730 (The Pile)

Max Activating Token Index: 913

Text #8

Max Range: 3.2612. Min Range: -3.2612

Max Act: 2.9482. Min Act: -0.0000

Data Index: 58963 (The Pile)

Max Activating Token Index: 976

Text #9

Max Range: 3.2612. Min Range: -3.2612

Max Act: 2.8202. Min Act: -0.0000

Data Index: 653069 (The Pile)

Max Activating Token Index: 310

Text #10

Max Range: 3.2612. Min Range: -3.2612

Max Act: 2.8302. Min Act: -0.0000

Data Index: 12359 (The Pile)

Max Activating Token Index: 832

Text #11

Max Range: 3.2612. Min Range: -3.2612

Max Act: 2.7908. Min Act: -0.0000

Data Index: 117793 (The Pile)

Max Activating Token Index: 858

Text #12

Max Range: 3.2612. Min Range: -3.2612

Max Act: 2.7831. Min Act: -0.0000

Data Index: 1591717 (The Pile)

Max Activating Token Index: 479

Text #13

Max Range: 3.2612. Min Range: -3.2612

Max Act: 2.7712. Min Act: -0.0000

Data Index: 1856745 (The Pile)

Max Activating Token Index: 339

Text #14

Max Range: 3.2612. Min Range: -3.2612

Max Act: 2.6233. Min Act: -0.0000

Data Index: 202032 (The Pile)

Max Activating Token Index: 826

Text #15

Max Range: 3.2612. Min Range: -3.2612

Max Act: 2.9106. Min Act: -0.0000

Data Index: 909095 (The Pile)

Max Activating Token Index: 953

Text #16

Max Range: 3.2612. Min Range: -3.2612

Max Act: 2.7551. Min Act: -0.0000

Data Index: 1003411 (The Pile)

Max Activating Token Index: 519

Text #17

Max Range: 3.2612. Min Range: -3.2612

Max Act: 2.5961. Min Act: -0.0000

Data Index: 658293 (The Pile)

Max Activating Token Index: 1021

Text #18

Max Range: 3.2612. Min Range: -3.2612

Max Act: 2.7235. Min Act: -0.0000

Data Index: 776252 (The Pile)

Max Activating Token Index: 416

Text #19

Max Range: 3.2612. Min Range: -3.2612

Max Act: 2.6987. Min Act: -0.0000

Data Index: 279535 (The Pile)

Max Activating Token Index: 457