Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 3099 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 1.9082. Min Range: -1.9082

Max Act: 1.9082. Min Act: -0.0000

Data Index: 1975037 (The Pile)

Max Activating Token Index: 786

Text #1

Max Range: 1.9082. Min Range: -1.9082

Max Act: 1.7507. Min Act: -0.0000

Data Index: 1848835 (The Pile)

Max Activating Token Index: 743

Text #2

Max Range: 1.9082. Min Range: -1.9082

Max Act: 1.6383. Min Act: -0.0000

Data Index: 1490849 (The Pile)

Max Activating Token Index: 321

Text #3

Max Range: 1.9082. Min Range: -1.9082

Max Act: 1.5224. Min Act: -0.0000

Data Index: 441971 (The Pile)

Max Activating Token Index: 308

Text #4

Max Range: 1.9082. Min Range: -1.9082

Max Act: 1.5865. Min Act: -0.0000

Data Index: 690289 (The Pile)

Max Activating Token Index: 992

Text #5

Max Range: 1.9082. Min Range: -1.9082

Max Act: 1.4794. Min Act: -0.0000

Data Index: 1551497 (The Pile)

Max Activating Token Index: 311

Text #6

Max Range: 1.9082. Min Range: -1.9082

Max Act: 1.3999. Min Act: -0.0000

Data Index: 1153645 (The Pile)

Max Activating Token Index: 732

Text #7

Max Range: 1.9082. Min Range: -1.9082

Max Act: 1.3958. Min Act: -0.0000

Data Index: 485178 (The Pile)

Max Activating Token Index: 135

Text #8

Max Range: 1.9082. Min Range: -1.9082

Max Act: 1.3043. Min Act: -0.0000

Data Index: 1222098 (The Pile)

Max Activating Token Index: 755

Text #9

Max Range: 1.9082. Min Range: -1.9082

Max Act: 1.4466. Min Act: -0.0000

Data Index: 250242 (The Pile)

Max Activating Token Index: 26

Text #10

Max Range: 1.9082. Min Range: -1.9082

Max Act: 1.3409. Min Act: -0.0000

Data Index: 1414336 (The Pile)

Max Activating Token Index: 450

Text #11

Max Range: 1.9082. Min Range: -1.9082

Max Act: 1.3020. Min Act: -0.0000

Data Index: 1551153 (The Pile)

Max Activating Token Index: 1023

Text #12

Max Range: 1.9082. Min Range: -1.9082

Max Act: 1.2941. Min Act: -0.0000

Data Index: 1578576 (The Pile)

Max Activating Token Index: 1003

Text #13

Max Range: 1.9082. Min Range: -1.9082

Max Act: 1.2790. Min Act: -0.0000

Data Index: 1246986 (The Pile)

Max Activating Token Index: 465

Text #14

Max Range: 1.9082. Min Range: -1.9082

Max Act: 1.2545. Min Act: -0.0000

Data Index: 19822 (The Pile)

Max Activating Token Index: 352

Text #15

Max Range: 1.9082. Min Range: -1.9082

Max Act: 1.2538. Min Act: -0.0000

Data Index: 1866890 (The Pile)

Max Activating Token Index: 755

Text #16

Max Range: 1.9082. Min Range: -1.9082

Max Act: 1.1676. Min Act: -0.0000

Data Index: 1363541 (The Pile)

Max Activating Token Index: 836

Text #17

Max Range: 1.9082. Min Range: -1.9082

Max Act: 1.2193. Min Act: -0.0000

Data Index: 441370 (The Pile)

Max Activating Token Index: 486

Text #18

Max Range: 1.9082. Min Range: -1.9082

Max Act: 1.1232. Min Act: -0.0000

Data Index: 836780 (The Pile)

Max Activating Token Index: 643

Text #19

Max Range: 1.9082. Min Range: -1.9082

Max Act: 1.1947. Min Act: -0.0000

Data Index: 397904 (The Pile)

Max Activating Token Index: 888