Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 97 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 5.2958. Min Range: -5.2958

Max Act: 5.2958. Min Act: -0.0000

Data Index: 1410209 (The Pile)

Max Activating Token Index: 801

Text #1

Max Range: 5.2958. Min Range: -5.2958

Max Act: 4.9447. Min Act: -0.0000

Data Index: 1674499 (The Pile)

Max Activating Token Index: 532

Text #2

Max Range: 5.2958. Min Range: -5.2958

Max Act: 5.1357. Min Act: -0.0000

Data Index: 1434645 (The Pile)

Max Activating Token Index: 1000

Text #3

Max Range: 5.2958. Min Range: -5.2958

Max Act: 4.7688. Min Act: -0.0000

Data Index: 1595087 (The Pile)

Max Activating Token Index: 727

Text #4

Max Range: 5.2958. Min Range: -5.2958

Max Act: 4.9023. Min Act: -0.0000

Data Index: 931043 (The Pile)

Max Activating Token Index: 852

Text #5

Max Range: 5.2958. Min Range: -5.2958

Max Act: 4.5237. Min Act: -0.0000

Data Index: 1661648 (The Pile)

Max Activating Token Index: 176

Text #6

Max Range: 5.2958. Min Range: -5.2958

Max Act: 4.2104. Min Act: -0.0000

Data Index: 1530124 (The Pile)

Max Activating Token Index: 683

Text #7

Max Range: 5.2958. Min Range: -5.2958

Max Act: 4.0982. Min Act: -0.0000

Data Index: 282243 (The Pile)

Max Activating Token Index: 195

Text #8

Max Range: 5.2958. Min Range: -5.2958

Max Act: 4.0559. Min Act: -0.0000

Data Index: 1240926 (The Pile)

Max Activating Token Index: 273

Text #9

Max Range: 5.2958. Min Range: -5.2958

Max Act: 4.2409. Min Act: -0.0000

Data Index: 446049 (The Pile)

Max Activating Token Index: 720

Text #10

Max Range: 5.2958. Min Range: -5.2958

Max Act: 3.9325. Min Act: -0.0000

Data Index: 17243 (The Pile)

Max Activating Token Index: 63

Text #11

Max Range: 5.2958. Min Range: -5.2958

Max Act: 3.7358. Min Act: -0.0000

Data Index: 68301 (The Pile)

Max Activating Token Index: 20

Text #12

Max Range: 5.2958. Min Range: -5.2958

Max Act: 3.8868. Min Act: -0.0000

Data Index: 427204 (The Pile)

Max Activating Token Index: 928

Text #13

Max Range: 5.2958. Min Range: -5.2958

Max Act: 3.8497. Min Act: -0.0000

Data Index: 1407282 (The Pile)

Max Activating Token Index: 495

Text #14

Max Range: 5.2958. Min Range: -5.2958

Max Act: 3.5995. Min Act: -0.0000

Data Index: 269217 (The Pile)

Max Activating Token Index: 611

Text #15

Max Range: 5.2958. Min Range: -5.2958

Max Act: 3.5060. Min Act: -0.0000

Data Index: 1590802 (The Pile)

Max Activating Token Index: 656

Text #16

Max Range: 5.2958. Min Range: -5.2958

Max Act: 3.6758. Min Act: -0.0000

Data Index: 694982 (The Pile)

Max Activating Token Index: 193

Text #17

Max Range: 5.2958. Min Range: -5.2958

Max Act: 3.8151. Min Act: -0.0000

Data Index: 220715 (The Pile)

Max Activating Token Index: 318

Text #18

Max Range: 5.2958. Min Range: -5.2958

Max Act: 3.7933. Min Act: -0.0000

Data Index: 373629 (The Pile)

Max Activating Token Index: 232

Text #19

Max Range: 5.2958. Min Range: -5.2958

Max Act: 3.5983. Min Act: -0.0000

Data Index: 366041 (The Pile)

Max Activating Token Index: 891