Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 158 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 1.2332. Min Range: -1.2332

Max Act: 1.2332. Min Act: -0.0000

Data Index: 1054712 (The Pile)

Max Activating Token Index: 1007

Text #1

Max Range: 1.2332. Min Range: -1.2332

Max Act: 1.1262. Min Act: -0.0000

Data Index: 1521546 (The Pile)

Max Activating Token Index: 460

Text #2

Max Range: 1.2332. Min Range: -1.2332

Max Act: 1.0311. Min Act: -0.0000

Data Index: 1598594 (The Pile)

Max Activating Token Index: 416

Text #3

Max Range: 1.2332. Min Range: -1.2332

Max Act: 1.1390. Min Act: -0.0000

Data Index: 796957 (The Pile)

Max Activating Token Index: 488

Text #4

Max Range: 1.2332. Min Range: -1.2332

Max Act: 1.0659. Min Act: -0.0000

Data Index: 1998149 (The Pile)

Max Activating Token Index: 800

Text #5

Max Range: 1.2332. Min Range: -1.2332

Max Act: 1.0339. Min Act: -0.0000

Data Index: 514429 (The Pile)

Max Activating Token Index: 521

Text #6

Max Range: 1.2332. Min Range: -1.2332

Max Act: 1.0720. Min Act: -0.0000

Data Index: 167802 (The Pile)

Max Activating Token Index: 851

Text #7

Max Range: 1.2332. Min Range: -1.2332

Max Act: 0.9549. Min Act: -0.0000

Data Index: 1217922 (The Pile)

Max Activating Token Index: 149

Text #8

Max Range: 1.2332. Min Range: -1.2332

Max Act: 0.8253. Min Act: -0.0000

Data Index: 404699 (The Pile)

Max Activating Token Index: 681

Text #9

Max Range: 1.2332. Min Range: -1.2332

Max Act: 1.0092. Min Act: -0.0000

Data Index: 1633707 (The Pile)

Max Activating Token Index: 899

Text #10

Max Range: 1.2332. Min Range: -1.2332

Max Act: 0.9636. Min Act: -0.0000

Data Index: 1415907 (The Pile)

Max Activating Token Index: 164

Text #11

Max Range: 1.2332. Min Range: -1.2332

Max Act: 1.0211. Min Act: -0.0000

Data Index: 1207956 (The Pile)

Max Activating Token Index: 62

Text #12

Max Range: 1.2332. Min Range: -1.2332

Max Act: 0.8839. Min Act: -0.0000

Data Index: 1294576 (The Pile)

Max Activating Token Index: 669

Text #13

Max Range: 1.2332. Min Range: -1.2332

Max Act: 0.9598. Min Act: -0.0000

Data Index: 598032 (The Pile)

Max Activating Token Index: 241

Text #14

Max Range: 1.2332. Min Range: -1.2332

Max Act: 0.9821. Min Act: -0.0000

Data Index: 1978330 (The Pile)

Max Activating Token Index: 732

Text #15

Max Range: 1.2332. Min Range: -1.2332

Max Act: 1.0242. Min Act: -0.0000

Data Index: 23494 (The Pile)

Max Activating Token Index: 979

Text #16

Max Range: 1.2332. Min Range: -1.2332

Max Act: 0.9810. Min Act: -0.0000

Data Index: 1465658 (The Pile)

Max Activating Token Index: 666

Text #17

Max Range: 1.2332. Min Range: -1.2332

Max Act: 0.9339. Min Act: -0.0000

Data Index: 57275 (The Pile)

Max Activating Token Index: 714

Text #18

Max Range: 1.2332. Min Range: -1.2332

Max Act: 0.9056. Min Act: -0.0000

Data Index: 45168 (The Pile)

Max Activating Token Index: 584

Text #19

Max Range: 1.2332. Min Range: -1.2332

Max Act: 0.9086. Min Act: -0.0000

Data Index: 560195 (The Pile)

Max Activating Token Index: 967