Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 4097 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 1.5173. Min Range: -1.5173

Max Act: 1.5173. Min Act: -0.0000

Data Index: 1858048 (The Pile)

Max Activating Token Index: 708

Text #1

Max Range: 1.5173. Min Range: -1.5173

Max Act: 1.4204. Min Act: -0.0000

Data Index: 1490411 (The Pile)

Max Activating Token Index: 787

Text #2

Max Range: 1.5173. Min Range: -1.5173

Max Act: 1.0821. Min Act: -0.0000

Data Index: 1132064 (The Pile)

Max Activating Token Index: 202

Text #3

Max Range: 1.5173. Min Range: -1.5173

Max Act: 0.9135. Min Act: -0.0000

Data Index: 1317080 (The Pile)

Max Activating Token Index: 701

Text #4

Max Range: 1.5173. Min Range: -1.5173

Max Act: 0.9074. Min Act: -0.0000

Data Index: 1704555 (The Pile)

Max Activating Token Index: 697

Text #5

Max Range: 1.5173. Min Range: -1.5173

Max Act: 0.8884. Min Act: -0.0000

Data Index: 1571024 (The Pile)

Max Activating Token Index: 766

Text #6

Max Range: 1.5173. Min Range: -1.5173

Max Act: 0.8696. Min Act: -0.0000

Data Index: 228296 (The Pile)

Max Activating Token Index: 249

Text #7

Max Range: 1.5173. Min Range: -1.5173

Max Act: 0.8228. Min Act: -0.0000

Data Index: 594446 (The Pile)

Max Activating Token Index: 590

Text #8

Max Range: 1.5173. Min Range: -1.5173

Max Act: 0.9051. Min Act: -0.0000

Data Index: 379506 (The Pile)

Max Activating Token Index: 898

Text #9

Max Range: 1.5173. Min Range: -1.5173

Max Act: 0.8591. Min Act: -0.0000

Data Index: 783676 (The Pile)

Max Activating Token Index: 347

Text #10

Max Range: 1.5173. Min Range: -1.5173

Max Act: 0.8232. Min Act: -0.0000

Data Index: 1767346 (The Pile)

Max Activating Token Index: 842

Text #11

Max Range: 1.5173. Min Range: -1.5173

Max Act: 0.7980. Min Act: -0.0000

Data Index: 1922264 (The Pile)

Max Activating Token Index: 657

Text #12

Max Range: 1.5173. Min Range: -1.5173

Max Act: 0.8396. Min Act: -0.0000

Data Index: 217601 (The Pile)

Max Activating Token Index: 921

Text #13

Max Range: 1.5173. Min Range: -1.5173

Max Act: 0.7222. Min Act: -0.0000

Data Index: 652538 (The Pile)

Max Activating Token Index: 1010

Text #14

Max Range: 1.5173. Min Range: -1.5173

Max Act: 0.7971. Min Act: -0.0000

Data Index: 1630077 (The Pile)

Max Activating Token Index: 787

Text #15

Max Range: 1.5173. Min Range: -1.5173

Max Act: 0.7176. Min Act: -0.0000

Data Index: 339124 (The Pile)

Max Activating Token Index: 864

Text #16

Max Range: 1.5173. Min Range: -1.5173

Max Act: 0.7956. Min Act: -0.0000

Data Index: 1138106 (The Pile)

Max Activating Token Index: 945

Text #17

Max Range: 1.5173. Min Range: -1.5173

Max Act: 0.7239. Min Act: -0.0000

Data Index: 450622 (The Pile)

Max Activating Token Index: 646

Text #18

Max Range: 1.5173. Min Range: -1.5173

Max Act: 0.7464. Min Act: -0.0000

Data Index: 503235 (The Pile)

Max Activating Token Index: 732

Text #19

Max Range: 1.5173. Min Range: -1.5173

Max Act: 0.7574. Min Act: -0.0000

Data Index: 490843 (The Pile)

Max Activating Token Index: 340