Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 3875 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 5.6222. Min Range: -5.6222

Max Act: 5.4910. Min Act: -0.0000

Data Index: 825123 (The Pile)

Max Activating Token Index: 43

Text #1

Max Range: 5.6222. Min Range: -5.6222

Max Act: 5.4410. Min Act: -0.0000

Data Index: 1523325 (The Pile)

Max Activating Token Index: 261

Text #2

Max Range: 5.6222. Min Range: -5.6222

Max Act: 5.6222. Min Act: -0.0000

Data Index: 1982109 (The Pile)

Max Activating Token Index: 537

Text #3

Max Range: 5.6222. Min Range: -5.6222

Max Act: 5.5459. Min Act: -0.0000

Data Index: 1271301 (The Pile)

Max Activating Token Index: 38

Text #4

Max Range: 5.6222. Min Range: -5.6222

Max Act: 5.1246. Min Act: -0.0000

Data Index: 1812345 (The Pile)

Max Activating Token Index: 361

Text #5

Max Range: 5.6222. Min Range: -5.6222

Max Act: 5.6063. Min Act: -0.0000

Data Index: 1860083 (The Pile)

Max Activating Token Index: 36

Text #6

Max Range: 5.6222. Min Range: -5.6222

Max Act: 5.1986. Min Act: -0.0000

Data Index: 544128 (The Pile)

Max Activating Token Index: 341

Text #7

Max Range: 5.6222. Min Range: -5.6222

Max Act: 5.1925. Min Act: -0.0000

Data Index: 47749 (The Pile)

Max Activating Token Index: 528

Text #8

Max Range: 5.6222. Min Range: -5.6222

Max Act: 5.3811. Min Act: -0.0000

Data Index: 1192563 (The Pile)

Max Activating Token Index: 229

Text #9

Max Range: 5.6222. Min Range: -5.6222

Max Act: 5.0009. Min Act: -0.0000

Data Index: 1960501 (The Pile)

Max Activating Token Index: 140

Text #10

Max Range: 5.6222. Min Range: -5.6222

Max Act: 5.1769. Min Act: -0.0000

Data Index: 1084171 (The Pile)

Max Activating Token Index: 611

Text #11

Max Range: 5.6222. Min Range: -5.6222

Max Act: 5.3440. Min Act: -0.0000

Data Index: 465463 (The Pile)

Max Activating Token Index: 544

Text #12

Max Range: 5.6222. Min Range: -5.6222

Max Act: 5.2878. Min Act: -0.0000

Data Index: 474008 (The Pile)

Max Activating Token Index: 409

Text #13

Max Range: 5.6222. Min Range: -5.6222

Max Act: 5.0998. Min Act: -0.0000

Data Index: 1545097 (The Pile)

Max Activating Token Index: 23

Text #14

Max Range: 5.6222. Min Range: -5.6222

Max Act: 5.0853. Min Act: -0.0000

Data Index: 1597640 (The Pile)

Max Activating Token Index: 203

Text #15

Max Range: 5.6222. Min Range: -5.6222

Max Act: 5.0481. Min Act: -0.0000

Data Index: 1813006 (The Pile)

Max Activating Token Index: 32

Text #16

Max Range: 5.6222. Min Range: -5.6222

Max Act: 5.0208. Min Act: -0.0000

Data Index: 1870285 (The Pile)

Max Activating Token Index: 371

Text #17

Max Range: 5.6222. Min Range: -5.6222

Max Act: 5.2119. Min Act: -0.0000

Data Index: 421316 (The Pile)

Max Activating Token Index: 74

Text #18

Max Range: 5.6222. Min Range: -5.6222

Max Act: 5.0151. Min Act: -0.0000

Data Index: 645330 (The Pile)

Max Activating Token Index: 626

Text #19

Max Range: 5.6222. Min Range: -5.6222

Max Act: 5.3819. Min Act: -0.0000

Data Index: 128210 (The Pile)

Max Activating Token Index: 285