Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 2620 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 0.8222. Min Range: -0.8222

Max Act: 0.8222. Min Act: -0.0000

Data Index: 1234220 (The Pile)

Max Activating Token Index: 320

Text #1

Max Range: 0.8222. Min Range: -0.8222

Max Act: 0.7665. Min Act: -0.0000

Data Index: 648665 (The Pile)

Max Activating Token Index: 404

Text #2

Max Range: 0.8222. Min Range: -0.8222

Max Act: 0.7475. Min Act: -0.0000

Data Index: 1841224 (The Pile)

Max Activating Token Index: 34

Text #3

Max Range: 0.8222. Min Range: -0.8222

Max Act: 0.7288. Min Act: -0.0000

Data Index: 1500749 (The Pile)

Max Activating Token Index: 57

Text #4

Max Range: 0.8222. Min Range: -0.8222

Max Act: 0.6392. Min Act: -0.0000

Data Index: 1892545 (The Pile)

Max Activating Token Index: 408

Text #5

Max Range: 0.8222. Min Range: -0.8222

Max Act: 0.7047. Min Act: -0.0000

Data Index: 1449247 (The Pile)

Max Activating Token Index: 1010

Text #6

Max Range: 0.8222. Min Range: -0.8222

Max Act: 0.6622. Min Act: -0.0000

Data Index: 1397235 (The Pile)

Max Activating Token Index: 232

Text #7

Max Range: 0.8222. Min Range: -0.8222

Max Act: 0.6078. Min Act: -0.0000

Data Index: 746306 (The Pile)

Max Activating Token Index: 19

Text #8

Max Range: 0.8222. Min Range: -0.8222

Max Act: 0.6089. Min Act: -0.0000

Data Index: 1172593 (The Pile)

Max Activating Token Index: 510

Text #9

Max Range: 0.8222. Min Range: -0.8222

Max Act: 0.5866. Min Act: -0.0000

Data Index: 1210044 (The Pile)

Max Activating Token Index: 233

Text #10

Max Range: 0.8222. Min Range: -0.8222

Max Act: 0.6210. Min Act: -0.0000

Data Index: 935998 (The Pile)

Max Activating Token Index: 728

Text #11

Max Range: 0.8222. Min Range: -0.8222

Max Act: 0.6181. Min Act: -0.0000

Data Index: 705165 (The Pile)

Max Activating Token Index: 635

Text #12

Max Range: 0.8222. Min Range: -0.8222

Max Act: 0.5745. Min Act: -0.0000

Data Index: 34542 (The Pile)

Max Activating Token Index: 641

Text #13

Max Range: 0.8222. Min Range: -0.8222

Max Act: 0.5639. Min Act: -0.0000

Data Index: 1623478 (The Pile)

Max Activating Token Index: 456

Text #14

Max Range: 0.8222. Min Range: -0.8222

Max Act: 0.5481. Min Act: -0.0000

Data Index: 105168 (The Pile)

Max Activating Token Index: 240

Text #15

Max Range: 0.8222. Min Range: -0.8222

Max Act: 0.6037. Min Act: -0.0000

Data Index: 274224 (The Pile)

Max Activating Token Index: 451

Text #16

Max Range: 0.8222. Min Range: -0.8222

Max Act: 0.5335. Min Act: -0.0000

Data Index: 361734 (The Pile)

Max Activating Token Index: 730

Text #17

Max Range: 0.8222. Min Range: -0.8222

Max Act: 0.5658. Min Act: -0.0000

Data Index: 7477 (The Pile)

Max Activating Token Index: 921

Text #18

Max Range: 0.8222. Min Range: -0.8222

Max Act: 0.5757. Min Act: -0.0000

Data Index: 428310 (The Pile)

Max Activating Token Index: 464

Text #19

Max Range: 0.8222. Min Range: -0.8222

Max Act: 0.5228. Min Act: -0.0000

Data Index: 568501 (The Pile)

Max Activating Token Index: 213