Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 4049 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 1.1213. Min Range: -1.1213

Max Act: 1.1213. Min Act: -0.0000

Data Index: 1791608 (The Pile)

Max Activating Token Index: 824

Text #1

Max Range: 1.1213. Min Range: -1.1213

Max Act: 1.0440. Min Act: -0.0000

Data Index: 152059 (The Pile)

Max Activating Token Index: 572

Text #2

Max Range: 1.1213. Min Range: -1.1213

Max Act: 0.7615. Min Act: -0.0000

Data Index: 412679 (The Pile)

Max Activating Token Index: 461

Text #3

Max Range: 1.1213. Min Range: -1.1213

Max Act: 0.7507. Min Act: -0.0000

Data Index: 16007 (The Pile)

Max Activating Token Index: 486

Text #4

Max Range: 1.1213. Min Range: -1.1213

Max Act: 0.6433. Min Act: -0.0000

Data Index: 1903079 (The Pile)

Max Activating Token Index: 836

Text #5

Max Range: 1.1213. Min Range: -1.1213

Max Act: 0.6961. Min Act: -0.0000

Data Index: 32741 (The Pile)

Max Activating Token Index: 713

Text #6

Max Range: 1.1213. Min Range: -1.1213

Max Act: 0.6744. Min Act: -0.0000

Data Index: 507161 (The Pile)

Max Activating Token Index: 113

Text #7

Max Range: 1.1213. Min Range: -1.1213

Max Act: 0.6443. Min Act: -0.0000

Data Index: 267541 (The Pile)

Max Activating Token Index: 223

Text #8

Max Range: 1.1213. Min Range: -1.1213

Max Act: 0.5913. Min Act: -0.0000

Data Index: 431655 (The Pile)

Max Activating Token Index: 613

Text #9

Max Range: 1.1213. Min Range: -1.1213

Max Act: 0.5786. Min Act: -0.0000

Data Index: 182647 (The Pile)

Max Activating Token Index: 834

Text #10

Max Range: 1.1213. Min Range: -1.1213

Max Act: 0.5631. Min Act: -0.0000

Data Index: 231839 (The Pile)

Max Activating Token Index: 288

Text #11

Max Range: 1.1213. Min Range: -1.1213

Max Act: 0.6112. Min Act: -0.0000

Data Index: 94038 (The Pile)

Max Activating Token Index: 628

Text #12

Max Range: 1.1213. Min Range: -1.1213

Max Act: 0.5336. Min Act: -0.0000

Data Index: 1336059 (The Pile)

Max Activating Token Index: 942

Text #13

Max Range: 1.1213. Min Range: -1.1213

Max Act: 0.5715. Min Act: -0.0000

Data Index: 1880971 (The Pile)

Max Activating Token Index: 841

Text #14

Max Range: 1.1213. Min Range: -1.1213

Max Act: 0.5219. Min Act: -0.0000

Data Index: 170746 (The Pile)

Max Activating Token Index: 414

Text #15

Max Range: 1.1213. Min Range: -1.1213

Max Act: 0.5426. Min Act: -0.0000

Data Index: 1357405 (The Pile)

Max Activating Token Index: 620

Text #16

Max Range: 1.1213. Min Range: -1.1213

Max Act: 0.5270. Min Act: -0.0000

Data Index: 1869988 (The Pile)

Max Activating Token Index: 900

Text #17

Max Range: 1.1213. Min Range: -1.1213

Max Act: 0.5348. Min Act: -0.0000

Data Index: 209362 (The Pile)

Max Activating Token Index: 970

Text #18

Max Range: 1.1213. Min Range: -1.1213

Max Act: 0.5452. Min Act: -0.0000

Data Index: 58771 (The Pile)

Max Activating Token Index: 827

Text #19

Max Range: 1.1213. Min Range: -1.1213

Max Act: 0.5293. Min Act: -0.0000

Data Index: 970670 (The Pile)

Max Activating Token Index: 857