Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: The Pile

Neuron 700 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l-pile')

Text #0

Max Range: 4.2628. Min Range: -4.2628

Max Act: 4.2628. Min Act: -0.0000

Data Index: 1832757 (The Pile)

Max Activating Token Index: 574

Text #1

Max Range: 4.2628. Min Range: -4.2628

Max Act: 3.7360. Min Act: -0.0000

Data Index: 1268516 (The Pile)

Max Activating Token Index: 454

Text #2

Max Range: 4.2628. Min Range: -4.2628

Max Act: 3.0930. Min Act: -0.0000

Data Index: 997493 (The Pile)

Max Activating Token Index: 51

Text #3

Max Range: 4.2628. Min Range: -4.2628

Max Act: 3.6327. Min Act: -0.0000

Data Index: 1854038 (The Pile)

Max Activating Token Index: 989

Text #4

Max Range: 4.2628. Min Range: -4.2628

Max Act: 3.2371. Min Act: -0.0000

Data Index: 1438612 (The Pile)

Max Activating Token Index: 238

Text #5

Max Range: 4.2628. Min Range: -4.2628

Max Act: 3.2611. Min Act: -0.0000

Data Index: 5328 (The Pile)

Max Activating Token Index: 210

Text #6

Max Range: 4.2628. Min Range: -4.2628

Max Act: 2.7905. Min Act: -0.0000

Data Index: 1276805 (The Pile)

Max Activating Token Index: 82

Text #7

Max Range: 4.2628. Min Range: -4.2628

Max Act: 3.0637. Min Act: -0.0000

Data Index: 1326481 (The Pile)

Max Activating Token Index: 443

Text #8

Max Range: 4.2628. Min Range: -4.2628

Max Act: 2.8317. Min Act: -0.0000

Data Index: 332690 (The Pile)

Max Activating Token Index: 52

Text #9

Max Range: 4.2628. Min Range: -4.2628

Max Act: 3.2221. Min Act: -0.0000

Data Index: 1577779 (The Pile)

Max Activating Token Index: 298

Text #10

Max Range: 4.2628. Min Range: -4.2628

Max Act: 3.1906. Min Act: -0.0000

Data Index: 1143528 (The Pile)

Max Activating Token Index: 778

Text #11

Max Range: 4.2628. Min Range: -4.2628

Max Act: 3.8348. Min Act: -0.0000

Data Index: 103860 (The Pile)

Max Activating Token Index: 142

Text #12

Max Range: 4.2628. Min Range: -4.2628

Max Act: 3.3274. Min Act: -0.0000

Data Index: 1131907 (The Pile)

Max Activating Token Index: 44

Text #13

Max Range: 4.2628. Min Range: -4.2628

Max Act: 3.2922. Min Act: -0.0000

Data Index: 1706949 (The Pile)

Max Activating Token Index: 716

Text #14

Max Range: 4.2628. Min Range: -4.2628

Max Act: 3.0149. Min Act: -0.0000

Data Index: 1116420 (The Pile)

Max Activating Token Index: 392

Text #15

Max Range: 4.2628. Min Range: -4.2628

Max Act: 2.9985. Min Act: -0.0000

Data Index: 383198 (The Pile)

Max Activating Token Index: 184

Text #16

Max Range: 4.2628. Min Range: -4.2628

Max Act: 2.9621. Min Act: -0.0000

Data Index: 569928 (The Pile)

Max Activating Token Index: 795

Text #17

Max Range: 4.2628. Min Range: -4.2628

Max Act: 2.8244. Min Act: -0.0000

Data Index: 1319138 (The Pile)

Max Activating Token Index: 541

Text #18

Max Range: 4.2628. Min Range: -4.2628

Max Act: 2.6606. Min Act: -0.0000

Data Index: 1625632 (The Pile)

Max Activating Token Index: 376

Text #19

Max Range: 4.2628. Min Range: -4.2628

Max Act: 2.9899. Min Act: -0.0000

Data Index: 1655779 (The Pile)

Max Activating Token Index: 562