Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: The Pile

Neuron 2404 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l-pile')

Text #0

Max Range: 5.3284. Min Range: -5.3284

Max Act: 5.3284. Min Act: -0.0000

Data Index: 1023911 (The Pile)

Max Activating Token Index: 540

Text #1

Max Range: 5.3284. Min Range: -5.3284

Max Act: 4.8214. Min Act: -0.0000

Data Index: 1665215 (The Pile)

Max Activating Token Index: 197

Text #2

Max Range: 5.3284. Min Range: -5.3284

Max Act: 4.9194. Min Act: -0.0000

Data Index: 1924063 (The Pile)

Max Activating Token Index: 98

Text #3

Max Range: 5.3284. Min Range: -5.3284

Max Act: 4.8560. Min Act: -0.0000

Data Index: 1817223 (The Pile)

Max Activating Token Index: 465

Text #4

Max Range: 5.3284. Min Range: -5.3284

Max Act: 4.1250. Min Act: -0.0000

Data Index: 997113 (The Pile)

Max Activating Token Index: 492

Text #5

Max Range: 5.3284. Min Range: -5.3284

Max Act: 4.1011. Min Act: -0.0000

Data Index: 1417158 (The Pile)

Max Activating Token Index: 118

Text #6

Max Range: 5.3284. Min Range: -5.3284

Max Act: 4.0653. Min Act: -0.0000

Data Index: 1499629 (The Pile)

Max Activating Token Index: 767

Text #7

Max Range: 5.3284. Min Range: -5.3284

Max Act: 3.8289. Min Act: -0.0000

Data Index: 452850 (The Pile)

Max Activating Token Index: 587

Text #8

Max Range: 5.3284. Min Range: -5.3284

Max Act: 3.3383. Min Act: -0.0000

Data Index: 773746 (The Pile)

Max Activating Token Index: 570

Text #9

Max Range: 5.3284. Min Range: -5.3284

Max Act: 3.9112. Min Act: -0.0000

Data Index: 308128 (The Pile)

Max Activating Token Index: 530

Text #10

Max Range: 5.3284. Min Range: -5.3284

Max Act: 3.6929. Min Act: -0.0000

Data Index: 276100 (The Pile)

Max Activating Token Index: 457

Text #11

Max Range: 5.3284. Min Range: -5.3284

Max Act: 3.2466. Min Act: -0.0000

Data Index: 1132210 (The Pile)

Max Activating Token Index: 476

Text #12

Max Range: 5.3284. Min Range: -5.3284

Max Act: 3.5502. Min Act: -0.0000

Data Index: 409064 (The Pile)

Max Activating Token Index: 218

Text #13

Max Range: 5.3284. Min Range: -5.3284

Max Act: 3.5398. Min Act: -0.0000

Data Index: 1265460 (The Pile)

Max Activating Token Index: 647

Text #14

Max Range: 5.3284. Min Range: -5.3284

Max Act: 3.7007. Min Act: -0.0000

Data Index: 346606 (The Pile)

Max Activating Token Index: 551

Text #15

Max Range: 5.3284. Min Range: -5.3284

Max Act: 3.2575. Min Act: -0.0000

Data Index: 1796529 (The Pile)

Max Activating Token Index: 165

Text #16

Max Range: 5.3284. Min Range: -5.3284

Max Act: 3.3931. Min Act: -0.0001

Data Index: 167593 (The Pile)

Max Activating Token Index: 588

Text #17

Max Range: 5.3284. Min Range: -5.3284

Max Act: 3.5786. Min Act: -0.0001

Data Index: 511264 (The Pile)

Max Activating Token Index: 584

Text #18

Max Range: 5.3284. Min Range: -5.3284

Max Act: 3.6547. Min Act: -0.0000

Data Index: 445577 (The Pile)

Max Activating Token Index: 636

Text #19

Max Range: 5.3284. Min Range: -5.3284

Max Act: 3.6539. Min Act: -0.0000

Data Index: 1663005 (The Pile)

Max Activating Token Index: 896