Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: The Pile

Neuron 2020 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l-pile')

Text #0

Max Range: 0.7924. Min Range: -0.7924

Max Act: 0.7924. Min Act: -0.0000

Data Index: 932425 (The Pile)

Max Activating Token Index: 326

Text #1

Max Range: 0.7924. Min Range: -0.7924

Max Act: 0.7220. Min Act: -0.0000

Data Index: 1705190 (The Pile)

Max Activating Token Index: 491

Text #2

Max Range: 0.7924. Min Range: -0.7924

Max Act: 0.6942. Min Act: -0.0000

Data Index: 45002 (The Pile)

Max Activating Token Index: 187

Text #3

Max Range: 0.7924. Min Range: -0.7924

Max Act: 0.6125. Min Act: -0.0000

Data Index: 281636 (The Pile)

Max Activating Token Index: 47

Text #4

Max Range: 0.7924. Min Range: -0.7924

Max Act: 0.6194. Min Act: -0.0000

Data Index: 127625 (The Pile)

Max Activating Token Index: 499

Text #5

Max Range: 0.7924. Min Range: -0.7924

Max Act: 0.6197. Min Act: -0.0000

Data Index: 1136962 (The Pile)

Max Activating Token Index: 627

Text #6

Max Range: 0.7924. Min Range: -0.7924

Max Act: 0.6179. Min Act: -0.0000

Data Index: 1023787 (The Pile)

Max Activating Token Index: 634

Text #7

Max Range: 0.7924. Min Range: -0.7924

Max Act: 0.5802. Min Act: -0.0000

Data Index: 218397 (The Pile)

Max Activating Token Index: 818

Text #8

Max Range: 0.7924. Min Range: -0.7924

Max Act: 0.5793. Min Act: -0.0000

Data Index: 6739 (The Pile)

Max Activating Token Index: 933

Text #9

Max Range: 0.7924. Min Range: -0.7924

Max Act: 0.5827. Min Act: -0.0000

Data Index: 150759 (The Pile)

Max Activating Token Index: 299

Text #10

Max Range: 0.7924. Min Range: -0.7924

Max Act: 0.5313. Min Act: -0.0000

Data Index: 1029324 (The Pile)

Max Activating Token Index: 34

Text #11

Max Range: 0.7924. Min Range: -0.7924

Max Act: 0.5306. Min Act: -0.0000

Data Index: 1626770 (The Pile)

Max Activating Token Index: 167

Text #12

Max Range: 0.7924. Min Range: -0.7924

Max Act: 0.5541. Min Act: -0.0000

Data Index: 1031924 (The Pile)

Max Activating Token Index: 257

Text #13

Max Range: 0.7924. Min Range: -0.7924

Max Act: 0.5745. Min Act: -0.0000

Data Index: 1551626 (The Pile)

Max Activating Token Index: 827

Text #14

Max Range: 0.7924. Min Range: -0.7924

Max Act: 0.5554. Min Act: -0.0000

Data Index: 1593270 (The Pile)

Max Activating Token Index: 580

Text #15

Max Range: 0.7924. Min Range: -0.7924

Max Act: 0.5397. Min Act: -0.0000

Data Index: 482278 (The Pile)

Max Activating Token Index: 506

Text #16

Max Range: 0.7924. Min Range: -0.7924

Max Act: 0.5562. Min Act: -0.0000

Data Index: 1934552 (The Pile)

Max Activating Token Index: 357

Text #17

Max Range: 0.7924. Min Range: -0.7924

Max Act: 0.5700. Min Act: -0.0000

Data Index: 932899 (The Pile)

Max Activating Token Index: 394

Text #18

Max Range: 0.7924. Min Range: -0.7924

Max Act: 0.5621. Min Act: -0.0000

Data Index: 585141 (The Pile)

Max Activating Token Index: 921

Text #19

Max Range: 0.7924. Min Range: -0.7924

Max Act: 0.5425. Min Act: -0.0000

Data Index: 1018423 (The Pile)

Max Activating Token Index: 744