Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 172 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 1.3177. Min Range: -1.3177

Max Act: 1.1785. Min Act: -0.0000

Data Index: 1016993 (The Pile)

Max Activating Token Index: 1021

Text #1

Max Range: 1.3177. Min Range: -1.3177

Max Act: 1.2364. Min Act: -0.0000

Data Index: 162483 (The Pile)

Max Activating Token Index: 2

Text #2

Max Range: 1.3177. Min Range: -1.3177

Max Act: 1.1960. Min Act: -0.0000

Data Index: 1922881 (The Pile)

Max Activating Token Index: 215

Text #3

Max Range: 1.3177. Min Range: -1.3177

Max Act: 1.2673. Min Act: -0.0000

Data Index: 167391 (The Pile)

Max Activating Token Index: 51

Text #4

Max Range: 1.3177. Min Range: -1.3177

Max Act: 1.3177. Min Act: -0.0000

Data Index: 1471260 (The Pile)

Max Activating Token Index: 851

Text #5

Max Range: 1.3177. Min Range: -1.3177

Max Act: 0.9435. Min Act: -0.0000

Data Index: 1081139 (The Pile)

Max Activating Token Index: 13

Text #6

Max Range: 1.3177. Min Range: -1.3177

Max Act: 0.8015. Min Act: -0.0000

Data Index: 812934 (The Pile)

Max Activating Token Index: 889

Text #7

Max Range: 1.3177. Min Range: -1.3177

Max Act: 1.1057. Min Act: -0.0000

Data Index: 551347 (The Pile)

Max Activating Token Index: 428

Text #8

Max Range: 1.3177. Min Range: -1.3177

Max Act: 0.9924. Min Act: -0.0000

Data Index: 137218 (The Pile)

Max Activating Token Index: 643

Text #9

Max Range: 1.3177. Min Range: -1.3177

Max Act: 1.0578. Min Act: -0.0000

Data Index: 1843195 (The Pile)

Max Activating Token Index: 438

Text #10

Max Range: 1.3177. Min Range: -1.3177

Max Act: 1.0777. Min Act: -0.0000

Data Index: 607184 (The Pile)

Max Activating Token Index: 655

Text #11

Max Range: 1.3177. Min Range: -1.3177

Max Act: 0.9741. Min Act: -0.0000

Data Index: 156532 (The Pile)

Max Activating Token Index: 635

Text #12

Max Range: 1.3177. Min Range: -1.3177

Max Act: 1.1350. Min Act: -0.0000

Data Index: 536600 (The Pile)

Max Activating Token Index: 1011

Text #13

Max Range: 1.3177. Min Range: -1.3177

Max Act: 0.9139. Min Act: -0.0000

Data Index: 739651 (The Pile)

Max Activating Token Index: 919

Text #14

Max Range: 1.3177. Min Range: -1.3177

Max Act: 0.9372. Min Act: -0.0000

Data Index: 99731 (The Pile)

Max Activating Token Index: 1005

Text #15

Max Range: 1.3177. Min Range: -1.3177

Max Act: 1.0783. Min Act: -0.0000

Data Index: 1633315 (The Pile)

Max Activating Token Index: 1002

Text #16

Max Range: 1.3177. Min Range: -1.3177

Max Act: 1.0372. Min Act: -0.0000

Data Index: 588461 (The Pile)

Max Activating Token Index: 267

Text #17

Max Range: 1.3177. Min Range: -1.3177

Max Act: 0.9851. Min Act: -0.0000

Data Index: 711331 (The Pile)

Max Activating Token Index: 1003

Text #18

Max Range: 1.3177. Min Range: -1.3177

Max Act: 0.9873. Min Act: -0.0000

Data Index: 448557 (The Pile)

Max Activating Token Index: 368

Text #19

Max Range: 1.3177. Min Range: -1.3177

Max Act: 1.0226. Min Act: -0.0000

Data Index: 1658729 (The Pile)

Max Activating Token Index: 840