Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 3792 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 2.8320. Min Range: -2.8320

Max Act: 2.8320. Min Act: -0.0000

Data Index: 1093429 (The Pile)

Max Activating Token Index: 220

Text #1

Max Range: 2.8320. Min Range: -2.8320

Max Act: 2.3053. Min Act: -0.0000

Data Index: 1409058 (The Pile)

Max Activating Token Index: 828

Text #2

Max Range: 2.8320. Min Range: -2.8320

Max Act: 2.4033. Min Act: -0.0000

Data Index: 610587 (The Pile)

Max Activating Token Index: 682

Text #3

Max Range: 2.8320. Min Range: -2.8320

Max Act: 2.2314. Min Act: -0.0000

Data Index: 186069 (The Pile)

Max Activating Token Index: 223

Text #4

Max Range: 2.8320. Min Range: -2.8320

Max Act: 2.1464. Min Act: -0.0000

Data Index: 1297219 (The Pile)

Max Activating Token Index: 94

Text #5

Max Range: 2.8320. Min Range: -2.8320

Max Act: 2.1308. Min Act: -0.0000

Data Index: 408618 (The Pile)

Max Activating Token Index: 419

Text #6

Max Range: 2.8320. Min Range: -2.8320

Max Act: 2.0179. Min Act: -0.0000

Data Index: 1485799 (The Pile)

Max Activating Token Index: 416

Text #7

Max Range: 2.8320. Min Range: -2.8320

Max Act: 1.9587. Min Act: -0.0000

Data Index: 1661355 (The Pile)

Max Activating Token Index: 212

Text #8

Max Range: 2.8320. Min Range: -2.8320

Max Act: 2.0591. Min Act: -0.0000

Data Index: 559724 (The Pile)

Max Activating Token Index: 691

Text #9

Max Range: 2.8320. Min Range: -2.8320

Max Act: 2.0188. Min Act: -0.0000

Data Index: 1665952 (The Pile)

Max Activating Token Index: 701

Text #10

Max Range: 2.8320. Min Range: -2.8320

Max Act: 1.9270. Min Act: -0.0000

Data Index: 944243 (The Pile)

Max Activating Token Index: 294

Text #11

Max Range: 2.8320. Min Range: -2.8320

Max Act: 1.9063. Min Act: -0.0000

Data Index: 1366080 (The Pile)

Max Activating Token Index: 574

Text #12

Max Range: 2.8320. Min Range: -2.8320

Max Act: 1.7932. Min Act: -0.0000

Data Index: 1127312 (The Pile)

Max Activating Token Index: 274

Text #13

Max Range: 2.8320. Min Range: -2.8320

Max Act: 1.9218. Min Act: -0.0000

Data Index: 782247 (The Pile)

Max Activating Token Index: 756

Text #14

Max Range: 2.8320. Min Range: -2.8320

Max Act: 1.8611. Min Act: -0.0000

Data Index: 927838 (The Pile)

Max Activating Token Index: 798

Text #15

Max Range: 2.8320. Min Range: -2.8320

Max Act: 1.8785. Min Act: -0.0000

Data Index: 1601377 (The Pile)

Max Activating Token Index: 941

Text #16

Max Range: 2.8320. Min Range: -2.8320

Max Act: 1.8149. Min Act: -0.0000

Data Index: 946624 (The Pile)

Max Activating Token Index: 762

Text #17

Max Range: 2.8320. Min Range: -2.8320

Max Act: 1.7051. Min Act: -0.0000

Data Index: 1650456 (The Pile)

Max Activating Token Index: 958

Text #18

Max Range: 2.8320. Min Range: -2.8320

Max Act: 1.7831. Min Act: -0.0000

Data Index: 1317612 (The Pile)

Max Activating Token Index: 314

Text #19

Max Range: 2.8320. Min Range: -2.8320

Max Act: 1.7906. Min Act: -0.0000

Data Index: 883834 (The Pile)

Max Activating Token Index: 132