Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 2956 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 4.8033. Min Range: -4.8033

Max Act: 4.8033. Min Act: -0.0000

Data Index: 1083722 (The Pile)

Max Activating Token Index: 70

Text #1

Max Range: 4.8033. Min Range: -4.8033

Max Act: 4.3031. Min Act: -0.0000

Data Index: 1899303 (The Pile)

Max Activating Token Index: 392

Text #2

Max Range: 4.8033. Min Range: -4.8033

Max Act: 3.9502. Min Act: -0.0000

Data Index: 854311 (The Pile)

Max Activating Token Index: 574

Text #3

Max Range: 4.8033. Min Range: -4.8033

Max Act: 3.7657. Min Act: -0.0000

Data Index: 704982 (The Pile)

Max Activating Token Index: 741

Text #4

Max Range: 4.8033. Min Range: -4.8033

Max Act: 3.5335. Min Act: 0.0001

Data Index: 529036 (The Pile)

Max Activating Token Index: 190

Text #5

Max Range: 4.8033. Min Range: -4.8033

Max Act: 3.6605. Min Act: -0.0000

Data Index: 1470496 (The Pile)

Max Activating Token Index: 990

Text #6

Max Range: 4.8033. Min Range: -4.8033

Max Act: 3.7160. Min Act: -0.0000

Data Index: 1871291 (The Pile)

Max Activating Token Index: 33

Text #7

Max Range: 4.8033. Min Range: -4.8033

Max Act: 3.8246. Min Act: -0.0000

Data Index: 700807 (The Pile)

Max Activating Token Index: 531

Text #8

Max Range: 4.8033. Min Range: -4.8033

Max Act: 3.3862. Min Act: 0.0000

Data Index: 1962132 (The Pile)

Max Activating Token Index: 373

Text #9

Max Range: 4.8033. Min Range: -4.8033

Max Act: 3.1879. Min Act: -0.0000

Data Index: 706475 (The Pile)

Max Activating Token Index: 15

Text #10

Max Range: 4.8033. Min Range: -4.8033

Max Act: 3.4204. Min Act: -0.0000

Data Index: 1157871 (The Pile)

Max Activating Token Index: 245

Text #11

Max Range: 4.8033. Min Range: -4.8033

Max Act: 3.2638. Min Act: -0.0000

Data Index: 230551 (The Pile)

Max Activating Token Index: 83

Text #12

Max Range: 4.8033. Min Range: -4.8033

Max Act: 3.5484. Min Act: -0.0000

Data Index: 848141 (The Pile)

Max Activating Token Index: 630

Text #13

Max Range: 4.8033. Min Range: -4.8033

Max Act: 3.2119. Min Act: -0.0000

Data Index: 814302 (The Pile)

Max Activating Token Index: 227

Text #14

Max Range: 4.8033. Min Range: -4.8033

Max Act: 3.1715. Min Act: -0.0000

Data Index: 1299405 (The Pile)

Max Activating Token Index: 462

Text #15

Max Range: 4.8033. Min Range: -4.8033

Max Act: 3.0082. Min Act: -0.0000

Data Index: 51684 (The Pile)

Max Activating Token Index: 60

Text #16

Max Range: 4.8033. Min Range: -4.8033

Max Act: 2.9171. Min Act: -0.0000

Data Index: 1183277 (The Pile)

Max Activating Token Index: 998

Text #17

Max Range: 4.8033. Min Range: -4.8033

Max Act: 2.7710. Min Act: -0.0000

Data Index: 253266 (The Pile)

Max Activating Token Index: 907

Text #18

Max Range: 4.8033. Min Range: -4.8033

Max Act: 2.6141. Min Act: -0.0000

Data Index: 774494 (The Pile)

Max Activating Token Index: 558

Text #19

Max Range: 4.8033. Min Range: -4.8033

Max Act: 2.4557. Min Act: -0.0000

Data Index: 62916 (The Pile)

Max Activating Token Index: 139