Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 2485 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 0.4108. Min Range: -0.4108

Max Act: 0.4108. Min Act: -0.0000

Data Index: 920399 (The Pile)

Max Activating Token Index: 37

Text #1

Max Range: 0.4108. Min Range: -0.4108

Max Act: 0.3432. Min Act: -0.0000

Data Index: 1555111 (The Pile)

Max Activating Token Index: 919

Text #2

Max Range: 0.4108. Min Range: -0.4108

Max Act: 0.2432. Min Act: -0.0000

Data Index: 758769 (The Pile)

Max Activating Token Index: 98

Text #3

Max Range: 0.4108. Min Range: -0.4108

Max Act: 0.2615. Min Act: -0.0000

Data Index: 933547 (The Pile)

Max Activating Token Index: 308

Text #4

Max Range: 0.4108. Min Range: -0.4108

Max Act: 0.2518. Min Act: -0.0000

Data Index: 410893 (The Pile)

Max Activating Token Index: 239

Text #5

Max Range: 0.4108. Min Range: -0.4108

Max Act: 0.2563. Min Act: -0.0000

Data Index: 690846 (The Pile)

Max Activating Token Index: 945

Text #6

Max Range: 0.4108. Min Range: -0.4108

Max Act: 0.2086. Min Act: -0.0000

Data Index: 808046 (The Pile)

Max Activating Token Index: 197

Text #7

Max Range: 0.4108. Min Range: -0.4108

Max Act: 0.2251. Min Act: -0.0000

Data Index: 669589 (The Pile)

Max Activating Token Index: 174

Text #8

Max Range: 0.4108. Min Range: -0.4108

Max Act: 0.2400. Min Act: -0.0000

Data Index: 1646018 (The Pile)

Max Activating Token Index: 166

Text #9

Max Range: 0.4108. Min Range: -0.4108

Max Act: 0.2249. Min Act: -0.0000

Data Index: 1980529 (The Pile)

Max Activating Token Index: 366

Text #10

Max Range: 0.4108. Min Range: -0.4108

Max Act: 0.2196. Min Act: -0.0000

Data Index: 453262 (The Pile)

Max Activating Token Index: 385

Text #11

Max Range: 0.4108. Min Range: -0.4108

Max Act: 0.2174. Min Act: -0.0000

Data Index: 1647167 (The Pile)

Max Activating Token Index: 352

Text #12

Max Range: 0.4108. Min Range: -0.4108

Max Act: 0.2101. Min Act: -0.0000

Data Index: 1501986 (The Pile)

Max Activating Token Index: 769

Text #13

Max Range: 0.4108. Min Range: -0.4108

Max Act: 0.2372. Min Act: -0.0000

Data Index: 879232 (The Pile)

Max Activating Token Index: 138

Text #14

Max Range: 0.4108. Min Range: -0.4108

Max Act: 0.2201. Min Act: -0.0000

Data Index: 884482 (The Pile)

Max Activating Token Index: 278

Text #15

Max Range: 0.4108. Min Range: -0.4108

Max Act: 0.2273. Min Act: -0.0000

Data Index: 289872 (The Pile)

Max Activating Token Index: 7

Text #16

Max Range: 0.4108. Min Range: -0.4108

Max Act: 0.2119. Min Act: -0.0000

Data Index: 1616134 (The Pile)

Max Activating Token Index: 362

Text #17

Max Range: 0.4108. Min Range: -0.4108

Max Act: 0.1903. Min Act: -0.0000

Data Index: 254538 (The Pile)

Max Activating Token Index: 102

Text #18

Max Range: 0.4108. Min Range: -0.4108

Max Act: 0.1985. Min Act: -0.0000

Data Index: 107524 (The Pile)

Max Activating Token Index: 255

Text #19

Max Range: 0.4108. Min Range: -0.4108

Max Act: 0.2085. Min Act: -0.0000

Data Index: 1267941 (The Pile)

Max Activating Token Index: 270