Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 847 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 7.5002. Min Range: -7.5002

Max Act: 7.5002. Min Act: -0.0000

Data Index: 1386303 (The Pile)

Max Activating Token Index: 506

Text #1

Max Range: 7.5002. Min Range: -7.5002

Max Act: 6.9911. Min Act: -0.0000

Data Index: 900323 (The Pile)

Max Activating Token Index: 687

Text #2

Max Range: 7.5002. Min Range: -7.5002

Max Act: 6.7693. Min Act: -0.0000

Data Index: 229100 (The Pile)

Max Activating Token Index: 42

Text #3

Max Range: 7.5002. Min Range: -7.5002

Max Act: 6.3592. Min Act: -0.0000

Data Index: 1479721 (The Pile)

Max Activating Token Index: 689

Text #4

Max Range: 7.5002. Min Range: -7.5002

Max Act: 6.4985. Min Act: -0.0000

Data Index: 1486743 (The Pile)

Max Activating Token Index: 473

Text #5

Max Range: 7.5002. Min Range: -7.5002

Max Act: 6.2942. Min Act: -0.0000

Data Index: 808525 (The Pile)

Max Activating Token Index: 969

Text #6

Max Range: 7.5002. Min Range: -7.5002

Max Act: 6.0567. Min Act: -0.0000

Data Index: 207384 (The Pile)

Max Activating Token Index: 236

Text #7

Max Range: 7.5002. Min Range: -7.5002

Max Act: 6.5664. Min Act: -0.0000

Data Index: 1389025 (The Pile)

Max Activating Token Index: 278

Text #8

Max Range: 7.5002. Min Range: -7.5002

Max Act: 6.1440. Min Act: -0.0000

Data Index: 1458334 (The Pile)

Max Activating Token Index: 523

Text #9

Max Range: 7.5002. Min Range: -7.5002

Max Act: 5.9245. Min Act: -0.0000

Data Index: 1715158 (The Pile)

Max Activating Token Index: 56

Text #10

Max Range: 7.5002. Min Range: -7.5002

Max Act: 5.9162. Min Act: -0.0000

Data Index: 1212463 (The Pile)

Max Activating Token Index: 482

Text #11

Max Range: 7.5002. Min Range: -7.5002

Max Act: 5.9168. Min Act: -0.0000

Data Index: 1553441 (The Pile)

Max Activating Token Index: 708

Text #12

Max Range: 7.5002. Min Range: -7.5002

Max Act: 6.2429. Min Act: -0.0000

Data Index: 971367 (The Pile)

Max Activating Token Index: 225

Text #13

Max Range: 7.5002. Min Range: -7.5002

Max Act: 5.7901. Min Act: -0.0000

Data Index: 360451 (The Pile)

Max Activating Token Index: 137

Text #14

Max Range: 7.5002. Min Range: -7.5002

Max Act: 5.9479. Min Act: -0.0000

Data Index: 190617 (The Pile)

Max Activating Token Index: 837

Text #15

Max Range: 7.5002. Min Range: -7.5002

Max Act: 6.1231. Min Act: -0.0000

Data Index: 107258 (The Pile)

Max Activating Token Index: 567

Text #16

Max Range: 7.5002. Min Range: -7.5002

Max Act: 5.8541. Min Act: -0.0000

Data Index: 1395117 (The Pile)

Max Activating Token Index: 330

Text #17

Max Range: 7.5002. Min Range: -7.5002

Max Act: 5.7914. Min Act: -0.0000

Data Index: 1239951 (The Pile)

Max Activating Token Index: 974

Text #18

Max Range: 7.5002. Min Range: -7.5002

Max Act: 5.5538. Min Act: -0.0000

Data Index: 1942357 (The Pile)

Max Activating Token Index: 526

Text #19

Max Range: 7.5002. Min Range: -7.5002

Max Act: 5.2782. Min Act: -0.0000

Data Index: 1963981 (The Pile)

Max Activating Token Index: 359