Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 3037 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 0.2846. Min Range: -0.2846

Max Act: 0.2846. Min Act: -0.0000

Data Index: 716236 (The Pile)

Max Activating Token Index: 674

Text #1

Max Range: 0.2846. Min Range: -0.2846

Max Act: 0.2738. Min Act: -0.0000

Data Index: 1892927 (The Pile)

Max Activating Token Index: 265

Text #2

Max Range: 0.2846. Min Range: -0.2846

Max Act: 0.2289. Min Act: -0.0000

Data Index: 872872 (The Pile)

Max Activating Token Index: 770

Text #3

Max Range: 0.2846. Min Range: -0.2846

Max Act: 0.2343. Min Act: -0.0000

Data Index: 168610 (The Pile)

Max Activating Token Index: 696

Text #4

Max Range: 0.2846. Min Range: -0.2846

Max Act: 0.2276. Min Act: -0.0000

Data Index: 1896861 (The Pile)

Max Activating Token Index: 325

Text #5

Max Range: 0.2846. Min Range: -0.2846

Max Act: 0.2338. Min Act: -0.0000

Data Index: 1406500 (The Pile)

Max Activating Token Index: 504

Text #6

Max Range: 0.2846. Min Range: -0.2846

Max Act: 0.2327. Min Act: -0.0000

Data Index: 344087 (The Pile)

Max Activating Token Index: 503

Text #7

Max Range: 0.2846. Min Range: -0.2846

Max Act: 0.1971. Min Act: -0.0000

Data Index: 1777647 (The Pile)

Max Activating Token Index: 846

Text #8

Max Range: 0.2846. Min Range: -0.2846

Max Act: 0.2050. Min Act: -0.0000

Data Index: 1926753 (The Pile)

Max Activating Token Index: 366

Text #9

Max Range: 0.2846. Min Range: -0.2846

Max Act: 0.2083. Min Act: -0.0000

Data Index: 877253 (The Pile)

Max Activating Token Index: 253

Text #10

Max Range: 0.2846. Min Range: -0.2846

Max Act: 0.2144. Min Act: -0.0000

Data Index: 989141 (The Pile)

Max Activating Token Index: 220

Text #11

Max Range: 0.2846. Min Range: -0.2846

Max Act: 0.1932. Min Act: -0.0000

Data Index: 608010 (The Pile)

Max Activating Token Index: 824

Text #12

Max Range: 0.2846. Min Range: -0.2846

Max Act: 0.2202. Min Act: -0.0000

Data Index: 1763499 (The Pile)

Max Activating Token Index: 919

Text #13

Max Range: 0.2846. Min Range: -0.2846

Max Act: 0.2064. Min Act: -0.0000

Data Index: 939912 (The Pile)

Max Activating Token Index: 490

Text #14

Max Range: 0.2846. Min Range: -0.2846

Max Act: 0.2120. Min Act: -0.0000

Data Index: 1968590 (The Pile)

Max Activating Token Index: 295

Text #15

Max Range: 0.2846. Min Range: -0.2846

Max Act: 0.2272. Min Act: -0.0000

Data Index: 694278 (The Pile)

Max Activating Token Index: 538

Text #16

Max Range: 0.2846. Min Range: -0.2846

Max Act: 0.1968. Min Act: -0.0000

Data Index: 1702619 (The Pile)

Max Activating Token Index: 138

Text #17

Max Range: 0.2846. Min Range: -0.2846

Max Act: 0.2190. Min Act: -0.0000

Data Index: 879120 (The Pile)

Max Activating Token Index: 195

Text #18

Max Range: 0.2846. Min Range: -0.2846

Max Act: 0.2170. Min Act: -0.0000

Data Index: 525664 (The Pile)

Max Activating Token Index: 147

Text #19

Max Range: 0.2846. Min Range: -0.2846

Max Act: 0.2068. Min Act: -0.0000

Data Index: 1296110 (The Pile)

Max Activating Token Index: 295