Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 353 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 0.8144. Min Range: -0.8144

Max Act: 0.8144. Min Act: -0.0000

Data Index: 1700600 (The Pile)

Max Activating Token Index: 213

Text #1

Max Range: 0.8144. Min Range: -0.8144

Max Act: 0.7742. Min Act: -0.0000

Data Index: 600204 (The Pile)

Max Activating Token Index: 475

Text #2

Max Range: 0.8144. Min Range: -0.8144

Max Act: 0.7345. Min Act: -0.0000

Data Index: 259998 (The Pile)

Max Activating Token Index: 636

Text #3

Max Range: 0.8144. Min Range: -0.8144

Max Act: 0.7769. Min Act: -0.0000

Data Index: 1139206 (The Pile)

Max Activating Token Index: 438

Text #4

Max Range: 0.8144. Min Range: -0.8144

Max Act: 0.6884. Min Act: -0.0000

Data Index: 343362 (The Pile)

Max Activating Token Index: 199

Text #5

Max Range: 0.8144. Min Range: -0.8144

Max Act: 0.7487. Min Act: -0.0000

Data Index: 1974081 (The Pile)

Max Activating Token Index: 201

Text #6

Max Range: 0.8144. Min Range: -0.8144

Max Act: 0.7079. Min Act: -0.0000

Data Index: 329772 (The Pile)

Max Activating Token Index: 98

Text #7

Max Range: 0.8144. Min Range: -0.8144

Max Act: 0.6783. Min Act: -0.0000

Data Index: 1622623 (The Pile)

Max Activating Token Index: 915

Text #8

Max Range: 0.8144. Min Range: -0.8144

Max Act: 0.7175. Min Act: -0.0000

Data Index: 1850501 (The Pile)

Max Activating Token Index: 745

Text #9

Max Range: 0.8144. Min Range: -0.8144

Max Act: 0.6464. Min Act: -0.0000

Data Index: 1411430 (The Pile)

Max Activating Token Index: 623

Text #10

Max Range: 0.8144. Min Range: -0.8144

Max Act: 0.6626. Min Act: -0.0000

Data Index: 1957063 (The Pile)

Max Activating Token Index: 283

Text #11

Max Range: 0.8144. Min Range: -0.8144

Max Act: 0.6248. Min Act: -0.0000

Data Index: 943397 (The Pile)

Max Activating Token Index: 461

Text #12

Max Range: 0.8144. Min Range: -0.8144

Max Act: 0.6544. Min Act: -0.0000

Data Index: 1366773 (The Pile)

Max Activating Token Index: 295

Text #13

Max Range: 0.8144. Min Range: -0.8144

Max Act: 0.6482. Min Act: -0.0000

Data Index: 718289 (The Pile)

Max Activating Token Index: 427

Text #14

Max Range: 0.8144. Min Range: -0.8144

Max Act: 0.6597. Min Act: -0.0000

Data Index: 1402659 (The Pile)

Max Activating Token Index: 214

Text #15

Max Range: 0.8144. Min Range: -0.8144

Max Act: 0.6747. Min Act: -0.0000

Data Index: 1064350 (The Pile)

Max Activating Token Index: 832

Text #16

Max Range: 0.8144. Min Range: -0.8144

Max Act: 0.6356. Min Act: -0.0000

Data Index: 1125786 (The Pile)

Max Activating Token Index: 478

Text #17

Max Range: 0.8144. Min Range: -0.8144

Max Act: 0.6511. Min Act: -0.0000

Data Index: 713993 (The Pile)

Max Activating Token Index: 682

Text #18

Max Range: 0.8144. Min Range: -0.8144

Max Act: 0.6221. Min Act: -0.0000

Data Index: 1203393 (The Pile)

Max Activating Token Index: 408

Text #19

Max Range: 0.8144. Min Range: -0.8144

Max Act: 0.5999. Min Act: -0.0000

Data Index: 546005 (The Pile)

Max Activating Token Index: 82