Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: The Pile

Neuron 71 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l-pile')

Text #0

Max Range: 7.1467. Min Range: -7.1467

Max Act: 7.1467. Min Act: -0.0000

Data Index: 506815 (The Pile)

Max Activating Token Index: 196

Text #1

Max Range: 7.1467. Min Range: -7.1467

Max Act: 6.3774. Min Act: -0.0000

Data Index: 1274505 (The Pile)

Max Activating Token Index: 40

Text #2

Max Range: 7.1467. Min Range: -7.1467

Max Act: 5.9090. Min Act: -0.0000

Data Index: 38749 (The Pile)

Max Activating Token Index: 361

Text #3

Max Range: 7.1467. Min Range: -7.1467

Max Act: 6.3114. Min Act: -0.0000

Data Index: 1886340 (The Pile)

Max Activating Token Index: 489

Text #4

Max Range: 7.1467. Min Range: -7.1467

Max Act: 5.9872. Min Act: -0.0000

Data Index: 1439231 (The Pile)

Max Activating Token Index: 188

Text #5

Max Range: 7.1467. Min Range: -7.1467

Max Act: 5.2986. Min Act: -0.0000

Data Index: 951756 (The Pile)

Max Activating Token Index: 749

Text #6

Max Range: 7.1467. Min Range: -7.1467

Max Act: 6.1075. Min Act: -0.0000

Data Index: 1588957 (The Pile)

Max Activating Token Index: 32

Text #7

Max Range: 7.1467. Min Range: -7.1467

Max Act: 5.9346. Min Act: -0.0000

Data Index: 275627 (The Pile)

Max Activating Token Index: 726

Text #8

Max Range: 7.1467. Min Range: -7.1467

Max Act: 5.6935. Min Act: -0.0000

Data Index: 765831 (The Pile)

Max Activating Token Index: 241

Text #9

Max Range: 7.1467. Min Range: -7.1467

Max Act: 5.6510. Min Act: -0.0000

Data Index: 1808737 (The Pile)

Max Activating Token Index: 336

Text #10

Max Range: 7.1467. Min Range: -7.1467

Max Act: 5.7852. Min Act: -0.0000

Data Index: 1781975 (The Pile)

Max Activating Token Index: 259

Text #11

Max Range: 7.1467. Min Range: -7.1467

Max Act: 6.3398. Min Act: -0.0000

Data Index: 996949 (The Pile)

Max Activating Token Index: 756

Text #12

Max Range: 7.1467. Min Range: -7.1467

Max Act: 5.5715. Min Act: -0.0000

Data Index: 271876 (The Pile)

Max Activating Token Index: 832

Text #13

Max Range: 7.1467. Min Range: -7.1467

Max Act: 5.7260. Min Act: -0.0000

Data Index: 1261892 (The Pile)

Max Activating Token Index: 600

Text #14

Max Range: 7.1467. Min Range: -7.1467

Max Act: 5.8956. Min Act: -0.0000

Data Index: 465298 (The Pile)

Max Activating Token Index: 645

Text #15

Max Range: 7.1467. Min Range: -7.1467

Max Act: 5.7573. Min Act: -0.0000

Data Index: 1490563 (The Pile)

Max Activating Token Index: 119

Text #16

Max Range: 7.1467. Min Range: -7.1467

Max Act: 5.4872. Min Act: -0.0000

Data Index: 1579815 (The Pile)

Max Activating Token Index: 220

Text #17

Max Range: 7.1467. Min Range: -7.1467

Max Act: 5.4565. Min Act: -0.0000

Data Index: 736068 (The Pile)

Max Activating Token Index: 477

Text #18

Max Range: 7.1467. Min Range: -7.1467

Max Act: 6.0370. Min Act: -0.0000

Data Index: 729807 (The Pile)

Max Activating Token Index: 175

Text #19

Max Range: 7.1467. Min Range: -7.1467

Max Act: 5.6212. Min Act: -0.0000

Data Index: 1425476 (The Pile)

Max Activating Token Index: 80