Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 3889 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 1.3244. Min Range: -1.3244

Max Act: 1.3244. Min Act: -0.0000

Data Index: 1946766 (The Pile)

Max Activating Token Index: 489

Text #1

Max Range: 1.3244. Min Range: -1.3244

Max Act: 1.2271. Min Act: -0.0000

Data Index: 1016030 (The Pile)

Max Activating Token Index: 199

Text #2

Max Range: 1.3244. Min Range: -1.3244

Max Act: 1.1166. Min Act: -0.0000

Data Index: 1336176 (The Pile)

Max Activating Token Index: 423

Text #3

Max Range: 1.3244. Min Range: -1.3244

Max Act: 1.2981. Min Act: -0.0000

Data Index: 1002598 (The Pile)

Max Activating Token Index: 914

Text #4

Max Range: 1.3244. Min Range: -1.3244

Max Act: 1.1399. Min Act: -0.0000

Data Index: 1717850 (The Pile)

Max Activating Token Index: 253

Text #5

Max Range: 1.3244. Min Range: -1.3244

Max Act: 1.0686. Min Act: -0.0000

Data Index: 385636 (The Pile)

Max Activating Token Index: 349

Text #6

Max Range: 1.3244. Min Range: -1.3244

Max Act: 0.9507. Min Act: -0.0000

Data Index: 1633502 (The Pile)

Max Activating Token Index: 589

Text #7

Max Range: 1.3244. Min Range: -1.3244

Max Act: 1.0669. Min Act: -0.0000

Data Index: 1102688 (The Pile)

Max Activating Token Index: 136

Text #8

Max Range: 1.3244. Min Range: -1.3244

Max Act: 0.9484. Min Act: -0.0000

Data Index: 95407 (The Pile)

Max Activating Token Index: 845

Text #9

Max Range: 1.3244. Min Range: -1.3244

Max Act: 0.9535. Min Act: -0.0000

Data Index: 759641 (The Pile)

Max Activating Token Index: 354

Text #10

Max Range: 1.3244. Min Range: -1.3244

Max Act: 1.0093. Min Act: 0.0000

Data Index: 1654660 (The Pile)

Max Activating Token Index: 750

Text #11

Max Range: 1.3244. Min Range: -1.3244

Max Act: 0.9457. Min Act: -0.0000

Data Index: 1512922 (The Pile)

Max Activating Token Index: 635

Text #12

Max Range: 1.3244. Min Range: -1.3244

Max Act: 0.9436. Min Act: -0.0000

Data Index: 983495 (The Pile)

Max Activating Token Index: 526

Text #13

Max Range: 1.3244. Min Range: -1.3244

Max Act: 0.8820. Min Act: -0.0000

Data Index: 1736727 (The Pile)

Max Activating Token Index: 237

Text #14

Max Range: 1.3244. Min Range: -1.3244

Max Act: 0.9799. Min Act: -0.0000

Data Index: 1301668 (The Pile)

Max Activating Token Index: 290

Text #15

Max Range: 1.3244. Min Range: -1.3244

Max Act: 0.9097. Min Act: -0.0000

Data Index: 1516898 (The Pile)

Max Activating Token Index: 332

Text #16

Max Range: 1.3244. Min Range: -1.3244

Max Act: 0.9298. Min Act: -0.0000

Data Index: 1653688 (The Pile)

Max Activating Token Index: 664

Text #17

Max Range: 1.3244. Min Range: -1.3244

Max Act: 0.7425. Min Act: -0.0000

Data Index: 1709129 (The Pile)

Max Activating Token Index: 704

Text #18

Max Range: 1.3244. Min Range: -1.3244

Max Act: 0.7599. Min Act: -0.0000

Data Index: 1131558 (The Pile)

Max Activating Token Index: 108

Text #19

Max Range: 1.3244. Min Range: -1.3244

Max Act: 0.8707. Min Act: -0.0000

Data Index: 1318055 (The Pile)

Max Activating Token Index: 394