Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 2008 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 1.5278. Min Range: -1.5278

Max Act: 1.5278. Min Act: -0.0000

Data Index: 1787133 (The Pile)

Max Activating Token Index: 375

Text #1

Max Range: 1.5278. Min Range: -1.5278

Max Act: 1.2185. Min Act: -0.0000

Data Index: 1548984 (The Pile)

Max Activating Token Index: 479

Text #2

Max Range: 1.5278. Min Range: -1.5278

Max Act: 1.2672. Min Act: -0.0000

Data Index: 730900 (The Pile)

Max Activating Token Index: 840

Text #3

Max Range: 1.5278. Min Range: -1.5278

Max Act: 1.2522. Min Act: -0.0000

Data Index: 1904865 (The Pile)

Max Activating Token Index: 553

Text #4

Max Range: 1.5278. Min Range: -1.5278

Max Act: 1.2022. Min Act: -0.0000

Data Index: 1692799 (The Pile)

Max Activating Token Index: 251

Text #5

Max Range: 1.5278. Min Range: -1.5278

Max Act: 1.1654. Min Act: -0.0000

Data Index: 1634112 (The Pile)

Max Activating Token Index: 535

Text #6

Max Range: 1.5278. Min Range: -1.5278

Max Act: 1.1452. Min Act: -0.0000

Data Index: 1089278 (The Pile)

Max Activating Token Index: 377

Text #7

Max Range: 1.5278. Min Range: -1.5278

Max Act: 1.0812. Min Act: -0.0000

Data Index: 409121 (The Pile)

Max Activating Token Index: 1006

Text #8

Max Range: 1.5278. Min Range: -1.5278

Max Act: 1.1709. Min Act: -0.0000

Data Index: 1381894 (The Pile)

Max Activating Token Index: 189

Text #9

Max Range: 1.5278. Min Range: -1.5278

Max Act: 1.0419. Min Act: -0.0000

Data Index: 85836 (The Pile)

Max Activating Token Index: 912

Text #10

Max Range: 1.5278. Min Range: -1.5278

Max Act: 1.0697. Min Act: -0.0000

Data Index: 324997 (The Pile)

Max Activating Token Index: 99

Text #11

Max Range: 1.5278. Min Range: -1.5278

Max Act: 0.9499. Min Act: -0.0000

Data Index: 760875 (The Pile)

Max Activating Token Index: 99

Text #12

Max Range: 1.5278. Min Range: -1.5278

Max Act: 1.0419. Min Act: -0.0000

Data Index: 469691 (The Pile)

Max Activating Token Index: 1015

Text #13

Max Range: 1.5278. Min Range: -1.5278

Max Act: 1.0836. Min Act: -0.0000

Data Index: 1828636 (The Pile)

Max Activating Token Index: 238

Text #14

Max Range: 1.5278. Min Range: -1.5278

Max Act: 1.0817. Min Act: -0.0000

Data Index: 1398416 (The Pile)

Max Activating Token Index: 184

Text #15

Max Range: 1.5278. Min Range: -1.5278

Max Act: 1.0522. Min Act: -0.0000

Data Index: 915497 (The Pile)

Max Activating Token Index: 520

Text #16

Max Range: 1.5278. Min Range: -1.5278

Max Act: 1.0150. Min Act: -0.0000

Data Index: 573084 (The Pile)

Max Activating Token Index: 39

Text #17

Max Range: 1.5278. Min Range: -1.5278

Max Act: 0.9511. Min Act: -0.0000

Data Index: 1894525 (The Pile)

Max Activating Token Index: 44

Text #18

Max Range: 1.5278. Min Range: -1.5278

Max Act: 1.0717. Min Act: -0.0000

Data Index: 692203 (The Pile)

Max Activating Token Index: 469

Text #19

Max Range: 1.5278. Min Range: -1.5278

Max Act: 1.0729. Min Act: -0.0000

Data Index: 558836 (The Pile)

Max Activating Token Index: 420