Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 3249 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 1.3304. Min Range: -1.3304

Max Act: 1.2685. Min Act: -0.0000

Data Index: 1986752 (The Pile)

Max Activating Token Index: 203

Text #1

Max Range: 1.3304. Min Range: -1.3304

Max Act: 1.3304. Min Act: -0.0000

Data Index: 250899 (The Pile)

Max Activating Token Index: 91

Text #2

Max Range: 1.3304. Min Range: -1.3304

Max Act: 1.2789. Min Act: -0.0000

Data Index: 152084 (The Pile)

Max Activating Token Index: 283

Text #3

Max Range: 1.3304. Min Range: -1.3304

Max Act: 1.1810. Min Act: -0.0000

Data Index: 1223101 (The Pile)

Max Activating Token Index: 697

Text #4

Max Range: 1.3304. Min Range: -1.3304

Max Act: 1.0815. Min Act: -0.0000

Data Index: 679582 (The Pile)

Max Activating Token Index: 548

Text #5

Max Range: 1.3304. Min Range: -1.3304

Max Act: 1.1345. Min Act: -0.0000

Data Index: 285929 (The Pile)

Max Activating Token Index: 577

Text #6

Max Range: 1.3304. Min Range: -1.3304

Max Act: 1.1504. Min Act: -0.0000

Data Index: 597722 (The Pile)

Max Activating Token Index: 341

Text #7

Max Range: 1.3304. Min Range: -1.3304

Max Act: 1.1486. Min Act: -0.0000

Data Index: 9996 (The Pile)

Max Activating Token Index: 169

Text #8

Max Range: 1.3304. Min Range: -1.3304

Max Act: 1.0663. Min Act: -0.0000

Data Index: 63774 (The Pile)

Max Activating Token Index: 107

Text #9

Max Range: 1.3304. Min Range: -1.3304

Max Act: 1.1870. Min Act: -0.0000

Data Index: 1977186 (The Pile)

Max Activating Token Index: 225

Text #10

Max Range: 1.3304. Min Range: -1.3304

Max Act: 1.0963. Min Act: -0.0000

Data Index: 574819 (The Pile)

Max Activating Token Index: 311

Text #11

Max Range: 1.3304. Min Range: -1.3304

Max Act: 0.9680. Min Act: -0.0000

Data Index: 1338348 (The Pile)

Max Activating Token Index: 750

Text #12

Max Range: 1.3304. Min Range: -1.3304

Max Act: 1.0767. Min Act: -0.0000

Data Index: 308957 (The Pile)

Max Activating Token Index: 325

Text #13

Max Range: 1.3304. Min Range: -1.3304

Max Act: 1.0322. Min Act: -0.0000

Data Index: 1173434 (The Pile)

Max Activating Token Index: 398

Text #14

Max Range: 1.3304. Min Range: -1.3304

Max Act: 1.1157. Min Act: -0.0000

Data Index: 208825 (The Pile)

Max Activating Token Index: 891

Text #15

Max Range: 1.3304. Min Range: -1.3304

Max Act: 1.1112. Min Act: -0.0000

Data Index: 1184773 (The Pile)

Max Activating Token Index: 1017

Text #16

Max Range: 1.3304. Min Range: -1.3304

Max Act: 1.0651. Min Act: -0.0000

Data Index: 493512 (The Pile)

Max Activating Token Index: 319

Text #17

Max Range: 1.3304. Min Range: -1.3304

Max Act: 1.0280. Min Act: -0.0000

Data Index: 640307 (The Pile)

Max Activating Token Index: 199

Text #18

Max Range: 1.3304. Min Range: -1.3304

Max Act: 0.9795. Min Act: -0.0000

Data Index: 1604091 (The Pile)

Max Activating Token Index: 625

Text #19

Max Range: 1.3304. Min Range: -1.3304

Max Act: 1.0684. Min Act: -0.0000

Data Index: 1108686 (The Pile)

Max Activating Token Index: 543