Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 3150 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 0.9304. Min Range: -0.9304

Max Act: 0.9304. Min Act: -0.0000

Data Index: 1161243 (The Pile)

Max Activating Token Index: 267

Text #1

Max Range: 0.9304. Min Range: -0.9304

Max Act: 0.8740. Min Act: -0.0000

Data Index: 1454232 (The Pile)

Max Activating Token Index: 87

Text #2

Max Range: 0.9304. Min Range: -0.9304

Max Act: 0.8793. Min Act: -0.0000

Data Index: 905788 (The Pile)

Max Activating Token Index: 478

Text #3

Max Range: 0.9304. Min Range: -0.9304

Max Act: 0.7398. Min Act: -0.0000

Data Index: 911863 (The Pile)

Max Activating Token Index: 380

Text #4

Max Range: 0.9304. Min Range: -0.9304

Max Act: 0.7850. Min Act: -0.0000

Data Index: 1389559 (The Pile)

Max Activating Token Index: 428

Text #5

Max Range: 0.9304. Min Range: -0.9304

Max Act: 0.7250. Min Act: -0.0000

Data Index: 561862 (The Pile)

Max Activating Token Index: 762

Text #6

Max Range: 0.9304. Min Range: -0.9304

Max Act: 0.7569. Min Act: -0.0000

Data Index: 1887001 (The Pile)

Max Activating Token Index: 793

Text #7

Max Range: 0.9304. Min Range: -0.9304

Max Act: 0.7534. Min Act: -0.0000

Data Index: 600201 (The Pile)

Max Activating Token Index: 259

Text #8

Max Range: 0.9304. Min Range: -0.9304

Max Act: 0.7065. Min Act: -0.0000

Data Index: 1974902 (The Pile)

Max Activating Token Index: 270

Text #9

Max Range: 0.9304. Min Range: -0.9304

Max Act: 0.7526. Min Act: -0.0000

Data Index: 132130 (The Pile)

Max Activating Token Index: 96

Text #10

Max Range: 0.9304. Min Range: -0.9304

Max Act: 0.7430. Min Act: 0.0000

Data Index: 979752 (The Pile)

Max Activating Token Index: 406

Text #11

Max Range: 0.9304. Min Range: -0.9304

Max Act: 0.7235. Min Act: -0.0000

Data Index: 358066 (The Pile)

Max Activating Token Index: 496

Text #12

Max Range: 0.9304. Min Range: -0.9304

Max Act: 0.6842. Min Act: -0.0000

Data Index: 1760736 (The Pile)

Max Activating Token Index: 656

Text #13

Max Range: 0.9304. Min Range: -0.9304

Max Act: 0.6971. Min Act: 0.0000

Data Index: 134250 (The Pile)

Max Activating Token Index: 735

Text #14

Max Range: 0.9304. Min Range: -0.9304

Max Act: 0.7039. Min Act: 0.0001

Data Index: 112052 (The Pile)

Max Activating Token Index: 323

Text #15

Max Range: 0.9304. Min Range: -0.9304

Max Act: 0.7108. Min Act: -0.0000

Data Index: 1495726 (The Pile)

Max Activating Token Index: 292

Text #16

Max Range: 0.9304. Min Range: -0.9304

Max Act: 0.6864. Min Act: -0.0000

Data Index: 1987973 (The Pile)

Max Activating Token Index: 363

Text #17

Max Range: 0.9304. Min Range: -0.9304

Max Act: 0.7107. Min Act: -0.0000

Data Index: 1948839 (The Pile)

Max Activating Token Index: 106

Text #18

Max Range: 0.9304. Min Range: -0.9304

Max Act: 0.6813. Min Act: 0.0000

Data Index: 151428 (The Pile)

Max Activating Token Index: 291

Text #19

Max Range: 0.9304. Min Range: -0.9304

Max Act: 0.6581. Min Act: -0.0000

Data Index: 778796 (The Pile)

Max Activating Token Index: 45