Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 175 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 3.9521. Min Range: -3.9521

Max Act: 3.6919. Min Act: -0.0000

Data Index: 1777653 (The Pile)

Max Activating Token Index: 508

Text #1

Max Range: 3.9521. Min Range: -3.9521

Max Act: 3.9015. Min Act: -0.0000

Data Index: 1980211 (The Pile)

Max Activating Token Index: 161

Text #2

Max Range: 3.9521. Min Range: -3.9521

Max Act: 3.6568. Min Act: -0.0000

Data Index: 324325 (The Pile)

Max Activating Token Index: 782

Text #3

Max Range: 3.9521. Min Range: -3.9521

Max Act: 3.5718. Min Act: -0.0000

Data Index: 1298630 (The Pile)

Max Activating Token Index: 159

Text #4

Max Range: 3.9521. Min Range: -3.9521

Max Act: 3.4544. Min Act: -0.0000

Data Index: 613862 (The Pile)

Max Activating Token Index: 298

Text #5

Max Range: 3.9521. Min Range: -3.9521

Max Act: 3.2172. Min Act: -0.0000

Data Index: 1078996 (The Pile)

Max Activating Token Index: 410

Text #6

Max Range: 3.9521. Min Range: -3.9521

Max Act: 3.1922. Min Act: -0.0000

Data Index: 1108978 (The Pile)

Max Activating Token Index: 30

Text #7

Max Range: 3.9521. Min Range: -3.9521

Max Act: 2.8379. Min Act: -0.0000

Data Index: 1299323 (The Pile)

Max Activating Token Index: 949

Text #8

Max Range: 3.9521. Min Range: -3.9521

Max Act: 3.3901. Min Act: -0.0000

Data Index: 690304 (The Pile)

Max Activating Token Index: 478

Text #9

Max Range: 3.9521. Min Range: -3.9521

Max Act: 3.1872. Min Act: -0.0000

Data Index: 1245499 (The Pile)

Max Activating Token Index: 228

Text #10

Max Range: 3.9521. Min Range: -3.9521

Max Act: 3.1648. Min Act: -0.0000

Data Index: 1456920 (The Pile)

Max Activating Token Index: 254

Text #11

Max Range: 3.9521. Min Range: -3.9521

Max Act: 3.9521. Min Act: -0.0000

Data Index: 1994734 (The Pile)

Max Activating Token Index: 912

Text #12

Max Range: 3.9521. Min Range: -3.9521

Max Act: 3.0709. Min Act: -0.0000

Data Index: 1252026 (The Pile)

Max Activating Token Index: 28

Text #13

Max Range: 3.9521. Min Range: -3.9521

Max Act: 2.9108. Min Act: -0.0000

Data Index: 56533 (The Pile)

Max Activating Token Index: 386

Text #14

Max Range: 3.9521. Min Range: -3.9521

Max Act: 3.0040. Min Act: -0.0000

Data Index: 879153 (The Pile)

Max Activating Token Index: 410

Text #15

Max Range: 3.9521. Min Range: -3.9521

Max Act: 2.8380. Min Act: -0.0000

Data Index: 759868 (The Pile)

Max Activating Token Index: 210

Text #16

Max Range: 3.9521. Min Range: -3.9521

Max Act: 2.9634. Min Act: -0.0000

Data Index: 96887 (The Pile)

Max Activating Token Index: 341

Text #17

Max Range: 3.9521. Min Range: -3.9521

Max Act: 2.7367. Min Act: -0.0000

Data Index: 72301 (The Pile)

Max Activating Token Index: 191

Text #18

Max Range: 3.9521. Min Range: -3.9521

Max Act: 2.0923. Min Act: -0.0000

Data Index: 1683388 (The Pile)

Max Activating Token Index: 974

Text #19

Max Range: 3.9521. Min Range: -3.9521

Max Act: 3.0920. Min Act: -0.0000

Data Index: 810285 (The Pile)

Max Activating Token Index: 567