Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 3316 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 3.6788. Min Range: -3.6788

Max Act: 3.6788. Min Act: -0.0000

Data Index: 1654795 (The Pile)

Max Activating Token Index: 344

Text #1

Max Range: 3.6788. Min Range: -3.6788

Max Act: 3.3293. Min Act: -0.0000

Data Index: 1680785 (The Pile)

Max Activating Token Index: 747

Text #2

Max Range: 3.6788. Min Range: -3.6788

Max Act: 3.6782. Min Act: -0.0000

Data Index: 1690759 (The Pile)

Max Activating Token Index: 223

Text #3

Max Range: 3.6788. Min Range: -3.6788

Max Act: 2.8427. Min Act: -0.0000

Data Index: 1984098 (The Pile)

Max Activating Token Index: 348

Text #4

Max Range: 3.6788. Min Range: -3.6788

Max Act: 2.8425. Min Act: -0.0000

Data Index: 1778861 (The Pile)

Max Activating Token Index: 816

Text #5

Max Range: 3.6788. Min Range: -3.6788

Max Act: 2.3397. Min Act: -0.0000

Data Index: 1364870 (The Pile)

Max Activating Token Index: 793

Text #6

Max Range: 3.6788. Min Range: -3.6788

Max Act: 2.1678. Min Act: -0.0000

Data Index: 523002 (The Pile)

Max Activating Token Index: 581

Text #7

Max Range: 3.6788. Min Range: -3.6788

Max Act: 2.8492. Min Act: -0.0000

Data Index: 683907 (The Pile)

Max Activating Token Index: 897

Text #8

Max Range: 3.6788. Min Range: -3.6788

Max Act: 2.1153. Min Act: -0.0000

Data Index: 183177 (The Pile)

Max Activating Token Index: 246

Text #9

Max Range: 3.6788. Min Range: -3.6788

Max Act: 2.3059. Min Act: -0.0000

Data Index: 723693 (The Pile)

Max Activating Token Index: 217

Text #10

Max Range: 3.6788. Min Range: -3.6788

Max Act: 2.1542. Min Act: -0.0000

Data Index: 519312 (The Pile)

Max Activating Token Index: 687

Text #11

Max Range: 3.6788. Min Range: -3.6788

Max Act: 1.7402. Min Act: -0.0000

Data Index: 494339 (The Pile)

Max Activating Token Index: 692

Text #12

Max Range: 3.6788. Min Range: -3.6788

Max Act: 1.8956. Min Act: -0.0000

Data Index: 437237 (The Pile)

Max Activating Token Index: 158

Text #13

Max Range: 3.6788. Min Range: -3.6788

Max Act: 2.6103. Min Act: -0.0000

Data Index: 1226529 (The Pile)

Max Activating Token Index: 937

Text #14

Max Range: 3.6788. Min Range: -3.6788

Max Act: 1.5769. Min Act: -0.0000

Data Index: 544433 (The Pile)

Max Activating Token Index: 965

Text #15

Max Range: 3.6788. Min Range: -3.6788

Max Act: 1.9399. Min Act: -0.0000

Data Index: 1604345 (The Pile)

Max Activating Token Index: 909

Text #16

Max Range: 3.6788. Min Range: -3.6788

Max Act: 1.8156. Min Act: -0.0000

Data Index: 1147346 (The Pile)

Max Activating Token Index: 33

Text #17

Max Range: 3.6788. Min Range: -3.6788

Max Act: 2.3145. Min Act: -0.0000

Data Index: 1569284 (The Pile)

Max Activating Token Index: 86

Text #18

Max Range: 3.6788. Min Range: -3.6788

Max Act: 2.6199. Min Act: -0.0000

Data Index: 1694962 (The Pile)

Max Activating Token Index: 752

Text #19

Max Range: 3.6788. Min Range: -3.6788

Max Act: 1.3466. Min Act: -0.0000

Data Index: 1162443 (The Pile)

Max Activating Token Index: 816