Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4279 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.4829. Min Range: -1.4829

Max Act: 1.3973. Min Act: -0.0000

Data Index: 543592 (C4 (Web Text))

Max Activating Token Index: 657

Text #1

Max Range: 1.4829. Min Range: -1.4829

Max Act: 1.4829. Min Act: -0.0000

Data Index: 1226679 (C4 (Web Text))

Max Activating Token Index: 787

Text #2

Max Range: 1.4829. Min Range: -1.4829

Max Act: 1.1504. Min Act: -0.0000

Data Index: 333595 (C4 (Web Text))

Max Activating Token Index: 1011

Text #3

Max Range: 1.4829. Min Range: -1.4829

Max Act: 1.0553. Min Act: -0.0000

Data Index: 51819 (C4 (Web Text))

Max Activating Token Index: 981

Text #4

Max Range: 1.4829. Min Range: -1.4829

Max Act: 1.0335. Min Act: -0.0000

Data Index: 160780 (C4 (Web Text))

Max Activating Token Index: 416

Text #5

Max Range: 1.4829. Min Range: -1.4829

Max Act: 1.0084. Min Act: -0.0000

Data Index: 296585 (C4 (Web Text))

Max Activating Token Index: 749

Text #6

Max Range: 1.4829. Min Range: -1.4829

Max Act: 0.9137. Min Act: -0.0000

Data Index: 556988 (C4 (Web Text))

Max Activating Token Index: 561

Text #7

Max Range: 1.4829. Min Range: -1.4829

Max Act: 0.9979. Min Act: -0.0000

Data Index: 854873 (C4 (Web Text))

Max Activating Token Index: 433

Text #8

Max Range: 1.4829. Min Range: -1.4829

Max Act: 0.9869. Min Act: -0.0000

Data Index: 447693 (C4 (Web Text))

Max Activating Token Index: 21

Text #9

Max Range: 1.4829. Min Range: -1.4829

Max Act: 0.9756. Min Act: -0.0000

Data Index: 210855 (C4 (Web Text))

Max Activating Token Index: 195

Text #10

Max Range: 1.4829. Min Range: -1.4829

Max Act: 0.9135. Min Act: -0.0000

Data Index: 503105 (C4 (Web Text))

Max Activating Token Index: 32

Text #11

Max Range: 1.4829. Min Range: -1.4829

Max Act: 0.9647. Min Act: -0.0000

Data Index: 1317335 (C4 (Web Text))

Max Activating Token Index: 716

Text #12

Max Range: 1.4829. Min Range: -1.4829

Max Act: 0.8492. Min Act: -0.0000

Data Index: 484646 (C4 (Web Text))

Max Activating Token Index: 559

Text #13

Max Range: 1.4829. Min Range: -1.4829

Max Act: 0.9158. Min Act: -0.0000

Data Index: 1112323 (C4 (Web Text))

Max Activating Token Index: 419

Text #14

Max Range: 1.4829. Min Range: -1.4829

Max Act: 0.8810. Min Act: -0.0000

Data Index: 1054495 (C4 (Web Text))

Max Activating Token Index: 705

Text #15

Max Range: 1.4829. Min Range: -1.4829

Max Act: 0.8869. Min Act: -0.0000

Data Index: 698433 (C4 (Web Text))

Max Activating Token Index: 124

Text #16

Max Range: 1.4829. Min Range: -1.4829

Max Act: 0.8449. Min Act: -0.0000

Data Index: 438555 (C4 (Web Text))

Max Activating Token Index: 332

Text #17

Max Range: 1.4829. Min Range: -1.4829

Max Act: 0.8140. Min Act: -0.0000

Data Index: 705694 (C4 (Web Text))

Max Activating Token Index: 81

Text #18

Max Range: 1.4829. Min Range: -1.4829

Max Act: 0.8490. Min Act: -0.0000

Data Index: 1099297 (C4 (Web Text))

Max Activating Token Index: 606

Text #19

Max Range: 1.4829. Min Range: -1.4829

Max Act: 0.8601. Min Act: -0.0000

Data Index: 1076062 (C4 (Web Text))

Max Activating Token Index: 385