Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3667 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.4527. Min Range: -2.4527

Max Act: 2.4527. Min Act: -0.0000

Data Index: 277733 (C4 (Web Text))

Max Activating Token Index: 493

Text #1

Max Range: 2.4527. Min Range: -2.4527

Max Act: 2.1971. Min Act: -0.0000

Data Index: 354124 (C4 (Web Text))

Max Activating Token Index: 533

Text #2

Max Range: 2.4527. Min Range: -2.4527

Max Act: 2.1610. Min Act: -0.0000

Data Index: 1293742 (C4 (Web Text))

Max Activating Token Index: 918

Text #3

Max Range: 2.4527. Min Range: -2.4527

Max Act: 2.1886. Min Act: -0.0000

Data Index: 576440 (C4 (Web Text))

Max Activating Token Index: 274

Text #4

Max Range: 2.4527. Min Range: -2.4527

Max Act: 2.0208. Min Act: -0.0000

Data Index: 531756 (C4 (Web Text))

Max Activating Token Index: 59

Text #5

Max Range: 2.4527. Min Range: -2.4527

Max Act: 1.9620. Min Act: -0.0000

Data Index: 545883 (C4 (Web Text))

Max Activating Token Index: 129

Text #6

Max Range: 2.4527. Min Range: -2.4527

Max Act: 1.8766. Min Act: -0.0000

Data Index: 574742 (C4 (Web Text))

Max Activating Token Index: 38

Text #7

Max Range: 2.4527. Min Range: -2.4527

Max Act: 1.8086. Min Act: -0.0000

Data Index: 830798 (C4 (Web Text))

Max Activating Token Index: 462

Text #8

Max Range: 2.4527. Min Range: -2.4527

Max Act: 1.7637. Min Act: -0.0000

Data Index: 646786 (C4 (Web Text))

Max Activating Token Index: 617

Text #9

Max Range: 2.4527. Min Range: -2.4527

Max Act: 1.6953. Min Act: -0.0000

Data Index: 713231 (C4 (Web Text))

Max Activating Token Index: 511

Text #10

Max Range: 2.4527. Min Range: -2.4527

Max Act: 1.7316. Min Act: -0.0000

Data Index: 1199540 (C4 (Web Text))

Max Activating Token Index: 590

Text #11

Max Range: 2.4527. Min Range: -2.4527

Max Act: 1.6028. Min Act: -0.0000

Data Index: 1170061 (C4 (Web Text))

Max Activating Token Index: 752

Text #12

Max Range: 2.4527. Min Range: -2.4527

Max Act: 1.6873. Min Act: -0.0000

Data Index: 896630 (C4 (Web Text))

Max Activating Token Index: 92

Text #13

Max Range: 2.4527. Min Range: -2.4527

Max Act: 1.5481. Min Act: -0.0000

Data Index: 1319499 (C4 (Web Text))

Max Activating Token Index: 721

Text #14

Max Range: 2.4527. Min Range: -2.4527

Max Act: 1.4707. Min Act: -0.0000

Data Index: 730774 (C4 (Web Text))

Max Activating Token Index: 289

Text #15

Max Range: 2.4527. Min Range: -2.4527

Max Act: 1.5205. Min Act: -0.0000

Data Index: 911061 (C4 (Web Text))

Max Activating Token Index: 594

Text #16

Max Range: 2.4527. Min Range: -2.4527

Max Act: 1.7837. Min Act: -0.0000

Data Index: 157240 (C4 (Web Text))

Max Activating Token Index: 765

Text #17

Max Range: 2.4527. Min Range: -2.4527

Max Act: 1.5177. Min Act: -0.0000

Data Index: 573823 (C4 (Web Text))

Max Activating Token Index: 26

Text #18

Max Range: 2.4527. Min Range: -2.4527

Max Act: 1.5054. Min Act: -0.0000

Data Index: 1306332 (C4 (Web Text))

Max Activating Token Index: 78

Text #19

Max Range: 2.4527. Min Range: -2.4527

Max Act: 1.4346. Min Act: -0.0000

Data Index: 642043 (C4 (Web Text))

Max Activating Token Index: 229