Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3644 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.6693. Min Range: -1.6693

Max Act: 1.6693. Min Act: -0.0000

Data Index: 924009 (C4 (Web Text))

Max Activating Token Index: 870

Text #1

Max Range: 1.6693. Min Range: -1.6693

Max Act: 1.6211. Min Act: 0.0000

Data Index: 746251 (C4 (Web Text))

Max Activating Token Index: 232

Text #2

Max Range: 1.6693. Min Range: -1.6693

Max Act: 1.4238. Min Act: -0.0000

Data Index: 966628 (C4 (Web Text))

Max Activating Token Index: 149

Text #3

Max Range: 1.6693. Min Range: -1.6693

Max Act: 1.3600. Min Act: -0.0000

Data Index: 874425 (C4 (Web Text))

Max Activating Token Index: 83

Text #4

Max Range: 1.6693. Min Range: -1.6693

Max Act: 1.3715. Min Act: -0.0000

Data Index: 965701 (C4 (Web Text))

Max Activating Token Index: 994

Text #5

Max Range: 1.6693. Min Range: -1.6693

Max Act: 1.2121. Min Act: -0.0000

Data Index: 1319045 (C4 (Web Text))

Max Activating Token Index: 300

Text #6

Max Range: 1.6693. Min Range: -1.6693

Max Act: 1.2434. Min Act: 0.0001

Data Index: 556735 (C4 (Web Text))

Max Activating Token Index: 646

Text #7

Max Range: 1.6693. Min Range: -1.6693

Max Act: 1.2100. Min Act: -0.0000

Data Index: 450769 (C4 (Web Text))

Max Activating Token Index: 34

Text #8

Max Range: 1.6693. Min Range: -1.6693

Max Act: 1.2589. Min Act: -0.0000

Data Index: 401693 (C4 (Web Text))

Max Activating Token Index: 660

Text #9

Max Range: 1.6693. Min Range: -1.6693

Max Act: 1.0807. Min Act: -0.0000

Data Index: 786337 (C4 (Web Text))

Max Activating Token Index: 56

Text #10

Max Range: 1.6693. Min Range: -1.6693

Max Act: 1.0713. Min Act: -0.0000

Data Index: 836321 (C4 (Web Text))

Max Activating Token Index: 241

Text #11

Max Range: 1.6693. Min Range: -1.6693

Max Act: 1.1345. Min Act: -0.0000

Data Index: 697035 (C4 (Web Text))

Max Activating Token Index: 108

Text #12

Max Range: 1.6693. Min Range: -1.6693

Max Act: 1.1961. Min Act: -0.0000

Data Index: 125142 (C4 (Web Text))

Max Activating Token Index: 266

Text #13

Max Range: 1.6693. Min Range: -1.6693

Max Act: 1.1292. Min Act: -0.0000

Data Index: 809757 (C4 (Web Text))

Max Activating Token Index: 530

Text #14

Max Range: 1.6693. Min Range: -1.6693

Max Act: 1.1869. Min Act: -0.0000

Data Index: 17221 (C4 (Web Text))

Max Activating Token Index: 990

Text #15

Max Range: 1.6693. Min Range: -1.6693

Max Act: 1.0485. Min Act: 0.0001

Data Index: 693851 (C4 (Web Text))

Max Activating Token Index: 430

Text #16

Max Range: 1.6693. Min Range: -1.6693

Max Act: 1.1087. Min Act: -0.0000

Data Index: 927527 (C4 (Web Text))

Max Activating Token Index: 463

Text #17

Max Range: 1.6693. Min Range: -1.6693

Max Act: 1.1344. Min Act: 0.0000

Data Index: 314643 (C4 (Web Text))

Max Activating Token Index: 44

Text #18

Max Range: 1.6693. Min Range: -1.6693

Max Act: 1.0028. Min Act: 0.0000

Data Index: 491244 (C4 (Web Text))

Max Activating Token Index: 50

Text #19

Max Range: 1.6693. Min Range: -1.6693

Max Act: 1.0471. Min Act: -0.0000

Data Index: 349813 (C4 (Web Text))

Max Activating Token Index: 13