Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 378 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.6368. Min Range: -0.6368

Max Act: 0.6368. Min Act: -0.0000

Data Index: 754342 (C4 (Web Text))

Max Activating Token Index: 470

Text #1

Max Range: 0.6368. Min Range: -0.6368

Max Act: 0.5469. Min Act: -0.0000

Data Index: 375729 (C4 (Web Text))

Max Activating Token Index: 245

Text #2

Max Range: 0.6368. Min Range: -0.6368

Max Act: 0.5023. Min Act: -0.0000

Data Index: 709930 (C4 (Web Text))

Max Activating Token Index: 402

Text #3

Max Range: 0.6368. Min Range: -0.6368

Max Act: 0.5133. Min Act: -0.0000

Data Index: 653045 (C4 (Web Text))

Max Activating Token Index: 706

Text #4

Max Range: 0.6368. Min Range: -0.6368

Max Act: 0.5143. Min Act: -0.0000

Data Index: 865011 (C4 (Web Text))

Max Activating Token Index: 171

Text #5

Max Range: 0.6368. Min Range: -0.6368

Max Act: 0.4785. Min Act: -0.0000

Data Index: 1309599 (C4 (Web Text))

Max Activating Token Index: 976

Text #6

Max Range: 0.6368. Min Range: -0.6368

Max Act: 0.4876. Min Act: -0.0000

Data Index: 1002317 (C4 (Web Text))

Max Activating Token Index: 227

Text #7

Max Range: 0.6368. Min Range: -0.6368

Max Act: 0.4879. Min Act: -0.0000

Data Index: 917445 (C4 (Web Text))

Max Activating Token Index: 831

Text #8

Max Range: 0.6368. Min Range: -0.6368

Max Act: 0.5008. Min Act: -0.0000

Data Index: 644969 (C4 (Web Text))

Max Activating Token Index: 754

Text #9

Max Range: 0.6368. Min Range: -0.6368

Max Act: 0.4786. Min Act: -0.0000

Data Index: 88060 (C4 (Web Text))

Max Activating Token Index: 559

Text #10

Max Range: 0.6368. Min Range: -0.6368

Max Act: 0.4785. Min Act: -0.0000

Data Index: 711696 (C4 (Web Text))

Max Activating Token Index: 577

Text #11

Max Range: 0.6368. Min Range: -0.6368

Max Act: 0.4524. Min Act: -0.0000

Data Index: 1347795 (C4 (Web Text))

Max Activating Token Index: 656

Text #12

Max Range: 0.6368. Min Range: -0.6368

Max Act: 0.4783. Min Act: -0.0000

Data Index: 171232 (C4 (Web Text))

Max Activating Token Index: 795

Text #13

Max Range: 0.6368. Min Range: -0.6368

Max Act: 0.4529. Min Act: -0.0000

Data Index: 426926 (C4 (Web Text))

Max Activating Token Index: 472

Text #14

Max Range: 0.6368. Min Range: -0.6368

Max Act: 0.4623. Min Act: -0.0000

Data Index: 783045 (C4 (Web Text))

Max Activating Token Index: 168

Text #15

Max Range: 0.6368. Min Range: -0.6368

Max Act: 0.4463. Min Act: -0.0000

Data Index: 1071324 (C4 (Web Text))

Max Activating Token Index: 842

Text #16

Max Range: 0.6368. Min Range: -0.6368

Max Act: 0.4467. Min Act: -0.0000

Data Index: 164440 (C4 (Web Text))

Max Activating Token Index: 946

Text #17

Max Range: 0.6368. Min Range: -0.6368

Max Act: 0.4440. Min Act: -0.0000

Data Index: 401362 (C4 (Web Text))

Max Activating Token Index: 573

Text #18

Max Range: 0.6368. Min Range: -0.6368

Max Act: 0.4379. Min Act: -0.0000

Data Index: 1265073 (C4 (Web Text))

Max Activating Token Index: 169

Text #19

Max Range: 0.6368. Min Range: -0.6368

Max Act: 0.4448. Min Act: -0.0000

Data Index: 559170 (C4 (Web Text))

Max Activating Token Index: 715