Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1522 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 9.2823. Min Range: -9.2823

Max Act: 9.2823. Min Act: -0.0000

Data Index: 1277364 (C4 (Web Text))

Max Activating Token Index: 175

Text #1

Max Range: 9.2823. Min Range: -9.2823

Max Act: 8.4621. Min Act: -0.0000

Data Index: 233398 (C4 (Web Text))

Max Activating Token Index: 21

Text #2

Max Range: 9.2823. Min Range: -9.2823

Max Act: 7.8041. Min Act: -0.0000

Data Index: 296005 (C4 (Web Text))

Max Activating Token Index: 16

Text #3

Max Range: 9.2823. Min Range: -9.2823

Max Act: 7.3563. Min Act: -0.0000

Data Index: 993194 (C4 (Web Text))

Max Activating Token Index: 253

Text #4

Max Range: 9.2823. Min Range: -9.2823

Max Act: 7.1956. Min Act: -0.0000

Data Index: 1015877 (C4 (Web Text))

Max Activating Token Index: 688

Text #5

Max Range: 9.2823. Min Range: -9.2823

Max Act: 6.8284. Min Act: -0.0000

Data Index: 618584 (C4 (Web Text))

Max Activating Token Index: 13

Text #6

Max Range: 9.2823. Min Range: -9.2823

Max Act: 6.8427. Min Act: -0.0000

Data Index: 461141 (C4 (Web Text))

Max Activating Token Index: 454

Text #7

Max Range: 9.2823. Min Range: -9.2823

Max Act: 6.6723. Min Act: -0.0000

Data Index: 293879 (C4 (Web Text))

Max Activating Token Index: 656

Text #8

Max Range: 9.2823. Min Range: -9.2823

Max Act: 6.3710. Min Act: -0.0000

Data Index: 521175 (C4 (Web Text))

Max Activating Token Index: 215

Text #9

Max Range: 9.2823. Min Range: -9.2823

Max Act: 6.1835. Min Act: -0.0000

Data Index: 1242728 (C4 (Web Text))

Max Activating Token Index: 676

Text #10

Max Range: 9.2823. Min Range: -9.2823

Max Act: 6.1442. Min Act: -0.0000

Data Index: 1008871 (C4 (Web Text))

Max Activating Token Index: 63

Text #11

Max Range: 9.2823. Min Range: -9.2823

Max Act: 6.0386. Min Act: -0.0000

Data Index: 1102975 (C4 (Web Text))

Max Activating Token Index: 314

Text #12

Max Range: 9.2823. Min Range: -9.2823

Max Act: 6.1512. Min Act: -0.0000

Data Index: 151708 (C4 (Web Text))

Max Activating Token Index: 14

Text #13

Max Range: 9.2823. Min Range: -9.2823

Max Act: 5.7736. Min Act: -0.0000

Data Index: 246490 (C4 (Web Text))

Max Activating Token Index: 721

Text #14

Max Range: 9.2823. Min Range: -9.2823

Max Act: 5.6161. Min Act: -0.0000

Data Index: 285328 (C4 (Web Text))

Max Activating Token Index: 159

Text #15

Max Range: 9.2823. Min Range: -9.2823

Max Act: 5.5613. Min Act: -0.0000

Data Index: 1134412 (C4 (Web Text))

Max Activating Token Index: 61

Text #16

Max Range: 9.2823. Min Range: -9.2823

Max Act: 5.5954. Min Act: -0.0000

Data Index: 1300125 (C4 (Web Text))

Max Activating Token Index: 286

Text #17

Max Range: 9.2823. Min Range: -9.2823

Max Act: 5.5589. Min Act: -0.0000

Data Index: 466883 (C4 (Web Text))

Max Activating Token Index: 360

Text #18

Max Range: 9.2823. Min Range: -9.2823

Max Act: 5.6987. Min Act: -0.0000

Data Index: 51000 (C4 (Web Text))

Max Activating Token Index: 294

Text #19

Max Range: 9.2823. Min Range: -9.2823

Max Act: 5.4360. Min Act: -0.0000

Data Index: 663749 (C4 (Web Text))

Max Activating Token Index: 40