Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5950 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.0110. Min Range: -2.0110

Max Act: 2.0110. Min Act: -0.0000

Data Index: 795733 (C4 (Web Text))

Max Activating Token Index: 104

Text #1

Max Range: 2.0110. Min Range: -2.0110

Max Act: 1.9203. Min Act: -0.0000

Data Index: 533651 (C4 (Web Text))

Max Activating Token Index: 541

Text #2

Max Range: 2.0110. Min Range: -2.0110

Max Act: 1.5426. Min Act: -0.0000

Data Index: 1255567 (C4 (Web Text))

Max Activating Token Index: 372

Text #3

Max Range: 2.0110. Min Range: -2.0110

Max Act: 1.5037. Min Act: -0.0000

Data Index: 180376 (C4 (Web Text))

Max Activating Token Index: 62

Text #4

Max Range: 2.0110. Min Range: -2.0110

Max Act: 1.5133. Min Act: -0.0000

Data Index: 1148137 (C4 (Web Text))

Max Activating Token Index: 63

Text #5

Max Range: 2.0110. Min Range: -2.0110

Max Act: 1.4752. Min Act: 0.0002

Data Index: 370505 (C4 (Web Text))

Max Activating Token Index: 141

Text #6

Max Range: 2.0110. Min Range: -2.0110

Max Act: 1.4115. Min Act: -0.0000

Data Index: 384468 (C4 (Web Text))

Max Activating Token Index: 124

Text #7

Max Range: 2.0110. Min Range: -2.0110

Max Act: 1.4693. Min Act: -0.0000

Data Index: 415493 (C4 (Web Text))

Max Activating Token Index: 92

Text #8

Max Range: 2.0110. Min Range: -2.0110

Max Act: 1.3277. Min Act: -0.0000

Data Index: 1013421 (C4 (Web Text))

Max Activating Token Index: 117

Text #9

Max Range: 2.0110. Min Range: -2.0110

Max Act: 1.3595. Min Act: -0.0000

Data Index: 40780 (C4 (Web Text))

Max Activating Token Index: 517

Text #10

Max Range: 2.0110. Min Range: -2.0110

Max Act: 1.3575. Min Act: -0.0000

Data Index: 883991 (C4 (Web Text))

Max Activating Token Index: 76

Text #11

Max Range: 2.0110. Min Range: -2.0110

Max Act: 1.2358. Min Act: 0.0000

Data Index: 491969 (C4 (Web Text))

Max Activating Token Index: 599

Text #12

Max Range: 2.0110. Min Range: -2.0110

Max Act: 1.2448. Min Act: -0.0000

Data Index: 804719 (C4 (Web Text))

Max Activating Token Index: 106

Text #13

Max Range: 2.0110. Min Range: -2.0110

Max Act: 1.2418. Min Act: -0.0000

Data Index: 45982 (C4 (Web Text))

Max Activating Token Index: 198

Text #14

Max Range: 2.0110. Min Range: -2.0110

Max Act: 1.2653. Min Act: 0.0000

Data Index: 250475 (C4 (Web Text))

Max Activating Token Index: 845

Text #15

Max Range: 2.0110. Min Range: -2.0110

Max Act: 1.2290. Min Act: -0.0000

Data Index: 577386 (C4 (Web Text))

Max Activating Token Index: 770

Text #16

Max Range: 2.0110. Min Range: -2.0110

Max Act: 1.2327. Min Act: -0.0000

Data Index: 656632 (C4 (Web Text))

Max Activating Token Index: 61

Text #17

Max Range: 2.0110. Min Range: -2.0110

Max Act: 1.1900. Min Act: -0.0000

Data Index: 799133 (C4 (Web Text))

Max Activating Token Index: 777

Text #18

Max Range: 2.0110. Min Range: -2.0110

Max Act: 1.2199. Min Act: 0.0000

Data Index: 368086 (C4 (Web Text))

Max Activating Token Index: 66

Text #19

Max Range: 2.0110. Min Range: -2.0110

Max Act: 1.2077. Min Act: -0.0000

Data Index: 247947 (C4 (Web Text))

Max Activating Token Index: 222