Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 11 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 3.0655. Min Range: -3.0655

Max Act: 2.9503. Min Act: -0.0001

Data Index: 1478852 (Python Code)

Max Activating Token Index: 907

Text #1

Max Range: 3.0655. Min Range: -3.0655

Max Act: 3.0655. Min Act: -0.0001

Data Index: 1505942 (Python Code)

Max Activating Token Index: 33

Text #2

Max Range: 3.0655. Min Range: -3.0655

Max Act: 2.5916. Min Act: -0.0001

Data Index: 1393673 (Python Code)

Max Activating Token Index: 140

Text #3

Max Range: 3.0655. Min Range: -3.0655

Max Act: 2.6039. Min Act: -0.0000

Data Index: 1433641 (Python Code)

Max Activating Token Index: 930

Text #4

Max Range: 3.0655. Min Range: -3.0655

Max Act: 2.4589. Min Act: -0.0001

Data Index: 1530686 (Python Code)

Max Activating Token Index: 956

Text #5

Max Range: 3.0655. Min Range: -3.0655

Max Act: 2.4812. Min Act: -0.0000

Data Index: 1428040 (Python Code)

Max Activating Token Index: 592

Text #6

Max Range: 3.0655. Min Range: -3.0655

Max Act: 2.4752. Min Act: -0.0001

Data Index: 1638468 (Python Code)

Max Activating Token Index: 641

Text #7

Max Range: 3.0655. Min Range: -3.0655

Max Act: 2.1863. Min Act: -0.0001

Data Index: 1362141 (Python Code)

Max Activating Token Index: 276

Text #8

Max Range: 3.0655. Min Range: -3.0655

Max Act: 2.2216. Min Act: -0.0001

Data Index: 1405634 (Python Code)

Max Activating Token Index: 956

Text #9

Max Range: 3.0655. Min Range: -3.0655

Max Act: 2.2031. Min Act: -0.0001

Data Index: 1366817 (Python Code)

Max Activating Token Index: 547

Text #10

Max Range: 3.0655. Min Range: -3.0655

Max Act: 2.0794. Min Act: -0.0001

Data Index: 1438552 (Python Code)

Max Activating Token Index: 497

Text #11

Max Range: 3.0655. Min Range: -3.0655

Max Act: 1.9913. Min Act: -0.0000

Data Index: 1509341 (Python Code)

Max Activating Token Index: 588

Text #12

Max Range: 3.0655. Min Range: -3.0655

Max Act: 2.0678. Min Act: -0.0001

Data Index: 1627097 (Python Code)

Max Activating Token Index: 652

Text #13

Max Range: 3.0655. Min Range: -3.0655

Max Act: 2.0372. Min Act: -0.0001

Data Index: 1456265 (Python Code)

Max Activating Token Index: 884

Text #14

Max Range: 3.0655. Min Range: -3.0655

Max Act: 2.1109. Min Act: -0.0000

Data Index: 1626728 (Python Code)

Max Activating Token Index: 889

Text #15

Max Range: 3.0655. Min Range: -3.0655

Max Act: 2.1030. Min Act: -0.0001

Data Index: 1540709 (Python Code)

Max Activating Token Index: 1014

Text #16

Max Range: 3.0655. Min Range: -3.0655

Max Act: 2.1105. Min Act: -0.0001

Data Index: 1488247 (Python Code)

Max Activating Token Index: 515

Text #17

Max Range: 3.0655. Min Range: -3.0655

Max Act: 2.0958. Min Act: -0.0001

Data Index: 1600441 (Python Code)

Max Activating Token Index: 49

Text #18

Max Range: 3.0655. Min Range: -3.0655

Max Act: 2.0739. Min Act: -0.0001

Data Index: 1509252 (Python Code)

Max Activating Token Index: 199

Text #19

Max Range: 3.0655. Min Range: -3.0655

Max Act: 2.0487. Min Act: -0.0000

Data Index: 1462417 (Python Code)

Max Activating Token Index: 498