Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5124 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.4635. Min Range: -1.4635

Max Act: 1.4635. Min Act: -0.0000

Data Index: 1377069 (Python Code)

Max Activating Token Index: 675

Text #1

Max Range: 1.4635. Min Range: -1.4635

Max Act: 1.3168. Min Act: -0.0000

Data Index: 337849 (C4 (Web Text))

Max Activating Token Index: 558

Text #2

Max Range: 1.4635. Min Range: -1.4635

Max Act: 1.3545. Min Act: -0.0000

Data Index: 348514 (C4 (Web Text))

Max Activating Token Index: 324

Text #3

Max Range: 1.4635. Min Range: -1.4635

Max Act: 1.3454. Min Act: -0.0000

Data Index: 1639017 (Python Code)

Max Activating Token Index: 158

Text #4

Max Range: 1.4635. Min Range: -1.4635

Max Act: 1.1657. Min Act: -0.0000

Data Index: 543597 (C4 (Web Text))

Max Activating Token Index: 730

Text #5

Max Range: 1.4635. Min Range: -1.4635

Max Act: 1.1990. Min Act: -0.0000

Data Index: 1603405 (Python Code)

Max Activating Token Index: 66

Text #6

Max Range: 1.4635. Min Range: -1.4635

Max Act: 1.1747. Min Act: -0.0000

Data Index: 485607 (C4 (Web Text))

Max Activating Token Index: 334

Text #7

Max Range: 1.4635. Min Range: -1.4635

Max Act: 1.1028. Min Act: -0.0000

Data Index: 1134068 (C4 (Web Text))

Max Activating Token Index: 312

Text #8

Max Range: 1.4635. Min Range: -1.4635

Max Act: 0.8457. Min Act: -0.0000

Data Index: 250377 (C4 (Web Text))

Max Activating Token Index: 785

Text #9

Max Range: 1.4635. Min Range: -1.4635

Max Act: 1.0350. Min Act: -0.0000

Data Index: 1614075 (Python Code)

Max Activating Token Index: 58

Text #10

Max Range: 1.4635. Min Range: -1.4635

Max Act: 1.0508. Min Act: -0.0000

Data Index: 1395189 (Python Code)

Max Activating Token Index: 154

Text #11

Max Range: 1.4635. Min Range: -1.4635

Max Act: 1.1604. Min Act: -0.0000

Data Index: 826548 (C4 (Web Text))

Max Activating Token Index: 716

Text #12

Max Range: 1.4635. Min Range: -1.4635

Max Act: 1.0698. Min Act: -0.0000

Data Index: 1629604 (Python Code)

Max Activating Token Index: 143

Text #13

Max Range: 1.4635. Min Range: -1.4635

Max Act: 0.9842. Min Act: -0.0000

Data Index: 1625928 (Python Code)

Max Activating Token Index: 81

Text #14

Max Range: 1.4635. Min Range: -1.4635

Max Act: 0.9990. Min Act: -0.0000

Data Index: 1414383 (Python Code)

Max Activating Token Index: 832

Text #15

Max Range: 1.4635. Min Range: -1.4635

Max Act: 0.8891. Min Act: -0.0000

Data Index: 1563700 (Python Code)

Max Activating Token Index: 727

Text #16

Max Range: 1.4635. Min Range: -1.4635

Max Act: 0.9329. Min Act: -0.0000

Data Index: 744610 (C4 (Web Text))

Max Activating Token Index: 418

Text #17

Max Range: 1.4635. Min Range: -1.4635

Max Act: 0.9084. Min Act: -0.0000

Data Index: 1589730 (Python Code)

Max Activating Token Index: 417

Text #18

Max Range: 1.4635. Min Range: -1.4635

Max Act: 0.9432. Min Act: -0.0000

Data Index: 1627706 (Python Code)

Max Activating Token Index: 149

Text #19

Max Range: 1.4635. Min Range: -1.4635

Max Act: 0.9142. Min Act: -0.0000

Data Index: 327349 (C4 (Web Text))

Max Activating Token Index: 755