Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5922 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.3030. Min Range: -3.3030

Max Act: 3.3030. Min Act: -0.0000

Data Index: 1576682 (Python Code)

Max Activating Token Index: 770

Text #1

Max Range: 3.3030. Min Range: -3.3030

Max Act: 3.1239. Min Act: -0.0000

Data Index: 1425836 (Python Code)

Max Activating Token Index: 276

Text #2

Max Range: 3.3030. Min Range: -3.3030

Max Act: 2.9632. Min Act: -0.0000

Data Index: 1450335 (Python Code)

Max Activating Token Index: 610

Text #3

Max Range: 3.3030. Min Range: -3.3030

Max Act: 3.1064. Min Act: -0.0000

Data Index: 1429282 (Python Code)

Max Activating Token Index: 562

Text #4

Max Range: 3.3030. Min Range: -3.3030

Max Act: 3.0891. Min Act: -0.0000

Data Index: 1567596 (Python Code)

Max Activating Token Index: 383

Text #5

Max Range: 3.3030. Min Range: -3.3030

Max Act: 3.0908. Min Act: -0.0000

Data Index: 1441872 (Python Code)

Max Activating Token Index: 410

Text #6

Max Range: 3.3030. Min Range: -3.3030

Max Act: 3.0756. Min Act: -0.0000

Data Index: 1485361 (Python Code)

Max Activating Token Index: 516

Text #7

Max Range: 3.3030. Min Range: -3.3030

Max Act: 3.1708. Min Act: -0.0000

Data Index: 1461081 (Python Code)

Max Activating Token Index: 402

Text #8

Max Range: 3.3030. Min Range: -3.3030

Max Act: 3.0526. Min Act: -0.0000

Data Index: 1591283 (Python Code)

Max Activating Token Index: 47

Text #9

Max Range: 3.3030. Min Range: -3.3030

Max Act: 3.0027. Min Act: -0.0000

Data Index: 1622122 (Python Code)

Max Activating Token Index: 175

Text #10

Max Range: 3.3030. Min Range: -3.3030

Max Act: 3.0056. Min Act: -0.0000

Data Index: 1647443 (Python Code)

Max Activating Token Index: 22

Text #11

Max Range: 3.3030. Min Range: -3.3030

Max Act: 2.8614. Min Act: -0.0000

Data Index: 1467410 (Python Code)

Max Activating Token Index: 95

Text #12

Max Range: 3.3030. Min Range: -3.3030

Max Act: 2.9707. Min Act: -0.0000

Data Index: 1423875 (Python Code)

Max Activating Token Index: 210

Text #13

Max Range: 3.3030. Min Range: -3.3030

Max Act: 2.9689. Min Act: -0.0000

Data Index: 1509390 (Python Code)

Max Activating Token Index: 619

Text #14

Max Range: 3.3030. Min Range: -3.3030

Max Act: 3.1009. Min Act: -0.0000

Data Index: 1451629 (Python Code)

Max Activating Token Index: 578

Text #15

Max Range: 3.3030. Min Range: -3.3030

Max Act: 2.9443. Min Act: -0.0000

Data Index: 1378029 (Python Code)

Max Activating Token Index: 52

Text #16

Max Range: 3.3030. Min Range: -3.3030

Max Act: 2.9227. Min Act: -0.0000

Data Index: 1527793 (Python Code)

Max Activating Token Index: 219

Text #17

Max Range: 3.3030. Min Range: -3.3030

Max Act: 2.7825. Min Act: -0.0000

Data Index: 1472591 (Python Code)

Max Activating Token Index: 76

Text #18

Max Range: 3.3030. Min Range: -3.3030

Max Act: 2.9147. Min Act: -0.0000

Data Index: 1549132 (Python Code)

Max Activating Token Index: 141

Text #19

Max Range: 3.3030. Min Range: -3.3030

Max Act: 2.8992. Min Act: -0.0000

Data Index: 1566601 (Python Code)

Max Activating Token Index: 102