Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 168 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.6027. Min Range: -1.6027

Max Act: 1.6027. Min Act: -0.0000

Data Index: 1390812 (Python Code)

Max Activating Token Index: 181

Text #1

Max Range: 1.6027. Min Range: -1.6027

Max Act: 1.5495. Min Act: -0.0000

Data Index: 1492307 (Python Code)

Max Activating Token Index: 659

Text #2

Max Range: 1.6027. Min Range: -1.6027

Max Act: 1.4591. Min Act: -0.0000

Data Index: 1386824 (Python Code)

Max Activating Token Index: 554

Text #3

Max Range: 1.6027. Min Range: -1.6027

Max Act: 1.3213. Min Act: -0.0000

Data Index: 1419270 (Python Code)

Max Activating Token Index: 741

Text #4

Max Range: 1.6027. Min Range: -1.6027

Max Act: 1.3082. Min Act: -0.0000

Data Index: 1626851 (Python Code)

Max Activating Token Index: 790

Text #5

Max Range: 1.6027. Min Range: -1.6027

Max Act: 1.0934. Min Act: -0.0000

Data Index: 1588904 (Python Code)

Max Activating Token Index: 934

Text #6

Max Range: 1.6027. Min Range: -1.6027

Max Act: 1.0516. Min Act: -0.0000

Data Index: 1509895 (Python Code)

Max Activating Token Index: 255

Text #7

Max Range: 1.6027. Min Range: -1.6027

Max Act: 1.0875. Min Act: -0.0000

Data Index: 1549730 (Python Code)

Max Activating Token Index: 768

Text #8

Max Range: 1.6027. Min Range: -1.6027

Max Act: 0.9974. Min Act: -0.0000

Data Index: 1583748 (Python Code)

Max Activating Token Index: 893

Text #9

Max Range: 1.6027. Min Range: -1.6027

Max Act: 0.9937. Min Act: -0.0000

Data Index: 1579148 (Python Code)

Max Activating Token Index: 380

Text #10

Max Range: 1.6027. Min Range: -1.6027

Max Act: 0.9106. Min Act: -0.0000

Data Index: 1500108 (Python Code)

Max Activating Token Index: 856

Text #11

Max Range: 1.6027. Min Range: -1.6027

Max Act: 0.9536. Min Act: -0.0000

Data Index: 1596331 (Python Code)

Max Activating Token Index: 264

Text #12

Max Range: 1.6027. Min Range: -1.6027

Max Act: 0.8540. Min Act: -0.0000

Data Index: 1378182 (Python Code)

Max Activating Token Index: 408

Text #13

Max Range: 1.6027. Min Range: -1.6027

Max Act: 0.7625. Min Act: -0.0000

Data Index: 1617257 (Python Code)

Max Activating Token Index: 458

Text #14

Max Range: 1.6027. Min Range: -1.6027

Max Act: 0.7614. Min Act: -0.0000

Data Index: 1407200 (Python Code)

Max Activating Token Index: 766

Text #15

Max Range: 1.6027. Min Range: -1.6027

Max Act: 0.8596. Min Act: -0.0000

Data Index: 1612048 (Python Code)

Max Activating Token Index: 489

Text #16

Max Range: 1.6027. Min Range: -1.6027

Max Act: 0.8278. Min Act: -0.0000

Data Index: 1570739 (Python Code)

Max Activating Token Index: 642

Text #17

Max Range: 1.6027. Min Range: -1.6027

Max Act: 0.7986. Min Act: -0.0000

Data Index: 1402350 (Python Code)

Max Activating Token Index: 64

Text #18

Max Range: 1.6027. Min Range: -1.6027

Max Act: 0.8203. Min Act: -0.0000

Data Index: 1641987 (Python Code)

Max Activating Token Index: 132

Text #19

Max Range: 1.6027. Min Range: -1.6027

Max Act: 0.7477. Min Act: -0.0000

Data Index: 1483545 (Python Code)

Max Activating Token Index: 216