Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1042 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.8033. Min Range: -1.8033

Max Act: 1.8033. Min Act: -0.0000

Data Index: 440644 (C4 (Web Text))

Max Activating Token Index: 687

Text #1

Max Range: 1.8033. Min Range: -1.8033

Max Act: 1.7481. Min Act: -0.0000

Data Index: 937248 (C4 (Web Text))

Max Activating Token Index: 158

Text #2

Max Range: 1.8033. Min Range: -1.8033

Max Act: 1.5371. Min Act: -0.0000

Data Index: 1144473 (C4 (Web Text))

Max Activating Token Index: 154

Text #3

Max Range: 1.8033. Min Range: -1.8033

Max Act: 0.9709. Min Act: -0.0000

Data Index: 1619920 (Python Code)

Max Activating Token Index: 617

Text #4

Max Range: 1.8033. Min Range: -1.8033

Max Act: 1.0921. Min Act: -0.0000

Data Index: 1621358 (Python Code)

Max Activating Token Index: 8

Text #5

Max Range: 1.8033. Min Range: -1.8033

Max Act: 0.8769. Min Act: -0.0000

Data Index: 540836 (C4 (Web Text))

Max Activating Token Index: 662

Text #6

Max Range: 1.8033. Min Range: -1.8033

Max Act: 0.8798. Min Act: -0.0000

Data Index: 1633111 (Python Code)

Max Activating Token Index: 907

Text #7

Max Range: 1.8033. Min Range: -1.8033

Max Act: 0.9182. Min Act: -0.0000

Data Index: 967995 (C4 (Web Text))

Max Activating Token Index: 987

Text #8

Max Range: 1.8033. Min Range: -1.8033

Max Act: 0.9656. Min Act: -0.0000

Data Index: 547170 (C4 (Web Text))

Max Activating Token Index: 909

Text #9

Max Range: 1.8033. Min Range: -1.8033

Max Act: 0.9331. Min Act: -0.0000

Data Index: 1639171 (Python Code)

Max Activating Token Index: 300

Text #10

Max Range: 1.8033. Min Range: -1.8033

Max Act: 0.9021. Min Act: -0.0000

Data Index: 1148970 (C4 (Web Text))

Max Activating Token Index: 705

Text #11

Max Range: 1.8033. Min Range: -1.8033

Max Act: 0.8555. Min Act: -0.0000

Data Index: 1504126 (Python Code)

Max Activating Token Index: 1017

Text #12

Max Range: 1.8033. Min Range: -1.8033

Max Act: 0.8250. Min Act: -0.0000

Data Index: 1400429 (Python Code)

Max Activating Token Index: 59

Text #13

Max Range: 1.8033. Min Range: -1.8033

Max Act: 0.8153. Min Act: -0.0000

Data Index: 849592 (C4 (Web Text))

Max Activating Token Index: 915

Text #14

Max Range: 1.8033. Min Range: -1.8033

Max Act: 0.8649. Min Act: -0.0000

Data Index: 1114229 (C4 (Web Text))

Max Activating Token Index: 486

Text #15

Max Range: 1.8033. Min Range: -1.8033

Max Act: 0.8095. Min Act: -0.0000

Data Index: 449374 (C4 (Web Text))

Max Activating Token Index: 648

Text #16

Max Range: 1.8033. Min Range: -1.8033

Max Act: 0.7953. Min Act: -0.0000

Data Index: 181859 (C4 (Web Text))

Max Activating Token Index: 264

Text #17

Max Range: 1.8033. Min Range: -1.8033

Max Act: 0.7803. Min Act: -0.0000

Data Index: 1377117 (Python Code)

Max Activating Token Index: 640

Text #18

Max Range: 1.8033. Min Range: -1.8033

Max Act: 0.7420. Min Act: -0.0000

Data Index: 1283771 (C4 (Web Text))

Max Activating Token Index: 663

Text #19

Max Range: 1.8033. Min Range: -1.8033

Max Act: 0.7855. Min Act: 0.0000

Data Index: 1466935 (Python Code)

Max Activating Token Index: 424