Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3564 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.7590. Min Range: -2.7590

Max Act: 2.4094. Min Act: -0.0000

Data Index: 1072379 (C4 (Web Text))

Max Activating Token Index: 274

Text #1

Max Range: 2.7590. Min Range: -2.7590

Max Act: 2.7590. Min Act: -0.0000

Data Index: 122335 (C4 (Web Text))

Max Activating Token Index: 901

Text #2

Max Range: 2.7590. Min Range: -2.7590

Max Act: 2.1396. Min Act: -0.0000

Data Index: 1038480 (C4 (Web Text))

Max Activating Token Index: 548

Text #3

Max Range: 2.7590. Min Range: -2.7590

Max Act: 2.0744. Min Act: -0.0000

Data Index: 660128 (C4 (Web Text))

Max Activating Token Index: 594

Text #4

Max Range: 2.7590. Min Range: -2.7590

Max Act: 2.1822. Min Act: 0.0000

Data Index: 440678 (C4 (Web Text))

Max Activating Token Index: 53

Text #5

Max Range: 2.7590. Min Range: -2.7590

Max Act: 2.0169. Min Act: -0.0000

Data Index: 458672 (C4 (Web Text))

Max Activating Token Index: 158

Text #6

Max Range: 2.7590. Min Range: -2.7590

Max Act: 1.9159. Min Act: -0.0000

Data Index: 951679 (C4 (Web Text))

Max Activating Token Index: 896

Text #7

Max Range: 2.7590. Min Range: -2.7590

Max Act: 1.9085. Min Act: -0.0000

Data Index: 1019012 (C4 (Web Text))

Max Activating Token Index: 85

Text #8

Max Range: 2.7590. Min Range: -2.7590

Max Act: 1.8501. Min Act: -0.0000

Data Index: 83857 (C4 (Web Text))

Max Activating Token Index: 850

Text #9

Max Range: 2.7590. Min Range: -2.7590

Max Act: 1.9006. Min Act: -0.0000

Data Index: 401877 (C4 (Web Text))

Max Activating Token Index: 393

Text #10

Max Range: 2.7590. Min Range: -2.7590

Max Act: 1.6363. Min Act: -0.0000

Data Index: 980266 (C4 (Web Text))

Max Activating Token Index: 419

Text #11

Max Range: 2.7590. Min Range: -2.7590

Max Act: 1.7394. Min Act: -0.0000

Data Index: 590202 (C4 (Web Text))

Max Activating Token Index: 417

Text #12

Max Range: 2.7590. Min Range: -2.7590

Max Act: 1.6911. Min Act: -0.0000

Data Index: 789016 (C4 (Web Text))

Max Activating Token Index: 211

Text #13

Max Range: 2.7590. Min Range: -2.7590

Max Act: 1.6899. Min Act: -0.0000

Data Index: 1227777 (C4 (Web Text))

Max Activating Token Index: 726

Text #14

Max Range: 2.7590. Min Range: -2.7590

Max Act: 1.6698. Min Act: -0.0000

Data Index: 556275 (C4 (Web Text))

Max Activating Token Index: 180

Text #15

Max Range: 2.7590. Min Range: -2.7590

Max Act: 1.7529. Min Act: -0.0000

Data Index: 1028203 (C4 (Web Text))

Max Activating Token Index: 608

Text #16

Max Range: 2.7590. Min Range: -2.7590

Max Act: 1.7531. Min Act: -0.0000

Data Index: 541126 (C4 (Web Text))

Max Activating Token Index: 561

Text #17

Max Range: 2.7590. Min Range: -2.7590

Max Act: 1.5440. Min Act: -0.0000

Data Index: 1039558 (C4 (Web Text))

Max Activating Token Index: 570

Text #18

Max Range: 2.7590. Min Range: -2.7590

Max Act: 1.6300. Min Act: -0.0000

Data Index: 1076383 (C4 (Web Text))

Max Activating Token Index: 587

Text #19

Max Range: 2.7590. Min Range: -2.7590

Max Act: 1.5258. Min Act: -0.0000

Data Index: 1122 (C4 (Web Text))

Max Activating Token Index: 428