Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3617 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.1936. Min Range: -0.1936

Max Act: 0.1936. Min Act: -0.0000

Data Index: 87976 (C4 (Web Text))

Max Activating Token Index: 911

Text #1

Max Range: 0.1936. Min Range: -0.1936

Max Act: 0.1436. Min Act: -0.0000

Data Index: 1160708 (C4 (Web Text))

Max Activating Token Index: 841

Text #2

Max Range: 0.1936. Min Range: -0.1936

Max Act: 0.1419. Min Act: -0.0000

Data Index: 684214 (C4 (Web Text))

Max Activating Token Index: 362

Text #3

Max Range: 0.1936. Min Range: -0.1936

Max Act: 0.1260. Min Act: -0.0000

Data Index: 886539 (C4 (Web Text))

Max Activating Token Index: 210

Text #4

Max Range: 0.1936. Min Range: -0.1936

Max Act: 0.1269. Min Act: -0.0000

Data Index: 1036025 (C4 (Web Text))

Max Activating Token Index: 543

Text #5

Max Range: 0.1936. Min Range: -0.1936

Max Act: 0.1244. Min Act: -0.0000

Data Index: 881756 (C4 (Web Text))

Max Activating Token Index: 168

Text #6

Max Range: 0.1936. Min Range: -0.1936

Max Act: 0.1156. Min Act: -0.0000

Data Index: 1120068 (C4 (Web Text))

Max Activating Token Index: 971

Text #7

Max Range: 0.1936. Min Range: -0.1936

Max Act: 0.1091. Min Act: -0.0000

Data Index: 620244 (C4 (Web Text))

Max Activating Token Index: 691

Text #8

Max Range: 0.1936. Min Range: -0.1936

Max Act: 0.1074. Min Act: -0.0000

Data Index: 303851 (C4 (Web Text))

Max Activating Token Index: 608

Text #9

Max Range: 0.1936. Min Range: -0.1936

Max Act: 0.1010. Min Act: -0.0000

Data Index: 1284698 (C4 (Web Text))

Max Activating Token Index: 726

Text #10

Max Range: 0.1936. Min Range: -0.1936

Max Act: 0.1055. Min Act: -0.0000

Data Index: 7943 (C4 (Web Text))

Max Activating Token Index: 374

Text #11

Max Range: 0.1936. Min Range: -0.1936

Max Act: 0.1043. Min Act: -0.0000

Data Index: 295121 (C4 (Web Text))

Max Activating Token Index: 942

Text #12

Max Range: 0.1936. Min Range: -0.1936

Max Act: 0.1007. Min Act: -0.0000

Data Index: 133659 (C4 (Web Text))

Max Activating Token Index: 631

Text #13

Max Range: 0.1936. Min Range: -0.1936

Max Act: 0.1004. Min Act: -0.0000

Data Index: 1305313 (C4 (Web Text))

Max Activating Token Index: 714

Text #14

Max Range: 0.1936. Min Range: -0.1936

Max Act: 0.1010. Min Act: -0.0000

Data Index: 795129 (C4 (Web Text))

Max Activating Token Index: 261

Text #15

Max Range: 0.1936. Min Range: -0.1936

Max Act: 0.0956. Min Act: -0.0000

Data Index: 64319 (C4 (Web Text))

Max Activating Token Index: 273

Text #16

Max Range: 0.1936. Min Range: -0.1936

Max Act: 0.0938. Min Act: -0.0000

Data Index: 614749 (C4 (Web Text))

Max Activating Token Index: 345

Text #17

Max Range: 0.1936. Min Range: -0.1936

Max Act: 0.0946. Min Act: -0.0000

Data Index: 566792 (C4 (Web Text))

Max Activating Token Index: 360

Text #18

Max Range: 0.1936. Min Range: -0.1936

Max Act: 0.0953. Min Act: -0.0000

Data Index: 709474 (C4 (Web Text))

Max Activating Token Index: 112

Text #19

Max Range: 0.1936. Min Range: -0.1936

Max Act: 0.0948. Min Act: -0.0000

Data Index: 617996 (C4 (Web Text))

Max Activating Token Index: 811