Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1222 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.1653. Min Range: -0.1653

Max Act: 0.1653. Min Act: -0.0000

Data Index: 706435 (C4 (Web Text))

Max Activating Token Index: 636

Text #1

Max Range: 0.1653. Min Range: -0.1653

Max Act: 0.1350. Min Act: -0.0000

Data Index: 287115 (C4 (Web Text))

Max Activating Token Index: 524

Text #2

Max Range: 0.1653. Min Range: -0.1653

Max Act: 0.1119. Min Act: -0.0000

Data Index: 330644 (C4 (Web Text))

Max Activating Token Index: 741

Text #3

Max Range: 0.1653. Min Range: -0.1653

Max Act: 0.1223. Min Act: -0.0000

Data Index: 1180515 (C4 (Web Text))

Max Activating Token Index: 111

Text #4

Max Range: 0.1653. Min Range: -0.1653

Max Act: 0.1297. Min Act: -0.0000

Data Index: 484404 (C4 (Web Text))

Max Activating Token Index: 161

Text #5

Max Range: 0.1653. Min Range: -0.1653

Max Act: 0.1202. Min Act: -0.0000

Data Index: 283049 (C4 (Web Text))

Max Activating Token Index: 603

Text #6

Max Range: 0.1653. Min Range: -0.1653

Max Act: 0.1232. Min Act: -0.0000

Data Index: 77839 (C4 (Web Text))

Max Activating Token Index: 101

Text #7

Max Range: 0.1653. Min Range: -0.1653

Max Act: 0.1100. Min Act: -0.0000

Data Index: 1033713 (C4 (Web Text))

Max Activating Token Index: 686

Text #8

Max Range: 0.1653. Min Range: -0.1653

Max Act: 0.1123. Min Act: -0.0000

Data Index: 836864 (C4 (Web Text))

Max Activating Token Index: 830

Text #9

Max Range: 0.1653. Min Range: -0.1653

Max Act: 0.1153. Min Act: -0.0000

Data Index: 189640 (C4 (Web Text))

Max Activating Token Index: 978

Text #10

Max Range: 0.1653. Min Range: -0.1653

Max Act: 0.1172. Min Act: -0.0000

Data Index: 193919 (C4 (Web Text))

Max Activating Token Index: 658

Text #11

Max Range: 0.1653. Min Range: -0.1653

Max Act: 0.1170. Min Act: -0.0000

Data Index: 128956 (C4 (Web Text))

Max Activating Token Index: 999

Text #12

Max Range: 0.1653. Min Range: -0.1653

Max Act: 0.1099. Min Act: -0.0000

Data Index: 56229 (C4 (Web Text))

Max Activating Token Index: 315

Text #13

Max Range: 0.1653. Min Range: -0.1653

Max Act: 0.1055. Min Act: -0.0000

Data Index: 531816 (C4 (Web Text))

Max Activating Token Index: 448

Text #14

Max Range: 0.1653. Min Range: -0.1653

Max Act: 0.1083. Min Act: -0.0000

Data Index: 359452 (C4 (Web Text))

Max Activating Token Index: 113

Text #15

Max Range: 0.1653. Min Range: -0.1653

Max Act: 0.1113. Min Act: -0.0000

Data Index: 313554 (C4 (Web Text))

Max Activating Token Index: 103

Text #16

Max Range: 0.1653. Min Range: -0.1653

Max Act: 0.1085. Min Act: -0.0000

Data Index: 833236 (C4 (Web Text))

Max Activating Token Index: 577

Text #17

Max Range: 0.1653. Min Range: -0.1653

Max Act: 0.1135. Min Act: -0.0000

Data Index: 394103 (C4 (Web Text))

Max Activating Token Index: 611

Text #18

Max Range: 0.1653. Min Range: -0.1653

Max Act: 0.1048. Min Act: -0.0000

Data Index: 1330311 (C4 (Web Text))

Max Activating Token Index: 265

Text #19

Max Range: 0.1653. Min Range: -0.1653

Max Act: 0.1077. Min Act: -0.0000

Data Index: 1011979 (C4 (Web Text))

Max Activating Token Index: 299