Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3054 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.1208. Min Range: -0.1208

Max Act: 0.1208. Min Act: -0.0000

Data Index: 1467883 (Python Code)

Max Activating Token Index: 318

Text #1

Max Range: 0.1208. Min Range: -0.1208

Max Act: 0.1167. Min Act: -0.0000

Data Index: 1069766 (C4 (Web Text))

Max Activating Token Index: 570

Text #2

Max Range: 0.1208. Min Range: -0.1208

Max Act: 0.1193. Min Act: -0.0000

Data Index: 1224237 (C4 (Web Text))

Max Activating Token Index: 36

Text #3

Max Range: 0.1208. Min Range: -0.1208

Max Act: 0.1092. Min Act: -0.0000

Data Index: 276547 (C4 (Web Text))

Max Activating Token Index: 123

Text #4

Max Range: 0.1208. Min Range: -0.1208

Max Act: 0.1100. Min Act: -0.0000

Data Index: 1489012 (Python Code)

Max Activating Token Index: 784

Text #5

Max Range: 0.1208. Min Range: -0.1208

Max Act: 0.1087. Min Act: -0.0000

Data Index: 1233394 (C4 (Web Text))

Max Activating Token Index: 291

Text #6

Max Range: 0.1208. Min Range: -0.1208

Max Act: 0.1129. Min Act: -0.0000

Data Index: 651034 (C4 (Web Text))

Max Activating Token Index: 343

Text #7

Max Range: 0.1208. Min Range: -0.1208

Max Act: 0.1071. Min Act: -0.0000

Data Index: 1464590 (Python Code)

Max Activating Token Index: 304

Text #8

Max Range: 0.1208. Min Range: -0.1208

Max Act: 0.1046. Min Act: -0.0000

Data Index: 859646 (C4 (Web Text))

Max Activating Token Index: 411

Text #9

Max Range: 0.1208. Min Range: -0.1208

Max Act: 0.1037. Min Act: -0.0000

Data Index: 924392 (C4 (Web Text))

Max Activating Token Index: 735

Text #10

Max Range: 0.1208. Min Range: -0.1208

Max Act: 0.1079. Min Act: -0.0000

Data Index: 1042318 (C4 (Web Text))

Max Activating Token Index: 668

Text #11

Max Range: 0.1208. Min Range: -0.1208

Max Act: 0.1052. Min Act: -0.0000

Data Index: 1533992 (Python Code)

Max Activating Token Index: 753

Text #12

Max Range: 0.1208. Min Range: -0.1208

Max Act: 0.1010. Min Act: -0.0000

Data Index: 707405 (C4 (Web Text))

Max Activating Token Index: 596

Text #13

Max Range: 0.1208. Min Range: -0.1208

Max Act: 0.1011. Min Act: -0.0000

Data Index: 799372 (C4 (Web Text))

Max Activating Token Index: 351

Text #14

Max Range: 0.1208. Min Range: -0.1208

Max Act: 0.0986. Min Act: -0.0000

Data Index: 955849 (C4 (Web Text))

Max Activating Token Index: 347

Text #15

Max Range: 0.1208. Min Range: -0.1208

Max Act: 0.1055. Min Act: -0.0000

Data Index: 1534030 (Python Code)

Max Activating Token Index: 354

Text #16

Max Range: 0.1208. Min Range: -0.1208

Max Act: 0.1086. Min Act: -0.0000

Data Index: 539573 (C4 (Web Text))

Max Activating Token Index: 593

Text #17

Max Range: 0.1208. Min Range: -0.1208

Max Act: 0.1018. Min Act: -0.0000

Data Index: 853808 (C4 (Web Text))

Max Activating Token Index: 978

Text #18

Max Range: 0.1208. Min Range: -0.1208

Max Act: 0.1011. Min Act: -0.0000

Data Index: 1400076 (Python Code)

Max Activating Token Index: 306

Text #19

Max Range: 0.1208. Min Range: -0.1208

Max Act: 0.0998. Min Act: -0.0000

Data Index: 89889 (C4 (Web Text))

Max Activating Token Index: 138