Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 480 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.1408. Min Range: -0.1408

Max Act: 0.1408. Min Act: -0.0002

Data Index: 741320 (C4 (Web Text))

Max Activating Token Index: 385

Text #1

Max Range: 0.1408. Min Range: -0.1408

Max Act: 0.1290. Min Act: -0.0002

Data Index: 180987 (C4 (Web Text))

Max Activating Token Index: 226

Text #2

Max Range: 0.1408. Min Range: -0.1408

Max Act: 0.1233. Min Act: -0.0002

Data Index: 541919 (C4 (Web Text))

Max Activating Token Index: 78

Text #3

Max Range: 0.1408. Min Range: -0.1408

Max Act: 0.1157. Min Act: -0.0002

Data Index: 1007363 (C4 (Web Text))

Max Activating Token Index: 678

Text #4

Max Range: 0.1408. Min Range: -0.1408

Max Act: 0.1184. Min Act: -0.0002

Data Index: 93100 (C4 (Web Text))

Max Activating Token Index: 491

Text #5

Max Range: 0.1408. Min Range: -0.1408

Max Act: 0.1185. Min Act: -0.0002

Data Index: 1168186 (C4 (Web Text))

Max Activating Token Index: 232

Text #6

Max Range: 0.1408. Min Range: -0.1408

Max Act: 0.1128. Min Act: -0.0002

Data Index: 1297120 (C4 (Web Text))

Max Activating Token Index: 79

Text #7

Max Range: 0.1408. Min Range: -0.1408

Max Act: 0.1127. Min Act: -0.0002

Data Index: 435295 (C4 (Web Text))

Max Activating Token Index: 613

Text #8

Max Range: 0.1408. Min Range: -0.1408

Max Act: 0.1107. Min Act: -0.0002

Data Index: 778462 (C4 (Web Text))

Max Activating Token Index: 132

Text #9

Max Range: 0.1408. Min Range: -0.1408

Max Act: 0.1106. Min Act: -0.0002

Data Index: 1125565 (C4 (Web Text))

Max Activating Token Index: 43

Text #10

Max Range: 0.1408. Min Range: -0.1408

Max Act: 0.1123. Min Act: -0.0002

Data Index: 361531 (C4 (Web Text))

Max Activating Token Index: 770

Text #11

Max Range: 0.1408. Min Range: -0.1408

Max Act: 0.1083. Min Act: -0.0002

Data Index: 413845 (C4 (Web Text))

Max Activating Token Index: 113

Text #12

Max Range: 0.1408. Min Range: -0.1408

Max Act: 0.1057. Min Act: -0.0002

Data Index: 133130 (C4 (Web Text))

Max Activating Token Index: 129

Text #13

Max Range: 0.1408. Min Range: -0.1408

Max Act: 0.1055. Min Act: -0.0002

Data Index: 521206 (C4 (Web Text))

Max Activating Token Index: 725

Text #14

Max Range: 0.1408. Min Range: -0.1408

Max Act: 0.1086. Min Act: -0.0002

Data Index: 77621 (C4 (Web Text))

Max Activating Token Index: 229

Text #15

Max Range: 0.1408. Min Range: -0.1408

Max Act: 0.1088. Min Act: -0.0002

Data Index: 621635 (C4 (Web Text))

Max Activating Token Index: 443

Text #16

Max Range: 0.1408. Min Range: -0.1408

Max Act: 0.1082. Min Act: -0.0002

Data Index: 445767 (C4 (Web Text))

Max Activating Token Index: 429

Text #17

Max Range: 0.1408. Min Range: -0.1408

Max Act: 0.0975. Min Act: -0.0002

Data Index: 328390 (C4 (Web Text))

Max Activating Token Index: 88

Text #18

Max Range: 0.1408. Min Range: -0.1408

Max Act: 0.1010. Min Act: -0.0002

Data Index: 330055 (C4 (Web Text))

Max Activating Token Index: 602

Text #19

Max Range: 0.1408. Min Range: -0.1408

Max Act: 0.0989. Min Act: -0.0002

Data Index: 215602 (C4 (Web Text))

Max Activating Token Index: 752