Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1036 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 0.1081. Min Range: -0.1081

Max Act: 0.1041. Min Act: -0.0004

Data Index: 901347 (C4 (Web Text))

Max Activating Token Index: 871

Text #1

Max Range: 0.1081. Min Range: -0.1081

Max Act: 0.0998. Min Act: -0.0004

Data Index: 72551 (C4 (Web Text))

Max Activating Token Index: 916

Text #2

Max Range: 0.1081. Min Range: -0.1081

Max Act: 0.0996. Min Act: -0.0004

Data Index: 623321 (C4 (Web Text))

Max Activating Token Index: 935

Text #3

Max Range: 0.1081. Min Range: -0.1081

Max Act: 0.0931. Min Act: -0.0004

Data Index: 1163800 (C4 (Web Text))

Max Activating Token Index: 537

Text #4

Max Range: 0.1081. Min Range: -0.1081

Max Act: 0.0960. Min Act: -0.0004

Data Index: 282165 (C4 (Web Text))

Max Activating Token Index: 314

Text #5

Max Range: 0.1081. Min Range: -0.1081

Max Act: 0.1044. Min Act: -0.0004

Data Index: 623527 (C4 (Web Text))

Max Activating Token Index: 249

Text #6

Max Range: 0.1081. Min Range: -0.1081

Max Act: 0.1005. Min Act: -0.0004

Data Index: 489838 (C4 (Web Text))

Max Activating Token Index: 322

Text #7

Max Range: 0.1081. Min Range: -0.1081

Max Act: 0.0995. Min Act: -0.0004

Data Index: 1241461 (C4 (Web Text))

Max Activating Token Index: 866

Text #8

Max Range: 0.1081. Min Range: -0.1081

Max Act: 0.1081. Min Act: -0.0004

Data Index: 1293812 (C4 (Web Text))

Max Activating Token Index: 969

Text #9

Max Range: 0.1081. Min Range: -0.1081

Max Act: 0.1000. Min Act: -0.0004

Data Index: 1081267 (C4 (Web Text))

Max Activating Token Index: 983

Text #10

Max Range: 0.1081. Min Range: -0.1081

Max Act: 0.1009. Min Act: -0.0004

Data Index: 570706 (C4 (Web Text))

Max Activating Token Index: 266

Text #11

Max Range: 0.1081. Min Range: -0.1081

Max Act: 0.0999. Min Act: -0.0004

Data Index: 869631 (C4 (Web Text))

Max Activating Token Index: 980

Text #12

Max Range: 0.1081. Min Range: -0.1081

Max Act: 0.0992. Min Act: -0.0004

Data Index: 613946 (C4 (Web Text))

Max Activating Token Index: 268

Text #13

Max Range: 0.1081. Min Range: -0.1081

Max Act: 0.1001. Min Act: -0.0004

Data Index: 640593 (C4 (Web Text))

Max Activating Token Index: 301

Text #14

Max Range: 0.1081. Min Range: -0.1081

Max Act: 0.0997. Min Act: -0.0004

Data Index: 661735 (C4 (Web Text))

Max Activating Token Index: 914

Text #15

Max Range: 0.1081. Min Range: -0.1081

Max Act: 0.0974. Min Act: -0.0004

Data Index: 1282171 (C4 (Web Text))

Max Activating Token Index: 169

Text #16

Max Range: 0.1081. Min Range: -0.1081

Max Act: 0.0979. Min Act: -0.0004

Data Index: 15853 (C4 (Web Text))

Max Activating Token Index: 632

Text #17

Max Range: 0.1081. Min Range: -0.1081

Max Act: 0.0971. Min Act: -0.0004

Data Index: 298026 (C4 (Web Text))

Max Activating Token Index: 228

Text #18

Max Range: 0.1081. Min Range: -0.1081

Max Act: 0.1013. Min Act: -0.0004

Data Index: 180255 (C4 (Web Text))

Max Activating Token Index: 229

Text #19

Max Range: 0.1081. Min Range: -0.1081

Max Act: 0.0931. Min Act: -0.0004

Data Index: 247795 (C4 (Web Text))

Max Activating Token Index: 832