Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3754 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 4.4081. Min Range: -4.4081

Max Act: 4.4081. Min Act: -0.0000

Data Index: 513639 (C4 (Web Text))

Max Activating Token Index: 74

Text #1

Max Range: 4.4081. Min Range: -4.4081

Max Act: 4.3743. Min Act: -0.0000

Data Index: 1349062 (C4 (Web Text))

Max Activating Token Index: 409

Text #2

Max Range: 4.4081. Min Range: -4.4081

Max Act: 4.2216. Min Act: -0.0000

Data Index: 903601 (C4 (Web Text))

Max Activating Token Index: 94

Text #3

Max Range: 4.4081. Min Range: -4.4081

Max Act: 4.2558. Min Act: -0.0000

Data Index: 811335 (C4 (Web Text))

Max Activating Token Index: 377

Text #4

Max Range: 4.4081. Min Range: -4.4081

Max Act: 4.0931. Min Act: -0.0000

Data Index: 683500 (C4 (Web Text))

Max Activating Token Index: 223

Text #5

Max Range: 4.4081. Min Range: -4.4081

Max Act: 4.0194. Min Act: -0.0000

Data Index: 1097942 (C4 (Web Text))

Max Activating Token Index: 549

Text #6

Max Range: 4.4081. Min Range: -4.4081

Max Act: 3.7939. Min Act: -0.0000

Data Index: 1059157 (C4 (Web Text))

Max Activating Token Index: 42

Text #7

Max Range: 4.4081. Min Range: -4.4081

Max Act: 3.9847. Min Act: -0.0000

Data Index: 745171 (C4 (Web Text))

Max Activating Token Index: 945

Text #8

Max Range: 4.4081. Min Range: -4.4081

Max Act: 3.5103. Min Act: -0.0000

Data Index: 282089 (C4 (Web Text))

Max Activating Token Index: 1012

Text #9

Max Range: 4.4081. Min Range: -4.4081

Max Act: 3.1655. Min Act: -0.0000

Data Index: 326667 (C4 (Web Text))

Max Activating Token Index: 314

Text #10

Max Range: 4.4081. Min Range: -4.4081

Max Act: 3.2631. Min Act: -0.0000

Data Index: 606006 (C4 (Web Text))

Max Activating Token Index: 548

Text #11

Max Range: 4.4081. Min Range: -4.4081

Max Act: 3.4097. Min Act: -0.0000

Data Index: 975380 (C4 (Web Text))

Max Activating Token Index: 935

Text #12

Max Range: 4.4081. Min Range: -4.4081

Max Act: 3.0527. Min Act: -0.0000

Data Index: 54448 (C4 (Web Text))

Max Activating Token Index: 966

Text #13

Max Range: 4.4081. Min Range: -4.4081

Max Act: 3.0883. Min Act: -0.0000

Data Index: 597744 (C4 (Web Text))

Max Activating Token Index: 253

Text #14

Max Range: 4.4081. Min Range: -4.4081

Max Act: 3.0867. Min Act: -0.0000

Data Index: 505322 (C4 (Web Text))

Max Activating Token Index: 581

Text #15

Max Range: 4.4081. Min Range: -4.4081

Max Act: 2.9430. Min Act: -0.0000

Data Index: 1236031 (C4 (Web Text))

Max Activating Token Index: 860

Text #16

Max Range: 4.4081. Min Range: -4.4081

Max Act: 2.9225. Min Act: -0.0000

Data Index: 579918 (C4 (Web Text))

Max Activating Token Index: 799

Text #17

Max Range: 4.4081. Min Range: -4.4081

Max Act: 2.9098. Min Act: -0.0000

Data Index: 279740 (C4 (Web Text))

Max Activating Token Index: 816

Text #18

Max Range: 4.4081. Min Range: -4.4081

Max Act: 2.9725. Min Act: -0.0000

Data Index: 1134926 (C4 (Web Text))

Max Activating Token Index: 405

Text #19

Max Range: 4.4081. Min Range: -4.4081

Max Act: 2.9765. Min Act: -0.0000

Data Index: 926292 (C4 (Web Text))

Max Activating Token Index: 351