Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2138 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.0121. Min Range: -0.0121

Max Act: 0.0121. Min Act: -0.0002

Data Index: 599548 (C4 (Web Text))

Max Activating Token Index: 21

Text #1

Max Range: 0.0121. Min Range: -0.0121

Max Act: 0.0118. Min Act: -0.0002

Data Index: 1001597 (C4 (Web Text))

Max Activating Token Index: 175

Text #2

Max Range: 0.0121. Min Range: -0.0121

Max Act: 0.0117. Min Act: -0.0002

Data Index: 934139 (C4 (Web Text))

Max Activating Token Index: 843

Text #3

Max Range: 0.0121. Min Range: -0.0121

Max Act: 0.0116. Min Act: -0.0002

Data Index: 1071253 (C4 (Web Text))

Max Activating Token Index: 6

Text #4

Max Range: 0.0121. Min Range: -0.0121

Max Act: 0.0117. Min Act: -0.0002

Data Index: 673659 (C4 (Web Text))

Max Activating Token Index: 9

Text #5

Max Range: 0.0121. Min Range: -0.0121

Max Act: 0.0116. Min Act: -0.0002

Data Index: 55892 (C4 (Web Text))

Max Activating Token Index: 44

Text #6

Max Range: 0.0121. Min Range: -0.0121

Max Act: 0.0116. Min Act: -0.0002

Data Index: 399332 (C4 (Web Text))

Max Activating Token Index: 508

Text #7

Max Range: 0.0121. Min Range: -0.0121

Max Act: 0.0115. Min Act: -0.0002

Data Index: 1052360 (C4 (Web Text))

Max Activating Token Index: 228

Text #8

Max Range: 0.0121. Min Range: -0.0121

Max Act: 0.0114. Min Act: -0.0002

Data Index: 11755 (C4 (Web Text))

Max Activating Token Index: 410

Text #9

Max Range: 0.0121. Min Range: -0.0121

Max Act: 0.0114. Min Act: -0.0002

Data Index: 1014965 (C4 (Web Text))

Max Activating Token Index: 844

Text #10

Max Range: 0.0121. Min Range: -0.0121

Max Act: 0.0114. Min Act: -0.0002

Data Index: 898951 (C4 (Web Text))

Max Activating Token Index: 263

Text #11

Max Range: 0.0121. Min Range: -0.0121

Max Act: 0.0115. Min Act: -0.0002

Data Index: 993240 (C4 (Web Text))

Max Activating Token Index: 243

Text #12

Max Range: 0.0121. Min Range: -0.0121

Max Act: 0.0110. Min Act: -0.0002

Data Index: 1132226 (C4 (Web Text))

Max Activating Token Index: 635

Text #13

Max Range: 0.0121. Min Range: -0.0121

Max Act: 0.0112. Min Act: -0.0002

Data Index: 118190 (C4 (Web Text))

Max Activating Token Index: 358

Text #14

Max Range: 0.0121. Min Range: -0.0121

Max Act: 0.0112. Min Act: -0.0002

Data Index: 1078640 (C4 (Web Text))

Max Activating Token Index: 171

Text #15

Max Range: 0.0121. Min Range: -0.0121

Max Act: 0.0111. Min Act: -0.0002

Data Index: 88867 (C4 (Web Text))

Max Activating Token Index: 70

Text #16

Max Range: 0.0121. Min Range: -0.0121

Max Act: 0.0113. Min Act: -0.0002

Data Index: 940684 (C4 (Web Text))

Max Activating Token Index: 1018

Text #17

Max Range: 0.0121. Min Range: -0.0121

Max Act: 0.0111. Min Act: -0.0002

Data Index: 686125 (C4 (Web Text))

Max Activating Token Index: 206

Text #18

Max Range: 0.0121. Min Range: -0.0121

Max Act: 0.0111. Min Act: -0.0002

Data Index: 1145310 (C4 (Web Text))

Max Activating Token Index: 738

Text #19

Max Range: 0.0121. Min Range: -0.0121

Max Act: 0.0111. Min Act: -0.0002

Data Index: 358728 (C4 (Web Text))

Max Activating Token Index: 399