Model: SoLU Model: 3 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 858 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-3l')

Text #0

Max Range: 0.0191. Min Range: -0.0191

Max Act: 0.0191. Min Act: -0.0005

Data Index: 486115 (C4 (Web Text))

Max Activating Token Index: 7

Text #1

Max Range: 0.0191. Min Range: -0.0191

Max Act: 0.0159. Min Act: -0.0006

Data Index: 294772 (C4 (Web Text))

Max Activating Token Index: 1019

Text #2

Max Range: 0.0191. Min Range: -0.0191

Max Act: 0.0157. Min Act: -0.0006

Data Index: 66142 (C4 (Web Text))

Max Activating Token Index: 78

Text #3

Max Range: 0.0191. Min Range: -0.0191

Max Act: 0.0156. Min Act: -0.0006

Data Index: 1228757 (C4 (Web Text))

Max Activating Token Index: 690

Text #4

Max Range: 0.0191. Min Range: -0.0191

Max Act: 0.0156. Min Act: -0.0006

Data Index: 1197375 (C4 (Web Text))

Max Activating Token Index: 164

Text #5

Max Range: 0.0191. Min Range: -0.0191

Max Act: 0.0152. Min Act: -0.0005

Data Index: 489741 (C4 (Web Text))

Max Activating Token Index: 18

Text #6

Max Range: 0.0191. Min Range: -0.0191

Max Act: 0.0150. Min Act: -0.0006

Data Index: 315441 (C4 (Web Text))

Max Activating Token Index: 982

Text #7

Max Range: 0.0191. Min Range: -0.0191

Max Act: 0.0150. Min Act: -0.0006

Data Index: 966959 (C4 (Web Text))

Max Activating Token Index: 70

Text #8

Max Range: 0.0191. Min Range: -0.0191

Max Act: 0.0146. Min Act: -0.0006

Data Index: 485273 (C4 (Web Text))

Max Activating Token Index: 182

Text #9

Max Range: 0.0191. Min Range: -0.0191

Max Act: 0.0146. Min Act: -0.0006

Data Index: 994190 (C4 (Web Text))

Max Activating Token Index: 33

Text #10

Max Range: 0.0191. Min Range: -0.0191

Max Act: 0.0144. Min Act: -0.0006

Data Index: 549908 (C4 (Web Text))

Max Activating Token Index: 253

Text #11

Max Range: 0.0191. Min Range: -0.0191

Max Act: 0.0143. Min Act: -0.0006

Data Index: 512993 (C4 (Web Text))

Max Activating Token Index: 86

Text #12

Max Range: 0.0191. Min Range: -0.0191

Max Act: 0.0143. Min Act: -0.0005

Data Index: 190592 (C4 (Web Text))

Max Activating Token Index: 780

Text #13

Max Range: 0.0191. Min Range: -0.0191

Max Act: 0.0142. Min Act: -0.0006

Data Index: 890169 (C4 (Web Text))

Max Activating Token Index: 559

Text #14

Max Range: 0.0191. Min Range: -0.0191

Max Act: 0.0138. Min Act: -0.0006

Data Index: 746269 (C4 (Web Text))

Max Activating Token Index: 821

Text #15

Max Range: 0.0191. Min Range: -0.0191

Max Act: 0.0142. Min Act: -0.0006

Data Index: 1197371 (C4 (Web Text))

Max Activating Token Index: 3

Text #16

Max Range: 0.0191. Min Range: -0.0191

Max Act: 0.0138. Min Act: -0.0006

Data Index: 264750 (C4 (Web Text))

Max Activating Token Index: 918

Text #17

Max Range: 0.0191. Min Range: -0.0191

Max Act: 0.0144. Min Act: -0.0006

Data Index: 110083 (C4 (Web Text))

Max Activating Token Index: 133

Text #18

Max Range: 0.0191. Min Range: -0.0191

Max Act: 0.0140. Min Act: -0.0006

Data Index: 782622 (C4 (Web Text))

Max Activating Token Index: 910

Text #19

Max Range: 0.0191. Min Range: -0.0191

Max Act: 0.0140. Min Act: -0.0006

Data Index: 188080 (C4 (Web Text))

Max Activating Token Index: 26