Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 538 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.4740. Min Range: -1.4740

Max Act: 1.4740. Min Act: -0.0002

Data Index: 335325 (C4 (Web Text))

Max Activating Token Index: 658

Text #1

Max Range: 1.4740. Min Range: -1.4740

Max Act: 1.3017. Min Act: -0.0002

Data Index: 281444 (C4 (Web Text))

Max Activating Token Index: 276

Text #2

Max Range: 1.4740. Min Range: -1.4740

Max Act: 1.2097. Min Act: -0.0002

Data Index: 509719 (C4 (Web Text))

Max Activating Token Index: 834

Text #3

Max Range: 1.4740. Min Range: -1.4740

Max Act: 1.1684. Min Act: -0.0002

Data Index: 311828 (C4 (Web Text))

Max Activating Token Index: 31

Text #4

Max Range: 1.4740. Min Range: -1.4740

Max Act: 1.1978. Min Act: -0.0002

Data Index: 741348 (C4 (Web Text))

Max Activating Token Index: 187

Text #5

Max Range: 1.4740. Min Range: -1.4740

Max Act: 1.3172. Min Act: -0.0002

Data Index: 330244 (C4 (Web Text))

Max Activating Token Index: 650

Text #6

Max Range: 1.4740. Min Range: -1.4740

Max Act: 1.1421. Min Act: -0.0002

Data Index: 659031 (C4 (Web Text))

Max Activating Token Index: 235

Text #7

Max Range: 1.4740. Min Range: -1.4740

Max Act: 1.1768. Min Act: -0.0002

Data Index: 60613 (C4 (Web Text))

Max Activating Token Index: 404

Text #8

Max Range: 1.4740. Min Range: -1.4740

Max Act: 1.1630. Min Act: -0.0002

Data Index: 583048 (C4 (Web Text))

Max Activating Token Index: 153

Text #9

Max Range: 1.4740. Min Range: -1.4740

Max Act: 1.2703. Min Act: -0.0002

Data Index: 472760 (C4 (Web Text))

Max Activating Token Index: 275

Text #10

Max Range: 1.4740. Min Range: -1.4740

Max Act: 1.0800. Min Act: -0.0002

Data Index: 618246 (C4 (Web Text))

Max Activating Token Index: 156

Text #11

Max Range: 1.4740. Min Range: -1.4740

Max Act: 1.1871. Min Act: -0.0002

Data Index: 1313076 (C4 (Web Text))

Max Activating Token Index: 233

Text #12

Max Range: 1.4740. Min Range: -1.4740

Max Act: 1.2198. Min Act: -0.0002

Data Index: 523931 (C4 (Web Text))

Max Activating Token Index: 331

Text #13

Max Range: 1.4740. Min Range: -1.4740

Max Act: 1.2031. Min Act: -0.0002

Data Index: 1139638 (C4 (Web Text))

Max Activating Token Index: 445

Text #14

Max Range: 1.4740. Min Range: -1.4740

Max Act: 1.1057. Min Act: -0.0002

Data Index: 1475 (C4 (Web Text))

Max Activating Token Index: 299

Text #15

Max Range: 1.4740. Min Range: -1.4740

Max Act: 1.1920. Min Act: -0.0002

Data Index: 926215 (C4 (Web Text))

Max Activating Token Index: 197

Text #16

Max Range: 1.4740. Min Range: -1.4740

Max Act: 1.1704. Min Act: -0.0002

Data Index: 1261971 (C4 (Web Text))

Max Activating Token Index: 369

Text #17

Max Range: 1.4740. Min Range: -1.4740

Max Act: 1.1648. Min Act: -0.0002

Data Index: 430821 (C4 (Web Text))

Max Activating Token Index: 441

Text #18

Max Range: 1.4740. Min Range: -1.4740

Max Act: 1.2172. Min Act: -0.0002

Data Index: 248130 (C4 (Web Text))

Max Activating Token Index: 235

Text #19

Max Range: 1.4740. Min Range: -1.4740

Max Act: 1.1236. Min Act: -0.0002

Data Index: 354600 (C4 (Web Text))

Max Activating Token Index: 832