Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1798 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 0.6356. Min Range: -0.6356

Max Act: 0.6356. Min Act: -0.0003

Data Index: 682101 (C4 (Web Text))

Max Activating Token Index: 739

Text #1

Max Range: 0.6356. Min Range: -0.6356

Max Act: 0.5844. Min Act: -0.0003

Data Index: 1120026 (C4 (Web Text))

Max Activating Token Index: 313

Text #2

Max Range: 0.6356. Min Range: -0.6356

Max Act: 0.5893. Min Act: -0.0003

Data Index: 86659 (C4 (Web Text))

Max Activating Token Index: 920

Text #3

Max Range: 0.6356. Min Range: -0.6356

Max Act: 0.5718. Min Act: -0.0003

Data Index: 1248445 (C4 (Web Text))

Max Activating Token Index: 167

Text #4

Max Range: 0.6356. Min Range: -0.6356

Max Act: 0.5494. Min Act: -0.0003

Data Index: 32224 (C4 (Web Text))

Max Activating Token Index: 611

Text #5

Max Range: 0.6356. Min Range: -0.6356

Max Act: 0.5592. Min Act: -0.0003

Data Index: 256417 (C4 (Web Text))

Max Activating Token Index: 165

Text #6

Max Range: 0.6356. Min Range: -0.6356

Max Act: 0.5401. Min Act: -0.0003

Data Index: 729899 (C4 (Web Text))

Max Activating Token Index: 171

Text #7

Max Range: 0.6356. Min Range: -0.6356

Max Act: 0.5623. Min Act: -0.0003

Data Index: 865935 (C4 (Web Text))

Max Activating Token Index: 734

Text #8

Max Range: 0.6356. Min Range: -0.6356

Max Act: 0.5067. Min Act: -0.0003

Data Index: 103409 (C4 (Web Text))

Max Activating Token Index: 1013

Text #9

Max Range: 0.6356. Min Range: -0.6356

Max Act: 0.5239. Min Act: -0.0003

Data Index: 1081289 (C4 (Web Text))

Max Activating Token Index: 445

Text #10

Max Range: 0.6356. Min Range: -0.6356

Max Act: 0.5706. Min Act: -0.0003

Data Index: 566358 (C4 (Web Text))

Max Activating Token Index: 815

Text #11

Max Range: 0.6356. Min Range: -0.6356

Max Act: 0.5483. Min Act: -0.0003

Data Index: 888643 (C4 (Web Text))

Max Activating Token Index: 845

Text #12

Max Range: 0.6356. Min Range: -0.6356

Max Act: 0.5158. Min Act: -0.0004

Data Index: 344158 (C4 (Web Text))

Max Activating Token Index: 519

Text #13

Max Range: 0.6356. Min Range: -0.6356

Max Act: 0.5129. Min Act: -0.0003

Data Index: 495699 (C4 (Web Text))

Max Activating Token Index: 783

Text #14

Max Range: 0.6356. Min Range: -0.6356

Max Act: 0.5084. Min Act: -0.0003

Data Index: 68053 (C4 (Web Text))

Max Activating Token Index: 405

Text #15

Max Range: 0.6356. Min Range: -0.6356

Max Act: 0.4901. Min Act: -0.0003

Data Index: 388392 (C4 (Web Text))

Max Activating Token Index: 354

Text #16

Max Range: 0.6356. Min Range: -0.6356

Max Act: 0.5021. Min Act: -0.0003

Data Index: 353362 (C4 (Web Text))

Max Activating Token Index: 584

Text #17

Max Range: 0.6356. Min Range: -0.6356

Max Act: 0.5142. Min Act: -0.0003

Data Index: 713248 (C4 (Web Text))

Max Activating Token Index: 451

Text #18

Max Range: 0.6356. Min Range: -0.6356

Max Act: 0.5119. Min Act: -0.0003

Data Index: 1218976 (C4 (Web Text))

Max Activating Token Index: 729

Text #19

Max Range: 0.6356. Min Range: -0.6356

Max Act: 0.5107. Min Act: -0.0003

Data Index: 90382 (C4 (Web Text))

Max Activating Token Index: 996