Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 589 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.7976. Min Range: -0.7976

Max Act: 0.7976. Min Act: -0.0000

Data Index: 479460 (C4 (Web Text))

Max Activating Token Index: 177

Text #1

Max Range: 0.7976. Min Range: -0.7976

Max Act: 0.6091. Min Act: -0.0000

Data Index: 296261 (C4 (Web Text))

Max Activating Token Index: 789

Text #2

Max Range: 0.7976. Min Range: -0.7976

Max Act: 0.6346. Min Act: -0.0000

Data Index: 181684 (C4 (Web Text))

Max Activating Token Index: 463

Text #3

Max Range: 0.7976. Min Range: -0.7976

Max Act: 0.5371. Min Act: 0.0000

Data Index: 608723 (C4 (Web Text))

Max Activating Token Index: 841

Text #4

Max Range: 0.7976. Min Range: -0.7976

Max Act: 0.5046. Min Act: -0.0000

Data Index: 1139945 (C4 (Web Text))

Max Activating Token Index: 188

Text #5

Max Range: 0.7976. Min Range: -0.7976

Max Act: 0.5365. Min Act: -0.0000

Data Index: 781370 (C4 (Web Text))

Max Activating Token Index: 890

Text #6

Max Range: 0.7976. Min Range: -0.7976

Max Act: 0.5305. Min Act: -0.0000

Data Index: 575258 (C4 (Web Text))

Max Activating Token Index: 206

Text #7

Max Range: 0.7976. Min Range: -0.7976

Max Act: 0.5143. Min Act: -0.0000

Data Index: 480858 (C4 (Web Text))

Max Activating Token Index: 449

Text #8

Max Range: 0.7976. Min Range: -0.7976

Max Act: 0.5460. Min Act: -0.0000

Data Index: 777634 (C4 (Web Text))

Max Activating Token Index: 795

Text #9

Max Range: 0.7976. Min Range: -0.7976

Max Act: 0.5247. Min Act: -0.0000

Data Index: 168287 (C4 (Web Text))

Max Activating Token Index: 361

Text #10

Max Range: 0.7976. Min Range: -0.7976

Max Act: 0.5058. Min Act: -0.0000

Data Index: 45381 (C4 (Web Text))

Max Activating Token Index: 842

Text #11

Max Range: 0.7976. Min Range: -0.7976

Max Act: 0.5161. Min Act: -0.0000

Data Index: 614559 (C4 (Web Text))

Max Activating Token Index: 339

Text #12

Max Range: 0.7976. Min Range: -0.7976

Max Act: 0.5072. Min Act: -0.0000

Data Index: 1125354 (C4 (Web Text))

Max Activating Token Index: 215

Text #13

Max Range: 0.7976. Min Range: -0.7976

Max Act: 0.4970. Min Act: -0.0000

Data Index: 1140381 (C4 (Web Text))

Max Activating Token Index: 432

Text #14

Max Range: 0.7976. Min Range: -0.7976

Max Act: 0.4987. Min Act: -0.0000

Data Index: 167622 (C4 (Web Text))

Max Activating Token Index: 987

Text #15

Max Range: 0.7976. Min Range: -0.7976

Max Act: 0.4903. Min Act: -0.0000

Data Index: 586355 (C4 (Web Text))

Max Activating Token Index: 934

Text #16

Max Range: 0.7976. Min Range: -0.7976

Max Act: 0.4904. Min Act: -0.0000

Data Index: 350645 (C4 (Web Text))

Max Activating Token Index: 334

Text #17

Max Range: 0.7976. Min Range: -0.7976

Max Act: 0.4847. Min Act: 0.0000

Data Index: 1218597 (C4 (Web Text))

Max Activating Token Index: 664

Text #18

Max Range: 0.7976. Min Range: -0.7976

Max Act: 0.4813. Min Act: 0.0000

Data Index: 739252 (C4 (Web Text))

Max Activating Token Index: 693

Text #19

Max Range: 0.7976. Min Range: -0.7976

Max Act: 0.4706. Min Act: -0.0000

Data Index: 897886 (C4 (Web Text))

Max Activating Token Index: 109