Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4753 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.2991. Min Range: -2.2991

Max Act: 2.2991. Min Act: -0.0001

Data Index: 149969 (C4 (Web Text))

Max Activating Token Index: 587

Text #1

Max Range: 2.2991. Min Range: -2.2991

Max Act: 1.9573. Min Act: -0.0000

Data Index: 341611 (C4 (Web Text))

Max Activating Token Index: 775

Text #2

Max Range: 2.2991. Min Range: -2.2991

Max Act: 1.9956. Min Act: -0.0001

Data Index: 289028 (C4 (Web Text))

Max Activating Token Index: 113

Text #3

Max Range: 2.2991. Min Range: -2.2991

Max Act: 1.9528. Min Act: -0.0001

Data Index: 384298 (C4 (Web Text))

Max Activating Token Index: 8

Text #4

Max Range: 2.2991. Min Range: -2.2991

Max Act: 1.7532. Min Act: -0.0000

Data Index: 1086127 (C4 (Web Text))

Max Activating Token Index: 112

Text #5

Max Range: 2.2991. Min Range: -2.2991

Max Act: 1.9176. Min Act: -0.0000

Data Index: 1331221 (C4 (Web Text))

Max Activating Token Index: 762

Text #6

Max Range: 2.2991. Min Range: -2.2991

Max Act: 1.7847. Min Act: -0.0001

Data Index: 1341399 (C4 (Web Text))

Max Activating Token Index: 35

Text #7

Max Range: 2.2991. Min Range: -2.2991

Max Act: 1.7788. Min Act: -0.0001

Data Index: 244465 (C4 (Web Text))

Max Activating Token Index: 403

Text #8

Max Range: 2.2991. Min Range: -2.2991

Max Act: 1.7462. Min Act: -0.0000

Data Index: 20239 (C4 (Web Text))

Max Activating Token Index: 318

Text #9

Max Range: 2.2991. Min Range: -2.2991

Max Act: 1.7544. Min Act: -0.0000

Data Index: 425762 (C4 (Web Text))

Max Activating Token Index: 623

Text #10

Max Range: 2.2991. Min Range: -2.2991

Max Act: 1.6234. Min Act: -0.0000

Data Index: 1338265 (C4 (Web Text))

Max Activating Token Index: 659

Text #11

Max Range: 2.2991. Min Range: -2.2991

Max Act: 1.7012. Min Act: -0.0001

Data Index: 96888 (C4 (Web Text))

Max Activating Token Index: 588

Text #12

Max Range: 2.2991. Min Range: -2.2991

Max Act: 1.6070. Min Act: -0.0000

Data Index: 958823 (C4 (Web Text))

Max Activating Token Index: 782

Text #13

Max Range: 2.2991. Min Range: -2.2991

Max Act: 1.8661. Min Act: -0.0001

Data Index: 206977 (C4 (Web Text))

Max Activating Token Index: 358

Text #14

Max Range: 2.2991. Min Range: -2.2991

Max Act: 1.7426. Min Act: -0.0001

Data Index: 1128944 (C4 (Web Text))

Max Activating Token Index: 216

Text #15

Max Range: 2.2991. Min Range: -2.2991

Max Act: 1.7450. Min Act: -0.0000

Data Index: 733535 (C4 (Web Text))

Max Activating Token Index: 184

Text #16

Max Range: 2.2991. Min Range: -2.2991

Max Act: 1.5904. Min Act: -0.0001

Data Index: 612510 (C4 (Web Text))

Max Activating Token Index: 238

Text #17

Max Range: 2.2991. Min Range: -2.2991

Max Act: 1.6795. Min Act: -0.0000

Data Index: 356394 (C4 (Web Text))

Max Activating Token Index: 308

Text #18

Max Range: 2.2991. Min Range: -2.2991

Max Act: 1.6211. Min Act: -0.0000

Data Index: 113411 (C4 (Web Text))

Max Activating Token Index: 244

Text #19

Max Range: 2.2991. Min Range: -2.2991

Max Act: 1.7473. Min Act: -0.0001

Data Index: 286485 (C4 (Web Text))

Max Activating Token Index: 383