Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2842 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.7199. Min Range: -0.7199

Max Act: 0.7199. Min Act: -0.0000

Data Index: 919560 (C4 (Web Text))

Max Activating Token Index: 61

Text #1

Max Range: 0.7199. Min Range: -0.7199

Max Act: 0.6673. Min Act: -0.0000

Data Index: 785502 (C4 (Web Text))

Max Activating Token Index: 327

Text #2

Max Range: 0.7199. Min Range: -0.7199

Max Act: 0.4866. Min Act: -0.0000

Data Index: 1635856 (Python Code)

Max Activating Token Index: 535

Text #3

Max Range: 0.7199. Min Range: -0.7199

Max Act: 0.4475. Min Act: -0.0000

Data Index: 1627537 (Python Code)

Max Activating Token Index: 46

Text #4

Max Range: 0.7199. Min Range: -0.7199

Max Act: 0.4152. Min Act: -0.0000

Data Index: 781175 (C4 (Web Text))

Max Activating Token Index: 49

Text #5

Max Range: 0.7199. Min Range: -0.7199

Max Act: 0.3947. Min Act: -0.0000

Data Index: 71374 (C4 (Web Text))

Max Activating Token Index: 107

Text #6

Max Range: 0.7199. Min Range: -0.7199

Max Act: 0.3756. Min Act: -0.0000

Data Index: 47984 (C4 (Web Text))

Max Activating Token Index: 744

Text #7

Max Range: 0.7199. Min Range: -0.7199

Max Act: 0.4011. Min Act: -0.0000

Data Index: 403070 (C4 (Web Text))

Max Activating Token Index: 296

Text #8

Max Range: 0.7199. Min Range: -0.7199

Max Act: 0.4140. Min Act: -0.0000

Data Index: 297476 (C4 (Web Text))

Max Activating Token Index: 681

Text #9

Max Range: 0.7199. Min Range: -0.7199

Max Act: 0.4000. Min Act: -0.0000

Data Index: 132797 (C4 (Web Text))

Max Activating Token Index: 350

Text #10

Max Range: 0.7199. Min Range: -0.7199

Max Act: 0.4087. Min Act: 0.0001

Data Index: 570743 (C4 (Web Text))

Max Activating Token Index: 899

Text #11

Max Range: 0.7199. Min Range: -0.7199

Max Act: 0.3890. Min Act: -0.0000

Data Index: 1574492 (Python Code)

Max Activating Token Index: 1021

Text #12

Max Range: 0.7199. Min Range: -0.7199

Max Act: 0.4106. Min Act: -0.0000

Data Index: 1606133 (Python Code)

Max Activating Token Index: 457

Text #13

Max Range: 0.7199. Min Range: -0.7199

Max Act: 0.3869. Min Act: -0.0000

Data Index: 1019167 (C4 (Web Text))

Max Activating Token Index: 392

Text #14

Max Range: 0.7199. Min Range: -0.7199

Max Act: 0.3933. Min Act: -0.0000

Data Index: 1101450 (C4 (Web Text))

Max Activating Token Index: 163

Text #15

Max Range: 0.7199. Min Range: -0.7199

Max Act: 0.3841. Min Act: -0.0000

Data Index: 12861 (C4 (Web Text))

Max Activating Token Index: 486

Text #16

Max Range: 0.7199. Min Range: -0.7199

Max Act: 0.3666. Min Act: -0.0000

Data Index: 1074942 (C4 (Web Text))

Max Activating Token Index: 667

Text #17

Max Range: 0.7199. Min Range: -0.7199

Max Act: 0.3744. Min Act: -0.0000

Data Index: 312334 (C4 (Web Text))

Max Activating Token Index: 982

Text #18

Max Range: 0.7199. Min Range: -0.7199

Max Act: 0.3721. Min Act: -0.0000

Data Index: 609964 (C4 (Web Text))

Max Activating Token Index: 507

Text #19

Max Range: 0.7199. Min Range: -0.7199

Max Act: 0.3584. Min Act: -0.0000

Data Index: 79493 (C4 (Web Text))

Max Activating Token Index: 355