Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2896 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.6974. Min Range: -1.6974

Max Act: 1.6974. Min Act: -0.0000

Data Index: 878583 (C4 (Web Text))

Max Activating Token Index: 814

Text #1

Max Range: 1.6974. Min Range: -1.6974

Max Act: 1.6419. Min Act: -0.0000

Data Index: 353798 (C4 (Web Text))

Max Activating Token Index: 334

Text #2

Max Range: 1.6974. Min Range: -1.6974

Max Act: 1.4010. Min Act: -0.0000

Data Index: 296784 (C4 (Web Text))

Max Activating Token Index: 455

Text #3

Max Range: 1.6974. Min Range: -1.6974

Max Act: 1.4290. Min Act: -0.0000

Data Index: 896305 (C4 (Web Text))

Max Activating Token Index: 659

Text #4

Max Range: 1.6974. Min Range: -1.6974

Max Act: 1.3354. Min Act: -0.0000

Data Index: 606757 (C4 (Web Text))

Max Activating Token Index: 827

Text #5

Max Range: 1.6974. Min Range: -1.6974

Max Act: 1.2997. Min Act: -0.0000

Data Index: 353093 (C4 (Web Text))

Max Activating Token Index: 512

Text #6

Max Range: 1.6974. Min Range: -1.6974

Max Act: 1.3727. Min Act: -0.0000

Data Index: 1011660 (C4 (Web Text))

Max Activating Token Index: 136

Text #7

Max Range: 1.6974. Min Range: -1.6974

Max Act: 1.2843. Min Act: -0.0000

Data Index: 264258 (C4 (Web Text))

Max Activating Token Index: 790

Text #8

Max Range: 1.6974. Min Range: -1.6974

Max Act: 1.2806. Min Act: -0.0000

Data Index: 1248070 (C4 (Web Text))

Max Activating Token Index: 34

Text #9

Max Range: 1.6974. Min Range: -1.6974

Max Act: 1.2718. Min Act: -0.0000

Data Index: 365744 (C4 (Web Text))

Max Activating Token Index: 841

Text #10

Max Range: 1.6974. Min Range: -1.6974

Max Act: 1.2652. Min Act: -0.0000

Data Index: 647543 (C4 (Web Text))

Max Activating Token Index: 250

Text #11

Max Range: 1.6974. Min Range: -1.6974

Max Act: 1.1700. Min Act: -0.0000

Data Index: 115246 (C4 (Web Text))

Max Activating Token Index: 987

Text #12

Max Range: 1.6974. Min Range: -1.6974

Max Act: 1.3055. Min Act: -0.0000

Data Index: 504277 (C4 (Web Text))

Max Activating Token Index: 715

Text #13

Max Range: 1.6974. Min Range: -1.6974

Max Act: 1.2161. Min Act: -0.0000

Data Index: 1194443 (C4 (Web Text))

Max Activating Token Index: 456

Text #14

Max Range: 1.6974. Min Range: -1.6974

Max Act: 1.2095. Min Act: -0.0000

Data Index: 1216584 (C4 (Web Text))

Max Activating Token Index: 271

Text #15

Max Range: 1.6974. Min Range: -1.6974

Max Act: 1.1652. Min Act: 0.0000

Data Index: 331479 (C4 (Web Text))

Max Activating Token Index: 942

Text #16

Max Range: 1.6974. Min Range: -1.6974

Max Act: 1.2233. Min Act: -0.0000

Data Index: 200504 (C4 (Web Text))

Max Activating Token Index: 522

Text #17

Max Range: 1.6974. Min Range: -1.6974

Max Act: 1.0830. Min Act: -0.0000

Data Index: 477224 (C4 (Web Text))

Max Activating Token Index: 168

Text #18

Max Range: 1.6974. Min Range: -1.6974

Max Act: 1.1470. Min Act: -0.0000

Data Index: 728762 (C4 (Web Text))

Max Activating Token Index: 604

Text #19

Max Range: 1.6974. Min Range: -1.6974

Max Act: 1.0766. Min Act: -0.0000

Data Index: 89585 (C4 (Web Text))

Max Activating Token Index: 232