Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4505 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.5599. Min Range: -0.5599

Max Act: 0.5599. Min Act: -0.0002

Data Index: 420117 (C4 (Web Text))

Max Activating Token Index: 175

Text #1

Max Range: 0.5599. Min Range: -0.5599

Max Act: 0.5084. Min Act: -0.0002

Data Index: 1278898 (C4 (Web Text))

Max Activating Token Index: 782

Text #2

Max Range: 0.5599. Min Range: -0.5599

Max Act: 0.5064. Min Act: -0.0002

Data Index: 619968 (C4 (Web Text))

Max Activating Token Index: 456

Text #3

Max Range: 0.5599. Min Range: -0.5599

Max Act: 0.4975. Min Act: -0.0002

Data Index: 828840 (C4 (Web Text))

Max Activating Token Index: 914

Text #4

Max Range: 0.5599. Min Range: -0.5599

Max Act: 0.4591. Min Act: -0.0002

Data Index: 599546 (C4 (Web Text))

Max Activating Token Index: 937

Text #5

Max Range: 0.5599. Min Range: -0.5599

Max Act: 0.4732. Min Act: -0.0002

Data Index: 639442 (C4 (Web Text))

Max Activating Token Index: 765

Text #6

Max Range: 0.5599. Min Range: -0.5599

Max Act: 0.4818. Min Act: -0.0002

Data Index: 562390 (C4 (Web Text))

Max Activating Token Index: 514

Text #7

Max Range: 0.5599. Min Range: -0.5599

Max Act: 0.4731. Min Act: -0.0002

Data Index: 7614 (C4 (Web Text))

Max Activating Token Index: 635

Text #8

Max Range: 0.5599. Min Range: -0.5599

Max Act: 0.4771. Min Act: -0.0002

Data Index: 165600 (C4 (Web Text))

Max Activating Token Index: 88

Text #9

Max Range: 0.5599. Min Range: -0.5599

Max Act: 0.4722. Min Act: -0.0002

Data Index: 485527 (C4 (Web Text))

Max Activating Token Index: 524

Text #10

Max Range: 0.5599. Min Range: -0.5599

Max Act: 0.4284. Min Act: -0.0002

Data Index: 1279604 (C4 (Web Text))

Max Activating Token Index: 257

Text #11

Max Range: 0.5599. Min Range: -0.5599

Max Act: 0.4369. Min Act: -0.0002

Data Index: 1256831 (C4 (Web Text))

Max Activating Token Index: 664

Text #12

Max Range: 0.5599. Min Range: -0.5599

Max Act: 0.4548. Min Act: -0.0002

Data Index: 544605 (C4 (Web Text))

Max Activating Token Index: 860

Text #13

Max Range: 0.5599. Min Range: -0.5599

Max Act: 0.4746. Min Act: -0.0002

Data Index: 256407 (C4 (Web Text))

Max Activating Token Index: 911

Text #14

Max Range: 0.5599. Min Range: -0.5599

Max Act: 0.4593. Min Act: -0.0002

Data Index: 348597 (C4 (Web Text))

Max Activating Token Index: 412

Text #15

Max Range: 0.5599. Min Range: -0.5599

Max Act: 0.4720. Min Act: -0.0002

Data Index: 79109 (C4 (Web Text))

Max Activating Token Index: 417

Text #16

Max Range: 0.5599. Min Range: -0.5599

Max Act: 0.4696. Min Act: -0.0002

Data Index: 555915 (C4 (Web Text))

Max Activating Token Index: 87

Text #17

Max Range: 0.5599. Min Range: -0.5599

Max Act: 0.4484. Min Act: -0.0002

Data Index: 25315 (C4 (Web Text))

Max Activating Token Index: 166

Text #18

Max Range: 0.5599. Min Range: -0.5599

Max Act: 0.4506. Min Act: -0.0002

Data Index: 1216634 (C4 (Web Text))

Max Activating Token Index: 3

Text #19

Max Range: 0.5599. Min Range: -0.5599

Max Act: 0.4436. Min Act: -0.0002

Data Index: 1057009 (C4 (Web Text))

Max Activating Token Index: 549