Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4983 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.7224. Min Range: -1.7224

Max Act: 1.6782. Min Act: -0.0000

Data Index: 1074430 (C4 (Web Text))

Max Activating Token Index: 310

Text #1

Max Range: 1.7224. Min Range: -1.7224

Max Act: 1.7224. Min Act: -0.0000

Data Index: 30372 (C4 (Web Text))

Max Activating Token Index: 958

Text #2

Max Range: 1.7224. Min Range: -1.7224

Max Act: 1.3811. Min Act: -0.0000

Data Index: 433530 (C4 (Web Text))

Max Activating Token Index: 903

Text #3

Max Range: 1.7224. Min Range: -1.7224

Max Act: 1.5958. Min Act: -0.0000

Data Index: 255345 (C4 (Web Text))

Max Activating Token Index: 316

Text #4

Max Range: 1.7224. Min Range: -1.7224

Max Act: 1.3705. Min Act: -0.0000

Data Index: 77340 (C4 (Web Text))

Max Activating Token Index: 267

Text #5

Max Range: 1.7224. Min Range: -1.7224

Max Act: 1.1786. Min Act: -0.0000

Data Index: 1267974 (C4 (Web Text))

Max Activating Token Index: 201

Text #6

Max Range: 1.7224. Min Range: -1.7224

Max Act: 1.2157. Min Act: -0.0000

Data Index: 1269608 (C4 (Web Text))

Max Activating Token Index: 432

Text #7

Max Range: 1.7224. Min Range: -1.7224

Max Act: 1.1804. Min Act: -0.0000

Data Index: 726912 (C4 (Web Text))

Max Activating Token Index: 959

Text #8

Max Range: 1.7224. Min Range: -1.7224

Max Act: 1.1055. Min Act: -0.0000

Data Index: 605288 (C4 (Web Text))

Max Activating Token Index: 569

Text #9

Max Range: 1.7224. Min Range: -1.7224

Max Act: 1.2268. Min Act: -0.0000

Data Index: 606128 (C4 (Web Text))

Max Activating Token Index: 869

Text #10

Max Range: 1.7224. Min Range: -1.7224

Max Act: 1.2834. Min Act: -0.0000

Data Index: 1054406 (C4 (Web Text))

Max Activating Token Index: 997

Text #11

Max Range: 1.7224. Min Range: -1.7224

Max Act: 1.0383. Min Act: -0.0000

Data Index: 271932 (C4 (Web Text))

Max Activating Token Index: 488

Text #12

Max Range: 1.7224. Min Range: -1.7224

Max Act: 1.2315. Min Act: -0.0000

Data Index: 659182 (C4 (Web Text))

Max Activating Token Index: 931

Text #13

Max Range: 1.7224. Min Range: -1.7224

Max Act: 1.1183. Min Act: -0.0000

Data Index: 896220 (C4 (Web Text))

Max Activating Token Index: 636

Text #14

Max Range: 1.7224. Min Range: -1.7224

Max Act: 1.0330. Min Act: -0.0000

Data Index: 708051 (C4 (Web Text))

Max Activating Token Index: 1010

Text #15

Max Range: 1.7224. Min Range: -1.7224

Max Act: 0.9320. Min Act: -0.0000

Data Index: 872162 (C4 (Web Text))

Max Activating Token Index: 534

Text #16

Max Range: 1.7224. Min Range: -1.7224

Max Act: 0.7873. Min Act: -0.0000

Data Index: 557659 (C4 (Web Text))

Max Activating Token Index: 761

Text #17

Max Range: 1.7224. Min Range: -1.7224

Max Act: 0.9410. Min Act: -0.0000

Data Index: 1140954 (C4 (Web Text))

Max Activating Token Index: 392

Text #18

Max Range: 1.7224. Min Range: -1.7224

Max Act: 0.8668. Min Act: -0.0000

Data Index: 1039356 (C4 (Web Text))

Max Activating Token Index: 498

Text #19

Max Range: 1.7224. Min Range: -1.7224

Max Act: 0.7992. Min Act: -0.0000

Data Index: 1062168 (C4 (Web Text))

Max Activating Token Index: 934