Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1463 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.8294. Min Range: -2.8294

Max Act: 2.8294. Min Act: -0.0000

Data Index: 1120777 (C4 (Web Text))

Max Activating Token Index: 954

Text #1

Max Range: 2.8294. Min Range: -2.8294

Max Act: 2.5189. Min Act: -0.0000

Data Index: 403071 (C4 (Web Text))

Max Activating Token Index: 857

Text #2

Max Range: 2.8294. Min Range: -2.8294

Max Act: 2.5498. Min Act: -0.0000

Data Index: 541215 (C4 (Web Text))

Max Activating Token Index: 60

Text #3

Max Range: 2.8294. Min Range: -2.8294

Max Act: 2.5036. Min Act: -0.0000

Data Index: 561571 (C4 (Web Text))

Max Activating Token Index: 705

Text #4

Max Range: 2.8294. Min Range: -2.8294

Max Act: 2.4938. Min Act: -0.0000

Data Index: 639918 (C4 (Web Text))

Max Activating Token Index: 881

Text #5

Max Range: 2.8294. Min Range: -2.8294

Max Act: 2.5561. Min Act: -0.0000

Data Index: 636550 (C4 (Web Text))

Max Activating Token Index: 891

Text #6

Max Range: 2.8294. Min Range: -2.8294

Max Act: 2.5655. Min Act: -0.0000

Data Index: 897663 (C4 (Web Text))

Max Activating Token Index: 833

Text #7

Max Range: 2.8294. Min Range: -2.8294

Max Act: 2.4304. Min Act: -0.0000

Data Index: 382532 (C4 (Web Text))

Max Activating Token Index: 1013

Text #8

Max Range: 2.8294. Min Range: -2.8294

Max Act: 2.3541. Min Act: -0.0000

Data Index: 579962 (C4 (Web Text))

Max Activating Token Index: 211

Text #9

Max Range: 2.8294. Min Range: -2.8294

Max Act: 2.4707. Min Act: -0.0000

Data Index: 944096 (C4 (Web Text))

Max Activating Token Index: 853

Text #10

Max Range: 2.8294. Min Range: -2.8294

Max Act: 2.3247. Min Act: -0.0000

Data Index: 751209 (C4 (Web Text))

Max Activating Token Index: 23

Text #11

Max Range: 2.8294. Min Range: -2.8294

Max Act: 2.3006. Min Act: -0.0000

Data Index: 1194510 (C4 (Web Text))

Max Activating Token Index: 764

Text #12

Max Range: 2.8294. Min Range: -2.8294

Max Act: 2.3005. Min Act: -0.0000

Data Index: 840975 (C4 (Web Text))

Max Activating Token Index: 572

Text #13

Max Range: 2.8294. Min Range: -2.8294

Max Act: 2.2670. Min Act: -0.0000

Data Index: 310859 (C4 (Web Text))

Max Activating Token Index: 699

Text #14

Max Range: 2.8294. Min Range: -2.8294

Max Act: 2.1543. Min Act: -0.0000

Data Index: 317363 (C4 (Web Text))

Max Activating Token Index: 793

Text #15

Max Range: 2.8294. Min Range: -2.8294

Max Act: 2.1427. Min Act: -0.0000

Data Index: 892674 (C4 (Web Text))

Max Activating Token Index: 874

Text #16

Max Range: 2.8294. Min Range: -2.8294

Max Act: 2.1131. Min Act: -0.0000

Data Index: 1200813 (C4 (Web Text))

Max Activating Token Index: 790

Text #17

Max Range: 2.8294. Min Range: -2.8294

Max Act: 2.3519. Min Act: -0.0000

Data Index: 710378 (C4 (Web Text))

Max Activating Token Index: 420

Text #18

Max Range: 2.8294. Min Range: -2.8294

Max Act: 2.1791. Min Act: -0.0000

Data Index: 402988 (C4 (Web Text))

Max Activating Token Index: 726

Text #19

Max Range: 2.8294. Min Range: -2.8294

Max Act: 2.1773. Min Act: -0.0000

Data Index: 366685 (C4 (Web Text))

Max Activating Token Index: 445