Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3611 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 4.1679. Min Range: -4.1679

Max Act: 4.1679. Min Act: -0.0000

Data Index: 484445 (C4 (Web Text))

Max Activating Token Index: 862

Text #1

Max Range: 4.1679. Min Range: -4.1679

Max Act: 3.7225. Min Act: -0.0000

Data Index: 455936 (C4 (Web Text))

Max Activating Token Index: 306

Text #2

Max Range: 4.1679. Min Range: -4.1679

Max Act: 3.3615. Min Act: -0.0000

Data Index: 1097129 (C4 (Web Text))

Max Activating Token Index: 212

Text #3

Max Range: 4.1679. Min Range: -4.1679

Max Act: 3.3222. Min Act: -0.0000

Data Index: 1104074 (C4 (Web Text))

Max Activating Token Index: 999

Text #4

Max Range: 4.1679. Min Range: -4.1679

Max Act: 3.2431. Min Act: -0.0000

Data Index: 1333736 (C4 (Web Text))

Max Activating Token Index: 744

Text #5

Max Range: 4.1679. Min Range: -4.1679

Max Act: 3.3481. Min Act: -0.0000

Data Index: 213493 (C4 (Web Text))

Max Activating Token Index: 228

Text #6

Max Range: 4.1679. Min Range: -4.1679

Max Act: 3.0098. Min Act: -0.0000

Data Index: 873836 (C4 (Web Text))

Max Activating Token Index: 947

Text #7

Max Range: 4.1679. Min Range: -4.1679

Max Act: 2.8560. Min Act: -0.0000

Data Index: 1041881 (C4 (Web Text))

Max Activating Token Index: 850

Text #8

Max Range: 4.1679. Min Range: -4.1679

Max Act: 2.8834. Min Act: -0.0000

Data Index: 312968 (C4 (Web Text))

Max Activating Token Index: 337

Text #9

Max Range: 4.1679. Min Range: -4.1679

Max Act: 3.0354. Min Act: -0.0000

Data Index: 1162987 (C4 (Web Text))

Max Activating Token Index: 614

Text #10

Max Range: 4.1679. Min Range: -4.1679

Max Act: 2.8571. Min Act: -0.0000

Data Index: 970827 (C4 (Web Text))

Max Activating Token Index: 708

Text #11

Max Range: 4.1679. Min Range: -4.1679

Max Act: 2.8077. Min Act: -0.0000

Data Index: 1218555 (C4 (Web Text))

Max Activating Token Index: 882

Text #12

Max Range: 4.1679. Min Range: -4.1679

Max Act: 2.6487. Min Act: -0.0000

Data Index: 964934 (C4 (Web Text))

Max Activating Token Index: 993

Text #13

Max Range: 4.1679. Min Range: -4.1679

Max Act: 2.7455. Min Act: -0.0000

Data Index: 530818 (C4 (Web Text))

Max Activating Token Index: 151

Text #14

Max Range: 4.1679. Min Range: -4.1679

Max Act: 2.7262. Min Act: -0.0000

Data Index: 320990 (C4 (Web Text))

Max Activating Token Index: 819

Text #15

Max Range: 4.1679. Min Range: -4.1679

Max Act: 2.6958. Min Act: -0.0000

Data Index: 208998 (C4 (Web Text))

Max Activating Token Index: 170

Text #16

Max Range: 4.1679. Min Range: -4.1679

Max Act: 2.6636. Min Act: -0.0000

Data Index: 490246 (C4 (Web Text))

Max Activating Token Index: 241

Text #17

Max Range: 4.1679. Min Range: -4.1679

Max Act: 2.6419. Min Act: -0.0000

Data Index: 181671 (C4 (Web Text))

Max Activating Token Index: 968

Text #18

Max Range: 4.1679. Min Range: -4.1679

Max Act: 2.4563. Min Act: -0.0000

Data Index: 748594 (C4 (Web Text))

Max Activating Token Index: 321

Text #19

Max Range: 4.1679. Min Range: -4.1679

Max Act: 2.5580. Min Act: -0.0000

Data Index: 1053227 (C4 (Web Text))

Max Activating Token Index: 432