Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3861 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.1995. Min Range: -2.1995

Max Act: 2.1995. Min Act: -0.0000

Data Index: 1119303 (C4 (Web Text))

Max Activating Token Index: 479

Text #1

Max Range: 2.1995. Min Range: -2.1995

Max Act: 2.0237. Min Act: -0.0000

Data Index: 619399 (C4 (Web Text))

Max Activating Token Index: 566

Text #2

Max Range: 2.1995. Min Range: -2.1995

Max Act: 1.7769. Min Act: -0.0000

Data Index: 727456 (C4 (Web Text))

Max Activating Token Index: 250

Text #3

Max Range: 2.1995. Min Range: -2.1995

Max Act: 2.1584. Min Act: -0.0000

Data Index: 229330 (C4 (Web Text))

Max Activating Token Index: 817

Text #4

Max Range: 2.1995. Min Range: -2.1995

Max Act: 1.9923. Min Act: -0.0000

Data Index: 535983 (C4 (Web Text))

Max Activating Token Index: 87

Text #5

Max Range: 2.1995. Min Range: -2.1995

Max Act: 1.9752. Min Act: -0.0000

Data Index: 446777 (C4 (Web Text))

Max Activating Token Index: 383

Text #6

Max Range: 2.1995. Min Range: -2.1995

Max Act: 1.8582. Min Act: -0.0000

Data Index: 715683 (C4 (Web Text))

Max Activating Token Index: 493

Text #7

Max Range: 2.1995. Min Range: -2.1995

Max Act: 1.7239. Min Act: -0.0000

Data Index: 1233907 (C4 (Web Text))

Max Activating Token Index: 277

Text #8

Max Range: 2.1995. Min Range: -2.1995

Max Act: 1.6890. Min Act: -0.0000

Data Index: 897820 (C4 (Web Text))

Max Activating Token Index: 658

Text #9

Max Range: 2.1995. Min Range: -2.1995

Max Act: 1.6764. Min Act: -0.0000

Data Index: 1050914 (C4 (Web Text))

Max Activating Token Index: 512

Text #10

Max Range: 2.1995. Min Range: -2.1995

Max Act: 1.6720. Min Act: -0.0000

Data Index: 415952 (C4 (Web Text))

Max Activating Token Index: 489

Text #11

Max Range: 2.1995. Min Range: -2.1995

Max Act: 1.6631. Min Act: -0.0000

Data Index: 944359 (C4 (Web Text))

Max Activating Token Index: 133

Text #12

Max Range: 2.1995. Min Range: -2.1995

Max Act: 1.6847. Min Act: -0.0000

Data Index: 827235 (C4 (Web Text))

Max Activating Token Index: 850

Text #13

Max Range: 2.1995. Min Range: -2.1995

Max Act: 1.3886. Min Act: -0.0000

Data Index: 10325 (C4 (Web Text))

Max Activating Token Index: 919

Text #14

Max Range: 2.1995. Min Range: -2.1995

Max Act: 1.6460. Min Act: -0.0000

Data Index: 507109 (C4 (Web Text))

Max Activating Token Index: 883

Text #15

Max Range: 2.1995. Min Range: -2.1995

Max Act: 1.6529. Min Act: -0.0000

Data Index: 550359 (C4 (Web Text))

Max Activating Token Index: 585

Text #16

Max Range: 2.1995. Min Range: -2.1995

Max Act: 1.6532. Min Act: -0.0000

Data Index: 274517 (C4 (Web Text))

Max Activating Token Index: 801

Text #17

Max Range: 2.1995. Min Range: -2.1995

Max Act: 1.7148. Min Act: -0.0000

Data Index: 816174 (C4 (Web Text))

Max Activating Token Index: 535

Text #18

Max Range: 2.1995. Min Range: -2.1995

Max Act: 1.6142. Min Act: -0.0000

Data Index: 485041 (C4 (Web Text))

Max Activating Token Index: 270

Text #19

Max Range: 2.1995. Min Range: -2.1995

Max Act: 1.5717. Min Act: -0.0000

Data Index: 1069602 (C4 (Web Text))

Max Activating Token Index: 879