Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3670 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.5506. Min Range: -2.5506

Max Act: 2.5506. Min Act: -0.0000

Data Index: 723343 (C4 (Web Text))

Max Activating Token Index: 358

Text #1

Max Range: 2.5506. Min Range: -2.5506

Max Act: 2.0009. Min Act: -0.0000

Data Index: 946226 (C4 (Web Text))

Max Activating Token Index: 581

Text #2

Max Range: 2.5506. Min Range: -2.5506

Max Act: 2.3455. Min Act: -0.0000

Data Index: 564166 (C4 (Web Text))

Max Activating Token Index: 1003

Text #3

Max Range: 2.5506. Min Range: -2.5506

Max Act: 2.1077. Min Act: -0.0000

Data Index: 530764 (C4 (Web Text))

Max Activating Token Index: 464

Text #4

Max Range: 2.5506. Min Range: -2.5506

Max Act: 1.7590. Min Act: -0.0000

Data Index: 691504 (C4 (Web Text))

Max Activating Token Index: 256

Text #5

Max Range: 2.5506. Min Range: -2.5506

Max Act: 1.9251. Min Act: -0.0000

Data Index: 1004728 (C4 (Web Text))

Max Activating Token Index: 494

Text #6

Max Range: 2.5506. Min Range: -2.5506

Max Act: 1.9175. Min Act: -0.0000

Data Index: 981117 (C4 (Web Text))

Max Activating Token Index: 27

Text #7

Max Range: 2.5506. Min Range: -2.5506

Max Act: 1.8002. Min Act: -0.0000

Data Index: 430657 (C4 (Web Text))

Max Activating Token Index: 380

Text #8

Max Range: 2.5506. Min Range: -2.5506

Max Act: 1.8638. Min Act: -0.0000

Data Index: 507376 (C4 (Web Text))

Max Activating Token Index: 927

Text #9

Max Range: 2.5506. Min Range: -2.5506

Max Act: 1.8498. Min Act: -0.0000

Data Index: 828917 (C4 (Web Text))

Max Activating Token Index: 903

Text #10

Max Range: 2.5506. Min Range: -2.5506

Max Act: 1.8019. Min Act: -0.0000

Data Index: 895339 (C4 (Web Text))

Max Activating Token Index: 940

Text #11

Max Range: 2.5506. Min Range: -2.5506

Max Act: 1.8716. Min Act: -0.0000

Data Index: 982591 (C4 (Web Text))

Max Activating Token Index: 648

Text #12

Max Range: 2.5506. Min Range: -2.5506

Max Act: 1.8018. Min Act: -0.0000

Data Index: 1225465 (C4 (Web Text))

Max Activating Token Index: 50

Text #13

Max Range: 2.5506. Min Range: -2.5506

Max Act: 1.7944. Min Act: -0.0000

Data Index: 722292 (C4 (Web Text))

Max Activating Token Index: 746

Text #14

Max Range: 2.5506. Min Range: -2.5506

Max Act: 2.0429. Min Act: -0.0000

Data Index: 117945 (C4 (Web Text))

Max Activating Token Index: 296

Text #15

Max Range: 2.5506. Min Range: -2.5506

Max Act: 1.8417. Min Act: -0.0000

Data Index: 406233 (C4 (Web Text))

Max Activating Token Index: 992

Text #16

Max Range: 2.5506. Min Range: -2.5506

Max Act: 1.7212. Min Act: -0.0000

Data Index: 659796 (C4 (Web Text))

Max Activating Token Index: 80

Text #17

Max Range: 2.5506. Min Range: -2.5506

Max Act: 1.7387. Min Act: -0.0000

Data Index: 228950 (C4 (Web Text))

Max Activating Token Index: 60

Text #18

Max Range: 2.5506. Min Range: -2.5506

Max Act: 1.9433. Min Act: -0.0000

Data Index: 887905 (C4 (Web Text))

Max Activating Token Index: 1010

Text #19

Max Range: 2.5506. Min Range: -2.5506

Max Act: 1.7181. Min Act: -0.0000

Data Index: 382923 (C4 (Web Text))

Max Activating Token Index: 446