Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3209 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.2122. Min Range: -2.2122

Max Act: 2.1018. Min Act: -0.0000

Data Index: 688368 (C4 (Web Text))

Max Activating Token Index: 783

Text #1

Max Range: 2.2122. Min Range: -2.2122

Max Act: 2.2122. Min Act: -0.0000

Data Index: 1350464 (C4 (Web Text))

Max Activating Token Index: 685

Text #2

Max Range: 2.2122. Min Range: -2.2122

Max Act: 2.1950. Min Act: -0.0000

Data Index: 244784 (C4 (Web Text))

Max Activating Token Index: 477

Text #3

Max Range: 2.2122. Min Range: -2.2122

Max Act: 2.0037. Min Act: -0.0000

Data Index: 1358605 (C4 (Web Text))

Max Activating Token Index: 758

Text #4

Max Range: 2.2122. Min Range: -2.2122

Max Act: 1.9926. Min Act: -0.0000

Data Index: 221397 (C4 (Web Text))

Max Activating Token Index: 817

Text #5

Max Range: 2.2122. Min Range: -2.2122

Max Act: 2.1371. Min Act: -0.0000

Data Index: 41239 (C4 (Web Text))

Max Activating Token Index: 359

Text #6

Max Range: 2.2122. Min Range: -2.2122

Max Act: 2.0036. Min Act: -0.0000

Data Index: 587073 (C4 (Web Text))

Max Activating Token Index: 520

Text #7

Max Range: 2.2122. Min Range: -2.2122

Max Act: 1.8864. Min Act: -0.0000

Data Index: 360780 (C4 (Web Text))

Max Activating Token Index: 839

Text #8

Max Range: 2.2122. Min Range: -2.2122

Max Act: 1.8684. Min Act: -0.0000

Data Index: 906096 (C4 (Web Text))

Max Activating Token Index: 734

Text #9

Max Range: 2.2122. Min Range: -2.2122

Max Act: 1.9470. Min Act: -0.0000

Data Index: 1514369 (Python Code)

Max Activating Token Index: 905

Text #10

Max Range: 2.2122. Min Range: -2.2122

Max Act: 1.9492. Min Act: -0.0000

Data Index: 712154 (C4 (Web Text))

Max Activating Token Index: 814

Text #11

Max Range: 2.2122. Min Range: -2.2122

Max Act: 1.9475. Min Act: -0.0000

Data Index: 919128 (C4 (Web Text))

Max Activating Token Index: 207

Text #12

Max Range: 2.2122. Min Range: -2.2122

Max Act: 2.0430. Min Act: -0.0000

Data Index: 176593 (C4 (Web Text))

Max Activating Token Index: 271

Text #13

Max Range: 2.2122. Min Range: -2.2122

Max Act: 1.8746. Min Act: -0.0000

Data Index: 159637 (C4 (Web Text))

Max Activating Token Index: 589

Text #14

Max Range: 2.2122. Min Range: -2.2122

Max Act: 1.8840. Min Act: -0.0000

Data Index: 857105 (C4 (Web Text))

Max Activating Token Index: 970

Text #15

Max Range: 2.2122. Min Range: -2.2122

Max Act: 1.7701. Min Act: -0.0000

Data Index: 661620 (C4 (Web Text))

Max Activating Token Index: 740

Text #16

Max Range: 2.2122. Min Range: -2.2122

Max Act: 1.8690. Min Act: -0.0000

Data Index: 353930 (C4 (Web Text))

Max Activating Token Index: 559

Text #17

Max Range: 2.2122. Min Range: -2.2122

Max Act: 1.9720. Min Act: -0.0000

Data Index: 1167329 (C4 (Web Text))

Max Activating Token Index: 599

Text #18

Max Range: 2.2122. Min Range: -2.2122

Max Act: 1.9234. Min Act: -0.0000

Data Index: 758484 (C4 (Web Text))

Max Activating Token Index: 322

Text #19

Max Range: 2.2122. Min Range: -2.2122

Max Act: 1.6890. Min Act: -0.0000

Data Index: 767823 (C4 (Web Text))

Max Activating Token Index: 200