Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4097 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.0593. Min Range: -1.0593

Max Act: 1.0593. Min Act: -0.0001

Data Index: 603238 (C4 (Web Text))

Max Activating Token Index: 890

Text #1

Max Range: 1.0593. Min Range: -1.0593

Max Act: 0.9092. Min Act: -0.0001

Data Index: 131378 (C4 (Web Text))

Max Activating Token Index: 792

Text #2

Max Range: 1.0593. Min Range: -1.0593

Max Act: 0.8714. Min Act: -0.0001

Data Index: 748608 (C4 (Web Text))

Max Activating Token Index: 800

Text #3

Max Range: 1.0593. Min Range: -1.0593

Max Act: 0.8818. Min Act: -0.0001

Data Index: 997507 (C4 (Web Text))

Max Activating Token Index: 1009

Text #4

Max Range: 1.0593. Min Range: -1.0593

Max Act: 0.8025. Min Act: -0.0001

Data Index: 563760 (C4 (Web Text))

Max Activating Token Index: 877

Text #5

Max Range: 1.0593. Min Range: -1.0593

Max Act: 0.7577. Min Act: -0.0001

Data Index: 643247 (C4 (Web Text))

Max Activating Token Index: 657

Text #6

Max Range: 1.0593. Min Range: -1.0593

Max Act: 0.7766. Min Act: -0.0001

Data Index: 215019 (C4 (Web Text))

Max Activating Token Index: 632

Text #7

Max Range: 1.0593. Min Range: -1.0593

Max Act: 0.7700. Min Act: -0.0001

Data Index: 560693 (C4 (Web Text))

Max Activating Token Index: 515

Text #8

Max Range: 1.0593. Min Range: -1.0593

Max Act: 0.7596. Min Act: -0.0000

Data Index: 36324 (C4 (Web Text))

Max Activating Token Index: 793

Text #9

Max Range: 1.0593. Min Range: -1.0593

Max Act: 0.7803. Min Act: -0.0001

Data Index: 43226 (C4 (Web Text))

Max Activating Token Index: 442

Text #10

Max Range: 1.0593. Min Range: -1.0593

Max Act: 0.7600. Min Act: -0.0001

Data Index: 968782 (C4 (Web Text))

Max Activating Token Index: 672

Text #11

Max Range: 1.0593. Min Range: -1.0593

Max Act: 0.7762. Min Act: -0.0001

Data Index: 740363 (C4 (Web Text))

Max Activating Token Index: 71

Text #12

Max Range: 1.0593. Min Range: -1.0593

Max Act: 0.7969. Min Act: -0.0001

Data Index: 474021 (C4 (Web Text))

Max Activating Token Index: 378

Text #13

Max Range: 1.0593. Min Range: -1.0593

Max Act: 0.7679. Min Act: -0.0001

Data Index: 1224536 (C4 (Web Text))

Max Activating Token Index: 475

Text #14

Max Range: 1.0593. Min Range: -1.0593

Max Act: 0.7204. Min Act: -0.0001

Data Index: 104640 (C4 (Web Text))

Max Activating Token Index: 824

Text #15

Max Range: 1.0593. Min Range: -1.0593

Max Act: 0.6830. Min Act: -0.0001

Data Index: 1101484 (C4 (Web Text))

Max Activating Token Index: 183

Text #16

Max Range: 1.0593. Min Range: -1.0593

Max Act: 0.7670. Min Act: -0.0001

Data Index: 1357112 (C4 (Web Text))

Max Activating Token Index: 697

Text #17

Max Range: 1.0593. Min Range: -1.0593

Max Act: 0.7691. Min Act: -0.0001

Data Index: 649096 (C4 (Web Text))

Max Activating Token Index: 560

Text #18

Max Range: 1.0593. Min Range: -1.0593

Max Act: 0.7342. Min Act: -0.0001

Data Index: 595420 (C4 (Web Text))

Max Activating Token Index: 436

Text #19

Max Range: 1.0593. Min Range: -1.0593

Max Act: 0.7381. Min Act: -0.0001

Data Index: 1003016 (C4 (Web Text))

Max Activating Token Index: 598