Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 231 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.7655. Min Range: -0.7655

Max Act: 0.7655. Min Act: -0.0000

Data Index: 492082 (C4 (Web Text))

Max Activating Token Index: 292

Text #1

Max Range: 0.7655. Min Range: -0.7655

Max Act: 0.7205. Min Act: -0.0000

Data Index: 939697 (C4 (Web Text))

Max Activating Token Index: 414

Text #2

Max Range: 0.7655. Min Range: -0.7655

Max Act: 0.7275. Min Act: -0.0000

Data Index: 1270566 (C4 (Web Text))

Max Activating Token Index: 430

Text #3

Max Range: 0.7655. Min Range: -0.7655

Max Act: 0.6137. Min Act: -0.0000

Data Index: 138705 (C4 (Web Text))

Max Activating Token Index: 843

Text #4

Max Range: 0.7655. Min Range: -0.7655

Max Act: 0.5874. Min Act: -0.0000

Data Index: 1102118 (C4 (Web Text))

Max Activating Token Index: 817

Text #5

Max Range: 0.7655. Min Range: -0.7655

Max Act: 0.6037. Min Act: -0.0000

Data Index: 322862 (C4 (Web Text))

Max Activating Token Index: 672

Text #6

Max Range: 0.7655. Min Range: -0.7655

Max Act: 0.5821. Min Act: -0.0000

Data Index: 464147 (C4 (Web Text))

Max Activating Token Index: 86

Text #7

Max Range: 0.7655. Min Range: -0.7655

Max Act: 0.5981. Min Act: -0.0000

Data Index: 512437 (C4 (Web Text))

Max Activating Token Index: 769

Text #8

Max Range: 0.7655. Min Range: -0.7655

Max Act: 0.5912. Min Act: -0.0000

Data Index: 1096330 (C4 (Web Text))

Max Activating Token Index: 1008

Text #9

Max Range: 0.7655. Min Range: -0.7655

Max Act: 0.5603. Min Act: -0.0000

Data Index: 363950 (C4 (Web Text))

Max Activating Token Index: 60

Text #10

Max Range: 0.7655. Min Range: -0.7655

Max Act: 0.5771. Min Act: -0.0000

Data Index: 517582 (C4 (Web Text))

Max Activating Token Index: 103

Text #11

Max Range: 0.7655. Min Range: -0.7655

Max Act: 0.5844. Min Act: -0.0000

Data Index: 883357 (C4 (Web Text))

Max Activating Token Index: 655

Text #12

Max Range: 0.7655. Min Range: -0.7655

Max Act: 0.5645. Min Act: -0.0000

Data Index: 685514 (C4 (Web Text))

Max Activating Token Index: 617

Text #13

Max Range: 0.7655. Min Range: -0.7655

Max Act: 0.5634. Min Act: -0.0000

Data Index: 192515 (C4 (Web Text))

Max Activating Token Index: 88

Text #14

Max Range: 0.7655. Min Range: -0.7655

Max Act: 0.5636. Min Act: -0.0000

Data Index: 1008240 (C4 (Web Text))

Max Activating Token Index: 95

Text #15

Max Range: 0.7655. Min Range: -0.7655

Max Act: 0.5779. Min Act: -0.0000

Data Index: 812464 (C4 (Web Text))

Max Activating Token Index: 821

Text #16

Max Range: 0.7655. Min Range: -0.7655

Max Act: 0.5398. Min Act: -0.0000

Data Index: 1329680 (C4 (Web Text))

Max Activating Token Index: 852

Text #17

Max Range: 0.7655. Min Range: -0.7655

Max Act: 0.5370. Min Act: -0.0000

Data Index: 402159 (C4 (Web Text))

Max Activating Token Index: 59

Text #18

Max Range: 0.7655. Min Range: -0.7655

Max Act: 0.5151. Min Act: -0.0000

Data Index: 1108704 (C4 (Web Text))

Max Activating Token Index: 840

Text #19

Max Range: 0.7655. Min Range: -0.7655

Max Act: 0.5193. Min Act: -0.0000

Data Index: 734575 (C4 (Web Text))

Max Activating Token Index: 506