Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2442 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 0.6438. Min Range: -0.6438

Max Act: 0.6438. Min Act: -0.0004

Data Index: 423564 (C4 (Web Text))

Max Activating Token Index: 133

Text #1

Max Range: 0.6438. Min Range: -0.6438

Max Act: 0.6253. Min Act: -0.0004

Data Index: 644772 (C4 (Web Text))

Max Activating Token Index: 278

Text #2

Max Range: 0.6438. Min Range: -0.6438

Max Act: 0.5665. Min Act: -0.0004

Data Index: 18048 (C4 (Web Text))

Max Activating Token Index: 169

Text #3

Max Range: 0.6438. Min Range: -0.6438

Max Act: 0.5461. Min Act: -0.0004

Data Index: 445277 (C4 (Web Text))

Max Activating Token Index: 830

Text #4

Max Range: 0.6438. Min Range: -0.6438

Max Act: 0.5617. Min Act: -0.0004

Data Index: 978268 (C4 (Web Text))

Max Activating Token Index: 47

Text #5

Max Range: 0.6438. Min Range: -0.6438

Max Act: 0.5750. Min Act: -0.0004

Data Index: 690913 (C4 (Web Text))

Max Activating Token Index: 282

Text #6

Max Range: 0.6438. Min Range: -0.6438

Max Act: 0.5357. Min Act: -0.0004

Data Index: 387653 (C4 (Web Text))

Max Activating Token Index: 110

Text #7

Max Range: 0.6438. Min Range: -0.6438

Max Act: 0.5829. Min Act: -0.0004

Data Index: 742638 (C4 (Web Text))

Max Activating Token Index: 916

Text #8

Max Range: 0.6438. Min Range: -0.6438

Max Act: 0.6029. Min Act: -0.0004

Data Index: 838045 (C4 (Web Text))

Max Activating Token Index: 366

Text #9

Max Range: 0.6438. Min Range: -0.6438

Max Act: 0.5407. Min Act: -0.0004

Data Index: 721406 (C4 (Web Text))

Max Activating Token Index: 780

Text #10

Max Range: 0.6438. Min Range: -0.6438

Max Act: 0.5756. Min Act: -0.0004

Data Index: 819410 (C4 (Web Text))

Max Activating Token Index: 482

Text #11

Max Range: 0.6438. Min Range: -0.6438

Max Act: 0.5478. Min Act: -0.0004

Data Index: 627216 (C4 (Web Text))

Max Activating Token Index: 734

Text #12

Max Range: 0.6438. Min Range: -0.6438

Max Act: 0.5281. Min Act: -0.0004

Data Index: 1288581 (C4 (Web Text))

Max Activating Token Index: 479

Text #13

Max Range: 0.6438. Min Range: -0.6438

Max Act: 0.5573. Min Act: -0.0004

Data Index: 106306 (C4 (Web Text))

Max Activating Token Index: 392

Text #14

Max Range: 0.6438. Min Range: -0.6438

Max Act: 0.5065. Min Act: -0.0004

Data Index: 448836 (C4 (Web Text))

Max Activating Token Index: 692

Text #15

Max Range: 0.6438. Min Range: -0.6438

Max Act: 0.5785. Min Act: -0.0004

Data Index: 125534 (C4 (Web Text))

Max Activating Token Index: 777

Text #16

Max Range: 0.6438. Min Range: -0.6438

Max Act: 0.5204. Min Act: -0.0004

Data Index: 682146 (C4 (Web Text))

Max Activating Token Index: 464

Text #17

Max Range: 0.6438. Min Range: -0.6438

Max Act: 0.5371. Min Act: -0.0004

Data Index: 539340 (C4 (Web Text))

Max Activating Token Index: 987

Text #18

Max Range: 0.6438. Min Range: -0.6438

Max Act: 0.4516. Min Act: -0.0004

Data Index: 780471 (C4 (Web Text))

Max Activating Token Index: 529

Text #19

Max Range: 0.6438. Min Range: -0.6438

Max Act: 0.5237. Min Act: -0.0004

Data Index: 635697 (C4 (Web Text))

Max Activating Token Index: 165