Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4834 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.7455. Min Range: -0.7455

Max Act: 0.7455. Min Act: -0.0000

Data Index: 618359 (C4 (Web Text))

Max Activating Token Index: 516

Text #1

Max Range: 0.7455. Min Range: -0.7455

Max Act: 0.7450. Min Act: -0.0000

Data Index: 277796 (C4 (Web Text))

Max Activating Token Index: 121

Text #2

Max Range: 0.7455. Min Range: -0.7455

Max Act: 0.6884. Min Act: -0.0000

Data Index: 837403 (C4 (Web Text))

Max Activating Token Index: 482

Text #3

Max Range: 0.7455. Min Range: -0.7455

Max Act: 0.7184. Min Act: -0.0000

Data Index: 465164 (C4 (Web Text))

Max Activating Token Index: 586

Text #4

Max Range: 0.7455. Min Range: -0.7455

Max Act: 0.6380. Min Act: -0.0000

Data Index: 329764 (C4 (Web Text))

Max Activating Token Index: 985

Text #5

Max Range: 0.7455. Min Range: -0.7455

Max Act: 0.5934. Min Act: -0.0000

Data Index: 1212111 (C4 (Web Text))

Max Activating Token Index: 102

Text #6

Max Range: 0.7455. Min Range: -0.7455

Max Act: 0.5709. Min Act: -0.0000

Data Index: 124622 (C4 (Web Text))

Max Activating Token Index: 62

Text #7

Max Range: 0.7455. Min Range: -0.7455

Max Act: 0.5806. Min Act: -0.0000

Data Index: 984410 (C4 (Web Text))

Max Activating Token Index: 753

Text #8

Max Range: 0.7455. Min Range: -0.7455

Max Act: 0.5139. Min Act: -0.0000

Data Index: 1181604 (C4 (Web Text))

Max Activating Token Index: 132

Text #9

Max Range: 0.7455. Min Range: -0.7455

Max Act: 0.4937. Min Act: -0.0000

Data Index: 1211376 (C4 (Web Text))

Max Activating Token Index: 228

Text #10

Max Range: 0.7455. Min Range: -0.7455

Max Act: 0.4815. Min Act: -0.0000

Data Index: 382097 (C4 (Web Text))

Max Activating Token Index: 592

Text #11

Max Range: 0.7455. Min Range: -0.7455

Max Act: 0.4228. Min Act: -0.0000

Data Index: 1045945 (C4 (Web Text))

Max Activating Token Index: 894

Text #12

Max Range: 0.7455. Min Range: -0.7455

Max Act: 0.3893. Min Act: -0.0000

Data Index: 181803 (C4 (Web Text))

Max Activating Token Index: 494

Text #13

Max Range: 0.7455. Min Range: -0.7455

Max Act: 0.3764. Min Act: -0.0000

Data Index: 1011954 (C4 (Web Text))

Max Activating Token Index: 871

Text #14

Max Range: 0.7455. Min Range: -0.7455

Max Act: 0.3967. Min Act: -0.0000

Data Index: 921533 (C4 (Web Text))

Max Activating Token Index: 739

Text #15

Max Range: 0.7455. Min Range: -0.7455

Max Act: 0.3718. Min Act: -0.0000

Data Index: 441442 (C4 (Web Text))

Max Activating Token Index: 876

Text #16

Max Range: 0.7455. Min Range: -0.7455

Max Act: 0.4358. Min Act: -0.0000

Data Index: 282549 (C4 (Web Text))

Max Activating Token Index: 517

Text #17

Max Range: 0.7455. Min Range: -0.7455

Max Act: 0.3917. Min Act: -0.0000

Data Index: 1045019 (C4 (Web Text))

Max Activating Token Index: 569

Text #18

Max Range: 0.7455. Min Range: -0.7455

Max Act: 0.3882. Min Act: -0.0000

Data Index: 270378 (C4 (Web Text))

Max Activating Token Index: 752

Text #19

Max Range: 0.7455. Min Range: -0.7455

Max Act: 0.3909. Min Act: -0.0000

Data Index: 941350 (C4 (Web Text))

Max Activating Token Index: 226