Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3035 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.5553. Min Range: -0.5553

Max Act: 0.5553. Min Act: -0.0000

Data Index: 873370 (C4 (Web Text))

Max Activating Token Index: 917

Text #1

Max Range: 0.5553. Min Range: -0.5553

Max Act: 0.4857. Min Act: -0.0000

Data Index: 264847 (C4 (Web Text))

Max Activating Token Index: 681

Text #2

Max Range: 0.5553. Min Range: -0.5553

Max Act: 0.4821. Min Act: -0.0000

Data Index: 1154643 (C4 (Web Text))

Max Activating Token Index: 821

Text #3

Max Range: 0.5553. Min Range: -0.5553

Max Act: 0.4651. Min Act: -0.0000

Data Index: 596845 (C4 (Web Text))

Max Activating Token Index: 869

Text #4

Max Range: 0.5553. Min Range: -0.5553

Max Act: 0.4898. Min Act: -0.0000

Data Index: 1252090 (C4 (Web Text))

Max Activating Token Index: 420

Text #5

Max Range: 0.5553. Min Range: -0.5553

Max Act: 0.4884. Min Act: -0.0000

Data Index: 814527 (C4 (Web Text))

Max Activating Token Index: 888

Text #6

Max Range: 0.5553. Min Range: -0.5553

Max Act: 0.4741. Min Act: -0.0000

Data Index: 1038037 (C4 (Web Text))

Max Activating Token Index: 120

Text #7

Max Range: 0.5553. Min Range: -0.5553

Max Act: 0.4612. Min Act: -0.0000

Data Index: 487256 (C4 (Web Text))

Max Activating Token Index: 39

Text #8

Max Range: 0.5553. Min Range: -0.5553

Max Act: 0.4729. Min Act: 0.0000

Data Index: 1348731 (C4 (Web Text))

Max Activating Token Index: 878

Text #9

Max Range: 0.5553. Min Range: -0.5553

Max Act: 0.4632. Min Act: -0.0000

Data Index: 1346934 (C4 (Web Text))

Max Activating Token Index: 883

Text #10

Max Range: 0.5553. Min Range: -0.5553

Max Act: 0.4463. Min Act: -0.0000

Data Index: 1305960 (C4 (Web Text))

Max Activating Token Index: 589

Text #11

Max Range: 0.5553. Min Range: -0.5553

Max Act: 0.4740. Min Act: 0.0000

Data Index: 998050 (C4 (Web Text))

Max Activating Token Index: 680

Text #12

Max Range: 0.5553. Min Range: -0.5553

Max Act: 0.4369. Min Act: -0.0000

Data Index: 611758 (C4 (Web Text))

Max Activating Token Index: 402

Text #13

Max Range: 0.5553. Min Range: -0.5553

Max Act: 0.4409. Min Act: -0.0000

Data Index: 569258 (C4 (Web Text))

Max Activating Token Index: 659

Text #14

Max Range: 0.5553. Min Range: -0.5553

Max Act: 0.4227. Min Act: -0.0000

Data Index: 817074 (C4 (Web Text))

Max Activating Token Index: 560

Text #15

Max Range: 0.5553. Min Range: -0.5553

Max Act: 0.4238. Min Act: -0.0000

Data Index: 2695 (C4 (Web Text))

Max Activating Token Index: 1010

Text #16

Max Range: 0.5553. Min Range: -0.5553

Max Act: 0.4354. Min Act: -0.0000

Data Index: 41985 (C4 (Web Text))

Max Activating Token Index: 589

Text #17

Max Range: 0.5553. Min Range: -0.5553

Max Act: 0.4216. Min Act: -0.0000

Data Index: 295238 (C4 (Web Text))

Max Activating Token Index: 673

Text #18

Max Range: 0.5553. Min Range: -0.5553

Max Act: 0.4137. Min Act: -0.0000

Data Index: 944495 (C4 (Web Text))

Max Activating Token Index: 569

Text #19

Max Range: 0.5553. Min Range: -0.5553

Max Act: 0.4029. Min Act: -0.0000

Data Index: 558923 (C4 (Web Text))

Max Activating Token Index: 874