Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3090 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.7877. Min Range: -1.7877

Max Act: 1.5718. Min Act: -0.0000

Data Index: 917830 (C4 (Web Text))

Max Activating Token Index: 963

Text #1

Max Range: 1.7877. Min Range: -1.7877

Max Act: 1.7877. Min Act: -0.0000

Data Index: 890333 (C4 (Web Text))

Max Activating Token Index: 20

Text #2

Max Range: 1.7877. Min Range: -1.7877

Max Act: 1.6742. Min Act: -0.0000

Data Index: 1200718 (C4 (Web Text))

Max Activating Token Index: 56

Text #3

Max Range: 1.7877. Min Range: -1.7877

Max Act: 1.4890. Min Act: -0.0000

Data Index: 362155 (C4 (Web Text))

Max Activating Token Index: 860

Text #4

Max Range: 1.7877. Min Range: -1.7877

Max Act: 1.5282. Min Act: -0.0000

Data Index: 288587 (C4 (Web Text))

Max Activating Token Index: 534

Text #5

Max Range: 1.7877. Min Range: -1.7877

Max Act: 1.5036. Min Act: -0.0000

Data Index: 559977 (C4 (Web Text))

Max Activating Token Index: 86

Text #6

Max Range: 1.7877. Min Range: -1.7877

Max Act: 1.3974. Min Act: -0.0000

Data Index: 650257 (C4 (Web Text))

Max Activating Token Index: 66

Text #7

Max Range: 1.7877. Min Range: -1.7877

Max Act: 1.4705. Min Act: -0.0000

Data Index: 606902 (C4 (Web Text))

Max Activating Token Index: 46

Text #8

Max Range: 1.7877. Min Range: -1.7877

Max Act: 1.4647. Min Act: -0.0000

Data Index: 805059 (C4 (Web Text))

Max Activating Token Index: 868

Text #9

Max Range: 1.7877. Min Range: -1.7877

Max Act: 1.3703. Min Act: -0.0000

Data Index: 87305 (C4 (Web Text))

Max Activating Token Index: 832

Text #10

Max Range: 1.7877. Min Range: -1.7877

Max Act: 1.5319. Min Act: -0.0000

Data Index: 910222 (C4 (Web Text))

Max Activating Token Index: 108

Text #11

Max Range: 1.7877. Min Range: -1.7877

Max Act: 1.4445. Min Act: -0.0000

Data Index: 1017910 (C4 (Web Text))

Max Activating Token Index: 511

Text #12

Max Range: 1.7877. Min Range: -1.7877

Max Act: 1.3616. Min Act: -0.0000

Data Index: 216476 (C4 (Web Text))

Max Activating Token Index: 117

Text #13

Max Range: 1.7877. Min Range: -1.7877

Max Act: 1.4494. Min Act: -0.0000

Data Index: 734948 (C4 (Web Text))

Max Activating Token Index: 775

Text #14

Max Range: 1.7877. Min Range: -1.7877

Max Act: 1.5147. Min Act: -0.0000

Data Index: 1201145 (C4 (Web Text))

Max Activating Token Index: 366

Text #15

Max Range: 1.7877. Min Range: -1.7877

Max Act: 1.4377. Min Act: -0.0000

Data Index: 440088 (C4 (Web Text))

Max Activating Token Index: 670

Text #16

Max Range: 1.7877. Min Range: -1.7877

Max Act: 1.3971. Min Act: -0.0000

Data Index: 1164865 (C4 (Web Text))

Max Activating Token Index: 86

Text #17

Max Range: 1.7877. Min Range: -1.7877

Max Act: 1.3804. Min Act: -0.0000

Data Index: 1010096 (C4 (Web Text))

Max Activating Token Index: 861

Text #18

Max Range: 1.7877. Min Range: -1.7877

Max Act: 1.3375. Min Act: -0.0000

Data Index: 227328 (C4 (Web Text))

Max Activating Token Index: 774

Text #19

Max Range: 1.7877. Min Range: -1.7877

Max Act: 1.3780. Min Act: -0.0000

Data Index: 278503 (C4 (Web Text))

Max Activating Token Index: 737