Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3163 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.7677. Min Range: -1.7677

Max Act: 1.7677. Min Act: -0.0000

Data Index: 831920 (C4 (Web Text))

Max Activating Token Index: 750

Text #1

Max Range: 1.7677. Min Range: -1.7677

Max Act: 1.4562. Min Act: -0.0000

Data Index: 895482 (C4 (Web Text))

Max Activating Token Index: 712

Text #2

Max Range: 1.7677. Min Range: -1.7677

Max Act: 1.5366. Min Act: -0.0000

Data Index: 647426 (C4 (Web Text))

Max Activating Token Index: 289

Text #3

Max Range: 1.7677. Min Range: -1.7677

Max Act: 1.5646. Min Act: -0.0000

Data Index: 1014938 (C4 (Web Text))

Max Activating Token Index: 314

Text #4

Max Range: 1.7677. Min Range: -1.7677

Max Act: 1.3404. Min Act: -0.0000

Data Index: 96526 (C4 (Web Text))

Max Activating Token Index: 447

Text #5

Max Range: 1.7677. Min Range: -1.7677

Max Act: 1.3229. Min Act: -0.0000

Data Index: 166044 (C4 (Web Text))

Max Activating Token Index: 902

Text #6

Max Range: 1.7677. Min Range: -1.7677

Max Act: 1.3135. Min Act: -0.0000

Data Index: 85260 (C4 (Web Text))

Max Activating Token Index: 534

Text #7

Max Range: 1.7677. Min Range: -1.7677

Max Act: 1.2485. Min Act: -0.0000

Data Index: 772891 (C4 (Web Text))

Max Activating Token Index: 453

Text #8

Max Range: 1.7677. Min Range: -1.7677

Max Act: 1.2314. Min Act: -0.0000

Data Index: 1071012 (C4 (Web Text))

Max Activating Token Index: 465

Text #9

Max Range: 1.7677. Min Range: -1.7677

Max Act: 1.2058. Min Act: -0.0000

Data Index: 1050912 (C4 (Web Text))

Max Activating Token Index: 87

Text #10

Max Range: 1.7677. Min Range: -1.7677

Max Act: 1.1261. Min Act: -0.0000

Data Index: 623332 (C4 (Web Text))

Max Activating Token Index: 118

Text #11

Max Range: 1.7677. Min Range: -1.7677

Max Act: 1.0699. Min Act: -0.0000

Data Index: 510577 (C4 (Web Text))

Max Activating Token Index: 253

Text #12

Max Range: 1.7677. Min Range: -1.7677

Max Act: 0.9750. Min Act: -0.0000

Data Index: 1115563 (C4 (Web Text))

Max Activating Token Index: 876

Text #13

Max Range: 1.7677. Min Range: -1.7677

Max Act: 0.9887. Min Act: -0.0000

Data Index: 476610 (C4 (Web Text))

Max Activating Token Index: 917

Text #14

Max Range: 1.7677. Min Range: -1.7677

Max Act: 0.9311. Min Act: -0.0000

Data Index: 1320236 (C4 (Web Text))

Max Activating Token Index: 652

Text #15

Max Range: 1.7677. Min Range: -1.7677

Max Act: 1.1634. Min Act: -0.0000

Data Index: 469960 (C4 (Web Text))

Max Activating Token Index: 711

Text #16

Max Range: 1.7677. Min Range: -1.7677

Max Act: 0.9680. Min Act: -0.0000

Data Index: 270708 (C4 (Web Text))

Max Activating Token Index: 691

Text #17

Max Range: 1.7677. Min Range: -1.7677

Max Act: 0.9529. Min Act: -0.0000

Data Index: 95496 (C4 (Web Text))

Max Activating Token Index: 724

Text #18

Max Range: 1.7677. Min Range: -1.7677

Max Act: 0.9605. Min Act: -0.0000

Data Index: 509721 (C4 (Web Text))

Max Activating Token Index: 776

Text #19

Max Range: 1.7677. Min Range: -1.7677

Max Act: 0.9428. Min Act: -0.0000

Data Index: 446413 (C4 (Web Text))

Max Activating Token Index: 617