Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2636 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.5891. Min Range: -3.5891

Max Act: 3.5891. Min Act: -0.0001

Data Index: 1598653 (Python Code)

Max Activating Token Index: 393

Text #1

Max Range: 3.5891. Min Range: -3.5891

Max Act: 3.2745. Min Act: -0.0001

Data Index: 615935 (C4 (Web Text))

Max Activating Token Index: 803

Text #2

Max Range: 3.5891. Min Range: -3.5891

Max Act: 3.5475. Min Act: -0.0001

Data Index: 1601144 (Python Code)

Max Activating Token Index: 323

Text #3

Max Range: 3.5891. Min Range: -3.5891

Max Act: 3.3919. Min Act: -0.0001

Data Index: 1651237 (Python Code)

Max Activating Token Index: 337

Text #4

Max Range: 3.5891. Min Range: -3.5891

Max Act: 3.4021. Min Act: -0.0001

Data Index: 1563603 (Python Code)

Max Activating Token Index: 581

Text #5

Max Range: 3.5891. Min Range: -3.5891

Max Act: 3.0323. Min Act: -0.0001

Data Index: 88787 (C4 (Web Text))

Max Activating Token Index: 491

Text #6

Max Range: 3.5891. Min Range: -3.5891

Max Act: 3.0228. Min Act: -0.0001

Data Index: 412776 (C4 (Web Text))

Max Activating Token Index: 180

Text #7

Max Range: 3.5891. Min Range: -3.5891

Max Act: 2.9815. Min Act: -0.0001

Data Index: 1346836 (C4 (Web Text))

Max Activating Token Index: 181

Text #8

Max Range: 3.5891. Min Range: -3.5891

Max Act: 2.8492. Min Act: -0.0001

Data Index: 217130 (C4 (Web Text))

Max Activating Token Index: 603

Text #9

Max Range: 3.5891. Min Range: -3.5891

Max Act: 2.9478. Min Act: -0.0001

Data Index: 1647417 (Python Code)

Max Activating Token Index: 624

Text #10

Max Range: 3.5891. Min Range: -3.5891

Max Act: 2.8128. Min Act: -0.0000

Data Index: 1198246 (C4 (Web Text))

Max Activating Token Index: 704

Text #11

Max Range: 3.5891. Min Range: -3.5891

Max Act: 2.7066. Min Act: -0.0001

Data Index: 895164 (C4 (Web Text))

Max Activating Token Index: 411

Text #12

Max Range: 3.5891. Min Range: -3.5891

Max Act: 2.8652. Min Act: -0.0000

Data Index: 819913 (C4 (Web Text))

Max Activating Token Index: 8

Text #13

Max Range: 3.5891. Min Range: -3.5891

Max Act: 2.7183. Min Act: -0.0001

Data Index: 938106 (C4 (Web Text))

Max Activating Token Index: 482

Text #14

Max Range: 3.5891. Min Range: -3.5891

Max Act: 3.0852. Min Act: -0.0001

Data Index: 1118627 (C4 (Web Text))

Max Activating Token Index: 358

Text #15

Max Range: 3.5891. Min Range: -3.5891

Max Act: 3.0908. Min Act: -0.0000

Data Index: 513709 (C4 (Web Text))

Max Activating Token Index: 989

Text #16

Max Range: 3.5891. Min Range: -3.5891

Max Act: 2.7929. Min Act: -0.0001

Data Index: 702371 (C4 (Web Text))

Max Activating Token Index: 141

Text #17

Max Range: 3.5891. Min Range: -3.5891

Max Act: 2.9150. Min Act: -0.0000

Data Index: 1596054 (Python Code)

Max Activating Token Index: 456

Text #18

Max Range: 3.5891. Min Range: -3.5891

Max Act: 2.6706. Min Act: -0.0001

Data Index: 758853 (C4 (Web Text))

Max Activating Token Index: 933

Text #19

Max Range: 3.5891. Min Range: -3.5891

Max Act: 2.5408. Min Act: -0.0001

Data Index: 1632441 (Python Code)

Max Activating Token Index: 309