Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4449 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.9393. Min Range: -3.9393

Max Act: 3.9393. Min Act: -0.0000

Data Index: 1153749 (C4 (Web Text))

Max Activating Token Index: 29

Text #1

Max Range: 3.9393. Min Range: -3.9393

Max Act: 3.5760. Min Act: -0.0000

Data Index: 616242 (C4 (Web Text))

Max Activating Token Index: 506

Text #2

Max Range: 3.9393. Min Range: -3.9393

Max Act: 3.2309. Min Act: -0.0000

Data Index: 1128738 (C4 (Web Text))

Max Activating Token Index: 403

Text #3

Max Range: 3.9393. Min Range: -3.9393

Max Act: 3.1598. Min Act: -0.0000

Data Index: 1101088 (C4 (Web Text))

Max Activating Token Index: 514

Text #4

Max Range: 3.9393. Min Range: -3.9393

Max Act: 2.9508. Min Act: -0.0000

Data Index: 635523 (C4 (Web Text))

Max Activating Token Index: 601

Text #5

Max Range: 3.9393. Min Range: -3.9393

Max Act: 2.8820. Min Act: -0.0000

Data Index: 208995 (C4 (Web Text))

Max Activating Token Index: 751

Text #6

Max Range: 3.9393. Min Range: -3.9393

Max Act: 3.1197. Min Act: -0.0000

Data Index: 678533 (C4 (Web Text))

Max Activating Token Index: 636

Text #7

Max Range: 3.9393. Min Range: -3.9393

Max Act: 2.8277. Min Act: -0.0000

Data Index: 227904 (C4 (Web Text))

Max Activating Token Index: 571

Text #8

Max Range: 3.9393. Min Range: -3.9393

Max Act: 2.9497. Min Act: -0.0000

Data Index: 379700 (C4 (Web Text))

Max Activating Token Index: 73

Text #9

Max Range: 3.9393. Min Range: -3.9393

Max Act: 2.8033. Min Act: -0.0000

Data Index: 5868 (C4 (Web Text))

Max Activating Token Index: 201

Text #10

Max Range: 3.9393. Min Range: -3.9393

Max Act: 2.6088. Min Act: -0.0000

Data Index: 470322 (C4 (Web Text))

Max Activating Token Index: 695

Text #11

Max Range: 3.9393. Min Range: -3.9393

Max Act: 2.6248. Min Act: -0.0000

Data Index: 1100986 (C4 (Web Text))

Max Activating Token Index: 189

Text #12

Max Range: 3.9393. Min Range: -3.9393

Max Act: 2.7418. Min Act: -0.0000

Data Index: 179454 (C4 (Web Text))

Max Activating Token Index: 193

Text #13

Max Range: 3.9393. Min Range: -3.9393

Max Act: 2.6429. Min Act: -0.0000

Data Index: 188786 (C4 (Web Text))

Max Activating Token Index: 873

Text #14

Max Range: 3.9393. Min Range: -3.9393

Max Act: 2.4649. Min Act: -0.0000

Data Index: 523006 (C4 (Web Text))

Max Activating Token Index: 543

Text #15

Max Range: 3.9393. Min Range: -3.9393

Max Act: 2.5977. Min Act: -0.0000

Data Index: 185162 (C4 (Web Text))

Max Activating Token Index: 357

Text #16

Max Range: 3.9393. Min Range: -3.9393

Max Act: 2.4475. Min Act: -0.0000

Data Index: 257064 (C4 (Web Text))

Max Activating Token Index: 509

Text #17

Max Range: 3.9393. Min Range: -3.9393

Max Act: 2.5694. Min Act: -0.0000

Data Index: 503387 (C4 (Web Text))

Max Activating Token Index: 386

Text #18

Max Range: 3.9393. Min Range: -3.9393

Max Act: 2.5452. Min Act: -0.0000

Data Index: 55227 (C4 (Web Text))

Max Activating Token Index: 122

Text #19

Max Range: 3.9393. Min Range: -3.9393

Max Act: 2.5053. Min Act: -0.0000

Data Index: 203971 (C4 (Web Text))

Max Activating Token Index: 281