Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4084 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.9684. Min Range: -2.9684

Max Act: 2.9684. Min Act: -0.0000

Data Index: 1604395 (Python Code)

Max Activating Token Index: 812

Text #1

Max Range: 2.9684. Min Range: -2.9684

Max Act: 2.6352. Min Act: -0.0000

Data Index: 1075665 (C4 (Web Text))

Max Activating Token Index: 331

Text #2

Max Range: 2.9684. Min Range: -2.9684

Max Act: 2.8433. Min Act: 0.0001

Data Index: 137232 (C4 (Web Text))

Max Activating Token Index: 99

Text #3

Max Range: 2.9684. Min Range: -2.9684

Max Act: 2.7734. Min Act: -0.0000

Data Index: 936002 (C4 (Web Text))

Max Activating Token Index: 212

Text #4

Max Range: 2.9684. Min Range: -2.9684

Max Act: 2.5733. Min Act: -0.0000

Data Index: 220296 (C4 (Web Text))

Max Activating Token Index: 634

Text #5

Max Range: 2.9684. Min Range: -2.9684

Max Act: 2.5419. Min Act: -0.0000

Data Index: 697010 (C4 (Web Text))

Max Activating Token Index: 416

Text #6

Max Range: 2.9684. Min Range: -2.9684

Max Act: 2.4482. Min Act: -0.0000

Data Index: 880603 (C4 (Web Text))

Max Activating Token Index: 422

Text #7

Max Range: 2.9684. Min Range: -2.9684

Max Act: 2.4077. Min Act: -0.0000

Data Index: 450740 (C4 (Web Text))

Max Activating Token Index: 541

Text #8

Max Range: 2.9684. Min Range: -2.9684

Max Act: 2.3886. Min Act: 0.0001

Data Index: 278488 (C4 (Web Text))

Max Activating Token Index: 80

Text #9

Max Range: 2.9684. Min Range: -2.9684

Max Act: 2.5320. Min Act: -0.0000

Data Index: 332019 (C4 (Web Text))

Max Activating Token Index: 562

Text #10

Max Range: 2.9684. Min Range: -2.9684

Max Act: 2.4142. Min Act: -0.0000

Data Index: 312797 (C4 (Web Text))

Max Activating Token Index: 704

Text #11

Max Range: 2.9684. Min Range: -2.9684

Max Act: 2.2375. Min Act: 0.0001

Data Index: 507828 (C4 (Web Text))

Max Activating Token Index: 129

Text #12

Max Range: 2.9684. Min Range: -2.9684

Max Act: 2.3717. Min Act: -0.0000

Data Index: 708226 (C4 (Web Text))

Max Activating Token Index: 895

Text #13

Max Range: 2.9684. Min Range: -2.9684

Max Act: 2.3623. Min Act: 0.0000

Data Index: 77031 (C4 (Web Text))

Max Activating Token Index: 238

Text #14

Max Range: 2.9684. Min Range: -2.9684

Max Act: 2.4802. Min Act: -0.0000

Data Index: 698150 (C4 (Web Text))

Max Activating Token Index: 523

Text #15

Max Range: 2.9684. Min Range: -2.9684

Max Act: 2.4411. Min Act: -0.0000

Data Index: 789820 (C4 (Web Text))

Max Activating Token Index: 526

Text #16

Max Range: 2.9684. Min Range: -2.9684

Max Act: 2.3274. Min Act: -0.0000

Data Index: 1329562 (C4 (Web Text))

Max Activating Token Index: 641

Text #17

Max Range: 2.9684. Min Range: -2.9684

Max Act: 2.3993. Min Act: -0.0000

Data Index: 328334 (C4 (Web Text))

Max Activating Token Index: 907

Text #18

Max Range: 2.9684. Min Range: -2.9684

Max Act: 2.3281. Min Act: -0.0000

Data Index: 420041 (C4 (Web Text))

Max Activating Token Index: 574

Text #19

Max Range: 2.9684. Min Range: -2.9684

Max Act: 2.3034. Min Act: -0.0000

Data Index: 652600 (C4 (Web Text))

Max Activating Token Index: 727