Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2009 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.3470. Min Range: -1.3470

Max Act: 1.3470. Min Act: -0.0000

Data Index: 396522 (C4 (Web Text))

Max Activating Token Index: 13

Text #1

Max Range: 1.3470. Min Range: -1.3470

Max Act: 1.1540. Min Act: -0.0000

Data Index: 29994 (C4 (Web Text))

Max Activating Token Index: 85

Text #2

Max Range: 1.3470. Min Range: -1.3470

Max Act: 1.1016. Min Act: -0.0000

Data Index: 1233599 (C4 (Web Text))

Max Activating Token Index: 435

Text #3

Max Range: 1.3470. Min Range: -1.3470

Max Act: 1.0257. Min Act: -0.0000

Data Index: 173794 (C4 (Web Text))

Max Activating Token Index: 338

Text #4

Max Range: 1.3470. Min Range: -1.3470

Max Act: 0.9674. Min Act: -0.0000

Data Index: 939332 (C4 (Web Text))

Max Activating Token Index: 428

Text #5

Max Range: 1.3470. Min Range: -1.3470

Max Act: 0.9530. Min Act: -0.0000

Data Index: 1038849 (C4 (Web Text))

Max Activating Token Index: 725

Text #6

Max Range: 1.3470. Min Range: -1.3470

Max Act: 0.8890. Min Act: -0.0000

Data Index: 765809 (C4 (Web Text))

Max Activating Token Index: 153

Text #7

Max Range: 1.3470. Min Range: -1.3470

Max Act: 0.9733. Min Act: -0.0000

Data Index: 1095975 (C4 (Web Text))

Max Activating Token Index: 376

Text #8

Max Range: 1.3470. Min Range: -1.3470

Max Act: 0.9413. Min Act: -0.0000

Data Index: 1046340 (C4 (Web Text))

Max Activating Token Index: 22

Text #9

Max Range: 1.3470. Min Range: -1.3470

Max Act: 0.9349. Min Act: -0.0000

Data Index: 591993 (C4 (Web Text))

Max Activating Token Index: 370

Text #10

Max Range: 1.3470. Min Range: -1.3470

Max Act: 0.8663. Min Act: -0.0000

Data Index: 1328835 (C4 (Web Text))

Max Activating Token Index: 345

Text #11

Max Range: 1.3470. Min Range: -1.3470

Max Act: 0.9091. Min Act: -0.0000

Data Index: 918740 (C4 (Web Text))

Max Activating Token Index: 494

Text #12

Max Range: 1.3470. Min Range: -1.3470

Max Act: 0.8531. Min Act: -0.0000

Data Index: 703638 (C4 (Web Text))

Max Activating Token Index: 84

Text #13

Max Range: 1.3470. Min Range: -1.3470

Max Act: 0.8729. Min Act: -0.0000

Data Index: 1273719 (C4 (Web Text))

Max Activating Token Index: 405

Text #14

Max Range: 1.3470. Min Range: -1.3470

Max Act: 0.8658. Min Act: -0.0000

Data Index: 480773 (C4 (Web Text))

Max Activating Token Index: 922

Text #15

Max Range: 1.3470. Min Range: -1.3470

Max Act: 0.8645. Min Act: -0.0000

Data Index: 527148 (C4 (Web Text))

Max Activating Token Index: 364

Text #16

Max Range: 1.3470. Min Range: -1.3470

Max Act: 0.8120. Min Act: -0.0000

Data Index: 396693 (C4 (Web Text))

Max Activating Token Index: 1006

Text #17

Max Range: 1.3470. Min Range: -1.3470

Max Act: 0.8265. Min Act: -0.0000

Data Index: 1013581 (C4 (Web Text))

Max Activating Token Index: 467

Text #18

Max Range: 1.3470. Min Range: -1.3470

Max Act: 0.8232. Min Act: -0.0000

Data Index: 1185708 (C4 (Web Text))

Max Activating Token Index: 831

Text #19

Max Range: 1.3470. Min Range: -1.3470

Max Act: 0.8419. Min Act: -0.0000

Data Index: 477688 (C4 (Web Text))

Max Activating Token Index: 587