Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2462 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.6484. Min Range: -0.6484

Max Act: 0.6484. Min Act: -0.0000

Data Index: 1350750 (C4 (Web Text))

Max Activating Token Index: 392

Text #1

Max Range: 0.6484. Min Range: -0.6484

Max Act: 0.6155. Min Act: -0.0000

Data Index: 217482 (C4 (Web Text))

Max Activating Token Index: 276

Text #2

Max Range: 0.6484. Min Range: -0.6484

Max Act: 0.6055. Min Act: -0.0000

Data Index: 521725 (C4 (Web Text))

Max Activating Token Index: 80

Text #3

Max Range: 0.6484. Min Range: -0.6484

Max Act: 0.6066. Min Act: -0.0000

Data Index: 67456 (C4 (Web Text))

Max Activating Token Index: 555

Text #4

Max Range: 0.6484. Min Range: -0.6484

Max Act: 0.5552. Min Act: -0.0000

Data Index: 819162 (C4 (Web Text))

Max Activating Token Index: 545

Text #5

Max Range: 0.6484. Min Range: -0.6484

Max Act: 0.5085. Min Act: -0.0000

Data Index: 62105 (C4 (Web Text))

Max Activating Token Index: 826

Text #6

Max Range: 0.6484. Min Range: -0.6484

Max Act: 0.4716. Min Act: -0.0000

Data Index: 571292 (C4 (Web Text))

Max Activating Token Index: 563

Text #7

Max Range: 0.6484. Min Range: -0.6484

Max Act: 0.4628. Min Act: -0.0000

Data Index: 526743 (C4 (Web Text))

Max Activating Token Index: 276

Text #8

Max Range: 0.6484. Min Range: -0.6484

Max Act: 0.4431. Min Act: -0.0000

Data Index: 518899 (C4 (Web Text))

Max Activating Token Index: 38

Text #9

Max Range: 0.6484. Min Range: -0.6484

Max Act: 0.4217. Min Act: -0.0000

Data Index: 1256257 (C4 (Web Text))

Max Activating Token Index: 914

Text #10

Max Range: 0.6484. Min Range: -0.6484

Max Act: 0.4367. Min Act: -0.0000

Data Index: 318744 (C4 (Web Text))

Max Activating Token Index: 919

Text #11

Max Range: 0.6484. Min Range: -0.6484

Max Act: 0.4602. Min Act: -0.0000

Data Index: 511049 (C4 (Web Text))

Max Activating Token Index: 57

Text #12

Max Range: 0.6484. Min Range: -0.6484

Max Act: 0.4168. Min Act: -0.0000

Data Index: 1003383 (C4 (Web Text))

Max Activating Token Index: 780

Text #13

Max Range: 0.6484. Min Range: -0.6484

Max Act: 0.3828. Min Act: -0.0000

Data Index: 497468 (C4 (Web Text))

Max Activating Token Index: 347

Text #14

Max Range: 0.6484. Min Range: -0.6484

Max Act: 0.3718. Min Act: -0.0000

Data Index: 1101017 (C4 (Web Text))

Max Activating Token Index: 647

Text #15

Max Range: 0.6484. Min Range: -0.6484

Max Act: 0.4009. Min Act: -0.0000

Data Index: 724304 (C4 (Web Text))

Max Activating Token Index: 113

Text #16

Max Range: 0.6484. Min Range: -0.6484

Max Act: 0.3908. Min Act: -0.0000

Data Index: 1045590 (C4 (Web Text))

Max Activating Token Index: 687

Text #17

Max Range: 0.6484. Min Range: -0.6484

Max Act: 0.3781. Min Act: -0.0000

Data Index: 816184 (C4 (Web Text))

Max Activating Token Index: 143

Text #18

Max Range: 0.6484. Min Range: -0.6484

Max Act: 0.4289. Min Act: -0.0000

Data Index: 805926 (C4 (Web Text))

Max Activating Token Index: 375

Text #19

Max Range: 0.6484. Min Range: -0.6484

Max Act: 0.3886. Min Act: -0.0000

Data Index: 474944 (C4 (Web Text))

Max Activating Token Index: 445