Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4109 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.5494. Min Range: -0.5494

Max Act: 0.5494. Min Act: -0.0000

Data Index: 500310 (C4 (Web Text))

Max Activating Token Index: 705

Text #1

Max Range: 0.5494. Min Range: -0.5494

Max Act: 0.4695. Min Act: -0.0000

Data Index: 713279 (C4 (Web Text))

Max Activating Token Index: 205

Text #2

Max Range: 0.5494. Min Range: -0.5494

Max Act: 0.4977. Min Act: 0.0000

Data Index: 1195564 (C4 (Web Text))

Max Activating Token Index: 404

Text #3

Max Range: 0.5494. Min Range: -0.5494

Max Act: 0.4383. Min Act: -0.0000

Data Index: 785409 (C4 (Web Text))

Max Activating Token Index: 22

Text #4

Max Range: 0.5494. Min Range: -0.5494

Max Act: 0.4461. Min Act: 0.0000

Data Index: 116843 (C4 (Web Text))

Max Activating Token Index: 78

Text #5

Max Range: 0.5494. Min Range: -0.5494

Max Act: 0.4227. Min Act: 0.0000

Data Index: 631700 (C4 (Web Text))

Max Activating Token Index: 435

Text #6

Max Range: 0.5494. Min Range: -0.5494

Max Act: 0.4182. Min Act: -0.0000

Data Index: 1312387 (C4 (Web Text))

Max Activating Token Index: 751

Text #7

Max Range: 0.5494. Min Range: -0.5494

Max Act: 0.4298. Min Act: 0.0000

Data Index: 5782 (C4 (Web Text))

Max Activating Token Index: 445

Text #8

Max Range: 0.5494. Min Range: -0.5494

Max Act: 0.3867. Min Act: -0.0000

Data Index: 999728 (C4 (Web Text))

Max Activating Token Index: 651

Text #9

Max Range: 0.5494. Min Range: -0.5494

Max Act: 0.4107. Min Act: -0.0000

Data Index: 482700 (C4 (Web Text))

Max Activating Token Index: 55

Text #10

Max Range: 0.5494. Min Range: -0.5494

Max Act: 0.4351. Min Act: 0.0000

Data Index: 830622 (C4 (Web Text))

Max Activating Token Index: 101

Text #11

Max Range: 0.5494. Min Range: -0.5494

Max Act: 0.4274. Min Act: -0.0000

Data Index: 688375 (C4 (Web Text))

Max Activating Token Index: 230

Text #12

Max Range: 0.5494. Min Range: -0.5494

Max Act: 0.4005. Min Act: -0.0000

Data Index: 1073603 (C4 (Web Text))

Max Activating Token Index: 407

Text #13

Max Range: 0.5494. Min Range: -0.5494

Max Act: 0.4172. Min Act: -0.0000

Data Index: 1281859 (C4 (Web Text))

Max Activating Token Index: 870

Text #14

Max Range: 0.5494. Min Range: -0.5494

Max Act: 0.3924. Min Act: -0.0000

Data Index: 128309 (C4 (Web Text))

Max Activating Token Index: 26

Text #15

Max Range: 0.5494. Min Range: -0.5494

Max Act: 0.4137. Min Act: -0.0000

Data Index: 1067393 (C4 (Web Text))

Max Activating Token Index: 360

Text #16

Max Range: 0.5494. Min Range: -0.5494

Max Act: 0.3843. Min Act: -0.0000

Data Index: 530224 (C4 (Web Text))

Max Activating Token Index: 350

Text #17

Max Range: 0.5494. Min Range: -0.5494

Max Act: 0.4201. Min Act: -0.0000

Data Index: 129977 (C4 (Web Text))

Max Activating Token Index: 747

Text #18

Max Range: 0.5494. Min Range: -0.5494

Max Act: 0.3945. Min Act: -0.0000

Data Index: 894906 (C4 (Web Text))

Max Activating Token Index: 557

Text #19

Max Range: 0.5494. Min Range: -0.5494

Max Act: 0.3872. Min Act: -0.0000

Data Index: 127299 (C4 (Web Text))

Max Activating Token Index: 834