Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2505 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.6783. Min Range: -1.6783

Max Act: 1.6783. Min Act: -0.0000

Data Index: 1141895 (C4 (Web Text))

Max Activating Token Index: 95

Text #1

Max Range: 1.6783. Min Range: -1.6783

Max Act: 1.6319. Min Act: -0.0000

Data Index: 898213 (C4 (Web Text))

Max Activating Token Index: 318

Text #2

Max Range: 1.6783. Min Range: -1.6783

Max Act: 1.6162. Min Act: 0.0000

Data Index: 1107845 (C4 (Web Text))

Max Activating Token Index: 235

Text #3

Max Range: 1.6783. Min Range: -1.6783

Max Act: 1.5727. Min Act: -0.0000

Data Index: 620458 (C4 (Web Text))

Max Activating Token Index: 474

Text #4

Max Range: 1.6783. Min Range: -1.6783

Max Act: 1.4839. Min Act: -0.0000

Data Index: 77088 (C4 (Web Text))

Max Activating Token Index: 518

Text #5

Max Range: 1.6783. Min Range: -1.6783

Max Act: 1.3896. Min Act: 0.0000

Data Index: 609940 (C4 (Web Text))

Max Activating Token Index: 71

Text #6

Max Range: 1.6783. Min Range: -1.6783

Max Act: 1.3778. Min Act: -0.0000

Data Index: 609587 (C4 (Web Text))

Max Activating Token Index: 562

Text #7

Max Range: 1.6783. Min Range: -1.6783

Max Act: 1.3768. Min Act: 0.0000

Data Index: 1319482 (C4 (Web Text))

Max Activating Token Index: 262

Text #8

Max Range: 1.6783. Min Range: -1.6783

Max Act: 1.4267. Min Act: -0.0000

Data Index: 405943 (C4 (Web Text))

Max Activating Token Index: 379

Text #9

Max Range: 1.6783. Min Range: -1.6783

Max Act: 1.2548. Min Act: -0.0000

Data Index: 675176 (C4 (Web Text))

Max Activating Token Index: 907

Text #10

Max Range: 1.6783. Min Range: -1.6783

Max Act: 1.3004. Min Act: -0.0000

Data Index: 395646 (C4 (Web Text))

Max Activating Token Index: 362

Text #11

Max Range: 1.6783. Min Range: -1.6783

Max Act: 1.2836. Min Act: -0.0000

Data Index: 1175056 (C4 (Web Text))

Max Activating Token Index: 486

Text #12

Max Range: 1.6783. Min Range: -1.6783

Max Act: 1.3730. Min Act: 0.0000

Data Index: 223321 (C4 (Web Text))

Max Activating Token Index: 805

Text #13

Max Range: 1.6783. Min Range: -1.6783

Max Act: 1.2528. Min Act: -0.0000

Data Index: 233649 (C4 (Web Text))

Max Activating Token Index: 579

Text #14

Max Range: 1.6783. Min Range: -1.6783

Max Act: 1.1667. Min Act: -0.0000

Data Index: 489250 (C4 (Web Text))

Max Activating Token Index: 1002

Text #15

Max Range: 1.6783. Min Range: -1.6783

Max Act: 1.2190. Min Act: 0.0000

Data Index: 907255 (C4 (Web Text))

Max Activating Token Index: 584

Text #16

Max Range: 1.6783. Min Range: -1.6783

Max Act: 1.0915. Min Act: -0.0000

Data Index: 900910 (C4 (Web Text))

Max Activating Token Index: 605

Text #17

Max Range: 1.6783. Min Range: -1.6783

Max Act: 1.1223. Min Act: 0.0000

Data Index: 39847 (C4 (Web Text))

Max Activating Token Index: 1021

Text #18

Max Range: 1.6783. Min Range: -1.6783

Max Act: 1.1066. Min Act: -0.0000

Data Index: 763279 (C4 (Web Text))

Max Activating Token Index: 910

Text #19

Max Range: 1.6783. Min Range: -1.6783

Max Act: 1.0908. Min Act: -0.0000

Data Index: 468867 (C4 (Web Text))

Max Activating Token Index: 788