Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2804 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.4843. Min Range: -0.4843

Max Act: 0.4843. Min Act: -0.0000

Data Index: 921986 (C4 (Web Text))

Max Activating Token Index: 866

Text #1

Max Range: 0.4843. Min Range: -0.4843

Max Act: 0.4729. Min Act: -0.0000

Data Index: 102536 (C4 (Web Text))

Max Activating Token Index: 914

Text #2

Max Range: 0.4843. Min Range: -0.4843

Max Act: 0.4726. Min Act: -0.0000

Data Index: 779600 (C4 (Web Text))

Max Activating Token Index: 425

Text #3

Max Range: 0.4843. Min Range: -0.4843

Max Act: 0.4574. Min Act: -0.0000

Data Index: 89817 (C4 (Web Text))

Max Activating Token Index: 933

Text #4

Max Range: 0.4843. Min Range: -0.4843

Max Act: 0.4413. Min Act: -0.0000

Data Index: 1048198 (C4 (Web Text))

Max Activating Token Index: 403

Text #5

Max Range: 0.4843. Min Range: -0.4843

Max Act: 0.4495. Min Act: -0.0000

Data Index: 99234 (C4 (Web Text))

Max Activating Token Index: 61

Text #6

Max Range: 0.4843. Min Range: -0.4843

Max Act: 0.4267. Min Act: -0.0000

Data Index: 574631 (C4 (Web Text))

Max Activating Token Index: 594

Text #7

Max Range: 0.4843. Min Range: -0.4843

Max Act: 0.4036. Min Act: -0.0000

Data Index: 1224471 (C4 (Web Text))

Max Activating Token Index: 278

Text #8

Max Range: 0.4843. Min Range: -0.4843

Max Act: 0.4403. Min Act: -0.0000

Data Index: 775780 (C4 (Web Text))

Max Activating Token Index: 291

Text #9

Max Range: 0.4843. Min Range: -0.4843

Max Act: 0.4012. Min Act: -0.0000

Data Index: 313440 (C4 (Web Text))

Max Activating Token Index: 538

Text #10

Max Range: 0.4843. Min Range: -0.4843

Max Act: 0.4255. Min Act: -0.0000

Data Index: 1226879 (C4 (Web Text))

Max Activating Token Index: 283

Text #11

Max Range: 0.4843. Min Range: -0.4843

Max Act: 0.3937. Min Act: -0.0000

Data Index: 49554 (C4 (Web Text))

Max Activating Token Index: 697

Text #12

Max Range: 0.4843. Min Range: -0.4843

Max Act: 0.3817. Min Act: -0.0001

Data Index: 93326 (C4 (Web Text))

Max Activating Token Index: 714

Text #13

Max Range: 0.4843. Min Range: -0.4843

Max Act: 0.3890. Min Act: -0.0000

Data Index: 1039978 (C4 (Web Text))

Max Activating Token Index: 85

Text #14

Max Range: 0.4843. Min Range: -0.4843

Max Act: 0.3902. Min Act: -0.0000

Data Index: 371310 (C4 (Web Text))

Max Activating Token Index: 561

Text #15

Max Range: 0.4843. Min Range: -0.4843

Max Act: 0.3737. Min Act: -0.0000

Data Index: 224588 (C4 (Web Text))

Max Activating Token Index: 972

Text #16

Max Range: 0.4843. Min Range: -0.4843

Max Act: 0.3729. Min Act: -0.0000

Data Index: 747768 (C4 (Web Text))

Max Activating Token Index: 317

Text #17

Max Range: 0.4843. Min Range: -0.4843

Max Act: 0.3840. Min Act: -0.0000

Data Index: 69716 (C4 (Web Text))

Max Activating Token Index: 541

Text #18

Max Range: 0.4843. Min Range: -0.4843

Max Act: 0.3758. Min Act: -0.0000

Data Index: 1150996 (C4 (Web Text))

Max Activating Token Index: 350

Text #19

Max Range: 0.4843. Min Range: -0.4843

Max Act: 0.3889. Min Act: -0.0000

Data Index: 1097550 (C4 (Web Text))

Max Activating Token Index: 403