Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 547 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.5589. Min Range: -1.5589

Max Act: 1.4428. Min Act: -0.0001

Data Index: 1352619 (C4 (Web Text))

Max Activating Token Index: 89

Text #1

Max Range: 1.5589. Min Range: -1.5589

Max Act: 1.5497. Min Act: -0.0001

Data Index: 1148810 (C4 (Web Text))

Max Activating Token Index: 240

Text #2

Max Range: 1.5589. Min Range: -1.5589

Max Act: 1.5389. Min Act: -0.0001

Data Index: 979580 (C4 (Web Text))

Max Activating Token Index: 417

Text #3

Max Range: 1.5589. Min Range: -1.5589

Max Act: 1.5589. Min Act: -0.0001

Data Index: 1310543 (C4 (Web Text))

Max Activating Token Index: 183

Text #4

Max Range: 1.5589. Min Range: -1.5589

Max Act: 1.4399. Min Act: -0.0001

Data Index: 341290 (C4 (Web Text))

Max Activating Token Index: 774

Text #5

Max Range: 1.5589. Min Range: -1.5589

Max Act: 1.5129. Min Act: -0.0001

Data Index: 1164895 (C4 (Web Text))

Max Activating Token Index: 717

Text #6

Max Range: 1.5589. Min Range: -1.5589

Max Act: 1.3770. Min Act: -0.0001

Data Index: 685558 (C4 (Web Text))

Max Activating Token Index: 359

Text #7

Max Range: 1.5589. Min Range: -1.5589

Max Act: 1.4932. Min Act: -0.0001

Data Index: 572778 (C4 (Web Text))

Max Activating Token Index: 168

Text #8

Max Range: 1.5589. Min Range: -1.5589

Max Act: 1.4107. Min Act: -0.0000

Data Index: 1080144 (C4 (Web Text))

Max Activating Token Index: 263

Text #9

Max Range: 1.5589. Min Range: -1.5589

Max Act: 1.4039. Min Act: -0.0001

Data Index: 1058167 (C4 (Web Text))

Max Activating Token Index: 462

Text #10

Max Range: 1.5589. Min Range: -1.5589

Max Act: 1.4392. Min Act: -0.0001

Data Index: 555269 (C4 (Web Text))

Max Activating Token Index: 562

Text #11

Max Range: 1.5589. Min Range: -1.5589

Max Act: 1.4653. Min Act: -0.0001

Data Index: 1072988 (C4 (Web Text))

Max Activating Token Index: 173

Text #12

Max Range: 1.5589. Min Range: -1.5589

Max Act: 1.2367. Min Act: -0.0001

Data Index: 277468 (C4 (Web Text))

Max Activating Token Index: 884

Text #13

Max Range: 1.5589. Min Range: -1.5589

Max Act: 1.1840. Min Act: -0.0001

Data Index: 791552 (C4 (Web Text))

Max Activating Token Index: 705

Text #14

Max Range: 1.5589. Min Range: -1.5589

Max Act: 1.3190. Min Act: -0.0001

Data Index: 187616 (C4 (Web Text))

Max Activating Token Index: 788

Text #15

Max Range: 1.5589. Min Range: -1.5589

Max Act: 1.2043. Min Act: -0.0001

Data Index: 1182054 (C4 (Web Text))

Max Activating Token Index: 273

Text #16

Max Range: 1.5589. Min Range: -1.5589

Max Act: 1.2848. Min Act: -0.0001

Data Index: 454731 (C4 (Web Text))

Max Activating Token Index: 730

Text #17

Max Range: 1.5589. Min Range: -1.5589

Max Act: 1.2240. Min Act: -0.0001

Data Index: 87935 (C4 (Web Text))

Max Activating Token Index: 933

Text #18

Max Range: 1.5589. Min Range: -1.5589

Max Act: 1.2889. Min Act: -0.0001

Data Index: 1077440 (C4 (Web Text))

Max Activating Token Index: 367

Text #19

Max Range: 1.5589. Min Range: -1.5589

Max Act: 1.2591. Min Act: -0.0001

Data Index: 732185 (C4 (Web Text))

Max Activating Token Index: 533