Model: SoLU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1113 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-4l')

Text #0

Max Range: 0.0941. Min Range: -0.0941

Max Act: 0.0925. Min Act: -0.0002

Data Index: 642935 (C4 (Web Text))

Max Activating Token Index: 935

Text #1

Max Range: 0.0941. Min Range: -0.0941

Max Act: 0.0941. Min Act: -0.0002

Data Index: 386828 (C4 (Web Text))

Max Activating Token Index: 346

Text #2

Max Range: 0.0941. Min Range: -0.0941

Max Act: 0.0930. Min Act: -0.0002

Data Index: 526066 (C4 (Web Text))

Max Activating Token Index: 507

Text #3

Max Range: 0.0941. Min Range: -0.0941

Max Act: 0.0920. Min Act: -0.0002

Data Index: 559456 (C4 (Web Text))

Max Activating Token Index: 893

Text #4

Max Range: 0.0941. Min Range: -0.0941

Max Act: 0.0898. Min Act: -0.0002

Data Index: 1112258 (C4 (Web Text))

Max Activating Token Index: 246

Text #5

Max Range: 0.0941. Min Range: -0.0941

Max Act: 0.0897. Min Act: -0.0002

Data Index: 951522 (C4 (Web Text))

Max Activating Token Index: 682

Text #6

Max Range: 0.0941. Min Range: -0.0941

Max Act: 0.0870. Min Act: -0.0002

Data Index: 1096848 (C4 (Web Text))

Max Activating Token Index: 1001

Text #7

Max Range: 0.0941. Min Range: -0.0941

Max Act: 0.0908. Min Act: -0.0002

Data Index: 996356 (C4 (Web Text))

Max Activating Token Index: 543

Text #8

Max Range: 0.0941. Min Range: -0.0941

Max Act: 0.0855. Min Act: -0.0002

Data Index: 1177431 (C4 (Web Text))

Max Activating Token Index: 707

Text #9

Max Range: 0.0941. Min Range: -0.0941

Max Act: 0.0862. Min Act: -0.0002

Data Index: 120115 (C4 (Web Text))

Max Activating Token Index: 400

Text #10

Max Range: 0.0941. Min Range: -0.0941

Max Act: 0.0859. Min Act: -0.0002

Data Index: 325837 (C4 (Web Text))

Max Activating Token Index: 415

Text #11

Max Range: 0.0941. Min Range: -0.0941

Max Act: 0.0829. Min Act: -0.0002

Data Index: 631252 (C4 (Web Text))

Max Activating Token Index: 887

Text #12

Max Range: 0.0941. Min Range: -0.0941

Max Act: 0.0852. Min Act: -0.0002

Data Index: 1031641 (C4 (Web Text))

Max Activating Token Index: 174

Text #13

Max Range: 0.0941. Min Range: -0.0941

Max Act: 0.0790. Min Act: -0.0002

Data Index: 561672 (C4 (Web Text))

Max Activating Token Index: 984

Text #14

Max Range: 0.0941. Min Range: -0.0941

Max Act: 0.0832. Min Act: -0.0002

Data Index: 17271 (C4 (Web Text))

Max Activating Token Index: 329

Text #15

Max Range: 0.0941. Min Range: -0.0941

Max Act: 0.0824. Min Act: -0.0002

Data Index: 487321 (C4 (Web Text))

Max Activating Token Index: 611

Text #16

Max Range: 0.0941. Min Range: -0.0941

Max Act: 0.0846. Min Act: -0.0002

Data Index: 327811 (C4 (Web Text))

Max Activating Token Index: 687

Text #17

Max Range: 0.0941. Min Range: -0.0941

Max Act: 0.0807. Min Act: -0.0002

Data Index: 328882 (C4 (Web Text))

Max Activating Token Index: 142

Text #18

Max Range: 0.0941. Min Range: -0.0941

Max Act: 0.0819. Min Act: -0.0002

Data Index: 701199 (C4 (Web Text))

Max Activating Token Index: 432

Text #19

Max Range: 0.0941. Min Range: -0.0941

Max Act: 0.0785. Min Act: -0.0002

Data Index: 481943 (C4 (Web Text))

Max Activating Token Index: 130