Model: SoLU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 679 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-4l')

Text #0

Max Range: 1.2681. Min Range: -1.2681

Max Act: 1.2681. Min Act: -0.0005

Data Index: 1098414 (C4 (Web Text))

Max Activating Token Index: 855

Text #1

Max Range: 1.2681. Min Range: -1.2681

Max Act: 1.0796. Min Act: -0.0005

Data Index: 327871 (C4 (Web Text))

Max Activating Token Index: 992

Text #2

Max Range: 1.2681. Min Range: -1.2681

Max Act: 0.9502. Min Act: -0.0005

Data Index: 887999 (C4 (Web Text))

Max Activating Token Index: 933

Text #3

Max Range: 1.2681. Min Range: -1.2681

Max Act: 0.8515. Min Act: -0.0005

Data Index: 135878 (C4 (Web Text))

Max Activating Token Index: 874

Text #4

Max Range: 1.2681. Min Range: -1.2681

Max Act: 0.9527. Min Act: -0.0005

Data Index: 215983 (C4 (Web Text))

Max Activating Token Index: 268

Text #5

Max Range: 1.2681. Min Range: -1.2681

Max Act: 1.0337. Min Act: -0.0005

Data Index: 833399 (C4 (Web Text))

Max Activating Token Index: 866

Text #6

Max Range: 1.2681. Min Range: -1.2681

Max Act: 0.8408. Min Act: -0.0005

Data Index: 609748 (C4 (Web Text))

Max Activating Token Index: 509

Text #7

Max Range: 1.2681. Min Range: -1.2681

Max Act: 0.8579. Min Act: -0.0005

Data Index: 278916 (C4 (Web Text))

Max Activating Token Index: 107

Text #8

Max Range: 1.2681. Min Range: -1.2681

Max Act: 0.8556. Min Act: -0.0005

Data Index: 1225726 (C4 (Web Text))

Max Activating Token Index: 729

Text #9

Max Range: 1.2681. Min Range: -1.2681

Max Act: 0.8519. Min Act: -0.0005

Data Index: 334632 (C4 (Web Text))

Max Activating Token Index: 794

Text #10

Max Range: 1.2681. Min Range: -1.2681

Max Act: 0.7937. Min Act: -0.0005

Data Index: 1056123 (C4 (Web Text))

Max Activating Token Index: 702

Text #11

Max Range: 1.2681. Min Range: -1.2681

Max Act: 0.8473. Min Act: -0.0005

Data Index: 174045 (C4 (Web Text))

Max Activating Token Index: 725

Text #12

Max Range: 1.2681. Min Range: -1.2681

Max Act: 0.7934. Min Act: -0.0005

Data Index: 1269685 (C4 (Web Text))

Max Activating Token Index: 846

Text #13

Max Range: 1.2681. Min Range: -1.2681

Max Act: 0.8373. Min Act: -0.0005

Data Index: 582719 (C4 (Web Text))

Max Activating Token Index: 73

Text #14

Max Range: 1.2681. Min Range: -1.2681

Max Act: 0.7930. Min Act: -0.0005

Data Index: 1333748 (C4 (Web Text))

Max Activating Token Index: 953

Text #15

Max Range: 1.2681. Min Range: -1.2681

Max Act: 0.8179. Min Act: -0.0005

Data Index: 1225779 (C4 (Web Text))

Max Activating Token Index: 427

Text #16

Max Range: 1.2681. Min Range: -1.2681

Max Act: 0.8229. Min Act: -0.0005

Data Index: 741386 (C4 (Web Text))

Max Activating Token Index: 22

Text #17

Max Range: 1.2681. Min Range: -1.2681

Max Act: 0.8080. Min Act: -0.0005

Data Index: 1248509 (C4 (Web Text))

Max Activating Token Index: 262

Text #18

Max Range: 1.2681. Min Range: -1.2681

Max Act: 0.8096. Min Act: -0.0005

Data Index: 551448 (C4 (Web Text))

Max Activating Token Index: 561

Text #19

Max Range: 1.2681. Min Range: -1.2681

Max Act: 0.8118. Min Act: -0.0005

Data Index: 980591 (C4 (Web Text))

Max Activating Token Index: 216