Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2083 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 0.7826. Min Range: -0.7826

Max Act: 0.7826. Min Act: -0.0000

Data Index: 993721 (C4 (Web Text))

Max Activating Token Index: 955

Text #1

Max Range: 0.7826. Min Range: -0.7826

Max Act: 0.7790. Min Act: -0.0000

Data Index: 1096695 (C4 (Web Text))

Max Activating Token Index: 973

Text #2

Max Range: 0.7826. Min Range: -0.7826

Max Act: 0.6080. Min Act: -0.0000

Data Index: 113241 (C4 (Web Text))

Max Activating Token Index: 464

Text #3

Max Range: 0.7826. Min Range: -0.7826

Max Act: 0.5862. Min Act: -0.0000

Data Index: 934852 (C4 (Web Text))

Max Activating Token Index: 385

Text #4

Max Range: 0.7826. Min Range: -0.7826

Max Act: 0.5421. Min Act: -0.0000

Data Index: 702567 (C4 (Web Text))

Max Activating Token Index: 176

Text #5

Max Range: 0.7826. Min Range: -0.7826

Max Act: 0.5102. Min Act: -0.0000

Data Index: 1129116 (C4 (Web Text))

Max Activating Token Index: 199

Text #6

Max Range: 0.7826. Min Range: -0.7826

Max Act: 0.5064. Min Act: -0.0000

Data Index: 1168384 (C4 (Web Text))

Max Activating Token Index: 350

Text #7

Max Range: 0.7826. Min Range: -0.7826

Max Act: 0.5238. Min Act: -0.0000

Data Index: 1114136 (C4 (Web Text))

Max Activating Token Index: 695

Text #8

Max Range: 0.7826. Min Range: -0.7826

Max Act: 0.5003. Min Act: -0.0001

Data Index: 340998 (C4 (Web Text))

Max Activating Token Index: 576

Text #9

Max Range: 0.7826. Min Range: -0.7826

Max Act: 0.4925. Min Act: -0.0001

Data Index: 660053 (C4 (Web Text))

Max Activating Token Index: 758

Text #10

Max Range: 0.7826. Min Range: -0.7826

Max Act: 0.4848. Min Act: -0.0000

Data Index: 390512 (C4 (Web Text))

Max Activating Token Index: 907

Text #11

Max Range: 0.7826. Min Range: -0.7826

Max Act: 0.5150. Min Act: -0.0000

Data Index: 30481 (C4 (Web Text))

Max Activating Token Index: 20

Text #12

Max Range: 0.7826. Min Range: -0.7826

Max Act: 0.4768. Min Act: -0.0000

Data Index: 632659 (C4 (Web Text))

Max Activating Token Index: 652

Text #13

Max Range: 0.7826. Min Range: -0.7826

Max Act: 0.4914. Min Act: 0.0001

Data Index: 987907 (C4 (Web Text))

Max Activating Token Index: 947

Text #14

Max Range: 0.7826. Min Range: -0.7826

Max Act: 0.4754. Min Act: -0.0001

Data Index: 340327 (C4 (Web Text))

Max Activating Token Index: 339

Text #15

Max Range: 0.7826. Min Range: -0.7826

Max Act: 0.4718. Min Act: -0.0000

Data Index: 1060152 (C4 (Web Text))

Max Activating Token Index: 13

Text #16

Max Range: 0.7826. Min Range: -0.7826

Max Act: 0.4795. Min Act: -0.0000

Data Index: 919948 (C4 (Web Text))

Max Activating Token Index: 858

Text #17

Max Range: 0.7826. Min Range: -0.7826

Max Act: 0.4584. Min Act: -0.0000

Data Index: 183071 (C4 (Web Text))

Max Activating Token Index: 875

Text #18

Max Range: 0.7826. Min Range: -0.7826

Max Act: 0.4383. Min Act: -0.0000

Data Index: 207741 (C4 (Web Text))

Max Activating Token Index: 644

Text #19

Max Range: 0.7826. Min Range: -0.7826

Max Act: 0.4524. Min Act: -0.0000

Data Index: 940755 (C4 (Web Text))

Max Activating Token Index: 213