Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1696 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.3934. Min Range: -0.3934

Max Act: 0.3934. Min Act: -0.0002

Data Index: 26394 (C4 (Web Text))

Max Activating Token Index: 168

Text #1

Max Range: 0.3934. Min Range: -0.3934

Max Act: 0.3624. Min Act: -0.0002

Data Index: 1193065 (C4 (Web Text))

Max Activating Token Index: 821

Text #2

Max Range: 0.3934. Min Range: -0.3934

Max Act: 0.3635. Min Act: -0.0002

Data Index: 1046333 (C4 (Web Text))

Max Activating Token Index: 113

Text #3

Max Range: 0.3934. Min Range: -0.3934

Max Act: 0.3449. Min Act: -0.0002

Data Index: 364802 (C4 (Web Text))

Max Activating Token Index: 514

Text #4

Max Range: 0.3934. Min Range: -0.3934

Max Act: 0.3411. Min Act: -0.0002

Data Index: 332277 (C4 (Web Text))

Max Activating Token Index: 845

Text #5

Max Range: 0.3934. Min Range: -0.3934

Max Act: 0.3644. Min Act: -0.0002

Data Index: 1077111 (C4 (Web Text))

Max Activating Token Index: 788

Text #6

Max Range: 0.3934. Min Range: -0.3934

Max Act: 0.3381. Min Act: -0.0002

Data Index: 558341 (C4 (Web Text))

Max Activating Token Index: 972

Text #7

Max Range: 0.3934. Min Range: -0.3934

Max Act: 0.3079. Min Act: -0.0002

Data Index: 421458 (C4 (Web Text))

Max Activating Token Index: 244

Text #8

Max Range: 0.3934. Min Range: -0.3934

Max Act: 0.3496. Min Act: -0.0002

Data Index: 649226 (C4 (Web Text))

Max Activating Token Index: 885

Text #9

Max Range: 0.3934. Min Range: -0.3934

Max Act: 0.3323. Min Act: -0.0002

Data Index: 1001520 (C4 (Web Text))

Max Activating Token Index: 799

Text #10

Max Range: 0.3934. Min Range: -0.3934

Max Act: 0.3340. Min Act: -0.0002

Data Index: 928211 (C4 (Web Text))

Max Activating Token Index: 837

Text #11

Max Range: 0.3934. Min Range: -0.3934

Max Act: 0.3187. Min Act: -0.0002

Data Index: 1220877 (C4 (Web Text))

Max Activating Token Index: 62

Text #12

Max Range: 0.3934. Min Range: -0.3934

Max Act: 0.3167. Min Act: -0.0002

Data Index: 270102 (C4 (Web Text))

Max Activating Token Index: 965

Text #13

Max Range: 0.3934. Min Range: -0.3934

Max Act: 0.3040. Min Act: -0.0002

Data Index: 810932 (C4 (Web Text))

Max Activating Token Index: 561

Text #14

Max Range: 0.3934. Min Range: -0.3934

Max Act: 0.3287. Min Act: -0.0002

Data Index: 44497 (C4 (Web Text))

Max Activating Token Index: 948

Text #15

Max Range: 0.3934. Min Range: -0.3934

Max Act: 0.2958. Min Act: -0.0002

Data Index: 716641 (C4 (Web Text))

Max Activating Token Index: 942

Text #16

Max Range: 0.3934. Min Range: -0.3934

Max Act: 0.3097. Min Act: -0.0002

Data Index: 57668 (C4 (Web Text))

Max Activating Token Index: 236

Text #17

Max Range: 0.3934. Min Range: -0.3934

Max Act: 0.3167. Min Act: -0.0002

Data Index: 1042842 (C4 (Web Text))

Max Activating Token Index: 206

Text #18

Max Range: 0.3934. Min Range: -0.3934

Max Act: 0.3043. Min Act: -0.0002

Data Index: 1165379 (C4 (Web Text))

Max Activating Token Index: 1004

Text #19

Max Range: 0.3934. Min Range: -0.3934

Max Act: 0.3048. Min Act: -0.0002

Data Index: 1063364 (C4 (Web Text))

Max Activating Token Index: 130