Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 768 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 0.3212. Min Range: -0.3212

Max Act: 0.3212. Min Act: -0.0001

Data Index: 1151302 (C4 (Web Text))

Max Activating Token Index: 973

Text #1

Max Range: 0.3212. Min Range: -0.3212

Max Act: 0.3194. Min Act: -0.0001

Data Index: 722306 (C4 (Web Text))

Max Activating Token Index: 1007

Text #2

Max Range: 0.3212. Min Range: -0.3212

Max Act: 0.3021. Min Act: -0.0001

Data Index: 15046 (C4 (Web Text))

Max Activating Token Index: 443

Text #3

Max Range: 0.3212. Min Range: -0.3212

Max Act: 0.3091. Min Act: -0.0001

Data Index: 996121 (C4 (Web Text))

Max Activating Token Index: 963

Text #4

Max Range: 0.3212. Min Range: -0.3212

Max Act: 0.2678. Min Act: -0.0001

Data Index: 576029 (C4 (Web Text))

Max Activating Token Index: 870

Text #5

Max Range: 0.3212. Min Range: -0.3212

Max Act: 0.2913. Min Act: -0.0001

Data Index: 1277596 (C4 (Web Text))

Max Activating Token Index: 344

Text #6

Max Range: 0.3212. Min Range: -0.3212

Max Act: 0.2932. Min Act: -0.0001

Data Index: 932444 (C4 (Web Text))

Max Activating Token Index: 285

Text #7

Max Range: 0.3212. Min Range: -0.3212

Max Act: 0.2683. Min Act: -0.0001

Data Index: 379404 (C4 (Web Text))

Max Activating Token Index: 1006

Text #8

Max Range: 0.3212. Min Range: -0.3212

Max Act: 0.2689. Min Act: -0.0001

Data Index: 1346802 (C4 (Web Text))

Max Activating Token Index: 301

Text #9

Max Range: 0.3212. Min Range: -0.3212

Max Act: 0.2705. Min Act: -0.0001

Data Index: 533212 (C4 (Web Text))

Max Activating Token Index: 185

Text #10

Max Range: 0.3212. Min Range: -0.3212

Max Act: 0.2492. Min Act: -0.0001

Data Index: 110393 (C4 (Web Text))

Max Activating Token Index: 88

Text #11

Max Range: 0.3212. Min Range: -0.3212

Max Act: 0.2635. Min Act: -0.0001

Data Index: 1231962 (C4 (Web Text))

Max Activating Token Index: 905

Text #12

Max Range: 0.3212. Min Range: -0.3212

Max Act: 0.2454. Min Act: -0.0001

Data Index: 805470 (C4 (Web Text))

Max Activating Token Index: 935

Text #13

Max Range: 0.3212. Min Range: -0.3212

Max Act: 0.2706. Min Act: -0.0001

Data Index: 329182 (C4 (Web Text))

Max Activating Token Index: 621

Text #14

Max Range: 0.3212. Min Range: -0.3212

Max Act: 0.2491. Min Act: -0.0001

Data Index: 1070352 (C4 (Web Text))

Max Activating Token Index: 1014

Text #15

Max Range: 0.3212. Min Range: -0.3212

Max Act: 0.2558. Min Act: -0.0001

Data Index: 137821 (C4 (Web Text))

Max Activating Token Index: 672

Text #16

Max Range: 0.3212. Min Range: -0.3212

Max Act: 0.2427. Min Act: -0.0001

Data Index: 1291213 (C4 (Web Text))

Max Activating Token Index: 495

Text #17

Max Range: 0.3212. Min Range: -0.3212

Max Act: 0.2410. Min Act: -0.0001

Data Index: 525679 (C4 (Web Text))

Max Activating Token Index: 82

Text #18

Max Range: 0.3212. Min Range: -0.3212

Max Act: 0.2493. Min Act: -0.0001

Data Index: 481795 (C4 (Web Text))

Max Activating Token Index: 549

Text #19

Max Range: 0.3212. Min Range: -0.3212

Max Act: 0.2568. Min Act: -0.0001

Data Index: 229773 (C4 (Web Text))

Max Activating Token Index: 806