Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 696 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 0.4485. Min Range: -0.4485

Max Act: 0.4485. Min Act: -0.0001

Data Index: 1096449 (C4 (Web Text))

Max Activating Token Index: 189

Text #1

Max Range: 0.4485. Min Range: -0.4485

Max Act: 0.4227. Min Act: -0.0001

Data Index: 270837 (C4 (Web Text))

Max Activating Token Index: 690

Text #2

Max Range: 0.4485. Min Range: -0.4485

Max Act: 0.3996. Min Act: -0.0001

Data Index: 512287 (C4 (Web Text))

Max Activating Token Index: 367

Text #3

Max Range: 0.4485. Min Range: -0.4485

Max Act: 0.3451. Min Act: -0.0001

Data Index: 307656 (C4 (Web Text))

Max Activating Token Index: 682

Text #4

Max Range: 0.4485. Min Range: -0.4485

Max Act: 0.4075. Min Act: -0.0001

Data Index: 1241452 (C4 (Web Text))

Max Activating Token Index: 688

Text #5

Max Range: 0.4485. Min Range: -0.4485

Max Act: 0.3556. Min Act: -0.0001

Data Index: 110247 (C4 (Web Text))

Max Activating Token Index: 767

Text #6

Max Range: 0.4485. Min Range: -0.4485

Max Act: 0.3865. Min Act: -0.0001

Data Index: 1175476 (C4 (Web Text))

Max Activating Token Index: 231

Text #7

Max Range: 0.4485. Min Range: -0.4485

Max Act: 0.3824. Min Act: -0.0001

Data Index: 551876 (C4 (Web Text))

Max Activating Token Index: 760

Text #8

Max Range: 0.4485. Min Range: -0.4485

Max Act: 0.3254. Min Act: -0.0001

Data Index: 1242003 (C4 (Web Text))

Max Activating Token Index: 806

Text #9

Max Range: 0.4485. Min Range: -0.4485

Max Act: 0.3545. Min Act: -0.0001

Data Index: 980652 (C4 (Web Text))

Max Activating Token Index: 205

Text #10

Max Range: 0.4485. Min Range: -0.4485

Max Act: 0.3725. Min Act: -0.0001

Data Index: 820420 (C4 (Web Text))

Max Activating Token Index: 754

Text #11

Max Range: 0.4485. Min Range: -0.4485

Max Act: 0.3362. Min Act: -0.0001

Data Index: 864235 (C4 (Web Text))

Max Activating Token Index: 914

Text #12

Max Range: 0.4485. Min Range: -0.4485

Max Act: 0.3335. Min Act: -0.0001

Data Index: 169170 (C4 (Web Text))

Max Activating Token Index: 898

Text #13

Max Range: 0.4485. Min Range: -0.4485

Max Act: 0.3885. Min Act: -0.0001

Data Index: 776262 (C4 (Web Text))

Max Activating Token Index: 900

Text #14

Max Range: 0.4485. Min Range: -0.4485

Max Act: 0.3406. Min Act: -0.0001

Data Index: 226457 (C4 (Web Text))

Max Activating Token Index: 655

Text #15

Max Range: 0.4485. Min Range: -0.4485

Max Act: 0.3018. Min Act: -0.0001

Data Index: 208211 (C4 (Web Text))

Max Activating Token Index: 33

Text #16

Max Range: 0.4485. Min Range: -0.4485

Max Act: 0.3659. Min Act: -0.0001

Data Index: 905914 (C4 (Web Text))

Max Activating Token Index: 272

Text #17

Max Range: 0.4485. Min Range: -0.4485

Max Act: 0.3402. Min Act: -0.0001

Data Index: 280826 (C4 (Web Text))

Max Activating Token Index: 228

Text #18

Max Range: 0.4485. Min Range: -0.4485

Max Act: 0.3297. Min Act: -0.0001

Data Index: 1215408 (C4 (Web Text))

Max Activating Token Index: 732

Text #19

Max Range: 0.4485. Min Range: -0.4485

Max Act: 0.3334. Min Act: -0.0001

Data Index: 421086 (C4 (Web Text))

Max Activating Token Index: 669