Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 802 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('solu-12l')

Click the toggle to see the full text!

Text #0

Max Range: 0.0447. Min Range: -0.0447

Max Act: 0.0447. Min Act: -0.0001

Data Index: 113735 (C4 (Web Text))

Max Activating Token Index: 335

Text #1

Max Range: 0.0447. Min Range: -0.0447

Max Act: 0.0434. Min Act: -0.0001

Data Index: 337274 (C4 (Web Text))

Max Activating Token Index: 71

Text #2

Max Range: 0.0447. Min Range: -0.0447

Max Act: 0.0430. Min Act: -0.0001

Data Index: 1303518 (C4 (Web Text))

Max Activating Token Index: 1023

Text #3

Max Range: 0.0447. Min Range: -0.0447

Max Act: 0.0428. Min Act: -0.0001

Data Index: 899718 (C4 (Web Text))

Max Activating Token Index: 95

Text #4

Max Range: 0.0447. Min Range: -0.0447

Max Act: 0.0418. Min Act: -0.0001

Data Index: 124953 (C4 (Web Text))

Max Activating Token Index: 1023

Text #5

Max Range: 0.0447. Min Range: -0.0447

Max Act: 0.0412. Min Act: -0.0001

Data Index: 764041 (C4 (Web Text))

Max Activating Token Index: 1023

Text #6

Max Range: 0.0447. Min Range: -0.0447

Max Act: 0.0410. Min Act: -0.0001

Data Index: 1117640 (C4 (Web Text))

Max Activating Token Index: 415

Text #7

Max Range: 0.0447. Min Range: -0.0447

Max Act: 0.0413. Min Act: -0.0001

Data Index: 911144 (C4 (Web Text))

Max Activating Token Index: 337

Text #8

Max Range: 0.0447. Min Range: -0.0447

Max Act: 0.0410. Min Act: -0.0001

Data Index: 397969 (C4 (Web Text))

Max Activating Token Index: 648

Text #9

Max Range: 0.0447. Min Range: -0.0447

Max Act: 0.0410. Min Act: -0.0001

Data Index: 447201 (C4 (Web Text))

Max Activating Token Index: 349

Text #10

Max Range: 0.0447. Min Range: -0.0447

Max Act: 0.0409. Min Act: -0.0001

Data Index: 717684 (C4 (Web Text))

Max Activating Token Index: 812

Text #11

Max Range: 0.0447. Min Range: -0.0447

Max Act: 0.0409. Min Act: -0.0001

Data Index: 256990 (C4 (Web Text))

Max Activating Token Index: 157

Text #12

Max Range: 0.0447. Min Range: -0.0447

Max Act: 0.0399. Min Act: -0.0001

Data Index: 393710 (C4 (Web Text))

Max Activating Token Index: 458

Text #13

Max Range: 0.0447. Min Range: -0.0447

Max Act: 0.0396. Min Act: -0.0001

Data Index: 655852 (C4 (Web Text))

Max Activating Token Index: 215

Text #14

Max Range: 0.0447. Min Range: -0.0447

Max Act: 0.0403. Min Act: -0.0001

Data Index: 403221 (C4 (Web Text))

Max Activating Token Index: 621

Text #15

Max Range: 0.0447. Min Range: -0.0447

Max Act: 0.0401. Min Act: -0.0001

Data Index: 144755 (C4 (Web Text))

Max Activating Token Index: 370

Text #16

Max Range: 0.0447. Min Range: -0.0447

Max Act: 0.0396. Min Act: -0.0001

Data Index: 1112414 (C4 (Web Text))

Max Activating Token Index: 827

Text #17

Max Range: 0.0447. Min Range: -0.0447

Max Act: 0.0393. Min Act: -0.0001

Data Index: 161778 (C4 (Web Text))

Max Activating Token Index: 927

Text #18

Max Range: 0.0447. Min Range: -0.0447

Max Act: 0.0392. Min Act: -0.0001

Data Index: 1337543 (C4 (Web Text))

Max Activating Token Index: 661

Text #19

Max Range: 0.0447. Min Range: -0.0447

Max Act: 0.0388. Min Act: -0.0001

Data Index: 1259324 (C4 (Web Text))

Max Activating Token Index: 554