Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1373 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 1.2613. Min Range: -1.2613

Max Act: 1.2613. Min Act: -0.0001

Data Index: 42029 (C4 (Web Text))

Max Activating Token Index: 504

Text #1

Max Range: 1.2613. Min Range: -1.2613

Max Act: 1.1217. Min Act: -0.0001

Data Index: 1168480 (C4 (Web Text))

Max Activating Token Index: 17

Text #2

Max Range: 1.2613. Min Range: -1.2613

Max Act: 1.0606. Min Act: -0.0001

Data Index: 572629 (C4 (Web Text))

Max Activating Token Index: 111

Text #3

Max Range: 1.2613. Min Range: -1.2613

Max Act: 1.0650. Min Act: -0.0001

Data Index: 1254365 (C4 (Web Text))

Max Activating Token Index: 23

Text #4

Max Range: 1.2613. Min Range: -1.2613

Max Act: 0.9270. Min Act: -0.0001

Data Index: 96545 (C4 (Web Text))

Max Activating Token Index: 359

Text #5

Max Range: 1.2613. Min Range: -1.2613

Max Act: 1.1043. Min Act: -0.0001

Data Index: 232700 (C4 (Web Text))

Max Activating Token Index: 674

Text #6

Max Range: 1.2613. Min Range: -1.2613

Max Act: 0.9235. Min Act: -0.0001

Data Index: 717376 (C4 (Web Text))

Max Activating Token Index: 769

Text #7

Max Range: 1.2613. Min Range: -1.2613

Max Act: 0.8602. Min Act: -0.0001

Data Index: 762628 (C4 (Web Text))

Max Activating Token Index: 680

Text #8

Max Range: 1.2613. Min Range: -1.2613

Max Act: 0.9654. Min Act: -0.0001

Data Index: 676614 (C4 (Web Text))

Max Activating Token Index: 95

Text #9

Max Range: 1.2613. Min Range: -1.2613

Max Act: 0.9596. Min Act: -0.0001

Data Index: 419605 (C4 (Web Text))

Max Activating Token Index: 46

Text #10

Max Range: 1.2613. Min Range: -1.2613

Max Act: 0.9184. Min Act: -0.0001

Data Index: 785316 (C4 (Web Text))

Max Activating Token Index: 369

Text #11

Max Range: 1.2613. Min Range: -1.2613

Max Act: 0.9400. Min Act: -0.0001

Data Index: 662580 (C4 (Web Text))

Max Activating Token Index: 248

Text #12

Max Range: 1.2613. Min Range: -1.2613

Max Act: 0.9179. Min Act: -0.0001

Data Index: 288132 (C4 (Web Text))

Max Activating Token Index: 144

Text #13

Max Range: 1.2613. Min Range: -1.2613

Max Act: 0.7687. Min Act: -0.0001

Data Index: 314594 (C4 (Web Text))

Max Activating Token Index: 667

Text #14

Max Range: 1.2613. Min Range: -1.2613

Max Act: 0.8980. Min Act: -0.0001

Data Index: 1051804 (C4 (Web Text))

Max Activating Token Index: 163

Text #15

Max Range: 1.2613. Min Range: -1.2613

Max Act: 0.8553. Min Act: -0.0001

Data Index: 1234473 (C4 (Web Text))

Max Activating Token Index: 148

Text #16

Max Range: 1.2613. Min Range: -1.2613

Max Act: 0.9323. Min Act: -0.0001

Data Index: 303733 (C4 (Web Text))

Max Activating Token Index: 151

Text #17

Max Range: 1.2613. Min Range: -1.2613

Max Act: 0.8251. Min Act: -0.0001

Data Index: 475883 (C4 (Web Text))

Max Activating Token Index: 147

Text #18

Max Range: 1.2613. Min Range: -1.2613

Max Act: 0.8540. Min Act: -0.0001

Data Index: 1129337 (C4 (Web Text))

Max Activating Token Index: 556

Text #19

Max Range: 1.2613. Min Range: -1.2613

Max Act: 0.7940. Min Act: -0.0001

Data Index: 2635 (C4 (Web Text))

Max Activating Token Index: 713