Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3243 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.0379. Min Range: -1.0379

Max Act: 0.9802. Min Act: -0.0001

Data Index: 818061 (C4 (Web Text))

Max Activating Token Index: 778

Text #1

Max Range: 1.0379. Min Range: -1.0379

Max Act: 0.9580. Min Act: -0.0001

Data Index: 310295 (C4 (Web Text))

Max Activating Token Index: 228

Text #2

Max Range: 1.0379. Min Range: -1.0379

Max Act: 1.0379. Min Act: -0.0001

Data Index: 710699 (C4 (Web Text))

Max Activating Token Index: 451

Text #3

Max Range: 1.0379. Min Range: -1.0379

Max Act: 0.9755. Min Act: -0.0001

Data Index: 833421 (C4 (Web Text))

Max Activating Token Index: 928

Text #4

Max Range: 1.0379. Min Range: -1.0379

Max Act: 0.9457. Min Act: -0.0001

Data Index: 358821 (C4 (Web Text))

Max Activating Token Index: 591

Text #5

Max Range: 1.0379. Min Range: -1.0379

Max Act: 0.9809. Min Act: -0.0001

Data Index: 1102061 (C4 (Web Text))

Max Activating Token Index: 19

Text #6

Max Range: 1.0379. Min Range: -1.0379

Max Act: 0.9327. Min Act: -0.0001

Data Index: 591950 (C4 (Web Text))

Max Activating Token Index: 102

Text #7

Max Range: 1.0379. Min Range: -1.0379

Max Act: 0.9080. Min Act: -0.0001

Data Index: 725683 (C4 (Web Text))

Max Activating Token Index: 149

Text #8

Max Range: 1.0379. Min Range: -1.0379

Max Act: 0.9613. Min Act: -0.0001

Data Index: 614666 (C4 (Web Text))

Max Activating Token Index: 10

Text #9

Max Range: 1.0379. Min Range: -1.0379

Max Act: 0.9707. Min Act: -0.0001

Data Index: 915215 (C4 (Web Text))

Max Activating Token Index: 805

Text #10

Max Range: 1.0379. Min Range: -1.0379

Max Act: 0.9430. Min Act: -0.0001

Data Index: 457997 (C4 (Web Text))

Max Activating Token Index: 27

Text #11

Max Range: 1.0379. Min Range: -1.0379

Max Act: 0.8609. Min Act: -0.0001

Data Index: 59837 (C4 (Web Text))

Max Activating Token Index: 710

Text #12

Max Range: 1.0379. Min Range: -1.0379

Max Act: 0.9254. Min Act: -0.0001

Data Index: 1320418 (C4 (Web Text))

Max Activating Token Index: 668

Text #13

Max Range: 1.0379. Min Range: -1.0379

Max Act: 0.8953. Min Act: -0.0001

Data Index: 781772 (C4 (Web Text))

Max Activating Token Index: 89

Text #14

Max Range: 1.0379. Min Range: -1.0379

Max Act: 0.9236. Min Act: -0.0001

Data Index: 417173 (C4 (Web Text))

Max Activating Token Index: 542

Text #15

Max Range: 1.0379. Min Range: -1.0379

Max Act: 0.8125. Min Act: -0.0001

Data Index: 1148556 (C4 (Web Text))

Max Activating Token Index: 536

Text #16

Max Range: 1.0379. Min Range: -1.0379

Max Act: 0.9147. Min Act: -0.0001

Data Index: 208901 (C4 (Web Text))

Max Activating Token Index: 185

Text #17

Max Range: 1.0379. Min Range: -1.0379

Max Act: 0.9071. Min Act: -0.0001

Data Index: 731595 (C4 (Web Text))

Max Activating Token Index: 950

Text #18

Max Range: 1.0379. Min Range: -1.0379

Max Act: 0.9497. Min Act: -0.0001

Data Index: 641831 (C4 (Web Text))

Max Activating Token Index: 771

Text #19

Max Range: 1.0379. Min Range: -1.0379

Max Act: 0.8381. Min Act: -0.0001

Data Index: 282868 (C4 (Web Text))

Max Activating Token Index: 371