Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 363 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.1084. Min Range: -1.1084

Max Act: 1.1084. Min Act: -0.0000

Data Index: 921211 (C4 (Web Text))

Max Activating Token Index: 140

Text #1

Max Range: 1.1084. Min Range: -1.1084

Max Act: 0.9294. Min Act: -0.0000

Data Index: 1061050 (C4 (Web Text))

Max Activating Token Index: 877

Text #2

Max Range: 1.1084. Min Range: -1.1084

Max Act: 0.9333. Min Act: -0.0000

Data Index: 3075 (C4 (Web Text))

Max Activating Token Index: 632

Text #3

Max Range: 1.1084. Min Range: -1.1084

Max Act: 0.7747. Min Act: -0.0000

Data Index: 636388 (C4 (Web Text))

Max Activating Token Index: 327

Text #4

Max Range: 1.1084. Min Range: -1.1084

Max Act: 0.7627. Min Act: -0.0000

Data Index: 1524338 (Python Code)

Max Activating Token Index: 604

Text #5

Max Range: 1.1084. Min Range: -1.1084

Max Act: 0.6810. Min Act: -0.0000

Data Index: 886312 (C4 (Web Text))

Max Activating Token Index: 127

Text #6

Max Range: 1.1084. Min Range: -1.1084

Max Act: 0.7462. Min Act: -0.0000

Data Index: 1424561 (Python Code)

Max Activating Token Index: 430

Text #7

Max Range: 1.1084. Min Range: -1.1084

Max Act: 0.6850. Min Act: -0.0000

Data Index: 1571977 (Python Code)

Max Activating Token Index: 513

Text #8

Max Range: 1.1084. Min Range: -1.1084

Max Act: 0.6257. Min Act: -0.0000

Data Index: 181624 (C4 (Web Text))

Max Activating Token Index: 912

Text #9

Max Range: 1.1084. Min Range: -1.1084

Max Act: 0.6445. Min Act: -0.0000

Data Index: 1509729 (Python Code)

Max Activating Token Index: 360

Text #10

Max Range: 1.1084. Min Range: -1.1084

Max Act: 0.6560. Min Act: -0.0000

Data Index: 363881 (C4 (Web Text))

Max Activating Token Index: 1012

Text #11

Max Range: 1.1084. Min Range: -1.1084

Max Act: 0.8377. Min Act: -0.0000

Data Index: 1532813 (Python Code)

Max Activating Token Index: 753

Text #12

Max Range: 1.1084. Min Range: -1.1084

Max Act: 0.6074. Min Act: -0.0000

Data Index: 287892 (C4 (Web Text))

Max Activating Token Index: 698

Text #13

Max Range: 1.1084. Min Range: -1.1084

Max Act: 0.6703. Min Act: -0.0000

Data Index: 1463107 (Python Code)

Max Activating Token Index: 553

Text #14

Max Range: 1.1084. Min Range: -1.1084

Max Act: 0.5538. Min Act: -0.0000

Data Index: 827716 (C4 (Web Text))

Max Activating Token Index: 491

Text #15

Max Range: 1.1084. Min Range: -1.1084

Max Act: 0.5692. Min Act: -0.0000

Data Index: 1636350 (Python Code)

Max Activating Token Index: 726

Text #16

Max Range: 1.1084. Min Range: -1.1084

Max Act: 0.5707. Min Act: -0.0000

Data Index: 1629758 (Python Code)

Max Activating Token Index: 64

Text #17

Max Range: 1.1084. Min Range: -1.1084

Max Act: 0.5055. Min Act: -0.0000

Data Index: 1530899 (Python Code)

Max Activating Token Index: 94

Text #18

Max Range: 1.1084. Min Range: -1.1084

Max Act: 0.5161. Min Act: -0.0000

Data Index: 1410660 (Python Code)

Max Activating Token Index: 485

Text #19

Max Range: 1.1084. Min Range: -1.1084

Max Act: 0.5448. Min Act: -0.0000

Data Index: 1434691 (Python Code)

Max Activating Token Index: 726