Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1644 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.8704. Min Range: -0.8704

Max Act: 0.8704. Min Act: -0.0000

Data Index: 129956 (C4 (Web Text))

Max Activating Token Index: 766

Text #1

Max Range: 0.8704. Min Range: -0.8704

Max Act: 0.8217. Min Act: -0.0000

Data Index: 265604 (C4 (Web Text))

Max Activating Token Index: 437

Text #2

Max Range: 0.8704. Min Range: -0.8704

Max Act: 0.7881. Min Act: -0.0000

Data Index: 51472 (C4 (Web Text))

Max Activating Token Index: 186

Text #3

Max Range: 0.8704. Min Range: -0.8704

Max Act: 0.7708. Min Act: -0.0000

Data Index: 925768 (C4 (Web Text))

Max Activating Token Index: 245

Text #4

Max Range: 0.8704. Min Range: -0.8704

Max Act: 0.6967. Min Act: -0.0000

Data Index: 1233671 (C4 (Web Text))

Max Activating Token Index: 87

Text #5

Max Range: 0.8704. Min Range: -0.8704

Max Act: 0.6592. Min Act: -0.0000

Data Index: 764323 (C4 (Web Text))

Max Activating Token Index: 395

Text #6

Max Range: 0.8704. Min Range: -0.8704

Max Act: 0.6078. Min Act: -0.0000

Data Index: 803181 (C4 (Web Text))

Max Activating Token Index: 443

Text #7

Max Range: 0.8704. Min Range: -0.8704

Max Act: 0.6440. Min Act: -0.0000

Data Index: 1032671 (C4 (Web Text))

Max Activating Token Index: 890

Text #8

Max Range: 0.8704. Min Range: -0.8704

Max Act: 0.5843. Min Act: -0.0000

Data Index: 1065077 (C4 (Web Text))

Max Activating Token Index: 118

Text #9

Max Range: 0.8704. Min Range: -0.8704

Max Act: 0.6030. Min Act: -0.0000

Data Index: 364967 (C4 (Web Text))

Max Activating Token Index: 1001

Text #10

Max Range: 0.8704. Min Range: -0.8704

Max Act: 0.5825. Min Act: -0.0000

Data Index: 684965 (C4 (Web Text))

Max Activating Token Index: 195

Text #11

Max Range: 0.8704. Min Range: -0.8704

Max Act: 0.5845. Min Act: -0.0000

Data Index: 836288 (C4 (Web Text))

Max Activating Token Index: 150

Text #12

Max Range: 0.8704. Min Range: -0.8704

Max Act: 0.5830. Min Act: -0.0000

Data Index: 1217083 (C4 (Web Text))

Max Activating Token Index: 542

Text #13

Max Range: 0.8704. Min Range: -0.8704

Max Act: 0.5840. Min Act: -0.0000

Data Index: 675810 (C4 (Web Text))

Max Activating Token Index: 371

Text #14

Max Range: 0.8704. Min Range: -0.8704

Max Act: 0.5653. Min Act: -0.0000

Data Index: 403918 (C4 (Web Text))

Max Activating Token Index: 224

Text #15

Max Range: 0.8704. Min Range: -0.8704

Max Act: 0.5769. Min Act: -0.0000

Data Index: 1201885 (C4 (Web Text))

Max Activating Token Index: 28

Text #16

Max Range: 0.8704. Min Range: -0.8704

Max Act: 0.5693. Min Act: -0.0000

Data Index: 770167 (C4 (Web Text))

Max Activating Token Index: 670

Text #17

Max Range: 0.8704. Min Range: -0.8704

Max Act: 0.5442. Min Act: -0.0000

Data Index: 1189800 (C4 (Web Text))

Max Activating Token Index: 99

Text #18

Max Range: 0.8704. Min Range: -0.8704

Max Act: 0.5701. Min Act: -0.0000

Data Index: 550638 (C4 (Web Text))

Max Activating Token Index: 710

Text #19

Max Range: 0.8704. Min Range: -0.8704

Max Act: 0.5413. Min Act: -0.0000

Data Index: 107985 (C4 (Web Text))

Max Activating Token Index: 372