Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4893 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 4.3609. Min Range: -4.3609

Max Act: 4.3516. Min Act: -0.0000

Data Index: 425213 (C4 (Web Text))

Max Activating Token Index: 589

Text #1

Max Range: 4.3609. Min Range: -4.3609

Max Act: 4.3344. Min Act: -0.0000

Data Index: 702429 (C4 (Web Text))

Max Activating Token Index: 41

Text #2

Max Range: 4.3609. Min Range: -4.3609

Max Act: 4.2472. Min Act: -0.0000

Data Index: 701108 (C4 (Web Text))

Max Activating Token Index: 53

Text #3

Max Range: 4.3609. Min Range: -4.3609

Max Act: 4.0842. Min Act: -0.0000

Data Index: 1223793 (C4 (Web Text))

Max Activating Token Index: 102

Text #4

Max Range: 4.3609. Min Range: -4.3609

Max Act: 4.2308. Min Act: -0.0000

Data Index: 1268526 (C4 (Web Text))

Max Activating Token Index: 162

Text #5

Max Range: 4.3609. Min Range: -4.3609

Max Act: 3.8557. Min Act: -0.0000

Data Index: 1255589 (C4 (Web Text))

Max Activating Token Index: 1002

Text #6

Max Range: 4.3609. Min Range: -4.3609

Max Act: 4.3609. Min Act: -0.0000

Data Index: 1119673 (C4 (Web Text))

Max Activating Token Index: 549

Text #7

Max Range: 4.3609. Min Range: -4.3609

Max Act: 4.1619. Min Act: -0.0000

Data Index: 727732 (C4 (Web Text))

Max Activating Token Index: 684

Text #8

Max Range: 4.3609. Min Range: -4.3609

Max Act: 4.1564. Min Act: -0.0000

Data Index: 707557 (C4 (Web Text))

Max Activating Token Index: 457

Text #9

Max Range: 4.3609. Min Range: -4.3609

Max Act: 4.1481. Min Act: -0.0000

Data Index: 215578 (C4 (Web Text))

Max Activating Token Index: 773

Text #10

Max Range: 4.3609. Min Range: -4.3609

Max Act: 4.1114. Min Act: -0.0000

Data Index: 684714 (C4 (Web Text))

Max Activating Token Index: 270

Text #11

Max Range: 4.3609. Min Range: -4.3609

Max Act: 3.9397. Min Act: -0.0000

Data Index: 64736 (C4 (Web Text))

Max Activating Token Index: 92

Text #12

Max Range: 4.3609. Min Range: -4.3609

Max Act: 4.2595. Min Act: -0.0000

Data Index: 621612 (C4 (Web Text))

Max Activating Token Index: 177

Text #13

Max Range: 4.3609. Min Range: -4.3609

Max Act: 3.9325. Min Act: -0.0000

Data Index: 570582 (C4 (Web Text))

Max Activating Token Index: 346

Text #14

Max Range: 4.3609. Min Range: -4.3609

Max Act: 3.6107. Min Act: -0.0000

Data Index: 302201 (C4 (Web Text))

Max Activating Token Index: 207

Text #15

Max Range: 4.3609. Min Range: -4.3609

Max Act: 3.9036. Min Act: -0.0000

Data Index: 991821 (C4 (Web Text))

Max Activating Token Index: 115

Text #16

Max Range: 4.3609. Min Range: -4.3609

Max Act: 3.7220. Min Act: -0.0000

Data Index: 973936 (C4 (Web Text))

Max Activating Token Index: 540

Text #17

Max Range: 4.3609. Min Range: -4.3609

Max Act: 3.7096. Min Act: -0.0000

Data Index: 670653 (C4 (Web Text))

Max Activating Token Index: 94

Text #18

Max Range: 4.3609. Min Range: -4.3609

Max Act: 3.8168. Min Act: -0.0000

Data Index: 158978 (C4 (Web Text))

Max Activating Token Index: 33

Text #19

Max Range: 4.3609. Min Range: -4.3609

Max Act: 3.8032. Min Act: -0.0000

Data Index: 232926 (C4 (Web Text))

Max Activating Token Index: 80