Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 559 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 7.2577. Min Range: -7.2577

Max Act: 7.2577. Min Act: -0.0000

Data Index: 25659 (C4 (Web Text))

Max Activating Token Index: 830

Text #1

Max Range: 7.2577. Min Range: -7.2577

Max Act: 7.0734. Min Act: -0.0000

Data Index: 870636 (C4 (Web Text))

Max Activating Token Index: 901

Text #2

Max Range: 7.2577. Min Range: -7.2577

Max Act: 6.9500. Min Act: -0.0000

Data Index: 869540 (C4 (Web Text))

Max Activating Token Index: 625

Text #3

Max Range: 7.2577. Min Range: -7.2577

Max Act: 6.9337. Min Act: -0.0000

Data Index: 175022 (C4 (Web Text))

Max Activating Token Index: 492

Text #4

Max Range: 7.2577. Min Range: -7.2577

Max Act: 7.0421. Min Act: -0.0000

Data Index: 9850 (C4 (Web Text))

Max Activating Token Index: 67

Text #5

Max Range: 7.2577. Min Range: -7.2577

Max Act: 6.7784. Min Act: -0.0000

Data Index: 163232 (C4 (Web Text))

Max Activating Token Index: 143

Text #6

Max Range: 7.2577. Min Range: -7.2577

Max Act: 6.7049. Min Act: -0.0000

Data Index: 381425 (C4 (Web Text))

Max Activating Token Index: 66

Text #7

Max Range: 7.2577. Min Range: -7.2577

Max Act: 6.6487. Min Act: -0.0000

Data Index: 795525 (C4 (Web Text))

Max Activating Token Index: 41

Text #8

Max Range: 7.2577. Min Range: -7.2577

Max Act: 6.4012. Min Act: -0.0000

Data Index: 1311665 (C4 (Web Text))

Max Activating Token Index: 911

Text #9

Max Range: 7.2577. Min Range: -7.2577

Max Act: 6.5685. Min Act: -0.0000

Data Index: 759125 (C4 (Web Text))

Max Activating Token Index: 225

Text #10

Max Range: 7.2577. Min Range: -7.2577

Max Act: 6.2484. Min Act: -0.0000

Data Index: 129877 (C4 (Web Text))

Max Activating Token Index: 519

Text #11

Max Range: 7.2577. Min Range: -7.2577

Max Act: 6.2496. Min Act: -0.0000

Data Index: 282119 (C4 (Web Text))

Max Activating Token Index: 578

Text #12

Max Range: 7.2577. Min Range: -7.2577

Max Act: 6.0335. Min Act: -0.0000

Data Index: 779607 (C4 (Web Text))

Max Activating Token Index: 660

Text #13

Max Range: 7.2577. Min Range: -7.2577

Max Act: 5.6961. Min Act: -0.0000

Data Index: 1165840 (C4 (Web Text))

Max Activating Token Index: 238

Text #14

Max Range: 7.2577. Min Range: -7.2577

Max Act: 5.9238. Min Act: -0.0000

Data Index: 1275252 (C4 (Web Text))

Max Activating Token Index: 218

Text #15

Max Range: 7.2577. Min Range: -7.2577

Max Act: 5.5908. Min Act: -0.0000

Data Index: 1434630 (Python Code)

Max Activating Token Index: 386

Text #16

Max Range: 7.2577. Min Range: -7.2577

Max Act: 5.6955. Min Act: -0.0000

Data Index: 1345270 (C4 (Web Text))

Max Activating Token Index: 935

Text #17

Max Range: 7.2577. Min Range: -7.2577

Max Act: 5.6484. Min Act: -0.0000

Data Index: 823051 (C4 (Web Text))

Max Activating Token Index: 512

Text #18

Max Range: 7.2577. Min Range: -7.2577

Max Act: 5.6131. Min Act: -0.0000

Data Index: 1075826 (C4 (Web Text))

Max Activating Token Index: 342

Text #19

Max Range: 7.2577. Min Range: -7.2577

Max Act: 5.6177. Min Act: -0.0000

Data Index: 591576 (C4 (Web Text))

Max Activating Token Index: 67