Model: SoLU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 364 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-4l')

Text #0

Max Range: 0.0659. Min Range: -0.0659

Max Act: 0.0659. Min Act: -0.0005

Data Index: 222331 (C4 (Web Text))

Max Activating Token Index: 339

Text #1

Max Range: 0.0659. Min Range: -0.0659

Max Act: 0.0563. Min Act: -0.0005

Data Index: 1022339 (C4 (Web Text))

Max Activating Token Index: 61

Text #2

Max Range: 0.0659. Min Range: -0.0659

Max Act: 0.0567. Min Act: -0.0005

Data Index: 1095205 (C4 (Web Text))

Max Activating Token Index: 312

Text #3

Max Range: 0.0659. Min Range: -0.0659

Max Act: 0.0527. Min Act: -0.0005

Data Index: 1182408 (C4 (Web Text))

Max Activating Token Index: 396

Text #4

Max Range: 0.0659. Min Range: -0.0659

Max Act: 0.0537. Min Act: -0.0005

Data Index: 1304655 (C4 (Web Text))

Max Activating Token Index: 394

Text #5

Max Range: 0.0659. Min Range: -0.0659

Max Act: 0.0543. Min Act: -0.0005

Data Index: 801486 (C4 (Web Text))

Max Activating Token Index: 257

Text #6

Max Range: 0.0659. Min Range: -0.0659

Max Act: 0.0531. Min Act: -0.0005

Data Index: 773711 (C4 (Web Text))

Max Activating Token Index: 163

Text #7

Max Range: 0.0659. Min Range: -0.0659

Max Act: 0.0526. Min Act: -0.0005

Data Index: 120892 (C4 (Web Text))

Max Activating Token Index: 595

Text #8

Max Range: 0.0659. Min Range: -0.0659

Max Act: 0.0505. Min Act: -0.0005

Data Index: 677567 (C4 (Web Text))

Max Activating Token Index: 644

Text #9

Max Range: 0.0659. Min Range: -0.0659

Max Act: 0.0530. Min Act: -0.0005

Data Index: 530942 (C4 (Web Text))

Max Activating Token Index: 148

Text #10

Max Range: 0.0659. Min Range: -0.0659

Max Act: 0.0527. Min Act: -0.0005

Data Index: 896829 (C4 (Web Text))

Max Activating Token Index: 210

Text #11

Max Range: 0.0659. Min Range: -0.0659

Max Act: 0.0503. Min Act: -0.0005

Data Index: 800698 (C4 (Web Text))

Max Activating Token Index: 267

Text #12

Max Range: 0.0659. Min Range: -0.0659

Max Act: 0.0507. Min Act: -0.0005

Data Index: 244389 (C4 (Web Text))

Max Activating Token Index: 301

Text #13

Max Range: 0.0659. Min Range: -0.0659

Max Act: 0.0498. Min Act: -0.0005

Data Index: 731374 (C4 (Web Text))

Max Activating Token Index: 911

Text #14

Max Range: 0.0659. Min Range: -0.0659

Max Act: 0.0517. Min Act: -0.0005

Data Index: 26523 (C4 (Web Text))

Max Activating Token Index: 910

Text #15

Max Range: 0.0659. Min Range: -0.0659

Max Act: 0.0502. Min Act: -0.0005

Data Index: 318539 (C4 (Web Text))

Max Activating Token Index: 725

Text #16

Max Range: 0.0659. Min Range: -0.0659

Max Act: 0.0504. Min Act: -0.0005

Data Index: 1165169 (C4 (Web Text))

Max Activating Token Index: 179

Text #17

Max Range: 0.0659. Min Range: -0.0659

Max Act: 0.0513. Min Act: -0.0005

Data Index: 1306975 (C4 (Web Text))

Max Activating Token Index: 62

Text #18

Max Range: 0.0659. Min Range: -0.0659

Max Act: 0.0504. Min Act: -0.0005

Data Index: 1166325 (C4 (Web Text))

Max Activating Token Index: 424

Text #19

Max Range: 0.0659. Min Range: -0.0659

Max Act: 0.0522. Min Act: -0.0005

Data Index: 1110748 (C4 (Web Text))

Max Activating Token Index: 224