Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2519 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.1857. Min Range: -0.1857

Max Act: 0.1797. Min Act: -0.0000

Data Index: 1093604 (C4 (Web Text))

Max Activating Token Index: 313

Text #1

Max Range: 0.1857. Min Range: -0.1857

Max Act: 0.1769. Min Act: -0.0000

Data Index: 248324 (C4 (Web Text))

Max Activating Token Index: 878

Text #2

Max Range: 0.1857. Min Range: -0.1857

Max Act: 0.1857. Min Act: -0.0000

Data Index: 1089233 (C4 (Web Text))

Max Activating Token Index: 804

Text #3

Max Range: 0.1857. Min Range: -0.1857

Max Act: 0.1656. Min Act: -0.0000

Data Index: 1171650 (C4 (Web Text))

Max Activating Token Index: 598

Text #4

Max Range: 0.1857. Min Range: -0.1857

Max Act: 0.1650. Min Act: -0.0000

Data Index: 1248693 (C4 (Web Text))

Max Activating Token Index: 75

Text #5

Max Range: 0.1857. Min Range: -0.1857

Max Act: 0.1714. Min Act: -0.0000

Data Index: 416308 (C4 (Web Text))

Max Activating Token Index: 968

Text #6

Max Range: 0.1857. Min Range: -0.1857

Max Act: 0.1636. Min Act: 0.0001

Data Index: 454764 (C4 (Web Text))

Max Activating Token Index: 654

Text #7

Max Range: 0.1857. Min Range: -0.1857

Max Act: 0.1517. Min Act: -0.0000

Data Index: 1168608 (C4 (Web Text))

Max Activating Token Index: 705

Text #8

Max Range: 0.1857. Min Range: -0.1857

Max Act: 0.1513. Min Act: -0.0000

Data Index: 1244410 (C4 (Web Text))

Max Activating Token Index: 208

Text #9

Max Range: 0.1857. Min Range: -0.1857

Max Act: 0.1511. Min Act: -0.0000

Data Index: 381501 (C4 (Web Text))

Max Activating Token Index: 75

Text #10

Max Range: 0.1857. Min Range: -0.1857

Max Act: 0.1526. Min Act: -0.0000

Data Index: 209138 (C4 (Web Text))

Max Activating Token Index: 577

Text #11

Max Range: 0.1857. Min Range: -0.1857

Max Act: 0.1619. Min Act: -0.0000

Data Index: 983857 (C4 (Web Text))

Max Activating Token Index: 226

Text #12

Max Range: 0.1857. Min Range: -0.1857

Max Act: 0.1529. Min Act: -0.0000

Data Index: 1011130 (C4 (Web Text))

Max Activating Token Index: 239

Text #13

Max Range: 0.1857. Min Range: -0.1857

Max Act: 0.1379. Min Act: -0.0000

Data Index: 242723 (C4 (Web Text))

Max Activating Token Index: 519

Text #14

Max Range: 0.1857. Min Range: -0.1857

Max Act: 0.1495. Min Act: -0.0000

Data Index: 944706 (C4 (Web Text))

Max Activating Token Index: 678

Text #15

Max Range: 0.1857. Min Range: -0.1857

Max Act: 0.1428. Min Act: -0.0000

Data Index: 1319105 (C4 (Web Text))

Max Activating Token Index: 312

Text #16

Max Range: 0.1857. Min Range: -0.1857

Max Act: 0.1408. Min Act: -0.0000

Data Index: 761784 (C4 (Web Text))

Max Activating Token Index: 411

Text #17

Max Range: 0.1857. Min Range: -0.1857

Max Act: 0.1458. Min Act: -0.0000

Data Index: 453451 (C4 (Web Text))

Max Activating Token Index: 567

Text #18

Max Range: 0.1857. Min Range: -0.1857

Max Act: 0.1334. Min Act: 0.0000

Data Index: 1059367 (C4 (Web Text))

Max Activating Token Index: 404

Text #19

Max Range: 0.1857. Min Range: -0.1857

Max Act: 0.1405. Min Act: 0.0000

Data Index: 496436 (C4 (Web Text))

Max Activating Token Index: 549