Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1729 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 0.1956. Min Range: -0.1956

Max Act: 0.1956. Min Act: -0.0001

Data Index: 43302 (C4 (Web Text))

Max Activating Token Index: 1007

Text #1

Max Range: 0.1956. Min Range: -0.1956

Max Act: 0.1821. Min Act: -0.0001

Data Index: 598072 (C4 (Web Text))

Max Activating Token Index: 101

Text #2

Max Range: 0.1956. Min Range: -0.1956

Max Act: 0.1813. Min Act: -0.0001

Data Index: 222925 (C4 (Web Text))

Max Activating Token Index: 838

Text #3

Max Range: 0.1956. Min Range: -0.1956

Max Act: 0.1854. Min Act: -0.0001

Data Index: 496199 (C4 (Web Text))

Max Activating Token Index: 700

Text #4

Max Range: 0.1956. Min Range: -0.1956

Max Act: 0.1503. Min Act: -0.0001

Data Index: 478271 (C4 (Web Text))

Max Activating Token Index: 615

Text #5

Max Range: 0.1956. Min Range: -0.1956

Max Act: 0.1638. Min Act: -0.0001

Data Index: 519519 (C4 (Web Text))

Max Activating Token Index: 795

Text #6

Max Range: 0.1956. Min Range: -0.1956

Max Act: 0.1841. Min Act: -0.0001

Data Index: 250386 (C4 (Web Text))

Max Activating Token Index: 907

Text #7

Max Range: 0.1956. Min Range: -0.1956

Max Act: 0.1654. Min Act: -0.0001

Data Index: 463266 (C4 (Web Text))

Max Activating Token Index: 468

Text #8

Max Range: 0.1956. Min Range: -0.1956

Max Act: 0.1580. Min Act: -0.0001

Data Index: 272365 (C4 (Web Text))

Max Activating Token Index: 429

Text #9

Max Range: 0.1956. Min Range: -0.1956

Max Act: 0.1570. Min Act: -0.0001

Data Index: 401562 (C4 (Web Text))

Max Activating Token Index: 88

Text #10

Max Range: 0.1956. Min Range: -0.1956

Max Act: 0.1728. Min Act: -0.0001

Data Index: 956882 (C4 (Web Text))

Max Activating Token Index: 389

Text #11

Max Range: 0.1956. Min Range: -0.1956

Max Act: 0.1545. Min Act: -0.0001

Data Index: 99806 (C4 (Web Text))

Max Activating Token Index: 1009

Text #12

Max Range: 0.1956. Min Range: -0.1956

Max Act: 0.1483. Min Act: -0.0001

Data Index: 357583 (C4 (Web Text))

Max Activating Token Index: 456

Text #13

Max Range: 0.1956. Min Range: -0.1956

Max Act: 0.1706. Min Act: -0.0001

Data Index: 702363 (C4 (Web Text))

Max Activating Token Index: 519

Text #14

Max Range: 0.1956. Min Range: -0.1956

Max Act: 0.1540. Min Act: -0.0001

Data Index: 582336 (C4 (Web Text))

Max Activating Token Index: 679

Text #15

Max Range: 0.1956. Min Range: -0.1956

Max Act: 0.1566. Min Act: -0.0001

Data Index: 762052 (C4 (Web Text))

Max Activating Token Index: 673

Text #16

Max Range: 0.1956. Min Range: -0.1956

Max Act: 0.1463. Min Act: -0.0001

Data Index: 1076561 (C4 (Web Text))

Max Activating Token Index: 753

Text #17

Max Range: 0.1956. Min Range: -0.1956

Max Act: 0.1376. Min Act: -0.0001

Data Index: 338422 (C4 (Web Text))

Max Activating Token Index: 946

Text #18

Max Range: 0.1956. Min Range: -0.1956

Max Act: 0.1309. Min Act: -0.0001

Data Index: 207828 (C4 (Web Text))

Max Activating Token Index: 144

Text #19

Max Range: 0.1956. Min Range: -0.1956

Max Act: 0.1460. Min Act: -0.0001

Data Index: 1005439 (C4 (Web Text))

Max Activating Token Index: 965