Model: SoLU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1207 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-4l')

Text #0

Max Range: 0.1838. Min Range: -0.1838

Max Act: 0.1838. Min Act: -0.0002

Data Index: 531162 (C4 (Web Text))

Max Activating Token Index: 233

Text #1

Max Range: 0.1838. Min Range: -0.1838

Max Act: 0.1691. Min Act: -0.0002

Data Index: 138156 (C4 (Web Text))

Max Activating Token Index: 93

Text #2

Max Range: 0.1838. Min Range: -0.1838

Max Act: 0.1760. Min Act: -0.0002

Data Index: 219899 (C4 (Web Text))

Max Activating Token Index: 716

Text #3

Max Range: 0.1838. Min Range: -0.1838

Max Act: 0.1642. Min Act: -0.0002

Data Index: 286181 (C4 (Web Text))

Max Activating Token Index: 424

Text #4

Max Range: 0.1838. Min Range: -0.1838

Max Act: 0.1587. Min Act: -0.0002

Data Index: 286213 (C4 (Web Text))

Max Activating Token Index: 95

Text #5

Max Range: 0.1838. Min Range: -0.1838

Max Act: 0.1516. Min Act: -0.0002

Data Index: 1130739 (C4 (Web Text))

Max Activating Token Index: 80

Text #6

Max Range: 0.1838. Min Range: -0.1838

Max Act: 0.1526. Min Act: -0.0002

Data Index: 782256 (C4 (Web Text))

Max Activating Token Index: 415

Text #7

Max Range: 0.1838. Min Range: -0.1838

Max Act: 0.1557. Min Act: -0.0002

Data Index: 151662 (C4 (Web Text))

Max Activating Token Index: 985

Text #8

Max Range: 0.1838. Min Range: -0.1838

Max Act: 0.1682. Min Act: -0.0002

Data Index: 816312 (C4 (Web Text))

Max Activating Token Index: 312

Text #9

Max Range: 0.1838. Min Range: -0.1838

Max Act: 0.1541. Min Act: -0.0002

Data Index: 754645 (C4 (Web Text))

Max Activating Token Index: 40

Text #10

Max Range: 0.1838. Min Range: -0.1838

Max Act: 0.1467. Min Act: -0.0002

Data Index: 1061934 (C4 (Web Text))

Max Activating Token Index: 194

Text #11

Max Range: 0.1838. Min Range: -0.1838

Max Act: 0.1432. Min Act: -0.0002

Data Index: 224438 (C4 (Web Text))

Max Activating Token Index: 53

Text #12

Max Range: 0.1838. Min Range: -0.1838

Max Act: 0.1474. Min Act: -0.0002

Data Index: 1040179 (C4 (Web Text))

Max Activating Token Index: 49

Text #13

Max Range: 0.1838. Min Range: -0.1838

Max Act: 0.1514. Min Act: -0.0003

Data Index: 329065 (C4 (Web Text))

Max Activating Token Index: 771

Text #14

Max Range: 0.1838. Min Range: -0.1838

Max Act: 0.1471. Min Act: -0.0002

Data Index: 1075823 (C4 (Web Text))

Max Activating Token Index: 429

Text #15

Max Range: 0.1838. Min Range: -0.1838

Max Act: 0.1410. Min Act: -0.0002

Data Index: 1283715 (C4 (Web Text))

Max Activating Token Index: 112

Text #16

Max Range: 0.1838. Min Range: -0.1838

Max Act: 0.1469. Min Act: -0.0002

Data Index: 636128 (C4 (Web Text))

Max Activating Token Index: 987

Text #17

Max Range: 0.1838. Min Range: -0.1838

Max Act: 0.1456. Min Act: -0.0002

Data Index: 419605 (C4 (Web Text))

Max Activating Token Index: 106

Text #18

Max Range: 0.1838. Min Range: -0.1838

Max Act: 0.1383. Min Act: -0.0002

Data Index: 819738 (C4 (Web Text))

Max Activating Token Index: 463

Text #19

Max Range: 0.1838. Min Range: -0.1838

Max Act: 0.1478. Min Act: -0.0002

Data Index: 436671 (C4 (Web Text))

Max Activating Token Index: 1006