Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4218 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.3776. Min Range: -0.3776

Max Act: 0.3776. Min Act: -0.0002

Data Index: 109839 (C4 (Web Text))

Max Activating Token Index: 203

Text #1

Max Range: 0.3776. Min Range: -0.3776

Max Act: 0.3609. Min Act: -0.0002

Data Index: 751975 (C4 (Web Text))

Max Activating Token Index: 775

Text #2

Max Range: 0.3776. Min Range: -0.3776

Max Act: 0.3013. Min Act: -0.0002

Data Index: 1310745 (C4 (Web Text))

Max Activating Token Index: 458

Text #3

Max Range: 0.3776. Min Range: -0.3776

Max Act: 0.2929. Min Act: -0.0002

Data Index: 483192 (C4 (Web Text))

Max Activating Token Index: 902

Text #4

Max Range: 0.3776. Min Range: -0.3776

Max Act: 0.2821. Min Act: -0.0002

Data Index: 1298595 (C4 (Web Text))

Max Activating Token Index: 411

Text #5

Max Range: 0.3776. Min Range: -0.3776

Max Act: 0.2717. Min Act: -0.0002

Data Index: 637011 (C4 (Web Text))

Max Activating Token Index: 1000

Text #6

Max Range: 0.3776. Min Range: -0.3776

Max Act: 0.2605. Min Act: -0.0002

Data Index: 1307920 (C4 (Web Text))

Max Activating Token Index: 791

Text #7

Max Range: 0.3776. Min Range: -0.3776

Max Act: 0.2572. Min Act: -0.0002

Data Index: 1280260 (C4 (Web Text))

Max Activating Token Index: 503

Text #8

Max Range: 0.3776. Min Range: -0.3776

Max Act: 0.2481. Min Act: -0.0002

Data Index: 288160 (C4 (Web Text))

Max Activating Token Index: 426

Text #9

Max Range: 0.3776. Min Range: -0.3776

Max Act: 0.2459. Min Act: -0.0002

Data Index: 686540 (C4 (Web Text))

Max Activating Token Index: 888

Text #10

Max Range: 0.3776. Min Range: -0.3776

Max Act: 0.2256. Min Act: -0.0002

Data Index: 235771 (C4 (Web Text))

Max Activating Token Index: 524

Text #11

Max Range: 0.3776. Min Range: -0.3776

Max Act: 0.2323. Min Act: -0.0002

Data Index: 1038350 (C4 (Web Text))

Max Activating Token Index: 979

Text #12

Max Range: 0.3776. Min Range: -0.3776

Max Act: 0.1978. Min Act: -0.0002

Data Index: 836993 (C4 (Web Text))

Max Activating Token Index: 297

Text #13

Max Range: 0.3776. Min Range: -0.3776

Max Act: 0.2031. Min Act: -0.0002

Data Index: 924860 (C4 (Web Text))

Max Activating Token Index: 924

Text #14

Max Range: 0.3776. Min Range: -0.3776

Max Act: 0.2037. Min Act: -0.0002

Data Index: 3686 (C4 (Web Text))

Max Activating Token Index: 169

Text #15

Max Range: 0.3776. Min Range: -0.3776

Max Act: 0.1831. Min Act: -0.0002

Data Index: 1230099 (C4 (Web Text))

Max Activating Token Index: 142

Text #16

Max Range: 0.3776. Min Range: -0.3776

Max Act: 0.1936. Min Act: -0.0002

Data Index: 926730 (C4 (Web Text))

Max Activating Token Index: 515

Text #17

Max Range: 0.3776. Min Range: -0.3776

Max Act: 0.1805. Min Act: -0.0002

Data Index: 451770 (C4 (Web Text))

Max Activating Token Index: 340

Text #18

Max Range: 0.3776. Min Range: -0.3776

Max Act: 0.1898. Min Act: -0.0002

Data Index: 1326233 (C4 (Web Text))

Max Activating Token Index: 493

Text #19

Max Range: 0.3776. Min Range: -0.3776

Max Act: 0.1817. Min Act: -0.0002

Data Index: 1402338 (Python Code)

Max Activating Token Index: 911