Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4415 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.6879. Min Range: -0.6879

Max Act: 0.6879. Min Act: -0.0002

Data Index: 647311 (C4 (Web Text))

Max Activating Token Index: 824

Text #1

Max Range: 0.6879. Min Range: -0.6879

Max Act: 0.6581. Min Act: -0.0002

Data Index: 133042 (C4 (Web Text))

Max Activating Token Index: 1005

Text #2

Max Range: 0.6879. Min Range: -0.6879

Max Act: 0.6627. Min Act: -0.0002

Data Index: 196154 (C4 (Web Text))

Max Activating Token Index: 942

Text #3

Max Range: 0.6879. Min Range: -0.6879

Max Act: 0.6827. Min Act: -0.0002

Data Index: 799607 (C4 (Web Text))

Max Activating Token Index: 908

Text #4

Max Range: 0.6879. Min Range: -0.6879

Max Act: 0.6188. Min Act: -0.0002

Data Index: 325440 (C4 (Web Text))

Max Activating Token Index: 132

Text #5

Max Range: 0.6879. Min Range: -0.6879

Max Act: 0.6243. Min Act: -0.0002

Data Index: 130539 (C4 (Web Text))

Max Activating Token Index: 181

Text #6

Max Range: 0.6879. Min Range: -0.6879

Max Act: 0.5470. Min Act: -0.0002

Data Index: 1292860 (C4 (Web Text))

Max Activating Token Index: 869

Text #7

Max Range: 0.6879. Min Range: -0.6879

Max Act: 0.5618. Min Act: -0.0002

Data Index: 428195 (C4 (Web Text))

Max Activating Token Index: 722

Text #8

Max Range: 0.6879. Min Range: -0.6879

Max Act: 0.6331. Min Act: -0.0002

Data Index: 152494 (C4 (Web Text))

Max Activating Token Index: 302

Text #9

Max Range: 0.6879. Min Range: -0.6879

Max Act: 0.6106. Min Act: -0.0002

Data Index: 107700 (C4 (Web Text))

Max Activating Token Index: 382

Text #10

Max Range: 0.6879. Min Range: -0.6879

Max Act: 0.6018. Min Act: -0.0002

Data Index: 1355432 (C4 (Web Text))

Max Activating Token Index: 483

Text #11

Max Range: 0.6879. Min Range: -0.6879

Max Act: 0.6270. Min Act: -0.0002

Data Index: 1064642 (C4 (Web Text))

Max Activating Token Index: 583

Text #12

Max Range: 0.6879. Min Range: -0.6879

Max Act: 0.5383. Min Act: -0.0002

Data Index: 614148 (C4 (Web Text))

Max Activating Token Index: 619

Text #13

Max Range: 0.6879. Min Range: -0.6879

Max Act: 0.6022. Min Act: -0.0002

Data Index: 807269 (C4 (Web Text))

Max Activating Token Index: 710

Text #14

Max Range: 0.6879. Min Range: -0.6879

Max Act: 0.4441. Min Act: -0.0002

Data Index: 341143 (C4 (Web Text))

Max Activating Token Index: 786

Text #15

Max Range: 0.6879. Min Range: -0.6879

Max Act: 0.5542. Min Act: -0.0002

Data Index: 118794 (C4 (Web Text))

Max Activating Token Index: 44

Text #16

Max Range: 0.6879. Min Range: -0.6879

Max Act: 0.5520. Min Act: -0.0002

Data Index: 482763 (C4 (Web Text))

Max Activating Token Index: 183

Text #17

Max Range: 0.6879. Min Range: -0.6879

Max Act: 0.6023. Min Act: -0.0002

Data Index: 650179 (C4 (Web Text))

Max Activating Token Index: 848

Text #18

Max Range: 0.6879. Min Range: -0.6879

Max Act: 0.5829. Min Act: -0.0002

Data Index: 956402 (C4 (Web Text))

Max Activating Token Index: 170

Text #19

Max Range: 0.6879. Min Range: -0.6879

Max Act: 0.5651. Min Act: -0.0002

Data Index: 773303 (C4 (Web Text))

Max Activating Token Index: 592