Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2751 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.3495. Min Range: -2.3495

Max Act: 2.3495. Min Act: -0.0000

Data Index: 1264189 (C4 (Web Text))

Max Activating Token Index: 422

Text #1

Max Range: 2.3495. Min Range: -2.3495

Max Act: 1.7938. Min Act: -0.0000

Data Index: 224831 (C4 (Web Text))

Max Activating Token Index: 830

Text #2

Max Range: 2.3495. Min Range: -2.3495

Max Act: 1.6962. Min Act: -0.0000

Data Index: 1015034 (C4 (Web Text))

Max Activating Token Index: 48

Text #3

Max Range: 2.3495. Min Range: -2.3495

Max Act: 1.7815. Min Act: -0.0000

Data Index: 919167 (C4 (Web Text))

Max Activating Token Index: 558

Text #4

Max Range: 2.3495. Min Range: -2.3495

Max Act: 1.6676. Min Act: 0.0000

Data Index: 1313971 (C4 (Web Text))

Max Activating Token Index: 876

Text #5

Max Range: 2.3495. Min Range: -2.3495

Max Act: 1.6300. Min Act: -0.0000

Data Index: 514227 (C4 (Web Text))

Max Activating Token Index: 223

Text #6

Max Range: 2.3495. Min Range: -2.3495

Max Act: 1.6245. Min Act: -0.0000

Data Index: 295259 (C4 (Web Text))

Max Activating Token Index: 348

Text #7

Max Range: 2.3495. Min Range: -2.3495

Max Act: 1.5903. Min Act: -0.0000

Data Index: 187837 (C4 (Web Text))

Max Activating Token Index: 273

Text #8

Max Range: 2.3495. Min Range: -2.3495

Max Act: 1.4544. Min Act: 0.0000

Data Index: 664113 (C4 (Web Text))

Max Activating Token Index: 917

Text #9

Max Range: 2.3495. Min Range: -2.3495

Max Act: 1.4396. Min Act: -0.0000

Data Index: 343746 (C4 (Web Text))

Max Activating Token Index: 103

Text #10

Max Range: 2.3495. Min Range: -2.3495

Max Act: 1.4174. Min Act: -0.0000

Data Index: 574050 (C4 (Web Text))

Max Activating Token Index: 905

Text #11

Max Range: 2.3495. Min Range: -2.3495

Max Act: 1.3410. Min Act: -0.0000

Data Index: 912885 (C4 (Web Text))

Max Activating Token Index: 254

Text #12

Max Range: 2.3495. Min Range: -2.3495

Max Act: 1.3268. Min Act: -0.0000

Data Index: 592648 (C4 (Web Text))

Max Activating Token Index: 413

Text #13

Max Range: 2.3495. Min Range: -2.3495

Max Act: 1.4004. Min Act: -0.0000

Data Index: 146047 (C4 (Web Text))

Max Activating Token Index: 237

Text #14

Max Range: 2.3495. Min Range: -2.3495

Max Act: 1.4011. Min Act: -0.0000

Data Index: 1098869 (C4 (Web Text))

Max Activating Token Index: 117

Text #15

Max Range: 2.3495. Min Range: -2.3495

Max Act: 1.3862. Min Act: 0.0000

Data Index: 212695 (C4 (Web Text))

Max Activating Token Index: 760

Text #16

Max Range: 2.3495. Min Range: -2.3495

Max Act: 1.3829. Min Act: 0.0000

Data Index: 113584 (C4 (Web Text))

Max Activating Token Index: 902

Text #17

Max Range: 2.3495. Min Range: -2.3495

Max Act: 1.3572. Min Act: -0.0000

Data Index: 50542 (C4 (Web Text))

Max Activating Token Index: 683

Text #18

Max Range: 2.3495. Min Range: -2.3495

Max Act: 1.3625. Min Act: -0.0000

Data Index: 398619 (C4 (Web Text))

Max Activating Token Index: 938

Text #19

Max Range: 2.3495. Min Range: -2.3495

Max Act: 1.2879. Min Act: -0.0000

Data Index: 985643 (C4 (Web Text))

Max Activating Token Index: 270