Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4575 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.6733. Min Range: -1.6733

Max Act: 1.6733. Min Act: -0.0000

Data Index: 946735 (C4 (Web Text))

Max Activating Token Index: 795

Text #1

Max Range: 1.6733. Min Range: -1.6733

Max Act: 1.5011. Min Act: -0.0000

Data Index: 32012 (C4 (Web Text))

Max Activating Token Index: 288

Text #2

Max Range: 1.6733. Min Range: -1.6733

Max Act: 1.4711. Min Act: -0.0000

Data Index: 787625 (C4 (Web Text))

Max Activating Token Index: 498

Text #3

Max Range: 1.6733. Min Range: -1.6733

Max Act: 1.1830. Min Act: 0.0000

Data Index: 994939 (C4 (Web Text))

Max Activating Token Index: 115

Text #4

Max Range: 1.6733. Min Range: -1.6733

Max Act: 1.1594. Min Act: -0.0000

Data Index: 104725 (C4 (Web Text))

Max Activating Token Index: 316

Text #5

Max Range: 1.6733. Min Range: -1.6733

Max Act: 1.1380. Min Act: -0.0000

Data Index: 652046 (C4 (Web Text))

Max Activating Token Index: 233

Text #6

Max Range: 1.6733. Min Range: -1.6733

Max Act: 1.0145. Min Act: -0.0000

Data Index: 293334 (C4 (Web Text))

Max Activating Token Index: 650

Text #7

Max Range: 1.6733. Min Range: -1.6733

Max Act: 1.0065. Min Act: -0.0000

Data Index: 1086102 (C4 (Web Text))

Max Activating Token Index: 94

Text #8

Max Range: 1.6733. Min Range: -1.6733

Max Act: 1.0157. Min Act: -0.0000

Data Index: 710649 (C4 (Web Text))

Max Activating Token Index: 864

Text #9

Max Range: 1.6733. Min Range: -1.6733

Max Act: 0.9802. Min Act: -0.0000

Data Index: 1218533 (C4 (Web Text))

Max Activating Token Index: 191

Text #10

Max Range: 1.6733. Min Range: -1.6733

Max Act: 0.9360. Min Act: -0.0000

Data Index: 531756 (C4 (Web Text))

Max Activating Token Index: 784

Text #11

Max Range: 1.6733. Min Range: -1.6733

Max Act: 0.9050. Min Act: -0.0000

Data Index: 406542 (C4 (Web Text))

Max Activating Token Index: 281

Text #12

Max Range: 1.6733. Min Range: -1.6733

Max Act: 0.8823. Min Act: -0.0000

Data Index: 1295219 (C4 (Web Text))

Max Activating Token Index: 932

Text #13

Max Range: 1.6733. Min Range: -1.6733

Max Act: 0.9391. Min Act: -0.0000

Data Index: 997962 (C4 (Web Text))

Max Activating Token Index: 358

Text #14

Max Range: 1.6733. Min Range: -1.6733

Max Act: 0.8930. Min Act: -0.0000

Data Index: 1177138 (C4 (Web Text))

Max Activating Token Index: 382

Text #15

Max Range: 1.6733. Min Range: -1.6733

Max Act: 0.9229. Min Act: -0.0000

Data Index: 310986 (C4 (Web Text))

Max Activating Token Index: 745

Text #16

Max Range: 1.6733. Min Range: -1.6733

Max Act: 0.8886. Min Act: 0.0001

Data Index: 455488 (C4 (Web Text))

Max Activating Token Index: 918

Text #17

Max Range: 1.6733. Min Range: -1.6733

Max Act: 0.8824. Min Act: -0.0000

Data Index: 252715 (C4 (Web Text))

Max Activating Token Index: 225

Text #18

Max Range: 1.6733. Min Range: -1.6733

Max Act: 0.8370. Min Act: 0.0001

Data Index: 788828 (C4 (Web Text))

Max Activating Token Index: 290

Text #19

Max Range: 1.6733. Min Range: -1.6733

Max Act: 0.8312. Min Act: -0.0000

Data Index: 387442 (C4 (Web Text))

Max Activating Token Index: 185