Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 482 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.5944. Min Range: -2.5944

Max Act: 2.4617. Min Act: -0.0000

Data Index: 1185855 (C4 (Web Text))

Max Activating Token Index: 795

Text #1

Max Range: 2.5944. Min Range: -2.5944

Max Act: 2.3129. Min Act: -0.0000

Data Index: 36846 (C4 (Web Text))

Max Activating Token Index: 729

Text #2

Max Range: 2.5944. Min Range: -2.5944

Max Act: 2.5944. Min Act: -0.0000

Data Index: 1081423 (C4 (Web Text))

Max Activating Token Index: 756

Text #3

Max Range: 2.5944. Min Range: -2.5944

Max Act: 2.3509. Min Act: -0.0000

Data Index: 1027491 (C4 (Web Text))

Max Activating Token Index: 495

Text #4

Max Range: 2.5944. Min Range: -2.5944

Max Act: 2.0053. Min Act: -0.0000

Data Index: 327548 (C4 (Web Text))

Max Activating Token Index: 721

Text #5

Max Range: 2.5944. Min Range: -2.5944

Max Act: 1.9373. Min Act: -0.0000

Data Index: 390029 (C4 (Web Text))

Max Activating Token Index: 260

Text #6

Max Range: 2.5944. Min Range: -2.5944

Max Act: 1.8745. Min Act: -0.0000

Data Index: 890459 (C4 (Web Text))

Max Activating Token Index: 707

Text #7

Max Range: 2.5944. Min Range: -2.5944

Max Act: 1.8479. Min Act: -0.0000

Data Index: 86462 (C4 (Web Text))

Max Activating Token Index: 59

Text #8

Max Range: 2.5944. Min Range: -2.5944

Max Act: 1.7759. Min Act: -0.0000

Data Index: 1006975 (C4 (Web Text))

Max Activating Token Index: 979

Text #9

Max Range: 2.5944. Min Range: -2.5944

Max Act: 1.7439. Min Act: -0.0000

Data Index: 1313 (C4 (Web Text))

Max Activating Token Index: 491

Text #10

Max Range: 2.5944. Min Range: -2.5944

Max Act: 1.7474. Min Act: -0.0000

Data Index: 1125280 (C4 (Web Text))

Max Activating Token Index: 110

Text #11

Max Range: 2.5944. Min Range: -2.5944

Max Act: 1.5943. Min Act: -0.0000

Data Index: 357695 (C4 (Web Text))

Max Activating Token Index: 387

Text #12

Max Range: 2.5944. Min Range: -2.5944

Max Act: 1.6542. Min Act: -0.0000

Data Index: 1250594 (C4 (Web Text))

Max Activating Token Index: 796

Text #13

Max Range: 2.5944. Min Range: -2.5944

Max Act: 1.6308. Min Act: -0.0000

Data Index: 1152130 (C4 (Web Text))

Max Activating Token Index: 208

Text #14

Max Range: 2.5944. Min Range: -2.5944

Max Act: 1.6859. Min Act: -0.0000

Data Index: 915543 (C4 (Web Text))

Max Activating Token Index: 711

Text #15

Max Range: 2.5944. Min Range: -2.5944

Max Act: 1.5807. Min Act: -0.0000

Data Index: 158992 (C4 (Web Text))

Max Activating Token Index: 126

Text #16

Max Range: 2.5944. Min Range: -2.5944

Max Act: 1.5834. Min Act: -0.0000

Data Index: 397887 (C4 (Web Text))

Max Activating Token Index: 568

Text #17

Max Range: 2.5944. Min Range: -2.5944

Max Act: 1.5567. Min Act: -0.0000

Data Index: 178122 (C4 (Web Text))

Max Activating Token Index: 208

Text #18

Max Range: 2.5944. Min Range: -2.5944

Max Act: 1.5358. Min Act: -0.0000

Data Index: 722027 (C4 (Web Text))

Max Activating Token Index: 639

Text #19

Max Range: 2.5944. Min Range: -2.5944

Max Act: 1.5330. Min Act: -0.0000

Data Index: 1048734 (C4 (Web Text))

Max Activating Token Index: 349