Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1532 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 5.2706. Min Range: -5.2706

Max Act: 5.2706. Min Act: -0.0001

Data Index: 756320 (C4 (Web Text))

Max Activating Token Index: 860

Text #1

Max Range: 5.2706. Min Range: -5.2706

Max Act: 4.7010. Min Act: -0.0000

Data Index: 1187701 (C4 (Web Text))

Max Activating Token Index: 171

Text #2

Max Range: 5.2706. Min Range: -5.2706

Max Act: 4.1379. Min Act: -0.0001

Data Index: 796127 (C4 (Web Text))

Max Activating Token Index: 310

Text #3

Max Range: 5.2706. Min Range: -5.2706

Max Act: 4.1356. Min Act: -0.0001

Data Index: 750391 (C4 (Web Text))

Max Activating Token Index: 793

Text #4

Max Range: 5.2706. Min Range: -5.2706

Max Act: 4.2449. Min Act: -0.0001

Data Index: 655003 (C4 (Web Text))

Max Activating Token Index: 986

Text #5

Max Range: 5.2706. Min Range: -5.2706

Max Act: 4.2085. Min Act: -0.0001

Data Index: 966172 (C4 (Web Text))

Max Activating Token Index: 314

Text #6

Max Range: 5.2706. Min Range: -5.2706

Max Act: 4.0257. Min Act: -0.0001

Data Index: 821699 (C4 (Web Text))

Max Activating Token Index: 225

Text #7

Max Range: 5.2706. Min Range: -5.2706

Max Act: 3.9998. Min Act: -0.0001

Data Index: 536960 (C4 (Web Text))

Max Activating Token Index: 658

Text #8

Max Range: 5.2706. Min Range: -5.2706

Max Act: 3.6726. Min Act: -0.0001

Data Index: 460830 (C4 (Web Text))

Max Activating Token Index: 644

Text #9

Max Range: 5.2706. Min Range: -5.2706

Max Act: 3.7551. Min Act: -0.0000

Data Index: 372185 (C4 (Web Text))

Max Activating Token Index: 921

Text #10

Max Range: 5.2706. Min Range: -5.2706

Max Act: 3.4873. Min Act: -0.0001

Data Index: 374089 (C4 (Web Text))

Max Activating Token Index: 815

Text #11

Max Range: 5.2706. Min Range: -5.2706

Max Act: 3.7166. Min Act: -0.0001

Data Index: 168568 (C4 (Web Text))

Max Activating Token Index: 553

Text #12

Max Range: 5.2706. Min Range: -5.2706

Max Act: 3.4820. Min Act: -0.0000

Data Index: 1096920 (C4 (Web Text))

Max Activating Token Index: 584

Text #13

Max Range: 5.2706. Min Range: -5.2706

Max Act: 3.5370. Min Act: -0.0001

Data Index: 939834 (C4 (Web Text))

Max Activating Token Index: 596

Text #14

Max Range: 5.2706. Min Range: -5.2706

Max Act: 3.5452. Min Act: -0.0001

Data Index: 657231 (C4 (Web Text))

Max Activating Token Index: 785

Text #15

Max Range: 5.2706. Min Range: -5.2706

Max Act: 3.6514. Min Act: -0.0001

Data Index: 32087 (C4 (Web Text))

Max Activating Token Index: 694

Text #16

Max Range: 5.2706. Min Range: -5.2706

Max Act: 3.4009. Min Act: -0.0001

Data Index: 612708 (C4 (Web Text))

Max Activating Token Index: 495

Text #17

Max Range: 5.2706. Min Range: -5.2706

Max Act: 3.3516. Min Act: -0.0001

Data Index: 1185150 (C4 (Web Text))

Max Activating Token Index: 490

Text #18

Max Range: 5.2706. Min Range: -5.2706

Max Act: 3.4765. Min Act: -0.0000

Data Index: 1295089 (C4 (Web Text))

Max Activating Token Index: 42

Text #19

Max Range: 5.2706. Min Range: -5.2706

Max Act: 3.5444. Min Act: -0.0001

Data Index: 682875 (C4 (Web Text))

Max Activating Token Index: 538