Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 864 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.5847. Min Range: -1.5847

Max Act: 1.5847. Min Act: -0.0000

Data Index: 410755 (C4 (Web Text))

Max Activating Token Index: 137

Text #1

Max Range: 1.5847. Min Range: -1.5847

Max Act: 1.3132. Min Act: -0.0000

Data Index: 1035478 (C4 (Web Text))

Max Activating Token Index: 399

Text #2

Max Range: 1.5847. Min Range: -1.5847

Max Act: 1.4173. Min Act: 0.0000

Data Index: 474995 (C4 (Web Text))

Max Activating Token Index: 224

Text #3

Max Range: 1.5847. Min Range: -1.5847

Max Act: 1.3659. Min Act: -0.0000

Data Index: 802526 (C4 (Web Text))

Max Activating Token Index: 111

Text #4

Max Range: 1.5847. Min Range: -1.5847

Max Act: 1.2644. Min Act: -0.0000

Data Index: 1200196 (C4 (Web Text))

Max Activating Token Index: 729

Text #5

Max Range: 1.5847. Min Range: -1.5847

Max Act: 1.2700. Min Act: -0.0000

Data Index: 251723 (C4 (Web Text))

Max Activating Token Index: 305

Text #6

Max Range: 1.5847. Min Range: -1.5847

Max Act: 1.1764. Min Act: -0.0000

Data Index: 647517 (C4 (Web Text))

Max Activating Token Index: 201

Text #7

Max Range: 1.5847. Min Range: -1.5847

Max Act: 1.2339. Min Act: -0.0000

Data Index: 33062 (C4 (Web Text))

Max Activating Token Index: 730

Text #8

Max Range: 1.5847. Min Range: -1.5847

Max Act: 1.2281. Min Act: -0.0000

Data Index: 1312961 (C4 (Web Text))

Max Activating Token Index: 391

Text #9

Max Range: 1.5847. Min Range: -1.5847

Max Act: 1.1688. Min Act: -0.0000

Data Index: 609574 (C4 (Web Text))

Max Activating Token Index: 490

Text #10

Max Range: 1.5847. Min Range: -1.5847

Max Act: 1.1151. Min Act: -0.0000

Data Index: 465363 (C4 (Web Text))

Max Activating Token Index: 585

Text #11

Max Range: 1.5847. Min Range: -1.5847

Max Act: 1.0801. Min Act: -0.0000

Data Index: 855586 (C4 (Web Text))

Max Activating Token Index: 170

Text #12

Max Range: 1.5847. Min Range: -1.5847

Max Act: 1.0348. Min Act: -0.0000

Data Index: 1353884 (C4 (Web Text))

Max Activating Token Index: 238

Text #13

Max Range: 1.5847. Min Range: -1.5847

Max Act: 1.0535. Min Act: -0.0000

Data Index: 175465 (C4 (Web Text))

Max Activating Token Index: 168

Text #14

Max Range: 1.5847. Min Range: -1.5847

Max Act: 1.0046. Min Act: -0.0000

Data Index: 82698 (C4 (Web Text))

Max Activating Token Index: 654

Text #15

Max Range: 1.5847. Min Range: -1.5847

Max Act: 1.0465. Min Act: 0.0000

Data Index: 1340920 (C4 (Web Text))

Max Activating Token Index: 535

Text #16

Max Range: 1.5847. Min Range: -1.5847

Max Act: 1.0126. Min Act: -0.0000

Data Index: 847474 (C4 (Web Text))

Max Activating Token Index: 42

Text #17

Max Range: 1.5847. Min Range: -1.5847

Max Act: 1.0045. Min Act: -0.0000

Data Index: 1042938 (C4 (Web Text))

Max Activating Token Index: 845

Text #18

Max Range: 1.5847. Min Range: -1.5847

Max Act: 1.0181. Min Act: -0.0000

Data Index: 1267953 (C4 (Web Text))

Max Activating Token Index: 803

Text #19

Max Range: 1.5847. Min Range: -1.5847

Max Act: 0.9574. Min Act: -0.0000

Data Index: 769673 (C4 (Web Text))

Max Activating Token Index: 297