Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3433 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.6751. Min Range: -0.6751

Max Act: 0.6751. Min Act: -0.0000

Data Index: 1017239 (C4 (Web Text))

Max Activating Token Index: 198

Text #1

Max Range: 0.6751. Min Range: -0.6751

Max Act: 0.4843. Min Act: -0.0000

Data Index: 67850 (C4 (Web Text))

Max Activating Token Index: 484

Text #2

Max Range: 0.6751. Min Range: -0.6751

Max Act: 0.4767. Min Act: -0.0000

Data Index: 616749 (C4 (Web Text))

Max Activating Token Index: 451

Text #3

Max Range: 0.6751. Min Range: -0.6751

Max Act: 0.4642. Min Act: -0.0000

Data Index: 426265 (C4 (Web Text))

Max Activating Token Index: 168

Text #4

Max Range: 0.6751. Min Range: -0.6751

Max Act: 0.4415. Min Act: -0.0000

Data Index: 1208598 (C4 (Web Text))

Max Activating Token Index: 85

Text #5

Max Range: 0.6751. Min Range: -0.6751

Max Act: 0.4360. Min Act: -0.0000

Data Index: 478110 (C4 (Web Text))

Max Activating Token Index: 192

Text #6

Max Range: 0.6751. Min Range: -0.6751

Max Act: 0.4775. Min Act: -0.0000

Data Index: 1311745 (C4 (Web Text))

Max Activating Token Index: 57

Text #7

Max Range: 0.6751. Min Range: -0.6751

Max Act: 0.4235. Min Act: -0.0000

Data Index: 943669 (C4 (Web Text))

Max Activating Token Index: 695

Text #8

Max Range: 0.6751. Min Range: -0.6751

Max Act: 0.3652. Min Act: -0.0000

Data Index: 879625 (C4 (Web Text))

Max Activating Token Index: 956

Text #9

Max Range: 0.6751. Min Range: -0.6751

Max Act: 0.3670. Min Act: -0.0000

Data Index: 857713 (C4 (Web Text))

Max Activating Token Index: 548

Text #10

Max Range: 0.6751. Min Range: -0.6751

Max Act: 0.3912. Min Act: -0.0000

Data Index: 529581 (C4 (Web Text))

Max Activating Token Index: 361

Text #11

Max Range: 0.6751. Min Range: -0.6751

Max Act: 0.4127. Min Act: -0.0000

Data Index: 725198 (C4 (Web Text))

Max Activating Token Index: 387

Text #12

Max Range: 0.6751. Min Range: -0.6751

Max Act: 0.3879. Min Act: -0.0000

Data Index: 892918 (C4 (Web Text))

Max Activating Token Index: 579

Text #13

Max Range: 0.6751. Min Range: -0.6751

Max Act: 0.3494. Min Act: -0.0000

Data Index: 510008 (C4 (Web Text))

Max Activating Token Index: 246

Text #14

Max Range: 0.6751. Min Range: -0.6751

Max Act: 0.3801. Min Act: -0.0000

Data Index: 1221811 (C4 (Web Text))

Max Activating Token Index: 598

Text #15

Max Range: 0.6751. Min Range: -0.6751

Max Act: 0.3496. Min Act: -0.0000

Data Index: 25468 (C4 (Web Text))

Max Activating Token Index: 259

Text #16

Max Range: 0.6751. Min Range: -0.6751

Max Act: 0.3708. Min Act: -0.0000

Data Index: 1118084 (C4 (Web Text))

Max Activating Token Index: 188

Text #17

Max Range: 0.6751. Min Range: -0.6751

Max Act: 0.3586. Min Act: -0.0000

Data Index: 282043 (C4 (Web Text))

Max Activating Token Index: 341

Text #18

Max Range: 0.6751. Min Range: -0.6751

Max Act: 0.4311. Min Act: -0.0000

Data Index: 1212034 (C4 (Web Text))

Max Activating Token Index: 719

Text #19

Max Range: 0.6751. Min Range: -0.6751

Max Act: 0.3648. Min Act: -0.0000

Data Index: 457754 (C4 (Web Text))

Max Activating Token Index: 659