Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 957 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.8153. Min Range: -0.8153

Max Act: 0.8153. Min Act: -0.0000

Data Index: 88910 (C4 (Web Text))

Max Activating Token Index: 109

Text #1

Max Range: 0.8153. Min Range: -0.8153

Max Act: 0.7081. Min Act: -0.0000

Data Index: 936933 (C4 (Web Text))

Max Activating Token Index: 785

Text #2

Max Range: 0.8153. Min Range: -0.8153

Max Act: 0.5646. Min Act: -0.0000

Data Index: 1309178 (C4 (Web Text))

Max Activating Token Index: 34

Text #3

Max Range: 0.8153. Min Range: -0.8153

Max Act: 0.4694. Min Act: -0.0000

Data Index: 951468 (C4 (Web Text))

Max Activating Token Index: 274

Text #4

Max Range: 0.8153. Min Range: -0.8153

Max Act: 0.4578. Min Act: -0.0000

Data Index: 729378 (C4 (Web Text))

Max Activating Token Index: 353

Text #5

Max Range: 0.8153. Min Range: -0.8153

Max Act: 0.4619. Min Act: -0.0000

Data Index: 461226 (C4 (Web Text))

Max Activating Token Index: 487

Text #6

Max Range: 0.8153. Min Range: -0.8153

Max Act: 0.4322. Min Act: -0.0000

Data Index: 166128 (C4 (Web Text))

Max Activating Token Index: 305

Text #7

Max Range: 0.8153. Min Range: -0.8153

Max Act: 0.4435. Min Act: -0.0000

Data Index: 297541 (C4 (Web Text))

Max Activating Token Index: 84

Text #8

Max Range: 0.8153. Min Range: -0.8153

Max Act: 0.4437. Min Act: -0.0000

Data Index: 49037 (C4 (Web Text))

Max Activating Token Index: 217

Text #9

Max Range: 0.8153. Min Range: -0.8153

Max Act: 0.4483. Min Act: -0.0000

Data Index: 1056779 (C4 (Web Text))

Max Activating Token Index: 677

Text #10

Max Range: 0.8153. Min Range: -0.8153

Max Act: 0.4047. Min Act: -0.0000

Data Index: 846126 (C4 (Web Text))

Max Activating Token Index: 612

Text #11

Max Range: 0.8153. Min Range: -0.8153

Max Act: 0.3993. Min Act: -0.0000

Data Index: 477323 (C4 (Web Text))

Max Activating Token Index: 596

Text #12

Max Range: 0.8153. Min Range: -0.8153

Max Act: 0.3985. Min Act: -0.0000

Data Index: 97727 (C4 (Web Text))

Max Activating Token Index: 523

Text #13

Max Range: 0.8153. Min Range: -0.8153

Max Act: 0.3944. Min Act: -0.0000

Data Index: 615538 (C4 (Web Text))

Max Activating Token Index: 656

Text #14

Max Range: 0.8153. Min Range: -0.8153

Max Act: 0.3667. Min Act: -0.0000

Data Index: 785435 (C4 (Web Text))

Max Activating Token Index: 478

Text #15

Max Range: 0.8153. Min Range: -0.8153

Max Act: 0.3774. Min Act: -0.0000

Data Index: 384830 (C4 (Web Text))

Max Activating Token Index: 639

Text #16

Max Range: 0.8153. Min Range: -0.8153

Max Act: 0.4114. Min Act: -0.0000

Data Index: 793001 (C4 (Web Text))

Max Activating Token Index: 375

Text #17

Max Range: 0.8153. Min Range: -0.8153

Max Act: 0.3904. Min Act: -0.0000

Data Index: 809489 (C4 (Web Text))

Max Activating Token Index: 170

Text #18

Max Range: 0.8153. Min Range: -0.8153

Max Act: 0.3839. Min Act: -0.0000

Data Index: 316418 (C4 (Web Text))

Max Activating Token Index: 244

Text #19

Max Range: 0.8153. Min Range: -0.8153

Max Act: 0.3895. Min Act: -0.0000

Data Index: 1321083 (C4 (Web Text))

Max Activating Token Index: 97