Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 580 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.7993. Min Range: -1.7993

Max Act: 1.7993. Min Act: -0.0000

Data Index: 1059497 (C4 (Web Text))

Max Activating Token Index: 247

Text #1

Max Range: 1.7993. Min Range: -1.7993

Max Act: 1.4678. Min Act: -0.0000

Data Index: 436956 (C4 (Web Text))

Max Activating Token Index: 952

Text #2

Max Range: 1.7993. Min Range: -1.7993

Max Act: 1.2487. Min Act: -0.0000

Data Index: 1161062 (C4 (Web Text))

Max Activating Token Index: 1022

Text #3

Max Range: 1.7993. Min Range: -1.7993

Max Act: 1.2477. Min Act: -0.0000

Data Index: 889132 (C4 (Web Text))

Max Activating Token Index: 768

Text #4

Max Range: 1.7993. Min Range: -1.7993

Max Act: 1.1611. Min Act: -0.0000

Data Index: 818524 (C4 (Web Text))

Max Activating Token Index: 162

Text #5

Max Range: 1.7993. Min Range: -1.7993

Max Act: 1.1646. Min Act: -0.0000

Data Index: 184788 (C4 (Web Text))

Max Activating Token Index: 419

Text #6

Max Range: 1.7993. Min Range: -1.7993

Max Act: 1.2314. Min Act: -0.0000

Data Index: 374340 (C4 (Web Text))

Max Activating Token Index: 764

Text #7

Max Range: 1.7993. Min Range: -1.7993

Max Act: 1.0851. Min Act: -0.0000

Data Index: 977726 (C4 (Web Text))

Max Activating Token Index: 144

Text #8

Max Range: 1.7993. Min Range: -1.7993

Max Act: 1.1044. Min Act: -0.0000

Data Index: 244845 (C4 (Web Text))

Max Activating Token Index: 590

Text #9

Max Range: 1.7993. Min Range: -1.7993

Max Act: 1.1242. Min Act: -0.0000

Data Index: 231399 (C4 (Web Text))

Max Activating Token Index: 424

Text #10

Max Range: 1.7993. Min Range: -1.7993

Max Act: 1.0940. Min Act: -0.0000

Data Index: 1062854 (C4 (Web Text))

Max Activating Token Index: 699

Text #11

Max Range: 1.7993. Min Range: -1.7993

Max Act: 1.1210. Min Act: -0.0000

Data Index: 590901 (C4 (Web Text))

Max Activating Token Index: 780

Text #12

Max Range: 1.7993. Min Range: -1.7993

Max Act: 1.0740. Min Act: -0.0000

Data Index: 1191213 (C4 (Web Text))

Max Activating Token Index: 917

Text #13

Max Range: 1.7993. Min Range: -1.7993

Max Act: 1.0511. Min Act: -0.0000

Data Index: 546534 (C4 (Web Text))

Max Activating Token Index: 453

Text #14

Max Range: 1.7993. Min Range: -1.7993

Max Act: 1.0716. Min Act: -0.0000

Data Index: 776061 (C4 (Web Text))

Max Activating Token Index: 510

Text #15

Max Range: 1.7993. Min Range: -1.7993

Max Act: 1.1623. Min Act: -0.0000

Data Index: 526243 (C4 (Web Text))

Max Activating Token Index: 271

Text #16

Max Range: 1.7993. Min Range: -1.7993

Max Act: 1.1165. Min Act: -0.0000

Data Index: 1296343 (C4 (Web Text))

Max Activating Token Index: 105

Text #17

Max Range: 1.7993. Min Range: -1.7993

Max Act: 1.1013. Min Act: -0.0000

Data Index: 1158912 (C4 (Web Text))

Max Activating Token Index: 730

Text #18

Max Range: 1.7993. Min Range: -1.7993

Max Act: 0.9919. Min Act: -0.0000

Data Index: 857154 (C4 (Web Text))

Max Activating Token Index: 380

Text #19

Max Range: 1.7993. Min Range: -1.7993

Max Act: 0.9517. Min Act: -0.0001

Data Index: 209950 (C4 (Web Text))

Max Activating Token Index: 233