Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3818 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.9035. Min Range: -1.9035

Max Act: 1.9035. Min Act: -0.0000

Data Index: 348356 (C4 (Web Text))

Max Activating Token Index: 409

Text #1

Max Range: 1.9035. Min Range: -1.9035

Max Act: 1.4095. Min Act: -0.0000

Data Index: 484847 (C4 (Web Text))

Max Activating Token Index: 334

Text #2

Max Range: 1.9035. Min Range: -1.9035

Max Act: 1.4715. Min Act: -0.0000

Data Index: 96564 (C4 (Web Text))

Max Activating Token Index: 294

Text #3

Max Range: 1.9035. Min Range: -1.9035

Max Act: 1.3474. Min Act: -0.0000

Data Index: 229607 (C4 (Web Text))

Max Activating Token Index: 224

Text #4

Max Range: 1.9035. Min Range: -1.9035

Max Act: 1.3274. Min Act: -0.0000

Data Index: 899543 (C4 (Web Text))

Max Activating Token Index: 415

Text #5

Max Range: 1.9035. Min Range: -1.9035

Max Act: 1.2830. Min Act: -0.0000

Data Index: 43316 (C4 (Web Text))

Max Activating Token Index: 315

Text #6

Max Range: 1.9035. Min Range: -1.9035

Max Act: 1.1850. Min Act: -0.0000

Data Index: 1071947 (C4 (Web Text))

Max Activating Token Index: 904

Text #7

Max Range: 1.9035. Min Range: -1.9035

Max Act: 1.2384. Min Act: -0.0000

Data Index: 559215 (C4 (Web Text))

Max Activating Token Index: 537

Text #8

Max Range: 1.9035. Min Range: -1.9035

Max Act: 1.2607. Min Act: -0.0000

Data Index: 488229 (C4 (Web Text))

Max Activating Token Index: 96

Text #9

Max Range: 1.9035. Min Range: -1.9035

Max Act: 1.2804. Min Act: -0.0000

Data Index: 1015911 (C4 (Web Text))

Max Activating Token Index: 412

Text #10

Max Range: 1.9035. Min Range: -1.9035

Max Act: 1.1518. Min Act: -0.0000

Data Index: 134086 (C4 (Web Text))

Max Activating Token Index: 410

Text #11

Max Range: 1.9035. Min Range: -1.9035

Max Act: 1.1810. Min Act: -0.0000

Data Index: 82699 (C4 (Web Text))

Max Activating Token Index: 68

Text #12

Max Range: 1.9035. Min Range: -1.9035

Max Act: 1.1855. Min Act: -0.0000

Data Index: 337842 (C4 (Web Text))

Max Activating Token Index: 89

Text #13

Max Range: 1.9035. Min Range: -1.9035

Max Act: 1.0791. Min Act: -0.0000

Data Index: 831351 (C4 (Web Text))

Max Activating Token Index: 382

Text #14

Max Range: 1.9035. Min Range: -1.9035

Max Act: 0.9793. Min Act: -0.0000

Data Index: 228820 (C4 (Web Text))

Max Activating Token Index: 310

Text #15

Max Range: 1.9035. Min Range: -1.9035

Max Act: 1.1163. Min Act: -0.0000

Data Index: 850954 (C4 (Web Text))

Max Activating Token Index: 585

Text #16

Max Range: 1.9035. Min Range: -1.9035

Max Act: 1.2548. Min Act: -0.0000

Data Index: 696119 (C4 (Web Text))

Max Activating Token Index: 235

Text #17

Max Range: 1.9035. Min Range: -1.9035

Max Act: 1.1045. Min Act: -0.0000

Data Index: 1298124 (C4 (Web Text))

Max Activating Token Index: 188

Text #18

Max Range: 1.9035. Min Range: -1.9035

Max Act: 1.1746. Min Act: -0.0000

Data Index: 223169 (C4 (Web Text))

Max Activating Token Index: 517

Text #19

Max Range: 1.9035. Min Range: -1.9035

Max Act: 1.0652. Min Act: -0.0000

Data Index: 943742 (C4 (Web Text))

Max Activating Token Index: 832