Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3373 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.3411. Min Range: -0.3411

Max Act: 0.3411. Min Act: -0.0002

Data Index: 992152 (C4 (Web Text))

Max Activating Token Index: 263

Text #1

Max Range: 0.3411. Min Range: -0.3411

Max Act: 0.3153. Min Act: -0.0002

Data Index: 935932 (C4 (Web Text))

Max Activating Token Index: 285

Text #2

Max Range: 0.3411. Min Range: -0.3411

Max Act: 0.3086. Min Act: -0.0002

Data Index: 931508 (C4 (Web Text))

Max Activating Token Index: 357

Text #3

Max Range: 0.3411. Min Range: -0.3411

Max Act: 0.2830. Min Act: -0.0002

Data Index: 138802 (C4 (Web Text))

Max Activating Token Index: 344

Text #4

Max Range: 0.3411. Min Range: -0.3411

Max Act: 0.2900. Min Act: -0.0002

Data Index: 596773 (C4 (Web Text))

Max Activating Token Index: 870

Text #5

Max Range: 0.3411. Min Range: -0.3411

Max Act: 0.2756. Min Act: -0.0002

Data Index: 796203 (C4 (Web Text))

Max Activating Token Index: 447

Text #6

Max Range: 0.3411. Min Range: -0.3411

Max Act: 0.2762. Min Act: -0.0002

Data Index: 1169485 (C4 (Web Text))

Max Activating Token Index: 159

Text #7

Max Range: 0.3411. Min Range: -0.3411

Max Act: 0.2686. Min Act: -0.0002

Data Index: 181991 (C4 (Web Text))

Max Activating Token Index: 258

Text #8

Max Range: 0.3411. Min Range: -0.3411

Max Act: 0.2572. Min Act: -0.0002

Data Index: 888307 (C4 (Web Text))

Max Activating Token Index: 279

Text #9

Max Range: 0.3411. Min Range: -0.3411

Max Act: 0.2572. Min Act: -0.0002

Data Index: 896880 (C4 (Web Text))

Max Activating Token Index: 341

Text #10

Max Range: 0.3411. Min Range: -0.3411

Max Act: 0.2583. Min Act: -0.0002

Data Index: 1286508 (C4 (Web Text))

Max Activating Token Index: 182

Text #11

Max Range: 0.3411. Min Range: -0.3411

Max Act: 0.2662. Min Act: -0.0002

Data Index: 1116642 (C4 (Web Text))

Max Activating Token Index: 345

Text #12

Max Range: 0.3411. Min Range: -0.3411

Max Act: 0.2558. Min Act: -0.0002

Data Index: 561412 (C4 (Web Text))

Max Activating Token Index: 470

Text #13

Max Range: 0.3411. Min Range: -0.3411

Max Act: 0.2463. Min Act: -0.0002

Data Index: 1348138 (C4 (Web Text))

Max Activating Token Index: 232

Text #14

Max Range: 0.3411. Min Range: -0.3411

Max Act: 0.2510. Min Act: -0.0002

Data Index: 645644 (C4 (Web Text))

Max Activating Token Index: 497

Text #15

Max Range: 0.3411. Min Range: -0.3411

Max Act: 0.2518. Min Act: -0.0002

Data Index: 964874 (C4 (Web Text))

Max Activating Token Index: 282

Text #16

Max Range: 0.3411. Min Range: -0.3411

Max Act: 0.2475. Min Act: -0.0002

Data Index: 580757 (C4 (Web Text))

Max Activating Token Index: 323

Text #17

Max Range: 0.3411. Min Range: -0.3411

Max Act: 0.2384. Min Act: -0.0002

Data Index: 211452 (C4 (Web Text))

Max Activating Token Index: 84

Text #18

Max Range: 0.3411. Min Range: -0.3411

Max Act: 0.2405. Min Act: -0.0002

Data Index: 1257628 (C4 (Web Text))

Max Activating Token Index: 263

Text #19

Max Range: 0.3411. Min Range: -0.3411

Max Act: 0.2524. Min Act: -0.0002

Data Index: 312096 (C4 (Web Text))

Max Activating Token Index: 638