Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2105 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('solu-12l')

Click the toggle to see the full text!

Text #0

Max Range: 0.7707. Min Range: -0.7707

Max Act: 0.7707. Min Act: -0.0002

Data Index: 296696 (C4 (Web Text))

Max Activating Token Index: 669

Text #1

Max Range: 0.7707. Min Range: -0.7707

Max Act: 0.7377. Min Act: -0.0002

Data Index: 915545 (C4 (Web Text))

Max Activating Token Index: 448

Text #2

Max Range: 0.7707. Min Range: -0.7707

Max Act: 0.7082. Min Act: -0.0002

Data Index: 514809 (C4 (Web Text))

Max Activating Token Index: 623

Text #3

Max Range: 0.7707. Min Range: -0.7707

Max Act: 0.7150. Min Act: -0.0002

Data Index: 1230750 (C4 (Web Text))

Max Activating Token Index: 814

Text #4

Max Range: 0.7707. Min Range: -0.7707

Max Act: 0.7001. Min Act: -0.0002

Data Index: 893520 (C4 (Web Text))

Max Activating Token Index: 390

Text #5

Max Range: 0.7707. Min Range: -0.7707

Max Act: 0.7042. Min Act: -0.0002

Data Index: 382538 (C4 (Web Text))

Max Activating Token Index: 612

Text #6

Max Range: 0.7707. Min Range: -0.7707

Max Act: 0.6935. Min Act: -0.0002

Data Index: 960585 (C4 (Web Text))

Max Activating Token Index: 760

Text #7

Max Range: 0.7707. Min Range: -0.7707

Max Act: 0.6766. Min Act: -0.0002

Data Index: 471812 (C4 (Web Text))

Max Activating Token Index: 145

Text #8

Max Range: 0.7707. Min Range: -0.7707

Max Act: 0.6720. Min Act: -0.0002

Data Index: 653524 (C4 (Web Text))

Max Activating Token Index: 910

Text #9

Max Range: 0.7707. Min Range: -0.7707

Max Act: 0.6764. Min Act: -0.0002

Data Index: 326681 (C4 (Web Text))

Max Activating Token Index: 515

Text #10

Max Range: 0.7707. Min Range: -0.7707

Max Act: 0.6778. Min Act: -0.0002

Data Index: 1264930 (C4 (Web Text))

Max Activating Token Index: 164

Text #11

Max Range: 0.7707. Min Range: -0.7707

Max Act: 0.6672. Min Act: -0.0002

Data Index: 527318 (C4 (Web Text))

Max Activating Token Index: 1013

Text #12

Max Range: 0.7707. Min Range: -0.7707

Max Act: 0.6500. Min Act: -0.0002

Data Index: 402262 (C4 (Web Text))

Max Activating Token Index: 950

Text #13

Max Range: 0.7707. Min Range: -0.7707

Max Act: 0.6692. Min Act: -0.0002

Data Index: 149290 (C4 (Web Text))

Max Activating Token Index: 621

Text #14

Max Range: 0.7707. Min Range: -0.7707

Max Act: 0.6567. Min Act: -0.0002

Data Index: 159962 (C4 (Web Text))

Max Activating Token Index: 884

Text #15

Max Range: 0.7707. Min Range: -0.7707

Max Act: 0.6493. Min Act: -0.0002

Data Index: 54652 (C4 (Web Text))

Max Activating Token Index: 381

Text #16

Max Range: 0.7707. Min Range: -0.7707

Max Act: 0.6466. Min Act: -0.0002

Data Index: 1036817 (C4 (Web Text))

Max Activating Token Index: 45

Text #17

Max Range: 0.7707. Min Range: -0.7707

Max Act: 0.6430. Min Act: -0.0002

Data Index: 285215 (C4 (Web Text))

Max Activating Token Index: 683

Text #18

Max Range: 0.7707. Min Range: -0.7707

Max Act: 0.6446. Min Act: -0.0002

Data Index: 1192204 (C4 (Web Text))

Max Activating Token Index: 956

Text #19

Max Range: 0.7707. Min Range: -0.7707

Max Act: 0.6370. Min Act: -0.0002

Data Index: 372312 (C4 (Web Text))

Max Activating Token Index: 100