Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1828 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.4444. Min Range: -0.4444

Max Act: 0.4157. Min Act: -0.0000

Data Index: 1328150 (C4 (Web Text))

Max Activating Token Index: 339

Text #1

Max Range: 0.4444. Min Range: -0.4444

Max Act: 0.4444. Min Act: -0.0000

Data Index: 293505 (C4 (Web Text))

Max Activating Token Index: 584

Text #2

Max Range: 0.4444. Min Range: -0.4444

Max Act: 0.4380. Min Act: -0.0000

Data Index: 93274 (C4 (Web Text))

Max Activating Token Index: 434

Text #3

Max Range: 0.4444. Min Range: -0.4444

Max Act: 0.4295. Min Act: -0.0000

Data Index: 573773 (C4 (Web Text))

Max Activating Token Index: 548

Text #4

Max Range: 0.4444. Min Range: -0.4444

Max Act: 0.3997. Min Act: -0.0000

Data Index: 141690 (C4 (Web Text))

Max Activating Token Index: 584

Text #5

Max Range: 0.4444. Min Range: -0.4444

Max Act: 0.3817. Min Act: -0.0000

Data Index: 1179866 (C4 (Web Text))

Max Activating Token Index: 202

Text #6

Max Range: 0.4444. Min Range: -0.4444

Max Act: 0.3885. Min Act: -0.0000

Data Index: 1340783 (C4 (Web Text))

Max Activating Token Index: 458

Text #7

Max Range: 0.4444. Min Range: -0.4444

Max Act: 0.3417. Min Act: -0.0000

Data Index: 147223 (C4 (Web Text))

Max Activating Token Index: 338

Text #8

Max Range: 0.4444. Min Range: -0.4444

Max Act: 0.4141. Min Act: -0.0000

Data Index: 1130363 (C4 (Web Text))

Max Activating Token Index: 739

Text #9

Max Range: 0.4444. Min Range: -0.4444

Max Act: 0.3827. Min Act: -0.0000

Data Index: 535793 (C4 (Web Text))

Max Activating Token Index: 906

Text #10

Max Range: 0.4444. Min Range: -0.4444

Max Act: 0.4030. Min Act: -0.0000

Data Index: 1241507 (C4 (Web Text))

Max Activating Token Index: 697

Text #11

Max Range: 0.4444. Min Range: -0.4444

Max Act: 0.3514. Min Act: -0.0000

Data Index: 419934 (C4 (Web Text))

Max Activating Token Index: 245

Text #12

Max Range: 0.4444. Min Range: -0.4444

Max Act: 0.3262. Min Act: -0.0000

Data Index: 455316 (C4 (Web Text))

Max Activating Token Index: 909

Text #13

Max Range: 0.4444. Min Range: -0.4444

Max Act: 0.4014. Min Act: -0.0000

Data Index: 931562 (C4 (Web Text))

Max Activating Token Index: 959

Text #14

Max Range: 0.4444. Min Range: -0.4444

Max Act: 0.3688. Min Act: -0.0000

Data Index: 1112779 (C4 (Web Text))

Max Activating Token Index: 805

Text #15

Max Range: 0.4444. Min Range: -0.4444

Max Act: 0.3394. Min Act: -0.0000

Data Index: 399476 (C4 (Web Text))

Max Activating Token Index: 87

Text #16

Max Range: 0.4444. Min Range: -0.4444

Max Act: 0.3754. Min Act: -0.0000

Data Index: 431802 (C4 (Web Text))

Max Activating Token Index: 903

Text #17

Max Range: 0.4444. Min Range: -0.4444

Max Act: 0.3356. Min Act: -0.0000

Data Index: 868089 (C4 (Web Text))

Max Activating Token Index: 831

Text #18

Max Range: 0.4444. Min Range: -0.4444

Max Act: 0.3493. Min Act: -0.0000

Data Index: 224685 (C4 (Web Text))

Max Activating Token Index: 460

Text #19

Max Range: 0.4444. Min Range: -0.4444

Max Act: 0.3336. Min Act: -0.0000

Data Index: 1143780 (C4 (Web Text))

Max Activating Token Index: 512