Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5361 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.3785. Min Range: -1.3785

Max Act: 1.3785. Min Act: -0.0000

Data Index: 605993 (C4 (Web Text))

Max Activating Token Index: 223

Text #1

Max Range: 1.3785. Min Range: -1.3785

Max Act: 1.3444. Min Act: -0.0000

Data Index: 564639 (C4 (Web Text))

Max Activating Token Index: 81

Text #2

Max Range: 1.3785. Min Range: -1.3785

Max Act: 1.2627. Min Act: -0.0000

Data Index: 136725 (C4 (Web Text))

Max Activating Token Index: 705

Text #3

Max Range: 1.3785. Min Range: -1.3785

Max Act: 1.2966. Min Act: -0.0000

Data Index: 162810 (C4 (Web Text))

Max Activating Token Index: 480

Text #4

Max Range: 1.3785. Min Range: -1.3785

Max Act: 1.0889. Min Act: -0.0000

Data Index: 940402 (C4 (Web Text))

Max Activating Token Index: 618

Text #5

Max Range: 1.3785. Min Range: -1.3785

Max Act: 1.2006. Min Act: -0.0000

Data Index: 477393 (C4 (Web Text))

Max Activating Token Index: 901

Text #6

Max Range: 1.3785. Min Range: -1.3785

Max Act: 1.1605. Min Act: -0.0000

Data Index: 110287 (C4 (Web Text))

Max Activating Token Index: 209

Text #7

Max Range: 1.3785. Min Range: -1.3785

Max Act: 1.0878. Min Act: -0.0000

Data Index: 388108 (C4 (Web Text))

Max Activating Token Index: 339

Text #8

Max Range: 1.3785. Min Range: -1.3785

Max Act: 1.1101. Min Act: -0.0000

Data Index: 1113077 (C4 (Web Text))

Max Activating Token Index: 979

Text #9

Max Range: 1.3785. Min Range: -1.3785

Max Act: 1.1362. Min Act: -0.0000

Data Index: 753570 (C4 (Web Text))

Max Activating Token Index: 818

Text #10

Max Range: 1.3785. Min Range: -1.3785

Max Act: 1.0501. Min Act: -0.0000

Data Index: 843315 (C4 (Web Text))

Max Activating Token Index: 977

Text #11

Max Range: 1.3785. Min Range: -1.3785

Max Act: 1.0614. Min Act: -0.0000

Data Index: 836791 (C4 (Web Text))

Max Activating Token Index: 245

Text #12

Max Range: 1.3785. Min Range: -1.3785

Max Act: 1.0499. Min Act: -0.0000

Data Index: 1137745 (C4 (Web Text))

Max Activating Token Index: 982

Text #13

Max Range: 1.3785. Min Range: -1.3785

Max Act: 0.9715. Min Act: -0.0000

Data Index: 611308 (C4 (Web Text))

Max Activating Token Index: 1000

Text #14

Max Range: 1.3785. Min Range: -1.3785

Max Act: 1.0134. Min Act: -0.0000

Data Index: 683839 (C4 (Web Text))

Max Activating Token Index: 239

Text #15

Max Range: 1.3785. Min Range: -1.3785

Max Act: 0.9384. Min Act: -0.0000

Data Index: 1034703 (C4 (Web Text))

Max Activating Token Index: 9

Text #16

Max Range: 1.3785. Min Range: -1.3785

Max Act: 0.9109. Min Act: -0.0000

Data Index: 186254 (C4 (Web Text))

Max Activating Token Index: 760

Text #17

Max Range: 1.3785. Min Range: -1.3785

Max Act: 0.7726. Min Act: -0.0000

Data Index: 516476 (C4 (Web Text))

Max Activating Token Index: 845

Text #18

Max Range: 1.3785. Min Range: -1.3785

Max Act: 0.9415. Min Act: -0.0000

Data Index: 1300188 (C4 (Web Text))

Max Activating Token Index: 389

Text #19

Max Range: 1.3785. Min Range: -1.3785

Max Act: 0.9314. Min Act: -0.0000

Data Index: 830939 (C4 (Web Text))

Max Activating Token Index: 68