Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2296 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.3557. Min Range: -1.3557

Max Act: 1.3557. Min Act: -0.0000

Data Index: 679019 (C4 (Web Text))

Max Activating Token Index: 262

Text #1

Max Range: 1.3557. Min Range: -1.3557

Max Act: 1.1827. Min Act: -0.0000

Data Index: 1167123 (C4 (Web Text))

Max Activating Token Index: 161

Text #2

Max Range: 1.3557. Min Range: -1.3557

Max Act: 1.1205. Min Act: -0.0000

Data Index: 972987 (C4 (Web Text))

Max Activating Token Index: 144

Text #3

Max Range: 1.3557. Min Range: -1.3557

Max Act: 1.0428. Min Act: -0.0000

Data Index: 102323 (C4 (Web Text))

Max Activating Token Index: 165

Text #4

Max Range: 1.3557. Min Range: -1.3557

Max Act: 1.0331. Min Act: -0.0000

Data Index: 431536 (C4 (Web Text))

Max Activating Token Index: 900

Text #5

Max Range: 1.3557. Min Range: -1.3557

Max Act: 0.9774. Min Act: -0.0000

Data Index: 1222651 (C4 (Web Text))

Max Activating Token Index: 589

Text #6

Max Range: 1.3557. Min Range: -1.3557

Max Act: 1.0136. Min Act: -0.0000

Data Index: 1245866 (C4 (Web Text))

Max Activating Token Index: 378

Text #7

Max Range: 1.3557. Min Range: -1.3557

Max Act: 0.9492. Min Act: -0.0000

Data Index: 6704 (C4 (Web Text))

Max Activating Token Index: 293

Text #8

Max Range: 1.3557. Min Range: -1.3557

Max Act: 0.8315. Min Act: -0.0000

Data Index: 865933 (C4 (Web Text))

Max Activating Token Index: 298

Text #9

Max Range: 1.3557. Min Range: -1.3557

Max Act: 0.8774. Min Act: -0.0000

Data Index: 1306348 (C4 (Web Text))

Max Activating Token Index: 514

Text #10

Max Range: 1.3557. Min Range: -1.3557

Max Act: 0.8552. Min Act: -0.0000

Data Index: 957099 (C4 (Web Text))

Max Activating Token Index: 119

Text #11

Max Range: 1.3557. Min Range: -1.3557

Max Act: 0.8533. Min Act: -0.0000

Data Index: 1252249 (C4 (Web Text))

Max Activating Token Index: 829

Text #12

Max Range: 1.3557. Min Range: -1.3557

Max Act: 0.7908. Min Act: -0.0000

Data Index: 931620 (C4 (Web Text))

Max Activating Token Index: 225

Text #13

Max Range: 1.3557. Min Range: -1.3557

Max Act: 0.7885. Min Act: -0.0000

Data Index: 1180527 (C4 (Web Text))

Max Activating Token Index: 921

Text #14

Max Range: 1.3557. Min Range: -1.3557

Max Act: 0.7787. Min Act: -0.0000

Data Index: 375418 (C4 (Web Text))

Max Activating Token Index: 797

Text #15

Max Range: 1.3557. Min Range: -1.3557

Max Act: 0.7965. Min Act: -0.0000

Data Index: 1302723 (C4 (Web Text))

Max Activating Token Index: 84

Text #16

Max Range: 1.3557. Min Range: -1.3557

Max Act: 0.8142. Min Act: -0.0000

Data Index: 585829 (C4 (Web Text))

Max Activating Token Index: 290

Text #17

Max Range: 1.3557. Min Range: -1.3557

Max Act: 0.8160. Min Act: -0.0000

Data Index: 1081106 (C4 (Web Text))

Max Activating Token Index: 393

Text #18

Max Range: 1.3557. Min Range: -1.3557

Max Act: 0.7811. Min Act: -0.0000

Data Index: 207468 (C4 (Web Text))

Max Activating Token Index: 101

Text #19

Max Range: 1.3557. Min Range: -1.3557

Max Act: 0.7635. Min Act: -0.0000

Data Index: 1200170 (C4 (Web Text))

Max Activating Token Index: 26