Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5523 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.4007. Min Range: -3.4007

Max Act: 3.4007. Min Act: -0.0000

Data Index: 965805 (C4 (Web Text))

Max Activating Token Index: 371

Text #1

Max Range: 3.4007. Min Range: -3.4007

Max Act: 2.5846. Min Act: -0.0000

Data Index: 424614 (C4 (Web Text))

Max Activating Token Index: 574

Text #2

Max Range: 3.4007. Min Range: -3.4007

Max Act: 2.5401. Min Act: -0.0000

Data Index: 872473 (C4 (Web Text))

Max Activating Token Index: 933

Text #3

Max Range: 3.4007. Min Range: -3.4007

Max Act: 2.4133. Min Act: -0.0000

Data Index: 1118317 (C4 (Web Text))

Max Activating Token Index: 4

Text #4

Max Range: 3.4007. Min Range: -3.4007

Max Act: 2.4150. Min Act: -0.0000

Data Index: 500639 (C4 (Web Text))

Max Activating Token Index: 795

Text #5

Max Range: 3.4007. Min Range: -3.4007

Max Act: 2.4067. Min Act: -0.0000

Data Index: 467837 (C4 (Web Text))

Max Activating Token Index: 656

Text #6

Max Range: 3.4007. Min Range: -3.4007

Max Act: 2.2689. Min Act: -0.0000

Data Index: 1310734 (C4 (Web Text))

Max Activating Token Index: 102

Text #7

Max Range: 3.4007. Min Range: -3.4007

Max Act: 2.7921. Min Act: -0.0000

Data Index: 508366 (C4 (Web Text))

Max Activating Token Index: 832

Text #8

Max Range: 3.4007. Min Range: -3.4007

Max Act: 2.3700. Min Act: -0.0000

Data Index: 1243890 (C4 (Web Text))

Max Activating Token Index: 905

Text #9

Max Range: 3.4007. Min Range: -3.4007

Max Act: 2.3334. Min Act: -0.0000

Data Index: 684136 (C4 (Web Text))

Max Activating Token Index: 375

Text #10

Max Range: 3.4007. Min Range: -3.4007

Max Act: 2.4512. Min Act: -0.0000

Data Index: 906219 (C4 (Web Text))

Max Activating Token Index: 534

Text #11

Max Range: 3.4007. Min Range: -3.4007

Max Act: 2.4770. Min Act: -0.0000

Data Index: 52285 (C4 (Web Text))

Max Activating Token Index: 148

Text #12

Max Range: 3.4007. Min Range: -3.4007

Max Act: 2.2555. Min Act: -0.0000

Data Index: 617293 (C4 (Web Text))

Max Activating Token Index: 827

Text #13

Max Range: 3.4007. Min Range: -3.4007

Max Act: 2.2422. Min Act: -0.0000

Data Index: 337245 (C4 (Web Text))

Max Activating Token Index: 342

Text #14

Max Range: 3.4007. Min Range: -3.4007

Max Act: 2.5008. Min Act: -0.0000

Data Index: 42592 (C4 (Web Text))

Max Activating Token Index: 146

Text #15

Max Range: 3.4007. Min Range: -3.4007

Max Act: 2.0697. Min Act: -0.0000

Data Index: 356310 (C4 (Web Text))

Max Activating Token Index: 31

Text #16

Max Range: 3.4007. Min Range: -3.4007

Max Act: 2.4138. Min Act: -0.0000

Data Index: 941987 (C4 (Web Text))

Max Activating Token Index: 320

Text #17

Max Range: 3.4007. Min Range: -3.4007

Max Act: 2.0424. Min Act: -0.0000

Data Index: 1080562 (C4 (Web Text))

Max Activating Token Index: 283

Text #18

Max Range: 3.4007. Min Range: -3.4007

Max Act: 2.0355. Min Act: -0.0000

Data Index: 832582 (C4 (Web Text))

Max Activating Token Index: 272

Text #19

Max Range: 3.4007. Min Range: -3.4007

Max Act: 2.1323. Min Act: -0.0000

Data Index: 905972 (C4 (Web Text))

Max Activating Token Index: 23