Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3558 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.1906. Min Range: -1.1906

Max Act: 1.1873. Min Act: -0.0000

Data Index: 1175999 (C4 (Web Text))

Max Activating Token Index: 35

Text #1

Max Range: 1.1906. Min Range: -1.1906

Max Act: 1.1034. Min Act: -0.0000

Data Index: 244551 (C4 (Web Text))

Max Activating Token Index: 550

Text #2

Max Range: 1.1906. Min Range: -1.1906

Max Act: 1.1906. Min Act: -0.0000

Data Index: 367748 (C4 (Web Text))

Max Activating Token Index: 704

Text #3

Max Range: 1.1906. Min Range: -1.1906

Max Act: 1.0562. Min Act: -0.0000

Data Index: 200296 (C4 (Web Text))

Max Activating Token Index: 128

Text #4

Max Range: 1.1906. Min Range: -1.1906

Max Act: 1.0463. Min Act: -0.0000

Data Index: 653719 (C4 (Web Text))

Max Activating Token Index: 556

Text #5

Max Range: 1.1906. Min Range: -1.1906

Max Act: 1.0036. Min Act: -0.0000

Data Index: 802106 (C4 (Web Text))

Max Activating Token Index: 633

Text #6

Max Range: 1.1906. Min Range: -1.1906

Max Act: 1.0020. Min Act: -0.0000

Data Index: 920944 (C4 (Web Text))

Max Activating Token Index: 319

Text #7

Max Range: 1.1906. Min Range: -1.1906

Max Act: 1.0312. Min Act: -0.0000

Data Index: 952527 (C4 (Web Text))

Max Activating Token Index: 338

Text #8

Max Range: 1.1906. Min Range: -1.1906

Max Act: 0.9594. Min Act: -0.0000

Data Index: 928200 (C4 (Web Text))

Max Activating Token Index: 402

Text #9

Max Range: 1.1906. Min Range: -1.1906

Max Act: 0.9539. Min Act: -0.0000

Data Index: 562130 (C4 (Web Text))

Max Activating Token Index: 227

Text #10

Max Range: 1.1906. Min Range: -1.1906

Max Act: 1.0118. Min Act: -0.0000

Data Index: 1248504 (C4 (Web Text))

Max Activating Token Index: 154

Text #11

Max Range: 1.1906. Min Range: -1.1906

Max Act: 0.9807. Min Act: -0.0000

Data Index: 818796 (C4 (Web Text))

Max Activating Token Index: 747

Text #12

Max Range: 1.1906. Min Range: -1.1906

Max Act: 0.9675. Min Act: -0.0000

Data Index: 1151813 (C4 (Web Text))

Max Activating Token Index: 627

Text #13

Max Range: 1.1906. Min Range: -1.1906

Max Act: 0.8549. Min Act: -0.0000

Data Index: 830438 (C4 (Web Text))

Max Activating Token Index: 551

Text #14

Max Range: 1.1906. Min Range: -1.1906

Max Act: 0.8958. Min Act: -0.0000

Data Index: 13523 (C4 (Web Text))

Max Activating Token Index: 649

Text #15

Max Range: 1.1906. Min Range: -1.1906

Max Act: 0.8269. Min Act: -0.0000

Data Index: 1339862 (C4 (Web Text))

Max Activating Token Index: 640

Text #16

Max Range: 1.1906. Min Range: -1.1906

Max Act: 0.7191. Min Act: -0.0000

Data Index: 717062 (C4 (Web Text))

Max Activating Token Index: 494

Text #17

Max Range: 1.1906. Min Range: -1.1906

Max Act: 0.7658. Min Act: -0.0000

Data Index: 1308622 (C4 (Web Text))

Max Activating Token Index: 175

Text #18

Max Range: 1.1906. Min Range: -1.1906

Max Act: 0.7825. Min Act: -0.0000

Data Index: 180885 (C4 (Web Text))

Max Activating Token Index: 80

Text #19

Max Range: 1.1906. Min Range: -1.1906

Max Act: 0.7983. Min Act: -0.0000

Data Index: 373286 (C4 (Web Text))

Max Activating Token Index: 46