Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 997 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.8993. Min Range: -0.8993

Max Act: 0.8993. Min Act: -0.0000

Data Index: 1166517 (C4 (Web Text))

Max Activating Token Index: 385

Text #1

Max Range: 0.8993. Min Range: -0.8993

Max Act: 0.8922. Min Act: -0.0000

Data Index: 473871 (C4 (Web Text))

Max Activating Token Index: 599

Text #2

Max Range: 0.8993. Min Range: -0.8993

Max Act: 0.8056. Min Act: -0.0000

Data Index: 779437 (C4 (Web Text))

Max Activating Token Index: 667

Text #3

Max Range: 0.8993. Min Range: -0.8993

Max Act: 0.8343. Min Act: -0.0000

Data Index: 809725 (C4 (Web Text))

Max Activating Token Index: 353

Text #4

Max Range: 0.8993. Min Range: -0.8993

Max Act: 0.8369. Min Act: -0.0000

Data Index: 271734 (C4 (Web Text))

Max Activating Token Index: 364

Text #5

Max Range: 0.8993. Min Range: -0.8993

Max Act: 0.8139. Min Act: -0.0000

Data Index: 1032897 (C4 (Web Text))

Max Activating Token Index: 933

Text #6

Max Range: 0.8993. Min Range: -0.8993

Max Act: 0.6948. Min Act: -0.0000

Data Index: 579630 (C4 (Web Text))

Max Activating Token Index: 601

Text #7

Max Range: 0.8993. Min Range: -0.8993

Max Act: 0.7065. Min Act: -0.0000

Data Index: 659694 (C4 (Web Text))

Max Activating Token Index: 785

Text #8

Max Range: 0.8993. Min Range: -0.8993

Max Act: 0.7046. Min Act: -0.0000

Data Index: 1120222 (C4 (Web Text))

Max Activating Token Index: 366

Text #9

Max Range: 0.8993. Min Range: -0.8993

Max Act: 0.6942. Min Act: -0.0000

Data Index: 1130377 (C4 (Web Text))

Max Activating Token Index: 992

Text #10

Max Range: 0.8993. Min Range: -0.8993

Max Act: 0.6258. Min Act: -0.0000

Data Index: 692330 (C4 (Web Text))

Max Activating Token Index: 227

Text #11

Max Range: 0.8993. Min Range: -0.8993

Max Act: 0.6112. Min Act: -0.0000

Data Index: 970278 (C4 (Web Text))

Max Activating Token Index: 837

Text #12

Max Range: 0.8993. Min Range: -0.8993

Max Act: 0.6509. Min Act: -0.0000

Data Index: 627593 (C4 (Web Text))

Max Activating Token Index: 487

Text #13

Max Range: 0.8993. Min Range: -0.8993

Max Act: 0.6769. Min Act: -0.0000

Data Index: 1034498 (C4 (Web Text))

Max Activating Token Index: 763

Text #14

Max Range: 0.8993. Min Range: -0.8993

Max Act: 0.6339. Min Act: -0.0000

Data Index: 60480 (C4 (Web Text))

Max Activating Token Index: 995

Text #15

Max Range: 0.8993. Min Range: -0.8993

Max Act: 0.6317. Min Act: -0.0000

Data Index: 27393 (C4 (Web Text))

Max Activating Token Index: 534

Text #16

Max Range: 0.8993. Min Range: -0.8993

Max Act: 0.6014. Min Act: -0.0000

Data Index: 203273 (C4 (Web Text))

Max Activating Token Index: 622

Text #17

Max Range: 0.8993. Min Range: -0.8993

Max Act: 0.5953. Min Act: -0.0000

Data Index: 694611 (C4 (Web Text))

Max Activating Token Index: 398

Text #18

Max Range: 0.8993. Min Range: -0.8993

Max Act: 0.6308. Min Act: -0.0000

Data Index: 991192 (C4 (Web Text))

Max Activating Token Index: 728

Text #19

Max Range: 0.8993. Min Range: -0.8993

Max Act: 0.6039. Min Act: -0.0000

Data Index: 571877 (C4 (Web Text))

Max Activating Token Index: 77