Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1658 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.9317. Min Range: -0.9317

Max Act: 0.9220. Min Act: -0.0000

Data Index: 493951 (C4 (Web Text))

Max Activating Token Index: 210

Text #1

Max Range: 0.9317. Min Range: -0.9317

Max Act: 0.9317. Min Act: -0.0001

Data Index: 936136 (C4 (Web Text))

Max Activating Token Index: 153

Text #2

Max Range: 0.9317. Min Range: -0.9317

Max Act: 0.8788. Min Act: -0.0001

Data Index: 1318069 (C4 (Web Text))

Max Activating Token Index: 294

Text #3

Max Range: 0.9317. Min Range: -0.9317

Max Act: 0.8239. Min Act: -0.0001

Data Index: 524857 (C4 (Web Text))

Max Activating Token Index: 656

Text #4

Max Range: 0.9317. Min Range: -0.9317

Max Act: 0.8150. Min Act: -0.0001

Data Index: 879124 (C4 (Web Text))

Max Activating Token Index: 697

Text #5

Max Range: 0.9317. Min Range: -0.9317

Max Act: 0.8256. Min Act: -0.0001

Data Index: 668564 (C4 (Web Text))

Max Activating Token Index: 885

Text #6

Max Range: 0.9317. Min Range: -0.9317

Max Act: 0.7899. Min Act: -0.0001

Data Index: 534475 (C4 (Web Text))

Max Activating Token Index: 937

Text #7

Max Range: 0.9317. Min Range: -0.9317

Max Act: 0.8292. Min Act: -0.0001

Data Index: 600687 (C4 (Web Text))

Max Activating Token Index: 868

Text #8

Max Range: 0.9317. Min Range: -0.9317

Max Act: 0.8108. Min Act: -0.0000

Data Index: 391790 (C4 (Web Text))

Max Activating Token Index: 475

Text #9

Max Range: 0.9317. Min Range: -0.9317

Max Act: 0.8184. Min Act: -0.0001

Data Index: 159773 (C4 (Web Text))

Max Activating Token Index: 900

Text #10

Max Range: 0.9317. Min Range: -0.9317

Max Act: 0.7899. Min Act: -0.0000

Data Index: 841310 (C4 (Web Text))

Max Activating Token Index: 511

Text #11

Max Range: 0.9317. Min Range: -0.9317

Max Act: 0.7447. Min Act: -0.0001

Data Index: 755980 (C4 (Web Text))

Max Activating Token Index: 102

Text #12

Max Range: 0.9317. Min Range: -0.9317

Max Act: 0.7606. Min Act: -0.0001

Data Index: 135317 (C4 (Web Text))

Max Activating Token Index: 454

Text #13

Max Range: 0.9317. Min Range: -0.9317

Max Act: 0.8021. Min Act: -0.0001

Data Index: 334885 (C4 (Web Text))

Max Activating Token Index: 952

Text #14

Max Range: 0.9317. Min Range: -0.9317

Max Act: 0.7538. Min Act: -0.0000

Data Index: 815483 (C4 (Web Text))

Max Activating Token Index: 925

Text #15

Max Range: 0.9317. Min Range: -0.9317

Max Act: 0.7764. Min Act: -0.0000

Data Index: 531038 (C4 (Web Text))

Max Activating Token Index: 905

Text #16

Max Range: 0.9317. Min Range: -0.9317

Max Act: 0.7237. Min Act: -0.0001

Data Index: 1207646 (C4 (Web Text))

Max Activating Token Index: 60

Text #17

Max Range: 0.9317. Min Range: -0.9317

Max Act: 0.7470. Min Act: -0.0000

Data Index: 253885 (C4 (Web Text))

Max Activating Token Index: 792

Text #18

Max Range: 0.9317. Min Range: -0.9317

Max Act: 0.8240. Min Act: -0.0001

Data Index: 1252702 (C4 (Web Text))

Max Activating Token Index: 241

Text #19

Max Range: 0.9317. Min Range: -0.9317

Max Act: 0.6848. Min Act: -0.0001

Data Index: 7817 (C4 (Web Text))

Max Activating Token Index: 726