Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3309 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.2291. Min Range: -1.2291

Max Act: 1.2291. Min Act: -0.0000

Data Index: 1145883 (C4 (Web Text))

Max Activating Token Index: 232

Text #1

Max Range: 1.2291. Min Range: -1.2291

Max Act: 1.1916. Min Act: -0.0000

Data Index: 751808 (C4 (Web Text))

Max Activating Token Index: 654

Text #2

Max Range: 1.2291. Min Range: -1.2291

Max Act: 1.1609. Min Act: -0.0000

Data Index: 1132782 (C4 (Web Text))

Max Activating Token Index: 842

Text #3

Max Range: 1.2291. Min Range: -1.2291

Max Act: 1.1457. Min Act: -0.0000

Data Index: 985971 (C4 (Web Text))

Max Activating Token Index: 35

Text #4

Max Range: 1.2291. Min Range: -1.2291

Max Act: 1.0980. Min Act: -0.0000

Data Index: 628723 (C4 (Web Text))

Max Activating Token Index: 129

Text #5

Max Range: 1.2291. Min Range: -1.2291

Max Act: 1.0958. Min Act: -0.0000

Data Index: 546302 (C4 (Web Text))

Max Activating Token Index: 953

Text #6

Max Range: 1.2291. Min Range: -1.2291

Max Act: 0.9911. Min Act: -0.0000

Data Index: 816950 (C4 (Web Text))

Max Activating Token Index: 880

Text #7

Max Range: 1.2291. Min Range: -1.2291

Max Act: 1.0462. Min Act: -0.0000

Data Index: 462703 (C4 (Web Text))

Max Activating Token Index: 783

Text #8

Max Range: 1.2291. Min Range: -1.2291

Max Act: 1.0711. Min Act: -0.0000

Data Index: 1240920 (C4 (Web Text))

Max Activating Token Index: 397

Text #9

Max Range: 1.2291. Min Range: -1.2291

Max Act: 0.9739. Min Act: -0.0000

Data Index: 444620 (C4 (Web Text))

Max Activating Token Index: 980

Text #10

Max Range: 1.2291. Min Range: -1.2291

Max Act: 1.0313. Min Act: -0.0000

Data Index: 327466 (C4 (Web Text))

Max Activating Token Index: 506

Text #11

Max Range: 1.2291. Min Range: -1.2291

Max Act: 1.0977. Min Act: -0.0000

Data Index: 5439 (C4 (Web Text))

Max Activating Token Index: 776

Text #12

Max Range: 1.2291. Min Range: -1.2291

Max Act: 1.0048. Min Act: -0.0000

Data Index: 848836 (C4 (Web Text))

Max Activating Token Index: 1013

Text #13

Max Range: 1.2291. Min Range: -1.2291

Max Act: 0.9807. Min Act: -0.0000

Data Index: 30905 (C4 (Web Text))

Max Activating Token Index: 893

Text #14

Max Range: 1.2291. Min Range: -1.2291

Max Act: 0.9767. Min Act: -0.0000

Data Index: 163339 (C4 (Web Text))

Max Activating Token Index: 663

Text #15

Max Range: 1.2291. Min Range: -1.2291

Max Act: 1.0057. Min Act: -0.0000

Data Index: 699138 (C4 (Web Text))

Max Activating Token Index: 636

Text #16

Max Range: 1.2291. Min Range: -1.2291

Max Act: 0.9716. Min Act: -0.0000

Data Index: 304723 (C4 (Web Text))

Max Activating Token Index: 488

Text #17

Max Range: 1.2291. Min Range: -1.2291

Max Act: 0.9365. Min Act: -0.0000

Data Index: 1218093 (C4 (Web Text))

Max Activating Token Index: 283

Text #18

Max Range: 1.2291. Min Range: -1.2291

Max Act: 0.9071. Min Act: -0.0000

Data Index: 876179 (C4 (Web Text))

Max Activating Token Index: 53

Text #19

Max Range: 1.2291. Min Range: -1.2291

Max Act: 0.9647. Min Act: -0.0000

Data Index: 1193282 (C4 (Web Text))

Max Activating Token Index: 34