IIRC the corners of the ten-mile square are due north/south/east/west of the district's original center point - so it's aligned 45 degrees. (I had thought that the Capitol was dead-center within the original ten-mile square, but it's not.)

Also there are a few borders set to not be in the middle of the river - New Jersey-Delaware is another one, and there are bits of land on the Jersey side that technically belong to Delaware. (But New Jersey-Pennsylvania isn't, even though Delaware used to be part of Pennsylvania.)