Try this. Just rotate the wheel by hand 1 revolution and see what it returns. I would assume this would be as inconsistent as running with PWM, but who knows, maybe it will be a clue.
Ringo
This is a strange problem. The fact that it works without the motor being powered means to me that it is probably either a power issue or a noise issue. Why the problem stays with 1 port and not another is weird. Do you happen to have a different (smaller) motor you can try and see what happens? Any chance you have an O-scope you can use to look at the encoder signals?
Someone else had a problem with the board acting weird using motors sorta like these and it was fixed by adding a large capacitor at the battery inputs to the serializer. Do you happen to have something like a 470uf or bigger cap you can try? If not let me know, I probably have something in my junk pile I can mail you to see if it helps.
Once we figure out what the issue is, there will be another problem. These motors draw .5A with no load, but 17A at stall, so that means with some load the current can quickly go above the serializers capacity. After we get the encoder problem solved, you are going to have to change to smaller motors or go to a bigger H-bridge, so It may be worth it to make that decision now in case that change fixes the problem. We have some 10A Dual H-bridges on the site that will probably work with these motors ok. I call them 10A to be conservative. All the parts are rated for 17A for very short times, so as long as your software monitors your wheels and stops if the wheels stall you are probably ok.