Looking past the slot, in this picture, the pocket limits lateral movement of the drumshaft. I referred to that as the "stop". You just want to eliminate the possibility of the end of your drumshaft hanging up against this surface.
3mm off center shouldn't normally be a problem. Indeed, since it's still a tight fit on port with the Drive reversed, it's not likely your culprit. Still, you can center the shaft by tapping it with a block or mallet to be sure.
Sorry to hear that you're having a problem.