Another place you will see this problem is if you pinch-zoom the document. If you pinch-zoom without flinging first, the document will zoom from the top left corner, rather than from the middle of the zoom location. You can see this in the Radaee PDF Viewer app.
Unfortunately, I found that my fix posted above creates a new problem: calling vSetY immediately after vSetX aborts the x movement. So when zooming the document, the left position doesn't move, and when panning the document (1 finger drag) it will only pan in the y direction. So I made a new vSetXY method that combines the x and y movement into a single m_scroller.startScroll action. This solves both the zooming and panning problems for me.
You might find a more direct solution that doesn't require combining the vSetX and vSetY function calls into a single function call. But I need this approach in my code because I also added an animated movement option, by using the optional fifth argument of m_scroller.startScroll. So I'm now using this instead of vSetX and vSetY:
public void vSetXY(int x, int y, boolean animated) {
if (x > m_tw - m_w) x = m_tw - m_w;
if (x < 0) x = 0;
if (y > m_th - m_h) y = m_th - m_h;
if (y < 0) y = 0;
int duration = (animated) ? 250 : 0 ;
lastX = x; // this lets us get the new x while the scroll is still animating
lastY = y; // this lets us get the new y while the scroll is still animating
int startX = vGetX();
int startY = vGetY();
int distanceX = Math.round(x - startX);
int distanceY = Math.round(y - startY);
m_scroller.forceFinished(true);
m_scroller.startScroll(startX, startY, distanceX, distanceY, duration);
}