That's true and a point worth making, but as a practical matter, breaching almost anyone's perimeter is game-over. To not have that be the case, you need to design from the ground up so that internal services don't trust their own deployment network; it's difficult, time consuming, and in many cases confining (ie, it makes some services prohibitively difficult to deploy).
It's for this reason that pentesters learn quickly that the "make an arbitrary HTTP query from the target's own server" bug is usually sev:critical; for instance, in virtually any Fortune 500 network, that pivot gets you (with a little effort and 50 lines of code) to a JMX console somewhere, and from there code execution.
There's no good reason not to do both (ensuring that your internal services are authenticated reasonably and don't expose functionality or information pre-auth, AND setting up a VPN). But the VPN is the most valuable step.
This is essentially something you need to worry about anyway, for other attack reasons.