I agree. I think there are two distinct usability issues to address: content size and font size. That is, how much real estate the virtual viewport occupies in proportion to the display AND the size of font relative to the reader's ability to read comfortably. The trick is sensible adjustments to style attributes to keep layout working properly on the vast majority of devices no matter the zoom level or default font size, within reason.

As a user example, when I load the HN front-page on an iOS device in portrait mode, I'm lazy and switch to rotate to landscape mode to see more without trimming edges and scrolling around. On most sites, I expect to be able to double-tap on a paragraph column to toggle maximize-fit-to-display-width mode and return regular all-content-on-display-width mode.

If some site prevents this, I will not buy their products or use their services, because it's a signal they don't care about usability and will inevitably waste people's time. That's a form of laziness and failure that should not be rewarded. There are usually competitors that want you to switch to their brand (except with govts). Those annoying jQuery Mobile one-sized-fits-all sites deployed by pharmacies, banks, govts and so on.