It's not just that they don't overlap; it's that they don't even really relate or affect each other. In other words, one is pointing North and the other West. They aren't opposites; just different.
Both of you are correct. In programming languages / the IT world, object A is orthogonal to object B if object A can be used without thinking about the potential consequences to object B.
In mathematics, orthogonal means perpendicular in a geometric-sense (think two vectors) but can also be used in other contexts with a different meaning.
Even more generally, two vectors are orthogonal if their inner product is zero.
This is why the word "perpenidcular" is not used, as sometimes your vectors don't really have "directions" in the intuitive sense of the word (e.g. the inner product space of functions).
I think that comes from the ideas behind vector decomposition (remember physics 101? ;-)).
When you decompose a goal into several linearly independent sub-goals - those sub-goals are said to be 'orthogonal' since they don't have any interaction/interdependence with one another.
Orthogonal is an appropriate synonym for complementary in a lot of cases. It wasn't an appropriate usage here though. Features complement each other by not overlapping, hence being orthogonal, which is probably the only usage that most programmers hear and the source of the confusion.
Sorry - people misappropriating math jargon into the mainstream is a little pet peeve of mine. I agree with your main point though.