Be careful with the setrlimit/ulimit API family, generally it doesn't do what you want. You can limit virtual memory (but... why?) or specific segments like stack, etc. There is also RLIMIT_RSS which sounds like what you'd want, but alas:

    RLIMIT_RSS
        This is a limit (in bytes) on the process's resident set (the number of virtual pages resident in RAM).  This limit has effect only in Linux 2.4.x,  x  <  3 and there affects only calls to madvise(2) specifying MADV_WILLNEED.

I also disagree with the conclusion "No hardware can compensate for a query gone wrong". There are concepts like 'quality of service' and 'fairness' which PG has chosen to not implement.

That does not cap memory per query, which should be implemented by postgres explicitly to capture only the memory that really counts towards "the query" (fine-grained memory limits; ulimit is not that).