The File example is a good illustration of why Java is _not_ a capability-secure language. Every File object in Java has a getParentFile() method that allows you to navigate up the hierarchy right to the root and then from there access every file on the filesystem. Java’s standard library is full of these kinds of design flaws. So in practice you can only apply capability-based thinking to small subsets of a codebase and have to fallback on the (much weaker) stack walking checks if you want strong isolation.
The problem with Java’s stack walking is that it is too complex and too easy to find privileged code that can be coaxed into performing unintended operations. There are plenty of old write ups of Java sandbox bypass bugs due to this, eg http://benmmurphy.github.io/blog/2015/10/21/zdi-13-075-2013-...
I shouldn't have used File as an example, that was confusing. I was trying to explain capabilities and stack walking in an abstract sense but was also using Java as a concrete example. Big mistake.
You're right that java.io.File isn't a capability. It just represents a file path with a few utility methods to list files, and therefore does a stack walk when you try to access the filesystem. A FileChannel is a file capability in the sense I meant above, because it represents an opened file, not a path. There's an access check once, when it's opened, and then the rest of the time there aren't any stack walks.
It's a pity that Ben Murphy didn't write up all his bugs. There are only two listed there. A few patterns cropped up repeatedly in the old sandbox escapes:
1. Accessing internal code you weren't meant to have access to. Often, some sorts of privileged pseudo-reflection API. Fixing this is the goal of Jigsaw.
2. Serialization acting as a back door, allowing the internal private state of classes to be tampered with. Serialization security has been improved with time and they're now working on a feature that will make it harder to screw this up, by allowing serialization to use normal constructors instead of this ad-hoc form of reflection.
3. Overly general frameworks that allowed attackers to construct nearly arbitrary programs out of privileged objects by chaining them together (this crops up in gadget attacks too). There's probably no platform level fix for this, people just have to be aware of the risks when working in sandboxed context.
I don't think a pure capability language is workable, to be honest. At least not at a large scale. In the purest sense you need a god object passed into the start of your program which vends all possible capabilities, and every library would require the user to construct and pass in all resources it needs externally, including things it might possibly need. And that code isn't easily modularised because you can't just use helpers provided by the library itself: that's the very same code you're trying to sandbox. There's no good way to make that usable or extensible. The combination of stack walking and capabilities lets you find the right balance in terms of API design between simplicity of use and sandbox simplicity.
Are you aware of the history of object-capability programming languages? There are multiple actual demonstrations of real-world ocaps programming languages and projects built with them:
It's actually not at all unworkable to use object-capability for large programs. In fact, one of the main benefits of ocaps is how well it aligns with well-established good software design principles such as dependency injection, avoiding singletons, avoiding global mutable state, and so on.
I know about E and Midori. I haven't looked at the others. As far as I know the only one that could realistically be said to have been used for large programs was Midori but very little about it was ever published, just a few blog posts. And Midori was cancelled. Presumably it wasn't so compelling.
I'd like to see a more modern attempt that wasn't as totally obscure as those other languages. However, nobody is doing that.
That’s just the first example. As the author of that series writes, most of the exploits are not due to memory corruption. Most are confused deputy attacks where privileged code can be tricked into performing dangerous operations.
The problem with Java’s stack walking is that it is too complex and too easy to find privileged code that can be coaxed into performing unintended operations. There are plenty of old write ups of Java sandbox bypass bugs due to this, eg http://benmmurphy.github.io/blog/2015/10/21/zdi-13-075-2013-...