If the server cannot be trusted, it will extract your encrypted data, since it serves up the code. The server, if compromised/subpoenaed, merely needs to serve you some JavaScript that sends home the encryption key, and your data is no longer just your data.
The redeeming quality is that compromised servers are usually detected and shutdown quickly. Traditional servers are dumped and the data is compromised. Encrypted blob storage makes the dump nearly impossible to decrpyt.
It's OK to improve security incrementally. Better security is the path to the best security.
We are still working on new sets, though obviously the rate of new sets is pretty low. The mailing list is basically unmonitored at this point, but everything we've got is on the site. (This is a vast improvement on the previous state, where we regularly failed to set out challenges to people who emailed us, due to overload.)
> Penetration tests, when done by a good firm like Matasano, are incredibly useful, but lose their value the next time you push code.
I'd like to nicely but firmly push back on this one, and have longitudinal analysis of clients' applications to back it up. We put a lot of effort into helping our customers improve over time, both formally (writing helpful recommendations) and informally (educating developers during and after the test). There exist customers that ignore our advice, and don't improve, but most have a dramatic improvement in new code quality after the first assessment, and continue to year after year.
Ah, you misunderstood what I meant. I didn't mean to imply that penetration tests, when done well, have no lasting value. I simply meant to imply that without a code freeze, there is always the chance of a new vulnerability creeping in no matter how well you follow checklists, best practices, or retain knowledge about types of vulnerabilities and how not to build them.
For that reason, automated testing on a continuous basis is important.
This is the same reason that you don't QA an application once a year. UIs change, requirements change, and for that you write integration tests, unit tests, etc.
Does that clarify things a bit? I didn't mean to imply Matasano did a poor job of educating their customers; in fact, I think you're among the best.
Note that our work-sample tests are, not-insanely, done in the comfort of your own home, at your own pace, on your own schedule, and represent the work we actually do. As co-head of recruiting for NCC US (aka head of recruiting for Matasano), I think the answer to the in-person interview question of "write code to do this," is "thank you for your time, I'll see myself out."
Most of our candidates drop out before the work sample. On the other hand, almost none of our candidates are qualified to work for us when they initially apply.
We make it really clear that there will be work samples before people even apply. I think the real question would be "how many qualified applicants don't bother applying because of the work sample," which is a question we can't answer. Given the paucity of unemployed qualified infosec folks, we're comfortable with the tradeoff.
Here are a couple of resources that I tend to hand out to startups that we do work for at Matasano. No charge :-)
Not trying to be a salesperson, but I feel like most startups get more value out of sitting down with a security consultant for a couple days and talking about architecture and dev processes then they do getting a full penetration test. Like the presentations say, the big risk in the early days is lack of interest, not security. I feel like a startup's big security concern it doing something that's going to make them have to rewrite everything later on.
In general, my feeling is that the Matasano process (which I currently manage) works outstandingly well where there isn't a flood of qualified candidates. If you have a glut of folks who are ready to start working, you can get away with a terrible process.
We do free-form technical interviews, but only to try to detect candidates who really aren't ready for the work-sample challenges. Our in-person interviews are standardized and try to evaluate consulting/architecture skills that are hard (impossible?) to measure without people. These involve open-ended intermediary questions, but the final answers are structured.
The bottom line is this: you cannot compare candidates using free-form interviews. You must compare candidates who are going to be doing similar work. Thus, free-form interviews have no evaluative value.
