So if I understand correctly, there was dirty/malformed data which was difficult to interpret, and when sent to a ML algorithm not tuned by a domain expert led to bad results.
This applies to all ML work, why is Watson exempt from it?
Conversely if this the canonical failure case, why's everyone so harsh on Watson?
Conversely if this the canonical failure case, why's everyone so harsh on Watson?
Because they keep claiming Watson can handle this kind of thing, they charged $60M for it, and failed.
That was $60M which could be spent on actual, real cancer research, or models which actually work.
For example, MSKCC[1] currently has a Kaggle competition[2] to do a roughly analogous task. The prize for that is $15,000, and they'll get something more useful for that than MD Anderson did for $60,000,000. Even taking into account it is probably costing MSKCC double the prize to have Kaggle run the competition it seems like IBM was ripping everyone off.
What I sincerely don't get about IT is why customers sign contracts where they fork over the mulah even when nothing of value is delivered to them. Why the actual fuck?
I mean, if Delta ordered a super-advanced new fancy 787 as a tech demo, and Boeing came back with "sorry, our experts were unable to make anything work, so we're going to take all your money and not give you anything back", Delta would rightly tell Boeing to fuck right off and demand that they refund the money.
If you buy a product, like Windows 10, you pay X and get the product, with whatever it contains. You can return that and get your money back.
If you buy a service, or a customized product, then you pay for the time and effort it takes to build / customize that product. Returning that time and effort would be difficult :)
It is more complicated than "pay X, based upon accomplishing Y". Most IT shops insist upon not just taking on the sustaining operations after the initial rollout, but doing it their way as well. When using and administering the products are nearly as complex as using and administering operating systems, I strongly suspect what is happening is our ecosystems have gotten much more complex than in the past, but our way of delivering them has not kept up. This is why AWS is so popular; those packaged services "only comes in black" forcibly eliminate much of the administration and operations idiosyncrasies many IT organizations insist upon for on-prem deployments, and accommodating those idiosyncrasies comes with costs, one of which a huge blurring of where the vendor leaves off and the IT staff takes over.
A lot of IT products don't stand alone in their own stack, and the wild variance in integration points from one site to the next drives a lot of IT project complexities and costs. It doesn't help that many of these integration points are considered essential by various stakeholders within an IT and business organization, yet are not managed by the organization to a necessary quality level. I've worked with failures that trace back to an LDAP cluster sometimes not synchronizing one of its nodes with a password update, for example. Even after tracing through the entire product stack to an outside service we depend upon that the organization delivers (LDAP in this case), the IT staff insisted it was the product. Client rages about a shitty product, then when I figured out the root cause, it was "oops, never mind", no apologies. The directory services team never fixed the sync issue, and instead I ended up cobbling together my own monitoring and notification to detect the issue and notify the administration team.
The administrator they assigned to take on the product after I rolled it out did not have the skills to diagnose and work around that issue, especially as fast as I worked it (within a few days). Unfortunately, this is a common situation. "Knowledge Transfer" is then leaned upon to fill the gap, which of course will fall short of expectations. It is a separate discussion on "knowledge transfer" commonly being a euphemism to for "give me the TL;DR within 100 pages so I don't have to read the stack of manuals and support notes of this product, but still be good enough to quickly troubleshoot if the product falls down". A lot of this "integration point failures that appear on the surface as product failures" comes down to very sysadmin'y troubleshooting, crossing lots of different specialized IT domains, often working on the fly side-by-side with specialized admins and administrator guides of products you've never seen before. Very hard to find people who are that broad, deep, flexible, and cool under pressure.
What I've seen happen a lot is the product vendor successfully rolls out their product, the organization's staff take over, and the stability and usefulness gradually degrades over time. This makes it particularly difficult to say the product never delivered. Sometimes it happens quickly (within a year), and other times it happens slowly (over a decade). It was a combination of human factors that caused the goal to fall short. A lot traces back to organizations wanting to cut costs, and the quickest way to do that when adopting a product is say, "we'll have our own staff do the day-to-day work, instead of paying an outside organization to come in and do it, or hiring new staff who are already experienced with it".
It's a hot mess, and my hope is eventually the cloud delivery model pushes deeper interoperability across the industry's various ecosystems, and increasingly automate more integration efforts not just in the cloud but in other delivery modes. Or at the very least cloud providers commoditize and stabilize increasing chunks of infrastructure and devops that products rely upon.
>> This applies to all ML work, why is Watson exempt from it?
>I'm not following, Watson was most assuredly not exempt from it.
Apologies, I meant it as not having clean data with badly tuned models should be obvious root cause of failure to the analysts who are familiar with this industry. As in it may take more work over time to improve the situation, and that this problem occurs commonly in other areas of AI/ML as well.
>> Conversely if this the canonical failure case, why's everyone so harsh on Watson?
>Because they were attempting to use Watson to help make potentially life-and-death decisions?
This applies to all ML work, why is Watson exempt from it?
Conversely if this the canonical failure case, why's everyone so harsh on Watson?