The one place I've used Bayes (hopefully properly!) is in a spaced repetition flash card program. Usually spaced repetition algorithms wait a certain amount of time based on how many times you have seen and remembered a card. The more times you have remembered it, the longer you wait. It then creates a schedule for each day. You review the cards that have "expired" their wait time.

I wanted to turn this upside down. Instead I sorted the cards by how likely you would be to remember them (a function of the number of times you have already remembered it and the amount of time that has passed since you last saw it). I put the least likely to be remembered cards first and the most likely to be remembered cards last.

As you reviewed the cards you would either remember them or not remember them. I supposed that cards grouped together had a similar probability of being remembered (only valid if my sort algorithm was correct, but I had some confidence on that since I based it on somebody else's research ;-) ). I then used Bayes to estimate the probability of getting the card correct.

So instead of scheduling the cards, I simply had the user keep reviewing until I had a certain confidence that there was a 90% or more of getting the cards correct. At which point I left the rest for another day (the probability goes down over time). Interestingly as the test is binary (remembered/not remembered) Bayes could be simplified down to getting it right a certain number of times in a row. This was simpler for the user to understand without significantly reducing the accuracy of the estimate.

Gratifyingly the system worked incredibly well. I added an option to continue drilling the cards after you hit the 90% mark and I very rarely got into a situation where the estimate was wrong.

That sounds very interesting. Is your program available anywhere?

See my reply to mtrimpe below. I maintained this program for quite a long time, but realistically my choice of development platform was a poor one ;-) Also my code was pretty awful as I was experimenting with several strange ideas and also writing Ruby code is if I had spent the last 20 years writing C++ code (which... um... might have been true...)

You can likely get it to work for some definitions of "work" on a Linux box, but anything else would require serious effort ;-)

Link in case you don't see the other message: https://github.com/mikekchar/JLDrill

That sounds fascinating. Is there anywhere you could look at the result (or code ;) of this work?

Umm... The code is horrific as I was experimenting with a few different ideas. It is written in Ruby for GTK+ and is not at all idiomatic Ruby. It's also somewhere between slightly and completely broken at the moment... But with all that in mind: https://github.com/mikekchar/JLDrill

Probably more interesting is simply my description of the scheduling algorithm: https://github.com/mikekchar/JLDrill/blob/master/web/src/Str...

There is also one detail missing which is forgetting. Because the items are sorted by the ratio of time waited to "ideal schedule", we can easily stick anything over a certain amount into a separate set (called the forgotten set). That way if you don't study for a long time you can "forget" those items and they are treated like a high priority "new" set until the set is empty. Probably that makes no sense, but if you read my strategy document, you will hopefully be able to understand.

That basically seems to ignore the exponential Ebbinghausian forgetting curve. I doubt it results in better scheduling of the cards.

It's not exponential. It has a gamma distribution. The nice thing about the gamma distribution is that within a short part of the tail it is nearly linear. So while it's hard to explain the math in an HN posting, below a certain probability your odds of getting a 90-95% confidence of a 90% recall rate (10% forgetting) rate is very, very low. So you will have to review all of those cards anyway. Once you get into the range where false positives are more likely (say above 80%) the curve starts to get more linear. This is especially true for cards with a very shallow curve (those that you have seen many times). So even when you get it wrong, the ones that are likely to be problematic are those that you will review again quickly anyway.

As for whether it results in a better scheduling of cards, it's hard to say. It has the advantage of being self adjusting and containing far fewer magic numbers than something like SM(insert-any-version-here).

Did you find out the base statistics for fatalities in skydiving? That that particular business had fatalities may just indicate that they do a lot more volume or more advanced types of skydiving than other companies (in which case they may actually be safer than the norm). For me, one of the big differences Bayes rule makes is spotting that information nearly always needs context to understand it correctly. You don't have the whole story unless you have a sensible prior.

Yeah I thought about this and could find a news article that said they did around 5000 jumps a year since they started in 2001 (and they're located in Las Vegas so they I assume most of their customers are first timers). So that is 75000 jumps. In the news I could find two fatalities, and two serious injuries (but let's ignore those for now). Apparently, statisticians use the term "micromorts" which means a death per million activities when talking about risks from things like this. So for this business the track record is 26.7 micromorts. Compared to the base statistics in skydiving--which is 9--this means that this business has 3 times the likely risk. Am I getting this right?

Sounds like it. But like you said they're almost all probably first-timers, given the location. It's hard to tell if that completely explains the micromort difference or not.

Also, it is possible that due to those fatalities, the business implemented better safety protocols as a response and may actually be safer now than it was then.

It's my Jack in the Box theory. Back in the 90s there were a couple high profile food poisoning cases (salmonella I think) where customers had died after eating at Jack in the Box. It nearly ruined the company. But my feeling is that the safest time to eat there is a month or so after those events. When all the restaurants would be kicking their food-handling practices into high gear.

So now I eat at Chipotle more often than I did before. Shorter lines and probably the cleanest food. I hope. My "theory" may be bullshit.

You aren't applying Bayes correctly. As a general guide, you need to add the word "given" to your problem statement, assign two "situation A" and "situation B" variables, and then work the math.

For example, assigning variables:

A = You died B = You're skydiving at that particular dropzone

Then: "What is the risk of mortality GIVEN that I am skydiving at this drop zone?" (P(A|B))

"What's the chance that I'm skydiving at this dropzone, given the fact that I died?" (P(B|A))

"What's the risk of my mortality while skydiving?" (P(A))

"What is my probability of skydiving at this dropzone?" (P(B))

P(A | B) = (P(B | A) * P(A)) / P(B)

So you calculated P(A | B) in a non-Bayesian way, without finding out the other information to calculate it using Bayes, and then stopped there, like most people do. This is why Bayes is often difficult for people to understand and apply correctly, and, honestly, it's probably not the equation you want for the situation you're looking at.

Another approach -- and this seems to be the one you want -- is to calculate a 95% confidence interval using a binomial distribution, to find out if their statistics are really anomalous. Death is a relatively rare event, and, even if they're distributed perfectly randomly, you'll find odd-looking clusters here and there.

To figure out if it's anomalous, many people would use the normal approximation to the binomial confidence interval, which would be wrong -- the probabilities of death are so relatively tiny, that they can't be approximated normally (rule of thumb is P(A) x P(not A)x sample size > 5 to use the normal distribution, which this fails), so we need to do an exact calculation. I've done this by hand before, but it's a pain in the butt. That's why we have calculators!

http://epitools.ausvet.com.au/content.php?page=CIProportion (If you don't trust it, you can use another one)

You enter your numbers:

sample size = 75000

number of deaths = 2

This gives the exact binomial confidence interval as: [3.23e-06, 9.633e-05]

This means that their actual death rate could be anything from .00000323 (that's 3 deaths per million) to .000096 (that's nearly 100 deaths per million)

Clearly, you cannot say with any meaningful level of confidence whether or not this drop zone is safer, or less safe, than average. Sorry!

Edit: In my Bayes example earlier, I realized that you could actually use it in an interesting-ish way (I guess?) to find out an unknown: "What are the odds that I was skydiving at this particular dropzone, given that I died skydiving?"

So: A = You're skydiving at that particular dropzone

B = You died :(

P(A | B) = (P(B | A) * P(A)) / P(B)

We know that: P(B | A) = 2/75000 = .0000267

P(A) = their skydives / all skydives = (75000 / 3,300,000 x 15) = 0.0015

Note: I found out that there were 3.3 million skydives in the US in 2012, so, let's extrapolate that out 15 years as a rough approximation, and limit us to just the US.

P(B) = .000009

Then:

P(A | B) = (.0000267 * .0015) / .000009 = 0.00445

So, if you died, then the probability that you were skydiving at that dropzone is 0.4%!

Note though, that there's some uncertainty in the calculation of P(B | A) (the probability of dying while skydiving at that dropzone) which you need to use confidence intervals, above, to actually figure out. Anyway, I certainly clarified some of my thoughts while writing this, and I hope that it helps you too, figuring out when to use confidence intervals, and when to use Bayes, and why it matters!

This is great. Thank you

I think your anecdote is good. Knowing the formula for Bayes theorem isn't necessary, and I've never needed to try to do the math in my head or anything. But understanding probability theory, and having an intuitive feel for what probability is, is really important.

I think your friends were right. Peoples lives are not the same as a coin flip. You could almost surely say that whatever caused the accidents in the past has been a point of focus for the agency specifically so it never happens again. You can't really apply your statistical thinking in that scenario.

Maybe they were but they actually statistically came to that conclusion. In the same sense that people think that given that it has been all heads for the past 10 flips then the chance it is tails is higher in the next (it's not)

>You can't really apply your statistical thinking in that scenario.

The main idea behind Bayes is that you can apply probabilistic reasoning to anything where you don't have complete knowledge (which means, short of 1+1, is everything)

Weren't your friends right? Since the incident happened, it's quite likely they fired an incompetnet instructor or tightened security protocols to such an effect that future dives would be safer.

Maybe but 3 years after the fatalities they had 2 serious injuries which makes me give less credence to this. (Also see my reply to sibling comment, my friends reasoned about this statistically while falling for the gamblers fallacy, sort of)