A Coin-Flip Thought Experiment

Your avatar retention is also poor.

Thanks for the elaboration. 

I linked to the original thread/context in my OP, but basically Cham313 and I were debating the importance of priors (the initial probability a hypothesis is true) when it comes to scientific evidence. He was arguing that priors really don't matter after we get experimental evidence, while I was arguing that whether or not priors or evidence matters more depends on the prior and evidence in question. We also discussed whether or not one can even have any kind of precision when assessing the prior probability of a hypothesis. Cham had once claimed that all that mattered was which hypothesis fit the evidence better, while I argued that knowing the fit doesn't mean much without the prior.

I devised this thought experiment as a way of showing how even when your priors are imprecise--in this experiment, the prior would be the initial likelihood the coin is fair/trick before you've flipped it--and even though you don't have any definitive evidence right away, eventually you will be convinced the coin is a trick one. Further, once you are convinced you can retroactively calculate your prior algebraically. I thought it also presented a more common illustration of how scientific evidence works; ie, there's rarely one experiment that provides overwhelming evidence for a hypothesis, but rather there are a series of experiments that provide progressively more evidence for a hypothesis until the evidence is overwhelming, even in cases when we can't put precise numbers on the priors OR the conditionals. 

Basically, this my nefarious attempt at turning the forum into proper Bayesians. Gooble Gobble One of Us! 

Not having a precise prior is not really a problem for Bayes and part of what I wanted to demonstrate with this thought experiment. For this I'm going to use "T" for "trick coin" and "F" for flips to keep things straight.

Asking "what is P(T)" is a fine question to ask. But you can just as easily ask "how many flips does it take to convince you the coin is a trick coin?" Once you answer that--and the answer will be quite subjective, even though I'm sure we'd all agree some answers are more or less reasonable--then we can find P(T) algebraically. If we wanted to set our level of certainty at 95%, and let's say we agreed on 20 flips to be that convinced, then we'd find P(T) like this: .95 = (1T) / ((1T) + ((1-T)0.000001)) which would be about 0.002%. So answering how much evidence it takes to convince you is just another way of defining your priors.

All of the questions you ask ("Where was the coin found?" etc.) would just be additional info that would modify the prior. Let's say the coin was found randomly on the street; how many flips to convince you? Let's say the coin was found on the floor of a magic shop; how many now? Presumably in the latter case you will be convinced with fewer flips, but why? You can no more put a precise prior on T on the streets as T on the floor of a magic shop. You just assume (not irrationally) that the latter is more common.

The crucial point is this: even when you can't define your priors, the evidence will still eventually convince you it's a trick coin. Debating how much evidence it takes is just another way of debating your priors.

When you describe dropping Bayes, you're not actually dropping Bayes. What you're doing is describing performing experiments that would give one of the conditionals--E(vidence)|~T or E|T--a value of zero. Finding experiments that achieve that is an ideal that science aspires to, but in the vast majority of fields for the vast majority of history that's not how it's worked. One thing I'm showing with this thought experiment is that, even without knowing our priors precisely, and without having the benefit of an experiment that can rule out one of our hypotheses, we are still capable of being convinced by the evidence of the flips. Indeed, it would be rather absurd to never be convinced through the flips alone.

First, I worded my OP as "you've found a coin and immediately start flipping it," and since I didn't specify you were a magician with this ability it shouldn't be assumed. Second, in your scenario, since P(F|T) and P(F|~T) are both 1, then the flips don't provide any evidence for whether the coin is fair or trick. Your guess as to whether it's fair or not would be identical to your prior. Checking the coin thrower would be more science that would be factored into Bayes rather than having nothing to do with it.

Ruling out examining the coin is a way of showing how often in science we aren't able to just "examine the coin." That's not anti-science, that's just a fact. The coin is just a metaphor here. Physicists don't actually put cats in boxes with poison that might or might not kill them either... at least I hope not!

First, I must make a distinction between Bayes on three levels: ideal, descriptive, and practical. Bayes is ideal because it's a complete description of how correct evidence processing works. Bayes is descriptive in that any and all correct evidence processing can be described in Bayesian terms. When you talk about examples from science, you can describe experiments/evidence in terms of conditionals, you can describe the hypothesis as the prior, and the posterior represents how the experiment/evidence alters the prior. Mostly what you're talking about in this section of your post is Bayes as practical, meaning you're asking how working scientists can explicitly use Bayes the way that statisticians use Bayes. I would say that they don't and probably don't need to. It's more important to understand the principles and try to apply them as best as possible rather than putting actual numbers to priors, conditionals, and posteriors. However, that scientists don't and don't need to do this doesn't mean we can't describe what they do in those terms. It's much the same my profession of poker, where I can't explicitly work out hands using Bayes at the table, but I can try to think in Bayesian terms while making decisions even on an intuitive level (and work them out explicitly afterwards). 

As for your examples, taking Newton first, before QM I think it's fine to simply say that the evidence for Newton's laws was overwhelming and our estimation of their probability of being true was extremely close to 100%. It's not enough for the skeptic just to say "well, they could be wrong, we don't know for sure" to offer any kind of serious challenge to Newton. All we have to do to account for that "maybe we could be wrong" is not put our level of certainty at 100%. I don't really see how this is a challenge to Bayes though; it's really just a lesson on not assuming certainty (which is a lesson Bayes can impart as well since if you were 100% certain about anything then new evidence could never change your mind). 

With natural selection and God the situation is a bit different. This is really more of an issue of Occam and Priors rather than Bayes, because in both cases the hypotheses are attempting to explain the same set of data, so which is more likely has everything to do with their complexity, including how much is known VS how much is being assumed. In the case of natural selection, we're basically just taking what we know happens--genetic change across generations--and extrapolating that from the origin of life to the present. You can use natural selection to make certain predictions and retrodictions that you can't do with God. So with God all you've done is add complexity without adding any more predictive power. As I said elsewhere, in these situations, without the aid of Solomonoff Induction, it's most practical to simply dismiss/ignore the more complex hypothesis unless it produces evidence (in the form of unique predictions) above and beyond what the simpler hypothesis can. Thus, this is more of a Occamian principle than a Bayesian (but Solomonoff incorporates both Occam and Bayes into its formalism).

I'd highly encourage you to read that QM thread I linked to because this is a whole different can of worms. I do not believe the wavefunction is probabilistic. I think we treat it is as probabilistic because that's how we experience it. I used this example in that QM thread: imagine you have a line of gunpowder. The gunpowder forks into three paths. You are the fire, the gunpowder is the wavefunction, the split is the measurement. As the fire, you will experience having gone left, middle, OR right after the split/measurement. You can ask "what's the probability of me going left VS going middle or right?" and produce an answer of "33%," but the truth is that the probability of you going left, right, AND middle is 100%. You will go all three ways, even though the "you" after the split will have experienced only having gone left, middle, OR right. It will seem to you as if the other two "paths" have disappeared.

But I have to stress that the probabilistic nature of QM is entirely dependent on interpretation, not science. The equation that models how quantum systems behave (Shrodinger Wave Equation) is deterministic, and you have to interpret it by adding to it a stochastic collapse in order to make it probabilistic. The simplest interpretation is Many-Worlds, which just takes the Shrodinger Wave Equation as a complete description of QM, and like I said above, in such situations it's best to just assume the simplest explanation until unique evidence is produced that favors the more complex explanations. So you can't really turn to QM in order to make the argument that probability is inherent in reality. In my strong opinion, probability only exists in the mind. It expresses various states of ignorance about reality. Any time we're using probability, we're acknowledging there's some facet of reality we don't know or can't account for.

If you want to know how you can (ideally, again, this is currently completely impractical) precisely model the probability that an interpretation/hypothesis/theory is correct then you need to research Solomonoff Induction; in particular concepts like Kolmogorov complexity that's about describing reality and our theories about reality in binary code. Once you've described them in binary code, it's just a matter of comparing the length of code to generate a probability that they're correct.

But, again, that's only ideally. Practically speaking, we just have to do the best we can. To go back to my OP's thought experiment, nobody knows the ratio of trick coins to fair coins in the world. So we can all say we're uncertain about that. If you put a number to that uncertainty like "I think the ratio might be something like 10,000:1," are you any more or less sure by doing that than in just saying "I don't know?" I don't think there's anything wrong with putting a number to our uncertainty as long is doesn't make us think the number is accurate, and thus unduly lessen our uncertainty. But like I've said, whether you know the ratio of fair to trick coins, you DO know that if you flip heads enough times, you'll eventually be convinced it's a trick coin. That's what I mean by using Bayes to be descriptive. You can give a reasonable answer about how many flips it will take to convince you. You can do that without ever once doing an explicitly Bayesian calculation. But once you've become convinced, once you've put a number to how many flips it takes, it's possible to describe this entire process in Bayesian terms, including having the ability to put a number on your prior of the ratio of trick to fair coins.