|
|
Post by Aj_June on Jul 5, 2017 8:39:24 GMT
Eva YojimboRemember sometime back on old IMDB when I needed your help regarding how to quickly solve a Bayes' theorem question? Your answer was awesome. I am much quicker with Bayes and TP rule than I was previously. I made some changes in approaching the questions and one important thing that has helped me a lot is to use proper terminologies. For example, if the question is about economic expansion and probability of expansion is 0.6 then I properly mark compliment probability as P(E c ) = 0.4, simple thing I was not doing before. Also, using words like prior, posterior and conditional etc. helps a lot. But going through the same question I asked you I got a different concern today. I will repeat the question. Ok, I solved it this time in less than a minute and the answer is 0.667. But if you look at the question carefully then we can say that the probability of tech sector outperforming the market when economy expands is some sort of empirical probability. It is probably based on historical data. But probability that economy will expand is most probably a subjective probability. I mean come on nobody uses empirical probability when it comes to estimating economic expansion especially in a global economy which is never like it was in history. So my question is how relevant do you think is Bayes' when it comes to such questions that involve considerable subjective priority? I mean probably the real world application are based more on empirical data but do you think the resulting answer in such questions are really close to truth?
|
|
|
|
Post by Arlon10 on Jul 5, 2017 9:59:03 GMT
Eva Yojimbo Remember sometime back on old IMDB when I needed your help regarding how to quickly solve a Bayes' theorem question? Your answer was awesome. I am much quicker with Bayes and TP rule than I was previously. I made some changes in approaching the questions and one important thing that has helped me a lot is to use proper terminologies. For example, if the question is about economic expansion and probability of expansion is 0.6 then I properly mark compliment probability as P(E c ) = 0.4, simple thing I was not doing before. Also, using words like prior, posterior and conditional etc. helps a lot. But going through the same question I asked you I got a different concern today. I will repeat the question. Ok, I solved it this time in less than a minute and the answer is 0.667. But if you look at the question carefully then we can say that the probability of tech sector outperforming the market when economy expands is some sort of empirical probability. It is probably based on historical data. But probability that economy will expand is most probably a subjective probability. I mean come on nobody uses empirical probability when it comes to estimating economic expansion especially in a global economy which is never like it was in history. So my question is how relevant do you think is Bayes' when it comes to such questions that involve considerable subjective priority? I mean probably the real world application are based more on empirical data but do you think the resulting answer in such questions are really close to truth? I share your concerns about the data given, its means of derivation and its accuracy. I would though for the purposes of the problem assume that the data given is correct so long as it is not self defeating. You know me. I never make assumptions unless I remember that they are just that, assumptions. Given that 'O' (outperformance by tech) is higher in 'E' (an expanding economy), if the news is that O is low ('O'=zero, 'not O'=1) then the expectation of 'E' should go down, but 0.667 > 0.6 !?!? I suspect it should be much lower than 0.60. Show and label your work. Let's use O as the event of tech outperforming the market. Let's use E as the event of the economy expanding. We are given that P(E)=0.60 P(O|E)=0.70 P(O|not E)=0.10 and finally P(not O)=1 (the outperformance by tech event did NOT occur) We are asked to find the new P(E). A cautionary note here, you might be tempted to use P(O|not E) where you actually need P(not O|E). The first is 10 percent the other is 30 percent. I know how bizarre that appears, but we are accepting the given data so long as it is not self defeating. When I plugged the numbers into Baye's Theorem I got 18 percent. However I do not see much use in that. I have serious concerns about the given data, just as you do. I don't believe this problem is a good example of applying the Theorem. Edit > Notice that if you know whether tech outperforms than it's time you knew what the economy did. It's bizarre that they are even asking that question. |
|
|
|
Post by tickingmask on Jul 5, 2017 12:49:41 GMT
When I plugged the numbers into Baye's Theorem I got 18 percent. What formula did you plug the numbers into, O Wise One? Please show us humble ordinary people how you did this calculation so we can all marvel at your magnificence. |
|
The Lost One
Junior Member
@lostkiera
Posts: 2,677
Likes: 1,302
|
Post by The Lost One on Jul 5, 2017 14:41:29 GMT
When I plugged the numbers into Baye's Theorem I got 18 percent. I think AJ's right here: Baye's Theorem is P(A|B)= (P(B|A)P(A))/P(B) A is that the economy will not expand. B is that the technology sector doesn't outperform the market. P(A)=0.4 P(B|A)=0.9 P(B)=(0.4x0.9)+(0.6x0.3)=0.54 Therefore P(A|B)=(0.9x0.4)/0.54=0.667 |
|
|
|
Post by tickingmask on Jul 5, 2017 15:37:41 GMT
P(B)=(0.4x0.9)+(0.6x0.3)=0.54 No, you are completely wrong. P(B) = 1 because the outperformance by tech event did NOT occur. Arlon said this, so it must be the truth.
|
|
|
|
Post by general313 on Jul 5, 2017 16:19:34 GMT
P(B)=(0.4x0.9)+(0.6x0.3)=0.54 No, you are completely wrong. P(B) = 1 because the outperformance by tech event did NOT occur. Arlon said this, so it must be the truth.
I think Kiera and AJ are right. If in doubt and the problem is simple enough, a quick Monte Carlo sim can shed some light, as in the following Python code:
On my computer I get the following output: calc probability = 0.666988 |
|
|
|
Post by tickingmask on Jul 5, 2017 16:50:53 GMT
On my computer I get the following output: calc probability = 0.666988Then your Python code is obviously wrong. The fault probably lies in the random number generator you are using. You should have written the program in C++ instead. |
|
|
|
Post by general313 on Jul 5, 2017 17:22:41 GMT
On my computer I get the following output: calc probability = 0.666988Then your Python code is obviously wrong. The fault probably lies in the random number generator you are using. You should have written the program in C++ instead. It's quite amusing how some people think that a compiled language will somehow have a much higher quality random number generator, and that an interpreted language like Python could have such a poor random function as to be completely off. There may be a coding error in my program but it's certainly not the fault of the random number generator. |
|
|
|
Post by tickingmask on Jul 5, 2017 19:08:23 GMT
It's quite amusing how some people think that a compiled language will somehow have a much higher quality random number generator, and that an interpreted language like Python could have such a poor random function as to be completely off. Don't challenge me on this. I have argued with random number generators and won. Your inferior language will be using some kind of pseudo random method that simply doesn't satisfy my superior intellect and isn't really random at all. It should be using some genuinely random event, such as the emission of an alpha particle from a radioactive material, or a correction by Arlon of an erroneous mathematical calculation that somebody has made. |
|
|
|
Post by maya55555 on Jul 5, 2017 19:16:32 GMT
Are using this to try to calculate the PE of a stock?
Are you trying to evaluate the status of a company?
|
|
|
|
Post by general313 on Jul 5, 2017 19:18:42 GMT
It's quite amusing how some people think that a compiled language will somehow have a much higher quality random number generator, and that an interpreted language like Python could have such a poor random function as to be completely off. Don't challenge me on this. I have argued with random number generators and won. Your inferior language will be using some kind of pseudo random method that simply doesn't satisfy my superior intellect and isn't really random at all. It should be using some genuinely random event, such as the emission of an alpha particle from a radioactive material, or a correction by Arlon of an erroneous mathematical calculation that somebody has made.  |
|
|
|
Post by phludowin on Jul 5, 2017 20:10:49 GMT
When I plugged the numbers into Baye's Theorem I got 18 percent. I think AJ's right here: Baye's Theorem is P(A|B)= (P(B|A)P(A))/P(B) A is that the economy will not expand. B is that the technology sector doesn't outperform the market. P(A)=0.4 P(B|A)=0.9 P(B)=(0.4x0.9)+(0.6x0.3)=0.54 Therefore P(A|B)=(0.9x0.4)/0.54=0.667 This is correct. Statistic class, first year. Arlon10 is hilarious as he was at the time on the old IMDb when he argued the probability of passengers in an airplane getting their assigned seat. He even started multiple threads about it; each of them demonstrating how wrong he was. He should do comedy for math geeks. |
|
|
|
Post by Aj_June on Jul 5, 2017 20:46:31 GMT
Eva Yojimbo Remember sometime back on old IMDB when I needed your help regarding how to quickly solve a Bayes' theorem question? Your answer was awesome. I am much quicker with Bayes and TP rule than I was previously. I made some changes in approaching the questions and one important thing that has helped me a lot is to use proper terminologies. For example, if the question is about economic expansion and probability of expansion is 0.6 then I properly mark compliment probability as P(E c ) = 0.4, simple thing I was not doing before. Also, using words like prior, posterior and conditional etc. helps a lot. But going through the same question I asked you I got a different concern today. I will repeat the question. Ok, I solved it this time in less than a minute and the answer is 0.667. But if you look at the question carefully then we can say that the probability of tech sector outperforming the market when economy expands is some sort of empirical probability. It is probably based on historical data. But probability that economy will expand is most probably a subjective probability. I mean come on nobody uses empirical probability when it comes to estimating economic expansion especially in a global economy which is never like it was in history. So my question is how relevant do you think is Bayes' when it comes to such questions that involve considerable subjective priority? I mean probably the real world application are based more on empirical data but do you think the resulting answer in such questions are really close to truth? I share your concerns about the data given, its means of derivation and its accuracy. I would though for the purposes of the problem assume that the data given is correct so long as it is not self defeating. You know me. I never make assumptions unless I remember that they are just that, assumptions. Given that 'O' (outperformance by tech) is higher in 'E' (an expanding economy), if the news is that O is low ('O'=zero, 'not O'=1) then the expectation of 'E' should go down, but 0.667 > 0.6 !?!? I suspect it should be much lower than 0.60. Show and label your work. Let's use O as the event of tech outperforming the market. Let's use E as the event of the economy expanding. We are given that P(E)=0.60 P(O|E)=0.70 P(O|not E)=0.10 and finally P(not O)=1 (the outperformance by tech event did NOT occur) We are asked to find the new P(E). A cautionary note here, you might be tempted to use P(O|not E) where you actually need P(not O|E). The first is 10 percent the other is 30 percent. I know how bizarre that appears, but we are accepting the given data so long as it is not self defeating. When I plugged the numbers into Baye's Theorem I got 18 percent. However I do not see much use in that. I have serious concerns about the given data, just as you do. I don't believe this problem is a good example of applying the Theorem. Edit > Notice that if you know whether tech outperforms than it's time you knew what the economy did. It's bizarre that they are even asking that question. Arlon, Bayes' theorem is used to update a prior probability, using new information. You believe New England Patriots has a certain probability of winning Super Bowl but 2 days before the Super Bowl Tom Brady gets injured and won't play the final game. You adjust your probability of New England's winning based on this new information that Tom Brady is injured. Tech Sector can outperform the market whether economy expands or not albeit it has different probabilities of doing so depending on whether economy expands or not. Bayes' theorem tells you if tech sector did outperform then what is the probability the economy expanded . The tricky part was the wording of the question which asked Given the new information that the technology sector will not outperform the market, the probability that the economy will not expand is closest to? Bayes' is meant to work this way: P(Event|New information) =P(New information|Event)P(Event)/ [P(New information|Event)P(Event) + P(New information|Event C )P(Event C)] However, more simply, it is represented as Kiera wrote, P(A|B)= (P(B|A)P(A))/P(B) It is derived this way: P(A|B) = P(AB) / P(B) (From conditional probability formula) First step -> Make P(AB) the subject by algebraic adjustments P(AB) = P(A|B) × P(B) In the same way we also know from conditional probability formula that P(AB) = P(A|B) × P(B) Second step -> Make P(BA) the subject by same algebraic adjustments P(BA) = P(B|A) × P(A) Since P(AB) and P(BA) are equal, P(A|B) × P(B) = P(B|A) × P(A) Make P(B|A) the subject and we have P(B|A) = P(A|B) × P(B) / P(A)
or alternatively
P(A|B) = P(B|A) × P(A) / P(B) |
|
|
|
Post by Eva Yojimbo on Jul 6, 2017 1:27:28 GMT
So my question is how relevant do you think is Bayes' when it comes to such questions that involve considerable subjective priority? I mean probably the real world application are based more on empirical data but do you think the resulting answer in such questions are really close to truth? First, thanks for the kudos and you're very welcome for the help. Second, and to answer your question: it really depends. On the most abstract level, all beliefs and evidence ideally works on Bayes' Theorem; you simply can't process evidence correctly and arrive at correct beliefs without it. That said, the problem of priors--where they come from, how do you make sure they're accurate--is a key problem, and obviously the less empirical data you have the more inaccurate your updates are going to be; it's the old garbage in, garbage out aphorism. But, again, there's really no other or better way to do it. Saying you're fuzzy on the priors is tantamount to saying that you don't understand whatever phenomena that you're asking about. The more data you gather, the more accurate your numbers will be; beyond that all you can do is give your best guess. I developed my Bayesian skills playing poker, and you don't get much "fuzzier" than that because you don't have an infinite amount of time to study how individuals play certain hands in certain situations so all you can do is make some educated guesses based on a very limited data set and just hope you get more accurate with time. However, having a basic grasp on how it works and being able to make even fairly accurate general assessments is often enough to be a winning player, especially when the alternative for most is just playing from their gut and guessing much of the time. |
|
|
|
Post by Arlon10 on Jul 6, 2017 9:14:47 GMT
When I plugged the numbers into Baye's Theorem I got 18 percent. I think AJ's right here: Baye's Theorem is P(A|B)= (P(B|A)P(A))/P(B) A is that the economy will not expand. B is that the technology sector doesn't outperform the market. P(A)=0.4 P(B|A)=0.9 P(B)=(0.4x0.9)+(0.6x0.3)=0.54 Therefore P(A|B)=(0.9x0.4)/0.54=0.667 I'm sorry, I am not a fan of the mathematics used in statistical analysis since more than anything else good results depend on good data collection and most people fail good data collection. I do recognize most of the math though. I have used computer sims and found them to be untrustworthy in some cases probably because of patterns in the "random" numbers. There was no use of Baye's Theorem in my classes and the copy of it I found online has no '+' in it. Edit > Never mind this. I stand by the rest though. I stand by my original point that outperformance by tech is higher in an expanding economy therefore knowledge that it does not outperform would mean a decrease in the likelihood of expansion. At the same time, I also stand by my point that the word problem uses questionable data. A good statistician would perform other tests to get a better understanding of the relationship between outperformance and expansion which is not likely so simple as given. There are too many possible variables there. In this I agree with the person who complained about it before. |
|
|
|
Post by Arlon10 on Jul 6, 2017 10:33:29 GMT
P(B)=(0.4x0.9)+(0.6x0.3)=0.54 No, you are completely wrong. P(B) = 1 because the outperformance by tech event did NOT occur. Arlon said this, so it must be the truth. I'm sorry, but yes I did (mis)read "Given the new information that the technology sector will not outperform the market" as probability of 1, that is the event is "given." On further study I see that is not what the wording in the problem meant. As I already said I have not used the theorem in any of my classes and have no clear idea what use it is. I have no practice applying it. The one practical use I have ever seen for the theorem is showing how error can accumulate in data. That error does accumulate in data is the result of bad data collection methods and understanding in the first place. What I studied were methods of identifying why the data is in error, not simply that it is. It means revising the collection method in light of the lurking variable. It was obvious to me from the outset the data for this problem was not collected properly to make any predictions. The lesson from Baye's Theorem is that you need better data. |
|
|
|
Post by Arlon10 on Jul 6, 2017 11:28:34 GMT
No, you are completely wrong. P(B) = 1 because the outperformance by tech event did NOT occur. Arlon said this, so it must be the truth.
I think Kiera and AJ are right. If in doubt and the problem is simple enough, a quick Monte Carlo sim can shed some light, as in the following Python code: ... calc probability = 0.666988 I suppose that is the "correct" answer according to Baye's Theorem. I don't see the use of it though. I don't see any realistic scenario where you can benefit from such information. How did anyone know that tech would not outperform the economy without some new idea how the economy would perform? If you have some idea how the economy will perform shouldn't that be factored in? I'm sorry I would not encourage the use of that theorem. One use it does seem to have is showing how bad some guesses can be. |
|
|
|
Post by Arlon10 on Jul 6, 2017 12:38:58 GMT
I think AJ's right here: Baye's Theorem is P(A|B)= (P(B|A)P(A))/P(B) A is that the economy will not expand. B is that the technology sector doesn't outperform the market. P(A)=0.4 P(B|A)=0.9 P(B)=(0.4x0.9)+(0.6x0.3)=0.54 Therefore P(A|B)=(0.9x0.4)/0.54=0.667 This is correct. Statistic class, first year. Arlon10 is hilarious as he was at the time on the old IMDb when he argued the probability of passengers in an airplane getting their assigned seat. He even started multiple threads about it; each of them demonstrating how wrong he was. He should do comedy for math geeks. I'm sorry you wasted your time on such unrealistic scenarios. Yes, Baye's Theorem can help you see flaws in your data collection, but what other use is it? Notice you're treating the "new" information that tech does not outperform as one data item, but it's two. It requires new information about tech specifically and the market as well. Knowing "outperformance" requires two numbers even if you are only given one of them. I suppose if the guy in the cubicle next to you tells you that tech does not outperform the economy then you could use Baye's Theorem to make a better guess about the economy. However, since he had to know something new about the economy, why don't you just ask him? Or why don't you use whatever method he used to find the new information? Instead of the bad data you have? |
|
The Lost One
Junior Member
@lostkiera
Posts: 2,677
Likes: 1,302
|
Post by The Lost One on Jul 6, 2017 12:39:26 GMT
I stand by my original point that outperformance by tech is higher in an expanding economy therefore knowledge that it does not outperform would mean a decrease in the likelihood of expansion. Yes you're right - 0.667 is the probability that the market won't expand, not the probability that it will. That threw me at first too. Proponents of Bayes' theorem would probably acknowledge this problem (ie if you're not sure how accurate your assessment of P(B) is then you're not going to have much certainty about what you calculate P(A|B) to be). I think what they would say is well what other model do you have that's going to give you a more accurate prediction? |
|
|
|
Post by Arlon10 on Jul 6, 2017 12:56:25 GMT
I stand by my original point that outperformance by tech is higher in an expanding economy therefore knowledge that it does not outperform would mean a decrease in the likelihood of expansion. Yes you're right - 0.667 is the probability that the market won't expand, not the probability that it will. That threw me at first too. Proponents of Bayes' theorem would probably acknowledge this problem (ie if you're not sure how accurate your assessment of P(B) is then you're not going to have much certainty about what you calculate P(A|B) to be). I think what they would say is well what other model do you have that's going to give you a more accurate prediction? It's my fault for turning the question around to expansion rather than not expansion. So I got an 82% chance that it would not expand. The reason for that is I assumed P(not outperformance)=1 based on my misreading of "Given the new information that the technology sector will not outperform the market," What??? What new information? Notice if you divide .18 by the correct .54 you get 0.333333 which is the probability the economy does expand (1-0.66667). I suppose it's a pretty cool formula for what it does. I just don't see where in real life you are going know such a bizarre assembly of statistics or ever depend on them. |
|
