Re: So, the most recent XKCD as of this writing

1

XKCD sucks.


Posted by: Merganser | Link to this comment | 11-10-12 8:58 PM
horizontal rule
2

It's better mockery of the P

I am due to be converted to Bayesianism, presumably by Jaynes, but haven't gotten there yet.


Posted by: clew | Link to this comment | 11-10-12 8:59 PM
horizontal rule
3

Duh. Of the P < 0.05 than of frequentism per se.


Posted by: clew | Link to this comment | 11-10-12 8:59 PM
horizontal rule
4

(But even I know that the Bayesians believe we know that the possibility the sun has exploded is minuscule, minuscule, so that if the detector says Yes you bet it's wrong. Besides, if the sun has exploded saving the money won't do much good.)


Posted by: clew | Link to this comment | 11-10-12 9:01 PM
horizontal rule
5

The Origin of Statistics in the Breakdown of Prior Expectations?


Posted by: nosflow | Link to this comment | 11-10-12 9:01 PM
horizontal rule
6

That cartoon moved me into the neb view of xkdc.


Posted by: Moby Hick | Link to this comment | 11-10-12 9:03 PM
horizontal rule
7

That and the continuing wisdom emitted by his brain.


Posted by: Moby Hick | Link to this comment | 11-10-12 9:08 PM
horizontal rule
8

(Also, if the bayesian is so confident that the sun hasn't exploded, why are they consulting the machine in the first place?)


Posted by: nosflow | Link to this comment | 11-10-12 9:11 PM
horizontal rule
9

5 is awesome.


Posted by: Beefo Meaty | Link to this comment | 11-10-12 9:12 PM
horizontal rule
10

Andrew Gelman doesn't like the cartoon.


Posted by: Beefo Meaty | Link to this comment | 11-10-12 9:14 PM
horizontal rule
11

It's a really stupid cartoon. He should add dinosaurs.


Posted by: Moby Hick | Link to this comment | 11-10-12 9:16 PM
horizontal rule
12

Gelman doesn't address neb's question about the calculation, though, nor has anyone in this thread.


Posted by: teofilo | Link to this comment | 11-10-12 9:21 PM
horizontal rule
13

I am confident that Cosma will be along in due course and set me straight.


Posted by: nosflow | Link to this comment | 11-10-12 9:22 PM
horizontal rule
14

Does he have grading to do or something?


Posted by: teofilo | Link to this comment | 11-10-12 9:24 PM
horizontal rule
15

To answer the OP question, though, the frequentist is supposed to set ((the sun has exploded and it rolls at most one six) at 0 since it hasn't happened and or(the sun hasn't exploded) at 1.Then 1 in 36 falls out after that.

Of course, if the chance of the sun exploding is really 0, there's no need to ask the machine, and the final answer contradicts the premise, but that's not really a defect in the cartoon.


Posted by: beamish | Link to this comment | 11-10-12 9:24 PM
horizontal rule
16

Sorry:

(the sun has exploded and it rolls at most one six) at 0 and (the sun hasn't exploded) at 1.


Posted by: beamish | Link to this comment | 11-10-12 9:26 PM
horizontal rule
17

I can't tell if the sun has exploded because it's night.


Posted by: Moby Hick | Link to this comment | 11-10-12 9:29 PM
horizontal rule
18

While we're criticizing Munroe's presentation of statistics, this post is good on how this comic is also misleading.


Posted by: teofilo | Link to this comment | 11-10-12 9:31 PM
horizontal rule
19

Of course, if the chance of the sun exploding is really 0, there's no need to ask the machine, and the final answer contradicts the premise, but that's not really a defect in the cartoon.

I think if the cartoon is internally incoherent in multiple respects, that is a flaw.


Posted by: nosflow | Link to this comment | 11-10-12 9:37 PM
horizontal rule
20

Check the moon


Posted by: beamish | Link to this comment | 11-10-12 9:39 PM
horizontal rule
21

I think if the cartoon is internally incoherent in multiple respects, that is a flaw.

It's supposed to be a reductio ad absurdum of frequentist statistics.


Posted by: beamish | Link to this comment | 11-10-12 9:40 PM
horizontal rule
22

I mean it occurred to me as well that if you set p(the sun hasn't exploded) to 1 then you can arrive at 1/36; that's why I said, in the post, that we're assuming that we don't know whether the sun has exploded (and I am assuming that if we are willing to set p(X) to 1 then we at least consider ourselves to know that X). After all, as the cartoon header says: "we're not sure". P(X) = 1 sounds like being sure.


Posted by: nosflow | Link to this comment | 11-10-12 9:41 PM
horizontal rule
23

Surely one can be a frequentist without believing that p = 0.05 is a magical value.


Posted by: nosflow | Link to this comment | 11-10-12 9:43 PM
horizontal rule
24

… and the absurdity of the frequentist's position seems to depend on his treating that way: since he knows already that the sun hasn't exploded, p(yes) = 1/36, which is less than 0.05, therefore, we must be in the has-exploded-no-lie case. Everything turns on the frequentist's taking being less than 0.05 as somehow decisive, so an equally good case against frequentism could have been constructed thus:

A frequentist playing with dice rolls two sixes. Since the probability of his having done so is 1/36, he concludes that he didn't do so.


Posted by: nosflow | Link to this comment | 11-10-12 9:48 PM
horizontal rule
25

22

The null hypothesis is that the sun hasn't exploded. Since in that case there is only 1/36 of observing a "yes" and we observe a "yes" we can reject the null hypothesis (using a 5% significance level).


Posted by: James B. Shearer | Link to this comment | 11-10-12 9:50 PM
horizontal rule
26

Ok, so when the frequentist says that there's a 1/36 chance of "that happening" he doesn't mean there's a 1/36 chance of observing a "yes" sans phrase, just that there's a 1/36 chance of observing a "yes" given that the following hypothesis, that the sun hasn't exploded.

I can accept that.

I remain confused about the use of the 5% significance level.


Posted by: nosflow | Link to this comment | 11-10-12 9:53 PM
horizontal rule
27

The sun may have exploded, but we won't know about it for another 8 minutes.


Posted by: Spike | Link to this comment | 11-10-12 9:55 PM
horizontal rule
28

the author clearly thinks the unthinking use of 5% significance as dispositive is moronic and that this is widespread. Why are you pretending not to get this? This is pose is stupider by far than the comic, and seems to be your sole lens on xkcd. That seems a shame.


Posted by: Turgid Jacobian | Link to this comment | 11-10-12 9:59 PM
horizontal rule
29

The unthinking use of 5% significance as dispositive is moronic. Is it a key point in the dispute between bayesians and frequentists, with the former opposing and the latter supporting it? That would be surprising.

I'm not sure what "this is pose" means, so I don't know what is supposed to be my sole lens on xkcd, but I've certainly linked/alluded positively to it in the past.


Posted by: nosflow | Link to this comment | 11-10-12 10:04 PM
horizontal rule
30

There is no difficulty in proving any statistical approach whatever to be absurd, if we suppose universal idiocy to be conjoined with it. … Men really ought to leave off talking a kind of nonsense on this subject.


Posted by: JS Mill | Link to this comment | 11-10-12 10:11 PM
horizontal rule
31

is
pose=pretending not to get it. I guess I may be incorrect to assume you have sufficient good will to go along with your more than sufficient intelligence.


Posted by: Turgid jacobian | Link to this comment | 11-10-12 10:12 PM
horizontal rule
32

Just to make sure that I get it (I have enough good will, it's my knowledge that's lacking), the connection between the P value and frequentist statistics is that the frequentist has to use such tests, since she can't appeal to prior odds for unprecedented events?


Posted by: beamish | Link to this comment | 11-10-12 10:19 PM
horizontal rule
33

Oh, I wasn't sure if "pose" was a typo for "post", you see, in addition to the extraneous "is". Anyway, I've certainly been critical of xkcd in the past without pretending not to get, or actually not getting (I believe), what he's on about, and I wouldn't even say that in this case I've been pretending not to get what he's on about. It really seems to me as if he's just conjoining unrelated things. Perhaps I am, in fact, simply not getting it, and mindlessly applying a 5% cutoff is the province of all and only frequentists.


Posted by: nosflow | Link to this comment | 11-10-12 10:23 PM
horizontal rule
34

26

I remain confused about the use of the 5% significance level.

By longstanding convention observations of some purported effect are called significant if there is a less than 5% probability that they would have occured by chance absent the effect.

So if for example you want to test whether painting classrooms purple improves student performance you take a bunch of classrooms and paint half of them purple at random. You then look at how the students do. If the students in the purple classrooms do better to an extent that would occur less than 5% of the time by chance (if purple paint made no difference) then you have a significant result which you can publish and get tenure and government grants. If the students in the purple classrooms do better but by an amount that would occur by chance 6% of the time then you don't have a significant result, your paper will probably be rejected and you won't get tenure.

This creates various bad incentives which are widely deplored but not to the extent of doing much of anything about them.


Posted by: James B. Shearer | Link to this comment | 11-10-12 10:29 PM
horizontal rule
35

29.1: no.


Posted by: Beefo Meaty | Link to this comment | 11-10-12 10:29 PM
horizontal rule
36

33: Bayesians tend to argue against any kind of threshold for meaningfulness in statistical analysis. Frequentists tend to be less worried about that as a problem. But these are mostly sociological descriptions, as opposed to strongly methodological ones, I think.


Posted by: Beefo Meaty | Link to this comment | 11-10-12 10:32 PM
horizontal rule
37

34: I mean I know that; the question is why it's being trotted out here.


Posted by: nosflow | Link to this comment | 11-10-12 10:32 PM
horizontal rule
38

Anyhow Shearer isn't broadly right; in many cases you have to do a fair bit more justification than just pointing to a significant result in order to get through peer review. Unless you're Daryl Bem.


Posted by: Beefo Meaty | Link to this comment | 11-10-12 10:35 PM
horizontal rule
39

Isn't it 1/36 because the cartoon was already written to pick on a p-value of less than .05 and no other value than 1/36 derived from a pair of dice could meet that requirement? This isn't science or math here. It's xkcd.


Posted by: fake accent | Link to this comment | 11-10-12 10:38 PM
horizontal rule
40

"Already written"? It's not as if he got the cartoon complete with the two dice and had only to write in a fraction somewhere in the lower left panel.


Posted by: nosflow | Link to this comment | 11-10-12 10:45 PM
horizontal rule
41

I just mean that if you've got two dice and want to make fun of the .05 thingy, you've got only 1/36 to work with.


Posted by: fake accent | Link to this comment | 11-10-12 10:54 PM
horizontal rule
42

I'm going to have to conclude that my knowledge of statistics is insufficient to understand why the meaning of this cartoon isn't as clear & obvious as it appears.


Posted by: bingobangoboy | Link to this comment | 11-10-12 10:58 PM
horizontal rule
43

I have no knowledge of the baseball stat world, but I thought article on Nate Silver was interesting.


Posted by: fake accent | Link to this comment | 11-10-12 11:00 PM
horizontal rule
44

Although what I know of Macleans makes me doubt what it publishes.


Posted by: fake accent | Link to this comment | 11-10-12 11:03 PM
horizontal rule
45

37

34: I mean I know that; the question is why it's being trotted out here.

As has been stated by others XKCD is making fun of people who use a (totally arbitrary) 5% cutoff for significance.


Posted by: James B. Shearer | Link to this comment | 11-10-12 11:05 PM
horizontal rule
46

I have already responded to those others, James, even in comments prior to the one you're answering. As JS Mill pointed out, a reductio of frequentists is ineffective if it only works if the frequentist happens to hold additional absurd positions.


Posted by: nosflow | Link to this comment | 11-10-12 11:08 PM
horizontal rule
47

The article in 43 is indeed interesting. Maybe we should declare Sam Wang our new god instead.


Posted by: teofilo | Link to this comment | 11-10-12 11:12 PM
horizontal rule
48

46

I have already responded to those others, James, even in comments prior to the one you're answering. As JS Mill pointed out, a reductio of frequentists is ineffective if it only works if the frequentist happens to hold additional absurd positions.

So you are expecting a cartoon to be fair and balanced?


Posted by: James B. Shearer | Link to this comment | 11-10-12 11:21 PM
horizontal rule
49

I think people fetishizing statistical significance too much is a problem, but I don't really think it's a frequentist vs Bayesian thing. The comic amused me, but it seems like it's targeting people who don't think when they apply statistics, not frequentists.


Posted by: essear | Link to this comment | 11-11-12 3:18 AM
horizontal rule
50

I agree with Shearer, essear, and Turgid Jacobian: the point is to make fun of unthinking incoherent ways of thinking about statistics, not about frequentism per se.

That said, my understanding of frequentism is that it's an incoherent theory-less cookbook-style approach to statistics. I've often found that people who are frequentists hold absurd positions. (e.g. Sash/a Vol/okh's old argument that if the margin of error of a poll is 3 and it shows one candidate up 2.9% then you literally haven't learned anything and you can't say that one candidate is up.)


Posted by: Unfoggetarian: "Pause endlessly, then go in" (9) | Link to this comment | 11-11-12 4:50 AM
horizontal rule
51

50.2: But I feel like the debate among real statisticians is fundamentally different. Doesn't Cosma identify more as a frequentist?


Posted by: essear | Link to this comment | 11-11-12 5:45 AM
horizontal rule
52

I had a discussion with another physicist recently where he was saying that he found the following problem clarified his thinking on Bayesian vs. frequentist: you are told that a coin has been flipped 14 times and landed heads 10 times. The coin is now going to be flipped two more times. Would you take a bet at even odds that the coin will land heads both times? Supposedly the frequentist will judge that the probability of heads, based on the observed trials, is ~71% and so it has a slightly better than 1/2 chance of landing heads in two trials, and will take the bet. The Bayesian will sum over all assumptions about the probability of heads, weighted by the existing information, and decide not to take the bet.

My objection to this is that it sounds like we're dealing with a stupid frequentist and a stupid Bayesian. A reasonable frequentist might also say "I know that most coins land heads 50% of the time, so let's call that the null hypothesis. On the basis of the observed information, I can't reject the null hypothesis at 95% confidence level. So I'll proceed assuming the null hypothesis is true, which means I expect the chance of winning the bet is 25% and I won't take it." A reasonable Bayesian would probably take not a uniform prior but one peaked around 50/50 heads/tails.

I don't think these kinds of toy examples are very useful-- I think probably people can do reasonable things from a frequentist point of view or a Bayesian point of view, and on a case-by-case basis it's going to involve making judgements that can't be discussed purely as airy abstractions.


Posted by: essear | Link to this comment | 11-11-12 5:56 AM
horizontal rule
53

I think probably people can do reasonable things from a frequentist point of view or a Bayesian point of view

Nice prior.


Posted by: JP Stormcrow | Link to this comment | 11-11-12 6:03 AM
horizontal rule
54

I'm never able to understand what a "null hypothesis" is other than a strongly peaked prior...


Posted by: Unfoggetarian: "Pause endlessly, then go in" (9) | Link to this comment | 11-11-12 6:08 AM
horizontal rule
55

The other xkcd foray into this area was the one that showed 20 studies, one coming up with p<0.05 and a headline touting that result. No mention of false detection rate correction.


Posted by: SP | Link to this comment | 11-11-12 6:11 AM
horizontal rule
56

I'm curious about whether there was any historical link between the ideas, but it seems like there is some kind of cocktail of Karl Popper and old-fashioned frequentist cookbook reject-the-null-hypothesis stuff that strongly influences the way people (including scientists) talk about science in a really annoying way. Lots of "science can never prove things, only disprove them" or "we're allowed to rule out hypotheses but never to rule one in" or whatever. It's all obvious nonsense-- the same people will turn around and talk about what the data is actually telling us is true in their next breath-- but it still seems kind of pernicious to me.


Posted by: essear | Link to this comment | 11-11-12 6:16 AM
horizontal rule
57

Here is the link to the comic mentioned in 55


Posted by: Spike | Link to this comment | 11-11-12 6:26 AM
horizontal rule
58

55: On the one hand, you're obviously right that it's unfair to ignore the more sophisticated methods that people actually use. On the other hand, there's something deeply unsatisfying about the approach where you notice that you're getting the wrong answers so you add a new ad-hoc fix on top of that, rather than scrapping the original approach because it produced nonsense.


Posted by: Unfoggetarian: "Pause endlessly, then go in" (9) | Link to this comment | 11-11-12 6:28 AM
horizontal rule
59

Two Karls, one one cup?


Posted by: Moby Hick | Link to this comment | 11-11-12 6:29 AM
horizontal rule
60

59 to 56.


Posted by: Moby Hick | Link to this comment | 11-11-12 6:29 AM
horizontal rule
61

Heh. Two Karls P, even.


Posted by: essear | Link to this comment | 11-11-12 6:34 AM
horizontal rule
62

56

... It's all obvious nonsense ...

It doesn't seem like nonsense (much less obvious nonsense) to me.


Posted by: James B. Shearer | Link to this comment | 11-11-12 6:37 AM
horizontal rule
63

58: Constructing and testing a reasonable hypothesis isn't a ad hoc fix to frequentist statistics.


Posted by: Moby Hick | Link to this comment | 11-11-12 6:38 AM
horizontal rule
64

Munroe replies to Gelman.


Posted by: essear | Link to this comment | 11-11-12 6:39 AM
horizontal rule
65

55

... No mention of false detection rate correction.

Possibly because most people (even people who know about 5% significance levels) have never heard of it (I had to google it).

Anyway it would be a bit hard to apply if the 20 studies were done by 20 different groups.


Posted by: James B. Shearer | Link to this comment | 11-11-12 6:43 AM
horizontal rule
66

an equally good case against frequentism could have been constructed thus:

A frequentist playing with dice rolls two sixes. Since the probability of his having done so is 1/36, he concludes that he didn't do so.

That would make a good cartoon!

I have twice found $20 bills on sidewalks. Am I not allowed to joke about economics-inspired oversimplifications?


Posted by: politicalfootball | Link to this comment | 11-11-12 6:49 AM
horizontal rule
67

62: Why not? There are less facile arguments to make, but for one thing, at a purely logical level "ruling out A" and "ruling in not-A" mean basically the same thing.

I still haven't finished my stupid grant proposal so I should stop commenting here today.


Posted by: essear | Link to this comment | 11-11-12 6:50 AM
horizontal rule
68

I found four twenty dollar bills in Lake Michigan. I got out of the water because there must have been a dead guy in the water whose pockets were being washed empty.


Posted by: Moby Hick | Link to this comment | 11-11-12 6:52 AM
horizontal rule
69

67

62: Why not? There are less facile arguments to make, but for one thing, at a purely logical level "ruling out A" and "ruling in not-A" mean basically the same thing.

OK, you have to be talking about the kind of hypothesis that can be falsified by a single observation (ignoring errors). Like "there are no black swans".


Posted by: James B. Shearer | Link to this comment | 11-11-12 7:05 AM
horizontal rule
70

64: d'oh. Do your research, Munroe!


Posted by: Beefo Meaty | Link to this comment | 11-11-12 7:30 AM
horizontal rule
71

I'm never able to understand what a "null hypothesis" is other than a strongly peaked prior...

Me neither!


Posted by: heebie-geebie | Link to this comment | 11-11-12 7:54 AM
horizontal rule
72

In fact, I often feel like I should really go take a statistics class and learn some of the basics.


Posted by: heebie-geebie | Link to this comment | 11-11-12 7:55 AM
horizontal rule
73

56: I assume the historical connection is that Fisher was the one who originally suggested 0.05 as a rule of thumb.


Posted by: Beefo Meaty | Link to this comment | 11-11-12 8:02 AM
horizontal rule
74

I think the issue is that frequentist statistics are the methods most frequently taught to non-statisticians and to people who have the attitude 'give me the recipe so I don't have to think about it'. They are the people most likely to do cartoon-able things. If everyone was taught Bayesian statistics I'm sure laughable things would happen based on only ever using the prior you got taught that one time in your one required stats class for your major.


Posted by: PGD | Link to this comment | 11-11-12 8:02 AM
horizontal rule
75

The thing to remember about the null hypothesis is that almost nobody appreciates it when you look at H0 and say "Ho." And they get really angry when you imply anything about their mother related to Hos.


Posted by: Moby Hick | Link to this comment | 11-11-12 8:02 AM
horizontal rule
76

The trouble with the comic isn't that it's not getting across any kind of point, it's that one of the perks of xkcd has been that it holds up to at least casual scrutiny. The one linked in 55 might be about an error which well-established methods exist to correct, but it does reflect real life (when lots of similar small studies are done all the time, considered as separate studies, then have publication bias and media bias stacked on). I do get that the current comic is about p and rejecting the null, but beyond that it makes about as much sense as something an editorial cartoonist would make if p-standards somehow became a national issue. The frequentist knows there's a dice-roller in the machine.

I have more sympathy for him against the critique in 18. There really were pundits explicitly rejecting the concept of poll aggregation and basing their forecasts on feelings and momentum, over whom all poll aggregators and analysts, including Nate Silver, collectively triumphed. (And the comic doesn't mention Nate Silver - it wasn't changed, was it?)


Posted by: Minivet | Link to this comment | 11-11-12 8:06 AM
horizontal rule
77

56: I assume the historical connection is that Fisher was the one who originally suggested 0.05 as a rule of thumb.

As I understand it the original 0.05 rule of thumb was proposed as a rule of thumb for the particular problem for which it was introduced, where 5% made sense for calling some result "significant", not as a general rule about what statistical significance is.


Posted by: nosflow | Link to this comment | 11-11-12 8:06 AM
horizontal rule
78

Not that it is directly relevant to the cartoon under discussion, but I've been seeing things that seem to imply that Bayesian:frequentist::Nate Silver:Karl Rove. That's just bizarre since Rove was adjusting based on priors also. He was just picking really stupid priors.


Posted by: Moby Hick | Link to this comment | 11-11-12 8:08 AM
horizontal rule
79

56: agreed on the annoying cocktail, but surely crude positivism ('the data tell us XXX') is no better. I think the reason Popper is so popular among working scientists (even though philosophers of science have long since moved past) is that he lets you do a nod to the problems with crude positivism without really coming to grips with the role unproven assumptions perform in science in rationalizing contradictions, ignoring inconsistencies, etc. Which if a science is predicting well most working scientists do get to ignore in their ordinary work.


Posted by: PGD | Link to this comment | 11-11-12 8:09 AM
horizontal rule
80

77: I don't know the story but that sounds very plausible. I have been told that he really did not intend for it to be a general rule used by everyone.


Posted by: Beefo Meaty | Link to this comment | 11-11-12 8:10 AM
horizontal rule
81

78: that is awfully stupid.


Posted by: Beefo Meaty | Link to this comment | 11-11-12 8:11 AM
horizontal rule
82

77 is incorrect---the 5% level was very much a general rule about what significance is. But he did reconsider later in life, and especially in his last book, as suggested in 80.


Posted by: Kreskin | Link to this comment | 11-11-12 8:16 AM
horizontal rule
83

65- I don't think "most people have heard of it" is a criteria for inclusion in an xkcd strip. And as can clearly be seen by the detailed goggles on the stick figure, they're definitely all the same research group.


Posted by: SP | Link to this comment | 11-11-12 8:22 AM
horizontal rule
84

73 56: I assume the historical connection is that Fisher was the one who originally suggested 0.05 as a rule of thumb.

But what does Fisher have to do with Popper, if anything?


Posted by: essear | Link to this comment | 11-11-12 8:23 AM
horizontal rule
85

The guy from Blues Traveler?


Posted by: heebie-geebie | Link to this comment | 11-11-12 8:25 AM
horizontal rule
86

The passage I had in mind is from, I believe, The Design of Experiments: "it is usual and convenient for experimenters to take 5 percent as a standard level of significance, in the sense that they are prepared to ignore all results which fail to reach this standard." But his writings on the topic were not at all consistent.


Posted by: Kreskin | Link to this comment | 11-11-12 8:26 AM
horizontal rule
87

Would love to follow the thread, but I must be going...


Posted by: Kreskin | Link to this comment | 11-11-12 8:26 AM
horizontal rule
88

84: oh, I see. Yeah, dunno. Somebody wake up ttaM...


Posted by: Beefo Meaty | Link to this comment | 11-11-12 8:27 AM
horizontal rule
89

86: somewhat differently, but apparently consistently, stated here:

If one in twenty does not seem high enough odds, we may, if we prefer it, draw the line at one in fifty (the 2 per cent. point), or one in a hundred (the 1 per cent. point). Personally, the writer prefers to set a low standard of significance at the 5 per cent. point, and ignore entirely all results which fail to reach this level. A scientific fact should be regarded as experimentally established only if a properly designed experiment rarely fails to give this level of significance.


Posted by: Beefo Meaty | Link to this comment | 11-11-12 8:29 AM
horizontal rule
90

I'm going to wait for 36 comments by other people before I accept that Kreskin is really gone.


Posted by: Moby Hick | Link to this comment | 11-11-12 8:31 AM
horizontal rule
91

I wonder where I got the idea in 77, then.


Posted by: nosflow | Link to this comment | 11-11-12 8:31 AM
horizontal rule
92

83

65- I don't think "most people have heard of it" is a criteria for inclusion in an xkcd strip. And as can clearly be seen by the detailed goggles on the stick figure, they're definitely all the same research group.

OK, the comic is a little incoherent in that the blame is implicitly being placed on the media instead of on the scientists for failing to present their results properly but the end result is the same.


Posted by: James B. Shearer | Link to this comment | 11-11-12 8:38 AM
horizontal rule
93

89-That's certainly how newspapers report polls. Within MOE so they're really tied! Who knew political pundits were so knowledgable about the history of science.


Posted by: SP | Link to this comment | 11-11-12 8:39 AM
horizontal rule
94

The quote in 89 is interesting because it shows how frequentist methodology is designed for repetition of experiments (see the last sentence in the quote). Lots of softer science areas don't place much institutional emphasis on repeating experiments, use historical data points as though they were repeated experiments on the same underlying process, etc. Also, it makes little sense to talk about statistical significance outside of a theory -- every fact you pull from a full population survey will be statistically significant at the highest possible level but your theory is the only thing telling you if they will still be true a week from now. This is obvious in one sense but worth repeating as most of the real uncertainty is usually in the relationship between the finding and the theory.


Posted by: PGD | Link to this comment | 11-11-12 8:49 AM
horizontal rule
95

[Gah, previous attempt at any entry eaten]

0. To answer the question in the OP, it's really

Pr("Yes"|Sun) = 1/36

so the measurement is relatively improbable under the hypothesis that the Sun is still there. Since 1/36 < 1/20, the conventional threshold, the frequentist idiot is supposed to blindly reject the hypothesis that the Sun is still there, and conclude that it is not there.

The 0.05 p-value threshold owes its prominence, I think, to three factors. (I) Fisher's use of it, though as I recall he thought of this more or less as the highest acceptable threshold, not an especially compelling one. (II) 5% corresponds very nearly to 2 standard deviations with a Gaussian distribution, and even doctors can multiply by two. (III) Self-reinforcing conventions, especially outside the community of statisticians. I'm quite sure that no actual frequentist statistician today would say this was anything more than a convention, and few of them would say it was a wise convention.

To treat the cartoon more seriously than it deserves, I'd actually say that it would show some, very weak, evidence for the sun having blown up. The probability of "Yes" when the Sun's blown up is fairly high (35/36), but low otherwise (1/36), so it's sensitive to what we're trying to infer (it's not just a Gygax test), and while the error probabilities are really quite high, it's not worthless. One can elaborate on this.

Ideally, null hypotheses are actually models which try to account for apparently interesting patterns in the data by means of boring or already-known processes. (E.g., might this apparent clumping of events really be due to mere fluctuations? Can we explain the spread of this mutant through selectively-neutral processes?) Fetishizing parameter values of zero is stupid.

Turning to the Bayesian side, the Bayesian statistician actually is much more convinced that the Sun has blown up after seeing the machine say "Yes" than they were before. Specifically, the ratio of belief in "Sun kaboom" to belief in "Sun fine" has increased by a factor of 35. According to the scale published by some aggressively-Bayesian forensic scientists, this is "moderate support" for the Sun having blown up.

Of course, there is nothing in the machinery of Bayesian inference which tends towards true, reliable answers, except under assumptions where there are also reliable frequentist methods (and sometimes no Bayesian method can work, while there are reliable frequentist ones). This has a lot to do with why the sensible practices of Bayesian data analysts have so little to do with the ideology they profess.

Shorter me: Randall Munroe is wrong on the Internet.


Posted by: Cosma Shalizi | Link to this comment | 11-11-12 8:52 AM
horizontal rule
96

So "due course" for Cosma is about eleven and a half hours. Good to know!


Posted by: nosflow | Link to this comment | 11-11-12 8:56 AM
horizontal rule
97

You'd really need more trials to establish a reasonable CI on that.


Posted by: Moby Hick | Link to this comment | 11-11-12 8:58 AM
horizontal rule
98

84: The statistical theorists of hypothesis testing (Fisher, Neyman and Pearson) and Popper seem to have completely ignored each other; they may not even have been aware of each other's existence. The first author I know of to find this puzzling was R. B. Braithwaite (Scientific Explanation, 1953).

Gerg Gigerenzer has some papers on the theme of how so much of the social sciences got caught up in a weird Fisher/Neyman-Pearson/positivist/Popperian mish-mash of rejecting hypotheses that parameters are exactly 0, at the 5% level. (See e.g. the group volume The Empire of Chance.) Whether other historians disagree, I don't know.


Posted by: Cosma Shalizi | Link to this comment | 11-11-12 9:04 AM
horizontal rule
99

My 2/3 completion of the statistics 101 course on Udacity.com has left me unable to participate meaningfully in this discussion. The Internet is a pack of lies and false promises.


Posted by: Robert Halford | Link to this comment | 11-11-12 9:05 AM
horizontal rule
100

||

Anyone have restaurant recs for New Orleans for places that might have a few vegetarian options? If not, I'm open to a citywide beignet crawl.

|>


Posted by: Sir Kraab | Link to this comment | 11-11-12 9:08 AM
horizontal rule
101

Dammit, now I'm being lured back into procrastination-land by all of Cosma's links.

Also by the basic cognitive dissonance that: I'm applying for a grant so I can hire a postdoc; in my field good postdocs are expected to be independent and so the person I hire might not collaborate with me at all; but to decide whether this is a good use of their money the government requires me to write 15 pages about what I will do, rather than asking the hypothetical future postdoc what s/he will do. I guess things make more sense in labs or whatnot.


Posted by: essear | Link to this comment | 11-11-12 9:10 AM
horizontal rule
102

99: Oh, that one.


Posted by: Cosma Shalizi | Link to this comment | 11-11-12 9:10 AM
horizontal rule
103

101: "The postdoctoral fellow will assist in all aspects of this work."


Posted by: Cosma Shalizi | Link to this comment | 11-11-12 9:14 AM
horizontal rule
104

103: Yeah, but, that's almost certainly false.


Posted by: essear | Link to this comment | 11-11-12 9:15 AM
horizontal rule
105

96, 97: So far as I can tell, this is a pretty good account of my response times.


Posted by: Cosma Shalizi | Link to this comment | 11-11-12 9:15 AM
horizontal rule
106

104: Distinguishing it from any other sentence in a grant proposal how, exactly?


Posted by: Cosma Shalizi | Link to this comment | 11-11-12 9:16 AM
horizontal rule
107

Hey, Cosma, I see you got tenure. Congrats.


Posted by: nosflow | Link to this comment | 11-11-12 9:18 AM
horizontal rule
108

107: Thanks, but here promotion to associate professor is a pre-tenure up-or-out step. I come up for tenure in 2014. (And if that happens, promotion to full professor a few years later.)


Posted by: Cosma Shalizi | Link to this comment | 11-11-12 9:20 AM
horizontal rule
109

Hey, Cosma, I see you work at a bats institution.


Posted by: nosflow | Link to this comment | 11-11-12 9:21 AM
horizontal rule
110

Congrats on getting associate professorhood.


Posted by: Moby Hick | Link to this comment | 11-11-12 9:23 AM
horizontal rule
111

106: Hmm. I guess I'm not yet cynical enough.


Posted by: essear | Link to this comment | 11-11-12 9:31 AM
horizontal rule
112

Yeah, but, that's almost certainly false.

What's the p-value on that?


Posted by: AcademicLurker | Link to this comment | 11-11-12 9:36 AM
horizontal rule
113

re that link of Cosma's in 105, now I want to know more about the "numbers of men in ten Prussian army corps killed by the kick of a horse..."

Congrats Cosma.


Posted by: Barry Freed | Link to this comment | 11-11-12 9:41 AM
horizontal rule
114

Thanks, all!

109: I have never seen any bats around campus in the seven years I've been here. But there are raptors (falcons, maybe?) which sometimes perch on the edge of the building opposite my office window, looking about with infinite disdain.

113: Enjoy, enjoy.


Posted by: Cosma Shalizi | Link to this comment | 11-11-12 9:48 AM
horizontal rule
115

||

A thing I did not formerly know: Hubert Sumlin's funeral was paid for by Mick Jagger and Keith Richards.

|>


Posted by: nosflow | Link to this comment | 11-11-12 11:11 AM
horizontal rule
116

We have hawks that perch on our building and the buildings nearby and swoop around outside my window. It's pretty great.


Posted by: Beefo Meaty | Link to this comment | 11-11-12 11:46 AM
horizontal rule
117

5!!


Posted by: Nworb Werdna | Link to this comment | 11-11-12 11:47 AM
horizontal rule
118

||

Today's cultural confusion: why do all the female mannequins on display here have such prominent nipples?

|>


Posted by: essear | Link to this comment | 11-11-12 12:18 PM
horizontal rule
119

118: Kindness?


Posted by: JP Stormcrow | Link to this comment | 11-11-12 12:22 PM
horizontal rule
120

Too much air conditioning.


Posted by: Moby Hick | Link to this comment | 11-11-12 12:22 PM
horizontal rule
121

Isn't it cold where you are? I can't keep track of your travels.


Posted by: nosflow | Link to this comment | 11-11-12 12:22 PM
horizontal rule
122

By the sidebar, I was pwned.


Posted by: Moby Hick | Link to this comment | 11-11-12 12:25 PM
horizontal rule
123

And it only hurt for little bit at first.


Posted by: JP Stormcrow | Link to this comment | 11-11-12 12:35 PM
horizontal rule
124

The link in 102 is quite devastating.


Posted by: teofilo | Link to this comment | 11-11-12 1:42 PM
horizontal rule
125

102, 124: I've now removed Udacity Stats 101 from my list of things I feel like I should do but probably never will.

Although in an odd way, I am now more likely to investigate the course, because I often need examples of how not to do online education.


Posted by: rob helpy-chalk | Link to this comment | 11-11-12 1:55 PM
horizontal rule
126

To Udacity's credit, Thrun responded pretty quickly on the Udacity blog and linked to that critical review acknowledged lots of problems with the course, said they'd be revising it, and I think the revisions might already be in the course now.


Posted by: fake accent | Link to this comment | 11-11-12 2:58 PM
horizontal rule
127

||

Cockatoolmaker!

|>


Posted by: teofilo | Link to this comment | 11-11-12 3:00 PM
horizontal rule
128

I haven't actually done the stats course, but I get Udacity's email announcements since I've done/am doing other courses. I really thought the CS101 was done well.


Posted by: fake accent | Link to this comment | 11-11-12 3:01 PM
horizontal rule
129

126: Huh, so he did. The author of the original critique also responded.


Posted by: teofilo | Link to this comment | 11-11-12 3:11 PM
horizontal rule
130

Speaking of doctors and statistics--and since people who tend to know things are around--I've been a bit curious about something I saw a short report on in Science News a few months back. It discussed work that purported to describe how to build an "evidence" scale which was analogous to temperature. This seems to be the key paper: "Measurement of statistical evidence on an absolute scale following thermodynamic principles" - V. J. Vieland, J. Das, S. E. Hodge, S.-C. Seok.

I'm going to presume that it is either wrong or trivial/not very informative or useful (and based on my quick peruse of the paper I'd guess the latter). So having done the hard work of noticing it in Science News and finding the paper in arXiv, I leave it to others to judge and report back.


Posted by: JP Stormcrow | Link to this comment | 11-11-12 3:53 PM
horizontal rule
131

That's a really long paper.


Posted by: Moby Hick | Link to this comment | 11-11-12 5:50 PM
horizontal rule
132

I was kind of hoping for a figure that started with "Likert scale where the coders kept forgetting whether 5 or 1 was 'strongly agree'" and ended with "atomic clock."


Posted by: Moby Hick | Link to this comment | 11-11-12 5:54 PM
horizontal rule
133

Is there anything funnier than Likert scales?


Posted by: Beefo Meaty | Link to this comment | 11-11-12 5:59 PM
horizontal rule
134

1) Definitely no.
2) Probably no.
3) I don't care.
4) Probably yes.
5) Definitely yes.


Posted by: Beefo Meaty | Link to this comment | 11-11-12 6:00 PM
horizontal rule
135

134: bravo


Posted by: Turgid Jacobian | Link to this comment | 11-11-12 6:12 PM
horizontal rule
136

130: It's horribly organized, so it's very hard for me to say what they're actually doing. Looking at their Table 1, their "evidence" looks like it's basically the ratio

(likelihood at its maximum)/(integrated likelihood with a flat prior)

What they are taking this to be evidence for; why they think that statistics should look just like the ideal gas law, when most of physics doesn't look like the ideal gas law; why they make a big deal about having an equation of state for variables they make up so it has the form they like, as opposed to physical quantities which can be independently measured; God alone knows.

I'd say, not worth anyone's time.


Posted by: Cosma Shalizi | Link to this comment | 11-11-12 7:18 PM
horizontal rule
137

136: Thanks. As I suspected.

But before I die I want our whole understanding of the universe to unify around some information/entropy formulation that explains everything and it's the end of science and everyone gives up and goes home and that's that. So I keep my eye out.


Posted by: JP Stormcrow | Link to this comment | 11-11-12 7:27 PM
horizontal rule
138

Oh, and is simple enough for me to understand.


Posted by: JP Stormcrow | Link to this comment | 11-11-12 7:27 PM
horizontal rule
139

I'd say, not worth anyone's time.

Roger that!

Hey Cosma, I heard a reasonably spirited defense of Bayesian cognitive models (as, basically, just as good an approach as any other on the computational level) the other day, you will no doubt be thrilled to hear.


Posted by: Beefo Meaty | Link to this comment | 11-11-12 7:29 PM
horizontal rule
140

Boy, I killed all the threads. Look!


Posted by: Beefo Meaty | Link to this comment | 11-11-12 8:19 PM
horizontal rule
141

Can't make me.


Posted by: JP Stormcrow | Link to this comment | 11-11-12 8:25 PM
horizontal rule
142

just as good an approach as any other on the computational level
So, lacking evidence, solid theoretical support, and a realistic implementation?


Posted by: Eggplant | Link to this comment | 11-11-12 8:43 PM
horizontal rule
143

So much for that attempt to start conversation.


Posted by: Eggplant | Link to this comment | 11-11-12 9:07 PM
horizontal rule
144

Sorry, Sifu already killed all the threads.


Posted by: teofilo | Link to this comment | 11-11-12 9:08 PM
horizontal rule
145

My trollery is too crude for thread resuscitation.


Posted by: Eggplant | Link to this comment | 11-11-12 9:16 PM
horizontal rule
146

Maybe people have started going to bed or something? I guess not everyone has tomorrow off.


Posted by: teofilo | Link to this comment | 11-11-12 9:17 PM
horizontal rule
147

142: that would be my take, sure. But so is it best to ignore that level of analysis entirely?


Posted by: Beefo Meaty | Link to this comment | 11-11-12 9:20 PM
horizontal rule
148

(I would have responded sooner but I got distracted trying to figure out why the New Yorker published an enormous, fawning profile of Kid Rock.)


Posted by: Beefo Meaty | Link to this comment | 11-11-12 9:21 PM
horizontal rule
149

148: That is a puzzler. Is he trying for a comeback or something?


Posted by: teofilo | Link to this comment | 11-11-12 9:23 PM
horizontal rule
150

Well, apparently he has become very popular as a country rocker, and wrote Romney's campaign song. Still weird, though.


Posted by: Beefo Meaty | Link to this comment | 11-11-12 9:26 PM
horizontal rule
151

But so is it best to ignore that level of analysis entirely?
It's central to our existence and the only area of science still at the alchemical stage. I wouldn't.


Posted by: Eggplant | Link to this comment | 11-11-12 9:32 PM
horizontal rule
152

151 to 148.


Posted by: Eggplant | Link to this comment | 11-11-12 9:33 PM
horizontal rule
153

Have scientists really made no inroads at all into the study of Kid Rock?


Posted by: teofilo | Link to this comment | 11-11-12 9:34 PM
horizontal rule
154

Kid Rock is Magma?


Posted by: JP Stormcrow | Link to this comment | 11-11-12 9:38 PM
horizontal rule
155

Presumably, everyone was off watching the Bears play a shitty football game.


Posted by: Stanley | Link to this comment | 11-11-12 9:42 PM
horizontal rule
156

151: well, right. So then, the argument goes, Bayesian models of inference are at least analytically and computationally tractable and relatively intuitive, even if they're not immediately, obviously plausible.


Posted by: Beefo Meaty | Link to this comment | 11-11-12 9:44 PM
horizontal rule
157

(I would have responded sooner but I got distracted trying to figure out why the New Yorker published an enormous, fawning profile of Kid Rock.)

Was it written by SF/J?


Posted by: nosflow | Link to this comment | 11-11-12 9:54 PM
horizontal rule
158

157: I don't believe so!


Posted by: Beefo Meaty | Link to this comment | 11-11-12 9:56 PM
horizontal rule
159

139: Unless they have a good fix for the problem pointed out by Eberhardt and Danks, I'd say they're in very deep trouble even as computational models. (Whereas one can do perfectly respectable Bayesian statistics with, say, ACT-R models.)


Posted by: Cosma Shalizi | Link to this comment | 11-11-12 10:23 PM
horizontal rule
160

Seems to be by someone named Kelefa Sanneh. That abstract is like the perfect distillation of the ridiculousness of this kind of article.


Posted by: teofilo | Link to this comment | 11-11-12 10:24 PM
horizontal rule
161

And speaking of distillation, I'm currently polishing off the bottle of Evan Williams I bought a while back. Anyone have recommendations on other whiskeys I should try?


Posted by: teofilo | Link to this comment | 11-11-12 10:31 PM
horizontal rule
162

Bulleit, and their rye's not too shabby either.


Posted by: Stanley | Link to this comment | 11-11-12 10:39 PM
horizontal rule
163

ISTR that the Hudson Baby Bourbon is quite nice, but it's v. spendy. (I just had it once at a bar.)


Posted by: nosflow | Link to this comment | 11-11-12 10:47 PM
horizontal rule
164

"A rapping rock star" is funny.


Posted by: nosflow | Link to this comment | 11-11-12 10:51 PM
horizontal rule
165

But before I die I want our whole understanding of the universe to unify around some information/entropy formulation that explains everything and it's the end of science and everyone gives up and goes home and that's that.

There's no shortage of people claiming to do things like that. You can just pretend like one of them isn't a crackpot. We'll lie to you as you're dying, if you want.


Posted by: essear | Link to this comment | 11-12-12 1:06 AM
horizontal rule
166

Don't listen to him, JP. Physicists have their reasons for delaying the time when there's a grand unified theory of everything.


Posted by: teofilo | Link to this comment | 11-12-12 1:10 AM
horizontal rule
167

re that link of Cosma's in 105, now I want to know more about the "numbers of men in ten Prussian army corps killed by the kick of a horse..."

You ask, we answer. It's from Ladislaus Bortkiewicz "The Law of Small Numbers" (1898); not obviously available online, but according to this biography (http://www-history.mcs.st-andrews.ac.uk/Biographies/Bortkiewicz.html) over twenty years, fourteen army corps of the Prussian army suffered between 0 and four soldiers killed per year from the kick of a horse.

You'd think that, after a bit, they would have figured it out and shot the horse.


Posted by: ajay | Link to this comment | 11-12-12 3:05 AM
horizontal rule
168

I doubt I'll learn enough statistics to fully understand a thread like this, much less the stuff linked from it, but I'd really like to have a better understanding of Bayesian statistics. It seems like I've been running into Bayes rule everywhere lately.


Posted by: fake accent | Link to this comment | 11-12-12 3:08 AM
horizontal rule
169

But before I die I want our whole understanding of the universe to unify around some information/entropy formulation that explains everything

Check the quantitative biology section of the Arxiv regularly. I've found that several times a year someone deposits a paper claiming to have discovered the secret of Life, The Universe, And Everything. Said papers invariably involve information theory.


Posted by: AcademicLurker | Link to this comment | 11-12-12 4:44 AM
horizontal rule
170

Oh, and bonus points if it's a cellular automaton. Maybe like a new kind of shit.


Posted by: JP Stormcrow | Link to this comment | 11-12-12 5:22 AM
horizontal rule
171

170: Yeah, that's what I was thinking of with my comment about crackpots above.


Posted by: essear | Link to this comment | 11-12-12 5:51 AM
horizontal rule
172

I kind of see how information is important, but it also seems like a case of it becoming a dominant metaphor in our time because of the prevalence of computers. Just like people used to have a "clockwork universe" and then in the 19th century thinking in terms of heat engines became popular. Metaphors driving the way we think about the world track technology. I'm sure I've said this before. Probably someone wrote a book about it or something.


Posted by: essear | Link to this comment | 11-12-12 6:11 AM
horizontal rule
173

159: I mean, I think in this case they weren't even talking about being tied to the data at a level that would privilege the bayesian explanation particularly; it was more that, hey, these models are easy to create and simulate, and they could be a synecdoche for a replicator process or whatever, so let's see if people behave at least consistently with hierarchical versions of these models because maybe that'll give us some information about representational structure.

The "synecdoche for a replicator process or whatever" is my gloss; the person making this case skipped the "a replicator process or" part of the phrase.


Posted by: Beefo Meaty | Link to this comment | 11-12-12 6:20 AM
horizontal rule
174

If recent history has taught us nothing, its that fawning biographers are usually banging their subjects.


Posted by: Spike | Link to this comment | 11-12-12 6:40 AM
horizontal rule
175

Oh, now I think I get "null hypothesis".


Posted by: heebie-geebie | Link to this comment | 11-12-12 7:02 AM
horizontal rule
176

If 175 is to 174, you've got a great stats text.


Posted by: Moby Hick | Link to this comment | 11-12-12 7:05 AM
horizontal rule
177

Or some really weird porn.


Posted by: Moby Hick | Link to this comment | 11-12-12 7:07 AM
horizontal rule
178

175 was indeed to 174. I was being extremely hilarious.


Posted by: heebie-geebie | Link to this comment | 11-12-12 7:08 AM
horizontal rule
179

So you can't help me find weird porn?


Posted by: Moby Hick | Link to this comment | 11-12-12 7:12 AM
horizontal rule
180

Be the weird porn you want to see in the world, Mobes.


Posted by: heebie-geebie | Link to this comment | 11-12-12 7:15 AM
horizontal rule
181

170, 171: I was also thinking of Edward Fredkin.

172: Yes, I think there is a tendency to go all in* on metaphors derived from the predominant technology** of the age. Somewhere I read (and have frustratingly not found again) a Twain essay where he gives his late 19th century view. There probably is a book, I first saw it discussed in a critique of Minsky back in the 80s.

*In the pre-Petraeus sense.


Posted by: JP Stormcrow | Link to this comment | 11-12-12 7:19 AM
horizontal rule
182

why they think that statistics should look just like the ideal gas law, when most of physics doesn't look like the ideal gas law;

Because so many in the professoriate are full of hot air?

(The above came to me in a dream.)


Posted by: nosflow | Link to this comment | 11-12-12 8:15 AM
horizontal rule
183

Huh.


Posted by: Eggplant | Link to this comment | 11-12-12 8:22 AM
horizontal rule
184

I actually owned the book in 170 at one point. That fucker weighed like 6 pounds.


Posted by: Spike | Link to this comment | 11-12-12 8:25 AM
horizontal rule
185

Statistics are annoying because my college library appears to have stopped buying physical books in 2004 in favor of all-electronic offerings. So I don't know where to go to familiarize myself with the basics of bioinformatics. Presumably whatever books are available would be largely collections of glorified software package manuals, but at least it would be a book instead of a collection of unprintoutable computer files accessible from a small number of computers.


Posted by: Cryptic ned | Link to this comment | 11-12-12 8:36 AM
horizontal rule
186

Is there not a Bioinformatics for Dummies?


Posted by: Spike | Link to this comment | 11-12-12 8:46 AM
horizontal rule
187

Well I'll be damned. I guess that would be where to start.

Latest edition published in 2006.


Posted by: Cryptic ned | Link to this comment | 11-12-12 8:51 AM
horizontal rule
188

I'm sure nothing has changed in the field over the past 6 years.


Posted by: Spike | Link to this comment | 11-12-12 9:03 AM
horizontal rule
189

At least nothing new that a dummy can understand.


Posted by: Moby Hick | Link to this comment | 11-12-12 9:04 AM
horizontal rule
190

Thanks to 114 and 167 everything I know about Poisson distributions I know from reading "Gravity's Rainbow" and links posted in Unfogged threads.


Posted by: Barry Freed | Link to this comment | 11-12-12 9:06 AM
horizontal rule
191

everything I know about Poisson distributions

It's just about counting. You learned it in kindergarten.


Posted by: essear | Link to this comment | 11-12-12 1:22 PM
horizontal rule
192

Right. One fish, two fish. Rouge fish, bleu fish.


Posted by: JP Stormcrow | Link to this comment | 11-12-12 3:00 PM
horizontal rule