## Rain: stochastic processes and dummy pronouns

Today's xkcd:

Mouseover title: "Hi, I'm your new meteorologist and a former software developer. Hey, when we say 12pm, does that mean the hour from 12pm to 1pm, or the hour centered on 12pm? Or is it a snapshot at 12:00 exactly? Because our 24-hour forecast has midnight at both ends, and I'm worried we have an off-by-one error."

I'll leave it to readers to compose the corresponding jokes for economists, physicists, anthropologists, literary theorists, stand-up comedians, and so on.

Update — Topher Cooper explains in the comments what the PoP really means:

The probability of precipitation (PoP) is the product of confidence that it will rain within the area being analyzed and the expected percentage of the area that the rain will fall on if it does rain. Its no wonder people are confused. This is a more sensible metric then it sounds, since it boils down to the probability that any particular person in the area who hears the forecast will be rained on.

So call it PoEP = Probability of Experiencing Precipitation". And if each hour's PoP were statistically independent of the others, we could combine them by inverting them relative to 1 (thus turning PoP into PoNEP = "Probability of Not Experiencing Precipitation"), taking the product, and then inverting again. (Of course the hourly PoPs are not independent, but that sort of thing never stops a statistician…)

Excellent comments, overall – but I'm disappointed not to get any scripts for an economist-meteorologist, who might analyze humidity and temperature in terms of supply-demand curves, or weather-system movement in terms of the balance of trade. After all, our U.S. weather is largely the result of competition between Canadian and Mexican air masses, right? Does rational choice theory really apply here?

Or a script for a literary-theorist-turned meteorologist, who might undertake to deconstruct meteorological aquacentrism as revealed in the experience of precipitation, etymologically "falling forward", which we might better call decipitation to underline the post-hegemonic opposition between prescription and description.

1. ### Yerushalmi said,

April 26, 2018 @ 9:06 am

All of these questions actually *have* bugged me for years. I was positively gleeful to see them reflected in xkcd.

2. ### Yerushalmi said,

April 26, 2018 @ 9:09 am

Oh, and a little mouseover-title follow-up in-joke for those familiar with the Talmud: עד ועד בכלל או עד ולא עד בכלל?

3. ### Robert Coren said,

April 26, 2018 @ 9:50 am

I'm pretty sure that the use of a numerical probability (expressed as a percentage) of precipitation in weather forecasts arose sometime during my childhood (1950s-60s), and I recall my father objecting to it on the grounds that it made no sense to talk about the probability of an isolated event.

4. ### Yerushalmi said,

April 26, 2018 @ 9:58 am

5. ### BZ said,

April 26, 2018 @ 10:49 am

I am a programmer. I minored in mathematics, and, as evidenced by my posting here, have an interest in linguistics as well. Oh, and more than a passing interest in meteorology. I *have* thought about the probability questions in the past, but in fact it's really easy. The probability is the *confidence* the forecasters have in their forecast. And it's not time or area-affected. In fact your probability will be the *maximum* not the average of probabilities over times and locations. If there is a big difference in probabilities across area or time it will be called out explicitly. You can see this if you poke around the National Weather Service websites. You can get point forecasts and hourly forecasts which all list probabilities.

6. ### Ellen K. said,

April 26, 2018 @ 11:06 am

@BZ

If the probability is the confidence they have in their forecast, then shouldn't it be a 80% chance of not raining?

7. ### Ralph Hickok said,

April 26, 2018 @ 3:26 pm

@Robert Coren:
I've heard your father's objection a number of times from various sources and I really don't understand it.

Surely it makes sense to say, "If I flip this coin, there's a 50% chance that it will come up heads" or "If I roll this die, there is a 16.67% chance that it will be a 6."

8. ### david said,

April 26, 2018 @ 4:01 pm

Ellen,
It's about 33% 0.8 to the fifth power

Ralph,

What you say is true if you have previously flipped the coin or rolled the die many times and know they are "fair". There are coins and dice that don't behave as you assume this one will.

9. ### Draconaes said,

April 26, 2018 @ 4:54 pm

@david:
In that case, the issue is with the specific percentage given, no? Not with the concept of giving a percentage at all?

10. ### Xmun said,

April 26, 2018 @ 9:27 pm

Harrumph. Neither 12pm nor 12am exist (despite their prevalence in print). It should be 12 noon and 12 midnight.

11. ### D.O. said,

April 26, 2018 @ 10:10 pm

But what would someone who majored in meteorology say?

12. ### Topher Cooper said,

April 26, 2018 @ 10:42 pm

I didn't major in meteorology but I did work as a programmer for a few years for a subsidiary of the Weather Company (formerly called The Weather Channel, which is now another subsidiary of The Weather Company; it was an odd job in many ways — one of the things I worked on was cranking out numerical predictions not of the weather but of what the next forecast would predict about the weather). The probability of precipitation (PoP) is the product of confidence that it will rain within the area being analyzed and the expected percentage of the area that the rain will fall on if it does rain. Its no wonder people are confused. This is a more sensible metric then it sounds, since it boils down to the probability that any particular person in the area who hears the forecast will be rained on. Here is the National Weather Service's description of the matter: https://www.weather.gov/ffc/pop

13. ### david said,

April 27, 2018 @ 7:52 am

Draconaes,

It's for those numbers. Topher Cooper explained it inthe update.

14. ### Ellen K. said,

April 27, 2018 @ 12:48 pm

Topher Cooper, thanks.

15. ### John Swindle said,

April 27, 2018 @ 3:33 pm

@Xmun: The ones that worry me are 12:00N and 12:00M and their variants as sometimes found on parking signs. A.M. and P.M. are from Latin. Is 12:00N noon or noctis? Is 12:00M midnight or meridi-something?

16. ### Robert Coren said,

April 27, 2018 @ 3:37 pm

I can't ask my father what he was thinking, but my inclination is to suggest that, while it's possible to run the experiment with flipping a coin as may times as you like, the weather prediction basically says that "in 80% of the cases which exactly these conditions, it will rain", but the objection is that "exactly these conditions" are most likely only going to occur once.

17. ### Topher Cooper said,

April 27, 2018 @ 7:56 pm

This isn't linguistics, though, so I think I'll resist any temptation to go on any further, even if someone has questions — that'll be hard if anyone does.

18. ### MattF said,

April 27, 2018 @ 9:55 pm

Some years ago, I met a meteorologist who was gleeful about meeting a physicist– he was pleased that it wouldn't be necessary to explain what 'probability' meant. The embarassing thing was that I was just about to ask him that.

19. ### peterv said,

April 28, 2018 @ 1:09 am

@MattF

You should not feel bad about not knowing what probability is. We’ve had a calculus for manipulating probability statements since the middle 1660s and a formal axiom system for these statements since the 1920s. But there is still no agreement, even among people who work with the modeling of uncertainty in particular domains, about the semantics of probability statements. The subject is inherently difficult.

If you wish to know more about the primary alternative semantics for statements of probability, I would recommend the short book of Donald Gillies: Philosophical Theories of Probability.

20. ### Andy Stow said,

April 28, 2018 @ 7:52 am

If they want the probability that a particular person will be rained on, shouldn't they also multiply it by the percent of people who are outdoors? It seldom rains in my house.

I'll get my coat.

21. ### Topher Cooper said,

April 28, 2018 @ 8:10 am

@peterv … and it is hard not to take sides, as you did when you used the phrase "modeling uncertainty" as the goal. That puts you squarely on the "subjectivist probability" (the basis of "Bayesian" statistics) side rather than the "frequentist probability" (the basis of "traditional" statistics; excellently illustrated by Robert Coren's view of probability as being about statements like "'in 80% of the cases which exactly these conditions, it will rain'") which is the major opinion divide.

I would also strongly recommend Barnett's "Comparative Statistical Inference," It does a brilliant job of balancing the practical and philosophical aspects with clear writing. Get it from the library or used since the \$300 price tag for the latest (not very recent, '99) edition is a bit hard to justify unless your degree is on the line, even for one of my favorite academic books.

22. ### Topher Cooper said,

April 28, 2018 @ 8:18 am

@Andy. People complain about my prolix writing, but when I try to be brief, relying on readers having some common sense, someone like you busts my chops about it. :-b

23. ### Robert Coren said,

April 28, 2018 @ 9:45 am

@Topher Cooper" Thanks for the thorough explanation; I withdraw my implied objection. However, I'm pretty sure they weren't doing all that 55-60 years ago when my father made his complaint.

24. ### Topher Cooper said,

April 28, 2018 @ 10:57 am

@Robert Coren: Your statement is true with probability very, very close to 1.0 since computers only got strong enough to allow scientists to start experimenting with meteorological "numerical forecasting" a bit more than 50 years ago — and the results were pretty worthless for quite a while. To be fair, though, probability statements about a weather forecast can be justified in terms of the subjective probability interpretation. Bayesian statistics was in no way mainstream back then, but subjective probabilities — as rational measures of rational uncertainty — is what everyone thinks a probability statement means, and that has always been true.. If you would answer to "I just flipped a normal coin. What is the probability that it landed 'heads'" with "50%" or "close to 50%", then you are taking the question to be about a subjective probability. A ridiculously rigid frequentist would either answer "The question is nonsensical, since probability is about the frequency of a kind of event in a sequence of similar events, and so does not apply to specific events" or "Taking that event as a sequence of length one, the answer is either "0" or "1", I don't know which."

25. ### peterv said,

April 28, 2018 @ 11:18 am

@Topher Cooper

Your comment caused great laughter in this household. In fact, I am, and have always been, a strong supporter of frequentist interpretations of probability. I am not, however, a proponent of probability theory as the best or only model for uncertainty.

26. ### Topher Cooper said,

April 28, 2018 @ 3:54 pm

@peterv I have frequently described myself as a philosophical, and theoretical Bayesian but a strong believer in traditional statistics for almost all practical uses. As far as frequentist vs subjectivist probability theories: like all correctly done mathematics, they are neither right nor wrong, they just are (in this case, just are both valid "models" (in the mathematical sense) of the same formal theories. Problems created by applying either one to a specific, real world, situation is not their fault.

27. ### loonquawl said,

April 29, 2018 @ 6:27 am

@Topher Cooper : "But the spread of these different runs lets you estimate how much variation there is in the model, which allows you to put a probability that if you ran the model, say, thousands of times," – I loved that part of your explanation. So there is a weather model, which gets modeled. And the meta-model is the one actually making the predictions? I now realized that in watching (and building an opinion on the reliability of) weather predicitons, i actually ran meta-meata -models of the actual weather. And depending on you view of what human conciousness actually is, the meat does not stop there.
——
Another xkcd quote: "I'm so meta even this acrnonym"

28. ### mg said,

April 29, 2018 @ 2:32 pm

(Of course the hourly PoPs are not independent, but that sort of thing never stops a statistician…)

I'm a statistician, and that sort of thing absolutely stops us – we think about it whenever we're setting up a study or an analysis model and have ways to account for lack of independence.

The people it *doesn't* stop are those who are not trained in statistics but have access to the ever-easier to use (or, more often, mis-use) statistical software that's out there.

29. ### Topher Cooper said,

April 29, 2018 @ 4:51 pm

@ML "but I'm disappointed not to get any scripts for an economist-meteorologist".

Ironically, my job as a programmer for a meteorology company was for a group which produced apps for customers who were concerned about meteorological factors in economics — specifically for the energy sector. We provided tools for wind farms to determine when the wind is likely to be too high or too low. We provided tools for managers of solar power arrays with predictions of incident sunlight. But the bread-and-butter for the group, and what I primarily worked on, was tools for "energy traders" — people who speculated on short term oil futures. They looked for advantage in whether or not it would be cold in a particular region, so that they could predict heating fuel use, and looked for advantage in whether or not it would be hot in a particular region so that power plants would need more or less fuel to generate electricity to run air-conditioners. It's the job I had (over close to 50 years in computers) with the least potential social positives; but I judge that it was socially neutral rather than negative — the energy traders were mostly trading money back and forth between them, betting with each other who could better predict fuel demands.

So I can't give you what you were looking for specifically, but our group's meteorologist could be said to be a meteorologist-economist, though not an economist-meteorologist.

30. ### Topher Cooper said,

April 29, 2018 @ 5:21 pm

@ML "(Of course the hourly PoPs are not independent, but that sort of thing never stops a statistician…)" As @mg said, that is something that very much concerns statisticians. It also concerns meteorologists, and software engineers who munge meteorological (or other statistical) data. There is nothing wrong with the PoPs not being independent — no meteorological data is truly independent of any other meteorological data if there is enough time, given their distance from each other, for their influence to connect. That is precisely what the "butterfly effect" (a term coined by — hey, we're back to linguistics! — by Edward Lorenz, one of the first people to experiment with numeric meteorological forecasting, and the closest thing to a specific discover of the near ubiquity of chaos in non-linear systems of equations) refers to.

Who it doesn't "stop" is people who perform calculations on dependent random variables that are only appropriate for independent variables. For example, a hopeful picnic-er isn't stopped from saying while standing in a drenching rain at 4:50PM "Don't worry, this is probably going to stop in 10 minutes because the 5:00PM forecast is for a 15% PoP". And apparently linguists can't be stopped either — since they, not statisticians — sometimes (at least once, anyway) use formulas requiring independence on dependent values.

31. ### Topher Cooper said,

April 29, 2018 @ 5:39 pm

@loonquawl: "So there is a weather model, which gets modeled." No there is a weather model which gets run — repeatedly which is how one produces a result from such a model. This is precisely what one does in any statistical simulation (a.k.a a Monte-Carlo simulation). It is one of the foundations of technological civilization, even ignoring weather forecasting. Like much of what engineering consists of, the public has no idea the details of what is being done and rarely cares about it — they only see the results. If you are concerned about the validity of the technique, you better never fly in an airplane again — or drive in a car or bus or train, or cross a modern bridge, or use modern medicines, or … In fact, you probably should go to an old house, sit in the basement and never come out.

32. ### maidhc said,

April 30, 2018 @ 3:13 am

Where I live (SF Bay area) our weather is affected by physical characteristics such as the presence of mountain ranges, yet weather forecats rarely take this into account.

33. ### mg said,

April 30, 2018 @ 9:35 am

@maidhc – I'm really surprised. In the Boston area, forecasts always include the variation between coast vs. inland, Cape Cod area vs. further north, etc. No one would pay attention to a meteorologist who didn't, since things like snow amounts vary drastically over the area.

34. ### Topher Cooper said,

April 30, 2018 @ 10:54 am

@maidhc: Be assured that they are taken into account in the models. But the models are calculated — and reported — on a grid which, at least when I was dealing with them several years ago, were at the very highest long range precision, was ½°×½° of lat/long (the models themselves would sometimes shift dynamically into a finer local grid for a small region but only as part of calculating the values for the reported grid points; also I'm ignoring that the grid is actually 3D and the altitude is key in dealing with topographic features like you report). At the latitude of SF, that's a rectangle about 35 miles by 21 miles. It is the job of local forecasters, and companies like AccuWeather and Intellicast (the product of my subsidiary company, which scored number one in accuracy of temperature forecasts of the national level alternatives while I was there) to interpret the grid forecast to what is going to go on on a more localized level.

If they aren't taking those mountains into account — well, the stereotype of SF media forecasters as getting lazy because the weather patterns are so regular there, might actually be true.

35. ### Topher Cooper said,

April 30, 2018 @ 10:57 am

@maidhc and mg:
Actually, I've noticed a similar, but more subtle, apparent neglect of a local topographically induced pattern here in the Boston area.

It is sometimes very noticeable that there are distinct micro-climates on either side of route 128 (a.k.a., Interstate 95, but it is a point of local pride to never call it that), which circles Boston. Crossing it one frequently finds that it is raining on one side but only misting or drizzling on the other, or there will be a several degree difference between the two sides, particularly noticeable when it creates a "rain-snow" line (just think about what that might mean, anyone puzzled by that phrase who is from further south). Presumably this is a product of a narrow heat-island effect over the highway, perhaps with some effect of turbulence caused by the traffic.

But, although the highway is frequently treated as a convenient point of reference in local forecasts ("Outside of 128 winds will be more gusty") they are talking about it as an arbitrary landmark, not as something of intrinsic meaningfulness.

36. ### mg said,

April 30, 2018 @ 11:02 am

@Topher – what makes you think they are using 128 merely as a marker of convenience? Everyone here knows that it's a real change marker. Same with 495. While it certainly is very convenient that there are highways in the right places to use as referents for weather reports, that doesn't mean it's the only reason the meteorologists use them in their reports. Just because they don't state the obvious each time doesn't mean that they and residents aren't aware of it.

37. ### Topher Cooper said,

April 30, 2018 @ 11:03 am

@mg: "No one would pay attention to a meteorologist who didn't, since things like snow amounts vary drastically over the area." Yes, Mark Twain could have said "If you don't like the weather in New England where you are, just walk a few hundred yards" instead of what he did say. Of course, he was probably talking about Connecticut, where he lived, which is New England Lite.

38. ### Topher Cooper said,

April 30, 2018 @ 1:07 pm

@mg: Of course the residents are aware of it, or at least one is — now I know two are. I've been living one town away from 128 for the last 35 years. Actually, I never doubted that it was commonplace observation — its pretty obvious. (I worked near 495 — for those not from here, a larger, outer circumferential highway — for a number of years, and have had lots of reasons to cross it over the years, but never noticed as strong an effect, but there is no question in my mind that that might be observational bias).

I know they are just (or in the vast majority of cases) using it as a convenient, recognizable line on the map, because they never, never refer to it any differently than they refer to actually arbitrary markers like "North of Worcester", "The Connecticut border," and "The New Hampshire border". Also they are referring to everything outward from 128 or between 128 and 495, or from 128 out to Worcester, while the effect I'm referring too blends back into a smooth progression outward from Boston or from the sea within a mile or two of the 128 transition — it is a micro-climate, not just an abrupt transition.

Neither do I doubt that the local forecasters know about it as well, my observation is that they do not seem to incorporate it into their forecasts, like they do the Cape Cod Canal (which marks the border of the Cape — which is technically now an island rather than a cape).

Not really a complaint, mind you, or at least not a confident one. It may be too hard to turn the general observation into specific forecasts and warning. But I have heard "Watch out for possible high wind gusts after crossing the Bourne and Sagamore bridges [which cross the CC Canal]", though in decades of living here have never heard "Watch out for spots of black ice just after going from Lexington to Lincoln" (a border that is roughly along 128), or "It may be a bit slippery near Minuteman High School [just past 128 from most of Lexington] so be careful dropping off students." Yet I am always alert for such when the temperature is hovering around 32°F, there is some dampness in the air or on the ground and I'm heading out that way.

39. ### Robert Coren said,

May 1, 2018 @ 9:37 am

When various circumstances in my life had me driving into Boston from the west (on Route 2) and from the north (on Route 93) on a fairly regular basis, I noticed that on both of those routes there is a crest from which one gets a broad view of the Boston skyline, and I have long believed that these points are part of the boundary between what forecasters call "the Boston area" and "the suburbs". (Both are inside Route 128, for what that's worth.) I haven't particularly noticed such a spot coming from the south (which I don't do as often), but I remember many decades ago having a book called "The Boston Basin Bicycle Book", which described various possible bike rides in the immediate Boston area, and realizing that the boundaries referred to above were also part of the defining border of the "Boston Basin".

40. ### Robert Coren said,

May 1, 2018 @ 9:41 am

route 128 (a.k.a., Interstate 95, but it is a point of local pride to never call it that)

I believe that, when (most of) 128 was designated as I-95, Massachusetts got a special dispensation allowing it to continue to use the "128" designation in addition to the Interstate one. Also, as implied above, there is about 20 miles of 128 (Peabody to Gloucester) that is not I-95.

41. ### Robert Coren said,

May 1, 2018 @ 9:43 am

@maidhc: My experience suggests that there are so many micro-climates in the Bay Area that a weather forecast that covered them all would go on for so long that the weather would have changed by the time it was done.

42. ### mg said,

May 1, 2018 @ 9:44 am

And there's that delightful stretch around Westwood near the Rte. 128 Amtrak station (yes, that's the official name), where 128 is simultaneously 95 North and 93 South (or vice versa).

43. ### Topher Cooper said,

May 1, 2018 @ 1:34 pm

@Robert Coren

"When various circumstances in my life had me driving into Boston from the west (on Route 2) … on a fairly regular basis, I noticed that on [this route] there is a crest from which one gets a broad view of the Boston skyline". Yeah, I have a somewhat shifted version of that view from my attic window. When I moved in 30 years ago, I had it from my bedroom window, but our yew and other foliage grew up.

That spot where Boston appears is just below a large rock, which used to be the highest point west of Boston until you get to the Berkshires. When the Mormons moved into the area they wanted to build their church, apparently, on a spot that guaranteed that their steeple was higher than that of any other denominations, so they bought the land on top, but had it sheered down a bit in the process. I haven't heard the claim since then — I regret that this was necessary according to their values.

"I have long believed that these points are part of the boundary between what forecasters call 'the Boston area' and 'the suburbs'." It is a rather nice, poetic choice of boundary, but sorry, that spot is on the border between Belmont and Arlington (which 2 runs along at that point). The Boston area (not to be confused with "The Greater Boston Area" which has an official, demographically derived definition) seems pretty consistently to mean (west of Boston) Boston, Cambridge, Newton and Somerville. Arlington, Belmont, Lexington, Medford, Winchester, Malden etc., fall into "the suburbs" as well as "inside 128". It's not even real close, since it is near Arlington's western border (its eastern boundary, part of the outer edge of "the Boston area", is at the "Route 2 rotary" — ah, touching on linguistics, we do call them "rotaries" 'round here, not "traffic circles" or any of the other alternatives).

44. ### Topher Cooper said,

May 1, 2018 @ 2:01 pm

@mg "And there's that delightful stretch around Westwood near the Rte. 128 Amtrak station (yes, that's the official name)"

Its official abbreviated designation is even "RTE". It is also referred to, though, as the Westwood station — in fact, annoyingly, Google Maps seems to think that is the proper designation and, last time I checked, demands clarification if you ask for the route 128 station.

"where 128 is simultaneously 95 North and 93 South (or vice versa)."

And what's more, at the point coming up from the south where you can see prominent signs announcing their contrary directions, they are all three (95, 93 and 128) running East/West.

45. ### Robert Coren said,

May 2, 2018 @ 10:06 am

Again, I don't go south of Boston as often as north or west, but once a year I make three round trips to Mansfield in three days — this year it was the weekend before last — which includes the southernmost portion of I-93 in the state, and I've never seen any indication that any part of 93 is labeled 128, although it might have been once upon a time. (My vague impression is that what used to be the "southernmost" portion of 128 — beyond the present junction of 93 and 95 — no longer exists as such.)

But as long as we're talking about strange combinations of route directions int he Boston area, I give you Alewife Brook Parkway in Cambridge, which for a brief stretch is Route 2 West, Route 16 East, and Route 3 North — all while going approximately northwest.

46. ### Topher Cooper said,

May 2, 2018 @ 12:25 pm

@Robert Corn
Seems you are right, and I am out of date. I checked on Wikipedia.

Before the RTE train station, in Islington, RT1 North, enters RT 95/RT 128 going South (physically SE) and joins them. The 3-in-1 highway continues until just past the RTE train station, where, in Canton, the roadbed becomes RT 93 North, RT 1 North accompanies it, RT 95 South exits SSW, and 128 terminates. Until the 90s RT 128 South continued with RT 93 and RT 1 North until they junctured with RT 3 North (which is the main route to Cape Cod) at the "Braintree Split" (in Braintree), where RT 1 and RT 128 used to both terminate, and RT 1, still does. Locals still consider this segment to be part of RT 128 (just like New Yorkers continued to call the major street between 5th and 7th Avenues "6th Avenue" even though it was officially "The Avenue of the Americas" — now, after decades it is again 6th Avenue). In fact, that segment is still named "The Yankee Division Highway" which used to be the name of the road which had been designated to be RT 128 — the first limited access circumferential route in the US (before RT 128 became limited access and The YADH in the early 50s, it was just a sequence of local roads designated with that route number). Given the constant local effort that it has taken to keep signs along the highway and at the official entry-exit ramps including reference to it as 128, I had not given any attention to such signs disappearing beyond the roadbed merging with 93.

(I suppose there is some linguistics connection to these issues of naming and designation, along with conflicts between actual, local, popular usage and official, centralized authoritarian prescriptivism).