From Matthew Yglesias:
A few of us at work were talking about why it's adviser and protester but professor and and auditor and after bullshitting around for 10 minutes I thought "maybe I should ask a linguist." Have you ever blogged on this?
I don't think that we have, though you can find well-informed discussions elsewhere, e.g. here or here/here. The executive summary is that -er is (originally) Germanic while -or is (basically) Latin, often via French.
But this doesn't help much with the particular examples you cite, since all four words are from Latin via French. Like most things about English morphology and spelling, the full answer is complicated, and also more geological than logical. But the OED seems to have the whole story — lifted from the depths of the discussion, the key point is that
Many derivatives [formed with -er as an agentive suffix] existed already in Old English, and many more have been added in the later periods of the language. In modern English they may be formed on all vbs., excepting some of those which have [Latin- or French-derived] agent nouns ending in -or, and some others for which this function is served by ns. of different formation (e.g. correspond, correspondent). The distinction between -er and -or as the ending of agent nouns is purely historical and orthographical.
For a (much) longer treatment — you have been warned — press onward.