In the month of Poe's death (in October 1849), and of course the month of Hallowe'en, BBC television showed a program called "Edgar Allan Poe: Love, Death and Women". As I watched it, and listened to the speaker of standard American English that they used for the readings from his poetry (as so many radio and TV programs have done before), reminded me that I have always thought the dialect Poe had in mind when he wrote The Raven couldn't possibly have been one of the rhotic ones. It has to be one of the non-rhotic New England accents. That is, I don't think anybody who pronounces nevermore with an r sound at the end is reading The Raven correctly.
The sound made by ravens and crows is not [kɔɹ], like a typical Midwesterner saying core; it is much more like [kɔ:], reminiscent of a speaker with an Oxford accent saying the word (rhyming it with the educated Southern British pronunciation of law). The repeated nevermore has to sound like that, or the poem makes no sense as far as I can see. The raven's invariant repetition of "Nevermore!" at the end of each stanza has to sound like a raven cawing. Ravens don't have postvocalic r.
[Added later: Having expressed this opinion, let me now add very humbly that (1) I am not an expert on poetry (though I know enough to know that Poe's essay "The Philosophy of Composition" doesn't settle the rhoticity question); (2) I am not an expert on the details of Poe's native dialect (whatever it may have been, rhotic or non-rhotic — though he does seem to have had early experience of at least three places where the local accents are mostly non-rhotic); and (3) I am not an expert on corvids either (I have heard big black carrion-nibbling birds going "caw" without an r on three continents, but who knows, maybe some of the birds I took for ravens were crows or something else). What I do consider myself expert at, though, is expressing an opinion that will instantly get fifty or sixty people tussling with each other to disagree with me, or agree with me, or squabble wildly with each other. Read on and see… Mission. F***ing. Accomplished!]