Deep learning stumbles again
« previous post | next post »
At least I think that's what happened here. Gita Jackson, "Tumblr's New Algorithm Thinks Garfield Is Explicit Content", Kotaku 12/4/2018:
Yesterday, Tumblr announced that it will ban all adult content starting December 17th. As users logged into their accounts, they have seen that some of their posts now have a red banner across them, marking them as flagged for explicit content. The problem is, a lot of these posts are hilariously far from being pornographic.
It’s pretty clear that these flags are being done based on an algorithm, and the algorithm is finding false positives. Here’s a list of things that got flagged: a fully clothed woman, a drawing of a dragon, fan-art of of characters from the anime Haikyu!!, art from the children’s book The Princess Who Saved Herself that the author of said book posted, a drawing of a bowl of fruit with mouths, a video of abstract blurs, Garfield.
https://twitter.com/heyshanmurphy/status/1069703550870142977
Some additional coverage, mostly focusing on perceived problems with the policy rather than its implementation — "Fandom’s Fate Is Not Tied to Tumblr’s: If Tumblr doesn’t learn from history, it will be headed for the same fate as LiveJournal"; "Tumblr’s porn ban is depressing. Here’s why."; "Before Tumblr announced plan to ban adult content, it was a safe space for exploring identity".
Without any evidence, I'm going to guess that Tumblr execs were freaked out at being removed from the App Store over child porn, and accepted the assurances of some overconfident techies whose only knowledge of pattern recognition is that Deep Learning Rulez. So surely pushing a bunch of porn sites through PyTorch would solve the problem, right? Or then again:
I like imagining the Tumblr anti-porn bot was purposefully badly written. Like some rogue chaotic-good developers that wanted to show the higher ups that it's a horrible idea. Training the AI on Garfield comics and fully dressed women lul
— Dani Lewandowski (@webdevdani) December 5, 2018
No doubt they'll manage to lower the false positive rate over time — but that may not be enough to save the platform.
So far the ban only applies to images, not to text — though allegedly Tumblr's own text-only announcement of the new policy was flagged? — so we don't have any funny examples of banning bible passages or nursery rhymes. But given Apple's continued policing of the App Store, such things will no doubt be increasingly common in our future.
There's a serious general problem here — how to detect Bad Things on the internet, and what to do about them? And this episode adds to the evidence that we're very far from having real solutions.
Update — as noted in the comments, the IPA chart is apparently also flagged as pornographic:
Philip Taylor said,
December 6, 2018 @ 9:00 am
Leaving aside the very real problem of false positives, surely a far more important problem is that one man's Bad Things is another's everyday part of life. To some, pornography is bad, to others something normal; to some, any mention of the Tiananmen Square massacre, Independence for Tibet, or the rights of a Muslim minority is So Bad It Must Be Suppressed at All Costs, whilst to others they are things that it essential to discuss. My personal belief is that no social medium which seeks to serve the whole community (as opposed to a social medium set up specifically to address the needs of a particular group) should seek or attempt to pass value judgements on what is Good, or Bad, or Acceptable, at all.
[(myl) But at the other end of scale, we've got criminal phishing, systematic cyberwarfare based on spreading content from spoofed sources to targeted recipients, etc. etc. Also what Joe Tello said below…]
Philip Taylor said,
December 6, 2018 @ 9:02 am
Sorry, please mentally delete "not" in "should not seek" — I forgot that I had used "No social medium" rather than "any social medium". Please bring back (or introduce) the ability to edit one's own content, at least for a couple of days after posting …
[(myl) Your "not" has been removed. Comment editing is a matter for WordPress — if anyone knows how to add it to this site without major php hacking, please let me know.]
Joe Tello said,
December 6, 2018 @ 9:19 am
lol Philip – go ahead de-genderize "one man's Bad Things" while you're editing for accuracy. No indication that the algorithm self-identifies as male.
Philip Taylor said,
December 6, 2018 @ 9:51 am
Mark — I know next to nothing about WordPress, but this seems a possible solution.
Joe — Sorry, for me the aphorism "one man's meat is another man's poison" is so old that it can't (and shouldn't) be expected to conform to today's ideas of Political Correctness.
Joe Tello said,
December 6, 2018 @ 10:52 am
lol Philip – perhaps post-modernize your old aphorism with "one person's pharmakon is another's pharmakon". This post is a kind of Socratic treatment of Deep Learning as the Answer to Everything the way writing was critiquedin the Phaedrus.
Ralph Hickok said,
December 6, 2018 @ 11:05 am
Perhaps the problem is that Garfield is a pussy cat?
Francois Lang said,
December 6, 2018 @ 11:21 am
@Philip Taylor
One man's meat is another man's poison
I know this is old hat, but it might still get a chuckle
One man's Mede is another man's Persian
Kristian said,
December 6, 2018 @ 11:59 am
Isn't Tumblr just deciding to go with the same standards that most mainstream social media/blogging sites have? Most of these seem fairly successful at keeping out pornography.
lysa said,
December 6, 2018 @ 12:45 pm
@Philip – I'm sorry to hear that the ability to use your preferred phrasing for a certain idiom is important enough to be worth erasing hundreds of women and non-binary people. Having respect for others' identities is just basic decency, not political correctness.
Ellen K. said,
December 6, 2018 @ 2:20 pm
lol Philip – go ahead de-genderize "one man's Bad Things" while you're editing for accuracy. No indication that the algorithm self-identifies as male.
Or even that it identifies as human.
Philip Taylor said,
December 6, 2018 @ 2:29 pm
Lysa — to my mind, they are not "erased" at all; they are subsumed. For centuries, it has been the rule that "the male encompasses the female", so in (for example) the apophthegm "Man proposes, God disposes" (the title of one of Turner's more striking paintings), "Man" clearly refers to any human of either sex or none.
lysa said,
December 6, 2018 @ 3:05 pm
@Philip – If you look "through the hat band," so to speak, I think you'll see that that's not quite the case.
Philip Taylor said,
December 6, 2018 @ 3:16 pm
Genuinely puzzled. Lysa — I was not familiar with the idiom "to look through one's hat band", so I asked Google, and as far as I can see it is not attested. Would you be kind enough to explain, please ?
Trogluddite said,
December 6, 2018 @ 3:27 pm
Have Tumblr's coders not heard of "Rule 34"? ;-)
Chandra said,
December 6, 2018 @ 3:49 pm
@Philip Taylor – Legions of non-male people have for decades been asserting that we do not appreciate being "subsumed" under male nomenclature as it does, in fact, contribute to our erasure, whether or not you, as a man, are able to understand that experience. You are of course entirely free to continue to prioritize archaism and tradition over respecting the wishes of half or more of the population, but please do not try to tell us what we do or do not experience.
lysa said,
December 6, 2018 @ 3:51 pm
@Philip – Where I come from, "looking through the hatband" has the connotation of being "on the fence," so to speak.
Paul Garrett said,
December 6, 2018 @ 6:04 pm
(As @lysa says, it's just about basic decency, not "political correctness", the latter … as we know all too well … a phrase most often used dismissively by people _not_ on the receiving end of marginalization…)
Philip Taylor said,
December 6, 2018 @ 6:15 pm
Chandra — I would never dream of trying to tell you (or anyone) what you do or do not experience: how could I possibly know ? I simply put my position, with which I accept that not necessarily everyone will agree. But we are discussing language and my argument is that it is a mistake to allow (or worse, require) long-established idioms to be sacrificed on the altar of Newspeak or Political Correctness (e.g., "rule of thumb", "manhole cover" or "accident blackspot") [1].
——–
[1] https://www.thetimes.co.uk/article/rule-of-thumb-blacklisted-as-pcs-go-pc-9x8rmz0v7fg
chris said,
December 6, 2018 @ 7:34 pm
we don't have any funny examples of banning bible passages or nursery rhymes
Or serious examples, either. Even granting for the sake of argument the premise that sexual content is somehow *more* banworthy than (e.g.) God ordering people to commit genocide and them obeying, ISTR the Bible *also* has explicit sexual content in it.
Nursery rhyme examples might be limited to false positives, IDK. Unless there are nursery rhymes about grass-mud horses (and if there aren't yet, I'm sure someone can invent some).
@Ralph Hickok: Well, you have to admit he IS fully nude in that picture. I predict it will be quite tricky to draw a line on which beings are allowed to appear nude, when fictional characters are involved. How much consideration should be given to deliberate omission of crucial details, as (not) seen in this picture?
Defining pornography is famously beyond the capacity of even some well-educated humans, who can merely claim "I know it when I see it". What hope does the present state of artificial intelligence have?
If Tumblr doesn't realize they're *at least* going to need an appeal process for false positives, then they are egregious fools who haven't made even a cursory study of the history of attempts to filter user-generated content. Not even mentioning the possibility that the Internet will "route around it" by a substantial fraction of their userbase deserting them for a less prudish competitor.
J.W. Brewer said,
December 6, 2018 @ 7:51 pm
Through the magic of the google books corpus, you can find the phrase "one person's meat is another person's poison" in a piece by Edwin L. Sabin titled "Stomachitis" published in 1911 in Lippincott's Monthly Magazine. Presumably neither Mr. Sabin nor his editors were responding to the sort of politicized controversy about gendered language that has arisen in more recent decades, and if he'd chosen the more conventional phrasing it would presumably have been printed in that form.
That said, the google books n-gram viewer shows "one person's meat" to be much rarer (like 95%+ less common) than "one man's meat" well into the 21st century.
If anyone is interested in returning to the subject of the original post, you can google up a story from 2009 reporting that "Garfield is obscene" in the judgment of whatever then-state-of-the-art internet filtering software was being used by the Communist authorities in mainland China. (That's the preview but I'm having trouble clicking through to the actual website.) So maybe Tumblr has just reached the level of competence the PRC was at a decade ago?
Y said,
December 6, 2018 @ 8:29 pm
Headline in the Guardian today: "Thanks for the vegan idioms, Peta, but there are bigger fish to fry."
Rebecca said,
December 6, 2018 @ 10:49 pm
@chris: cartoon animal nudity will indeed be tricky to process, given the way some clothed cartoon animals dress. Mickey Mouse seems pretty conventional, but they better not train their algorithms on Pooh or Donald Duck
Brett said,
December 6, 2018 @ 11:26 pm
@Rebecca: There is an old joke about the fact that Mickey Mouse wears pants but no shirt, while Donald Duck wears a shirt but no pants.
rcalmy said,
December 6, 2018 @ 11:49 pm
Bringing it closer to home, Gretchen McCulloch at All Things Linguistic reblogged the image of a vowel chart that got flagged. Apparently the algorithm sees something salacious in IPA symbols.
[(myl) From here:
]
Michèle Sharik Pituley said,
December 7, 2018 @ 10:31 am
@Chandra & lysa — thank you! I, too, am so tired of being told that I’m “subsumed” or “encompassed” or “included” in male-gendered language. It’s not rocket surgery to make language inclusive and to show respect to “the other half” of humanity; it’s just basic decency.
Also so tired of the term “political correctness”. Excluding groups of people because you don’t want to change your habits is rude. Yes, long-standing habits of language are difficult to change, but that doesn’t mean one shouldn’t try. If one is quoting something of historical importance (such as Shakespeare or Chaucer) then fine, but make it clear it’s a quote (so the meat/poison thing might be ok if it is clearly a quote) and not just general obtuseness or cluelessness.
Language changes. Try to keep up. Thank you.
/rant
Ps please bring back the option to subscribe to comments. The disappearance of this option has been mentioned several times now (by me & by others), but never addressed or even acknowledged. Is it a WP thing? Or a deliberate change?
Philip Taylor said,
December 7, 2018 @ 2:18 pm
Michèle, Lysa, Chandra, et al. I am sorry that you feel excluded when a writer such as myself uses idioms such as "One man's meat is another man's poison", or "Man proposes, God disposes". Exclusion is certainly not intended (quite the reverse, in fact), but I accept that it might be inferred. Nonetheless, I shall continue to use such phrases, since whilst language itself might evolve, idiolects, at least for those of a certain age, tend to be rather more conservative. But I would ask you two questions, if I may ? (q1) What is your position on "mankind", when referring to the entire human race ? For me, the term is completely unexceptionable, and for this I have the support of the Oxford English Dictionary, which I regard as authoritative in such matters :
and (2) what is your position on "waitperson" ? I have seen this used in writing on more than one occasion (but never heard anyone use it in speech), and it always strikes me as one of the most ridiculous and facile additions to our language that there has ever been. We have waiters (who are male) and waitresses (who are female), and ignoring for the purposes of my question the small percentage (circa 0,4%) of trans-gender individuals who might perform this rôle, are there really perceived problems with identifying the sex of the rôle-holder as we always have ? Already the term "actress" is under attack (especially in the Guardian) — how many more such useful words/distinctions are we to lose, I am forced to wonder.
P.S. Does anyone really say "waitperson" in real life ? Serious question.
Michèle Sharik Pituley said,
December 7, 2018 @ 4:43 pm
I’m almost 50 years old and have learned to use gender-neutral language, so age is no excuse. Also: intent is not magic.
To answer your questions:
I prefer “humanity” to “mankind”. “Humankind” seems a bit awkward, though I concede it might be useful in formal situations where “humanity” might be considered too casual.
I waited tables for years to put myself through university. At the restaurant in which I worked, we were all “servers” or sometimes “the wait staff” but never “wait people” or “a waitperson”, which just sounds awkward to me. (I waited tables from 1987-1996, so a simple gender-neutral option existed way back then.)
I think it’s important to ask: Why is it so important to distinguish people by sex or gender? Why can’t we all just be people? Why do you find this distinction “useful”? (I mean in a general sense. Of course a distinction can be useful in a specific sense, but now that I think about it, maybe only if it’s the only or most obvious distinction — and then can’t you just say “third one from the left”? This bears more thought, which unfortunately I don’t have time for right now, as I’m leaving for the evening.)
FWIW, I feel the same way about “Congressperson”. I prefer “Representative” or “Congressional Representative” if I have to be more specific for some reason. That’s what they’re supposed to *do*, after all.
Nipo said,
December 7, 2018 @ 5:24 pm
@Michèle Sharik Pituley
I have nothing to do with the comments you replied to, but I want to reply your question "Why is it so important to distinguish people by sex or gender?"
Not so surprisingly, I don't. There is no point. However, in other languages there are other approaches than de-genderifying the language.
E.g. in Swedish there are still sjuksköterskor (sick-carer-ess-es), the female nurses, as well as sjukskötare (sick-carer). There are still "fröken" (miss) in the day care. However, the day care "miss" can be named Peter, and the difference between the different nurses seems to be in where they work at – every nurse in mental health are called "sick-carers" and every other nurse is a "sick-carer-ess".
An ex-chairman of Finnish parliament was a woman – she chose to be addressed as "Mrs. Chairman", because she found the alternatives awkward. Also, a national worry in Finland is that it is hard to find male law-men (laki/mies, lawyer) to hire anymore.
Of course there are no "proper" genders in either Swedish or Finnish, so our troubles are not exactly the same. Finnish has no genders to speak of, and Swedish genders are neutrum and utrum. Speakers of neither language seem to actually differentiate between the people by their genitals, even if the language still has the words to do so. Even though there is no point of denoting whether a server is a man or woman or machine, one can also choose what one hears.
But could someone explain to a non-native English learner: why is the word "server" gender-neutral but "waiter" is not? They seem to be built the very same way.
Philip Taylor said,
December 7, 2018 @ 6:30 pm
Michèle asked "why is it so important to distinguish people by sex or gender ?". In some cases it is important, in other cases far less so. For a case where it is important, consider the stage or film — you could not employ a actress if the role required a man, nor could you employ a actor if the role required a woman. In the case of waiter/waitress it is less important (although my wife, who runs a hotel, would undoubtedly advertise for a waiter or a waitress but not for "a person to wait at table" or any similar circumlocution), but it is traditional, as are, for example 'bus conductor/conductress (the latter frequently referred to as a "clippy" — there was no analogous term for a conductor), governor/governess (again, an important distinction), tailor/tailoress (unimportant) and so on.
As to whether one can (or should) deliberately adapt one's speech or writing to conform to current norms, this must surely be a personal choice, and my choice is informed by discussions with women of my own age group (70+), very few of whom feel excluded by expressions such as "one man's …" and most of whom find "waitperson" as ridiculous as I do. I have little doubt that as one goes down the age spectrum, the ratio of those who feel excluded to those who do not will rise, and I think that such people are perfectly entitled to expect members of their own age group to conform to the norms to which that age group subscribes, but by the same token I do not think that they are entitled to expect their elders to adapt their speech just to conform to the expectations of, or to cater to the sensibilities of, a younger generation.
chris said,
December 7, 2018 @ 10:49 pm
But could someone explain to a non-native English learner: why is the word "server" gender-neutral but "waiter" is not? They seem to be built the very same way.
An interesting point. I think the only plausible answer is historical: because there is a female-specific form "waitress" in recent/current usage, "waiter" became effectively male-specific.
If "waitress" had never existed, then we probably wouldn't see "waiter" as any more gendered than "driver" or "painter" or "singer".
Chandra said,
December 7, 2018 @ 10:52 pm
@Philip Taylor –
1) I simply put my position, with which I accept that not necessarily everyone will agree. My point is that you can’t have a valid “position” on other people’s experiences. It’s not an opinion or a feeling, it’s a thing that actually happens, and you can no more “disagree” with it than I can “disagree” with what you ate for breakfast this morning.
2) I say “humankind” or “humanity”, and “server”. I’m not sure why you bringing up dictionaries, as the question at hand is about social context, not definitions.
Also most transgender people prefer to be referred to by binary gender terms; you’re thinking of nonbinary people. What do you deem the critical percentage point for a group of people to be worthy of basic consideration?
3) As I said, you’re free to continue doing whatever you want. I’m also free to continue holding opinions about what your words and actions say about you. Frankly I couldn’t care less about outdated aphorisms; it’s the way you’ve chosen to respond to people’s requests for more inclusive language, on a public forum whose readership is comprised of more than just 70-year-olds, that is most telling.
Both my parents are in their 70s and have no difficulty with this concept. Also it’s interesting to note that you had no trouble whatsoever substituting “Bad Things”/”everyday part of life” for “meat”/”poison” in said aphorism – almost as if you’re perfectly capable of being flexible with language after all, when it suits you.
Philip Taylor said,
December 8, 2018 @ 4:10 am
Chandra —
"My point is that you can’t have a valid “position” on other people’s experiences" — I agree. My "position" was simply concerned with which words and phrases I preferred to use. As I explained previously, I have never sought to comment on other's experiences.
"What do you deem the critical percentage point for a group of people to be worthy of basic consideration". Infinitely small. I sought only to exclude them from the domain of my question, not from consideration.
"it’s the way you’ve chosen to respond to people’s requests for more inclusive language" — My response is simply that I hear their requests, but that I will continue decide for myself which words or phrases I will use, and that my decisions will be based primarily on my own feelings but influenced by discussions with people (in this context, primarily but not solely women) of my own social circle and age group.
"you’re perfectly capable of being flexible with language when it suits you". Agreed — when I choose to do so, not when others seek to dictate it.
Chandra said,
December 9, 2018 @ 12:40 am
Your position was that we are not erased. That is an incorrect statement about our experiences.
No one is dictating anything to you. We have expressed how this kind of language affects us – as we are well within our rights to do when someone decides to pull “political correctness” into a discussion that had nothing at all to do with that in the first place – and you have made it abundantly clear that you don’t care enough to change your habits. I am now stating for the third time that you are free to do whatever you want, which is exactly the opposite of dictating what you should do. We are also free to express why we disapprove. If you dislike our disapproval, that is your own problem to deal with.
Philip Taylor said,
December 9, 2018 @ 2:20 am
Chandra, I clearly wrote "To my mind, they [i.e., you] are not 'erased' at all; they [you] are subsumed". Thus this is not a statement about your experiences, about which I know absolutely nothing, but about my perception of the phrase and what its use entails. Can you see the difference ? And of course I don't dislike your disapproval, I just choose to ignore it when deciding what words or phrases I will use..
speedwell said,
December 9, 2018 @ 4:15 am
About the idea that changing outdated language is "sacrificing it on the altar of political correctness" (leaving out the question of whether "political correctness" is actually a meaningful term instead of simply a sneer at people with manners): It's only "sacrificial" in the same sense that, when you change your underwear, you "sacrifice" the old knickers to the washing machine.
ajay said,
December 10, 2018 @ 12:31 pm
If "waitress" had never existed, then we probably wouldn't see "waiter" as any more gendered than "driver" or "painter" or "singer".
Or indeed "editor" and "aviator", both of which used to have female forms which are now completely unused.
Chandra said,
December 10, 2018 @ 2:02 pm
@Philip Taylor –
To my mind you didn’t have eggs for breakfast this morning. Can you see that your perception is utterly irrelevant to the situation?
Perhaps you mean to say that you did not have the intention of erasing anyone. That may be so, but your intention doesn’t change our reality, in which plenty of research shows that male-coded terms used as a supposed generic in fact cause people to assume the referents are men (e.g. Moulton, Robinson & Elias (1978); Miller & James (2009)), and reinforce biased views of women (e.g. Briere & Lanktree (1983)).
We are demonstrably erased, and this kind of language is demonstrably harmful to us. It is incredibly frustrating and exhausting to have people with no direct experience of these effects constantly talk over us – implying that we are merely being frivolously difficult (i.e. “politically correct”) when we ask for consideration – instead of actually listening and trying to understand.
And on that note, I have no further energy for this debate and will not be returning to this thread to read or respond.
Peter said,
December 10, 2018 @ 2:59 pm
chris wrote:
Given the back-stories that are supposed to be behind many nursery rhymes, I expected that it wouldn't be hard to find something of a salacious nature, and indeed the original text of "Rub-a-dub-dub" does seem to refer to gentlemen observing young ladies in their bath.
Of course, one could also cite the lyric of Tom Lehrer's song "Smut":
Philip Anderson said,
December 11, 2018 @ 6:09 pm
Philip Taylor:
I agree that we would usually refer to a woman taking a female role as an actress, in that context, but she would also be an actor by profession or vocation so you would be employing an actor. But other artistic professions have long lost their gendered forms, e.g. authoress and poetess.
Some of your "traditional" forms were dated when I was young (perhaps you were a young fogey?); I can barely remember bus conductors, but a train might have a (male or female) conductor, never a 'conductress'. Is "clippy" still current in the North of England? I have only met it in a short story.
There is an important distinction between governors and governesses, but it is not primarily a gender issue; a governess was, and perhaps still is in some frozen-in-time families, a chaperone-cum-teacher, whereas a male or female governor governs something or somewhere, e.g. a school or the Bank of England.
Why should your choice of terminology only be informed by discussions with women of your own age group? One of the reasons comments are allowed here is so that we can listen to people with different experiences and points of view, including on our own language.
Philip Anderson said,
December 11, 2018 @ 6:12 pm
Peter:
Remember that most origin stories for nursery rhymes are modern inventions.
Philip Taylor said,
December 12, 2018 @ 8:57 am
I think that this digression has gone on far too long (and I am sure others will agree with me), so I apologise in advance for what follows. Let me respond to Peter's questions, and then I promise to remain silent on this topic.
'Bus conductor/conductress/clippy — my father was a 'bus driver, and as a child I would frequently travel on his 'bus; he had a favourite clippy with whom he preferred to work, and she was known as "Jack's clippy". This was in South-East London.
As to "why [my] choice of terminology should be informed only by discussions with women of [my] own age group ?", there is no reason at all why it should be; it simply is. Though as I explained earlier, not solely informed, just mainly. I think that we tend to choose our friends on the basis of shared values, and most of my friends have little interest in adopting modern usages such as "Ms", "Chair" (for Chairman) or "actor" for an actress. I fully accept that these views are by no means fashionable, and I believe that those who hold different views are fully entitled to state them, which is why I earlier said to Chandra "Of course I don't dislike your disapproval, I just choose to ignore it when deciding what words or phrases I will use …". Or to mis-quote Voltaire "I disagree with virtually everything that you say, but I will defend to the death your right to say it".