Language Log

« previous post | next post »

This morning, let's take a break from the analysis of headlines, and look at some phonetics research. Recently, Jiahong Yuan and I have begun working with a woman who aims to revise and improve tests that are used to certify court reporters. (For some background about the techniques and devices that the court reporters must learn to use, see here or here.)  The (preliminary) results of our (pilot) experiment may be interesting to some of you; I think that they also point to a broader opportunity for linguistic research of other kinds.

Among the questions that our partner needs to answer is what speaking rates, in words or syllables per minute, a court reporter is likely to encounter. Obviously, there's a great deal of variation, depending on the nature of the situation and the characteristics of the speaker. She doesn't want to make the tests too easy, by using material that's unusually slow or unusually simple, or by allowing too many errors at normal rates. Nor does she want to make the tests too hard, by using material that's atypically fast and complex, or by excessive penalties for errors at abnormally fast speeds.

In short, she'd like to be able to make predictions, based on the test results, about the error rates that testees will achieve across the range of situations they are likely to encounter in doing their job. In order to do that, she needs to be able to characterize what that range of situations really is. And an important dimension of characterization is speaking rate.

Now, overall speaking rate is easy to determine. You just count words in the transcript, measure time in the recording, and divide: result, words per minute. For syllables, you just look all the words up in a pronouncing dictionary, and add up the syllables, and divide by the same measure of elapsed time. (There are plenty of conceptual issues here about what a word is, exactly, and how many syllables to attribute to certain classes of sounds, and how to treat reduced pronunciations in rapid speech, and so on. But these are all essentially irrelevant to the topic under discussion, because the same metrics are used on the test materials as in the norms.)

However, overall speaking rate is not the right measure. A deposition or hearing will have some periods of silence, and there will be other times when a speaker is speaking slowly, choosing words with great care. Those periods, where there are relatively few words per minute, will then be balanced by periods when the rate is faster.

For norming the tests, it's important to know what the distribution of local rates actually is. A would-be court reporter may be able to keep up easily in transcribing speech at the overall rate or below, but may be completely lost when things get just a little faster — and it might be that (say) 40% of the material would then be beyond their capabilities.

In order to get some information about what the distribution of local speaking rates is really like, we plan to process a large and representative sample of real-life examples. To establish and validate the procedures, we began with an initial sample of 40-odd recordings with transcripts. The recordings were .wav files, made under normal conditions in the course of various depositions and court proceedings, and the transcripts were MS Word files, in the standard format for court documents of the relevant kind.

After some preprocessing, we ran these through a system that aligns the words to the audio, using techniques from speech-recognition technology. (See here, here, and here for some additional discussion of this same set of programs.) Spot checks indicate that the alignment was very accurate, as we've come to expect after testing this system on many different kinds of material.

The results of the word-level alignment look something like this (the two numbers on each line are the starting and ending time of the corresponding word, in seconds counting from the start of the recording):

404.443310658 404.553061224 "IT'S"
404.553061224 404.722675737 "NOT"
404.722675737 404.872335601 "YOUR"
404.872335601 405.251473923 "JOB"
405.251473923 405.371201814 "TO"
405.371201814 405.560770975 "DIG"
405.560770975 405.740362812 "INTO"
405.740362812 405.929931973 "ALL"
405.929931973 406.069614512 "THAT"
406.069614512 406.448752834 "STUFF"
406.448752834 406.57845805 "IS"
406.57845805 406.638321995 "IT"

And internally, the aligner has also produced an alignment of the phonetic segments involved (as per the available dictionary pronunciations, anyhow), like this:

404.443310658 404.47324263 "AH0"
404.47324263 404.503174603 "T"
404.503174603 404.553061224 "S"
404.553061224 404.582993197 "N"
404.582993197 404.692743764 "AA1"
404.692743764 404.722675737 "T"
404.722675737 404.802494331 "Y"
404.802494331 404.842403628 "UH1"
404.842403628 404.872335601 "R"
404.872335601 405.031972789 "JH"
405.031972789 405.22154195 "AA1"
405.22154195 405.251473923 "B"

We then wrote a script that scans these files and produces local speaking-rate estimates, over a moving 30-second window that scans from the start of the transcript to the end. Here's what the resulting time-function looks like, for one deposition selected at random:

a>

(Click on the image for larger version. If you have a small screen, you might want to right-click and view the image in separate browser window.)

Here's another example of the same sort:

a>

For the second example, the average rate was 164 wpm, and the median 165. The maximum rate (in a 30-second window anywhere in this particular deposition) was 239; the 90th percentile was 200; the 95th percentile was 210.

In terms of syllables rather than words, the mean rate was 226 per minute, the median 232, and the maximum 322. (The average ratio of syllables to words was 1.38, with a minimum (for any 30-second window) of 1.05 and a maximum of 1.82.

For the (small) sample we've looked at so far — 41 transcribed passages amounting to around 10 hours in total — the maximum over any 30-second window was 284 wpm. About 1.7% of the 30-second segments were faster than 225 wpm, and only around 0.25% were faster than 250 wpm.

According to the Wikipedia article on Court Reporting

The minimum speed needed to become certified by the NCRA is 225 words per minute. The NVRA requires a minimum speed of 250 words per minute to qualify for certification.

Skilled individuals can handle speeds well over 300 wpm. My understanding is that tested rates are adjusted downwards for errors, as in the case of typing tests.  The purpose of the work described here is to provide a better quantitive basis for predicting what such numbers mean in terms of coverage, and for deciding what the standards for certification should be.

---

There are many kinds of science. Some research is hypothesis-driven, where the goal is to confirm or refute a theory, or to determine which of two alternative theories is better. In other cases, the goal of the research is just to see what's out there, so to speak. Such mapping expeditions can be motivated by abstract curiosity, by relevance to theoretical questions, or by practical applications.

This particular piece of research is obviously a mapping expedition, motivated by a particular practical question, which was posed originally in a piece of email from a stranger. What I've described in this post is just the beginning of the project, not the end; but I'm now confident that the techniques we've developed for other applications will work on this material as well, and will enable us to give a useful answer to the practical problem at hand.

In this case, the practical application is not of great interest outside the community of court reporters. But there's a broader context that's worth thinking about.  There are millions of hours of transcribed audio out there: court proceedings, oral histories,  broadcast interviews, political speeches, audiobooks, and so on.  The techniques used in this case, with some modifications and adaptations, can make this stuff available as raw material for research in all of the scientific disciplines that deal with spoken language.

[Update 3/24/2009: Some interesting commentary here, including this:

I'm currently in a trial and during cross of the defendant today, the WPM meter never went below 250. It's hot, it's heavy. Can I get it? Absolutely. Do I stop the proceedings if I can't get it? Absolutely. But I try to hang on as best as I can, knowing they will eventually stop to look at an exhibit or take a breath at some point.

]

March 21, 2009 @ 10:21 am · Filed by Mark Liberman under Phonetics and phonology

Permalink