The Art of Hyping Machine Translation - 上海翻译公司论坛

上海翻译公司论坛 → 翻译业界交流 → 翻译行业动态 → The Art of Hyping Machine Translation

The Art of Hyping Machine Translation
	发起人：eging3　　回复数：1　　浏览数：6663　　最后更新：2022/9/28 19:29:47 by nihaota

发表新帖

　帖子排序：

2019/6/27 10:08:50

[只看该作者] #1

eging3

角　　色：管理员
发帖数：1914
注册时间：2015/7/22

回复编辑删除

The Art of Hyping Machine Translation

Baidu is

China’s top search engine, one of the country’s leading proponents of artificial intelligence, and an excellent study in public relations handiwork.

In mid-October 2018, Baidu’s research team published a

research paper titled “STACL: Simultaneous Translation with Integrated Anticipation and Controllable Latency.” The publication came in the run up to the 2018 edition of the

Conference on Empirical Methods in Natural Language Processing, as thousands of the world’s leading researchers in machine translation and natural language processing descend on Brussels to share their latest findings.

At the time, the Baidu paper seemed like just another incremental development in

neural machine translation (NMT) research. But then Baidu’s PR machinery began churning.

Baidu’s PR agency sent press releases, the research paper itself, and a

GitHub demo page to numerous media outlets for coverage—Slator included. The agency wrote that STACL “can translate just a few seconds into the speaker’s speech and finish just a few seconds after the speaker finishes, very much like a simultaneous interpreter.” Within hours, widespread media coverage ensued:

“Linguists, update your resumes because Baidu thinks it has cracked fast AI translation,” by

The Register
“Baidu’s AI Can Do Simultaneous Translation Between Any Two Languages,” by

IEEE Spectrum
“Baidu’s Chinese-to-English translator finishes your sentence for you,” from

MIT Technology Review
“Baidu Looks to Amaze With A.I. Translation in Real Time,” from

Fortune.com
“Baidu creates the world’s first simultaneous translation system,” from

SiliconAngle
“China’s Baidu challenges Google with A.I. that translates languages in real-time,” by

CNBC
“Baidu to debut simultaneous machine translation in latest challenge to Google,” from

SCMP

Other headlines were more straightforward yet still took the well spun PR message at face value:

“Baidu launches simultaneous language translation AI,” from

VentureBeat
“Baidu develops its own take on real-time translation (Google has some fresh competition),” from

Engadget
“Baidu releases a new AI translation system, STACL, that can do simultaneous interpretation,” by

Packt Hub

Links to the media coverage the paper garnered including logos of the outlets were displayed on the Baidu Research group’s Github demo page.

As these articles and headlines stand uncorrected, they are now beginning to be picked up by other media outlets over time. Case in point,

Axios linked to the

IEEE Spectrum article in a piece on the dominance of English in scientific discourse.

As part of the PR push, Baidu conducted a public demo of STACL at the November 1 annual Baidu World Congress. During the event, two screens at either side of the main display displayed automatic speech recognition output and STACL’s simultaneous translation at the same time.

Baidu STACL Demo: Two screens show automatic speech recognition output on one and STACL simultaneous translation in anotherHowever, the simultaneous interpretation available on the live stream was still provided by human interpreters.

Unpacking STACL

So what is STACL and does the research merit such coverage? Compared to a regular NMT engine that “processes” an entire sentence and then translates it, STACL is “fed” the source sentence word by word, and begins translating the input once it reaches a specified number of words. Baidu’s research team called it a wait-k model, where users can specify how many words STACL should wait for before beginning translation.

STACL translates the partial input as it goes and attempts to predict words that may only appear towards the end of a source sentence. For example, in languages like German where important parts of speech such as verbs and negation typically appear at the end of the sentence, STACL will need to

guess the verb or negation as it translates a German source sentence into, say, English. STACL’s prediction method also uses context from the language data on which it is trained.

The researchers conducted experiments on English-to-German and Chinese-to-English

translation tasks.

The catch? At a wait-5 model (system waits five words before beginning translation), STACL’s output quality is slightly below the current state-of-the-art. Go lower, such as a wait-3 model (system waits three words), and the predicted words can be completely wrong.

Indeed, the research paper points out that the more words the system has to wait for, the better the output becomes, so it is essentially a balancing act between how simultaneous the translation is and how fluent it becomes.

Time to Update Interpreter Resumes?

Baidu’s press release said the research was “inspired by human simultaneous interpreters, who routinely anticipate or predict materials that the speaker is about to cover in a few seconds into the future.”

In the research paper’s introduction, the endgame is made apparent: “Simultaneous translation has the potential to automate simultaneous interpretation.”

So should interpreters heed The Register’s words and update their resumes? Nope. Slator reached out to NMT experts to get their take on STACL:

“Firstly, I think we still need to be careful not to classify every new piece of research as a breakthrough,” said Dr. John Tinsley, Co-Founder and CEO of Iconic Translation Machines. He said simultaneous translation “has been around for a while, and there have been a few implementations in the context of NMT already.”

[indent]“We still need to be careful not to classify every new piece of research as a breakthrough” — John Tinsley, Co-Founder and CEO, Iconic Translation Machines

[/indent]Tinsley noted that exploring this technology particularly for interpretation is “enticing,” adding that “it’s positive to have good research teams working in this area and taking steps forward. That being said, I think that’s what we’re looking at here—steps forward.” He also informed Slator that he and his team will be “digging a little deeper under the hood” of the STACL paper in the future.

A Solution in Search of a Problem

Professor Andy Way, Full Professor at Dublin City University and Deputy Director of the ADAPT Centre for Digital Content Technology, said: “It looks to me like this is a solution in search of a problem.”

“If this is not a tool to support interpreters, but instead is intended to

replace them, then I think you know based on my track record what I would say about that,” he added.

Prof. Way said simultaneous interpretation is one of the hardest tasks in the business, and human interpreters are “great at what they do.” With that in mind, he noted that STACL’s lower BLEU scores raises deployment issues.

[indent]“The situations in which interpreters are deployed are usually places where critical decisions need to be made. Would you trust a machine in such a situation?” — Professor Andy Way, Full Professor at Dublin City University

[/indent]“Not a Scientific Breakthrough”

As for Dr. Jean Senellart, the Systran Global CTO provided Slator with an impromptu, brief version of a review he said he would give if the paper was submitted to a conference. He noted that after conferring with Chinese colleagues, he was told the announcement made a lot of noise and it seemed like it was presented “as being equivalent in importance to the paper of Microsoft [on achieving human parity].”

Dr. Senellart’s first point was that the presentation was misleading because in an actual consecutive interpreting scenario, while the interpreter may be translating after every full sentence, he or she is also part of the communication flow, to the point that the speaker may pause to allow the interpreter to ask questions or clarifications before translating.

“In this workflow, we are indeed doubling the time needed for the communication,” he said.

In the case of MT, Senellart clarified that waiting a certain number of words before translating, as STACL does, is not necessary. “The machine can listen and compute simultaneously, there would not be a need to pause the communication so both would be qualified as an additive latency,” he said.

Senellart said that on a 30 minute speech, an NMT engine that translated after each full sentence and STACL “would only differ by a few seconds.” He added that those few seconds “would not justify a loss of quality of several BLEU points.”

Ordering Food

Slator also reached out to two of the authors of Baidu’s STACL paper: Mingbo Ma and Liang Huang. Huang addressed the above concern, conceding that it is “true only for speech-to-text translation in a conference setting.” However, he offered what he called “a more common scenario: 1-on-1 dialog (e.g., ordering food).”

“Here this full-sentence translation strategy would cause a 2x delay,” he said.

[indent]“While our current technology is still speech-to-text, we are also working on expanding it to simultaneous speech-to-speech translation.” — Liang Huang, Principal Scientist, Baidu Silicon Valley AI Lab

[/indent]In his impromptu review, Senellart continued that “the use case is wrongly presented and wants to drive the reader to some understanding that the machine has reached a new ‘superhuman ability’.”

To this point, Huang simply stated: “We never claimed “superhuman ability”.”

Senellart also addressed a claim in the paper where the authors said STACL’s overhead is “much more desirable” because it was additive instead of multiplicative like the overhead of an NMT engine that translates after every full sentence.

“It is wrong,” he said. “Because even if this statement is true (consecutive vs. simultaneous in the human world), the method they are trying to replace (waiting to the end of the sentence) is falling in the same category than their presented method (both are additive).”

As for the science behind it, Senellart said “there is no new idea” behind STACL: “the solution they propose would be the default solution anyone who did not read the paper would set up.” He clarified that this is not necessarily bad, “but this is not a scientific breakthrough.” Indeed, simultaneous translation has been looked at before, prior to NMT, and there have been

proposals for incorporating it into an NMT engine as early as 2016.

Small BLEU Penalty; Huge Translation Error

Finally, Senellart noticed that there was “almost no qualitative or quantitative evaluation [performed], and this absence most likely covers for big errors.” He pointed out in one of the examples given in the research paper and the GitHub demo page, STACL correctly predicted and translated “to meet.”

“But here what if the final verb is not meet, but “call”, or something totally different?” Senellart said. “The change of the verb will be only a small BLEU penalty but a huge translation error—how does the model try to recover (if it does) when it finally encodes the verb and “realizes” that the full sentence is wrong?”

Huang confirmed that they did only use BLEU, but also said that “we predict (or “anticipate”) all kinds of words, not just verbs, and if there is a major problem in the prediction, it should reflect in BLEU score as well.” He added that they intend to incorporate human evaluations on prediction accuracy.

“We also acknowledge that (a) our current work can’t fix a previous prediction mistake, and (b) it can’t even “detect” such a mistake,” he said. “But we are working on ways to address these problems.”

Perhaps anticipating a bit of pushback from the machine translation community—with Microsoft’s controversial

“human parity” claim a cautionary tale—Baidu did add a closer to their press release stating that “STACL is not intended to replace human interpreters, who will continue to be depended upon for their professional services for many years to come.”

According to them, it was merely meant to make simultaneous translation “more accessible.”

专业翻译公司 http://www.e-ging.com