Exploring AI's Potential for Augmenting Post-Editing in Machine Translation Across Multiple Domains: Challenges and Opportunities

Jeehaan Algaraady ^1* Mohammad Mahyoob ^2,3*

• ¹ Taiz University, Taiz, Yemen
• ² Technical Community College, Taiz, Yemen
• ³ Taibah University, Medina, Al Madinah, Saudi Arabia

Post-editing is a significant process in machine-generated translation (MGT), serving to amend errors in machine-generated translation (MGT) and improving its quality, cohesion, and coherence. The current research aims to comprehensively investigate the capabilities of Large Language Models (LLMs), specifically ChatGPT4o, as a natural language processing tool in translation post-editing services across various domains. Our investigation assesses ChatGPT4o 's potential to boost MGT's productivity, accuracy, consistency, and overall quality. In our experiment, a set of Arabic-translated texts from different domains generated by Google Translate is comprehensively posted by two professional human translators and ChatGPT-4o. Three other human professional post-editors then evaluate the two post-edited versions. A paired t-test is applied for statistical analysis and to compare the quality metrics of human and ChatGPT4o post-edits. The results revealed that human post-editors excelled ChatGPT-4o in most metrics, while ChatGPT-4o outperformed human editors in efficiency with a positive tstatistic of 8.00 and a p-value of 0.015. In fluency, both show no statistical significance, with a tstatistic of -3.5 and a p-value of 0.074, suggesting that both methods perform similarly in ensuring natural text flow. ChatGPT-4o performed competitively in English-to-Arabic translation as it provides a high degree of fluent, accurate, coherent, and efficient editing with consistency in style, tone, and register throughout the content due to its conversational nature. However, ChatGPT-4o struggles with grammatical and syntactic aspects, specific idioms, terminologies, and complex terms in medical and sports texts. The study demonstrates that ChatGPT-4o holds promise for improving translation quality in the post-editing process and shows its effectiveness in assisting human translators in post-editing translation services. Moreover, the tool's effectiveness is further enhanced in many translation domains when using task-specific prompts. However, the appeal for human post-editing remains essential for ensuring high-quality translations.