A Model for Designing Decision-based Translation Tests

Translation tests are generally based on the penalty system and deduction of marks for errors. Marks are usually deducted from a total of 100 (or whichever marking scales are adopted). The deducted marks are added up and subtracted from the total. In many cases, by the time the marker has finished the deductions, there is nothing left to be deducted (with negative marking sometimes used).

This penalty model is source text-focused and examiner-centred. It is largely concerned with the microstructure of text – translation at word and sentence levels, and with how word-pieces fit into their corresponding slots (as in a jigsaw puzzle).

This model raises the following fundamental question:

Does a translation, no matter how remote it may be from the deemed-correct translation, really have a zero value?

If the answer is yes, it raises serious doubts about the test admission criteria. If no, a more positive approach is needed, assuming that the candidates have already qualified for the examination and have already attained a certain standard that will qualify them for accreditation on passing the test.

Competence and performance assessment

What is the purpose of an accreditation translation test? First and foremost, an accreditation translation test seeks to establish whether a candidate is qualified to work as a professional translator. It validates this largely by evaluating a snapshot of the translation product of a timed test — in other words, the outcome of the translation process. The time factor is supposed to test the candidate's competence or adequate ability and knowledge to carry out translation work within set time limits. But what it really does is establish the candidate's level of performance within given parameters for that instance only and does not go beyond the translation snapshot (that is the final version of text on paper submitted for marking).

While a shift from product-oriented to process-oriented models has been recently observed in translator training (Gile, 1993), translation tests, especially those run by accreditation bodies, remain product-oriented. Generally speaking, any form of translator performance assessment should have the following basic characteristics: uniformity, validity and reliability.

Uniformity

Uniformity refers to a standardized procedure for designing, managing and marking tests that ensures the test designer and test marker follow the same norms and standards and have the same expectations about the test. Uniformity also refers to the consistency of interpretation by all examiners of such a procedure.

Validity

Validity refers to how well the test fulfils its function, the degree to which the translation test actually measures what it is supposed to measure and the means and techniques by which such measurement is achieved. As Kline (1991) confirms, a test is valid if it measures what it claims to measure. What exactly does a translation test measure?

Reliability

Reliability refers to how objective and consistently dependable a test is. Kline (1991) identifies two aspects of test reliability. One refers to reliability over time — the consistency of scores obtained by the same person when re-examined with the same translation test on different occasions and the correlation between scores by different examiners of the same test taken by the same candidate. The other refers to the internal consistency of the test (45). Accordingly, a translation test should enable the candidate to produce more or less the same results under the same conditions. In other words, it should bring out the best in him or her. It should also enable the marker to replicate the resulting score consistently. Both aspects are extremely important in translation tests.

Without appropriate criteria against which a translation test can be validated, it cannot be a reliable tool for measuring the candidates competence.

Designer-candidate-marker relationship

One of the major problems, if not the main problem, in assessing translation candidates is the mismatch between the test designer’s assumptions about the test, the candidate’s analysis of the test requirements, strategies and intentions, and the test marker’s expectations of the test.

The translation test designer’s assumptions or design criteria are not apparent to the translation candidate simply because they are not spelled out in the test paper. The candidate’s intentions are not apparent to the test marker, and the marker’s expectations and evaluation criteria are not defined. To illustrate, the following scenario is always a possibility.

1. The test designer chooses a technical medical text that is aimed at medical professionals assuming that the translated version will be aimed at the same type of reader.
   

2. For whatever reason, the candidate translates the text for a non-technical reader, and
   

3. The marker assesses the translation with another reader in mind.
   

In this case, there is a clear mismatch between the criteria all three players apply to each of their tasks. No doubt, this results in discrepancies and incorrect assessment and does not reflect real-life situations.

The translation process

Translation is a dichotomous, cumulative and iterative process that musters a host of cognitive activities relating to language, rhetoric, linguistics, communication, and culture. It is dichotomous because it deals with two languages that have different linguistic and cultural norms and patterns and different communication codes. It is cumulative because knowledge gained in translating one piece of discourse is applied to translating another piece of discourse. And it is iterative because it undergoes a repetition of a sequence of events yielding results successively closer to the finished text.

That is why finding out whether a translation candidate has selected the correct equivalent of a word should not be the sole focus of a translation test. The focus should rather be on how the candidate has gone about producing the translation. Dwelling on isolated errors and dissecting these errors into spelling, grammatical and syntactical within the same error is a gross error in itself. Many examiners are bogged down in this area and completely lose sight of the purpose of the test.

Process, procedure and methodology

There is research evidence to indicate that most translators and translation theorists fail to distinguish between the processes, procedures and methodologies that make up the translation activity. The confusion seems to stem from the definition of the translation process rather than from the relationship between object and conclusions. While translation literature contains many references to the translation process, that process refers to the process of producing translations outside the internal cognitive world of the translator. In fact, the translation process is two processes in one that run in parallel: internal process and external process. The internal process relates to the translator’s cognitive processes while the external process relates to the activities that yield a finished translation product ( that is translation, editing, revision, and so on). The lack of a clear distinction between the two processes seems to be responsible for the confusion.

Translation as a decision-making process under constraints

If we look at translation as a decision making process under constraints, we might be tempted to say choosing a word or deciding to use a certain spelling, perhaps erroneously but consistently throughout the translation, is probably far better than choosing the correct word here and using another somewhere else so long as the communicative value is not compromised.

A translation should be seen as an integral, holistic product. Shifting the emphasis from one component to another within the overall entity of the product or focusing on lexical units within the translation yields lopsided results and gives a false sense of quality. A translation test should be analyzed in terms of strategies rather than lexical units.

A collaborative exercise

Translation is a collaborative exercise. In real-life situations, translators often work in teams. They consult and collaborate with each other to produce the final translation. They also consult authorities and specialists, or are at least supposed to in order to solve problems, clarify ambiguities, and even check the accuracy of the information content.

Therefore, translation tests should be concerned with more than just linguistic competence and meaning transfer. Translation tests should examine the protocols and processes employed by the translation candidate. That is why it is important to analyze the draft copy of the test as wells as the finished product to gain insight into the process.

There is no right or wrong translation

Barring mistranslations, errors of meaning, syntax, lexis and so on, no translation strategy or approach should be deemed better than another except in terms of how effective it is in meeting the requirements of the translation product. In this sense, translation quality is relative to purpose. Therefore, irrespective of approach — be it literal, communicative, semantic, reader-centred, text-oriented and so forth — the main criterion in assessing the quality of a translation and in this instance a translation test is to ascertain whether the translator has succeeded in applying the chosen approach to the translation product to meet the information needs and requirements.

How translation is taught

Before we test would-be translators, we need to examine how we teach translation in the classroom and what approaches we take to train translation students to produce translation work in the real work environment.

A typical translation course usually consists of the following components:

• translation theory
  

• contrastive analysis
  

• discourse analysis
  

• cultural studies
  

• language skills
  

• translation workshops
  

More often than not, these components are disjointed, unrelated and isolated as if translator training is a matter of mere exposure to various disciplines and specialities —a bit of this and a bit of that!

Translation theory is important. But theory alone is insufficient unless it has obvious practical application. Many translation courses do not show the link between theory and practice and discussion of theory is often esoteric and in the abstract. Translation workshops are also useful drills to develop the student's skills of analysis and problem solving. But workshops are limited if they do not tackle the basic issues of who, why and how. Traditionally, translation workshops consist of translating a piece of text in fragments and sentence by sentence. Sometimes the students take turns translating individual sentences either offhand or after a brief review and sometimes a short text is translated then the translation is discussed. The treatment of text is often restricted to basic properties such as grammar, syntax and accuracy.

The translation process in its practical sense is rarely discussed on such a course. Translation is presented as a single activity. The student translates, translates and translates. No explanation of the various phases and stages is offered, except maybe casually, and no attempt is made to provide a methodical approach to the iterative process of translation. Furthermore, there is no distinction between the internal (cognitive) processes, external (mechanical) processes, and procedures of translation.

A proposed translation syllabus

A translation syllabus should prepare the student in the following areas:

• communication and writing strategies
  

• translation strategies
  

• research and analysis
  

• terminology
  

• information analysis
  

• audience analysis
  

• translation management
  

Communication and writing strategies should focus on the communication process and rhetorical techniques within a system of meaning and should explain how these affect translation. They should also focus on thought patterns and how these affect communication as a whole and translation as a communication transaction.

Translation strategies should focus on the various approaches to translation and on the advantages and disadvantages of each approach. They should also deal with code switching and meaning making and remaking and with factors affecting loss of information and loss of meaning in switching codes.

Research and analysis should focus on the various ways of research and on text analysis. Text analysis should deal with the various types of text and with the various types of paragraph. It should show the relationship between the types of text/paragraph and their logical underlay. In this connection, text analysis should draw upon logic, communication, contrastive analysis and rhetoric.

Terminology should focus on the methodologies for formulating terms. It should develop an understanding and an awareness of terminology problems and techniques. Terminology training should also deal with how concepts are expressed in a concept-term relationship and how the system of concepts varies among languages and cultures.

This component, which is often overlooked by translation course designers, is important because terminology accounts for 50 to 60 percent of translation errors and because the translators are in the forefront of those who are responsible for information transfer and terminology formulation and who frequently have to deal with terminological problems.

Information analysis should focus on techniques for analyzing the content of the text, while audience analysis should focus on how to profile the translation user.

Finally, translation management should focus on how to organize research, use dictionaries and references, validate information, estimate time and size up text.

A translation course should look at translation as a re-engineering process in totality. It should cultivate an understanding of the purpose of translation in various settings and environments.

A decision-based translation process

The following flowchart shows the initial decisions in the translation process, from the translator’s perspective.

A model for designing decision-based translation tests

Within this framework, a translation test should:

• simulate a real-life situation
  

• include a translation user profile
  

• include a checklist of translation strategies indicating the choice a candidate makes (semantic, communicative and so on)
  

• insist on a draft copy to enable the examiner to analyze the approach taken by the candidate
  

• allow editing and other corrective conventions in the finished copy to enable the examiner to analyze the conscious decisions made by the candidate
  

A translation test should also answer the following questions:

• Who is the audience of the source language?
  

• Who is the audience of the target language?
  

• What are the source language audience's expectations?
  

• What are the target language audience's expectations?
  

• What are the source language audience's information needs?
  

• What are the target language audience's information needs?
  

• How is the source language audience expected to use the text?
  

• How is the target language audience expected to use the translation?
  

• What type of translation is appropriate in this case?
  

In marking tests, let us reverse the penalty system and assume that the translation candidate develops the translation from scratch with a set of options and within constraints. Instead of deducting marks, let us credit the candidate "for getting it right" and then add the marks to produce the final result. In designing a translation test, the passage can be selected to contain a number of sentences (or logical translation units). Each unit is given a number of credits. The overall passage is given a total number of credit marks, with extra credit marks reserved for overall performance. The translation test should be assessed in terms of comprehension and production, that is the candidate’s ability to comprehend the source language and to produce that which is comprehended in the target language. Within this overall division, the errors should also be judged in terms of interference and intervention, that is whether the candidate has unjustifiably interfered with the text or legitimately intervened, for linguistic, cultural, or information needs considerations and so on, to render the text intelligible to the target language reader.

Within this framework, the test should be marked on a rising scale from 1 to 5 as follows:

Table 1 Marking Criteria

Information integrity refers to the ability to retain the same information in terms of accuracy, correctness, completeness and intention (both informative and communicative).

Table 2 Information Integrity

Linguistic integrity refers to the ability to render the text in a sound language in terms of grammar, structure (both micro and macro levels), coherence and cohesion.

Table 3 Linguistic Integrity

Translation dexterity refers to the candidate’s translation skills and adroitness in terms of strategies, comprehension, production, matchability, and approximation.

Table 4 Translation dexterity

Fitness for purpose refers to the ability to render the text suitable for its purpose.

Table 5 Fitness for Purpose

Aesthetic effect refers to the ability to employ appropriate rhetorical techniques. This is an overall artistic impression, in terms of "tellability".

Checklists, templates, prototypes and blueprints can be developed to ensure that these factors are considered iteratively, that is on first, second and third passes etc. The first pass for example considers these factors at sentence level, the second pass at paragraph level, and the third pass at passage level (macro structure).

Informed Performers

In real-life situations, professionals are supposed to be informed performers. In other words, they are clearly briefed on what is expected of them and the tasks they are assigned to perform. Translators should not be any different. As professionals who know the processes, procedures and protocols of their profession, translators are briefed on the task at hand or seek clear instructions from their clients and customers. Therefore, translation tests should take this aspect into consideration if they are to be realistic and not haphazard or erratic. Accordingly, a translation test should include a clear brief about the purpose of the translation, a clear profile of the translation users and a description of their information needs. This should enable the candidate to perform better to meet specific user requirements, and produce usable translations that should be so assessed.

Conclusion

Certainly, the proposed approach demands additional effort on the examiner's part in the first place. It requires training and retraining in new methods of assessment and a total overhaul of the accreditation approach to translation that will bring it into line with modern trends in large international organizations. This approach focuses on translation engineering. It takes the process of translator assessment out of the confines of the school teacher mindset into the practical world of business and industry. Above all, this approach helps ensure that translation tests are reliable and valid.

It is inevitable in any objective endeavour to have an element of subjectivity. However, the extent of such subjectivity can be controlled and the urge to inject one’s own opinion into the process of assessment can be curbed by imposing stringent measurement criteria. This paper has advocated a positive approach to translator assessment that takes the issue of informed performers as its central point. In the age of the information superhighway, it is no longer the translator’s sole responsibility to unravel the mysteries of text and to wrestle with ambiguities and antiquated practices. Accordingly, the translation test should reflect this reality. Accreditation bodies are in the business of accrediting translators who are going to be working in the community, who will have access to texts written for the community dealing with community issues and who will have access to the authors of the original. As such, translation tests should become in tune with real-life translation processes and requirements.

References

© 1995 Ali Darwish
All rights reserved. No part of the contents of this document may be copied, reproduced, or stored in any retrieval system, without the express permission of the author.