Australian Review of Applied Linguistics. Supplement Series - Volume 13, Issue 1, 1996

The past decade in Australia has been characterised by a striking growth in the application of language testing, frequently in contexts governed by macro-political pressures. Despite this, the literature to date has rarely placed its first emphasis on the pragmatic considerations which typically give rise to test development, and impact on test design as well as administration outcomes. To illustrate the significance of this, this chapter explores the recent pressures surrounding the evolution and implementation of the Special Test of English Proficiency (step test) – a form of ESL testing designed not merely to assess linguistic competence, but to play a central role in the determination of residential status for substantial numbers of asylum seekers.

This article examines the decisions underpinning language test design in circumstances where existing tests are of questionable validity for particular subpopulations, or where no suitable test exists. Three strands of test development work are described, all concerned with the assessment of language abilities in bilinguals and speakers of English as a second language in Australia. The specific contexts discussed are: (1) the assessment of residual language-processing abilities following stroke; (2) the assessment of communicative ability following stroke; and (3) the assessment of first and second language abilities in Alzheimer’s dementia. For all of these purposes, tests need to be suitable for people of a range of ethnolinguistic backgrounds and degrees of bilingualism. Each context, however, raises its own set of considerations, and has therefore necessitated a different approach. In the first case, the starting point for the development of appropriate test content has been the investigation of a published aphasia test already available in many of the required languages. In the second case, an analysis of the communicative needs of a sample of stroke patients of non-English-speaking background was conducted, and content for a new measure selected on the basis of the results. In the third case, content selection has been informed primarily by the findings of empirical research on language decline in dementia.

The two most common approaches to rating second language performance pose problems of reliability and validity. An alternative method utilizes rating scales that are empirically derived from samples of learner performance; these scales define boundaries between adjacent score levels rather than provide normative descriptions of ideal performances; the rating process requires making two or three binary choices about a language performance being rated. A procedure, that consists of a series of five explicit tasks, is used to construct a rating scale. The scale is designed for use with a specific population and a specific test task.

A group of primary school ESL teachers used this procedure to make two speaking tests, including elicitation tasks and rating scales, for use in their school district. The tests were administered to 255 sixth grade learners. The scales were found to be highly accurate for scoring short speech samples, and were quite efficient in time required for scale development and rater training. Scales exhibit content relevance in the instructional setting. Development of this type of scale is recommended for use in high-stakes assessment.

As part of a research study on language maintenance and shift in the Indo-Fijian community in Wellington, New Zealand, a performance test was developed to assess the speaking and listening skills in Fiji Hindi of a sample of the Indo-Fijian teenagers. The design of the test needed to take account of the fact that Fiji Hindi is a preliterate vernacular language with no role in education and an ambivalent status within its own speech community. The test consisted of three main parts: a naturalistic conversation, two structured speaking tasks and a structured listening task. This paper focuses on some facets of the test administration, including the decision to administer it in the test-takers’ homes; the influence of various personal attributes of the interviewer; the ways of dealing with the lack of a script for Fiji Hindi; and the issue of live versus tape-based assessment of the test-takers’ performance. Both the test-takers and an independent rater provided feedback on the test that was generally very positive. The paper concludes with a discussion of various factors that may have influenced the reliability and validity of this somewhat unconventional language test.

Oral performance tests designed to assess candidates’ ability to function in a particular professional role typically involve role plays which simulate aspects of professional interaction between a candidate, in his/her professional role, and an interlocutor, in the role of client or consumer of the professional services. The validity of such tests is dependent both on the incorporation of appropriate content and on the extent to which the interaction between candidate and interlocutor simulates authentic professional interaction. Materials for such role plays are commonly produced according to specifications developed by language testers following consultation with professional informants, but there has been little investigation of how well tasks reflect their purpose (i.e. the aims/intentions of the professional experts). Test interlocutors, generally language teachers, normally undergo some form of training in the test administration procedure, but there has been little analysis of how well their behaviour reflects client behaviour.

This paper reports on an investigation into the role which expert professional informants can play in providing post hoc content validation of tasks used in this kind of occupation-specific test, focusing on both the content of the tasks and the behaviour of the interlocutors. The study draws on data from the oral interaction component of an occupation-specific test, the Occupational English Test (McNamara 1996), a 4-skills test for overseas-trained health professionals (with attention in this study to nurses). The study involved a two-day workshop where representatives of the nursing profession, in conjunction with the researchers, analysed both written test materials and audiotaped test data, contributing to an analysis of the realism of the tasks and the features of interlocutor behaviour.

This paper describes a methodology for exploring the validity of the Rasch scaling procedure in relation to a multiple-choice test of reading Chinese as a foreign language. The validation procedure involved a post hoc content analysis of test items to identify factors which the research literature on early reading indicated were likely to be related to item difficulty. A comparison was then made between these difficulty factors and the item-difficulty ranking produced by a Rasch analysis of test items. The analysis revealed that the dimension mapped out by the Rasch scaling (and to some extent the corresponding wording of the reporting descriptors) does indeed reflect certain elements thought to be related to early reading ability.

Test score interpretation and use are the staple of construct validity. As such, in addition to the concern with measurement accuracy, it is imperative that the meaning of test scores and their intended use(s) be also documented. Along these lines, qualitative speech analyses are undertaken in the present paper to help in the interpretation of the dimensions underlying student performance on oral tasks. Results of these analyses yield rich information that explicate the meaning of the dimensions by delineating their specific features as manifested in the speech samples. Also discussed in the paper are the ramifications of these results for pedagogical use. Insights that linguistic accuracy and communicative skills in general, and their specific features specifically, provide for instructional material and activities are addressed. Furthermore, a case is made for curricular improvements to help learners develop well-rounded L2 abilities and to improve their use of the language for real-life communication. Finally, with regard to assessment, it is argued that generic assessment criteria do not reflect the critical features operating in a given context, and assessment practitioners are urged to study their contexts of use and to tailor their criteria according to the particulars of those contexts.

In Japan it has long been considered that university entrance exams have a negative impact on teaching and learning of English in schools. Recent research, however, suggests that the relationship between testing and teaching is not so simple and requires finegrained ethnographic research in order to fully understand its complexities. This paper argues the case for the necessity of conducting empirical research before any conclusions about the presence or absence of washback can be drawn. An example of this type of research is offered and the processes involved in investigating washback in the Japanese context are described in such a way as to provide a model for future research in this area.