The Common European Framework for testing and teaching

By Jennifer Wallace
Anhui Gongye Daxue, Ma'anshan, China

Lots of us are trying to develop tests appropriate for the situations we're teaching in. One document I'd recommend, because I've found it enormously helpful, is the Council of Europe Frameowrk, which is on the Internet, as a downloadable pdf file (for which you need to have Adobe Acrobat Reader on your machine). I like the document for several reasons.

The work behind it is the work of a large number of experts across Europe, who've developed one framework to cover the teaching (and testing) of any of the languages taught and used in Europe - which of course includes a variety of non-European languages. In other words, the whole thing is language independent. I understand it to be very much a reflection of the most up to date understanding we have of measuring language performance. The particular document in question is the latest version, the result of many revisions.

The document addresses the fundamental questions in all this, and looks at every dimension conceivable - so I can use it as a basis for testing speaking, listening, reading, anything. It looks at things on general levels and on detailed specific levels - so you can home in on the level that is relevant for you at the moment.

Because this framework is as comprehensive as it is, it's let me think up a variety of activities for the form of my tests, activities that reflect the students experiences and what they've done in a course. But at the same time it's kept me very much on track, enabling me to see clearly what level our target it.

Because it's not language-specific, you can test yourself (there's one section on self-testing) for your Chinese to see how this sort of approach works.

Someone also commented about examiners' ability not to be swayed - well, I think what allows me to be more objective is using a number of scales and criteria when I test. For example, this semester my college end-of-first-year students will get some marks for pronunciation (because we've done quite a bit of pronunciation work on their Oral English classes), some marks for fluency, some marks for grammar, some marks for vocabulary/lexis and some marks for coherence. I'm also thinking about including some marks for how they deal with problems - repair work, asking for help, paraphasing, miming, using fillers to gain thinking time and to fill a silence, and the suchlike - what's called strategic competence. My criteria for vocab/lexis and grammar will not be whether they demonstrate use of anything in particular, but in how effective they are at communicating successfully - do their errors interfere with communication, or hinder it, or render it impossible! This is because I teach college English majors - I think testing for specific aspects of these dimensions is the responsibility of other teachers in other classes. but at the same time, my students do realise that I consider grammar and lexis to be seriously important.

As regards a quick test, my experience, and the experience of other testing large numbers quickly for summer schools (in UK language schools), is that in an informal chat of around 5 minutes, grading only on a 5 pint scale (with very easy to understand scoring 5) is a remarkably effective tool in the hands of a native speaker. Even on the most mundane of topics (your home town, your family), it sorts the lower from the higher from the in betweens. I did this at the beginning of this year with my 225 new students, and on subsequent reflection, having taught them now for 2 semesters, remarkably few of my initial assessments were wrong, and none were way off. What's interesting is looking back at their subsequent development! The value for me is how much respect I have for the students who got a low rating at the beginning who would only now get a middle rating - but wow, what progress! In each band, I can see students who have really made big efforts and made progress, and I can also see students who've made almost no progress. Of those, a small number are not interested in the effort it entails (basketball etc is more important), but I also have one or two who I realise are making efforts but little progress. I think that initial testing and placement has really helped me, and I plan to do it for future Oral English classes.

One thing I did was use the test results to make groups according to level, and that's been very successful as well.

On testing oral English

By George

To accurately test my students, I give them oral exams which are recorded on tape. These exams have two parts. The first part is Q&A covering things we have covered in class. They almost always have a memorized response for the basic questions. I tend to ignore these. I focus on their responses to the followup questions. For example, I've told them that we might discuss their grandparents, soI might ask

"Are your grandparents alive?" "How many children did they have?" How many boys and how many girls.? "Do you know your aunt's and uncles?" "O.K let's talk about your youngest aunt" Here is where they begin to breakdown because they didn't think to prepare for a discussion about their youngest aunt. I've also begun ba asking about a favorite middle-school teacher and them focus on the teacher they liked the least. Once I gotten to the real subject I'll begin with what is the persons name, age etc. and gradually lead to more complex questions. Then I start looking for syntactic, grammatical and vocabulary failure. In many cases the exam has ended in 2 or 3 minutes and some have gone as long as 30 or 40 minutes. In all cases I use subjects they are familiar with. Family, School, Friends and Hometowns. If I knew more about sports I would dwell on that. I have been know to ask a student to explain what a mid-fielder, a striker or a goalie does if they play those positions in football or the role of Guards, the Center or Forwards in basketball. I've even asked guitar playing students to explain how to play a particular son. In short they give me a guitar lesson.

To test for middle school, determine what is grade appropriate and start from there.

Again, start simple and progress to the complex. At what level do they abandon an answer or the topic entirely. The second part is a short oral reading which incorporates most of the english phonemes. I sometimes give the samples to practice with but they get a new reading for the exam. The must read cold.

Also, I've just begun developing a set of reading passages tha will begin at about fifth or sixth grade level for native speakers using Flesch-Kincaide RGL measures and which become progressively more advanced. This way I can determine the level at which they begin to break down, identified by their rate of word abandonment. In the first year I will be mainly concerned with phonetic identification and production. As we progress, stress and intonation will become more of a factor.

Oral exams can be quantified, but I don't like using them as the basis for a grade. I tell the school that grades shoud be considered as a report of a student's speaking level and how much they have improved. In my classes, the only one's who acutally fail are those who only show up for exams and the rare film. Those who come to class but aren't there count as absent. Our school weeds them out pretty quick. Last term eight of my students flunked out including two who were pretty good english speakers. Six were expelled for cheating on Chinese teacher's exams.

A speaking test examiner teaches speaking

By Jennifer Wallace - Anhui Gongye Daxue, Ma’anshan City, Anhui Province, China

When I came to teach here, although I‘d been a speaking test examiner for more than 10 years (for UCLES exams) I‘d actually never had to set an oral English exam before. I’d taught always in situations where the students were either taking no exam or were working towards an external exam. So if I did have to set tests, they were very much on the mock-exam model. I’d never taught a modern language within a university/college setting where this was the student’s main subject (major). Although I had a short training specific to coming to this post in China, provided by the NGO who sponsor my post, I came with some sort of assumption that there would be a syllabus, there would be designated attainment targets (although not necessarily expressed in that way). Well, you all know the reality here. I was timetabled for first year Oral English classes who were provided with one of the ORAL ENGLISH WORKSHOP series of books. If anything, I found that was worse than arriving to nothing. It implied someone somewhere thought the content of this course book was what my students should be mastering.

Anyway, after a semester of muddling along and getting some sort of impression about what might be possible, I realised that the lowest of the UCLES EFL exams I’d been a speaking test examiner for was probably within the reach of everyone in the class. I’d been warned about the tradition of everyone in the class passing the exams. Remember I’d done those UCLES tests for years. I could remember the type of tasks set in the exam, and I produced a parallel. Those UCLES tests are taken in pairs, but I chose to give each student an individual exam - partly as a public relations exercise about oral exams within my department. I was interlocuter as well as
assessor. So I recorded all the exams and marked them from the tapes. I was right in that all my students were capable of attaining that first level in the UCLES hierarchy, which means that in a grander scheme of things they had all achieved the Council of Europe Basic User level. The descriptors for this (in summary) are:

Can understand sentences and frequently used expressions related to areas of most immediate relevance (e.g. very personal and family information, shopping, local geography, environment).

Can communicate in simple and routine tasks requiring a simple and direct exchange of information on familiar and routine matters.

Can describe in simple terms aspects of his/her background, immediate environment and matters of immediate need.

At the end of the first semester all my students could do that - although a good number could only just do it with a very sympathetic interlocutor. Others walked through it. Which gave me a good spread of marks. And on that basis I decided to model the exam at the end of the second semester on the next level up (which I’d also done examining work for). By that stage my classes had included a fair amount of group work, and so the exam was done in small, randomly selected groups of 4, not including me. This year I’ve done lots more group work, but am actually planning to give one-to-one exams at this stage instead - partly for comparison.

So my decisions were based on a combination of what was within my own capabilities as well as the students. I’m not an expert on language testing. My only teacher training is a CELTA, and in the past when I’ve had to devise and construct college tests it was done under the supervision of a very experienced head of department. But also I’m not into re-inventing the wheel. The Council of Europe stuff - which relates to ALL the languages taught in Europe (and that includes teaching non-European languages) - is the result of mega-input from experts over heaven knows how many years now. I feel I’d be deluding myself if I thought I could devise any better sort of structure to work within - so I’m using it. I do also like it - I find it clear and easy to get my head around.

Developing tests for academic listening & speaking

By Erlyn Baack - ITESM, Campus Queretaro, Mexico

A teacher asked about "Listening/Speaking exams to assess low intermediate students if they are ready for high intermediate level and then high intermediate students if they are ready for advanced level of ESL. Putting an emphasis on better academic preparation, in Listening we stress listening to lectures and note-taking, and in Speaking, we stress academic vocabulary, grammar correctness, and presentation techniques. We consider (1) exam content, (2) testing efficiency considering the number of students, (3) a rubric for Listening/Speaking, (4) involvement of outside instructors to make testing more objective.

The teacher asked a whole series of interrelated questions involving testing of 40 low intermediate and 80 high intermediate students, and while the questions were specifically about "Listening/Speaking exams, the emphasis is much broader including listening and note-taking and academic vocabulary, grammar correctness, and presentation techniques.

Those questions, therefore, are all-encompassing questions that must be asked early on in the semester or even before the semester begins.

Teachers at both levels and the school administration should ask themselves (1) What do we want the students at each level to know (or be able to do) when they finish the level, (2) how are we going to teach it, and (3) how are we going to test it? Essentially, therefore, teachers at both levels should be (or should have been) addressing these questions in their lesson plans and classroom activities since the beginning of the semester. If daily or weekly classroom activities are designed for students to achieve the successes desired by the time of the final exam, their probability of success is high.

It would seem to me that listening to lectures and note-taking would be the easiest to teach and test although it takes collaboration among teachers and administrators on setting it up. For a simple classroom activity, I would recommend going to a website like Living on Earth at http://loe.org , download some MP3 files of radio stories, play them for the class while they take notes, and then test using ten multiple choice questions, for example. (Parenthetically, it is EASY to get written permission from LOE to use their MP3s in this manner, even including putting their files on CDs for the reserve in the library or for free distribution.) After the quiz (or before in some instances) students at both levels should have the opportunity to discuss the stories because they are interesting, sometimes polemic, and they can generate a lot of discussion. The last radio show is at consisting of seven stories ranging from three minutes to 17 minutes, and finally, an additional benefit is that the text is always available to these MP3 stories as well.

Regarding testing, to be fair, academic vocabulary and grammar correctness can be taken from materials (like Living on Earth) that students have already been exposed to. The text can be used for that, and at the levels Inna Braginsky is asking about, students can always benefit from focused attention toward S-V agreement, pronouns, modals, simple, compound, and complex sentences, adjective clauses, parallel structure, etc. It is up to the teachers and administrators to address the most obvious weaknesses at each level and focus on a few specifics students can study, so when they take their multiple choice texts on these things, they can succeed. (No apologies for multiple choice tests here! ;-) )

The hardest part is the speaking exam because it requires an enormous amount of consensus among TRAINED teachers.

First, teachers must agree on a series of possible questions or topics which may be announced beforehand or not, their preference. I think to be reliable the topic should come from topics students have already been exposed to, much like the Living on Earth topics suggested above (a dozen topics would not be too many for students to prepare for). After students are given three or four minutes to present a (1) SUMMARY or an (2) ARGUMENT for something or a (3) RATIONALE against something (for examples), teachers should ask a pertinent follow-up question or two, and both teachers/administrators and students should at least feel that the speech topic was fair; it was a topic the student was at least exposed to and had the chance to study.

Then, teachers should assess the speaking abilities of the students (both the prepared and extemporaneous parts), and ideally they will have worked together long enough with the parts of their rubric to arrive at a consensus independently. If four teachers, for example, assign the student a four of six points on an aspect of his speaking performance, they should be very proud of that!

I didn't address bringing a bunch of non-ESL-trained teachers into the mix, but their opinions count! Spaces both formal and informal should be available for "mainstream" teachers to explain the "greatest weaknesses" of the ESL students, but I wouldn't invite them to exams.

Using rubrics to evaluate spoken English

By Maria Spelleri - Manatee Community College, Florida, USA

One way to get a sense of structure with the evaluation of student oral production is by using a rubric. Here's an example of a speaking rubric for an ESL program in a US elementary school system: RUBRIC and here's a site with programs to help you develop a rubric: DEVELOP RUBRIC

To create a rubric for a speaking activity such as retelling a story, you need to break the activity down into its most basic elements. For example, speech is comprised of vocabulary, grammar, pronunciation/stress/intonation, logical meaning and order, purpose, and in the case of the story, an element of cohesion. For the specific task, you might want to also consider the accuracy of the retelling, the amount of detail included, number or length of pauses and inappropriate filler noises, etc. Then, for each category, set the possible performance/assessment levels, for example, "excellent", "satisfactory", and "needs improvement". I prefer to work with a basic set of 3 as it is easier for me to break down a production into bad, so-so, and good instead of more subtle variations- although plenty of instructors use 4 and 5 categories.

If you do a Google search using key words like "ESL Speaking rubric", you should find many ideas to help you create a rubric that will meet your needs.

By the way, I would suggest recording the assessment either audio or audio/video because it can be hard to listen to content, mentally evaluate, and complete a rubric at the same time. Replaying the audio gives you time to better analyze the students' work and assess more fairly. Playing back the recording for the student who can then watch him or herself and compare the recording to the completed rubric assessment is a valuable learning tool as well.

Evaluating L2 socialization skills

By Nik Bramblett - UCF, Orlando FL, USA

Sometimes we need to evaluate L2 socialization skills using an alternative assessment and not a paper test.

Here's what I would do:

(a) Work with students (using appropriate combination of whole group, breakout small-group, and/or individual/paired strategies) to develop a rubric for a role-playing activity. Discuss what "socialization skills" means and how you might measure mastery of them. Let the students decide what's important and what they will be graded on (with appropriate guidance from you as necessary, of course).

(b) Have students work in pairs or trios for the assessment... students would randomly select a social problem-solving situation from a collection that you created on cards or whatever... "You need make an important call [make up a specific scenario] and your cell phone is dead; there are two strangers nearby [perhaps it's a bus stop or whatever]. Interact with those people to solve your problem." for example. Students would have a brief period to plan/rehearse, and would then more-or-less improv a scene.

(c) Both you and the student audience would use the rubric you designed together (and reviewed clearly and modeled and practiced before these presentations began) to measure the ability of the students to perform whatever specific tasks, roles, etc. you had decided were the measurable objectives. Students' ability to effectively judge their peers' performance would (rightly) be part of the grade. This would not only measure the mastery of the skills but also the metacognition behind the skills.

Scenario-based assessment of socializing skills

By Noriko Ishihara - University of Minnesota, USA / Hosei University

[An excellent way to test students language abilities is in a realistic setting. But how can that be done? Noriko Ishihara explains.]

How to do a scenario-based assessment of socializing skills. In my view, it's very close to assessing sociolinguistic/pragmatic ability, which has usually been done with a situational approach.

In this instruction and assessment, learner language is elicited using realistic scenarios and the teacher chooses from a range of language- and culture-focused features to assess, for example,

- directness, politeness, and formality
- organization/discourse structure
- language form, semantic strategies, word choice
- tone (verbal and non-verbal cues)
- understanding and use of sociocultural norms
- the extent to which the speaker's intentions match the listener's most likely interpretation

The selected feature(s) can be assessed using various rubrics and/or checklists by the teacher and learners themselves, which can be used as rather formal assessment or part of everyday instruction/informal assessment. If anyone is interested, I'd be happy to share a paper in press that details this approach with various sample scenarios, learners language, and sample assessment using authentic learner language.

Fighting plagiarism with Google

By Chuck in China

I was helping a teacher who was grading some papers she was not sure about. She gave me the two papers and I quickly did a Google search and located both papers on the Web. She was shocked at how quickly I sniffed out the plagiarism. So she went back through all of her papers and all night she has been coming to my room and dropping off papers. Not all of them, just the ones that she, in her heightened state of awareness, found suspect.

All but one I found to be taken from the web. I found them all easily through Google. Just pick out an uncommon phrase from the paper and run it through Google and it pops up. Interestingly, all of the papers that were used came from various Chinese websites "dedicated" to English learning.

I used this method this semester when I suspected quite a few essays in my Western Lit class. I quickly found 6 plagiarists. This doesn't surprise me at all.

The quick lessons for those teaching substantive courses here is:

1. Be careful in assigning essay assignments. Don't pick a broad topic (as my colleague did) such as: "Write a persuasive essay on an important subject" or (as I did the first time I made this mistake) "Choose one of Shakespeare's sonnets, analyze it, and tell me why it appeals to you". Their are thousands of these essays floating around on the Internet on the Chinese websites alone (as a quick search of Google will show). The next time I chose an essay topic, I wanted to use O. Henry. But I first did a Google search to see which of his stories had the least number of references on the Web. I certainly wasn't going to use "Cop and the Anthem" (which all Chinese students know by heart anyway).

2. Use Google to check the originality of any essays you assign. It takes less than a minute to check each paper. You may be surprised how quick and how many plagiarists you find.

3. In my writing classes, I have decidedly swung away from giving take-home essays as assignments. Now, I give writing assignments in class to be completed in class. Apart from the plagiarism issue, it hones the students' skills in actually writing under some kind of immediate pressure. It forces them to produce good English on the spot. Also, as Chinese students who are forced to deal with the exam system here (CET, TEM, PET), a writing component is a part of all those exams so giving them time pressure to perform will help them in the short run, too.

Anyway, that's how I spent the first night of my summer vacation - tracking down plagiarism for a fellow colleague and, (I hope) imparting some useful advice for those who are new to the game here.

Foreign teachers in China

By Don YD Chen - Liuzhou, Guangxi, China

I have been working with foreign teachers working in Chinese institutions for many years and have alway been in good terms with them. I, as a Chinese teacher of English, fully understand their situations in this vast land, where the culture is so diverse that one can hardly avoid continuous shocks such culture shocks, food shocks or shocks of whatever one can imagine, in the first few months (or a year). Worse, there are students whose enthusiasm shrinks soon after they discover that foreign teachers are not good at teaching them 'to pass exams'.

Assessing with the Council of Europe Framework

By Jennifer Wallace - Anhui Gongye Daxue, Ma'anshan, China

Lots of us are trying to develop tests appropriate for the situations we're teaching in. One document I'd recommend, because I've found it enormously helpful, is the Council of Europe Framework, which is on the Internet, as a downloadable pdf file (for which you need to have Adobe Acrobat Reader on your machine). I like the document for several reasons.

The work behind it is the work of a large number of experts across Europe, who've developed one framework to cover the teaching (and testing) of any of the languages taught and used in Europe - which of course includes a variety of non-European languages. In other words, the whole thing is language independent. I understand it to be very much a reflection of the most up to date understanding we have of measuring language performance. The particular document in question is the latest version, the result of many revisions.

The document addresses the fundamental questions in all this, and looks at every dimension conceivable - so I can use it as a basis for testing speaking, listening, reading, anything. It looks at things on general levels and on detailed specific levels - so you can home in on the level that is relevant for you at the moment.

Because this framework is as comprehensive as it is, it lets me think up a variety of activities for the form of my tests, activities that reflect the students experiences and what they've done in a course. But at the same time it's kept me very much on track, enabling me to see clearly what level our target it.

Because it's not language-specific, you can test yourself (there's one section on self-testing) for your Chinese to see how this sort of approach works.

Someone also commented about examiners' ability not to be swayed - well, I think what allows me to be more objective is using a number of scales and criteria when I test. For example, this semester my college end-of-first-year students will get some marks for pronunciation (because we've done quite a bit of pronunciation work on their Oral English classes), some marks for fluency, some marks for grammar, some marks for vocabulary/lexis and some marks for coherence.

I'm also thinking about including some marks for how they deal with problems - repair work, asking for help, paraphrasing, miming, using fillers to gain thinking time and to fill a silence, and the suchlike - what's called strategic competence. My criteria for vocab/lexis and grammar will not be whether they demonstrate use of anything in particular, but in how effective they are at communicating successfully -do their errors interfere with communication, or hinder it, or render it impossible! This is because I teach college English majors - I think testing for specific aspects of these dimensions is the responsibility of other teachers in other classes. but at the same time, my students do realise that I consider grammar and lexis to be seriously important.

As regards a quick test, my experience, and the experience of other testing large numbers quickly for summer schools (in UK language schools), is that in an informal chat of around 5 minutes, grading only on a 5 point scale (with very easy to understand scoring 5) is a remarkably effective tool in the hands of a native speaker. Even on the most mundane of topics (your home town, your family), it sorts the lower from the higher from the in betweens. I did this at the beginning of this year with my 225 new students, and on subsequent reflection, having taught them now for 2 semesters, remarkably few of my initial assessments were wrong, and none were way off.

What's interesting is looking back at their subsequent development! The value for me is how much respect I have for the students who got a low rating at the beginning who would only now get a middle rating - but wow, what progress! In each band, I can see students who have really made big efforts and made progress, and I can also see students who've made almost no progress. Of those, a small number are not interested in the effort it entails (basketball etc is more important), but I also have one or two who I realise are making efforts but little progress. I think that initial testing and placement has really helped me, and I plan to do it for future Oral English classes. One thing I did was use the test results to make groups according to level, and that's been very successful as well.

Speaking of oral exams

By George -

To accurately test my students, I give them oral exams which are recorded on tape. These exams have two parts. The first part is Q&A covering things we have covered in class. They almost always have a memorized response for the basic questions. I tend to ignore these. I focus on their responses to the follow-up questions. For example, I've told them that we might discuss their grandparents, so I might ask

"Are your grandparents alive?" "How many children did they have?" How many boys and how many girls? "Do you know your aunt's and uncles?" "O.K let's talk about your youngest aunt" Here is where they begin to breakdown because they didn't think to prepare for a discussion about their youngest aunt. I've also begun by asking about a favorite middle-school teacher and then focus on the teacher they liked the least. Once I get to the real subject I'll begin with what is the person’s name, age etc. and gradually lead to more complex questions. Then I start looking for syntactic, grammar and vocabulary failure. In many cases the exam ends in 2 or 3 minutes and some have gone as long as 30 or 40 minutes. In all cases I use subjects they are familiar with: Family, School, Friends and Hometowns. If I knew more about sports I would dwell on that. I have been known to ask a student to explain what a mid-fielder, a striker or a goalie does if they play those positions in football or the role of guards, the center or forwards in basketball. I've even asked guitar playing students to explain how to play a particular song. In short they give me a guitar lesson.

To test for middle school, determine what is grade appropriate and start from there.

Again, start simple and progress to the complex. At what level do they abandon an answer or the topic entirely. The second part is a short oral reading which incorporates most of the English phonemes. I sometimes give the samples to practice with but they get a new reading for the exam. They must read cold.

Also, I've just begun developing a set of reading passages that will begin at about fifth or sixth grade level for native speakers using Flesch-Kincaid RGL measures and which become progressively more advanced. This way I can determine the level at which they begin to break down, identified by their rate of word abandonment. In the first year I will be mainly concerned with phonetic identification and reproduction. As we progress, stress and intonation will become more of a factor.

I've not seen the CET tests, so I can't comment on those. Oral exams can be quantified, but I don't like using them as the basis for a grade. I tell the school that grades should be considered as a report of a student's speaking level and how much they have improved. In my classes, the only one's who actually fail are those who only show up for exams and the rare film. Those who come to class but aren't there count as absent. Our school weeds them out pretty quick. Last term eight of my students flunked out including two who were pretty good English speakers. Six were expelled for cheating on Chinese teacher's exams.

Testing spoken English

By George

To accurately test my students, I give them oral exams which are recorded on tape. These exams have two parts. The first part is Q&A covering things we have covered in class. They almost always have a memorized response for the basic questions. I tend to ignore these.

I focus on their responses to the follow-up questions. For example, I've told them that we might discuss their grandparents, so I might ask, "Are your grandparents alive?" "How many children did they have?" How many boys and how many girls.? "Do you know your aunt's and uncles?" "O.K let's talk about your youngest aunt" Here is where they begin to breakdown because they didn't think to prepare for a discussion about their youngest aunt.

I've also begun ba asking about a favorite middle-school teacher and them focus on the teacher they liked the least. Once I gotten to the real subject I'll begin with what is the persons name, age, etc. and gradually lead to more complex questions.

Then I start looking for syntactic, grammatical and vocabulary failure. In many cases the exam has ended in 2 or 3 minutes and some have gone as long as 30 or 40 minutes. In all cases I use subjects they are familiar with. Family, School, Friends and Hometowns. If I knew more about sports I would dwell on that. I have been know to ask a student to explain what a mid-fielder, a striker or a goalie does if they play those positions in football or the role of guards, the center or forwards in basketball. I've even asked guitar playing students to explain how to play a particular song. In short they give me a guitar lesson.

To test for middle school, determine what is grade appropriate and start from
there. Again, start simple and progress to the complex. At what level do they abandon an answer or the topic entirely. The second part is a short oral reading which incorporates most of the english phonemes. I sometimes give the samples to practice with but they get a new reading for the exam. The must read cold.

Also, I've just begun developing a set of reading passages tha will begin at about fifth or sixth grade level for native speakers using Flesch-Kincaide RGL measures and which become progressively more advanced. This way I can determine the level at which they begin to break down, identified by their rate of word abandonment. In the first year I will be mainly concerned with phonetic identification and production. As we progress, stress and intonation will become more of a factor.

Oral exams can be quantified, but I don't like using them as the basis for a grade. I tell the school that grades shoud be considered as a report of a student's speaking level and how much they have improved.

In my classes, the only one's who acutally fail are those who only show up for exams and the rare film. Those who come to class but aren't there count as absent. Our school weeds them out pretty quick. Last term eight of my students flunked out including two who were pretty good english speakers. Six were expelled for cheating on Chinese teacher's exams.

Difficulties in scoring oral exams

Tony Lee - Shengda College, Zhengzhou, Henan, China

My marking of the first half semester exam last year was a disaster of sorts. Totally inexperienced, I was far more subjective than I should have been but the kids seemed satisfied with their placings. The marks were not a problem because another techer with zero grading experience of any sort scored the whole class between 90 and 100 so to make it fair we had to do the same.

Next exam I used a 'proper' grading rubric and slavishly referred to it while listening to each member of the groups of three. It still resulted in a bunched up result - spread of 10%.out of a possible spread of 40%. Changed the rubric again and this last lot of results was just as closely bunched.

I know that you can turn them into any spread you like but that is a bad way of correcting my own testing "incompetance". My problem is I lack the killer instinct of a practiced examiner I find it impossible to nit-pick to the extent required and then ruthelessly apply the marking scale.

As others have commented -- perhaps with oral it does not matter quite so much if the ability to speak is what we are testing because provided the students are given at least a couple of minutes to prepare they all do a pretty good job of speaking (I see them in 3's and each student has a different topic to talk on and they have had 10 minutes to prepare. After each has spoken I get involved in a general discussion on some safe personal topic and gauge how well they cope).

The problem is I do not really test their ability to cope with a typical staff meeting or questioning from an audience or even a job interview situation. I have had interview sessions in the classroom and the typical result is that neither side has a clue what the other is saying.they are so busy worrying about working out what they are going to say that they have nothing to spare for listening what the other side is saying -- a situation many native speakers also find themselves in. No answers from me I'm afraid - just the same old problems.

Our limited experience with failures here and a bit of insider info from the Chinese teachers leads me to the conclusion that there are no real failures, just those who just scrape through.

Difficulties in scoring oral exams

Tony Lee - Shengda College, Zhengzhou, Henan, China

My marking of the first half semester exam last year was a disaster of sorts. Totally inexperienced, I was far more subjective than I should have been but the kids seemed satisfied with their placings. The marks were not a problem because another techer with zero grading experience of any sort scored the whole class between 90 and 100 so to make it fair we had to do the same.

Next exam I used a 'proper' grading rubric and slavishly referred to it while listening to each member of the groups of three. It still resulted in a bunched up result - spread of 10%.out of a possible spread of 40%. Changed the rubric again and this last lot of results was just as closely bunched.

I know that you can turn them into any spread you like but that is a bad way of correcting my own testing "incompetance". My problem is I lack the killer instinct of a practiced examiner I find it impossible to nit-pick to the extent required and then ruthelessly apply the marking scale.

As others have commented -- perhaps with oral it does not matter quite so much if the ability to speak is what we are testing because provided the students are given at least a couple of minutes to prepare they all do a pretty good job of speaking (I see them in 3's and each student has a different topic to talk on and they have had 10 minutes to prepare. After each has spoken I get involved in a general discussion on some safe personal topic and gauge how well they cope).

The problem is I do not really test their ability to cope with a typical staff meeting or questioning from an audience or even a job interview situation. I have had interview sessions in the classroom and the typical result is that neither side has a clue what the other is saying.they are so busy worrying about working out what they are going to say that they have nothing to spare for listening what the other side is saying -- a situation many native speakers also find themselves in. No answers from me I'm afraid - just the same old problems.

Our limited experience with failures here and a bit of insider info from the Chinese teachers leads me to the conclusion that there are no real failures, just those who just scrape through.

Testing oral English

By Jennifer Wallace - Anhui Gongye Daxue, Anhui Province, China

When I came to teach here, although I‘d been a speaking test examiner for more than 10 years (for UCLES exams) I‘d actually never had to set an oral English exam before. I’d taught always in situations where the students were either taking no exam or were working towards an external exam. So if I did have to set tests, they were very much on the mock-exam model.

I’d never taught a modern language within a university/college setting where this was the student’s main subject (major). Although I had a short training specific to coming to this post in China, provided by the NGO who sponsor my post, I came with some sort of assumption that there would be a syllabus, there would be designated attainment targets (although not necessarily expressed in that way). Well, you all know the reality here.

I was timetabled for first year Oral English classes who were provided with one of the ORAL ENGLISH WORKSHOP series of books. If anything, I found that was worse than arriving with nothing. It implied someone somewhere thought the content of this course book was what my students should be mastering.

Anyway, after a semester of muddling along and getting some sort of impression about what might be possible, I realised that the lowest of the UCLES EFL exams I’d been a speaking test examiner for was probably within the reach of everyone in the class. I’d been warned about the tradition of everyone in the class passing the exams. Remember I’d done those UCLES tests for years.

I could remember the type of tasks set in the exam, and I produced a parallel. Those UCLES tests are taken in pairs, but I chose to give each student an individual exam - partly as a public relations exercise about oral exams within my department. I was interlocuter as well as assessor.

So I recorded all the exams and marked them from the tapes. I was right in that all my students were capable of attaining that first level in the UCLES hierarchy, which means that in a grander scheme of things they had all achieved the Council of Europe Basic User level. The descriptors for this (in summary) are:

Can understand sentences and frequently used expressions related to areas of most immediate relevance (e.g. very personal and family information, shopping, local geography, environment).

Can communicate in simple and routine tasks requiring a simple and direct exchange of information on familiar and routine matters.

Can describe in simple terms aspects of his/her background, immediate environment and matters of immediate need.

At the end of the first semester all my students could do that - although a good number could only just do it with a very sympathetic interlocutor. Others walked through it. Which gave me a good spread of marks. And on that basis I decided to model the exam at the end of the second semester on the next level up (which I’d also done examining work for). By that stage my classes had included a fair amount of group work, and so the exam was done in small, randomly selected groups of 4, not including me. This year I’ve done lots more group work, but am actually planning to give one-to-one exams at this stage instead - partly for comparison.

So my decisions were based on a combination of what was within my own capabilities as well as the students. I’m not an expert on language testing. My only teacher training is a CELTA, and in the past when I’ve had to devise and construct college tests it was done under the supervision of a very experienced head of department. But also I’m not into re-inventing the wheel.

The Council of Europe stuff - which relates to ALL the languages taught in Europe (and that includes teaching non-European languages) - is the result of mega-input from experts over heaven knows how many years now. I feel I’d be deluding myself if I thought I could devise any better sort of structure to work within - so I’m using it. I do also like it - I find it clear and easy to get my head around.

I’m also interested to find that now I’ve got hold of an English translation of the syllabus for our English & Education majors, which details Band 2, 4 and 6 targets, I’m starting to be able to relate them meaningfully to the band descriptors I’m used to using. I’ve come across this syllabus too late to affect how I teach and examine this year, but it’s certainly going to help me next year.

It also makes me realise that we tend to see the Band exam stuff from the student-obsession perspective, while underlying it there is the same sort of work I’ve been used to being aware of in a European context. Here, I realise I see the students’ type of understanding - the seeing only the tip of the iceberg - because I‘m not included in my department as a colleague with access to formal and informal discussion about all this.

From my students I have an impression of teaching that’s come down to a lowest common denominator sort of level influenced by the need to get students through those exams to allow them to graduate. But I think I can also see where my own college department is failing to meet the specifications in that syllabus, both in intention and reality. Which in itself is interesting as this is a department that an international NGO thought warranted support in its development.

Helping students with vocabulary

Terry Pruett-Said - English for Academic Purposes, Macomb Community College, Michigan, USA

In my extensive/academic reading class, students have to hand in a short book report each week. As part of that book report, they have to write down 4 new vocab words with its definition, and the sentence it came from in the book. Advanced students have to also find another example sentence and write a sentence of their own.

Sometimes when we are reading in class, I just ask the students to stop and share a word with the class. At other times I ask students to write down two words each on a piece of paper I hand around. Sometimes I ask students to share one thing they've learned from what they are reading, and they often give a word.

From these methods I choose words that I think will be useful often based on the academic word list. I make a crossword puzzle. I already have many of the definitions because the students have written them. When I don't, I use my online Longman dictionary to quickly look up the definition on my computer while I'm making the crossword puzzle.

I also make questions with the vocabulary words that students have to discuss in class. Some of these usually show up on their vocabulary quiz. My vocabulary quiz usually consists of a matching exercise, a true/false exercise, a fill-in-the blank
that usually comes from sentences that I get and sometimes modify from my online Longman dictionary too. Then I add some of the questions that they had in their list. Having students choose their own words to some extent is time-consuming, but seems to be motivating.

In my writing/grammar classes students don't choose the vocabulary words. I choose them from readings in their grammar books. Again I try to choose words that will be useful to them, and I find the academic word list a good start. I give the students a dictation using sentences with the vocabulary words. Usually just three at the beginning of class, but I often try to get 2-4 vocab words in each sentence.

Sometimes I then assign homework with the vocabulary words--usually I have them write sentences with the words and sometimes definitions too. The definitions and sentences have to be the same meaning as in the reading which seems to be one of the biggest challenges. I also require students to use 2-3 of the new vocabulary words in their paragraphs and essays. The vocabulary shows up again on unit tests.

I teach at a community college and most of my students aren't in my class because they want to be, but because they have to be. These are not the students who are going to improve their vocabulary on their own. I feel in these circumstances it is really necessary to create a situation where students have to notice and learn new words. It's like chipping away at an iceberg, but every little chip adds up.

English teaching problems in China

By Wu Jun - Luoyang Foreign Languages University, Henan

Why can't Chinese speak fabulous English? Well, in the past I think some of Chinese did speak good English in church sponsored schools. But these schools were wiped out after Communist Party took power. Then Chinese people learned Russian for over ten years and no English was taught in most schools. After 1978, English was found to be so important as to so many people undertook the task of teaching English even though they used to teach Russian.

As a great nation, China has also produced many many English language experts, or professors or doctors with linguistic, literature or translation degree. They thrive in this country because there is the need to teach so many people. I suspect that they are quite happy with the status quo because more people will spend more time and money on English learning. Some of them just don't care about how to teach students to learn English, just teach them English: whether their approach is good or not.

I often tell my adult students that they are victims of this Chinese characteristic English language teaching. And in most cases, they agreed.

Personally I don't want to teach English any more. When there are so many tests that students must take and so many books or experts claiming that they can help them pass the tests, I don't think I can teach students anything.

Dr. Bland's Ten Commandments for teaching English

By Merton Bland

Introduction

A few years ago the author was assigned to the prime TESOL institution in southern Vietnam, the University of Social Sciences and Humanities. What he found was quite a challenge: an educational system, centrally administrated, mired in traditional practices. The grammar/translation methodology, a legacy of the French, held sway in the classrooms, producing, in spite of six years of English in the secondary schools, a nation of graduates unable to communicate in the target language. Discussions with Vietnamese colleagues, usually trained abroad, resulted in plans for a lecture tour to the major teacher training institutions of Vietnam (including Hanoi, DaNang, Hue, and about a half-dozen others) with a message stressing alternatives to the status quo, and the present format was developed. The author was subsequently invited to speak at institutions of ideological training (Communist Party Cadre) and information diffusion (schools for journalists) previously off-limits to Westerners, as well as the teacher training institutions previously noted.

The Commandments

(1) Do not teach English. Teach something, anything, IN English, using English as a vehicle of communication rather than an object of study. This is sometimes called the content-based curriculum.

(2) Do not teach grammar. Ingesting rules can be counterproductive: We are all familiar with students who are unable to apply rules learned through rote memorization. Instead, the grammar of English is best acquired inductively by the students formulating their own hypotheses. (This reflects Krashen's acquisition vs. learning.)

(3) Do not teach vocabulary. The schema, the concept pods which constitute the lexigraphical units of language, vary from language to language, even from person to person. No language is a direct translation of any other. Thus, vocabulary must be forged within the target language itself in a manner not unlike that of first language acquisition. To do otherwise is to risk forging the chains which prevent the bifurcation of the native and target languages and forever making your students translate in their heads word for word.

(4) Do not teach pronunciation. There is no longer any standard English. Well over two-thirds of the world's 1.5 billion English speakers are non-native speakers. Their English is certainly as acceptable as the Received Pronunciation (RP) of a tiny fraction of the British or the Broad Midwestern of Hollywood--as long as their English is comprehensible to the greatest number of persons who do not share that particular accent.

(5) Do not give tests. While testing is well embedded in many parts of the world, scaling is to be preferred to testing. Usually tests only require the regurgitation of knowledge. Scaling, placing people on a scale from beginner to educated native, has much more validity.

(6) Do not use lesson plans. Teach students, not lesson plans. Many teachers come away from their teacher training institutions with a mandated compulsion to spend hours writing lesson plans. Such planning is quite counterproductive since in an actual teaching situation the teacher must be alert to the reactions of the students--stressing pragmatic considerations, putting more time and effort where the lesson needs it and shortening or eliminating parts where the students seem to be in command of the concept being stressed. Yes, the teacher should have a general idea of the objectives of the lesson. Certainly the teacher should have available any materials which will be needed. Most importantly the teacher should leave time after the lesson to reflect on it and evaluate its strengths and weaknesses. But the focus of any teaching should be on the students, not on the constraints imposed by any preconceived lesson plan.

(7) Do not use the native language in the classroom: Never, never, never! If our aim is the successful bifurcation of the native and target languages, any use of the native language is by definition counterproductive. Draw a chalkline on the doorsill and proudly use the native language outside the classroom, but create an immersion situation inside.

(8) Do not use textbooks. You know your own students better than any textbook author. Authentic materials are all around you. For example: Record the news from the VOA or the BBC. Videotape CNN or Australian TV. Bring in any expatriate Anglophone in town and have him chat with the students. Have your school subscribe to the "International Herald Tribune" or "Time" or "Newsweek." Borrow English language videos. If they have subtitles put a book in front of the bottom of the monitor to cover up those subtitles. Buy, with your own money if necessary, paperbacks. After you read them they can be the nucleus of an individualized reading program (each student reads his own book and then reports on it to the class). Have your class keep journals in English, and write their own English to English vocabulary lists. Have the class write their own book.

(9) Do not teach the microskills: reading, writing, speaking, listening. English is one language, indivisible. And English is a living language; one only dissects the dead.

(10) Do not teach. Empower your students to take responsibility for their own learning. This reflects a general trend, especially in North American education, to deemphasize the role of the teacher as the font of all knowledge and provide the students with the means to further their own educative process beyond the classroom. This is called the student-centered classroom (as opposed to the teacher-centered classroom).

Thus Hath Dr. Bland Spoke

On the international plane, a focus on communication is overtaking traditional methodologies. This is reflected in most of the commandments. On the other hand, some of the commandments, i.e. Number 4, were reactions to local controversies. Number 4 was a response to the discussions as to which English represented the standard: British (RP), Midwest American, or even, in the Vietnamese context, Australian English. The answer was none of the above, but rather a Vietinglish comprehensible to the greatest number of non-Vietnamese.
Obviously, no immediate revolution was planned, nor did one occur. The aim, rather, was to present some alternatives and allow them to foment. Someday one of those teachers-in-training will become minister of education, and perhaps he or she will remember Dr. Bland's seminar and institute some of those reforms.

(Versions of this list have appeared in other publications, including the "WATESOL Journal.")

Merton L. Bland has worked as a K-12 teacher in the USA; a US foreign service officer in Africa, Asia and Australia; and as an ESL/EFL teacher and teacher trainer in the US, Asia, Africa, and Europe. <mert_bland@yahoo.com>

News: Electronics Giants Step Up English Interviews

Korean businesses are trusting TOEIC or TOEFL test scores less and realizing the importance of oral English interview more. Read about it here.

Speaking test, frame & rubric

By Jennifer Wallace - Anhui University of Technology, China

This exam was for first year/freshmen college diploma and university degree course students (all English majors). In other words this is the first oral exam they've ever had. This pretty simple test was successful in that the poorest were able to respond appropriately, albeit in one word responses, and asking me almost one word questions, while the higher level students you could barely shut up! I recorded the exams, and did the marking afterwards. I've given my marking scale at the end of all this.

This follows the format used for the speaking test in the first level of Cambridge (UCLES) EFL examination: Key English Test (KET). It uses a very fixed script: this means every student gets the same - no extra help, no forgetting things, etc.

Each student is allowed between 5 and 8 minutes.

Part 1

The examiner interviews the student: the student responds to questions, including one extended response. Functions covered: giving factual information about yourself, explaining and giving reasons, talking about past experiences

Part 2

The examiner gives a topic for the student to ask questions about (which uses verbal prompts). The examiner answers the questions, as a genuine interlocutor. Functions covered: requesting information (This may include information on: likes / dislikes, habits, factual information)

Part 1

What’s your full name?
How do you spell that?
What’s today’s date?
What time is it? (I HAVE A CLOCK IN THE ROOM)
Where are you from in Anhui?
Did you go to middle school there? (Where did you go to middle school?)
At school, what subjects (ONE OF:) did you like best, were you good at, were the most difficult, didn‘t you like?
Why?

Any 3 of these questions:
What do you usually do at the weekends?
What did you do last weekend?
What are you going to do this weekend?
What do you do in your free time?
Why do you like ...?
How often do you play / listen / watch ... ?
Have you been to any other towns / places in China?
(Which towns have you visited?)
What did you think of ....... ?
(Did you like ... ?) (What did you / didn’t you like?)

1 of these topics:
Tell me something about your family.
Tell me something about (name of home town).
Tell me something about your hobbies.

Part 2

I’m going to give you a card.
Ask me some questions about ....
Use the words on the card to help you.
Ask me 4 questions.
I’ll answer your questions.
Do you understand - shall I repeat that?

FAVOURITE FOOD
What? How often? When? Expensive?

SPARE TIME
What? How often? When? Expensive?

MUSIC
Like? What type? Favourite? When ... listen?

FAVOURITE SPORT
What? Play? Watch? When?

That’s the end of your examination. - Thank you very much.

ASSESSMENT SCALE

Grade A (100 - 90)
Deals effectively with all tasks. Able to communicate effectively. Interacts in an appropriate manner throughout. Meaning is conveyed despite limitations in linguistic resources and there is an adequate range of vocabulary and paraphrase strategies to deliver the intended message. Speech can generally be understood with ease. There are hesitations and attempts at long turns may lack coherence, but not such as to strain the listener.

Grade B (89 - 80)
Some of the features of A and some of the features of C.

Grade C (79 - 70)
Adequate achievement of tasks. Able to communicate appropriately most, but not all, of the time. Generally interacts in an appropriate manner. Breakdowns occur in communication resulting from inadequacies in the use of linguistic resources and paraphrase strategies. Speech is sometimes difficult to understand. Long hesitations sometimes demand undue patience of the listener. Utterances are often halting and disconnected with turns sometimes left undeveloped.

Grade D (69 - 60)
Some of the features of C and some of the features of E.

Grade E (59 and below)
Non-achievement of most tasks. Generally unable to communicate appropriately. Often interacts in an inappropriate manner. Gross inadequacies in the use of linguistic resources means message is often destroyed. Speech is often unintelligible. Utterances are often abandoned.