Reflection

Refection unit 4

We learned that preparing tests is something that we have to handle with extra care. We should make sure that our tests have all the basic abilities such as listening, writing, reading, and speaking. We also learned about the importance of being coherent with what we teach and what we test (if we have taught speaking and listening skills in class, our test should focus on those skills). When we read the suggestions by Harris on how to prepare tests, we understood that those techniques could help us to check tests easily and to focus on the aspect that we intend to evaluate. It is important to be aware of the drawbacks and the positive sides of the testing techniques in order to use them in a planned and conscious way. We think that this unit is one of the most important ones because it will help us to be aware of what we should test and how.

Four Items

Four items

Writing

Instructions:  write a short text about what you are going to do in the next celebration.

Ex: For Christmas I am going to bake a cake and …….

______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

(We know that this item is not very objective, but we think that free writing is very important.)

Reading

Instructions: read the following text and write the meaning of the underlined words   (definition, synonym).

·         People pull on your earlobe, one pull for each year. (Maria, Brazil)

·         Your nose gets greased with butter for good luck. (Tom, Canada)

·         We have noodles for lunch and every one wishes you a long life. (Ping, China)

·         You are blindfolded and have to break a piñata. Everyone shares the treats. (Jose, Mexico)

·         You get a birthday pie not a cake. (Vladimir, Russia)

·         Little ones sit on a chair and grown-ups lift them up, once for each year and one more time for good luck. (Amy, Israel)

·         You wear a colorful dress and pass out chocolates at school. (Deepa, India)

·         Birthday cakes candles are lit at sunrise and burn all day. At dinner, you blow out the candles and make a wish. If you blow out the candles in one try, your wish comes true. (Katrina, Germany)

_________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

Listening

Instructions: listen to the dialog and answer some questions. (Try to write in complete sentences even though it will not be evaluated)

A: Hi, how are you going to celebrate your birthday?

B: My parents are going to bring me a cake and some presents. And you, how are you going to celebrate your birthday?

A: I’ m going to visit Disneyland.

B: What are you going to do there?

A: Well, I’m going to spend two days in a beautiful hotel, I’m going to dance with Cinderella, and I’m going to take a photograph of Mickey Mouse.

B: And, what are you going to do for Christmas?

A: Well, I’m going to visit my grandparents.

B: Are you going to give them presents?

A: No, I won’t because I don’t have any money.

Questions:

1.  When is student A.  going to photograph Mickey Mouse?

_____________________________________________________________________________

2. What is student A.  going to do for Christmas?

_____________________________________________________________________________

3. What is student B.  going to eat on her birthday?

_____________________________________________________________________________

4. With who is student A.  going to dance in Disneyland?

_____________________________________________________________________________

Speaking

Instructions: speak about your favorite holiday for 3 minutes.

These questions may help you prepare your little talk:

On what date do you celebrate the holiday?

What do you eat?

What do you wear?

What kind of holiday is it? (Religious holiday, a national holiday, etc.)

Why is it your favorite holiday?

Activity 1

Reading:

I liked both identifying the topic (match the title with the text) and linking (what does the underlined word refer to?) because both of them are good ways of checking whether students understood the reading.

Writing:

I would use guided writing, summary and combined (read the letter and write a reply) because I think that they are checking writing skills students will need in the real world.

Speaking:

I think it is very difficult to assess speaking. I guess I would use free interviews.

Listening:

I liked the text completion because it is a “real world problem” where the students are listening for specific information.  I also liked the sequencing of texts/pictures after having listened to a story. I just think that the story should be funny and not too difficult for the students.

Language:

In this section I found many interesting options for specific language skills, but the ones I especially  liked were word sequencing,  gap-fill (selected words in a text are blanked out), cued sentences, and personalization.  I think that gap-fill and word sequencing are really good for testing structures while cued sentences and personalization are good for testing the writing of beginners. 

Melanie L. 

For reading and listening I liked the activity of Text completion eg. “Listen and complete the information about the film” because it about a topic that almost every body will enjoy and for me I think it will be fun even though it is for an exam.

For the writing I liked the test format “Guided writing” using pictures, notes, diagrams because the teacher is giving the students the tools to make the exercise (pictures, notes, diagrams, etc.) so it can be easier.

In the speaking past I liked the “Role play” because it is fun and they are speaking; besides the students will not be that afraid to participate.

In language I liked the “Gap-fill” activity because it is good for testing the grammatical structures.

Sarahi A. 

We have to consider before testing our students, their learning style and abilities. It is also important that our tests contain the contents of the units seen in class and make sure if the exercises in the test are related with the skills developed during the class; besides, we must consider the age and level of the students because if they are elementary students, they are not totally responsable of their learning development, so surely their parents will ask the teacher about the result of the test. In the case of university students, they will check their test results and will be aware of their self knowledge obtained during the course.

The tests that I would like to use in my class are: Proficiency Test and Progress Test. Also I would like to consider the suggestions given by Harries and McCann when we have to do a test:

a)    Balance formal assessment test with informal.

b)    Consider the assessment task program.

c)    Assessment Weighting.

d)    Syllabus Objectives

I would like to use the next techniques to assess my students:

1)    In writing,  I’d use the “guided writing” because Ss’ have less mistakes and it is easier for the teacher to evaluate students due to the guide given works as limit and everybody “understands” how to do the test and in indirect manner how is going to be evaluated the writing.

2)    In the case of speaking, I want to mention that speaking has been a difficult factor to evaluate, even using and preparing a lesson plan to be used during all the semester, the time limits  the use of the type of test. So I will use free interviews, but before doing this test I will explain the students what aspects of conversation I will assess and I will share them new vocabulary. I have used Role play, because Ss’ get fun, improve their speaking, get relax and they usually forget that are being evaluated. Days before the play I give them a list of possible words to use during the play. The problem is to evaluate four or five student at the same time.

 3)    In Listening, I would like to work with Text Completion because Ss’ would put in practice the new vocabulary and structures given before, beside they had to read and understand the context to write the appropriate words to fill in the blanks.

4)    In reading, I’d like to use word attack because the Ss’ must infer the meaning of new words and this  test might help them to build more vocabulary, also this technique obligates students to read the context in order to understand if their new words are correct or not.

Alfonso L.

Reflection Unit 3

We found this unit very interesting because it really opened our eyes to some very important aspects of testing, like the importance of practicality, validity, and reliability of the tests that we use to evaluate students.

 

After reading the two texts about tests, we noticed that we cannot believe or trust in all institutions’ exams. We analyzed our own tests, and we had to recognize that they were not too bad, but that they could be better if we had known a little more about the differences and content of English testing. Now we are more aware of the different types of testing; for example, formal and informal testing, and the importance of knowing when to use each one.

 

Even though this unit was quite theoretical, we liked it and think that it contained essential and interesting information. This knowledge will help us a lot to create better tests for our students.  

“Five principles of testing” “Reliability in testing” and “Validity in testing”.

Validity

Construct validity: General notion of validity, but without empirical evidence.

Content validity: A test has content validity when its content tests all the language skills and/or structures it should be testing.

Criterion-related validity: A test has criterion-related validity if the results on the test agree with those provided by some independent and highly dependable assessment of the candidate’s ability.

Concurrent validity: A test has concurrent validity if the test and the criterion are administered at about the same time

Predicative validity: A test has predicative related validity if it can predict the future performance of candidates.

Face validity: A test has face validity if it looks as if it measures what it is supposed to measure.

Reliability

Reliability coefficient: The reliability coefficient is the quantification of the reliability of a test; it allows us to compare the reliability of different tests.

Scorer reliability: A test has scorer reliability when it is possible to quantify the level of agreement given by same or different scorers on different occasions by means of scorer reliability coefficient.

Backwash is the effect that tests have on learning and teaching.

Practicality: A test is practical if it is easy, quick and cheap to administer, score, and interpret.

Melanie

When a test have content validity its content constitutes a representative sample of the language skills,   proper relevant structures, etc. with which it is mean to be concerned. If we want to know if a test has content validity, we need a specification of skills structures that it is mean to cover.

There are some components of test validity, first we have to consider, the grater a test’s content validity, so this is the more likely it is to be an accurate measure of what it is supposed to measure. And we have also to consider if the items in the test are representative of all structures and skills in question.

There are two kinds of criterion related validity. First, Concurrent validity: It is established when the test and the criterion are administered at about the same time; besides, it might revel a strong relationship between students’ performance on the test and supervisors’ assessment of the reading ability. Second, predictive validity: It concerns the degree to which a test can predict candidates’ future performance. We have to consider that is difficult to predict when there are so many factors such as intelligence knowledge, motivation, health etc.

One way to obtain evidence about the construct validity of our test is to investigate what test takers actually do when they respond to an item, so we can use two methods to gather this information: think aloud and retrospection. In the think aloud, test takers voice their thoughts as they respond to the item. In retrospection, they try to recollect what their thinking was as they respond.

Validity in scoring

If we want that our tests have validity we have to consider the manner that it has been done for example in a reading test may call for short written responses, if these responses take into account spelling and grammar, then it is not valid. By measuring more than one ability, it makes the measurement less accurate.

Face validity

A test to face validity has to measures what is supposed to measure. If a test has to measure pronunciation but it doesn’t need that the student speaks then this exam might be thought to lack face validity. So to make tests more valid we have to write explicit specification for the test and whenever feasible, use direct testing; besides, make sure that scoring of responses relates directly to what is being taste and finally, do everything possible to make the test reliable.

Reliability

To have reliability in a test we should consider two important components the performance of the candidates from occasion to occasion, and the reliability of scoring. Firstly we will consider the reliability coefficient, it is like validity coefficients because thy allow us to compare the reliability of different test. The ideal reliability coefficient test is 1.The test with a reliability coefficient of 1 is one which would give precisely the same results for particular set candidates regardless of when it happened to be administered.

Scorer reliability

It is possible to quantify the level of agreement given by the same or different scorers on different occasions by mean of scorer reliability coefficient. In the case of the multiple choice test, the scorer reliability coefficient would be 1. When scoring requires no judgment, and could in principle or in practice be carried out by computer, the test is said be objective. We must consider that if the scoring of a test is not reliable, then the test cannot be reliable either, so there is a relationship between test reliability and scorer reliability.

If we want to make test more reliable, we must consider the performance from candidates then these suggestions could help us to score reliability: Exclude items which do not discriminate well between weaker and strong students, do not allow candidates too much freedom, write unambiguous items, provide clear and explicit instructions, use items that permit scoring which is as objective as possible, Identify candidates by numbers, not name, etc.

Backwash

Backwash is the effects that test have on learning and teaching; back wash is seen as a part of the impact a rest may have on learners and teachers, on educational systems in general, and on society at large. We have to ensure the test is known and understood by students and teachers because however good the potential backwash effect of a test may be, the effect will not be fully realized if students and those responsible for teaching don not know and understand what the test demands of them. It is important to give students specific comments (good and bad) and feedback during and after the test.

Counting the cost

One important and desirable quality of test which trips quiet readily off the tongue of many testers, after validity and reliability, is that of practicality. What will be the cost of not achieving beneficial backwash? The production and distribution of sample test and teachers training will also be costly. Some experts will argue, therefore that such procedures are impractical. When we compare costs, efforts and avoid the waste of time in activities that are not going to give us beneficial backwash effects. So we have to continue supplying information to increase test reliability in order to be more practical and more benefits.

Alfonso

Validity: for me validity is when the teacher puts in an exam all the knowledge that the students are supposed to know.

Reliability: is when he teacher makes an exam that is easy for students to understand. Also if a teacher applies that exam two times at the same person in two different occasions the student will have to had the almost the same result as the first time.

Practicality: is when the teacher makes an exam that is not to easy that the students will answer in 5 to 10 min or to hard that it will take 10 hours. It is supposed to be an exam according to the needs of the students.

Achieving beneficial backwash

It has some principals, one is that teachers should make oral tests but usually they don’t, it also says that the teachers should not focus only in the test because there are a lot of other material to assess; we as teachers need to focus in direct testing; teachers need to be careful with the activities they put in the test, they need to make sure that the activities are right for that level; we also need to make tests with and objective, for example, what they have achieved in a certain period of time; we also need to make sure that the test is understandable, that the students understand what they are going to do;  for me the most important is that teachers should need to be making different tests as the years past, I hate when the teachers give the same tests to students year thru year.

Sarahi

Analysis of a test

1.    What is the purpose of the test?

a.    The purpose of this test is to know the English level of the students.

2.    Does it represent direct or indirect testing (or a mixture of both)?

a.    They represent both direct and indirect testing, direct because it provides the information about the students’ abilities that we want to assess and it also has authentic conversations in the listening area. Indirect because it has a section where the student selects the appropriate answers that are underlined and we want to measure and that are used in informal and formal Standard English.

3.    Are the items discrete point or integrative (or mixture of both)?

a.    Most of them are discrete because it measures one element at a time. It also has some small part of integrative.

4.    Which items are objectives, and which are subjective? Can you order the subjective items according to degree of subjectivity?

a.    It is objective because has multiple choice.

5.    Is the text norm-referenced or criterion-referenced?

a.    Norm-reference

6.    Does the test measure communicative abilities? Would you describe it as a communicative test? Justify your answer

a.    Yes, because in the first part there are 9 conversations that you heard and read in order to answer the questions. And it also has a reading part where students have to read and answer question, and a part that is called the Language use section where students answer a 34 multiple choice questions choosing the correct one according to their own experiences in every day’s language

7.    What relationship is there between the answers to question 6 and the answers to the other questions?

a.    In all the other questions, as well in the number six, what they tried to show us is a manner to measure the ability that a student has to communicate in his/her second language; probably some test are better for some institutions or the tests were created for a specific purpose, but basically the main objective of everything was to test the communicative ability of the individuals.

Types of testing

Tests can be categorized according the information that they provide. Tests give us the information that we need. There are four types of tests that we are going to mention they are: Proficiency tests, achievement tests, diagnostic tests, and placement tests.

Proficiency Tests

This test measures people’s ability in language and any kind of training that they might have in the language that they have learned, but  proficiency tests are not based on the content, they are based on a specification of what candidates are able to  do in the language to be considered “proficient”. An example of a proficiency test is the Cambridge First Certificate in English examination (CPE).

Achievement test

These tests are related to language courses; we use these tests to establish how successful students or groups of individuals or the courses have been in achieving objectives. There are two kind of achievement test: 1) final achievement test and 2) progress achievement test.

Final Achievement test is related to the content of the course with which it is concerned; also they are written by members of teaching institutions, ministries of education, etc. These tests are administered at the end of the courses. Some testers think that final achievement tests should be based on the books or material used during the course, but the problem would be if that material or the book were not chosen well or if the syllabus is badly designed.

One alternative to avoid this problem is to base the test content directly on the objective of the course.

Progress Achievement test. This test intends to measure what progress students are making.  We must remember that “progress” is towards the achievement of course objectives, so these tests should relate to objectives. To avoid any kind of problems because students could feel frustrated because their bad grades during these type of tests, teachers should establish well-defined short-term objectives and show the students a clear progressions towards the final test. An example of this test might be “pop quizzes” because such tests will not form part of formal assessment or any other test made by the teacher that leads students to the final objective.

Diagnostic test. This tests is used to identify learners weaknesses and strengths; the problem is that we, as teacher can create our diagnostic test with the objective of measuring a student’s learning process, but unfortunately, there are no good diagnostic test. It would be better to use a proficiency test if we really want to see the weak or strong areas of our students. As it was mentioned because the lack of good diagnostic tests to measure any specific area of learners there are possible tests that might help us that has been developed trough the computer system.  This interesting web- based development is called DIALANG, and was created to make diagnostic tests in many countries of Europe.

Direct versus indirect testing

Direct Testing. We use it when the candidate is required to perform precisely the skill that we wish to measure. If we need that our candidate to speak, write or do any other activity, we have to use the most authentic material or elements possible. The acts of speaking or writing provide us the necessary information about the candidate abilities, but also the candidate has to do the activities successfully. Although we can make our own direct test in a very valid and high reliable way, we still don’t have tests that accurately measure the abilities of candidates.

Indirect testing. It tries to measure the abilities that underlie the skills in which we are interested. An example of this exam is when you answer a test of pronunciation ability identifying pairs of words which rhyme with each other.

It is possible that the main appeal of indirect testing is that it offers the possibility of measuring a finite number of abilities instead of the large number of manifestation of them. An example of this test is the TOEFL because you might be reading a sentence and the objective is to find the wrong word, so automatically you are being evaluated in writing.

There exists another type of test which is called Semi-direct. This test although it is not direct stimulates direct testing. An example is the TOIC test, when somebody asks you a question and you answer while your voice is recorded.

Discrete point versus integrative testing

Discrete point testing, It refers to a te sting of one element at a time, item by item, example testing grammar we focus on one element after other and other. Discrete point test are always indirect.

Integrative testing: The candidate has to combine many languages elements in the task, example you are listening a play recorder taking notes. Integrative testing will tend to be direct.

Norm- reference versus criterion- referenced testing

Norm reference: It gives information about a candidate performance to the of other candidates, but we are not told what directly what the candidate is capable of doing in the language; for example, in your evaluation you are into the 10% who did the better than 100% who took it. Exam with this characteristic is the Interagency Language Roundtable (ILR) or the Michigan Test of English Language Proficiency.

Criterion- referenced testing: The purpose of this test is to classify people according whether or not they are able to perform some task or set of task satisfactorily. This kind of test sets a meaningful standard for students.  Example: Berkshire German certificate. 

Objective testing versus subjective testing

Basically the difference between them is between methods of scoring. If we don’t have to judge one part of the scorer, then the scoring is objective. When there is a multiple choice test with correct responses unambiguously identify, that would be a case in point. If judgment is called for the scoring is said to be subjective.

Computer adaptive testing

This system of testing is an easy way to predict people performance of low ability on difficult topics simply by seeing their consistently incorrect responses in easy items. There is no need that strong candidates to attempt easy items the computer will predict everybody’s performance. We have to remember that computer adaptive testing offers a potentially more efficient way of collecting information on people’s abilities. This exam is done in the computer.

Communicative language testing

This kind of test wants to assess the communicative ability of the student, including reading and listening.