Calibration Instructions

Reasons to Calibrate

As we prepare to rate the Language Acquisition Tests (LATs), we need to ensure there is intra-rater reliability and inter-rater reliability.

• Intra-Rater Reliability: Being consistent with yourself. 

  • Do not change the way you rate in the middle of rating.
  • Follow the rubric for every student you rate.
  • Do not rate based on the level and students you are teaching this semester.
  • Take breaks so you don’t get fatigued.
  • Rate in a place with limited distractions.

• Inter-Rater Reliability: Being consistent with the other raters.

  • Follow the rubric for every student you rate. If everyone follows the rubric diligently, the ratings will be more reliable.

Biases and Errors

The biases and errors will be discussed in detail during the calibration meetings that you must attend after you finish the calibration rating. For the calibration rating, focus on following the rubrics the best you can.

Using the Rubrics

The rubrics are tables that include columns for Text Type, Content, and Accuracy and rows for proficiency levels ranging from 0 (Novice Low-Novice Mid) to 7 (Advanced High). The level descriptions in the rubric are designed to be read top to bottom and left to right. You will notice that the rubrics are divided into groups.

ELC students do not typically reach the Superior level while attending the ELC, and our curriculum is not designed to reach that level. Because of this, the Superior level is not included in the rubric, but the general description is included because the Advanced Mid level is reaching for it and the Advanced High level is mostly there but not consistent.

To use the rubric when rating:

• identify the student’s proficiency level (Novice, Intermediate, or Advanced) based on the general descriptions.

• narrow down to a specific sublevel (Low, Mid, or High) within the proficiency level.

  • Move from left to right in the rubric. Start with Text Type, then move to Content, and then Accuracy.
• find the best overall level with emphasis placed on Text Type, then Content, and then Accuracy.
• go with the lower of two levels if you are debating between two.

Holistic Rating

When rating students’ LATs, we rate holistically. Because each prompt is designed to elicit responses at different proficiency levels, the rubric does not work for rating each prompt individually. For example, the Advanced section of the rubric cannot be used to rate a Novice level prompt. If students have 20 seconds to describe the room they are in, we would not expect an advanced response with highly organized paragraph-like speech with discourse markers. Instead, we would expect words, phrases, or sentences for that response. However, that one prompt does not show the range of their abilities, so we need to listen to or read multiple responses to determine their proficiency holistically.

• Speaking LAT: 12 prompts ranging from Novice to Superior.

  • You do not have to listen to all of their responses, but you do need to listen to enough of them to give an accurate rating.

    • You do not need to listen to the responses in order of the prompts. It is usually more time-consuming and more time-wasting if you listen to all of the responses in order.
    • The prompts are labeled according to the proficiency level they are designed to elicit. Start with an Intermediate prompt, see how they do, and then move to lower, the same, or higher proficiency prompts based on how they do. If they continue to do well, go to higher proficiency prompts. When their language starts to break down, go to lower proficiency prompts. Continue doing that until you determine the proficiency level where they are showing their best English.

• Writing LAT: 2 prompts with one being Intermediate and the other one Advanced/Superior.

  • There are two approaches you can take.

    • Start with the Advanced/Superior prompt (30-minute response).

      • If you would rate it at a 3 (Intermediate Mid) or lower, look at the Intermediate prompt for a more accurate sample of level-appropriate writing.

    • Start with the Intermediate prompt (10-minute response).

      • If there is clear evidence of a well-written paragraph, move on to the Advanced/Superior prompt for a more accurate sample of level-appropriate writing.

Rating Website

Calibration ratings must be finished by the last day of class so that the Coordinator of Assessment and Technology can analyze the data and prepare the materials for the calibration meetings. To rate, follow the steps below.

• Go to https://elctesting.byu.edu and log in.

  • Your username is your BYU ID number, and your password is your BYU NetID (unless you changed it to something else in the past).

    • If you do not remember your password, click on Forgot your password to reset it. You will get an email with instructions to reset it. If you do not get an email, contact the Coordinator of Assessment and Technology.

• Click on Ratings and it will take you to a new page listing your rating events. 

  • Choose the calibration rating event for the current semester. It will be at the top of the list and will show the number of ratings you have remaining.
  • After you click on the rating event, it will take you to a page listing your assigned speaking and/or writing ratings. Click on the skill area you are rating.
• On the rating page, you will see the different prompts and responses on the left and the rating options on the right. When you click on the rating options, you will see the descriptions of that level from the rubric. Those descriptions are helpful, but it is beneficial to have the whole rubric open while you are rating. It can be a little easier to narrow it down when you can see all of the level descriptions at once. Determine the appropriate proficiency level, click on it, and submit it. The page will automatically move you through all of the students you need to rate.
• When you need to take a break or if you need to finish later, exit or close the rating event so that it pauses your time spent rating.

Calibration Meetings

The calibration meetings are held on the first day of LATs. There is one meeting for speaking and one for writing. They usually occur at the same time, so if you are rating both skill areas, attend the meeting of the skill area you are least confident in rating. Then, talk with the supervisor that led the other meeting so you can get the information you need.

During the meeting, you will see your calibration rating results, see how all raters compare to one another, learn of possible biases, errors, or tendencies you may have, and go through at least one rating sample together for discussion. These meetings are mandatory. If you are unable to attend the meeting at the scheduled time because of a class you have on campus or because of an emergency, contact the Coordinator of Assessment and Technology and the supervisor leading the calibration meeting. You will have to watch the recording of the meeting and then meet with the supervisor that led the meeting before you can start rating LATs.

Rating LATs

You can begin rating LATs as soon as the calibration meetings end. Half of the students take the Speaking LAT the first day and the Writing LAT the second day while the other half of the students do the opposite, so there are students taking the speaking and writing test both days. Because of this, you will be assigned half of your ratings on the first day of LATs and the second half of your ratings on the second day. You can either work on your ratings both days, or you can do all of them on the second day. I recommend dividing it between the two days so that you do not get fatigued rating them all the second day. I also recommend rating during the time you would normally be teaching if you are having a hard time finding the time to rate. You have had that time blocked off the whole semester; it can be blocked off for rating on those days too.

LATs ratings must be completed by midnight of the last day of LATs. They need to be all done by then so that any triple ratings can be done the morning of the day after LATs.