DORA Studies, Subtest Specifications, and Correlations
Accuracy and Refinement
Assessment, by its very nature, attempts to measure a skill with test items that sample across a given domain. Common sense dictates that the shorter the test, the less accurate it is. For this reason, the Diagnostic Online Reading Assessment (DORA) sees itself as in a class separate from the 10-15 minute reading assessments offered by multiple other educational publishers. Our assessment takes approximately 45 to 60 minutes and adapts to each test-taker as he or she undertakes an assessment. The detailed report that we produce for each student is far more accurate and diagnostic than what a "short" test can offer. Of course, assessment improvement is an ongoing process. By performing ongoing item analysis with specific pools of students, we are continuing to improve our assessment.Test-Retest Study: Q3 2003
Test-Retest is the ability of a test to be taken once and then immediately again and have similar results. Let’s Go Learn undertook a second test-retest study in Q3 of 2003.This data was added to an earlier study performed in Q1 of 2003 in order to provide a larger sampling and to reduce the margin of error. The combined results were once again excellent. Variability was low, meaning that the LGL Reading Assessment is very precise and can be re-administered with low bias. Sample size: n=225
Grade level deltas are calculated by first taking the absolute value of the change (or delta) between each test and retest subtest score. Next the mean is calulated from these deltas. All scores are in grade levels, so a grade level delta of 0.5 means one half of a grade level. SE is standard error for mean delta in grade levels.
- High-Frequency Words - Grade level delta: 0.39 SE=0.11
- Word Recognition - Grade level delta: 0.19 SE=0.12
- Word Analysis - Grade level delta : 0.16 SE=0.07
- Word Meaning - Grade level delta: 0.60 SE=0.19
- Spelling - Grade level delta: 0.27 SE=0.10
- Silent Reading - Grade level delta: 0.35 SE=0.13
Subtest Specifications
High-Frequency Words: 72 criterion-referenced words. 24 words per grade from 1st to 3rd grade. Word Recognition: 120 criterion-referenced words. 10 words per grade from 1st to 12th grade. Phonics (Word Analysis): 80 criterion-referenced words. 20 words per grade from 1st to 4th grade. Oral Vocabulary (Word Meaning): 60 criterion-referenced words. 5 words per grade from 1st to 12th grade. Spelling: 60 criterion-referenced words. 5 words per grade from 1st to 12th grade. Reading Comprehension (Silent Reading): 12 Flesch-Kincaid leveled passages with 6 questions per passage. 1 passage per grade (with three sets of comprehension passages used, students cycle through A, B, and C passages on subsequent assessments).Reading Level Calculation Adherence
In the silent reading subtest of the LGL Reading Assessment, the following method for reading level calculation was chosen. The sixth subtest, silent reading, is made up of 12 reading passages for grades 1 through 12. These passages were systematically constructed to adhere to the Flesch-Kincaid Reading Grade Level Index. This index calculates the number of words, syllables, and sentences in a given passage. It then utilizes the average syllables per word and words per sentence to articulate a readability formula. After researching the methods used by textbook publishers and children’s book publishers, our research found this to be the most reliable and widely accepted system for leveling reading material.Let’s Go Learn Correlates Significantly to the Nationally Recognized CAL Reads Program
In its FIRST comparison to one-on-one paper-and-pencil assessments performed by CAL Reads reading specialists, Let’s Go Learn achieved high correlations with statistical significance beyond the a= .01 level. CAL Reads reading specialists used the following assessments in their one-on-one assessments with the students in the study:
Sight-word familiarity: Fry’s high-frequency word list Word recognition: Diagnostic Assessments of Reading (DAR) published by Riverside Publishing Word meaning: Diagnostic Assessments of Reading (DAR) published by Riverside Publishing Spelling: Diagnostic Assessments of Reading (DAR) published by Riverside Publishing Silent Reading: Qualitative Reading Inventory (QRI) published by Pearson, Allyn & Bacon.
- Sight-word familiarity r= .89 (n=17)
- Word recognition r= .81 (n=20)
- Word meaning r= .60* (n=20)
- Spelling r= .78 (n=20)
- Silent reading r= .89 (n=19)
This study was conducted in the Tahoe/Truckee Unified School District in California in February 2002. Students were tested both by CAL Reads reading specialists and online using Let’s Go Learn within a three-week time period. The Word Analysis subtests were not compared because of incompatible methods with which CAL Reads and Let’s Go Learn reported their final results. See our paper, SBIR U.S. Department of Education Pilot Results for more information surrounding this pilot. *Lower correlation results in Word Meanings were traced to two confusing items in this initial study. Shortly after this initial study, the Word Meaning component of DORA was modified to improve its correlation.
Item Analysis Major Revision: Q1, 2003
Item analysis was performed across all LGL assessment systems.
A pool of 1000 students was used. Items across all six sub-tests with more than 75% of students answering correctly or fewer than 25% answering correctly were flagged. These extreme values represent errors that are outside the range for which the items were designed.
Example 1: "and" received an unusually high error rate. The word "an" was a distracter that was selected with a high percentage. Conclusion: The audios of "and" and "an" are too similar. Students might not be hearing the /d/ sound and thus may think the target word is "an."
Example 2: Target vocabulary word: "Caravan." Too often students chose a picture of one car and one van. Overall the error rate was too high for this particular word. Conclusion: Many students define "caravan" as the Dodge Caravan vehicle and not as a line of camels walking through the desert.
LGL Reading Assessment Comparison to Nationally-Normed Paper-and-Pencil Assessments: April 2003
Tested students (Grade range: 2-6) within one week’s time on DORA and the following paper-and-pencil tests.
LGL HFW subtest and the Slosson Oral Reading Test
LGL WR subtest and the Woodcock Word Identification Test
LGL WA subtest and the Woodcock Word Attack
Correlation (HFW & SORT): r=0.95 SE=0.073 (n=21)
Correlation (WR & WI-W): r=0.92 SE=0.088 (n=21)
Correlation (WA & W-WA): r=0.91 SE=0.097 (n=21)
High correlation demonstrates criterion validity of the LGL Reading Assessment.
Recalculation of April SP and SR results + New Tests October 2003
LGL SP subtest and the WRAT LGL SR subtest and the Gray Oral Reading Test Correlation (SP &WRAT): r=0.85 SE=0.210 (n=21) Correlation (SR &GORT):* r=0.65 SE=0.250 (n=21) Medium to high correlations demonstrate concurrent validity of the LGL Reading Assessment. *Lower correlations to the GORT are attributed to a high variability observed in the GORT results. Students inconsistently tested well above their grade levels on the GORT. Subsequent SR comparisons with a more consistent paper-and-pencil assessments is recommended.Research Awards
In September 2001, the U.S. Department of Education awarded Let’s Go Learn a prestigious grant reserved for private companies. This SBIR grant sought to support innovative new technologies that bring with them the ability to realize scalable solutions in education. The U.S. Department of Education recognized Let’s Go Learn’s unique team of experts and product plan and gave Let’s Go Learn extremely high marks across the entire spectrum of review criteria.