In a recent Brief produced through the Quality Initiatives Research and Evaluation Consortium – INQUIRE – Zellman and Fiene (2012) provide a framework to guide QRIS validation and examples of the activities that could be conducted as part of validation efforts. The current Brief serves as a companion to the 2012 INQUIRE Brief by providing detailed examples and findings from the validation activities in four states: Indiana, Maine, Minnesota and Virginia. The purpose of this Brief is to demonstrate how different states have approached QRIS validation, to compare findings, and to highlight challenges in designing and conducting QRIS validation studies.
The picture that emerges from the synthesis of findings across the four states and across the validation approaches is mixed. For instance, the results of efforts to validate the quality standards and indicators in QRIS generally have been successful. Efforts to review how well measures are functioning, however, reveal concerns about limited variation on some measures and QRIS structures that are producing skewed distribution of programs across the rating levels. There are some indications that QRIS levels are distinct with respect to measures of observed quality, but only in the QRIS that used the observational measures as part of the rating process. Finally, validation studies that included measures of children’s developmental progress indicate limited support for linkages between these measures of children’s growth, QRIS ratings and program quality elements. The findings suggest that further work is needed to strengthen the ability of QRIS ratings to serve as meaningful markers of program quality. A key theme discussed in the brief is that the information gained from validation efforts can serve as a critical tool for guiding initial design of QRIS, redesign efforts and continuous quality improvement. Zellman and Fiene (2012) emphasize that validation studies do not produce “yes” or “no” answers about QRIS but provide data that can support QRIS in a process of refining and improving. As such, validation efforts must be timed appropriately and aligned with a clear decision-making framework for how the findings will be used. In the four states highlighted in this Brief, researchers partnered with state agency leaders and other QRIS stakeholders to assist in developing a validation plan that could support QRIS development as well as a process for reviewing and interpreting findings so that the results could be applied appropriately. As states continue implementation of QRIS, administrators and stakeholders are encouraged to engage in validation efforts that can inform their systems and move progressively toward the provision of effective services. (author abstract)