Back
to Center for Effective Collaboration and Practice Home
Page.Volume II: Using Evaluations Data to Manage, Improve, Market, and Sustain Children's Services
|
|
INTRODUCTION | |
| Providers and evaluators of
mental health services to children and their families are often challenged by the task of
translating evaluation findings into clear and meaningful reports that can illustrate the
resources, gaps, expenditures, and outcomes of their programs. The broad range of data
typically collected by providers of public services includes demographic descriptions of
clients, service utilization, system costs, consumer satisfaction, and behavioral and
emotional indicators. Effective analysis, interpretation, and presentation of these data
elements require a blend of science, art, technology, and communication skills. Once
produced and disseminated, however, evaluation reports can be powerful tools for improving
service delivery, marshalling public support, validating managerial decisions, and
sustaining emotional and financial involvement in the service systems. Effective evaluation data reports can be powerful tools for improving and sustaining interagency service delivery systems for children and families. The purpose of this monograph is to describe promising practices in the use of evaluation data at sites funded by the federal Center for Mental Health Services as part of the Comprehensive Community Mental Health Services for Children and Their Families Program. The sites showcased in this monograph have been developing and implementing their evaluation programs for at least five years as a requirement of their federal funding. These selected systems of care have been deemed some of the most successful in going beyond their funding obligations to become true data-driven systems committed to: (a) gauging the effectiveness of their local services through strategic data analysis; (b) instilling timely and consistent evaluation feedback mechanisms into their practices; and (c) responding to evaluation findings with data-based decision making and system improvements. It is the intent of this monograph to share a wealth of ideas and experiences from these sites about using local data in ways that can impact the delivery, management, and sustainability of community-based services for children and families. Selected promising practice sites for evaluation reporting include: Community Wraparound Initiative, Illinois; Families First/Access, Vermont; KanFocus, Parsons, Kansas; Multiagency Integrated System of Care (MISC), Santa Barbara, California; Stark County Family Council, Ohio; Texas Department of Mental Health and Mental Retardation Children’s Services (DMHMR)*; Wings for Children and Families, Maine *Note: Texas DMHMR is not a federally-funded system of care site
|
||
| top | Study Design | |
| The research questions guiding much of the
work for this monograph focused on the experiences of multiple stakeholders with reporting
and utilizing evaluation data. The authors’ intent was threefold: (a) to describe how
a supportive evaluation "culture" was garnered in each selected site to
reinforce and sustain data utilization; (b) to illustrate major "take-away
messages" and describe how these messages were developed in the sampling of
evaluation products; and (c) to capture specific examples where data were critical to
decision-making processes and/or brought about program and policy changes. This monograph
is not a "how to" guide for developing evaluation programs; rather it provides
examples of effective uses of evaluation data when they are collected with valid and
reliable measures. It illustrates how selected sites have developed supportive evaluation
processes to generate effective data reporting products that have impact at local, state,
and national levels. Interviews were scheduled with multiple stakeholders of those sites specifically nominated by local and national experts as outstanding examples of data-driven systems. The interviews were semi-structured and based on the research questions described above, providing detailed descriptions of the development and utilization of evaluation reports. In all, 19 interviews were conducted by phone or in person with family members, site directors, evaluators, and service providers in seven nominated sites for inclusion in the monograph. In addition, the authors reviewed data reporting products developed at each site, including community report cards, descriptive outcome reports, newsletters, and conference presentations.
|
||
| top | Findings | |
Analysis of the interviews and
evaluation products in the promising practices sites reveal patterns in the establishment
of support for evaluation and the use of data. Recurring techniques to garner buy-in from
multiple stakeholders and establish an "evaluation culture" include the
following processes:
The sites also have developed creative, innovative, and effective solutions for the analysis and publication of their service and outcome data. Some of the more informative and compelling products developed in the sites include: descriptive outcome reports of children and families served by programs, feedback reports of individually-administered assessment instruments, conference presentations and other academic publications, newsletters and/or informational brochures, and reports of data collection completion rates. The major "take-away messages" of these evaluation reports have been: (a) outcome information can be a powerful catalyst for changing and developing programs; (b) data are pivotal to improving individualization and effectiveness of service delivery; and (c) evaluation can provide compelling evidence of accomplishments to support sustainability and to build an evaluation culture. Data have been utilized to support activities and the sites’ commitment to service delivery principles as well as:
The evaluators in the service sites illustrated in this monograph have developed ways to blend science, communications, and graphical skills in presentations of their evaluation data that have given their services public exposure, encouraged system improvements, and supported program sustainability. They have been flexible and responsive to the needs of their system of care by making their data visible in meaningful ways that leave local, state, and national audiences with concrete take-away messages, comprehensible facts, and ideas for promoting effective system modifications and sustainability. A supportive evaluation culture is the foundation, as well as the result, of producing effective data reports that have impact at multiple levels.
|
||
| top | Implications | |
| The results of this descriptive study indicate that the
utility and effectiveness of evaluation feedback is intricately tied to: (a) the
comprehensiveness and complexity of the service delivery system; (b) the integration of
the evaluation program with service delivery; (c) the quality of the management
information system in place; and even (d) the political climate supporting child and
family services. In addition, systematic and functional data reporting mechanisms require
multifaceted processes to support an "evaluation culture", and highly visible
products that are timely, meaningful, and practical for various audiences. Once these
methods are in place, useful evaluation reports can become the dividends of an effective
evaluation program—offering stakeholders deserving returns on their personal and
financial investments. Only through these challenging yet do-able data feedback processes
can systems truly adapt, grow, and endure in the sometimes unsteady waters of managed
behavioral health services. Useful evaluation reports are the dividends of an effective evaluation program—offering stakeholders deserving returns on their investments.
|
Back to Volume II: Using Evaluation Data to Manage, Improve, Market, and Sustain Children's Services
Back
to Center for Effective Collaboration and Practice Home
Page.
Questions?
Comments? E-mail the CECP.