Listserv Session:
Evaluation Framework (February 2-9, 2004)

---

Draft ADRC Evaluation Framework and Potential Indicators

Click here for a PDF version of the Draft ADRC Evaluation Framework and Potential Indicators for better printing.

At AoA and CMS’ request, the staff of the Technical Assistance Exchange (TAE) drafted a framework for examining the Aging and Disability Resource Centers (ADRCs) within the environment they operate, as well as potential indicators to measure performance and impact. We are sharing the framework and indicators with you, the grantees, to solicit your feedback. We wanted to work with you to refine the framework and indicators to meet your data collection, monitoring and outcome tracking needs. We consider the concepts and measures described here a starting point for dialogue. We think it is critical to get your input about these items because they will be the basis for both establishing the grant reporting requirements and possibly standardizing some indicators across the grants for a national evaluation.

Evaluation Framework

We placed the ADRC evaluation framework within the larger Department of Health and Human Services (DHHS) framework and AoA-CMS major initiatives and system change goals. Both AoA and CMS view your efforts as central to providing models for delivery systems nationwide. As a result, it is critical to develop a framework that will allow us to gather lessons learned and quantify the effects of your programs.

Exhibit 1 presents the broad DHHS systems framework for striving to support individuals with disabilities to engage in community life on their own terms. The ADRCs embody the Access portion of the coherent systems management efforts. Click here for Exhibit 1.



The ADRCs also constitute a joint AoA-CMS initiative that complement several of each agencies own initiatives encouraging system change related to access, efficiency, effectiveness and a consumer focus This is shown in Exhibit 2. Click here for Exhibit 2.

Using these two broader frameworks, we developed an ADRC specific evaluation framework that delineates the ADRC interventions, mediating factors, intermediate outcomes and desired long-term impacts (Exhibit 3). The interventions fall into the three broad mandates for the grants to: 1) increase awareness and provide reliable information; 2) provide assistance in seeking services and making decisions; and 3) simplifying and streamlining access to public programs through a physical or virtual one stop shop. Each grantee’s progress will be influenced by a number of mediating factors, including their starting point related to infrastructure, previous experience, collaboration, and leadership, the broader political and economic environment, as well as geographic considerations, and the audiences chosen to target. Intermediate outcomes expected to manifest over the next three years include:

• Awareness among a number of internal and external groups, including those critical pathways where institutional versus community placement decision are often made;


• Visibility of the ADRCs and the aging network role in the access to long term care supports;


• Fostering understanding among the target populations, providers and the critical pathways of available resources and options through the awareness and information activities;


• Gaining the trust of consumers, their families, providers and others by providing responsive information, assistance and systems of access; and


• Integrated access to public programs through combining intake, eligibility and assessment processes that, along with visibility, fosters ease of access.

The evaluation framework is displayed in Exhibit 3. Click here for Exhibit 3.


The interventions and intermediate outcomes will manifest in the long-term impacts of consumer focus, enhanced access, efficiency, and visibility/awareness.

Start-Up Process Measures

Within an evaluation framework, we would hope to identify indicators that demonstrate progress toward or accomplishment of the desired systems change goals. However, during the planning and initial implementation phase of the ADRCs, these types of indicators do not lend themselves to potential reporting requirements because most activity is focused on decision making, model development, and start-up. Therefore, prior to presenting potential indicators relevant to the post-start-up period, we present potential start-up process measures that could serve as candidates for the first year’s reporting requirements in Exhibit 4. Click here for Exhibit 4.

Our plan is to work with the grantees, AoA and CMS to add to, delete from, and/or refine the list of measures for Year One progress to be part of the initial semi-annual reporting requirement. For those familiar with the new web-based annual reporting mechanism for the Real Choices Grants, we anticipate that the semi-annual reports required of the ADRC grantees will also be web-based, but tailored more specifically for the ADRCs. The input required of you could be narrative or we could strive to have drop-down menu choices where possible.

Potential Indicators Following Implementation

Following consideration of possible year one reporting requirements, we turned to identifying potential indicators for each of the desired systems change goals/long-term impacts. As noted above, most of these indicators become relevant only after the start of operations. The indicators are classified based on whether the measure captures:

• Structure – elements likely to be established in the development of the ADRCs that tend to be more tangible


• Process – activities likely to occur in the implementation of the ADRCs that facilitate the system change goals


• Output – measures of activities or products of the ADRCs


• Outcome—results of the ADRC that indicate progress toward goals


• Impact – longer range measures of changes related to goals that can be attributable to the ADRCs (i.e., establishing cause and effect by assessing results relative to if the program had not been implemented by controlling for factors other than the ADRCs to the extent possible).

Exhibit 5 presents the first cut at potential indicators. For each indicator, we also identify the likely data source for each through the color scheme. The list builds on indicators included in many of your applications, however, it is not exhaustive, nor would all grantees necessarily have activities that would result in, or provide information to support, all of the indicators. We also acknowledge that the list is rather comprehensive in its range. Please consider these only as possibilities and be assured that we will work with you collectively to come up with a set that makes sense. Click here for Exhibit 5.

Similar to the Year One proposed measures, our plan is to work with the grantees, AoA and CMS to identify any additional indicators and then to develop a smaller set of agreed upon indicators to be part of later semi-annual reporting requirements. We think it is important to begin this process early because some of the indicators require specific information to be tracked in your management information systems that many of you are making decisions about. In identifying the subset of indicators for a possible reporting requirement, the focus will likely be on those indicators that you can self report major milestones (e.g., establishment of an advisory group) or can be supported by the information systems that you will be putting in place as part of your ADRCs (e.g., number of institutional level of care determinations by source of referral). Indicators that would support continuous quality improvement efforts and could begin to make the case for continuation beyond the grant period will also be strong candidates.

Soliciting Feedback

We were very interested in your feedback regarding the evaluation framework, the potential Year One process measures, and the potential longer-term indicators. The only way for the framework and the indicators to successfully capture program progress and impact is if the framework rings true in your daily operations and the indicators provide you with meaningful measures of your achievements. Once finalized, the framework and indicators will likely be used for a national evaluation. They will also be used to develop a subset of indicators for the grants’ semi-annual reporting requirement. Finally, we will use the framework and indicators to provide requested technical assistance with your own evaluative efforts. If there is interest, we could pursue developing common IT elements, measures and survey instruments to facilitate consistency in the information collected, as well as the ability to aggregate across and compare between states.

If you have comments that you want to share directly with us, please e-mail them to ADRC-TAE@lewin.com or call either Lisa Alecxih (703-269-5542) or Karen Linkins (703-269-5681).