Primary data: Data collected by an evaluation team specifically for the evaluation study. Probability sampling: The selection of units from a population based on the principle of randomization.
Every unit of the population has a calculable non-zero probability of being selected. Process evaluation: The systematic collection of information to document and assess how a program was implemented and operates.
Propriety: The extent to which the evaluation has been conducted in a manner that evidences uncompromising adherence to the highest principles and ideals including professional ethics, civil law, moral code, and contractual agreements. Qualitative data: Observations that are categorical rather than numerical, and often involve knowledge, attitudes, perceptions, and intentions. Quasi-experimental design: Study structures that use comparison groups to draw causal inferences but do not use randomization to create the treatment and control groups.
The treatment group is usually given. The control group is selected to match the treatment group as closely as possible so that inferences on the incremental impacts of the program can be made. Random digit dialing: In telephone interviewing, a technique used to select a sample.
Randomization: Use of a probability scheme for choosing a sample. This can be done using random number tables, computers, dice, cards, and so forth. Regression artifacts: Pseudo-changes in program results occurring when persons or treatment units have been selected for the program on the basis of their extreme scores. Regression artifacts are a threat to internal validity. Reliability: The extent to which a measurement, when repeatedly applied to a given situation consistently produces the same results if the situation does not change between the applications.
Reliability can refer to the stability of the measurement over time or to the consistency of the measurement from place to place. Replicate sampling: A probability sampling technique that involves the selection of a number of independent samples from a population rather than one single sample. Each of the smaller samples is termed a replicate and is independently selected on the basis of the same sample design. Resources: Assets available and anticipated for operations.
They include people, equipment, facilities, and other things used to plan, implement, and evaluate programs. Sample size formula: An equation that varies with the type of estimate to be made, the desired precision of the sample and the sampling method, and which is used to determine the required minimum sample size.
Sampling error: The error attributed to sampling and measuring a portion of the population rather than carrying out a census under the same general conditions.
Sampling frame: Complete list of all people or households in the target population. Sampling method: The method by which the sampling units are selected such as systematic or stratified sampling. Sampling unit: The unit used for sampling. Secondary data: Data collected and recorded by another usually earlier person or organization, usually for different purposes than the current evaluation. Selection and program interaction: The uncharacteristic responsiveness of program participants because they are aware of being in the program or being part of a survey.
This interaction is a threat to internal and external validity. Selection bias: When the treatment and control groups involved in the program are initially statistically unequal in terms of one or more of the factors of interest. Setting and program interaction: When the setting of the experimental or pilot project is not typical of the setting envisioned for the full-scale program.
This interaction is a threat to external validity. Stakeholders: People or organizations that are invested in the program or that are interested in the results of the evaluation or what will be done with results of the evaluation. Standard: A principle commonly agreed to by experts in the conduct and use of an evaluation for the measure of the value or quality of an evaluation e. It indicates how closely individual measurements cluster around the mean.
Statistical analysis: The manipulation of numerical or categorical data to predict phenomena, to draw conclusions about relationships among variables or to generalize results.
Statistical model: A model that is normally based on previous research and permits transformation of a specific impact measure into another specific impact measure, one specific impact measure into a range of other impact measures, or a range of impact measures into a range of other impact measures. Statistically significant effects: Effects that are observed and are unlikely to result solely from chance variation.
These can be assessed through the use of statistical tests. Stratified sampling: A probability sampling technique that divides a population into relatively homogeneous layers called strata, and selects appropriate samples independently in each of those layers. Subjective data: Observations that involve personal feelings, attitudes, and perceptions. Subjective data can be measured quantitatively or qualitatively. Surveys: A data collection method that involves a planned effort to collect needed data from a sample or a complete census of the relevant population.
The relevant population consists of people or entities affected by the program or of similar people or entities. Testing bias: Changes observed in a quasi-experiment that may be the result of excessive familiarity with the measuring instrument. This is a potential threat to internal validity. Treatment group: In research design, the group of subjects that receives the program.
Also referred to as the experimental or program group. Utility: The extent to which an evaluation produces and disseminates reports that inform relevant audiences and have beneficial impact on their work. Skip directly to site content Skip directly to page options Skip directly to A-Z link.
Section Navigation. Facebook Twitter LinkedIn Syndicate. Minus Related Pages. A key decision is whether there are existing data sources— secondary data collection—to measure your indicators or whether you need to collect new data— primary data collection. Depending on your evaluation questions and indicators, some secondary data sources may be appropriate.
Some existing data sources that often come into play in measuring outcomes of public health programs are:. Before using secondary data sources, ensure that they meet your needs. Although large ongoing surveillance systems have the advantages of collecting data routinely and having existing resources and infrastructure, some of them e. Primary data collection methods also fall into several broad categories. Among the most common are:. Choosing the right method from the many secondary and primary data collection choices must consider both the context How much money can be devoted to collection and measurement?
How soon are results needed? Are there ethical considerations? Is it about a behavior that is observable? Is it something the respondent is likely to know? Some methods yield qualitative data and some yield quantitative data. If the question involves an abstract concept or one where measurement is poor, using multiple methods is often helpful. Each method comes with advantages and disadvantages depending on the context and content of the data collection see Table 4.
The text box to the right lists possible sources of information for evaluations clustered in three broad categories: people, observations, and documents. Keep in mind that budget issues alone should not drive your evaluation planning efforts. The four evaluation standards can help you reduce the enormous number of data collection options to a manageable number that best meet your data collection situation. Here is a checklist of issues — based on the evaluation standards — that will help you choose appropriately:.
Mixed data collection refers to gathering both quantitative and qualitative data. Mixed methods can be used sequentially, when one method is used to prepare for the use of another, or concurrently.
An example of sequential use of mixed methods is when focus groups qualitative are used to develop a survey instrument quantitative , and then personal interviews qualitative and quantitative are conducted to investigate issues that arose during coding or interpretation of survey data. An example of concurrent use of mixed methods would be using focus groups or open-ended personal interviews to help affirm the response validity of a quantitative survey.
Different methods reveal different aspects of the program. Consider some interventions related to tobacco control:. When the outcomes under investigation are very abstract or no one quality data source exists, combining methods maximizes the strengths and minimizes the limitations of each method. Using multiple or mixed methods can increase the cross-checks on different subsets of findings and generate increased stakeholder confidence in the overall findings.
Table 4. A quality evaluation produces data that are reliable, valid, and informative. An evaluation is reliable to the extent that it repeatedly produces the same results, and it is valid if it measures what it is intended to measure. If you are designing your own evaluation tools, you should be aware of the factors that influence data quality:. A key way to enhance the quality of primary data collection is through a pretest.
The pretest need not be elaborate but should be extensive enough to determine issues of the logistics of data collection or the intelligibility of instruments prior to rollout. Obtaining quality data involves trade-offs i. You will also need to determine the amount of data you want to collect during the evaluation. There are cases where you will need data of the highest validity and reliability, especially when traditional program evaluation is being supplemented with research studies.
But there are other instances where the insights from a few cases or a convenience sample may be appropriate. If you use secondary data sources, many issues related to the quality of data—such as sample size—have already been determined.
If you are designing your own data collection tool and the examination of your program includes research as well as evaluation questions, the quantity of data you need to collect i. You will also need to determine the jurisdictional level for which you are gathering the data e. Improve delivery mechanisms to be more efficient and less costly - Over time, product or service delivery ends up to be an inefficient collection of activities that are less efficient and more costly than need be.
Evaluations can identify program strengths and weaknesses to improve the program. Verify that you're doing what you think you're doing - Typically, plans about how to deliver services, end up changing substantially as those plans are put into place. Evaluations can verify if the program is really running as originally planned. Program evaluation can: 4.
Facilitate management's really thinking about what their program is all about, including its goals, how it meets it goals and how it will know if it has met its goals or not. Produce data or verify results that can be used for public relations and promoting services in the community. Produce valid comparisons between programs to decide which should be retained, e.
Fully examine and describe effective programs for duplication elsewhere. This may seem too obvious to discuss, but before an organization embarks on evaluating a program, it should have well established means to conduct itself as an organization, e.
To effectively conduct program evaluation, you should first have programs. That is, you need a strong impression of what your customers or clients actually need. You may have used a needs assessment to determine these needs -- itself a form of evaluation, but usually the first step in a good marketing plan. Next, you need some effective methods to meet each of those goals.
These methods are usually in the form of programs. It often helps to think of your programs in terms of inputs, process, outputs and outcomes. Inputs are the various resources needed to run the program, e. The process is how the program is carried out, e.
The outputs are the units of service, e. Outcomes are the impacts on the customers or on clients receiving services, e. Often, management wants to know everything about their products, services or programs. However, limited resources usually force managers to prioritize what they need to know to make current decisions.
Your program evaluation plans depend on what information you need to collect in order to make major decisions. Usually, management is faced with having to make major decisions due to decreased funding, ongoing complaints, unmet needs among customers and clients, the need to polish service delivery, etc.
For example, do you want to know more about what is actually going on in your programs, whether your programs are meeting their goals, the impact of your programs on customers, etc? You may want other information or a combination of these. Ultimately, it's up to you. There are trade offs, too, in the breadth and depth of information you get. The more breadth you want, usually the less depth you get unless you have a great deal of resources to carry out the evaluation. On the other hand, if you want to examine a certain aspect of a program in great detail, you will likely not get as much information about other aspects of the program.
For those starting out in program evaluation or who have very limited resources, they can use various methods to get a good mix of breadth and depth of information. They can both understand more about certain areas of their programs and not go bankrupt doing so. Consider the following key questions when designing a program evaluation.
For what purposes is the evaluation being done, i. Who are the audiences for the information from the evaluation, e. From what sources should the information be collected, e. How can that information be collected in a reasonable fashion, e.
When is the information needed so, by when must it be collected? What resources are available to collect the information? When designing your evaluation approach, it may be helpful to review the following three types of evaluations, which are rather common in organizations. Note that you should not design your evaluation approach simply by choosing which of the following three types you will use -- you should design your evaluation approach by carefully addressing the above key considerations.
Often programs are established to meet one or more specific goals. These goals are often described in the original program plans. Goal-based evaluations are evaluating the extent to which programs are meeting predetermined goals or objectives. Questions to ask yourself when designing an evaluation to see if you reached your goals, are: 1.
How were the program goals and objectives, is applicable established? Was the process effective? What is the status of the program's progress toward achieving the goals? Will the goals be achieved according to the timelines specified in the program implementation or operations plan? If not, then why? Do personnel have adequate resources money, equipment, facilities, training, etc.
How should priorities be changed to put more focus on achieving the goals? Depending on the context, this question might be viewed as a program management decision, more than an evaluation question. How should timelines be changed be careful about making these changes - know why efforts are behind schedule before timelines are changed?
How should goals be changed be careful about making these changes - know why efforts are not achieving the goals before changing the goals?
Should any goals be added or removed? How should goals be established in the future? Process-based evaluations are geared to fully understanding how a program works -- how does it produce that results that it does.
These evaluations are useful if programs are long-standing and have changed over the years, employees or customers report a large number of complaints about the program, there appear to be large inefficiencies in delivering program services and they are also useful for accurately portraying to outside parties how a program truly operates e. There are numerous questions that might be addressed in a process evaluation. These questions can be selected by carefully considering what is important to know about the program.
What is required of employees in order to deliver the product or services? How are employees trained about how to deliver the product or services? How do customers or clients come into the program? What is required of customers or client? How do employees select which products or services will be provided to the customer or client?
What is the general process that customers or clients go through with the product or program? What do customers or clients consider to be strengths of the program? What do staff consider to be strengths of the product or program? Program evaluation with an outcomes focus is increasingly important for nonprofits and asked for by funders.
An outcomes-based evaluation facilitates your asking if your organization is really doing the right program activities to bring about the outcomes you believe or better yet, you've verified to be needed by your clients rather than just engaging in busy activities which seem reasonable to do at the time. Outcomes are benefits to clients from participation in the program. Outcomes are often confused with program outputs or units of services, e.
The following information is a top-level summary of information from this site. To accomplish an outcomes-based evaluation, you should first pilot, or test, this evaluation approach on one or two programs at most before doing all programs. The general steps to accomplish an outcomes-based evaluation include to: 1. Identify the major outcomes that you want to examine or verify for the program under evaluation.
You might reflect on your mission the overall purpose of your organization and ask yourself what impacts you will have on your clients as you work towards your mission.
For example, if your overall mission is to provide shelter and resources to abused women, then ask yourself what benefits this will have on those women if you effectively provide them shelter and other services or resources. As a last resort, you might ask yourself, "What major activities are we doing now?
This "last resort" approach, though, may just end up justifying ineffective activities you are doing now, rather than examining what you should be doing in the first place. Choose the outcomes that you want to examine, prioritize the outcomes and, if your time and resources are limited, pick the top two to four most important outcomes to examine for now.
For each outcome, specify what observable measures, or indicators, will suggest that you're achieving that key outcome with your clients. This is often the most important and enlightening step in outcomes-based evaluation. However, it is often the most challenging and even confusing step, too, because you're suddenly going from a rather intangible concept, e. It helps to have a "devil's advocate" during this phase of identifying indicators, i.
0コメント