Twenty years ago, many nonprofit organizations realized
their accounting systems could be better managed by computers.
Later, agencies expanded their computer usage to include databases
for mailing lists and donor information. According to a recent
survey, most nonprofits today use specialized accounting and mailing
list software and many have fundraising systems. A few organizations
now use information systems to track the outcomes of their programs.
This paper defines several forms of performance measurement, including
program evaluation, results mapping and outcomes measurement.
Next, it examines four case studies, based on interviews with
the staff of organizations that are at various stages of measuring
outcomes using computer software. Some general themes for designing
and implementing computerized outcomes measurement systems are
identified, and several software packages are discussed. Finally,
a framework for managers using technology to measure outcomes
is recommended. While some approaches may be more applicable to
particular fields, this paper offers a survey of current practices
and general guidance for human services and housing professionals.
The United Way of America describes performance measurement as a hierarchical chain:
Inputs are resources used, such as staff and money. Activities are the interventions or programs attempted. Outputs are the number of units of service or clients served. Outcomes are the benefits received by clients. Outcomes can be measured through outcome indicators, some of which are short-term, others that are long-term. For example, an initial outcome of a drug counseling session may be that the client better understands addiction, as demonstrated by their answer to questions on the subject. This is an indicator that the client may achieve longer-term outcomes, such as avoiding drug abuse. If clients gain knowledge, for example, then it may be assumed that they will modify their behavior. If they modify their behavior, then their status or condition will improve. The United Way of America describes this as an "if-then" chain, leading to intended outcomes. In essence, shorter-term, measurable outcomes act as a proxy for longer-term outcomes. By measuring initial outcomes instead of studying harder to quantify long-term outcomes, the United Way believes, "even smaller organizations are able measure their impact."
Outcomes indictors may be compared to targets set by the organization (80 percent of clients will pass the test on addiction) or to benchmarks from the past (last year 80 percent passed the test, next year 90 percent will).
There are many performance measurement methods, ranging from simple to complex:
INPUTS
ACTIVITIES
OUTPUTS
OUTCOMES
Often, all performance measurement is lumped into the category of "program evaluation." As the list above shows, program evaluation is just one tool available to organizations. Program evaluations are among the most expensive and time consuming methods of measuring performance. For this reason, most nonprofit programs do not conduct formal evaluations. Once one evaluation "proves" that an initial outcome leads to longer-term objectives, however, tracking initial outcomes as a proxy for longer-term ones can be validated for many organizations.
An outcomes measurement system often tracks numbers served and client demographics, the satisfaction of clients, as well as initial and intermediate outcomes. While this system could be managed manually, with papers kept in files, by automating information digitally, computers can streamline performance measurement. There are three primary reasons nonprofit managers should use IT systems to track outcomes: (1) digital information can be indexed, sorted and cross-tabulated in far more complex ways than are feasible in a manual system. (2) Computerized systems can store far more data much faster and, given an appropriate back-up system, more securely. (3) Computerized systems standardize data collection, much like using forms does, but also offers opportunities to integrate data systems from other areas in the agency, such as billing, to creating an enterprise-wide database that reduces redundant work efforts.
Qualitative data is typically recorded as text derived from interviews, focus groups and observations. This is often called "soft" data, due to its subjective and descriptive nature. Quantitative data comes from standardized testing instruments, surveys, budgets or other numerical accounts of programs. This "hard" data is more easily complied into statistics and reports. Qualitative data is more difficult to represent. It may seem that computerized systems would be superior for quantitative data, but less useful for qualitative data. In fact, however, computers are just as important for managing "softer" information. In order to analyze qualitative data, text must be reviewed and categorized. Once data is "coded" it can then be used more like quantitative data for reporting and analysis. Computerized systems excel at sorting and categorizing data.
An example using IT to analyzing qualitative data is "Results Mapping," developed by Barry Kibel at the Pacific Institute for Research and Evaluation. The results mapping approach records four variables for each unit of service: (1) the change agent, (2) the recipient, (3) the intervention, and (4) the initial/intermediate outcomes. Dr. Kibel has developed a standardized, one-sentence "story" so nonprofit staff simply describe their work in a conversational fashion, recording these four key pieces of information in a narrative form. This data is entered into a spreadsheet and then coded based on the type of client, staff and intervention. Over time, the results of particular interventions can be "mapped" in sequence for each client or each program by sorting and indexing the spreadsheet. Each point on the map is given a score based on its contribution toward desired outcomes. Scores from each point can be aggregated for clients or interventions to assess progress towards intended goals. Although ultimately results mapping is based on subjective, "soft" data, it has elements of quantitative data that make it possible for nonprofits to administer, use for management decisions and represent to funders.
The United Way of America recently surveyed the evaluation practices of United Way-funded organizations, yielding the following findings:
96 percent reported they measured outputs, such as the number of clients served.
76 percent stated they recorded client characteristics and demographics.
59 percent reported they assess management practices.
20 percent reported they measure participant satisfaction.
19 percent stated they measure program
outcomes.
Clearly, organizations tend to use outputs measures much more often than outcomes measures. Jed Emerson, Director of the Roberts Foundation, writes that nonprofits resist tracking outcomes because it removes resources and their attention away from service delivery. Also the staff of nonprofits generally do not view themselves as researchers and lack the skill and knowledge needed to track outcomes. Moreover, because of the nature of their clients, they will tend to track data only for those clients who remain in contact, a less than representative sample.
Sharon Oster, nonprofit management scholar, describes evaluating nonprofit programs as difficult because organizations have multiple goals and constituencies. It can be difficult to agree upon which outcome is important and how each data point should be measured. She also writes "evaluation is central to the continued ability of nonprofits to innovate, adapt and use scarce resources well in serving their missions." Yet, "data systems in a typical nonprofit often fail to provide management with meaningful information that could be used in performance reviews."
The United Way's own research finds that many organizations face enormous technical challenges to measuring outcomes. Not only do nonprofits have difficulty determining which variables to track, they lack the skill and information technology infrastructure to establish a tracking system.
Clearly, better methods of outcomes measurement are needed by nonprofit managers and IT could play an important role in facilitating measurement systems. The following mini-case studies review the experiences of four organizations, three that provide services directly to clients, and one that acts as an intermediary funder of other nonprofit organizations. These stories demonstrate the benefits and barriers of an IT-enabled outcomes measurement system.
This homeless shelter in downtown Boston developed a client-tracking database in early 1997. Based on Microsoft Access, the system automated the previous data collection system, which was based on paper forms filled out by the staff on each client. Beth Welty, program staff person and developer of the system, estimates she spent 80 to 100 hours designing the software, and continues to spend two or three hours per week on maintenance. All data is entered from paper forms into the system by designated data-entry staff. Ms. Welty designed the system after ANCHOR, a nationally available database for homeless shelters. Because Massachusetts may adopt ANCHOR as the standard for all area shelters in the future, this database was engineered to be easily transferable to that system. Ms. Welty, hired only a year ago, brought expertise in Access and knowledge of ANCHOR, as well as experience in the homeless field. Her unique skills allowed the development of a system without acquiring any new software or hiring a consultant, but also left her as the sole "expert" on the system. Over time, she says, other staff have climbed what has been a "steep learning curve" and are beginning to use the system.
The Rescue Mission typically has only a short-term relationship with clients. Given the transitory nature of their target population, it is difficult to establish the impact of programs. Recent contracts from the state of Massachusetts and the US Department of Housing and Urban Development (HUD), however, require specific outcomes measures. The trend is that each year's contract more explicitly requires agencies to track clients and demonstrate program effectiveness. Some of the staff at the shelter resent having to ask clients questions and track their behavior, believing it detracts from service provision. Yet, as competition for funding tightens, the Rescue Mission will not survive without adopting more careful outcomes measures.
The system has proven most valuable in the organization's advocacy efforts. For the first time, the Rescue Mission has consistent and easily organized data that indicates where clients come from and what other services they recently received. The organization can present evidence to policy makers that indicates a need for new programs or funding, such as the fact that a certain percent of homeless clients are recently released from the penal system. For the first time the agency can quickly run reports for funders that show the demographics of the population served. The database has also allows the agency to tailor its service delivery to the changing needs of clients.
Ms. Welty's goal is for this system to become more integrated into the organization's operations, and her long-term goal is to incorporate a follow-up component that would allow pre- and post-treatment comparisons. At this point, however, the focus is on measuring outputs and client status rather than outcomes. In addition to the difficulty of gathering this data from highly mobile clients, the Rescue Mission also lacks a funding source to pay for the follow-up activities required for outcomes measurement. Compared to the paper-based system the organization used a year ago, however, the agency has made great progress in implementing an IT system that allows it to track some indicators of its performance.
Beech Acres has served families and children since the 1800's. Originally an orphanage, the organization evolved into a large and sophisticated provider of out-patient and residential mental health counseling. The agency has a database to track clients and handle billing, much as many other organizations operating in a managed care environment. Two years ago, after hearing a presentation by Barry Kibel at the United Way, Beech Acres management adopted the results mapping approach. Using an Excel spreadsheet, a narrative, or "story," is written about each interaction with a client. Four data points, (1) the change agent, for example, "Counselor Smith," (2) the intervention, for example, "counseled on addiction," (3) the client, for example "Jane Doe," and, (4) the outcome, for example, "gained knowledge about alcoholism," are recorded in the spreadsheet. Any intermediary step toward the client's treatment goals is tracked. Each intervention is given a score, based on its significance and resiliency (sustainable, longer-term results receive higher scores). "The system is based on the theory that intermediate steps increase the probability of larger outcomes being achieved," said Joan Gallenstien, the director of Beech Acres results mapping project. Over time, the results of particular interventions or change agents can be tracked. Because the narratives are in a spreadsheet, the data can be sorted, indexed and searched in order or organize information for reports and to guide treatment programs.
The first year, Ms. Gallenstien, who is herself an experienced social worker, interviewed every clinician weekly, guiding their analysis of each intervention and structuring their results "story." Since she was the only person who entered information, she was able to standardize terminology. This year the some staff have begun to enter their own information, while other ("computer-phobic") staff fill out forms that are entered by Ms. Gallenstien. The quality of the data has declined since individual staff began entering their own information. Ms. Gallenstien is working to establish standardized language and to train staff to become more proficient at entering their clients' narratives.
Presently, clinicians may be required to report their work five separate times. Ms. Gallenstien is struggling to re-engineer other processes at the agency to streamline data collection. She has begun sitting in on weekly case consultations to try to glean results mapping stories from discussions of each client. That method has not been very successful, however. Because processing results mapping stories can take several hours per week, management is eager to speed up the process, but some staff enjoy the opportunity to de-brief and analyze their work. An ideal way to collect data has not yet been discovered.
Because results mapping is considered to be a subjective, "soft" form of outcomes measurement, Beech Acres combines reports from this process with more traditional standardized test scores, such as behavioral indexes completed by clinicians or clients. The agency also uses customer satisfaction surveys to round out its outcomes measures. The state of Ohio will soon require all human services agencies to use a standardized pre- and post-testing instrument for all clients. Based on changes in these test scores, those close to the funding process tell Ms. Gallenstien, the state will make decisions as to which agencies will be funded. Ms. Gallenstien described the prevailing attitude in government as a desire for a "positive test score that is statistically significant. It is easier for them [funders] to make decisions based on numbers than more subjective, but more accurate, I think, systems such as our results mapping approach."
NHS of Cincinnati has been providing housing rehabilitation loans and homebuyer counseling for nearly 20 years. Although the Neighborhood Reinvestment Corporation (NRC), developed a specialized outcome tracking database in 1997 called N-Works 2000, NHS of Cincinnati has not adopted it. Instead, it relies on a system developed in-house based on Q&A software, a flat (non-relational) DOS database (see Appendix: Types of Software). That system only tracks clients who apply for a loan. The system is administered by one staff person, Anna Sharp, who generates reports as required for the city, the NRC or for HUD. Ms. Sharp recognizes the limitations of this system, but generally feels it can provide evidence of NHS activities, including the performance of the loans that clients borrow. She would like to use the N-Works 2000 database the NRC developed, but does not know its base software (Microsoft Access) and lacks time to learn. She has that software on her desk, but has not installed it.
The greatest difficulty with measuring outcomes at NHS is deciding which programs and which clients should be included in the process. While the agency trains and counsels thousands of potential home buyers each year, those clients do no enter the data collection system until they take out a home rehabilitation loan or mortgage. More than 85 percent of the people who attend NHS homebuyer counseling do not take out a loan with NHS, although many probably will achieve NHS's intended outcome of homeownership through another lender. Even though N-Works 2000 does have fields to track these clients, NHS of Cincinnati does not have a process to collect information from these individuals, particularly after they leave a training session. IT will not solve this problem.
Technology can, however, improve how data is reported. Because NHS has a "patchwork" of funders, it has many constituencies to which it must present outcomes information. While local banks that fund the agency want to know the how many new homebuyers NHS created, the city wants to know the value of homes improved. For some funders, the use of geographic information software (GIS) would demonstrate NHS activities in high-risk neighborhoods. For others, an economic impact matrix that shows how the agency stimulates tax revenue is more valuable. Ms. Sharp attempts to satisfy each funder's need on an ad hoc basis, often creating reports based on summaries of written data or tabulations of paper forms, but her system cannot generate the range of information needed.
Despite the difficulties NHS has encountered, its Q&A system allows it to quickly and accurately pull data as needed. The experiences of another NHS provides a comparison. The agency tracks data on its activities on legal pads, summarizing data by calculator when reports are required. Periodically data is written on index cards and sorted to compile information. One funder noted that the quality the agency's often hand written reports is generally poor. Clearly, using even antiquated technology results in better quality information.
The United Way of America has backed outcomes measurement initiatives for several years. Plagued by scandals and reliant on donations from private corporations and their employees, the agency has taken this strategy because must prove its outcomes to donors, or risk losing support. The United Way of Cincinnati, one of the nation's stronger United Ways, has become a leader in the measurement of outcomes. An experienced field staff person and an MIS expert worked together to develop an Access-based database that tracks each of the 150 agencies that the United Way funds each year. In addition to basic information about the agency and its funding, the database includes a module that records the agency's goals for the next period. Each goal has specific outcomes associated with it, compiled in a narrative format. At the end of each period, the agency's results are compared with its goals. The system also tracks demographic data on who was served by each agency. The United Way's funding committees can assess each agency's progress relative to its goals and to other agencies. The United Way can also use data on who was served in its public relations and fundraising efforts.
Currently, each agency submits its data to the United Way on paper forms. A recent survey of funded agencies shows that about 60 percent have Microsoft Access and the ability to submit data electronically. A process for electronic data transfer may begin this year, although some agencies are so small that they will continue to submit data on paper.
The system has given the United Way better information more rapidly, enhancing its ability to make fund allocation decisions and to attract donors. It has also created a mechanism for funded agencies to set goals and meet performance targets. This system was highlighted at a national conference in late 1997, and the Cincinnati United Way is selling its system to other United Ways across the nation (for $5,000, installed). Ultimately the system is based on self-reporting and is subjective, but its meets the organization's current needs, as well as those of other United Ways.
Implementing a computerized database that will provide new information to management, staff and funders will cause significant changes in the dynamics of an organization. By planning for the operational changes caused by a new information system, the organization will create value as better information leads to improved services, affirmation of the staff's efforts and increased fundraising. Poorly managed, however, a computerized outcomes management system can lead to an organizational disaster. Carefully planning, implementation flexibility and a clear strategy will guide the process towards creating the maximum value.
As the case of the NHS of Cincinnati showed, it can be difficult to decide which outcomes should be measured. It is important to gather information from staff, board members and funders early in the process of designing the system Staff should have input into the design and learn how the system will operate. Each staff person should consider how their job will change or could be improved if they had better data on the outcomes of their work. If an outside consultant is used, the staff should calibrate the consultant's vision of the software with their own.
After installation, staff will have to be trained not only on how to input data, but also how to use the newly developed outcomes data to improve operations. The NHS's reluctance to install a new software system shows that time and resources must be allocated for introducing and learning a new system. It is likely any new database will have "bugs" initially, which will delay the project and create frustration. Although rare in nonprofits, designing some incentive for using the system can help motivate staff. The incentive may be monetary, or may simply represent a relief from other duties in exchange for supporting the system. Staff should want to use the system, recognizing it will improve their ability to perform. In all of the case studies described, only one or two people in the organization worked with the computer. Other staff, out of fear, ignorance or lack of time, do not use the system. Frequently the staff of nonprofits are less productive than they could be because they resist using technology. Obviously, leadership of the project at the highest level will encourage the staff to become involved in its implementation.
Outcomes software should measure the progress, outcomes, and satisfaction of clients, and also be interactive with other aspects of the agency, such as case management and financial administration. Beech Acres did not integrate its results mapping data with its other client data systems. By integrating systems, redundant data entry could be eliminated. Integration may not always be feasible, since client confidentiality limits access to some data. In a networked system, management must be careful to establish which staff are permitted to use each area of the database.
Once outcomes measures are generated by the system, the organization is likely to discover new uses for the information. Organizations may learn, as demonstrated by the Boston Rescue Mission, that reports used to tailor services to client needs may also be useful in efforts to advocate for changes in public policy.
Many nonprofits already have the software needed to track outcomes. With a spreadsheet and a database, such as Microsoft Excel and Access, many organizations, like the Boston Rescue Mission, can create their own system. Such a process is time-consuming and fraught trials and errors, but it can prove successful, particularly for smaller organizations. Having a model to help design screens and decide which variables to track, such as ANCHOR, can also be helpful. Based on the common use of Access in the case studies described above, as well as the widespread use of Access, many organizations base their internally-created systems on this software.
There are some pre-designed systems that could be effective for a few organizations. Several of these packages are reviewed in the Appendix, but each is designed for a specific type of organization. In general, however, the differences among organizations make using a pre-designed system difficult. Software designers often mention the 80/20 rule-an organization typically can buy a system that satisfies 80 percent of its needs for 20 percent of the cost of a customized system. In the case of outcomes tracking software, however, the huge variation among organizations may make finding a system that satisfies 80 percent of an agency's needs difficult. If an appropriate system is found, organizations should be careful to "test drive" software, entering examples of data it expects to use.
Many organizations hire consultants to design software, such the Neighborhood Reinvestment Corporation (NRC) did with the N-Works 2000 system. Because the NRC has over 200 affiliates to share the fixed costs of developing a database, this proved to be a cost-effective option. For most agencies, hiring a developer is expensive and time-consuming. In the long run, however, it will create a product that works well and closely matches the needs of the organization. If the software proves successful, as the United Way of Cincinnati's did, organizations may be able to re-coop its development costs by selling the software to similar agencies.
David Garvin defines a "learning organization" as one skilled at "creating, acquiring, and transferring knowledge, and at modifying its behavior to reflect new knowledge and insights."' A computerized outcomes measurement system will collect, sort and aggregate the results of the organization's programs, providing better information for line staff, administrators and funders. As Gary Pucket wrote in his article, Use Your Computer To Evaluate Your Programs, "a well-designed system will help your organization help people - effectively!"
Word processors are ubiquitous; if an agency has a PC, it has some word processing capacity. Newer systems can take text and create merge files that act in some ways like a database, but in general these systems were designed for typing and printing documents, not for collecting and analyzing outcomes data. Examples: WordPerfect and MS Word.
Spreadsheets allow easy data entry and analysis, as well as superior graphing and charting. Typically used for financial or other numerical data, textual data can also be used. Although very large datasets (tens of thousands of observations) are awkward to use in a spreadsheet, many versions have advanced data management modules, rivaling statistical packages. Examples: Lotus 123, Excel, Quattro Pro.
Statistical packages are designed exclusively to analyze datasets, using complicated mathematical operations on many variables at a time. Some systems can only use numeric data. Typically, data is exported from a database to a statistical package for analysis. Examples: Stata, SAS, SPSS.
Databases are the most common software used to track outcomes. Data can be recorded as text or numbers. Searching and coding data is usually simple, as is generating reports based any given variable. New variables can be generated from existing ones, allowing data to be stratified by type or kind. A decade ago, most databases were "flat," meaning information is kept in a single file, like a spreadsheet. Today, most databases are relational databases that join several tables of information based on common variables. Data is accessed though a viewer that can search for and assemble variables from separate files. There is great variation among systems, from those that are intuitive to those that only a programming expert can design. Examples: Dbase, Access, Approach, Oracle, FoxPro, Filemaker Pro.
| Software | ||||||
| Description | An evaluation software system designed specifically for nonprofits in the health and human services field. It includes basic client characteristics, treatments or activities and measurements of intermediate outcomes. | Sold by Cost Management Associates this program is designed to track community development data for agencies that receive federal Community Development Block Grant funding. Although outcomes data on individual clients is tracked by the system, its primary purpose is to generate standardized reports for funders. | Designed by staff of the Neighborhood Reinvestment Corporation and consultants specifically for its 200 member agencies. It includes data on clients, loans and other community development activities. While primarily designed to produce reports for funders, the system can be customized for each site. It is an affordable solution, although each site will require modification for its unique programs. | A database developed for the social services industry. Primarily designed to match clients to appropriate services, the program complies client intake data and produces reports for social workers on each clients' needs. It could be modified to measure outcomes, but it does not do so in its current form. | Produced by CASS Software. Specifically designed for mental health agencies to assess clients' mental status. The software can be downloaded for free, but the assessment scales, required for operation, must be purchases from CASS. For this very specific application, it provides a good measure of outcomes, based on pre- and post-treatment data. | Designed for mental health agencies. It tracks client treatments and demographics as well as clinical notes and other narrative data. Reports can be run that indicate intermediate outcomes measures. Based on a client-server system, this database integrates aspects of spreadsheets, word processors and databases. It is not easily modified and may require customization for each installation. |
| Base Software | Visual Basic | Visual Basic | Microsoft Access | Microsoft Access | Borland Database Engine | ODBC and SQL |
| Price | The software is released in modules, ranging from $3,500 to over $20,000. | Prices vary. | $350 per affiliate | N/A | Free, but assessment scales $500 and up | Sold in modules. $3,500 and up |
Human Service Information software, Echo Consulting Services, 1620 Main Street, P.O. Box 540, Center Conway, New Hampshire 03813-0540 (phone 603-447-5453 or 800-635-8209; fax 603-447-2037). http://www.echo.com
IPASS Program. Evaluation software, Agency Systems, Comprehensive Community Software, P.O. Box 37410, 7645 Production Drive, Cincinnati, Ohio 45237 (phone 800-685-6006 or 513-761-4610). http://www.agency-sys.com
Case Mananger 1.0. BMI Systems, Inc. Box 251 Boston, MA 02248 http://www.costmaninc.com/manage
N-Works 2000, Neighborhood Reinvestment Corp. New England District, Boston, (617-450-0410) talford@nw.org
Demos at Computer Use in Social Services Network, http://www.uta.edu/cussn/cussn.html:
CASS -Computer Assisted Social Services http://www.syspac.com/~walmyr/
TXPlan Human Services/Mental Health Tracking http://earleycorp.com
ASSIST Software for Human Services Client Tracking ftp://ftp.uta.edu/cussn/assist.zip
Information Please: http://www.dfw.net/homepage/
(ITRC) Information Technology Resource Center 6 N Michigan Ave-Suite 1405 Chicago, IL 60602 312/372-4872 itrc@igc.org http://www.mcs.net/~itrc and http://npo.net/itrc/ and http://www.igc.org/trc/
Applied Research Group Box 7401, N. Carolina State University, Raleigh, NC 919/515-1300 http://www.cuacs.ncsu.edu/arg/ers.htm
Cost Management Associate, Inc. Nonprofit
Solutions Dept. http://www.costmaninc.com/manage/
CompuMentor http://www.compumentor.org
Computers in Human Services Haworth Press, 10 Alice St., Binghamton, NY 13904-1580 (800/342-9678) email Hamis@spectra.net http://www.uta.edu/cussn/cussn.html
Harvard Family Research Project: "The Evaluation Exchange," http://www.harvard.edu/HFRP/
Office of the Assistant Secretary for Planning and Evaluation (OASPE), which oversees program evaluation in the U.S. Department of Health and Human Services. http://www.os.dhhs.gov
Support Centers of America's Technology Forum http://www.supportcenter.org/sf/mkttech.html
The NonProfit Times Online Directory http://www.nptimes.com/
The Philanthropy Journal Technology Section: http://www.philanthropy-journal.org/tech/tech.htm
United Way of America. http://www.unitedway.org
ACCESS-L: Microsoft Access Database Discussion List. To subscribe send "subscribe ACCESS-L" to LISTSERV@PEACH.EASE.LSOFT.COM
CUSSNET: Computer Use in Social Services
Network Discussion List, To subscribe send "SUBSCRIBE CUSSNET"
to LISTSERV@LISTSERV.UTA.EDU
Dave Curry, Researcher, Pacific Institute, Chapel Hill, NC, January 5, 1998.
David Dangler, Director, Rutland West (Vermont) Neighborhood Housing Services, December 16, 1997.
Kim Johnson, Senior Field Service Associate, United Way and Community Chest of Greater Cincinnati, Ohio, January 5, 1998.
Joan Gallenstien, Director of Program Evaluation, Beech Acres, Cincinnati, Ohio, January 5, 1998.
Meg Plantz, Senior Research Associate, United Way of America, Arlington, VA, January 6, 1998.
Gary Pucket, Sales Staff, Agency Systems, Cincinnati, Ohio, January 7, 1997.
Anna Sharp, Operations Manager, Neighborhood Housing Services of Cincinnati, Ohio, December 26, 1997.
Beth Welty, Program Associate, Boston
Rescue Mission, December 18, 1997.
_____ "10 secrets of computer wizardry." Nonprofit World. 12(3): 24-32. 1994 May/Jun.
_____ "Top problems facing nonprofits." Nonprofit World. 15(5): 55. 1997 Sep/Oct.
_____ "How you & other nonprofits are using technology: Latest survey results." Nonprofit World. 14(3): 36-39. 1996 May/Jun.
_____ Getting Smart, Getting Real Annie E. Casey Foundation 1995 Research and Evaluation Conference Report, 1996.
_____ Annotated Bibliography of Program Evaluation Program Evaluation Project, Center for Nonprofit Management, 1997
Alie, Raymond E. Seita, John R . "Who's using evaluation and how? New study gives insight." Nonprofit World. 15(5): 40-49. 1997 Sep/Oct.
Altshuler, Alan A. "10 Lessons Learned" http://ksgwww.harvard.edu/~innovat/10less.htm Innovations in Government Program, Kennedy School of Government, Harvard University.
Altshuler, Alan A. and William B. Parent "Breaking Old Rules: Four Themes for the 21st Century" http://ksgwww.harvard.edu/~innovat/4themes21st.htm Innovations in Government Program, Kennedy School of Government, Harvard University.
Anthony, Robert N. and David W. Young, Note on Performance Measurement in Nonprofit Organizations Crimson Press, Cambridge, MA 1997.
Applegate, Lynda M. Warren F. McFalan, and James L McKenney, Corporate Information Systems Management: The issues facing senior excutives, 4th Ed. Richard D. Irwin (publisher) 1996.
Asbrand, Deborah, "Habitat of Humanity Creates Better IS Blueprint" Infoworld Vol 17(13) 74-78, Mar 27, 1995.
Beckley, Rika. Elliott, Margaret A. Prickett, Jeanine M . "Closing the gap: Information technology & the nonprofit sector." Nonprofit World. 14(1): 36-42. 1996 Jan/Feb.
Connelly, Shannon. "1995 Nonprofit Software Directory" Nonprofit World. 26(8): 34-46. Oct, 1995.
Dixon, Greta KMarie . "Evaluating nonprofit programs: Now it's essential." Nonprofit World. 12(3): 44-46. 1994 May/Jun.
Drucker, Peter, "Management & Objectives: What Results Should You Expect? A User's Guide to MBO," in Martin Gruber (ed.), Management Systems in the Human Services, Philadelphia Temple University Press, 1981.
Emerson, Jed and Fay Twersky, eds. New Social Entrepreneurs: The Success, Challenge and Lessons of Nonprofit Enterprise Creation. The Roberts Foundation Homeless Economic Development Fund, San Francisco, September, 1996.
Feferman, Flavio "Qualitative Data and the Software that Loves Them" Star Systems Tech Talk. http://www.os.dhhs.gov
Feferman, Flavio "Software for Handling Your Quantitative Data" Star Systems Tech Talk http://www.os.dhhs.gov
Hatry, H, T. Van Houten, M. Plantz and M. Greenway, Measuring Program Outcomes: A Practical Approach, United Way of America, Washington, DC, 1996.
Jonas, Mort "Tough Decision: Choosing nonprofit software" http://philanthropy-journal.org October 25, 1996
Jorjani, Hamid . "The information highway and program evaluation: Challenges and opportunities." Optimum. 27(1): 7-13. Mar.1997
Keen, Peter, Every Manager's Guide to Information Technology: A glossary of key terms and concepts for today's business leader Harvard Business School Press, 2nd Ed, 1994.
Keyes, Marge "Quality Measurement Network (QMNet): Final Report on Changes Needed in CONQUEST's Software Based Programs" http://www.os.dhhs.gov, December, 1997.
Kibel, Barry M. Success Stories as Hard Data: An Introduction to Results Mappingsm Pacific Institute for Research and Evaluation, Chapel Hill, NC, Pre-edited Draft Copy, July 1997.
Lightworks Technology Foundation "Funding technology programs: A thumbnail guide for nonprofit organizations" Chapel Hill, North Carolina, April 18, 1997
Martin, Lawrence L. Kettner, Peter M ."Performance measurement: The new accountability." Administration in Social Work. 21(1): 17-29. 1997.
McKersie, William S . "Watchful waiting, and other evaluation lessons learned: A Chicago case study." Foundation News. 38(5): 32-38. Sep/Oct 1997.
Nonprofit Web Talk, "Making Databases Work for Your Nonprofit " Guest: Brian Larsen, Donor II Host: Sean Bailey, Philanthropy-Journal Transcript from April/May 1997 The Philanthropy Journal http://philanthropy-journal.org
Ostrooke, Gary. "Investing in Results" News and Views: United Way of America's Executive Newsletter, Vol. 2(7). October 6, 1997. http://www.unitedway.org/outcomes/invest.htm
Ouellette, Tim ."Nonprofits rely on IS innovation." Computerworld. 30(51): 6. Dec 16, 1996.
Plantz, Margaret, et al, "Outcomes Measurement: Showing Results in the Nonprofit Sector," in Kathryn Newcome, ed., New Directions for Program Evaluation Vol 75, San Francisco, CA: Jossey-Bass, Fall 1997. http://www.unitedway.org/outcomes/ndpaper.htm
Popjoy, Oveta. Information systems developed for nonprofits for profits" Journal of Information Systems Vol 43(10) 25-29, Oct. 1992.
Puckett, Gary W. and Jean A Vogt." Use your computer to evaluate your programs." Nonprofit World. 13(1): 21-22. Jan/Feb 1995.
Rossi, Peter and Howard Freeman, Evaluation, 5th Ed. Newbury Park, CA, Sage Publications, 1993.
Wolosky, Howard W . "Evaluating a nonprofit's accounting software needs." The Practical Accountant. 29(7): 55-61. July 1996.
Young, Dennis R., "What Does Feedback Have to
Do with Nonprofit Governance?", Nonprofit World, November-December
1996.
This document is also available at http://pages.prodigy.net/michael_collins/Outcomes_IT.doc
This trend is even more keen in government. The Government Performance & Results Act of 1993 (GPRA) mandates that beginning with fiscal year 1998, all federal departments must report performance measurement data, with an emphasis on outcomes.
______. "How you & other nonprofits are using technology: Latest survey results." Nonprofit World. 14(3) May/Jun. 1996, p. 38.
Hatry, H, T, 1996, p. 1
Dixon, 1994 p. 46
Results Mappingsm is a registered trademark of the Pacific Institute for Research and Evaluation
Feferman, http://www.os.dhhs.gov
Sumariwalla, 1991. p.
Emerson, p. 382
Oster, p. 147.
Hatry, 1996, p. 7
A national Congressionally-funded umbrella organization for all 200 NHS organizations across the US
A member of the funding evaluation team at the NRC made this comment in an interview about N-Works 2000
Puckett and Vogt, p. 21.
A study by the US Department of Human Services determined Access is preferable for many human services applications.
Keene 1994
Wolowsky, p 3
McKersie, p. 35
Ouellette, Tim, p 6.
Garvin 1993. p.
Puckett and. Vogt, p. 22