Inside This Issue
- Welcome to the SAAP V National Evaluation Bulletin
- National Evaluation of SAAP V - Overview
- SAAP V National Evaluation Framework (2005 – 2009)
- National Evaluation Research Agenda
- SAAP V Mid-term Review (March – December 2007)
Welcome to the SAAP V National Evaluation Bulletin
Welcome to this first issue of the SAAP V National Evaluation Bulletin prepared by the Evaluation Working Group (EWG) on behalf of the SAAP National Coordination and Development Committee (CAD). CAD established the EWG in late 2005 to provide advice on the National Evaluation of SAAP V. The EWG is chaired by the Australian Government Department of Families, Community Services and Indigenous Affairs (FaCSIA) and has representatives from each state and territory government and two community experts including a sector representative from the Australian Federation of Homelessness Organisations (AFHO). CAD has recently requested the EWG to provide a brief bulletin, every six months, to keep the SAAP sector informed of evaluation developments through the life of SAAP V. I trust you will find the SAAP V National Evaluation Bulletin both informative and interesting.
Clare Wall
Chairperson, SAAP National Coordination and Development Committee
National Evaluation of SAAP V - Overview
The SAAP V Multilateral Agreement (2005-10) requires a mid-term review (MTR) to be completed this year and a final evaluation (FE) of SAAP V to be completed by 30 June 2009.
CAD has agreed that the SAAP V National Evaluation will be a rolling ‘impact’ evaluation assessing whether or how well SAAP V has achieved its objectives, that is, the extent to which the program contributed to: '...self-reliance and independence for people who were homeless or at imminent risk of homelessness'.
The term 'rolling' evaluation is used to suggest a strategic focus on:
- early development of an appropriate evaluation strategy
- significant stakeholder engagement with the evaluation process
- integration with the ongoing work of the sector
- helping to drive the SAAP V reform agenda
- an action research approach where possible
- ongoing quantitative and qualitative information gathering (and sharing).
The EWG will conduct the MTR ‘in-house’ on behalf of CAD, and will also manage the National Evaluation Research Agenda. An independent evaluator will be engaged to conduct the final evaluation. To ensure a rigorous process, following the MTR and prior to appointing the independent evaluator, an evaluation expert will be engaged to conduct an independent assessment of the SAAP V National Evaluation Strategy before it is finalised.
SAAP V National Evaluation Framework
(2005 – 2009)
The SAAP V National Evaluation Framework consists of four key questions:
- To what extent has SAAP V implemented the Strategic Priorities?
- What impact have the SAAP V Strategic Priorities had on client outcomes?
- How well did SAAP V achieve its aim and goals (as defined in the
Program Logic)? - What needs to happen next in SAAP?
The SAAP V Strategic Priorities are to:
- Increase involvement in early intervention and prevention activities.
- Provide better assistance to people who have a number of support needs.
- Provide ongoing assistance to ensure stability for clients post crisis.
The SAAP V Program Logic identifies program inputs, processes, outputs and outcomes in a systematic way to illustrate the cause and effect relationships the program is seeking to achieve. SAAP V outcomes form a hierarchy of four levels and our ability to make assessments against the higher levels of the program logic outcomes is limited. Consequently, an increased use of qualitative assessment will be necessary for the SAAP V evaluation and this requirement will be supported by the development and implementation of Knowledge Management processes within the homelessness sector. The Program Logic is part of the SAAP V Multilateral Agreement. A copy of the Program Logic and can be found at http://www.facsia.gov.au/internet/facsinternet.nsf/aboutfacs/programs/house-saap_program_logic.htm
Naturally, each of the National Evaluation Framework questions (above) give rise to supplementary questions and these in turn give rise to further questions. A National Evaluation Matrix has been developed by the EWG to keep track of this cascade of questions and the sources of information required to answer them (and the information gaps that need to be addressed) before conducting the Final Evaluation.
CAD has requested AFHO to work with the EWG to develop a sector-wide consultation strategy focusing on the SAAP V National Evaluation Framework questions. The consultation strategy will be finalised during the MTR and conducted during the second half of 2008 for input to the Final Evaluation.
National Evaluation Research Agenda
The National Evaluation Research Agenda includes the following proposed projects:
- NDC data analysis by the NDCA at the AIHW to support the National Evaluation
- a scan of environmental factors that may impact on the ability of SAAP to meet its objectives
- National Sector Survey (twice) on implementation of the SAAP V Strategic Priorities
- NDC Referrals Project – looking at the effectiveness of SAAP referrals
- amend the Client Satisfaction Survey to include an Indigenous component and to consider a complementary post SAAP client survey
- AFHO to develop a consultation strategy for the sector focusing on evaluation questions
- longer term housing outcomes for SAAP clients entering Public Housing or Private Rental
- analysis of I&I Fund project evaluations for input to the final evaluation.
Please note, the EWG has identified a need to conduct a SAAP Sector Survey on the implementation of the SAAP V Strategic Priorities. It is planned to develop and administer the survey to the sector as soon as possible and to repeat the survey in early 2009 in order to provide the Final Evaluator with a comparative analysis of implementation of the Strategic Priorities during SAAP V.
SAAP V Mid-term Review (March – December 2007)
The EWG met on 7 and 8 March 2007 in Adelaide to plan and commence the SAAP V MTR. The MTR is focused on assessing whether the implementation of SAAP V is ‘on track’ and with ensuring the appropriate measures are in place to assess the program’s performance—for example, the MTR will identify gaps in the data collection and provide advice on those indicators that are not robust and require further work. The MTR will be conducted over the remainder of this year with the final report provided to CAD in December. The MTR Report will be published in early 2008 following agreement by Ministers.