Skip to content

National evaluation (2004–2008) of the Stronger Families and Communities Strategy 2004–2009

Appendix A: Methodological limitations

The methodology of this evaluation was designed to address the particular circumstances of the Stronger Families and Communities Strategy (SFCS) 2004–2009 relevant at its inception. Inevitably there are a number of limitations for such a complex enterprise. This appendix contextualises the evaluation methodology, and explains some of its challenges and limitations.

Cross-strategy evaluation

The evaluation request for tender stipulated a meta evaluation of the whole SFCS 2004–2009, and to this end the national evaluation has included a cross-strategy evaluation which was intended to address the SFCS 2004–2009 as a whole. However, although it is possible to comment to some extent on all the strands, it was not possible to conduct a meta-level analysis because the SFCS 2004–2009 did not operate as a coherent whole. Structurally, the three components—CfC, ItG and LA—were managed in different parts of FaHCSIA, and there was no overarching level of management. On the ground, there were some sites where CfC interfaced with LA or ItG, but there was no systematic coordination. In essence, these three strands should be seen as three separate funding streams, which addressed similar issues but which did not constitute three components of a holistic strategy.

Service user perspective

Normally an evaluation of this type would collect information about the services from the perspective of the service users as well as the service providers. It would have considerably strengthened the methodology if it had been possible to report on families' perspectives of the quality of CfC services, on how well they felt their needs were being met, and on whether the community was becoming more child-friendly. To some extent these issues are addressed in Stronger Families in Australia (SFIA), but this survey did not identify the individuals who used CfC-funded services or activities. There are no CfC-specific service user perspectives in the evaluation, except for the Engaging fathers in child and family services study. This is partly because of budget constraints, but it was also felt to be inappropriate for the national evaluation to conduct fieldwork in areas where local evaluations were already operating. The potential for confusion and overburdening people was very great and so we decided against this approach. As we report below, the extent to which local evaluations collected data from service users is not yet clear.

Minimal data on Local Answers

LA receives less attention in this evaluation in comparison with the other programs because its evaluation was allocated fewer resources within the SFCS 2004–2009 budget. This was a deliberate decision by FaHCSIA at an early stage of the evaluation. CfC was seen as a new way of delivering services, and it was decided to focus evaluation resources on this program, while ItG was given considerable resources for the local evaluations. Moreover, LA was an evolution of the SFCS 2000–2004 which was already being evaluated. In addition, the range and scale of LA projects meant that it was too diverse to be evaluated meaningfully in terms of outcomes. Finally, unlike the other two programs from SFCS 2004–2009, LA projects were not exclusively focused on the early years. They covered projects dealing with the whole range of children and young people.

Outcome indicator framework data

The outcome indicators framework was a challenging area for data reliability and relevance. This component of the evaluation was new and innovative, drawing on secondary data to measure outcomes within the communities and compare them to other similar areas. Before SFIA became the main outcome measurement of the evaluation, the outcome indicators framework was intended to be the principal vehicle for measuring community-level outcomes. It represents a very promising way of measuring outcomes in local areas-far cheaper than conducting primary research, and offering the potential to measure changes over much longer periods. An effective indicator framework could also help relieve the burden on service providers in some areas.

However, at the time of writing, the quality of local area data in Australia could only be described as appalling. Although there have been continuous improvements, it will be a long time before relevant data will be collected and reported systematically at a local area level. Even where there are local data available, these are reported using different units of analysis (postcodes, statistical local areas), which do not map easily onto CfC sites, and therefore provide limited information about changes in CfC sites compared to other areas.

In addition to the quality issue, another problem was the timescales for these data. Most administrative data are reported two years in arrears, so data published in 2008 generally refers to services provided in 2006. This means that we do not have any administrative data which can report reliably on outcomes in CfC communities, and will have to wait to 2010 for such data to be available. Furthermore, the latest census was conducted in 2006, in the middle of the SFCS 2004–2009 initiative, and therefore only in 2011 will census data be able to be used to measure outcomes.

Progress reports

The progress reports analysis was based on output data collected by SFCS services in their reports to FaHCSIA. Data were analysed quantitatively by Social Policy Research Centre researchers. All 45 CfC sites and all organisations funded under LA and ItG were required to complete progress reports.

Service providers and Facilitating Partners spent considerable time and effort completing progress reports, and FaHCSIA state and territory offices oversaw and supported this process. The reporting templates that Facilitating Partners and Community Partners were required to complete were complex and detailed. FaHCSIA changed the content and format of these templates in an effort to ease the reporting burden and complexity. While service providers appreciated FaHCSIA's responsiveness to their concerns, overall the reporting requirements were generally described by Facilitating Partners and Community Partners as challenging.

The reporting template used collected large amounts of qualitative and quantitative data. The reports covered information such as the type of activity, the number of participants, the allocation and expenditure of finances, partnerships, success factors and outcomes. The quality of the data varied between organisations and sites. Where data was of limited quality (such as large numbers of missing data or inconsistencies between questions and responses), it was not used in the evaluation.

The difficulties experienced in regard to the administrative dataset were not unusual. Many similar evaluations of complex initiatives such as SFCS 2004–2009 report similar findings, and some have failed altogether to use information provided by practitioners and service providers.

Government-funded programs require some level of reporting for contract management and evaluation purposes. Collecting that data in a consistent format, such as a database, from different organisations is important for the data to be useful. Lessons can be learnt from the SFCS 2004–2009 reporting template for collecting useful administrative data in future funded government programs.

The progress reporting from SFCS 2004–2009 reinforced the importance of keeping administrative reporting data indicators to a minimum, ensuring that questions are simple and all people completing them have a sound understanding of terminology used, limiting the collection of qualitative data to case studies and explanations for quantitative responses, and providing ongoing technological support and quality assurance mechanisms during the compilation of the reports.

Despite some challenges, analysis of the 641 CfC-funded activities and 360 LA projects included in the progress reporting databases was conducted. Some practical examples and case studies from the CfC progress reports have also been included throughout this report to provide insight into some of the practices occurring in communities.

Department of Finance and Administration review

The original design of the SFCS evaluation (2004–2008) was predicated on the fact that the program was due to be reviewed by the then Department of Finance and Administration (now Department of Finance and Deregulation) in September 2007. As such, it was originally planned that both rounds of the process evaluation, as well as the first two waves of SFIA, would be completed by early 2007 so that evaluation data could feed into this review. This, and the delays in accessing the data for the sample, considerably shortened the timescales for the intervals between SFIA waves (these issues will be reported on in more detail in the Stronger Families in Australia: the impact of Communities for Children report). When it was learned that the Department of Finance and Administration review would not take place, the second wave of fieldwork for the process evaluation was put back to late 2007 to allow for longer timescales between waves and for the program to mature further. Nevertheless, in a number of locations, the initiative still had 18 months to run when the second wave of fieldwork was undertaken, so these findings should not be seen as representing the fully mature CfC.

[ top ]