National evaluation (2004–2008) of the Stronger Families and Communities Strategy 2004–2009
3. Methodology
3.1 Introduction
3.2 Stronger Families in Australia Study (CfC)
3.3 Outcome Indicators Framework (CfC)
3.4 Community profiles (CfC)
3.5 Service mapping (CfC)
3.6 Service Coordination Study (CfC)
3.7 Partnership Model Study (CfC)
3.8 Costs and effects analysis (CfC)
3.9 Evaluation reports analysis (ItG)
3.10 Progress reports analysis (CfC/LA/ItG)
3.11 Themed studies (CfC/LA/ItG)
3.12 Promising practice profiles (CfC/LA/ItG)
3.1 Introduction
The methodology for the Stronger Families and Communities Strategy (SFCS) national evaluation was developed in consultation with FaHCSIA and outlined in the national evaluation framework (Stronger Families and Communities Strategy National Evaluation Consortium 2005). As already mentioned, the methodology was designed to make the national evaluation both formative (contributing to the development and refinement of policy and practice), and summative (addressing implementation and effectiveness). The evaluation sought to measure outcomes and relate them to the inputs, processes and outputs. It asked ‘what components of the model work?’, ‘why do they work?’ and ‘under what circumstances?’ The CfC component of the evaluation also examined for whom the early intervention works.
This section outlines the methodologies employed in the various evaluation strands and substrands. Further details about the methodologies can be found in the web appendixes relating to each strand of the evaluation. Methodological limitations are described in Appendix A at the end of this document.17
[ top ]
3.2 Stronger Families in Australia Study (CfC)
The primary component of the CfC outcomes evaluation was the Stronger Families in Australia (SFIA) study. It was a longitudinal, quantitative survey of families living in 10 CfC sites and five comparable communities (non-CfC sites known as contrast sites). The contrast sites provided the control group against which the impact of the CfC intervention could be estimated.
The survey followed 2,202 children (aged 2 years at the time of the first wave of the survey) and their families over three periods of time (at the beginning of the CfC in 2006, and again in 2007 and 2008). The findings from this component of the evaluation are available in a separate report (Edwards et al. 2009) and are summarised below.
[ top ]
3.3 Outcome Indicators Framework (CfC)
The Outcome Indicators Framework examined changes in the wellbeing of children, families and communities over the course of the SFCS at the community level (rather than the individual level) using secondary data from a range of sources.18
The Outcome Indicators Framework was initially set up to measure community level change from 2004 to 2008, but because of a serious lack of pre-existing indicator data, the Outcome Indicators Framework established a baseline. As CfC aims to achieve long-term results (like Sure Start in the United Kingdom), the Outcome Indicators Framework puts in place some baseline data to monitor long-term community level achievements. The Outcome Indicators Framework report presents 77 indicators representing CfC outcome domains of interest—early learning and care, healthy young families, supporting families and parenting, and child-friendly communities.19
[ top ]
3.4 Community profiles (CfC)
The national evaluators created comprehensive profiles of all 45 CfC communities. These included producing demographic profiles and mapping child and family services which pre-dated CfC. These profiles were created using customised and publicly available data from the Australian Bureau of Statistics (ABS) Census of Population and Housing, the results of the Australian Early Development Index (where it was available) and other relevant data. This data provided contextual information about the social and economic characteristics of CfC sites, and the demographic characteristics of children aged 0 to 5 years and their families. Comparing these data across sites helped to highlight the wide variety of contexts in which CfC was implemented.
[ top ]
3.5 Service mapping (CfC)
Service mapping involved compiling lists of existing services for 0 to 5 year olds and their families in all 45 CfC sites. Two rounds of service mapping were conducted for the national evaluation. The first round established a baseline of the range of services and initiatives in place prior to the establishment of local CfC initiatives, or early in their implementation. The second round tracked changes during the life of the program. The number and types of services were aggregated for this component of the evaluation—they were not spatially mapped. In addition, data from the service mapping exercise informed the Service Coordination Study.20
[ top ]
3.6 Service Coordination Study (CfC)
The logic model of CfC indicated that outcomes for children and families partly depend on how services and activities are coordinated. Therefore, the national evaluation included a Service Coordination Study, which aimed to measure the extent to which service agencies in CfC sites worked together and also the manner in which they collaborated.
The study consisted of two components: fieldwork interviews and a snapshot survey.21 The snapshot survey was completed by agencies in 41 CfC sites providing services for children aged 0 to 5 years and their families in 2006 and 2008.22 Survey results are based on 744 responses (442 in Wave 1 and 302 in Wave 2) from stakeholders within agencies in the contrast sites, those not funded by CfC, as well as those that were.23
The fieldwork involved visits to 10 CfC sites24 to conduct in-depth interviews with CfC key personnel in 2006 and late 2007. In total, 222 interviews were conducted (97 in Wave 1 and 125 in Wave 2) with Facilitating Partner and Community Partner staff, CfC Committee members, FaHCSIA state and territory offices, local evaluators, local government personnel, interagency group chairpersons, and other relevant community members.
[ top ]
3.7 Partnership Model Study (CfC)
The Facilitating Partner model used in the CfC program was an indirect way of funding community services. Rather than entering directly into contracts with service providers (as in the LA program), the government funded local non-government organisations (NGOs), the Facilitating Partners, as brokers in their sites to engage the community in CfC processes and services. As part of their role, Facilitating Partners subcontracted other NGOs to provide services. This made the Facilitating Partners intermediaries between government and the community sector.
The evaluation sought to ascertain how useful the Facilitating Partner model had been for improving and coordinating services in the CfC sites and for achieving outcomes. It also sought to determine how effectively FaHCSIA and the Facilitating Partner organisations managed the model’s implementation. These questions were investigated during two rounds of the intensive fieldwork with 222 stakeholders in the 10 CfC sites.
[ top ]
3.8 Costs and effects analysis (CfC)
The CfC costs and effects analysis assessed how effectively resources had been used by comparing the goals of the initiative with its outcomes and costs. It established whether the money spent on the CfC initiative produced tangible benefits.
The analysis used the data from the outcomes evaluation25 and financial management data obtained from FaHCSIA to provide information about the value added by the initiative. Outcomes were measured at Wave 3 of SFIA, and through changes over time in the CfC locations compared to similar areas where the initiative did not take place. Effectiveness was assessed in terms of the relative size of the change in outcomes compared to locations without CfC.
[ top ]
3.9 Evaluation reports analysis (ItG)
The national evaluators analysed 22 evaluation reports prepared by ItG local evaluators.26 The analysis is included in Web Appendix B27 and is summarised in this report. The evaluation reports describe ItG projects, the factors facilitating project implementation, challenges faced by the projects, and outcomes which had been achieved.
[ top ]
3.10 Progress reports analysis (CfC/LA/ItG)
The progress reports analysis was based on output data collected by SFCS services in their reports to FaHCSIA. All 45 CfC sites and all organisations funded under Local Answers (LA) and Invest to Grow (ItG) were required to complete progress reports. These reports covered information such as the type of activity, the number of participants, allocation and expenditure of finances, partnerships, success factors and outcomes. The quality of this data varies considerably between sites.
This report includes analysis of CfC, ItG and LA progress reports. The CfC data is based on a total of 641 CfC-funded activities (up to December 2007). Progress report data is available on 22 of the 26 ItG projects, which delivered more than 36,000 occasions of service. The LA database includes information on 360 projects, which provided over 120,500 occasions of service.
[ top ]
3.11 Themed studies (CfC/LA/ItG)
The national evaluation included three themed studies, which explored particular topics in greater depth to determine why and how things work.
The Indigenous families and children: coordination and provision of services study examined the impact of SFCS on service provision and coordination in communities with a high proportion of Aboriginal and Torres Strait Islander children. This investigation comprised a literature review, 25 telephone interviews with people working in 12 CfC communities, 125 interviews with CfC stakeholders in 10 sites (additional questions were added as part of the 2007 fieldwork), two focus groups, document analysis of ItG evaluation reports, and additional analysis of the SFIA data (Flaxman, Muir & Oprea 2009).
The Engaging hard-to-reach families and children study explored how, and how effectively, CfC, LA and ItG projects engaged children and families that may be considered hard-to-reach. The study also investigated the challenges encountered and how additional supports might enhance engagement. The methodology included a literature review, document analysis and 20 interviews with key stakeholders in a sample of 14 ItG and six LA projects. Additional interview data was also collected during the 125 interviews conducted in 10 CfC sites in 2007 (Cortis, Katz & Patulny forthcoming).
The Engaging fathers in child and family services: participation, perceptions and good practice study explored how SFCS services managed to engage the fathers of young children, and the effectiveness of services in assisting this client group. The study investigated father participation in selected SFCS programs and identified successful strategies for engaging with fathers. The research included survey responses from 59 SFCS program managers about father engagement, 17 interviews with service managers and facilitators across seven projects, and seven focus groups with a total of 34 fathers (Berlyn, Wise & Soriano 2008).
[ top ]
3.12 Promising practice profiles (CfC/LA/ItG)
One of the key objectives of the national evaluation was to identify what works in early childhood and community development, and to disseminate some of these practices. The promising practice profiles (PPP) aimed to provide a resource for practitioners, policy makers and researchers working in the early childhood sector, by providing access to information about examples of effective practices in different settings. The PPPs aimed to assist in the planning of appropriate programs, provide a vehicle for peer learning and information sharing, and enhance the quality of services provided to families and communities.
A validation process identified promising practices in 57 CfC, LA and ItG programs through a call for proposals over three rounds. All proposals received were subject to a semi-blind independent review process. The PPPs will be published online in 2008 and 2009. A summary report of the PPP findings is also available (see Soriano, Berlyn & Wise 2008).
[ top ]