Stronger Families in Australia study: the impact of Communities for Children

3. The SFIA evaluation study design

3.1 Evaluation design
3.2 Estimation methods
3.3 Measuring differential impacts of the CfC initiative
3.4 Selection of CfC and contrast sites
3.5 Sampling, recruitment, data collection and attrition
3.6 Outcome measures

3.1 Evaluation design

The intention of the CfC initiative was to bring about improvements in child, family and community outcomes at a population level; firstly by facilitating improvements in service coordination and delivery, and secondly by improving the quality of the community environment in which families live and children develop. These changes were expected to lead to better outcomes for children and their families than what would have otherwise been the case. Given the whole-of-community focus, all families with a 2 year-old child in CfC sites were targeted for inclusion in the evaluation, unlike most other evaluation studies, which have only included actual service users or clients. A consequence was that not all families had used services funded under the CfC initiative.

The key challenge in evaluating the impact of area-based initiatives such as the CfC was to estimate what the outcomes would have been for the children, their families and their community in the absence of the intervention (that is, the counterfactual). Approaches that could be used to examine these issues were categorised as those that involve the random assignment of people to an intervention (experimental design) and those that do not (quasi-experimental design). Because the allocation of funding for the CfC initiative did not involve randomisation, an experimental design was not possible.⁴ A quasi-experimental design was therefore implemented.

As discussed briefly in Section 1, the SFIA evaluation study was a longitudinal study of families with a child aged 2 years (at Wave 1) living in 10 communities in which there was a CfC program and five communities in which there was no CfC intervention (contrast sites). The contrast sites were selected to be comparable in terms of socioeconomic status to communities with a CfC intervention. The logic of the design was as follows: assuming that the outcomes in CfC and contrast sites would have been the same had CfC not been implemented, then any differences between the CfC sites and the contrast sites that occurred after the intervention could be attributed to CfC.

3.2 Estimation methods

In a matched case-control design study, the validity of all methods of estimating the impact of the intervention rests on the assumption that outcomes in CfC and contrast sites would have been the same in the absence of the CfC initiative (that is, construction of a credible counterfactual). The basic method of identifying intervention effects was to compare the child or family outcomes in CfC and contrast sites. The differences between the estimation models generally lay in the extent to which adjustments were made to ensure that the intervention and control sites were truly comparable.

In the SFIA evaluation study, two approaches were used to estimate the impact of the CfC intervention. The reasons for using both approaches are discussed below.

In the first approach, the impact of the intervention was estimated to be the difference in the outcome measures post-intervention (that is, at Wave 3). The validity of this approach depended upon there being no difference in the outcome measures pre-intervention (that is, at Wave 1). As discussed below, there were no statistically significant differences in the outcome measures between the CfC and non-CfC sites after controlling for socioeconomic and demographic characteristics at Wave 1.⁵ The similarity of the characteristics of children and families in the CfC and non-CfC sites pre-intervention supported the validity of this approach. Regression analysis was used to control for any differences in socioeconomic and demographic characteristics between CfC and contrast sites prior to the intervention.

The second approach was the difference-in-difference model. This method took account of any outcome differences between CfC and non-CfC sites pre-intervention (that is, at Wave 1). This was achieved by comparing the rate of change of outcomes in CfC and contrast sites over time. By modelling the rate of change in outcomes, differences at baseline in CfC and contrast sites were largely immaterial, assuming that scores at baseline did not affect the rate of change of the outcome variables. Perhaps the main advantage of the difference-in-difference approach was that it made adjustments for differences prior to the intervention, even if they were statistically non-significant. By comparing the rates of change in the means for the CfC and contrast groups, both group-specific and time-specific effects were allowed for (Wooldridge 2002, p. 130).

Figure 1 uses a hypothetical scenario to illustrate the logic of the difference-in-difference model for the case of a positive rate of change in the CfC site over the three waves of data collection. The rate of change in CfC and non-CfC sites at baseline (Wave 1) to Wave 2 is the same. Thereafter, the rate of change of the CfC site increases relative to the rate of change of the non-CfC site (from Wave 2 to Wave 3).

Figure 1: Hypothetical impact of the CfC initiative

Figure 1: Hypothetical impact of the CfC initiative

In this evaluation, the impact of the CfC initiative was measured only a short time after implementation and therefore any effects were likely to be small. This can make it difficult to find statistically significant effects, even though there may be positive effects of the intervention. Given these challenges, we used the two methods described above to estimate the impact of CfC. The statistical significance of the estimated effect of the CfC intervention may differ between the Wave 3 cross-section and difference-in-difference estimates. In the difference-in-difference model, small, non-significant differences pre-intervention and post-intervention may combine to produce a statistically significant change, while the Wave 3 cross-sectional model finds no difference pre or post-intervention. Alternatively, small non-significant differences pre-intervention may result in the difference-in-difference estimate being non-significant, while the post-intervention cross-section estimates are statistically significant and there is no difference pre-intervention. Further, ‘differencing’ may exacerbate attenuation bias due to measurement error (Deaton 1997).

Some of the outcome measures used to assess the impact of the CfC intervention were only used at Wave 3. In these instances, it was only possible to use the post-intervention cross-section estimates.

The validity of both estimation methods also required that there was no selective attrition between Wave 1 and Wave 3. Selective attrition occurs in longitudinal intervention studies when participants from intervention or control groups with certain characteristics drop out of the study at higher rates in a way that is systematically related to their outcomes. If this occurs, then any estimated effect of the intervention may be a function of selective attrition rather than the intervention. This issue is discussed in more detail below, but the overall conclusion was that there was no evidence of selective attrition in this evaluation.

The clustering of the sample (into 15 communities at the time of the first interview) also needed to be taken into account in the estimation procedure. It is likely that outcomes for families and their children within a community will be correlated. To the extent to which this occurs, standard errors will be underestimated and hence there is a risk of false inferences being drawn.⁶ This is because the clustering of the sample tends to result in a loss of sampling efficiency compared to that which would result if a simple random sample had been used (referred to as the design effect).⁷ The effects of the clustering of the data was taken into account in the calculation of standard errors using Stata’s survey commands (a statistical package).⁸

The technical aspects of the Wave 3 cross-sectional estimates and difference-in-difference models are now discussed in more detail.

Wave 3 cross-sectional model

For continuous variables at Wave 3, ordinary least squares (OLS) regression was used; for binary variables, logistic regression was used. Given that there was no evidence of differences at Wave 1 and no evidence of selective attrition, these estimates provided an unbiased estimate of the impact of the CfC intervention.⁹ The model estimated was:

Where:

Y_i = outcome for individual i post the CfC intervention

α= constant term

CfC = 1 if individual i lived in a CfC site pre-intervention and 0 if individual i lived in a contrast site pre-intervention

Z_i = a vector of individual family/child characteristics

e_i = random error term.

The control variables included in the regression analysis were: the gender and age of the child; whether the child was of Aboriginal or Torres Strait Islander origin; and maternal age, education and labour force status. Household income was included, as was whether at least one of the parents was born overseas. Finally, whether the father was ‘present and working’, ‘present and not working’ or ‘absent’ was included. All of these variables were measured at each of the three waves. Given that CfC and contrast sites were matched on the Socio-Economic Index for Areas (SEIFA),¹⁰ which comprises over 30 area-level variables, it was not necessary to include area characteristics as control variables.¹¹

Difference-in-difference model

The logic behind the difference-in-difference estimator was that the effect of the intervention could be obtained by estimating the difference between outcome measures for those receiving the intervention and the control group (Cobb-Clark & Crossley 2003). As a result of this estimation strategy, any baseline differences in the outcome variable would be ‘differenced out’.

As discussed in the introduction, Wave 3 was the appropriate point at which to test the effectiveness of CfC, as the implementation of CfC services and programs varied across the sites, commencing either shortly before or shortly after the Wave 2 data collection. To allow sufficient time for the CfC intervention to ‘work’, the difference-in-difference estimates were for changes between Wave 1 and Wave 3.

The Wave 1 to Wave 3 difference-in-difference (DD) estimate of the effects of the CfC intervention were given by:

Where:

= the average rate of change from Wave 1 to Wave 3 in contrast sites

= the average rate of change from Wave 1 to Wave 3 in CfC sites.

The difference-in-difference estimates included the same set of control variables as the Wave 3 cross-sectional analysis. The model estimated was:

Where:

Y_it = the outcome for individual i at time t

α= constant term

Z_it = a vector of individual family/child characteristics that are time varying

T₂ = indicator for Wave 2

T₃ = indicator for Wave 3

CfC = indicator for living in a CfC site at Wave 1

e_it = random error term that is normally distributed.

The coefficient for the interaction between the Wave 3 indicator and living in a CfC site at Wave 1 was the difference-in-difference estimate of the impact of the CfC intervention (β₆). The difference-in-difference model was estimated using respondents who participated in all three waves of the survey (the balanced panel). The balanced panel was used in order to eliminate the possibility that attrition had affected the change in outcomes in CfC and non-CfC sites and in order to provide comparability with the Wave 3 cross-sectional estimates.

3.3 Measuring differential impacts of the CfC initiative

One of the aims of the CfC service model was to improve outcomes for disadvantaged families in disadvantaged areas. However, the international evidence is that, in some cases, interventions have improved outcomes for less disadvantaged children but have not benefited the most disadvantaged (Belsky et al. 2006). Therefore, it is not only important that overall outcomes are improved, but also important to understand whether the CfC initiative has succeeded in addressing the needs of the most disadvantaged children and families in the community.

The National Evaluation of Sure Start impact study in the UK examined whether child and family outcomes varied according to the gender of the child, and whether the household was a lone-parent, teenage-parent, jobless, or very poor household. The differential impact of Sure Start on child and family outcomes was also examined for maternal employment status.

For the CfC initiative, hard-to-reach families were an important target group. For the purpose of the analysis, hard-to-reach was defined using three different methods: (1) combining a number of different hard-to-reach indicators at Wave 1; (2) whether the mother had a low level of educational attainment (Year 10 or less); and (3) whether the child lived in a low-income household (parental income was $485 per week or less).¹² Over half of the sample (60 per cent) were classified hard-to-reach as defined in this evaluation. Of households in the sample, 17.5 per cent had a mother who had Year 10 education or less and 16.0 per cent of the sample were low-income households at Wave 1.¹³

To define hard-to-reach, a combination of categories were identified from the literature and from interviews with service providers, which are described in the Engaging hard-to-reach families and children report (Cortis, Katz & Patulny forthcoming). Families in which any of the following were present at Wave 1 were defined as hard-to-reach: no father present in the household; mother unemployed or not in the labour force and father not working/not present; parental income $500 or less;¹⁴ maternal education Year 10 or less; a parent born overseas; and child is of Aboriginal or Torres Strait Islander origin. Persons in the SFIA sample who met any of these criteria were coded as ‘HTR’, and all remaining persons (who met none of these criteria) were coded as ‘not HTR’.¹⁵

The impact of CfC on each of the three subgroups was determined by running separate statistical models for each of the following groups: hard-to-reach and not hard-to-reach; low maternal education and higher maternal education; and low income and higher income groups.

Differential effects were not tested for other hard-to-reach groups, such as teenage parents, because there were too few children in these subgroups when separated into CfC and control sites to provide reliable estimates.

3.4 Selection of CfC and contrast sites

The first stage in sampling for the SFIA evaluation study was the selection of 10 CfC sites from the total pool of 45 sites in which the CfC initiative had been implemented. The CfC initiative was rolled out in three phases. As the design of the evaluation necessitated baseline (or pre-implementation) data, the sample was restricted to a small number of phase 1 sites and all phase 2 sites, as these were expected to commence in the 12 months following the first wave of data collection. To avoid respondent burden, areas that were sampled for the Longitudinal Study of Australian Children (LSAC) were excluded as well. The intention was to include one site from each Australian state and territory that was sufficiently large to meet the sampling quota.

The SFIA evaluation study was designed so that there were five contrast communities. The contrast sites were chosen from the same states and territories as the CfC sites and were also similar to the CfC sites in location, size and socioeconomic status (as measured by SEIFA). Sites were selected from the pool of possible communities considered for CfC funding, or from communities in which the Australian Early Development Index (AEDI) is administered.¹⁶

Of the five contrast sites, three were from the original list of potential CfC areas, which was developed by the Australian Bureau of Statistics, and one site was from one of the AEDI communities. The other contrast site was considered for round 2 of the AEDI trial, but was not in the final selection. These communities also met the other selection requirements; that is, they had a similar demographic profile to the CfC sites and a sufficient population size of 2 year-old children (the age of the study children in Wave 1).

The validity of the evaluation of the impact of the CfC intervention depended upon the assumption that the families residing in the contrast sites were not systematically different on the key outcome variables to the families residing in CfC sites after appropriate statistical controls had been introduced. We found that the contrast sites did provide a valid counterfactual and these results are discussed in detail in Section 4.

3.5 Sampling, recruitment, data collection and attrition

The sampling unit for the SFIA evaluation study was 2 year-old children from families receiving Family Tax Benefit (FTB) Part A or B in 2005 in CfC and contrast sites, identified by postcode. The sample was obtained from the administrative database for FTB.¹⁷ A total of 6,537 records from the FTB database formed the sampling frame, which included 105 additional records for one particular site. Of this number, 1,311 records were not allocated to interviewers because they did not fit the sampling specifications.¹⁸ Thus, the final sample for Wave 1 data collection was 5,226.

In April 2006, families in the selected sample were sent a Primary Approach Letter from FaHCSIA, followed by a Letter about Fieldwork sent by the fieldwork agency, I-view, which explained the study and indicated that an interviewer from I-view would be contacting the family to make an appointment for an interview.

The fieldwork agency was required to make up to six telephone call attempts or visits to the contact address to attempt initial contact with the family.¹⁹ After initial contact with the family was achieved and an appointment scheduled, interviewers were required to make at least three attempts to conduct the interview with Parent 1.

The longitudinal study involved face-to-face interviews with the person in the family who knew most about the child (Parent 1). In most cases, this was the mother. In a small number of cases, Parent 1 was the child’s resident father, a relative who was the primary carer for the child or a foster parent. In Wave 3, the study child was required to be present during the interview to undertake the objective measures (anthropometric measures and a language and vocabulary test).

In the first wave, 3,379 families were approached and 2,202 families were interviewed. The final sample of 2,202 families in Wave 1 represented 42 per cent of the total population of eligible families (n=5,226). Contact was not continued with the remaining 1,847 families once this quota had been achieved. Thus, the response rate was 65 per cent.

At Wave 2 and Wave 3, a similar methodology was used to re-interview study families. Interviewers were required to make up to six telephone call attempts or visits to the contact address to attempt initial contact with the family. Once contact was achieved, interviewers made at least three attempts to conduct the interview. Telephone interviews were conducted with families that had moved more than 400 kilometres from any CfC or contrast site. Postal and web-based communication was used to keep in touch with families between Waves 1 and 2 and between Waves 2 and 3.

The Wave 2 fieldwork response rate was 91.5 per cent. The main sample loss was 4.3 per cent refusal to interviewer, and 4.2 per cent unable to locate and/or contact Parent 1 and/or make a successful contact with Parent 1. At Wave 3 the fieldwork response rate was 90.3 per cent. The main sample loss was 1.8 per cent refusal to interviewer, and 7.9 per cent unable to locate and/or contact Parent 1 and/or make a successful contact with Parent 1. The fieldwork sample at each data collection wave, the number of interviews completed and the response rates are presented in Table 1.

Table 1: Sample, interviews and response rates across the three waves of data collection
	Total sample	Total interviews	Response rate (%)
Wave 1	3,379	2,202	65
Wave 2	2,202	2,026	92
Wave 3	2,034^(a)	1,836	90

(a) Includes eight non-participating families in Wave 2 who contacted I-view in Wave 3.

Source: SFIA.

As with any longitudinal study, not all families who were involved at Wave 1 participated in subsequent waves. As mentioned above, if the families who dropped out of the study differed systematically from those who remained in the study in ways that were related to their outcomes, then any estimated effect of the intervention may have been biased.

The extent to which sample attrition is selective can be determined by comparing the between-waves response rates for key demographic groups of families in CfC and contrast sites. Determining whether the rates of attrition vary for CfC and contrast sites on baseline outcome scores is another approach in examining selective attrition. For example, it may be that a positive difference observed at Wave 3 is the result of greater non-participation among low-scoring families in CfC sites than in contrast sites.

The approach adopted here was to assess whether response rates²⁰ differed systematically by site (contrast compared to CfC sites) for control variables and key outcome variables. There was no association between Wave 3 attrition and Wave 1 control and outcome variables when a logistic regression was used to model the probability of not responding to Wave 3. There was also no evidence of selective attrition—Wave 1 control and outcome variables were not more highly associated with non-response at Wave 1 for families residing in CfC compared to contrast sites (see Appendix A for details). Consequently, any differences in terms of the outcomes that were observed between CfC and contrast sites at Wave 3 were not a function of selective attrition of families between Waves 1 and 3.

The first wave of the study commenced in 2006. Data collection for the first wave covered the period 7 June to 31 August 2006. The data collection period for Wave 2 covered the period 7 March to 8 July 2007. In Wave 3, the fieldwork start date was 6 February 2008 and the completion date was 31 May 2008.

In Wave 1 and Wave 2, administration of the survey was paper-based. In Wave 3, computer-assisted personal interviewing (CAPI) was used. CAPI has the added advantages of combining data collection with date entry, and helps to eliminate errors in data recording. CAPI interviews are a standard data collection technique in social science fieldwork. The move to CAPI did not adversely affect response rates, or introduce bias in responses due to a change in the mode of administration. Moreover, any effects of CAPI would be expected to impact on responses from CfC and contrast sites equally.

3.6 Outcome measures

Nineteen primary outcome variables were modelled for the purposes of evaluating the impact of the CfC initiative. These were outcomes from the CfC intervention that were expected to be related to the overall objectives of the SFCS: healthy young families, supporting families and parents, early learning and care, and child-friendly communities. A detailed description of these measures, including internal consistency reliabilities (Cronbach’s α) for all scaled measures (that is, derived from multiple variables) are presented in Appendix B. Overall, the scaled measures demonstrated good internal consistency reliability. Parent report measures of children’s emotional and behavioural functioning and parenting hostility were also correlated with observer ratings of these variables.

Healthy young families

Five child and two parent physical and socio-emotional outcomes were used to assess ‘healthy young families’. The child outcomes included:

the number of child injuries requiring medical attention in the past 12 months
parental reports of child physical functioning, as assessed using the 8-item Pediatric Quality of Life Inventory (PedsQL) (Varni et al. 2003)
child emotional and behavioural problems, as assessed using the 20-item Total Difficulties scale of the Strengths and Difficulties Questionnaire (SDQ) (Goodman 1997)
child prosocial behaviour, as measured by the 5-item Prosocial subscale of the Strengths and Difficulties Questionnaire (Goodman 1997)
child overweight, determined by using the child’s length (or height), weight and girth to calculate body mass index (BMI), measured at Wave 3 only.

The parent outcomes included:

a global rating of parent physical health
parent mental health, as measured using the 6-item Kessler-6 scale (K-6) (Kessler et al. 2003).

Supporting families and parents

Four measures were included in the ‘supporting families and parents’ domain. Two of these were parenting measures:

parent self-ratings of hostile/irritable parenting, sourced from LSAC and the National Longitudinal Survey of Children and Youth (Statistics Canada & Human Resources Development Canada 1995)
a single-item measure of parenting self-efficacy.

Parental relationship conflict, assessed through a 5-item scale of the frequency of verbal and physical arguments derived from the LSAC study, and living in a jobless household were the other two measures.

Early learning and care

In relation to ‘early learning and care’, children’s receptive vocabulary achievement and verbal ability were assessed using the LSAC Short-Form of the Peabody Picture Vocabulary Test (PPVT) at Wave 3. The quality of the home learning environment was based on a 4-item scale developed for LSAC.

Child-friendly communities

Finally, ‘child-friendly communities’ was assessed in terms of:

support in raising children from partner/spouse, partner’s/spouse’s parents, own parents, other family, friends, neighbours, and community organisations, as used in LSAC
a single item measuring parent involvement in community service activities, such as volunteering at school or coaching a sports team
parent assessment of the neighbourhood as a place to bring up children
community social cohesion, measured on a 5-item scale relating to parent perceptions of trust, reciprocity and shared values existing between individuals in the local community (Sampson, Raudenbush & Earls 1997)
community facilities, in terms of access to basic services, shops, public transport and parks, as derived from LSAC
a measure of unmet service need, developed for the study, which assessed whether respondents had difficulty accessing developmental, support and adult mental health services if needed.

[ top ]