Stronger Families in Australia study: the impact of Communities for Children
- Previous: 4. Validity of contrast sites as the counterfactual
- Next: 6. Summary of the effectiveness of the Communities for Children initiative
5. Estimates of the impact of the CfC initiative on child, family and community outcomes
5.1 Overall effects of the CfC intervention
5.2 How large were the effects of the CfC intervention?
5.3 Differential effects of the CfC intervention
on specific subpopulations
This section presents the estimates of the overall effects of the CfC intervention on the 19 outcome variables for the full sample, and the extent to which there were differential impacts of the CfC initiative for hard-to-reach families, and families with a low level of maternal education and low income.
In order to aid the interpretation of the estimated impact of the CfC initiative for the various child, family and community outcome measures, the estimated effects are presented as a standardised effect size.22 Effect sizes can readily be used to assess the magnitude of the effect of the CfC intervention. A general rule of thumb is that effect sizes between 0.1 and 0.2 are ‘small’; effect sizes between 0.3 and 0.5 are ‘medium’ and effect sizes 0.6 and above are ‘large’ (Cohen 1988). However, these rules of thumb are very broad generalisations (Hill et al. 2007) and a more considered interpretation of effect sizes is discussed in more detail in Section 5.2. Despite these cautions, effect sizes are useful for interpreting the policy significance of differences between CfC and contrast groups. For example, a one standard deviation difference (an effect size of 1.0) between CfC and contrast sites represented a 34 per cent improvement in the mean of the whole population in the CfC site.
5.1 Overall effects of the CfC intervention
This section describes the estimated effects of the CfC initiative on the 19 outcome variables (Table 3). As discussed in Section 3.1, we present effect sizes based on difference-in-difference models, and regression estimates of the difference in outcomes at Wave 3. Some outcome variables were only collected at Wave 3 and consequently only regression estimates at Wave 3 could be modelled. Detailed results from each statistical model are provided in Appendix E.
(a) Only collected at Wave 3.
Note: * p<0.05 (statistically significant difference between the CfC and non-CfC sites at the 5 per cent confidence level). † p=0.06.
Source: SFIA.
Hostile parenting
By Wave 3, parents living in CfC sites reported lower levels of hostile or harsh parenting than parents in contrast sites. At Wave 1, levels of hostile parenting were higher, but non-significant. However, by Wave 3, levels of hostile parenting had increased for parents in the contrast sites but had stayed relatively constant in CfC sites. The Wave 3 regression estimates were statistically significant.
Parenting self-efficacy
By Wave 3, parents living in CfC sites reported higher levels of parental self-efficacy—self-confidence in their ability to parent effectively—than parents in contrast sites. For parents in both CfC and contrast sites, parenting self-efficacy decreased over time; however, for parents in contrast sites, the decline in parenting self-efficacy was much greater than in CfC sites, and by Wave 3 the regression estimates suggest that this gap was statistically significant at the p<0.06 level.
Jobless households
A significant negative effect was found for living in a jobless household, meaning children in CfC sites were less likely to be living in a jobless household at Wave 3, based on both the difference-in-difference and the regression estimates. Pre-intervention, the proportion of children living in a jobless household was higher for families living in contrast sites (21.0 per cent at Wave 1) than in CfC sites (17.0 per cent at Wave 1), but these differences were not statistically significant. At Wave 3, the proportion of children in contrast sites living in a jobless household was 20.0 per cent—very similar to that at Wave 1. However, in CfC sites, the percentage of jobless households had decreased to 13.2 per cent.
Simply, children who lived in CfC sites were 66.0 per cent less likely to live in a jobless household at Wave 3 than children who lived in contrast sites, using estimates from the regression analysis (odds ratio=0.66, p<0.05). The estimates from the difference-in-difference estimate were somewhat smaller, but still statistically significant (odds ratio=0.76, p<0.05).
Although there was a fairly large rate of joblessness in one contrast site (33.8 per cent in Wave 3), the differences between CfC and non-CfC sites were still robust (see Appendix F). When the proportion of jobless households in this contrast site was taken into account in the regression estimates, the gap in the rate of jobless households at Wave 3 dropped from 6.8 per cent to 5.5 per cent, but was approaching statistical significance at p=0.07 (t=–1.97, p=0.07). Moreover, the difference-in-difference estimates were still statistically significant (odds ratio=0.73, t=–2.74, p<0.05).
Child physical health
Children residing in CfC sites had significantly lower reported physical functioning than children in contrast sites at Wave 3, based on regression estimates, but not for the difference-in-difference estimate. At Waves 1 and 2, there were no significant differences between the mean level of child physical health in the CfC and contrast sites, but by Wave 3 there was a greater increase in mean levels of child physical functioning in contrast sites than in CfC sites.
Summary
While these results point to impacts of the CfC initiative on specific outcomes, when the findings on all outcomes were considered, both regression and difference-in-difference estimates suggested an overall trend in the data towards positive effects of the CfC intervention. Given the evaluation was examining short-run effects of an area-based intervention that did not directly impact on all families in the CfC communities, we would expect that many effects of CfC would be small and imprecisely estimated. Although there is a possibility that non-significant findings in a positive direction could be driven by chance variation, the pattern of results in Table 3 also suggests that the CfC intervention was generally positive.23 Fourteen of 19 regression estimates and 9 of 15 difference-in-difference estimates indicated better outcomes in CfC sites compared to contrast sites. Moreover, the pattern of estimates that suggested positive outcomes of the CfC intervention were not evenly dispersed across the four priority areas. The two estimation methods also provided a consistent picture. As can be seen from Table 3, effect sizes estimated using difference-in-difference models were consistent but generally smaller than those using OLS or logistic regression at Wave 3.
5.2 How large were the effects of the CfC intervention?
While there is evidence that the CfC intervention had a significant, positive impact on some child and parenting outcomes, it was important to consider the size of the effects as well (see Table 3). One way of assessing the size of the impacts was to compare the findings to those found in comparable studies.
Using Cohen’s (1988) criteria (see above), the effects observed would be considered small. However, as mentioned earlier, Hill and colleagues (2007) have argued that Cohen’s criteria are very broad generalisations. They suggest that in order to evaluate the practical significance of findings, the nature of the intervention, the target population and the outcome measures should be considered.
As discussed earlier, the Sure Start Impact Evaluation therefore provides a useful benchmark to evaluate the practical significance of the current CfC findings. The early impact intervention (referred to here as Sure Start [Early Impacts]), conducted three years after the implementation of Sure Start, is the most appropriate comparison point to the CfC initiative. In both cases, programs and services were not particularly well established and the study children had been exposed to changed conditions for a relatively short time during the preschool years. However, in the later impact study of Sure Start (referred to here as Sure Start [Later Impacts]), the Sure Start Local Programmes had been running for at least five years and study children had experienced changed conditions for their entire lives.
Table 4 provides effect size estimates for the SFIA evaluation study for the regression estimates at Wave 3 and the difference-in-difference estimates, and Sure Start (Early Impacts) and Sure Start (Later Impacts) for common outcomes. Estimates from the Wave 3 cross-section regressions from the SFIA evaluation study were the most comparable to the Sure Start findings, as Sure Start estimated impacts by comparing intervention and control groups at one time period (that is, did not use difference-in-difference estimates). Table 4 shows that the effect sizes for Sure Start (Early Impacts) were small, with the exception of parental hostility, which had a medium effect size. In this instance, however, parents in Sure Start were engaging in more hostile/harsh parenting. Table 4 also shows that effect sizes for the SFIA evaluation study were similar or greater in magnitude than Sure Start (Early Impacts), although it should be noted that there was a negative effect on parental mental health in the SFIA evaluation study, whereas Sure Start (Early Impacts) reported a positive effect on this outcome.
Although there was no directly comparable measure of community effects, the Sure Start (Early Impacts) evaluation suggested that the effect size of community outcomes was small (d=0.09). Although impacts on community outcomes were small overall, in the SFIA evaluation, there was a small to medium impact on maternal involvement in a community service activity (d=0.19) (effect sizes for Early Impacts were derived from Belsky et al. 2006).
Notes: Effect sizes for Sure Start (Early Impacts) were derived from Belsky et al. (2006); for Sure Start (Later Impacts), from the National Evaluation of Sure Start Team (2008). In Sure Start (Early Impacts) and Sure Start (Later Impacts), the estimates were based on families with 3 year olds. Odds ratios were converted to effect sizes using a method developed by Chinn (2000).
The effect sizes for Sure Start (Later Impacts) were also small, with the exception of harsh parenting practices, which had a medium effect. Compared to the CfC intervention, the evaluation of the Later Impacts of Sure Start was more effective at reducing the number of injuries requiring medical attention and harsh parenting practices. Sure Start (Later Impacts) had similar size effects to the CfC intervention with respect to children’s mental health and behavioural problems and prosocial behaviour, although the effects for CfC were more promising overall (in the case of both interventions, the effects were small). The CfC intervention also had a larger effect on children’s receptive vocabulary achievement and verbal ability than Sure Start (Later Impacts). Again, the evaluation of Sure Start (Later Impacts) for the quality of community was small (d=0.03) and comparable to findings from the SFIA evaluation study, with the exception of involvement in a community service activity (d=0.19).
5.3 Differential effects of the CfC intervention on specific subpopulations
This section describes the analyses designed to test whether there were specific subgroups of children or families who were differently affected by the CfC initiative. Differential effects were examined for hard-to-reach families, defined by a number of categories derived from the literature and from interviews with service providers and described in the hard-to-reach themed study report (Cortis, Katz & Patulny forthcoming). Two subpopulations of hard-to-reach families were also analysed separately: children of mothers with low education at Wave 1 (Year 10 or less), and children in low-income households at Wave 1 (where the combined parental income was in the lowest 20 per cent of the distribution at Wave 1, or $485 per week or less). Low maternal education and low income were chosen because: (a) there were sufficient numbers to conduct robust analyses; and (b) low income and low maternal education have been found to be associated with poorer child outcomes (for example, Morris, Duncan & Rodrigues 2004).
Differential effects according to maternal education, household income and hard-to-reach status are presented in Tables 5 to 7. Both the difference-in-difference and Wave 3 regression estimates are presented using effect sizes. Effect sizes in each column refer to the impact of CfC for a particular subpopulation; that is, the impact of CfC for low-education groups, which was produced by comparing low-education groups in CfC sites with low-education groups that were not in CfC sites. The estimates were not the impact of CfC for low-education compared with high-education groups.24
Statistically significant differences between CfC and contrast sites were found for each of the three subgroups (differential effects). However, these varied by whether a difference-in-difference or a regression model was employed. On balance, there was more evidence of statistically significant differential effects of the CfC intervention for the hard-to-reach, low maternal education and low-income groups with the difference-in-difference estimates. As the difference-in-difference model was comparing the average rates of change in the outcomes from Wave 1 to Wave 3 in CfC and contrast sites, small but statistically non-significant differences at Wave 1, coupled with small but not statistically significant differences at Wave 3, when taken together, may lead to statistically significant estimates of the impact of CfC that were not evident from Wave 3 cross-sectional regression estimates.
Low level of maternal education
The overall pattern of results suggested that, on balance, there were some positive effects of the CfC initiative, both for families in which the mother had a low level of educational attainment, and for families in which the mother had a higher level of educational attainment. There would appear to be stronger positive effects for households in which the mother had an educational attainment of Year 11 or higher than where the mother had not achieved Year 11 education.
The Wave 3 cross-section regression estimate suggests that, for a mother with low education, the CfC intervention had positive impacts on their child’s receptive vocabulary and verbal ability. The standardised effect size on this group of children was ‘medium’ (d=0.47).25 The Wave 3 cross-section regression estimate also suggested that CfC had a positive effect on involvement in community service activity for low-education mothers (d=0.39). Although this effect did not reach statistical significance, the difference-in-difference estimate was also in the same direction (d=0.21). The difference-in-difference estimate suggested that for low education families, the CfC intervention reduced the rate of household joblessness (d=–0.26, p<0.05). The Wave 3 cross-sectional regression estimate was similar but not statistically significant (d=–0.25, p=0.11).
In contrast, the difference-in-difference estimate suggested that the CfC intervention had a negative effect on the mental health of low-education mothers. This finding appears to be anomalous, given the other findings. The Wave 3 cross-section estimates did not find that CfC had a negative impact upon the health of low-education mothers (the effect was one-fifth the size of the difference-in-difference estimate, and it was not statistically significant at the 90 per cent confidence level). Although there is some doubt about the robustness of this finding, it is plausible that greater contact with services resulted in more diagnosis of health problems or generally greater awareness of maternal health. Similarly, there was a negative finding on reported physical health among children with mothers with Year 10 education or less.
In relation to mothers with Year 11 or higher educational attainment, two of the estimates suggest improved outcomes from the CfC initiative. First, there was evidence that the CfC intervention reduced the rate of household joblessness for this group when estimated using the difference-in-difference model. Although not statistically significant, the estimates from the Wave 3 cross-sectional regression model were in the same direction and magnitude as the difference-in-difference estimates. Second, for mothers who had a higher level of education, the Wave 3 cross-sectional estimates suggested that the intervention reduced levels of hostile or harsh parenting (at the 92 per cent confidence level). In contrast, there was no statistically significant difference between levels of hostile parenting of mothers with a lower level of education in the CfC and contrast sites. Finally, the Wave 3 cross-sectional estimates suggested that the CfC intervention may have had a negative impact on child physical functioning, irrespective of maternal education.
Taken together, the evidence from the overall pattern of estimates from the Wave 3 cross-sectional and difference-in-difference models and the statistically significant results suggested that residing in a CfC site had positive impacts for both groups of families—where the mother had low levels of education and where the mother had a higher level of education.
The overall trends in the pattern of estimates were also worthy of examination, given that small effects were likely because of the nature of the evaluation (a short-run evaluation of an area-based intervention that did not impact on all families in the community). The key findings from the overall trend of the estimates (see Table 5) were:
- For low education households, two-thirds of the Wave 3 cross-sectional regression estimates suggested that the CfC was having a positive effect, although not all of the effects were statistically significant. For the difference-in-difference estimates, half of the estimates indicated that CfC was having positive effects, although, once again, a number of the effects were not statistically significant.
- For higher education families (mother has an educational attainment of Year 11 or higher), two-thirds of the estimates from the Wave 3 cross-sectional regression models suggested a positive impact of the CfC initiative. For the difference-in-difference estimates, almost three-quarters of the estimates indicated positive effects from CfC, although only one was statistically significant.
(a) Only collected at Wave 3.
Note: All differences marked by * were statistically significant at the 5 per cent level at Wave 3 but were not at Wave 1. Differences marked by ** were statistically significant at the 1 per cent level at Wave 3 but were not at Wave 1. It should be noted that mental health was slightly higher (p=0.06) for the low education group at Wave 1. All estimates in Table 5 were rounded to two decimal places.
Source: SFIA.
Low-income households
Low-income households were households where the combined parental income was in the lowest 20 per cent of the distribution at Wave 1 ($485 per week or less). Table 6 presents estimates of the impact of CfC—low-income households living in CfC sites were compared with low-income households living in contrast sites, while not-low-income households living in CfC sites were also compared to their not-low-income counterparts in contrast sites.
(a) Only collected at Wave 3.
Note: The effect of CfC on these outcomes were estimated using OLS and logistic regression. All differences marked by * were statistically significant at the 5 per cent level at Wave 3 but not at Wave 1. It should be noted that mental health was significantly higher at Wave 1 for the low-income group (p=0.02). Community involvement was significantly lower for the low-income group at Wave 1 (odds ratio=0.53, p<0.05).
Source: SFIA.
For low-income households, there was a positive impact on maternal involvement in community service activities when estimated using the difference-in-difference model. Although not statistically significant, this finding was also consistent with the Wave 3 regression estimate. Consistent with the finding of a higher level of community engagement among mothers in low-income households in CfC sites, there was also evidence that CfC increased the reported community social cohesion of low-income mothers. Finally, the estimates suggested that CfC had the effect of reducing the number of jobless households for low-income households.
In contrast, in terms of reported child physical functioning, based on the Wave 3 regression estimates, children in low-income and not-low-income families in CfC sites had significantly lower levels of physical functioning than children in these two groups in contrast sites. The difference-in-difference estimates also suggested that the CfC intervention had a negative effect on the general health of low-income mothers. Although not statistically significant, the Wave 3 cross-sectional estimates also suggested that general health among low-income mothers deteriorated in CfC sites compared to low-income mothers in contrast sites.
In relation to not-low-income households, there was a statistically significant effect of the CfC intervention on hostile or harsh parenting (p=0.06) based on Wave 3 regression estimates. There was no evidence that the CfC intervention had a statistically significant effect for low-income households, although the effect sizes were bigger for the Wave 3 regression and difference-in-difference estimates, which suggested CfC had a bigger impact on hostile or harsh parenting of low-income families than for not-low-income families.
The overall pattern of results suggests there were positive effects of the CfC initiative for both low-income and not-low-income groups, but the pattern was more positive for low-income households:
- For the low-income households, 78 per cent of the regression estimates at Wave 3 and 69 per cent of the difference-in-difference estimates were in the positive direction.
- For the not-low-income households, 59 per cent of the regression estimates from Wave 3 and 57 per cent of the difference-in-difference estimates suggested positive effects.
The general pattern of estimates for both groups provided some further evidence that was consistent with the view that the CfC intervention had positive impacts.
Hard-to-reach families
There were a number of statistically significant effects of the CfC intervention for not-hard-to-reach families (Table 7). In particular, child emotional and behavioural problems were lower for not-hard-to-reach families in CfC sites than contrast sites at Wave 3, while receptive vocabulary achievement and verbal ability was higher in CfC sites (at the p<0.06 level of significance). There were also positive impacts of the CfC initiative on parenting among the hard-to-reach group. According to regression models at Wave 3, there were significantly lower rates of harsh or hostile parenting. Hard-to-reach families had lower rates of jobless households in CfC sites compared to contrast sites, based on difference-in-difference estimates, and the size and direction were similar for the Wave 3 regression model.
Child physical health (as reported by parents) was worse for hard-to-reach families in CfC sites, based on both regression models at Wave 3 and difference-in-difference estimates at the 0.09 level of statistical significance.
Among not-hard-to-reach families, unmet service needs increased in CfC sites in comparison to not-hard-to-reach families in contrast sites. This finding is unusual, as it does not match any other results observed in either the analysis of the full sample or the other groups in the differential analysis.
The overall pattern of results for the hard-to-reach group was consistent with the view that the CfC initiative was positive, with 65 per cent of regression and 65 per cent of difference-in-difference estimates indicating positive impacts. There was also a positive impact for the not-hard-to-reach families, with 88 per cent of regression and 67 per cent of the difference-in-difference estimates being in the positive direction.
(a) Only collected at Wave 3.
Note: The effect of CfC on these outcomes were estimated using OLS and logistic regression. All differences marked by * were statistically significant at the 5 per cent level at Wave 3 but not at Wave 1. Differences marked by ** were statistically significant at the 1 per cent level at Wave 3 but not at Wave 1.
- Previous: 4. Validity of contrast sites as the counterfactual
- Next: 6. Summary of the effectiveness of the Communities for Children initiative
[ top ]