Skip to content

Stronger Families in Australia study: the impact of Communities for Children

6. Summary of the effectiveness of the Communities for Children initiative

6.1 Summary of evaluation findings
6.2 Interpretation of the evaluation findings
6.3 How large were the effects of the CfC intervention?

6.1 Summary of evaluation findings

Research suggests that community disadvantage, characterised by poor access to services and other extra-familial supports, low levels of community cohesion, and high levels of crime and poverty, can place children at risk. This has been the catalyst for the development and implementation of area-based interventions, which aim to improve the quality of the community context in which families raise young children and, in turn, promote optimal child development. However, we are still at the early stages of understanding how community-level interventions can help vulnerable children and families. Area-based initiatives target the entire population of families with young children within a prescribed area, and little evidence has been available to suggest that families in greatest need of services actually receive them (one exception is the Pathways to Prevention Project: Homel et al. 2006). There has also been an absence of data to indicate whether investments in local services can actually ‘regenerate’ a community in social capital and social cohesion terms, and that these outcomes, if achieved, spill over to positively influence the lives of children and their families. One of the aims of the SFCS evaluation was to address these gaps in knowledge and to better understand if and how community interventions can enhance the wellbeing of young Australian children and their families.

In 2005, the CfC initiative was introduced into 45 local communities, with the aim of enhancing the life chances and development of Australian children living in disadvantaged areas. Stronger Families in Australia—with its large sample representing 42 per cent of the population in 10 CfC and five contrast sites, low attrition (not systematic) at longitudinal follow-up, sophisticated approach to design and analysis, robust measurement and appropriately matched comparison group—provides a unique opportunity to consider the effectiveness of this model of early childhood intervention in the Australian context.

Having established that families and children in the five contrast sites provided a valid counterfactual, the final analyses determined whether there were significant differences between CfC and contrast sites, and the magnitude of any effects. Both cross-sectional regression modelling on outcomes post-intervention (Wave 3) and difference-in-difference modelling were used to estimate the effects of the CfC intervention. Regression modelling was used to compare differences between CfC and contrast sites post-intervention (that is, at Wave 3). Difference-in-difference models were also used to estimate whether the rate of change of outcomes in CfC and contrast sites from Wave 1 to Wave 3 was statistically significant, after taking into account the effects of other variables. To aid the interpretation of the estimated effects of the CfC intervention, standardised effect sizes that indicated whether observed differences were ‘small’, ‘medium’ or ‘large’ were used.

The broad conclusion from the SFIA evaluation study is that there was clear evidence that the CfC intervention has had a positive effect on a range of outcome measures. Although negative effects were found on some health outcomes (based on parental reports), it was unclear whether subjects in CfC sites were actually less healthy than their counterparts in contrast sites at Wave 3, or whether the CfC initiative was a catalyst for greater diagnosis and awareness by parents of their own health and that of their children.

In terms of the effects that reached conventional levels of statistical significance, findings from the sample overall suggested:

In contrast to these positive findings, however, children in CfC sites had lower physical functioning than children in contrast sites using Wave 3 cross-sectional regression modelling (but not statistically significant using difference-in-difference modelling).

Positive, although not necessarily statistically significant, effects of CfC were found for:

The overall pattern of results, taken with the results that were statistically significant, provided evidence that the CfC intervention was having some positive effects.26 The effect sizes estimated using difference-in-difference models were smaller than those using the Wave 3 cross-sectional estimates. However, the general picture was consistent between the two estimation methods.

One of the key policy drivers of the CfC service model was to improve outcomes for disadvantaged families in disadvantaged areas. In practice, similar interventions have widened the gap. This is because some initiatives improved outcomes for the less disadvantaged children but did not reach the most disadvantaged sectors of the population. For example, the early impacts evaluation of Sure Start widened the gaps for some groups. Children from jobless households and lone-parent families had significantly lower verbal ability when living in SSLP areas than in comparison areas. Children of teen mothers also had lower verbal ability and social competence and higher behaviour problems than peers in comparison areas (NESS Team 2005). Thus, it was not only important that overall outcomes were improved, but also to understand the extent to which the CfC initiative had succeeded in closing the gaps between the most disadvantaged and the relatively less disadvantaged children in the community.

Analyses were therefore performed on three key subpopulations in which the numbers were sufficiently high to produce reliable estimates: children of mothers with low education, children in low-income families, and children in hard-to-reach families,27 as defined by a number of risk indicators. When the statistical significance of specific outcomes were considered, there was some evidence of the beneficial effects of the CfC initiative for disadvantaged families in relation to early learning and care, community-level outcomes and parenting, and some evidence of negative effects on child and maternal physical health and maternal mental health.

In relation to the three key subpopulations of interest, positive findings from the CfC intervention included:

In relation to these three key subpopulations, negative findings from the CfC intervention included:

Significant effects were also observed for higher income households, households comprising mothers with greater than Year 10 education and not-hard-to-reach households. The effects found included:

Table 8 summarises the statistically significant findings discussed above. It shows a consistent pattern of positive, statistically significant results for three of the four priority areas. When findings from the full sample and the subgroup analysis were considered together, supporting families and parents, child-friendly communities and early learning and care were all areas that appear to have benefited from the CfC intervention. Healthy young families was the one outcome that did not appear to have benefited from the CfC initiative, with all but one of the statistically significant findings suggesting negative impacts of CfC.

Consistent with the pattern of statistically significant results in Table 8, the overall pattern of results also provided some evidence of the positive impacts of the CfC intervention for these subpopulations, specifically:


Table 8: Significant effects of CfC on families, children and communities: families living in CfC sites compared to contrast sites
 

Outcome variable

Full sample

Hard-to-reach status

Mother’s education

Income

Hard-to-reach

Not hard-to-
reach

Year 10 or
less

Higher than
Year 10

Low income

Higher
income

Supporting families and parents

Less hostile parenting

tick(W3**)

tick(W3*)

tick(W3**)

 

tick(W3*)

 

tick(W3*)

Parenting self-efficacy

tick(W3*)

 

tick(W3**)

       

Service needs are met

   

cross(W3**)

       

Living in a household with a job

tick(W3**)

tick(DD** & W3*)

 

tick(DD**)

tick(DD**)

tick(DD**)

tick(W3**)

Child-friendly communities

Involvement in community service activities

     

tick(W3**)

 

tick(DD**)

 

Community social cohesion

         

tick(DD*)

 

Early learning and care

Child receptive vocabulary and verbal ability(a)

   

tick(W3*)

tick(W3**)

     

Healthy young families

Child has no/few emotional and behavioural problems

   

tick(W3**)

       

Reported child physical health

cross(W3**)

cross(W3** & DD*)

 

cross(W3**)

cross(W3**)

cross(W3**)

cross(W3**)

Parent-reported general health

 

cross(DD*)

     

cross(DD**)

 

Parent-reported mental health

     

cross(DD**)

     

(a) Outcomes only tested at Wave 3 and therefore are not available using difference-in-difference.

Notes: W3=result based on regression at Wave 3; DD=result based on difference-in-difference; * p<0.1; ** p<0.05.

tick =CfC result is ‘better’ than the control site; cross =CfC result is ‘worse’ than the control site.

6.2 Interpretation of the evaluation findings

The CfC initiative aimed to support positive child development in disadvantaged areas through an effective, comprehensive and integrated system of early childhood services, and a focus on broad systems of relationships and networks in the whole community to improve community ‘child-friendliness’ (that is, community ‘embeddedness’ or social capital). Success of the CfC initiative was measured here in terms of ‘downstream’ effects on the capacity of the home and community environment to support child development, and in terms of child functioning directly.

Findings on parenting

There was evidence that the CfC intervention led to improvements in parenting, with a reduction in hostile parenting being found in the full sample, as well as the hard-to-reach, the not-hard-to-reach, higher maternal education and higher household income subsamples. Effects on parenting were not found for families with a low maternal education or low income. This may be accounted for by the fact that these groups had fewer financial resources than all the others, and that financial stress is strongly linked to adverse parenting (for example, Conger & Donnellan 2007). The CfC intervention also appeared to have increased parenting self-efficacy for the full sample and the not-hard-to-reach subsample, but not for other groups, suggesting the effects of CfC activities on parenting confidence flow to the not-hard-to-reach sector of the community.

Findings on community outcomes

Of the six community-level outcomes examined, the effect of CfC on joblessness was most striking. Indeed, for the sample overall, and for all key subgroups except those who were not hard-to-reach, CfC had a small to medium effect size on reducing joblessness. Importantly, for the sample overall, findings emerged using both the difference-in-difference estimation and the regression modelling at Wave 3. Also, significant effects were found in analyses controlling for individual site differences. These were clear signals that the findings on joblessness were statistically robust. Although it is difficult to know the precise mechanisms whereby the CfC initiative produced this effect, it is possible that greater community engagement and participation of mothers, and denser networks of relationships surrounding families in CfC sites led to opportunities for employment (for example, Stone, Gray & Hughes 2003). Indeed, additional descriptive analyses suggested that increases in mothers’ (as opposed to fathers’) employment may have explained the lower rates of joblessness. There was a 9.4 per cent increase in mothers’ rates of employment from Wave 1 to Wave 3 compared to 4.9 per cent in contrast sites for the same period.

There were also significant findings on community-level outcomes for those households with low maternal education and low household income. In particular, households comprising mothers with Year 10 education or less and low-income households reported higher involvement in community service activities such as volunteering. The effect sizes were medium to large.

The apparent positive impacts on joblessness, community service activity and social cohesion suggested that new programs and enhanced inter-organisational networks in CfC sites may be the building blocks of community cohesion and social capital.

Findings on child outcomes (other than health)

While there were relatively few effects of the intervention found for child outcomes (early learning and care), positive effects for child receptive vocabulary/verbal ability were found for children in not-hard-to-reach families and those with a mother with Year 10 education or less. The benefits of CfC were evident for children of mothers with low education in particular; these children had a receptive vocabulary and verbal ability score that was 6 points higher than children of mothers with low education in contrast sites. This improvement was not only statistically significant but also of practical import. A six-point difference is almost one-half of a standard deviation—a ‘medium’ standardised effect size.

Follow-up analyses of children with receptive vocabulary and verbal ability one standard deviation below the population mean (see Love et al. 2002) suggested that children of mothers with low education who resided in CfC sites were 2.56 times more likely (p=0.05) than children with mothers with below average education from contrast sites not to have below average verbal skills. This suggested that, either as a result of children’s direct participation in CfC activities (such as playgroups) and/or through parental participation in CfC programs, children were observed to have made considerable gains in this aspect of their cognitive development in a very short time. Given the benefits of early literacy on children’s performance on entry to school, it would be instructive to follow the current sample to assess whether these early gains are sustained after the transition to school.

Findings on reported maternal health and child health

Health is the one area in which negative effects of the CfC intervention were estimated. While there is some evidence that the CfC intervention had a negative impact upon the mental health of low-education mothers, this finding was not found elsewhere. Similarly, the estimates suggested that the CfC intervention had a negative effect on parent general health for the hard-to-reach and low-education mothers. In contrast, the CfC intervention reduced children’s emotional and behavioural problems in the not-hard-to-reach group.

Across all of the groups examined, except for the not-hard-to-reach, there was evidence that the intervention had a negative effect on child physical functioning. However, this was inconsistent with the findings of no effects on children being overweight. These findings (maternal and child health) were neither expected nor readily explained with the data currently available.

Like many other measures included in the SFIA evaluation study, the assessment of child physical functioning was based on parental subjective reports (BMI and verbal/receptive skills are two primary exceptions). The eight items included in the PedsQL physical functioning subscale relate to gross motor skills and indicators of illness or a health condition.

One possible explanation is that the apparent negative effects on parent and child health were a result of the intervention leading to a greater awareness of their own and their child’s health problems rather than there being a negative effect on actual health. This may have occurred because the CfC intervention increased the likelihood that parents came in contact with health professionals and others who may have recognised undiagnosed health conditions, ailments or developmental issues for the child.

It is also possible that children in CfC sites may have come into greater contact with other children through increased participation in programs and activities, such as playgroups and preschool, thus contracting more common infectious diseases, such as colds, while children in contrast sites potentially spent less time in close contact with other children.

Both interpretations are entirely speculative, however. Further waves of the SFIA study will provide the empirical data that is needed to test these alternative hypotheses before firm conclusions can be drawn about the impact of the CfC initiative on child physical and maternal health. Although the current study did not collect information on the number and type of investments in health-related projects, such information may also be useful in understanding the pattern of results on child physical and maternal health.

6.3 How large were the effects of the CfC intervention?

As discussed, the size of the CfC impacts on all outcomes were small, but could be considered positive relative to what was observed in the early phase of the UK Sure Start evaluation. The current results were also comparable in size to those found in the Later Impact evaluation of the Sure Start program, where 3 year-old children were exposed to mature SSLPs throughout their entire lives.31

An important question, however, is the extent to which these effects compare with alternative early childhood interventions that target specific client groups and seek to enhance child outcomes through other processes, such as centre-based programs (for example, the Perry Preschool Project and Head Start), home visiting programs, case management interventions and parenting programs (for example, Triple P).

A 2005 review of the effectiveness of early childhood interventions found that a range of early childhood interventions were efficacious, especially in the domain of children’s cognitive skills and child outcomes in general. However, with the exception of Triple P, which reported greater impacts, studies reporting effect sizes on parenting and child outcomes were in the negligible to small range (Wise et al. 2005). This has important implications for decisions regarding future investments in the CfC initiative versus alternative service modalities, as the current findings suggest that the CfC is just as effective. This is especially true when one considers that the SFIA evaluation measured effects on an entire population, rather than on program participants as such. The CfC service model was also assessed at a very early stage of implementation. Indeed, programs and services in CfC communities were only likely to have been fully operational for a maximum of two years at Wave 3, or from the time study children were approximately 3 years old. As initiatives continue to be put into effect, one might expect even greater disparity in outcomes between the CfC and non-CfC sites. Given the focus of the CfC initiative was on children aged 0 to 5 years and their families, one might also expect larger positive effects among a new cohort of young children, as they would have greater exposure to programs and services, commencing from birth.

[ top ]