Australasian Evaluation Society international conference
On 2–4 September 2009, the Australian Evaluation Society (AES) held its annual international evaluation conference, of which FaHCSIA was a major sponsor. The theme of the conference was ‘Evidence and evaluation: gathering evidence, using evidence, evidence and stakeholders' and was held at the National Convention Centre in Canberra.
The conference was structured around six plenary sessions where keynote speakers presented to the entire conference on a broad range of evaluation and evidence-based topics.
The majority of presentations were run concurrently across several session times. Individuals presented evaluation papers on a wide variety of topics ranging across health, corporate, international development, social, education and environmental–based evaluation.
Throughout the conference several recurring topics developed. A number of presenters emphasised the importance of integrating evaluation and policy development. A keynote speaker discussed the role evaluations play in ensuring government policy is effective and efficient, while meeting citizens' needs. The speaker argued that evidence-based evaluation enhances accountability, democracy and trust between the government, public service and citizens.
Another topic widely discussed was the advantage of using less quantitative methods of evaluation. The presenter argued that quantitative-based analysis, such as administrative data or randomised experiments, has the potential to lead to false conclusions and that more qualitative methods of evaluation have greater capacity to attribute causality. Similarly, other presenters discussed the advantages and disadvantages of particular qualitative styles, such as using anecdotal evidence in evaluations.
Other presentations covered frameworks and strategic steps in conducting evaluations. One presenter argued that a common framework for evaluations is desirable. Outcomes theory's Five Building Blocks Model was presented as a potential framework that may help clarify the types of evidence and level of outcomes that are appropriate depending on the focus of the evaluation. The presentation also emphasised the importance of using logic models or a program logic to demonstrate the links between inputs, outcomes and outputs. Another presenter outlined a number of steps that are essential to conducting evaluations. The first step is to define a topic then clarify the objectives, decide what to measure, collect the evidence, ensure quality and credibility of the evidence, draw implications, ensure objectivity and consider biases and finally implement findings.
The most prominent theme throughout the conference was best captured in the opening plenary session by Professor William Trochim. Professor Trochim plays a leading role in the American Evaluation Association as well as being a Professor of Policy Analysis and Management at Cornell University.
Professor Trochim's presentation discussed the conflicting ideologies surrounding evaluation methodology. He argued that randomised control trials (RCTs) and other randomised experimental designs are seen by some evaluators as a gold standard for program evaluation. Professor Trochim notes that while these methods can contribute to some evaluations they are not appropriate for all evaluations and in some circumstances the validity of RCTs can be questioned.
Professor Trochim argued that a pluralist approach to evaluation is most appropriate, valid and has more rigour. Using multiple methods for an evaluation creates a more complete and accurate picture of the situation.
Professor Trochim also argued that different stages of a program often require a different method of evaluation. Premature or inappropriate methods of evaluation may not show the correct picture. Therefore, one needs to use a variety of different methods over the course of the evaluation to build and synthesise already accumulated knowledge.
Professor Trochim proposed that evaluation always takes time and one should research, compare, debate and trial different types and different combinations of methods for each evaluation.
The AES conference covered a wide variety of ideas and debates about evaluation and evidence. More information about the conference and the evaluation papers that were presented can be found on the AES website.
Australia's welfare 2009 conference
On 17 November 2009 the Australian Institute of Health and Welfare (AIHW) held its ‘Australia's welfare 2009' conference at the Queanbeyan Performing Arts Centre, NSW. A range of highly regarded Australian and international speakers, including FaHCSIA staff, tackled topics such as those relating to social inclusion and how well the needs of Australians are being met by government services and programs designed to promote wellbeing. Other topics included: disability and disability services; housing and housing assistance; children, youth and families; carers and informal care; ageing and aged care; and homelessness.
At the conference The Hon Jenny Macklin MP launched the AIHW report on welfare services in Australia. The report, Australia's welfare 2009, found that while most Australians enjoy a good standard of living, many are struggling to care for people with a disability, or with issues related to ageing, homelessness, children and families.
A full list of presenters and copies of their presentations can be found on the AIHW website.
[
top ]
Child protection as regulation: clarifying principles conference
Several FAHCSIA staff attended the ‘Child protection as regulation: clarifying principles conference' hosted by the Regulatory Institutions Network (RegNet) at the Australian National University in November 2009.
Drawing together regulatory scholars and national and international experts in child protection, the conference provided an interdisciplinary exchange exploring child protection as a form of regulation. Conceptualising child protection as a form of regulation may be an uncommon approach; however, it does offer ways of new thinking about the principles that governments and other interested parties might wish to endorse in their approaches to regulating families when it comes to children's safety.
Regulation is traditionally understood to mean rules and procedures. Regulatory scholars broaden this definition to regulation as a social activity or a process by which we all engage in when we intervene in our social world. This expanded definition allows for a wider repertoire of informal as well as formal strategies that can be utilised when aiming to influence or steer certain behaviour or the flow of events. While most families self regulate when it comes to the care and protection of their children, most need some informal help along the way. For example, friends help out with babysitting, or a grandparent provides much needed respite care. These activities are examples of regulatory processes which ensure children are provided with supervision and help parents meet their child rearing responsibilities. Child care programs and before and after school care are examples of formal assistance parents rely on in the process of raising their children.
When limits to self regulation and informal care fall short, intervention may come from state authorities when a child is considered at risk. Diversionary programs that help families access support services and encompass a ‘whole-of-family' approach offer a responsive alternative to heavy-handed, highly investigative, expensive and time consuming interventions that characterise many child protection systems. Conference presenters noted that families have great difficulty getting involved on their own terms and resent heavy-handed requirements for their ‘cooperation' with authorities, although families do tend to cooperate when child protection issues are raised. What is often missing but sorely needed, is someone to ‘open the space' for dialogue and deliberation, which in turn can lead to care plans that articulate what needs to be done for the child, by whom, how and by when. Most families do have someone they know who wants to or will help them when child protection authorities become involved. They come to planning meetings, prepare safety plans that are acceptable to professionals (including courts) and make their resources freely available. In New Zealand (NZ), data are showing that the chances of children re-entering the care and protection system is six times less likely if the family is engaged at the point of notification first being made.
Almost half of the states in the United States (US), for example, have now embraced the language of ‘engaging families' in child protection work; however, family engagement approaches in child protection are still evolving in the practice domain. Some key lessons that have been learnt from effective practice include: infusing the possibility of family involvement at any stage before ‘formal' involvement; bringing the family in when big decisions are at stake that need to be made quickly; keeping space open for ongoing work and planning; and making clear differentiation between danger and risk and what are complicating issues/factors. One such complicating issue identified during the conference was domestic violence. Discussion then centred on the NZ experience of child protection practice when domestic violence is an identified issue and when the risk to children is ambiguous.
Why does this idea of regulation matter? Conference participants—irrespective of their disciplinary backgrounds—recognised that regulatory systems impact on families and children. When it comes to child protection, the negative consequences of regulation have been documented in stories such as ‘Orphans of the living' and by groups such as the Alliance for Forgotten Australians. Conference presenters suggested that questions are not asked about the child protection regulatory system itself, nor are the assumptions in the system—the purpose, intervention by and outcomes of the system—adequately debated.
The conference—with its national and international participants—has provided support to a regulatory analysis of child protection systems and offered practical strategies based on a range of overseas and national evidence.
The conference was based on the special edition of the Journal, ‘Communities, children and families Australia,' vol. 4, no. 1 (2009), which can be accessed from the Australian Catholic University website or email ICPS@acu.edu.au.
[ top ]
Growing up in Australia: the Longitudinal Study of Australian Children (LSAC)
The Australian Institute of Family Studies (AIFS) hosted the second Growing up in Australia: the Longitudinal Study of Australian Children (LSAC) research conference on 3–4 December 2009 at Rydges on Swanston, Melbourne. The conference involved two concurrent sessions each day including 33 oral and seven poster presentations based on LSAC data. The following is a brief summary of presentations of particular relevance to the FaHCSIA portfolio.
Professor Alan Hayes, Director of AIFS, opened the conference and expressed his confidence that the event would build on the great success of the first LSAC conference held in 2007. Professor Hayes noted that ‘the goal of the study, this conference and, no doubt, many of your own projects, is to improve outcomes for future generations'. At the conclusion of the opening speech Professor Hayes launched the 2008–09 LSAC Annual Report.
The talk ‘Using LSAC to examine the adequacy of Australian children's living standards'explored how LSAC data could inform the extent to which children have the rights, described in the United Nations Convention on the Rights of the Child, ‘to develop to their fullest potential' and have a standard of living that is ‘adequate to their developmental needs'. LSAC data was used to compare three distinct approaches to assess this question: economic welfare, child capabilities and social inclusion. These approaches highlight similar patterns of disadvantage, and each approach has advantages and disadvantages in addressing different human rights principles.
The presentation ‘Measuring family disadvantage in the Longitudinal Study of Australian Children' described the development of a measure of social disadvantage for families within LSAC. The measure is adapted from the index of multiple disadvantage presented by the UK Social Exclusion Board Task Force. The proposed index is comprised of a range of parent-based factors including: poverty, joblessness, education, housing, mental health and physical health. This measure can assist in identifying at-risk families and in the implementation and evaluation of social inclusion policy interventions.
The presentation ‘Parenting in place: exploring parenting behaviours and socioeconomic position in the context of neighbourhoods' looked at how socioeconomic position (SEP) and neighbourhood characteristics are associated with overprotective parenting practices. Preliminary analyses indicated that SEP and some parent perceptions of neighbourhood are associated with parenting behaviours. The results suggest that researchers and policy makers need to consider not just parenting practices in isolation but also in the context in which they occur.
The ‘Breastfeeding and infant time use' presentation explored how infant time use may partly explain the well-established relationship between child cognitive outcomes and breastfeeding. The results showed that breastfed infants were more likely to spend more interactive time with parents and less time in a range of other activities such as sleeping, eating, drinking or having other foods compared to non-breastfed infants. This suggests that future research should further explore the relational benefits of breastfeeding as a mechanism for explaining improved cognitive outcomes for breastfed infants.
Another paper utilising time use data looked at how different time use activities influence children's cognitive and non-cognitive development. The researchers found that, as expected, time spent in educational activities was associated with higher levels of cognitive development. More surprisingly, the researchers also found that time using media and time spent socialising was more strongly associated with children's cognitive development than time spent in formal care and education. The researchers found that children's non-cognitive skills did not vary significantly depending on how they spent their time but was related to the warmth and strictness of the mother.
The paper ‘Talking to children clearly but respectfully: the effect of inductive reasoning and hostile parenting at age 4–5 on child's conduct problems at age 6–7 using LSAC Waves 1 and 2' found a significant gender difference in the relationship between hostile parenting and inductive reasoning (explaining the consequences of a child's misbehaviour on others) in their impact on children's later conduct problems. When parents inductively reasoned with boys there were lower levels of hostility and conduct problems and this was significantly lower than when parents communicated with boys in a hostile way. For girls, conduct problems were better explained by sociodemographic factors.
The keynote presentation, ‘Children's physical, cognitive and socio-emotional outcomes: do they share the same drivers?', tested the commonly held view that the ‘same, or similar risk factors account for a wide range of problematic child and adolescent outcomes'. This argument rests on the findings from different studies, each of which typically focuses on only one outcome, whereas LSAC offers the opportunity to directly test this proposition within a single dataset. The findings showed that while the ‘common drivers' proposition generally applied, there were interesting nuances both in the timing and specific impacts on particular child outcomes.
The paper ‘Parental leave, maternal mental wellbeing, couple relationship and quality of parenting: evidence from the Longitudinal Study of Australian Children' looked at the extent to which the use and duration of parental leave affected parental outcomes, which may in turn influence child outcomes. The paper focused on the relationship between maternity leave, maternal mental wellbeing, couple relationship and quality of parenting. Drawing on LSAC data from Waves 1, 1.5 and 2, the paper found that factors such as the presence of older siblings, mothers' education, age, family income and neighbourhood facilities are more likely to affect parental outcomes than the use and type of parental leave. The analysis highlighted the complexity of investigating the impact of parental leave.
The paper ‘The impact of child support payments on the labour supply decisions of resident mothers'investigated the impact of child support on mother's workforce participation; in particular, hours worked and their probability of employment. The paper found that the receipt of child support did not affect the labour supply for the majority of resident mothers. Specifically, there was a modest negative impact on the number of hours worked, but not on the probability of employment.
The presentation ‘The effect of mothers' experience of extended work–life tension on child outcomes'examined the impact of mother's work responsibilities on children's outcomes. Using the longitudinal characteristics reflected in multiple waves of LSAC, this research considered potential links between mothers' ongoing experience of work–life tension and the physical, social, emotional and learning outcomes of their children.
Based on an analysis of the Australian Temperament Study supplemented by LSAC Waves 1–3, the presentation ‘Intergenerational mobility in Australia: how do vulnerable kids fare?' assessed the extent to which intergenerational mobility is associated with social–emotional and cognitive learning, educational attainment, changes in household structure, access to child care/education and changes in benefit entitlements. The presentation compared the trajectories of children who formed the top and bottom 10 per cent of the distribution of learning and social–emotional outcomes at around the age of 4 years. The presentation revealed the extent to which these starting points (as indicators of their ‘personal characteristics') can predict outcomes in middle childhood and early adulthood, and the extent to which factors such as socioeconomic position amplify or confound children's trajectories.
The presentation ‘The negative effects of persistent poverty on children's physical, socio-emotional, and learning development' examined the negative effects of persistent poverty on children. The researchers divided families into ‘persistently poor', ‘out of poverty', ‘fell into poverty' and ‘never been poor' and assessed child outcomes for these groups. Consistent with previous literature, the results showed that children from persistently poor families exhibited poorer outcomes on all measures.
The presentation ‘Persistence and emergence of conduct difficulties in children with an intellectual limitation'found that children with intellectual limitations are at significantly greater risk of persistent conduct difficulties, are more likely to be exposed to environmental risk factors and are less resilient when exposed.
Overall, the conference was a great success and was well attended by a range of government and non-government agency representatives and academics. For detailed conference abstracts and notes see the AIFS website.
[
top ]
Maximising the value of longitudinal studies for policy and science: methodological and analysis issues
On 12–13 December 2009, FaHCSIA and the Australian Research Alliance for Children and Youth (ARACY) Australian Research Council (ARC) Research Network hosted a workshop titled ‘Maximising the value of longitudinal studies for policy and science: methodological and analysis issues' at the Pavilion hotel in Canberra. The workshop was jointly facilitated by Professor Ann Sanson, Network Coordinator, ARACY ARC Research Network, and Dr Helen Rogers, Longitudinal Study of Australian Children (LSAC) Section Manager within FaHCSIA. The workshop included nine oral presentations, three panel discussions and a poster session addressing a range of issues in longitudinal studies including: participant retention and attrition, data collection, weighting, imputation, data linkage and data harmonisation.
The first keynote presentation, ‘The value and challenges of doing longitudinal research well in the 21st century', drew on findings from the famous Dunedin Longitudinal Study to investigate study retention and gene environment interactions. The Dunedin study began in 1975 with 1,037 participants and has achieved a 96 per cent response rate at the most recent wave of data collection in 2004–05. The study investigators attributed this success to the time and effort devoted to tracing and engaging with study participants and demonstrating the value of the study by highlighting the quantity and quality of publications the study has generated. The researchers also showed the importance of minimising attrition by demonstrating that the 20 per cent of participants who were most difficult to trace were also significantly more disadvantaged on a range of measures.
The first panel discussion further highlighted the challenges of attrition. A range of strategies for engaging with participants were discussed including the use of cash payments, gifts and new communication technologies such as text messaging, emails and social networking sites.
The presentation on ‘Longitudinal scale performance: quality characteristics of ordinal measures' described statistical techniques for evaluating and creating ordinal measurement scales. Many longitudinal studies use these scales by combining several items to measure self-reported subjective phenomena like wellbeing or depression. It is important to establish measurement fidelity in these scales to ensure they reliably measure the same construct over time. The presentation described the benefits and practical implications of the covariance approach for assessing measurement fidelity.
The second keynote presentation, ‘The experience of the United Kingdom Millennium Cohort Study', described the methodological challenges and management strategies used in the Millennium Cohort Study (MCS). The MCS is a longitudinal birth cohort study following over 19,000 children born in the UK in 2000 and 2001. The presentation discussed a range of issues including minimising attrition, measurement quality, the special challenges of conducting research with children, and the achievements of researchers using MCS data to produce policy relevant research.
One presentation was devoted to the issue of weighting in the Household, Income and Labour Dynamics in Australia (HILDA) dataset. The presentation explained the importance of weighting as a way to adjust panel data so it more accurately represents the population from which the survey sample is drawn. Specifically, weighting adjusts for the unequal probabilities of selection into a panel survey, non-response to particular questions and non-response to an entire data collection wave. The presentation outlined some important considerations in producing weights and the various possible options. It went on to describe the process of developing and using the cross-sectional and longitudinal weights for the HILDA survey.
The presentation on ‘Imputation approaches for handling missing data in longitudinal studies' described the relative benefits of handling missing data using multiple imputation compared to traditional approaches of analysing complete cases only, non-response weighting, and single imputation. The presentation then covered the process, strengths and risks of multiple imputations.
The presentation on data linkage discussed how data linkage brings together two or more separate data records about the same individual. This method has become increasingly commonly used and is a powerful technique for using administrative datasets in research. The presentation provided an overview of a range of issues including consulting with data owners, privacy and confidentiality, and the data linkage process.
The workshop provided a good opportunity for researchers and people involved in the development and management of longitudinal surveys to come together and discuss relevant methodological and analysis issues.