Skip to content

Housing banner

YP4 – Joining up services for Homeless jobseekers

The Evaluation Agenda

Considerable energy was devoted in 2004 to establishing a rigorous evaluation framework for the trial, even before a single participant was recruited. To assist with this process, an Ethics and Evaluation Advisory Group was established as a standing subcommittee of the Inter Agency Coordinating Committee. The Ethics and Evaluation Advisory Group (EEAG) consists of eminent professors as well as representatives from all the major stakeholders who have partnered YP4.

The EEAG prepared an evaluation framework for YP4. The broad purpose of YP4's evaluation framework is to ensure that the lessons learnt from YP4 and its implications for program design and practice are documented, fully understood, shared and, wherever possible, realised.

There are three questions or groups of questions that drive the evaluation, each of which has a series of sub-questions associated with them.

The questions suggest three key components of the evaluation: summative, formative and financial evaluation. Attention is being given equitably to these three streams of the evaluation. Subsequent sections of this chapter detail, in turn, the approach taken for each of the three components of the evaluation.

[ top ]

The most significant feature of the evaluation agenda is that YP4 can be understood as a social experiment in that the outcome evaluation methodology centres on the existence of a 'control' group whose (employment and housing) outcomes can be directly compared to the outcomes of the 'treatment' group i.e. those who are receiving the service delivered by YP4. Importantly, the allocation of young homeless jobseekers to the 'control' or 'treatment' groups is, with a few exceptions, occurring randomly. This is made possible by the fact that there are more young homeless jobseekers in each of the catchment areas than there are places in YP4. Social experimentation of this type is rarely seen (or permitted) in Australian social policy environments.

Overall, the evaluation framework makes explicit the commitment of YP4 to the highest standards of ethical behaviour. The National Health and Medical Research Council (NHMRC) determine ethical standards for all research involving people in Australia and these standards are recorded in the “National Statement on Ethical Conduct in Research Involving Humans” (1999). All human research ethics committees in Australia are bound to follow the standards set out in the statement: They must register with the NHMRC, report to it annually, and are subjected to audits.

Many of the community agencies and government departments represented on the Ethics and Evaluation Advisory Group have formally constituted Human Research Ethics Committees (HREC) within their own organisations. Rather than secure ethics approval individually from each of these committees, the Ethics and Evaluation Advisory Group collectively applies the national standards of ethical conduct, and ensures broad compatibility with the internal ethical requirements of each member organisation. External contractors linked to research institutions who are appointed to conduct evaluation activities on behalf of YP4 are expected to secure more formal ethics approval from within their home institutions.

Protection of privacy for trial participants is another imperative for YP4. YP4 complies with all relevant privacy legislation. A requirement of participation in YP4 (including the control group) is the provision of informed consent to access personal data from specified sources.

[ top ]

Outcome evaluation

The outcome evaluation is being overseen by a principal investigator who is independent of all of the partner organisations, including the five government departments who are participating. The principal investigator is Dr. Marty Grace of Victoria University. The following extracts from the application for ethics approval from the Victoria University Human Research Ethics Committee provide a useful summary of the outcome evaluation of YP4.

1. Title of project

Outcome evaluation of the YP4 trial

2. Aim of project

The overall aim of the project is to contribute to knowledge and policy activism regarding effective ways of delivering services to young homeless jobseekers.

The specific aims of the project are:

The YP4 trial involves randomly-assigned participants who will receive services either in the standard way or in a new joined-up way that has been developed specifically for this trial.

[ top ]

The following research question will guide the research:

By joining up services and programs, does YP4 assist participants to progress along a pathway that will achieve more sustainable employment and housing outcomes than would current interventions and if so, do those outcomes persist over time?

3. Plain language statement of project

YP4 (formerly known as the Young Homeless Jobseeker Trial) seeks to demonstrate that joining up a range of services and programs in a client-centred manner will result in more sustainable employment and housing outcomes for young homeless jobseekers. YP4 is an initiative of four community organisations: Hanover Welfare Services, Brotherhood of St Laurence, Melbourne Citymission and Loddon Mallee Housing Services. The 'YP' represents young people. The '4' refers to the four 'p's: purpose (a job), place (a home), personal support (offered by the project) and proof (the evaluation). The number '4' also represents the number of partner agencies and the number of sites for the trial.

YP4 represents a new approach to assisting individuals who are experiencing both homelessness and unemployment, in recognition that existing forms of housing and employment assistance are fragmented, linear, ineffective and inefficient for homeless jobseekers. YP4 will offer homeless jobseekers a single point of contact to address employment, housing, educational and personal support goals in an integrated manner over a two-year period.

An Ethics and Evaluation Advisory Group (EEAG) provides expertise and advice to the Inter Agency Coordinating Committee (IACC) for YP4, regarding all evaluation and research processes and outcomes required by the Trial. An evaluation framework has been prepared and formally approved by the Ethics and Evaluation Advisory Group. The purpose of the evaluation is to ensure that what is learnt from YP4 and the implications for program design and practice are documented, fully understood, shared and wherever possible, realised both within the project and beyond.

There are three components of the YP4 evaluation: an outcome evaluation, a process evaluation and a financial evaluation. This Ethics Application is for the outcome evaluation only. The other aspects of the evaluation will not involve Victoria University directly.

[ top ]

4. Nature of research, including methodology and a list of all procedures to be used on human subjects.

This project falls within the tradition of critical social research. It is informed by contemporary critical social theory that synthesises postmodern insights regarding diversity and multiple sites of power with more traditional concerns regarding inequality. Critical social research has a central purpose of contributing to transformational change that will challenge and reduce the oppression of the people who experience the social problems that are the focus of the study.

The overall approach or methodology for this research is iterative, with cycles of data collection and analysis, feedback, reflection and action. These processes will be undertaken in partnership with project staff, the Ethics and Evaluation Advisory Group, and the Inter Agency Coordinating Committee (IACC). The 'action' could include changes to the project, changes to the evaluation plan, targeted advocacy and strategic dissemination of particular findings.

The evaluation will be structured around the documented objectives of the project:

Project documents detail the strategies to be used to achieve these objectives, the outcomes intended and the indicators that will be used as evidence. These details (Appendix 1) will provide the starting point for the outcome evaluation.

The trial design largely determines the outcome evaluation design, with two randomised groups receiving services in different ways. The group receiving services in the standard way will be called 'S' and the group receiving joined-up services will be called 'J'. The outcome evaluation will give priority to researching and comparing the outcomes for the two groups.

[ top ]

Data collection strategies and procedures

Project staff have already undertaken considerable work on the data collection plan, particularly in negotiating with Centrelink, other holders of data, and the service-providing agencies. The data collection and management for this project will be particularly complex, involving a number of sources, data in different formats, collections at different points in time, and the need to integrate the data into a single SPSS data set.

Data collection will include:

The project's research and evaluation officer will be responsible for establishing and maintaining a single merged data set. The Principal Investigator will provide consultation regarding the structure and management of this data set.

(i) Existing data sets

Data will be collected from existing data sets, in accordance with consent granted by participants. (NB: Provision of consent is a condition of entry to YP4. These data will relate to recorded income, employment, unemployment, and periods of ill health.

YP4 staff have developed protocols with holders of data to ensure timely access to all relevant data. The YP4 research and evaluation officer will undertake preliminary consolidation of all datasets, including de-identification. These data will be collected annually commencing March 2005 and continuing until January 2009.

[ top ]

(ii) Data from service providers

J group participants' case managers within YP4 will provide structured feedback on individual participants' progress at 3 months, 6 months, 9 months, 12 months, 15 months, 18 months, 21 months and 24 months. The indicators detailed in Appendix 1 will form the basis of this data collection.

(iii) Individual interviews with group S and J participants

Individual interviews, either face-to-face or by telephone, will be conducted as soon as possible after the participant joins the trial, and at 6 months, 24 months, 36 months and 48 months thereafter. Interview content will relate directly to the objectives of the YP4 trial (see above).

The YP4 project team has negotiated with Centrelink that their in-kind contribution to the project will include conducting individual interviews with all participants in both groups at set points in time following a participant's acceptance into the trial. These interviews will be up to one hour in duration. The possibility of using telephone surveys has been canvassed by Centrelink and is yet to be considered by the Ethics and Evaluation Advisory Group. The Principal Investigator will be involved in training the staff who will be conducting these interviews.

Participants will receive $30 worth of vouchers in return for each interview. Vouchers will be used in order not to compromise participants' eligibility for income support.

(iv) Focus groups

Four focus groups with YP4 case managers will be conducted over the two year period of YP4 service delivery i.e. March 2005 to June 2007. The purpose of these groups for the outcome evaluation will be largely to explore reasons for findings as they emerge from the other data.

Four focus groups with Group J participants will be conducted over the two year period of YP4 service delivery i.e. March 2005 to June 2007. The purpose of these groups for the outcome evaluation will be largely to explore reasons for findings as they emerge from the other data.

[ top ]

Data analysis

Data analysis will be carried out following each major data collection event, at present planned for 3 months, 6 months, 9 months, 12 months, 15 months, 18 months, 21 months and 24 months. The YP4 research and evaluation officer, as the person with responsibility for the hands-on maintenance of the data, will assist with the data analysis, and will be involved in review of output and plans for further analysis.

The Principal Investigator will report on findings quarterly in 2005 and 2006, and annually in 2007, 2008 and 2009. The reports will include a brief written report and a presentation. A major report will be presented by December 2007, with addendum reports in December 2008 and December 2009.

Dissemination of findings

Dissemination of findings will be a crucial aspect of this research. Considerable time and effort will go into working with the project manager, the Ethics and Evaluation Advisory Group, and the Inter Agency Coordinating Committee (IACC) to plan and prepare publications, media releases, submissions and presentations. While excellent work has been done to prepare the foundation paper and trial proposal, ongoing review of the literature will be necessary throughout the project.

5. Number, type and age range of subjects

A total of 480 persons aged 18-35 years will participate in the research. All participants will be homeless jobseekers.

[ top ]

6. Any other relevant comments

This research is an outcome evaluation for the YP4 trial. Arrangements for this trial have been the subject of about three years' work by the participating agencies, Hanover Welfare Services, Brotherhood of St Laurence, Melbourne Citymission and Loddon Mallee Housing Services. Early work involved noticing the unsatisfactory outcomes of work with homeless jobseekers, reflecting on the reasons, and examining the literature. Further work involved developing a model that would have a better chance of success with the target group. The proponents have obtained funding and in-kind contributions from a wide variety of government departments, philanthropic organisations and in-house sources.

Recent intensive work has involved negotiations among the service delivery agencies and development of protocols with relevant government departments.

Process Evaluation

The process of joining up government programs and services in client centred ways is complex and time-consuming. Some examples of the complexity of joining up were outlined in the 'Planning and Implementation' section of this report.

The process evaluation will enable a better understanding of the complexities of joining up. The YP4 manager doubles as the principal investigator of the process evaluation. The key questions guiding the process evaluation are: What is our experience of joining up services and programs for young homeless jobseekers, how best can we understand these processes, and what is their value?

Before answering these questions, it is important to consider what joined up services and programs might mean or might look like. The term 'joined up' can be understood as a catchy new name to describe a longstanding concern for better coordination between different parts of a system (Chandler 2000). Coordination of policy is an 'eternal' problem which dates back to Chadwickian reforms of British local government in the 1830s (6 2004). 'Joined up' is mostly used in relation to government, having been coined and then enshrined in legislation by the Blair government in the UK in the late 1990s. Most of the literature about joined up government assumes that both vertical and horizontal integration will result in more seamless services and an enhanced customer experience of interacting with government. Di Maio (2004) acknowledges that the target of efforts to join up are not limited to horizontal and vertical integration of government departments, but extend to process hierarchies like management, and operations as well as between public and private entities.

[ top ]

For the purposes of YP4, it is useful to understand joined up practice as a multi-dimensional issue that is also multi-layered (multilevel). Understanding joined up practice in this manner facilitates identification of three critical issues:

A schema has been developed to facilitate articulation of the lenses through which the process of joining things up can be understood. This schema guides the process evaluation, and is reproduced in appendix 2. The schema is an advance on the notion that joined up government is a matter of vertical and horizontal integration. Such a notion is implicitly two-dimensional. A multidimensional schema better captures the complexity of joined up practice. To explain further, the dimensions of joined-up practice as understood by YP4 are:

Experiential

Of great interest to YP4 is the lived experience of joining up as felt by clients, case managers, participating services, agencies who have relationships with YP4 providers, government departments and community members in YP4 localities. Integrated services may be experienced in qualitatively different ways than fragmented services.

[ top ]

Programmatic/Technical

The programmatic dimension of joined up practice is technical and practical - involving managing the interface and boundaries between programs and services. This dimension is mostly about implementation. For example, joining up programs may require the sharing of a database. In such instances, it is important to ensure that the right people see the right information on the right screen at the right time.

Conceptual

This dimension of joining up is about the way in which the joining up process is understood, for example, what is meant by joining up, how it is understood and thought about and what language, metaphors and models are used to explain it. Evidence of joined-up thinking must be apparent in the ways that service providers and government officials talk about planning, delivering and managing integrated services.

Systemic/Structural

This dimension of joining up operates at a macro-policy level. The policy platforms which ground the work of YP4, the contractual arrangements which shape it and the organisational structures which support it must be coherently integrated. For example, the accountability processes, leadership and governance of YP4 must all be joined up.

Cultural/Environmental

The cultural dimension concerns the impact of the environment in which joining up happens such as how the organisational culture allows/promotes or inhibits joining up, the extent to which there is a cultural match or mismatch between current practice and joined up practice and the extent to which there is an alignment of values and collaboration and partnerships are valued and rewarded.

The issues mapped here have different impacts on YP4's efforts at joining up, depending in part on where within the system energy is invested. This suggests that joined up practice needs to occur at various levels simultaneously.

[ top ]

Primary

ie. Internal to YP4. For example, the act of joining up services when they are delivered to clients, what it takes to deliver a joined up service, how to communicate through the various channels of the governance/management structure, how the different components of YP4 talk to each other and 'make it happen'.

Secondary

How do local trial providers network within their localities in order to develop the options needed for a joined up service, what different strategies are employed at a local level, or at the overall YP4 trial level, at the sectorial and departmental levels to ensure the secondary service systems work with YP4 and that services and programs can indeed be joined up.

Tertiary

How does the broader system of service delivery generally impact on YP4's efforts at 'joining up'? What are the challenges of joining up the 'silo's' of the service system in a more than temporary way? What could joining up the delivery of services look like… holistically - what will need to be done on the macro level for this to happen?? This dimension is about policy development and the political.

Consider row 1, column 1 of the schema. Clients' lived experience of receiving services is fraught. Often the client has to do the joining up of services themselves, they find long waiting lists for service, strict eligibility criteria and slight changes in status can lead to being ejected from a service and clients often have to fit in with the model of service being offered. For YP4, concessions have been secured that will enable fast-tracking of clients into services and will provide temporary exemptions from the business rules that are most likely to result in ejection from a service - like zero job search requirements for 6 months, like two years stay in transitional housing instead of the usual one. YP4 is committed to work with clients for two years regardless of changes in their status, so becoming a mother, going to prison or moving to another area does not automatically mean that the YP4 service will cease.

Consider also row 3, column 4 of the schema. This is where governance fits in. YP4 is overseen by an interagency coordinating committee which is comprised of the four partner agencies, representatives of each of five government departments which support YP4 and three peak bodies. A lot of important work occurs in committee meetings where members share a common goal and task.

[ top ]

Colloquially, the schema of joined up practice is being referred to within YP4 as the ice cube tray model of joining up. In the same way as packing the ice cube tray into the freezer when some of the little cubes do not contain water can be inefficient, so too YP4's hypothesis is that ignoring any one of the various dimensions of joined up practice will ultimately result in the potential of more sustainable employment and housing outcomes for young homeless jobseekers being less than fully realised.

To date, most of the effort invested in the process evaluation has gone into developing the framework or schema for understanding joined up practice. Future priorities for the process evaluation are to determine a methodology specific to each of the dimensions and levels of joined up practice and collate evidence specific to each dimension and level. At this stage, it is clear that the evaluation of the acts of joining up which occur within YP4 will be a participatory, organic and developmental process.

Financial Evaluation

The financial evaluation includes both a cost-benefit analysis and a cost-effectiveness analysis. It is being overseen by principal investigators who are independent of all the partner organisations. Professor Jeff Borland, Dr Roger Wilkins and Dr YiPing Tseng, researchers from the Department of Economics and the Melbourne Institute of Applied Economic and Social Research at the University of Melbourne, are the co-Chief Investigators undertaking the financial evaluation.

The following information is an excerpt from an application for funding submitted to the Australian Research Council by the co-Chief Investigators.

Cost-benefit analysis involves estimating the value of benefits and costs to society from a program or policy. In the financial evaluation of YP4, application of cost-benefit analysis will involve three main stages: (a) Identifying the main sources of costs and benefits associated with the YP4 trial; (b) Calculating an estimate of the monetary value of each specific benefit and cost; and (c) Aggregating estimates of costs and benefits to calculate the cost-benefit ratio for the program.

[ top ]

The main potential benefits of YP4 derive from its effects on employment, housing status, and health/well-being. Improved outcomes in these areas may yield benefits from: reduced direct cost to government (for example, reduced unemployment payments and rent assistance, and increased tax payments); and reduced usage of government and not-for-profit welfare services (for example, reduced demand for public housing and SAAP services, and reduced demands for health services and contact with criminal justice system).

The main costs associated with YP4 will be the extra time spent by case managers in managing YP4 participants, and extra expenditure on programs for participants relative to the benchmark of non-participants who would not have access to integrated services.

An important aspect of the project will be to describe in a precise manner the policy effect being identified. Estimates of the effects of a program on participants measure the effect of receiving services as a participant, relative to services provided to non-participants. In this project what is being received by participants, and not by non-participants, is the 'joining up' of services, rather than necessarily access to services per se. Hence, for the project to provide a meaningful basis for future policy development, it will be critical to document and describe how the experience of participants has differed from non-participants.

The methodology of the project is to apply the approach of cost-benefit analysis to the YP4 trial. Application of cost-benefit analysis involves estimating the value of benefits and costs to society of the YP4 trial. This involves the following three main stages:

Stage (a): Identifying the main sources of costs and benefits associated with the YP4 trial

[ top ]

Identification of the main potential benefits and costs of the program has already occurred in development of the methodology for data collection before and during the trial. (One of the Chief Investigators, Professor Jeff Borland, has been involved in development of this methodology.) In relation to potential benefits, data will be collected on 22 types of outcome measures from YP4 participants and members of the control group that relate to labour market, housing, health, and community engagement. Relating to potential costs, data will be collected on services accessed and payments made to YP4 trial participants and control group members, and on time and other costs associated with case management of trial participants. Part of this project would be collecting data on costs associated with case management.

Stage (b): Calculating an estimate of the monetary value of each specific benefit and cost

This stage involves two steps - first, estimating the impact of the YP4 trial on each outcome, and then calculating a monetary value of that outcome. For example, one type of outcome that will be examined is employment. The project will then provide an estimate of the effect of participation in the YP4 trial on the probability of being in paid work at various stages after commencement of participation in the trial. This can be converted into a monetary value by estimating the value of reduced/increased unemployment (or related) welfare payments and on tax payments to government due to that effect of the YP4 trial on participation in paid work. Other examples of outcomes to be examined will be tenure in secure housing; engagement with local community; health status; and participation in formal education programs.

The YP4 trial is being implemented as a random experiment. This means that the experimental approach is likely to constitute the basis of the empirical approach to estimation of impacts of the YP4 trial on outcomes. In an experimental approach, individuals in a population are randomly assigned between participation and non-participation in a program, and the outcome of interest is compared between those groups. Random assignment should generate groups of participants and non-participants where each group has the same average characteristics. Hence the comparison between the two groups can be thought of as a comparison between two 'individuals' who have the same characteristics, except for whether they are assigned to participate in the program. Comparison of outcomes for the two groups will therefore provide an estimate of the causal impact of program participation.

[ top ]

Often however it has been found that problems can arise with implementation of a random experiment methodology that mean that it is invalid to undertake empirical analysis assuming randomisation (see for example, Breunig et al., 2003). For this reason an important aspect of this project would be to validate whether the YP4 trial has been undertaken in a way that does allow it to be treated as an experiment. Besides a process evaluation of the trial, this involves performing statistical tests for whether the potential determinants of outcomes are independent of who has been assigned between the trial and the treatment group (that is, whether the distributions of individual characteristics are similar between treatment and control group). In the case that treatment and control groups are similar enough, the impact of the YP4 trial can be calculated (as has been described above) by comparing mean outcome measures between the participants and control group. In the case that assignment between the trial and control group is not random, then it is necessary to use a quasi-experimental matching approach. The matching method estimates the program impact by comparing outcomes for program participants and non-participants in the time period(s) after the program commences. That is, it uses data on outcomes of non-participants in the period after program commencement to estimate non-participation outcomes for the group of participants. The term 'matching' is used since the comparison is made conditional on observable covariates that affect both the outcome and whether individuals are assigned to the program. For evaluating this program it would be most appropriate to use the 'propensity-score' matching method (see Blundell and Costa-Dias, 2001, Sianesi, 2004, and Borland et al., 2005).

An alternative approach for estimating the impact of YP4 trial participation on outcomes is through estimation of hazard function models for the determinants of exit from unemployment or receipt of unemployment/welfare payments. In this approach the program impact is identified as a time-varying covariate for duration - see for example Abbring and Van Den Berg (2003), and Van Den Berg et al. (2004). For some outcome measures, such as tenure of secure housing, a duration type approach may be applied to estimate the impact of the YP4 trial.

The basis for the evaluation of the impact of the YP4 trial on outcomes will be a data set being compiled by Hanover Welfare Services. This data set will merge data from:

[ top ]

While this data source will be a primary input to estimating effects of the trial on outcomes, it is also expected (based on the investigators' previous experience of using data from administrative sources, and that data pooled with other sources) that it will be necessary to spend a large amount of time in converting the data set to a form that will be usable for empirical analysis - for example, validating and seeking clarification about issues that arise in using the data, and generating descriptive statistics. The other main task in this part of the project will be programming estimators to use to calculate the effects, and undertaking that estimation. Given the large number of outcome variables being used in this study (22), this is also likely to take a considerable amount of time.

Part of the approach in the project will be to generate measures of the impact of the YP4 trial, and cost-benefit ratio, at several points through the trial. This will have the advantage of providing an early feedback on outcomes, but as well will contribute to the international literature that suggests significant differences may exist between short-term and long-term program impacts (for example, Lechner et al., 2005).

The second step in estimating a monetary value of benefits will be to calculate the monetary value of impacts of the YP4 trial. This will involve collection of information on welfare payments and taxes (to estimate values of reduced receipt of payments), and costs of services (to estimate values of reduced usage of services); and matching that information to estimates of impacts on outcomes.

Calculating estimates of the costs of the YP4 trial will involve comparison of usage of services and receipt of payments between trial participants and the control group, and collection of data on welfare payments and costs of services in order to value those differences; and collection of data on time and other costs of case management and collection of data to value those costs. This latter activity is likely to involve visits to the collaborating organisations.

Stage (c): Aggregating estimates of costs and benefits to calculate the cost-benefit ratio for the program

[ top ]

In undertaking this stage it will be important to recognise what factors it may not be possible to have valued in monetary terms, and the implications for the cost-benefit ratio. The cost-benefit analysis will allow separate analysis of the private benefits to YP4 trial participants, and an overall analysis of social costs and benefits.

Main tasks and timeline

Tasks in the research project would be undertaken in an order, and with a sequencing, to match with the implementation of the YP4 trial. The 2-year period of the trial is expected to conclude in mid to late 2007. The proposed time line of activities is as follows:

Year 1 (2006)

[ top ]

Year 2 (2007)

Year 3 (2008)

Research findings

The specific research findings from this project would be expected to include:


[ top ]