3. Methodology

3.1 Ethics approval

Prior to commencement of the study, approval for the project was received from the UNSW Human Research Ethics Committee (HREC) (approval number HREC 06170).

3.2 Survey design

In order to inform the design of a survey instrument suitable for examining self-reliance among clients of SAAP services, the researchers undertook an extensive review of existing self-reliance measures, both nationally and internationally. This included systematic searches for validated and non-validated psychometric tools using academic databases, such as the Mental Measurements Yearbook, PsycINFO and the Social Sciences Index, as well as databases of non-published literature using the GrayLIT Network. Further to this, through networking and generic Internet searches, the researchers contacted government and non-government organisations that have examined the viability of measuring self-reliance, independence and/or resilience in client groups. These search efforts suggested that there were in fact very few tools already in existence that were relevant to or suitable for this task.

Those that were most relevant were those developed in earlier research specifically for SAAP, as discussed above (Browton, 2001; Baulderstone and Talbot, 2004; Kunnen and Martin, 2004; Kunnen, Lee and Martin, 2004). The researchers drew on this work and the facets of self-reliance outlined in the SAAP V Multilateral Agreement (Commonwealth of Australia, 2005).

To ensure thoroughness, the researchers also contacted all SAAP services in Australia, requesting information about any internal surveys used to gauge client change. While a number of services had devised tools for this purpose (and passed them on to the researchers), most were found to be too limited in scope. They were also designed for completion by service providers rather than by clients, and they did not explicitly focus on the concept of self-reliance.

In spite of the conceptual and methodological complexities outlined in the previous section, it was imperative that the questionnaire itself be short, clear and simple – a tool that would encourage participation by clients and services alike. SAAP services are typically over-worked and under-resourced, with staff members having only limited time to contribute to the study. Moreover, a significant proportion of clients are likely to have only limited literacy skills (Buhrich, Hodder and Teesson, 2003). Thus in order to secure cooperation from services the survey had to be simply worded and laid out in a way that clients found easy to understand and follow with only limited assistance.

3.3 Stakeholder consultations

Having completed an initial draft design for a survey instrument, the researchers undertook consultations with key stakeholders to discuss the instrument and the study more generally. This was important, not just to get comments on the draft questionnaire, but also to build support for the study as a whole in order to maximise service participation in the research.

The purpose of the consultations was three-fold:

to inform the homelessness sector about the study;
to invite the participation of key stakeholders in the study; and
to invite detailed comment on the survey instrument, in order to ensure its practicality, the appropriateness of its content and layout for the diverse client groups, and its ability to meet the overarching aim and objective of the project.

The study was publicised through the SAAP Newsletter and the researchers subsequently emailed all SAAP services to inform them about the study, to let them know that some would be chosen to participate in the survey, and to canvass ideas and experiences of any similar attempts to assess client outcomes. A number of services replied expressing interest in the study and offering comments, ideas and examples of tools used in their services.

Two consultation meetings were held, one in Sydney and one in Melbourne. These involved representatives of SAAP State departments, peak homelessness organisations, and SAAP services. Comments were sought by phone and email from a number of organisations whose representatives were unable to attend the meetings. Subsequently one housing service and one peak homelessness body, both in Melbourne, also offered to circulate the draft questionnaire to their consultative client reference groups for comment.

Collectively, those consulted were generally supportive of the project; they recognised the need to develop a standardised yet practical way to gauge client development within SAAP services. Using their knowledge of SAAP services and clients, they offered constructive views on the scope of the survey; the appropriateness of individual survey items; the phrasing of items; survey layout; and the practicalities of survey administration through the agencies. Modifications to the draft survey were made accordingly.

However, stakeholders also raised a number of concerns about the study.

There was a view from some service-provider and advocacy groups that the research was driven by the Commonwealth Government’s need to have a simple quantitative answer to what is a highly complex question. There was concern that this might produce data that did not reflect the realities but which could be used by the Commonwealth in ways that might adversely affect services.
Related to this was a view that self-reliance was a ‘program logic’ concept that had little to do with the reality of people’s lives and service delivery; in particular, it failed to recognise the differences between people which arise from their varied histories of homelessness and the consequent paths they may take out of homelessness.
It was argued that SAAP should not be seen primarily as a housing program but rather as a support program. Most clients are not in a position to become fully self-reliant through SAAP inputs alone, given the limitations of available funding. Thus a failure to achieve full self-reliance outcomes should not be seen as a sign of system failure.
It was argued that a key determinant of self-reliance is income through employment, but SAAP can only have a limited impact on client employability. The focus therefore needed to be on the ‘building blocks’ of support through case management that helps people on their way to self-reliance.
In this context, stakeholders argued that it was important to recognise that very small changes in behaviour can be crucial. Thus engagement with services is in itself often an important step on the road to greater self-reliance.
There was a view that determining on a national scale how far individual clients improved their circumstances as a result of SAAP support required a larger and more detailed study, with a longitudinal focus.
There was also concern about the differences in service approaches (e.g. ‘housing first’, or a focus on case management and ongoing support) and how these might influence people’s responses to the survey.

We return to discussion of these issues in the analysis of survey data and the concluding chapter of the report.

3.4 Survey piloting

Following the stakeholder consultations, the modified survey was piloted with four groups of SAAP clients – single men, young people (through two SAAP services in Sydney), and the two client reference groups in Melbourne. While attempts were made to include women with children in the pilot phase, this turned out not to be possible within the timeframe. To ensure that issues specific to this client group were taken into account, the survey instrument was reviewed by a peak non-governmental women’s agency, and comments were incorporated accordingly. Details of the pilot phase of the project are provided below.

Single men

The survey was first piloted in a large, metropolitan SAAP service located in Sydney, NSW. The service supports men with alcohol and other drug problems, aged 18 years and over, by providing safe, overnight accommodation, food, clothing and showering facilities, as well as specialised staff who facilitate rehabilitation.

Seven clients who visited the service during one evening were invited to complete the survey and to participate in a confidential interview with one of the researchers. The purpose of the interview was to understand their experiences with and views on the questionnaire survey, with specific reference to its scope, ease of completion, perceptions of the relevance of survey items to the concept of self-reliance, and whether any items caused distress.

Six clients completed the survey independently, while one required assistance because of a visual impairment. As there were only two staff members at the service and they were occupied with meal duties, one of the researchers assisted this client.

Overall, the clients found the survey to be simple and straightforward. It did not seem cluttered, the text was not too small, and the length of the survey was seen as acceptable. A few clients indicated that some individual survey items were lengthy, particularly where there was a long list of possible responses. However, this became less of a problem when they realised that many of the response options were not applicable to them.

All of the clients stated that the questions were simply worded and self-explanatory, and clarification about a survey item was sought only once. This suggests that the clients were able to understand the questionnaire and the attached instructions, including one respondent who later reported that he did not attend school beyond Year Seven. Reflecting on the skill levels of other friends and acquaintances who accessed SAAP services, several clients thought that they too would experience little difficulty in completing the survey.

Most of the clients were comfortable with answering all the survey items. Some advised that other surveys they had completed had requested far more personal information – so this one was relatively innocuous. However, one client spoke of feeling uncomfortable with questions that related to children and chose not to answer these. This was, he said, because he had been denied contact with his own two children.

One client suggested that the reasons clients present to a homelessness service are often complex and not as direct as suggested by the survey. Another was dubious about the suitability of such a survey for short-term services. He expressed some cynicism about the capacity of such services to facilitate self-reliance among clients when the service opens at 5.30 pm and clients are required to leave by 7.00 am the following day.

Survey response times ranged from five to 25 minutes, with an average of approximately 15 minutes. The clients saw this as acceptable, particularly those who had completed other surveys that sometimes took much longer.

Young people

The second pilot of the survey was conducted in a SAAP service providing short and medium-term accommodation for young people in Sydney. For reasons of client privacy and confidentiality, the service providers themselves preferred to pilot the survey with clients and to provide feedback to the researchers. Four young clients of the service, three of whom were young women, were invited to complete the survey and discuss their experiences with one of the service providers.

According to the service providers, the young clients generally considered the survey to be lengthy. One client only completed half of the survey for this reason. Notwithstanding this shortcoming, there were no identified problems with the content of the survey or sensitivity around particular questions. However, it should be noted that most of the clients apparently completed the survey with some support from staff.

Staff from the pilot agencies were also positive about the questionnaire; one manager for example, suggested that it was the most appropriate instrument of its kind that he had seen developed for the clients with whom he worked. An examination of the completed surveys confirmed that the clients appeared to have no major problems with the format or logic of the individual items. Consequently, there were no significant revisions to the questionnaire at this stage.

Client reference groups

Following the stakeholder consultation in Melbourne, representatives from one peak homelessness organisation and one SAAP service offered to discuss the survey with their respective client reference groups. These groups are comprised of current and former SAAP clients and meet regularly to inform policy development and service provision in the homelessness sector.

Both reference groups provided constructive comments on the survey. Their overall responses were overwhelmingly positive and they were interested in the themes explored within the survey, viewing it as a useful opportunity for personal reflection about their own achievements. Collectively, they advised that there were no serious problems with the format of the survey, the language or the comprehensibility of its items. All but one member of the first group completed the survey unaided, while the remaining member required assistance because of limited literacy skills. Members took between 10 and 25 minutes to complete the survey, which they all saw as acceptable

Members of the second reference group identified a number of ways to improve the survey, particularly for clients with limited literacy skills or limited confidence. Suggestions included rewording specific items; reconsidering the array of response options available; rearranging survey items that appeared thematically similar; and improving the aesthetics of the survey. Such feedback underlines the importance of participatory research practices that give key stakeholders the opportunity to inform the study and to maximise the value of the research.

3.5 Survey content

The questionnaire was amended to take account of the piloting and reference group comments described above, and the final version included items about the following areas (see Appendix A for full questionnaire).

homelessness histories and circumstances prior to accessing the SAAP service;
reasons for accessing the service;
understandings of the meaning of self-reliance;
factors and support systems that facilitate or hinder ‘getting back on your feet’;
changes in self-reliance since accessing the SAAP service, with reference to:
- accommodation
- income and employment
- self-care
- relationships
- contact with community and government organisations
- wellbeing
- mobility
- care of children living with them
- literacy among those whose first language is not English
Other sources of support;
forms of support still needed;
housing, employment and income status before approaching the SAAP services and now; and
client demographics.

3.6 Selecting the survey samples

Sampling framework

Because there was no direct way to contact homeless SAAP service users other than through the service agencies, it was decided to operate a two-stage sampling method, whereby the researchers would first select a sample of services and then invite those services to recruit clients for the survey.

FaHCSIA supplied the SPRC with a database of funded SAAP services, containing location and contact information, main service-types, primary and secondary client-types, and the most recently recorded annual number of completed service episodes or ‘support periods’ (as a proxy for service ‘size’). Agencies funded as peak bodies, new services only recently established, and those providing only telephone advice and support, were excluded as they were not regarded as relevant to the study. From this adjusted total of just under 1200 agencies, we drew a random sample pool of 200 services.

The selected services were provided with detailed information about the study and invited to participate. They were also advised that the final sample of services selected would be smaller, so that not all who were invited to participate would be included. A total of 113 services initially agreed in principle to participate, and from these a smaller sample of 65 services was chosen, based on a random sampling stratified by State/Territory jurisdiction, service-delivery model, primary client target-group, and agency size (as indicated by the number of support periods between 2004 and 2005). Some minor adjustments were then made to the sample to ensure a range of metropolitan, regional and rural locations, and a good representation of Indigenous-specific services.

Criteria for client selection

There was considerable discussion at the planning stage, and in the consultation meetings, about what criteria services should use for client recruitment. SAAP administrative data show that around 80 per cent of clients have only one support period in a financial year, but the present method of data collection is not able to fully capture support periods crossing years. Nor is there much known about repeat periods of service use in later years. While most clients have short support periods, much of the resources of SAAP services go into working with those with longer periods of support. As discussed above, the concept of building self-reliance through service support really only makes sense when applied to those with more than a passing use of services. Thus there was general agreement that the focus should be on this group. It was finally determined that clients would be eligible to participate in the study if they had been known to the service for a minimum of four weeks, either continuously or intermittently, and had received case-management support from the service during this period.

3.7 Survey administration

To facilitate data collection, each participating service was asked to nominate a staff member whose role it was to liaise with the researchers and collect data. These individuals were provided with a training module comprised of supporting documentation and additional telephone support.

The documentation outlined the client-screening criteria; the procedures to be followed during and after each administration of the survey; issues of confidentiality and privacy; quality control processes; reimbursements for participating services and clients; and contact details for the researchers and the Ethics Secretariat. The module was disseminated to the services, accompanied by survey materials and pencils, promotional material about the project, and a reply-paid, self-addressed envelope in which to return completed questionnaires. Participating clients were offered $10 vouchers as compensation for their time and contribution, while the services themselves were offered an administrative payment of $100-$200, depending on their quota of clients.

The initial aim was to recruit 750 clients through the 65 services. This figure was identified as sufficient to accommodate some potential attrition without jeopardising the sample size. The number to be recruited from each service was contingent on its size. ‘Small’ services (defined in terms of fewer than 100 support periods recorded in the database) were asked to recruit a maximum of 10 clients; services of medium size (100 to 500 support periods) were asked to recruit a maximum of 15 clients; and large services (more than 500 support periods) were asked to recruit a maximum of 20 clients.
The recruitment phase of the study was planned to occur during a specified two-week period from early February 2007. However, in light of recruitment difficulties (and the subsequent need to increase the initial sample), this was not possible. Some of the nominated staff members advised that current client numbers were relatively small, and they were unlikely to meet the suggested quota. This was a particular problem over the post-Christmas holiday season. As it turned out, the eligibility criteria also placed some restrictions on the ability of services to recruit the required numbers of clients, and from the questionnaires returned it also appeared that a few had been recruited who did not fully meet the criteria. Services not returning questionnaires within the prescribed period were contacted with reminders by email and telephone.

A further problem that emerged during the recruitment phase was that, in some cases, the recorded support periods supplied to the SPRC by the SAAP database did not correspond to the actual volume of clients and service-use. As well, there were a few services that reported having a different primary client-group or service-type from that recorded in the database; and a number of the contact details were incorrect or out of date.

Given the difficulties some agencies experienced in recruiting eligible clients, a decision was taken to increase the service sample to 95 (subsequently amended to 94 because of a duplication), again ensuring that the sample was representative of services nationally. The survey was closed off in early May 2007, with 75 services having participated and provided completed questionnaires for a total of 630 clients. While this was fewer than initially anticipated, it still provided a sample large enough for detailed analysis and one that appears to be broadly representative of SAAP clients as a whole (as shown in Section 4)

3.8 Limitations of the survey

There are inevitably some limitations involved in a survey of this kind. First, as a number of stakeholders have argued, one-off quantitative surveys are not necessarily the best method for capturing the complexities of the changes occurring in the lives of a highly diverse population of homeless people, especially when such changes are non-linear or discontinuous. Ideally, longitudinal follow-up of clients over time, using individual tools of goal attainment, might provide a richer and more complete picture of outcomes achieved through service use. However, the resources necessary to mount such a study on a national and representative level would be considerable and were not available for this particular study. That does not mean that the results of this one-off exercise could not be used to inform a future, more comprehensive study.

Another limitation is that reliance on SAAP services to recruit survey participants risks the possibility of selection bias in the sample. We have no evidence that this happened in the present study, although it does appear that a small number of participants may not have met in full the eligibility criteria prescribed.

As well, the use of a self-completion questionnaire risks excluding clients with limited literacy skills, although the pilot suggested this was rarely a problem and staff were encouraged to offer assistance to clients who needed it. The returns suggested that most of those participating understood most of the questions (though some problems arose from the layout of one set of questions), but we have no way of knowing to what extent different clients received more or less help from service staff, or whether clients with greater literacy or cognition difficulties may have chosen not to take part.

One other aspect of a self-completion questionnaire that might be seen as a limitation is that it relies on subjective self-perception by clients in their assessments of the extent to which they have progressed towards getting back on their feet, rather than being based on some ‘objective’ measure of achievement. However, while some corroborative view of achievement from case managers or other service staff might have been useful, it is our view that clients’ own assessment of their progress, or the lack of it, still provides an important measure of the impact of SAAP service use.

Bearing in mind all these limitations, we are confident overall that the results provide a valid picture of how a broadly representative group of homeless people across Australia view the progress they have achieved in getting back on their feet as a result of receiving services under the SAAP program.

3.9 Case studies

The final element of the research was the assembly of the illustrative case studies, based on in-depth interviews with clients and ex-clients whom SAAP services identified in their view as having made significant progress towards getting back on their feet after being homeless. The purpose of these was to explore in a more qualitative way issues which arose in the survey and to seek clients’ own more detailed explanations of how support from services had contributed to their increased self-reliance, what aspects of their lives had changed and how they saw their futures.

The clients were recruited through SAAP agencies that had taken part in the client survey and were based in the Sydney and Blue Mountains regions.

The topic guide for the in-depth interviews is presented as Appendix B to this report.

Return to top