National SAAP Client Satisfaction Survey

Previous: Appendix D: Verbatim Suggestions for Improvement

Appendix E: Technical Notes

9.1 Introduction

9.1.1 Background

The Supported Accommodation Assistance Program (SAAP) is governed by Commonwealth legislation. It is:

a 'support program assisting people who are homeless or at risk of homelessness, through a range of support and supported accommodation services. It is an important part of Australia's overall response to homelessness and of the broader social safety net designed to prevent disadvantage in the community'.³²

SAAP commenced as a joint Commonwealth and State/Territory program on 1 January 1985 and since then it has undergone growth and significant change in the services provided. At the outset, the program was focussed on providing a 'safe environment for the night'. Since then, agencies involved are much more focussed on the 'particular needs of clients and are designed to achieve maximum independence for SAAP clients'.

During its lifespan, SAAP has undergone reviews and evaluation, developing as it has passed through four agreement stages. The current agreement (SAAP IV) is scheduled to conclude in 2005.

A measure of client satisfaction is one component in evaluating SAAP. In 2001, the Australian Federation of Homelessness Organisations (AFHO) was commissioned, along with partners, to develop options for client satisfaction measures in SAAP. Findings were reported in June 2003.³³

CBSR was commissioned by FACS to progress to a further stage of collecting client satisfaction data at the national level. This involved further development and testing of a survey instrument to collect valid and reliable data on client satisfaction. Information and findings from this research will feed into an evaluation of the SAAP in early 2004.

9.1.2 Objectives of the research

The overall objective of the research is to measure client satisfaction by clients against key outcomes at state/territory and national levels, against:

quality of engagement and support by agency staff - competent, professional, tone, empathy, friendliness, interested,
quality of information and communication;
helpfulness of support programs offered;
standard of accommodation and other facilities;
outcomes achieved with regard to the service clients expect and whether their expectations/needs were met; and
timeliness and responsiveness of the service to meet client needs.

The research tasks to achieve this were to develop, coordinate and implement a client satisfaction survey on a national basis.

A scoping meeting involving the CBSR team; Barbara Beatty and Doug Limbrick from the Commonwealth Department of Family and Community Services; and representatives from New South Wales, Western Australia, South Australia and Victoria, was held on 16 September 2003. This meeting reconfirmed the objectives of the research and the proposed methodological approach.

[ top ]

9.1.3 Purpose of this document

This document is intended to provide a record of methodology employed in the 2003 National SAAP Client Satisfaction Survey.

9.2 Previous research

9.2.1 AFHO findings

The AFHO report included the following major findings:

There is general support from both SAAP clients and agencies for the introduction of client satisfaction measures in SAAP;
The development and implementation of client satisfaction measures needs to the consultative, flexible and meaningful at the agency level, fully resourced and not burdensome on agencies or clients;
Measuring client satisfaction is not equivalent to measuring overall service quality;
Development of client satisfaction measures in SAAP should be undertaken in the context of a broader quality improvement framework;
That only necessary information is collected should be ensured;
Relationships with consumers and consumer groups need to continue to be built to increase reliability and representativeness of client satisfaction data;
Accompanying children need to be addressed in a separate project; and
Best practice ethical procedures need to be developed with all stakeholders.

9.2.2 Aspects to measure in SAAP client satisfaction

Aspects of service delivery that are identified in the AFHO report as focal for client satisfaction measures include:

the quality of engagement and support by staff;
the quality of information and communication;
the helpfulness of support programs offered; and
the standard of accommodation and other facilities.

On this basis, AFHO developed a set of 'core' questions under these general headings. These questions, using a 5-point Likert scale of satisfaction were pilot tested in eight locations by AFHO. AFHO also developed a set of supplementary questions, relevant to different target audiences, but these were not piloted.

CBSR developed a first draft of the questionnaire and met with the Committee on 24

September 2003 to discuss the questionnaire content. CBSR undertook qualitative research to pilot test the questions and, after revisions, administered the survey on a national basis to SAAP clients.

[ top ]

9.3 Introduction to 2003 research methodology

9.3.1 Overview of research method

The research method adopted a five stage approach, as follows:

At stage 1, a scoping meeting was held with the Steering Committee to confirm and finalise the research plan (including confirmed methodology, recruitment protocols, sample plan, timelines and team responsibilities). This meeting was held on 16 September 2003. Ethics approval was sought at this time.
At stage 2 a two-tier stratified random sample plan was designed and implemented. Advance letters were drafted and distributed to agencies. Agencies were asked to nominate a 'champion' and a teleconference was held with all agency 'champions' to provide the agencies with information relating to the census of their clients, discuss and coach 'champions' on the recruitment. Following this, a second mail-out was conducted, whereby all agencies were invited to participate. This was in response to a low agency response rate to the first mail-out. The first tier sampling methodology is, in effect, a census of all agencies.
In stage 3, a draft questionnaire was refined through cognitive interviews and then pilot testing.
Stage 4 oversaw the implementation of the survey, adopting a call centre approach over a four-week period, with clients staying with the agency or using agency services at the time of the survey eligible for inclusion.
During stage 5, after implementation of the survey, CBSR presented FaCS with a report of findings, as well as recommendations for future surveys with this target audience.

9.3.2 Ethics approval

The AFHO report noted the requirement for ethics Committee approval prior to survey commencement, and raised the following ethical issues:

assurance of confidentiality, privacy and anonymity;
informed client consent for any use of information;
use of the information in a way that does not threaten services, workers or clients;
need for information that is critical of a service not to jeopardise the client's future access to that or other services;
if, when and how an agency worker might be informed if a client is critical of their behaviour/attitude;
possible overstating of client satisfaction or the selection of biased samples that may arise if client satisfaction information is used in decision-making about funding; and
commitment to disseminating and using client satisfaction information wherever possible to improve service delivery.

CBSR sought, and received, approval from the Market Research Society of Australia (MRSA) ethics Committee prior to undertaking any fieldwork. The following table (Table 39) shows ethical issues raised and a brief description of how each of these was addressed.

MRSA ethics approval documentation is shown at section 8.15.

[ top ]

Table 39: How ethical issues were addressed
ISSUE	HOW IT WILL BE ADDRESSED
Assurance of confidentiality, privacy and anonymity	MRSA's Code of Practice, to which CBSR is a signatory, provides strict protocols for client confidentiality. All data was collected and stored in keeping with our privacy and data security protocols. Electronic storage of data was password protected and physical data was kept on a secure premises (with security alarm system) at CBSR in Canberra. All data stored was de-identified. All interviews began with assurances to respondents, about both the confidentiality of the survey and anonymity of their responses. All communications to clients and agencies reiterated this guarantee. Furthermore, the standard quality control of monitoring calls by supervisors was not conducted during the survey to increase privacy for respondents. Along with the above procedures, including assurances, provision of information sheet and verbal explanations of the process and data use, no identifying information was disclosed. All CBSR staff involved signed a FaCS Confidentiality Agreement.
Client understanding of the research process and their involvement in it, including what is being asked of them when responding	Terminology was important throughout the process. All communications were kept clear and concise. CBSR developed communications within these parameters and all communications were approved by FaCS prior to distribution. Consultation occurred with members of each target audience in order to refine and test the draft questionnaires. Interviews in this process were moderated by CBSR staff experienced in qualitative and cognitive interviewing techniques. The questionnaire was designed to ensure that the client understood the question the way the researcher intended. All clients that were involved with an agency at the time of the survey were given an 'information sheet' by agency staff (prepared by and distributed by CBSR, with FaCS review and approval). The design of the sheet was creative to take into account low literacy or language comprehension. When given the sheet, all clients were verbally told the purpose of the research, given guarantees of confidentiality and anonymity and were told about the $5 incentive. The information sheet provided to clients included a translation (in 12 languages) directing NESB clients to a translation service. (This was provided by the Commonwealth Department of Multicultural and Immigration Affairs - DIMIA).
Informed client consent for any use of information	All clients in the service or leaving the service during the survey period were provided with the opportunity to participate. Agency staff and interviewers were respectful of the decision of any client not to participate, and did not coerce or in any other way pressure a client to participate if they opted not to. The information sheet provided to clients clearly explained that participation in the survey was voluntary. In addition, when agencies explained the survey process to clients verbally, the voluntary nature of the survey was reinforced. The purpose of the survey (to measure client satisfaction at a national level) was also communicated verbally with clients prior to them undertaking the survey, both by agency staff and also at the commencement of the survey.
Clients do not feel intimidated or discriminated against, including because of age, language, disability, ethnicity, race or sex	CBSR provided a 1800 phone number for clients to call, at no cost to the client or the agency to participate in the survey. Where possible, agencies provided a 'quiet room' with a telephone for the client for both the pilot and survey. Where not possible, clients used a public telephone, or a land-line telephone elsewhere to make the free-call. The survey was [ten] minutes in length, and was designed to minimise burden to respondents. Questions related to the service delivery from the SAAP agency and were not of a personal or sensitive nature. Interpreters were provided for all non-English speaking clients through the DIMIA service. Gender specific interviewers were used to conduct surveys with sensitive target audiences (e.g. female interviewers conducted telephone calls with clients from women refuges, unless the caller was clearly happy and willing to continue with a male interviewer). Components of the information sheet (providing access to the translation service) were translated into the 12 most common languages among clients. These languages were: Arabic; Amharic; Chinese; Creole; German; Italian; Serbo-Croatian; Somali; Spanish; Tagalog; Turkish; and Vietnamese. Protocols were built into interviewer training and briefing notes to reflect community and cultural requirements of Aboriginal/Torres Strait Islander people. Appropriate facilities were made available for clients with disabilities, e.g. TTY for hearing impaired, verbal explanations of written material for sight-impaired.
Use of the information in a way that does not threaten services, workers or clients	Adequate information and training were provided to SAAP agency staff to enable them to feel confident to participate. There was also extensive consultation, through a 'champion' at each agency to ensure that the information is used in an appropriate context, and to fully inform agencies of the use of the information collected. All information was de-identified, with no details disseminated at client level. All reporting that was distributed at a national or state/territory level did not identify anything at the agency level. The survey was completed with trained and specialised interviewers. These interviewers had prior experience in conducting sensitive surveys in the social welfare field. Prior to the survey, the interviewers were fully briefed (in person for the pilot test and over the telephone for the survey) by members of the CBSR project team. This briefing involved: a question-by-question discussion of the questionnaire; a detailed set of interviewer notes; and mock interviews and potential scenarios. Pilot testing and cognitive interviewing techniques were used to develop questions to which clients feel confident in responding honestly to without any fear of retribution.
Need for information that is critical of a service not to jeopardise the client's future access to that or other services	Clients were assured of their confidentiality throughout the process, with no identifying information disclosed and no identifying information relayed to the agency.
If, when and how an agency might be informed if a client is critical of their behaviour/attitude	Results, both positive and negative, were reported on in the aggregate only. Where abuse or harm were reported by the client, interviewers were briefed to provide details to CBSR, who then liaised with FaCS as to the appropriate protocols. All details about the client were kept confidential subject to duty of care considerations.
Possible overstating of client satisfaction or the selection of biased sample may arise if client satisfaction information is used in decision-making about funding	Clients included people from all target groups at different locations, gender and a wide range of services (e.g. Long-term, crisis). During the questionnaire refinement phase, cognitive interviews were conducted with clients purposively chosen to ensure representation from 'worst case' scenario clients (e.g. clients who have a disability). Participants were recruited with the support of several agencies in ACT, NSW and QLD. The cognitive interviews investigated in-depth the term 'satisfied' as well as potential use of the goal attainment scale. Pilot testing further investigated the use of various scales (e.g. an unbalanced 'happiness' scale and a good/bad scale) to ensure appropriate and accurate measures of satisfaction. Again, agencies were recruited with the assistance of agencies in NSW, VIS and QLD. Inclusion of all types and locations of SAAP services was addressed through the sampling strategy. Initially, the sampling methodology was a two-tier random stratified probability sample of agencies plus census of clients, minimising the potential for sample bias. However, due to low response rates in general combined with issues raised about number of clients in/leaving medium to long-term agencies, the 'pool' of agencies was increased to become a census - i.e. all agencies were invited to participate in the survey. After data collection, the sample of agencies who participated was verified against population parameters to identify any possible biases in sample selection. Careful monitoring of distribution of information sheets and other measures of non-response rates were carried out. Non-response was looked into in order to understand who was participating and who was not.
Commitment to disseminating and using client satisfaction information wherever possible to improve service delivery	CBSR placed emphasis on the communication of results in a way that facilitated decision-making. Regular progress reports (at least twice weekly) were provided to FaCS during the course of the fieldwork. These included provision of details to input into improving service delivery. There was a final overview report which covered information about client satisfaction and its reported impact on service delivery. Additional reporting will occur to the Coalition of Australian Governments, which will cover the overall SAAP program and feed into the overall evaluation of this program.

[ top ]

9.4 Sampling

9.4.1 Target audiences

The target audiences for this research were clients of SAAP services throughout Australia. This included:

young people;
families;
women;
men; and
Aboriginal and Torres Strait Islander people.

Whilst 'accompanying children' are also a target audience of SAAP, research by AFHO recommended that a separate project is conducted with this group. Based on discussions with FaCS, it was agreed that accompanying children would not be included in this research.

In the original brief, clients with 'high and complex needs' were also identified as a separate target audience; however, discussion during the course of the scoping meeting suggested that identifying these individuals would be too difficult to achieve. It has therefore been removed as a target audience, although some questions included in the questionnaire aimed to identify those with these needs. These are described as 'multiple needs' in this report.

9.4.2 Sampling plan

A two-tier stratified random sample planwas initially adopted, as described in this section. The rationale for stratified sampling was the degree of analysis required at the sub- group level. The following section describes the sampling carried out for the first mail-out to agencies.

Tier 1 sampling & agency involvement

Sample design

The population of agencies is N=1205.

The sample frame for sample selection is a list of agencies provided to CBSR by the

Department of Family and Community Services as of June 2003.

Over one-third of agencies (n=500) were selected randomly within key strata. The target sample size of agencies was n=400, but some over-selection was done due to anticipated non-participation or attrition rates.

The sample strata were:

State/Territory; and
Type of agency (short-term, medium/long-term and 'other'³⁴ agencies).

'Other' agencies include the following types of agencies:

Day support;
Outreach support;
Telephone/information referral;
Agency support;
Multiple (i.e. agencies which frequently use more than one service delivery model for the provision of SAAP Services); and
Other (i.e. agencies providing services according to a Service Delivery Model not specified above).

The sample of agencies was disproportionate within these strata and involved over-sampling of long term agencies. Table 40 provides detail regarding initial sample selection at Tier 1.

[ top ]

Table 40: Tier 1 sampling plan
State	Agency Delivery Code 1 (Short-term agencies)	Agency Delivery Code 2 (Medium to long-term agencies)	Agency Delivery Code 3 ('other' agencies)
ACT	11	10	10
New South Wales	43	41	41
Northern Territory	11	10	10
Queensland	30	29	29
South Australia	11	6	8 'other' and 31 'multiple'
Tasmania	13	5	14
Victoria	30	44	15
Western Australia	17	17	17
TOTAL	166	162	175

The pilot results suggested that the number of clients responding from any particular agency would be low. Further, two weeks after the date by which agencies were to respond to the invitation, one-half (53%) of agencies still had not responded. CBSR thus recommended (and FaCS agreed) to extend the invitation to all agencies. Therefore, following consultation with FaCS and the Steering Committee, CBSR conducted a second mail-out to the remainder of agencies (i.e. those not selected in the initial sampling). This sampling approach resulted in an effective censusof agencies.

This second round of invitations to 703 additional agencies was conducted via a combination of email (for the 546 agencies for which email addresses were available) and hard copy letter (to the other 157 agencies).

Advance letters to agencies

A background letter was sent to all agencies. The letter introduced the client satisfaction research, noting the activities of questionnaire refinement and outlining steps involved in the national survey of clients. The letter was printed on CBSR letterhead to confirm the

involvement of a third party in the research (an aspect noted in the AFHO report as important). A total of 503 letters were distributed in the first-mail out, with a further 703 distributed in the second mail-out.

A copy of the letter is available in Advance letter.

Two weeks after the initial mail-out, a reminder letter was sent to all agencies that had not yet responded (264 in total at that time). For the 184 agencies where an email address was available, the letter was sent via email. For the remaining 80 agencies, with no email address available, the letter was sent via fax. In addition to the reminders, the ACT and NT representatives from the Steering Committee volunteered to assist with following up (in- person or by telephone) with each agency that had not yet responded in their Territory. On

23 October, CBSR provided FaCS with a list of all agencies that had not yet responded to the first mail out, by State/Territory.

[ top ]

Agency 'champions'

Each agency was asked to nominate a 'champion' who then became the primary contact for all survey-related matters. The agency was asked to provide name and contact details of the 'champion' to CBSR. This ensured that at least one person in each participating agency felt ownership over the process. Agencies who declined to participate were asked to provide information about their reason for non-participation.

These reasons are summarised in Table 41.

Table 41: Reasons for agency non-participation - as at 28 October 2003
Reasons for non-participation³⁵ As at 28 October 2003
Lack of staff	35
Not relevant for agency	16
Lack of facilities (quiet room, telephone)	20
Short notice	2
Low case load	2
Closed for renovations	1
Shelter being de-funded/No SAAP funding	11
Already involved in data collection	5
No clients exiting	8³⁶
Previous surveys problematic	2
Conflict of interest	1
No reason given	16

Teleconference with agencies

Once all champions were nominated, CBSR held teleconferences with groups of agencies. The purpose of the teleconference was to provide the agencies with information relating to the upcoming research among clients, discuss and coach 'champions' on the recruitment approach for clients and answer any questions the champions and/or their agencies raised.

The number and date of each teleconference is summarised in Table 42.

[ top ]

Table 42: Agency champion teleconferences
Teleconference	Date of teleconference	Participating agencies
#1	5 November 2003	Samaritan House, Lowana Young Women's Service, Belleden Youth Refuge, Lasa Youth Centre, Inanna, Canberra Community Housing, George Lloyd House, Barnado's Young Mum's and Family Support Program
#2	5 November 2003	Woy Woy Women's Refuge, Toukley, Bringa Women's Refuge, Hastings, Christo Youth Service, Tanderra, The Anchor Men's Hostel, Wimlah, Marcellin Cottage, Carrie's Place, Early Intervention, Barnado's at Craybrook, SAF, Fairlight, The Station Limited, Namoi House
#3	5 November 2003	Salvation Army City Service, Sidestreet Counselling Service, Eyre Peninsula Women's and Children's Support Centre, Youth and Parent Services, Youthcare, Jireh House, Bethlehem House, Burnie Youth Shelter, NYSA, Karinya Young Women's Refuge, Hobart City Mission, COSS, CTSS, CTSS North, Anglicare Access
#4	5 November 2003	Warringu Women's Shelter, Rockhampton Women's Shelter, Ipswich Independent Youth Service, Oscare, CASA, Women's Shelter in Gladstone, Gateway, Noosa Youth Service, Lion's Emergency Accommodation Service
#5	5 November 2003	Onslow Safe House, NC Kays, Karratha Women's Refuge, Healing Centre, Marnja Jarndu Women's Refuge, Byanda, Catherine House, YWCA Palmerstone, Anglicare Youth Housing Program in Palmerstone
#6	6 November 2003	Rainbow Lodge, Don Bosco, Gender Centre, Harold Lodge, Veratis House, Centacare Youth Accommodation and Support, Community Youth Homes, Lillians, Southern Women's, Youth Assistance Program, The Annexe, Vinshez, Vince's, Alambie, Vincentian Village Farm and Women's Service, Women's Housing
#7	6 November 2003	Loddon Mallee Housing Services, WAYSS, Hanover Family Services North and West Division, Hanover Southern Housing and Support Service, Hanover Southbank, Horsham Salvation Army Family Services, Woorarra, Outreach Victoria, Community Adolescent Placement, Reconciliation and Mediation Service, Independent Young Person's Support Program, Mallee Accommodation and Support Service, Gippscare Cross Target, Gippscare DVO's, Gippscare Crisis Initiative Cross Target
#8	6 November 2003	Cyril Lindsay House, Noarungga Olga Fudge, Northern Family Accommodation Service, Boden Brompton Housing, Family Support, Centacare, Anglican Community Care Program, Seventy Council Tracer Place, Louise Place
#9	6 November 2003	Silky Oaks, Goldcoast Youth Service, Ipswich Accommodation Project, Caboolture Family Haven, CYA, BABI, Youth Housing Project, Zigzag
#10	6 November 2003	Freemantle Youth Service, St Bartholomew's, Centacare, Anglicare Youth Housing Program, Bakhita Centre, Anglicare Youth Housing Program in Palmerston
#11	10 November 2003	Homelinx Youth Housing Program, Warrina Women's Refuge, Rosa Refuge, Samaritans Youth Services Taree (Open Door), Ammberly's Single Women's Refuge, Northern Region Young Mothers Accommodation Project, Western Housing for Youth, Homeless Mens Support Centre, San Miguel Relocation Officer Project, Langunyah House Youth Refuge, Women Up North Women's Housing Scheme, Tweed Shire Women's Service, Allawah, Girls Accommodation Unit, Edward Eagar Lodge
#12	10 November 2003	Meerindoo Youth Accommodation Service, Anchor, Maryborough & District Accommodation Service, Castlemaine and District Accommodation and Resource Group, Sacred Heart Mission HomeFront (Women' Crisis Centre), Kardinia - Kyewong Women's Shelter, Werribee Support & Housing Group, Willaneen Outreach Service, Bairnsdale Koorie Women's Shelter, Women's Information Support & Housing in the North, Wesley Community Contact Centre, Wimmera Uniting Care
#13	10 November 2003	Port Pirie SAAP Accommodation Service, Domestic Violence Northern Metropolitan Region, Murray Mallee Men's Service, Mission Australia Family Service, St Josephs Family Accommodation Service, Majella House St Vincent's Community Services, Jebb House, Windana Support Centre, Chisholm, Housing Action Group Maryborough Mens Shelter-Coolooli House, Emergency Long Term Accommodation Moranbah, Jupiter Mossman Cooperative Society, Integrated Youth Services - Maroochydore Shelter, Bundaberg and District Womens DV and Resource Services Inc, Whitsunday Women's Services Inc
#14	10 November 2003	Streetsyde Emergency Youth Accommodation, Starick House, Pat Thomas Memorial Community House, Alice Springs Women's Shelter, Launceston Women's Shelter

CBSR assumed responsibility for organising and running each teleconference.

The primary reason for non-participation in teleconferences was the lack of time available to agency staff. In order to encourage as many agencies as possible to participate, CBSR attempted to speak to each agency 'champion' personally, and were available at all times to answer any questions the agency 'champion' had. At the completion of the teleconferences CBSR collated all the questions that were raised by participants and circulated this document along with answers to all agencies that were participating in the survey.

[ top ]

Briefing pack

Prior to the teleconference, a briefing pack was sent to the agency champion.

A copy of the materials provided in this pack is shown at Agency briefing packs.

Response rates at Tier 1

Table 43 shows agency participation rates.

Table 43: Agency participation rates
	n=	%
Agencies invited to participate	1205	100%
Agencies agreeing to participate	299	25%
Agencies refusing to participate (see reasons above)	102	8%
Agencies not responding	805	67%

Of those agencies that agreed to participate, 69% went on to participate.

The majority of agencies that agreed to participate, but did not end up taking part, did not provide reasons for withdrawal and there was no formal process to ascertain the reasons included within the scope of the research. The following reasons for withdrawal are based on anecdotal evidence provided by the CBSR manager of the project. The reasons listed are generally in order of more commonly provided reasons to less common but it should be noted that the order is indicative only.

perception of onerous workload despite desire to participate;
forgot about the survey (finding forms under a 'pile of paper' after close date);
dissatisfaction that an agency level report would not be provided;
concerns that if any negative feedback about agency was received that it might be reported to FaCS;
illness/absence of agency staff during survey period; or
client suicide at agency (one case).

Tier 2 sampling & client involvement

Sample design

A census was conducted with all clients either currently in an agency, or formally exiting from the participating agencies, during a four-week period.³⁷

Although the AFHO report recommended focusing on clients who were exiting from the agency, the agency recruitment and pilot stages found that: (1) many medium and long- term agencies would have no one exiting in the (initial) two-week period,³⁸ resulting in refusal to participate and an inevitable sample bias towards short-term agencies; and (2) the final client sample size would be lower than estimated and needed for state/territory and other sub-group analysis. It was decided, in consultation with FaCS and the Steering Committee, that the methodology be changed to include allclients accessing services from the agency in the period, not just those exiting. Although this introduced the risk of the same client accessing the survey multiple times, it was felt that this risk was minimal (due to agency monitoring and involvement and the method of supplying the incentive payment - see below) and that the benefits outweighed the potential costs.

[ top ]

Sampling protocols at agency level

Since the aim of Tier Two sampling was to achieve a census, and it was critically important not to introduce selection bias at this stage, the protocols adopted at this stage were viewed as very important.

Agency champions were briefed on a standard sampling protocol as follows:

Clients were offered a $5 incentive in exchange for their participation.
All clients either in the agency or exiting the agency were to be given an 'information sheet' (prepared and distributed to agencies by CBSR, with FaCS review and approval). For a copy of this information sheet, please see Client information sheet.
All clients were to be verbally told the purpose of the research, given guarantees of confidentiality and anonymity, and be told about the $5 incentive (provided in the form of a $5 supermarket voucher, except where a supermarket was not available in the community. In this case, cash was provided).

Quality of Tier 2 sample and client response rates

Quality of the Tier 2 sample is negatively impacted to the degree that:

Some clients were systematically excluded from the sample (i.e. the sampling protocols were not followed); and
The response rate among clients invited to participate is low.

We do not have information regarding how frequently the first problem might have occurred. Table 44 shows client participation rates, based on the number of unused and used client information sheets. It must be noted that this method of calculating a response rate is imperfect and probably overstated, since it was beyond the control of CBSR to stop agencies from photocopying client information sheets, although champions were instructed NOT to do this. As well, not all agencies returned unused information sheets. Of 205 participating agencies, 198 returned unused client information sheets.

Table 44: Client participation rates³⁹
	n=	%
Number of client information sheets	5933	100%
Unused information sheets	3630	61%
Clients invited to participate (based on used information sheets)	2344	40%
Clients actually participating	1000	43%⁴⁰

All possible mechanisms to improve the response rate were adopted. These included ensuring that the instruction sheet was clear and easy to comprehend; confidentiality and anonymity was reinforced at every stage, every phone call made by clients was answered in a timely fashion, interviewers were aware of sensitivities in managing the interview, agencies were trained to encourage clients to participate without making them feel coerced, payment of incentive to clients was prompt and interpreters for non-English speaking clients were provided.

[ top ]

Table 45: Response rate by state
	Forms distributed	Completed Surveys	% of total completed
ACT	98	45	5
NSW	534	319	32
NT	59	29	3
QLD	425	165	17
SA	567	188	19
TAS	135	76	8
VIC	325	116	12
WA	197	59	6
Not identified		4
Total	2344	1000

Table 46: Response rate by agency type
	Forms distributed	Completed Surveys	% of total completed
Sample response rates	Forms distributed	Completed Surveys	% of total completed
Crisis/short term	678	365	37
Medium to long term	749	238	24
Other	466	259	26
Multiple	374	135	14
Not identified	77	3
Total	2344	1000

[ top ]

9.5 Questionnaire refinement

9.5.1 Cognitive interviews

Design

Considerable research was conducted by AFHO on the optimum content and structure of 'core' questions to measure client satisfaction with SAAP. AFHO also strongly recommended further testing of the questions before progressing to a survey phase.

CBSR read the AFHO report in detail and developed a set of questions based on the findings reported in that document.

After designing and gaining approval on a first draft of questions, CBSR conducted cognitive testing of the questionnaire as the first step in testing the questions.

Interviews were conducted by CBSR staff experienced in qualitative and cognitive interviewing techniques. A total of 18 interviews were conducted, each lasting an average of one hour. Qualitative interviews were held in person.

Table 47: Cognitive interviews
Location
Target group	Canberra/Queanbeyan	Sydney	Melbourne
Women	2
Men			4
Families	3
Youth	3	4
Aboriginal and Torres Strait Islander	2

To encourage the participation of clients in the cognitive testing phase, an incentive of $30 was provided.

The questionnaire tested in cognitive interviewing is shown at Draft questionnaire tested in cognitive interviews.

Cognitive interview findings

During the cognitive interviews, two different scales (goal attainment and happiness scale) were tested, to identify differences in respondent interpretation and use of these scales. In addition, a 5-point Likert scale was tested, using a two-stage approach. For example, respondents were asked if they were 'happy' and then asked if they were 'very happy' or

'somewhat happy'.

Findings from this, and other general findings, are discussed below. See 'Question level analysis' for details about specific issues and recommendations.

[ top ]

Happiness vs. goal attainment scale

Expectations were low to non-existent for this target group. As a consequence, response to the goal attainment scale tended to be answered without any linkage by the respondent between their actual expectations and how their experience measured against those. That is, when asked the first question about expectations they often replied 'about what I expected' then when the follow-up question was asked about what their expectations were they answered 'I didn't have any', or 'I don't know'. We recommend that a version of the 'happiness' scale be used for all satisfaction questions.

3 point vs. 5 point

Although the five point scale was asked in two stages during the interviews, interviewers experienced little to no difficulty with respondents providing an answer using the 5 point scale. In fact, some respondents spontaneously wanted to provide greater differentiation at the first stage. The 5 point scale was not tested in one phase during the cognitive interviews. Therefore, we recommend that Likert scale questions be asked in one stage during the pilot test.

The cognitive interviews revealed that the vast majority of respondents answered 'happy' to satisfaction questions. This reinforces AFHO report findings that there is a tendency for respondents to answer in the positive. When asked the five point scale there was apparent differentiation in responses, with some respondents 'very' happy and some 'somewhat' happy. In order to be able to differentiate between extent of satisfaction, we suggest that the five point scale include 3 positive codes (i.e. a little bit happy, quite happy and extremely happy), 1 neutral code and 1 negative code. We suggest further testing this in the pilot interviews.

Respondent fatigue

Respondent fatigue/concentration is a significant issue with this target group. We recommend keeping the length of the survey to a minimum. Average survey length in field was twelve minutes and seven seconds, a length which was both appropriate and achievable with this target group.

Different agency types

The questionnaire was interpreted quite differently by respondents depending on what type of agency they were involved with. For example, respondents involved with short-term agencies had a different interpretation of 'accommodation' than those in longer term accommodation (eg 'bed' as opposed to a 'house'). Similarly, the concept of 'help finding next accommodation' was very different depending on whether the respondent was in a crisis refuge as opposed to a house with a tenancy agreement. In addition, different things were provided by different agency types (such as food in refuges but no food in longer term accommodation). To recognise such differences, it is recommended that at the opening of the survey (when it is conducted using the call-centre) the respondent provide the agency identifier number (to be obtained from the information sheet provided to clients) and that this identifier is linked in the CATI system to a 'type' of agency. Once the type of agency is identified (and confirmed with the respondent) it is recommended that some questions be skipped for different target audiences (e.g. skip the food question for respondents from longer term agencies).

Likely client participation

A brief description of the call centre approach for the main study was given and respondents were asked how they would have felt if they had been asked to participate in something like what had just been described. Responses ranged from 'no way' through to 'yeah, probably'. The $5 incentive was attractive to some target groups (particularly those in shelters/refuges) however for medium to long term agency respondents the $5 incentive did not greatly alter the likelihood of them participating. Respondents living in the agency (eg refuges/shelters) appeared more likely to participate because a telephone is available. Respondents in medium to long-term housing were interested but were unsure whether they could call a 1800 number using their phone (those who had in-coming call access only). The pilot test will provide a much more detailed investigation of likely participation rates.

The following table details specific changes to the questionnaire that were recommended.

[ top ]


Question	Scale	Issues	Suggested revision
How would you describe what you wanted from <INSERT NAME OF AGENCY> during this stay?	This question was asked as an open ended question.	The majority of respondents did not understand what this question was asking and the open ended nature of the question did not work. Prior to any probing, the most common responses were 'housing', 'accommodation', 'a place to stay', 'help', 'someone to talk to', 'somewhere to get my act together', 'a bed' or 'support'. A few respondents said they wanted a 'place with no drugs or alcohol'. There was also some mention of the difference between what respondents 'wanted' versus 'needed' from the agency. A small number of respondents also answered this based on what they 'got' rather than what they 'wanted'. One respondent interpreted it as 'which of the services offered by this agency did you want to use...' A small number of respondents answered based on what they wanted when they first arrived, which they differentiated from what they wanted from the agency now i.e. their needs had changed since they first arrived. 'THIS STAY - after the first cognitive interview the wording was revised from 'this visit' to 'this stay' as 'visit' clearly did not work for medium and long-long term agency clients. The understanding of the term 'this stay' varied depending on whether the respondent was involved with a short-term or a medium to long- term agency. Short-term respondents generally interpreted 'this stay' as 'this time' at the agency; however, respondents who have been in and out of the same refuge/hostel were confused about on what basis they should answer. For medium to long-term respondents, this 'stay' was interpreted as 'this' tenancy agreement that they held with the agency.	Suggest rewording question to: (1) 'Which of the following BEST describes what you needed from <INSERT NAME OF AGENCY> when you first arrived' READ. ACCEPT ONE RESPONSE ONLY. Suggested pre-codes: Just a bed for a night or two A place to live for some time. Neither of these (SPECIFY) (2) 'And did you need any of the following from <INSERT NAME OF AGENCY> when you first arrived' READ. ACCEPT MULTIPLE RESPONSES. Someone to talk to or get advice from Professional help (for example, counselling)
Thinking about this stay to <INSERT NAME OF AGENCY>, overall, was the help you got...? For the purpose of the cognitive interview only, this question was followed with a question about what the respondent's expectations were. When the question was asked using the 'happiness' scale was used, the question was followed with a question asking whether the respondent was 'very happy' or 'somewhat happy'.	Scale 1: Better than you expected About what you expected Not as good as you expected Scale 2: Happy Neither unhappy nor happy (volunteered) Unhappy	All respondents interviewed said that the service was 'better than' or 'about' what they expected. When a follow-up question was then asked about what they expected, the most common answer was that they expected nothing, with some respondents mentioning they were 'so desperate' they did not have any expectations. In a few cases, this was due to previous (and constant) disappointments) so they had ceased expecting anything. In general, even among those who did have some expectations, the level of expectation was very low. These included wanting help with mental issues or health problems, or help to receive counselling/AA. Some respondents mentioned they had been to other agencies and thus used those past experiences as a 'rule' against which to measure their expectations from the current agency. In these few cases, all respondents had low expectations as their previous places were not as good as the current agency. When probed, these few respondents were unable to recall what their expectations had been of the very first agency they were involved with. When the question was repeated using the 'happy' scale, all respondents said they were 'happy' with the service. Thus, some respondents were 'happy' and the service 'about met' expectations. Others were 'happy' and the service was 'better than' their expectations. In general there was more hesitation in answering the question using the goal attainment scale, with respondents clearly more comfortable using the 'happiness' scale. When the 5 point Likert scale was used, respondents had no hesitation giving an answer of either 'very' or somewhat'. In addition, some respondents volunteered the 'very' at the first stage of asking the question (i.e. when asked if they were 'happy or unhappy' they answered 'very happy'). The interview findings, thus, indicate little to no difficulty with a 5 point scale when asked in two stages. The responses provided showed good differentiation, with some respondents saying 'somewhat' (and providing reasons for not answering 'very') and some respondents said 'very' (and again were able to provide reasons why they said 'very'). 'HELP' - the word help was interpreted as being about everything the agency could offer - services, referrals, food, the speed of the services, staff. As the goal of this question is overall satisfaction, the question is working well; however, this raises issues later in the questionnaire when respondents were asked about 'overall with assistance/services' (discussed further below).	Use the 'happiness' scale as described above.
Thinking about the length of time it took for <INSERT NAME OF AGENCY> to help you, was it .... For the purpose of the cognitive interview only, a follow-up question was asked about what the respondents expected with regard to timeliness.	Faster than you expected About what you expected Slower than you expected	Generally respondents interpreted this question to refer to the length of time the agency took to find them accommodation. Less frequently, the question was interpreted as the length of time the agency helps them with a range of services such as referrals. The majority of respondents said the time taken was 'faster' than expected. A few respondents said the time taken was 'slower' than expected. One respondent qualified this as being due to her panic of becoming a first-time mother and she said that everything, including clothes/supplies for the baby and accommodation for them both, was slower than she wanted. Others said they were seeking transitional housing and this was a more time- consuming process than they had expected. Expectations of timeliness were generally very low. Many respondents believed that the time an agency might take to help is 'beyond the agency's control, as it is 'just a matter of waiting for a space to become available'.	Reword question to use a 'happiness scale' and define what the length of time taken is for (e.g. to find next accommodation, to refer services).
Thinking about this stay at <INSERT NAME OF AGENCY>, do you recall receiving any information about... What <INSERT NAME OF AGENCY> can offer and what it can not offer you The rules of the house Your rights and responsibilities as a tenant How to make a complaint Other	Yes No Not sure/guessing	After the first interview, an extra code for 'not sure/guessing' was added, as it was clear that the respondent was guessing about what they had received. The remainder of interviews showed that many respondents guessed about what information they had been provided with. Concentrated probing was required during this question to determine whether respondents understood the difference between each type of information. The extent that probing was required to obtain clear recall about individual items suggests that the question requires revision in order for it to work satisfactorily during the call centre survey. There was some inconsistency in temporal interpretations of information provision - some respondents interpreted this question as being about what they received when they 'entered' the agency and some referred to information they had picked up over the course of time during their stay.	Define whether information relates to verbal or written (or both). Also need to reword the question to specify the timeframe (e.g. upon entering or during involvement). Use the word 'remember' instead of recall. Suggest rotating item order. Provide examples of types of community services.
community services you can use		The majority of respondents interpreted the question to be in reference to written information provided when they 'entered' the agency, however a minority also understood the question to be about verbal information during the course of their 'stay'. One respondent did not know what 'recall' meant. When the wording was changed to 'remember' they understood what it meant. 'WHAT CAN/CAN'T BE OFFERED' - the majority of respondents recalled both receiving a brochure and having a verbal discussion with agency staff when they entered the service. If they remembered a booklet or pamphlet, it then became difficult to determine (requiring extensive probing by the interviewer) whether they were guessing about the following types of information (e.g. 'I got such a big pile of information, I am guess it must have been in there but I didn't read it'). 'RULES OF THE HOUSE' - this was the most clearly recalled item of information by all respondents. One respondent at a refuge said 'they have a copy on the back of the loo door'. Other respondents recalled, without hesitation, receiving a verbal 'run-down' of the rules when they first arrived at the agency or when they first took residence of their accommodation. Some respondents recalled having to sign a document about rules when they entered the house. Some respondents had been evicted from previous agencies for breaking rules and this encouraged them to know the rules of subsequent agencies. 'RIGHTS AND RESPONSIBILITIES' - respondents often understood this phrase to mean their rights as a tenant, and the majority of respondents from medium to long-term agencies mentioned their tenancy agreement when asked about information they received on this. Only one respondent asked 'do I have rights?'. There was some confusion between 'responsibilities' and 'rights', however when the question was phrased as 'rights as a tenant' the question was interpreted correctly. 'HOW TO MAKE A COMPLAINT' - this phrase was, on the whole, interpreted as being about making a complaint about the service/staff. However, respondents at refuges also interpreted the question as being about making a complaint against other residents (e.g. violence, noise) or about the food. One respondent from a long- term agency referred to making complaints about items requiring maintenance in their house. 'COMMUNITY SERVICES' - in one case the respondent understood this phrase to be about community service served by a court. In another case, the word 'community' was not understood (however, this respondent was from a non- English speaking background and had a very low level of English proficiency). In general, however, the phrase was interpreted as being about services such as legal aid, job seeker services, salvation army, and financial assistance services.	Remove 'rights and responsibilities'. Reword complaints to clarify what complaints the question is asking about. (IE complaints about accommodation or service received). When interpreting response to this question, bear in mind that some respondents will have guessed.
In general, was the information you got...?	Very useful to you A little bit useful to you Not useful to you	The vast majority of respondents answered that the information was 'very' useful, and many of these answered the question quickly. When probed further about why the information was useful, respondents were generally unable to answer why. With extensive probing, reasons the information was found to be useful included that it helped them to feel informed and was useful as a reference; however, it was clear that many had never looked at written information after it was handed to them. Expanding the question to include verbal information may help alleviate this problem. For a small number of respondents the information was 'useful' primarily if it was 'usable' i.e. had some pragmatic utility value. For example, one respondent said that the most useful were lists of phone numbers for community services and lists of alternative crisis housing agencies.	Retain question as is. Bear comments in mind during interpretation. Add a prior question about written information as follows: Thinking about the written information you received, have you ever read this? Yes No Don't recall (volunteered)
Thinking about this stay to <INSERT NAME OF AGENCY>, overall, was the accommodation ... ? For the purpose of the cognitive interview only, a follow-up question was then asked about the respondent's expectations in terms of accommodation.	Better than you expected About what you expected Not as good as you expected	Responses to this question were balanced between 'better' and 'about as good'. No respondents said the accommodation was 'not as good' as expected. When asked about what they expected, the majority said they had no expectations. Of those who did have expectations, some (especially those in refuges) had been to other refuges and used these as a basis for expectations of their current accommodation. Others had very low expectations of what the accommodation might be like. One respondent said 'I expected a small old fibro place, not a flash house on snob hill like I got'. Others said they 'just wanted it to be hygienic'. 'ACCOMODATION' - this word had very different meanings depending on what type of service the respondent was involved with. Short-term agency respondents mainly interpreted it as being about 'a place to stay' or a 'roof over head', whereas longer term agency respondents interpreted the word as 'the house' and some extended the meaning of the word to include 'the surrounding area'. One respondent included food and another included rules of the house in their interpretation of the word and another mentioned 'stability' as being included.	Suggest using 'happiness' scale. Suggest providing examples to specify what accommodation refers to.
What type of accommodation did you get from this <INSERT NAME OF AGENCY>?	This was asked as an open ended question.	Respondents found this question difficult as an open-ended question and significant probing was required. As with other questions, and as intended, the answer varied greatly depending on the type of agency the respondent was from. Short-term agency respondents generally said 'a bedroom'; however, some said they got 'friendly' accommodation. Respondents from medium and long-term agencies tended to interpret the question as either the length of accommodation (e.g. medium-term, long-term), the size of the accommodation (i.e. 3 bedroom, 2 bedroom, a flat) or the type of building (e.g. a townhouse). However, some of these respondents also answered based on what they perceived to have been the key benefit of the accommodation to them, for example, 'stable', 'a place to go in crisis'. When probed further about types of accommodation (i.e. shelter vs. refuge vs. community housing) many respondents saw 'refuges' as short-term and more 'home-like' with bedrooms shared between two people. Shelters were interpreted more as 'crisis only', with dorm- style bedrooms. Respondents were not sure what the term 'community housing' referred to.	Provide pre-codes. Suggested pre- codes: Crisis/ Shelter accommodation (for example, a stay of under 3 months) Short- term/Refuge (a stay of 3-12 months) Medium term housing (6 months - 1 year) Long term housing (more than 1 year)
Thinking about <INSERT NAME OF AGENCY> during this stay, how much do you agree or disagree that...? It is clean It is {modern/new} It feels warm and homely The accommodation is safe The neighbourhood is safe My belongings are safe here The food is good The kitchen has enough equipment The furnishings are good There is enough space It is there for me when I need it I could find privacy here if I wanted it	Agree Neither agree nor disagree (volunteered) Disagree Not applicable	The largest difficulty with this question was the scale used to answer (i.e. agree/disagree). Even with significant reminding of the scale, most respondents reverted to answering yes or no. This question was expanded, to be asked as a 5 point scale. Respondents had no difficulty with the 5 point scale and were able to expand upon why they had said 'very' or 'somewhat'. 'CLEAN' - no problem. 'MODERN' - most respondents thought of the quality of the house and furnishings with regard to the word modern. Some respondents thought of 'gadgets' such as TVs or internet access. 'NEW' - most respondents had difficulty with this concept; they were not sure what qualified as 'new'. The word modern worked much better. 'WARM AND HOMELY' - this was interpreted as either being comfortable or as a place where clients could have visitors, have their family around etc. 'ACCOMODATION IS SAFE' - safety with regard to accommodation was primarily interpreted as being about locks and security screens in medium-long term agencies. Respondents from short-term agencies mentioned safety comes from the workers being around. 'NEIGHBOURHOOD IS SAFE' - no problem. 'BELONGINGS ARE SAFE' - short-term agency respondents all said this depended greatly upon who was staying in the house at various times. 'FOOD IS GOOD' - only applicable for refuge accommodation. No problem. 'KITCHEN HAS ENOUGH EQUIPMENT' - no problems for those in medium to long term agencies. In refuges, some respondents said they were not allowed (and sometimes had never seen) the kitchen and therefore answered this based on the range of food available. 'FURNISHINGS ARE GOOD' - furnishings was widely interpreted as relating to furniture. No problem. 'ENOUGH SPACE' - space was interpreted as space inside the building and , for those in medium to long-term housing, it also encompassed the outside area. Where the respondent had children, the term space was related to space for the children to play. No problem. 'THERE FOR ME WHEN NEED IT' - this was not understood by any respondents in medium to long term agencies. Those in short-term agency accommodation understood this to mean being able to get a bed for the night when they needed it. 'PRIVACY IF WANTED' - no problem.	Reword stem of question so that respondents answer yes/no. Use 'modern'. Warm and homely - change to 'it is comfortable'. Reword item about belongings safe to 'belongings are generally safe'. Only ask food in refuges/shelters. Only ask kitchen question in medium/long term agencies. Reword 'furnishings' to 'furniture'. Only ask 'there for me when needed' in refuges/shelters.
Thinking about this stay to <INSERT NAME OF AGENCY>, overall, was the type of assistance and services they were able to give you...? For the purpose of the cognitive interview only, a follow-up question was asked of what respondents expected with regard to assistance services.	More than you expected About what you expected Not as much as you expected	Responses were balanced between either 'more than' expected or 'about' what expected. No respondents said 'not as much as expected'. Prior to probing about what they understood 'assistance and services' to mean, many respondents included accommodation, information and staff in their frame of reference. For these respondents, the question almost doubled as an overall satisfaction question, and one even said they had been 'asked this question before'. A small number of respondents excluded housing from their answer. When probed, they said housing had already been asked about, so did not think of it when answering this question. Following probing, some respondents were able to distinguish 'assistance and services' as relating specifically to services provided such as referrals, accompanying respondents to court, assisting with job seeking, youth services and general support. Once again, expectations were very low among all respondents.	Suggest clarifying question to focus on assistance and services only (to exclude housing). Provide examples to clarify. Also, use happiness scale.
Thinking about this stay, did you need...? Help with child care Counselling for yourself or someone else Help with budgeting Help with rebuilding your life Emergency short-term help (for example, food money) An interpreter or someone from your culture Referrals to other community services Help to work out your next accommodation	Yes No Not applicable	Some respondents answered all items based on what they felt they actually got from the agency, not what they needed. 'CHILD CARE' - no problem. 'COUNSELLING FOR YOURSELF OR SOMEONE ELSE' - for the minority of respondents there was confusion about whether this referred to counselling for themselves only or someone else only, or for a combination. One person said 'yes' on the basis that someone else in the refuge has required counselling for a crisis that did not involve the respondent. 'HELP WITH BUDGETING' - budgeting was widely understood as being able to write down a budget and, for example, to pay bills. No problem. 'HELP WITH REBUILDING YOUR LIFE' - rebuilding your life was commonly understood as being about getting one's feet back on the ground. No problem. 'EMERGENCY SHORT-TERM HELP (FOR EXAMPLE, FOOD MONEY)' - there was some confusion about whether this also included emergency short term help with accommodation. 'AN INTERPRETER OR SOMEONE FROM YOUR CULTURE' - no problem. 'REFERRALS TO OTHER COMMUNITY SERVICES' - the same difficulty as mentioned above about what a community service is arose. Otherwise, no problem. 'HELP TO WORK OUT YOUR NEXT ACCOMMODATION' - this was generally not understood by respondents from medium or long-term accommodation. For those in short-term agencies, there was a mix of interpretations. Some perceived the main role of the agency as assistance with finding any accommodation, and therefore did not distinguish this as different from their 'next' accommodation. Others interpreted this as referring to permanent or transitional housing and felt they had no problem with this question.	Add an introduction to the question: 'I am now going to ask about whether you needed some services during this stay. I would like you to answer about whether you needed the service, not whether you actually got the service.' Counselling - suggest rewording to only include 'yourself' Emergency short term help - clarify to exclude accommodation Referrals to community services - include examples as per previous question about community services Help to work out your next accommodation - ask only of those in short term accommodation such as shelter/refuge
Thinking about <INSERT TYPE OF SERVICE>, was the help you got from <INSERT NAME OF AGENCY> during this stay...?	More than you expected About what you expected Not as much as you expected Happy Neither happy or unhappy (volunteered) Unhappy	Each respondent was asked this for 1 or 2 of the services they had mentioned they required help with. All respondents said the help they got with the services was 'more than' or 'about' what they expected. No respondents said it was 'not as much' as they expected. Expectations among respondents were virtually nil, with many respondents saying that anything they received was a 'gift' - the fact that there ARE interventions to try and prevent this being the case was amazing to them. The question did not appear to work well with respondents. Respondent fatigue with the question was apparent by the time they had worked through the battery of items in the previous question. The question worked slightly better when it asked whether they were happy or unhappy with the service/received, although even then all respondents said 'happy' and therefore the question did not appear to differentiate responses.	Reword question to ask whether the service was received or not.
Thinking about this stay to <INSERT NAME OF AGENCY>, overall, were staff...? For the purpose of the cognitive interview only, a follow-up question was asked about what respondents expected in terms of staff.	More helpful than you expected About as helpful as you expected Less helpful than you expected	All responses were either 'more' or 'about' as helpful as expected. No respondents said staff were 'less helpful' than expected. Unlike with 'assistance and services', respondents were clearly able to answer this question in relation to staff only, as opposed to overall help provided by the agency. Expectations for staff were slightly higher than for other elements mentioned above; however, many respondents still said they did not have 'any' expectations. Those who did, said they expected them to be friendly and welcoming.	Reword question to use happiness scale.
Thinking about <INSERT NAME OF AGENCY> staff during this stay, how much do you agree or disagree that they ...? Are trustful of you Involve you in decisions Are understanding of your situation Are fair Are warm and friendly Are efficient and professional Do things on time Take you seriously Are knowledgeable Explain things well Listen to you Pay you enough attention Are available when you need them	Agree Neither agree or disagree (volunteered) Disagree	The vast majority of respondents agreed with the bulk of items asked; however, some respondents disagreed with some items. Again, respondents repeatedly reverted to answering with yes or no, despite constant reminding of the agree/disagree scale. 'ARE TRUSTFUL OF YOU' - no problem. 'INVOLVE YOU IN DECISIONS' - this was widely understood as referring to decisions about the respondent themselves, such as outings (youth refuge). Some respondents interpreted the question as being told what to do. 'ARE UNDERSTANDING OF YOUR SITUATION' - no problem. 'ARE FAIR' - no problem. 'ARE WARM AND FRIENDLY' - no problem. 'ARE EFFICIENT AND PROFESSIONAL' - efficient was generally interpreted as getting things done on time and professional was interpreted as knowledgeable (thus, a double up of other items asked). 'DO THINGS ON TIME' - no problem. 'TAKE YOU SERIOUSLY' - no problem. 'ARE KNOWLEDGEABLE' - no problem. 'EXPLAIN THINGS WELL' - no problem. 'LISTEN TO YOU' - no problem. 'PAY YOU ENOUGH ATTENTION' - no problem. 'ARE AVAILABLE WHEN YOU NEED THEM' - no problem.	Reword question so that respondents answer using yes/no codes. Reword 'involve you in decisions' to clarify if refers to decisions about the individual, or decisions about the agency. Delete item about 'efficient and professional' as it doubles up on other items about 'doing it on time' and 'knowledgeable'.
Thinking of this stay, what things could <INSERT NAME AGENCY> have done better?		Almost all respondents gave this question considerable thought and then answered 'nothing'. This reinforces that expectations are low. One respondent said the agency could possibly have provided transport and one respondent said the agency could provide information faster about where to go after the service.	No revisions.
Thinking of this stay, what is the one most important thing that <INSERT NAME AGENCY> could have done better?		This question did not work given that the bulk of respondents could not thing of anything the agency could have done better.	Delete question for sake of brevity.
I am now going to ask you some questions about your personal situation. You don't have to answer these if you don't want to. Do you, or have you ever ....? experienced a mental health illness (such as depression) had a problem with drugs or alcohol experienced violence in the home been seriously in debt had a serious health condition or disability other than a mental illness	Yes No Refused	This question was asked as a 'sentence completion' (i.e. complete the sentence 'I came to this agency because ....) in the first interview. The question did not elicit the information intended. Further, during the interview it became clear that the respondent was quite willing to volunteer direct information about multiple needs. The decision was then made to ask the question directly for the remainder of interviews. Respondents did not display discomfort at being asked the question directly and many respondents gave (what can only be assumed as) honest answers, with many 'yes' answers and no refusals. 'EXPERIENCED A MENTAL HEALTH ILLNESS' - Even though depression was noted here as an example, three respondents, despite having revealed their depression, did not answer yes to this question as they interpreted 'mental health illness' as brain damage or genetic defects. When asked for a better wording, they suggested that 'psychological problems' would be more likely to include depression. 'SERIOUSLY IN DEBT' - Some confusion about interpretation of 'seriously'. Several respondents felt that 'seriously in debt' meant owing $50K or more, while the others felt that it was owing more than you could ever hope to pay off on your current income. One respondent said that $1K was seriously in debt for him. Another said that he was seriously in debt, not because he owed a lot of money, but because he owed money to someone who 'would have no hesitation in killing me'. 'VIOLENCE IN THE HOME' - many respondents interpreted this as physical violence and did not include emotional or verbal abuse in their answers. 'HEALTH CONDITION' - respondents generally had difficulty with this question and could not make a clear distinction between injuries and health conditions. For instance, one respondent had an injury that was serious but was now healed (steel plates in his leg). This caused him no ongoing issues or problems but he still answered 'yes'.	Reword 'mental health illness' to 'experience psychological problems'. Suggest rewording 'violence in the home' to 'violence or abuse in the home'. Revisit interpretations when doing final analysis.
What year were you born?		No problem.	No revisions.
Do you identify as Aboriginal or Torres Strait Islander?	Yes No Refused/not answered	There were a few aboriginal respondents and when asked this question, they all said 'yes, aboriginal'.	Add extra codes: Yes, Aboriginal Yes, Torres Straight Islander No (neither) Refused
What country were you born in?	Australia Other [SPECIFY]	No problem.	No revisions.
Are you...?	Employed, full or part time Unemployed (that is, actively looking for work in the last 4 weeks) Or something else (eg on a pension or not looking for work)	Some respondents are looking for work and on benefits and therefore did not know which to answer. One respondent was looking for work but was also working part-time, cash in hand. He was confused as to which answer to choose.	Clarify definitions. Are you...? Employed, full or part-time. This includes cash-in- hand work. Unemployed but actively looking for work. Unemployed and not actively looking for work.
How many days or months have you been at this agency for during this stay? [FORCE A CODE - ENTRE ZERO FOR DAYS IF RESPONDENT ANSWERS IN MONTHS] ....................Months ....................days		No problem.	No revisions.
During this stay, did you come...?	Alone With your partner only With your partner and children With your children only	No problem.	No revisions.

[ top ]

9.5.2 Pilot testing

Design

A pilot test of the revised questionnaire was conducted, to provide information on average completion time and issues with the flow of the questionnaire or potential logistics problems within agencies. Cognitive interviewing detected most problems that may be encountered with the questionnaire, particularly with respect to question wording and structure. However, it was not able to provide information on average completion time or issues with the flow of the questionnaire or potential logistics problems within agencies.

Pilot interviews were conducted with clients from six pilot agencies over the course of one day. This, in essence, served as a 'dress rehearsal' of the main fieldwork. Prior to the pilot, CBSR contacted the selected agencies and requested their assistance in recruiting clients and in providing a quiet room with a telephone for the client - in other words the agreed protocol for the main fieldwork was followed.

A senior member of our CBSR research team was present at the Melbourne call centre to

'listen in' to the pilot interviews. Interviewers were also debriefed at the conclusion of the pilot.

Following the pilot, the CBSR senior researcher administered a semi-structured pilot feedback questionnaire to the agency 'champions' involved in the pilot.

The average survey length was reviewed to estimate participant burden, as well as the likely response rate and telephone traffic for the main fieldwork.

An incentive of $5 was provided to all callers. The incentive was mailed to the client, care of the agency. This allowed assessment of how the incentive was working.

The questionnaire tested in piloting is shown at Draft questionnaire tested in pilot.

Pilot findings

At the conclusion of the pilot, a pilot test report was compiled by CBSR. This included an assessment about whether any final modifications to the survey instrument or the agency protocol were needed for overall outcomes to be achieved. Findings from the pilot testing are discussed below.

Overall the pilot resulted in a low number of calls received. This section discusses findings from the calls received, as well as coverage of discussions held between CBSR and the agencies involved in the pilot and also with the interviewers at the call centre.

Pilot test response rate

All six of the agencies who were contacted to participate in the pilot agreed to be involved with the pilot.

A total of three calls were received during the pilot test; however, only two calls were completed. Table 48 shows the number of calls received by agency type.

[ top ]

Table 48: Completed pilot interviews
Short-term	Medium to long term	Other
2	0	0

The two completed interviews were from the same agency.

The length of time taken to complete the survey was 7-8 minutes. Thus, the length of the questionnaire did not require revision as the ideal length was for a completion time of under 10 minutes.

Completed interviews

The two interviews conducted ran very smoothly. Overall:

Both respondents displayed no difficulty with the questions asked, including all demographics asked.
During the pilot, no calls were received from females involved with crisis agencies. Thus, the issue of conducting the interview with a female interviewer was not pilot tested; however, CBSR believe that this process will operate well when the survey is conducted.
At the commencement of CATI interviews, there is a standard statement made that the call may be monitored for quality control by supervisors. For the purpose of the pilot, this was retained as Sarah Tink was listening into calls. However, it was felt that when the survey commences that this quality control check should be removed. The reason for this is that (as mentioned above) the target group often have issues with trust and it is felt that by mentioning 'call monitoring' at the opening of the survey may impact upon the caller's feelings of trust towards the interviewers.

Detailed feedback from interviewers included the following:

Both scales worked well. Interviewers felt they ran smoothly and were easily understood by the callers. This was mainly due to the casual nature of the wording used in the scale. Of the two scales, interviewers felt that the happy/unhappy scale worked slightly better than the good/bad scale for two reasons: 1) it was shorter (4 codes read compared to 6); and 2) it seemed easier for callers to give an answer for. Interviewers gave feedback that the word 'excellent' was unlikely to be used by the target group.

[ top ]

Agency feedback on pilot test:

All agencies involved in the pilot test were contacted on the day after the pilot day, to gain information on how they found the process of the pilot survey, their perceptions of client reactions to the survey, to understand possible refusal rates (i.e. clients who may have been provided an information sheet but did not call the call-centre), and the reason for the very poor response rate.

Unfortunately only four of the agencies were contactable on the day after the pilot test. Table 49 shows the contact made with agencies (during and after the pilot test) and also (for those who were contacted after the pilot test) a brief summary of their feedback. (See section 8.11 for the questions used as a guide for discussion during these calls).

Table 49: Detailed agency feedback
Type of Service	Contact made with agency	Agency feedback on pilot test
Short- term	Could not contact agency on day of pilot test. Also unable to contact agency on day of follow-up.	n/a
Short- term	Contacted on day of follow-up.	Two calls received from agency during pilot test. Two clients left on the day of the pilot. The agency contacted two clients who had left the service last week and asked them to participate. Four information sheets were handed out. Of these, two clients opted to use the agency phone and the other two said they would call from a phone elsewhere - one of each of these calls was received (i.e. only one client called from elsewhere and only one of the two clients who said they would use the agency phone actually made the call) No clients refused (all indicated they would call). The only logistical problem encountered was that the agency did not have a telephone in a private area. Generally client reactions to the information sheet were seen as positive. In general, the agency has approximately 50% of clients leave in a two-week period. Agency felt that the telephone works 'okay' with the clients (single men); however, the agency staff member believes that it is easier for them to provide false information over the phone than in a face-to- face situation, and therefore that a face-to-face approach would be much more suitable.
Short- term	Contacted on day of pilot test and on day on follow-up.	No calls received from agency during pilot test. No information sheets handed out. Agency indicated that they do not generally receive much warning of clients leaving (in fact, on many occasions clients reportedly leave in the middle of the night, to avoid the workers and also to avoid payment owing to the agency). No clients left agency on day of pilot and agency was unable to predict if anyone might be leaving during the few days following the pilot. One-quarter of clients leave within 1-3 days, one-fifth between 4-7 days and one-quarter leave after 1-2 weeks. Agency has previously done its own survey. The survey was conducted with 60 clients and the main finding was that clients do not like talking to anyone on the phone and that they would prefer face-to-face (the gender of the face-to-face person they spoke with did not matter). Agency felt that if clients are asked to make a telephone call on exit that the client will not make the call (as they are more interested in going to the next place).
Short- term	Contacted on day of pilot test and also on day of follow-up	No calls received from agency during pilot test. Agency had no clients leave and were unable to identify if any might be likely to leave in coming days. No information sheets handed out. No other feedback from agency on pilot test.
Medium to long- term	Called on day of pilot test and also on day of follow-up.	No calls received from agency during pilot test. No information sheets handed out. Agency reported that 100% of clients are migrant women who do not speak English. Due to language barriers, the agency said that significant lead-up time is required to contact the clients as an interpreter is needed. Despite best efforts from the agency, contact was only made with one client about the pilot test. When the agency called the client, the client starting speaking straight away about a personal crisis and the agency worker then had to assist with the crisis and made an assessment that it was not a good time to speak with the client about the pilot test. Agency feedback that if they were participating in the survey they would make a time to get all clients in one meeting together, where interpreters can be booked and on-site, and would then workshop through with clients' participation, questions and concerns. Agency felt that, even if this approach was taken, there would still be a large portion of clients whop would not pick up the phone to make the call. For instance, the agency said, that clients will not (even with an interpreter service available) call to connect electricity or services to their homes.
'Other'	Could not contact agency on day of pilot test. Also unable to contact agency on day of follow-up.	n/a

[ top ]

Conclusions from the pilot test were that:

The revised questionnaire appears to be working well with a very small sample of respondents.
There was anecdotal evidence from agencies that clients will not be interested in participating on exit. However, there is no evidence from the pilot itself to support this. Given that four information sheets were distributed and two interviews were completed, a 50% response rate is encouraging.
A variety of logistical constraints will impede agency involvement. These will be especially apparent for agencies with high populations of NESB or youth. Agency involvement will likely to the key barrier to a good response rate.
The 'on exit' requirement of the proposed methodology outlined in the tender brief is inappropriate for medium to long-term agencies, for reasons identified prior to the pilot during agency recruitment for the main study. An agreement has already been made to move to an approach that expands eligibility beyond 'at exit'. It is interesting to note, however, that the 'on exit' requirement appears less than ideal for short-term agencies as well.
It is highly likely that the requirement for a private room with telephone is unrealistic.

Following the pilot test findings, and follow-up calls with agencies, CBSR recommended that a meeting be held with representatives from FaCS and the Steering Committee to discuss implications for the methodology and to discuss options for action.

During the meeting, it was agreed that CBSR would conduct a second mail-out to all agencies who had not yet been invited to participate (see above). It was also agreed that all clients (both those in the agency and those exiting the agency during the survey period) would be eligible to participate (initially the methodology had been to only include those exiting an agency during the survey period). Third, it was agreed to extend the period for the survey from 2 weeks to 3-4 weeks.

9.5.3 Final survey form

A copy of the final questionnaire used is available at 'Final questionnaire' (below).

The final average completion time was seven to eight minutes, including time to establish rapport between the interviewer and respondent. The length of the survey posed no or little problem for respondents. Although no formal feedback on length of interview was gathered during the main fieldwork period, there were few 'hang-ups' or other disruptions to the smooth administration of the questionnaire.

9.6 Fieldwork

[ top ]

9.6.1 Client information sheet

Agencies provided all clients using their service in the survey period with an information sheet, inviting them to participate in the survey. The sheet included a 1800 number for the call centre, located at Colmar Brunton Research's Melbourne Computer Assisted Telephone Interviewing (CATI) facility. It also included an agency identifier that could be quoted by the client when speaking to the interviewer at the call centre. This would allow linking of the client survey data with information known about the agency, and potentially allow reporting at the agency level at a future date.⁴¹

The sheet included translation of two sentences into 12 languages directing NESB clients to a translation service (provided by DIMIA), should they require it. The 12 languages were based on the client data and the most frequently spoken languages in the client population. These were:

Arabic;
Amharic;
Chinese;
Creole (pidgin);
German;
Italian;
Serbo-Croatian;
Somali;
Spanish;
Tagalog;
Turkish; and
Vietnamese.

To address the potential issue of low literacy levels, the design of the sheet was simple. The agency protocol also involved agency staff verbally passing on certain information to clients when they handed the sheet to the client.

For a copy of the Information sheet, including translation, please see Client information sheet.

[ top ]

The agency identifier was a critical piece of information that clients needed to pass on to the interviewer (as this assisted with identifying the region and other characteristics of the agency to be used later for analysis purposes, as well as identifying whether a specific interviewer would be required to conduct the survey). Since this could have potentially appeared complicated to clients (as well as raise concerns about individual confidentiality) shapes were used to direct clients to the agency identifier. The following illustration provides an illustration of how this was done:

An illustration guided by the interviewer

Clients were then guided by the interviewer to read out the number in the house.

9.6.2 Quiet room

Where possible, agencies provided a 'quiet room' from which clients called the call centre. Where agencies were unable to provide a 'quiet room', clients called the free-call number from either a public telephone or a land-line telephone elsewhere.

9.6.3 1800 number and call centre

CBSR provided a 1800 phone number for clients to call, at no cost to the client or the agency, to participate in the survey.

The call centre was open to receive calls from 9am-9pm Monday to Friday and 9am-5pm Saturday and Sunday between 11 November and 5 December 2003.

[ top ]

9.6.4 Trained interviewers

The survey was completed using Computer Assisted Telephone Interviewing (CATI) techniques, with trained and specialised interviewers. A team of dedicated interviewers were assigned to the call centre. These interviewers have prior experience in conducting sensitive survey in the social welfare field.

Prior to conducting the survey, the team of interviewers were briefed fully via telephone by senior members of the CBSR project team. Representatives from FaCS were welcomed to attend this briefing.

This briefing involved:

a detailed set of interviewer notes;
question-by-question discussion of the questionnaire, including the rationale behind each question; and
mock interviews and potential scenarios. Interviewers were also debriefed after the pilot.

9.6.5 Client incentives

A $5 incentive was paid to every client respondent that completed a survey. The incentive was provided in the form of a $5 Coles supermarket voucher, unless the participant indicated that they were not located close to a Coles supermarket, and in those cases $5 cash was posted to the participant. At the end of the survey, participants were asked where they would like to have the voucher posted - either care of the agency or to another address. In the cases where participants chose to have the voucher posted to the agency, the agency then passed the voucher to the participant when it arrived.

9.6.6 Translation

All respondents were given the option of contacting a translation service if they required assistance. The Department of Immigration and Multiculturalism and Indigenous Affairs' translation service, TIS, was used. Directions on how to contact this service were provided with advance letters, in the 12 most frequently spoken languages. However, the translation service itself operates in over one hundred languages.

[ top ]

9.6.7 Final sample

The survey was cut off upon completion of 1,000 interviews on 5 December 2003. This was the target sample size.

In total, three respondents were unable to provide the interviewer with the agency number (see above). Two of these were later resolved by matching the stated name of the agency to the agency number in the sample database. However, one respondent provided an agency name that had no match. This respondent was removed from the dataset and the effective unweighted and weighted (see below) sample size is therefore n=999.

9.6.8 Logistical issues

There was no formal process, within the scope of this research, to ascertain any logistical issues at the agency level that may have impacted on client participation rates or the general smooth functioning of the survey. However, we are able to provide some anecdotal evidence of what went well/badly in terms of agency involvement in the process, and, in particular, in the fieldwork logistics.

Agencies were keen to participate and this eagerness was positive for agency involvement. However, this meant that management of, and liaison with, agencies, was labour intensive and some agencies had very high expectations regarding liaison.
The change of methodology mid-way through the process (particularly the amendment from 'exiting' clients only to all current clients) caused confusion for many agencies, and this prompted several calls during the survey fieldwork period for clarification of who could participate.
Anecdotal evidence from agencies suggests that the administration of incentives proved successful for clients ... only two incentives (of 1000) were reported as being received, with no client of that name at the agency to pass the incentive to.
The translation service provided by DIMIA caused confusion in one instance, with the service unable to establish what number needed to be called to complete the survey. In this case, DIMIA called the Canberra office of CBSR, liaised with the project manager and was able to connect the client to the call centre to complete the survey.

9.7 Analysis

9.7.1 Database

Data were stored in a secure location on CBSR premises. Access to the database was restricted to authorised personnel.

The data analysis package used was SPSS version 11.0.

[ top ]

9.7.2 Unique identifier

Each respondent was given a unique identification number.

9.7.3 Data weighting

To correct for disproportionate sampling, weights were applied to the data as shown in Table

50. A comparison of the sample to population parameters on state/territory and service/agency type was first done. This showed some disproportions in the sample. These were corrected through weighting factors.

The population information is based on the average number of clients each SAAP agency had each night during November/early December 2002.⁴² It is important to note that the population information may contain some duplication, as some clients could concurrently access SAAP services at more than one agency. However, it is the opinion of the Department of Family and Community Services that this situation is likely to be relatively rare, and does not pose any serious problem in terms of the distribution of client numbers as used for weighting purposes.

Table 50: Weighting
Weighting scheme based on support periods (November to early December 2002)
Agency type within state	POPULATION N=	N%	SAMPLE n=	n%	Proportionate n=	Weight	After weight applied
1.00 ACT/Crisis	133.125	0.009267	27	0.027027	9.25724354	0.342861	9.257244
2.00 ACT/Medium	97.625	0.006795	11	0.011011	6.788645262	0.61715	6.788645
3.00 ACT/Other	78.5	0.005464	7	0.007007	5.458731402	0.779819	5.458731
5.00 NSW/Crisis	1074.875	0.074819	113	0.113113	74.74463587	0.661457	74.74464
6.00 NSW/Medium	596.625	0.04153	69	0.069069	41.4880971	0.601277	41.4881
7.00 NSW/Other	1629	0.113391	137	0.137137	113.2773688	0.826842	113.2774
9.00 NT/Crisis	98.9375	0.006887	18	0.018018	6.879913861	0.382217	6.879914
10.00 NT/Medium	112.5	0.007831	8	0.008008	7.823022709	0.977878	7.823023
11.00 NT/Other	97.875	0.006813	3	0.003003	6.806029757	2.268677	6.80603
13.00 QLD/Crisis	1134.0625	0.078939	78	0.078078	78.86041504	1.011031	78.86042
14.00 QLD/Medium	475.3125	0.033085	52	0.052052	33.05227095	0.635621	33.05227
15.00 QLD/Other	540.8125	0.037645	36	0.036036	37.60700861	1.044639	37.60701
17.00 SA/Crisis	150.125	0.01045	24	0.024024	10.43938919	0.434975	10.43939
18.00 SA/Medium	113.25	0.007883	21	0.021021	7.875176194	0.375008	7.875176
19.00 SA/Other	269.9375	0.01879	9	0.009009	18.77090838	2.085656	18.77091
20.00 SA/Multiple	894.3125	0.062251	135	0.135135	62.18868442	0.460657	62.18868
21.00 TAS/Crisis	147.375	0.010258	39	0.039039	10.24815975	0.262773	10.24816
22.00 TAS/Medium	14.1875	0.000988	4	0.004004	0.986570086	0.246643	0.98657
23.00 TAS/Other	759.8125	0.052889	33	0.033033	52.83582616	1.601086	52.83583
25.00 VIC/Crisis	1059.75	0.073767	33	0.033033	73.69287392	2.233117	73.69287
26.00 VIC/Medium	4052.875	0.282111	49	0.049049	281.8287392	5.751607	281.8287
27.00 VIC/Other	95.25	0.00663	34	0.034034	6.623492561	0.194809	6.623493
29.00 WA/Crisis	469	0.032646	35	0.035035	32.61331245	0.931809	32.61331
30.00 WA/Medium	271.125	0.018872	24	0.024024	18.85348473	0.785562	18.85348
Total	14366.25	1	999	1	999		999

[ top ]

9.8 Survey schedule


Activity	Description	Timing	Responsibility
Scoping	Scoping meeting	16 Sep	CBSR/FaCS/Steering Committee
Ethics approval	Draft ethics approval document	18 Sep	CBSR
	Gain ethics Committee approval	24 Sep	CBSR
Survey Design Report	Draft Survey design report provided to FaCS (now Appendix E of report of findings)	24 Sep	CBSR
Sampling	Obtain revised lists of agencies for recruitment from FaCS	30 Sep	FaCS
	Sampling of agencies	1 Oct	CBSR
	Approval of sampling	2 Oct	FaCS
	Select agencies	3 Oct	CBSR
	Prepare lists of agencies for CATI	Same day that final questionnaire is scripted into CATI	CBSR
Advance letter	Draft advance letter to agencies provided to FaCS	26 Sep	CBSR
	Approval received from FaCS on draft advance letter	30 Sep	FaCS
	Revise and finalise advance letter	1 Oct	CBSR
	Receive final approval on revised advance letter	1 Oct	FaCS
	FaCS to advise on who to address letter to	1 Oct	FaCS
	Print advance letter ready for distribution	2 Oct	CBSR
	Distribute advance letter to agencies	3 Oct	CBSR
	Advance letter received by agencies	8 Oct	n/a
Information Sheet/Pack	Suggest 12 languages and provide FaCS with draft sentence for translation	29 Sep	CBSR
	Receive approval from FaCS on languages and sentence for translation	30 Sep	FaCS
	Draft information sheet provided to FaCS	9 Oct	CBSR
	Draft of other information for 'pack' provided to FaCS	9 Oct	CBSR
	Approval received from FaCS on draft information sheet and 'pack'	15 Oct	FaCS
	Revise and finalise information sheet and 'pack'	16 Oct	CBSR
	Revise information pack materials regarding extended timing and inclusion of clients not exiting	29 Oct	CBSR
	Print/collate information sheet and packs	24-31 Oct	CBSR
	Distribute information sheet/packs to recruited agencies	Commence 27 Oct	CBSR
Questionnaire development	Questionnaire design meeting	24 Sep 03	CBSR/FaCS/Steering Committee
	Provide draft questionnaire ready for cognitive testing to FaCS	30 Sep	CBSR
	Approval received from FaCS on draft questionnaire	2 Oct	FaCS
	Revise and finalise questionnaire ready for cognitive interviews	3 Oct	CBSR
Agency involvement	Send reminder email to agencies who have not replied	1 Oct	CBSR
	Extra mail out to increased 'pool' of agencies	By 27 Oct	CBSR
	Email to agencies sampled in first mail out (re: increased fieldwork time and revised to 'beyond' exit)	22 Oct	CBSR
	Receive agency 'champion' nominations	Up to 30 Oct	CBSR
	Advise champions of teleconference date/time	Commence 27 Oct	CBSR
	Conduct teleconferences with agencies	29 Oct - 7 Nov	CBSR
Cognitive interviews	Cognitive locations decided	26 Sep	FaCS
	Liaise with Steering Committee representatives from cognitive locations	29 Sep	CBSR
	Cognitive interview participants recruited	30 Sep - 3 Oct	State representatives
	Cognitive interviews conducted	8 - 13 Oct	CBSR
	Cognitive interview report written and provided to FaCS	15 Oct (COB)	CBSR
Pilot testing	Revise questionnaire ready for piloting	15 Oct	CBSR
	Provide revised questionnaire ready for piloting to FaCS	15 Oct	CBSR
	Final approval of questionnaire for pilot test	16 Oct	FaCS
	Pilot test questionnaire coded into CATI	17 Oct	CBR Melb
	Pilot test conducted (Melbourne)	20 Oct	CBSR
	Pilot test report written and provided to FaCS	22 Oct	CBSR
Final questionnaire	Revise questionnaire ready for 'final'	27 Oct	CBSR
	Approval received on final questionnaire from FaCS	28 Oct	FaCS
	Survey design report updated and provided to FaCS	28 Oct	CBSR
Survey fieldwork	Questionnaire coded into CATI	7 Nov	CBSR
	Interviewer briefing	10 Nov	CBSR
	Survey conducted	11 Nov - 7 Dec
Data analysis and reporting	Data analysis	9 - 18 Dec
	Verbal progress report to Steering Committee	19 Dec
	Presentation of findings	20 Jan 04
	Draft report of findings provided to FaCS	23 Jan 04
	Final report provided to FaCS	30 Jan 04

[ top ]

9.9 Advance letter

Dear Coordinator,

National SAAP client satisfaction research - your agency's participation

Colmar Brunton Social Research (CBSR) has been commissioned by the Commonwealth Department of Family and Community Services (FaCS) to collect data at a State/Territory and national level on client satisfaction with the Supported Accommodation Assistance Program (SAAP). You should have recently received a letter from Doug Limbrick of FaCS introducing this research.

This research builds on the extensive work done by the Australian Federation of Homelessness Organisations (AFHO) consortium. The overall aim of this research is to measure client satisfaction against key outcomes at State and national level, to feed into the National Evaluation of SAAP IV.

We are sure you agree that it is important for clients of SAAP agencies to be able to have their say. This is one of the key ways for client views to be included in the evaluation. Your participation will help give clients of your agency a chance to have their say.

The survey will be conducted using a call centre. Clients will call toll-free to participate in a short (10 to 15 minute) phone survey. This method has been chosen to minimise the impact on agency workloads. The survey will take place during a two week period in November.

We have randomly selected about one-third of all SAAP agencies to participate in this research. All agencies had an equal chance of being selected. We are now approaching your agency to request your involvement.

To be involved in this important work you will first need to nominate one person in your agency to become a 'champion' - this person will become the key contact for all activities related to this research. The responsibilities of the champion are detailed below. Once you have notified us of your agency champion, we will be contacting him or her to organise a briefing by telephone.

Once CBSR has received contact details for a champion in each of the nominated agencies a teleconference with groups of agencies will be arranged. The purpose of the teleconference is to provide the agencies with information relating to the upcoming census for their clients, discuss and coach 'champions' on the recruitment approach for clients and to answer any questions the champions and their agencies might have.

The date and time of the teleconference will be confirmed with your agency champion once we have received the champion's details.

[ top ]

What will be required of your agency?

Nominate a champion
Distribute information sheets to exiting clients over a 2 week period
Help explain the purpose of the survey to clients
Provide a 'quiet room', or similar, with a telephone over the course of the 2 week period
Return all undistributed information sheets to us (so that we can calculate participation rates)
Distribute $5 to all clients who participate in the survey

The role of the 'champion'

The champion will be required to coordinate activities related to the survey in your agency. This includes making sure that all clients who leave your service within the survey period have a chance to participate in the survey, those clients are provided with an information sheet (provided by CBSR); and clients are helped to understand the purpose of the survey. The champion will also be responsible for ensuring each client receives their $5 payment upon completion of the survey.

The roles of the champion will be provided in more detail in an information pack that will be provided by CBSR once you have nominated a champion.

Why participate?

Clients will have the chance to 'have their say'.
Clients will be provided with a $5 incentive, to thank them for their participation.
This will provide very useful information for the SAAP evaluation, which will be important in determining the future of SAAP.

Your agency's anonymity will be protected

All data relating to your agency will not be identifiable in any reporting from this survey. The results will be analysed and published at state/territory and national levels only.

[ top ]

Next steps?

Please complete the attached form whether or not you are able to participate.

There will be no negative consequences for your agency should you chose not to participate. However, your agency's participation is vital to the success of this initiative and we strongly encourage you to do so.

If you require further information about the research, you can contact Sarah Tink at Colmar

Brunton on 1800 004 446 or Barbara Beatty at FaCS on (02) 6212 9507. Yours sincerely

Joan Young

Managing Director

Colmar Brunton Social Research

3 October 2003

Please complete this form and return either by fax or email by [DATE]

Fax: (02) 6249 8588

Email: SAAP@canberra.cbr.com.au

.......Yes, this agency will participate in this research

Agency: ...........................................................................................

Street address: ...........................................................................................

Suburb/Town: ...........................................................................................

State: .........................Postcode: .........................

Name of champion: ...........................................................................................

Position/title of champion: ...........................................................................................

Phone number of champion: (0) .........................

Fax number:(0 ) .........................

.......No, this agency choses not to participate in this research. Reasons are provided below.

Please provide reason/s for non-participation: ................................................

...........................................................................................

[ top ]

9.10 Agency briefing packs

Cover letter

[DATE]

[PERSON'S NAME]

[ADDRESS] Dear [NAME]

National SAAP client satisfaction research

Thank you for agreeing to participate in the National SAAP client satisfaction research, by electing to be a champion for your agency. Colmar Brunton Social Research (CBSR) has been commissioned by the Commonwealth Department of Family and Community Services (FaCS) to collect data at a State/Territory and national level on client satisfaction with the Supported Accommodation Assistance Program (SAAP).

Please find included in this information pack:

Cover letter (this document)
Background information
Copy of the information letter that was distributed to agencies
Instructions for agency champions (including a survey timeline)
Information sheets for distribution to clients
Distribution and Participation Form
Contact details for both FaCS and CBSR
Courier Pack (to return all materials to CBSR at the completion of the survey phase)
Checklist

If you require further information about the research, you can contact Sarah Tink at Colmar Brunton on 1800 004 446 or Barbara Beatty at FaCS on (02) 6212 9507. Yours sincerely

Joan Young

Managing Director

Colmar Brunton Social Research

[ top ]

Background Information

Research on homelessness assistance

The Supported Accommodation Assistance Program (SAAP) assists people who are homeless, or at risk of homelessness, through a range of support and supported accommodation services. SAAP is a joint Commonwealth-State program that is coordinated nationally by the Department of Family and Community Services (FaCS) and administered by States and Territories. In 2001, the Australian Federation of Homelessness Organisations (AFHO) was commissioned, along with partners, to develop options for client satisfaction measures in SAAP.

Colmar Brunton Social Research (CBSR) has now been commissioned by FaCS to progress to the next step of collecting SAAP client satisfaction data at the national level. This challenging research has never been conducted before, and CBSR is delighted to be involved with this revolutionary study.

The overall objective of the research is measure client satisfaction against key outcomes at State/Territory and national levels. This will be achieved through the development and implementation of a survey, adopting an innovative call-centre method. Information and findings from this research will be used as a key component in full-scale SAAP IV evaluation, to be conducted in early 2004.

The purpose of this research is to provide information at a national and State/Territory level that can potentially lead to improvement in the way that SAAP is provided.

Copy of the information letter that was distributed to agencies

22 October 2003

Dear Coordinator,

National SAAP client satisfaction research - your agency's participation

The date and time of the teleconference will be confirmed with your agency champion once we have received the champion's details.

[ top ]

What will be required of your agency?

Nominate a champion
Distribute information sheets to all clients over a 4 week period
Help explain the purpose of the survey to clients
Provide a 'quiet room', or similar, with a telephone over the course of the 4 week period
Return all undistributed information sheets to us (so that we can calculate participation rates)
Distribute $5 to all clients who participate in the survey (the money is provided by us)

The role of the 'champion'

The champion will be required to coordinate activities related to the survey in your agency. This includes making sure that all clients in your service within the survey period have a chance to participate in the survey, those clients are provided with an information sheet (provided by CBSR); and clients are helped to understand the purpose of the survey. The champion will also be responsible for ensuring each client receives their $5 payment upon completion of the survey.

The roles of the champion will be provided in more detail in an information pack that will be provided by CBSR once you have nominated a champion.

Why participate?

Clients will have the chance to 'have their say'.
Clients will be provided with a $5 incentive, to thank them for their participation.
This will provide very useful information for the SAAP evaluation, which will be important in determining the future of SAAP.

Your agency's anonymity will be protected

All data relating to your agency will not be identifiable in any reporting from this survey. The results will be analysed and published at state/territory and national levels only.

Next steps?

Please complete the attached form whether or not you are able to participate.

If you require further information about the research, you can contact Sarah Tink at Colmar Brunton on 1800 004 446 or Barbara Beatty at FaCS on (02) 6212 9507.

Yours sincerely

Joan Young

Managing Director

[ top ]

Colmar Brunton Social Research

Instructions for agency champions

Instructions for Agency Champions

Thank you for agreeing to participate in the National SAAP client satisfaction research, by electing to be a champion for your agency. As a champion, you will assist with the survey both leading up to and during the survey period. Your participation is vital and will ensure that the project runs smoothly.

Colmar Brunton Social Research (CBSR) is going to be holding teleconferences with agency champions between the 31st October and 7th November. Due to the large number of agencies involved in the research, a number of teleconferences will be conducted during this period with approximately 20 champions involved with each teleconference. Each champion is required to participate in one teleconference only. The purpose of the teleconference is to:

provide background information on the research;
provide champions with training on what is involved in their role; and
provide champions with the opportunity to ask any questions about the research.

The survey is a census of all clients during the four week survey period between 11 November and 7 December. This means that all clients should be invited and encouraged to take part in the survey. The steps for the agency champions to follow in order to recruit participants are outlined below.

Step 1: Tell the client about the survey.

Step 2: Hand the information sheet to the client and verbally explain its contents.

Step 3: If applicable, show the client to a 'quiet room' that they can make the phone call from.

Step 4: If the client requests it, dial the number for the call centre then hand the phone to the client and leave the room.

Step 5: Record on the 'Distribution and participation sheet'' that an information sheet has been given to the client.

Step 6: If the client choses to have the $5 voucher posted care of the agency, upon receipt of the voucher give the voucher to the client.

At the end of the survey period, you will be asked to:

Complete the Distribution and Participation form (enclosed)
Return all unused information sheets
Return all to Colmar Brunton in the courier pack provided

The information pack you have received includes a sheet that has the times the call centre will be open, as well as contact phone number.

CBSR has highly trained interviewers to staff the call centre during the survey period. Where possible female clients will be interviewed by female interviewers.

CBSR is available to offer support and assistance to agencies throughout the research. Agency champions should feel free to call whenever the need arises.

If you require further information about the research, you can contact Sarah

Tink at Colmar Brunton on 1800 004 446 or Barbara Beatty at FaCS on

(02) 6212 9507. Yours sincerely

Joan Young

Managing Director

[ top ]

Colmar Brunton Social Research

Survey timeline


Dates for Survey	Times Call Centre is open	Contact number for client call centre	Contact number forany queries
Tuesday 11^th November - Friday 14^th November	9am - 9pm	1800 555 145	1800 004 446 (ask for Sarah)
Saturday 15^th November - Sunday 16^th November	9am - 5pm
Monday 17^th November - Friday 21^st November	9am - 9pm
Saturday 22^nd November - Sunday 23rd November	9am - 5pm
Monday 24^th November - Friday 28^th November	9am - 9pm
Saturday 29^th November - Sunday 30^th November	9am - 5pm
Monday 1^st December - Friday 5^th November	9am - 9pm
Saturday 6^th December - Sunday 7^th December	9am - 5pm

Note: All times are AEDT (Australian Eastern Daylight Time).

[ top ]

6. 'CLIENT information' sheet

We are inviting you to do a short interview about how happy you are with accommodation and service you have received from this place. If you do the survey it will take about 10 minutes and afterwards you will be sent a $5 voucher to thank you for your time.

To do the survey, find a telephone and call 1800 555 145. The telephone call is free.

If you want to talk to someone about the survey before taking part call 1800

004 446, ask for Sarah.

Please keep this sheet with you. During the phone call the interviewer will ask you to read the number in the house below.

number in the house image

Client satisfaction survey

Different languages

[ top ]

Distribution and Participation Form

Name of Agency:

This form is to be returned to CBSR upon completion of survey

(8 December)


Date	Number of 'Client Information' Sheets distributed to participants	Signature of Agency Champion
Tuesday 11-11-03
Wednesday 12-11-03
Thursday 13-11-03
Friday 14-11-03
Saturday 15-11-03
Sunday 16-11-03
Monday 17-11-03
Tuesday 18-11-03
Wednesday 19-11-03
Thursday 20-11-03
Friday 21-11-03
Saturday 22-11-03
Sunday 23-11-03
Monday 24-11-03
Tuesday 25-11-03
Wednesday 26-11-03
Thursday 27-11-03
Friday 28-11-03
Saturday 29-11-03
Sunday 30-11-03
Monday 1-12-03
Tuesday 2-12-03
Wednesday 3-12-03
Thursday 4-12-03
Friday 5-12-03
Saturday 6-12-03
Sunday 7-12-03
Total used:
Number returned to CBSR:	Date returned: ......./............../2003

[ top ]

Contact details for both FaCS and CBSR

NEED MORE INFORMATION?

8. Contact details

Colmar Brunton Social Research (CBSR)

Name: Sarah Tink

Phone Number: (02) 6249 8566 / 1800 004 446

Fax Number: (02) 6249 8588

Email: sarah.tink@canberra.cbr.com.au

Family and Community Services (FaCS)

Name: Barbara Beatty

Phone Number: (02) 6212 9507

Fax Number: (02) 6212 9520

Email: Barbara.Beatty@facs.gov.au

9. Checklist for champions

Set up a quiet room with a telephone, if possible
Tell the client about the survey
Hand one ' Client information' sheet (the green page) to each client and verbally explain
(If applicable) Show client to the ' quiet room'
(If asked by client) Dial call centre number (see ' Client information' sheet) , hand phone to client and leave room
Record on ' Distribution and participation sheet'
(If applicable) Give $ 5 voucher to client when received in post
Return unused ' Client information' sheets & completed ' distribution and participation sheet' to CBSR in supplied reply paid envelope

[ top ]

To do the survey, find a telephone and call 1800 555 145. The telephone call is free.

If you want to talk to someone about the survey before taking part call 1800

004 446, ask for Sarah.

Please keep this sheet with you. During the phone call the interviewer will ask you to read the number in the house below.

number in the house image

Client satisfaction survey

Different languages

[ top ]

9.11 Draft questionnaire tested in cognitive interviews

Introduction

Thank you for meeting with me today.

Later this year we are going to do a large survey to find out what people like you think about accommodation and assistance services like this one. I have asked you to talk to me today to help us develop a questionnaire that we will use for that survey. It will take about an hour and a half and at the end of the interview I will give you $30 to thank you for talking to me today.

So what we will do is go through the questions one by one. As we go through the questionnaire, I will be especially interested in how you came up with each of your answers. This helps me understand how you are interpreting the question. We want to find out what different words like 'satisfied' and 'happy' mean to different people, so some of the questions I will ask twice, to find out what you think about different ways the questions can be asked.

I'd like to ask you to think aloud as you answer each question and tell me everything that comes to your mind in answering the question. The more you can tell us, the more useful it will be to us as we try to develop better questions.

If I ask you any questions that you would prefer not to answer, just tell me and we'll skip that question.

Everything you say is between you and me and no-one from this service will know or find out what you have said to me. So that I can remember everything you have said, I would like to tape this interview today if that is OK with you? The tape won't be given to anyone from this service or any government people, it is only for me to use.

[IF OK TO TAPE, GIVE PARTICIPANT CONSENT FORM TO SIGN]

[IF PARTICIPANT CAN NOT READ CONSENT FORM, READ IT TO THEM AND GET THEIR VERBAL CONSENT AT BEGINNING OF TAPING]

Do you have any questions before we begin?

Before we start, let's start with a practice question. This question will give you an idea of what we're looking for from you during this discussion.

Q: What did you eat last Saturday?

Tell me what you were thinking as you tried to remember that
How did you remember it was Saturday?
You answered very quickly, why was it so easy to come up with that answer
You looked puzzled/curious, tell me what you were thinking
Was it hard to think of one main thing?

Thank you for that, you did really well. Let's move on.

[ top ]

Questions

NOTE TO INTERVIEWERS: THE PURPOSE OF THIS QUESTION IS TO UNDERSTAND WHAT NEEDS THE CLIENT HAD WHEN THEY APPROACHED THE AGENCY, ESPECIALLY, TO TRY TO DISTINGUISH THOSE WHO 'JUST NEEDED A BED FOR THE NIGHT', FROM THOSE WHO NEEDED LONGER TERM ASSISTANCE, FROM THOSE WHO WERE PRESENTING WITH MANY DIFFERENT TYPES OF PROBLEMS BESIDES ACCOMMODATION, ETC.

INSTRUCTION TO INTERVIEWERS: IF THE RESPONDENT APPEARS TO BE ON A VERY DIFFERENT TANGENT, EXPERIMENT WITH ALTERNATIVE WAYS OF ASKING THIS QUESTION. NOTE THE MOST SUCCESSFUL.

INSTRUCTION TO INTERVIEWERS: RECORD RESPONSES VERBATIM FOR THE PURPOSE OF POTENTIAL QUESTIONNAIRE PRECODES.

Q1 How would you describe what you wanted from <INSERT NAME OF AGENCY> during this stay?

[SPECIFY]

Mandatory cognitive probes:

Tell me what you were thinking as you tried to remember that
Is it hard to think of a main thing you wanted
Can you think of any other things you wanted
What were you thinking of when you thought of 'this stay'
Can you tell me in your own words what this question is asking

Optional probes (use if needed):

How did you get that answer
You answered very quickly, why was it so easy to come up with that answer
You looked puzzled/curious, tell me what you were thinking

[Encourage participant - that's great, that is what I want you to do]

NOTE TO INTERVIEWERS: Q2(A) IS EXPLORING THE CONCEPT OF OVERALL SATISFACTION.

Q2A Thinking about this stay to <INSERT NAME OF AGENCY>, overall, was the help you got...?

[READ OUT]

[SINGLE RESPONSE]

Better than you expected
About what you expected
Not as good as you expected

[ top ]

Mandatory cognitive probes:

Tell me what you were thinking as you tried to answer that
Can you tell me in your own words what this questions is asking

Optional probes (use if needed):

You answered very quickly, why was it so easy to come up with that answer
You looked puzzled/curious, tell me what you were thinking

Note to interviewers: At the end of all the interviews we want to know whether people's expectations were high or low. The purpose of asking this is to 'link' to the question above (i.e. once gained an idea of how it measured against their expectations, find out what those expectations are)

Q2Ai What were your expectations in terms of help from <INSERT NAME OF AGENCY>?

NOTE TO INTERVIEWERS: Q2(B) IS EXPLORING THE SAME CONCEPT OF OVERALL SATISFACTION, BUT USING A DIFFERENT SCALE.

INSTRUCTION TO INTERVIEWERS: EXPLORE THE DIFFERENCES/SIMILARIES IN ANSWERS TO Q2(A) AND Q2(B). DO THEY APPEAR TO BE ASKING THE SAME THING? IF NOT, WHY NOT? WHICH ONE WORKS BETTER? At the end of asking this question the two ways we want to know whether the scales are discriminating, and how they cross-match (e.g. is someone having their expectations met yet they are unhappy, or are they answering happy on everything).

Q2B I am now going to ask you the same question, but am going to give different words for you to use to answer. Thinking about this stay to <INSERT NAME OF AGENCY>, overall, are you happy or unhappy with the help you got?

[READ OUT]

[SINGLE RESPONSE]

Happy
Neither unhappy nor happy (volunteered)
Unhappy

Mandatory cognitive probes:

Tell me what you were thinking as you tried to answer that
Can you tell me in your own words what this questions is asking

NOTE TO INTERVIEWERS: Q2(C) IS ATTEMPTING TO REPLICATE A LIKERT SCALE THROUGH A STAGED APPROACH TO QUESTIONING.

INSTRUCTION TO INTERVIEWERS: DETERMINE IF THIS 2-STEP APPROACH IS EASY FOR, CLEAR TO RESPONDENTS.

Q2C IF HAPPY, And were you very happy or just somewhat happy with the help you got?

[READ OUT]

[SINGLE RESPONSE]

Very happy
Somewhat happy

[ top ]

[Encourage participant - that's great, you are doing really well]

Q2D IF UNHAPPY, And were you very unhappy or just somewhat unhappy with the help you got?

[READ OUT]

[SINGLE RESPONSE]

Somewhat unhappy
Very unhappy

Mandatory cognitive probes:

Tell me what you were thinking as you tried to answer that
Can you tell me in your own words what this questions is asking

NOTE TO INTERVIEWERS: Q2E is attempting to find out how satisfied they are with the length of time the service has taken to help them.

Q2E Thinking about the length of time it took for <INSERT NAME OF AGENCY> to help you, was it ....

Faster than you expected
About what you expected
Slower than you expected

Q2Ei What were your expectations in terms of length of time taken to help you?

NOTE TO INTERVIEWERS: Q3 is attempting to find out if they received any information on these 'items', which have been identified in prior research as being important to clients.

INSTRUCTION TO INTERVIEWERS: THE CRITICAL COGNITIVE QUESTION IS WHETHER THE 'ITEM' IS MEANINGFUL TO CLIENTS. COULD IT BE PHRASED A DIFFERENT WAY?

Q3 Thinking about this stay to <INSERT NAME OF AGENCY>, do you recall receiving any information about...

What <INSERT NAME OF AGENCY> can offer and what it can not offer you
The rules of the house or your rights as a tenant
Your rights and responsibilities
How to make a complaint
Other community services you can use

For each CODE

Yes
No
Not sure/guessing

[ top ]

Mandatory cognitive probes (ask after each item in battery):

Tell me what you were thinking as you tried to remember that
What do you think is meant by 'INCLUDE APPROPRIATE PHRASE (EG 'RIGHTS & RESPONSIBILITIES')

If code 1 in Q3

Q4 In general, was the information you got...? [READ OUT]

[SINGLE RESPONSE]

Very useful to you
A little bit useful to you
Not useful to you

Mandatory cognitive probes:

Can you tell me in your own words what this question is asking
In what way was the information useful/not useful
Tell me what kind of information, in general, is useful to you

[Encourage participant - that's great, you are doing really well]

NOTE TO INTERVIEWERS: Q5 is asking about accommodation, which can include aspects of the building, infrastructure, food, etc.

INSTRUCTION TO INTERVIEWERS: DETERMINE HOW 'ACCOMMODATION' IS BEING INTERPRETED. IS THE RESPONDENT'S INTERPRETATION BROAD ENOUGH? IF NOT, WHAT OTHER WORDS COULD WE USE. EXPERIMENT WITH

DIFFERENT WORDS.

Q5 Thinking about this stay to <INSERT NAME OF AGENCY>, overall, was the accommodation ... ?

[READ OUT]

[SINGLE RESPONSE]

Better than you expected
About what you expected
Not as good as you expected

[ top ]

Mandatory cognitive probes:

Tell me what you were thinking as you answered that question
What were you thinking of when I said, 'accommodation'
Are there other things that count as 'accommodation'? What are they?

Q5A What were your expectations in terms of accommodation?

NOTE TO INTERVIEWERS: Q6 IS TRYING TO FIND OUT WHAT KIND OF SERVICE THE CLIENT RECEIVED (IE SHELTER/REFUGE OR LONGER TERM HOUSING (EG COMMUNITY HOUSING)).

INSTRUCTION TO INTERVIEWERS: IF THE RESPONDENT APPEARS TO BE ON A VERY DIFFERENT TANGENT, EXPERIMENT WITH ALTERNATIVE WAYS OF ASKING THIS QUESTION. NOTE THE MOST SUCCESSFUL.

INSTRUCTION TO INTERVIEWERS: RECORD RESPONSES VERBATIM FOR THE PURPOSE OF QUESTIONNAIRE PRECODES.

Q6 What type of accommodation did you get from this <INSERT NAME OF AGENCY>?

Mandatory cognitive probes:

Tell me what you were thinking as you tried to answer that
Can you tell me in your own words what this questions is asking

[Encourage participant - that's great, you are doing really well]

NOTE TO INTERVIEWERS: Q7(A) IS EXPLORING THE DIFFERENT DIMENSIONS OF ACCOMMODATION. THESE HAVE BEEN IDENTIFIED IN PRIOR RESEARCH AS BEING OF IMPORTANCE TO CLIENTS.

INSTRUCTION TO INTERVIEWERS: THE CRITICAL COGNITIVE QUESTION IS HOW THE 'ITEM' IS INTERPRETED BY CLIENTS. COULD IT BE PHRASED A

DIFFERENT WAY? ALSO, DOES THE AGREE/DISAGREE SCALE WORK?

Q7A Thinking about <INSERT NAME OF AGENCY> during this stay, how much do you agree or disagree that...? [ROTATE]

[READ OUT]

It is clean
It is {modern/new} NOTE TO INTERVIEWERS: TEST ALTERNATE WORDING}
It feels warm and homely
The accommodation is safe
The neighbourhood is safe
My belongings are safe here
The food is good
The kitchen has enough equipment
The furnishings are good
There is enough space
It is there for me when I need it
I could find privacy here if I wanted it

For each CODE

Agree
Neither agree nor disagree (volunteered)
Disagree
Not applicable

[ top ]

NOTE TO INTERVIEWERS: Q7(B) IS ATTEMPTING TO REPLICATE A LIKERT SCALE THROUGH A STAGED APPROACH TO QUESTIONING.

INSTRUCTION TO INTERVIEWERS: ATTEMPT WITH ONE OR TWO OF THE BATTERY ITEMS. DETERMINE IF THIS 2-STEP APPROACH IS EASY FOR, CLEAR TO RESPONDENTS. DO CLIENTS STRUGGLE TO ANSWER? DOES IT TAKE

SIGNIFICANT TIME TO GET THROUGH?

Q7B1 IF AGREE, And do you agree strongly or just agree somewhat that...? [READ OUT]

[SINGLE RESPONSE]

Agree strongly
Agree somewhat

Q7B2 IF DISAGREE, And do you disagree strongly or just disagree somewhat that...? [READ OUT]

[SINGLE RESPONSE]

Disagree somewhat
Disagree strongly

[Encourage participant - that's great, you are doing really well]

Q8 Thinking about this stay to <INSERT NAME OF AGENCY>, overall, was the type of assistance and services they were able to give you...?

[READ OUT]

[SINGLE RESPONSE]

More than you expected
About what you expected
Not as much as you expected

Q8A What were your expectations in terms of assistance and services?

NOTE TO INTERVIEWERS: Q9 IS EXPLORING THE DIFFERENT SERVICES THAT MAY HAVE BEEN PROVIDED TO THE CLIENT. THESE HAVE BEEN IDENTIFIED IN

PRIOR RESEARCH AS BEING OF IMPORTANCE TO CLIENTS.

It is important that the word 'Need' is used in this question as the question as asking what they needed, as opposed to what they 'got'

INSTRUCTION TO INTERVIEWERS: THE CRITICAL COGNITIVE QUESTION IS HOW THE 'ITEM' IS INTERPRETED BY CLIENTS. COULD IT BE PHRASED

[ top ]

Q9 Thinking about this stay, did you need...? [ROTATE]

[READ OUT]

Help with child care
Counselling for yourself or someone else
Help with budgeting
Help with rebuilding your life
Emergency short-term help (for example, food money)
An interpreter or someone from your culture
Referrals to other community services
Help to work out your next accommodation

For each CODE:

Yes
No
Not applicable

[Encourage participant - that's great, you are doing really well]

NOTE TO INTERVIEWERS: FOR ALL YES in Q9, ASK Q10. ASK Q10 SEPARATELY

FOR EACH ONE MENTIONED IN Q9.

Q10 Thinking about <INSERT TYPE OF SERVICE>, was the help you got from <INSERT NAME OF AGENCY> during this stay...?

[READ OUT]

More than you expected
About what you expected
Not as much as you expected

[ top ]

Q10A What were expectations in terms of <INSERT TYPE OF SERVICE>?

NOTE TO INTERVIEWERS: Q10(B) IS EXPLORING THE SAME CONCEPT OF OVERALL SATISFACTION, BUT USING A DIFFERENT SCALE.

INSTRUCTION TO INTERVIEWERS: THE CRITICAL COGNITIVE QUESTION IS WHETHER THIS SCALE IS WORKING? IF IT IS NOT WORKING, EXPERIMENT WITH THE 'HAPPINESS' SCALE FOR ONE OR TWO ITEMS. EXPLORE THE DIFFERENCES/SIMILARIES IN ANSWERS TO Q10(A) AND Q10(B). DO THEY APPEAR TO BE ASKING THE SAME THING? IF NOT, WHY NOT? WHICH ONE

WORKS BETTER?

Q10B Thinking about <INSERT TYPE OF SERVICE>, overall, are you happy or unhappy

with the help you got? [READ OUT]

[SINGLE RESPONSE]

Happy
Neither happy or unhappy (volunteered)
Unhappy

Mandatory cognitive probes:

Tell me what you were thinking as you tried to answer that
Can you tell me in your own words what this questions is asking

[Encourage participant - that's great, you are doing really well]

INSTRUCTION TO INTERVIEWERS: THE CRITICAL COGNITIVE QUESTION IS

WHETHER THIS SCALE IS WORKING?

Q11 Thinking about this stay to <INSERT NAME OF AGENCY>, overall, were staff...? [READ OUT]

[SINGLE RESPONSE]

More helpful than you expected
About as helpful as you expected
Less helpful than you expected

[ top ]

Q11A What were your expectations in terms of staff?

INSTRUCTION TO INTERVIEWERS: THE CRITICAL COGNITIVE QUESTION IS

WHETHER THIS SCALE IS WORKING?

Q12 Thinking about <INSERT NAME OF AGENCY> staff during this stay, how much do you agree or disagree that they ...? [ROTATE]

[READ OUT]

Are trustful of you
Involve you in decisions
Are understanding of your situation
Are fair
Are warm and friendly
Are efficient and professional
Do things on time
Take you seriously
Are knowledgeable
Explain things well
Listen to you
Pay you enough attention
Are available when you need them

For each CODE:

Agree
Neither agree or disagree (volunteered)
Disagree

Mandatory cognitive probes:

Tell me what you were thinking as you tried to answer that
Can you tell me in your own words what this questions is asking

INSTRUCTION TO INTERVIEWERS: RECORD RESPONSES VERBATIM FOR THE PURPOSE OF POTENTIAL QUESTIONNAIRE PRECODES.

[ top ]

Q13 Thinking of this stay, what things could <INSERT NAME AGENCY> have done better? [SPECIFY, ACCEPT MULTIPLE]

Mandatory cognitive probes:

Tell me what you were thinking as you tried to remember that
Is it hard to think of the things you wanted
Can you think of any other things you wanted
What were you thinking of when you thought of 'this stay'
Can you tell me in your own words what this question is asking

Q14 Thinking of this stay, what is the one most important thing that <INSERT NAME AGENCY> could have done better?

[ACCEPT ONE ANSWER ONLY]

[SPECIFY]

Mandatory cognitive probes:

Tell me what you were thinking as you tried to remember that
Is it hard to think of one important thing
Can you tell me in your own words what this question is asking

Demographics

Thanks, you are doing really well.

What I would now like to do is go through some questions that ask a bit about you. I want you to think about whether you know what to answer and whether you feel comfortable giving answers. If you don't want to answer anything, you don't have to, just tell me and we'll go to the next question.

NOTE TO INTERVIEWERS: THE PURPOSE OF THIS QUESTION IS TO UNDERSTAND WHY THE CLIENT APPROACHED THE AGENCY, THAT IS WHAT ARE THE PROBLEMS THE CLIENT IS PRESENTING WITH - DRUG PROBLEMS, MENTAL HEALTH PROBLEMS, FINANCIAL PROBLEMS, DOMESTIC VIOLENCE, ETC.

INSTRUCTION TO INTERVIEWERS: IF THE RESPONDENT APPEARS TO BE ON A VERY DIFFERENT TANGENT, EXPERIMENT WITH ALTERNATIVE WAYS OF ASKING THIS QUESTION. NOTE THE MOST SUCCESSFUL.

INSTRUCTION TO INTERVIEWERS: RECORD RESPONSES VERBATIM FOR THE PURPOSE OF POTENTIAL QUESTIONNAIRE PRECODES.

Q15 I am now going to ask you some questions about your personal situation. You don't have to answer these if you don't want to.

Do you, or have you ever ....?

6. experienced a mental health illness (such as depression)

7. had a problem with drugs or alcohol

8. experienced violence in the home

9. been seriously in debt

10.had a serious health condition or disability other than a mental illness

Q16 Record Gender [DO NOT READ INTERVIEWER TO CODE]

Male
Female

Q17 What year were you born? [ENTER YEAR]

19...............

Q18 Do you identify as Aboriginal or Torres Strait Islander?"

Yes
No
Refused/not answered

Q19 What country were you born in?

Australia
Other [SPECIFY]

[ top ]

Q20 Are you...? [READ OUT]

[SINGLE RESPONSE]

Employed, full or part time
Unemployed (that is, actively looking for work in the last 4 weeks)
Or something else (eg on a pension or not looking for work)

Q21 How many days have you been at this agency for this stay?

[FORCE A CODE - ENTRE ZERO FOR DAYS IF RESPONDENT ANSWERS IN MONTHS]

...............Months...............days

Q22 During this stay, did you come...? [READ OUT]

[SINGLE RESPONSE]

Alone
With your partner only
With your partner and children

Thank you for talking to me today, I really appreciate your time and everything you have done. It will help us a lot.

[GIVE PARTICIPANT $30]

9.12 Draft questionnaire tested in pilot

Introduction

Hello, my name is [NAME OF INTERVIEWER]. Are you calling about the survey?

Can I confirm you were given an information sheet <NAME OF AGENCY>? [IF YES, CONTINUE, IF NO ASK RESPONDENT WHICH AGENCY THEY GOT THE SHEET FROM AND ENTER NAME OF AGENCY]

[IF HAS INFORMATION SHEET ASK] Can you tell me the number in the house on your information sheet please?

[INTERVIEWER TO ENTER NUMBER, IF RESPONDENT CAN'T PROVIDE NUMBER ASK NAME OF AGENCY]

Record Gender

[INTERVIEWER TO CODE]

Male
Female

[IF AGENCY = CRISIS AND GENER = FEMALE ENSURE INTERVIEWER IS FEMALE]

I now want to ask you some questions and it should only take 15 minutes. The questions are about how happy or unhappy you are with things like the accommodation, the staff, information and the services that you have got from <NAME OF AGENCY>. Everything you say is completely confidential, so nothing that you tell me will go back to <NAME OF AGENCY> or to anyone else.

Q1A Which of the following BEST describes what you needed from <INSERT NAME OF AGENCY> when you first arrived?

[READ OUT]

[SINGLE RESPONSE]

Just a bed for a night or two
A place to live for some time.
Neither of these (SPECIFY)

Q1B And did you need any of the following from <INSERT NAME OF AGENCY> when you first arrived?

[READ OUT]

[MULTIPLE RESPONSE]

Someone to talk to or get advice from
Referral for professional help (for example, counselling)

[ top ]

Q2A Thinking about this stay with <INSERT NAME OF AGENCY>, overall, how happy or unhappy are you with the help you got? Are you...

[READ OUT]

[SINGLE RESPONSE]

Extremely happy
Happy
A little bit happy
Neither unhappy nor happy (volunteered)
Unhappy

Q2B Thinking about the length of time it took for <INSERT NAME OF AGENCY> to help you with things like services or with getting accommodation, overall how happy or unhappy are you with how quickly this was done? Are you...

[READ OUT]

[SINGLE RESPONSE]

Extremely happy
Happy
A little bit happy
Neither unhappy nor happy (volunteered)
Unhappy

Q3A Thinking about when you first got to <INSERT NAME OF AGENCY>, or any time during this stay, do you remember receiving any information, either verbally or in writing, about ...

[ROTATE] [READ OUT]

What <INSERT NAME OF AGENCY> can offer and what it can not offer you
The rules of the house
How to make a complaint about accommodation or the service received
Other community or government services that you could use, like Salvation Army, Legal Aid or job seeker services

For each CODE

Yes
No
Not sure/guessing (VOLUNTEERED/ASSESSMENT OF INTERVIEWER)

[ top ]

IF YES TO ANY AT Q3A:

Q3B Thinking about the written information you received, have you ever read this?

Yes
No
Don't recall [VOLUNTEERED - DO NOT READ OUT]

Q3C In general, was the information you got...?

[READ OUT]

[SINGLE RESPONSE]

Extremely useful to you
Quite useful to you
A little bit useful to you
Not useful to you

Q4 What type of accommodation did you get from <INSERT NAME OF AGENCY> during this stay?

[READ OUT]

[SINGLE RESPONSE]

A room in a hotel/motel, boarding house or bed & breakfast
A bed in a dormitory
A shared room
A house/flat
Or something else [SPECIFY]

Q5 Thinking about this stay with <INSERT NAME OF AGENCY>, overall, how good or bad did you think the accommodation you were provided with was (that is, the room or house that you are staying in)? Is it...

[READ OUT]

[SINGLE RESPONSE]

Excellent
Really good
Good
Just OK
Quite bad
Bad

Q6 I am going to read you a list of things that might describe the accommodation provided to you by <INSERT NAME OF AGENCY> during this stay. For each, please tell me whether this is true or not. [ROTATE]

[READ OUT]

It is clean
It is comfortable
It is modern
Security such as door locks are good
The place feels safe because the workers are around
The neighbourhood is safe
My belongings are generally safe here
The food is good
The kitchen has enough equipment
The furniture is good
There is enough space
It is there for me when I need it
I could find privacy here if I wanted it

For each CODE

Yes
No
Not applicable

[ top ]

Q7 Thinking about this stay with <INSERT NAME OF AGENCY>, overall, how good or bad is the help and services they were able to give you other than accommodation)? Is it ...

[READ OUT]

[SINGLE RESPONSE]

Excellent
Really good
Good
Just OK
Quite bad
Bad

Q8. I am now going to ask about whether you needed some specific services during this stay. I would like you to say whether you neededthe service, not whether you actually gotthe service. Thinking about this stay, did you need...?

[ROTATE]

[READ OUT]

Help with child care
Counselling for yourself
Help with budgeting
Help with rebuilding your life
Emergency short-term help (for example, food or money but not including accommodation)
An interpreter or someone from your culture
Referrals to other community or government services (such as Salvation Army, Legal

Aid or job seeker services)

8. Help to work out your next accommodation [ASK ONLY IF RESPONDENT IN CRISIS/SHORT TERM ACCOMODATION]

For each CODE:

Yes
No
Not applicable [VOLUNTEERED - DO NOT READ OUT]

Q9 Thinking about <INSERT EACH TYPE OF SERVICE CODED 'YES' IN Q8A>, did you get the help you needed from <INSERT NAME OF AGENCY> during this stay...?

Yes
No
Not sure [VOLUNTEERED - DO NOT READ OUT]

[ top ]

Q10 Thinking about this stay, overall, do you feel the staff at <INSERT NAME OF AGENCY>

are ...

[READ OUT]

[SINGLE RESPONSE]

Excellent
Really good
Good
Just OK
Quite bad
Bad

Q11. Thinking about <INSERT NAME OF AGENCY> staff during this stay, do you feel that they are ...?

[ROTATE] [READ OUT]

Trustful of you
Involve you in decisions affecting you
Understanding of your situation
Fair
Warm and friendly
Do things on time
Take you seriously
Knowledgeable
Explain things well
Listen to you
Pay you enough attention
Available when you need them

For each CODE:

Yes
No
Don't know/not applicable [VOLUNTEERED - DO NOT READ OUT]

Q12 Thinking of this stay, what things could <INSERT NAME AGENCY> have done better? [RECORD VERBATIM. PROMPT, 'ANYTHING ELSE?' PROBE TO 'NOTHING ELSE'. ACCEPT

MULTIPLE]

[ top ]

Demographics

Q13 I am now going to ask you some questions about your personal situation. You don't have to answer these if you don't want to.

Do you, or have you ever ....? [ROTATE]

[READ OUT]

experienced psychological problems (such as depression)
had a problem with drugs or alcohol
experienced violence in the home (including physical or verbal abuse)
been seriously in debt
had a problem with gambling
had a serious health condition or disability other than psychological problems

For each CODE:

Yes
No
Refused [VOLUNTEERED - DO NOT READ OUT]

Q14 What year were you born? [ENTER YEAR]

Q15 Are you an Aboriginal or Torres Strait Islander?

Yes Aboriginal
Yes Torres Strait Islander
Yes both
No, neither
Refused/not answered [VOLUNTEERED - DO NOT READ OUT]

Q16 What country were you born in? [DO NOT READ]

Australia
Other [SPECIFY]

Q17 Are you...? [READ OUT]

[SINGLE RESPONSE]

Employed, full or part time. This includes cash-in-hand work.
Unemployed but actively looking for work
Unemployed and not looking for work

[ top ]

Q18 How many days or months have you been at this agency for during this stay? [FORCE A CODE - ENTER ZERO FOR DAYS IF RESPONDENT ANSWERS IN MONTHS]

Months days

Q19 During this stay, did you come...? [READ OUT]

[SINGLE RESPONSE]

Alone
With your partner only
With your partner and children
With your children only

Okay, so that is the end of the survey questions, thank you for your time.

We would now like to send you a $5 Coles voucher to thank you for doing the survey. We can send it to <INSERT NAME OF AGENCY> if that is easiest for you or we can send it somewhere else. Where would you like us to send it?

Agency [CONFIRM ADDRESS] <INSERT AGENCY ADDRESS>
Other [INSERT ADDRESS] CLOSE

9.13 Final questionnaire

Introduction

Hello, my name is [NAME OF INTERVIEWER]. Are you calling about the survey?

Can you tell me the number in the house on your information sheet please?

[INTERVIEWER PROMPT: IF RESPONDENT IS NOT SURE WHAT INTERVIEWER IS

REFERRING TO PROMPT WITH, THIS IS THE PIECE OF PAPER WITH A PICTURE OF A HOUSE ON IT WITH A NUMBER INSIDE]

[INTERVIEWER TO ENTER NUMBER, IF RESPONDENT CAN'T PROVIDE NUMBER ASK NAME OF AGENCY]

If agency 1301 - ask question:

What is your gender?

Male
Female
Transgender
Refused 9 [VOLUNTEERED/DO NOT READ OUT] Record Gender

[INTERVIEWER TO CODE]

Male
Female

[IF AGENCY = CRISIS AND GENDER = FEMALE ENSURE INTERVIEWER IS FEMALE, IF RESPONDENT IS HAPPY TO CONTINUE WITH A MALE INTERVIEWER THAN OKAY TO DO SO]

I now want to ask you some questions and it should only take 10 minutes. The questions are about how happy or unhappy you are with things like the accommodation, the staff, information and the services that you have got from <NAME OF AGENCY>. Everything you say is completely confidential, so nothing that you tell me will go back to <NAME OF AGENCY> or to anyone else and will not affect the services you are receiving.

[ top ]

Q1A Which of the following BEST describes what you needed from <INSERT NAME OF AGENCY> when you first arrived?

[READ OUT. ROTATE CODES 1 AND 2]

[SINGLE RESPONSE]

Just a bed for a night or two
A place to live for some time
Neither of these/Other SPECIFY [VOLUNTEERED ONLY. DO NOT READ. RECORD VERBATIM]

Q1B And did you need [READ OUT. ROTATE CODES 1 AND 2] from <INSERT NAME OF AGENCY> when you first arrived?

[READ OUT. ROTATE CODES 1 AND 2] [MULTIPLE RESPONSE]

Someone to talk to or get advice from
Referral for professional help [for example, counselling]
Anything else [SPECIFY]

For each CODE

Yes
No
Don't recall

Q2A Thinking about this stay with <INSERT NAME OF AGENCY>, overall[EMPHASISE], was the help you got...

[READ OUT]

[SINGLE RESPONSE]

Really good
Good
Just okay
Bad
Don't know/ No opinion [VOLUNTEERED. DO NOT READ].

Q2B And again, thinking about this stay with <INSERT NAME OF AGENCY>, overall

[EMPHASISE], was the help you got... [READ OUT]

[SINGLE RESPONSE]

Better than you expected
About what you expected
Not as good as you expected
Don't know/ No opinion [VOLUNTEERED. DO NOT READ]

Q2C Thinking about the length of time it took for <INSERT NAME OF AGENCY> to help you with things like services or with getting accommodation, overall was the timeliness of the help...

[READ OUT]

[SINGLE RESPONSE]

Really good
Good
Just okay
Bad
Don't know/ No opinion [VOLUNTEERED. DO NOT READ]

Q2D And again, thinking about the length of time it took <INSERT NAME OF AGENCY> to help you with things, was it...

[READ OUT]

[SINGLE RESPONSE]

Faster than you expected
About as quick as you expected
Slower than you expected
Don't know/ No opinion [VOLUNTEERED. DO NOT READ]

[ top ]

Q3A Thinking about when you first got to <INSERT NAME OF AGENCY>, or any time during this stay, do you remember receiving any information, either verbally or in writing, about ...

[ROTATE] [READ OUT]

What <INSERT NAME OF AGENCY> can offer and what it can not offer you
The rules of the house
How to make a complaint about accommodation or the service received
Other community or government services that you could use, like Salvation Army, Legal Aid or job seeker services

For each CODE

Yes
No
Not sure/guessing [VOLUNTEERED - DO NOT READ OUT]

IF YES TO ANY AT Q3A, ASK Q3B AND Q3C. OTHERWISE, SKIP TO Q4.

Q3B Thinking about the writteninformation you received, have you ever read this?

Yes
No
Don't recall [VOLUNTEERED - DO NOT READ OUT]

Q3C In general, was the information you got...?

[READ OUT]

[SINGLE RESPONSE]

Very useful to you
Quite useful to you
A little bit useful to you
Not useful to you
No opinion/ don't know [VOLUNTEERED. DO NOT READ]

Q4 What type of accommodation did you get from <INSERT NAME OF AGENCY> during this stay? Was it...

[READ OUT]

[SINGLE RESPONSE]

A room in a hotel/motel, boarding house, bed & breakfast or hostel
A bed in a dormitory
A shared room
A house/flat
Or was it something else [SPECIFY. RECORD VERBATIM]
Don't recall [DO NOT READ. VOLUNTEERED]

Q5 Thinking about this stay with <INSERT NAME OF AGENCY>, overall, how good or bad did you think the accommodation you are provided with is. By accommodation, we mean the room or house that you are staying in. Is it...

[READ OUT]

[SINGLE RESPONSE]

Really good
Good
Just okay
Bad
Don't know/ No opinion [VOLUNTEERED. DO NOT READ]
Not received accommodation

[READ OUT]

It is clean
It is comfortable
It is modern
Security, such as door locks, are good
There are always workers around
The place feels safe
The neighbourhood feels safe
My belongings are generally safe here
The food is good
The kitchen has enough equipment
The furniture is good
There is enough space
It is there for me when I need it
I could find privacy here if I wanted it

For each CODE [DO NOT READ]

Yes, true
No, not true
Not applicable [DO NOT READ OUT - VOLUNTEERED/INTERVIEWER ASSESSMENT]
Don't know [DO NOT READ OUT - VOLUNTEERED/INTERVIEWER ASSESSMENT]

Q7 Thinking about this stay with <INSERT NAME OF AGENCY>, overall, how good or bad are the help and services, otherthan accommodation [EMPHASISE], that they are able to give you? Are they ...

[READ OUT]

[SINGLE RESPONSE]

Really good
Good
Just okay
Bad
Don't know/ No opinion [VOLUNTEERED. DO NOT READ]

[ top ]

Q8. I am now going to ask about whether you needed some specific services during this stay. I would like you to say whether you neededthe service, regardless of whether you actually gotthe service. Thinking about this stay, did you need...?

[ROTATE]

[READ OUT]

Help with child care
Counselling for yourself
Help with budgeting
Help with rebuilding your life
Emergency short-term help [for example, food or money but not including accommodation]
An interpreter or someone from your culture
Referrals to other community or government services [such as Salvation Army, Legal Aid or job seeker services]
Help to work out your next accommodation

For each CODE:

Yes
No
Don't know [VOLUNTEERED - DO NOT READ OUT]

Q9 Thinking about <INSERT EACH TYPE OF SERVICE CODED 'YES' IN Q8>, did you get the help you needed from <INSERT NAME OF AGENCY> during this stay...?

Yes
No
Not sure [VOLUNTEERED - DO NOT READ OUT]

Q10 Thinking about this stay, overall, do you feel the staff at <INSERT NAME OF AGENCY> are ...

[READ OUT]

[SINGLE RESPONSE]

Really good
Good
Just okay
Bad
Don't know/ No opinion [VOLUNTEERED. DO NOT READ]

Q11. Thinking about <INSERT NAME OF AGENCY> staff during this stay, do you feel that they ...?

[ROTATE]

[READ OUT]

Are trustful of you
Involve you in decisions affecting you
Are understanding of your situation
Are fair
Are warm and friendly
Do things on time
Take you seriously
Are knowledgeable
Explain things well
Listen to you
Pay you enough attention
Are available when you need them

For each CODE:

Yes
No
Don't know [VOLUNTEERED - DO NOT READ OUT]

Q12 Thinking of this stay, what things could <INSERT NAME AGENCY> have done better? [RECORD VERBATIM. PROMPT, 'ANYTHING ELSE?' PROBE TO 'NOTHING ELSE'. ACCEPT MULTIPLE]

[ top ]

Demographics

Q13 I am now going to ask you some questions about your personal situation. You don't have to answer these if you don't want to.

Do you, or have you ever ....? [ROTATE]

[READ OUT]

experienced psychological problems [such as depression]
had a problem with drugs or alcohol
experienced violence in the home [including physical or verbal abuse]
been seriously in debt
had a problem with gambling
had a serious health condition or disability other than psychological problems

For each CODE:

Yes
No
Refused [VOLUNTEERED - DO NOT READ OUT]

Q14 In what year were you born? [ENTER YEAR]

...............

Q15 Are you an Aboriginal or Torres Strait Islander?

Yes Aboriginal
Yes Torres Strait Islander
Yes both
No, neither
Refused/not answered [VOLUNTEERED - DO NOT READ OUT]

Q16 What country were you born in?

[DO NOT READ]

Australia
United Kingdom [Great Britain/England/Northern Ireland/Scotland/Wales]
Ireland [not Northern Ireland - see UK]
New Zealand
Austria
Balkans [Yugoslavia/Croatia/Serbia/Montenegro/Bosnia-Herzegovina]
Chile
China
East Timor
El Salvador
Germany
Hungary
India
Iran
Italy
Laos
Lebanon
Netherlands [Holland]
Pacific Islands except PNG [Cook Islands/Tonga/Fiji/NIUE Island/Torres Straits/Badu Island/Vanuatu]
Papua New Guinea [PNG]
Philippines
Poland
Romania
Samoa
South Africa
Sri Lanka
Thailand
Vietnam
Other [SPECIFY]

[ top ]

Q17 Are you...? [READ OUT]

[SINGLE RESPONSE]

Employed, full or part time. This includes cash-in-hand work.
Unemployed but actively looking for work.
Unemployed and not looking for work.
Refused [VOLUNTEERED. DO NOT READ]

Q18 How many days or months have you been at this agency for during this stay?

...............Days and/or...............Months

Q19 How many more nights do you plan to stay at <INSERT AGENCY NAME>?

[DO NOT READ]

[SINGLE RESPONSE]

None
1 - 7 nights
8 - 30 nights
31 or more nights
Don't know [DO NOT READ OUT - VOLUNTEERED]

Q20 During this stay, did you come...? [READ OUT]

[SINGLE RESPONSE]

Alone
With your partner only
With your partner and children
With your children only

Okay, so that is the end of the survey questions, thank you for your time.

[If code 2009, read SAFEWAY instead of COLES] [DO NOT READ OUT]

Agency [CONFIRM ADDRESS. SHOULD BE PRE-POPULATED FROM DATABASE]
Other [INSERT ADDRESS]
Objection raised to Coles voucher [DO NOT READ OUT]

If code 3 in Q20

Q21 Do you have a Coles supermarket nearby?

If code 1 in Q21

Q22 Can I please get an address for us to send the voucher to?

Send voucher [INSERT ADDRESS]

If code 2 in Q21

Q23 Instead of a Coles voucher, we will send you cash. Can I please get an address for us to send the cash to?

Send cash [INSERT ADDRESS]

THANK AND CLOSE

[ top ]

9.14 MRSA Ethics Approval Letter

MRSA Ethics Approval Letter image

SAAP website.
Australian Federation of Homelessness Organisations. June 2003. Measurement of client satisfaction in the Supported Accommodation Assistance Program (SAAP): Final Report.
It was agreed between FaCS and CBSR that, in South Australia, the 'other' agencies would be divided into two groups - 'other' and 'multiple' as there was a high portion of agencies that operate as 'multiple' (i.e. significantly higher than in other states/territories) and it was felt important that these agencies be able to be separated for analysis purposes.
Several agencies gave more than 1 reason for non-participation, while many gave no reason.
CBSR contacted these eight agencies to advise of the revised methodology and to inform the agencies that they can participate if they wanted to.
Clients informally exiting were, by definition, excluded from the sample and the survey results. It must be noted that, since these clients may well have higher levels of dissatisfaction with the service provided, this may introduce a bias into the survey findings.
This was extended to four weeks after consultation with the Steering Committee. It was found that despite conducting a census of agencies and clients, four weeks was needed to achieve the target sample of n=1000 clients.
This information is based on agency data provided during and at the inclusion of the fieldwork period and may not be perfectly accurate.
Calculated as a percentage of those participating as a proportion of those invited to participate (1000/2344)
Agency level reporting has not been commissioned at this time.
As data for 2003 were not available, the 2002 figures were used as proxies.

Previous: Appendix D: Verbatim Suggestions for Improvement

[ top ]