Appendix C - Evaluation framework
| FHBH Evaluation | |||
|---|---|---|---|
| Key Program Objectives | Key Evaluation Questions | Data Sources | Performance Measures |
| 1. To improve the safety and functioning of housing within the Indigenous communities where FHBH has been implemented, and in a cost-effective way | 1.1 What was the state of Indigenous housing prior to FHBH? What problems were present? |
|
Problems in Indigenous Housing prior to FHBH |
| 1.2 What was the state of housing after FHBH occurred? What problems were fixed? |
|
Problems rectified in Indigenous Housing after FHBH & their level of seriousness | |
| Section 1 of the framework addresses the following outcomes as stated at A2 of the
Consultancy Contract:
A comprehensive analysis of the implementation and management phase of FHBH projects, with particular emphasis on client community involvement, sustainability and the development of housing maintenance management capacity within target communities. A thorough cost/benefit and/or cost-effectiveness analysis of the FHBH projects. |
1.3 What has been the effect of the passage of time on the outcomes of FHBH? Have improvements been sustained? Why or why not? |
|
FHBH improvements have been successfully built upon by the community |
| 1.4 Do the residents feel that their houses are safer and healthier since FHBH? |
|
Level of perceived safety of residents | |
| 1.5 What are the remaining problems within housing in Indigenous communities? |
|
Remaining problems & their level of seriousness | |
| 1.6 What have been the budgets for the FHBH projects? | Consultation with:
|
Budget information | |
| 1.7 On what items has the money been spent? What are the most expensive items? Is there room to achieve further efficiencies? |
|
Cost structure of monies spent | |
| 1.8 Approximately what proportion of problems (routine, damage, faulty) (essential, urgent, routine) within communities are being fixed through the budgets? | Consultation with & data from:
|
Percentage of problems being fixed | |
| 1.9 Are the most serious problems being fixed? Does this differ between communities? | Consultation with & data from:
|
Improvements in overall ratings of houses | |
| 1.10 What is the sensitivity of the level of money spent? That is, if we allocated 50 to 100 per cent more or 50 per cent less, what is the likely increase/decrease in the number of problems that will be fixed? | Analysis by Consultant Team | The practical effects of marginal increases / decreases in per house funds on improving the quality of houses | |
| 2. To transfer housing maintenance systems, skills and employment to the Indigenous communities (and ICHOs) in which FHBH has operated | 2.1 What level of community/ICHO involvement in employment, training and project management opportunities occured through FHBH? Has this been an appropriate level? Did communities want to be involved? |
|
No. of community/ICHO (Indigenous / non-Indigenous) members involved in the FHBH projects Community/ICHO perceptions regarding level of involvement - enough, not enough, too much? |
| 2.2 What housing maintenance systems and skills are communities/ICHOs and individual participants left with after FHBH? What did they have before? Is there new employment as a result of FHBH? |
|
Increased and/or improved systems/skills/employment transferred to the community/ICHO/community members | |
| Section 2 of the framework addresses the following Outcomes as stated at A2 of the
Consultancy Contract:
A comprehensive analysis of the implementation and management phase of FHBH projects, with particular emphasis on client community involvement, sustainability and the development of housing maintenance management capacity within target communities. |
2.3 Have the systems and skills that have been learnt through FHBH been used by communities/ICHOs towards housing maintenance? If so, where and under what circumstances? Have these systems and skills been used in other ways in the community? |
|
Re-use of skills gained through FHBH - for housing maintenance / in other ways |
| 2.4 Do the communities/ICHOs and community members who were involved in FHBH feel confident that they could maintain housing better now that they have obtained systems and skills through FHBH (or would they require further support applying these)? Do they use/prefer other systems and skills and if so why? |
|
Level of community/ICHO/community member confidence and success in applying FHBH systems and skills for housing maintenance | |
| 3. To encourage states and territories to adopt housing assessment and maintenance programs in their asset management systems | 3.1 Has any state or territory adopted the FHBH assessment (or something similar)? Why/why not? | Consultation with:
|
State or territory implementation of FHBH framework |
| Section 3 of the framework addresses the following Outcomes as stated at A2 of the
Consultancy Contract:
An assessment of the interrelationships between the FHBH projects and State and Territory Indigenous housing policy and program settings, including a description of how these interactions have impacted on the implementation of FHBH projects. |
3.2 Do the states & territories have a clearer understanding of maintenance requirements of Indigenous Housing as a result of FHBH? Has this understanding translated into improvements to documented (and budgeted) maintenance programs? | Consultation with:
|
Clearer understanding of maintenance requirements Presence of maintenance schedules |
| 3.3 Has FHBH influenced the allocation of state & territory funds with regard to maintaining Indigenous Housing? Have FHBH funds been used to leverage better outcomes? | Consultation with:
|
State and territory funding changes due to FHBH | |
| 4. To provide a point-in-time analysis of the quality of housing stock in Indigenous communities (to determine progress on BBF outcomes) | 4.1 Has a baseline understanding and framework for that understanding been developed that assesses the quality of housing stock in Indigenous communities before and after FHBH? How does this relate to NRF/CHINS/census analysis? | Consultation with:
|
Existence of a baseline understanding and framework
As good as or better than other frameworks / understandings |
| Section 4 of the framework addresses the following Outcomes as stated at A2 of the
Consultancy Contract:
An assessment of the historical, socioeconomic and political environment in which the FHBH projects were introduced and in which FHBH projects continue to operate, with a focus on how these conditions have advantaged and/or disadvantaged FHBH projects. An assessment of the development and design processes for FHBH projects, including how these processes have been influenced by governance factors, the behaviour of participants and the circumstances of client communities. A comprehensive analysis of the implementation and management phase of FHBH projects, with particular emphasis on client community involvement, sustainability and the development of housing maintenance management capacity within target communities. |
4.2 Has this framework allowed an 'any-point-in-time' analysis of the quality of the housing stock? |
|
Useful analysis of housing in Indigenous communities |
| 4.3 What proportion of Indigenous housing stock is analysed / assessed as part of FHBH? Is this adequate and effective? (see also program objective 5) | Consultation with:
|
Percentage of stock analysed | |
| 4.4 How (and why) has this framework changed over time? | Consultation with:
|
Documentation only (no performance measures) | |
| 4.5 Is the current framework still considered to be a useful measure of quality of housing in Indigenous communities? | Consultation with:
|
All serious problems within houses in Indigenous communities are being identified | |
| 4.6 Has the framework assisted the Government and Indigenous communities to understand/scope the capacity and context of Indigenous communities, and to undertake and systemise the maintenance requirements for Indigenous housing with regard to capacity and context? (see also program objective 4) | Consultation with:
|
Presence of maintenance schedules |