Introduction

It has become increasingly obvious that in order to adequately address violence in the long term, a multi-stakeholder, multi-sectoral and partnership-driven approach is required to deliver services that address the root causes of crime and violence. According to Rosenbaum (2002: 171), such partnerships are ‘widely praised as the ideal vehicle for implementing complex and comprehensive interventions.’ The very notion of social and developmental crime prevention is arguably premised on this assumption (Pelser 2002; Bruce & Gould 2008).

To assess integrated strategies aimed at preventing violence, evidence from South African experiences has been used to highlight some of the inherent difficulties in developing appropriate outcome indicators that adequately capture the impact and role of often diverse interventions, implemented by different partners. It has also proven difficult to attribute these outcomes to the different interventions. In an ideal scenario, any partnership requires some shared or common indicators, which in themselves should facilitate and promote partnerships (Morgan & Homel 2011). Particularly in data-scarce environments where official statistics may be flawed, inaccurate and collected infrequently, the collection of data for baseline and evaluation, and the development of data sources to inform appropriate indicators of measurement, is often a complicated, costly and time-consuming exercise. While local government is theoretically responsible for social crime prevention partnerships, it also often has a limited ability to collect and manage data.

In the case of civil society partners who are often primarily responsible for much of the service delivery, there is usually little capacity or budget to collect the appropriate and rigorous data needed to inform the performance framework and outcome assessments. This is particularly the case as the need for higher quality data and evaluation methodologies becomes imperative in the move toward a more evidence-led and systematic approach to crime and violence prevention.1 Where data does exist, either with government partners or civil society partners, the measurements or geographical boundaries of intervention data are often not clearly stated among the various partners.

Among these difficulties is the accountability for service delivery and performance between different government departments and between government, civil society and academic/research institutions. In short, as Rosenbaum (2002) argues in the case of social crime prevention initiatives, a partnership’s greatest strength is also its greatest weakness: the diversity of agencies and institutions, and thus the diversity of opinions (including measurements and outcomes), approaches, and orientation towards social problems. This, it is argued based on the experience of this paper’s authors, can also be extended to the diversity of resources.

The authors thus propose to explore these challenges within a South African context, drawing on the experience of a number of inter-agency, local level crime and violence prevention partnerships. The paper will focus on three pilot projects, each conducted in a separate province of South Africa, and detail how the performance framework and measurements were developed. Practical challenges and difficulties faced in the conceptualization, application and analysis of these projects, as well as the success achieved in the implementation of these frameworks will also be detailed and explored. The paper will not focus on the outcomes of the projects themselves, but rather on the development and implementation of the performance measures, and on how change, specifically behavioural change, is measured in such projects.

Outline of the Paper

To begin, the paper will outline the projects undertaken in 2008 by the Open Society Foundation of South Africa (OSF-SA), the Criminal Justice Initiative with the support of the Centre for Justice and Crime Prevention (CJCP) and the Department of Community Safety in three low-income and crime-ridden communities of South Africa. A discussion of these projects will assist in determining how performance measurements were developed in each safety audit and consequently in the safety plans adopted. This will be followed by an analysis of the use of these performance measurements in assessing the successes and challenges encountered during the implementation of these projects. This exercise will facilitate the process of identifying the worthwhile measures for participatory monitoring and evaluation. Following this, a discussion of the challenges encountered in measuring and evaluating the impact of these projects, as well as identifying the successful indicators, will be conducted.

The importance of developing measurement data in a way that is accessible, appropriate for all partners, useful in advocating change and in adapting and further developing specific interventions, will also be explored. To accomplish these goals it is recommended that a specific amendment is added, which explains how local partnership structures can use data, and the form in which data can best be used by such structures. The use of participatory monitoring and evaluation methodologies that can also be easily adopted by local structures (as a collective) and by the individual partners will be examined, and recommendations made as to how these can be best adapted to different environments within the urban environment. Finally, the paper will provide a discussion of how appropriate measurements of change and impact on social crime and violence prevention interventions can be adapted in environments that are particularly characterised by resource and data constraints.

Background to the Project Sites

Through multi-stakeholder engagement, safety audits and consequently safety plans were effected in three pilot sites that were identified with the assistance of the Department of Community Safety as being crime hotspots. These are: Orange Farm in Gauteng Province, Nompumelelo in the Eastern Cape, and Elsie’s River in the Western Cape. The responsibility of the multi-stakeholder forum was to create a platform to encourage collaboration among parties in the bid to identify challenges in these communities. Furthermore, their mandate was expanded to collectively develop solutions to the safety challenges experienced in the three sites. Additionally, the diverse stakeholders were brought together to encourage and ensure that each component of the project was monitored appropriately and that lines of communication between all the stakeholders were open (Tait & Ehlers 2010). The features of these sites will be briefly outlined in order to create a basis for the discussions.

Elsie’s River: Set in the Cape Flats, Elsie’s River has previously received support for urban renewal as it was selected as the site for the Cape Flats Renewal Strategy. However, it is evident that the crime prevention interventions in this area have not been carried through and have had little or no evaluation. The discontinuation of projects is attributed to the fact that most strategies have focused on gang reduction which has led new efforts or interventions to rely on this focus as the only basis for understanding what can still be done to improve safety (Tait & Ehlers 2010).

Nompumelelo: Conversely, Nompumelelo, originally a township which housed migrant labourers who wanted to be close to their places of employment, has improved standards of low-income housing. However, it is an overpopulated area, particularly during weekdays. Overpopulation undoubtedly brings various challenges for the population, especially in terms of resources and infrastructure which negatively affects community cohesion (Tait & Ehlers 2010).

Orange Farm: With a population of over one million and located 45 kilometres south of Johannesburg, Orange Farm is one of South Africa’s largest townships. It is underdeveloped and the majority of residents experience socio-economic challenges. Given that 40 per cent of the population is under the age of 18 years, this is to be expected. Many of these young people are believed to be involved in crime, substance abuse and violent behaviour. Linked to this are the high rates of teenage pregnancies and HIV/AIDS infections. However, it must be noted that there have been considerable infrastructural investments in Orange Farm, such as the construction of the new police station, railway station and victim support centre. Despite these community investments, a great percentage of the Orange Farm population do not have access to proper health care, education and recreational facilities. The focus is therefore on crime-combatting strategies such as the arrest of publicly intoxicated teenagers; however, there has been no investment in providing access and support for substance abuse treatment. This lack of support has therefore led to a vicious cycle of repeat offenders and arrests, rather than solutions to the substance abuse problem (Ehlers & Tait 2009).

Developing Indicators for the Measurement of Impact and Change

Measurement practices have long existed in the developed world, such as the Comprehensive Community Initiatives (CCIs) in the United States, which were developed in the 20th century as a collective measurement of violence and crime reduction (Halpern 1995; O’Connor 1995).2 Similarly, in recent years there has been a focus on delivering crime and violence prevention programmes through multi-stakeholder partnerships in the Global South. These partnerships aim to bring together public sector organisations and other actors, such as civil society organisations, community members and business people. The involvement of diverse individuals and groups demonstrates that there is a growing recognition that all sectors in society have the responsibility and ability to play a role in crime and violence prevention.

While there is no strict method to monitor and evaluate a project, it is important to develop an approach that is appropriately linked to the process being administered. A variety of ideas for broadening the design and methods of monitoring and evaluation practices was put forward by the Department for International Development (DFID), and these may be relied on as guidelines (Stern, Stame, Mayne, Forss, Davies & Befani 2012). In the case of the safety plans discussed here, there were initially underlying assumptions on the causes of crime and violence in these three sites that guided the development of indicators. Therefore, these assumptions with initiatives aimed at reducing violence were tested. Safety audits to understand the challenges faced in these communities were conducted and thereafter, safety plans were developed. Notably, community initiatives partnered with the local and provincial government in each site as well as with public sector actors in order to manage, monitor and facilitate the implementation of these safety plans.

In the three sites, the OSF-SA worked with a Multi-Stakeholder Forum (MSF) which acted as a fulcrum to coordinate the input of all the parties involved in the safety plan. Throughout this process, the OSF-SA used the monitoring and evaluation processes to implement the successful aspects back into the project, and to make these experiences available to a broader audience. To develop safety plans for the three sites, evidence-based assessments of crime and violence levels were undertaken of crime and violence indicators regarding the actual crimes taking place, their locations in each site and aspects of the victims and offenders.

The MSFs encouraged collaboration with a multidisciplinary project team of community members and local and provincial government departments in each site. Ehlers & Tait (2009) note that the MSFs are similar to the multi-disciplinary Community Safety Forum structures which existed as potential crime prevention initiatives. It was the OSF-SA’s goal to provide funding for the support of civil society interventions that would conduct work necessary to the safety plans. Through this grant-making, the OSF-SA aimed to ensure that government departments and civil society organisations would work together to improve safety through the MSFs. It was also anticipated that this process would illuminate collaboration problems between civil society and government departments. The safety audits provided a basis to develop indicators to analyse the existence and impact of the following:

  1. The profile of the community (location, demographics etc.)
  2. The nature and extent of recorded crime in the identified community
  3. The specific safety concerns in the community
  4. The available state resources to address these concerns
  5. The available non-state resources to address these concerns
  6. Current interventions being implemented in the community
    • The impact of these current interventions
    • The challenges still faced in implementing these interventions
  7. A database of stakeholders and role players in the community as well as the institutional mechanisms in place to support broad multi-stakeholder safety initiatives (Ehlers & Tait 2009)

The process of conducting safety audits on the three sites was valuable in two main ways. The data gathering did not only identify the relevant key stakeholders, their duties/activities and ensure their involvement and buy-in. It was also helpful for providing baseline data which is important for use in future evaluations.

Using the safety audits, the OSF-SA managed to develop detailed site-specific safety plans. A snapshot into the aims of each plan can be seen in the following table. The safety plans differed based on each of the pilot communities’ needs which were established in the safety audits. The activities of each are outlined in Table 1.

Table 1

Indicators for improved safety in target areas developed by stakeholder forums from safety audits (OSF-SA 2009).

Elsie’s river Nompumelelo Orange Farm

  • Support families and young people
  • Ensure programmes and services are available
  • Create opportunities for recreation and skills development for youth
  • Address alcohol and substance abuse
  • Develop positive awareness programmes and access to treatment and support
  • Integrate safe schools into the community and ensure their support by the community
  • Put in place an effective and trusted criminal justice system
  • Improve the quality of services and infrastructure
  • Increase employment
  • Promote safety and prosperity for all.
  • Improve municipal services and facilities
  • Support an integrated youth safety and development programme
  • Create a programme addressing facilitators of crime, particularly alcohol and drugs
  • Promote job creation
  • Promote a more effective and efficient criminal justice system.
  • Improve access to services
  • Strengthen youth and family initiatives
  • Improve access to social grants
  • Support community development
  • Build partnerships
  • Reduce alcohol- and drug-related crime

When considering the progress and changes that have been made, it is important to note that measuring the impact that each initiative has had on meeting a specific goal is not straightforward. Indicators of change and impact are diverse and their detection can be challenging. Some outcomes are easier to discern, while others take longer to appreciate and still others cannot be measured using quantitative or qualitative indicators.

Based on the goals outlined in the previous table, outcome measurements in Nompumelelo, measured after the last safety audit and development of the safety plan, can be seen in Table 2.

Table 2

Nompumelelo: detectable outcomes from the development of the safety plan (OSF-SA 2009).

Improving municipal services and facilities Promoting school safety Curbing alcohol and substance abuse Promoting an effective criminal justice system

  • A fence has been erected around the sports field and grass has been planted.
  • Half of the main road in Nompumelelo has been tarred.
  • A CCTV camera has been installed to monitor the area around the taxi rank.
  • Access to water and sanitation for residents of the informal settlement has been improved.
  • The South African Police Service (SAPS) visits schools regularly.
  • The National Institute for Crime Prevention and the Reintegration of Offenders’ (NICRO’s) Safety Ambassadors programme has been implemented, and a NICRO social worker was appointed to the Nompumelelo area.
  • An after-school care programme has been established at both schools and food gardens have been planted.
  • A code of conduct regulating closing times for taverns has been agreed upon by the tavern owners, and SAPS is enforcing the code.
  • Visible policing has been improved with the establishment of sector policing.

The two tables clearly demonstrate the discrepancies between developing and satisfying indicators when measuring the impact and outcomes of crime and violence prevention initiatives. Based on the progress made in Nompumelelo as seen in the preceding table, various sources of data and performance indicators were taken into consideration when assessing the progress that has been made. Moreover, measuring the effectiveness of crime and violence prevention mechanisms is challenging as it is theoretically grounded. The theoretical ideas behind some ideas may translate into unforeseen challenges when practically applied. For instance, while erecting a fence around a particular area is in theory expected to curb crime and send a message of protection, it may also be construed as contributing to exclusionary practices.

However, as demonstrated in the problematic issues raised by the community and the proposed solutions, the multi-faceted nature of the measures suggested demonstrate the diverse ways in which each multi-stakeholder approaches crime and violence prevention. Participatory initiatives for crime and violence prevention therefore bring innovative approaches to the challenges of citizen safety and security. Additionally, as seen in the actual initiatives and activities in Nompumelelo, the SAPS, Department of Public Works, NICRO and community members were all responsible for the safety improvement in various ways. This is evidence that multi-stakeholder partnerships draw together a range of mechanisms to achieve their goals. The forum benefits from the specific competencies and capacities that each stakeholder brings and, in so doing, larger networks are formed which enables greater engagement with the wider community and possibly improves the capacity to influence the policy agenda. The larger networks created from the engagement with multiple stakeholders also adds to the pool of resources. These come in the form of technical, human, physical and financial resources within each sector.

Nevertheless, there are evidently some challenges, many of which were experienced in these projects in the three sites. These may be detected in each of the planning, delivery and evaluation phases. The next section will highlight the key challenges encountered in the three project sites and some of the factors that led to these difficulties will be discussed.

Challenges Experienced in the MSFs

Similar to the challenges identified in CCIs in the USA, multi-stakeholder forums face the problem of operationalising their procedures in ways that are considered legitimate by the majority of stakeholders. More time would need to be invested in the operationalising procedures in order for the inputs of all stakeholders to be considered. The multi-stakeholder approach should also depend on funding from a variety of sources – if not, the key funders may influence the whole group with their ideas (Barchechat & Sansfaçon 2003: 13). This might not be appropriate as it could cause further complications. In regard to funding, the OSF-SA committed to provide funding to civil society actors working within the safety plans, and who are now producing commendable results. The collaboration in providing funds perhaps mitigated the challenge of working with a lack of resources, but it is unclear how this financial contribution has contributed to a competition for resources among the various stakeholders who are all working within the safety plans.

A key challenge that affects all phases of safety initiatives, particularly in countries such as South Africa, is the lack of available and reliable data. Ehlers & Tait (2009) highlight the lack of longitudinal evaluations in particular. The lack of data hampers the development of a crime prevention policy since there is no way to track the impact of interventions over time in order to gauge the viable options. The scarcity of data, especially of a longitudinal nature, is common in such multi-stakeholder partnerships in developing countries. In most cases, data is flawed or collected irregularly. Another challenge concerns the reluctance to share data. This hesitation may not be a result of poor relations between stakeholders but instead may be a consequence of a limited research capability which inhibits the ability of stakeholders to collect and disseminate data among each other. However, the existence of multiple partners creates an opportunity for these challenges to be alleviated since the little data from each source may be shared and used to develop worthwhile safety plans.

Related to the flexibility of the multi-stakeholder partnership is the difficulty of determining the amount of progress. Establishing the process of how interventions are measured as well as the subsequent decision that an intervention has been successful becomes more challenging with the increasing flexibility and the availability of many partners. This is because some milestones may not be considered to be progress by other partners. Diverse partners need to collectively recognise and agree on milestones that may not be central to their core business, but necessary to achieve overall safety. This challenge can be mitigated at the planning stage where all partners are engaged in the process of determining their indicators. Furthermore, when an initiative has been unsuccessful, the responsibility of rectifying and the accountability for the lack of success also tend to be more challenging among multi-stakeholder partnerships.

Yet another challenge associated with multiple stakeholders is the difficulty of aligning all parties’ process and outcome indicators with the broader overall project indicators and outcomes. The challenge then is to construct indicators common to all stakeholders, sectors and institutions. It is also important to take into account the different data sources that are derived from various data collection techniques among the broad group of stakeholders. Related to this is the challenge presented by different administrative data and data boundaries. For example, data for station precincts may reflect larger or smaller populations than the data from schools, national or local government department offices, as the geographical boundaries are often designed differently. Therefore, such disjunctures need to be taken into consideration.

These challenges also relate to the difficulties of measuring impact or behavioural change which, without developing appropriate and detectable indicators, may be impracticable. Such issues may be compounded by instances where external service providers to those included in the MSF may be involved in the safety plan. This calls for the mapping of all service providers who may not be in the MSF but are linked to stakeholders.

Mitigating Challenges: Lessons from the Multi-stakeholder Process

The acknowledgement from the outset that the inclusion of multi-stakeholders was essential in developing and implementing safety plans for all three sites was beneficial. Evaluating crime prevention programmes is riddled with challenges regarding issues such as the measurement of the actual impact from multi-stakeholders’ perspectives and the disjuncture between researchers’ and practitioners’ perceptions or gauges on the level of impact (Barchechat & Sansfaçon 2003: 13). The safety plans rolled out in the three sites provide insight into the ways in which some of the typical challenges can be met.

Multi-stakeholder forums as seen in the three cases provide for a more representative image of the community, and therefore have a greater overall impact if managed democratically. Their flexible structure also allows for experimentation or exploration (Barchechat & Sansfaçon 2003). Evidently Community Safety Forums, which play an integral role in planning safety initiatives, would best be located within these multi-stakeholder forums, while indicators for monitoring and evaluation should be developed as a collective within the group.

Developing Participatory Monitoring and Evaluation Practices

In order for safety plans to best achieve their aims, it is advised that they are developed in a way that the goals set are ‘SMART.’ This means that objectives should be: specific, measurable, achievable, relevant and time bound. Consequently, the monitoring and evaluation should be ‘SPICED’: subjective, participatory, interpreted and communicable, cross-checked and compared, empowering, and disaggregated (GIZ 2014). Additionally, it is evident that the safety plans developed for the three pilot sites would benefit from an approach which use the multi-stakeholder basis to develop performance indicators of a quantitative and qualitative nature as well as proxy indicators.

The importance of proxy indicators

While quantitative and qualitative indicators are useful for measuring the impact of the safety plans put in place, it is discernible from the progress noted after the implementation of the safety plans that proxy indicators (such as people’s feelings regarding safety) used to gauge safety are also important. The importance of proxy indicators is evident in that describing changes in behaviour is challenging and if the indicators are developed in a way that discounts some actions or results, then the progress might not be easily recorded. Also, quantitative data which shows increases or decreases in certain behaviours is not necessarily objective, particularly for crimes that are under-reported such as rape and domestic violence.

Detecting Impact and Change: Collectively Developing and Monitoring Progress Indicators

Similarly, the process of developing progress indicators requires as much attention as the actual implementation of the safety plan. In their systemic participatory safety planning toolkit, the Inclusive Violence and Crime Prevention Program (VCP) of the German Development Cooperation (GIZ) identify the need for monitoring and evaluation to include what is referred to as ‘progress indicators.’ These show the progress made through milestones as well as ‘impact indicators’ which indicate the impact of the prevention measures on the quality of life of the people in the project area (GIZ 2014).

Techniques used to chart progress and impact may take the form of ‘participatory rapid appraisals’ which encompass various methods to enable local people to analyse their own realities. Group exercises are employed to facilitate information sharing and analysis. Visualisation techniques, such as diagramming, are emphasised so that participants can discuss, modify and extend their ideas. These techniques are used as the basis for planning, monitoring and evaluating development activities which facilitate the development of indicators. Audio-visual techniques such as video diaries, story-telling, drama performances, songs, and photographs can be relied on to help identify and measure the communities’ perceptions of changes within their communities as a result of the project.

An additional benefit of an audio-visual technique, apart from its interactive nature, is that it recognises that multiple, non-linear events lead to change. It focuses on people and behavioural changes, as well as on the progress that development interventions have had on capacity-building and empowering the community. However, outcome mapping assumes only that a contribution has been made, and never attempts attribution unlike the aforementioned activities.

Philliber’s method, which uses charts to track and visualize the change that has occurred, is another such technique. Such an activity involves charting a causal sequence that involves the processes on one side of the chart, short-term outcomes in the centre of the chart, and long-term outcomes on the other end of the chart (Philliber 1998). Additionally, the ‘most significant change’ technique may be relied on to detect progress. This technique involves collecting stories about change that has occurred in the community. Stories collected from stakeholders are discussed and critically analysed to help determine the impact of the intervention. This technique is useful for monitoring intermediate outcomes and impact as each story highlights the experiences and changes detected.

However, it is important to point out that when introducing violence and crime prevention initiatives, change may only become noticeable over time. This highlights Ehlers & Tait’s assertion on the importance of longitudinal data as this kind of data assists in detecting and reflecting change. Therefore, it is necessary to develop indicators which are time specific. When the safety plans were put in place, various indicators to monitor and evaluate the changes that might occur as time went by were needed. The measurement of these changes depended on the extent to which the measures of the safety plans were appropriate and detectable using the selected indicators.

Furthermore, as identified in the challenges experienced, there is a need for baseline data as well as more information at each stage of the monitoring process so as to enable the viable evaluation of the projects.

Joint Assessments, Implementation, Monitoring and Evaluation

In order to reconcile the gap in knowledge and practice between researchers and practitioners and maximise the benefits from both groups, it is important to have collaborative planning, implementation, monitoring and evaluation systems (Barchechat & Sansfaçon 2003). It is also worthwhile to have a single expert or group of experts monitor and evaluate the safety plan from the outset in order to facilitate the use of appropriate methods for developing indicators and documenting the process. This would also allow for the monitoring and evaluation of both the outcomes and impact of safety plans/initiatives. The OSF-SA also highlights the importance of monitoring and evaluation information data to be written and produced in a way that is accessible to community members. This, and the need for cost-effective monitoring and evaluation methods in communities is evident (Griggs 2012).

It is important for a certain level of consistency in design and methods throughout the lifespan to exist. Also, a change of champion or lack of institutional memory in partners may potentially introduce another challenge. This may occur in instances where individuals or officials such as the police or heads of different organisations involved in the projects are redeployed. Therefore, there needs to be documentation of all the various processes of the projects in order to avoid losing valuable information along with a stakeholder.

Methods of Documenting Each Stage of the Safety Plan

One of the most significant aspects of the OSF-SA safety audit initiatives was the contribution of information on crime prevention, law enforcement and safety. As a result, an additional facet of the project was introduced which aimed at developing and implementing a longitudinal evaluation across the three sites. The realisation of the dearth of longitudinal data encouraged this initiative. In order to initiate the evaluation system, safety indicators were developed and tailored to each site, stakeholder and service provider. Since each site will have individualised indicators and systems, service providers in each site will be able to track and report on their progress. These processes will be aligned to a larger tracking system across the entire safety plan. These systems will facilitate an assessment of each site’s crime and safety project annually and consequently enable more comprehensive longitudinal evaluations in the future (Ehlers & Tait 2009).

Determining Each Stakeholder’s Responsibilities

Of great importance when planning and implementing an MSF is the need to outline each stakeholder’s responsibilities and what is expected of them. Evidently, the findings from the audits call for a broad approach when addressing crime and violence in each community. This involves collaborative efforts which work to both influence and transform the criminal justice system and impact individual attitudes towards behavioural change. There is also a role for institutions which focus on socio-economic issues, and on how these challenges influence crime and violence and how they are experienced and dealt with by the community.

Concluding Remarks

The experiences from the case studies in the three sites demonstrate that measuring impact and change effected through community and state violence prevention partnerships requires participatory monitoring and evaluation systems. It is important for the multi-stakeholder partnerships to be consensus-based, process-oriented and involve a substantial investment from the outset to promote the construction and the maintenance of that partnership. This will facilitate the identification of the monitoring and evaluation requirements and indicators which encourage change in the crime and violence prevention initiative.

Based on discussion, it is clear that multi-stakeholder involvement is necessary in order to contribute as much data as each party can provide. For example, a programme of early interventions that includes working with children, parents, families and schools at a community or site level, will likely necessitate the inclusion of process and outcome indicators drawing on diverse official datasets. These datasets include school enrolment, attrition and performance data, nutritional and health data, quality-of-life data and other proxy indicators, as well as more direct crime-related data such as official crime statistics or victimization data. Besides these, however, are common indicators relating to safety and well-being outcomes for the children, families and ultimately communities in which the interventions are located.