Chapter 6. Delivering and evaluating people-centred legal and justice services
This chapter considers the fourth step, including “what works” in delivering and evaluating legal and justice services. It provides an overview of good practices consistent with the goals of contributing to inclusive growth and enhanced individual and community well-being. It also consolidates what we know works in identifying, measuring, mapping, designing, delivering and evaluating people-centred legal and justice services into a preliminary outline of common criteria for people-centred access to justice.
Step 4 rationale and overview
The fourth and last stage of planning for people-centred legal and justice services involves matching the services to meet needs by focusing on how these services can be most effectively planned, delivered and evaluated. This stage brings together and builds on the first three stages of identifying and measuring legal needs, mapping needs and designing legal and justice services that will work to meet these needs. Planning the delivery of services involves the next level of policy questions:
-
Are those needs being met?
-
Are services being delivered where the need is?
-
Are appropriate types of services being matched to each type of need?
Figuring out what works in people-focused access to justice may require rethinking the traditional approaches to delivering the full continuum of legal and justice services that focus first and foremost on responding to people’s legal and justiciable needs. It requires countries to evaluate the effectiveness of services and measure their impact.
This chapter sets out the principles, criteria, evidence and good practices related to planning and evaluating the delivery of people-centred legal and justice services to foster inclusive growth, sustainable development, and individual and community well-being. The first section provides an overview of the planning of legal and justice service delivery: levels of planning, frameworks for planning, the use of research and the location of services. The second section identifies main criteria for people-centred delivery models: comprehensive and individual needs assessments; seamless referral and service integration; coherence; and effectiveness. The third section reviews issues related to implementing people-centred evaluation of legal and justice services and provides an overview of promising evaluation methodologies. A fourth section sets out key findings. A final concluding section brings together the key findings into a set of common criteria to guide the planning, design and delivery of people-centred legal and justice services.
Planning the delivery of legal and justice services
Multiple planning levels
Most countries have mechanisms in place for the planning of legal and justice services.1 These mechanisms may vary with the level and nature of legal and justice infrastructure and also in relation to the constitutional frameworks of each country. In most countries, this means that planning will take place across multiple levels and involve a range of entities. For example, in large federal jurisdictions, it is likely that planning will need to occur at multiple levels, including the:
-
national/strategic level
-
state/jurisdictional level
-
regional and/or local (tactical) levels.
Each level of governance is likely to have both shared and unique requirements in terms of legal needs assessments data and information.
While there are significant differences across countries, the national/strategic level’s focus tends to generally be the setting of priorities (for example, priority groups for assistance), securing cross-sectoral and cross-government collaboration (to allow for legal and other human services to work collaboratively to address the holistic needs of citizens), and the allocation of resources on a needs basis to the state jurisdictional levels. To carry out this task, national/strategic level policymakers and planners would benefit from legal needs assessments of the national and jurisdictional level, as well as service delivery administrative data.
Depending on the size (geographically and demographically) of the state or province, policymakers and planners at the state/jurisdictional level may well focus in a similar way to those at the national level. That is, they will focus on setting priorities for addressing legal needs within their jurisdiction, securing and facilitating cross-sectoral collaboration, and then allocation of resources by region and/or other criteria to facilitate the detailed planning the delivery of services at a more appropriate level. To carry out this work, jurisdictional level policymakers will use appropriate legal needs assessment data and service delivery data at least down to a regional level.
Except in very large jurisdictions (where it may be necessary to have regional and local levels of planning), it is likely that regional and local planning, which would include the planning of individual legal and justice services, will be merged. At this level, service providers and planners will be primarily focused on detailed planning within their jurisdiction. This will include planning of individual services delivery to priority clients. It is necessary to consider the context of planning in collaboration with other local services to ensure their coordination and integration as to cover as many legal needs as possible. Overall, planning may ideally occur in a holistic way, featuring key criteria for the delivery of people-focused legal and justice services.
As discussed in Chapter 3, to carry out this work, service providers and planners will gain from legal needs assessment data approximated down to the lowest geographic entity level at which legal service administrative data is also obtained and can be compared. In addition, the local/tactical level is where local information and other legal and non-legal datasets, as well as other complementary and supplementary legal needs assessments, will be important.
Framework for planning service delivery
The development of people-centred legal and justice services occurs in the context of existing socio-economic, geographic, political and service environments, with their opportunities and constraints in terms of resources and infrastructure (Coumarelos et al., 2015[1]). These pre-existing environments can pose challenges to planning, including historic fragmentation of accountability, policy, funding and delivery. This fragmentation is an understandable product of the incremental evolution of legal and justice policy and service delivery, particularly where there are multiple levels of government responsibility, multi-portfolio government responsibilities, institutional independence within the justice sector, etc.
The planning of service delivery, therefore, occurs within pre-existing strategic frameworks and national priorities. In addition, limited funding can impose considerable constraints on planning and improving legal services. Thus, sometimes, change may only be possible through reconfiguring existing resources and working methodologies (e.g. via service innovation, collaboration or efficiency).
Source: Based on Pleasence, P. et al. (2014[2]), Reshaping Legal Assistance Services: Building on the Evidence Base, Law and Justice Foundation of New South Wales, Sydney.
Any consideration of priorities and approaches in the delivery of people-centred legal and justice services must be undertaken within the context of each country’s particular circumstances. Each country will have specific geographic, demographic and economic factors influencing its justice policies. They each have existing service and infrastructure frameworks, which, in turn, will be shaped by national priorities and approaches to access to justice. Research and data collection and monitoring processes should accord with national (and state) priorities and objectives. Justice policies and approaches can also be attuned to the broader national socio-economic priorities for inclusive growth and sustainable development. The results of local legal needs assessments can also be woven into these broader frameworks.
For example, differing national priorities may manifest to varying degrees in how competing access to justice considerations are weighed. Prioritisation may be particularly important in the context of an expansive understanding of legal needs within the framework of inclusive growth and sustainable development. Traditionally, legal and justice services have focused on the resolution of legal problems through the formal justice system (courts and administrative tribunals). The people-focused recognition of the importance of everyday legal problems and their impact on socio-economic status and health leads to a more extensive, encompassing concept of access to justice. One metaphor is the iceberg: the 10% above the water reflects the approximately 7% to 10% of people who make use of the formal justice system to resolve their problems but 90% of the problems lie below the water line. Delivering services to meet these needs can be challenging especially in the context of inadequate resources and static budgets. Examples of competing considerations and national priorities are set out here to illustrate this point.
One central choice in the context of prioritising legal and justice services is providing assistance to the majority of the population (or a large proportion of it) or to alternatively focus on providing services to the most in need of public assistance. Legal and justice processes and concepts are often difficult and alienating for much of the population. Targeting “light” services to provide a minimum effective level of assistance (information, self-help guides and advice) can address the legal needs of many highly capable citizens. On the other hand, targeting services at the most disadvantaged and the most in need is more likely to require a greater level of investment per client as “light” services will often not be sufficient to meet their needs (Pleasence et al., 2014[2]; McDonald and Wei, 2016[3]).
If targeting those “most in need”, different countries will likely have different target groups identified for priority attention and support. For example, in France, the contemporary legal aid system has two components: i) accès au droit, which gives all low-income earners the necessary resources to have equal access to legal information; and ii) aide juridictionnelle, which co-ordinates funding public legal representation to those who fall below a certain financial threshold (Pro Bono Institute, 2016[4]). The essential criterion for receiving aide juridictionnelle is financial need. The system distinguishes between full and partial aid, depending on the claimant’s financial situation. If full legal aid is granted, it will cover all the costs of the proceedings, including the fees paid directly to the attorneys or other practitioners (bailiff, notary, etc.). People who benefit from specific welfare benefits (namely the Solidarity Allowance for Elderly, the Income of Active Solidarity or the Temporary Waiting Allowance) and victims of particularly serious crimes (torture, rape, etc.) do not need to provide proof of income and can automatically benefit from the aide juridictionnelle.2
In Australia, national priorities for legal assistance services are outlined in the National Partnership Agreement on Legal Assistance Services (NPA). Under the NPA, the national government (the “Commonwealth”) outlines how they distribute funding to Legal Aid Commissions and Community Legal Centres in each individual state and territory. The NPA outlines the priority groups that are to receive this funding. It stipulates that services should be targeted at people experiencing financial disadvantage, and, where appropriate, to also plan and target their services at people falling within certain groups (Council of Australian Governments, 2010[5]). The selection of these groups is informed by the legal needs research, the LAW Survey (Coumarelos et al., 2012[6]) in particular, and includes the following:
-
Children and young people (up to 24 years).
-
Indigenous Australians.
-
Older people (aged over 65 years).
-
People experiencing, or at risk of, family violence.
-
People experiencing, or at risk of, homelessness.
-
People in custody and prisoners.
-
People residing in rural or remote areas.
-
People who are culturally and linguistically diverse.
-
People with a disability or mental illness.
-
People with low education levels.
-
Single parents (Council of Australian Governments, 2010[5])
When targeting those “most in need,” some countries (or states within federal countries) may target particular areas of law for priority attention and support. For example, in Canada, in the province of British Columbia, civil legal aid for representation is available for low-income earners (whose income is below the threshold the Legal Services Society [LSS] defined in its financial guidelines) and whose legal matter falls within one of the following four categories:
-
1. Family law.
-
2. Child apprehension.
-
3. Mental health and prison-related issues.
-
4. Immigration.
Legal services prioritise (sometimes exclusively) criminal legal assistance, or broader assistance to include civil and family law matters. Traditionally, much attention to legal procedure and legal aid focuses on criminal law matters, as these matters have the potential to impact the life and liberty of citizens in the most obvious manner. Legal needs research over at least the last two decades has highlighted the importance of civil and family law legal needs experienced by citizens in many jurisdictions. Nevertheless, a key theme of difference between countries is likely to be the relative priorities and proportions of legal assistance effort devoted to the different areas of law and the different institutions in the legal system.
Another prioritisation question is whether to focus services mainly on the provision of legal advice in court proceedings or taking a broader, holistic approach to the inclusion of out-of-court (and sometimes quasi-legal and non-legal) services to meet particular legal needs. Within a context emphasising criminal law, the focus of legal assistance is often in court-based proceedings and representation. However, as legal needs research has suggested recently, the importance of placing greater attention on civil and family law matters. Moreover, the attention has increasingly begun to be placed on services outside of court contexts. In its 2014 biannual assessment of the efficiency of legal services in Europe, the Council of Europe Commission for the Efficiency of Justice (CEPEJ) began for the first time to seek data from countries in relation to the legal aid services provided out of court (CEPEJ, 2014[7]). Countries also have different infrastructure policies and priorities for providing legal assistance. For example, some countries (such as France) may emphasise a primarily centrally managed and structured legal assistance sector. In some other countries, some legal assistance sectors are left largely to the private and non-government sectors, sometimes with little funding provided by government. For many countries, however, the legal assistance infrastructure includes a mix of:
-
State-funded formal legal aid services.
-
State-supported community legal or quasi-legal services, often principally operated by non-governmental organisations (NGOs).
-
Pro bono and reduced fee services provided in certain circumstances by legal professionals, and by courts and tribunals.
-
National priorities may also be evident in legislative and policy developments. For example, approaches to address key national problems or legal needs priorities can be seen in key legislative and policy work, such as the legislative and policy implementation to combat discrimination in Finland (Committee on the Elimination of Racial Discrimination, 2012[8]).
Using research evidence in planning and delivering legal and justice services
Research evidence is one of a number of factors to consider in planning and establishing the delivery of people-centred legal and justice services at different levels. Table 6.1 describes the types of questions that are likely to be relevant to planning services that the legal needs research evidence can help inform. The figure also presents some other key considerations for planning and provides examples of other data and information relevant to these considerations. It is important to note that the figure does not provide an exhaustive list of questions or information to be considered in planning, and the relevant considerations will vary depending on the level (such as national/regional/local) and type of planning and the specific context.
Importantly, service planning is necessarily shaped by strategic priorities, available resources and the existing political, socio-economic and service environment. Thus, strategic frameworks concerning priorities, funding and the current service environment are key considerations for legal planning. Other potentially useful data sources include data held by local councils and data contained in planning and research reports and toolkits. On-the-ground knowledge held by local legal and human services, as well as other stakeholders, can also contribute usefully to planning legal services.
Matching service delivery to meet need: Where the services are delivered
The goal of the planning process is to provide the most appropriate services based on the specific legal needs and capabilities of the particular individual, and are delivered where and when they are needed. One of the first steps in matching service delivery to meet legal needs is to gain an understanding of where services should be delivered. Also relevant to service planning is geography and existing infrastructure within a jurisdiction. For instance, in some rural areas, distance, poor public transport and fewer on-the-ground services may be relevant considerations in planning services. Conversely, population density and diversity, and complex human service environments may influence planning in urban areas. In addition, there may be a benefit in service providers considering their role within the broader system of available services. What gaps exist in service provision? How can these gaps be filled to successfully meet the needs and capabilities of clients, avoiding doubling up services, and capitalising on relevant opportunities for networking and co-ordination of services?
The mapping of legal needs described in Chapter 4 can be used both for the purposes of identifying and measuring legal needs and as an important input for planning the location of services. Experience shows that attempts to map the delivery of services should be based on existing and available legal services infrastructure.
Service delivery mapping can be carried out at two levels – the macro level and the micro level.
Macro-level mapping could involve plotting the locations of service facilities and identifying the types of services delivered at various points within a region. This could reveal that legal services appropriate for particular target communities are located in the areas of greatest need and/or greatest number of those target groups. However, it may not often have the information about what services are actually being delivered, to which clients in that particular area, as well as which services are provided remotely into that target area by out-of-area services, such as specialist services, etc. For example, Figures Figure 6.2 and Figure 6.3 contain main service locations for legal assistance services in Sydney, Australia, and then New South Wales, plotted against the Need for Legal Assistance Services (NLAS). These maps have the potential to also provide information about part-time and outreach location service points.
Sources: Based on Australian Bureau of Statistics data; Mirrlees-Black, C. and S. A. Williams (2015[9]), Collaborative Planning Resource - Jurisdictional Data: NSW, Law and Justice Foundation of NSW, Sydney.
Sources: Based on Australian Bureau of Statistics data (ABS); Mirrlees-Black, C. and S. A. Williams (2015[9]), Collaborative Planning Resource - Jurisdictional Data: NSW, Law and Justice Foundation of NSW, Sydney.
While the macro level of planning can provide an “at a glance” picture of whether legal services are located in the general proximity of where the greatest legal need is expected, it may not provide sufficient insight as to where the legal services are actually being delivered. In other words, a service being located in a certain region does not necessarily mean that the services are actually targeted to priority clients within that region. It also provides limited information on which citizens within a certain region are receiving services from locally based legal service providers or remotely from jurisdictional-level specialist legal services.
For governments, service planners, legal and justice service providers and others to gain a sound picture of where legal services are being delivered and whether these coincide with where the proportion of levels of legal needs exist, it could be important to undertake micro-planning, which could provide a greater level of detail.
Mapping at the “micro” level may involve the use of detailed administrative data –particularly client tracking data – collected on each individual for each service rendered to that person. This could include demographic profiles and descriptors of the type of legal matter processed by the service provider. Using this data facilitates mapping the types and quantum of services where they are actually delivered.
Administrative data can provide important information in mapping service delivery against need, particularly if it is collected robustly and consistently within services and on the range of services available within a jurisdiction. This data can help identify the location of delivery of services and can be mapped to contrast with where the relevant measure or proxy of legal needs (such as the Australian need for legal assistance services [NLAS]) indicates the level of need or the likely demand for legal assistance services.
The increasing range of data analysis and mapping software tools provide an opportunity for easy matching of service delivery against legal need data. For example, Figure 6.4 compares the rate of legal service provision (per thousand persons per annum, by local government area) in the Newcastle area of Australia, with a recognised measure of disadvantage (Australian Socio-Economic Index for Areas, SEIFA) used as a proxy for legal needs. This map could have been developed based upon a different geographic entity (such as postcode) and using numbers of services provided as opposed to a rate per thousand persons.
Note: Australian Socio-Economic Index for Areas (SEIFA) by Local Government Areas (LGA).
Source: LJF Legal Assistance Services Data Digest: LawAccess NSW, Legal Aid NSW (Advice) and NSW Community Legal Centres 2015.
Alternatively, this data contrast could be represented as in Figure 6.5. In this example, the approach taken has been one that develops a visual indicator (the different sized circles) to indicate the size of the ratio of legal services provided to NLAS capability.
Note: This map does not use actual data. It is an example for illustrative purposes only.
Source: Mirrlees-Black, C. and S. Williams (2016[10]), Collaborative Planning Resource: Presentation to Federation of Community Legal Centres, (Victoria) Inc., Melbourne, Victoria.
While not representing legal or justice sector data; Figure 6.6 contrasts the number of places in early childhood centres (i.e. supply) with the number of infants of employed parents (i.e. demand) in the Saint-Denis area in France. It is recognised that legal need is a less precise quantity and the services provided in the justice sector are much more diverse than the simple indicator of the availability of places in early childhood centres (that is, each individual legal service provided to a client will be different depending upon the client and the particular legal need, and the services provided by each different service will be different in nature as well). The map nevertheless demonstrates the potential usefulness of such a data collection and mapping exercise.
Note : 2009 data
Source: CG93-DEF-PMI, CAF, Insee RP in Observatoire Départemental - Seine Saint Denis (2013[11]), Panorama Cartographique d’Indicateurs Sociaux.
Despite the potential of this approach to mapping, there are also has a number of limitations. For example, the administrative data collected can be quite variable across (and even within) different jurisdictions. Service fragmentation where people use multiple services may overestimate the number of people receiving the service. It may also vary not just in terms of the scope of the collected information but in data quality. As noted earlier, variation in terms, definitions, how legal matters are defined, how services are defined, categorised and provided, may potentially impinge upon the quality and reliability of the data – at least until same standard protocols and practices are adopted. This approach would be most effective if the administrative data capture a sufficiently comprehensive range of variables from the full range of legal and related service providers that are providing legal services to the community. For that to occur, it may require an appropriate system of common data definitions and data collection protocols to ensure that the data collected are consistent and comparable between and within organisations across jurisdictions.
What works in the delivery of legal and justice services?
Determining what works in the delivery of people-centred legal and justice services involves tailoring services to meet the actual needs of individuals, families, communities, and small and medium-sized enterprises (SMEs) based on the evidence gathered through the processes to identify, measure and map legal needs. Tailoring services involve identifying:
-
the most effective means to meet particular legal needs for particular groups in particular locations
-
how to ensure that those services are delivered when and where they are needed by the most efficient means.
The concept of people-centred legal and justice services itself underscores the notion that “one size does not fit all”. The promising practices regarding designing these services set out in Chapter 5 provide a guide, but need to be applied to the particular needs of the particular individual, in their particular circumstances. The challenge in developing workable delivery systems is brought into high relief by what we know about legal needs:
-
the clear inequality in the experience of legal problems
-
legal problems not existing in isolation from other problems
-
the links between this inequality in experienced legal needs and socio-economic disadvantage
-
the links between socio-economic disadvantage and lower personal capability for resolving legal problems.
It is important to bear in mind that there is not a single perfect solution for any particular circumstance but also in the context of what is possible and realistic given the geography, infrastructure and other relevant factors.
The evidence suggests that there will be no single or “ideal” model of service delivery. Rather, strategies are better “made to measure” – appropriately matched to the diverse legal need and capability of clients, and informed by the existing service infrastructure and gaps across the community.
Taking a similar approach as in Chapter 5 on what works for designing services, this section takes a range of access-to-justice guiding criteria and principles, and integrates them with promising practices for the delivery of people-centred legal and justice services.
The vast number of access to justice principles and the survey of promising practices can be synthesised into three people-centred delivery criteria:
-
evidence-based planning
-
collaboration and integration
-
effectiveness.
These delivery criteria build on the general design criteria for what works with a view to ensuring that legal and justice services actually work to meet the needs of individuals, families, communities and SMEs in specific contexts (Figure 6.7).
Evidence-based planning
Access to justice programmes is most successful when they address needs identified by reliable research. Planning for people-centred legal and justice services is often found effective when it is based on knowledge of the client group and the environment. In the past, and still to a large extent today, services tend to be problem-focused (e.g. type of legal issue, area of law) rather than client-focused. In these cases, it can be difficult to detect and address related legal and non-legal problems and may lead to poorer, less sustainable outcomes. People-centred service delivery often implies that service responses “focus on mitigating the total impact of legal problems on a person’s life, rather than considering each legal problem separately”.
Chapters 3 and 4 outlined interesting practices for comprehensive needs assessments, including: legal needs surveys, qualitative needs assessments, collection of data on use and satisfaction, and mapping projects of legal and justice services relative to need. These assessments provide important information about the prevalence of legal problems experienced by specific groups, in particular circumstances, and/or by location. Life cycle methodologies can also be employed to understand patterns of legal problems as they are commonly experienced by people at specific stages of life, for example upon entering the workforce or in old age, e.g. legal health checks tailored to specific age groups (Box 6.1).
A determination of the type and depth of legal justice service that will work best in a given situation requires further individual consideration of the capability of the person and the importance/complexity of the legal problem they face. Country experience shows that service models need to be flexible and nuanced to be responsive to individual client needs and informed by mapping legal needs with the location of services.
Legal empowerment is impaired under some conditions including, for example, resource deficiency, lack of personal skills and/or abilities and knowledge, power misbalances in important relationships and institutional failures (Pleasence et al., 2014[2]). A wide range of factors determines individual legal capacity, and legal capability (and literacy) is often particularly low among disadvantaged groups.3 These factors are not static and there is an important dynamic between legal problems and capabilities. Some life events affect personal capability (e.g. relationship breakdown, victims of domestic violence) and also give rise to legal issues. Legal problems can contribute to illness, which in turn affects capability. This relationship can be seen in specific situations but also in persistent patterns.
It is difficult to overstate the importance of integrating an understanding of personal and legal capability into the design and delivery of legal and justice services. The impact can be summed up as follows:
“Legal capability has the potential to confound socio-legal studies and vex evaluation of legal service provision. Unless differential client legal capability is taken into account, evaluation of legal services may not provide an accurate picture of whether or not certain forms and modes of legal service provision ‘work’. Capability factors may obscure the particular problem and people circumstances affecting appropriateness.” (Pleasence et al., 2014, p. 161[2])
Promising practices include legal capability assessment and screening tools (Marchiori, 2015[12]). Assessments cannot be made solely on a one-time basis, since some clients, even disadvantaged ones, initially appear capable and problems only become noticeable later in the process.
Legal needs research demonstrated that legal and non-legal problems often tend to cluster. These clusters of problems can also be seen as coinciding with specific periods in a person’s life and/or with transitions in a person’s lifetime. Service delivery approaches in other sectors often incorporate life-cycle and life-event methodologies to facilitate and tailor the delivery of services to clients to ensure great accessibility. Some examples of legal services tailored to different needs and legal problems depending on life stage or life events are highlighted below.
Children
The Office of the Children’s Lawyer in Canada, Ontario, represents children under the age of 18 in court cases involving custody and access and child protection, as well as in civil, and estates and trusts cases. The Office of the Children’s Lawyer employs both lawyers and clinicians (social workers), who work on a fee-for-service basis across the province. Clinicians prepare reports for the court and help lawyers who are representing children.
Youth
In 2010, the General Secretariat for Youth of the Greek State launched a programme called “Youth Legal Aid” aiming to provide free legal aid to minors and socially vulnerable target groups of young citizens (up to 30 years old).
Elderly
The Elderly Legal Assistance Programme (ELAP) enables older Americans 60 years of age and older to receive access to lawyers and paralegals who provide representation in non-criminal legal cases, information on many issues of interest to seniors, and community education to prevent harm that may potentially be caused by not having access to legal assistance. ELAP addresses issues regarding income; consumer protection; abuse, neglect and exploitation; healthcare; housing; long-term care and more. Services are provided in every county. The only kinds of cases and issues addressed by this programme are civil matters. The programme provides direct services to those persons 60 years of age and older who are in the greatest social and/or economic need, limited English speaking persons, and rural or low-income minorities.
Sources: General Secretariat for Youth, Greece; Ministry of the Attorney, Ontario, Canada (n.d.[13]), The Office of the Children's Lawyer, https://www.attorneygeneral.jus.gov.on.ca/english/family/ocl; Department of Human Services, Georgia, US (n.d.[14]), Elderly Legal Assistance Program, https://aging.georgia.gov/elderly-legal-assistance-program.
Collaboration and integration
The next step in expanding access to justice is providing integrated and holistic services. This is fundamental to the everyday legal problems’ paradigm of access to justice that views legal problems as aspects of the normal activities of everyday life and, therefore, experiencing legal problems as a human process. In addition, it is well established that legal problems trigger other legal problems and legal problems trigger and are triggered by a range of non-legal problems. Thus, many people, particularly the disadvantaged, experience clusters of interconnected legal and non-legal problems that, like Gordian knots, cannot be disentangled. Indeed, legal needs research demonstrates that people can experience problems that can give rise to needs for a range of services – not only legal and justice services – and that they undertake a wide range of pathways to justice. Legal and justice services, while increasingly aiming to centre on clients, tend to be generally focused more on types of legal problems rather than on clients’ needs, which may mean that people need to assess several services to get effective help.
Fragmentation of legal and justice service providers and limited co-ordination are often reported as other systemic difficulties in some countries. Legal and justice services are often provided by a diverse range of private and public institutions, which sometimes also differ across regions within countries. Some legal and justice service providers operate in a sectoral fashion and fairly autonomously, with different types of legal jurisdiction and eligibility criteria, which requires citizens (and economic actors as appropriate) to deal with them separately (Coumarelos et al., 2012[6]; Forell, McCarron and Schetzer, 2005[15]).4
The bigger the range of separate services and service providers, the greater the difficulty an individual may have in navigating the justice system. Fragmentation and lack of integration create obstacles for individuals seeking assistance, particularly members of vulnerable and disadvantaged groups, and can lead to delays. Research showed that individuals who are turned away from more than one service can suffer from “referral fatigue” and are more likely to leave legal problems unresolved even when they have serious consequences (Buckley, 2010[16]; Coumarelos et al., 2012[6]; Pleasence, 2006[17]). Promising practices include robust seamless referral systems such as a “no wrong number, no wrong door policy” (Pleasence et al., 2014[2]) and active or “warm” referrals between agencies for people who access the justice system through a “side door” (Forell and Gray, 2009[18]; Buckley, 2010[16]; Coumarelos et al., 2012[6]; Pleasence, 2006[17]).
The requirement for integrated services extends beyond legal and justice services to include referrals to other types of human services in order to address an individual’s problems comprehensively and holistically. This flows from a people-centred approach and addresses the reality that many people experience problems that have both legal and other dimensions. Fragmented approaches and limited co-ordination of both public and private (pro bono) legal services, as well as social services, can present a particular challenge for those in especially difficult situations and facing multiple legal problems. This means that these groups may be required to identify separate legal and other service providers for each problem, which may have different eligibility criteria and in itself present a barrier to accessing justice (Pleasence et al., 2014[2]). A related challenge which was identified in a number of studies is linked to long waiting times for legal services and assistance, which may aggravate the legal problem as well as complicate and increase the cost of the resolution process.
In order to overcome these problems, the planning of delivery of people-centred legal and justice services should be guided by the principles of collaboration among service providers and integration of services. As noted in the review of lessons from other sectors, citizen or user-centric services include a focus on the availability and alignment of service delivery channels and optimising the service delivery in each of them. Good practices for facilitating collaboration and integration of services include:
-
Recognising multiple pathways to justice.
-
Facilitating navigation through simple gateways.
-
Providing seamless referral and service integration.
-
Conceptualising service delivery as including non-traditional legal/justice actors and institutions.
-
Promoting co-operative service delivery.
-
Working toward greater coherence within the justice sector.
The provision of a continuum of legal services and a spectrum of justice services increases the availability of options that can be tailored to meet the needs of an individual, dealing with a specific problem or problems, at a given time and in particular circumstances. An effective intake system that is able to diagnose an individual’s situation and legal needs is a critical first step in targeting services. Another important empirical finding from the legal needs research is that people pursue a range of justice pathways (Pleasence, 2016[19]). Differing forms of assistance may be required at various points along these justice pathways – from prevention to post-resolution support. From a public expenditure perspective, integrated services can enjoy competitive and economic advantages (Currie, 2015[20]). Legal assistance websites operating on a jurisdiction-wide basis (national, regional, state, province or municipality) are effective entry points for citizens navigating the justice system (Ribadeneyra, 2012[21]). Legal needs research has demonstrated that many people, particularly vulnerable populations, do not know where to seek legal help and these portals can address this initial barrier (LSC, 2013[22]). Australian national access to justice website, Foolkit5 (stands for free legal toolkit) version 7.0, would be a relevant example. One author concludes: “It is difficult to overestimate the importance of these state-wide websites as foundational building blocks for transformational delivery changes. These sites provide the Internet framework on which to hang new services and new approaches to collaboration. Their authenticity and interface consistency make these sites viable platforms for information and service delivery innovation across the country” (Staudt, 2009[23]).
Joined-up services or one-stop shops are an important form of service integration and seamless referral, best exemplified by problem-solving judicial or non-judicial initiatives. In the Czech Republic, the Probation and Mediation Service also focuses on integration with social services.6 In Chile, a public multiservice network seeks to link the benefits and services provided by public institutions with citizens, through different service channels: points of care throughout the country, an online portal with simple and direct access to information and a call centre delivering orientations on public services.
Some OECD countries have established problem-solving courts. These courts use their authority “to address the underlying problems of defendants, the structural problems of the justice system including jail and prison overcrowding by diverting certain offenders from incarceration and custody, and the social issues of communities”. They apply a holistic view and the “judicial case processing is partnered with treatment providers and community groups to follow up and support victims and offenders alike in order to reduce recidivism” (OECD, 2016[24]).
Most OECD countries have streamlined problem-solving principles in certain fields of criminal justice, i.e. without introducing a fully-fledged court. In this context, the case-management process or track is differentiated according to the circumstances of the case or of the defendants (e.g. juvenile offenders). These diversion schemes embody a problem-solving approach outside of a problem-solving court.
Family-Justice Centres
Family-Justice Centres are initiatives that aim to improve the position of domestic-violence victims. Although not a problem-solving court, they share the ambition of domestic violence courts to help victims and hold offenders accountable. Family-Justice Centres provide co-located, one-stop, multidisciplinary services to victims of family violence. A Family-Justice Centre may “offer comprehensive medical and legal services, counselling to victims and children, links to the court system, as well as access to on-site professionals providing civil legal services, job training and placement assistance, public- benefits assistance, advocacy, and safety planning” (OECD, 2016[24]).
Family-Justice Centres aim to fill the gap between collaboration and service integration by creating collaborative service arrangements leading to enhanced services for clients through specifically created partnerships. This continuum as envisaged in Family-Justice Centres links together various steps, e.g. co-operation, co-ordination and collaboration leading to the creation of integrated, wrap-around services.
Family and domestic violence courts
Domestic violence courts exclusively deal with cases related to intimate abuse. Although they function under the umbrella of problem-solving courts, they are different in the sense that protection of the victim rather than treatment and recovery of the offender has priority. Victims are provided with extensive services such as counselling, shelter and advocacy. Offenders are commonly required to participate in batterers’ intervention programmes and report to the judge on compliance. Yet, most domestic violence courts still emphasise the offenders’ accountability over his or her rehabilitation. Using a “carrot and stick” approach, the offender will receive a suspended sentence (probation) depending on his or her treatment or protective-order compliance. However, domestic violence cases qualify for diversion programmes under certain court models.
In England and Wales (United Kingdom), Specialist Domestic Violence Courts (SDVCs) are led by the public prosecution service. SDVCs’ primary concern is with victims. Lay magistrates preside over court sessions. SDVCs are characterised by multi-agency information sharing, fast-tracking of cases and the offer of comprehensive and immediate victim services. Safe courthouses and facilities are features of this innovation. Review hearings are not part of the court procedure. An exception to that rule is the Croydon court where a defendant’s compliance with community rehabilitation orders is reviewed after three months. Different types of domestic violence court models exist. An important distinction is between dedicated courts and integrated court systems. In an integrated domestic violence court, one judge handles criminal cases related to domestic violence as well as the accompanying civil matters such as custody, visitation, civil protection orders and matrimonial matters. These courts work on the “one family, one judge” principle. In non-integrated courts, the family court judge might be uninformed of a protection order issued during criminal proceedings. Similarly, a criminal court judge might be unaware of relevant family court proceedings relating, for example, to divorce or child-protection. The objectives of an integrated family domestic violence court are: “1) a more holistic and multi-disciplinary approach to family problems; 2) more effective judicial monitoring to increase accountability for offenders and compliance with court orders; 3) improved judicial decision making as a result of the judge having more information about the family; and 4) better access to and co-ordination of support services (i.e. legal and social services) for the victims and children” (OECD, 2016[24]).
Sources: OECD (2016[24]), Northern Ireland (United Kingdom): Implementing Joined-up Governance for a Common Purpose, https://doi.org/10.1787/9789264260016-en; Government of New Zealand, Ministry of Justice.
Family and/or victim-focus models are developed in OECD countries (Box 6.2). In Austria, the White Ring is a victims’ support hotline which is integrated with other service providers.7 In Australia, the Aboriginal Family Violence Prevention and Legal Service Victoria ensures that legal services are integrated with related service provision.8 The Family Violence Court in Victoria, Australia, “employs a ‘one stop shop’ model, which deals with all related matters within the jurisdiction of the Magistrates’ Court” (Productivity Commission, 2013[25]). An evaluation found that “there appear to be considerable benefits, from the perspectives of victims and families, in enabling specialised courts to address issues across different legal frameworks” (Productivity Commission, 2013[25]). The Australian Productivity Commission recommends the extension of this practice so that all family violence courts deal with both criminal proceedings and protection orders in relation to family violence and provide related services to meet legal and related needs. In New Zealand, reducing domestic violence and improving the family court system remains a constant priority (Box 6.3).
A cross-government initiative aims at reducing the impact of family and sexual violence. Within this framework, the Family and Whānau Violence Legislation Bill aims to make changes toward breaking the pattern of family violence and reducing the harm and cost inflicted on those who suffer violence and wider New Zealand society. Changes include increasing access to risk and needs assessments and services, more accurately recording family violence offending in the criminal justice system, enabling the introduction of codes of practice and new information sharing provisions.
Changes include increasing access to risk and needs assessments and services, more accurately recording family violence offending in the criminal justice system, enabling the introduction of codes of practice, and new information sharing provisions.
In addition to modernising the Domestic Violence Act, the Australian government has proposed several targeted improvements, including:
-
Extending the maximum duration of Police Safety Orders from 5 to 10 days to provide victims with more time to put safety arrangements in place.
-
Enhancing the principles so that decision-makers recognise that family violence:
-
includes behaviour that may appear to be minor or trivial when viewed in isolation, but forms a pattern of behaviour that causes cumulative harm, and
-
often includes coercive or controlling behaviour.
-
-
Amending the definition of family violence to explicitly recognise that violence includes dowry-related violence, in order to increase public awareness of the issue.
-
Amending the definition of family relationship to make it clear that the definition can cover a caretaker/carer and a recipient of a care relationship.
-
Requiring assessors and providers to take into account victims’ views, unless it is unsafe or otherwise inappropriate to do so.
These are targeted improvements which will work to modernise the Domestic Violence Act and will ensure that the new Family Violence Act provides a greater focus on victims’ needs and is usable for the wider social and justice sector.
Source: Government of New Zealand, Ministry of Justice.
Joined-up services in Australia “increased collaboration and co-operation between legal assistance providers themselves and other service providers to ensure clients receive “joined up” service provision to address legal and other problems” (Productivity Commission, 2013[25]). There are an almost infinite variety of joined-up initiatives. They can be place-, issue-, client- or education-based, to name just a few (Productivity Commission, 2013[25]). One of the promising practices is to have input from clients in planning stages.
Another area of integrated legal and justice services relates to insolvency and bankruptcy needs (Box 6.4).
Addressing all facets of the problem leads to better and more lasting outcomes. From a public expenditure perspective, integrated services can enjoy competitive and economic advantages (Currie, 2015[20]). For example, Ireland found that it was insufficient to enact innovative personal insolvency legislation in 2013 to address the needs of people in chronic mortgage arrears. Reviews of the effectiveness of the legislation over several years revealed that a purely legal solution was insufficient, even after further law reform measures were undertaken. In 2016, taking a people/client focus, the Ministry of Justice talked to those working with people in mortgage, including the State Courts agency, the Ministry for Social Protection, the national Insolvency Service, the Citizens Information Service, the State Legal Aid Board and the Ministry for Finance. Working together, an effective cross-agency solution was designed to address a cross-agency problem. Today, the Abhaile (“Homeward”) Service provides people with serious mortgage arrears on their homes with a single point of contact in the state, which analyses their financial circumstances, working with them and negotiating solutions with their lenders or referring them to experts for free legal or insolvency advice. This service creates and funds the path to resolution of difficulties for the people concerned and accompanies them until they find the solution. The service also provides duty lawyers to assist mortgage arrears clients at each court session dealing with repossession cases. The money advice advisors are present in the court building on such days and it is now normal for judges to recommend these services to a person with mortgage arrears arriving alone in court.
In Korea, courts also developed a integrated and co-ordinated one-stop shop approach for people facing bankruptcy. The courts took the lead in creating a business agreement with the various service providers and entities involved in bankruptcy cases. The starting point is the person’s needs: what is their final goal in a bankruptcy case? How can the courts and partners help them to achieve this goal? By visiting a court, the individual can consult on bankruptcy/debtor rehabilitation, access education and skills for managing credit and finance, get assistance to develop skills for getting a job, access job referrals and receive advice on family law problems. Korean courts have also taken a problem-solving justice approach in the Juvenile Division to understand and address the underlying needs of youth coming into conflict with the law. The services provided extend to educational and job opportunities.
Sources: Currie, A. (2015[20]), Extending the Reach of Legal Aid: Report on the Pilot Phase of the Legal Health Check-Up Project, https://www.legalhealthcheckup.ca/bundles/legalcheck/pdf/legal-health-check-up-pilot-evaluation.pdf; OECD Policy Roundtables on Equal Access to Justice.
Seamless referral and integrated services require greater co-operation and collaboration among legal and justice service providers and with other human services, including information sharing. Evidence demonstrated that collaboration and co-operation between legal assistance providers themselves, organisations and institutions, breaking down the silos between the justice system and other government services, can contribute to effective service delivery, better outcomes and cost-efficiency (Pleasence et al., 2014[2]).
Whānau Violence Legislation Bill introduces provisions relating to enabling safe and appropriate information sharing to prevent and stop family violence. The new information-sharing provisions allow family violence agencies and social services practitioners to share information for specified and limited purposes relating to family violence.
An agency or practitioner who receives a request to disclose personal information for those purposes must consider disclosing the information but cannot be compelled to do so. The bill will provide immunity from civil, criminal or disciplinary proceedings for sharing information, unless the information is shared in bad faith.
The Ministry of Justice developed a guidance document, which seeks to clarify and simplify the rules around information sharing in an accessible form.
Source: New Zealand Ministry of Justice (n.d.[26]), “Sharing information safely: Guidance on sharing personal information under the Family and Whānau Violence Legislation Bill”, https://consultations.justice.govt.nz/policy/information-sharing-guidance-fv-sector/user_uploads/draft-guidance-fv-information-sharing.pdf.
A recent evaluation of the Cooperative Legal Services Delivery Program of New South Wales found that this central agency is “a high value for money programme and a successful model for increasing networking between legal and non-legal agencies, sharing information, improving referral paths, increasing knowledge of non-legal services about legal issues, and in providing additional legal services to address the gaps for disadvantaged populations” (Ryan and Ray, 2014[27]) (Box 6.6). Indeed, a 2015 OECD report on integrated social services highlighted the value of co-ordinated policies and services for members of vulnerable groups (OECD, 2015[28]).
Enhanced collaboration is not simply a question of institutional arrangements. It begins with an attitude and predisposition that informs legal aid service delivery attuned to client needs: “The challenge is to encourage people to affect change in a fundamental way to ensure that they have appropriate assistance (“no wrong door”) to address their broader needs (e.g. need for mental health services, need for housing)” (Department of Justice Canada, 2014[29]).
Legal and justice services will work to meet people’s needs to the extent they are part of a coherent system. The administration of justice is a process involving a chain of decisions by several actors and, therefore, “the system needs to be addressed as a whole, from the entry point to the end point of the process” (UNDP, 2014[30]). Coherence requires “systems thinking”, a continuum of services that dovetail together, integration of services, collaboration among service providers and an effective triage or referral function that enables people to be directed to the most appropriate destination for resolution, irrespective of how they make contact with the system. In Canada and the United States, access to justice commissions or committees is established to bring more coherence to the administration of justice.9 While focused on the superior courts rather than the broader sector, a similar function is played by the Civil Justice Councils in the United Kingdom as a whole10 and Scotland in particular.11 One specific example is Canada’s Action Committee on Access to Justice in Civil and Family Matters which undertook a multi-year collaborative strategy to promote and measure progress with respect to the Justice Development Goals set out in its 2013 Action Plan.12
The Cooperative Legal Service Delivery (CLSD) programme comprises regionally-based partnerships between legal and related non-legal services that aim to improve access to legal services for disadvantaged people in regional and remote New South Wales (NSW). The CLSD partnerships operate on the premise that disadvantaged people with multiple, complex legal and other needs and low legal capability ideally require integrated or joined-up legal and non-legal services to successfully negotiate barriers to justice. It also works on the theory that better co-ordination and co-operation in the planning and delivery of legal services will enhance the efficient and effective use of scarce resources and improve access to justice for disadvantaged people. The CLSD programme aims for continuous improvements to service delivery through better alignment of planning, programme design and service delivery within and across the legal and related sectors covered by the programme. There are currently 12 CLSD partnerships across regional NSW, each with a funded CLSD Program Regional Coordinator based at a local organisation. The below evaluation findings demonstrate a range of positive impacts of the partnerships and strong support from regional and federal stakeholders.
Increased networking, information sharing and referral pathways between services have been seen as core advantages resulting from CLSD activity in all regions, highlighting an improvement in the ability and capacity of participating agencies to assist disadvantaged clients in their legal needs.
At the regional level, the key benefits included information sharing between partners, improved referral pathways, networking, increased legal knowledge of community service providers, increased access to legal services by disadvantaged people and identification of legal needs and gaps. Central agencies reported valuing the local knowledge gained from CLSD regions and having access to a regional network to enhance their own initiatives.
The CLSD was found to help identify and address the legal needs of disadvantaged populations, which is seen as a raison d’être for the CLSD partnership.
Source: Legal Aid NSW (2012[31]), Report: Evaluation of the Cooperative Legal Services Delivery Program, https://www.legalaid.nsw.gov.au/__data/assets/pdf_file/0020/15176/Evaluation-CLSD-Final-Report-August-2012.pdf.
Central Agent and the Multi-Agency Team
As a part of the 2018 budget, a new Family and Sexual Violence Central Agent was appointed to lead and co-ordinate the family and sexual violence system. The funding provided will cover the preliminary policy work for setting up the position, which will lead to much-needed co-ordination of public-sector and NGO efforts to address domestic and sexual violence.
The work of the Central Agent will follow on from that of the government’s Multi-Agency Team (MAT). The MAT is made up of officials drawn from the Ministry of Justice, Ministry of Social Development, Department of Corrections, Te Puni Kokiri, Ministry for Women and New Zealand Police, and is tasked with driving the family and sexual violence work programme.
Integrated Safety Response pilot
The Integrated Safety Response (ISR) pilot is a part of the family and sexual violence work programme, and is operating on two sites; Christchurch and the Waikato area. The pilot brings together a team of police, Child, Youth and Family Agency (CYF) - now the Ministry for Vulnerable Children, corrections, health, specialist family violence non-governmental organisations and Māori service providers to support victims and their families. During the pilot phase, each incident of family violence reported to theNew Zealand Police and high-risk prison releases will be triaged daily, with a Family Safety Plan developed to address the risks and needs of the family.
Source: New Zealand Ministry of Justice.
The New Zealand Government also established a Centre for Dispute Resolution (GCDR) to provide continuous stewardship and advice to support a systems-based, best practice approach to dispute resolution,13 including the review, design and development of agencies’ dispute resolution thinking, policies and schemes (Box 6.8).
The Government Centre for Dispute Resolution (GCDR) was established to provide leadership and stewardship to support a systems-based, best practice approach to dispute resolution in New Zealand. The GCDR is the lead advisor to government on dispute resolution. It develops best practice dispute resolution guidance as well as various tools and resources, and also assists agencies directly in the review, design and development of their dispute resolution thinking, policies and schemes. The GCDR also serves as an advocate for improvements to the overall government dispute resolution system.
Source: New Zealand Ministry of Justice.
The delivery of people-centred legal and justice services involves a wide range of responsibilities across governments and civil society organisation. Coherence could also require rethinking assumptions about “what is possible” and about how government departments or ministries work together, with other associated governmental bodies and with non-governmental bodies “to soften their boundaries so that they can work together more effectively to solve problems, and work better on their systems for listening to consumers of legal and dispute resolution services”.14 Overcoming policy and service fragmentation will take “systemic vigilance” to guard against blockages, failures and injustices.
Effectiveness
Implementing service change and working towards best practice service delivery involves not only planning and establishing services that reach relevant clients and are appropriately tailored to their legal needs and capabilities, but also integrating ongoing monitoring and evaluation into service provision to inform decision-making. While there is no consensus regarding a precise definition of effective access to justice, a central point is that the services provide meaning assistance, assistance that makes a difference, to the individuals, families, communities and SMEs.
From a people-centric perspective, key measures of success of legal services are the level of satisfaction held by clients or users and the broader outcomes achieved as a result of the service. For example, an evaluation of the Family Dispute Resolution (FDR) in Australia included broad measurements of client satisfaction including the justice service delivered and its outcomes. Indicators include: satisfaction with the process including the information provided and time taken; active participation of parties in resolving their dispute; narrowing of legal issues; better understanding of the relevant issues; settlement arising from the service; and, eventually, the ability to resolve future disputes with another party without outside help. Not only was FDR found to offer a timely and low-cost option to litigation but it was also found to narrow the issues in disputes, to voluntarily participate in the process and determine the outcome and focus on the best interests of the child (CBA, 2013[32]).
The effectiveness of people-centred legal and justice services is a cumulative measure of the justice system which depends upon the fulfilment of the other criteria discussed throughout this report. It is particularly closely linked to the requirement for outcome-focused and fair services. The question of effectiveness also engages broader considerations of priorities and allocation of resources and integrates a people-centred perspective. The Australian Strategic Framework for Access to Justice defines effectiveness: “The interaction of the various elements of the justice system should be designed to deliver the best outcomes for users. Justice initiatives should be considered from a system-wide perspective rather than on an institutional basis”. Effectiveness is promoted through ongoing evaluation including from a client or user perspective, and the employment of evidence-based best practices (Australian Government, 2009[33]).
Effectiveness and outcomes are strongly interrelated: effectiveness is usually defined as meeting the desired outcome. Sound arguments have been presented for measuring effectiveness relative to a broad range of outcomes and impacts that extend well beyond a specific case outcome: “Civil justice research must step back from narrow definitions of effectiveness that are limited to case outcomes and consider the broader, systemic effects of representation on individuals and those around them”.
There is an important opportunity to develop connections between research into the effectiveness of models of legal assistance on individuals and the impact of providing this service on the broader community. Chapter 2 of this report concludes that there is a partial business case but that much more evidence is needed to develop a clear understanding of the interactions between effective legal aid and positive social and economic outcomes at the individual, community and state levels as seen from short-, medium- and long-term perspectives. Countries also pointed to limited evidence about the effects legal problems have on the social, health and economic lives of citizens, both in the short term and over time. The key link between studying the effectiveness of specific models of assistance and the business case for legal aid is the definition of outcomes employed in these studies. Emerging methodologies encompass more thoughtful conceptions of “outcomes” of legal and justice processes but there remain important gaps at both the conceptual and operational levels. Similarly, definitions of effectiveness can be broadened to include measurements of impact beyond the individual to connect legal aid provision to the justice system and broader societal goals of development and social and economic equality.
The effectiveness of legal and justice services is enhanced through an ongoing cycle of planning, implementation and evaluation, with evaluations feeding back into the design and delivery of people-centred services. For example, Portugal’s Mais Proxima justice reform process envisions people-centred change through a cycle of ten interlocking steps: vision/strategy, action plan, monitoring, collaborative working, agile methods, active listening, proof of concepts, pilot projects, evaluation and roll out.
Ensuring effectiveness also calls for regular evaluation of introduced changes and reforms. For example, in New Zealand, in 2014, major changes were made to the family justice system, including requiring mediation before parents could apply to the Family Court and removing lawyers from the early stages of some court proceedings. While these reforms aimed to help people resolve parenting disputes without having to go to court, the Minister of Justice asked an independent panel to examine the changes and consider how they have impacted separating families and their children.15
People-centred evaluation and measurement
Challenges
The main challenge in delivering people-oriented access to justice services is to enhance responsiveness and ensure that provision extends to the “right” mix of services, to the “right” clients, in the “right” areas of law and in the “right” locations and at the “right” time. In order to meet this challenge, we need to know people’s problems, what methods respond to these problems, for what type of client and under what conditions. While impact assessment takes many forms (e.g. piloting, embedding research and analysis into the design and implementation of access to justice policies), we need a sound understanding of which legal and justice services are effective in which circumstances in order to achieve these objectives. This is a challenging task for a number of reasons. Evaluation of the effectiveness of various models of legal assistance is a relatively new field and there is a multitude of approaches to legal service delivery. A lack of common definitions and a common framework means that, to date, there is a relatively low ability to generalise from results. There is no widely accepted definition of the parameters of different types of justice services, “effective” or “meaningful” access to justice and there is no common framework for developing process and outcome indicators that would facilitate comparison of results. Another gap is the lack of attention that is often paid to individual legal capabilities in the assessment of the effectiveness of services. Studies have demonstrated that a sophisticated understanding of legal capability is a critical aspect of developing and delivering responsive legal assistance services (Pleasence et al., 2014[2]). Yet few research methodologies adequately meet this requirement. Furthermore, it can be difficult to control variations in service delivery (e.g. quality of services provided can vary even within a single programme).
Another challenge is a financial one. Evaluation can cost more than the service and the peripheral nature of justice policy means that few resources are dedicated to this purpose. As discussed earlier, administrative data in the justice sector is limited and there is often a lack of comparability of data between different bodies even within the same jurisdiction. There is a need to create more comprehensive data that would facilitate evaluation and make this data available to researchers.
A related point is that because of the fragmentation of the justice sector that exists in many countries, there are overlapping responsibilities and no overall ownership of the system as a whole. Coupled with the diffuse consequences of legal and justice services, these challenges can result in a vicious cycle where, for example, pilot projects are not evaluated or are not scaled up even when evaluations demonstrate that the services are effective in meeting legal needs.
Measuring access to justice
At present, the most efficient instrument to evaluate and measure access to justice in a people-centred approach is legal needs surveys. Many countries have developed very structured methodologies and continual progress is being made on this front. As noted above, administrative data can be used to complement surveys to present a more complete picture of access to justice in a given country. The United States Working Group on Access to Justice Indicators and Data Collection is working to improve the statistical infrastructure in the United States that measures access to justice (Box 6.9).
The EU Justice Scoreboard helps its member country measure the quality of justice systems through meta-indicators including access to justice, adequate resources, effective assessment tools and appropriate standards. The current quantitative measurements for access to justice incorporate: online information about the justice system; income threshold for legal aid in a consumer case; court fee to start a judicial proceeding in a commercial case; and a number of complaints to the online dispute resolution mechanism.
EU member states and courts are continuing to develop a more citizen-oriented approach to justice indicators through the Court Quality Framework Design (CQFD) project designed to fill this gap, measuring what works from a people-centred perspective. CQFD is creating a new tool to support the design of local solutions to improve the quality of day-to-day justice. This tool will empower heads of courts by providing them with keys to designing people-centred services and will help justice policymakers initiate a peer dialogue with courts of their choice in Europe with a positive record of quality services. The methodology is based on experience and evidence of what works: pilot courts are being studied in order to look for core standards of best practices that can be shared with others.
Source: OECD Roundtables on Access to Justice.
Measuring the service effectiveness
There are several approaches to measuring the effectiveness of legal services and access to justice, including but not limited to: programme evaluation of specific legal assistance or justice services, justice pathway evaluation, cost-benefit and cost-effectiveness analysis and outcome or impact evaluations. In each case, a range of methodologies can be employed to measure indicators of effectiveness (Digiusto, 2012[34]). Five main questions are posed in evaluation studies: is it a good model of service in theory? Is it being implemented in the world? Does the model accomplish its goal? Does the model improve the life of the individual? Does the model reduce poverty/inequality in society? Box 6.6 sets out key consideration in the assessment of the effectiveness of legal and justice services.
Evaluation plays many roles. A four-stage taxonomy has proved useful in the legal services domain, distinguishing between evaluation centred on service planning, establishment, operation and outcomes, each of which is associated with distinct evaluation questions and forms of evaluation.
The first stage is associated with needs assessment and gap analysis for example. The second and third stages are associated with, for example, process evaluation and monitoring. The fourth stage concerns whether services are effective, efficient, cost-effective and cost-beneficial.
In general, the desired outcomes of public legal assistance services relate to one or more of usage (e.g. by target clients or in relation to priority issues), citizen capability (including empowerment), “case” outcomes; procedural impact; and wider (e.g. socio-economic) impact.
Evaluation is applied social science. Methods follow questions. The most critical task of evaluation is articulating evaluation purpose and questions.
A simple categorisation of research designs includes experimental, quasi-experimental, cross-sectional, longitudinal, case study and comparative. Data sources include interviews/self-completion questionnaires, focus groups, administrative data, content, observation and participation.
While process evaluation is often associated with qualitative approaches, it can also follow quantitative approaches. In contrast, outcome evaluation is often associated with quantitative approaches, but can also follow qualitative approaches. However, qualitative research falls short when quantification of impact is required.
A significant number of studies have looked to quantify the impact of legal services on outcomes but there have been few randomised experiments.
The benefit of randomised experiments over other outcome evaluation designs is their high level of interval validity, an important element in the context of causation. However, randomised experiments are often infeasible and can be too narrowly focused in the context of scarce resources.
Not all approaches to “outcome” evaluation rely on establishing cause and effect. For example, the benefits of services can be explored through inquiry into the value placed on them.
Challenges to impact centred cost-benefit analysis include the availability of measures and the diffuse nature of the wider impact.
With limited resources available for evaluation, they must be of good quality, useful and sustainable. At a minimum, they should address the right questions, fully describe the nature of the evaluation, adopt appropriate social research methods, engage (and respect) stakeholders, be impartial, be shared and adhere to norms and standards.
The quality, utility and sustainability of evaluation in the legal services sphere does not only depend on efforts within individual exercises. Evaluation requires an enabling environment.
There is an important role for meta-evaluation and other synthesis methods to consolidate learning and explore emerging themes.
Prior to commencing evaluation exercises, prior research should be reviewed.
In drawing together examples of legal service evaluations, evaluations could be indexed according to location of service, nature of service, commissioning organisation, evaluation lead, purpose of evaluation, evaluation questions, research methods, protocols followed, methodological recommendations, stakeholder engagement, use made of evaluation and publication.
Source: Pleasence, P. (2016[19]), ‘Legal Needs’ and Legal Needs Surveys: A Background Paper, OSJI.
Programme evaluation
Governments and public legal service providers employ a range of methodologies to evaluate the quality and impact of the legal services they provide. Traditionally these evaluations have focused on experience in service delivery and some measurement of client satisfaction (Buckley, 2013[35]).
Studies show that clients receiving legal assistance services have a high level of satisfaction with the services received but it is more difficult to gauge the effects on case outcomes (Engler, 2010[36]). Researchers have noted the danger of focusing narrowly on client satisfaction as an indication of the effectiveness of service (Curran, 2012[37]) since “[c]lient viewpoints, while important, tell us very little about the key issues for quality, such as correct advice and appropriate help” (Moorhead, 2003[38]). Nor do client viewpoints tell us whether litigants receiving help actually were able to perform the tasks required of them or whether they prevailed in their cases as a result.16 Curran urges a shift to “client feedback,” which can explore the level of “quality” of the service (Curran, 2012[37]). For example, the Australian Strategic Framework Report expressed caution in over-emphasising the extent that general information can be adequate assistance. Results have to be closely scrutinised.
It would be wrong to conclude on the basis of the strong rating for the Internet (as well as other sources of information) that direct assistance is less effective. This is because when more than one advisor is used, the Internet was rated as the most useful sources in 34.6% of cases (still a strong outcome) whereas a private lawyer was rated as the most useful in 72.7% of cases – significantly higher than all other sources (Australian Government, 2009[33]).
More recently, methodologies have expanded to the quality of procedure and of outcomes experienced by individuals who receive assistance. It is key to have “[…] an eye toward measuring the effectiveness of the assistance received in terms of case outcome, rather than accepting the mere facts of assistance as sufficient” (Engler, 2010[39]). Particularly in the context of limited resources and unmet demand, “[M]easuring effectiveness is about measuring whether or not the programme achieved its aim or intended outcome” (Digiusto, 2012[34]). In rare cases, studies investigate the longer-term impact of legal assistance such as “whether clients gained new confidence, knowledge or reassurance as a result of the legal services they received” (Focus Consultants, 2012[40]).
Procedure and outcomes can also be assessed from a systemic perspective, for example, a reduction in the length of a hearing or achievement of a broader policy goal such as reduction of poverty. Ideally, evaluation provides the feedback needed to ensure both quality of service and as a feedback loop facilitating systemic reform where needed.
A wide range of methodologies is utilised in measuring the effectiveness of legal assistance services including:
-
interviews
-
focus groups
-
surveys with users of services and justice system stakeholders
-
online evaluations
-
feedback cards distributed along with resource and analysis of returned cards
-
tracking distribution and usage of and traffic patterns for online services
-
experimental design methodologies (including randomisation).
In her in-depth comparative analysis of methodologies, Curran concludes that the most useful methodologies are:
-
The “Most Significant Change Technique (TMSCT)” described as: “[...] a participatory form of evaluation that uses a story approach to explore the impact of a service or program. This challenges the conventional evaluation so commonly used and discussed above with its focus on predefined indicators. TMSCT is a process that ensures that the many stakeholders, including client, community, service providers and government are involved in deciding on what kind of impact and change is important and records and reflects on these. Case studies are often used in this approach” [note this technique is more often referred to by the acronym MSCT].
-
“Survey Research” which “involves the use of questionnaires and structured interviews to collect quantitative data at a single point in time which is examined to identify patterns and relationships”.
-
The “Case Design Approach” which “involves a range of qualitative and quantitative evaluation methods including interviews, questionnaires, participant observation (difficult in legal assistance research due to client professional privilege issues) and document analysis. It focuses on a very in-depth analysis of a case or service program and examines these to develop in-depth understandings rather than causal explanations. Such approaches reveal particularity and diversity and are good at enabling greater sense to be made of a situation that might not be evident with a more superficial study”.
-
“Participatory action research” which “involves the evaluator working with the client/service/community to identify research questions, to collect the data and analyse it. This approach sits within a critical theory which is designed to contribute to learnings and empower people in the process and into the future by learning about their situation and working with the service/community/client to work out ways of making improvements. It uses ‘quality criteria’ which involves reliability, measurement validity, credibility, transferability, dependability and confirmability. It requires that the participation be authentic and ensures that the cause and effect relationship holds” (Curran, 2012[41]).
In some cases, services are evaluated on a broader basis that allows for a comparison of a variety of providers. For example, Community Legal Education Ontario undertook an in-depth assessment of effective formats and delivery channels for reaching low-income and disadvantaged communities in Ontario with information about their legal rights (CLEO, 2015[42]). The report noted the many barriers people experience in effectively accessing and utilising legal information and concluded that the very best approach is a multi-layered response tailored to individual needs. One US study carried out an in-depth evaluation and assessment of the outcomes of five hotline information services over a three-year period (Pearson and Davis, 2002[43]). This study was completed in 2002 but is considered to be have been “pioneering” in the field and to have “withstood the test of time” in terms of identifying key issues (Smith, 2013[44]). Smith undertook a meta-analysis of seven studies and draws a number of tentative conclusions about hotlines as a means of providing legal advice. He highlights the fact that hotlines vary substantially and “cannot really be measured against each other”. He also suggests that studies show: “The benefit of the hotline expands with the depth of services offered. The best results are obtained when the hotline is the ‘front end’ of a system that can extend through assistance to full representation”.
One of the most thorough evaluation plans established by a service provider was used by the UK Legal Services Research Centre (LSRC) at the time Community Legal Advice Centres (CLACs) and Community Legal Advice Networks (CLANs) were introduced in England and Wales in 2009. This effort was designed to “constitute a new approach to the way that civil legal and advice services are funded, purchased and delivered” in that country. The LSRC developed a research framework based on four key areas against which the services were to be evaluated: accessibility, seamlessness, integration and tailored services. The published studies provide a comprehensive review of the initial period of these new service delivery models.
Methodological approaches included:
-
Observation of advice sessions.
-
Separate interviews conducted with both clients and advisors immediately following the advice sessions, and through in-depth, follow-up interviews two weeks after observations.
-
Fieldwork in some of the centres.
-
Face-to-face footfall survey of clients in reception areas of the five operating CLACs and their outreach locations: Questionnaires were administered to clients who visited the services during a one-week period, thereby providing a snapshot of CLAC users.
-
The development of a reporting tool through a collaborative approach with stakeholders and the implementation and analysis of the reporting tool.
As part of the reporting process, the LSRC developed a set of outcome criteria for various types of legal matters against which results were to be measured (an overview is presented in Chapter 5). Other examples of the development of substantive standards for the evaluation of outcomes exist in Australia and the United States.
Justice pathway evaluation
Another people-centred approach to evaluation focuses on assessing an individual’s experience across a justice pathway rather than from the perspective of a specific programme or services. A quantitative assessment of justice pathways is proposed by the “Measuring Access to Justice” (MA2J) project developed by Tilburg University, the Tilburg Institute for Interdisciplinary Studies of Civil Law and Conflict Resolution Systems (TISCO) and the Hague Institute for the Innovation of Law (HiiL). MA2J developed a research methodology aimed at measuring access to justice through the perceptions and attitudes of people who have travelled a “path to justice”. This technique is based on using a legal needs survey to understand the performance of different institutions.
The “path” approach’s units of measurement are individuals who had a legal problem and acted to solve it with the means of state or non-state intervention. The definition of the path to justice is simple: a commonly applied process that users address to cope with their legal problem. Three major “pillars” of the experiences of the users are measured: the costs of justice, the quality of the procedure and the quality of the outcome. Each of the three pillars is modelled as a multi-faceted indicator consisting of sub-indicators. The individual scores of the sub-indicators form the scores of the three cost, procedure and outcome indicators. Eventually, the MA2J project intends to aggregate the information on costs, quality of the procedure and quality of the outcome into one composite figure, the Access to Justice Index. The goal is for this index to provide focused information about the measured paths to justice (Gramatikov, Barendrecht and Verdonschot, 2011[45]).
The indicators and sub-indicators have been developed over a period of years during which conceptual and methodological problems were addressed. More than twelve pilot applications of the methodology were conducted to test, validate and refine the measurement instruments and the overall methodological framework. While acknowledging the findings from civil legal needs research that many people who have a problem for which there is a legal solution do not take steps to address that problem, MA2J focus on measuring justice needs “from the moment when a person first takes a step toward resolving the problem”. This “demand-oriented” approach focuses on the most urgent legal problems experienced by citizens. MA2J identifies 12 categories of legal problems “that appear to be urgent in many, if not most, legal systems and locations”.
Measurement is undertaken from the perspective of the “users” based on their experience on the path of justice: “After completing a path to justice, people think about the costs incurred, the procedure and the outcome. In its essence, the methodology asks the users of justice to reflect on their experiences and formulate a quantitative account of the particular path to justice” (Gramatikov, Barendrecht and Verdonschot, 2011[45]). The quality of procedures and outcomes are measured not against an objective characterisation of justice, but against the user’s perception of procedure and with the outcome being the perceived result of the procedure and the perceived quality of the received outcome. Data is collected through surveys of random samples of persons who have engaged in and used the services within various paths to justice.
Costs of justice, quality of the procedure and quality of the outcome are multi-faceted categories and MA2J measures their different dimensions with one or more variables. For example, the costs of the procedure are defined as the resources the user would need to travel from the beginning to the end of a path to justice. Within this indicator, a set of sub-indicators reflect different types of procedural costs: out-of-pocket expenses, time, and other opportunity and intangible costs (stress, emotions, etc.) (OECD-OSF, 2016[46]). The three indicators will be drawn together into composite access to justice index based on a rating of the importance of the indicators and sub-indicators (with the more important dimensions given more weight within the index) (Gramatikov and Laxminarayan, 2008[47]).
MA2J is expected to assist in measuring, defining and operationalising the paths of justice that meet the demands for resolution of urgent legal problems. The researchers foresee that the project “will provide vital data for the accessibility of the legal processes put in place as well as the subjective perceptions of the users” (Gramatikov, Barendrecht and Verdonschot, 2011[45]). Comparisons will also be possible with alternative paths to justice, between different users or groups of users, or with baseline data collected through a separate process. It will be possible to measure changes in the paths to justice and along the three main indicators over time. The information gathered will also provide feedback to providers of justice system services, identify bottlenecks and may make it possible to predict use of different paths17.
The Evolving Legal Services Research Project (ELSRP) initiated by Community Legal Education Ontario is utilising a similar research design method. It is an innovative longitudinal research project and examines the effectiveness of public legal education and information (PLEI) in helping low- and modest-income people address their legal problems. One of the main differences between the MA2J methodology and that of the ELSRP is that the Canadian project is measuring process and outcome issues against both a subjective (perspective of the individual) and objective (integrating legal norms) standard.18
Finally, Abel recommends a three-prong approach for employing process mapping and analysis to evaluate the effectiveness of the spectrum of access to justice interventions. These are: i) identifying the tasks required; ii) identifying the obstacles from completing the required tasks; and iii) identifying the access to justice interventions that can overcome the obstacles. She too emphasises that this analysis must extend to considering outcomes and whether access was meaningful and effective. As she notes, while several evaluations have found that self-help interventions can improve the ability of unrepresented litigants to perform certain tasks, they do not show that they “enable them to perform those tasks at a level sufficient to enable the court to reach fair and accurate decisions” (Abel, 2010[48]).
Cost-effectiveness, cost-benefit and economic analysis
Meeting the legal and justice services needs of various groups of the population through the delivery of people-focused services in the context of fiscal constraints is a challenging task. It requires an understanding of the relative costs of different strategies to achieving the desired outcome for specific groups of the population (for specific issues) to ensure that they generate value for money. There are several economic evaluation methods (such as cost-effectiveness and cost-benefit analysis) that can be used for this purpose (Box 6.11).
Cost-effectiveness analysis compares the relative costs and the outcomes (effects) of two or more interventions that are potentially competing for the same resources. It is used to determine which intervention achieves the most favourable outcomes or the greatest quantity of particular outcomes at a given cost.
Cost-benefit analysis (CBA) involves comparing the total expected cost of two or more options (i.e. including any negative consequences) against the total expected net benefits to see whether the benefits outweigh the costs and by how much. CBA generally requires that all outcomes are measured in (or converted into) monetary terms to make them comparable to each other and to the costs. CBA can be used to examine which of two or more interventions achieves a given level of benefit at the lowest cost and it can provide a basis for deciding how best to allocate a budget in order to achieve the maximum “overall benefit”.
Sources: Digiusto, E. (2012[34]), “Effectiveness of public legal assistance services”, Justice Issues, Vol. 16, Law and Justice Foundation of New South Wales; Bamberger, M., J. Rugh and L. Mabry (2012[49]), Real World Evaluation: Working Under Budget, Time, Data, and Political Constraints, 2nd Ed., Sage, USA.
In addition, economic impact analysis techniques that aim to understand the general economic value of investments in legal and justice services have also been developed and employed. There are several models of economic impact evaluation methodologies, including the Perryman Group’s input-output assessment model, which use a variety of data (from surveys, industry information and other sources) to describe the various goods and services required to produce another good/service. Examples of these studies were discussed in Chapter 2 as they form the basis for the business case for people-centred legal and justice services (The Perryman Group, 2013[50]).
Outcome or impact evaluations
A promising emerging practice is an evaluation of services based on outcomes or impacts – a critical development to further our understanding of “what works”. To date, most of these empirical impact studies are measuring the correlation or contribution of a service to a specific end, rather than causation per se. Nevertheless, they provide important insights into the design and delivery of people-centred legal and justice services.
Outcome measurement requires legal and justice service providers to follow up with clients to find out if assistance was helpful and ask about the results. A growing number of service providers carry out client satisfaction surveys19 which can be seen as an important first step in this direction. One American legal aid pilot is investigating the effects of full civil legal assistance on women experiencing domestic violence, in cases of divorce, custody, child support and civil protective orders (Houseman, 2013[51]). The study is measuring whether receiving legal representation enhanced client safety, psychological well-being, positive functioning and longer-term economic self-sufficiency.
The measure of outcome is by definition a relative one, “the likelihood of obtaining a better result” with or without the legal and justice services (Steinberg, 2011[52]). Further, it is difficult to define favourable outcomes since clients can have more than one goal; “legal outcomes tell only part of the overall story” (Greiner and Pattanayak, 2011[53]). Albiston and Sandefur present a strong argument for measuring effectiveness relative to a broad range of outcomes and impacts extending well beyond a specific case outcome: “Civil justice research must step back from narrow definitions of effectiveness that are limited to case outcomes and consider the broader, systemic effects of representation on individuals and those around them” (2013[54]).
One concern is how to address key contextual variables such as individual legal capability, types of legal problems/legal processes or fora, and quality and responsiveness of the service(s). A partial list of specific factors includes: characteristics of the individual/group experiencing the legal problem; personal and legal capability; relationship with other party/parties; characteristics of the legal problem; type(s) of dispute resolution process; complexity of tasks required by the process; type(s) of legal information and assistance available; and mechanisms for provision of support and assistance including diagnosis, triage and referral systems. Clarity on these types of factors is central to the generation of useful data and the ability to compare and generalise results.
Many US studies have investigated the impact of counsel in various settings that handle civil cases. These reports consistently show that representation is a significant variable affecting a claimant’s chances for success in a vast range of legal proceedings including eviction, custody and debt collection cases, as well as administrative proceedings (Greiner and Pattanayak, 2011[53]). The type of proceeding involved has varied from uncontested to claims adjustment, mediation, arbitration, various types of administrative adjudications, and court proceedings (including specialised courts of limited jurisdiction) (Engler, 2010[36]). A recent California study carried out an in-depth evaluation of partial legal services in housing-related cases (Steinberg, 2011[52]). The outcomes assessed in this study were both procedural and substantive in nature. The findings indicate was that the unbundled legal services programme was successful in furthering procedural justice, but that its impact on substantive case outcomes was quite limited.
Until recently, the vast majority of these studies were observational in nature, that is the research design was to compare outcomes in cases where the individual was represented by counsel to similar cases where the individual was unrepresented. This approach has been the subject of a lively debate among scholars.
There are a number of recent and ongoing US projects carrying out what many consider more empirically rigorous randomised evaluations of the need for legal representation, in which participants are randomly selected to receive a type of legal assistance service and whose experience and outcome are compared to a cohort which did not receive the service. Concerns are also raised about the ethical and practical limitations of randomised studies and the limitations of their utility in providing a meaningful evaluation of civil legal services and policy formulation (Greiner and Pattanayak, 2011[53]; Curran, 2012[37]; Albiston and Sandefur, 2013[54]).
The Boston Bar Association’s Civil Gideon Task Force20 and the Shriver Pilot Projects in California21 have established multi-year pilot projects engaging a range of research methods to evaluate the experience of persons who have full legal representation to those who receive more limited forms of assistance in a variety of court-based processes. Methodologies include analysis of randomised studies, assessment of the court dockets, efforts to follow litigants after the period of the study, and interviews with judges, advocates and other personnel involved. These evaluation methodologies were developed through a participatory process engaging practitioners, policymakers, experts and other stakeholders. This multi-pronged methodological approach with its mix of quantitative and qualitative research methods is considered to have strong potential for evaluating the effectiveness of legal services (Greiner, Pattanayak and Hennessy, 2012[55]). Early results from the Boston project conclude that people who receive full legal representation fare much better in court-based processes, both in terms of legal and socio-economic outcomes.
Longer term objectives for outcome assessment would be to evaluate the link between people-centred legal and justice services with outcomes that facilitate inclusive growth and sustainable development goals. At present, this remains a difficult task. The OECD could play an important role in facilitating outcome-based methodologies measuring the impact of people-centred legal and justice services on inclusive growth over a longer time horizon.
Toward a common methodology
There are a number of lessons to be derived from experience designing access to justice research programmes and that can be drawn together to establish good practices for people-centred evaluation and measurement. For example, with several decades of experience with legal needs surveys, best practices have now been identified through OECD-OSF projects (OECD-OSJI, 2019[56]). Another insight is that research that utilises probability samples is likely to be more useful than random samples because legal needs tend to be concentrated within specific groups rather than evenly spread across an entire jurisdiction.
Good practices can also be shared concerning common pitfalls in designing research programmes in this sector and how to overcome them. For example, in studying the efficacy of specific services, there is a tendency to focus on impact and to neglect to ask about use and the competency and quality of the service. Similarly, in reviewing the sustainability of a measure and whether it is a good candidate for scaling up and lasting, we may study costs and benefits (return on investment) but rarely consider the perceived value by integral stakeholders.
Despite the progress made, there is much work to be done: many questions, audiences, methods and examples increase the difficulties faced by countries, especially in the context of resource constraints. The huge range of potential approaches underscore the importance of co-ordination and a common framework can foster best practice, enable findings to be synthesised and lessons to be communicated. The twin goals of survey evaluation and research are to reduce the current data deficit and to create actionable intelligence. Two of the main obstacles to facing these goals are resource constraints and political challenges.
These obstacles could be overcome, and our capacity for people-centred evaluation and measurement advanced through co-operation and the development of a common methodological framework. Sandefur has proposed that knowledge resources could be leveraged through a concerted effort to focus on four key ingredients of equal access to justice research agenda:22
-
Common core of research questions so that each is asking about things we most want to understand.
-
Short menu of accessible measures so that studying it is manageable for many jurisdictions.
-
A design framework for the things studied so that we know what other things we are learning about at the same time (generalising to other objects).
-
A context framework for research sites so that we know in what other contexts we might expect similar results (generalising to other contexts).
While the people-centric focus is on the now well-understood concept of legal need, understanding what works also requires taking into account the contextual framework within which service delivery takes place. Elements of a contextual framework could include:
-
income level
-
strength of legal monopoly
-
literacy level
-
social diversity
-
rural vs. urban
-
density of auxiliary (human, social) services
-
source of services (largely government – a little government plus civil society)
-
co-ordination of the justice sector (fragmented – centralised).
The adoption of a shared contextual framework would enable key findings to be synthesised and lessons communicated and increase the comparability of results across locales and jurisdictions. A common methodology for people-centred evaluation and measurement could include prioritising evaluation of the more strategic impact of services and developing shared realistic standards for evaluation programmes.
Key findings
-
Figuring out what works in people-focused access to justice may require rethinking the traditional approaches to delivering the full continuum of legal and justice services that focus first and foremost on responding to people’s legal and justiciable needs. It requires countries to evaluate the effectiveness of services and measure their impact.
-
The goal of the planning process is to provide the most appropriate services based on the particular legal need and on the capabilities of the particular individual and are delivered where and when they are needed. Research evidence and mapping of existing services are important inputs.
-
One central choice in the context of prioritising legal and justice services is providing assistance to the majority of the population (or a large proportion of it) or to alternatively focus on providing services to the most in need of public assistance.
-
The evidence suggests that there will be no single or “ideal” model of service delivery. Rather, strategies are better “made to measure” – appropriately matched to the diverse legal needs and capability of clients, and informed by the existing service infrastructure and gaps across the community.
-
Three people-centred delivery criteria can assist in matching service to needs: evidence-based planning, collaboration and integration and effectiveness.
-
A determination of the type and depth of legal justice service that will work best in a given situation must be evidence-based: it requires individual consideration of the capability of the person and the importance/complexity of the legal problem they face. Promising practices, in addition to legal needs surveys, include legal capability assessment and screening tools.
-
Fragmentation of legal and justice service providers and limited co-ordination are often reported as other systemic difficulties in some countries. Legal and justice services are often provided by a diverse range of private and public institutions, which sometimes also differ across regions within countries. This fragmentation can be overcome by collaboration between service providers and integration of services.
-
There is no consensus regarding a precise definition of effective access to justice, however, a central point is that the services provide meaningful assistance –assistance that makes a difference – to individuals, families, communities and SMEs.
-
From a people-centric perspective, key measures of success of legal services are the level of satisfaction held by clients or users and the broader outcomes achieved as a result of the service. Effectiveness and outcomes are strongly interrelated: effectiveness is usually defined as meeting the desired outcome. Sound arguments have been presented for measuring effectiveness relative to a broad range of outcomes and impacts that extend well beyond a specific case outcome.
-
Evaluation of the effectiveness of various models of legal assistance is a relatively new field and there is a multitude of approaches to legal service delivery. A lack of common definitions and a common framework means that to date there is a relatively low ability to generalise from results.
-
At present, the most efficient instrument to evaluate and measure access to justice in a people-centred approach is legal needs surveys. Other important methodologies to measure the effectiveness of legal services and access to justice, include but are not limited to: programme evaluation of specific legal assistance or justice services; justice pathway evaluation; cost-benefit and cost-effectiveness analysis; and outcome or impact evaluations.
-
A people-centred evaluation and measurement agenda can be advanced through increased co-operation, co-ordination and the development of a common research framework in order to foster best practice, enable findings to be synthesised and lessons communicated.
Toward common criteria for people-centred legal and justice services
This report weaves together lessons derived from people-centred service delivery, access to justice principles and indicators and promising practices in the delivery of legal and justice services in OECD and partner countries. It provides a useful starting point for the development of common criteria about “what works” to ensure equal access to justice and contribute to inclusive growth, sustainable development, and individual and community well-being. What we know works in identifying, measuring, designing, delivering and evaluating people-centred legal and justice services can be consolidated into a preliminary outline of common criteria.
The key overlapping themes can be grouped along the pathways to justice. A people-focused starting point for the development of services means employing effective evidence-based needs and capabilities assessment, paying attention to the diverse needs of client groups to ensure inclusion and work toward inclusion, and integrating participatory design processes that involve users/clients. A second theme concerns what works with respect to service content. The key service content concepts are accessibility, availability of a continuum of legal services and spectrum of justice services, people-centredness, empowerment, prevention, responsiveness and integrated services. Next, the focus shifts to service quality: appropriateness, responsiveness, timeliness, proactivity, outcome focus and fairness. A final theme is the overall delivery system quality, which includes tailoring based on evidence, seamless referral, no “wrong door” service delivery, intelligent processing, collaboration and integration, coherence, evaluation, assessment from the user/client perspective and effectiveness.
Effective access to legal and justice services presupposes an enabling framework or structure consisting of elements discussed above under the principle of “availability”: legal frameworks setting out rights, entitlements and obligations, an independent judiciary and legal profession, access to justice institutions (formal and informal), all of which are free of corruption. This enabling framework defines “the space within which citizens and the state can negotiate access to justice and justice outcomes” (Marchiori, 2015[12]). At the same time, access to justice principles and common criteria can also be applied to innovation within this enabling framework.
There is currently a global challenge to identify indicators that meaningfully reflect the impact of access to justice interventions on how peoples’ legal problems are resolved and in turn what impact these interactions with the justice system have on larger goals of poverty reduction and social inclusion. The delineation of common criteria to guide legal and justice services could support OECD Member countries and others as they adopt and evaluate initiatives to foster equal access to justice. As noted at the outset, common criteria can assist by:
-
Improving data-collecting capacity.
-
Building stakeholder support and ownership.
-
Informing project design.
-
Gauging project effectiveness and informing practice.
-
Increasing transparency and accountability.
-
Including the perspectives of vulnerable and marginalised groups (UNDP, 2014[30]).
This project is timely: “The new 2030 agenda reinforces the need to collect reliable, suitable and comparable data to support the development and monitoring of legal aid services and provide an accurate picture of rule of law in practice” (IDLO, 2015[57]). Common criteria could help to shape indicators and evaluation questions, in turn, enriching our understanding of “what works”.
Selecting a limited set of common criteria for “what works” to promote equal access to justice does not pose challenges that vary markedly from those encountered in other measurement domains (Dandurand, Kittayarak and MacPhail, 2015[58]). There are a number of trade-offs to be made in specifying indicators or common criteria. One study highlighted the importance of balancing “depth and breadth”, “aggregation and granularity” and “flexibility and consistency” when developing and reporting on quality indicators (Carinci et al., 2015[59]). Criteria should be dynamic and flexible, offer balanced coverage of areas of greatest concern or relevance, including structure, process and outcome (Green Growth Knowledge Platform, 2013[60]).
The following factors have been applied in selecting the common criteria proposed:
-
importance (in terms of its impact of access to justice)
-
policy relevance to both legal and justice services
-
analytical or empirical soundness
-
measurability
-
operational experience in countries
-
multi-dimensionality
-
capacity for evolution
-
usefulness in communication.
Furthermore, the aim is that the common criteria be “evergreen,” in the sense of being capable of maintaining their value over time and across different justice systems.23
There is a growing understanding that access takes place within a complex justice system, consisting of many institutions and actors, formal and informal and the system needs to be addressed as a whole. The justice system must be comprehended as a series of pathways or justice chains from the perspective of individuals, families and communities. Common criteria should help to guide systemic, comprehensive reform from entry point to end point.
For discussion purposes, it is proposed that people-centred legal and justice services should be developed, designed, delivered and evaluated according to ten common criteria: i) evidence-based planning; ii) accessibility; iii) availability; iv) prevention, proactivity, and timeliness; v) appropriateness and responsiveness; vi) empowerment; vii) equality and inclusion; viii) outcome-focused and fairness; xi) collaboration and integration; and x) effectiveness. Figure 6.8 consolidates the earlier partial figures into an overarching framework and Box 6.12 defines in global terms the ten criteria and provides examples of good practices for each one.
Common criteria bring together what is currently known about effective legal and justice service delivery, clarify and extend the discussion about good practices to enhance equal access to justice. Making the most of common criteria involves an ongoing process as we learn more about what works.
These criteria can be applied across all the elements of the justice system: structures, processes, outcomes and the system as a whole. They also should be applied across the key stages of the justice pathway from promoting legal health, capability and literacy through to dealing with specific and/or chronic legal problems and disputes in a variety of processes and fora, enforcement and post-resolution support that further reinforces legal capability and resilience to future legal problems.
The common criteria proposed here are not meant to be static. They are a starting point for a process of continual learning and evolution of evidence-based best practices to ensure high quality legal and justice service delivery. Life can be breathed into common criteria by using them as a base for: evaluation of services; the development of standards for infrastructure and procedures; research; inter-programme consultation and collaboration on best practices; and reviewing comparable developments in other sectors, documenting which criteria and related best practices are most valuable (Charn and Youlls, 2014[61]).
Evidence-based planning
People-centred legal and justice services are based on and respond to an empirical understanding of legal needs and legal capabilities of those who require or seek assistance.
Good practice example: legal needs assessments, mapping of legal and justice services relative to need, individual capabilities assessment by service providers, screening tools.
Accessibility
People-centred legal and justice services are accessible and designed to actively overcome the range of barriers to the assistance they require.
Good practice example: sensitive use of information and communications technology (ICT), programmes to overcome accessibility barriers faced by at-risk groups, simplification of legal language and procedures, reform of the substantive law to facilitate legal clarity, civic engagement and co-design of services.
Availability
People-centred legal and justice services are available across the justice chain and provided in a range of formats, programme and services types.
Good practice example: range of services provided along the continuum of legal assistance services and the spectrum of justice services, strategies deployment of these services.
Prevention, proactivity and timeliness
People-centred legal and justice services are proactive and contribute to the prevention of legal problems and a timely resolution. Recurring legal problems are addressed on a systemic basis to address underlying causes thereby preventing reoccurrences.
Good practice example: proactive outreach, hospital-based legal advice programmes (medical-legal partnerships), improving co-ordination of services in legal information and advice sector, adapting entry points to the justice system to the needs of marginalised groups (i.e. using trusted intermediaries in community), enhanced capacity for diagnosis, triaging and referral, problem-solving courts, systemic advocacy, justice institutions with systemic/preventative mandates.
Appropriateness and responsiveness
People-centred legal and justice services are appropriate and responsive to the individual, the issues they face and their situation. They are tailored, proportionate and efficient and flexible to accommodate local circumstances.
Good practice example: simplifying proceedings, tailoring procedures to make them proportionate to types of matters, increased use of specialised procedures, expanding dispute resolution options within courts and tribunals and outside of them, making courts and tribunals multi-service centres, developing court, tribunal and/or community based triage systems, matching level of legal assistance/representation to individual capacity and situation, use of ICT to deliver legal and justice services, localised flexibility.
Empowerment
People-centred legal and justice services are empowering, enable people’s meaningful participation in the justice system and build people’s legal capabilities.
Good practice example: making legal information available in a range of formats, using ICT to deliver legal information and skills, legal awareness initiatives, building legal literacy and legal capabilities through self-help and guided help, employing legal health strategies that encourage people to manage their legal affairs to prevent problems, legal health checklists, simplifying justice services to make them more “user friendly,” culturally appropriate services, post-resolution support and follow up that builds resilience.
Equality and inclusion
People-centred legal and justice services are inclusive and targeted at those most in need, responsive to specific access needs of particular groups likely to suffer from social and economic disadvantage or are otherwise marginalised or vulnerable and those with complex needs. They are designed to contribute to equality, poverty reduction and social inclusion.
Good practice example: priority-setting based on needs assessments of vulnerable groups, outreach services, culturally appropriate services, and legal and justice resources available in a range of accessible formats.
Outcome focus and fairness
People-focused legal and justice services contribute to fair process and fair outcomes and to better and more sustainable procedural, substantive and systemic outcomes, including increased trust and confidence in the justice system and better justice system performance, and to the attainment of societal objectives such as socio-economic inclusion.
Good practice example: development of fairness standards, client evaluation of fairness of procedures, evaluation of processes and outcomes on an individual and systemic basis (e.g. result of how legal and justice services work together), outcome objectives are set for individual legal and justice services and for integrated services, services are evaluated according to these objectives, long-term follow up studies of client/user outcomes.
Collaboration and integration
People-centred legal and justice services are part of a coherent system that provide seamless referrals and integrated services through collaboration among legal, justice and other human service providers. People get access to all the services they need to solve the legal and related non-legal aspects of their problems holistically, regardless of the entry point for assistance.
Good practice example: developing multiple, diverse and integrated access points and service responses, seamless referral systems, one-stop shops, problem-solving courts, holistic approaches to service delivery, holistic practices, team delivery of services (including non-legal service providers), collaborative governance structures to facilitate co-ordination between legal and justice service providers (including client/user representatives).
Effectiveness
People-focused legal and justice services are effective and continually improved through evaluation, evidence-based learning and the development and sharing of best practices.
Good practice example: involvement of clients/users in planning and evaluation, integrate “plan-do-study” cycle, sharing of good practices, standardisation of tools, development of quality standards/matrices.
References
[48] Abel, L. (2010), “Evidence-based access to justice”, Journal of Law and Social Change, Vol. 295, p. 305.
[64] Action Committee on Access to Justice in Civil and Family Matters (2013), Access to Civil and Family Justice: A Roadmap for Change, Ottowa, Canada, http://www.cfcj-fcjc.org/sites/default/files/docs/2013/AC_Report_English_Final.pdf.
[54] Albiston, C. and R. Sandefur (2013), “Expanding the empirical study of access to justice”, Wisconsin Law Review, Vol. 101, p. 111.
[33] Australian Government (2009), A Strategic Framework for Access to Justice – A Guide for Future Action, Australian Government, Attorney-General’s Department, Canberra.
[49] Bamberger, M., J. Rugh and L. Mabry (2012), Real World Evaluation: Working Under Budget, Time, Data, and Political Constraints, 2nd Ed., Sage, USA.
[68] Beqiraj, J. and L. McNamara (2014), International Access to Justice: Barriers and Solutions, Bingham Centre for the Rule of Law Report 02/2014, International Bar Association.
[35] Buckley, M. (2013), Evolving Legal Services: Review of Current Literature, Community Legal Education Ontario, Toronto.
[16] Buckley, M. (2010), Moving Forward on Legal Aid – Research on Needs and Innovative Approaches, Canadian Bar Association, Ottowa.
[59] Carinci, F. et al. (2015), “Towards actionable international comparisons of health system performance: Expert revision of the OECD framework and quality indicators”, International Journal for Quality in Health Care, Vol. 27/2, pp. 137-146, https://doi.org/10.1093/intqhc/mzv004.
[32] CBA (2013), Reaching Equal Justice – An Invitation to Envision and Act, Canadian Bar Association, Ottawa, Canada, https://www.cba.org/CBAMediaLibrary/cba_na/images/Equal%20Justice%20-%20Microsite/PDFs/EqualJusticeFinalReport-eng.pdf.
[7] CEPEJ (2014), “European judicial systems: Efficiency and quality of justice”, CEPEJ Studies, European Commission for the Efficiency Justice.
[61] Charn, J. and R. Youlls (2014), A Question of Quality, LSC’s Equal Justice, http://www.lsc.gov/sites/default/files/AQuestionofQuality.pdf.
[42] CLEO (2015), Don’t Smoke, Don’t Be Poor, Read Before Signing: Linking Health Literacy and Legal Capability, Centre for Research and Innovation, Toronto, Canada.
[8] Committee on the Elimination of Racial Discrimination (2012), Concluding Observations on the 20-22th Periodic Reports of Finland, adopted by the Committee at its 81 session, https://tbinternet.ohchr.org/_layouts/treatybodyexternal/Download.aspx?symbolno=CERD/C/FIN/CO/20-22&Lang=En.
[6] Coumarelos, C. et al. (2012), Legal Australia-Wide Survey: Legal Need in Australia, Law and Justice Foundation of New South Wales, Sydney.
[1] Coumarelos, C. et al. (2015), Collaborative Planning Resource - Service Planning, Law and Justice Foundation of NSW, Sydney.
[5] Council of Australian Governments (2010), National Partnership Agreement on Legal Assistance Services, https://www.ag.gov.au/LegalSystem/Legalaidprogrammes/Documents/NationalPartnershipAgreementOnLegalServices.pdf.
[37] Curran, L. (2012), A Literature Review: Examining the Literature on How to Measure the ‘Successful Outcomes’: Quality, Effectiveness and Efficiency of Legal Assistance Services, College of Law, Australian National University.
[41] Curran, L. (2012), We Can See There’s a Light at the End of the Tunnel Now – Demonstrating and Ensuring Quality Service to Clients, Legal Aid ACT.
[20] Currie, A. (2015), Extending the Reach of Legal Aid: Report on the Pilot Phase of the Legal Health Check-Up Project, Canadian Forum on Civil Justice, York University, Toronto, https://www.legalhealthcheckup.ca/bundles/legalcheck/pdf/legal-health-check-up-pilot-evaluation.pdf.
[66] Currie, A. (2004), Riding the Third Wave: Rethinking Criminal Legal Aid within an Access to Justice Framework, Statistics Division, Department of Justice Canada, Research Report, Ottawa.
[58] Dandurand, Y., K. Kittayarak and A. MacPhail (2015), Justice Indicators and Criminal Justice Reform - A Reference Tool, International Centre for Criminal Law Reform and Criminal Justice Policy, Vancouver.
[14] Department of Human Services, Georgia, US (n.d.), Elderly Legal Assistance Program, Division of Aging Services, https://aging.georgia.gov/elderly-legal-assistance-program.
[29] Department of Justice Canada (2014), Report of the Deputy Minister Advisory Panel on Criminal Legal Aid, Department of Justice Canada, Ottowa, http://www.justice.gc.ca/eng/rp-pr/cal.sernam.cl/?m=programa&i=9.
[34] Digiusto, E. (2012), “Effectiveness of public legal assistance services – A discussion paper”, Justice Issues, p. 16.
[36] Engler, R. (2010), “Connecting self-representation to civil gideon: What existing data reveal about when counsel is most needed”, Fordham Urban Law Journal, Vol. 37, p. 39.
[39] Engler, R. (2010), “Reflections on a civil right to counsel and drawing lines: When does access to justice mean full representation by counsel, and when might less assistance suffice?”, Seattle Journal for Social Justice, Vol. 97/138, p. 123.
[69] European Union Agency for Fundamental Rights (FRA) (2011), Access to Justice in Europe: an Overview of Challenges and Opportunities.
[62] Family Court of Australia (2015), Court User Satisfaction Survey, http://www.courtexcellence.com/~/media/Microsites/Files/ICCE/User%20Satisfaction%20Survey.ashx.
[40] Focus Consultants (2012), Evaluation of Family Legal Services, Final Report, Legal Services Society.
[18] Forell, S. and A. Gray (2009), “Outreach legal services to people with complex needs: What works?”, Justice Issues, Vol. 12.
[15] Forell, S., E. McCarron and L. Schetzer (2005), No Home, No Justice? The Legal Needs of Homeless People in NSW, Law and Justice Foundation of NSW, Sydney.
[65] Genn, H. (1993), “Tribunals and informal justice”, Modern Law Review, Vol. 393, p. 410.
[45] Gramatikov, M., M. Barendrecht and J. Verdonschot (2011), “Measuring the costs and quality of paths to justice: Contours of a methodology”, Hague Journal on the Rule of Law, Vol. 349, p. 349.
[47] Gramatikov, M. and M. Laxminarayan (2008), “Weighting justice: Constructing an index of access to justice”, TISCO Working Paper Series on Civil Law and Conflict Resolution Systems, Tilburg University, Netherlands.
[60] Green Growth Knowledge Platform (2013), Moving Towards a Common Approach on Green Growth Indicators - A Green Growth Platform Scoping Paper.
[53] Greiner, J. and C. Pattanayak (2011), “Randomized evaluation in legal assistance: What difference does representation (offer and actual use) make? Draft”, http://ssrn.com/abstract=1708664.
[55] Greiner, J., C. Pattanayak and J. Hennessy (2012), “The limits of unbundled legal assistance: A randomized study in a Massachusetts District Court and prospects for the future, draft.”.
[51] Houseman, A. (2013), Civil Legal Aid in the United States: An Update for 2013.
[57] IDLO (2015), “Indonesian-Dutch comparative justice”, Policy Workshop, 10-11 December 2015, The Hague, https://www.idlo.int/sites/default/files/pdfs/events/Report%20-%20Comparative%20Justice%20Policy%20Workshop.pdf.
[31] Legal Aid NSW (2012), Report: Evaluation of the Cooperative Legal Services Delivery Program, Legal Aid Commission of New South Wales, https://www.legalaid.nsw.gov.au/__data/assets/pdf_file/0020/15176/Evaluation-CLSD-Final-Report-August-2012.pdf.
[63] Legal Services Society (2015), 2015 Client Satisfaction Survey, Sentis Market Research, Vancouver, http://www.legalaid.bc.ca/about/LSSClientSurveyAug2015.pdf.
[22] LSC (2013), Report of the Summit on the Use of Technology to Expand Access to Justice, Legal Services Corporation, Washington, http://www.lsc.gov/sites/default/files/LSC_Tech%20Summit%20Report_2013.pdf.
[12] Marchiori, T. (2015), A Framework for Measuring Access to Justice Including Specific Challenges Facing Women, Report commissioned by UN Women, in partnership with the Council of Europe.
[3] McDonald, H. and Z. Wei (2016), How People Solve Legal Problems: Level of Disadvantage and Legal Capability, Law and Justice Foundation of New South Wales, Sydney, http://www.lawfoundation.net.au/ljf/site/articleIDs/4752B67A5D6A030FCA257F6A0004C3C5/$file/JI_23_Disadvantage_legal_capability.pdf.
[13] Ministry of the Attorney, Ontario, Canada (n.d.), The Office of the Children’s Lawyer, https://www.attorneygeneral.jus.gov.on.ca/english/family/ocl.
[10] Mirrlees-Black, C. and S. Williams (2016), Collaborative Planning Resource: Presentation to Federation of Community Legal Centres, (Victoria) Inc., Melbourne, Victoria.
[9] Mirrlees-Black, C. and S. Williams (2015), Collaborative Planning Resource - Jurisdictional Data: NSW, Law and Justice Foundation of NSW, Sydney, http://ttp://www.lawfoundation.net.au/ljf/site/templates/resources/$file/CPR_NSW_Jurisdictional_Data_2015WEB.pdf.
[38] Moorhead, R. (2003), “Contesting professionalism: Legal aid and non-lawyers in England and Wales”, Law and Society Review, Vol. 765, p. 785.
[26] New Zealand Ministry of Justice (n.d.), “Sharing information safely: Guidance on sharing personal information under the Family and Whānau Violence Legislation Bill”, https://consultations.justice.govt.nz/policy/information-sharing-guidance-fv-sector/user_uploads/draft-guidance-fv-information-sharing.pdf.
[11] Observatoire Départemental - Seine Saint Denis (2013), Panorama Cartographique d’Indicateurs Sociaux.
[24] OECD (2016), Northern Ireland (United Kingdom): Implementing Joined-up Governance for a Common Purpose, OECD Public Governance Reviews, OECD Publishing, Paris, https://doi.org/10.1787/9789264260016-en.
[28] OECD (2015), Integrating Social Services for Vulnerable Groups: Bridging Sectors For Better Service Delivery, OECD Publishing, Paris, https://doi.org/10.1787/9789264233775-en.
[46] OECD-OSF (2016), Understanding Effective Access to Justice.
[56] OECD-OSJI (2019), Legal Needs Surveys and Access to Justice, OECD Publishing, Paris.
[43] Pearson, J. and L. Davis (2002), The Hotline Outcomes Assessment Study Final Report - Phase III: Full-Scale Telephone Survey (Center for Policy Research), Commissioned by the Project for the Future of Equal Justice [US Hotline Assessment].
[19] Pleasence, P. (2016), ‘Legal Needs’ and Legal Needs Surveys: A Background Paper, OSJI.
[17] Pleasence, P. (2006), Causes of Action: Civil Law and Social Justice, 2nd Edn., Stationery Office, Norwich.
[2] Pleasence, P. et al. (2014), Reshaping Legal Assistance Services: Building on the Evidence Base, Law and Justice Foundation of New South Wales, Sydney.
[4] Pro Bono Institute (2016), A Survey of Pro Bono Practices and Opportunities in 84 Jurisdictions, Latham & Watkins LLP, http://www.probonoinst.org/wpps/wp-content/uploads/Global-Survey-2016.pdf.
[25] Productivity Commission (2013), Annual Report 2012-13, Annual Report Series, Productivity Commission, Australian Government, https://www.pc.gov.au/about/governance/annual-reports/2012-13.
[21] Ribadeneyra, J. (2012), “Using technology to enhance access to justice”, Harvard Journal of Law & Technology, Vol. 26/1.
[27] Ryan, P. and K. Ray (2014), Evaluation of the Cooperative Legal Services Delivery Program, Legal Aid New South Wales, Sydney.
[44] Smith, R. (2013), “Telephone hotlines and legal advice: A preliminary discussion paper”, (unpublished), Legal Aid Group.
[23] Staudt, R. (2009), “All the wild possibilities: Technology that attacks barriers to access to justice”, Loyola of Los Angeles Law Review, Vol. 1117.
[52] Steinberg, J. (2011), “In pursuit of justice? Case outcomes and the delivery of unbundled legal services”, Georgetown Journal on Poverty Law & Policy, Vol. 453, p. 501.
[50] The Perryman Group (2013), Current and Potential Economic Benefits of Legal Aid Services in Texas, http://www.teajf.org/news/docs/Impact/LSSClientSurveyAug2015.pdf.
[30] UNDP (2014), Why, What and How to Measure? A User’s Guide to Measuring Rule of Law, Justice and Security Programmes, United Nations Development Programme, New York, https://www.undp.org/content/dam/undp/library/crisis%20prevention/UNDP_CPR_ROLMEGuide_August2014.pdf.
[67] University of Toronto Faculty of Law (2011), “Middle income access to civil justice colloquium, Background paper”, Middle Income Access to Civil Justice Initiative, Middle Income Access to Civil Justice Initiative Steering Committee.
Notes
← 1. This section is based upon presentation by G. Mulherin, “The Future of Civil Procedures: Innovation & Inertia: Thoughts on the Planning, Funding and Responsibilities of Legal Assistance Services” delivered at Monash University, Melbourne, June 2016.
← 2. Ibid.
← 3. Ibid. Factors include: poor knowledge of legal rights, legal remedies and the justice system; poor literacy, language and communication skills; poor health; personal efficacy; living free from violence; having a steady job; access to resources; having trust and confidence in institutions; readiness to take action; more pressing basic needs; services that are not culturally sensitive or appropriate; physical and systemic barriers to justice system; distance; poor infrastructure; and fragmentation of justice system and services.
← 4. See also Currie (2004[66]), University of Toronto Faculty of Law (2011[67]), Beqiraj, J. and McNamara, L. (2014[68]), FRA (2011[69]).
← 5. See: Foolkit (http://www.foolkit.com.au).
← 6. See: Probation and Mediation Service of the Czech Republic (www.pmscr.cz/en/).
← 7. See: Weiser Ring (www.weisser-ring.at).
← 8. See: National Family Violence Prevention Service (http://www.nationalfvpls.org).
← 9. For a list of US Access to Justice Commissions, see the American Bar Association (www.americanbar.org/groups/legal_aid_indigent_defendants/initiatives/resource_center_for_access_to_justice/atj-commissions/commission-directory.html). Access to justice committees have been established in most Canadian provinces and territories following the recommendation in the report of the Action Committee on Access to Justice in Civil and Family Matters (2013[64]).
← 10. See: Courts and Tribunals Judiciary (www.judiciary.gov.uk/related-offices-and-bodies/advisory-bodies/cjc/cjc-publications/guidance-for-the-instruction-of-experts-in-civil-claims).
← 11. See: Scottish Civil Justice Council: www.scottishciviljusticecouncil.gov.uk.
← 12. See: Access to Civil and Family Justice - A Roadmap for Change (www.cfcj-fcjc.org/sites/default/files/docs/2013/AC_Report_English_Final.pdf).
← 13. See: New Zealand Ministry of Business, Innovation and Employment (www.mbie.govt.nz/about/our-work/roles-and-responsibilities/government-centre-dispute-resolution).
← 14. Presentation by Mr Conan McKenna, 2017 OECD Roundtable.
← 15. See: New Zealand Ministry of Justice (http://www.justice.govt.nz/justice-sector-policy/key-initiatives/family-court-rewrite/, https://www.justice.govt.nz/about/news-and-media/news/family-court-research-documents/).
← 16. See Genn (1993[65]) (questioning “whether subjective perceptions of fairness on the part of applicants or litigants in informal hearings should be a sufficient goal, or whether fair procedures must be related to just outcomes”).
← 17. In summary, data from MA2J could offer the users the knowledge to weigh the cost and benefits of a (legal) process. Better assessment of the cost, quality of the procedure and quality of the outcome would lead to improved access to justice. This data can also assist in evaluating different systemic properties such as accessibility, predictability, fairness and equality.
← 18. For more detailed information on the project, please see PLE Learning Exchange Ontario (www.plelearningexchange.ca/research/research-evolving-legal-services- research-project).
← 19. See for example: Family Court of Australia (2015[62]); Legal Services Society (2015[63]).
← 20. For background to this project see: Boston Bar Association (2008), “Gideon’s New Trumpet: Expanding the Civil Right to Counsel in Massachusetts” and updates on this website: www.bostonbar.org.
← 21. The California Sargent Shriver Civil Counsel Act Pilot Programs are funded by the California legislature to provide legal services, including direct representation, to low income self-represented litigants in select areas including housing, child custody and guardianship. This project has been called “the most ambitious civil right to counsel pilot project”. The legislation establishing the USD 10 million programme specifies that an evaluation of the effectiveness of the 9 pilots be conducted by 2015 and reported on in early 2016. The report must include an assessment of the benefits and impact of the programmes on both the individuals receiving assistance and the courts.
← 22. Presentation during the 2017 OECD Roundtable on Equal Access to Justice.
← 23. Based on discussions with Dr Ab Currie, Senior Fellow, Canadian Forum on Civil Justice.