3. Model Questions, Model Structure and Short-Form Illustrative Questionnaire
This Chapter sets out a series of model core legal needs survey questions, along with explanations of their form. It also describes the range of topics that have been addressed through past legal needs surveys, and then situates the model questions within an illustrative short-form questionnaire.
3.1. Model core legal needs survey questions
The following paragraphs introduce and explain model questions for: identifying justiciable problems, gauging problem seriousness, determining problem impact, identifying sources of help, categorising problem resolving behaviour, establishing the nature of processes used, ascertaining if and how problems have been concluded, investigating (if relevant) reasons for not obtaining independent advice, exploring people’s perceptions of dispute resolution processes and outcomes, estimating the cost of problem resolution, examining legal capability, and finding out problem start and end dates. The model questions are informed by the full range of legal needs surveys undertaken to date. Explanations link to the broader methodological discussion in Chapter 2.
The questions are primarily designed for use in short-form questionnaires but are also nested in the illustrative long-form questionnaire set out in Annex B. Indication is provided as to how questions can be adapted to collect more or less granular data if necessary.
The diversity of justice institutions, services, practices and norms around the world entails that questions may sometimes need to be adapted to reflect the understanding, experience and options available for specific sample populations. While the questions have been designed to be broadly applicable, appropriate scrutiny and testing should always be undertaken ahead of implementation.
3.1.1. Problem identification (Annex A; Figure A.1)
A model question for identifying justiciable problems is:
I am going to read you a list of problems and disputes that people commonly experience in everyday life. In each case, can you tell me whether you have personally experienced such a problem in the past two years; by which I mean a problem that started since [DATE] or started before then, but continued afterwards?
Please only include problems that you have had yourself, not problems experienced by a business you run, in the course of self-employment or by an employer, and not situations where you represented or helped somebody else with their problem. And please only mention problems once.
This question is appropriate for both long- and short-form questionnaires. For longer questionnaires, all problems of interest should be presented to respondents individually. Show cards, or even comprehensive problem type booklets, have commonly been used to facilitate this process.1 For shorter questionnaires, problem categories, along with brief descriptions and examples, can be presented instead. Care should be taken to ensure that descriptions and examples are sufficiently clear to indicate the full range of problems in a particular category, while minimising the likelihood that non-justiciable problems will be reported.2
The model question for problem identification uses the phrase “problems and disputes” to indicate the character of the issues under study. These terms have been adopted as a pair in more than half of the national legal needs surveys detailed in Table 1.1 and Table 1.2.3
To avoid any reference to law, the question refers to problems and disputes commonly experienced “in everyday life”. To promote reliability, respondents are presented with a simple (dichotomous) question asking whether they have experienced problems, rather than how many problems they have experienced. If, as is generally the case, the total number of problems experienced is of interest, this can be ascertained through an immediate follow-up question (e.g. “How many such problems have you experienced in the past two years?”)
The reference period is two years. Although uncommon in the past,4 a two-year period is increasingly considered a good timeframe for achieving a balance between maximising problem reporting, data accuracy and contemporaneity.5 Reflecting this, the World Justice Project’s General Population Poll moved to a two-year reference period in 2017, and the same was adopted for the 2017-2018 Nepalese survey and 2017-2018 South African Governance Public Safety and Justice Survey Pilot.
Allied to the reference period, the model question makes clear that problems should be reported if they existed within the reference period, irrespective of when they started. Many of the surveys detailed in Table 1.1 and Table 1.2 have followed this Paths to Justice survey inspired rule of inclusion, but others have been silent on the matter. Here, a rule is made explicit to ensure clarity, and the choice of rule reflects the fact that the main focus of interest in legal needs surveys tends to be on problem resolving behaviour and experience, which manifests across the full lifetime of a problem. The rule increases the number of problems that will be reported. Importantly, it also increases the number of problems that have been concluded and problems of a more serious nature about which data is captured in a manner that does not undermine the coherence of the sample.6
The question is directed to identifying problems experienced by respondents personally, rather than within households, etc., to promote data accuracy, best reflect the general nature of experience and provide flexibility for data disaggregation and aggregation, as discussed in the previous chapter.
The question clearly excludes problems experienced by a respondent in running a business, an employer, or anyone respondents may have represented or helped to resolve a problem. It also emphasises that problems should only be reported once, as there is a significant risk of double-counting when potentially overlapping problem types or categories are presented to respondents. This is often the case when problems are defined in broad terms but can also occur with multi-dimensional problems.
An example set of 12 core problem categories, descriptions and examples – for use in short-form questionnaires – is set out in Table 3.1.7 The examples used to illustrate the categories have been drawn from previous surveys, with phrasing modified for greater relevance across all jurisdictions. If individual problem types need to be identified, this can be done through follow-up.8
Randomisation should be employed when presenting problem types or categories to respondents in order to mitigate issues stemming from fatigue and satisficing behaviour.
However, when a problem might reasonably fall within a number of categories or types, questions should be designed to ensure reliability. To do this, some categories or types may need to be presented in a particular order. If this is the case, randomisation should apply to the groups so constituted, rather than to individual categories.
Table 3.2 sets out additional categories that are often incorporated into legal needs surveys, but which are distinct in character from the categories set out in Figure 2.1. These additional categories are concerned with business and crime. As well as being distinct, the three categories in Figure 2.1 should be asked about separately, unless measures have been taken to prevent conceptual conflation as discussed in Chapter 2. . Reflecting this, the model question expressly excludes business related problems from its scope.
The problem descriptions in Table 3.1 and Table 3.2 are drafted in lay terms to promote recognition and avoid overly narrow interpretations that might stem from public misconceptions of law. As far as is possible, they are drafted to make a legal dimension inevitable.
As the law varies between jurisdictions, some problem descriptions will not be appropriate universally and others may require further explanation. However, care has been taken to select examples that have broad applicability. Nevertheless, when using the descriptions in Table 3.1 and Table 3.2, they should always be reviewed by legal and community experts ahead of use.
As sociocultural norms vary among jurisdictions, errors introduced into surveys by inaccurate responses (respondent error) will also vary. For example, social desirability bias is likely to vary considerably among jurisdictions in relation to domestic violence. It is known that “in countries with strong cultural pressure to keep violence behind closed doors or simply to accept is as natural, non-fatal violence is likely to be underreported” (Fraga, 2016, p. 78).
While the majority of past legal needs surveys have included domestic violence – domestic violence has been found to be catalytic in relation to wider population experience (Pleasence et al., 2003) – it is to be noted that the United Nations Department of Economic and Social Affairs Guidelines for Producing Statistics on Violence Against Women – Statistical Surveys argue that “surveys designed to address a broad array of crime- or health-related or other issues cannot accommodate the broad range of questions needed to study violence against women in all its complexity” (United Nations Department of Economic and Social Affairs, 2014, p. 8). They continue:
“Certain drawbacks are evident when a module on violence against women is inserted into an already lengthy questionnaire on other topics. For example, the question wording and ordering may not facilitate disclosures of violence, especially if introductory statements or questions that cue respondents to think about violence occurring in private settings or incidents involving intimate partners have not been introduced. Compared to dedicated surveys, the breadth of questions that can be included in surveys on other topics is also limited, thereby reducing the opportunities for disclosure of experiences of violence. Finally, less attention is usually paid to the sensitisation of interviewers to violence-related issues during their training, the need for interviewers to develop a rapport with respondents, privacy and confidentiality issues surrounding the interview and other ethical and safety issues, all of which can have a significant negative impact on the willingness of respondents to report violence.” (United Nations Department of Economic and Social Affairs (2014, p. 8-9)
Importantly, if domestic violence (or other forms of violence or harassment in, say, a neighbourhood or employment context) is included in legal needs surveys, then – as the guidelines further add – “ethical considerations are of upmost importance … [and] care must be taken … to consider how each aspect of the survey design and implementation will affect the safety and well-being of the respondents and, indeed, of the interviewers.” (United Nations Department of Economic and Social Affairs (2014, p. 7).
To obtain a clearer picture of the nature of problems reported through the model problem identification question, the simplest method is to follow it up, for example, with questions asking what problems were about and/or whether and with whom the problems were shared.
3.1.2. Problem seriousness (Annex A; Figure A.2)
A model question for gauging problem seriousness is:
Thinking about the problem as a whole, consider a scale of 1 to 10, where 1 represents the least serious type of problem you could face and 10 represents the most serious.
To provide some examples, a score of 8 might be [ANCHOR 1] and a score of 2 might be [ANCHOR 2].
What number best represents the seriousness of your problem?
Again, this question is appropriate for both long- and short-form questionnaires. It provides a basis for both substantive analysis and the filtering out of trivial problems from follow-up (i.e. by omitting from follow-up any problem not perceived to be as serious as the lower anchor problem description). It is based on the seriousness question developed for the 2010 English and Welsh Civil and Social Justice Panel Survey. While the psychometric properties of the scale have not been tested, and the question is likely to be less reliable than a multiple item scale (no examples of which have yet been developed),9 it has proved to be simple to implement.
The form of the question is textual, rather than graphical, so that it can be used across surveys delivery via different means without the need for adaptation.
No anchor problem descriptions are included in the model question text, as suitable problem descriptions are likely to vary between populations. 10 It is important to have a high degree of accord as to the seriousness of selected problem descriptions. Moreover, the wording of anchors must be considered carefully, as it can make a significant difference to measurement.11
3.1.3. Problem impact (Annex A; Figure A.3)
A model question for determining problem impact is:
Did you experience any of the following as part of or as a result of this problem?
a) ill-health or injury
b) high levels of stress
c) damage to a family relationship
d) being harassed, threatened or assaulted
e) damage to your property
f) loss of employment
g) having to move home
h) financial loss
i) fear or loss of confidence
This question is fairly concise, but can be lengthened or shortened if required. Some surveys have investigated a greater range of impact areas – including stigma, alcohol/drug problems, denial of public services, problems concerning education and problems concerning documentation – but those in the model question represent the most common (and commonly asked about).
There may also be interest in particular aspects of model question impact areas, such as unemployment (as an aspect of loss of employment) and homelessness (as an aspect of moving home).
The model question extends to elements of justiciable problems, as well as impacts that follow from justiciable problems. Both are relevant to the cost of justiciable problems, as well as to the potential benefits of interventions that might prevent justiciable problems from arising. If interest is only in impacts following from justiciable problems, the words “as part of” can be removed. However, if this is done, it should be emphasised that harms experienced “as part of” problems should not be reported.
3.1.4. Identifying sources of help (Annex A; Figure A.4; Figure A.5; Figure A.6)
Two model questions which together can be used to identify sources of help are:
1. Did you, or somebody acting on your behalf, obtain information from any of the following sources, to help you better understand, resolve or prepare to resolve [the problem]?
a) A website or “app”
b) A leaflet, book or self-help guide
c) Newspapers or magazines
d) Television, video or radio
2. (Apart from anything you have told me about already) Did you, or someone acting on your behalf, obtain information, advice or representation from any of the following people or organisations to help you better understand or resolve [the problem]?
Please exclude any help provided by the other party.
The first question asks about information obtained via the media (i.e. all modes of mass communication); the second about information, advice and representation otherwise obtained from people or organisations.
Both questions ask about help obtained personally or through someone acting on the respondent’s behalf. This is to more fully capture the totality of help seeking. The questions also require help to have been “obtained” so as to exclude instances of unsuccessful help seeking. In the absence of additional questions in this area, this formulation provides greatest insight into unmet legal needs. In a longer questionnaire, a more comprehensive account of problem resolving behaviour can be acquired by asking about both successful and unsuccessful attempts to obtain help. Unsuccessful attempts to obtain help may also be investigated through the model residual activity question presented in the next chapter, although details will be limited.
Both questions refer to help obtained “to help better understand or resolve” problems rather than “to try to resolve” problems to clearly incorporate help obtained prior to, or separate from, decisions to take action.
The second question asks about “information, advice or representation” to clarify the types of help of interest and prevent reporting of people or organisations responsible for dispute resolution processes. The term “help” has been avoided, as it is broader in scope than information, advice and representation.
The second question also uses the phrase “people or organisations”. Although not a critical design issue in the case of closed questions, the phrase is the most appropriate for open ones. It indicates that all sources of help should be included. However, the question explicitly states that help obtained from the other party should be excluded. The question will also naturally exclude information obtained via mass media, which was asked about in the first question. To emphasise the distinction, the second question starts by excluding anything respondents have already reported.
As always, questions concerning sources of help should be constructed using lay language whenever possible. If the identity of a particular, say, legal service needs to be known, then it is preferable to use generic terminology in the first instance and follow-up with more specific questions about identity.
Finally, the second question centres upon a list of categories of sources of help, an example of which is set out in Figure A.6. Figure A.6 list will prompt respondents to recall help sought from the full range of sources in Figure 2.3 and thus enable data to be recorded in a manner that is consistent with that table’s structure.
For shorter-form questionnaires, an open version of the second question could be used instead. If an open question is adopted, additional sources of help should be probed for. An open form of the second question is:
(Apart from anything you have told me about already) did you, or someone acting on your behalf, obtain information, advice or representation from any person or organisation to help you better understand or resolve [the problem]? For example, from family or friends; from a lawyer, professional advisor or advice service; a court, government body, or the police; a trade union or employer; a religious or community leader or organisation; an [NGO/civil society organisation/charity]12, a trusted person or organisation or anybody else?
Please exclude any help provided by the other party.
3.1.5. Process (Annex A; Box A.1)
A model question and follow-ups for establishing the nature, and initiator, of processes are:
1. (Apart from anything you have told me about already) Did any of the following things happen as part of [the problem] or sorting it out? When I say “you” here, I mean you or somebody acting on your behalf.
a) You communicated with the other party
b) You or the other party made a claim to, or made use of, a court (or tribunal)
c) [If applicable] You or the other party made a claim to, or made use of, an [Indigenous/ customary] dispute resolution process (e.g. [examples])
d) [The problem] was reported to the police (or other prosecution authority)
e) You or the other party turned to, or action was taken by, a formal designated authority or agency, such as [examples, e.g. Ombudsman, regulator (e.g. [example]) or enforcement authority (e.g. consumer protection authority)]
f) You or the other party turned to, or action was taken by, another state authority (e.g. [examples])
g) You or the other party turned to, or action was taken by, a religious authority
h) You or the other party turned to, or action was taken by, a community leader or organisation (e.g. [example])
i) You participated in formal mediation, conciliation or arbitration (e.g. [examples])
j) You or the other party made use of a formal appeals process operated by the other party or independently
k) You, the other party or somebody else turned to, or action was taken by, another third party for adjudication, mediation or intervention
l) There was no negotiation or third party involvement
2. [For each positive response (a to k), respondents to then be asked]
Who initiated this action? [PROMPT ONLY IF NECESSARY]
a) You
b) The other party
c) The third party responsible for the process
d) Another third party.
3. [If the respondent did not initiate processes b to h or j and k]
Did you respond to this action?
The model process question begins by excluding anything that respondents have already reported. This emphasises that the question concerns processes and not information, advice or representation, already addressed in earlier questions.
The question then asks whether any of 11 distinct processes “happen[ed] as part of [the problem] or sorting it out”. These processes follow the typology set out in Table 2.3 and are defined with reference to the body responsible for the process rather than with the process (which is often but not always implicit). The 11 process categories in the question include four state provided processes, two community processes, one religious authority process, negotiation between the parties and three “other” categories of process. In five of the 11 categories, the question requires examples or detail to be included, so as to make clear the types of authority/ process. For example, category (c) refers to traditional dispute resolution processes, which are still common in many jurisdictions.
Reflecting the need for appropriate terms and phrases, the model process question uses seven different phrases to identify the 11 process. The most common phrase, “turn to”, appears five times, “made use of” three times and “made a claim to” twice. The term “contact” is too general. Phrases such as “appear at” and “participate in” are also inappropriate in all cases other than mediation (which requires engagement), as they suggest a need for engagement on the part of the respondent. The phrase “appeal to” may be inappropriate if it suggests prior process when there need be none.
Some of the response categories for the model process question refer only to respondents’ activities, some to the activities of respondents and the other party, and some to any activity. The distinctions mirror the nature of the processes concerned. Thus, only the respondent and other party in a dispute are relevant to the advancement of civil court process; the same is not true of, for example, criminal investigation.
For each process identified, the first follow-up question asks who initiated the process. The appropriate options will vary depending upon the nature of the process.
If a respondent did not initiate an identified process, the second follow-up question asks whether the respondent responded to any action.
For shorter-form questionnaires an open process question may be necessary. A model open process question and follow-up are:
1. Did you, somebody acting on your behalf, the other party or anybody else, make a claim to a court (or tribunal), or turn to any other third-party individual or organisation – such as [institutional examples] or a community or religious leader [or respected family member] – to adjudicate, mediate or intervene to help resolve [the problem]?
2. [For each process specified]
Did you, or somebody acting on your behalf, initiate or respond to this action?
a) Initiated action
b) Responded to action
c) Neither initiated, nor responded to, action
If necessary, a further follow-up question could be used to identify who initiated processes, if not respondents.
The reference to a respected family member should be included if a family member might potentially act to adjudicate, mediate or intervene to resolve a problem (other than by representing the respondent in negotiation or another dispute resolution process; which should be coded accordingly). As the report of the 2015 Uganda survey commented, “The family is a central institution for dispute resolution in Uganda, along with the broader informal network consisting of friends and neighbours. This is not surprising given the traditional value of the family in dispute resolution as well as the lack of access to the formal justice system, which incentivises a heterogeneity of responses to justice problems, particularly for women and especially for those who are poor” (Piest et al., 2016, p. 80).
Given the complexity of the model process question, and sometimes substantial variation in the nature of dispute processes between jurisdictions, it is particularly important to review process questions through cognitive testing and piloting.
3.1.6. Residual problem resolving behaviour (Annex A; Figure A.7)
In order to appropriately categorise problem resolving behaviour, it is necessary to capture data concerning any form of problem resolving behaviour on the part of the respondent that is not captured by the model sources of help questions or the model process question.
A model question for identifying residual problem resolving behaviour is:
(Apart from anything you have told me about already) Did you, or somebody acting on your behalf, do anything else to help you better understand or resolve [the problem], such as obtain or organise evidence or make an insurance claim?
The question is presented so as not to limit responses to pre-determined behaviours. A similar question was used in the Paths to Justice surveys, to ask about the residual activity of those who did not contact sources of help. The examples provided (namely, “obtaining or organising evidence” and “making an insurance claim”) represent a potentially significant component of problem resolving behaviour. Consideration of options, and communication with the other side that falls short of “trying to resolve the problem” (asked about in the model process question), might also fall within residual activity.
Importantly, the residual problem resolving behaviour question should be asked after the model process question. If the short form of model process question is used, then the residual problem resolving behaviour question should also include the example of “communicate with the other party”.
3.1.7. Fact and manner of conclusion (Annex A; Figure A.8 and Figure A.9)
A model question for identifying the fact of problem conclusion is:
Is [the problem] ongoing or done with? By “done with” I mean that the problem is either resolved or that it persists, but you and everybody else have permanently given up all efforts to resolve it further. [PROBE FULLY]
a) Ongoing
b) Too early to say
c) Done with - problem persists, but all have given up trying to resolve it further
d) Done with - problem resolved
This question is appropriate for both long- and short-form questionnaires. It seeks to ascertain whether or not problems are ongoing or “done with”, in the sense that there is no prospect of any future attempts being made to resolve them. The question requires careful coding and – if relevant – careful follow-up, as it allows for problems to be treated as if they have been concluded, even though they may persist. This accurately reflects the reality of lived problems, which are often simply “put up with”.
The phrase “done with” is preferred to other previously used wording (notably “over” or “resolved”), as it suggests no change in the future rather than the non-existence of a problem. However, as the intended meaning is relatively complex, the question goes on to provide a definition. The definition of “done with” is central to the question, so it should not be shortened or removed to save time. It is important that it refers to the intentions of both the respondent and the other party and that the intentions be “permanent” in nature.
A model question for identifying the manner of problem conclusion is:
Which of the following statements best reflects how the problem outcome was ultimately brought about?
The problem outcome was ultimately brought about by:
a) a court (or tribunal) judgment
b) a decision or intervention by another formal authority
c) mediation, conciliation or arbitration
d) action by another third party
e) agreement between you and the other party
f) the other party independently doing what you wanted
g) you independently doing what the other party wanted
h) your moving away from the problem (e.g. moving home, changing job)
i) the problem sorting itself out
j) you and/or all other parties giving up trying to resolve the problem
Again, the question is appropriate for both long- and short-form questionnaires. It is presented in a closed form, as the data sought is particular and an open question is less conducive to a succinct and relevant response.
The question is designed to be comprehensive. The model covers all 30 categories referenced in past surveys.
There is some scope for more succinct delivery of the question. For example, process questions can be used to restrict the categories read/shown to respondents. For instance, if there has been no court process, then the corresponding (first) outcome category is redundant. However, such efficiencies place additional burden on earlier questions and must be carefully considered.13
3.1.8. Reasons for advice not being obtained14 (Annex A; Figure A.10)
A model question for investigating, if relevant, reasons for independent advice having not been obtained is:
Why didn’t you obtain independent advice to help resolve the problem?
The question can be either open or closed. In the above instance, the range of potential responses is well understood (with model categories set out in Figure A.10). The main considerations concerning the form of the question will be the length of the question and data quality. A speedily delivered open question is likely to be quicker, while a closed question is likely to generate a broader range of responses from individual respondents.
The question is designed to allow multiple responses. If there is interest in the relative significance of response items in individual cases, then respondents could be asked to identify the main reason, rank responses or indicate the degree of importance of each reason. Although each of these requires more time, they yield a greater level of insight.
Perceptions of the quality of process and outcome (Annex A); Model questions for exploring perceptions of the dispute resolution process and problem outcome are:
Do you feel the outcome of this problem was basically fair to everybody concerned?
Regardless of the outcome of this problem, do you feel the process through which the outcome was reached was basically fair to everybody concerned?
The model questions are most suited to short-form questionnaires. In their suggested form, they are best suited to capturing binary data. If the aim is to investigate multiple dimensions of perceptions of process and outcome, longer-form questionnaires are preferable because of their greater scope. They can also be used in addition to a shorter questionnaire.
The model questions explore the dispute resolution process and problem outcome as a whole rather than individual processes (e.g. specific court processes, mediation, etc.) and individual process outcomes. Past legal needs surveys have tended to adopt this approach, as it requires fewer questions and reflects the relative rarity of most processes (thus limiting options for analysis and reporting). If, as in the Paths to Justice surveys, there is interest in the quality of individual processes and process outcomes, then the model questions should be prefaced by explicit reference to those processes. For example, they could be prefaced in a manner such as, “Thinking about the claim made to a court as part of this problem …”
Both model questions ask about fairness; the most fundamental dimension of process and outcome quality. They ask about “basic” fairness, to indicate that minor deficiencies of fairness should be disregarded. They also ask about fairness “to everybody concerned”, to shift the focus away from respondents’ personal satisfaction.
The process question is clearly differentiated from the outcome question by an explicit instruction to disregard outcome.
To provide a degree of measurement, the model questions can be reformulated to use, say, a 4-point Likert scale (e.g. “very fair”, “somewhat fair”, “somewhat unfair” and “not fair at all”15):
How fair do you feel the outcome of this problem was to everybody concerned?
Regardless of the outcome, how fair do you feel the process through which the outcome was reached was to everybody concerned?
Again using a Likert scale (e.g. “strongly agree”, “agree”, “disagree”, “strongly disagree”), the model questions can also be incorporated into a question matrix:
To what extent do you agree or disagree with the following statements about the problem?
- The outcome of this problem was fair to everybody concerned.
- Regardless of the outcome, the process through which the outcome was reached was fair to everybody concerned.
If there is interest in investigating the different dimensions of process quality and outcome, then additional questions are required. The Justice Needs and Satisfaction Surveys have included a substantial question-set to permit relatively complex measurement of process and outcome quality. Other surveys have sought only a basic indication of one or more quality dimensions. This requires fewer questions, but care must be taken to align questions with the concepts of interest.
If single questions are used to reflect the dimensions of process and outcome quality detailed in Chapter 2 (procedural, interpersonal, informational, distributive and restorative justice, along with outcome functionality and transparency), the starting point for question design should be definitions of these dimensions. This was the approach taken for the 2017 Sierra Leonean survey, in which the questions concerning outcome quality were closely tied to the “criteria for evaluating outcomes of paths to justice” (Verdonschot et al., 2008, p. 12). set out in the course of the Measuring Access to Justice in a Globalising World project.
Example questions covering the seven dimensions, again using a Likert scale for responses, are:
To what extent do you agree or disagree with the following statements?
a) The process was fair, and I had opportunity to explain my position.
b) I was treated with respect.
c) Processes and decisions made were clearly explained.
d) The outcome involved a fair distribution of benefits and burdens.
e) Any loss or harm arising from the problem (e.g. financial or concerning relationships) has been made good.
f) The problem was solved in a timely matter and is unlikely to recur.
g) The outcome was much the same as for other people in similar situations.
3.1.9. The cost of justiciable problem resolution (Annex A; Figure A.12)
A model question for estimating the cost of justiciable problem resolution is:
Excluding indirect payments – such as insurance premiums or membership subscriptions – but including payments made by family members and friends
Did you, personally, have to pay for any of the following in order to resolve the problem?
a) Lawyer and other advisor fees
b) Court, mediation or other administrative fees
c) Telephone calls and correspondence
d) Collecting information or obtaining evidence (incl. reimbursement of witnesses’ costs)
e) Travel (e.g. bus fares or petrol to visit an advisor)
f) Lost business or salary, from taking time off work (e.g. to obtain advice)
g) Bribes / kick-backs (Remember, your answer is confidential)
h) Incidental domestic costs (e.g. childcare)
The model cost question provides a platform for investigating both the affordability and proportionality of the cost of problem resolution. The question is designed to establish whether respondents have personally incurred any financial costs in acting to resolve a specific justiciable problem. The form of the question was inspired by the approach taken by HiiL’s Justice Needs and Satisfaction Surveys and the 2014 Everyday Legal Problems and the Cost of Justice in Canada Survey. The wording of the question also draws on the 2010 Ukrainian, 2014 English and Welsh and 2017 Indian surveys.
The scope of the question is defined and clarified in the preamble, which states that it relates to payments made “in order to resolve” a specified problem, rather than costs incurred simply as a result of encountering a problem. The preamble also excludes indirect payments by respondents, such as insurance premiums or membership (e.g. union) subscriptions, but it includes payments made by family members and friends. In both cases, if the scope of the question is not explicitly stated, there is a risk that respondents will interpret the question inconsistently.
The question centres on a list of cost items. The item list serves both to define the scope of the question and to assist respondents in recalling expenditures they may have incurred. The question permits data to be captured for each list item, although global data could also be captured. If interest only extends to legal costs, the list can be shortened or the question adapted to reference legal costs without a list being used.
In asking about bribes and kick-backs, the question includes a confidentiality reminder to allay concerns respondents may have.
To establish the level of costs incurred by respondents, one of three approaches can be adopted. The first is subjective. Respondents are asked how expensive, or difficult to meet, the costs were (similar to in the 2015 Polish survey and 2016 Argentinian survey). This gives insight into the extent to which costs present an obstacle to accessing justice. If this approach is adopted, the follow-up question might be: “How difficult was it to find the money to pay for this/these things?”
The second approach is more objective. The respondent is asked a single question in order to establish the aggregate costs incurred. For example, the 2014 Canadian survey simply asked, “Approximately how much in total did it cost to deal with this problem?”
This question may be harder for respondents who have incurred costs of multiple types and so may be less reliable than the third approach.
The third approach is to ask about the amount of each type of cost separately, as was done in the Justice Needs and Satisfaction Surveys, and calculate the total from the figures supplied: “Approximately how much did you have to pay for [cost item]?”
The last two approaches are attractive in that they provide a basis for cost-benefit type calculations. However, they do not naturally indicate the extent to which costs were a burden to respondents.
Beyond affordability and proportionality – the latter of which can be investigated either through a subjective value-for-money type question, such as employed in the 2012 Colombian survey,16 or through enquiring as to respondents’ objectives and/or seeking to quantify outcomes – if there is interest in particular forms of financial support for legal services or processes, these can be asked about in a number of ways. One way is to ask:
Apart from family or friends, did anybody, or any organisation, pay or part-pay for [the help you received from [advisor]/[process] fees17]?
Then an open question can be used to ask who provided the financial support. Alternatively, specific forms of support can be enquired about through single questions, such as:
Did you receive financial assistance from legal aid?
Or, a list-based question can be used:
Did any of the following pay or part-pay for [the help you received from [advisor]/[process] fees18]?
3.1.10. Legal capability (Annex A; Figure A.13)
A first model question for examining legal capability in relation to an identified problem is:
To what extent do you agree or disagree with the following statements about the problem?
a) I understood or came to understand my legal rights and responsibilities.
b) I knew where to get good information and advice about resolving the problems.
c) I was able to get all the expert help I needed.
d) I was confident I could achieve a fair outcome.
This question is appropriate for both long- and short-form questionnaires. It addresses a respondent’s awareness of legal rights and responsibilities, awareness of services and legal confidence in relation to an identified problem. It is optimised for identifying unmet legal need. For speed and efficiency, the question employs a common stem to ask respondents about the extent to which they agree with statements relating to these three aspects of legal capability, along with a further statement that concerns whether respondents obtained all the help they felt they needed. The first statement contains elements of questions used in past surveys to investigate knowledge of rights and responsibilities at the time the problem arose, modified to incorporate knowledge acquired subsequently. This is because, if seeking to identify unmet legal need, it is important to recognise knowledge acquired during the problem resolution process. The second statement investigates knowledge of legal services. It adopts phrasing from the 2014 Canadian survey.19 The third statement concerns legal confidence and uses phrasing similar to that used in the 2016 Tajik survey.
A second, complementary, model question for examining legal capability in relation to an identified problem is:
Which of the following describe the problem? You can choose more than one option, or none.
a) Bad luck / part of life
b) Bureaucratic
c) A family or private matter
d) Legal
e) Political
f) A social or community matter
g) Economic
h) None of these
This question concerns the ability of respondents to recognise legal issues. It is appropriate for both long- and short-form questionnaires, but its length means its inclusion in shorter questionnaires may be problematic.
The question is based on one introduced in the 2010 English and Welsh Civil and Social Justice Panel Survey, a version of which was also included in the 2016 Argentinian survey. It gives insight into both awareness of law and the way in which problems are characterised. This links to perceptions of appropriate fora for problem resolution. By mixing the “legal” response category with other potential descriptions, the question avoids being leading.20 The other potential descriptions included in the question constitute common characterisations of justiciable problems. Other descriptions, such as “criminal”, have also been included in the past. If there is particular interest in characterisation, then a response of “none of these” could be followed up with a question such as, “How would you characterise the problem?”
An issue with the question as formulated is that it is set in the present, meaning it reveals characterisation in hindsight rather than when the problem began. However, the question is simpler to administer in this form and is likely to benefit from greater accuracy than one seeking to situate respondents at the time when a problem began. Moreover, characterisation is of significant interest at all stages of problem experience.
Together, the two model legal capability questions address the four broad aspects of legal capability discussed in Chapter 2. The questions do this in relation to particular problems, but legal capability can also be asked about in general.
Questions concerning general recognition of legal issues have never been asked, and only the English and Welsh Civil and Social Justice Survey has asked about general legal understanding. This was done through a series of lengthy vignettes in which legal knowledge questions were embedded.21 Past general capability questions have tended to relate to only awareness of services and legal confidence.
In the case of awareness of services, a mixture of open and closed questions have been used. The former have asked respondents where they might get help to deal with specific problems (e.g. the 2016 Moldovan survey), the latter whether they have heard of (or “know something about”22) services presented to them in a list or whether they know where they could get help, if needed (e.g. the 2012 Tajik survey). There are advantages and disadvantages to using both open and closed questions. For example, “the strength of the open questions is that there are virtually no false positives,” while “the disadvantage of the open form of the question is that it may provide a low estimate of active knowledge, because some people who could recognise the correct answer, or retrieve it given more time, will fail to retrieve it in a survey situation” (Fowler, 1995, p. 69).
As regards the form of open questions, the 2016 Argentinian survey asked about sources of “legal advice”, the 2016 Mongolian survey asked about help for problems such as those detailed in earlier problem identification questions, and the 2014 English and Welsh survey asked about help for hypothetical problems. All three approaches are legitimate, but because of people’s generally narrow interpretation of things “legal”, there are concerns about using the first approach in jurisdictions in which legal advice is frequently provided outside traditional legal services. A model form of question, which can be easily adopted to each approach, is:
Where can people get independent expert advice about X?”
Turning to closed questions, past surveys have sometimes presented respondents with a list of sources of help to determine their knowledge about them. A model form of such a question is:
Which of the following do you know something about?23
It is also possible to ask respondents how much they known about where help can be obtained, although this again requires clarifying the nature of the help and/or the problems it relates to. A model form of such a question is:
How well do you know where people can get independent expert advice about [problem description]?
In relation to legal confidence, both the HiiL SLE questions (which are presented in blocks) and the recently developed standardised measures of legal confidence (such as the GLC scale) provide excellent models.24 An example of the SLE form of question is:
Imagine you had a conflict with your employer, for example, a conflict over your dismissal. How likely is it that you would get a fair solution to the problem?
The GLC scale, the most robust of three standardised legal confidence scales – at least, for use in the United Kingdom – is constructed as follows (with a 4-point Likert scale response-set: “very confident”, “quite confident”, “not very confident”, “not confident at all”):
If you found yourself facing a significant legal dispute – such as being unreasonably sacked by your employer, injured as a result of someone else’s negligence, involved in a dispute over money as part of a divorce, or facing eviction from your home – how confident are you that you could achieve an outcome that is fair and you would be happy with in the following situations?
-
Disagreement is substantial and tensions are running high.
-
The other side says they “will not rest until justice is done”.
-
The other side refuses to speak to you except through their solicitor.
-
A notice from court says you must complete certain forms, including setting out your case.
-
The problem goes to court, a barrister represents the other side, and you are on your own.
-
The court makes a judgement against you, which you see as unfair. You are told you have a right to appeal.”
3.1.11. Finding out problem start and end dates
Model questions for finding out the start and end dates of identified problems are:
1. Can you tell me roughly what month and year the problem started?
2. [If the problem is done with, but persists]
And when did you and everybody else give up all actions to resolve the problem?
3. [If the problem is done with and fully resolved]
And when did it conclude?
3.2. Beyond core questions
The model questions presented above are intended to serve as a core set of questions relevant for national and global statisticians and policymakers. They are not intended to inhibit the collection of more in-depth data, nor are they intended to limit the scope of future legal needs surveys. Rather, they are intended to provide a form and phraseology for key questions that heeds the lessons of past surveys. They are also intended to provide a sufficient basis for meaningful indicators of access to justice, including basic measurement of levels of legal need and unmet legal need. This is discussed further in Chapter 4.
3.2.1. Legal needs surveys: additional topics of investigation
Past legal needs surveys have investigated a broad array of topics. Many of these topics have been explored in this and the previous chapter. However, there are many others that legal needs surveys have and can legitimately explore. Annex C sets out a complete list of topics addressed in past surveys. The following paragraphs provide a summary.
In relation to problem experience, aside from topics already discussed in this chapter, topics have included:
-
The substance of problem (e.g. money, property, changing behaviour, apology, etc.)
-
Who is considered to be responsible for the problem
-
Links to other identified justiciable problems (including “problem clustering”)
-
The nature of other party/parties (including their demographics and relative power)
-
Relationships with other party/parties
-
Whether problems involve discrimination
-
The existence and extent of disagreement
-
Whether problems are shared with other people (households and communities)
In relation to obtaining help, aside from topics already discussed in this chapter, topics have included:
-
How respondents find out about/chose sources of information and help
-
Factors in choice (e.g. distance, cost, reputation, ethnicity, etc.)
-
The extent to which options are researched
-
Nature of Internet use
-
Obstacles/barriers to access (e.g. opening hours, distance, cost, language, etc.)
-
Distance and mode of travel to sources of help
-
When help is obtained
-
The timeliness of assistance
-
The sequence of sources of information/help
-
Links between the use of different sources (e.g. signposting, referral, etc.)
-
The nature of information/help sought
-
The nature of information/help obtained
-
Whether any information/advice suggests objectives would not be met
-
The form of communication with sources of help
-
Whether help is obtained through an intermediary
-
Satisfaction with/utility of information/help obtained
-
Reasons for satisfaction/dissatisfaction
-
Responses to being dissatisfied (complaint, advice, etc.)
-
The impact of obtaining help on social, health, and economic circumstances
-
Details of unsuccessful attempts to obtain information/help
-
Whether there was consideration of (any/further) information/help
In relation to dispute resolution processes, aside from topics already discussed in this chapter, topics have included:
-
Whether and how many hearings/sessions
-
Whether respondent attends hearings/sessions
-
Tasks undertaken by respondent in hearings/sessions
-
Whether the respondent (and other party) is represented, and by who
-
Whether the respondent is pressed to give particular testimony
-
Bribery/threats as part of processes
-
Reasons for choice of process
-
The duration of process
-
The sequence of processes
-
The language used in processes and availability of translation
-
Whether processes are discriminatory
-
The utility of processes
And in relation to problem resolution in general, aside from topics already discussed in this chapter, topics have included:
-
Whether respondents thought problems would be resolve without action
-
The delay until first action was taken
-
Whether previous experience influenced strategy
-
Whether respondents regret how they handled problems
-
What they wish they had done/had known
-
Whether and what help would have improved outcome
By and large, legal needs surveys routinely ask a bank of general attitudinal questions concerning the justice system. Some mirror the questions discussed in relation to perspectives on process or outcome; others are concerned with trust in the justice system, equality of justice and accessibility of justice.25
And, apart from justiciable problem related data, all surveys collect (sometimes extensively26) demographic data, which is used to explore the social patterning of problem experience and behaviour. Demographic data has also been incorporated into analyses of the impact of justiciable problems, the general clustering of problems (both justiciable and other), and links between justiciable problems and wider social, economic and health problems.
Given the strong association between justiciable problem experience and morbidity/disability, there is good reason to consider health/disability status a core aspect of demographic data. Similarly, broader associations between justiciable problem experience, disadvantage and poverty make a similar case for data relating to employment, family status, housing type, income, language, migration, social safety net assistance, etc.
As with other data, for demographic data to be comparable between surveys, it must be collected in comparable form. Thus, unless there is good reason to do otherwise, standard/common forms of demographic questions should be adopted whenever possible; and if international norms exist, they should be adopted. Not only does this promote comparability, it also promotes data quality – as significant focus is placed on refining demographic questions by national and supranational statistical agencies.
3.3. From questions to questionnaire
As discussed in relation to the model legal needs survey structure in Figure 2.4, longer legal needs survey questionnaires are best constructed as a combination of specific structural and topic-based modules, which link to data structure and the various topics of study discussed in the preceding text. This helps to appropriately represent data structure within questionnaires and tie questionnaires to their defining research questions; clarifying which topics are central and which are peripheral.
This section situates the model questions discussed above in an illustrative short-form questionnaire, which is introduced with an explanation of the content and ordering of the questionnaire. Summary explanatory notes are also provided alongside the questionnaire components in Table 3.3. The questionnaire set out in Table 3.3 collects only the smallest amount of data required to build a basic picture of the experience of justiciable problems. This is done in a conceptually coherent manner, and it permits a rudimentary measurement of legal and unmet legal needs using the framework set out in Figure 2.1. Such measurement would involve data from questions 3, 6, 14, 17, 18, 21, 22 and 23. It is not suggested that these questions represent the ideal measurement tool - far from it. Professed knowledge of rights is a poor substitute for an objective test of legal knowledge. Thus, the latter is preferable. However, an objective test – even a short quiz – is unlikely to be practicable in any but the most comprehensive (or narrow) surveys. In fact, professed knowledge can be “disastrously wrong” (Sandefur, 2016, p. 453). Similarly, legal confidence would best be assessed with a standardised measure, such as those recently developed in England and Wales (Pleasence and Balmer, forthcoming), and process fairness would be best addressed through detailed measures rather than single questions.
Measurement of legal needs and access to justice is discussed further in Chapters 2 and 4.
The questionnaire set out in Table 3.3 is too short to fully reflect the model legal needs survey structure in Figure 2.4, but it is built on the same conceptual foundations. Moreover, indication is provided of the points at which supplementary questions and looping question modules might be added.
While care has been taken to draft questions suitable to a variety of modes of delivery, the illustrative short-form questionnaire is most suited to face-to-face delivery, with show-cards to assist delivery. Adaptation to other modes of delivery should be relatively straightforward.
An illustrative expanded version of the questionnaire is set out in Annex B.
As noted in relation to the model questions, the diversity of justice institutions, services, practices and norms around the world means that questions may need to be adapted to specific populations. As with all questionnaires, questions should be carefully scrutinised and tested. The complexity of the subject matter also increases the importance of training for those collecting and/or coding data.
3.3.1. Illustrative short-form legal needs survey questionnaire
The illustrative short-form legal needs survey questionnaire set out in Table 3.3 commences with an introduction that sets the scene for the survey. It introduces the interviewer and the nature of the survey,27 and provides information necessary to ensure that the ethical requirements for personal interviews and collection of personal data are met. These requirements vary from jurisdiction to jurisdiction, and care should always be taken to use appropriate text when introducing a survey.
Following the introduction, initial demographics are obtained. If there is an interest in general legal capability, questions addressing that should be included at this point, as long as they are formulated to avoid drawing attention to the “legal” focus of the questionnaire. Placing such questions here will help to engage respondents, particularly those who go on to report no justiciable problems. It also means that respondents will not have been exposed to questions providing names of sources of help, processes, etc.
Then comes the foundation question upon which the remainder of the questionnaire is built, namely the problem identification question (Q1). This question determines the scope of the survey. If the question is poorly worded, then out-of-scope problems will be followed-up and/or in-scope problems will be missed. The greater and more accurate the detail of problem descriptions, the more effective the question. Q1 references the problem descriptions set out in Table 3.1.
For each category in which problems are reported, Q2 ascertains the number of problems that have been experienced. This then leads to Q3, which asks about the seriousness of problems. Q3 is asked at this early point in the questionnaire as the data it generates will be used to identify the pool of problems sufficiently serious for follow-up. Data generated by this question is also of broader interest. It can contribute to measuring unmet legal need using the framework set out in Figure 2.1, enable the relative seriousness of problems to be investigated, and explored as a predictor of strategy choices, etc. For efficiency, Q2 and Q3 should be asked immediately after respondents report problems in any problem category.
If there is interest in asking about problems experienced in a business capacity, an appropriate identification question should be asked at this point.
Q4 asks about the nature of those problems selected for follow-up. These are randomly selected from the pool of identified problems with seriousness scores over a defined threshold. The number of problems followed-up will depend upon the time available for interviews. Although Q4 is not one of the core model questions discussed above, it validates Q3 data and provides examples for reporting.
If there is interest in identifying problems that are shared (e.g. within households, communities, etc.), this should be asked about at this point, when the focus is on the nature of problems.
Q5 and Q6 ask about information, advice and representation. As discussed above, these two questions are used to distinguish between help obtained via mass communication channels and help received personally. For respondents who received no help, Q20 later asks for reasons, provided that problems have concluded. Although it is possible to ask a variant of Q20 immediately after Q6, it is not known at this point whether the problems have been concluded, so further response categories would need to be added to reflect that respondents may intend to obtain advice in the future. When interest is primarily in legal advice, then the routing for this section can be amended to ask about only that. Q6 also contributes to measuring unmet legal need using the Figure 2.1 framework.
Q7 then asks about process. As discussed above, questions about process should be asked separately to questions about help and other problem-solving behaviour. This is important because people do not necessarily choose or even engage with process. When processes are identified in Q7, the following question establishes who initiated them. For the sake of efficiency, Q8 should be asked of each process type as soon as it is reported. Likewise, Q9 – which asks whether respondents responded to formal process brought against them – should immediately follow all instances of Q8 in the manner of a loop.
Q10 is the problem and resolving behaviour “catch-all” question, without which it is impossible to be certain whether respondents took any action to understand or resolve their problems.
Q11 determines whether problems have been concluded. If they have, Q12 establishes the manner of conclusion. It is important not to conflate process (addressed in previous questions) with the manner in which problems are concluded. They are linked but distinct.
Q13 and Q14 concern quality of outcome and process. Q14 can also contribute to measuring unmet legal need using the Figure 2.1 framework.
Q15 and Q16 ask about the costs of problem resolution. First, respondents are presented with a series of cost items and asked if they incurred them. Then, they are asked to estimate associated costs. If interest is limited to legal service costs and process fees, then a shorter Q15 is appropriate, although it will not provide the same insight in benefit-cost analysis. Q15 can also be shortened by not asking about each cost item separately, but the breadth of items will result in the data being ambiguous.
Together, Q17 and Q18 address the four broad aspects of legal capability discussed in Chapter 2. The two questions also provide data that can be used as part of the process of measuring unmet legal need using the Figure 2.1 framework. Q18 comprises a question matrix, which can be expanded to explore further aspects of legal capability.
Q19 asks about problem impact. If there is a particular interest in benefit-cost analysis, Q19 items can be followed-up for the purposes of estimating costs. For example, use of health services can be investigated, or welfare claims.
Finally, in terms of problem data, Q21 to Q23 establish problem start and (if relevant) end dates. This provides reasonable estimates of problem duration that, as well as being of interest in their own right, provide greater flexibility in the forms of analysis that can be employed with legal needs survey data. Sampling both ongoing and concluded problems raises the issue of how to deal with ongoing problems in analysis and reporting. Including ongoing problems in estimates of, say, use of lawyers poses the problem that ongoing problems may involve lawyer use after the time of interview. Analysis needs to consider “censored’” observations to arrive at an accurate estimate. This can be achieved by using appropriate forms of analysis, such as event history analysis, which takes into account both concluded and on-going problems when modelling problem duration.28
If there is interest in asking about attitudes to the justice system, the appropriate place to do so is after the problem data has all been collected. This placement means that responses will benefit from respondents’ reflections on their own experience of justiciable issues. Attitudinal questions usually concern levels of access to justice, equality of justice and trust in the justice system. They can be asked globally or of constituent elements of the justice system. Care should be taken to avoid technical language, for the reasons set out elsewhere in this Guide.
Finally, in terms of substantive data, core demographic data is obtained (other than that already obtained at the outset). If there is interest in the social patterning of problems and/or links between justiciable problems and wider social, economic and health problems, then core demographic data should cover age, gender, ethnicity, employment status, family status, health/disability status, housing type, income, language, migration status, social safety net assistance, etc.
The questionnaire ends with closing remarks and, if necessary, a request for permission to recontact the respondent.
3.5. References
Fowler, F.J. Jr. (1995), Improving Survey Questions: Design and Evaluation, Sage, Thousand Oaks.
Fraga, S. (2016), “Methodological and ethical challenges in violence research”, in Porto Biomedical Journal, Vol. 1(2), p. 77-80.
Ipsos MORI (2012), The English and Welsh Civil and Social Justice Panel Survey: Wave Two Technical Report, Ipsos MORI, London.
Kim, S. and S. Choi (2018), “Reflections and lessons from the 2018 Nationwide Legal Needs and Access to Justice Survey in South Korea”, unpublished report prepared for the Judiciary of the Republic of Korea.
Piest, J., M. Gramatikov, S. Muller, K. Heijstek-Ziemann and J. Sallali (2016), Justice Needs in Uganda: Legal Problems in Daily Life, HiiL, The Hague.
Pleasence, P. and N.J. Balmer (forthcoming), “Development of a general legal confidence scale: A first implementation of the Rasch measurement model in empirical legal studies,” Journal of Empirical Legal Studies.
Pleasence, P. and N.J. Balmer (2018), “Measuring the accessibility and equality of civil justice”, Hague Journal on the Rule of Law, Vol. 10(2), pp. 255-294.
Pleasence, P. and N.J. Balmer (2013), In Need of Advice: Findings of a Small Business Legal Needs Benchmarking Survey, PPSR (for the Legal Services Board), Cambridge.
Pleasence, P. and N.J. Balmer (2012), “Ignorance in bliss: Modeling knowledge of rights in marriage and cohabitation”, in Law and Society Review, Vol. 46(2), pp. 297-333.
Pleasence, P., N.J. Balmer, A. Buck, A. O’Grady, M. Maclean and H. Genn (2003), “Family problems: What happens and to whom?” in Family Law, Vol. 33, pp. 497-501.
Pleasence, P., N.J. Balmer and C. Denvir (2017), “Wrong about rights: Public knowledge of key areas of consumer, housing and employment law in England and Wales”, Modern Law Review, Vol. 80(5), pp. 836-859.
Pleasence, P., N.J. Balmer and C. Denvir (2015), How People Understand and Interact with the Law, Legal Education Foundation, London.
Sandefur, R.L. (2016), “What we know and need to know about the legal needs of the public”, Carolina Law Review, Vol. 67, pp. 443-459.
Seymour, R.A., J.M. Simpson, J.E. Charlton and M.E. Phillips (1985), “An evaluation of length and end-phrase of visual analog scales in dental pain”, Pain, Vol. 21, pp. 177-185.
Steele, F. (2008), “Multilevel models for longitudinal data”, Journal of the Royal Statistical Society. Series A: Statistics in Society, Vol. 171(1), pp. 5-19.
United Nations Department of Economic and Social Affairs (2014), Guidelines for Producing Statistics on Violence Against Women, United Nations, New York.
Verdonschot, J.H., M. Barendrecht, L. Klaming and P. Kamminga (2008), “Measuring access to justice: The quality of outcomes”, TISCO Working Paper Series on Civil Law and Conflict Resolution Systems, Vol. 007/2008, Tilburg University, Tilburg.
Notes
← 1. Examples of the content of show cards can be found in Ipsos MORI (2012). Beyond show cards, the World Justice Project’s General Population Poll made use of booklets to present respondents with lists of the justiciable problems asked about.
← 2. In drafting problem descriptions within justiciable problem identification questions, a central concern is to ensure they suggest both the full range of justiciable problems of interest and only justiciable problems. Evidently, as descriptions become broader, risks associated with the former tend to improve at the expense of risks to the latter. In the case of longer surveys, both risks can be addressed by increasing the number of problem descriptions included (at both the problem category and constituent problem type levels). This enables scope to be maintained alongside more rigorous problem specification. However, in the case of shorter surveys, compromise can sometimes be unavoidable.
← 3. 28 of 47 surveys, for which information is available, including all 10 HiiL Justice Needs and Satisfaction surveys and all five instances of the English and Welsh Civil and Social Justice Survey, a survey that involved substantial developmental work over a period of more than a decade. The terms were also used together – along with the terms “abuses” and “crimes” – in the 2009 Bangladesh survey. The terms have also been used separately from each other in a majority (14) of the remaining surveys.
← 4. Only the 2004 Slovakian survey, the 2017-2018 Nepalese survey and the 2017 iteration of the World Justice Project’s General Population Poll adopted a two-year reference period.
← 5. If surveys are repeated more regularly than every two years or the focus is a shorter time span, then a shorter reference period may be appropriate. For example, the panel form of the English and Welsh Civil and Social Justice Survey adopted an 18-month reference period to mirror the period between waves of the survey. Periods longer than two years offer diminishing returns (in terms of maximising problem reporting). However, it may sometimes be necessary to extend a reference period beyond two years to yield the sample size required for proposed forms of analysis. For example, after pre-testing, the 2018 Nationwide Legal Needs and Access to Justice Survey in South Korea moved from a two year to a four year reference period to increase the volume of problem data the survey would yield (Kim and Choi, 2018).
← 6. In the absence of information on problem start dates, the adoption of this rule means that data conveys problem prevalence rather than incidence (as these concepts are understood in epidemiology). “Prevalence” relates to cases existing in a time period; “incidence” to cases occurring within a time period.
← 7. The surveys detailed in Table 1.1 normally incorporated between 10 and 20 problem categories (with 14 to 16 the most common numbers). A small number of surveys incorporated more.
← 8. The 2006 New Zealand survey – which initially asked about only five problem categories – followed-up by individually asking about 92 constituent problem types, when relevant. As most respondents reported no problems in most categories, this approach resulted in a substantial amount of time being saved. The approach was reprised for the 2017 New Zealand survey.
← 9. The reliability of a scale is related to the number of items included.
← 10. For reference, the English and Welsh Civil and Social Justice Survey anchors at the “serious” end alluded to regular physical abuse by a partner and at the “less serious” end to a faulty “moderately expensive electrical item”. Elsewhere, the World Justice Project’s General Population Poll incorporated anchors using the following text: “For example, a score of 2 might be a problem with a neighbor over a noisy party and a score of 8 might be being made homeless.”
← 11. See, for example, Seymour et al. (1985).
← 12. Include the most appropriate term
← 13. In addition, some responses to questions on outcomes are recorded under the first category. Again, this restricts the number of categories available to respondents. However, there are instances in which multiple processes are used within problem resolution, and relying on the ordering of the category list requires both that it be hierarchical (so that if two categories being potentially applicable, the first is always the appropriate choice) and that respondents understand that they should clarify say how the process ultimately brought about the problem outcome. While it may be argued that the categories are hierarchical and that the question clearly indicates that the selected process must have ultimately brought about the problem outcome, the safest route to accurate data is communication of the full category list.
← 14. This question is included here, in preference to a question concerning reasons for inaction, as it more directly links to the example approach to measuring unmet legal discussed in Chapter 2.
← 15. As in the case of the 2010 and 2012 English and Welsh Civil and Social Justice Surveys.
← 16. The 2012 Colombian survey asked about respondents’ satisfaction with processes on the basis of “the money you spent compared to the result you obtained.”
← 17. The appropriate text will reflect survey sponsor interest and responses to earlier sources of help and process questions.
← 18. The appropriate text will reflect survey sponsor interest and responses to earlier sources of help and process questions.
← 19. The 2014 Canadian survey also asked a series of questions relating to legal capability: recognition of legal issues, knowledge of where “to obtain good information and advice about resolving the problem,” knowledge of the type of assistance needed, and knowledge necessary “to deal with the problem.” However, the questions did not closely mirror the aspects of legal capability set out in Chapter 2, nor did they benefit from the greater efficiency of a “grid” style formulation.
← 20. This is in contrast to the alternative approach of asking if respondents are “aware … the problem was related to the law” (as adopted in the 2005 Japanese survey).
← 21. In the case of both recognition of legal issues and legal understanding, general inquiry is complicated by the need for substantial questionnaire space. A range of subject areas must be explored, and sufficient contextual detail must be provided to enable clarity and certainty. While it would be possible to ask a simple question about a respondent’s legal knowledge (such as, “In general, to what extent do you know about legal rights and responsibilities in everyday situations, such as the workplace, accidents, the family, and housing?”), this would inevitably involve self-assessment, and self-assessed legal understanding may differ markedly from actual understanding (Pleasence and Balmer 2012, Pleasence et al. 2015, Pleasence et al. 2017).
← 22. 2010 English and Welsh Civil and Social Justice Panel Survey.
← 23. A more nuanced version of the same question would be, “How much do you know about what each of the following do?”
← 24. In the latter case, see Pleasence and Balmer (forthcoming).
← 25. In addition to the standardised measures of legal capability mentioned above, two standardised measures of attitudes to justice have also been developed, both demonstrating reasonable psychometric properties. One is the nine-item Perceived Inaccessibility of Justice (IOJ) scale, the other the six-item Perceived Inequality of Justice (PIJ) scale (see Pleasence and Balmer (2018).
← 26. For example, in addition to information collected about the identity and relationships between members of households surveyed, the 2010 English and Welsh Civil and Social Justice Panel Survey asked 73 separate demographic questions, including 24 questions concerning respondents’ health.
← 27. Engaging respondents and motivating them to complete questionnaires can be challenging if the subject matter of the survey is not of personal concern. Careful consideration should therefore be given to how surveys can best be introduced to respondents. Sometimes incentives, such as money or postage stamps, are used to boost response rates.
← 28. Steele (2008) presents a common form of analysis; Pleasence and Balmer (2013) give an example in a legal needs survey context.