Chapter 6. Corrupt influence in textbook procurement in Ukraine

This chapter scrutinises recent reforms in textbook procurement and examines shortcomings which expose the procurement process to the risk of corrupt influence. It identifies weaknesses in recent reforms to the textbook acquisition process that have permitted continued integrity breaches, such as publicising the names of those chosen to evaluate textbooks, exposing them to influence by textbook publishers and authors. Reducing opportunities and incentives for corrupt influence in textbook acquisition can be achieved through improvements to the confidentiality and conflict of interest regulations that govern expert staff in screening textbooks for review, and by providing improved guidance and support to teachers who choose among texts, including dedicated review time and simplified options from which to choose.

Regulatory and policy background

The adoption of new state standards for primary and secondary education in 2011 (Annex 6.A1, ref. 1) triggered changes to curriculum and study plans, and required the gradual acquisition of new textbooks for all compulsory subjects in all years of primary and secondary school. After three rounds of textbook procurement between 2012 and 2014 and the Euromaidan Revolution of 2014, the textbook tendering process was overhauled in an attempt to eliminate concerns over the quality of textbooks and corruption in their acquisition. The new competition procedures were designed to be more participatory, transparent, and abuse-proof (Annex 6.A1, ref. 2), and much of the handling of documentation and communication was transferred into a public system of electronic management of textbook proposals and evaluation procedures. Nonetheless, the procurement of textbooks in 2014-15 using these new arrangements revealed some important shortcomings in the reform implementation. They prompted the Ministry of Education and Science (MoES) to improve the implementation planning for the 2015-16 edition of the tendering process, in time for the procurement cycle for Year 8. The next sub-sections describe the most up-to-date competition procedures for tendering of textbooks in Ukraine. These procedures were meant to guide the procurement cycle in 2016, and they form the basis for the integrity analysis in this chapter.

Summary of the competitive process for textbook acquisition

Since 2015, public procurement in Ukraine is being gradually transferred to a centralised, electronic system (ProZorro) to simplify the procurement procedures and make the procurement process more transparent and resistant to abuse (Annex 6.A1, ref. 3). The procurement reform covers all public sectors, including education, but it does not include the procurement of school textbooks. The procurement of textbooks follows a separate set of rules and has its own trajectory of improvement, with specific challenges.

There are three main stages of a typical procurement process: pre-tendering, which includes assessment of needs, planning and preparation for tendering process; the tender itself, including announcement, evaluation and award; and post-tendering (post-award), the stage at which the contract with the supplier is managed, the order is placed and the payment for the services is processed (OECD, 2009).

These stages describe the competitive process of procuring textbooks in Ukraine as well, which takes place in two phases. In the first phase, publishers are invited to submit an electronic version of their textbook proposals to the MoES for evaluation. The proposals, which must be provided in paste-up layout, are uploaded in an online, publicly accessible Repository. This requirement of paste-up layout is traditional for state procurement for textbooks but tends to be burdensome for independent authors and small publishers who cannot afford to risk money in advance of procurement for layout and illustrations. The MoES charges external teams of experts with the evaluation of the drafts in the Repository and provides a seal of approval to those that correspond to all content and layout-related requirements. In the second phase, teachers are invited to log into the Repository, evaluate the approved textbook proposals and select one per subject for their school. Through its regional education departments, the MoES collects these orders and funds the printing and delivery of the selected textbooks, sharing half of the financial burden with the local authorities (cities and rayons) (Annex 6.A1, ref. 4). Table 6.1 shows an overview of these key steps.

Table 6.1. Key stages in the competition for textbook procurement in Ukraine (2016)
Procurement stage	Textbook procurement	Description
Pre-tendering	Phase I	– Announcing a tender – Setting up competition commissions for each subject – Setting up an appeals commission – Setting up evaluation teams and defining evaluation requirements – Preparation of supporting materials for evaluators and tender participants
Tendering	Phase I and II	– Invitation to tender – Collection of textbook proposals – Evaluation of textbook proposals by experts – Evaluation of approved proposals by teachers – Choice of textbooks by teachers (award)
Post-award	Phase II	– Collection of orders – Order of textbooks and print – Delivery
Source: Orders of the Ministry of Education and Science of Ukraine No. 1359; No. 9; No. 12; and No. 5.

Procedural details and distribution of responsibilities

Pre-selection – preparation

The textbook procurement competitions are initiated and announced by the MoES. Before bidders (publishers) can submit their textbook proposals, the MoES, together with the Institute for Modernisation of the Content of Education (Institute), a body under the MoES authority, issues a list of subjects and names of textbooks that need to be procured for the year. Together with the Academy of Sciences (Academy), a self-governing state research body, the MoES develops the official text of the tender to be published, and jointly with the Institute it prepares methodical-instructional materials that will guide the evaluation of textbooks. All materials must be published on the website of the MoES (Annex 6.A1, ref. 4).

In parallel, the Institute compiles a pool with names of individuals who are willing and able to participate in evaluating the initial textbook proposals. These individuals can be proposed by the MoES, the Academy of Sciences, the Academy of Pedagogical Sciences, and civil society organisations. Each evaluator can evaluate a maximum of two textbook proposals, and each proposal is evaluated by not more than one team comprising two evaluators. The names of experts appointed as evaluators are published on the website of the MoES. The evaluation teams for each textbook proposal are set up by the MoES ahead of the evaluation, and must consist of one teacher, one methodology supervisor and one scientist.

To avoid conflict of interest, the new procurement regulations prohibit employees of the MoES, the Institute for Modernisation of Educational Content, authors/co-authors, and publishers or staff working in the publishing houses submitting the proposals, to participate as evaluators. If an evaluator discovers his/her assignment to a textbook proposal prepared by a bidder with whom there is an affiliation, he/she must, in written form, declare it as a conflict of interest within three days (Annex 6.A1, ref. 5).

Before they can start evaluating, the evaluators must undergo training and receive guidance from the Institute in the form of methodological materials and tools. In 2015, these tools included 14 criteria for evaluating the layout and content of textbooks (Annex 6.A1, ref. 5). In 2016, the criteria were organised in three groups: relevance to the curriculum; relevance to the aims and tasks of education; and criteria to analyse the structural elements of the textbook, such as age-appropriateness of illustrations and text, methodological soundness and gender sensitivity. Further improvements proposed in 2016 stipulated that the evaluation should also verify that the textbook proposals are not discriminatory. The MoES and the Institute are responsible also for the development of methodical-instructional materials to help teachers select their textbooks from the electronic Repository.

Finally, the Institute is responsible for setting up Competition Commissions for each subject, an Appeals Commission, verifying the technical readiness of the electronic systems, assuring the completeness of methodical-instructional materials that will guide the work of the evaluators, and for training evaluators of textbook proposals.

After these preparatory steps are completed, the authors of textbooks are invited to submit their proposals, together with a description of their draft and a full list of people involved in the preparation and submission. The drafts are uploaded in the Repository and remain publicly accessible throughout the process.

Phase I – evaluation

The experts must present the results of their evaluation in the form of a report. The evaluation reports do not have a predefined structure, but they must be detailed, assess the textbook proposals against predefined criteria and conclude with a judgment about the textbook proposal (approved/not approved). Starting from 2016, evaluators are also obliged to provide recommendations on how to improve the textbook proposals that they assess (Annex 6.A1, ref. 4). The evaluation reports must also be certified by the head of the institution where the evaluator is employed. When all evaluation reports are collected, the Institute processes the results and uploads the approved textbooks for schools to choose from in the next phase of tendering.

Phase II – first round (award)

The teachers of subjects and years covered in the tender are given an opportunity to become acquainted with the approved textbook proposals, consult with their colleagues and agree on the choice of textbooks. Their recommendations are communicated to the principal of their school, who fills out a textbook order form in which they indicate which of the approved textbooks in the Repository they wish to order. By selecting a textbook, teachers in practice decide which bidder will be awarded a contract, which is a central decision in the procurement process (OECD, 2009).

Phase II – second round (contracting)

In the second round of Phase II, the textbook order forms are submitted to the city or rayon education departments responsible for the school. The orders are then sent to the Institute, which passes them on to the MoES for the post-award phase, which includes the printing and distribution of the textbooks. The choices of schools are posted on the Repository website.

The special role of competition commissions

The national Competition Commissions are established for each subject or subject area covered to monitor compliance with the rules of the tendering process. The tasks of the Commissions include: verifying that the evaluation experts have submitted their evaluation according to the rules; summarising the evaluation results; and preparing the justification for approval or rejection of the textbook proposal. The Commissions also review the aggregate orders received from the schools through the city and rayon education departments and assess which ones are ordered in sufficient quantities (over 2 000 copies) for print. The size of the order is the only criterion that is taken into consideration; otherwise they are obliged to follow the choices of schools, no matter how diverse these might be. Books ordered below the threshold of 2 000 are not procured at public expense, but parents are free to purchase them privately (Annex 6.A1, ref. 4).1 The work and decisions of the Commission are documented in protocols, which are published on the website of the MoES.

Opportunity to appeal

The Institute for the Modernisation of Content of Education is also responsible for setting up an Appeals Commission. The decisions of this Commission are public and its members are bound by conflict of interest regulations – they cannot be members of the evaluation teams or in any way affiliated with the bidders submitting the textbook proposals. In Phase I of the textbook competition, participants can appeal after the publication of assessment results. The Commission then decides whether there are sufficient grounds to consider the appeal or not. The Commission must take a decision within a day from the submission of a complaint. In case of infringement of rules and procedures, the tender procedure can be cancelled.

A. Description of integrity risk and violation

Public procurement is among the government activities most vulnerable to fraud and corruption (OECD, 2009). A recent anti-corruption monitoring report on Ukraine notes that between 2010 and 2014, about 75% of funds allocated for procurement in that country have been defrauded (UNIAN Information Agency, 2014; OECD, 2015).

The procurement of textbooks is a sector-specific form of public purchase, but it is exposed to similar integrity risks like procurement in other domains of the public sector. In Ukraine it was plagued by irregularities as well. In 2014, the National Accounting Chamber concluded that about 35% of the additional cost of textbook procurement in 2013/14, or UAH 26.8 million, was used inefficiently because the MoES allowed single source procurement (contracts awarded without competition), and failed to safeguard the quality of textbooks and competitive nature of tendering. Other irregularities reported by news organisations included exclusion of qualified bidders by preventing their manuscripts from participating in tenders, problems with the quality of textbook content, inflated prices, and bid rigging in view of awarding contracts to publishing companies close to those in charge of the textbook procurement process.

To make the tendering process more abuse-resistant and restore trust in its outcomes, one of the early actions of the new government in 2014 was to introduce the procurement rules described in the introductory section of this chapter. They were first applied in 2015 for the procurement of textbooks for Years 4 and 7 and revealed weaknesses at the tendering stage of procurement, which affected the integrity of procurement at the stages of textbook evaluation and of selection of textbooks by teachers. The weaknesses put these two key stages of tendering at risk of corrupt influence, and require attention, as described in the next sub-sections.

Risk of corrupt influence on expert evaluators in procurement phase I

Corrupt influence can be manifested in different ways, including qualifying an unqualified bidder, improper contract awards, paying too much or buying inappropriate items, or accepting low quality or non-compliant goods and services, etc. (Kramer, 2012). The evaluation of bids is considered a particularly vulnerable step in the procurement process (OECD, 2007a).

Official statements by the MoES and media reports indicate that problems arose with the process of textbook evaluation during the 2015 procurement. The selection of evaluators and the quality of their judgments led to results that left the door open to public speculations about incompetence and corrupt influence. The Competition Commissions determined that many positive evaluations by experts were not properly justified. Furthermore, only about 10% of more than 180 textbook proposals were proposed for rejection, which the MoES interpreted as a sign that the evaluation had effectively failed its task of screening the proposals. In 2016 only 22 of 199 proposals (11%) were rejected. In an official statement, the ministry acknowledged the failure of the evaluation component of the procurement campaign in 2015 and pointed out that it was due to the unprofessional behaviour of evaluators: “Despite . . . maximum transparency during the competition, there was a manifest problem with the expert evaluations. Parts of the evaluations were both unfair and unprofessional. . .” (MoES, 2015a).

This is an important failure for a reform project that - judging by the supporting materials and legislation - was otherwise carefully planned and regulated. It points towards possible difficulties of ensuring that the evaluators – one of two key groups of players in the new textbook procurement cycle - are of good quality and not susceptible to outside influence.

The problems experienced in the selection process arose, in part, from the design of the process itself. The names of evaluators selected for each subject are made public ahead of the evaluation. This put them at risk of influence by textbook publishers and authors, who could identify who the evaluators of their textbook draft are and try to influence their decisions. Some of the education professionals whom the review team met were textbook authors and publishers, and they suggested that in 2015, publishing houses, especially bigger ones with means and good networks, might have done just that. In the words of one lecturer who is also an author of textbooks in mathematics: “Some publishers were trying to influence in favour of their books, especially those evaluators who were teachers”.

Risk of corrupt influence on teachers in procurement phase II

The newly introduced responsibility of teachers for the selection of textbooks to be procured created another source of integrity risk. The new regulations stipulated that teachers were to be the main actors at this stage of tendering, while the role of school principals and regional education authorities in the textbook selection is limited to safeguarding compliance with selection procedures.

Award decisions are key decisions of procurement that have consequences for the overall fairness of the procurement process (OECD, 2009). They are also an area of multiple risks of corruption which emerge at the stage of evaluation and manifest themselves in the results of the approval and award decisions (OECD, 2007a). The integrity and transparency of decisions at this stage requires proper documentation, but also knowledge, diligence, good criteria for the selection and - most importantly - impartiality.

The site visits for this review included discussions with numerous teachers, in their schools and in teacher training institutes. Some of them admitted to having relied on the opinions and suggestions of third parties in making their choices, including city and rayon education authorities and their methodologists, senior teachers or the principal of their school. In practice, they delegated the responsibility of contract award decision.

For bidders who wished to influence the decision of teachers in favour of a textbook, this meant that the tasks of influencing teacher choices could effectively be focused on school principals and education authorities. In the words of a person who was closely involved in the textbook procurement process, “Publishers reached out to the schools and promised textbook discounts, and tried to exert influence on the methodologists in the education departments, who in turn advise the teachers”. In 2015, decisions under the new reformed process continued to favour the two publishing companies that dominated the market between 2011 and 2014 (Figures 6.1 and 6.2), and which were involved in irregularities criticised by the Accounting Chamber. Some of these irregularities, such as sub-standard quality of textbooks, were decried in 2014 by teachers themselves. Aware of such practices, in 2016 the MoES through its textbook Repository portal published a warning against lobbying by parties not participating in the selection process, for the choice of particular textbooks. The warning reminded participants that it is illegal, and warned against the circulation of lists with recommended textbooks to choose from.

B. Factors that create opportunities for the violation

Deficient provisions on confidentiality and conflict of interest

The desire of regulators to make each step in the textbook procurement process as transparent and participatory as possible is commendable. However, some provisions may have done more harm than good. Specifically, the publication of names of evaluators ahead of the evaluation of textbook proposals opened an opportunity for bidders to identify the evaluators of their textbooks, providing them an opportunity to influence the evaluation process.

A related shortcoming that reinforced opportunity for corrupt influence was the failure of regulations (Annex 6.A1, refs. 4 and 6) guiding this stage of procurement to make the disclosure of conflict of interest mandatory. Conflict of interest regulations required evaluators to declare conflicts of interest present at the time when evaluators were nominated. This excluded conflict of interest situations that might have emerged at a later stage, for instance as a result of experts becoming affiliated with a bidder in the course of evaluation of a textbook proposal. The regulations also failed to establish the liability of evaluators who chose to become influenced by publishers and did not clarify who was responsible for declaring the conflict of interest.

Limited guarantees of independence for teachers selecting the textbooks

The role that the new regulations have given to teachers in the textbook procurement process provides them with the opportunity to select the textbook option that is best for them, which promotes diversity and the relevance of teaching materials (Erickson and Formalont, 1979; Watts-Taffe, 2006).

Box 6.1. The United States’ experience with rules on disclosure of conflict of interest

Amendments were introduced to the US procurement legislation following scandals involving intermediaries and consultant companies that were paid to obtain proprietary information on contracts. Participants in the procurement process must now:

Certify that they have no knowledge of or did not improperly release procurement information.
Attend training sessions and certify attendance of those sessions.
Provide financial disclosure requirements (to rule out conflict of interest). Since this is a voluntary effort, officials can of course withhold information. Companies must also certify that they did not receive nor solicit procurement information. These steps have been helpful to accredit knowledge and build cases regarding corrupt activities involving the improper release of procurement information.

Source: OECD (2007b), Bribery in Public Procurement: Methods, Actors and Counter-measures, https://doi.org/10.1787/9789264013964-en.

By involving its teachers in textbook selection, Ukraine is following the example of many OECD countries. Across the OECD, in 2012 close to 65% of 15-year-olds attended schools whose principals reported that they and/or teachers had considerable responsibility for selecting which textbooks are used in the school. In some countries, for instance the Czech Republic, Italy, New Zealand, Sweden and the United Kingdom, that share was well above 90% (OECD, 2013).

However, there are several reasons why teachers in Ukraine find it difficult to act effectively on the role they have been assigned. First, MoES regulations fail to guide the teachers’ choices towards independence and impartiality. The supporting materials do not outline what types of co-operation, consultation, and support are permissible for teachers, and which ones represent an integrity risk. Furthermore, the involvement of regional education authorities in the collection and processing of textbook preferences opens an informal channel of influence that can jeopardise the impartiality of teacher choices as well. The paper format of submission, as requested in the latest version of the regulations, adds to this problem. It allows education authorities to identify what textbook a teacher has chosen, and hold him/her accountable for that choice, which might not correspond to their interests, expectations and (informal) agreements.

C. Factors that create incentives for the violation

Choice of textbooks as additional workload

A survey of 499 teachers from 2015 revealed widespread dissatisfaction with the production and content quality of textbooks. Some 54% of the teachers surveyed stated that they are not satisfied with the quality of textbooks they work with, and 42% indicated that the improvement of teaching materials is among the three most significant factors for improving the quality of teaching (IED, 2015).

The choice of new and better textbooks is an important opportunity for teachers to improve their work, at least for those who are dissatisfied with the quality of their teaching materials. Teachers who were willing to share their experience with the new opportunity to select textbooks in 2015 admitted, however, that they were overwhelmed by the amount of time and effort required to assess textbook proposals and make an informed choice. They reported a wide gap between the time needed and the time they had at their disposal to read through the manuscripts and reach a decision. For Year 7 for instance, in some subjects the number of textbooks to choose from was as high as 18 (Table 6.2). According to a teacher from a group undergoing professional development in one of the cities visited by the review team, reading through textbook drafts can only take place in the time outside of teaching hours, but this time is increasingly limited because of administrative and volunteer work, and rising demand for student consultations.

Table 6.2. Number of textbook proposals for teachers to choose from in Year 7, by subject (2015)
Subject	Number of textbook proposals
Ukrainian language and literature	18
Literature	13
Russian	10
English	9
History	9
Geography	9
Physics	9
Chemistry	9
Geometry	8
Algebra	7
Biology	7
History	6
Informatics	5
German	5
Spanish	2
Source: Center for Educational Communications (2015), Репозитарій навчального контенту [Repository of Educational Content], http://ua.lokando.com/portal/statpage.php?id=common&page=textbooks-2015 (accessed 15 December 2016).

Research and experience show that individuals who must make complex decisions under constraints rely heavily upon shortcuts or heuristics to simplify their decision problem, taking cues from known and trusted sources of advice (Gigerenzer and Todd, 1999; Nevid, 2013). The experience with the failed evaluation stage of the 2015 procurement round shows that there are limits to the amount of time and effort teachers can realistically invest in such tasks in addition to their regular workload and supplementary activities.

The better the quality and more limited the quantity of textbooks approved for procurement, the more likely it is that the teachers themselves - and not substitutes - will be choosing the new textbooks. If time and cognitive burden persists, teachers are likely to informally delegate their responsibility or accept shortcuts in the form of recommendations from outside their school.

D. Policy options

Closing the opportunities for malpractice

Improve confidentiality and conflict of interest regulations

Integrity in public procurement requires that countries balance public access to information and legitimate confidentiality in the submission and evaluation of tenders (OECD, 2009). The publication of names of evaluators prior to completing evaluation of the textbook proposal is a source of integrity risk, since it creates opportunities for unauthorised contacts and influence. The procurement process should be revised to keep names of evaluators confidential until the evaluation is completed. Evaluators should be publicly associated with the proposals they have evaluated only in case of an appeal.

Furthermore, a mandatory disclosure of conflict of interest among evaluators should be introduced. This will ensure that evaluators remain liable for their affiliations even if these emerge after the nomination, in the course of textbook evaluation. It is also desirable to ensure that there are clearly defined consequences for non-compliance, such as the exclusion of publishers who violate these rules from the tender.

Provide comprehensive guidance on the selection of textbooks by teachers

Since teachers are provided with limited guidance about what counts as permissible or acceptable practice, it would be useful for the Institute to develop (and the MoES to approve) guidance for teachers on the integrity of the textbook selection process, which defines what co-operation, consultation and external support is permissible, and what should not be allowed (e.g. informal lists with “best of” books to choose from), as well as the consequences in case of non-compliance. In addition, the submission of choices by schools should be done in electronic form directly to the central site managing the procurement of textbooks instead of in paper form through the local education authorities. This will help to limit the leverage of local education authorities and the number of potential sources of undue influence.

Eliminating the incentives for engaging in malpractice

Create dedicated teacher time for textbook review

While engaging teachers in textbook selection is, in principle, good practice, teachers must be appropriately supported in meeting this responsibility. To that end, the MoES should expand Phase II of the textbook procurement procedure, by decreeing that the choice of textbooks takes place in the form of a fixed selection period, with a firm starting and ending day. The purpose should be to create dedicated time for the selection of textbooks that does not overlap with the other, regular obligations of teachers and provides them with sufficient time to focus on the important task of selecting their textbooks. Selections could take place once a year until 2019, outside of the time allocated for teaching in class, last longer than was the case so far (for instance, several weeks) and be scheduled to deliver results in time for the timely procurement of the textbooks.

Present teachers with feasible choices

With the help of the Institute for Modernisation of Educational Content, the MoES should invest in improving the effectiveness of evaluation of textbook proposals. They should ensure that the number of approved textbooks from which teachers choose is feasible to cope with within their time and capabilities. This could be achieved for instance by setting a maximum number of textbook proposals per subject that can pass the evaluation process. At the same time, introducing a more differentiated system of evaluation criteria would help to justify the ceiling. The maximum number of proposals per subject should be commensurate with the time that teachers are realistically expected to have during the selection campaign.

Box 6.2. Aspects of teacher participation in textbook selection in the United States evaluation committees

In the USA, most of the work of evaluating textbooks is done by evaluation committees. In adoption states and in large school districts, there usually is an effort - at least a purported one - to compose a committee that is representative of the state or district as a whole, although it appears that committees often are homogeneous. Research [. . .] suggests that ideal committee membership includes teachers who are knowledgeable in the subject matter content and pedagogy, from diverse ethnic backgrounds, and educated and experienced in teaching a wide variety of students. Parent participation on the committee would represent the diversity within the student population.

Compensation

The volunteer status of the committee is not commensurate with what often is an overwhelming task. Providing payment for committee service might enhance the pool of teachers interested in participating and might increase the quality of their participation.

Evaluation criteria

Criteria for evaluation vary widely. Most school districts incorporate a uniform evaluation form, checklist, or rubric, while others allow evaluators to frame their evaluations entirely on their own. Most sets of criteria for evaluation focus on accuracy of content; scope, sequence, and pacing of content; usability for teachers; and alignment with state or district curriculum or testing benchmarks. It is recommendable to use evaluation tools that require more than a checkmark. In some cases, evaluation forms are designed to require reviewers to comment or rate materials on quality. In others, reviewers are checking for the existence of certain characteristics rather than the quality of these characteristics. Forms also vary in level of detail. Some provide general guidelines left to the individual interpretation of committee members, while other forms are more specific. The more powerful tools require the reviewer to rate or write.

Time for preparation and review

Preparing the reviewers to evaluate textbooks effectively is critical. It is not uncommon for evaluation committee members to have no prior experience and receive no formal training. Training should not be limited to an orientation. It also should include a review of the latest research in the subject of study. It is also imperative that sufficient time be provided for the review to enable reviewers to read and evaluate entire units of study. It is better for individual reviewers to evaluate small sections in depth and “planfully” than for reviewers, overwhelmed with the charge of reading several complete texts, to engage in the “flip test”. Of course, the ideal is for individuals to read entire texts in a systematic manner.

Source: Watts-Taffe, S. (2006), “Textbook selection and respect for diversity in the United States” in E. Roberts-Schweitzer (ed.), Promoting Social Cohesion through Education: Case Studies and Tools for Using Textbooks and Curricula, World Bank, Washington, DC.

References

Center for Educational Communications (2015), Репозитарій навчального контенту [Repository of Educational Content], http://ua.lokando.com/portal/statpage.php?id=common&page=textbooks-2015 (accessed 15 December 2016).

Erickson, L. and M. Formalont (1979), “Criteria for the evaluation of reading materials”, in T. Hatcher and L. Erickson (eds.), Indoctrinate or Educate?, International Reading Association, Newark.

Gigerenzer, G. and P.M. Todd (1999), Simple Heuristics That Make Us Smart, Oxford University Press, Oxford.

IED (2015), Середня освіта в Україні: думка вчителів та батьків [Secondary Education in Ukraine: Attitudes of Teachers and Parents], Institute for Education Development, Kyiv, http://iro.org.ua/ua/main/research/23.

Kramer, M.W. (2012), The Most Common Procurement Fund Schemes and Their Primary Red Flags, IACRC (International Anti-Corruption Resource Center), Washington, DC, http://iacrc.org/procurement-fraud/the-most-common-procurement-fraud-schemes-and-their-primary-red-flags/.

MoES (2015a), Звернення Міністерства освіти і науки України до вчителів, які беруть участь в конкурсному відборі підручників для 4-х і 7-х класів [Address of the MoES to the Teachers Participating in the Competitive Selection of Textbooks for the 4th and 7th Graders], Kyiv, www.mon.gov.ua/usi-novivni/novini/2015/03/30/zvernennya-ministerstva-osviti-i-nauki/.

MoES (2015b), Результати конкурсного відбору підручників для учнів 4 та 7 класів загальноосвітніх навчальних закладів станом на 13 травня 2015 року [Results of the Competitive Selection of Textbooks for Students in Years 4 and 7 of Secondary Schools as of May 13, 2015], http://mon.gov.ua/usi-novivni/povidomlennya/2015/06/12/rezultati-konkursnogo-vidboru/.

Nevid, J.S. (2013), Psychology: Concepts and Applications, Cengage Learning, New York.

OECD (2015), Anti-Corruption Reforms in Ukraine: Round 3 of Monitoring of Implementation of the Istanbul Action Plan, OECD Publishing, Paris, www.oecd.org/daf/anti-bribery/Ukraine-Round-3-Monitoring-Report-ENG.pdf.

OECD (2013), PISA 2012 Results: What Makes Schools Successful (Volume IV), OECD Publishing, Paris, https://doi.org/10.1787/9789264201156-en.

OECD (2009), OECD Principles for Integrity in Public Procurement, OECD Publishing, Paris, https://doi.org/10.1787/9789264056527-en.

OECD (2007a), Integrity in Public Procurement: Good Practice from A to Z, OECD Publishing, Paris, https://doi.org/10.1787/9789264027510-en.

OECD (2007b), Bribery in Public Procurement: Methods, Actors and Counter-measures, OECD Publishing, Paris, https://doi.org/10.1787/9789264013964-en.

UNIAN (2014), Янукович и Ко разворовывали 150 млрд гривень ежегодно с помощью госзакупок Подробности читайте на [Yanukovych and Co. Looted 150 Billion Hryvnia per Year through Public Procurement], Ukrainian Independent Information Agency of News, http://economics.unian.net/finance/960764-yanukovich-i-ko-razvorovyivali-150-mlrd-griven-ejegodno-s-pomoschyu-goszakupok.html.

Watts-Taffe, S. (2006), “Textbook selection and respect for diversity in the United States”, in E. Roberts-Schweitzer (ed.), Promoting Social Cohesion through Education: Case Studies and Tools for Using Textbooks and Curricula, World Bank, Washington, DC.

ANNEX 6.A1. References of legal sources

Order of the Cabinet of Ministers of Ukraine No. 1392 of 25 November 2011.
Order of the Ministry of Education and Science of Ukraine No. 1149 of 7 October 2014.
Law of Ukraine on Public Procurement No. 9 (922-VIII) of 15 December 2015.
Order of the Ministry of Education and Science of Ukraine No. 1361 of 25 December 2015.
Order of the Ministry of Education and Science of Ukraine No. 9 of 12 January 2015.
Order of the Ministry of Education and Science of Ukraine No. 5 of 6 January 2015.

References cited as sources of Table 6.1:

Order of the Ministry of Education and Science of Ukraine No. 1359 of 10 November 2016.

Order of the Ministry of Education and Science of Ukraine No. 9 of 12 January 2015.

Order of the Ministry of Education and Science of Ukraine No. 12 of 14 January 2015.

Order of the Ministry of Education and Science of Ukraine No. 5 of 6 January 2015.

Note

← 1. According to a proposed amendment of the regulations, the threshold shall be raised to 40 000 nationwide orders per book.