Home-based in Kazakhstan: National consultant for Causal Machine Learning to Enhance Policy Design (160 working days) - Europe and Central Asia Regional Office (ECARO)

Contract

This is a Consultancy contract. More about Consultancy contracts.

UNICEF works in some of the world’s toughest places, to reach the world’s most disadvantaged children. To save their lives. To defend their rights. To help them fulfill their potential.

Across 190 countries and territories, we work for every child, everywhere, every day, to build a better world for everyone.

And we never give up.

For every child, innovation.

Purpose of Activity/Assignment

The usual approach to policy design roughly follows this pattern:

1. Design and implement a policy
2. Ask, “did the policy work on average?”
3. Expand or adjust the program based on the answer to 2.

While valuable, this approach has limitations. Chief among them is that it struggles at identifying if a policy works differently for one group or another because it relies on averages. Average population effects compounds cases where a policy works better than average for some people with those where it doesn’t work for a different group of people.

There are ways to tease out these differences. For example, we could ask “did the policy work on average for X?”. However, answering this question in a repeated fashion is expensive and time consuming. With this approach, it is operationally impossible to get a complete mapping of how a policy works for all the different groups of people it targets.

Recent developments in machine learning can enable governments to change this paradigm. The most robust evaluation methods rely on random control trials (RCTs) or other quasi-experimental methods to estimate average treatment effects (ATEs). Machine learning enables policymakers to shift to a model that looks at the heterogenous effects of policies in more detail, allowing them to adjust and target policy according to the needs of specific sub-groups. Rather than answer “what works on average”, governments will answer “what works for whom”. This will ideally lead to a more impactful, adaptive, and cost-efficient mode of policy making.

Causal machine learning introduces developments in machine learning to the world of policy evaluation. At a high level, causal forests are composed of decision trees which partition datasets into population clusters. These clusters can then be used to predict and compare the effect of policies at the individual level. This helps policymakers estimate individual average treatment effects (IATE) and group average treatment effects (GATE). Unlike ATEs, these two measures do capture heterogeneity in the effect of a policy across population distributions.

Building on the above, UNICEF, along with the Government of Kazakhstan, aims to explore the use of causal machine learning (ML) to enhance social protection policies. Causal ML employs AI algorithms, statistical techniques, and big data to understand the causal effects of interventions such as government programmes. Leveraging Kazakhstan’s administrative data platforms, the project aims to enhance policy effectiveness, generate fiscal savings, and increase the impact of government interventions.

As such and considering the complexity and novelty of the causal machine learning field, UNICEF, together with the government of Kazakhstan, have identified the need for a national expert to work with the data science team of the Ministry of Innovation to develop a minimum viable algorithm to use causal machine learning to estimate the effect of policies in Kazakhstan. Note that the expert will be embedded with the government’s national team and will be trained and advised by a causal ML international expert.

Scope of Work

The exercise will produce a minimum viable open source pipeline (including data ingestion, algorithm, and analysis of inferred treatment effects) to estimate the causal effect of a policy in Kazakhstan. The model will be trained with government official administrative data. Preliminarily, the government has identified a dataset of about 90 socioeconomic family level indicators, that covers most of the Kazakh population, for the exercise.

The national consultant will be the main focal point for project implementation, will coordinate exchanges with UNICEF’s AI advisor, will help coordinate the Ministry of Digital Development’s team assigned to the project, and will be responsible for the write-up of all final deliverables (github repository, reports, PPTs, monthly progress reports, etc.).

Objectives

The expert will work together with UNICEF’s international causal ML advisor as well as the data science team within Kazakhstan’s “Ministry of Digital Development, Innovations and Aerospace Industry” to:

a. In consultation with UNICEF’s AI advisor, perform quality assurance on the administrative data set to be used for the exercise. b. Select a policy relevant question to build and test a minimum viable model. UNICEF and the government of Kazakhstan have pre-identified some possible questions, such as: (i) What is the effect of cash transfers on children with disabilities? (ii) What is the effect of cash transfers on the transition of youth from education into the labor market? (iii) What is the effect of cash transfers on education completion? c. Lead the development of an algorithm, using open-source tools, that uses causal machine learning approaches to answer one or more such questions. The algorithm should estimate heterogeneous treatment effects and output an analysis of these effects to inform decision making and policy recommendations. d. In consultation with UNICEF’s AI advisor and the ministry, design a plan of action to further increase the use of causal ML by the government of Kazakhstan. The plan should consider technical and political bottlenecks. e. The expert should become familiar with Kazakhstan’s data protection regulations, as well as the EU’s General Data Protection Regulation, and ensure the project complies with both, as well as any relevant ethical standards.

Guiding questions

The exercise will explore the feasibility to use causal machine learning to answer policy relevant impact evaluation questions. Item “a” above describes a few potential questions, but in general terms the exercise will answer the following evaluation question:

a. What is the effect (on average, by sub-group, and individual) of X policy on Y child-related outcomes? The outcomes Y and the policy X will be defined jointly with UNICEF and the Government of Kazakhstan.

In addition, and considering the capacity building nature of the project, the following questions will also be considered:

a. What is the minimum quality of administrative data needed by countries to effectively implement their own causal machine learning model? b. To what extent might quality issues lead to biased predictions and hence biased policy? c. How do causal machine learning models compare to traditional statistical approaches for policy evaluation? d. What role can causal machine learning play in policy evaluation and design? e. What kind of child-related policy questions can be addressed using causal machine learning models? f. What is a replicable and low-cost model for knowledge transfer on causal machine learning to several countries? g. Would AI policy recommendations be welcomed by policy makers? How can AI be positioned as another tool at their disposal instead of a threat?

Methodology

The above-mentioned questions will be answered primarily by the data science team within Kazakhstan’s “Ministry of Digital Development, Innovations and Aerospace Industry”.

The expert’s role will be that of leading the implementation of all phases of the project, including scoping and assessing data quality, defining feasible policy questions, work-planning, supervising the implementation of the causal forests algorithm, analyzing output and debugging, liaising with the international causal ML advisor, and preparing and presenting the final report.

The national expert will also be responsible for keeping version control of the project’s code repository, preparing well documented and clean versions of the code, and preparing visually appealing reports with the project’s findings (in English).

At the project’s onset, and at later stages, the expert will also help organize remote and in-person training sessions for the ministry’s data science team. The sessions will be led by UNICEF’s AI advisor. The expert is expected to attend said trainings and continue to engage with UNICEF’s AI advisor to gain expertise in causal inference throughout the project.

A steering committee with participation from members of the Ministry of Digital Development, Innovations and Aerospace Industry and UNICEF will be established and take an advisory role throughout the whole project.

The exercise will be managed by the Evaluation Specialist from UNICEF’s Europe and Central Asia Regional Office (ECARO) in close coordination with the Child Rights Monitoring and Evaluation Specialist from UNICEF’s Kazakhstan Country Office.

Work Assignment Overview

Tasks / Milestone******Deliverables / Outputs******Timeline / DeadlineProject design including preliminary assessment of data, feasibility of policy relevant question, and training plansIn coordination with AI advisor, short inception report , workplan, analysis plan.15 working days;

1 month after signing contract

Data quality diagnosisAssessment report of data quality, completeness, and availability40 working days;

2 months after signing the contract (and continuously afterwards if needed)

Training in causal MLOrganize and co-facilitate Training workshops (online and offline) which will be led by UNICEF’s AI advisor20 working days;

3 months after signing the contract

Lead development of prototype algorithm and report (approved by steering committee and causal ML advisor)Monthly progress reports and GitHub updates.

Code in github that produces an analysis of the individual level, group level, and aggregate level impact of a preselected policy (focus on children), report with results, quality assurance, compliance with ethical and legal frameworks.

75 working days;

6 months after signing contract

Validate results with non-technical government counterpartsMeeting minutes, brief report with findings10 working days;

8 months after signing contract

Estimated Duration of the Contract

160 working days between July 2024 and March 2025.

Consultant's Work Place and Official Travel

The Consultant will be remote/home-based with no travels foreseen.

Estimated Cost of the Consultancy & Payment Schedule

Payment will be made on submission of an invoice and satisfactory completion of the above-mentioned deliverables. UNICEF reserves the right to withhold all or a portion of payment if performance is unsatisfactory, if work/outputs are incomplete, not delivered or for failure to meet deadlines. All materials developed will remain the copyright of UNICEF and UNICEF will be free to adapt and modify them in the future.

Please submit a professional fee (in USD) based on 160 working days to undertake this assignment.

To qualify as an advocate for every child you will have…

• Master's degree in economics, statistics, computer science, data science, evaluation, political science, public policy, business, and other relevant areas.
• At least three (3) years of experience in the structuring and use of big data sets.
• At least one (1) year of experience in the use and adaptation of AI models (broadly speaking).
• Experience with random forest, decision trees, nearest neighbour, modified causal forests, or other relevant machine learning algorithms.
• At least five (5) years of experience in the structuring and use of big data sets.
• Experience in assessing the quality of big data sets.
• Experience in the design and application of data science solutions for policy relevant questions (or private sector/business use cases).
• Experience with the structuring and use of government administrative records for statistical analysis would be an asset.
• Knowledge in principles of causal inference would be a strong asset (AB testing, uplift modelling, Rubin’s causal framework, RCTs, matching, etc.).
• Fluency in Python is required.
• Knowledge of causal ML would be desirable.
• Working knowledge of English is required.
• Fluency in Kazakh or Russian is required.

For every Child, you demonstrate…

UNICEF’s core values of Care, Respect, Integrity, Trust, Accountability, and Sustainability (CRITAS), and core competencies in Communication, Working with People and Drive for Results.

To view our competency framework, please visit here.

UNICEF is here to serve the world’s most marginalized children and our global workforce must reflect the diversity of those children. The UNICEF family is committed to diversity and inclusion within its workforce, and encourages all candidates, irrespective of gender, nationality, religious and ethnic backgrounds, including persons living with disabilities, to apply to become a part of the organization.

UNICEF has a zero-tolerance policy on conduct that is incompatible with the aims and objectives of the United Nations and UNICEF, including sexual exploitation and abuse, sexual harassment, abuse of authority and discrimination. UNICEF also adheres to strict child safeguarding principles. All selected candidates will be expected to adhere to these standards and principles and will therefore undergo rigorous reference and background checks. Background checks will include the verification of academic credential(s) and employment history. Selected candidates may be required to provide additional information to conduct a background check.

Remarks:

Please include a full CV and Cover Letter in your application. Additionally, indicate your availability and professional fee (in USD) to undertake the terms of reference above. Applications submitted without a professional fee will not be considered. Only shortlisted candidates will be contacted and advance to the next stage of the selection process.

Individuals engaged under a consultancy or individual contract will not be considered “staff members” under the Staff Regulations and Rules of the United Nations and UNICEF’s policies and procedures and will not be entitled to benefits provided therein (such as leave entitlements and medical insurance coverage). Their conditions of service will be governed by their contract and the General Conditions of Contracts for the Services of Consultants and Individual Contractors. Consultants and individual contractors are responsible for determining their tax liabilities and for the payment of any taxes and/or duties, in accordance with local or other applicable laws.

The selected candidate is solely responsible to ensure that the visa (applicable) and health insurance required to perform the duties of the contract are valid for the entire period of the contract. Selected candidates are subject to confirmation of fully-vaccinated status against SARS-CoV-2 (Covid-19) with a World Health Organization (WHO)-endorsed vaccine, which must be met prior to taking up the assignment. It does not apply to consultants who will work remotely and are not expected to work on or visit UNICEF premises, programme delivery locations or directly interact with communities UNICEF works with, nor to travel to perform functions for UNICEF for the duration of their consultancy contracts.

UNICEF offers reasonable accommodation for consultants with disabilities. This may include, for example, accessible software, travel assistance for missions or personal attendants. We encourage you to disclose your disability during your application in case you need reasonable accommodation during the selection process and afterwards in your assignment.