16 July 2020
Marta Monterrubio, a specialist from the Triangular Cooperation project for Public Policy Evaluation in Latin America and the Caribbean, tells us about this discipline in the context of COVID-19
As part of the EVALÚA project led by the FIIAPP and financed by the European Union, a methodological guide to the experimental and behavioural economics methodologies applied to public policy evaluation has been prepared. The author of the guide, Diego Aycinena Abascal, a professor at the Universidad del Rosario in Bogotá and a visiting researcher at the Economic Science Institute at Chapman University in Orange, has compiled some of the most significant findings of behavioural experiments since the discipline was created 50 years ago, with the aim of making these easy for decision-makers to access and use.
What is the impact of biases and non-standard preferences during the COVID-19 crisis?
Classical political economy holds that human beings are what is known as “homus economicus”, which basically means that we make decisions by weighing up the cost and the benefit of our actions, so that the latter outweighs the former. However, since the mid 20th century, experiments have shown that the courses of action that we choose tend to deviate from this model.
Non-standard preferences, non-standard beliefs and non-standard decision-making
These concepts show that human beings do not always behave in a rational manner, because we are influenced by biases, intuitions and false beliefs. One very common behaviour is loss aversion (bias) which implies we would rather not lose now than gain more later. Myopic loss aversion is a combination of the above with risk, and can prompt us to act in ways that lead to a negative medium-term result. Our perception of loss is completely different to our perception of profit when a risk is involved, as shown by the theory of the prospects. When faced with a risk, we would rather not lose now rather than make a profit later. Otherwise put, when faced with a potential loss, we take risks, but when presented with a possible gain, we look for security.
Present bias and self-control problems involve inconsistent behaviours and show that our willpower has limits. We might make a decision and then put it off more or less indefinitely. This frequently manifests in spending money to make up for a lack of motivation or action, for example, by buying “miracle products” instead of starting the diet we had planned.
Observation of social norms suggests that when choosing a course of action, we consider more than the potential losses and gains. We also consider whether the action is in line with what our peers (our social circle) tend to do and consider appropriate in such a situation. Because of this, our decisions are influenced greatly by the societies in which we live, which can weigh more than result of the cost/benefit equation.
Hindsight bias is the tendency for people with knowledge of an outcome to falsely believe that they would have predicted the outcome of an event, exaggerating the similarity between their ex-post beliefs and their beliefs prior to an informational event.
Prominence preference, although an irrational strategy, steers people to choose the most striking option or one that stands out for spurious reasons, such as aesthetics or a prominent physical position (supermarket shelf, for example).
Ultimately, many of these phenomena (self-control issues, social preferences, social norms, over-projection of preferences) can be chalked up to emotions and feelings. Psychological literature reveals the role of emotions as a mechanism that mediates our behaviour. However, these psychological findings have recently begun to be incorporated into public policy.
Biases, false beliefs and non-standard preferences during the COVID-19 crisis
They influence our decisions and daily lives more than we think. For example, since the pandemic appeared we have seen loss aversion bias influence many governments’ decision-making, particularly in the beginning, when they were still ignorant of the magnitude of what lay ahead. Many put off imposing draconian isolation measures for fear of economic hardship, failing to grasp that the initial losses would avoid having to pay a high price later on. This is closely related to the prospect theory, which includes the previous bias in a risk situation. Other biases like optimism and the illusion of control have also been common, causing many people to resort to futile remedies, pseudo-science and superstition, and the spread of an enormous number of hoaxes and lies about the pandemic.
The hindsight bias stated above is frequent, among the multitude of opinions around us, to the extent that one might be forgiven for thinking that almost everyone already knew what was going to happen and which would be the best decisions to make right from the start.
Social norms have clearly influenced our behaviour during confinement, with non-compliance or stricter compliance with the rules conditioned by what has been happening in the immediate environment (family, neighbourhood, municipality).
We have also seen some political or social leaders urging disobedience with regulations, questioning their effectiveness or legitimacy, albeit on a more psychological than economic level. Authority bias makes people predisposed to believing that if an authority gives us permission to break the law and even cross the moral line, we feel prone to do so, which has recently happened in certain cases.
The optimistic bias leads us to project our own wishes onto objective data. As already mentioned, which has been observed during the de-escalation. We tend to think that the risk is lessening, causing people to stop taking precautions which are clearly necessary if we analyse the data carefully.
These are just a few examples of how we behave irrationally and take irrational decisions, both large and small. Understanding this irrationality and the mental shortcuts we make can be helpful when we make personal decisions, but it is crucial for public policy decision makers.
What are nudges?
Nudges are behavioural statements frequently used to promote public policies. Nudges are designed to influence our decisions by modifying the decision-making architecture, without substantially modifying incentives or restricting options using premises of behavioural economics.
Nudges have become extremely popular because they are suitable for making low-cost statements based on soft or libertarian paternalism without resorting to prohibitions or restrictions. There are several successful documented cases of statements using nudges, for example, to encourage organ donation, quit smoking through commitments, reduce OR deaths with check-lists, improve loan repayment rates with personalised reminders, increase compliance with tax payments, among others.
However, nudges must be designed carefully, otherwise they may be ineffective, counter-productive or used for other purposes. Some of these biases may lead us to overestimate the likelihood that nudges will be a success. The statements that lead to success get the most attention and those that fail are often ignored or forgotten. In addition, it has been seen that the effects of some statements are short-lived and the lack of long-term improvement is ignored. Therefore, the likelihood of success of a behavioural statement or a nudge is only as good as the behavioural foundation on which it is based.
Behavioural experiments applied to public policies.
Behavioural experiments have been widely used in consumer affairs, political marketing, investment and finance, and other fields. Their application to public policy, in terms of design, management and evaluation, is far more recent. However, there is some experience that allows us to extract some important learnings:
- Experimental data is replicable insofar as it allows knowledge to be built based on previous findings. This facilitates a cumulative and systematic process of experimental learning. Consequently, a store of generalised knowledge is built that supports the design of public policiesand makes them relevant and viable.
- The findings of these experiments allow us to observe variables directly, something that would not be possible otherwise. For example, observing antisocial behaviours that people would try to conceal outside the laboratory (it would only be possible to observe complaints), such as taking money earned by third parties, or filing false income statements to avoid taxes and obligations.
- Field experiments can be used in an enormous number of subject areas, for example, they have been successful for evaluating actions aimed at increasing electoral participation, measuring corruption in educational qualifications, measuring reductions in water or energy consumption and to increase recycling and determine responses to improve tax compliance.
- There are different types of experiments. By their nature, laboratory, artefactual, and framing experiments are cheaper and have a comparative advantage in helping us understand mechanisms and giving us valuable information before implementing a policy or programme through scale testing. Experiments in the natural field have a comparative advantage for evaluating the impact of previously implemented policies and programmes, and for accurately measuring their effects on the specific population of interest in its natural context. Their approaches can be complementary, as has been seen in combinations of artefactual field experiments with natural field experiments.
The Methodological Guide is a tool available to institutions to help select the most suitable type of experiment and for dissecting the roadmap for implementation, defining the steps for development and describing the advantages and limitations.
17 October 2019
Marta Monterrubio, Public Policy Evaluation specialist from the Evalúa project, tells us about the policy evaluation carried out after the earthquake that took place in Ecuador in 2016
When it comes to preparing the National Evaluation Agendas, different goals come into play: accountability, policy improvement or programme evaluation (its design or management), transparency promotion as a democratic tool, and ultimately institutional or managerial learning.
In terms of Disaster Risk Management, this evaluation is relevant to all of these matters. It is also a particularly sensitive matter: in addition to exposing the major vulnerabilities that afflict a large part of the world’s population, there are known cases of regrettable deficiencies in fund management for emergency and reconstruction. The unanimous opinion of specialists, also included in the Sendai Framework, is that in terms of disaster risk having a solid prevention system helps prevent the loss of human lives, as well as material loss and the loss of basic goods for population survival. It will also make a difference when facing subsequent reconstruction.
On April 16, 2016, an earthquake of magnitude 7.8 (Mw)3 was recorded on the north-east coast of Ecuador. 671 people died and 6,277 were injured. The damage affected four provinces, and fourteen cantons were declared to be in a state of emergency.
After assisting the first moments of the emergency, the Ecuadorian government approved the 2016 Post-Earthquake Reconstruction Plan, framed within its Risk Management regulations and in the National Decentralized Risk Management System (SNDGR). The Reconstruction Plan’s aim is territorial recovery, canalise the reconstruction and recovery processes of post-earthquake livelihoods under the criteria of resilience and sustainability through intersectoral and multilevel coordinated interventions.
What has emergency and reconstruction assistance coverage been like for the population in the affected areas? How many families benefited from this asisstance and for how long? How and in what way were the shelter, rental and food aid distributed? Were the most vulnerable people included? To what degree was infrastructure rehabilitated? How many public health facilities were rebuilt and rehabilitated? What is the degree of citizen satisfaction with regard to medical care and services? Is Ecuador’s National Decentralized Risk Management System working as well as it could to prevent and manage disasters of this nature? What improvements should be made to minimize the consequences of possible future disasters?
These are just some of the questions that the evaluation will answer: Transparency, improvement, learning.
The consulting team hired by EVALÚA has already completed the field work. In the next few weeks we will have the answers.
18 May 2017
Public policy evaluation in Latin America.
Some time ago I had the chance to talk to an investigation official of the national police of a Latin American country at a meeting I was invited to attend as an evaluator.
A great deal is being done in this area through the technical support of institutions, including Spanish ones, whilst scrupulously avoiding the application of “models” from other countries to a national reality that is totally foreign and extremely complex. The United States, for example, has focused particularly on the issues of drugs and youth gangs. In Spain, we have great experience, and here I can mention, for example, the work done in Guatemala to reduce the number of crimes against life and amount of violence against women.
The official I was speaking with said that, in his country, the police are under great stress, that they’ve been doing their job under great pressure for several years, and that many of them would like to leave the field and get a university degree or do something else or, at least, have a better system of shifts that would give them more down time.
He also told me that they had learned that a small number of criminals commit the vast number of crimes, and that if there are, say, 10,000 homicide victims annually, these could be attributed to some 1,000 aggressors. That is, with a little investigation, with good police instincts and better coordination between prosecutors, judges and law enforcement, the problem of citizen insecurity could be handled much better.
“Don’t leave us on our own”, he said, worried about the low level of support they had internally and the idea that these issues could cease to be a cooperation priority. That’s probably the way things are seen by some people who are tired of supporting an institution that we can characterise as weak-strong-absent-present.
Weak in terms of the strategic direction, given that many bodies work on the same issues in an uncoordinated way: criminal investigation, prosecutors, judges, prisons, institutions that fight against organised crime… Strong, because at times these institutions think the only acceptable answer for society is the incarceration of young people; this accomplishes nothing except filling up prisons, where, on the other hand, there are few rehabilitation programmes to help them abandon the cycle of violence. Absent, because the police and judges are never where they’re needed, and they don’t reach many places where criminals are still being lynched. And, lastly, present, because the government is eager for media coverage and to demonstrate that citizen insecurity is its great priority, without us knowing what that actually means.
Public policy evaluation
That’s why it’s necessary to improve our diagnoses by using tools like public policy evaluation to be able to improve programmes and address these challenges that generate such contradictory responses from institutions.
And we have an extensive experience working with many institutions to be able to introduce evaluations as a requirement in implementing cooperation programmes for institutional reform. And we have also generated the necessary trust, which will enable us to conduct not only qualitative analyses but also quantitative ones, with baselines and surveys after several years.
At first glance, it seems easier to quantify the impact of aid in the case of social programmes or poverty-reduction programmes, which are where these types of evaluations have traditionally been done. However if we observe carefully from the institutional standpoint and, more specifically, from one of capacity building, there are many interesting options.
If the programmes are designed from the start with the objective of being evaluated, we could predict their impact on police performance; to give an example, the impact of one type of training or another, and eventually, the impact this has on institutional change and crime reduction, which is the important thing. It would also be a great advance, and we’re capable of doing it, to learn what effect the training of criminal investigation units has on the impunity variable. That is, in terms of how much our cooperation helps in terms of solving cases and the ability to bring them to trial.
That way the institutions themselves will be more aware of the limitations and opportunities that exist for carrying out larger-scale reforms and for gradually building a new culture of institutional development.
Miguel Angel Lombardo works at FIIAPP on South-South cooperation for public policy evaluation in Latin America.