As M2 students, I assume that you are already proficient with the exercise of reading and discussing a research paper. Well-written journal articles are structured to help the reader understand the main scientific issue, methods, data and results. Here, the exercise is somewhat different. On the one hand, you may not be working with research article. Briefcase reports, institutional reports, case studies, etc. do not go follow the same process of peer-review, and face different editorial constraints. Here is a set of question you should be able to answer at the end of a document — if only to say that the information was not present.
This framework should also help you designing your own behavioural projects.
10.1 Describing an intervention
In what I call the analytic part, the focus is on the intervention itself: what has been done, to whom, and with which results? At this stage, the goal is to collect readily available information about the intervention – what is said explicitly in the document you have. A good description should specify:
Why? What are the stated objectives of the intervention?
How? How was the intervention conducted?
Who? This is the target population: who are the people whose behaviour has been influenced? The level of detail should be relevant to the understanding of the intervention. For example, are age, ethnicity, neighbourhood type stated as important factors in the behaviours or the intervention itself?
Where? Most interventions are conducted first on a reduced perimeter. In terms of scalability, it begs the question of the comparability of this perimeter to other prospective ones. The description should highlight the features which support or undermine a reproduction of the intervention elsewhere.
When? Similarly, early experiments often have a narrow time frame. The consequences of this time frame should be made explicit. For example, an intervention based on outdoor sports practice may not rely on the same sports and infrastructures in winter than in spring.
This part provides the basic understanding of an intervention: with these elements, you can produce an elevator pitch of the intervention, and prepares you for the second part.
10.2 Analyzing the intervention
This second part is a deep dive into the intervention. Here, what is not said can be as important as what is said. Missing information should not be ignored, it should be tagged as missing, since this would be critical points if you had to reproduce the intervention.
10.2.1 Policy issue
This section describes the policy case for an intervention. Specifically, it should be expressed in terms which would be familiar to a policy-maker or a civil servant: behavioural concepts will be used later. The goals here is to understand the problem as it was presented to the BI team.
What is the goal of the policy? Usually, an intervention starts with the failure to achieve a given policy objective. You should be careful to chain the policy objective from large, overarching goals of public action to the precise policy involved. It helps avoiding a confusion between the means and the ends. For example, promoting general health is the general objective of the policy behind graphic warnings on cigarette packets. The intermediate objective is to reduce cigarette consumption.
What is the actual behaviour? By definition, a behavioural intervention targets a behaviour which is deemed detrimental to oneself or to others, with an impact sufficient to warrant public action. Try to describe as precisely as possible the set of behaviours the intervention attempts to change.
What is the intended behaviour? In most cases, and intervention will be about behavioural change. Thus, what is the new behaviour which is expected after the intervention? In the smoking example, it is to have people stop smoking (altogether), smoke less or avoid starting smoking?
10.2.2 Whodunit?
This section identifies the involved actors, both from a policy and research perspective.
Who asked for it? Even if some interventions are conducted by researchers at their own initiative, most originate from a third party, politician, administration, or non-profit institution, who provides both the problem and the funds.
Who conducted the intervention? The identity of the team conducting the intervention provide clues as to why some choices were made. For example, academic-laid interventions will have a strong incentive to provide novel, publishable results, since these are the success metric in academia. It will likely push towards new solutions and insist on a very robust identification strategy for the impact, usually randomization. Other teams who face cost efficiency constraints may prefer more tried-and-tested approaches, transposing what has been done elsewhere.
How did they publish their results? Often, the document you’ll have will not be the only support through which the intervention was described. Briefcase papers often have companion academic articles (including at the working paper stage) which may provide useful additional information. Conversely, a companion policy brief may help you understand an intervention you discovered first in an academic paper.
10.2.3 Diagnosis
The goal of this section is to understand how the behaviour described in the descriptive part was modelled by the BI team.
What are the structural obstacles to the desired behaviour? By “structural”, I understand here obstacles to the desired behaviour which do not depend on people changing their behaviour. They can be economic or practical constraints. For example: why do people use their cars to commute in most middle-sized cities in France? Lack of dedicated biking lanes and expensive and unreliable public transport are structural factors.
What are the behavioural obstacles to the desired behaviour? This steps translates behaviour described in laypeople terms into behavioural science concepts. From example, the fact that most people select the middle option when a menu of three sizes is offered, even if it is in fact larger and most expansive than what they’d have selected with another menu (think Starbucks) is a framing effect.
What are the behavioural levers used? Similarly, this step describes in behavioural terms the concepts leveraged to change behaviours. Communicating that 85% of students are on time for class is an example of social norm use.
How similar it is to well-known interventions? Unless you are doing a literature review, the idea here is to gather the similar interventions mentioned in the document, as well as reference ones you already know through your training and experience. The key point is to identify the key differences from other similar interventions, and why the team had to make this adaptations.
10.2.4 Intervention
This step describes the intervention itself, in a way which links the diagnosis to what has been actually done. If the previous section moved from laypeople notions to behavioural ones, this section takes the inverse path: from the behaviourally-expressed solution to the practical implementation of it.
How did they implement their intervention? In this step, imagine you are providing instruction to civil servants on how they could run the same intervention, with a careful attention to practicalities. Going back to the student punctuality statement example: how was it broadcast? Teachers during their classes, posters, mailing campaigns? Who sent the mails, a generic institutional account, the president of the institution, a well-respected academic, a student body? At which point of the term? Was it sent to the university mail accounts, or did the students specify a preferred mail account?
How well the intervention matches the diagnosis? Most of the time, practical constraints entail some degree of adaptation in the implementation of the intervention. This step highlights the gaps from the theoretical intervention and the actual one which was actually run. Keeping with the same example, lack to access to the mail addresses for students from partner institutions may have reduced the scope of the mailing, or a loss aversion element may have been included in the wording of the message without it being present in the initial design.
10.2.5 Impact assessment
Let us be frank: there is a tendency in our field (and many others) to overstate claims of impact in publications. Without going into actual malpractices and fraudulent data, even good papers may present their results in the most flattering light possible, since it demonstrably increases the chances of publication. You should therefore be very critical in your reading of the impact assessment.
How robust if their identification strategy? I basically assume that you know enough about inferential and causal identification to assess if the identification strategy described corresponds to the causal claims made in the paper. Specific attention should be paid to the quality and granularity of the randomization (if a RCT), control of spillover effects or the quality of the natural experiment used.
How replicable is it? Replicability means here the possibility to obtain the same results from the same data. Is the dataset available? The codes for producing the results? Was it pre-registered?
How strong and plausible are the effects? The current replication crisis has highlighted how often results too good to be true were indeed just that. You should try to assess how large the effects are relative to other, comparable experiments (ideally, this kind of benchmark should be in the paper), and how confident you are that they are in a reasonable ballpark.
How long do the effects last? Long-term behavioural change is difficult. Too often, follow-up is limited due to lack of funding or loss of interest from third parties who are content with short-term results. Thus, information about longer-term impacts is especially valuable.
How scalable is the intervention? Most documented intervention are intended to be prototypes: small in scale, to be followed by a larger implementation. In practice, scalability is a issue, with the intervention relying on voluntary participation from civil servants or the public, or leveraging a specific training which would be costly or complicated to provide to a large number of people.
Even in such an academic exercise, we are as subject to behavioural biases as any other Human. It can thus be a good idea to separately rate how sympathetic or concerned you feel about the policy issue, and how attractive you find the document or the writing. It has been documented that making thus kind of separate assessment may help limit the halo effect of your analysis (Kahneman et al. 2021Kahneman, Daniel, Olivier Sibony, and Cass R. Sunstein. 2021. Noise: a flaw in human judgment. William Collins.).