Displacement and discrepancies: the data on global health aid

Donations for global health programs have risen from $5.6 billion in 1990 to more than $21 billion by 2007. Most of this global health aid comes from our contributions as taxpayers to government-based aid agencies like the US Agency for International Development (USAID) and the UK Department for International Development (DFID):

Suppose we donate $1 of our taxes to a global health program. How much should we expect to actually make it into public health or healthcare services in the recipient country? If we look at the financial reports of leading non-profits like Doctors Without Borders, we see that about 85% of each donation is ultimately spent on healthcare (the rest goes to management and fundraising). But if we look at the OECD dataset describing government-based foreign aid, we find that only about 37 cents of every aid $1 given through government agencies actually makes it into the health budgets of recipient countries.

Where does the rest go? Corruption? Bureaucracy? Actually, when we look at the data, we find a strikingly different answer…

Two key studies last year found that foreign aid for health programs appears to be producing a “substitution effect” by “displacing” domestic health spending. That is, when $1 of aid is received for a health program, the government decreases its own spending on health, so that the $1 added in aid only results in a net increase of about 37 cents. The other 63 cents that the country would have spent on health gets shuttled off into the national reserve.

The mechanisms of displacement

Isn’t this a type of corruption? It’s easy to misinterpret this finding as evidence of corruption. But there are actually several explanations for the statistical association between higher aid to governments and lower government spending on health. In particular, over the last several years, several multinational financial institutions purposely instructed government ministers to perform this substitution in health budgets. Perhaps most prominently, the International Monetary Fund (IMF, a global lending agency) instructed nations to divert aid to national reserves, under the premise that such substitution would reduce the likelihood of financial instability, because aid is so unpredictable. In other words, a country would use the aid in place of its usual spending on health, to use when times got rough; some of the money that would have been spent on health is also spent in other sectors that might be underfunded in comparison (of course, this defeats some of the purpose of the aid, which is to increase overall funding available for health by about $1 for each $1 spent in aid).

We empirically tested whether this macroeconomic policy could explain the substitution effect being observed. (Wonkish sidenote: some other authors also tried to do this, but incorrectly correlated overall health spending with overall levels of aid to see if this macroeconomic policy was a factor; that type of statistical approach can produce spurious findings, because different countries will have different absolute aid levels based on numerous political factors. The appropriate statistical model is to analyze whether changes to the existing level of aid produce changes to the existing level of domestic spending on health, such that an additional $1 in aid produces an additional 63 cents of diversion).

Like previous authors, we found that each $1 of new aid is associated with only $0.37 of increased health spending. But then we disaggregated the data and found that there were really two subgroups of spending patterns:

In one group of countries, those that were subject to IMF policies, each additional $1 of aid resulted in less than $0.01 added to the health system on average (nearly complete displacement). In countries that did not borrow from the IMF, however, each additional $1 of aid resulted in about $0.45 added to the health system on average. Furthermore, the average growth of health system spending in non-IMF countries was twice as high as in IMF countries. We found that these results were robust to various time-varying effects and country-specific factors, as described in further detail here.

Caveats to conclusions

Naturally, there are important caveats to this finding. First, we didn’t explain why even the non-IMF countries still displaced many of their funds, and this remains a subject to investigate further. Second, is it the fault of the IMF or of donor countries if aid is substituted? One could argue that if aid were more reliable, the IMF would not advise countries to stash their funds and would allow countries to actually spend that money to build more hospitals and clinics. Alternatively, it is true that other countries not subject to the IMF’s restrictive guidelines have done much better in terms of their health system development when they actually used aid as intended rather than substituting their domestic spending for aid. And the only developing countries to have developed robust public health indicators in the last several decades have done so through further development of the government-based health sector, not through private provision that the IMF has promoted, or through non-governmental organizations.

Nevertheless, there’s a great limitation to the data being used in these and other assessments of global health aid. Frankly, the data just suck. Where global health aid money is coming from and going to is highly unclear; only about one of every three dollars allocated to health can be assigned an identifiable purpose. In addition, the datasets used by prior authors extensively imputed (in the most widely-cited prior analysis, 44% of the data set was missing for low-income countries, and was filled-in by assuming a trend between available data points). It’s often easier to find where a specific agency has donated its funds, such as in this display of theUK’s foreign aid spending:

Recent debates about aid have used this kind of data to make dramatically different conclusions: one group has accused governments of attempting to create “aid dependency” (Moyo) as part of “aid imperialism” (Easterly) that tries to co-opt foreign governments; conversely, others have presented aid as a panacea for chronic social and economic problems (Sachs).

Heat and light

What’s reasonable and unreasonable to conclude about global health aid from the available data? If we look at the data itself, we see a few key difficulties with its analysis and with the claims being made in the ongoing aid debate:

[1] Confirmation bias and homogeneity assumptions: There is such heterogeneity in types of aid (both a small NGO’s medical clinic fundraiser and the World Bank’s initiative to privatize African health systems are referred to as “aid”). Hence, selective use of individual case studies serves as an easy way to argue that aid is either disastrous or life-saving.

[2] Lack of a counterfactual: Would a country have been better off without aid? Many aid critics use theoretical examples of what results from aid (such as Moyo’s fictional Republic of Dongo) but aid is often directed precisely to those countries that are worst off (a type of selection bias)–so more aid could be associated with suffering and disaster, making aid look like the culprit when it was actually helping stave off worse disaster. One statistical approach to the problem is to simulate what would have happened if everything but the aid were ‘held equal’ (a counterfactual analysis), and to make mathematical adjustments for factors associated with aid and outcomes (such as natural disasters), but such measures are never perfect.

[3] Using the Wrong Variable: Most analyses do not assess the effects of aid (the money actually given), but the effects of aid commitments (donor promises to give money in the future), a different – and in some cases, unrelated – variable. The majority of statistical evidence is based on the analysis of aid data in the OECD Creditor Reporting System (CRS) or Development Assistance Committee (DAC) databases, which cover donor aid from 1973 to 2007. Until the mid-2000s, coverage of aid disbursements was very limited. For example, prior to 2003, the estimated mean of country health aid commitments was about ten times higher than the estimated mean of country health aid disbursements. Thus, much empirical research does not actually investigate the effects of aid, but the effects of promises to give aid.  Before 2000, aid disbursements and aid commitments were statistically unrelated variable, as shown in the graph below (the x-axis is disbursements, and both axes are in $US per capita):

[4] Statistical Power and Type-II Errors: One of the first things a student of statistics learns is that data have “noise”. Measurement errors make it harder to detect an effect of one variable on another, should a relationship truly exist. When the signal on a radio is poor, it becomes difficult to hear lyrics. This doesn’t mean the song isn’t playing properly or isn’t in English. While some critics of aid fail to find a relationship between aid and net improvements in GDP or related measures, they do not account for the large measurement errors in the data. This noise in the data risks a “type-II error”, in which the power of the data is not sufficient to identify an effect even if one actually exists. If the real effect is small or localized to a particular community, it will not be likely to show up, and thus our emphasis on finding statistical effects of aid can be unfair to on-the-ground programs with important impact that is not captured by commonly-used statistical variables.

[5] Net Effects and Positive Externalities: One of the greatest errors being made in global health analysis today is to fail to analyze a program’s externalities, both positive and negative. For example, an analysis of HIV programs recently concluded that funds for antiretroviral delivery were of less benefit than educational programs; the analysis ignored, however, the impact of the medication programs on the overall health system, which included the development of medical schools and the education of providers now who treat other diseases. (As a side-note, the analysis also ignored that HIV prevalence declined due to high death rates, not simply because less people were getting HIV).

Foreign aid during a recession

So in the midst of this heated debate, what are we to conclude: should aid be cut or expanded? Ever since the economic crisis starting in 2007, increasing pressure has mounted on government officials to cut foreign aid funds, or to direct them narrowly towards programs that produce good statistical results (a controversy in its own right).

The data available do not clearly reveal what’s worked or failed about aid, and the question of how much aid should be provided is fundamentally still a moral and political choice. One thing that we can conclude from the data is that it is unusual for aid to be cut during economic downturns. Aid has decreased overall as a percentage of income in donor countries, but independent of recessions:

In a recent analysis, we looked more closely at the relationship between economic downturns and the official health aid provided by 15 OECD countries between 1975 and 2007. Specifically, we analyzed the fluctuations in employment, business cycles and per capita gross domestic product (GDP), and related these fluctuations to disbursements of aid. We found no statistically significant association between aid and the state of recession, fluctuations in GDP or changes in unemployment rates in donor countries. In other words, if a decrease in global health aid were to occur during or after the current economic recession, it would have little historical precedent, and claims of “inevitability” would be unjustifiable.

So if significant quantities of foreign aid are to continue being spent in the health sector, we should ultimately try to determine “what works” in terms of which aid programs produce better health outcomes. But the data are so poor, and so much of the aid is being displaced, that the task is proving difficult.

What can we do to improve the data? The Publish What You Fund effort seems to be addressing this issue. The campaign urges donors to disclose their aid information regularly and promptly, in a standardised format that will be comparable with other countries and openly accessible via the Internet. Their Transparency Assessment has already provided novel insights into how open different governments and agencies have been with their ledger books. The next step will be to correlate such data to actual local outcomes, in a way that is meaningful to recipient communities. A prerequisite will be that aid must actually be used instead of simply displacing other funds, and that the aid being given is recorded in real disbursements rather than in promises and press releases.

4 responses to “Displacement and discrepancies: the data on global health aid

  1. Pingback: More on Aid Spending « Health and Development

  2. Pingback: Global Health Aid « The Pulse

  3. Pingback: Does international aid “crowd out” government funding for healthcare? | [ EpiAnalysis ]

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s