By William Church
Director, GLCSS
 

 

The following dialogue is in response to World Bank Governance Report Blunders in Central Africa by the Great Lakes Centre for Strategic Studies Director William Church. The World Bank response is unedited except for limited excerpts used by William Church in his reply. GLCSS urges the reader to examine the excerpted text to ensure the reply is in context.
 

The original article:
 

World Bank Governance Report Blunders in Central Africa
http://www.bloggernews.net/2006/10/world-bank-governance-report-blunders.html

World Bank Comment: William Church, Director of the Great Lakes Center for Strategic Studies, criticizes the latest release of the Worldwide Governance Indicators (WGI) for making various “blunders” in its rating of countries in Central Africa.  As one of the principal authors of this study I want to disagree with Mr. Church’s assessments of the data we produce.

Together with Daniel Kaufmann, Massimo Mastruzzi, and in earlier installments Pablo Zoido we have been producing aggregate indicators of six dimensions of governance for virtually all countries in the world, since 1996.  They measure (1) Voice and Acountability, (2) Political Stability/Absence of Violence/Terrorism, (3) Government Effectiveness, (4) Regulatory Quality, (5) Rule of Law, and (6) Control of Corruption.  The aggregate indicators, the underlying data on which they are based, a graphical interface, and complete documentation are available on the web at www.govindicators.org.

Mr. Church questions the WGI because of how it ranks some countries relative to others.  He questions for example why the WGI assigns higher scores to the Central African Republic on Political Stability/Absence of Violence/Terrorism than it does for Kenya or Uganda.  

What Mr. Church fails to note is that the differences between the countries he identifies are actually very small relative to the margins of error that we transparently report for all countries.  As we stress repeatedly in all of the documentation of the data, any effort to measure different dimensions of governance will involve an element of uncertainty.  What is unique about the WGI is that we allow users to take this uncertainty into account, through our reported margins of error.  Had Mr. Church heeded our warnings and considered these margins of error, he would have realized that one should not interpret any of the small differences he identifies between countries as being at all practically meaningful.  In fact, these margins of error are hard not to notice, as they appear automatically in all of the graphs available on our website, as well as in all the data tables that users can download.

William Church Comments: Ironically, the World Bank actually supports the GLCSS argument when they reply “the differences between the countries he identifies are actually very small relative to the margins of error that we transparently report for all countries.”
 

This is the exactly the point of GLCSS criticism. There are very large differences between Uganda and the Central African Republic and the Central African Republic and Rwanda. Their statistical results even with a margin of error factors should have revealed these differences.
 

GLCSS recognizes their point regarding margin of error but the reality is whether the Central African Republic ranked higher or the same it still supports the position—which the World Bank agrees with—that there is a need for more reference points.  Both Rwanda and the Central African Republic had limited reference points and this caused a distortion in the research. The practical assessment of the difference is that they are not ranked the same.
 

In terms of political stability, even after statistical adjustments, Rwanda and Uganda should show that they are significantly more stable that the Central African Republic, which is a country with an active rebellion, been described as a “fragile democracy” and “volatile” by on-the ground UN assessments and other reports.
 

In essence, the World Bank weakens its own study when it uses statistical results that do not reflect common sense reality and seasoned observers.
 

World Bank Comment: To be more specific, Mr. Church is concerned at the difference between the Political Stability/Absence of Violence ratings for Central African Republic (16.5th percentile) and Uganda (10.4th percentile), which conflicts with his views on the relative positions of these two countries.  We certainly respect Mr. Church’s views, and want only to point out that they are not inconsistent with what the WGI show.  Here’s why.  In the units of our governance indicators, the CAR has a score of -1.13, while Uganda has a score of -1.31, or a difference of 0.18 points.  When we consider that the difference between the best and worst countries in the world in 2005 on this indicator is 4.4 points, this difference between the CAR and Uganda is pretty small at less than 5 percent of the entire range of the indicator across countries!  Moreover, as noted there are margins of error.  What these mean is that for the CAR, our “best guess” for this indicator would range from -1.6 to -0.6, while for Uganda our “best guess” would range from -1.7 to -1.0.  Within these two ranges we think there is plenty of room for reasonable people to disagree over the precise relative standing of these two countries that in fact score quite close to each other.

The same point applies to Mr. Church’s criticism of trends in the aggregate indicators for the Central African Republic, five of which register small improvements between 2004-2005 (and perhaps more interestingly and more consistent with his views, all of which register sharp declines since the late 1990s).  The changes between 2004 and 2005 are tiny relative to the clearly-reported margins of error, and so should not be interpreted as evidence of significant changes in one direction or the other.


 

William Church Comments: Once again the World Bank negates the possibility of their report to be used as a measurement tool for a country’s progress when the margin of error explanation produces a conclusion that “these two countries score quite close to each other.”
 

GLCSS urges the World Bank to conduct a survey of the African diplomatic community and ask if they see Uganda and the Central African Republic as relatively the same in terms of Political Stability. Common sense tells us that the UN description of the Central African Republic as a “fragile” democracy is not close to the reality of Uganda as implied in the World Bank research.
 

 

World Bank Comments: Mr. Church is concerned that the WGI data for some countries in Central Africa, particularly Rwanda, is based on very few underlying data sources.  So are we. Unfortunately, internationally comparable quantitative data on various dimensions of governance is scarce, particularly for small, and poor, countries.  Our approach in the WGI has been to use as much of the available information as possible.  In countries like Rwanda where data is scarce, this scarcity is reflected in larger margins of error, properly indicating a greater degree of uncertainty about the estimates of governance for such countries.  Again, had Mr. Church consulted these margins of error, he would perhaps not have concluded that the WGI data show a significant decline for a country like Rwanda — the observed decline is well within the margins of error for that country, and so as producers of the data we do not interpret it as showing evidence of any significant trend for that country.

William Church Comments: First, the point needs to be made that the World Bank has agreed with GLCSS that there are limited WGI data points for Central Africa. The key issue is the impact this has on the World Bank study. This is revealed in the following World Bank response: “the observed decline is well within the margins of error for that country (Rwanda), and so as producers of the data we do not interpret it as showing evidence of any significant trend for that country
 

Once again, GLCSS believes that the World Bank has just supported the reality of the criticism of its analysis. Other organizations and international observers have noted the progress in Rwanda. GLCSS wrote about other studies in its original paper that supports the findings that there has been progress in Rwanda.
If the World Bank study can not measure that progress or says that it can not measure a significant trend then perhaps there are flaws in the study for this one particular country.
 

 

World Bank Comments: Mr. Church is concerned with changes in the composition of data sources from year to year, again using Rwanda as an example.  It is true that such changes can in principle affect scores on the aggregate indicators, although as we have shown in our past work on average these effects are small.  Still, it is worth knowing how much this matters for a particular case.  Mr. Church seems to suggest some capriciousness on our part, “dropping” one source and “replacing” it with another.  Actually, things are much more mundane — every year some of our sources are no longer updated, while other new ones become available.  Thus for Rwanda it was not our decision to “drop” the Columbia University State Capacity Project, rather this source was simply not updated for 2005.  Similarly, we did not “replace” this with the U.S. State Department/Amnesty International Human Rights Reports.  This source has been continuously available for Rwanda since 1996 and we have used it each time.  These kind of facts can easily be verified by any users, since we provide on our website details on the composition of sources for each country and year in our dataset, as well as access to virtually all of the data itself from each source.

William Church Comments: Once again the World Bank concurs with GLCSS when it agrees that changes in sources can affect the aggregate score. This is exactly what happened in the case of Rwanda and it was compounded by the limited data sources. GLCSS did not suggest a capricious act on the part of the World Bank in this case. (If it can be read with that meaning GLCSS apologizes to the World Bank) GLCSS recognized that the available data sources are limited; however, it is important to note the replacement did have an impact. This was the point of our comment and GLCSS did not suggest that the World Bank replaced Columbia University on whim.
 

 
World Bank Comments: Mr. Church is also concerned that our indicators “contradict” other data sources.  We would prefer the term “complement”, recognizing that there are many different data sources with different perspectives.  To take a specific example, Mr. Church is concerned that our aggregate indicator of Regulatory Quality shows no improvement for Rwanda, while the Heritage Foundation’s Index of Economic Freedom for Rwanda shows improvements.  The Heritage Index is part of our aggregate indicator, and together with another source, the Bertelsmann Transformation Index, shows small improvements between 2004 and 2005.  But another one of our sources, Global Insight’s Business Conditions Index, shows a decline, resulting in a flat score overall for the aggregate indicator.  This example illustrates an important premise of our aggregate indicators project — that looking at more data sources is better than looking at less, recognizing that all data sources may not agree about levels or trends in governance.

William Church Comments: The above example proves the primary criticism of the World Bank study.  Limited data sources produced a distorted result. This is supported by their own statement. Kenya and Uganda had nearly double the data sources and produced a balanced analysis in this category. GLCSS urges the World Bank to improve their analysis with additional survey sources and additional African sources. The World Bank response also does not address the fact that even other World Bank studies show that Rwanda is one of the regional leaders in this field.
 
World Bank Comment: Mr. Church is concerned about the lack of Africa-based data sources used in the WGI.  In fact, the WGI uses a wide range of data sources, some of which are Africa-based.  Most notable in terms of coverage are the assessments of the African Development Bank, which provides data on some 50 countries in Africa, that enter into four of our six governance indicators since 1998 (thus hardly in “rare circumstances” as he writes).  We also use surveys of firms and surveys of households with coverage in the Africa region, including the Global Competitiveness Survey, and the Afrobarometer Survey, although these unfortunately do not nearly have the comprehensive coverage that the African Development Bank has.  

William Church Comments: The World Bank has supported the arguments of GLCSS. The World Bank had African sources available and they did not use them for specifically Rwanda and other Central African countries. A close examination of the data sources for Rwanda show that the World Bank did use the African Development Bank. On this point we agree; however, in none of the years examined did they use Global Competitive Survey or the Afrobarometer. This was exactly the point of GLCSS. They had these sources available and they did not use them. Also GLCSS pointed out that the sources were overwhelming US-based sources.
 

Once again GLCSS apologizes if the World Bank took offense at the use of the word “rare”.  However, GLCSS stands by the usage of this word in terms of comparison to other sources. The fact remains is that in the area of Rule of Law the African Development Bank is used but it is dwarfed by seven other sources most of which are US-based sources.  This is true in nearly all of the measurement categories but not quite in that same ratio.
 


World Bank Comments: Finally, two small points of clarification.  (1) The Worldwide Governance Indicators are not used by the World Bank in its lending decisions.  The World Bank does use another internally-generated measure of governance, the Country Policy and Institutional Assessments, in the allocation of concessional loans from the International Development Association (IDA) across countries.  For information, and the 2005 CPIA assessments and the IDA allocation process, visit www.worldbank.org/ida.  (2)  Mr. Church claims that the Worldwide Governance Indicators project lacks outside review.  This is incorrect.  The WGI were begun initially as a research project in the late 1990s, and a number of papers detailing the WGI methodology have appeared in peer-reviewed scholarly publications.  The data and methodology have been freely available to all interested parties and we have engaged with many different stakeholders over the years on both the overall methodology and the scores for particular countries.
 

William Church Comments: We are sorry for the misunderstanding. GLCSS was addressing the issue that some government international donors have started using this as a criteria and did not mean to imply that the World Bank uses its own study in lending decisions.
 

Compiled and posted by the Great Lakes Centre for Strategic Studies 3 October 2006.
 

Be Sociable, Share!