Hailed as the latest technological advance that could revolutionise development and agriculture (along with other sectors), “big data” has been the focus of several recent articles, most notably a series of articles published by SciDev.Net. In June 2013 a UN High level panel called for a “data revolution” emphasising the need for better data to track progress towards development goals. But what is big data and how can it aid poverty and hunger eradication?
Big data is not just large amounts of information but rather it’s about integrating infrastructure to collect data at every step of the development process and designing new data collection methods that can track development goals effectively. In particular, big data is being hailed as the big fix for the lack of reliable official statistics in developing countries. But there is no clear (agreed upon) definition of big data, one article stating “it is data generated through our increasing use of digital devices and web-supported tools and platforms in our daily lives”. Due to our increasingly digital society, the amount of data (from social media platforms, mobile phones, online financial services etc.) has grown enormously. A much quoted statistic states that up to 90% of the world’s data was created over just two years (2010–2012). The aim for big data is to use this sizeable knowledge source to add value to society. Driving interest in evidence-based policy making, big data is also being termed a movement, one that aims to turn data into decision making.
In May 2012 Global Pulse published a White Paper entitled Big Data for Development: Opportunities & Challenges, which highlighted the opportunities big data provides. In particular they explore the role of big data in describing what is happening, predicting what may happen and explore the reasons behind why things happen.
For agriculture, big data means information can be collected along the whole supply chain including from supermarkets, weather sensing equipment, digital images, and research papers. These data sets can then be transformed through analytics into actionable information. But this conversion is rife with complexities in terms of managing, processing, sharing and using huge amounts of data.
Some examples of developments in big data in the development and agricultural fields include:
- In Rwanda, data collected on mobile money transfers was used to analyse the motivations and behaviour of aid givers after the 2008 earthquake. A worrying outcome was that donations were more likely to help wealthier people.
- Call detail record (CDR) analysis has been used to study the spread of infectious diseases and their control in an urban slum in Kibera, Kenya.
- Vital Herd has developed a sensor, or e-Pill that can be swallowed by cows and used to collect information about the animal’s vital signs and health that is transmitted to receivers and to software, which then alerts farmers to any health problems. In the US, total economic loss from animal sickness and death is more than $5bn a year, with global losses amounting to 12 times this.
- The Climate Corporation, a company purchased by Monsanto in 2013, operates a cloud-based farming information system that uses weather measurements and soil observations, to generate 10 trillion weather simulation data points and predictions on weather for the next 7 days at a resolution of around 1/3rd of a square mile. This allows farmers to better plan when to spray fertiliser or pesticide. Such data can also aid in the development of weather insurance-related products.
- Tech Mahindra, an IT services company in India, has developed a system called Farm-to-Fork which allows the monitoring of conditions in food shipping containers. When conditions such as temperature, humidity and oxygen levels change, alerts are sent out and the problem can be rectified either remotely or manually. An estimated 10% to 15% of food that is transported chilled spoils during transport.
Of course knowing the right data to collect is difficult, even more so when decisions that can affect people’s lives are based on that data. GDP, a measure on which policy decisions are commonly based, can be fraught with inaccuracies and the method of calculation may differ between countries, making comparisons misleading. In poorer countries, the capacity to collect data on economic activity is made harder by high degrees of informal market activity and low public spending. Investing in countries’ ability to collect data efficiently is needed but is big data the answer? Much of the big data is collected without prior thought to its collection and analysis, which can lead to problems in trying to analyse it with any kind of rigour. Big data analytics, is a field driving the analysis of big data through advances in statistical machine-learning and algorithms that are able to process huge amounts of digital data and identify patterns.
Some believe big data to be a big risk, particularly if policy-makers rely on it without question. Data collection methods such as surveys go to great lengths to ensure they are unbiased and representative. The big question is how reliable are data collected through more informal mechanisms and should we then base decisions on them? One concern is that through relying on the results of analytics to predict and mitigate future events they will ignore tackling the causes of the problems. As an example distribution of police resources based on the likelihood of crimes occurring in certain locations from past patterns, as is happening in some US and UK cities, fails to investigate why certain areas have higher crime rates than others.
There is also the fear that these sophisticated technological data collection and analysis tools will widen the divide between the rich and powerful and the poor and not so powerful. The capacity to use analytics and process huge amounts of data is so far only in the hands of a few organisations. As one article states, “People with the most data and capacities would be in the best position to exploit big data for economic advantage, even as they claim to use them to benefit others”.
Privacy is also a big issue, as even anonymous data can be tracked back to individuals, and relatively easily. Are we then heading into a “Big Brother” state? There are also concerns over how this data will be used. A study by researchers at Northeastern University found that based on mobile phone data on past movements they could predict with over 93% accuracy where a person was located at any given time.
Still in its infancy when it comes to development, big data is likely here to stay given that digital data is on the rise. Addressing the issues surrounding big data will involve the academic community, the development of legal frameworks and international regulation and will likely spark the demand by society for greater openness and accountability. Indeed parallel to the rise of big data is the open data movement. The report from the UN High level panel recommends, as part of the data revolution, the establishment of a new global partnership on development data, designed to address data gaps, issues accessing data and motivate the production of baseline figures against which to measure progress on the post-2015 development goals. But a truly transformative data revolution will mean this data brings about real change, targeting limited resources more effectively, and empowering people all around the world.