• The UN Secretary General's Independent Expert Advisory Group
  • on a Data Revolution for Sustainable Development
Search Menu

Data Innovation: big data and new technologies

The consultation has ended and comments are closed.

The world of data changes every day and every hour. New innovations have hugely increased the quantity of data and the possibilities available to people and institutions who want to collect and use it.  The challenge, and the opportunity, is to make this new world of data useful and useable to improve people’s lives.

In defining the Sustainable Development Goals to take us through to the year 2030, we have an opportunity to discover new ways of assessing wellbeing, measuring global development and making swift interventions in times of crisis. Looking at the changes over 15 years gives us an indication of how different the world could be over the next 15.

Since 2000 when the Millennium Development Goals began, there has been a surprisingly swift uptake of technology even in developing nations. In 2014, the developing world accounts for more than three-quarters of the world’s mobile phone subscriptions.



Today, in the private sector, analysis of big data – data sets too large and complex to be studied without software- is commonplace – with consumer profiling, personalised services, and predictive analysis being used to optimise sales. Similar techniques could be adopted to gain real-time insights into people’s wellbeing and to target aid interventions to vulnerable groups.  Such innovations offer exciting new opportunities, but also throw up big challenges around privacy, public trust, and the potential abuses of data. Legal frameworks have not yet caught up with rapidly advancing technology.

Public sector bodies are also starting to use big data and new technologies. Public health researchers are gaining valuable insights from using anonymised mobile phone data on human migration and linking this to the spread of malaria and dengue fever.

The increasing use of internet-enabled devices with sensors will provide still more opportunities both to improve the way services are delivered and also to harness that data to gain faster insights into whether interventions are working. Mobile technology services are also drivers of information that can empower citizens, be it apps that tell farmers when to optimally plant crops, micro-loans for fledgling enterprises or medical information for front-line health practitioners in remote settings.

Much of the big data with the most potential to be used for public good is collected by the private sector. As such, public-private partnerships are likely to become more widespread. The challenge will be ensuring they are sustainable over time, and that clear frameworks are in place to clarify roles and expectations on all sides.

The Independent Expert Advisory group welcomes input regarding how the opportunities of emerging technology and methodologies can best be realised for public good.  In particular:

  1. Creating incentives and regulatory structures to encourage private enterprises to share data and pilot new ways of working.
  2. Encouraging links between public and private sectors to research new methods, pilot their use, and encourage dissemination.
  3. Developing privacy frameworks that protect the privacy of individuals without hampering life-saving uses of data science for public good
  4. Capacity-boosting on new sources of data and the use of new technologies in the field of international development, humanitarian assistance and statistics.
  5. Examples of new innovations and methods that have demonstrated potential for improving the frequency, reliability, accessibility and usefulness of data.
  6. Examples of success in bringing together official and non-official data sources, and traditional and non-traditional methods. What are the factors behind the success stories, and what have been some of the challenges?
  7. Supporting widespread adoption of new and innovative ways of working where proven effective.



  1. Really good initiative!
    Sharing our experience here in Developing World, currently I work as data scientist at Rio de Janeiro City Hall, in Brazil.
    Crossing data from several databases, some of them in real time such as from transportation (GPS buses, car accidentes, traffic jams, lights, etc.) we could improve the system of buses and the velocity of cars in all the city.
    Another interesting report, using Big Data Analytics we have been crossing data from twiters, rains and dengue disease. The numbers fro this year were really good: reduction of 93% of cases in Rio de Janeiro.
    Lastly, our open Data portal helped not only enterprises and civil society, but journalists to create headlines based on the more than 30.000 files at

  2. Congratulations to the great Initiative. UNDP Kosovo has used the crowdsourcing method to identify priorities of Kosovans by selecting the upcoming Human Development Reports topics. The topics/most pressing challenges have been found in economic growth, education system and poor healthcare provision of existing Institutions. Furthermore, the Policy and Research Team will ‘crowdsource’ the Recommendations given by people themselves. This means, since the development should be ‘people-centred’ referring to the human development concept/approach people should also have an opinion on solutions of the existing problems which Kosovo has to solve in the (near) future. The crowdsourcing method is the preferable way of collecting data commensurable with the human development approach reflected in development projects implemented by the UN(DP) worldwide.

  3. The NZ Data Futures Forum set out some ideas for how to support data sharing and use – see in particular their third report at One proposal was to appoint a champion to drive innovation and sharing. The idea was that a high profile, energetic, passionate, entrepreneurial, connected individuals can do a lot to generate innovation across a sector, advocate for opportunities, promote the value and facilitate others to collaborate. They also had some suggestions on effective privacy frameworks.

  4. Beyond Data Monitoring – Achieving the Sustainability Development Goals Through Intelligence (Decision-Support) Integrating Holistic Analytics, True Cost Economics, and Open Source Everything

    Document Online at

    Executive Summary

    As the United Nations (UN) contemplates its most important new economic and social initiative, the seventeen new Sustainability Development Goals (SDG), to be manifest in the Global Sustainable Development Report and related UN System activities, it is essential that the Secretary-General be afforded an opportunity to recognize the radical changes that are taking place in the external environment – and how the UN can capitalize on them to accelerate achievement of the SDGs.

    At a time when The UN is focused on data as a statistical artifact necessary to monitoring the current state and future progress, the world is experiencing the five stages of collapse identified by Dmitry Orlov: financial, commercial, political, social, and cultural. Monitoring is necessary but insufficient if the UN is to stabilize – stop – this systemic collapse, and enable achievement of the SDGs.

    Beyond data monitoring – and a reliance on modest donor promises, many of which will fail to materialize – there is a brilliant world of holistic analytics, true cost economics, and open source everything engineering. This approach – pro-active and centered on ethical evidence-based decision-support – could – if implemented within the UN with a fraction of the promised funding for the SDGs – mobilize vastly greater resources; speed implementation of the seventeen SDGs, and therefore support the mission of the UN and its Member States in a manner much more effective than now possible.

    Secretary-General Ban Ki-moon has since 2012 been seeking a solution – a tangible foundation – for moving beyond Government in harmonizing understanding, spending, and outcomes in relation to the UN Mission – particularly the SDGs. Intelligence (decision-support) is the means by which the UN can illuminate true costs, educate the public, eradicate corruption, and harmonize field effect.

    The reality is that the Specialized Agencies (SA) and their information stove-pipes as well as their human networks are far removed from useful access and exploitation by the core elements of the UN responsive to the Secretary-General. Similarly, the data silos of all other organizations scattered across the eight information “tribes” that must be brought together to achieve hybrid governance (academic, civil society including labor and religion, commerce especially small business, government especially local, law enforcement, media, military, and non-government/non-profit) are all beyond any possible UN construct for near-real-time big data monitoring and sense-making.

    A human-centric United Nations Open-Source Decision-Support Information Network (UNODIN) is proposed as a counterpart to the established data monitoring capability. UNODIN offers an opportunity, at very low-cost, to mobilize donations from over one hundred billionaires seeking impact investments far beyond the capabilities of the thousands of smaller lesser non-governmental organizations – while also helping tens of thousands of Chief Executive Officers (CEO) redirect their corporate spending in favor of sustainable profits that are directly tied to the seventeen SDGs.

    By using intelligence (decision-support) to educate local to global publics away from unsustainable products, services, policies, and behaviors, and by promulgating open source everything solutions within each of the SDGs, the UN will accomplish its mission – its specific goals – faster, better, cheaper than anyone might have imagined. We must create an education-intelligence-research revolution.

    Such a revolution would place the UN via UNODIN at the center of a global to local network of humans able to leap-frog past the obstacles inherent in data monitoring, able to achieve near-real-time understanding, self-governance, localized enforcement, and most importantly, localized self-sustainability. By combining a holistic analytic model useful at all levels, a commitment to rapidly documenting and promulgating true cost economic information, and a leadership role in creating open source everything engineering solutions relevant to each aspect of each SDG, the UN would become the catalyst for a financial, commercial, political, social, and cultural revolution. Nothing less will do.

  5. (5) The World Inequality Database in Education (, developed by the EFA Global Monitoring Report, presents an innovative way of showing progress in education (or the lack thereof) by different sub-groups over time. By drawing together different sources of data, including DHS, MICS, national household surveys and learning assessments, WIDE is able to highlight the powerful influence of factors such as wealth, gender, location and ethnicity on select education indicators and outcomes. Work drawing on the WIDE database has highlighted significant disparities across countries, age cohorts and between populations within countries, and helped inform policy formation and public debate.

  6. Hi. Very happy see this initiative. Am just finishing up a PhD on the role of open data and web innovation in environmental sustainability. No time to prepare a more substantial submission, but would like to share a few of my conclusions, which relate to all four of your consultation areas. Get in touch if you would like to discuss further.
    Jack Townsend

    Sustainable development is highly complex (“wicked problems”), so the biggest contribution of the data revolution may be to enable experimentation, feedback and learning as to what works where. Target sustainability metrics at all levels are key to this process, above all the global Sustainable Development Goals. An open approach of democratic accountability and transparency to the agreement of all these metrics is important (i.e. what counts as sustainable), along with the recording and calculating of them, to build up trust so they are effective.
    It appears that the most important thing that we can do for environmental sustainability is systematically decreasing global and national constraints on the extraction and consumption of natural resources (such as meaningful carbon pricing). The data revolution has big role to play to track resource extraction extraction and use by both governments and corporations (e.g. carbon accounting) and to do this as transparently as possible to prevent fraud and build vital trust. (see call from Hans Rosling for timely accurate carbon data data

    There are many analogies between getting people to contribute to the information commons of open data, and getting them to look after the natural commons of the environment. Indeed, the ideas of Elinor Ostrom on how to build frameworks of governance that protect the commons have currency on both sides of the fence. Sustainability challenges are highly interdisciplinary, so the ability to readily integrate datasets (as per the vision of “linked data” is crucial to enabling real benefit. An open approach is important not just to get data flowing, but to empower a broader range of actors and counter concentration of power in the hands of the data elite. Privacy (not just for individuals, but for countries, companies and species) is a major concern in the new data economy, whether explicitly open data or not. New personal data architectures (e.g. Remote Storage may offer technical ways to mitigate privacy loss whilst keeping data flowing.

    Data aids research and accountability, but it also underlies the development of web companies. These can bring about rapid economic, social and environmental change, and some companies that centre on ICT systems and data may be achieving large-scale environmental outcomes. Examples of such “cleanweb” companies include Recyclebank (behaviour change of recycling), Opower (energy use feedback), Solar City (solar energy marketing and provision), Zipcar (collaborative consumption of car use), Google Nest (smart heatings systems), and Abundance Generation (crowd-funding renewable energy projects). Recently published a report on the growing cleanweb sector in the UK that identified over 250 companies, as well as a taxonomy of web for sustainability systems that identifies 30 genres, in six major groups (see PowerPoint presentation from ICT4S2014, or recent interview ).

    Some relevant resources:
    – The Sustainability Stream of the 2012 OKFest, a global opportunity to think about the role of open data in environmental sustainability
    – The Open Sustainability working group of Open Knowledge, an international discussion list on open data, open knowledge for environmental sustainability
    – The work of Jorge Zapico on the role of data and openness in sustainability, in particular the “hacker ethic”.
    – The TransforMap project to map out local sustainability resources
    – My TEDx talk on open knowledge and sustainability
    – The Climate Knowledge Brokers initiative to better integrate linked data for climate change response
    – Scidatacon 2014 conference on integrating science data for global sustainability

  7. As a good example of success in bringing together official and non-official data sources, and traditional and non-traditional methods, I would like to highlight an initiative called “Bigger Data”, promoted by private companies (both big companies and SMEs) and public research centres in our region, which intends to enhance and enrich the remarkable heritage of data available through the program “Open Data”2 by local governments and enterprise, in order to obtain information of high utility to the socio-economic cities environment, immediately useful for community and for the improvement of public and private services.
    After an initial analysis of the state of the art related to existing solutions for the exploitation of large amounts of data provided by the “sensor area” and the “open data provider”, the project aims to provide an innovative solution that will enable, starting from identified data, to provide a range of services focused on the following topics: Data collection and processing of road infrastructure, Energy efficiency in public lighting, Improvement of mobile services for visually impaired people.
    To achieve these objectives, the project will use an ICT platform that will serve to collect data from heterogeneous sources, to process data and to make them available for the provision of services through applications on mobile devices (smartphones, tablets, etc.). Please do not hesitate to contact us for any further information.

  8. Jack,

    All really great stuff. I hope you have a chance to read my UNODIN paper,, it is both full text online for translation, and can be downloaded as a document (3.4 is the latest version, and so posted).

    Open Data without Open Everything is largely worthless — sorry to be so blunt, but as we have found over the past two decades, all Open Data does, in the absence of Open Software, Open Hardware, and so on, is empower the proprietary vendors of over-priced licenses and maintenance. The UN is perfectly suited to empower us all toward going “all in” on Open Source Everything, and that is the point of my submission as well as my last book (free stuff at

    I want very much to integrate your hard-earned wisdom and hope you will get in touch.

    With best wishes,

  9. Regarding 1. “Creating incentives”, there needs to be a dialogue between public and private sector concerning ethics and privacy, concerning the fact that good privacy safe guards are in the self-interest of all for maintaining buy-in from the public.
    Regarding 2. “Encouraging links”, public funded research (e.g. research funded by UK research councils) has a responsibility to lead by example, showing that ethical approaches can be done without blocking the ability to innovate.
    Regarding 3. “Developing privacy”, privacy regulations need to be applied to private enterprise the same as we do to government funded research. With growing data sets, privacy becomes a growing issue. Simple annonymization techniques do not work (and are often not really applied anyway). We need methods that are “ethical by design”. Data harvesting tools need to be tested to make sure that they are inherently privacy protecting, before being allowed to be used.
    Regarding 5. “Examples of new innovations”, the Horizon Digital Economy Research hub at the University of Nottingham is involved in a range of project aimed at developing ethical, privacy protecting, ways of using data from social media, smart devices and other sources.

  10. What happens to people when they get to 60?

    It’s a little discussed subject. Data broken down by age is limited, but bringing together what is available from the UN, World Bank and Gallup gives us a snapshot of what is happening to older people now and how we can change things for the better.

    The Global AgeWatch Index brings together data on income, health, employment, social connections and personal security into one number and ranks countries accordingly.

    The ranking is accompanied by a global report – this year focusing on income security in old age – and country report cards that highlight innovative responses by government, growing citizen action and some frightening gaps in policy.

    New questions and insights
    What makes China and Bangladesh rank higher than India? How do people fare in the lowest ranked countries – Afghanistan, West Bank and Gaza and Malawi?

    Why does Bolivia do so well in comparison? Why is Turkey, a country with high economic growth, so far behind Mexico?

    What are the emerging issues facing governments at the top of the Index, Norway, Finland, Ireland and Argentina, where populations aged 60 plus make up between 15% and 26% of the total?

    These are some of the questions the Index explores.

    Two new features of the 2014 Global AgeWatch Index help explain the issues behind the figures.

    Included in our country report cards are radar charts that benchmark individual countries against regional averages. And 34 of the report cards include detailed commentaries, written by in-country experts, adding a richness to the data.

    Living without a pension
    The 2014 Index report points out that 150 million people aged 65 or over in Index countries live without a pension of any kind – For example only 29% of older Indians receive a pension, 4% of older Malawians and barely anyone in Myanmar. However 95% of older Bolivians get a social pension which not only helps them individually but is also credited with reducing household poverty by 13.5%. And 74% of older Chinese now have a pension – that is 130 million people.

    The 2014 Index report shows that pension coverage is rising, particularly in Latin America, but adequacy is still a major issue. For example in Kyrgyzstan the pension is worth US$98 a month, US$6 below the subsistence level of US$104. Research shows that heating and other bills eat up to 70% of this income. The situation is compounded by low economic activity amongst older people in Central Asia compared to other regions.

    The Index measures older people’s capabilities through economic activity and educational status. Some 22% of people aged 80-plus are looking for a job in Indonesia, and 92% of people aged 55-64 in Tanzania work. Many people in Colombia aged 60 and over want or need to go on working but face age discrimination. Job adverts routinely specify young applicants, for example.

    Global postcode lottery
    It is not only which you country you live in that determines your wellbeing in older age. Reports from individual countries show that provincial responsibility for services means that these are often very different in rural and urban areas, depending on local government providers.

    For example, supplementary health services in Canada, access to day care centres in Colombia and ambulances services in Kyrgyzstan, vary greatly between different parts of the country.

    Rising issues
    Caregiver burnout and the demand for individual care and community services to enable people to stay in their own homes in their later years are issues of concern for civil society, governments and professionals the world over.

    The transition into old age is inevitable, but it is not adequately being addressed. We hope the Global AgeWatch Index will stimulate demand for better data and debate.

    We invite you to explore the report and website to find out more.

  11. Pingback: Open letter to the UN Data Revolution Group - The hi:project

  12. Hi everyone, very useful initiative!
    Working mostly with data for a child care org, I do sometimes have the sense that things that matter most—might be slipping by unnoticed. I guess that data analysis that takes place in most organizations produces only a small fraction of its potential for useful insights. Very often this is due to the fact that capacity building is missing or there is still not enough awarness on the importance of data.
    Often there is the opinion that software tools can solve all our problems, which of course is not true. Vendors sell big data tools but there is not even a clear definition what big data is.
    Beside capacity building for analysts and data specialists we need clear metrics, measures and make data comparable and accessible for everyone which includes that data becomes easy to understand and visual, visual, visual.
    I really hope this revolution leads us this way.
    BR Christian

  13. Humans matter most. The open source everything tools, apart from eliminating licensing and maintenance feeds that are unaffordable, open up information-sharing and shared sense-making (conversations creating choice) among humans.

    The key point is that Open Data is largely worthless in the absence of going “all in” on the rest of the Open Source Ecology, particularly cloud, hardware, software, and spectrum.

    For a sense of Big Data myths and mal-practice, see Big Data @ Phi Beta Iota, at

    It will be most interesting to see if your loyal dissident comments are noticed — even more interesting if they are acted upon.

  14. There is no doubt that big data offers significant and exponential potential for international development work across multiple fields. That this Independent Advisory Group has been formed is clear evidence that the United Nations is ready to see how big data can help, an exciting initiative for any of us who have straddled the worlds between international development and big data technologies in their life’s work.

    In this initiative, I believe we are hoping for and seeking paths for a robust collaboration that brings local, regional, and global community groups, national governments, and international representatives together with the gamut of current and future big data technologies that will enable them to do better, live longer, be happier, or otherwise make a difference. There is no end to the technologies, platforms, or research initiatives that are available to enable robust use of big data within international development. And likewise there is no end of international development use cases to which big data technologies could be fit.

    But as we move forward, we must remember that much of our success will fall upon a common understanding of the potential and current limitations of big data. To that end, I would like to offer four key insights from my last 10 years of working on the technical side of massive open and public data projects for use in decision making circles.

    1) Data Quality and Assurance Matters

    It is commonly known that roughly 80% of a data scientist’s, engineer’s, or analyst’s time can be spent cleaning up data so it can be used in conjunction with a software platform to produce a compelling story. Of course, resolving this issue varies greatly depending on the size of the project or whether the project is being produced from a single license platform from a laptop or an enterprise-level solution within which multiple users are networked. It could also depend on the level of technical skill in databases and Extract, Transform, and Load (ETL) processes, the mechanisms for otherwise ingesting and aggregating data, the access and security protocols in place for assuring that data is protected, and the multitude of other issues involved in using big data to create good narratives. But bottom line, if the data isn’t workable to begin with, the user will loose trust in the data and in the process, and we will end up with a phenomena which is frequently referred to as “garbage in, garbage out”. This benefits no one.

    2) Data Provenance Matters

    In many data science circles, and particularly in many data science labs, the source of the data matters not. Those working on the methods and algorithms to make the data sciences come to life don’t care because it is not important to making the technology operational. As a result, few platforms provide the means for documenting data provenance from the inception of a data science project. While it doesn’t matter in the research world, for the purposes of the governments and international organizations who would use big data outputs to make policy, it matters greatly. Indeed, many big data projects have failed to be adopted in policy making circles because of the lack of understanding among big data platform developers of how important it is to be able to trace back to a source and determine its reliability when making life-affecting decisions.

    3) Context and Use Cases Matter

    There is no one size fits all approach to big data, especially when it comes to international development. And much of the success of adopting big data approaches will depend on providing experts and world citizens with the right tools to match a need. For example, current analytics software and data visualization packages are great for producing nationalized, aggregated demographic type of data or metadata outputs and beautiful visuals. But they will not likely be as useful for tracking the moment by moment of humanitarian disaster relief as a geo-spatial mapping platform, tracking agricultural price fluctuations in a micro community as crowd sourcing solutions, or providing a mechanism for the International Criminal Court to upload and sift through the exorbitant amounts of physical data they gather during a field mission to find the nuggets relevant to their cases. These are all different types of big data problems, each of which requires an understanding and formalization of existing processes, what parts can be automated, which will remain manual until advances in big data enabling technology can catch up to needs, and where humans will continuously need to intercede in the process.

    4) Flexible Open Data Standards Matter

    As discussions move forward on the foundations for a legal framework for agreement, I urge stakeholders to assure that whatever outcomes produced do not restrict innovations in big data technologies. The international community is ready to adopt technologies as they stand now, but there is a lot that big data innovators can learn about how the international community uses and needs to use big data. Aside from some forays into international business marketing, international data and how it is used in different countries and within different domains is a largely unexplored area. Certainly, the technology that exists today can provide significant insight, but there is so much more to learn. To that end, I hope that whatever standards are developed take into account that the standards that will assure legal, international collaboration are not necessarily the same as those that will enable collaboration for the purpose of technological innovation. Big Data innovators will need open standards for data formats, accessing data, developing APIs, and other considerations that fall under the auspices of GNU Free Documentation License developed by the Free Software Foundation or a Creative Commons license.

    The pendulum has swung dear readers. Big Data technologies have reached a moment where inputs from international stakeholders can help fuel a new generation of advancements, many of which have yet to be defined. And the opportunities presented in Open Data initiatives such as the UN Secretary General’s Independent Expert Advisor Group are in a great position to help frame the myriad of international problems needing practical solutions. I look forward to a process that enables the adoption of big data technologies and open data initiatives within broader international circles and to the Big Data innovations that could emerge as a result.

  15. I would like to cross post this to Phi Beta Iota. Can you send me permission along with a head and shoulder photograph and a pointer to your preferred website? Email robert.david.steele.vivas [at] gmail [dot] com.

  16. Pingback: Volunteering the future: A call to arms | Voices from Eurasia - We help build better lives.

  17. Public bodies should back private bodies to develop added-value business for the good of the citizens. This can be achieved through the following conditions:
    • Public bodies should be funded to support the partner industry (SME, start-up) developing new datasets or services.
    • Information should be provided along with the data itself, so as to ensure that any user knows about the ins and outs of the data used. This refers to quality insurance, metadata, and licence of use. In this respect, the notion of authoritative data (“sustainable and neutral data mastered by countries, which users can trust, upon which thematic data can be based, and with temporal and historical dimension”) is relevant. In this context, public bodies are in a good position to produce authoritative data able to support private bodies.
    • The authoritative data policy should as well ensure that the private body can develop business by easily testing and fabricating its product of service.

  18. Privacy and confidentiality are the most important things in statistical work , and we must maintain privacy when we publishing the data . Central Bureau of Statistics , Sudan has experience in sharing of the raw data with others through memorandums of understanding, we release the micro data exclusively for the purposes of scientific analysis .

  19. it will be interesting to standardize the format in which data is collected, to facilitate the use and make it also cheaper.

  20. Pingback: The data revolution we need | Goal 18

  21. Pingback: The Data Revolution We Need | Goal 18

  22. Pingback: Role of Data in the Post-2015 Development Agenda | Sustainable Development Solutions Network Northern Europe

  23. Pingback: Global Partnership on Sustainable Development Data Launched in NY | Sustainable Development Solutions Network Northern Europe

  24. Pingback: La innovación de los datos: big data y nuevas tecnologías – Cúcuta, Smart City de la frontera

Comments are closed.