Policy Action Tracker: AI supporting evidence-based policy-making

In the policy analysis realm, tracking the implementation of climate strategies, such as the Nationally Determined Contributions (NDCs), presents a daunting challenge. The sheer volume of policy documents to analyze makes manual tracking an uphill task for policy advisors. Considering this, we began experimenting with a novel, AI-powered solution: The Policy Action Tracker.

The journey started from a cooperation between the Sustainable Development Solution Network (SDSN) and the GIZ Data Lab who handed this challenge in at the Hack4Good hackathon at ETH Zürich. Four students developed a search algorithm that could sift through policy documents, identifying contextual references to specific topics beyond simple keywords.

The Paris Agreement

The resulting algorithm was an encouraging success, marking the first step towards an easier policy analysis. However, to run the program code, coding expertise is necessary– a common accessibility hurdle in the world of AI.

To bridge this gap, we transformed the code into a user-friendly web application.

The Policy Action Tracker is designed to analyze individual documents. To initiate the process, users have the option to upload a document or select from example documents of the application, such as South Africa's Low Emission Strategy. The application then dissects the document into distinct paragraphs that serve as the foundation for the ensuing analysis.

 The application claims three key analytical features:

• contextual search,
• classification of Sustainable Development Goals (SDG),
• comparison of national NDC documents.

These features were specifically tailored in collaboration with policy experts from the SDSN, designed to facilitate their policy analysis and goal tracking endeavors.

The search feature uses the contextual search algorithm developed by the students, to help users navigating extensive policy documents and identifying contextual references to specified topics. The user can use questions or words for the contextual search, but also has the option for keyword search if needed “Show only Exact Matches”.

The second feature is designed to pinpoint references to the SDGs. Classifying these references aids policy advisors by providing a clear overview of the SDG topics discussed within a document and enabling them to track progress towards these objectives. By identifying and categorizing SDG mentions, policy advisors can discern gaps and prioritize areas for action. For this feature we used OSDG’s dataset, which comprises labelled text paragraphs based on their relevance for the SDGs. Learning from these examples, the model can subsequently predict references within new texts. Thanks to this specialized training, the model's precision outperforms results from the contextual search.

Employing this model, we are able to scrutinize an entire document and provide a distribution overview. For example, when analyzing the Low Emission Strategy South Africa, the model detects SDG 13 (Climate Action) and SDG 7 (Affordable and Clean Energy) as the most prominent topics.

The final feature, our NDC comparison tool, uses data from the German Institute of Development and Sustainability’s NDC Explorer, and integrates those with the document in question. The NDC Explorer catalogs topics noted in a country's NDCs – such as "climate risk droughts" or "anticipated costs of future climate-related hazards."

Our application identifies relevant paragraphs in the policy document that correspond to these topics. To illustrate, when linking the Low Emission Strategy to the South African NDC, one would simply select "South Africa" from a dropdown menu in the sidebar. The app also enables users to choose between "adaptation" and "mitigation" options. The results are then presented by topic, highlighting the top matches.

 This feature aims to provide policy advisors with a swift overview of thematic connections between the document and the country's national climate strategy, thereby aiding in evaluating the coherence of policy implementation.

All elements of our application – the documents, training data, models, and hosting infrastructure – are open-source. We've assembled what was already available, fine-tuned a model on SDGs, and developed the application code, which we've also made open source. This means that anyone can freely use this app, as well as reuse or replicate it in whole or in part. Our app maintains full transparency regarding the training data used and even the CO2 emissions generated.

 To facilitate this, we depend on Hugging Face's infrastructure, which provides pre-trained LLMs and server capacity to run the app. While the provided infrastructure is fundamental, upgrading would allow us to utilize larger models, resulting in improved outcomes. This demo aims to illustrate how even small teams can develop AI applications, leveraging open resources like the Hugging Face ecosystem, pre-trained LLMs, and data from OSDG and the NDC Explorer.

We hope our work underscores the importance of collaboration and open-source principles. By sharing our code, models, and data, we can stimulate innovation, learn from one another, and create better tools to address our era's challenges. The Policy Action Tracker serves as an example of the potential we can unlock when we collaborate and share within the open-source community.

Link to the application:

Policy Action Tracker - SDSN Demo