Delphi Demo - CAGs from Text¶
July 30, 2018
This is a Jupyter notebook created to showcase the design and capabilities of the Delphi package, available at https://github.com/ml4ai/delphi.
A rendered HTML version of this notebook can also be found at
http://vision.cs.arizona.edu/adarsh/export/delphi_demo_wm_pi_meeting_2018.html
This demo has been tested with the version of Delphi corresponding to the commit hash below.
In [1]:
!git rev-parse HEAD
6c2188d18b2a916e2a8b00e9231f5ff62f951acd
Construct and visualize CAG corresponding to use case¶
In [2]:
%load_ext autoreload
%autoreload 2
from delphi.utils import get_data_from_url
import pickle
from delphi.AnalysisGraph import AnalysisGraph
from delphi.visualization import visualize
from delphi.manipulation import merge_nodes
import pandas as pd
from delphi.inspection import statements
import delphi.jupyter_tools as jt
from delphi.quantification import map_concepts_to_indicators
WARNING: [2018-10-04 14:13:43] indra/eidos - Could not instantiate Eidos reader, text reading will not be available.
In [3]:
url = 'http://vision.cs.arizona.edu/adarsh/export/demos/data/pi_mtg_demo.pkl'
sts = pickle.load(get_data_from_url(url))
In [4]:
G = AnalysisGraph.from_statements(sts)
visualize(G, rankdir='LR')
Out[4]:
Inspecting and editing CAGs¶
In [5]:
merge_nodes(G, 'food_security', 'food_insecurity', same_polarity=False)
visualize(G, rankdir='LR', nodes_to_highlight='food_insecurity')
Out[5]:
Inspecting statements¶
In [6]:
pd.options.display.max_colwidth=1000
pd.options.display.width=1000
jt.create_statement_inspection_table(statements(G))
Out[6]:
| obj_polarity | subj_polarity | |||
|---|---|---|---|---|
| un_groundings | Source API | Sentence | ||
| ('conflict', 'food_availability') | eidos | The conflict has severely disrupted families ' access to animal milk , which is a major source of nutrition for children in cattle-keeping and agro-pastoralist parts of the country .16 The need for milk for children was highlighted by participants in focus group discussions for the HNO in the Bor , Juba , Malakal and Bentiu PoC sites . | -1 | 1 |
| ('conflict', 'food_insecurity') | eidos | Further escalation of the conflict would likely result in increased food insecurity and displacement while humanitarian access and space may become further restricted . | 1 | 1 |
| Households Duk 28.7 % poor 29.8 % severe in Ayod , Nyirol , and Uror have limited physical access to markets , 75.1 % moderate ; a change since April 2017 when large-scale conflict in Uror spread 30.2 % borderline ; 24.1 % severe Uror to Nyirol and parts of Ayod , destroying markets in the areas . | 1 | 1 | ||
| SOUTH SUDAN Food Security Outlook Update December 2017 Some households likely in Catastrophe ( IPC Phase 5 ) , while conflict has increased in several areas FEWS NET produces forward-looking food security analysis and IPC compatible mapping several times a year for 30 countries , including South Sudan . | -1 | 1 | ||
| ➤ ➤ The intensification of the conflict also increases people 's food insecurity due to the disruption of trade and livelihoods . | 1 | 1 | ||
| hume | Conflict Risk of famine due to armed conflict | 1 | 1 | |
| Nonetheless, the indications are clear that the global food system is facing increasing risks due to more frequent extreme weather. | 1 | 1 | ||
| Nyalen’s case is but one example among millions of how conflict causes hunger, destroys individual and household resilience, and undermines rural livelihoods. | 1 | 1 | ||
| The overstretching of current humanitarian resources and capabilities during the projected worsening of food insecurity is a distinct possibility, raising the risk of an insufficient response to further deterioration. | 1 | 1 | ||
| The decrease in 2015 estimated national production is essentially due to the impact on yields of unfavourable weather conditions on first season crops in Central and Eastern Equatoria states and on main crops in Western Bahr el Ghazal State as well as to the disruption of agricultural activities in Western Equatoria State due to insecurity. | 1 | 1 | ||
| ('conflict', 'food_production') | eidos | Other areas of concern include : 1 ) Western Bahr el Ghazal , where conflict is also limiting agricultural activities , driving displacement , and disrupting normal market functioning ; 2 ) Northern Bahr el Ghazal , where prices are extremely high and households are especially dependent on markets to access food ; and 3 ) Greater Equatoria where ongoing conflict has disrupted crop production and restricted the movement of local populations . | -1 | 1 |
| ('conflict', 'human_migration') | eidos | Further escalation of the conflict would likely result in increased food insecurity and displacement while humanitarian access and space may become further restricted . | 1 | 1 |
| IDPIDP POPULATION BYsite SITEtype TYPE population by in informal settlements 1.86 million internally displaced with host communities Source : IOM/CCCM , Nov 2017 Source : IOM/CCCM , Nov 2017 18 18 Denotes a score of 0 Sources : CCCM , IOM/DTM Sources : CCCM , IOM/DTM During 2017 , new fighting and violence in Central Equatoria , Jonglei , Upper Nile and Western Bahr el Ghazal increased the population of IDPs in Wau by 20,000 and in Bor by 800 . | 1 | 1 | ||
| hume | Although the dispute within the Sudan People’s Liberation Movement (SPLM) was primarily political, spiraling violence quickly led the SPLM and the national army (SPLA) to split, resulting in the formation of the SPLM/A in Opposition (SPLA-IO). | 1 | 1 | |
| In addition to the life, the violence has caused displacement, restricted households’ movement towards fields, and forced aid workers to relocate, all of which are contributing to continued extreme acute food insecurity. | 1 | 1 | ||
| Likewise, conflict between Oromia and Somali Regions in Ethiopia has also caused the displacement of hundreds of thousands of people in the two regions. | 1 | 1 | ||
| Nyalen’s case is but one example among millions of how conflict causes hunger, destroys individual and household resilience, and undermines rural livelihoods. | 1 | 1 | ||
| The conflict is driving continued large-scale displacements, particularly of farming households from the country’s greenbelt. | 1 | 1 | ||
| Thousands of people living with HIV have seen their life-sustaining treatment interrupted without possibility of resumption and fresh violence in Western and Central Equatoria is affecting populations with the highest prevalence rates of HIV/AIDS in South Sudan. | 1 | 1 | ||
| Violence and displacement in the Greater Equatoria region have severely affected populations with the highest prevalence of HIV/AIDS and disrupted access to life-saving treatment. | 1 | 1 | ||
| Armed clashes will lead to further internal and external displacement. | 1 | 1 | ||
| sofia | Additionally , recent clashes have reportedly led to the displacement of at least 25 000 civilians around Ibba , Maridi , Mundri East , Mundri West and Yambio counties in Western Equatoria State . | 1 | 1 | |
| Conflict continues to drive large-scale displacement , and the majority of those fleeing the country are leaving Greater Equatoria for Uganda ( Figure 4 ) . | 1 | 1 | ||
| In the Greater Equatoria region , food security is likely to worsen significantly due to the ongoing conflict that has led to displacement of the local population which has disrupted the second season cropping . | 1 | 1 | ||
| ('conflict', 'market') | hume | Economic need is also linked to increased criminality in many areas, including the Equatorias, where civilians—including humanitarian workers—are increasingly being affected by theft, looting and related violent attacks. | 1 | 1 |
| Farm households without their own plough team must borrow equipment from others, often only available rather late in the season and they suffer lower yields as a result, due to sowing much of their grain on unploughed land and to the slowness of hand weeding. | 1 | 1 | ||
| Instability and hunger has created a surge of survival-criminality that further exacerbates the problem through stealing, looting and the prevention of free-flowing commerce”, he said. | 1 | 1 | ||
| Most crop damages were caused by droughts or insect attacks and crop diseases (12). | 1 | 1 | ||
| The vast lack of access to maternal, child and public health services, exacerbated by poor hygiene and sanitation practices, hyperinflation and high food prices and insecurity will continue to drive the level of malnutrition across the country. | 1 | 1 | ||
| Under this set of preconditions the production shocks considered here would almost certainly result in a more dramatic price response. | 1 | 1 | ||
| When national governance fails, as in the case of Somalia, conflict can lead to large-scale food insecurity or even famine. | 1 | 1 | ||
| ('drought', 'food_insecurity') | hume | Between 2004 and 2009 approximately 55 000 people a year lost their lives as a direct result of conflict or terrorism.2 In contrast, as a result of famine caused by drought and conflict, more than 250 000 died in Somalia alone between 2010 and 2012.3 | 1 | 1 |
| •➤ Although drought conditions have severely affected agro-pastoral livelihoods, farmers who benefitted from livelihood assistance were able to cope better and incur lower losses. | 1 | 1 | ||
| •➤ In Ethiopia, prolonged drought conditions are severely affecting the livelihoods in most southern and southeastern pastoral and agro-pastoral areas of Southern Nations, Nationalities and Peoples Region, southern Oromia and southeastern Somali Regions, where cumulative seasonal rainfall was up to 60 percent below average. | 1 | 1 | ||
| ('flooding', 'conflict') | eidos | Print HOME | NEWS WEDNESDAY 13 SEPTEMBER 2017 Floods displace hundreds in wartorn in South Sudan September 12 , 2017 ( JUBA ) -- Floods caused by rain have displaced more than 100,000 people in South Sudan , an official said , raising fears about the devastating impact this could have on food security in the wartorn nation . | 1 | 1 |
| ('flooding', 'food_insecurity') | hume | If a household lives outside a flood plain then the exposure to flooding is low and therefore the risk of a flood causing the household to become food insecure is | 1 | 1 |
| UNICEF together with partners responded to over 20,000 households affected by the flooding in Northern Bahr el Ghazal with provision of supplies, including household water treatment products and water containers. | 1 | 1 | ||
| ('flooding', 'food_production') | eidos | Furthermore , serious flooding experienced in parts of Jonglei , Warrap and Upper Nile states continues to inhibit crop production . | -1 | 1 |
| ('flooding', 'human_migration') | hume | Widespread flooding severely affected traders’ movements and commodities flow to the three conflict affected states, resulting into rising prices of the main staples in the local markets. | 1 | 1 |
| ('flooding', 'market') | hume | Food insecurity levels worsen further with each lean season, a time of year when food stocks are typically depleted, food prices are at their highest and heavy rains disrupt markets and restrict humanitarian access. | -1 | 1 |
| ('food_availability', 'food_insecurity') | eidos | Devaluation of the local currency , soaring food prices and reduced food trade as a result of the conflict , are undermining the food security of cattle keeping communities . | 1 | -1 |
| Expanding domestic food production in 2016 is then essential to prevent escalating food insecurity in 2017 . | -1 | 1 | ||
| HUMANITARIAN NEEDS AND DRIVERS The conflict and increasing levels of food insecurity worsened by rising staple food prices and collapse of the market , have had the most direct impact on people 's nutrition status . | -1 | 1 | ||
| ('food_availability', 'market') | eidos | As households always rely on markets to a significant degree and generally exhaust their stocks around the second quarter of the following year , usually the prevalence of high food expenditure is highest in February or June . | -1 | 1 |
| Food insecurity levels worsen further with each lean season , a time of year when food stocks are typically depleted , food prices are at their highest and heavy rains disrupt markets and restrict humanitarian access . | -1 | 1 | ||
| ('food_insecurity', 'conflict') | cwms | Food insecurity can trigger conflict. | 1 | 1 |
| eidos | INCREASE OF PRICES FOR STAPLE FOODS 150 South Sudanese Pounds DISPLACEMENT The cumulative effects of poor living conditions , malnutrition and inadequate services in a context of conflict and massive displacement have weakened resistance to illnesses and made them more life-threatening for crisis-affected people . | 1 | 1 | |
| International Experience The notion of a food security stock often elicits deep skepticism because in many countries such stocks have been central to relatively complex systems aimed at price stabilization involving domestic purchases and sales of grain , usually at fixed prices . | 1 | -1 | ||
| Such risks include sexual and gender based violence when women and men attempt to access assistance as well as other necessities , such as charcoal or firewood , ii ) ensuring that risks such as forced recruitment of children are prevented when large populations gather to receive food assistance , ( iv ) ensuring access to food assistance for the most marginalized and vulnerable groups , and ( v ) ensuring that food assistance does not exacerbate tension between different social groups . | 1 | -1 | ||
| hume | Competition for scarce resources has led to high tensions between refugee and host communities, further aggravating the situation for refugees. | 1 | 1 | |
| Competition for scarce resources has led to tensions between refugee and host communities, further aggravating the situation for refugees. | 1 | 1 | ||
| Due to the scare resources, increased tensions between host communities and Rohingya refugees have been reported. | 1 | 1 | ||
| Fuel shortages have constrained activity and led to theft and insecurity, while long gaps and inconsistency in salary payments to public sector employees have impacted the provision of health-care and education services, and the rule of law. | 1 | 1 | ||
| In the first half of 2017, these factors resulted in closure of services and disrupted treatment for more than 15,000 mal- nourished children and pregnant or breastfeeding women. | 1 | 1 | ||
| Relief items pre-positioned in the field are also ROADS SITUATION vulnerable to localized conflicts, resulting in destruction and looting. | 1 | 1 | ||
| The same study also found that drought affects conflict by causing changes in livestock prices, and suggests that drought- caused livestock price shocks are a main driver of local conflict. | 1 | 1 | ||
| With nearly 90 per cent of refugees living in camps in Upper Nile and Unity States where the South Sudan conflict has been particularly intense, tensions over scarce resources have increased between refugees and local populations. | 1 | 1 | ||
| sofia | Greater Equatoria , and particularly some of the countrys most productive Greenbelt counties from Yei , Lainya , Morobo , Kajo Keji and Magwi , are facing Crisis ( IPC Phase 3 ) and Emergency ( IPC Phase 4 ) levels of acute food insecurity , driven largely by armed conflict that has had severe effects on agricultural activities and markets . | 1 | -1 | |
| The worsening food insecurity is primarily driven by continued conflict and displacements , which have contributed to the reduced crop production , disruptions to pastoralist livelihoods and persistent macroeconomic deterioration . | 1 | -1 | ||
| ('food_insecurity', 'drought') | hume | In addition, many displaced households were unable to plant and therefore will not benefit from the harvest season. | 1 | 1 |
| ('food_insecurity', 'food_availability') | eidos | Between February and May , food security will deteriorate even further as households deplete their food stocks and are unable to purchase sufficient food at extremely high prices . | -1 | 1 |
| Between February and May , food security will further deteriorate as households will have depleted their harvests and natural sources of food further decline . | -1 | 1 | ||
| However , nationwide , the food insecure caseload ( IPC Phases 3 , 4 and 5 ) increased from about 5 million in February to a record high of 6 million in June as food access continues to be severely constrained by widespread insecurity , large scale displacements , high food prices , market disruptions , macro-economic collapse and exhaustion of households ' coping mechanisms . | -1 | 1 | ||
| Negative coping mechanisms included relying on less expensive foods ( 90 percent ) , reducing the number of daily meals ( 69 percent ) and restricting adult consumption ( 68 percent ) . | -1 | 1 | ||
| ('food_insecurity', 'food_insecurity') | eidos | In the Greater Equatoria region , which is 6,000 normally the primary food basket for the country , food 5,000 security is expected to worsen unusually due to conflict and insecurity since July that has prevented the 4,000 population from accessing their farms . | 1 | 1 |
| Widespread fighting , displacement and poor access to services , disease outbreaks , extremely poor diet ( in terms of both quality and quantity ) , low coverage of sanitation facilities and poor hygiene practices are the key drivers of the high levels of acute malnutrition across South Sudan . | 1 | 1 | ||
| ('food_insecurity', 'food_production') | eidos | Although , the short-and long-term rainfall forecasts are favorable for agricultural areas of the western and central highlands , production of most crops is likely to be limited due to insecurity and lack of availability and/or access to farm inputs due to ongoing conflict . | -1 | 1 |
| ('food_insecurity', 'human_migration') | eidos | Intensive and abnormal migrations due to insecurity have seriously depleted the health conditions for animal health throughout the country . | 1 | 1 |
| People 's vulnerability to disease and malnutrition has compounded due to multiple displacements . | 1 | 1 | ||
| sofia | The worsening food insecurity is primarily driven by continued conflict and displacements , which have contributed to the reduced crop production , disruptions to pastoralist livelihoods and persistent macroeconomic deterioration . | 1 | -1 | |
| ('food_insecurity', 'market') | eidos | In many areas , though , physical insecurity is also limiting trade flows and households ' physical access to markets . | -1 | 1 |
| hume | Many farmers may be in a similar situation, needing to sell livestock, labour or land in markets where an excess supply of these commodities has reduced their value. | -1 | 1 | |
| Poor or deteriorating livestock conditions have been reported in localised areas, consequently decreasing market prices. | -1 | 1 | ||
| Such labor can, however, be limited by the need to regulate body temperature under conditions of high ambient temperature, high radiation and humidity, and low wind. | -1 | 1 | ||
| drought, dry spell or irregular rains (30 percent), reduced income (22 percent) or loss of employment illness (17 percent) and epidemics (5 percent), crop pests and diseases (14 percent), high fuel/transport costs and other non-food prices (14 percent) and death of a working household member (9 percent). | -1 | 1 | ||
| • Low-income consumers in poor or fragile countries are protected from price rises through cash transfers and social protection arrangements, quelling unrest and reducing the incentives for governments to impose export controls. | -1 | 1 | ||
| • Market functionality is severely limited by the appalling road conditions and widespread insecurity, which effectively make Bentiu an economy under siege, with hardly any fuel available. | -1 | 1 | ||
| sofia | However , such a reduction will also reduce the pressure on herders ' incomes and their need to sell stock , leading to lower supply of and higher prices for livestock . | 1 | -1 | |
| Persistent insecurity and massive displacement have led to widespread disruption of livelihood activities , including agricultural , fisheries and livestock production , as well as limited access to local food markets among producers , traders and consumers . | 1 | -1 | ||
| ('food_production', 'food_insecurity') | eidos | In countries such as India , where legumes rather than animals are the preferred source of protein , these changes in the quality of food crops will accelerate the largely neglected epidemic of `` hidden hunger '' or micronutrient deficiency ( 24 ) . | -1 | 1 |
| ('human_migration', 'food_availability') | eidos | In times of drought , this migration flow becomes especially important , as it both reduces the number of people to be fed from village granaries and provides a supplementary income for buying food . | -1 | 1 |
| ('human_migration', 'food_insecurity') | eidos | Intensive and abnormal migrations due to insecurity have seriously depleted the health conditions for animal health throughout the country . | 1 | 1 |
| The conflict-related displacement of over 200,000 people from northern , central , and eastern former Jonglei has severely disrupted livelihoods and access to social services , thus severely undermining food security in the State . | 1 | 1 | ||
| ('human_migration', 'market') | hume | . Due to instability in neighbouring countries, the refugee population in South Sudan has increased. | 1 | 1 |
| A small number of host community members, who have used their already limited resources to provide for the displaced, also require assistance. | 1 | 1 | ||
| Additionally, the movement of populations of cattle from Jonglei State into Western Equatoria through Mundri West and across Mundri East counties caused an enormous threat to the host to avoid | 1 | 1 | ||
| sofia | Persistent insecurity and massive displacement have led to widespread disruption of livelihood activities , including agricultural , fisheries and livestock production , as well as limited access to local food markets among producers , traders and consumers . | 1 | 1 | |
| ('market', 'conflict') | eidos | In May , the already rising trend in food price had a sharp acceleration as the conflict intensified strongly in some parts of the country ( in particular in southern Unity state and some western counties of Western Equatoria State ) , causing an increase in cereal prices by up to 50 percent in most markets in just one month . | 1 | 1 |
| ('market', 'food_availability') | eidos | Figure 15 : South Sudan - Proportion of households in food expenditure class , February 2011-November 2015 The impact of the 2014 conflict on the supply of food and other commodities to local markets led to a sharp increase in overall food expenditure levels in mid-2014 and mid-2015 , when the countrywide proportion of households with high food expenditure reached an all-time high of more than 70 percent ( versus the previous record of 46 percent in June 2012 ) . | 1 | 1 |
| However , nationwide , the food insecure caseload ( IPC Phases 3 , 4 and 5 ) increased from about 5 million in February to a record high of 6 million in June as food access continues to be severely constrained by widespread insecurity , large scale displacements , high food prices , market disruptions , macro-economic collapse and exhaustion of households ' coping mechanisms . | -1 | -1 | ||
| ('market', 'food_insecurity') | eidos | The economic crisis , poor functioning of markets and resulting high prices have enhanced the vulnerability of South Sudanese households , whose very high levels of expenditures on food imply that they have little room to accommodate further price rises . | -1 | -1 |
| hume | A combination of average to below average crop production and the continuing economic crisis in the country characterized by depreciation in the currency, inflation, increasing costs of transportation and reduced imports, among other factors will keep the prices of imported food at higher than normal levels. | 1 | 1 | |
| By contrast, Wood's survey of farmers in the northern highlands of Ethiopia in 1974 found that most livestock losses were the result of distress sales in order to raise cash rather than deaths due to inadequate fodder (Wood, 1976). | 1 | 1 | ||
| Conflict and the lower international oil price are both driving extremely poor macroeconomic conditions in South Sudan. | 1 | 1 | ||
| From August to December 2015, harvesting-time missions were conducted in several counties (essentially where security levels were acceptable to guarantee staff safety) in Central Equatoria, Eastern Equatoria, Lakes, Jonglei, Upper Nile, Northern Bahr el Ghazal State, Warrap State and the Abyei Administrative Area. | 1 | 1 | ||
| Further, uncertain fuel supply networks (frequent disruption or delays in fuel deliveries) often lead to fuel shortages and high price disparities. | 1 | 1 | ||
| Given expected very low food access during this time, there remains a risk of Famine (IPC Phase 5) in a worst-case scenario of an extended absence of assistance. | 1 | 1 | ||
| Malnutrition has spiked as a result of lack of access to safe water and sanitation, increasing disease rates, rising food insecurity and conflict. | 1 | 1 | ||
| NeedS & key FigUReS the world’s newest nation, South Sudan, is faced with myriad challenges, some of which arise from historical marginalization, conflict and under-development whilst others are driven by more recent developments, including the armed conflict since december 2013. | 1 | 1 | ||
| Particularly in the north, protection is compromised by people remaining under emergency or makeshift shelters after years of displacement, with little capacity to improve these conditions. | 1 | 1 | ||
| While the current HIV prevalence in South Sudan is 2.7 percent, there are concerns that the high rates of malnutrition and food insecurity linked to the emergency could further exacerbate the situation, particularly when associated with additional risks of increased incidence, poor adherence to treatment, and the related mortality. | 1 | 1 | ||
| opportunities to improve resource use. | 1 | 1 | ||
| ('market', 'human_migration') | eidos | ➤ ➤ The staggering economy as well as the poor agricultural season impact the labour market and wages , subsequently preventing many IDPs from finding work and meeting their basic needs . | -1 | 1 |
Removing incorrect edges¶
In [7]:
G.remove_edges_from([('food_insecurity', 'drought'), ('food_insecurity', 'food_insecurity')])
visualize(G, rankdir='LR', nodes_to_highlight='food_insecurity')
Out[7]:
Mapping concepts to indicators¶
In [8]:
url = 'http://vision.cs.arizona.edu/adarsh/export/demos/data/concept_to_indicator_mapping.txt'
mapping = get_data_from_url(url)
map_concepts_to_indicators(G, 2, url)
visualize(G, indicators=True)
Out[8]:
In [9]:
from delphi.random_variables import Indicator
G.nodes['flooding']['indicators'] = [Indicator('PRECIPITATION', 'CYCLES')]
G.nodes['conflict']['indicators'] = G.nodes['conflict']['indicators'][1:]
G.nodes['market']['indicators'] = G.nodes['market']['indicators'][1:]
G.nodes['drought']['indicators'] = None
G.nodes['human_migration']['indicators'] = None
visualize(G, indicators=True)
Out[9]:
In [10]:
from datetime import datetime
from delphi.parameterization import parameterize
date = datetime(2014, 1, 1)
url = 'http://vision.cs.arizona.edu/adarsh/export/demos/data/south_sudan_data.csv'
df = pd.read_csv(url, sep="|", index_col="Indicator Name")
parameterize(G, datetime(2014, 1,1), df)
visualize(G, indicators=True, indicator_values = True,
graph_label=f'Causal Analysis Graph for South Sudan, {date.year}')
Out[10]:
Infer transition model¶
In [11]:
url = 'http://vision.cs.arizona.edu/adarsh/export/demos/data/adjectiveData.tsv'
G.infer_transition_model(get_data_from_url(url), 100)
Set initial parameters¶
In [12]:
from delphi.export import export_default_initial_values
export_default_initial_values(G, variables_file='variables.csv')
s0 = pd.read_csv('variables.csv', index_col=0, header=None,
error_bad_lines=False)[1]
s0.loc['∂(flooding)/∂t'] = 0.1
s0.to_csv('variables.csv')
s0
Out[12]:
0 human_migration 1.0 ∂(human_migration)/∂t 0.0 food_insecurity 1.0 ∂(food_insecurity)/∂t 0.0 market 1.0 ∂(market)/∂t 0.0 food_availability 1.0 ∂(food_availability)/∂t 0.0 conflict 1.0 ∂(conflict)/∂t 0.0 food_production 1.0 ∂(food_production)/∂t 0.0 drought 1.0 ∂(drought)/∂t 0.0 flooding 1.0 ∂(flooding)/∂t 0.1 Name: 1, dtype: float64
Execute model¶
In [13]:
%matplotlib inline
from delphi.bmi import *
from matplotlib import pyplot as plt
import seaborn as sns
from delphi.utils import _insert_line_breaks
initialize(G, 'variables.csv')
concept1 = 'flooding'
concept2 = 'food_insecurity'
ind1 = G.nodes[concept1]['indicators'][0]
ind2 = G.nodes[concept2]['indicators'][0]
plot_data = {variable: {'xs':[], 'ys':[],
'ylabel': variable.replace('_', ' ').capitalize(), 'units': '',
'ax_number': i}
for i, variable in enumerate((concept1, concept2, ind1.name, ind2.name))}
plot_data[ind1.name]['units'] = f"({ind1.unit})"
plot_data[ind2.name]['units'] = f"({ind2.unit})"
n_timesteps = 5
for day in range(n_timesteps):
update(G)
for concept in (concept1, concept2):
for datapoint in G.nodes[concept]['rv'].dataset:
plot_data[concept]['xs'].append(day+1)
plot_data[concept]['ys'].append(datapoint)
indicator = G.nodes[concept]['indicators'][0]
for datapoint in indicator.dataset:
if indicator.name != ind2.name:
plot_data[indicator.name]['xs'].append(day+1)
plot_data[indicator.name]['ys'].append(datapoint)
else:
if 0 < datapoint < 100:
plot_data[indicator.name]['xs'].append(day+1)
plot_data[indicator.name]['ys'].append(datapoint)
fig, axes = plt.subplots(1,4, figsize=(20, 4))
for variable, plot_data in plot_data.items():
ax = axes[plot_data['ax_number']]
ax.set_xlabel('Time step')
ax.set_xticks(range(6))
ax.set_title(_insert_line_breaks(' '.join((plot_data['ylabel'], plot_data['units']))))
sns.lineplot(plot_data['xs'], plot_data['ys'], ax = ax)
/Users/adarsh/.local/share/virtualenvs/delphi-K80NehlR/lib/python3.7/site-packages/scipy/stats/stats.py:1713: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result. return np.add.reduce(sorted[indexer] * weights, axis=axis) / sumval
Still todo: truncate probability distributions from 0 to 100 for percentages.
In [14]:
from delphi.inspection import inspect_edge
inspect_edge(G, 'flooding', 'food_insecurity')
Out[14]:
| obj_polarity | subj_polarity | |||
|---|---|---|---|---|
| un_groundings | Source API | Sentence | ||
| ('flooding', 'food_insecurity') | hume | If a household lives outside a flood plain then the exposure to flooding is low and therefore the risk of a flood causing the household to become food insecure is | 1 | 1 |
| UNICEF together with partners responded to over 20,000 households affected by the flooding in Northern Bahr el Ghazal with provision of supplies, including household water treatment products and water containers. | 1 | 1 |
Exploring the unknown unknowns¶
In [15]:
url = 'http://vision.cs.arizona.edu/adarsh/export/demos/data/pi_mtg_demo_unfiltered.pkl'
sts=pickle.load(get_data_from_url(url))
In [16]:
from delphi.subgraphs import get_subgraph_for_concept_pair
G = AnalysisGraph.from_statements(sts)
merge_nodes(G, 'food_security', 'food_insecurity', same_polarity=False)
G = get_subgraph_for_concept_pair(G, 'drought', 'food_insecurity', cutoff=2)
visualize(G, nodes_to_highlight=['drought','food_insecurity'])
Out[16]:
Causal analysis graphs from Software¶
Original Fortran program¶
In [17]:
jt.display(f'../data/program_analysis/crop_yield.f')
Out[17]:
1 ************************************************************************ 2 * UPDATE_EST - Updates the estimated yield of magic beans given 3 * some additional amount of rainfall 4 ************************************************************************ 5 * 6 * VARIABLES 7 * 8 * INPUT RAIN = Additional rainfall 9 * 10 * INPUT YIELD_EST = Crop yield to update 11 * 12 ************************************************************************ 13 SUBROUTINE UPDATE_EST(RAIN, TOTAL_RAIN, YIELD_EST) 14 DOUBLE PRECISION RAIN, YIELD_EST, TOTAL_RAIN 15 TOTAL_RAIN = TOTAL_RAIN + RAIN 16 17 * Yield increases up to a point 18 IF(TOTAL_RAIN .le. 40) THEN 19 YIELD_EST = -(TOTAL_RAIN - 40) ** 2 / 16 + 100 20 21 * Then sharply declines 22 ELSE 23 YIELD_EST = -TOTAL_RAIN + 140 24 ENDIF 25 26 END SUBROUTINE UPDATE_EST 27 28 ************************************************************************ 29 * CROP_YIELD - Estimate the yield of magic beans given a simple 30 * model for rainfall 31 ************************************************************************ 32 * 33 * VARIABLES 34 * 35 * INPUT MAX_RAIN = The maximum rain for the month 36 * INPUT CONSISTENCY = The consistency of the rainfall 37 * (higher = more consistent) 38 * INPUT ABSORPTION = Estimates the % of rainfall absorbed into the 39 * soil (i.e. % lost due to evaporation, runoff) 40 * 41 * OUTPUT YIELD_EST = The estimated yield of magic beans 42 * 43 * DAY = The current day of the month 44 * RAIN = The rainfall estimate for the current day 45 * 46 ************************************************************************ 47 PROGRAM CROP_YIELD 48 IMPLICIT NONE 49 50 INTEGER DAY 51 DOUBLE PRECISION RAIN, YIELD_EST, TOTAL_RAIN 52 DOUBLE PRECISION MAX_RAIN, CONSISTENCY, ABSORPTION 53 54 MAX_RAIN = 4.0 55 CONSISTENCY = 64.0 56 ABSORPTION = 0.6 57 58 YIELD_EST = 0 59 TOTAL_RAIN = 0 60 61 DO 20 DAY=1,31 62 * Compute rainfall for the current day 63 RAIN = (-(DAY - 16) ** 2 / CONSISTENCY + MAX_RAIN) * ABSORPTION 64 65 * Update rainfall estimate 66 CALL UPDATE_EST(RAIN, TOTAL_RAIN, YIELD_EST) 67 PRINT *, "Day ", DAY, " Estimate: ", YIELD_EST 68 69 20 ENDDO 70 71 PRINT *, "Crop Yield(%): ", YIELD_EST 72 73 END PROGRAM CROP_YIELD
In [18]:
%cd ../delphi/program_analysis/autoTranslate/
!./autoTranslate ../../../data/program_analysis/crop_yield.f
%cd ../../../notebooks/
/Users/adarsh/ml4ai/delphi/delphi/program_analysis/autoTranslate + java fortran.ofp.FrontEnd --class fortran.ofp.XMLPrinter --verbosity 0 ../../../data/program_analysis/crop_yield.f + python scripts/translate.py -f crop_yield.xml -g crop_yield.py + python scripts/genPGM.py -f crop_yield.py -p pgm.json -l lambdas.py /Users/adarsh/ml4ai/delphi/notebooks
AST in XML Format¶
In [19]:
jt.display('../delphi/program_analysis/autoTranslate/crop_yield.xml')
Out[19]:
1 <?xml version="1.0" encoding="UTF-8" standalone="no"?> 2 <ofp version="0.8.4"> 3 <file path="/Users/adarsh/ml4ai/delphi/delphi/program_analysis/autoTranslate/../../../data/program_analysis/crop_yield.f"> 4 <subroutine name="UPDATE_EST"> 5 <header> 6 <arguments count="3"> 7 <argument name="RAIN"/> 8 <argument name="TOTAL_RAIN"/> 9 <argument name="YIELD_EST"/> 10 </arguments> 11 <subroutine-stmt hasArgSpecifier="true" hasBindingSpec="false" hasDummyArgList="true" hasPrefix="false" name="UPDATE_EST" rule="1232"/> 12 </header> 13 <body> 14 <specification declarations="1" implicits="0" imports="0" uses="0"> 15 <declaration type="variable"> 16 <type hasKind="false" hasLength="false" name="DOUBLE"> 17 <intrinsic-type-spec hasKindSelector="false" keyword1="DOUBLE" keyword2="PRECISION" rule="403" type="402"/> 18 <declaration-type-spec rule="502" type="500" udtKeyword=""/> 19 </type> 20 <variables count="3"> 21 <variable hasInitialValue="false" name="RAIN"> 22 <entity-decl hasArraySpec="false" hasCharLength="false" hasCoarraySpec="false" hasInitialization="false" id="RAIN" rule="503008"/> 23 </variable> 24 <variable hasInitialValue="false" name="YIELD_EST"> 25 <entity-decl hasArraySpec="false" hasCharLength="false" hasCoarraySpec="false" hasInitialization="false" id="YIELD_EST" rule="503008"/> 26 </variable> 27 <variable hasInitialValue="false" name="TOTAL_RAIN"> 28 <entity-decl hasArraySpec="false" hasCharLength="false" hasCoarraySpec="false" hasInitialization="false" id="TOTAL_RAIN" rule="503008"/> 29 </variable> 30 <variable/> 31 </variables> 32 <type-declaration-stmt eos=" " numAttributes="0" rule="501"/> 33 </declaration> 34 <declaration/> 35 </specification> 36 <statement> 37 <assignment> 38 <target> 39 <name hasSubscripts="false" id="TOTAL_RAIN" type="variable"> 40 <data-ref numPartRef="1" rule="612"/> 41 <designator hasSubstringRange="false" rule="603"/> 42 </name> 43 </target> 44 <value> 45 <operation type="multiary"> 46 <operand> 47 <name hasSubscripts="false" id="TOTAL_RAIN" type="ambiguous"> 48 <data-ref numPartRef="1" rule="612"/> 49 </name> 50 </operand> 51 <operator operator="+"/> 52 <operand> 53 <name hasSubscripts="false" id="RAIN" type="ambiguous"> 54 <data-ref numPartRef="1" rule="612"/> 55 </name> 56 </operand> 57 </operation> 58 </value> 59 </assignment> 60 </statement> 61 <if> 62 <header> 63 <operation type="multiary"> 64 <operand> 65 <name hasSubscripts="false" id="TOTAL_RAIN" type="ambiguous"> 66 <data-ref numPartRef="1" rule="612"/> 67 </name> 68 </operand> 69 <operator operator=".le."/> 70 <operand> 71 <literal type="int" value="40"> 72 <int-literal-constant digitString="40" kindParam="" rule="406"/> 73 </literal> 74 </operand> 75 </operation> 76 </header> 77 <body> 78 <statement> 79 <assignment> 80 <target> 81 <name hasSubscripts="false" id="YIELD_EST" type="variable"> 82 <data-ref numPartRef="1" rule="612"/> 83 <designator hasSubstringRange="false" rule="603"/> 84 </name> 85 </target> 86 <value> 87 <operation type="multiary"> 88 <operand> 89 <operation type="unary"> 90 <operator operator="-"/> 91 <operand> 92 <operation type="multiary"> 93 <operand> 94 <operation type="multiary"> 95 <operand> 96 <operation type="multiary"> 97 <operand> 98 <name hasSubscripts="false" id="TOTAL_RAIN" type="ambiguous"> 99 <data-ref numPartRef="1" rule="612"/> 100 </name> 101 </operand> 102 <operator operator="-"/> 103 <operand> 104 <literal type="int" value="40"> 105 <int-literal-constant digitString="40" kindParam="" rule="406"/> 106 </literal> 107 </operand> 108 </operation> 109 </operand> 110 <operator operator="**"/> 111 <operand> 112 <literal type="int" value="2"> 113 <int-literal-constant digitString="2" kindParam="" rule="406"/> 114 </literal> 115 </operand> 116 </operation> 117 </operand> 118 <operator operator="/"/> 119 <operand> 120 <literal type="int" value="16"> 121 <int-literal-constant digitString="16" kindParam="" rule="406"/> 122 </literal> 123 </operand> 124 </operation> 125 </operand> 126 </operation> 127 </operand> 128 <operator operator="+"/> 129 <operand> 130 <literal type="int" value="100"> 131 <int-literal-constant digitString="100" kindParam="" rule="406"/> 132 </literal> 133 </operand> 134 </operation> 135 </value> 136 </assignment> 137 </statement> 138 <statement/> 139 </body> 140 <body type="else"> 141 <statement> 142 <assignment> 143 <target> 144 <name hasSubscripts="false" id="YIELD_EST" type="variable"> 145 <data-ref numPartRef="1" rule="612"/> 146 <designator hasSubstringRange="false" rule="603"/> 147 </name> 148 </target> 149 <value> 150 <operation type="multiary"> 151 <operand> 152 <operation type="unary"> 153 <operator operator="-"/> 154 <operand> 155 <name hasSubscripts="false" id="TOTAL_RAIN" type="ambiguous"> 156 <data-ref numPartRef="1" rule="612"/> 157 </name> 158 </operand> 159 </operation> 160 </operand> 161 <operator operator="+"/> 162 <operand> 163 <literal type="int" value="140"> 164 <int-literal-constant digitString="140" kindParam="" rule="406"/> 165 </literal> 166 </operand> 167 </operation> 168 </value> 169 </assignment> 170 </statement> 171 <statement/> 172 </body> 173 </if> 174 <statement> 175 <execution-part rule="208"/> 176 </statement> 177 </body> 178 <end-subroutine-stmt name="UPDATE_EST" rule="1234"/> 179 </subroutine> 180 <program name="CROP_YIELD"> 181 <header/> 182 <body> 183 <specification declarations="3" implicits="1" imports="0" uses="0"> 184 <declaration subtype="none" type="implicit"/> 185 <declaration type="variable"> 186 <type hasKind="false" hasLength="false" name="INTEGER"> 187 <intrinsic-type-spec hasKindSelector="false" keyword1="INTEGER" keyword2="" rule="403" type="400"/> 188 <declaration-type-spec rule="502" type="500" udtKeyword=""/> 189 </type> 190 <variables count="1"> 191 <variable hasInitialValue="false" name="DAY"> 192 <entity-decl hasArraySpec="false" hasCharLength="false" hasCoarraySpec="false" hasInitialization="false" id="DAY" rule="503008"/> 193 </variable> 194 <variable/> 195 </variables> 196 <type-declaration-stmt eos=" " numAttributes="0" rule="501"/> 197 </declaration> 198 <declaration type="variable"> 199 <type hasKind="false" hasLength="false" name="DOUBLE"> 200 <intrinsic-type-spec hasKindSelector="false" keyword1="DOUBLE" keyword2="PRECISION" rule="403" type="402"/> 201 <declaration-type-spec rule="502" type="500" udtKeyword=""/> 202 </type> 203 <variables count="3"> 204 <variable hasInitialValue="false" name="RAIN"> 205 <entity-decl hasArraySpec="false" hasCharLength="false" hasCoarraySpec="false" hasInitialization="false" id="RAIN" rule="503008"/> 206 </variable> 207 <variable hasInitialValue="false" name="YIELD_EST"> 208 <entity-decl hasArraySpec="false" hasCharLength="false" hasCoarraySpec="false" hasInitialization="false" id="YIELD_EST" rule="503008"/> 209 </variable> 210 <variable hasInitialValue="false" name="TOTAL_RAIN"> 211 <entity-decl hasArraySpec="false" hasCharLength="false" hasCoarraySpec="false" hasInitialization="false" id="TOTAL_RAIN" rule="503008"/> 212 </variable> 213 <variable/> 214 </variables> 215 <type-declaration-stmt eos=" " numAttributes="0" rule="501"/> 216 </declaration> 217 <declaration type="variable"> 218 <type hasKind="false" hasLength="false" name="DOUBLE"> 219 <intrinsic-type-spec hasKindSelector="false" keyword1="DOUBLE" keyword2="PRECISION" rule="403" type="402"/> 220 <declaration-type-spec rule="502" type="500" udtKeyword=""/> 221 </type> 222 <variables count="3"> 223 <variable hasInitialValue="false" name="MAX_RAIN"> 224 <entity-decl hasArraySpec="false" hasCharLength="false" hasCoarraySpec="false" hasInitialization="false" id="MAX_RAIN" rule="503008"/> 225 </variable> 226 <variable hasInitialValue="false" name="CONSISTENCY"> 227 <entity-decl hasArraySpec="false" hasCharLength="false" hasCoarraySpec="false" hasInitialization="false" id="CONSISTENCY" rule="503008"/> 228 </variable> 229 <variable hasInitialValue="false" name="ABSORPTION"> 230 <entity-decl hasArraySpec="false" hasCharLength="false" hasCoarraySpec="false" hasInitialization="false" id="ABSORPTION" rule="503008"/> 231 </variable> 232 <variable/> 233 </variables> 234 <type-declaration-stmt eos=" " numAttributes="0" rule="501"/> 235 </declaration> 236 <declaration/> 237 </specification> 238 <statement> 239 <assignment> 240 <target> 241 <name hasSubscripts="false" id="MAX_RAIN" type="variable"> 242 <data-ref numPartRef="1" rule="612"/> 243 <designator hasSubstringRange="false" rule="603"/> 244 </name> 245 </target> 246 <value> 247 <literal type="real" value="4.0"> 248 <real-literal-constant kindParam="" realConstant="4.0" rule="417"/> 249 </literal> 250 </value> 251 </assignment> 252 </statement> 253 <statement> 254 <assignment> 255 <target> 256 <name hasSubscripts="false" id="CONSISTENCY" type="variable"> 257 <data-ref numPartRef="1" rule="612"/> 258 <designator hasSubstringRange="false" rule="603"/> 259 </name> 260 </target> 261 <value> 262 <literal type="real" value="64.0"> 263 <real-literal-constant kindParam="" realConstant="64.0" rule="417"/> 264 </literal> 265 </value> 266 </assignment> 267 </statement> 268 <statement> 269 <assignment> 270 <target> 271 <name hasSubscripts="false" id="ABSORPTION" type="variable"> 272 <data-ref numPartRef="1" rule="612"/> 273 <designator hasSubstringRange="false" rule="603"/> 274 </name> 275 </target> 276 <value> 277 <literal type="real" value="0.6"> 278 <real-literal-constant kindParam="" realConstant="0.6" rule="417"/> 279 </literal> 280 </value> 281 </assignment> 282 </statement> 283 <statement> 284 <assignment> 285 <target> 286 <name hasSubscripts="false" id="YIELD_EST" type="variable"> 287 <data-ref numPartRef="1" rule="612"/> 288 <designator hasSubstringRange="false" rule="603"/> 289 </name> 290 </target> 291 <value> 292 <literal type="int" value="0"> 293 <int-literal-constant digitString="0" kindParam="" rule="406"/> 294 </literal> 295 </value> 296 </assignment> 297 </statement> 298 <statement> 299 <assignment> 300 <target> 301 <name hasSubscripts="false" id="TOTAL_RAIN" type="variable"> 302 <data-ref numPartRef="1" rule="612"/> 303 <designator hasSubstringRange="false" rule="603"/> 304 </name> 305 </target> 306 <value> 307 <literal type="int" value="0"> 308 <int-literal-constant digitString="0" kindParam="" rule="406"/> 309 </literal> 310 </value> 311 </assignment> 312 </statement> 313 <loop type="do"> 314 <header> 315 <index-variable name="DAY"> 316 <do-variable id="DAY" rule="831"/> 317 <lower-bound> 318 <literal type="int" value="1"> 319 <int-literal-constant digitString="1" kindParam="" rule="406"/> 320 </literal> 321 </lower-bound> 322 <upper-bound> 323 <literal type="int" value="31"> 324 <int-literal-constant digitString="31" kindParam="" rule="406"/> 325 </literal> 326 </upper-bound> 327 </index-variable> 328 <loop-control doConstructType="1701" hasOptExpr="false" rule="818008" whileKeyword=""/> 329 </header> 330 <do-stmt digitString="20" doKeyword="DO" eos=" " hasLoopControl="true" id="" rule="827"/> 331 <body> 332 <statement> 333 <assignment> 334 <target> 335 <name hasSubscripts="false" id="RAIN" type="variable"> 336 <data-ref numPartRef="1" rule="612"/> 337 <designator hasSubstringRange="false" rule="603"/> 338 </name> 339 </target> 340 <value> 341 <operation type="multiary"> 342 <operand> 343 <operation type="multiary"> 344 <operand> 345 <operation type="unary"> 346 <operator operator="-"/> 347 <operand> 348 <operation type="multiary"> 349 <operand> 350 <operation type="multiary"> 351 <operand> 352 <operation type="multiary"> 353 <operand> 354 <name hasSubscripts="false" id="DAY" type="ambiguous"> 355 <data-ref numPartRef="1" rule="612"/> 356 </name> 357 </operand> 358 <operator operator="-"/> 359 <operand> 360 <literal type="int" value="16"> 361 <int-literal-constant digitString="16" kindParam="" rule="406"/> 362 </literal> 363 </operand> 364 </operation> 365 </operand> 366 <operator operator="**"/> 367 <operand> 368 <literal type="int" value="2"> 369 <int-literal-constant digitString="2" kindParam="" rule="406"/> 370 </literal> 371 </operand> 372 </operation> 373 </operand> 374 <operator operator="/"/> 375 <operand> 376 <name hasSubscripts="false" id="CONSISTENCY" type="ambiguous"> 377 <data-ref numPartRef="1" rule="612"/> 378 </name> 379 </operand> 380 </operation> 381 </operand> 382 </operation> 383 </operand> 384 <operator operator="+"/> 385 <operand> 386 <name hasSubscripts="false" id="MAX_RAIN" type="ambiguous"> 387 <data-ref numPartRef="1" rule="612"/> 388 </name> 389 </operand> 390 </operation> 391 </operand> 392 <operator operator="*"/> 393 <operand> 394 <name hasSubscripts="false" id="ABSORPTION" type="ambiguous"> 395 <data-ref numPartRef="1" rule="612"/> 396 </name> 397 </operand> 398 </operation> 399 </value> 400 </assignment> 401 </statement> 402 <statement> 403 <call> 404 <name hasSubscripts="true" id="UPDATE_EST" type="procedure"> 405 <subscripts count="3"> 406 <subscript type="simple"> 407 <name hasSubscripts="false" id="RAIN" type="ambiguous"> 408 <data-ref numPartRef="1" rule="612"/> 409 </name> 410 </subscript> 411 <subscript type="simple"> 412 <name hasSubscripts="false" id="TOTAL_RAIN" type="ambiguous"> 413 <data-ref numPartRef="1" rule="612"/> 414 </name> 415 </subscript> 416 <subscript type="simple"> 417 <name hasSubscripts="false" id="YIELD_EST" type="ambiguous"> 418 <data-ref numPartRef="1" rule="612"/> 419 </name> 420 </subscript> 421 <subscript/> 422 </subscripts> 423 <data-ref numPartRef="1" rule="612"/> 424 </name> 425 <call-stmt callKeyword="CALL" eos=" " hasActualArgSpecList="false" rule="1218"/> 426 </call> 427 </statement> 428 <statement> 429 <print> 430 <print-format type="*"/> 431 <outputs count="4"> 432 <output> 433 <literal type="char" value=""Day ""> 434 <char-literal-constant digitString-kind-param="" identifier-kind-param="" rule="427" str=""Day ""/> 435 </literal> 436 </output> 437 <output> 438 <name hasSubscripts="false" id="DAY" type="ambiguous"> 439 <data-ref numPartRef="1" rule="612"/> 440 </name> 441 </output> 442 <output> 443 <literal type="char" value="" Estimate: ""> 444 <char-literal-constant digitString-kind-param="" identifier-kind-param="" rule="427" str="" Estimate: ""/> 445 </literal> 446 </output> 447 <output> 448 <name hasSubscripts="false" id="YIELD_EST" type="ambiguous"> 449 <data-ref numPartRef="1" rule="612"/> 450 </name> 451 </output> 452 </outputs> 453 <print-stmt eos=" " hasOutputItemList="true" printKeyword="PRINT" rule="912"/> 454 </print> 455 </statement> 456 <statement/> 457 <label lbl="20" rule="313"/> 458 <do-term-action-stmt doKeyword="" endKeyword="ENDDO" eos=" " id="" inserted="false" label="20" rule="838"/> 459 </body> 460 </loop> 461 <statement> 462 <print> 463 <print-format type="*"/> 464 <outputs count="2"> 465 <output> 466 <literal type="char" value=""Crop Yield(%): ""> 467 <char-literal-constant digitString-kind-param="" identifier-kind-param="" rule="427" str=""Crop Yield(%): ""/> 468 </literal> 469 </output> 470 <output> 471 <name hasSubscripts="false" id="YIELD_EST" type="ambiguous"> 472 <data-ref numPartRef="1" rule="612"/> 473 </name> 474 </output> 475 </outputs> 476 <print-stmt eos=" " hasOutputItemList="true" printKeyword="PRINT" rule="912"/> 477 </print> 478 </statement> 479 <statement> 480 <execution-part rule="208"/> 481 </statement> 482 </body> 483 <end-program-stmt endKeyword="END" eos=" " id="CROP_YIELD" programKeyword="PROGRAM" rule="1103"/> 484 <main-program hasExecutionPart="true" hasInternalSubprogramPart="false" hasProgramStmt="true" rule="1101"/> 485 </program> 486 </file> 487 </ofp>
Equivalent Python code¶
In [20]:
jt.display('../delphi/program_analysis/autoTranslate/crop_yield.py')
Out[20]:
1 from typing import List 2 3 def UPDATE_EST(RAIN: List[float], TOTAL_RAIN: List[float], YIELD_EST: List[float]): 4 TOTAL_RAIN[0] = (TOTAL_RAIN[0] + RAIN[0]) 5 if (TOTAL_RAIN[0] <= 40): 6 YIELD_EST[0] = (-((((TOTAL_RAIN[0] - 40) ** 2) / 16)) + 100) 7 else: 8 YIELD_EST[0] = (-(TOTAL_RAIN[0]) + 140) 9 10 def CROP_YIELD(): 11 DAY: List[int] = [0] 12 RAIN: List[float] = [0.0] 13 YIELD_EST: List[float] = [0.0] 14 TOTAL_RAIN: List[float] = [0.0] 15 MAX_RAIN: List[float] = [0.0] 16 CONSISTENCY: List[float] = [0.0] 17 ABSORPTION: List[float] = [0.0] 18 MAX_RAIN[0] = 4.0 19 CONSISTENCY[0] = 64.0 20 ABSORPTION[0] = 0.6 21 YIELD_EST[0] = 0 22 TOTAL_RAIN[0] = 0 23 for DAY[0] in range(1, 31+1): 24 RAIN[0] = ((-((((DAY[0] - 16) ** 2) / CONSISTENCY[0])) + MAX_RAIN[0]) * ABSORPTION[0]) 25 UPDATE_EST(RAIN, TOTAL_RAIN, YIELD_EST) 26 print("Day ", DAY, " Estimate: ", YIELD_EST) 27 print("Crop Yield(%): ", YIELD_EST) 28 29 CROP_YIELD()
Extracted lambda functions¶
In [21]:
jt.display('../delphi/program_analysis/autoTranslate/lambdas.py')
Out[21]:
1 def UPDATE_EST__lambda__TOTAL_RAIN_0(TOTAL_RAIN, RAIN): 2 TOTAL_RAIN = (TOTAL_RAIN+RAIN) 3 return TOTAL_RAIN 4 5 def UPDATE_EST__lambda__IF_1_0(TOTAL_RAIN): 6 return (TOTAL_RAIN<=40) 7 8 def UPDATE_EST__lambda__YIELD_EST_0(TOTAL_RAIN): 9 YIELD_EST = (-((((TOTAL_RAIN-40)**2)/16))+100) 10 return YIELD_EST 11 12 def UPDATE_EST__lambda__YIELD_EST_1(TOTAL_RAIN): 13 YIELD_EST = (-(TOTAL_RAIN)+140) 14 return YIELD_EST 15 16 def CROP_YIELD__lambda__MAX_RAIN_0(): 17 MAX_RAIN = 4.0 18 return MAX_RAIN 19 20 def CROP_YIELD__lambda__CONSISTENCY_0(): 21 CONSISTENCY = 64.0 22 return CONSISTENCY 23 24 def CROP_YIELD__lambda__ABSORPTION_0(): 25 ABSORPTION = 0.6 26 return ABSORPTION 27 28 def CROP_YIELD__lambda__YIELD_EST_0(): 29 YIELD_EST = 0 30 return YIELD_EST 31 32 def CROP_YIELD__lambda__TOTAL_RAIN_0(): 33 TOTAL_RAIN = 0 34 return TOTAL_RAIN 35 36 def CROP_YIELD__lambda__RAIN_0(DAY, CONSISTENCY, MAX_RAIN, ABSORPTION): 37 RAIN = ((-((((DAY-16)**2)/CONSISTENCY))+MAX_RAIN)*ABSORPTION) 38 return RAIN
DBN-JSON file¶
In [22]:
dbn_json = '../delphi/program_analysis/autoTranslate/pgm.json'
jt.display(dbn_json)
Out[22]:
1 { 2 "start": "CROP_YIELD", 3 "functions": [ 4 { 5 "name": "UPDATE_EST__assign__TOTAL_RAIN_0", 6 "type": "assign", 7 "target": "TOTAL_RAIN", 8 "sources": [ 9 "TOTAL_RAIN", 10 "RAIN" 11 ], 12 "body": [ 13 { 14 "type": "lambda", 15 "name": "UPDATE_EST__lambda__TOTAL_RAIN_0", 16 "reference": 4 17 } 18 ] 19 }, 20 { 21 "name": "UPDATE_EST__condition__IF_1_0", 22 "type": "assign", 23 "target": "IF_1", 24 "sources": [ 25 "TOTAL_RAIN" 26 ], 27 "body": [ 28 { 29 "type": "lambda", 30 "name": "UPDATE_EST__lambda__IF_1_0", 31 "reference": 5 32 } 33 ] 34 }, 35 { 36 "name": "UPDATE_EST__assign__YIELD_EST_0", 37 "type": "assign", 38 "target": "YIELD_EST", 39 "sources": [ 40 "TOTAL_RAIN" 41 ], 42 "body": [ 43 { 44 "type": "lambda", 45 "name": "UPDATE_EST__lambda__YIELD_EST_0", 46 "reference": 6 47 } 48 ] 49 }, 50 { 51 "name": "UPDATE_EST__assign__YIELD_EST_1", 52 "type": "assign", 53 "target": "YIELD_EST", 54 "sources": [ 55 "TOTAL_RAIN" 56 ], 57 "body": [ 58 { 59 "type": "lambda", 60 "name": "UPDATE_EST__lambda__YIELD_EST_1", 61 "reference": 8 62 } 63 ] 64 }, 65 { 66 "name": "UPDATE_EST__decision__YIELD_EST_0", 67 "type": "assign", 68 "target": "YIELD_EST", 69 "sources": [ 70 "IF_1_0", 71 "YIELD_EST_2", 72 "YIELD_EST_1" 73 ] 74 }, 75 { 76 "name": "UPDATE_EST", 77 "type": "container", 78 "input": [ 79 { 80 "name": "RAIN", 81 "domain": "real" 82 }, 83 { 84 "name": "TOTAL_RAIN", 85 "domain": "real" 86 }, 87 { 88 "name": "YIELD_EST", 89 "domain": "real" 90 } 91 ], 92 "variables": [ 93 { 94 "name": "TOTAL_RAIN", 95 "domain": "real" 96 }, 97 { 98 "name": "RAIN", 99 "domain": "real" 100 }, 101 { 102 "name": "IF_1", 103 "domain": "boolean" 104 }, 105 { 106 "name": "YIELD_EST", 107 "domain": "real" 108 } 109 ], 110 "body": [ 111 { 112 "name": "UPDATE_EST__assign__TOTAL_RAIN_0", 113 "output": { 114 "variable": "TOTAL_RAIN", 115 "index": 1 116 }, 117 "input": [ 118 { 119 "variable": "TOTAL_RAIN", 120 "index": 0 121 }, 122 { 123 "variable": "RAIN", 124 "index": 0 125 } 126 ] 127 }, 128 { 129 "name": "UPDATE_EST__condition__IF_1_0", 130 "output": { 131 "variable": "IF_1", 132 "index": 0 133 }, 134 "input": [ 135 { 136 "variable": "TOTAL_RAIN", 137 "index": 1 138 } 139 ] 140 }, 141 { 142 "name": "UPDATE_EST__assign__YIELD_EST_0", 143 "output": { 144 "variable": "YIELD_EST", 145 "index": 1 146 }, 147 "input": [ 148 { 149 "variable": "TOTAL_RAIN", 150 "index": 1 151 } 152 ] 153 }, 154 { 155 "name": "UPDATE_EST__assign__YIELD_EST_1", 156 "output": { 157 "variable": "YIELD_EST", 158 "index": 2 159 }, 160 "input": [ 161 { 162 "variable": "TOTAL_RAIN", 163 "index": 1 164 } 165 ] 166 }, 167 { 168 "name": "UPDATE_EST__decision__YIELD_EST_0", 169 "output": { 170 "variable": "YIELD_EST", 171 "index": 3 172 }, 173 "input": [ 174 { 175 "variable": "IF_1", 176 "index": 0 177 }, 178 { 179 "variable": "YIELD_EST", 180 "index": 2 181 }, 182 { 183 "variable": "YIELD_EST", 184 "index": 1 185 } 186 ] 187 } 188 ] 189 }, 190 { 191 "name": "CROP_YIELD__assign__MAX_RAIN_0", 192 "type": "assign", 193 "target": "MAX_RAIN", 194 "sources": [], 195 "body": { 196 "type": "literal", 197 "dtype": "real", 198 "value": "4.0" 199 } 200 }, 201 { 202 "name": "CROP_YIELD__assign__CONSISTENCY_0", 203 "type": "assign", 204 "target": "CONSISTENCY", 205 "sources": [], 206 "body": { 207 "type": "literal", 208 "dtype": "real", 209 "value": "64.0" 210 } 211 }, 212 { 213 "name": "CROP_YIELD__assign__ABSORPTION_0", 214 "type": "assign", 215 "target": "ABSORPTION", 216 "sources": [], 217 "body": { 218 "type": "literal", 219 "dtype": "real", 220 "value": "0.6" 221 } 222 }, 223 { 224 "name": "CROP_YIELD__assign__YIELD_EST_0", 225 "type": "assign", 226 "target": "YIELD_EST", 227 "sources": [], 228 "body": { 229 "type": "literal", 230 "dtype": "integer", 231 "value": "0" 232 } 233 }, 234 { 235 "name": "CROP_YIELD__assign__TOTAL_RAIN_0", 236 "type": "assign", 237 "target": "TOTAL_RAIN", 238 "sources": [], 239 "body": { 240 "type": "literal", 241 "dtype": "integer", 242 "value": "0" 243 } 244 }, 245 { 246 "name": "CROP_YIELD__assign__RAIN_0", 247 "type": "assign", 248 "target": "RAIN", 249 "sources": [ 250 "DAY", 251 "CONSISTENCY", 252 "MAX_RAIN", 253 "ABSORPTION" 254 ], 255 "body": [ 256 { 257 "type": "lambda", 258 "name": "CROP_YIELD__lambda__RAIN_0", 259 "reference": 24 260 } 261 ] 262 }, 263 { 264 "name": "CROP_YIELD__loop_plate__DAY_0", 265 "type": "loop_plate", 266 "input": [ 267 "CONSISTENCY", 268 "MAX_RAIN", 269 "ABSORPTION", 270 "RAIN", 271 "TOTAL_RAIN", 272 "YIELD_EST" 273 ], 274 "index_variable": "DAY", 275 "index_iteration_range": { 276 "start": { 277 "type": "literal", 278 "dtype": "integer", 279 "value": 1 280 }, 281 "end": { 282 "value": 32, 283 "dtype": "integer", 284 "type": "literal" 285 } 286 }, 287 "body": [ 288 { 289 "name": "CROP_YIELD__assign__RAIN_0", 290 "output": { 291 "variable": "RAIN", 292 "index": 0 293 }, 294 "input": [ 295 { 296 "variable": "DAY", 297 "index": -1 298 }, 299 { 300 "variable": "CONSISTENCY", 301 "index": -1 302 }, 303 { 304 "variable": "MAX_RAIN", 305 "index": -1 306 }, 307 { 308 "variable": "ABSORPTION", 309 "index": -1 310 } 311 ] 312 }, 313 { 314 "function": "UPDATE_EST", 315 "output": {}, 316 "input": [ 317 { 318 "variable": "RAIN", 319 "index": 0 320 }, 321 { 322 "variable": "TOTAL_RAIN", 323 "index": -1 324 }, 325 { 326 "variable": "YIELD_EST", 327 "index": -1 328 } 329 ] 330 }, 331 { 332 "function": "print", 333 "output": {}, 334 "input": [ 335 { 336 "variable": "DAY", 337 "index": -1 338 }, 339 { 340 "variable": "YIELD_EST", 341 "index": -1 342 } 343 ] 344 } 345 ] 346 }, 347 { 348 "name": "CROP_YIELD", 349 "type": "container", 350 "input": [], 351 "variables": [ 352 { 353 "name": "DAY", 354 "domain": "integer" 355 }, 356 { 357 "name": "RAIN", 358 "domain": "real" 359 }, 360 { 361 "name": "YIELD_EST", 362 "domain": "real" 363 }, 364 { 365 "name": "TOTAL_RAIN", 366 "domain": "real" 367 }, 368 { 369 "name": "MAX_RAIN", 370 "domain": "real" 371 }, 372 { 373 "name": "CONSISTENCY", 374 "domain": "real" 375 }, 376 { 377 "name": "ABSORPTION", 378 "domain": "real" 379 } 380 ], 381 "body": [ 382 { 383 "name": "CROP_YIELD__assign__MAX_RAIN_0", 384 "output": { 385 "variable": "MAX_RAIN", 386 "index": 2 387 }, 388 "input": [] 389 }, 390 { 391 "name": "CROP_YIELD__assign__CONSISTENCY_0", 392 "output": { 393 "variable": "CONSISTENCY", 394 "index": 2 395 }, 396 "input": [] 397 }, 398 { 399 "name": "CROP_YIELD__assign__ABSORPTION_0", 400 "output": { 401 "variable": "ABSORPTION", 402 "index": 2 403 }, 404 "input": [] 405 }, 406 { 407 "name": "CROP_YIELD__assign__YIELD_EST_0", 408 "output": { 409 "variable": "YIELD_EST", 410 "index": 2 411 }, 412 "input": [] 413 }, 414 { 415 "name": "CROP_YIELD__assign__TOTAL_RAIN_0", 416 "output": { 417 "variable": "TOTAL_RAIN", 418 "index": 2 419 }, 420 "input": [] 421 }, 422 { 423 "name": "CROP_YIELD__loop_plate__DAY_0", 424 "inputs": [ 425 "CONSISTENCY", 426 "MAX_RAIN", 427 "ABSORPTION", 428 "RAIN", 429 "TOTAL_RAIN", 430 "YIELD_EST" 431 ], 432 "output": {} 433 }, 434 { 435 "function": "print", 436 "output": {}, 437 "input": [ 438 { 439 "variable": "YIELD_EST", 440 "index": 2 441 } 442 ] 443 } 444 ] 445 } 446 ], 447 "name": "pgm.json", 448 "dateCreated": "2018-10-04" 449 }
Executable DBN - Loop plate representation¶
In [23]:
from delphi.program_analysis.scopes import *
root = Scope.from_json(dbn_json)
A = root.to_agraph()
jt.display_image(A.draw(format='png', prog='dot'))
Out[23]:
High-level representation of CAG from program¶
In [24]:
import sys
sys.path.append('../delphi/program_analysis/autoTranslate')
import lambdas
from delphi.program_analysis.ProgramAnalysisGraph import ProgramAnalysisGraph
from delphi.bmi import *
G = ProgramAnalysisGraph.from_agraph(A, lambdas)
initialize(G)
from delphi.visualization import visualize
visualize(G, show_values = True)
Out[24]:
In [25]:
update(G)
visualize(G, show_values = True)
Out[25]:
In [26]:
update(G)
visualize(G, show_values = True)
Out[26]:
Sensitivity Analysis¶
In [27]:
%matplotlib inline
import seaborn as sns
sns.set_style('darkgrid')
from IPython.display import set_matplotlib_formats
set_matplotlib_formats('retina')
import numpy as np
from matplotlib import pyplot as plt
from delphi.utils import compose, rcompose
from delphi.program_analysis.ProgramAnalysisGraph import ProgramAnalysisGraph
def make_plots(n_samples, deterministic = True):
variables = ('RAIN', 'TOTAL_RAIN', 'YIELD_EST')
vals = {k:[] for k in variables}
days = {k:[] for k in variables}
palette = sns.color_palette()
colors = {k:palette[i] for i, k in enumerate(vals)}
fig, axes = plt.subplots(1,len(vals), figsize=(18, 5))
ax = {k:axes[i] for i, k in enumerate(vals)}
for _ in range(n_samples):
G = ProgramAnalysisGraph.from_agraph(A, lambdas)
if not deterministic:
G.nodes['MAX_RAIN']['init_fn'] = lambda: np.random.normal(4, 1)
initialize(G)
for i in range(1,31):
update(G)
for k in vals:
vals[k].append(G.nodes[k]['value'])
days[k].append(G.nodes['DAY']['value'])
for k in vals:
sns.lineplot(days[k], vals[k], ax = ax[k], label=k, color=colors[k])
ax[k].set_xlabel('DAY', fontsize=20)
ax[k].set_ylabel(k, fontsize=20)
plt.tight_layout()
make_plots(10, deterministic=False)
In [ ]: