People are looking for the edge, where is the information that I can leverage to get ahead of the rest of the market? Unstructured data seems to be the untapped source, whether it is news, social or both.
The Financial industry clearly calls out for a new approach to predictive analysis or what Saffron refers to as anticipation or “anticipating the future”. Prediction in this industry has been characterized by the use of a massive amount of historical market data reduced to models about the past to create forecasts about the future. But when the present is characterized by the increasing complexity of financial instruments and the increasing volume, variability and velocity of available data, then models based on the past become inadequate, uninformed and misguided.
Some news organizations, like Reuters and their work with OpenCalais, have been trying to help by bringing unstructured text into this analysis. OpenCalais automatically “reads” Reuters content and marks up the people, places and things (entities) that are being written about in the news article.
Dow Jones takes a similar approach to entity markup and makes these details available to algorithmic traders for analysis. In the past, accurately representing the people, places and things in data has been a big challenge.
But the question remains, even after the markups are done, how do we capture all of this knowledge and better exploit it to inform decision-making and help us anticipate the future?
To this end, change is underway on a variety of fronts. Take the recent sentiment analysis of Twitter and the news to help anticipate socio-political change. By analyzing public sentiment, as expressed in text, you can identify changes in public opinion or attitude. Arab Spring is a good example:
A new study by Computer scientist Kalev Leetaru, at the University of Illinois at Urbana-Champaign, claims you could have foreseen the Arab Spring if only you’d been paying enough attention to the news. Leetaru shows how data mining of news reportage can reveal the possibility of future crises well before they happen. Leetaru mined news reporting to identify the sentiment, as reported, on Tunisia, Egypt and Libya. “In all cases”, he says, “there was a clear, steady trend towards a negative tone for about a decade before the revolts.”
Monday morning “quarterbacking” is always easier. Leetaru knew that long-established dictatorial political leaders had been deposed. In other words, he knew what to look for in this case. Despite that, his argument is that sentiment analysis can be an indicator of an impending event, crisis or opportunity.
The ability to analyze unstructured data sources is an important component of analysis, such as anticipating socio-political unrest or mitigating brand reputation risk or determining the best approach to replacing damaged non-inventoried components. Incorporating unstructured data sources in business decision-making is proving to be a viable component of business intelligence. It provides context, facts and additional insight about a topic or transaction that cannot be expressed by predefined structured data.
The automatic combination of text or unstructured data with structured data creates a new, unified knowledge source. This marriage of structured data with unstructured sources not only changes the way analysis is done but creates a more informed, more complete and more accurate knowledge base to exploit for decision making and anticipation.
Without this, text analysis systems are just saying that the overall sentiment toward Egypt is bad (and you know this by reading and looking at the overall trend).
But a deeper understanding of the current (and past) relationship of people, places, things and events makes anticipating potential outcomes possible and thereby creates actionable, informed intelligence for risk management, reputation management, and a variety of business decisions.
In our personal finances, we want to know “should I invest in this stock or not? Should I divest of this stock or not? Should I stay in equities or move to cash?”
Elevating business intelligence to answer such questions cannot be accomplished by just analyzing unstructured text sources.
At Saffron we’ve created a new school of business intelligence that combines text sources with more traditional structured, transactional data sources. We do this through a specialized data index based on associations. Saffron stores the semantic structure of your data, along with frequency counts so that relevance metrics can be computed. Now you have an agile knowledge source to exploit, in real time, for decision support and anticipation applications. You can now answer questions such as “which of our customers will churn given a competitors new product announcement? Who are these customers? Why will they churn? What has happened to provoke churn?”, without predefined models or rules, and instead using all the information available about a customer, your interactions with them, all of their transactions with you and ultimately their interests and concerns as expressed in their communications with you.
Understanding how and why markets and customers move as supported by a variety of real time and historical data about indicators, transactions, socio-political events, and public sentiment is the Holy Grail of Analytics. Bringing all this together into a naturally predictive system is the Holy Grail of the Financial Industry.
On second thought, isn’t this the Holy Grail for all industries?
You have to first understand that there is a plethora of information locked away in unstructured sources. The ability to get to that data is somewhat convoluted.
Saffron’s experience with national security pre-9/11 has highlighted this ‘big data’ problem particularly with data coming from news, email and many other sources of unstructured content. Saffron figured out that you could use plenty of tools, called ‘entity extractors’, as one element of dealing with text. So if you read documents and comments it is not just about storing the text or data, but about reading it in some sense of finding what:
What things are?
What are the parts?
What are the problem descriptions?
Who are the people?
What are the companies mentioned?
What products are mentioned?
When you start using these ‘entity extractors’ to find, and mark, the people, places and things the situations, problems, actions and outcomes give Saffron a memory base. This leads to opportunities to learn about those things and connections between them, the connections between people and the connections between problems and solutions.
It’s in the text and part of the solution to address text is first to use these technologies of marking people, places and things, situations, actions and outcomes. If you mark these then Saffron as a memory base can help you learn about all those people, places and things, situations, actions and outcomes and how they are connected. That’s the knowledge, and the connectivity of the data. It is not just the raw data itself and this is how Saffron gets into the unstructured text to exploit it.
While applicable in virtually any business environment, Sense Making is well known for its value in military Intelligence analysis. Saffron has first-hand experience applying this dimension of our robust data analytics solution to the problem of IED (Improvised Explosive Device) defeat in Iraq.
Sense Making Objective
Draw on Experience to save lives by finding enemy insurgents and stopping random bombings before they occur.
The U.S. Army and its coalition partner war-fighters in Iraq needed the ability to decide and act at the tactical level to eliminate, capture or exploit insurgent targets directly or indirectly involved in IED activity. To be effective, this targeting had to be done prior to actual IED attacks (and at the time of our forces’ choosing) in order to prevent loss of life for both civilians and war-fighters.
Saffron Technology played a lead role on the team formed to enable and enhance the analytic workflow of war-fighting intelligence analysts. SaffronAnalyst was developed by Saffron to provide an analyst user interface consistent with the workflow and thought processes of an actual Intelligence analyst. Although the system was already capable of identifying associations and their frequencies between basic entities like people, places and things — Saffron was asked to provide this analysis within the existing analyst workflow by integrating Saffron’s Sense Making capabilities with other third- party tools to provide an end-to-end analytic workbench.
The Customer-Saffron Solution
We worked closely with military Intelligence analysts, and Saffron’s engineers were trained in all the current methods of analysis used by the customer. The customer’s existing processes involved manually extracting entities and building a co-occurrence matrix. Saffron identified immediate opportunities where the Saffron Natural Intelligence Platform could improve the “time and speed to results” over their current manual process.
Saffron engineers observed Intelligence analysts in action and spent time on-site witnessing their analytical process in real time, against real targets. This made it easier to identify where SaffronAnalyst should integrate into the analysts’ workflow, so as to enhance — but not disrupt — current procedures.
The solution called for adding new features and capabilities to Saffron’s existing product to incorporate the desired components of the current manual process. At the same time, the customer created new approaches to parsing unstructured data. To ensure a streamlined and accelerated data-analysis process for the customer’s smaller operational units, Saffron and the customer teamed with a third-party ETL (data Extract, Transform and Load ) vendor.
Integration of the two products was developed, tested and implemented in under four weeks. Intelligence analysts reduced the time for creating their Critical Entity Link Charts by 80% in the final benchmark test. And Saffron further enhanced its Sense Making tool.
Many organizations face a similar challenge of quickly analyzing large volumes of intelligence data with a small team of people. Saffron Natural Intelligence Platform’s Sense Making capabilities make it ideal for helping organizations and their workgroups analyze large volumes of such data in a short amount of time.
Instead of two-dimensional models and manually designed tables, Saffron uses dynamic, multi-dimensional memories to quickly identify connections and their frequencies of occurrence within massively dense and “noisy” data sets.
In this particular case, Saffron was integrated with entity extraction tools including SRI, Inc.’s NetOwl and Basis Technology, Inc.’s Rosette, and ingested varied unstructured data sources into SaffronMemory Base. Saffron’s analytic results were then integrated with i2, Inc.’s Analyst Notebook to provide complete transportability of the Connections, Analogies and other supporting evidence from Saffron directly to the Intelligence analyst’s desktops. In addition, new visualization capabilities for SaffronAnalyst were added, including new link analysis capabilities, entity management, export to ESRI’s ArcGIS product, analyst interest collaboration, and more. Throughout, Saffron’s Connections, Analogies and Network Reasoning Methods were used in this Sense Making implementation.
The value emerges in the extraordinary speed and accuracy with which the Saffron Natural Intelligence Platform returned its data analysis, giving Intelligence analysts enough confidence to act, and enough time to save lives.
Experience Management by Sense Making is Saffron’s proven, sophisticated method of fast, comprehensive and pertinent data analysis.