People are looking for the edge, where is the information that I can leverage to get ahead of the rest of the market? Unstructured data seems to be the untapped source, whether it is news, social or both.
The Financial industry clearly calls out for a new approach to predictive analysis or what Saffron refers to as anticipation or “anticipating the future”. Prediction in this industry has been characterized by the use of a massive amount of historical market data reduced to models about the past to create forecasts about the future. But when the present is characterized by the increasing complexity of financial instruments and the increasing volume, variability and velocity of available data, then models based on the past become inadequate, uninformed and misguided.
Some news organizations, like Reuters and their work with OpenCalais, have been trying to help by bringing unstructured text into this analysis. OpenCalais automatically “reads” Reuters content and marks up the people, places and things (entities) that are being written about in the news article.
Dow Jones takes a similar approach to entity markup and makes these details available to algorithmic traders for analysis. In the past, accurately representing the people, places and things in data has been a big challenge.
But the question remains, even after the markups are done, how do we capture all of this knowledge and better exploit it to inform decision-making and help us anticipate the future?
To this end, change is underway on a variety of fronts. Take the recent sentiment analysis of Twitter and the news to help anticipate socio-political change. By analyzing public sentiment, as expressed in text, you can identify changes in public opinion or attitude. Arab Spring is a good example:
A new study by Computer scientist Kalev Leetaru, at the University of Illinois at Urbana-Champaign, claims you could have foreseen the Arab Spring if only you’d been paying enough attention to the news. Leetaru shows how data mining of news reportage can reveal the possibility of future crises well before they happen. Leetaru mined news reporting to identify the sentiment, as reported, on Tunisia, Egypt and Libya. “In all cases”, he says, “there was a clear, steady trend towards a negative tone for about a decade before the revolts.”
Monday morning “quarterbacking” is always easier. Leetaru knew that long-established dictatorial political leaders had been deposed. In other words, he knew what to look for in this case. Despite that, his argument is that sentiment analysis can be an indicator of an impending event, crisis or opportunity.
The ability to analyze unstructured data sources is an important component of analysis, such as anticipating socio-political unrest or mitigating brand reputation risk or determining the best approach to replacing damaged non-inventoried components. Incorporating unstructured data sources in business decision-making is proving to be a viable component of business intelligence. It provides context, facts and additional insight about a topic or transaction that cannot be expressed by predefined structured data.
The automatic combination of text or unstructured data with structured data creates a new, unified knowledge source. This marriage of structured data with unstructured sources not only changes the way analysis is done but creates a more informed, more complete and more accurate knowledge base to exploit for decision making and anticipation.
Without this, text analysis systems are just saying that the overall sentiment toward Egypt is bad (and you know this by reading and looking at the overall trend).
But a deeper understanding of the current (and past) relationship of people, places, things and events makes anticipating potential outcomes possible and thereby creates actionable, informed intelligence for risk management, reputation management, and a variety of business decisions.
In our personal finances, we want to know “should I invest in this stock or not? Should I divest of this stock or not? Should I stay in equities or move to cash?”
Elevating business intelligence to answer such questions cannot be accomplished by just analyzing unstructured text sources.
At Saffron we’ve created a new school of business intelligence that combines text sources with more traditional structured, transactional data sources. We do this through a specialized data index based on associations. Saffron stores the semantic structure of your data, along with frequency counts so that relevance metrics can be computed. Now you have an agile knowledge source to exploit, in real time, for decision support and anticipation applications. You can now answer questions such as “which of our customers will churn given a competitors new product announcement? Who are these customers? Why will they churn? What has happened to provoke churn?”, without predefined models or rules, and instead using all the information available about a customer, your interactions with them, all of their transactions with you and ultimately their interests and concerns as expressed in their communications with you.
Understanding how and why markets and customers move as supported by a variety of real time and historical data about indicators, transactions, socio-political events, and public sentiment is the Holy Grail of Analytics. Bringing all this together into a naturally predictive system is the Holy Grail of the Financial Industry.
On second thought, isn’t this the Holy Grail for all industries?
You have to first understand that there is a plethora of information locked away in unstructured sources. The ability to get to that data is somewhat convoluted.
Saffron’s experience with national security pre-9/11 has highlighted this ‘big data’ problem particularly with data coming from news, email and many other sources of unstructured content. Saffron figured out that you could use plenty of tools, called ‘entity extractors’, as one element of dealing with text. So if you read documents and comments it is not just about storing the text or data, but about reading it in some sense of finding what:
What things are?
What are the parts?
What are the problem descriptions?
Who are the people?
What are the companies mentioned?
What products are mentioned?
When you start using these ‘entity extractors’ to find, and mark, the people, places and things the situations, problems, actions and outcomes give Saffron a memory base. This leads to opportunities to learn about those things and connections between them, the connections between people and the connections between problems and solutions.
It’s in the text and part of the solution to address text is first to use these technologies of marking people, places and things, situations, actions and outcomes. If you mark these then Saffron as a memory base can help you learn about all those people, places and things, situations, actions and outcomes and how they are connected. That’s the knowledge, and the connectivity of the data. It is not just the raw data itself and this is how Saffron gets into the unstructured text to exploit it.
Natural approach sees 10 to 20 times return on mission.
When people make decisions, we recall our past experiences to find the analogies and similarities. What was like this situation? What was different? What did I do?
How did it turn out?
This is the way the world works… except in business where increasing focus on data has meant more rule based decision-making.
“Less than 20% of information we need to make decisions is stored in the company’s data warehouses. A new approach is needed… DBMS are not capable of handling and searching through all the types of data we have – email, text, voice, video… ” – Donald Feinberg, Gartner VP
High Velocity, High Complexity, High Diversity
A fuzzy world of diverse, complex and disorganized data sources can’t accurately be explained via tables and SQL queries. Indeed, 80% of the answers can’t even be found within your business.
Big Data is not just big in volume but big in velocity as intelligence goes real time.
According to Forrester – current BI technologies are “falling short in the ever-faster race” to meet business needs. (Download report free)
Unlocking experience unlocks ROI
Like natural human intelligence, Saffron focuses on how People, Places and Things fit together.
Like a true corporate memory, it doesn’t require rules, models or statisticians. You don’t even have to decide what to include or exclude. Everything you can know is included.
One global manufacturer has experienced a 10x return on cost of ownership versus savings in year 1.
And because there are always new problems to solve, this is a recurring business case. In year 2, ROI is expected to grow to 20x.
Democratized and operationalized.
BI is no longer just for forecasting, or just for management. Democratized data means real time intelligence within your company. Available to anyone and everyone as needed to get the job done.
Saffron Advantage: experience intelligence without rules.