The term "big data" is typically used to describe extremely large sized data sets. Financial data management has come to the forefront as service firms looking at ways to control data more effectively. The amount of data continues to grow on an exponential basis, while regulatory requirements force firms to take proactive approaches for issues such as risk management.
More users inside financial firms continue to require access to growing data sets to help uncover market opportunities, product development possibilities and customer trends. Financial reference data has become particularly critical with regard to risk factors and regulatory reporting aspects.
Data management has always been a challenge for capital markets. As such, the industry has spent literally billions of dollars on a collective basis in an attempt to create accurate and complete datasets.
This so-called "golden copy" has improved, particularly for certain asset classes. However, data sharing across financial institutions remains a challenge. Business units often prefer to refer to their own specific set of data for calculations, which makes business-wide data analysis extremely difficult.
There are increasing numbers and types of unstructured data for collection, analysis and storage. For example, traders who are interested in the latest news concerning specific companies and industries have developed useful tools to analyze real-time news. Many advanced trading operations have also developed tools to assist in decision making.
More financial reference data is also transmitted using a variety of digitized sources such as video, audio and social media outlets such as Twitter. As a result, some firms have tried to devise various ways in order to analyze these massive amounts of data. Firms have also begun analyzing data from documents, websites, surveys and search traffic, among many more digital media outlets.
For risk management and regulatory issues, users must look at data from across the entire business. Often, it needs to be compared to data from both markets and other data sources. As terabytes have grown into petabytes, the existing structures of relational databases are having difficultly keeping up.
Wall Street financial institutions are continually searching for more effective ways to handle larger data sets. Big data techniques are also being used to manage advanced analytics and regulatory compliance.
Providers of larger databases also offer a wide variety of products at customizable price points based upon applications, data set sizes, processing strength and other variables.
In short, the price tag for developing financial data management systems is high. However, in the interest of technological science, open source developers are working diligently on big data solutions for financial institutions.