Machine Learning in Finance

This reading describes how financial data science draws on big data: its types, sources and characteristics, and how the data are processed and visualized. It then explains data mining and machine learning (supervised, unsupervised and reinforcement), overfitting and underfitting, the split into training, validation and test datasets, neural networks, and natural language processing and large language models.

LOS 11.a — Big data, machine learning and AI in finance

What financial data science is for

Financial data science applies both quantitative and qualitative analysis to generate insight into a specific financial question, for example how customers behave or where market prices may move next. Among its main uses in finance are (1) evaluating investment opportunities, (2) optimizing portfolios and (3) mitigating risk. Fintech is where finance and technology meet. Of most interest in investment management are the fintech developments that aid quantitative analysis. The word is also used more broadly for technology developments applied to financial services and for the companies that build them.

The objective is the insight itself. Data-quality work, described below under the characteristics of big data and under curation, supports that objective but is not the objective.

Big data: types and sources

Big data is the broad term for all the potentially useful information being generated. By format it is:

Data come from traditional sources (markets, company financial reports, regulatory filings, government statistics) or alternative data sources:

• Individuals: social media activity, clicks, searches, time spent on web pages (mostly unstructured and growing fast).
• Business processes: transaction and corporate records such as sales data or banking records (usually structured). They can act as leading or real-time indicators, whereas quarterly reports give a lagging view.
• Sensors: smartphones, connected cars, smart buildings, satellites. The network of these devices is the Internet of Things (IoT); it produces the largest volumes, mostly unstructured.

Buying alternative data or collecting it by web scraping (automated extraction from websites) raises legal and ethical issues: some of what is captured may be personal information protected by privacy rules.

The "Vs" of big data

A terabyte is 1,000 gigabytes and a petabyte is 1,000 terabytes. Faster data generate more of it: volumes rise from batch collection (megabytes) through periodic (gigabytes) and near-real-time (terabytes) to real-time feeds (petabytes). More complex structures also need more storage.

The first three are the classic 3 Vs (volume, velocity, variety); veracity is the fourth V that matters especially for financial data. Other data-quality concerns are outliers, bad or missing data, sampling biases, and whether the amount collected is sufficient and appropriate for the intended use.

Data science: processing and visualization

Data science is the field concerned with extracting information from big data. Its processing methods:

A data lake is a central repository that keeps structured, semi-structured and unstructured data in their raw, original format. Distributed computing frameworks let data be processed and analyzed on many machines at once.

Visualization: charts and graphs suit structured data; less structured data call for word clouds (show how often words occur in a text sample), heat maps, 3D graphics and mind maps (show logical relations between concepts).

Typical inputs include share prices and trading volumes, asset returns, interest rates, exchange rates, accounting figures and macroeconomic series. The work calls for expertise in finance as well as in data handling, because much of the data is sequential, time dependent and sometimes cyclical. Financial data bring special problems: noise, nonstationarity, interdependencies, nonlinearity, fat tails (extreme events more likely), very high-frequency data in trading, and too little data in some markets (e.g., emerging markets). Regulated firms must meet strict rules on data management, risk assessment and model validation; firms outside financial regulation still have to comply with business law and their contracts, although they have more freedom in how they handle data. Data privacy, thorough model testing and human review of model decisions matter for every firm.

Data mining, AI and machine learning

Data mining searches large datasets (usually structured) for patterns that can be used to predict outputs; pattern identification rests on conditional probabilities (e.g., Bayes' theorem). Its key risk is overfitting.

Artificial intelligence (AI) refers to computer systems that simulate human thinking. Machine learning (ML) is a major subcategory of AI: an algorithm receives input data, without any assumption about how they are distributed, possibly with target outputs, and learns without human help to model the outputs or to find patterns.

A supervised model trained on a large, reliable dataset should predict well but may not adapt to new situations. Reinforcement learning is costly: designing a reward system that actually improves the model can take much training and many resources. Whatever the method, people with subject expertise are needed to choose the model, clean the inputs, limit bias and interpret the output.

Deep learning uses many layers of neural networks to find patterns of increasing complexity and may use supervised, unsupervised or semi-supervised learning (image and speech recognition, natural language processing).

Overfitting vs. underfitting

Other ML issues: results can be a "black box" (hard to explain), and data leakage occurs when information from outside the training dataset influences what the model learns.

Training, validation and test datasets

• Training dataset: the model looks for relationships here; it is the largest set, about 60%–80% of the data (upper end for large datasets).
• Validation dataset: used to refine (tune) the relationships found in training.
• Test dataset: used last, to judge the model's predictive ability.
  The validation and test sets share the remaining 20%–40% equally. Observations are normally assigned at random, which reduces bias, overfitting and data leakage. Time series data must keep their chronological order instead:
• Time-based split: the oldest 60%–80% of observations go to training, the next 10%–20% to validation and the most recent 10%–20% to testing.
• Rolling-window validation: train on one window, validate on a later period, then shift the window forward and repeat.
  Either way, the span covered by validation plus test data should match the intended forecast horizon.

Neural networks, NLP and LLMs

Neural networks suit nonlinear relationships with many interdependencies: an input layer, several hidden layers of weighted nodes (neurons), and an output layer. They can learn in a supervised or unsupervised way, and each neuron's weight reflects its importance. They are often trained by backpropagation (a form of supervised learning): a forward pass produces a prediction that is compared with the actual outcome, the loss is computed, the loss gradient shows how weight changes affect the error, and gradient descent adjusts the weights step by step to minimize the loss.

• Generative adversarial networks (GANs) pit a generator network against a discriminator network to create realistic synthetic data (e.g., plausible market scenarios for stress tests, trained on historical market data). Because GANs can also produce convincing fake material, their output needs careful review.
• Variational autoencoders (VAEs) compress data and then reconstruct them using encoder and decoder networks. They are probabilistic generative models, used to reduce dimensionality and spot anomalies.
• Copulas (e.g., the Gaussian copula) are a more traditional mathematical tool for nonlinear relationships that uses marginal probability distributions to capture dependencies between random variables. Exam convention: the Cholesky decomposition is given as an example of a copula. Current practice: the Cholesky decomposition is a matrix factorization, used to generate the correlated normal draws in a Gaussian-copula simulation.

Natural language processing (NLP) lets computers interpret human language: speech recognition, translation, scanning employee communications for compliance, and detecting sentiment shifts in research reports or earnings-call transcripts. Text analytics works mainly with unstructured material, such as filings and research reports or recordings of earnings calls and surveys. NLP converts that material into usable data much more quickly than a person could and searches it for patterns; lexical analysis, for example, measures how often each word appears. Large language models (LLMs) are generative AI that produce human-like text, using neural networks to read context and meaning in qualitative inputs (nonfinancial inputs such as election polls can matter too). Their output is text such as a summary or a forecast based on tone and sentiment. They are typically trained on a very large dataset through self-supervised learning, an unsupervised approach in which the model generates the labels itself from raw data. They are not built to catch errors in financial data, so outputs need expert review and should be used alongside traditional data-driven models.

Common tools are Python (fintech apps), R (statistics), Java (runs across platforms) and C/C++ (high-speed algorithmic trading). Common databases are SQL (structured data, on a server), SQLite (structured data, not on a server; common in mobile apps) and NoSQL (unstructured data).