Conditionalism as a Methodology of Forecasting, Decision-Making and Machine Learning ()
1. Introduction
The process of understanding physical, economic, or social reality and subsequent forecasting and decision-making inherently follows a strict protocol. Initially, observations or experimental data are collected and analyzed. This initial data sample, i.e., learning or training data, is used primarily to discover existing relationships or laws. Then, the findings from the learning data sample are formalized, generalized, and tested on more observations or experiments. After sufficient satisfactory testing, we use these findings to produce more forecasts and decisions.
The outlined protocol works perfectly well in natural sciences because experiments can reliably reproduce the data, and the learning data is limitless in practical terms. One can always run more experiments with fixed controls to produce additional missing data and refine conclusions.
We tend to follow a similar approach in areas of knowledge dealing with day-to-day life covered by “soft” rather than “hard” disciplines, such as social sciences, economics, finance, etc. We also start with a sample of learning or training data. Here, the concept of the training data is more general than a series of numbers: it can be a sequence of historical events, business decisions, or specific court cases. The scientific protocol also analyzes the learning data to establish relationships, dependencies, or form opinions and judgments. As in the natural sciences, we produce forecasts and make decisions based on the findings from the learning data sample.
However, the data we use in worldly disciplines is fundamentally different from natural sciences. Unlike scientific data, the data sample in these fields is limited. A bulk of data comes from history, bounded by a particular historical time. Often the number of available data points is scarce, without any opportunity of obtaining more observations, such as people with specific experience, a particular sequence of legal cases, results of irreproducible economic policies, etc. Most importantly, the data usually is colored by historical background, specifics of the experimental conditions, or other factors that may lie well outside of our understanding or even realization of their existence. In scientific terms, it is difficult, if not impossible, to select or produce a clean sample of data with proper controls. It makes any data available to researchers in these fields fundamentally biased, to a higher or lower degree.
In this paper, we address the problem of the existence of hidden conditionality, or biases, in the learning or training data samples. We show that the distribution of outcomes that follows our forecasts and subsequent decisions based on this biased learning data is not an unconditional distribution in mathematical terms. On the contrary, our choices strongly depend on the conditions inherent to our chosen learning data. We argue that we often forecast and decide based on conditional rather than unconditional data so that our projections and decisions bear similar conditionality as the learning data.
The initial analysis often misses the realization of this conditionality (Siniscalchi & Battigalli, 2003; Bradley, 2011). The conditionality is difficult or impossible to quantify, or we do not know if it exists. Without this realization, the rationale for our decision-making bears the same conditionality as when we form our decision-making framework. As a result, the same conditions that went into the decision may no longer exist, but since we continue making decisions on the same basis, they may turn out completely wrong.
In what follows, we analyze several examples from different areas to illustrate how an understanding of conditionality leads to forecasts and decisions opposite to conclusions based on unconditional distributions (Bradley, Dietrich, & List, 2014). Our analysis results are a methodological forecasting and decision-making framework with a strong emphasis on understanding the conditionality of the learning data sample, which we call Conditionalism.
2. Logic Puzzles
To show that our immediate perception can produce results that drastically differ from exact math by missing a hidden conditionality, we will look at a well-known mathematical puzzle. A coin has been tossed ten times in this puzzle, with “heads” coming up every time. What is the probability of “heads” coming up on the eleventh toss?
A mathematically inclined person will immediately get a correct answer that this probability is precisely 1/2, as in any consecutive toss. However, this answer may not sit well with the rest of us: it seems virtually impossible that the “heads” will be coming up eleven times in a row as the probability of it happening appears relatively small. Our question is not a solution to the puzzle, but why is there such a vast gap between the exact mathematical answer and what seems to be intuitive common sense?
The answer lies in our inability to perceive conditional probabilities and distributions. In a common-sense approach, we view the likelihood of “heads” outcome of the eleventh toss as an unconditional probability, correctly arriving at the probability of the eleven straight “heads” as
, which indeed is extremely small. However, we fail to account that the event of eleventh straight “heads” is conditional on the ten “heads” that has already happened, which has the unconditional probability of
. Applying the law of conditional probabilities, we arrive at the correct answer:
Here
is the conditional probability of having eleven straight “heads” given that ten straight “heads” already happened,
and
are the unconditional probabilities of having ten and eleven straight “heads,” respectively.
This trivial example illustrates how ignoring conditionality leads to entirely wrong conclusions. The reason we employed a mathematical puzzle before considering real-life cases is that here our argument of missed inherent conditionality has its purest form, free from any additional considerations that typically go into any real-life situation.
3. Hiring Practices
Accounting for the conditionality of applicants’ backgrounds can often influence the evaluation of prospective candidates in employment practices.
Let us assume two candidates interview for a job opening. They both will be college graduates without any significant prior work experiences for a straightforward comparison. In this situation, college credentials and perceived abilities are the primary basis for hiring. A standard practice to get a sense of these skills in an interview is to administer an informal test, usually by asking to solve some problems and answer questions requiring a certain amount of knowledge and cognitive ability. To illustrate our argument, we will be offering a job based solely on the results of these tests and their interpretation. No additional considerations such as personality, communication skills, etc., commonly a factor, will influence our decision.
In our interview process, two candidates, A and B, undergo this type of evaluation. Assume that candidate A scored 90 on the informal test, while candidate B scored 85. The job will likely go to A because this candidate is ahead.
When we decide to offer the job to candidate A, our decision is based purely on the unconditional distribution of the test scores. Conditionality, in this case, reflects both candidates’ educational backgrounds. Let us suppose it turns out that candidate A came from an advanced program, while candidate B’s program has a relatively average academic standing. If we were to quantify each candidate’s program standing, A’s would be, say, 95. This score means that students from the same program as A will receive 95 on our test, on average. The B’s program score would be 70, reflecting a lower education quality.
Our test results may look completely different after accounting for the differential in educational backgrounds. With proper conditioning, A is an under-performer by 5 points because, coming from an advanced educational experience, we expect A to test at least 95 to be above that program’s average. Meanwhile, B is almost a star scoring 15 points above B’s average peer in the program.
The realization of conditionality can radically change our employment decision. Despite a lower absolute test score, we may see candidate B as a significantly more able person.
4. Investment Advice
Proper realization of conditionality is crucial to viewing asset values and issuing investment advice. Same financial data produces diametrically opposite conclusions, depending on the “market context.” “Market context” is an investment industry jargon for the hidden conditionality of data.
In this example, we will be looking at XYZ’s company and will attempt to evaluate whether owning XYZ’s stock is a good investment.
Let us assume that after looking through financial reports and filings for the company, we see that XYZ’s business is a reliable generator of cash. XYZ also does not have any debt or loans, not owing anything to any creditors. Combining this financial data signals a stable company, which business is in good shape and no need to borrow money, impacting a company’s financial standing.
So, what is the proper value of XYZ’s stock, and is it a good investment? It depends on the “market context,” or conditions under which we observe all XYZ’s desirable accounting features. During the crisis of 2008-09, owning XYZ’s shares would be a good investment. In that period, investors were worried about companies’ ability to carry their debt, which would not exist for XYZ due to the complete absence of borrowing. Investors were also worried about the sustainability of a company’s business. They looked carefully at whether the company makes enough immediate cash from its operations, a problem XYZ also does not have.
Based on this same financial data, the analysis results would be the opposite for 2013-14. XYZ not having any debt is negative because borrowing was easy and cheap. The absence of debt could imply XYZ’s management’s failure to pursue business opportunities such as potential expansions or acquisitions. From the perspective of competitiveness, the cash flow that XYZ generates, although presently strong, would be viewed as temporary and prone to erosion as XYZ’s peers potentially catch up as they pursue available business opportunities. As a result, in such a stock market, XYZ’s stock would not get very far, despite excellent financials.
The fact that the same data can lead to diametrically opposite conclusions further illustrates the importance of understanding the conditions under which we obtain the data and draw conclusions, rather than treating the data as unconditional and applying results of the financial analysis without accounting for the “market context.”
5. Model-Based Forecasting and Decision-Making
In today’s world, business, management, and other decisions are made not exclusively by individuals but increasingly, sometimes even solely by mathematical models. As mathematical models rule ever-increasing aspects of our lives well beyond strictly scientific or engineering applications, the models are not free of similar conditionality issues, no matter how mathematically complex or rigorous they are.
Economic forecasting is essential for today’s financial system (Armstrong, 2001). Reliable forecasts of economic data and decisions based on these forecasts are vital for investment institutions, banks, and regulatory bodies, to name a few (Green & Armstrong, 2015). Sophisticated mathematical and financial models are often responsible for producing economic forecasts. They are often directly involved in making economic and financial decisions, in some cases at an automatic level. Each of these models uses numerical parameters which are specific to the model. In a calibration procedure, numerical values of the model parameters are determined by looking at what the model would have predicted historically and matching past predictions of the model with observed history by adjusting numerical values of the model parameters. The use of calibration procedures is relatively widespread and necessary. Virtually any economic or financial model undergoes at least some degree of calibration.
Calibrations rarely consider the conditionality of the historical data points. Each historical data point or sub-series may turn out to be conditioned by a separate set of factors, not reflected in the model. Proper conditioning is not an easy task as economic reality can be complicated. Quite often, specific economic factors or combinations of causes may not even be entirely transparent. Thus, economic models are frequently calibrated based on the unconditional distributions of historical data, without clear realization and adjustments for these dependencies.
Hidden conditionality explains why some economic models seemingly work well to reproduce historical data. They arrive at correct past would-be decisions but then perform poorly afterward, failing to explain contemporaneous data resulting in suboptimal decision-making. In the simplest case, a model’s parameters reflect a state of the economy that existed in the past. These parameters need appropriate adjustment as the underlying conditions change.
In a more complex and more typical case, a model was calibrated over a period when the underlying conditionality was changing. The set of parameter values obtained in this case usually is unreliable. Limited ability to account for the conditionality of historical data is often behind inaccurate forecasts. Thus, a model’s forward-looking predictions are reliable only when the underlying conditional dependencies are weak or fully reflected by the model, the latter being quite rare given the general complexity of the economic driving forces.
The forecasting deficiency based on the conditionality of data is rather fundamental and inherent to economic forecasting. The conditional dependence can be complicated, and there is usually no large enough data sample to fully capture it. The forecasting error can be utterly unpredictable since the underlying conditionality is not transparent and challenging to detect. Since this error is of modeling rather than statistical type, one cannot reliably estimate it by existing probabilistic techniques.
We suggest interpretative oversight as a practical solution to the deficiency of forecasting data. With a profound understanding of a particular model’s workings and signaling, the model is useful solely when concurrent conditions bear sufficient similarity, i.e., conditionality, with the data of the learning sample or period where the model is known to perform well. We will discuss this approach in a more general context of machine learning.
6. Interpretative Oversight in Machine Learning
Various machine learning and artificial intelligence approaches are not immune to the conditionality of the learning data set (Russell & Norvig, 2010). The discussion of model-based forecasting of Section V fully applies here since computer-based learning rests on sophisticated mathematical and computer science methods, often being a suite of several integrated models and techniques.
Projections of the artificial intelligence methods depend on the analysis of the training data of the learning sample to choose specific parameter values, just as in the calibration of economic models over a set of historical data. Machine learning results will possess the same conditionality as the learning sample. Thus, after a specific machine learning method has been “trained,” its predictions and forecasts will only be applicable when similar conditions persist or re-occur in the current data.
An ability to detect or recognize the specific conditionality of the current or training data may lie outside a particular model or method. Thus a decision to deploy it can be exogenous to the technique. Still, if the only choice exogenous to the model is whether the model is “on” or “off,” while strictly following other model-generated results are, interpretative oversight does not reduce the systematic nature of machine learning. Altering or re-interpreting machine results makes the decision-making process style discretionary.
Accounting for the possibility of unknown conditionality of the learning data is vital when dealing with nonlinear systems. In an example of self-improving systems, nonlinearity determines the principal part of the dynamics of a system (Krakovsky, 2016, 2018). Any change in a system’s nonlinearity will result in a drastic difference in the system’s behavior. If machine learning fails to capture the existence of such nonlinearity due to a different regime in a learning stage, the results of machine forecasting would be particularly unreliable.
7. Implementation in Portfolio Management
Any financial manager relying on mathematical or machine learning models is subject to the conditionality of those models’ training or calibration data. Preference for nonparametric models and interpretative oversight become essential selection principles for practical implementations:
Nonparametric models do not explicitly rely on parameters or variables. Model parameters typically require constant calibration or maintenance associated with statistical distributions and may even be distribution-free (Wasserman, 2007). The advantage of these decision-making methods is evident as they do not explicitly rely on the training data, thus virtually avoiding dependency on the conditionality inherent to any such data. These are some of the most sought-after models and are very difficult to come by. Nonparametric models significantly reduce or eliminate the number of model parameters that need periodic adjustments in the decision-making.
Interpretative oversight plays a crucial role in dealing with the models requiring calibrations and training data. In the spirit of this technique, expert knowledge of the market conditions infers a potential conditionality of the training data and associated limitations in choosing particular model parameters. A model is applied only when similar conditions re-occur. This approach requires a solid understanding of the market conditions and insight into the relationship between a model’s mathematics and market developments. In other words, one needs to explain what economic events produce which mathematical signals, and vice versa, what model signals come from what kind of plausible economic developments.
Interpretative oversight also helps develop particular machine learning techniques or models. Observation of incremental successes or failures of model predictions based on current data may allow gradual integration of specific conditionality into the model, increasing the scope of the model’s applicability.
Interpretative oversight is very different from a common approach of “results interpretation,” which combines model-generated decisions with an outside judgment. In this widely practiced approach of handling model inefficiencies, the model results are often overwritten in a discretionary way by “common sense” or “industry knowledge” considerations, reducing the role of mathematical models in decision-making to only one of the many contributing factors.
Despite common practice, “result interpretation” has two significant limitations. Overwriting the model results and replacing them with discretionary exogenous decisions outside of the model’s scope breaks the systematic nature of the decision-making process. It makes decision-making predominantly discretionary, with the model-generated signals as only one of the components of investment decisions.
Unlike “results interpretation,” interpretative oversight preserves the systematic nature of the decision-making process because once a model is deemed applicable, one should not overwrite model-generated decisions.
8. Generational Gap
A curious and entertaining example of the principles of proper conditioning is a millennia-old problem of the generational gap, as one can relate it to the conditionality of the training data.
Any generation has a specific cultural, economic, political, and historical background. The background inevitably influences the decision-making framework of representatives of each generation. In a more formal language, the learning data sample that we inherently use to train or “calibrate” our future decision-making rationale will bear the same conditionality as the learning data itself, namely the combination of socio-economic factors prevalent for that generation. As a result, the same conditionality that went into the corresponding learning data colors decision made by each person. For example, people who grew up during the Great Depression may be conservative in their financial choices (McComb, 2006).
A consecutive generation will have their own set of learning data that goes into their decisions, influenced by their own set of socio-economic factors. If these factors turn out to be sufficiently different between two consecutive generations, representatives of both generations may arrive at other decisions in the same situation because their decision-making rationale bears very different conditionality.
9. Conclusion
We showed that a proper understanding of the conditions prevailing over the data sets used to drive our forecasts and decisions when the data are limited or intrinsically biased is vital for correct decision-making. The problem of conditionality in the training or learning data set is relevant for both human and machine learning. Ignoring this conditionality and treating available data as unconditional may result in wrong forecasts and subsequent decisions.
Conditionalism, a methodological approach to such situations, pays particular attention to the conditionality of the training or learning data. This decision-making methodology is essential to active practitioners in areas where data samples for learning are limited, scarce, or subject to unquantifiable biases. These areas cover vast parts of our daily lives, such as economics, business and management science, social and behavior sciences, to name a few. Correct implementation of these practices leads to better understanding and interpretation of reality and better decision-making.
Acknowledgements
The author would like to thank Prof. Ilia Tsetlin of INSEAD, Prof. Peter Carr of NYU, and Dr. Raphael Douady for discussions and valuable suggestions. Special thanks are due to Prof. Alexei Chekhlov of Columbia U. for his opinions in systematic trading and encouragement.
I am grateful to Dimitri Sogoloff, CEO of Horton Point LLC, for his strategic advice. This work owes the term Conditionalism to Dr. Rada Sumareva-Ukrainsky.
Last but not least, I would like to thank the paper’s referees for their comments that helped improve the manuscript.