Is the Market Truly in a Random Walk? Searching for the Efficient Market Hypothesis with an AI Assistant Economist

Lizhi Xin, Kevin Xin^*

XINVISIONQ, INC., Chicago, USA.

DOI: 10.4236/tel.[phone]                

Abstract

The equity market is known for its uncertainty and randomness. While the market and the participating traders may seem like independent entities in their own right, but it is the foray of traders that makes the market in a random walk, as the market’s volatility influences the traders’ judgement on which action to take; the market and traders are “entangled together” in this way. This paper presents a methodology to model both the market’s volatility and traders’ actions by drawing on the concept of quantum superposition to illustrate that it is indeed the “interactions” of both the market and traders that result in the random walk, fully conforming to the efficient market hypothesis. We’ve also developed an AI assistant economist that’s powered by a quantum-like evolutionary algorithm to produce short-horizon predictions of the future trend of the market based on Darwinian natural selection.

Keywords

Random Walk, Efficient Market Hypothesis, Genetic Programming, Machine Learning, Quantum-Like Evolutionary Algorithm, AI Assistant Economist

1. Introduction

The stock market has been known to be a volatile place (Shiller, 1991; Knight, 1921; Baaquie, 2007; Fox, 2009). In 1900, Louis Bachelier wrote his Ph.D. thesis, The Theory of Speculation, which formulated the first mathematical theory of the stochastic processes of the market (Bachelier et al., 2006). In 1965, Eugene Fama developed the Efficient Market Hypothesis (EMH), which states that financial markets are “informatively efficient” and that no one can accurately predict the market’s future by any means possible (Fama, 1970). EMH complements the fact that the market is indeed on a random walk, and that the current price of stocks reflects all the available information. In 1973, Fisher Black and Myron Scholes developed the BSM model that describes the market’s behavior in pure mathematical terms (Black & Scholes, 1973), which provides a theoretical framework to derive a price estimate of European call and put options. Their log-normal distribution of the returns is based on Brownian motion (Broadbent et al., 1928).

The mainstream ways of describing the market have been to model the state of the market and the traders’ actions separately, with the market being treated as a physical particle or mathematical entity, and the traders are seen as a completely separate external factor that should be left alone when attempting to describe the market.

To model the behavior of traders, Amos Tversky and Daniel Kahneman formulated prospect theory (Kahneman & Tversky, 1979; Kahneman, 2013), which attempts to factor in the “humanity” aspect of trading; arguing that traders value gains and losses differently that’s known as loss aversion theory. They proposed that losses will always carry a greater emotional impact on an individual, thus gains are generally perceived as greater probability wise.

When studying the market, it is crucial to factor in both the market itself and the participating traders involved and not separately, because it is together, both the market and traders’ overall is what makes up the movement of the market; it is the market’s volatility that hampers the participating traders’ decisive decision-making ability (traders are initially hesitant whether to buy or sell) and in turn it is the “collective effort” of all the traders (some will buy while others will sell) that eventually determine the market’s trend direction (increase or decrease). Thus, the market and traders are intertwined or “entangled” with each other, the market’s volatility affects the traders and the traders’ actions in turn determine the trend of the market and vice versa, in this continuous looping cycle, which fully reflects that the market is truly in a random walk.

We present a methodology to model both the market’s volatile movement (increase or decrease) and the participating trader’s actions (buy or sell) by utilizing the concept of the quantum principle of superposition and illustrate that it is the two “entangled” that causes the market to be in a random walk which fully conforms to the observed market movement as stated by the efficient market hypothesis (Xin & Xin, 2021).

Again, established agent-based and stochastic models tend to treat the market and traders separately, however, when modeling market behavior, we have to keep in mind that the market’s “behavior” emerges from the interactions between the market and traders together as a collective whole of a complex system, essentially 1 + 1 > 2. Our quantum-like framework is a unified framework that models both the market and participating traders together as a collective whole all under complex Hilbert Space. Our framework establishes that the market’s random walk arises from the complex interactions between the traders and the external environment rather than from the simple random fluctuations that are commonly described by stochastic models.

Building off our methodology, we’ve developed an AI assistant economist (one that can aid human economists to analyze and forecast the financial market) powered by our quantum-like evolutionary algorithm that can produce a short horizon prediction (one week) of the market’s future movement by studying one month of data from the Dow Jones Index (Xin & Xin, 2024).

The rest of the paper is structured as follows: Section 2 details the methodology. Section 3 is the results. Section 4 is the conclusion.

2. Methods

The volatility of the market and the hesitation of the traders’ actions are essentially intertwined; the uncertain nature of the market hampers traders’ decision-making ability of when to buy and sell, and in turn it is the “collective” actions of all the participating traders that determines the markets’ closing price in this ever-changing cycle between the market and traders.

To effectively model both the volatility of the market and the traders’ actions of buy and sell, the concept of quantum superposition principle (Silverman, 2008; Feynman, 2015; Dirac, 1958) can be utilized; by “superposing” both the market’s states and the traders’ actions. This can be modeled as in (1) and (2).

$|Q\rangle =c_1|q_1\rangle +c_2|q_2\rangle$ (1)

where $|q_1\rangle$ denotes the market increases; $|q_2\rangle$ denotes the market decreases. ${\omega }_1={\left|c_1\right|}^2$ is the objective frequency that the market increases; ${\omega }_2={\left|c_2\right|}^2$ is the objective frequency that the market decreases.

$|A\rangle ={\mu }_1|a_1\rangle +{\mu }_2|a_2\rangle$ (2)

where $|a_1\rangle$ denotes the trader believes that the market increases; $|a_2\rangle$ denotes the trader believes that the market decreases. $p_1={\left|{\mu }_1\right|}^2$ is the trader’s degree of belief that the market increases; $p_2={\left|{\mu }_2\right|}^2$ is the trader’s degree of belief that the market decreases.

The market and all the participating traders can be described as a complex system, as (3).

$|\psi \rangle = c_1|q_1\rangle \otimes {\displaystyle \prod _{i=1}^N|a_1^i\rangle }+c_2|q_2\rangle \otimes {\displaystyle \prod _{i=1}^N|a_2^i\rangle }$ (3)

where N is the number of traders in the group. The density operator of the complex system can be described as (4).

$\begin{array}{l}{\rho }_{\text{market}+\text{traders}}=|\psi \rangle \langle \psi |\\ ={\omega }_1|q_1\rangle \langle q_1|+{\omega }_2|q_2\rangle \langle q_2|+\left[c_1{c}_2^{*}|q_2\rangle \langle q_1|\otimes {\displaystyle \prod _{i=1}^N\langle a_1^i|a_2^i\rangle }+\text{H}\text{.C}.\right]\end{array}$ (4)

where the third term is a non-diagonalization term that represents the superposition of the market either increasing or decreasing as well as the traders’ being unable to deduce whether the market will increase or decrease. Traders will tend to randomly “guess” that the market is increasing or decreasing; the traders’ believing whether the market is increasing or decreasing is “orthogonal”, and when the number of participating traders is very large then the expectations of all the traders for whether the market will increase or decrease are then zero as (5).

${\displaystyle \prod _{i=1}^N\langle a_1^i|a_2^i\rangle }\stackrel{N\to \infty }{\to }0$ (5)

(4) then becomes (6).

${\rho }_{\text{market}+\text{traders}}\stackrel{N\to \infty }{\to }{\omega }_1|q_1\rangle \langle q_1|+{\omega }_2|q_2\rangle \langle q_2|$ (6)

When there is a vast number of participating traders involved ($N\to \infty$ ), the market and all the participating traders as a whole tends to be in a random walk (${\rho }_{\text{market}+\text{traders}}\approx {\rho }_{\text{market}}$ ) as outlined by the efficient market hypothesis. ${\rho }_{\text{market}}$ as (7) is the actual observed density operator of the market; where ${\omega }_1$ is the observed objective frequency that the market will increase and ${\omega }_2$ is the observed objective frequency that the market will decrease (${\omega }_1\approx {\omega }_2=0.5$ ).

${\rho }_{\text{market}}={\omega }_1|q_1\rangle \langle q_1|+{\omega }_2|q_2\rangle \langle q_2|$ (7)

We have shown above that it is the market and the participating traders as a collective whole that make the market become a random walk. It is widely acknowledged that nobody can accurately predict the future trend of the market in the long run. Now the question becomes: is it possible to produce a short-horizon forecast of the market’s future trend?

To answer this question, we’ve developed a quantum-like evolutionary algorithm to power the AI assistant economist that utilizes both the quantum superposition principle and Genetic Programming (GP) (Holland, 1975; Koza, 1992, 1994) to produce possible short-horizon predictions of the market’s future trend by machine learning historical trading data.

For the AI assistant economist (AI agent), we can hypothesize that before the AI agent makes its decision, “believes” whether the market will increase or decrease, are “superposed simultaneously” in its “mind”, which can be described by the density operator as in (8).

${\rho }_{\text{agent}}=|A\rangle \langle A|=p_1|a_1\rangle \langle a_1|+p_2|a_2\rangle \langle a_2|+{\mu }_1{\mu }_2^{*}|a_1\rangle \langle a_2|+{\mu }_1^{*}{\mu }_2|a_2\rangle \langle a_1|$ (8)

where $p_1$ is the AI agent’s degree of belief that the market increases, $p_2$ is the AI agent’s degree of belief that the market decreases. The third and fourth terms in (8) are the “quantum interference” terms that indicate the AI agent’s “mind” is undecided on whether the market will increase or decrease, where the AI agent can “think” that the market is both increasing and decreasing.

When an AI agent actually “decides” on whether the market increases or decreases, a projection of pure state to mixed state happens in the AI agent’s “mind” as (9), which describes the decision-making process of an AI agent.

${\rho }_{\text{agent}}\stackrel{\text{Decide}}{\to }{{\rho }^{\prime }}_{\text{agent}}=p_1|a_1\rangle \langle a_1|+p_2|a_2\rangle \langle a_2|$ (9)

The decision-making process is essentially just a projection from pure state to mixed state, where GP, an algorithm based on Darwinian natural selection (Darwin, 1859), is utilized to evolve a satisfactory pure state. The pure state is essentially just a 2 × 2 matrix, where (9) can be described by the matrix form represented in (10).

${\rho }_{\text{agent}}=\left[\begin{array}{cc}{\rho }_{11}& {\rho }_{12}\\ {\rho }_{21}& {\rho }_{22}\end{array}\right]\stackrel{\text{projection}}{\to }{{\rho }^{\prime }}_{\text{agent}}=\left[\begin{array}{cc}{p}_1& 0\\ 0& p_2\end{array}\right]=p_1|a_1\rangle \langle a_1|+p_2|a_2\rangle \langle a_2|$ (10a)

$|a_1\rangle =\left[\begin{array}{c}1\\ 0\end{array}\right],|a_2\rangle =\left[\begin{array}{c}0\\ 1\end{array}\right];|a_1\rangle \langle a_1|=\left[\begin{array}{cc}1& 0\\ 0& 0\end{array}\right],|a_2\rangle \langle a_2|=\left[\begin{array}{cc}0& 0\\ 0& 1\end{array}\right]$ (10b)

Because the pure density operator ${\rho }_{\text{agent}}$ is just an arbitrary 2 × 2 matrix, we can then approximately construct this density operator with the 8 most basic quantum gates as (11), leading it to become a “matrix tree” (Xin et al., 2023).

$\left\{\begin{array}{c}H=\frac{1}{\sqrt{2}}\left[\begin{array}{cc}1& 1\\ 1& -1\end{array}\right]X=\left[\begin{array}{cc}0& 1\\ 1& 0\end{array}\right]Y=\left[\begin{array}{cc}0& -\text{i}\\ \text{i}& 0\end{array}\right]Z=\left[\begin{array}{cc}1& 0\\ 0& -1\end{array}\right]\\ S=\left[\begin{array}{cc}1& 0\\ 0& \text{i}\end{array}\right]D=\left[\begin{array}{cc}0& 1\\ -1& 0\end{array}\right]T=\left[\begin{array}{cc}1& 0\\ 0& {\text{e}}^{\text{i}\pi /4}\end{array}\right]I=\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right]\end{array}\right\}$ (11)

This “matrix tree” is the tree that’s constructed from the 8 basic quantum gates in (11) by the 3 operators’ addition (+), multiplication (×), and or (//); the final form that the matrix tree takes is a 2 × 2 matrix, hence the name matrix tree. After constructing an individual “matrix tree”, we can then construct a population of “matrix trees”, and then by using the fitness function as the evaluation criteria, the most satisfactory density matrix ${\rho }_{\text{agent}}$ from the population is evolved through generations of natural selection. The “matrix tree” is essentially a decision tree that guides the AI agent which strategies to “take” with corresponding actions. At any given time, the expected value under the current environment (the market is increasing or decreasing) and the corresponding actions (the AI agent “thinks” that the market is increasing or decreasing) can be represented as (12).

$\begin{array}{l}{\rho }_{\text{market}}\otimes {\rho }_{\text{agent}}\\ ={\omega }_1{p}_1{\left|\langle q_1||a_1\rangle \right|}^2+{\omega }_1{p}_2{\left|\langle q_1||a_2\rangle \right|}^2+{\omega }_2{p}_1{\left|\langle q_2||a_1\rangle \right|}^2+{\omega }_2{p}_2{\left|\langle q_2||a_2\rangle \right|}^2\end{array}$ (12)

where (12) is the composite system of the market and the AI agent. Essentially, (12) describes the four possible outcomes of every “decision” made by the AI agent; if the market is increasing or decreasing and the AI agent “thinks” or “doesn’t think so” and vice versa; when the AI agent “thinks” correctly in line with the corresponding motion of the market it’s “rewarded”, if not it’s “punished”. The expected value for the AI agent is the possible scenarios of what the outcome could be paired with the state of the market that’s being observed, as in (13). If the training data has N number of values, then the fitness function for the “matrix tree” is defined as (14), and it is the total sum of all the expected values of each “decision made” by the AI agent.

$E{V}_t=\{\begin{array}{l}{\omega }_1{p}_1,\text{market increases and AI agent}"\text{thinks}"\text{so with probaility}{p}_1\\ -{\omega }_1{p}_2,\text{market increases and AI agent doesn<msup> n <mo>′</mo> </msup> t}"\text{think}"\text{so with probability}{p}_2\\ -{\omega }_2{p}_1,\text{market decreases and AI agent does<msup> n <mo>′</mo> </msup> t}"\text{think}"\text{so with probability}{p}_1\\ {\omega }_2{p}_2,\text{market decreaes and AI agent}"\text{thinks}"\text{so with probability}{p}_2\end{array}$ (13)

${\text{fitness}}_{\text{matrixTree}}={\displaystyle \sum _{t=1}^{N}E{V}_t}$ (14)

If there are M number of individuals in a population of “matrix trees”, the most satisfactory “matrix tree” is the one that possesses the maximum fitness function that can be described as in (15).

${\rho }_{\text{agent}}^{\text{output}}=\arg \underset{a}{\max }\left\{{\text{fitness}}_{\text{matrixTree}},k=1,\cdots ,M\right\}$ (15)

By learning historical data, the more rewards that are reaped, then the more accurate chance, there is of predicting the next outcome of whether the market will increase or decrease.

3. Results

In this paper, we produced short-horizon forecast outcomes by studying a small sample of data. Data from October 4th, 2024 to November 1st, 2024 of the Dow Jones Industrial Average Index was used as training data; the following week of November 4th, 2024 to November 8th, 2024 was forecasted. The data was trained twice consecutively, with 6 possible forecast outcomes produced each session by 3 AI agents. From the 12 total possible forecast outcomes produced by means of majority rules, a final trend sequence of whether the market will increase or decrease is produced to analyze the future trend of the market. The fitting results of the two training sessions are shown in Figure 1 and Figure 2. The trend sequence produced by means of majority rules from the 12 individual possible forecast outcomes is shown in Table 1.

Table 1. Final action sequence produced by logic tree.

Date	DJIA Trend	Trend Sequence
11/04/2024	Decreased	0
11/05/2024	Increased	0
11/06/2024	Increased	0
11/07/2024	Decreased	1
11/08/2024	Increased	0

Figure 1. The fitting results of the first training session.

Figure 2. The fitting results of the second training session.

Using this action sequence produced, the future trend of the market and the market’s volatility can be analyzed. The actual recorded trend of the Dow Jones for the following week is listed in the DJIA Trend column, while the trend sequence that’s produced is listed in the trend sequence column, where 0 represents the AI agent “believes” that the market will increase and 1 represents the AI agent “believes” that the market will decrease.

For this action sequence produced, the predicted trend of the market will be {Increase, Increase, Increase, Decrease, Increase}, in which only the first value predicted was wrong, thus resulting in odds of 80% accuracy. The main emphasis of the 80% accuracy in this case only applies to the short forecast horizon, i.e. the next trading week of 5 data points; for a longer period of time, for example if the forecast horizon was extended to 3 months (60 data points), then it would not be possible for our algorithm to obtain odds of 80%. Essentially, for a longer forecast, the odds would still be closer to 50-50.

4. Conclusion

In this paper, we presented a methodology to describe both the volatility of the market and the participating traders’ actions together in an intertwined model. By utilizing the concept of quantum superposition principle to model both the state of the market and all the participating traders’ possible actions as a whole, we show that the market is indeed in a random walk as stated by the efficient market hypothesis. Unlike traditional methods that don’t factor in the participating traders, which treat the market as a mathematical or physical entity (particle) where calculus is needed to describe it; we are able to subtly model both the market without a statistical approach.

Since the future is inherently unpredictable, no matter what we do, there is no universal omnipotent method that can find the crystal ball to peer into and see the future, therefore, if a black swan suddenly landed, there is really nothing we can do to foresee it. Thus, in this paper and our methodology’s approach in general, we have deliberately chosen a small sample of historical data of about 20 data points to produce a short horizon forecast of 5 future points, assuming that the market will “keep” its “relative” trend of the recent past in the near future, and even in this case, our method is not 100% fail-proof—odds may not always reach 80%. However, if a black swan really does alight, our AI assistant economist won’t be able to predict this either, in which we will just have to “trust our instincts” to “throw” the dice back and hope for the best outcome.

Acknowledgements

Sincere thanks to the managing editor Joy Deng for a rare attitude of high quality. This work received no funding.

Conflicts of Interest

The authors declare no conflicts of interest regarding the publication of this paper.

References

[1]	Baaquie, B. E. (2007). Quantum Finance: Path Integrals and Hamiltonians for Options and Interest Rates. Cambridge University Press.
[2]	Bachelier, L., Davis, M. H. A., & Etheridge, A. (2006). Theory of Speculation: The Origins of Modern Finance. Princeton University Press.
[3]	Black, F., & Scholes, M. (1973). The Pricing of Options and Corporate Liabilities. Journal of Political Economy, 81, 637-654. https://doi.org/10.1086/260062
[4]	Broadbent, T. A., Einstein, A. B., Hertz, H., Dryden, H. L., Murnaghan, F. P., & Bateman, H. (1928). Investigations on the Theory of the Brownian Movement. The Mathematical Gazette, 41, 231.
[5]	Darwin, C. (1859). On the Origin of Species by Means of Natural Selection, or, the Preservation of Favoured Races in the Struggle for Life. John Murray. https://doi.org/10.5962/bhl.title.68064
[6]	Dirac, P. A. M. (1958). The Principles of Quantum Mechanics. Oxford University Press.
[7]	Fama, E. F. (1970). Efficient Capital Markets: A Review of Theory and Empirical Work. The Journal of Finance, 25, 383-417.
[8]	Feynman, R. P. (2015). The Feynman Lectures on Physics. Volume III, Quantum Mechanics. Basic Books.
[9]	Fox, J. (2009). The Myth of the Rational Market a History of Risk, Reward, and Delusion on Wall Street. HarperCollins.
[10]	Holland, J. (1975). Adaptation in Natural and Artificial System. University of Michigan Press.
[11]	Kahneman, D. (2013). Thinking, Fast and Slow. Farrar, Straus and Giroux Macmillan.
[12]	Kahneman, D., & Tversky, A. (1979). Prospect Theory: An Analysis of Decision under Risk. Econometrica, 47, 263-291. https://doi.org/10.2307/1914185
[13]	Knight, F. (1921). Risk, Uncertainty, and Profit. Houghton Mifflin Company.
[14]	Koza, J. R. (1992). Genetic Programming, on the Programming of Computers by Means of Natural Selection. MIT Press.
[15]	Koza, J. R. (1994). Genetic Programming II, Automatic Discovery of Reusable Programs. MIT Press.
[16]	Shiller, R. J. (1991). Market Volatility. MIT Press.
[17]	Silverman, M. P. (2008). Quantum Superposition: Counterintuitive Consequences of Coherence, Entanglement, and Interference. Springer.
[18]	Xin, H. W., Xin, L. Z. (2021). Quantum Finance: Principle, Mechanism and Algorithm of Uncertain Market. University of Science and Technology of China Press.
[19]	Xin, K., & Xin, L. Z. (2024). QxEAI: Assisting Equity Trading with Quantum-Like Evolutionary Algorithm. https://doi.org/10.51094/jxiv.1011
[20]	Xin, L., Xin, K., & Xin, H. (2023). On Laws of Thought—A Quantum-Like Machine Learning Approach. Entropy, 25, Article 1213. https://doi.org/10.3390/e25081213