A Geometric View on Inner Transformation between the Variables of a Linear Regression Model

In the teaching and researching of linear regression analysis, it is interesting and enlightening to explore how the dependent variable vector can be in-ner-transformed into regression coefficient estimator vector from a visible geometrical view. As an example, the roadmap of such inner transformation is presented based on a simple multiple linear regression model in this work. By applying the matrix algorithms like singular value decomposition (SVD) and Moore-Penrose generalized matrix inverse, the dependent variable vector lands into the right space of the independent variable matrix and is meta-morphosed into regression coefficient estimator vector through the three-step of inner transformation. This work explores the geometrical relationship between the dependent variable vector and regression coefficient estimator vector as well as presents a new approach for vector rotating.


Introduction
A matrix can be factorized into the product of several matrices with special properties. Particularly, by singular value decomposition (SVD) which is widely used in regression analysis [1] [2] [3], a matrix can be factorized into the product of three matrices with orthogonal or diagonal properties respectively. SVD can be formularized like T X U V θ = [4] [5] [6] [7]. Moore-Penrose generalized inverse is a special case of generalized matrix in-verse [8] [9] [10] [11] [12] and can be applied in regression analysis [13] [14] and least square analysis [15]. For example, in the study of Tian and Zhang [13], the linear unbiased estimator of partial coefficients is derived through Moore-Penrose generalized inverse algorithm. Herein, multiplied by Moore-Penrose generalized inverse of the independent variable matrix, the dependent variable vector is transformed into the new coordinate systems, or left space and then right space of the independent variable matrix [16]. In fact, this process can also be regarded as a vector rotating algorithm. In addition to Moore-Penrose's generalized inverse algorithm, singular value decomposition (SVD) facilitates such transformation by dividing the transformation process into three-step to present a visible geometrical view.

y Can Be Transformed into β
In order to avoid the irrelevant calculation minutia when adopting an independent matrix with a large rank and only to highlight inner transformation, herein, a simple multiple linear regression model ˆŷ Xβ ε = + is adopted with the independent matrix U and V are respectively the left and right singular vector matrix of X, and θ is eigen value matrix of X [6] [7] [17]. Because X is a matrix with full rank column, β can be demonstrated as ˆX y β + = [13] [16], X + signifies the Moore-Penrose generalized inverse of matrix X. And β can also be demonstrated as below when X is substituted by its SVD form presented in equation (E.1), The Equation (E.2) can be regarded as a process that the vector y is transformed 3 times from right to left multiplied by ( ) Let the coordinate axes of a 3-dimension original coordinate system be symbo- . The quintessence of the equation which is located in a coordinate system constructed by 1 2 , u u and 3 u , or left space of X. In fact, y and 1 y are at the same spatial location. However, y is in the coordinate system constructed by 1 2 , a a and 3 a , meanwhile 1 y is presented by the coordinate system constructed by 1 2 , u u and 3 u , in another word, During this transformation, multiplied by θ 's Moore-Penrose inverse θ + , the vector 1 y in 1 2 , u u and 3 u coordinate system is transformed into the vector 2 y which is in the same coordinate system as 1 y . Such a transformation can be called "Vector Stretching". During this transformation, the coordinate value of 1 y in 1 u axis diminishes, meanwhile, the coordinate value of 1 y in 2 u axis gets enlarged. The coordinate value of 1 y in 3 u axis vanishes. The "Vector Stretching" of 1 y into 2 y can be demonstrated as below in Figure 2.
The vanishing coordinate value of 1 y in 3 u axis can signify the degree of freedom of X from a geometric view.
X's right singular vectors can be denoted as   The quintessence of Equation (E.5) is that the vector 2 y in the coordinate system is constructed by 1 u and 2 u can be embodied by the vector β in the coordinate system T 1 v and T 2 v . This transformation is demonstrated as below in Figure 3.
Though the vector β has the same spatial position as 2 y in Figure 3, these two vectors are situated in two different coordinate systems constructed respectively by T T 1 2 , v v (the right space of X) and 1 2 , u u (the left space of X). In another word, the distinction between the vector β and 2 y is that β is demonstrated by the coordinate system of T 1 v and T 2 v , meanwhile, 2 y is demonstrated by 1 u and 2 u . As demonstrated by the aforementioned 3 steps of transformation, the dependent variable y vector is inner-transformed into parameter β vector.

y Can Be Transformed into ŷ
The independent variable X's left singular vector matrix U [6] [7] [17]) can be blocked into two sub-matrics 1 U and 2 U as below.
[ ] Let ŷ symbolize the estimator of the dependent variable y. For that 1 U is the left space of X, ŷ can be demonstrated as  [12]. So, ŷ can be demonstrated as below: Based on E.6, y can be transformed into ŷ by two steps as below.
In Transformation 4, y transforms into y * by being projected into 1 U , the DOI: 10.4236/am.2021.1210061 936 Applied Mathematics left space of X. Another word, y * is the projection of y in X.
In Transformation 5, multiplied by 1 U , y * transforms into ŷ and is presented in the coordinate system constructed by 1 2 , a a and 3 a . Their spatial locations are demonstrated in Figure 4.
In Figure 4, it can be found that the error estimator ε is located in the same direction with 3 u which is the left null space of matrix X, and can be expressed as below: ŷ is the projection of y into the left space of X, but presented in the original coordinate system constructed by 1 2 , a a and 3 a . ε is located in the left null space of matrix X and perpendicular with ŷ . This perpendicular result can be geometrically demonstrated in Figure 4 as well as in the following multiplication.

Finding and Conclusion
By applying SVD and Moore-Penrose generalized inverse of the independent variable X in a multiple linear regression model, the dependent variable y can be transformed into the regression coefficient estimator vector β and its own estimator ŷ . This process presents a new geometric perspective to study the relationship between X, y, β and ŷ , through the inner-transformation algorithm.
As demonstrated from Figures 1-3, y transforms into β by transforming from the original coordinate system of 1 2 , a a and 3 a into the right space of X constructed by T 1 v and T 2 v . Through this process, y transforms into 1 y by transferring into the coordinate system of 1 2 , u u and 3 u , that is presented by  As demonstrated in Figure 4, y transforms into y * by projecting into 1 U , the left space of X. And then, multiplied by 1 U , y * transforms into ŷ and returns back into the original coordinate system constructed by 1 2 , a a and 3 a . ŷ is the projection of y into the left space of X, and perpendicular with error estimator εˆ which is located in the left null space of X.
With the aid of the algorithms like matrix decomposition and Moore-Penrose generalized matrix inverse, the dependent variable y of a multiple linear regression model can be inner-transformed into the regression coefficient estimator β and its own estimator ŷ . This process is a new approach to illustrate the inner-transformation between variables from a geometric view as well as presenting the spatial locations of the variables. To date, there is no study to explore the relationship between the variables of the multiple linear regression model from the view of geometric transformation. This study fills such a gap and provides a new perspective for studying multiple linear regression.
The limitation of this work is that a simple example of the multiple linear model is adopted to present such intricate inner transformation. But, it can demonstrate more inference art of inner transformation if more complex examples of multiple linear regression model are adopted in future studies.