A Solution of Generalized Cosine Equation in Hilbert’s Fourth Problem ()
1. Introduction
The integral-geometric approach to Hilbert’s fourth problem. The fourth problem in Hilbert’s famous collection of 1900, asks for the geometries, defined axiomatically, in which there exists a notion of length for which line segments are the shortest connections of their endpoints. The problem has later seen many transformation and interpretation. It was shown by G. Hamel that it is the same to ask (see [1] [2] ): Given an open convex subset
of
, determine all complete projective metrics on
.
A metric
is called projective if it is continuous and linearly additive. Here
is the n-dimensional Euclidean space. There are two classical examples of projective metrics, already given by D. Hilbert. The first example is
with the metric induced by a norm (a Minkowski space). Such metrics are the translation invariant projective metrics on
. The second example is what is now called a Hilbert geometry.
The modern approaches make it clear that the problem is at the basis of integral geometry, inverse problems and Finsler geometry (see [3] -[5] ). There is anintegral-geometric approach suggested by H.Busemann to construct class of projective metrics (see [3] ).
We denote by
(
)—the space of hyperplanes in
,
-the unit sphere in
(the space of unit vectors),
-the oriented great circle with pole at
. By
we denote the bundle of hyperplanes containing the point
. Let
be a measure on
which satisfies
(1)
and
(2)
where
is the segment with endpoints
and
.
If we define
(3)
for
, then
is a projective metric.
The question arises whether this construction produces all projective metrics on
. First comes the A. V. Pogorelov-R. V. Ambartzumian-R. Aleksander result ([2] [6] [7] ) stating that in two dimensional case every projective metrics can be obtained by (3) with a measure in the space of lines in the plane. This essentially solves Hilbert’s fourth problem in two dimensional case (see also [8] ).
In dimension greater than two, the situation is different: not every projective metric on
can be obtained by (3) with some measure
in
. This is already seen from the following construction.
Now let
be a norm on
. Suppose that the projective metric
can be generated by (3) with translation invariant measure
on
. The translation invariant measure
can be decomposed: there exists a finite even measure
on
such that
(see ([9] ), where
is the usual parametrization of a plane e: p is the distance of e from the origin o;
is the direction normal to e. We assume that
. Now the assumed representation (3), gives
(4)
The equation
(5)
where
is a given even function while
is the unknown function, is known as the zonoid equation. By a result of W. Blaschke the following is known (see ([10] ). If the even function
is sufficiently often differentiable then (5) has a uniquely determined continuous even solution not necessarily positive. If
is the solution of (5), then we can define a translation invariant measure
on
which satisfies (4).
From now on, we restrict ourselves to Finsler metrics on
, since sufficiently smooth projective metrics are induced from Finsler metrics, and projective metrics can be approximated, uniformly on compact sets, by smooth projective Finsler metrics (see [2] [4] ).
We define a Finsler metric on
as a continuous function
with the property that
is a norm on
, for each
.
We consider locally finite signed measure
in the space
, which posses density with respect to the standard Euclidean motion invariant measure, i.e. (see [11] )
(6)
where
is an element of the standard measure. To define function
on
we consider the restriction of
onto
as a function on the hemisphere, since a direction completely determines a plane from
. Then we extend the restriction to
by symmetry. Thus

where
is the plane with normal
. Below
we call the restriction of
onto
. In [2] , A. V. Pogorelov showed the following result.
Theorem 1 If
is a smooth projective Finsler metric in
(
), then there exists a uniquely determined locally finite signed measure
in the space
, with continuous density function
, such that, for 
(7)
Here
is the restriction of
onto
,
denotes the spherical Lebesgue measure on
.
The measure
is also called a Crofton measure for the Finsler metric h (see [4] ). Different from the approach in [2] , Theorem 1 was proved by R.Schneider [12] using expansions in spherical harmonics.
(7) defines a transform

and we pose the problem of its inversion (i.e. reconstruction of h for given h). The Equation (7) where h is a given function and h is required, we call generalized cosine equation. We are interested in the solution of the generalized cosine equation defined by (7). Note that, the Equation (7) can have either no or exactly one continuous solution (see [10] ).
In case
is a translation invariant measure on
(
), (7) represents the zonoid Equation (5) playing an important role in convexity (see [10] ).
An inversion of the generalized cosine equation. The problem of finding the solution of (7) we reduce to find the solution of an other integral equation appearing in Combinatorial Integral Geometry. The concept of a flag density which was introduced and systematically employed by R. V. Ambartzumian, in [9] [11] will be of basic importance below.
We consider the so-called directed flags (below just a flag). A flag is a triad
, where x is a point in
called the location of f, g is a directed line containing the point x, and e is an oriented plane (a plane with specified positive normal direction) containing g. There are two equivalent (and dual to each other) representations of a flag:

where
is the normal of e and
is the angular coordinate of the direction of g in
, while
is the spatial direction of g and
is the angular coordinate of the normal of e in
.
We use locally finite signed measure
in the space
to define the following function (flag function) in the space of flags
(so-called sine-square transform) (see [13] [14] )
(8)
Here
is the restriction of h onto
. To explain
we write
. Then
is the angle between g and the trace
, where
is the plane through the origin orthogonal to e. If we represent
then
(9)
where
is the angular coordinate of the projection of
into the plane of the flag
. Note that (9) does not depend on the choice of the reference point on the plane of the flag
. The function
defined by (8) we call flag density of measure
.
If the flag density
is sufficiently often differentiable, then (8) has a unique continuous solution
(see [9] [14] ). In [14] [15] (see also [16] -[18] in case
is a translation invariant measure) by author of the present paper using integral and stochastic geometry methods was found an inversion formula for (8) and reconstruct the density
of signed measure
in terms of its smooth flag density
.
The problem of finding the solution of (7) we reduce to finding the flag density
for which
(10)
and using inversion formula of (8).
Now we describe the inversion formula of (7). We need to give the definitions of certain partial derivatives of
. With each directed flag we associate three orthogonal axes through x: by definition, axis
coincides with the direction of g; axis
lies within e, is orthogonal to
and is directed into the right half of e bounded by g; axis
is coincides with the positive normal to
. We require that the axes
,
,
form a left triad.
By
we denote the derivative of
at f which corresponds to positive rotation of f around the axis g. By definition, the positive rotation of the space around the axis g appears clockwise, when we look in the direction of the axis g.
Also we denote by
the partial derivative in the argument x which is taken in the direction
. In the special cases where
coincides with directions of the axes
or
respectively, the values of 
will be denoted as
and
correspondingly.
Our main result is the following. Let H be a sufficiently often differentiable function (
). Also we assume that the Equation (7) has a solution. For every fixed
, we now solve the zonoid equation for the function
. Since
is smooth, there exists a smooth even solution
on
. Then we put the function
into Equation (8) instead of
and found the flag function 
(11)
which depends on
.
For a given plane e and a point
, the corresponding so-called bundle of flags we denote by

We consider so-called bundle mass of flag density
at
:
(12)
Note that the first integral in (12) does not depend on
.
By
we denote the restriction of
onto
. The notation
is reasonable since
completely determines a flag from
.
Theorem 2 Let H be a sufficiently often differentiable function (
) and h defined on
be the solution of (7). For a given plane
the following representation is valid
(13)
where x is a point on e,
is defined by (11),
is the restriction of
onto
,
is the bundle mass function of
.
In Setion 3, we present the expression for
in terms of the derivatives of
with respect to the parameters involved.
2. Convex Bodies and Measures in the Space of Planes
Equation (8) naturally emerges in Integral Geometry (see [13] [14] ). It proved in [14] that Equation (8) has the unique solution in the class of continuous functions and found an inversion formula. Here we present a short version of the proof of the formula for completeness.
Note that, in [16] the same problem was considered for the case
is a translation invariant measure on
(
). In [16] (see also [17] ) was obtained integral expression for the value of
for a spherical domain bounded by a piecewise smooth curve and an inversion formula was found for a case
has a density, which first was found in [11] (see also [18] ).
To invert Equation (8) we do the following. Let
be a smooth function on the space of flags that is defined by (8) of a measure
in
possessing continuous density h. We consider the restriction of h onto the set of planes tangent to a spherical domain. Then, by integral geometry methods we find the integral of the restriction over a spherical disc in terms of
. Using this integrals we find an inversion formula for h.
We need some results from integral geometry. Let
be a signed measure on
, possessing density h with respect to the invariant measure, i.e.
. Given a subset
, by
we denote the set of planes, that intersect a. Let
be a convex body with a sufficiently smooth boundary
. By
we denote the principal normal curvatures of
at a point
, and by
,
we denote the flag, where t is the plane tangent to
at the point
,
is the directed line whose direction coincides with the i-th principal direction of curvature at
. In [14] (see also [19] ) the following representation has been obtained.
Theorem 3 Let
be a signed measure on
, possessing density h with respect to the invariant measure. For any sufficiently smooth convex body
we have the following representation:
(14)
where
is the flag density of
defined by (8),
is an area element on
.
For the case
is a translation invariant measure on
the representation first was found in [20] .
We will need some further definitions. By
we denote the sphere with center
and radius
,
will stand for
.
Assume h is the density of
. We define the following function on 

where
is the plane tangent to
at
.
The measure
, where
is an area element on
, we call the conditional measure on
, generated by
(or h).
Let
be a geodesically convex domain on
, which is contained in some hemisphere. Let
,
be the convex body bounded by
, planes tangent to
at points of
and the conical surface with the vertex Q and base A. In [14] , the following result was shown.
Theorem 4 Let
be a signed measure on
with continuous density h with respect to the invariant measure, and let
be an open convex domain contained in some hemisphere. Then
(15)
Using Theorems 3 and 4, one can calculate the values of the “conditional measure” for various domains
.
Now we calculate the conditional measure of a spherical disc
of spherical radius
and using this result we find an inversion formula for (8). For translation invariant case the result was obtained in [16] (see also [17] ). Without loss of generality, one can consider Q as the origin O. On
we consider usual spherical coordinates
. The center of the disc chosen for the pole. We have:
.
A flag f we call a tangent flag to A at
, if f is located at l, the plane of f is tangent to
at 
and the positive normal of the plane of f coincides with the outer normal to
at
, the line of f is tangent to
at l and the direction on the line of f corresponds to the motion along
, which leaves (locally) A on the left hand side. By
we denote the set of the tangent flags of A. The tangent flag at l is unique and therefore
is a reasonable notation for the value of
at the tangent flag located at l. By
we denote the restriction of a flag function
onto
.
Theorem 5 Let
be a
smooth flag function that is defined by (8) of a signed measure
in
with continuous density h with respect to the invariant measure. Then for any spherical disc 
(16)
here n is the outer normal direction to
at s,
denotes the Lebesgue measure on
,
is the bundle mass of
at s.
Proof of the Theorem 5. According to Theorem 4 we have to calculate
and
. In order to apply Theorem 3, instead of
and
we consider their smooth versions
and
for some
. We have
(17)
We divide the surfaces
and
into domains (part of conical surface and part of spherical surface) and using (14) we get the expressions for
and
in terms of flag density. The last expressions we put into (15) and obtain (16). The realization of this procedure can be found in [14] [15] .
3. An Inversion Formula for (8)
To derive an inversion formula for (8), we express h in terms of given function
. Let
,
is the bundle of flags.
Theorem 6 Let
be a
smooth function on the space of flags in
, that is defined by transform (8) of a signed measure
in
with continuous density h. For a given plane e the following representation is valid
(18)
where x is a point on e,
is the restriction of
onto
,
is the bundle mass of
at x.
Proof of the Theorem 6. Let
be a point. We consider a unit sphere
tangent to e at
and denote by
the spherical disk centered at x with the spherical radius
. By the mean-value theorem
(19)
where
is the area of
.
On
, we consider usual spherical coordinates
with x chosen for the pole. We consider the restriction of the bundle mass
onto
and by
we denote the value of the restriction at
. Also, we consider the restriction of
onto
, where
is the spherical disk centered at x with the spherical radius
. By
we denote the value of the restriction at
.
We represent the integral in (19) according to (16) and find the limit by decomposing the resulting terms in powers of
. We have
(20)
To decompose other terms we need the following lemma.
Lemma 1 For any differentiable flag function 
(21)
In its formulation,
is defined assuming that the positive normal of e is parallel to the outer normal to
at x. Note, that
defined on the bundle of flags
.
We have
(22)
(23)
(24)
After proper substitution and taking into account that
, we find
(25)
Here
is the derivative of
at
, in the direction of normal to
which is outer normal to
.
It is easy to prove that
(26)
where
denotes the second derivative in the spatial direction which corresponds to the direction 
on e. After substitution (26) into (25) we obtain (18). Theorem 6 is proved.
The second integral in (13) contains the derivative
. To present the expression for
in terms of the derivatives of the function with respect to the parameters involved we introduce on
spherical coordinates in which
is specified by a pair
, where
denotes the latitude of
, and
denotes its longitude. The corresponding derivatives with respect to
are given by the expressions (see. [17] ):
(27)
Using these formulas one can write the expression for
.
4. The Connection between Equations (7) and (8)
For a given line
and a point
, the corresponding bundle of flags we denote by

Now we integrate (8) over bundle of flags
with respect to Lebesgue measure
on
, where
is the direction of
and
is the great circle on
with pole at
. We obtain
(28)
since from the definition of flag density we have (see [9] )
(29)
Note that one can obtain (29) using the cosine theorem of spherical geometry.
It follows from (28) if
is a smooth projective Finsler metric in
and
is a flag density such that
(30)
then Equations (7) and (8) have the same unique solution. Thus the problem of finding the solution of (7) we reduce to find flag density
which satisfy (30) for a given smooth projective Finsler metric
.
Note that for a fixed
the restriction
of a smooth projective Finsler metric
onto
is uniquely determined by the restriction
of the unique solution
of (7) onto the bundle
. The restriction
can be found by solution of the zonoid Equation (5) for
.
Also, note that for a fixed
the restriction
of the flag density
of the signed measure
(where h is the unique solution of (7)) is uniquely determined by the restriction
.
Hence for every fixed
, we solve the zonoid equation for the function
. Since
is a smooth function, there exists a smooth even solution
on
. Then we put the function
into Equation (8) instead of
and find the flag density
which depends on
. Thus we find the flag density
which satisfy (30). Substituting the flag density
into (18) we get the solution of (8) which coincides with the solution of (7). Theorem 2 is proved.
Fundings
This work was supported by State Committee Science MES RA, in frame of the research project SCS 13-1A244.