A Solution of Generalized Cosine Equation in Hilbert ’ s Fourth Problem

A solution of Hilbert’s fourth problem leads to the integral equation which can be called the generalized cosine equation. In the present paper, we propose an inversion formula for the solution of the generalized cosine equation using integral and stochastic geometry methods.


Introduction
The integral-geometric approach to Hilbert's fourth problem.The fourth problem in Hilbert's famous collection of 1900, asks for the geometries, defined axiomatically, in which there exists a notion of length for which line segments are the shortest connections of their endpoints.The problem has later seen many transformation and interpretation.It was shown by G. Hamel that it is the same to ask (see [1] [2]): Given an open convex subset C of n R , determine all complete projective metrics on C .A metric d is called projective if it is continuous and linearly additive.Here n R is the n-dimensional Euclidean space.There are two classical examples of projective metrics, already given by D. Hilbert.The first example is n R with the metric induced by a norm (a Minkowski space).Such metrics are the translation invariant projective metrics on n R .The second example is what is now called a Hilbert geometry.The modern approaches make it clear that the problem is at the basis of integral geometry, inverse problems and Finsler geometry (see [3]- [5]).There is anintegral-geometric approach suggested by H.Busemann to construct class of projective metrics (see [3]).
We denote by n E ( E E = 3 )-the space of hyperplanes in n R , 1 n− S -the unit sphere in n R (the space of unit vectors), where xy is the segment with endpoints x and y .If we define for , n x y ∈ R , then d is a projective metric.The question arises whether this construction produces all projective metrics on n R .First comes the A. V. Pogorelov-R.V. Ambartzumian-R.Aleksander result ( [2] [6] [7]) stating that in two dimensional case every projective metrics can be obtained by (3) with a measure in the space of lines in the plane.This essentially solves Hilbert's fourth problem in two dimensional case (see also [8]).
In dimension greater than two, the situation is different: not every projective metric on 3 R can be obtained by ( 3) with some measure µ in 3 E .This is already seen from the following construction.Now let ⋅ be a norm on 3 R .Suppose that the projective metric ( ) ,  d x y x y = − can be generated by (3) with translation invariant measure µ on E .The translation invariant measure µ can be decomposed: there exists a finite even measure m on 2 S such that ( ) ), where ( ) , p ξ is the usual para- metrization of a plane e: p is the distance of e from the origin O; 2 ξ ∈ S is the direction normal to e.We assume that ( ) ( ) The equation where 0 H ≥ is a given even function while h is the unknown function, is known as the zonoid equation.By a result of W. Blaschke the following is known (see ( [10]).If the even function H is sufficiently often differentiable then (5) has a uniquely determined continuous even solution not necessarily positive.If h is the solution of (5), then we can define a translation invariant measure ( ) From now on, we restrict ourselves to Finsler metrics on 3 R , since sufficiently smooth projective metrics are induced from Finsler metrics, and projective metrics can be approximated, uniformly on compact sets, by smooth projective Finsler metrics (see [2] [4]).
We define a Finsler metric on 3 R as a continuous function [ ) We consider locally finite signed measure µ in the space E , which posses density with respect to the standard Euclidean motion invariant measure, i.e. (see [11])

d e h e e h e p
where de is an element of the standard measure.To define function x h on 2 S we consider the restriction of h onto [ ] x as a function on the hemisphere, since a direction completely determines a plane from [ ] x .Then we extend the restriction to 2 S by symmetry.Thus ( ) ( ) is the plane with normal ξ .Below x h we call the restriction of h onto [ ] x .In [2], A. V. Pogorelov showed the following result.
Theorem 1 If H is a smooth projective Finsler metric in 3 R ( [ ) ), then there exists a uniquely determined locally finite signed measure µ in the space E , with continuous density function h , such that, for , , d , . x Here x h is the restriction of h onto ] [x , dξ denotes the spherical Lebesgue measure on 2 S .
The measure µ is also called a Crofton measure for the Finsler metric H (see [4]).Different from the approach in [2], Theorem 1 was proved by R.Schneider [12] using expansions in spherical harmonics.(7) defines a transform h H → and we pose the problem of its inversion (i.e.reconstruction of h for given H).The Equation (7) where H is a given function and h is required, we call generalized cosine equation.We are interested in the solution of the generalized cosine equation defined by (7).Note that, the Equation ( 7) can have either no or exactly one continuous solution (see [10]).
An inversion of the generalized cosine equation.The problem of finding the solution of ( 7) we reduce to find the solution of an other integral equation appearing in Combinatorial Integral Geometry.The concept of a flag density which was introduced and systematically employed by R. V. Ambartzumian, in [9] [11] will be of basic importance below.
We consider the so-called directed flags (below just a flag).A flag is a triad ( ) , ,  f x g e = , where x is a point in 3 R called the location of f, g is a directed line containing the point x, and e is an oriented plane (a plane with specified positive normal direction) containing g.There are two equivalent (and dual to each other) representations of a flag: where 2 ω ∈ S is the normal of e and ϕ is the angular coordinate of the direction of g in ω S , while 2 Ω ∈ S is the spatial direction of g and Φ is the angular coordinate of the normal of e in Ω S .
We use locally finite signed measure ( ) ( ) in the space E to define the following function (flag function) in the space of flags  (so-called sine-square transform) (see [13] [14]) Here x h is the restriction of h onto [ ] x .To explain ( ) x g e = .Then ( ) , f α ξ is the angle between g and the trace e e ξ ∩ , where e ξ is the plane through the origin orthogonal to e.If we represent ( ) , , sin cos f where ψ is the angular coordinate of the projection of ξ into the plane of the flag f .Note that (9) does not depend on the choice of the reference point on the plane of the flag f .The function ρ defined by (8) we call flag density of measure µ .If the flag density ρ is sufficiently often differentiable, then (8) has a unique continuous solution h (see [9] [14]).In [14] [15] (see also [16]- [18] in case µ is a translation invariant measure) by author of the present paper using integral and stochastic geometry methods was found an inversion formula for (8) and reconstruct the density h of signed measure µ in terms of its smooth flag density ρ .
The problem of finding the solution of ( 7) we reduce to finding the flag density ρ for which ( ) ( ) and using inversion formula of (8).Now we describe the inversion formula of (7).We need to give the definitions of certain partial derivatives of ( ) . With each directed flag we associate three orthogonal axes through x: by definition, axis 1 x coincides with the direction of g; axis 2 x lies within e, is orthogonal to 1 x and is directed into the right half of e bounded by g; axis 3 x is coincides with the positive normal to e .We require that the axes 1 x , 2 x , 3 x form a left triad.By ( ) we denote the derivative of ρ at f which corresponds to positive rotation of f around the axis g.
By definition, the positive rotation of the space around the axis g appears clockwise, when we look in the direction of the axis g.
Also we denote by x η ∂ ∂ the partial derivative in the argument x which is taken in the direction 2 η ∈ S .In the special cases where η coincides with directions of the axes 2 x or 3 x respectively, the values of x η ρ ∂ ∂ will be denoted as y ρ′ and n ρ′ correspondingly.
Our main result is the following.Let H be a sufficiently often differentiable function ( [ ) ).Also we assume that the Equation ( 7) has a solution.For every fixed which depends on 3 x ∈ R .For a given plane e and a point x e ∈ , the corresponding so-called bundle of flags we denote by We consider so-called bundle mass of flag density ρ at Note that the first integral in (12) does not depend on ) and h defined on E be the solution of (7).For a given plane e ∈ E the following representation is valid where x is a point on e, ρ is defined by (11), ( ) ρ ⋅ is the restriction of ρ onto , x e , ( ) M ⋅ is the bundle mass function of ρ .
In Setion 3, we present the expression for in terms of the derivatives of ρ with respect to the parameters involved.

Convex Bodies and Measures in the Space of Planes
Equation ( 8) naturally emerges in Integral Geometry (see [13] [14]).It proved in [14] that Equation ( 8) has the unique solution in the class of continuous functions and found an inversion formula.Here we present a short version of the proof of the formula for completeness.
Note that, in [16] the same problem was considered for the case µ is a translation invariant measure on E ( ( ) In [16] (see also [17]) was obtained integral expression for the value of m for a spherical domain bounded by a piecewise smooth curve and an inversion formula was found for a case m has a density, which first was found in [11] (see also [18]).
To invert Equation (8) we do the following.Let ρ be a smooth function on the space of flags that is defined by (8) of a measure µ in 3 E possessing continuous density h.We consider the restriction of h onto the set of planes tangent to a spherical domain.Then, by integral geometry methods we find the integral of the restriction over a spherical disc in terms of ρ .Using this integrals we find an inversion formula for h.
We need some results from integral geometry.Let µ be a signed measure on E , possessing density h with respect to the invariant measure, i.e. de e h de A we denote the set of planes, that intersect A. Let B be a convex body with a sufficiently smooth boundary ∂B .By 1 2 , k k we denote the principal normal curvatures of ∂B at a point s ∈ ∂B , and by ( ) we denote the flag, where t is the plane tangent to ∂B at the point s , i g is the directed line whose direction coincides with the i-th principal direction of curvature at s ∈ ∂B .In [14] (see also [19]) the following representation has been obtained.
Theorem 3 Let µ be a signed measure on E , possessing density h with respect to the invariant measure.
any sufficiently smooth convex body B we have the following representation: [ ] ( ) ( ) ( ) ( ) where ρ is the flag density of µ defined by (8), ds is an area element on ∂B .For the case µ is a translation invariant measure on E the representation first was found in [20].
We will need some further definitions.By ( ) we denote the sphere with center Q and radius R , 2 S will stand for ( ) Assume h is the density of µ .We define the following function on ( ) where ( ) t s is the plane tangent to ( )

−
, where ds is an area element on ( ) , we call the conditional measure on ( ) Let A be a geodesically convex domain on ( ) , which is contained in some hemisphere.Let be the convex body bounded by ( ) at points of A ∂ and the conical surface with the vertex Q and base A. In [14], the following result was shown.
Theorem 4 Let µ be a signed measure on E with continuous density h with respect to the invariant measure, and let ( ) be an open convex domain contained in some hemisphere.Then Using Theorems 3 and 4, one can calculate the values of the "conditional measure" for various domains 2 A ⊂ S .Now we calculate the conditional measure of a spherical disc ( ) of spherical radius < π 2 α and using this result we find an inversion formula for (8).For translation invariant case the result was obtained in [16] (see also [17]).Without loss of generality, one can consider Q as the origin O. On 2 S we consider usual spherical coordinates ( ) , ϕ ν .The center of the disc chosen for the pole.We have: A flag f we call a tangent flag to A at l A ∈ ∂ , if f is located at l, the plane of f is tangent to 2 S at A l ∈ ∂ and the positive normal of the plane of f coincides with the outer normal to 2 S at A l ∈ ∂ , the line of f is tangent to A ∂ at l and the direction on the line of f corresponds to the motion along A ∂ , which leaves (locally) A on the left hand side.By A ∂ we denote the set of the tangent flags of A. The tangent flag at l is unique and therefore ( ) for some > 0 δ .We have oriented great circle with pole at 2 ω ∈ S .By ] [x we denote the bundle of hyper- planes containing the point n x ∈ R .Let µ be a measure on n

2 ωTheorem 2
we denote the restriction of ρ onto , x e .The notation ( ) Let H be a sufficiently often differentiable function ([)

Theorem 5 n is the outer normal direction to 2 S
notation for the value of ρ at the tangent flag located at l.By ( ) ρ ⋅ we denote the restriction of a flag function ρ onto A ∂ .Let ρ be a 2 C smooth flag function that is defined by (8) of a signed measure µ in E with continuous density h with respect to the invariant measure.Then for any spherical disc at s, dϕ denotes the Lebesgue measure on [ ) ρ at s. Proof of the Theorem 5.According to Theorem 4 we have to calculate