^{1}

^{*}

^{2}

^{*}

This elucidation investigates the Hausdorff dimension of the output space of multi-layer neural networks. When the factor map from the covering space of the output space to the output space has a synchronizing word, the Hausdorff dimension of the output space relates to its topological entropy. This clarifies the geometrical structure of the output space in more details.

The multi-layer neural networks (MNN, [1,2]) have received considerable attention and were successfully applied to many areas such as signal processing, pattern recognition ([3,4]) and combinatorial optimization ([5,6]) in the past few decades. The investigation of mosaic solution is the most essential in MNN models due to the learning algorithm and training processing. In [7-9], the authors proved that the output solutions space of a 2-layer MNN forms a so-called sofic shift space, which is a factor of a classical subshift of finite type. Thus, MNN model indeed produces abundant output patterns and makes learning algorithm more efficient. A useful quantity to classify the output solution space is the topological entropy ([

From the dynamical system (DS) point of view, the topological entropy reveals the complexity of the global patterns. However, it provides less information of the inner structure of a given DS, e.g., self-similarity or recurrent properties. The possible quantity reveals that such properties are the Hausdorff dimension (HD, [^{1}, the previous work ([

This paper is organized as follows. Section 2 contains a brief disscussion for the computation of topological entropy in [

A one-dimensional multi-layer neural network (MNN) is realized as

for some, and. The finite subset indicates the neighborhood, and the piecewise linear map

is called the output function. The template is composed of feedback template with

controlling template

and threshold

where

for.

A solution

of (1) is called mosaic if

for, and for some.

of a mosaic solution is called a mosaic pattern, where

.

The solution space of (1) stores the patterns, and the output space of (1) is the collection of the output patterns; more precisely,

A neighborhood is called the nearest neighborhood if. In [

Since the neighborhood is finite and is invariant for each, the output space is determined by the so-called basic set of admissible local patterns. Replace the pattern -1 and 1 by - and +, respectively; the basic set of admissible local patterns of the first and second layer is a subset of

and, respectively, where denotes

To ease the notation, we denote

by. Given a template, the basic set of admissible local pattern

is determined, where and are the basic set of admissible local patterns of the first and second layer, respectively. Let

denote the parameter space of (1). Theorem 2.1 asserts that can be partitioned into finitely many subregions so that two templates in the same partition exhibit the same basic set of admissible local patterns.

Theorem 2.1. (See [

1).

2) if.

3) Templates for some if and only if.

Since the template of MNNs is spatially invariant, the so-called transition matrix is used to investigate the complexity of MNNs. The transition matrix is defined by

herein is presented as for. Furthermore, the transition matrix of the second layer

is defined by

the transition matrix of the first layer

is defined by

where

Write

as four smaller matrices. Define by

Ban and Chang [

Theorem 2.2. (See [

where is a matrix with all entries being 1’s; and are the Hadamard and Kronecker product, respectively.

As being demonstrated in [7-9,13], the solution space is a so-called shift of finite type (SFT, also known as a topological Markov shift) and the output space is a sofic shift. More specifically, a SFT can be represented as a directed graph and a sofic shift can be represented as a labeled graph for some labeling and finite alphabet. A labeled graph is called right-resolving if the restriction of to is one-to-one for all, where consists of those edges starting from. If is not right-solving, there exists a labeled graph, derived by applying subset construction method (SCM) to, such that the sofic shift represented by is identical to the original space. A detailed instruction is referred to [

One of the most frequently used quantum for the measure of the spatial complexity is the topological entropy. Let be a symbolic space and let denote the collection of the patterns of length in. The topological entropy of is defined by

Herein

indicates the cardinality of.

Theorem 2.3. (See [7,9,13]) Let be the labeled graph obtained from the transition matrix of (1). The topological entropies of and are and

respectively, where is the transition matrix of the labeled graph which is obtained by applying SCM to.

Aside from the topological entropy, the Hausdorff dimension characterizes its geometrical structure. The concept of the Hausdorff dimension generalizes the notion of the dimension of a real vector space and helps to distinguish the difference of measure zero sets. Let be a finite set with cardinality, which we consider to be an alphabet of symbols. Without the loss of generality, we usually take

.

The full -shift is the collection of all biinfinite sequences with entries from. More precisely,

It is well-known that is a compact metric space endowed with the metric

Suppose is a subspace of. Set

It follows that and can be embedded in the close interval separately. Moreover, and can be mapped onto the close interval, and is identified with the direct product. This makes the elucidation of the Hausdorff dimension of the output space comprehensible. (Recall that the alphabet of is).

Suppose are shift spaces and is a factor map. We say that has a synchronizing word if there is a finite word such that each element in admits the same terminal entry. More precisely, for any

satisfying

we have.

Suppose is a labeled graph representation of the output space of (1). Denote by the SFT represented by the graph if is right-resolving; otherwise, denote by the SFT represented by the graph, where is obtained by applying SCM to. It follows that is a covering space of and there is a factor map which is represented by the labeling (or). Theorem 2.4 asserts that the Hausdorff dimension of the output space relates to the topological entropy of its covering space if has a synchronizing word.

Theorem 2.4. Along with the same assumption of Theorem 2.3. Let, which is represented by if is right-resolving and is represented by otherwise, be the covering space of. Suppose the factor map, which is represented by the labeling (or), has a synchronizing word. Then

Restated,

Proof. Suppose is a SFT and is an invariant probability measure on. The Variational Principle indicates that the topological entropy of is the supremum of the measure-theoretic entropy of; more precisely,

A measure is called maximal if attains the supremum. Let be a Markov measure which is derived from the transition matrix of. Then is the unique measure that satisfies

if is topologically transitive (cf. [

where is a maximal measure of (see [

.

Theorem 2.3 demonstrates that the topological entropy of the output space

(respectively) if is not rightresolving (respectively is right-resolving). A straightforward examination infers that

.

Hence we have

this completes the proof.

Suppose with

and. The transition matrices for the first and second layer are

respectively. Therefore, the transition matrix and the symbolic transition matrix of the MNN are

respectively, where

It is seen from the symbolic transition matrix that the labeled graph is not right-resolving, and applying SMC to derives a right-resolving labeled graph (cf.

Theorem 2.3 indicates that

where

is the golden mean.

The symbolic transition matrix of is

It is seen that both and are synchronizing words of. Theorem 2.4 demonstrates that

The fractal set of the output space is seen in