1. Introduction
In 1936, Erich Hecke (see [1] ) introduced the groups 
generated by two linear-fractional transformations
and
. Hecke showed that 
is discrete if and only if
,
, 
or
. Hecke group 
is isomorphic to the free product of two finite cyclic group of order 2 and q, and it has a presentation
The first few of these groups are
, the full modular group having special interest for mathematicians in many fields of Mathematics, 
and
.
A non-empty set 
with an action of the group G on it, is said to be a G-set. We say that 
is a transitive G-set if, for any 
in 
there exists a g in G such that
. Let
, where 
and m is a square free positive integer. Then
is the set of all roots of primitive second degree equations
, with reduced discriminant 
equal to n and
is the disjoint union of 
for all k. If 
and its conjugate 
have opposite signs then 
is called an ambiguous number [2] . The actual number of ambiguous numbers in 
has been discussed in [3] as a function of n. The classification of the real quadratic irrational numbers 
of 
in the forms 
modulo n has been given in [4] [5] . It has been shown in [6] that the action of the modular group
, where 
and
, on the rational projective line 
is transitive. An action of
, where 
and 
and its proper subgroups on 
has been discussed in [7] [8] .
invariant under the action of modular group G but 
is not invariant under the action of H. Thus it motivates us to establish a connection between the elements of the groups G and H and hence to deduce a common subgroup 
of both groups such that each of 
and 
is invariant under H* and hence we find G-subsets of 
and H-subsets of 
or 
according as 
or 
and 
for all non-square n. Also the partition of 
has been discussed depending upon classes 
modulo
.
2. Preliminaries
We quote from [5] [6] and [8] the following results for later reference. Also we tabulate the actions on 
of 
and
, the generators of G and H respectively in Table 1.
Theorem 2.1 (see [5] ) Let
,
. Then 
and 
are both G-subsets of
.
Theorem 2.2 (see [5] ) Let
. Then 
and 
are both G-subsets of
.
Theorem 2.3 (see [6] ) If
, then 
and 
are exactly two disjoint G-subsets of 
depending upon classes 
modulo 4.
Theorem 2.4 (see [6] ) If
, then 
and
Table 1. The action of elements of G and H on
.
are both G-subsets of
.
Lemma 2.5 (see [8] ) Let
. Then:
1) If 
then 
if and only if
.
2) 
if and only if
.
Theorem 2.6 (see [8] ) The set
, is invariant under the action of H.
Theorem 2.7 (see [8] ) For each non square positive integer
, 
is an H-subset of
.
3. Action of 
on 
We start this section by defining a common subgroup of both groups 
and
, where
, 
, 
and
. For this, we need the following crucial results which show the relationships between the elements of G and H.
Lemma 3.1 Let 
and 
be the generators of G and H respectively defined above. Then we have:
1) 
and
.
2) 
and
.
3) 
and
.
4) 
and
.
5) 
and
.
6) 
and
. In particular 
and
.
Following corollary is an immediate consequence of Lemma 3.1.
Corollary 3.2 1) By Lemma 3.1, since 
and 
so 
is a common subgroup of G and H where 
are the transformations defined by 
and
.
2) As
, 
, so 
is a proper subgroup of
.
3) 
and
.
Since for each integer n, either 
or 
for each odd prime p. So in the following lemmawe classify the elements of 
in terms of classes 
with 0 modulo p or qr, qnr nature of a, b and c modulo p.
Lemma 3.3 Let 
be prime and
. Then 
consists of classes
, 
, 
, 
, 
, 
, 
or
.
Proof. Let 
be any class of
. Then 
leads us to exactly three cases. If 
then exactly one of 
is 
and the other is qr or qnr of 
as otherwise 
and hence the class 
is one of the forms
, 
, 
,
. If 
then 
and the class takes the form 
or
. In third case if 
then 
so again
. This yields the class in the forms 
or
. Hence the result. 
Lemma 3.4 Let 
and let 
be the class of 
of
. Then:
1) If 
then 
has the forms
, 
, 
, 
, 
, 
, 
, 
, 
, 
, 
or 
only.
2) If 
then 
has the forms
, 
, 
, 
, 
, 
, 
, 
, 
, 
, 
or 
only.
Proof. Let 
be the class of 
with
. As 
so if 
then 
according as 
or
. Thus we have
, 
if 
and
, 
if
. If 
then
, so we get
, 
, 
, 
, 
, 
, 
, 
, 
or 
only. This proof is now complete. 
Lemma 3.5 Let 
and let 
be the class of 
of
. Then:
1) If
then 
has the forms
, 
, 
, 
, 
or 
only.
2) If 
then 
has the forms
, 
, 
, 
, 
or 
only.
Proof. The proof is analogous to the proof of Lemma 3.4. 
Note: If 
then
, 
and 
are three classes of 
in modulo 2. If n is an odd then three classes of 
are
, 
and 
modulo 2. These are the only classes of 
if
. But if 
then 
is also a class of 
and there are no further classes. These classes in modulo 2 of 
do not give any useful information during the study of action of 
on 
except that if 
then the set consisting of all elements of 
of the form 
is invariant under the action of the group G. Whereas the study of action of H* on 
gives some useful information about these classes. The following crucial result determines the H*- subsets of 
depending upon classes 
modulo 2. It is interesting to observe that 
splits into 
and 
in modulo 2. Each of these two H*-subsets further splits into proper H*-subsets in modulo 4.
Lemma 3.6 
and 
are two distinct H*-subsets of 
depending upon classes 
modulo 2.
Theorem 3.7 and Remarks 3.8 are extension of Lemma 3.6 and discuss the action of H* on 
depending upon classes 
modulo 4. Proofs of these follow directly by the equations
, 
and classes 
modulo 4 given in [6] .
Theorem 3.7 Let n be any non-square positive integer. Then 
splits into two proper H*- subsets
,
. Similarly 
splits into two proper H*-subsets 
and 
.
Remark 3.8 1) Let
. Then 
and 
are H*-subsets of
. In particular if
, then 
and 
are H*-subsets of
. Whereas if
, then
, 
, 
and 
are H*-subsets of
. Specifically, 
,
.
2) As we know that if 
and 
are even, then 
must be even as
. If
, then 
and
.
3) If
, then 
or 
is empty according as 
or
. As we know that if n and c are even, then a must be even as
. However
, 
are proper H*-subsets of 
depending upon classes 
modulo 4.
Lemma 3.9 Let n be any non-square positive integer. Then 
and 
are distinct H*-subsets of an H-set
.
Proof. Follows by the equations 
and vice versa. Hence 
is equivalent to
.
Clearly 
where 
denotes the set of all ambigious numbers in
(see [8] ).
Remark 3.10 1) Each G-subset X of 
splits into two H*-subsets 
and 
and
.
2) Each H-subset Y of 
splits into two H*-subsets 
and
.
3) Each H-subset Y of
, 
splits into two H*-subsets 
and
.
4) Each H-subset Y of
, 
splits into two H*-subsets 
and
.
Theorem 3.11 a) If A is an H*-subset of 
or
, then 
is a G-subset of
.
b) If A is an H*-subset of
, then 
is an H-subset of 
or 
according as 
or
.
c) If A is an H*-subset of
, then 
is an H-subset of 
for all nonsquare n.
Proof. Proof of a) follows by the equation
.
Proof of b) follows by the equations 
or 
according as 
or
.
Proof of c) follows by the equation
. 
Following examples illustrate the above results.
Example 3.12 1) Let
. Then 
but
. Also 
but
. Similarly 
whereas
. Also
,
. So 
has exactly 4 orbits under the action of H whereas 
splits into two G-orbits namely
,
.
2) 
splits into nine H-orbits. Also 
and 
. Whereas 
splits into four 
-orbits namely
, 
and
. (see Figure 1) 
Theorem 3.13 Let p be an odd prime factor of n. Then 
and 
are two H*-subsets of
. In particular, these are the only H*- subsets of 
depending upon classes 
modulo p.
Proof. Let 
be the class of
. In view of Lemma 3.3, either both of 
are qrs or qnrs and the two equations
,
fix b, c modulo p. If 
then 
or 
according as 
or
. similarly for
. This shows that the sets 
and 
are H*-subsets of 
depending upon classes modulo p. 
The following corollary is an immediate consequence of Lemma 3.6 and Theorem 3.13.
Corollary 3.14 Let p be an odd prime and
. Then 
splits into four proper H*- subsets depending upon classes modulo 2p.
Proof. Since 
implies that
. This is equivalent to congruences 
and
. By Theorem 3.13
, 
are H*-subsets and then, by Lemma 3.6, each of 
and
further splits into two H*-subsets
, 
, 
and
. 
The next theorem is more interesting in a sense that whenever
, 
is itself an H*-set depending upon classes 
modulo p.
Theorem 3.15 Let p be an odd prime and
. Then 
is itself an H*-set depending upon classes 
modulo p.
Proof. follows from Lemmas 3.4, 3.5 and the equations 
and 
given in Table 1.
Let us illustrate the above theorem in view of Theorem 3.4. If
, then the set
is an H*-set.
That is, 
is itself an H*-set depending upon classes 
modulo 3. Similarly for
.
Theorem 3.16 Let p be an odd prime and n is a quadratic residue (quadratic non-residue) of 2p. Then 
is the disjoint union of three H*-subsets
, 
and 
depending upon classes 
modulo 2p.
Proof. Follows by Theorems 2.6, 2.7 and 3.15. 
The following example justifies the above result.
Example 3.17 Since
, then 
splits into these three H*-subsets
, 
,
. 
The next theorem is a generalization of Theorem 3.13 to the case when n involves two distinct prime factors.
Theorem 3.20 Let 
and 
be distinct odd primes factors of n. Then
, 
, 
and 
are four H*-subsets of
. More precisely these are the only H*subsets of 
depending upon classes 
modulo
.
Proof. Let 
be any class of 
with
. Then 
implies that
(1)
This is equivalent to congruences 
and
. By Theorem 3.14, the congruence 
gives two H*-subsets 
and
of
. As
, again applying Theorem 3.13 on each of 
and 
we have four H*-subsets
, 
, 
and 
of
.