Introduction

Any closed, simple, piecewise curve defines a Hamiltonian flow, the billiard flow inside the billiard table T [29, 32, 31, 16]. The billiard map of the table T is the Poincaré map defined by the standard cross-section of the billiard flow. The study of the billiard map for a convex table T goes back to Birkhoff [2] and the theory is still far from being completed. In this work we assume that T is convex and and use the standard facts about the phase space of the billiard map .

A homotopically nontrivial invariant circle defines a closed curve in the interior of T. It is called the caustic. While in general the caustics of T may be a complicated curve, the caustics in this paper are convex. The invariant circle consists then of the supporting rays of and the table T is obtained from by the string construction [29, 31]: One takes an unstretchable string having length , wraps it around , pulls it tight at a point M and drags it around . The point M then traces the boundary of the table.

The geometric description of an ellipse with given foci is a special case of the string construction.

Let now be a closed convex curve with perimeter . By varying the string length we obtain all billiard tables with the caustic . The corresponding billiard maps form a one-parameter family of twist maps which is determined by . Our investigatigation of this family emphasis on the relationship between the geometric shape of and the dynamics of the family of billiard maps.

Let be the invariant circle of corresponding to . Due to the time-reversal symmetry, there are two invariant circles in associated with . We make a choice of by requiring that the rotation number of is less than or equal to 1/2. In the special case when is an interval, the tables are the ellipses with foci and . The invariant circle is then the lower of the two invariant circles in with the rotation number 1/2 (see Figure 3).

Let be the restriction of the twist map to . With a uniform parameter on , the maps form a continuous monotone family of orientation preserving circle homeomorphisms. The rotation number is a continuous, nondecreasing function which contains significant amount of information on the family of twist maps. For instance, is identically 1/2 if and only if is an interval or a point. In all other cases, one has and . If is a polygon with n sites then . Otherwise .

The invariant circle moves monotonically upward in the phase space as runs from to infinity (Figure 4).

For small , the curve is near the bottom of . As increases, moves up and approaches for the equator . It remains in general below the equator but not necessarily strictly as the ellipse shows.

We state now the main results of our work. There are two theorems. Our first result is that there is a Baire generic set of caustics for which the rotation function is a devil's staircase. More precisely, has the property that for any rational p/q, the interval has a nonempty interior. Some geometrical properties of imply that has nonempty interior. We apply this to show that contains all polygons and curves with a flat point.

In the second theorem, we study then the motion of Birkhoff periodic orbits of a fixed rotation number in the phase space . Let be the interval of the set of parameters for which there exists a Birkhoff periodic orbit on the curve . For , all Birkhoff periodic orbits with rotation number p/q are above . For , they are below . Global index arguments imply that for the curve separates the set of Birkhoff periodic orbits into two nonempty subsets. We prove that for near a Birkhoff periodic set of index -1 approaches from above. This set continues as a set of index +1 below for near . At , a bifurcation occurs and two Birkhoff periodic sets of index -1 are created on . The picture for near is obtained by an obvious symmetry from the case at : a Birkhoff periodic set of index +1 approaches from above the curve , collides with two Birkhoff periodic sets on and leaves as a Birkhoff periodic set of index -1. (see Figures 12,13,14).

The theme of relations between the shape of the billiard table and the billiard dynamics is well reflected in the literature. Here, we mention only a few papers which are especially close to our direction.

A problem which is closely related to our work is how to decide whether a specific billiard table has an invariant circle or not. We mention the theorem of Mather [19] about the global absence of invariant circles in the case when the billiard table has a flat point, the result of Hubacher [13] about the absence of invariant circles near the table if the curvature of the table has a discontinuity. An other related result is the theorem of Gruber [6, 7] which tells that generically in the Hausdorff topology on the set of convex tables, a billiard table has no caustics. On the other hand, if the table is smooth enough and the curvature is strictly positive, Lazutkin's result [17] assures in this case the existence of infinitely many caustics near the boundary of the table. The inverse problem of finding billiard tables with a special invariant circle or caustics is interesting both from the geometrical and from the dynamical point of view. Bialy [1] showed that if the phase space of the billiard map is foliated by a continuous family of homotopically non-trivial invariant circles, then the billiard table is a disc. There are billiard tables with flat invariant circles different from the equator [8]. The size of the area which is free of caustics can be estimated from the shape of the table [9]. For the billiard in an ellipse, there are caustics for any rotation number and the table can be obtained from any of them by the string construction. Elliptical billiards are believed to be special. According to a conjecture attributed to Birkhoff and stated by Poritsky [26], they are the only integrable billiards in the sense that a subset of full Lebesgue measure of the phase space is foliated by invariant circles. This Birkhoff-Poritsky problem is still open.

The paper is organized as follows. In section 2, we establish notation and collect preliminary results. In section 3, we investigate the rotation function and show that it is Baire generically a devil's staircase. If there exists some for which every orbit on is periodic, a caustic is called exceptional. We will show in section 4 that this property is quite restrictive. While tables with exceptional caustics with rotation number 1/3 have been constructed in [14] and his construction probably generalizes to any possible rational number, it is not known whether a table exists which has two exceptional invariant caustics. In section 5, we investigate the passage of the Birkhoff periodic orbits through the invariant circle as varies from to .