Evolution of the Internet at the Autonomous Systems Level During a Decade

We empirically study the evolution of the Internet at the Autonomous Systems (ASes) level examining how many properties of its undirected graph change during the decade 2008-2017. In order to analyze local as well as global properties we consider three classes of metrics related to structure, connectivity and centrality. We find that the Internet almost doubled its size preserving the sparse nature and becoming more connected. The Internet has a small internal nucleus, composed of high degree ASes, much more stable and connected than external components. There are indications of an underlying hierarchical organization where a small fraction of big ASes are connected to many regions with high internal cohesiveness containing low and medium degree ASes and these regions are poorly connected among them. The overall trend of the average shortest path length of the Internet is a slight decrease over time. Betweenness centrality measurements suggest that during its evolution the Internet became less congested.


INTRODUCTION
The Internet is a highly engineered communication infrastructure continuously growing over time. It consists of Autonomous Systems (ASes) each of which can be considered a network, with its own routing policy, administrated by a single authority. ASes peer with each other to exchange traffic and use the Border Gateway Protocol (BGP) [23] to exchange routing and reachability information in the global routing system of the Internet. Therefore, the Internet can be represented by a graph where ASes are nodes and BPG peering relationships are links. The structure of the Internet has been studied by many authors and the literature on the subject is vast. One of the most used methods is the statistical analysis of different metrics characterizing the AS graph [21], [24], [16], [15]. There are not many studies concerning the evolution of the Internet over time [11], [25] and because the amount of data to analyze tends to grow dramatically often only a limited number of properties are considered. The purpose of this work is to study the evolution of the Internet considering features related to both its topology and data traffic. To achieve this goal we consider for each year of the period 2008-2017 a snapshot of the undirected AS graph, introduce three classes of metrics related to structure, connectivity, centrality and analyze how they change over time. The paper is organized as follows: in Section 2 we describe the datasets; in Section 3 we define the adopted metrics and for each of them explain its importance; we report the results in Section 4. Finally, in Section 5 we summarize the results and make the final considerations.

DATA SETS
The ASes graphs have been constructed from the publicly available IPv4 Routed /24 AS Links Dataset provided by CAIDA [5]. AS links are derived from traceroute-like IP measurements collected by the Archipelago (Ark) [4] infrastructure, a globally distributed hardware platform of network path probing monitors. The association of an IP address with an AS is based on the RouteViews [1] BGP data and the probed IP paths are mapped into AS links. We exclude multi-origin ASes and AS sets because they may introduce distortion in the association process. The sizes of the ASes graphs analyzed in this work are shown in Tab. 1.

DESCRIPTION OF METRICS
In this section we introduce the metrics chosen for this analysis whose summary scheme is shown in Tab. 2. For each metric we give a short description and briefly discuss its importance. We use the notation G = (N , E) to indicate an AS graph which has N nodes and E edges.

Degree distribution
The degree distribution P(k) is the probability that a random chosen node has degree k. If a graph has N k nodes with degree k then P(k) = N k /N . Since P(k) is a probability distribution it satisfies the normalization condition k max k =k min P(k) = 1 where k min and k max are the minimum and maximum degree, respectively. From P(k) we can calculate the average degreek = k max k=k min kP(k). For a random network P(k) follows a binomial distribution and in the limit of sparse network δ ≪ 1, where δ is the link density, it is well approximated by a Poissonian. The Internet, as many other real networks, can be considered sparse and, moreover, it is scalefree which means that it contains both small and very high degree nodes and this feature can not be reproduced by a Poissonian. Many studies agree that the degree distribution follows a power law P(k) ∝ k −α though deviations have been observed [12], [7], [17]. For each snapshot of the AS graph we calculate the best fit power law parameters k P L min and α and verify the statistical plausibility of this model.

K-core decomposition
A k-core of a graph is obtained by removing all nodes with degree less than k. Therefore, the k-core is the maximal subgraph in which all nodes have at least degree k. The 0-core is the full graph and coincides with the 1-core if there are s no isolated nodes. The kcore decomposition is a way of peeling the graph by progressively removing the outermost low degree layers up to the innermost high degree core which we call nucleus. We denote by k cor e max the coreness of the nucleus, and by N n (N k ) and E n (E k ) the number of ASes  Year   2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  # Nodes  28838  31892  35149  38550  41527  47407  47581  50856  51736  52361  # Edges 135723 152447 184071 213870 281596 282939 298355 347518 379652 414501   and edges in the nucleus (in the k-core). In the case of the Internet the analysis of the k-core decomposition over time is useful for understanding whether its nucleus, composed of high degree ASes, evolves differently from its periphery.

Clustering coefficient
The local clustering coefficient C i of a node i of degree k is the ratio of the actual number of edges E i connecting its neighbours to the maximum possible number of edges that could connect them. For an undircted graph C i = 2E i /k(k − 1). By averaging over all nodes we obtain the global clustering coefficient C = i C i /N . For a random network C is independent of the node's degree and decreases with the size of the graph as C ∼ N −1 . Scale-free networks exhibit a quite different behavior. For example the clustering coefficient of a scale-free network obtained from the Barabasi-Albert model [2] follows C ∼ (lnN ) 2 /N , which for large N is higher that that of a random network. An important quantity is C(k), the average clustering coefficient of degree k nodes. It has been shown [22] that it is the three-points correlation function which is the probability that a degree k node is connected to others two nodes which in their turn are joined by an edge. C(k) can be used to study the hierarchical structure of networks [21], [19].

Shortest path length
The shortest path length between two nodes is the minimum number of hops needed to connect them. Of course, for any pair of nodes there may be several shortest paths connecting them. The shortest path length distribution s(h) provides, for a given number h of hops, the number of shortest paths of length h. We call S the average shortest path length. The diameter D is the longest shortest path. The importance of the shortest paths is mainly related to routing. Many routing algorithms are based on the shortest path length. Adaptive algorithms allow to change routing decision to optimize traffic load and prevent congestions. The knowledge of the available shortest paths is then crucial for routing efficiency.

Closeness centrality
The closeness centrality Γ of a node i is the inverse of its average shortest path length to all other nodes: is the shortest path length between i and j. Nodes with high Γ are those closest to all others and can be considered central in the network. On the contrary, nodes with low Γ are, on average, far away from the others and can be considered peripheric.

Betweenness centrality
The concept of betweenness centrality applies to both nodes and edges. The betweenness centrality of a node i is defined as B n (i) = j,k ∈V σ (j, k | i)/σ (j, k) where the sum is over all pairs of nodes, σ (j, k) is the number of shortest paths and σ (j, k | i) is the number of those passing through i. If j = k then σ (j, k) = 1 and if i ∈ {j, k }, σ (j, k | i) = 0. The betweenness centrality B e of an edge e is defined in the same way. In this case σ (j, k | e) is the number of shortest paths containing e. Efficient routing policies exploit as much as possible available shortest paths, hence a node (edge) with high betweenness centrality carries large traffic load. In [14] the betweenness centrality was used to investigate the evolution of networks whose nodes may break down due to overload and in [13] it was used to define the load of a node for studying the problem of data packet transport in power law scale free networks.

RESULTS
In this section we compare the measurements of the metrics concerning the Internet AS graphs obtained for each year of the decade 2008-2017 and report the corresponding results.

Degree distribution
In Fig. 1 are shown the node probability degree distributions and their complementary cumulative functions (CCDF). For all data sets the peak of the degree distribution is for k = 2, a result already reported in [17] where it is claimed that it is due to the AS number assignment policies. While the edge density is around δ ≈ 3 × 10 −4 during the decade 2008-2017, the general trend is a growth over time for bothk and k max as shown in Tab. 3. This means that the Internet has become more connected preserving its sparse nature. All degree distributions have a similar form. In order to verify the statistical plausibility of the power law model we perform a goodness of fit test based on the Kolmogorov-Smirnov statistic [9] which provides a p-value. The power law has statistical support if p > 0.1. From Tab. 3 we see that even if the best fit exponent is always around the value α ≈ 2.1 the power law can be considered a reliable model only for the distributions of the years 2008, 2010 and 2011. Since for the majority of the most larger data sets p ≤ 0.1 we could say that at the AS level the evolution of the Internet can not be explained by models which predict a pure power law degree distribution.  [25] found no clear evidence of the exponential growth of k cor e max and observed a stability of its value after 2003. They also found that the size of the nucleus exhibits large fluctuations over time. We now examine in Fig. 4 how the fraction of ASes and edges varies in the periphery of the Internet from the 2 to the 10-core. We start from the core of order 2 because in our case there are no isolated ASes. Compared to the evolution of the nucleus it is evident that the periphery evolves with a different dynamics. In Tab. 5 we compare for each k-core the number of ASes and edges it contained in 2008 and 2017 and report the percentage variation. Results clearly show that the nucleus is much more stable than the periphery. The connectivity of each core can be studied by looking at its edge density which is defined as Fig.5 is shown δ k as a function of N and in Tab. 5 is reported its average value. The edge density increases with the coreness showing that the inner is the core the more it is connected. It is interesting to note that the edge density of the Internet nucleus is three order of magnitude higher than that of the most external 2-core.
Tauro et al [20] studied the topology of the Internet from the end of 1997 to the middle of 2000. They introduced the concept of importance of a node on the base of its degree and effective eccentricity defined as the minimum number of hops required to reach at least 90% of all other nodes. The most important nodes have high degree and low effective eccentricity. The found that the structure of the Internet is hierarchical with a highly connected core surrounded by layers of nodes of decreasing importance.

Clustering coefficient
The clustering coefficient has been used to investigate the hierarchical organization of real networks. The hierarchy could be a consequence of the particular role of the nodes in the network. A stub AS does not carry traffic outside itself and is connected to a transit AS that, on the contrary, is designed for this purpose. The hierarchy of the Internet is rooted in its geographical organization in international, national backbones, regional and local areas. International ASes do not need to be connected to local networks but only to national backbones. This is the skeleton of the Internet. National backbones in their turn are connected to regional networks which finally connect local areas to the Internet, implementing in such a way a best and less expensive strategy. It is reasonable to suppose that this hierarchical structure introduces correlations in    the connectivity of the ASes. A. Vázquez et al [21] showed that the hierarchical structure of the Internet is captured by the scaling C(k) ∼ k −γ and found γ = 0.75. Ravasz and Barabasi [19] proposed a deterministic hierarchical model for which C(k) ∼ k −1 and using a stochastic version of the model showed that the hierarchical topology is again well described by the scaling C(k) ∼ k −γ even if the value of γ can be tuned by varying other network parameters. In Fig. 6 is shown the average clustering coefficient as a function of the size of the AS graph and in the fourth column of Tab. 6   For the deterministic hierarchical model studied in [19] C is independent of N . The weak dependence of C on N might indicate the presence of a hierarchical organization in the structure of Edge density Size of the AS graph 2-core 3-core 4-core 5-core 6-core 7-core 8-core 9-core 10-core k max -core   the Internet. To further investigate on this point we study C(k). In Fig. 7 we plot C(k) for the AS graph only for the year 2017 because for all other years the plots are almost overlapping. The best fit with the power law k −γ provides for all the years values of γ which differ only by ∼0.1% obtaining, on average, γ = 1.08±0.01. In Fig. 7 is also shown the slope of the function C(k) ∼ k −1 and even if it nicely follows the slope of the experimental points the goodness of fit test does not give any statistical support to the scaling C(k) = k −γ . However, data show that C(k) decreases with k especially for k > 100. Low degree ASes have high neighbourhood connectivity and, on the contrary, neighbours of big hub ASes are slightly connected among them. This is consistent with a hierarchical organization in which big ASes are connected to many regions with high internal cohesiveness and composed of low or medium degree ASes, and these regions are poorly connected among them. Since the C(k) plots of the AS graph snapshots overlap, to study the evolution of the clustering coefficient over years we compare the CCDF of C(k) in Fig. 8. For our convenience we consider in more detail three degree regions: high (k > 1000), medium 100 < k ≤ 1000, low k ≤ 100 and also plot them in the same figure. We observe that in the high degree region the CCDF distributions are very intertwined indicating that during the decade 2008-2017 this region was rather static. In the medium degree region a clear separation emerges between the CCDF of the different years and for a given value of C(k) the CCDF increases over time. The gap is even more pronounced in the peripheric low degree region. This result suggests that the evolution of the Internet from 2008 to 2017 was not uniform and the most significant changes mainly affected its middle and even more its periphery, and the neighbourhood connectivity in these regions increased over time.

Shortest path length
In Fig. 9   Year S ± 0.6 D C  In Fig. 10 are shown, for different lengths, the number of shortest paths over time. The 3-hops shortest paths are the most numerous, as expected, and their number increases over time.

Closeness centrality
The closeness centrality Γ as a function of the node's degree k is shown in Fig. 11  Here hn indicates a shortest path whose length is n hops.
years have similar slope. Their curves lie in between of those plotted and are not shown in the figure for better readability because they overlap in the region 100 < k < 1000. We observe that Γ increases with the degree which means that big hub ASes are in the center of the Internet while low degree ASes are peripheric. We consider Γ in three regions: k ≤ 100, 100 < k ≤ 1000 and k > 1000 corresponding respectively to low, medium and high degree and we find that within errors it is almost constant over the period 2008-2017 with average values of 0.392 ± 0.007, 0.434 ± 0.004 and 0.484 ± 0.004.

Betweenness centrality
The average node betweenness centrality as a function of the degree k is shown in Fig. 12 for the AS graphs of the years 2008 and 2017. As in the case of the closeness centrality, we do not plot the curves of the other years for readability reasons. However, for all the years the average node betweenness centrality increases with the degree which means that the higher is the degree of an AS the more is the number of shortest paths passing through it. There is evidence of an overall slight decrease of B n (k) during the evolution of the Internet from 2008 to 2017. The overall average values of B n calculated in 2008 and 2017 are ∼7.1×10 −5 and ∼3.6×10 −5 respectively. In Tab. 7 are reported the average values of B n (k) calculated in the degree regions k ≤ 100, 100 < k ≤ 1000 and k > 1000.
In order to study B e we represent an edge as a point of the xy plane whose coordinates (k x , k y ) are the degrees of the nodes it connects. In Fig. 13 is shown, for each year of the decade 2008-2017, the colored 3D map of the average B e . The highest B e is associated to edges which have at least a high degree (k > 1000) AS as a terminal. Edges connecting low or medium degree ASes have lower B e . This is what one would expect considering that high degree ASes are the backbone of the Internet and the most part of the shortest routes should cross them. We also observe a slight decrease of B e over time. The overall average B e was ∼2.2×10 −5 in 2008 and ∼0.7×10 −5 in 2017, indicating that somehow the Internet has become less congested although it has expanded. By looking at the colored contour maps shown on the right side of Fig. 13 we infer that during its evolution the lowering of B e affected first the part of the Internet containing low and medium degree ASes (k < 1000) and subsequently the backbone.

CONCLUSION
We studied the evolution of the Internet at the AS level during the decade 2008-2017. For each year of the decade we considered a snapshot of the AS undirected graph and analyzed how a wide range of metrics related to structure, connectivity and centrality varies over time. During the decade 2008-2017 the Internet almost doubled its size preserving its sparse nature and becoming more connected. The Internet is a scale free network because it contains both very high and low degree ASes. For all the years 2008-2017 the best fit of the degree distributions with a power law P(k) ∼ k −α provides values of the exponent very close to each other and around α ≃ 2.1. However, the statistical analysis shows that a pure power law model fails to explain the scale free properties.
The study of the k-core decomposition shows that the number of ASes and edges in each core increases over time. The Internet has a small internal nucleus composed of high degree ASes. The fraction of nodes and edges in the nucleus is quite stable over time and is ∼0.4% and ∼4%, respectively. We analyzed the first external cores from order 2 to 10 and we found that the fraction of nodes and edges they contain exhibits much more larger fluctuations. This result indicates that the external cores evolve differently from the nucleus. Starting from the most external 2-core, the average edge density calculated over all the years 2008-2017, increases approaching the internal nucleus form ∼42.1×10 −5 to ∼64.1×10 −2 indicating that the ASes in the nucleus are much more connected than those in the periphery.
We investigated the hierarchical organization of the Internet by studying the average clustering coefficient C. Our measurements show that C decreases with the degree k. The presence of a hierarchy is captured by the scaling C(k) ∼ k −γ [19], [21]. We found that even if a pure scaling behavior does not have statistical support, for all the years 2008-2017 the best fit power law of C(k) provides values of γ which differ only by ∼0.1% with an average of γ = 1.08. This value is very close to 1 that was found in [19] for a deterministic hierarchical model. Moreover, for this model C is independent on the size N of the network. We found that, apart from the year 2013, C(N ) weakly increases over time. Its minimum and maximum values are ∼0.59 and ∼0.68 measured in 2008 and 2017 respectively. For our convenience we defined three degree regions: high (k > 1000), medium (100 < k ≤ 1000) and small (k ≤ 100) and compared the CCDF of C(k) in these regions for all the years 2008-2017. We found that in the high degree region the distributions overlap while in the other two regions they are well separated. The gap is more pronounced in the low degree region and for a given value of k the neighbourhood connectivity increases over time. Hence, there are indications of an overall hierarchical organization of the Internet where a small fraction of big ASes (k > 1000) are connected to many regions with high internal cohesiveness containing low and medium degree ASes and these regions are slightly connected among them. During the evolution of the Internet in the decade 2008-2017 the neighbourhood connectivity of the ASes belonging to the high degree region was rather static while it increased over time in the other regions.
The average shortest path length S of the Internet slightly decreased during the decade 2008-2017 form ∼3.1 to ∼2.9 measured in 2008 and 2016-2017 respectively. Previous measurements [26] found that S was ∼3.5 in 2001 and ∼3.3 in 2006. This reinforces the fact that a pure power law model could not explain the evolution of the Internet because for 2 < α < 3 it predicts S ∼ lnlnN [10], [3].
Regardless of the analyzed year, the closeness centrality Γ of an AS increases with its degree. We found that Γ can be considered constant over the decade 2008-2017 in the small, medium and high degree regions with average values of ∼0.39, ∼0.43 and ∼0.48, respectively. Hence, big ASes are in the center of the Internet and low degree ASes are in the periphery.
It is reasonable to assume that the traffic load of an AS or carried by an edge is proportional to the number of shortest paths passing through the AS and containing the edge. These measurements can be quantified by the average node and edge betweenness centrality B n and B e . We found that B n increases with the degree indicating that big ASes manage much more traffic than small ASes. Moreover, the highest level of data traffic flows through edges that connect  Table 7: Average node betweenness centrality B n in the degree regions k ≤ 100, 100 < k ≤ 1000 and k > 1000. These quantities were measured also in [17] for three sources of data. Authors found that for the AS graph of the Internet constructed from the CAIDA Skitter [8] repository with data collected in March 2004 B n and B e were ∼11.0×10 −5 and ∼5.4×10 −5 . This is a further confirmation that during the evolution of the Internet the traffic load somehow decreases. This may be due to the adoption of more efficient routing policies and to infrastructural upgrades with more advanced network devices.