Restoration of grassland such as exclusion of grazing has been considered to increase aboveground plant diversity and soil fertility. However, knowledge on the effect of long-term exclusion of grazing on soil bacterial community structure and diversity is not well understood. The two sites were selected in the Inner Mongolian grassland, i.e., one fenced off since 1979 (UG79) and the other continually grazed by sheep (FG) all along. Soil microbial biomass was measured using fumigation method and bacterial community structure and diversity were assessed using methods of Denaturing Gradient Gel Electrophoresis (DGGE) and clone library. Results showed that the UG79 soil had significantly higher microbial biomass carbon and nitrogen compared with the FG soil. There was a clear separation in soil bacterial community structure, but not in bacterial diversity between the two sites. Moreover, 55 clones from the UG79 soil and 56 clones from the FG soil were selected and sequenced. Phylogenetic analysis of all clone sequences indicated that bacterial communities were dominated by the groups of Actinomycetes, Proteobacteria and Bacteroidetes, but there were no significant differences in bacterial diversity between the two sites, consistent with the results obtained from DGGE. The results highlighted that although long-term exclusion of grazing increased soil microbial biomass, but it did not harbor higher bacterial diversity compared with freely grazed site.