Research on the Evaluation of Urban Open Data


Urban open data is the key to the construction of smart city. Through the research on evaluation of urban open data, the concept, types, characteristics and other basic problems of urban open data are systematically summarized. From perspective of “quality”, “opening” and “acquisition”, a complete urban open data evaluation framework and index system is built. And the corresponding weights of evaluation indexes and score and overall rating methods are determined, so as to objectively evaluate the conditions of urban open data, and describe, monitor, guide and promote the construction and development of urban open data.

Share and Cite:

Liu, Y. , Jiang, C. and Li, S. (2017) Research on the Evaluation of Urban Open Data. World Journal of Engineering and Technology, 5, 122-134. doi: 10.4236/wjet.2017.53B014.

1. Introduction

In many areas and links of the construction of smart cities, urban open data has been considered as a prerequisite and basic work. The four main features of a “smart city”―comprehensive and thorough perception, ubiquitous interconnection of broadband, application of intelligent convergence and human-based sustainable innovation”―are based on open urban data. It can be said that urban open data is the key to “smart city”. Through urban open data, we can maximize the use of data by the whole society, and realize added value of data; more importantly, we can establish an ecological system operating around data, so that individuals, organizations, enterprises and government in the society are not only data producers, but also data analyzers, explorers and application users, so as to inspire infinite creativity and make the city really smart.

Urban open data is a complex problem involving multiple disciplines. This paper will systematically summarize the concept, types and features of urban open data and other basic problems through research on evaluation of urban open data, and put forward evaluation factors and the evaluation system of urban open data, so as to have a comprehensive and clear understanding of urban open data. From the practical point of view, quality and levels of urban open data at home and abroad are uneven. The research on evaluation of urban open data can standardize and deepen the content of urban open data, and comprehensively and objectively evaluate the quality and value of urban open data, so as to effectively describe, monitor, guide and promote the urban open data work.

In existing literatures, there are few researches on the evaluation of urban open data. We can mainly borrow from researches on open access (OA), open repository resources and open government data.

In terms of open access (OA), J. Beall established a number of criteria to judge the poor journals; Jiang Jing pointed out the comprehensive evaluation index system of open access journals, including academic information content, inclusion, quality of information released, copyright policy, academic influence, etc. [1]; Gu Liping, et al. discussed about the evaluation and the selection of open access journals from the perspective of quality level, and the degree of opening service ability [2]; and Chen Ming constructed the evaluation model of open access journals, including 16 evaluation indexes such as the number of articles published, time delay of article publishing, the total citation frequency, impact factor, journal h index and visits [3].

In terms of open repository resources, through project planning, satisfaction of academic goals of institution, the allocation and usage patterns of funds, the relation with related digital project, platform interoperability, measurement of document use and other indexes, M. Westell built the evaluation model of repository resources of institution [4]; Y. H. Kim, et al. developed and strengthened the evaluation system of open repository resources from three aspects, including system index, content index and management index [5]; and Sun Tan, et al. evaluated [6] and studied repository resources of foreign institutions from the perspective of system construction, content organization, service management, etc.

In terms of open government data, T. Davies evaluated the open data portal of government. The main standards included allowing users to directly find their desired fact, data visualization, supporting more efficient work, supporting innovation and reuse, etc.; T. Berners-Lee established a five-star evaluation system of linked open data, and each star corresponded to the specific evaluation content [7]; and Open Data Institute pointed out that excellent open data could be provided in correlated and structured format, and had guaranteed availability, consistency and traceability, etc.

From the existing literatures, we can see that researches on evaluation of open data are mainly done by foreign countries, and the evaluation objects are mainly open government data, scientific data, etc. In addition, researches on evaluation of open data are still in the initial stage. There is no mature and unified understanding of the selection of principles, methods and indexes of evaluation, weight distribution, etc. Researches on evaluation need to be enriched and improved.

2. Basic Problems of Urban Open Data

2.1. Concept

In broad sense, urban data contains all the data related to a city. Due to the complexity of a city, the urban data under the concept is too broad. This paper defines urban data as a series of data directly produced by urban activities or which can directly affect urban activities; and not all urban data is open city. Therefore, urban open data can be defined as data suitable for being opened directly produced by urban activities or which can directly affect urban activities. In short, it can be understood as the intersection of urban data and open data (Figure 1).

2.2. Classification

The content of urban data is very extensive. From the perspective of dynamic data produced by urban information service, some scholars divided urban data into 8 classes, including data of maps and points of interest, GPS data, passenger flow data, mobile phone data, LBS location service data, video monitoring data, environmental and meteorological data and social activity data [8]. These data may not be suitable for being open or data owners are reluctant to provide data for a variety of reasons. Urban open data is only part of them. Taking urban data of government involved in the actual business operation that has been opened involved in the standard, combined with classification and research of data on domestic and foreign urban open data portals, the classification of urban open data in the research is summarized. It is divided into three categories, including social and economic data, public service data and urban construction data. On this basis, it is further subdivided. There are a total of nine categories, including urban economy, social activities, public facilities, landscape environment, tour- ism development, road traffic, municipal facilities, land use and planning management (Table 1).

Figure 1. Relationship between urban open data, open data and urban data.

Table 1. Classification and content description of urban open data.

2.3. Features

Urban open data inherits the common features of open data, and it also has its own features. As open data, it has originality, readability, interactivity and relevance; as urban data, it has the following features. It is characterized by time and space. Different from other types of data, urban data often has space and time attributes. The data are based on composition and distribution of urban space. At the same time, changes of urban activities in different periods of time give data more meanings. It is comprehensive. City is a complex comprehensive system. The factors cross and superpose in the city, and form different attribute characteristic of the city. City, a comprehensive system, can be divided into several small systems, such as ecosystem, hydrogeology, pipe network system, power system and so on. Therefore, urban data is bound to be complex and diversified. Urban open data is structured. Acquisition and management of urban open data is clearly goal-oriented. Data is often recorded according to certain rules [9], so the data has strong practicality and stability, and a high value density.

3. Evaluation System of Urban Open Data

3.1. Framework of the Evaluation System

The framework of evaluation is foundation and support for the establishment of the evaluation system. The main idea of the construction of the evaluation framework of urban open data is based on the analytic hierarchy process (AHP). The hierarchical index evaluation structure model is used for multi-level and multi-angle control of evaluation indexes, and the objective and comprehensive evaluation behaviors are ensured. All the evaluation indexes of urban open data are divided into Level 1 Evaluation Indexes, Level 2 Evaluation Indexes and Level 3 Evaluation Indexes. All levels of indexes are linked together and progressive level by level. Different levels correspond to content of indexes with different precision and depth. Level 1 Evaluation Indexes are highly general, programmatic and concise; Level 2 Evaluation Indexes decompose Level 1 Evaluation Indexes, and form more specific types of indexes; and Level 3 Evaluation Indexes are further subdivision of Level 2 Evaluation Indexes, and complete the final index selection. They are rich in content and operational (Figure 2).

Figure 2. Hierarchical structure of the evaluation system of urban open data.

3.2. Selection of Evaluation Indexes

After construction of the evaluation framework of urban open data, it is important to determine the content in the framework, i.e. evaluation indexes. The selection of evaluation indexes starts from main points and standards of the evaluation of urban open data. With reference to the relevant evaluation results of open resources, the specific index items are determined with targets. Second, in the selection of evaluation indexes, both rationality and feasibility shall be considered, and the evaluation indexes selected shall be available.

1) Selection of Level 1 Evaluation Indexes

a) Quality evaluation indexes

Quality indexes cover the form of urban open data and evaluation indexes of the content value category. The direction of evaluation is the quality of the data itself. False and low-quality open data will directly lead to wrong conclusions, and waste manpower, material and other social resources. This aspect shall be first focused on.

b) “Open” evaluation indexes

“Open” indexes cover elements which urban open data should have in the link of data release. They focus on the evaluation of public release of urban open data to the whole society, and involve data security, data update and other key issues. They are the basic export-oriented features which should be guaranteed by urban open data.

c) “Acquisition” evaluation indexes

Acquisition indexes cover the evaluation of a series of factors in the process of acquiring the target data by the user. Such category of evaluation indexes gives more emphasis on customer service and experience feeling. The content includes users’ needs of convenience, feasibility and diversity in the process of data positioning, data searching, data browsing and data downloading. Data acquisition is the communication between users and data, and it is the key point to implement urban open data at the actual operation level.

2) Selection of Level 2 and Level 3 Evaluation Indexes

a) Decomposition of quality evaluation index

The quality of urban open data is mainly embodied in three aspects, including authenticity, integrity and spatiality. Among them, it is difficult to verify authenticity in the actual operation process of evaluation, so it is not directly considered as an evaluation index, and it can be replaced from an authoritative point of view. To sum up, Level 2 Evaluation Indexes of “quality” can be decomposed into integrity, space and authority.

Integrity can be judged by the breadth, depth, and the degree of repetition of data. The breadth of data is reflected in the coverage of data types, and can be evaluated from the data types involved. Depth aims at accurateness and level of detail of the same type of data information provided. The degree of repetition of data can be direct evaluated by the number of data repeated. Therefore, Level 3 Evaluation Indexes corresponding to integrity index are set as data type, information depth and number of data repeated.

Spatial indexes mainly focus on the spatial attribute of urban open data. Spatiality is an important feature of urban open data. The spatiality of data can be mainly judged from two aspects. One aspect is the accurate degree of spatial orientation of data. The other aspect is the performance of spatial orientation of data (such as just text description or spatial data). Therefore, Level 3 Indexes corresponding to spatial index are set as spatial granularity and spatial orientation.

The content evaluated by authoritative index mainly is mainly whether sources of data are authoritative and reliable, and have evidences. On the one hand, data provided by an authoritative and professional organization or department usually can guarantee a certain degree of quality; on the other hand, if the source or link of data is provided, the data is well documented, and it is also a guarantee of the quality of data. Seen from representation of authority, data provider is the primary basis. Metadata that can be acquired (i.e., data that describes data) is also included. Therefore, Level 3 Evaluation Indexes corresponding to authoritative index are set as data provider and metadata.

b) Decomposition of “public” evaluation index

The openness of urban open data can be mainly embodied in three aspects, including security, timeliness and relevance. Among them, timeliness is essentially the control of the timeliness of data. Therefore, “open” Level 2 Evaluation Indexes are decomposed into security, timeliness and relevance.

Security has always been one of the unavoidable problems of urban open data. It mainly can be judged by the sensitivity of information provided by the data, and whether the data itself contains risk factors. For processing of sensitive data, there are many technical means, which can effectively protect sensitive information from being disclosed. Whether the data itself is dangerous can be evaluated by whether the download link of test data contains virus or phishing links and other means. To sum up, Level 3 Evaluation Indexes corresponding to security are set as sensitive information and dangerous link.

The timeliness index mainly focuses on the validity of data at the time level. The timeliness of data can be judged from three aspects. First, the release time of urban open data can reflect the time background and the significance of data well; second, the update time of urban open data can reflect whether the data is latest and timely; third, different urban open data has different statistical cycle requirements, so rationality and stability of the update cycle is also one of the important aspects of timeliness. Therefore, Level 3 Evaluation Indexes corresponding to timeliness are set as release time, update time and update cycle.

The main content investigated by linkage index is the degree of interoperability between different data with reference to Linked Data. Linkage requires that local data (sets) be other external data (sets) are linked together. Therefore, it can be judged by observing whether urban open data has related external links or whether external links contain return link pointing to the data. Therefore, Level 3 Evaluation Indexes corresponding to linkage are set as being linked to data and being pointed to by data.

c) Decomposition of “acquisition” evaluation index

The acquisition of urban open data mainly includes machine reading, convenience and degree of interaction. Therefore, Level 2 Evaluation Indexes of “acquisition” are decomposed into machine reading, convenience and interactivity.

The machine reading index mainly includes two aspects. First, data format is correct and complete, and has no missing or damage. It will not hinder or have adverse effects on data reading and reuse; second, the diversity of data format, which means that the data format provided shall ensure the common general format, and provide different kinds of data format as many as possible, in order to meet different needs of users. Therefore, Level 3 Evaluation Indexes corresponding to machine reading are set as format integrity and format diversity.

The convenience index mainly considers the efficiency level in the first phase of data acquisition behavior, and it is the service function performance based on the user experience. On the one hand, the platform carrying data is one of the most important factors, and high-quality interface design allows users to quickly and accurately find the required data resources; on the other hand, restrictions on user identity may also affect the convenience to some extent, such as the need for registration, filling out the questionnaire, etc. In summary, Level 3 Indexes corresponding to the convenience index can be set as user restriction and interface design.

The interaction index mainly refers to feedback and evaluation of users' data utilization. For any urban open data, the feedback given by user after using the data is important data evaluation basis. The common form of feedback includes comment system and scoring (star) system; in addition, by providing APP software and other means, interest and enthusiasm for user feedback, upload and reuse results or other related data resources can be stimulated and improved. To sum up, Level 3 Evaluation Indexes corresponding to the interaction index can be set as comment system, scoring system and data application.

The summary of evaluation indexes of urban open data at different levels are shown in Table 2.

3.3. Weight of Evaluation Indexes

Determining the specific weight value of each evaluation index is an important link in the establishment of the evaluation system of urban open data. The weight value reflects the position and influence of the index in the overall evaluation, and directly decides the evaluation results of urban open data. Therefore, the weight value is an important manifestation of rationality and scientificity of the evaluation system. This paper adopts the analytic hierarchy process, combines with meaning and connotation of each index, and considers city planning science, statistics, information science, library science, science of public management and other disciplines of knowledge; at the same time, with the expert

Table 2. Selection of each level of evaluation indexes of urban open data.

method, 30 experts of city planning and other professional fields are consulted, and the judgment matrix of any two indexes at each level of the evaluation system of urban open data is obtained. With the help of matlab software, the normalization processing and consistency test of the judgment matrix are completed, and finally the weight value of each evaluation index and the overall ranking result of different levels are obtained (Table 3).

4. Scoring and General Comment of Evaluation Indexes

After the text edit has been completed, the paper is ready for the template. Duplicate the template file by using the Save As command, and use the naming convention prescribed by your journal for the name of your paper. In this newly created file, highlight all of the contents and import your prepared text file. You are now ready to style your paper.

4.1. Scoring of Evaluation Indexes

Scoring of evaluation indexes refers to scoring of the final level of evaluation indexes according to certain standards, so as to derive the score of the upper level

Table 3. The overall ranking result of each level of evaluation indexes.

of indexes, and complete the overall scoring of urban open data. In the evaluation system in this paper, all three levels of evaluation indexes will be scored. The scoring process follows the 5-level scoring system. Each level has corresponding judgment criteria and scores. The corresponding score will be assigned to Level 2 Index that meets a certain level of judgment criteria. Specific scoring settings are shown in Table 4.

4.2. Overall Comment

After scoring, Level 3 Index is multiplied by its weight to obtain the final score of the index. The final scores of all Level 3 Evaluation Indexes are added to obtain the final score of Level 2 Evaluation Indexes. In this way, finally the total score of urban open data is calculated. The total score is essentially the weighted average of urban open city. It still complies with the classification rules of the 5-level scoring system.

After calculation of the total score of urban open data, according to different

Table 4. Grading standards for three levels of evaluation indexes of urban open data.

evaluation purposes, the overall comment of urban open data can be given by area. The overall comment by area aims to summarizing and analyzing the scoring results by distinguishing different levels, so as to draw the final evaluation conclusion and select the urban open data required by different evaluation purposes.

The overall comment is mainly made as the total score complies with the 5- point system. The total score is divided into several areas. When all three levels of evaluation indexes have the lowest score of 1 point, the lowest score of the area is 1 point. In the same way, the highest score of the area is 5 points. On this basis, it is divided into 5 areas. Different areas correspond to different levels of evaluation. When the score is on the node, it shall be divided according to the following rules. When the score is 5 points, it belongs to the scope of Area 1. When the score is on other nodes, it belongs to the scope of the area on the left (Figure 3). The mathematical symbols: [5, 4.2] (4.2, 3.4] (3.4, 2.6] (2.6, 1.8] (1.8, 1].

5. Conclusion

Research on the evaluation of urban open data can help us to have a more profound and comprehensive understanding of the connotation of urban open data, strengthen the monitoring and management of urban open data, and effectively guide and promote the construction and development direction of urban open data.

This paper is a preliminary discussion about evaluation of urban open data. The established related evaluation system and evaluation methods are proved to have good operability, and can find problems and guide the urban open data practice through the verification of application of open data in Shanghai, Beijing and other cities. In spite of this, on the one hand, because of various limitations, the research still has many disadvantages; on the other hand, the urban open data itself is developing and changing, and the understanding of its evaluation methods is not changeless. We hope that scholars can join us and further explore and research urban open data and its evaluation. In the future, urban open data and its evaluation research work can be expanded from the following aspects.

1) We shall strengthen the research of front-end problems of urban open data, including the related basic theories of urban open data, the construction of law and regulation environment, the government's encouragement and guidance policies for urban open data. A great deal of content still needs pioneering work.

2) We shall strengthen the research on evaluation of the quality of urban open data. So far, the evaluation theory and means of quality of urban open data are not deep enough, and the operational capacity is low. The data quality is the material basis for urban open data and is of great significance. Due to the impact of the concept of big data, the scale of urban open data increases unprecedentedly, bringing the decrease of overall value density of data and many problems

Figure 3. Overall comment on different areas of urban open data.

of private data security management. Facing such a situation, we need to develop more scientific and standardized quality assessment methods, and strictly control the quality of open data of all types of cities, which can effectively promote the development and progress of urban open data.

3) We shall actively do relevant research on the construction of urban open data platform, including the research on platform construction standards. At present, many cities in China have joined the team of open data, and many cities are setting up their own open data platforms, but compared with foreign countries, the construction scale and quality of platform is not enough. The system of urban open data construction has not been formed.

4) We shall strengthen the research on popularization of urban open data and its evaluation and related cultural construction. For individuals, organizations or the government, we shall state that the construction of urban open data is not only material, but also the construction of data culture which is required more. We can start from the cultivation of data literacy, improvement of data awareness and participation of the public, guarantee of right to data, creation of collaborative governance data of different stakeholders and other aspects, and explore the construction of the ecological system of data suitable for the development of urban open data.

Conflicts of Interest

The authors declare no conflicts of interest.


[1] Jiang, J. (2011) Research on Comprehensive Evaluation Index System of Open Access Journals. East China Normal University, Shanghai.
[2] Gu, L.P., Zhang, X.L., Chu, J.L., et al. (2013) Evaluation and Selection of Open Access Journals: Quality Level, Openness and Service Capability. Library and Information Service, No. 1, 49-54.
[3] Chen, M. (2010) Construction of Evaluation Model for Open Access Journals. Library and Information Service, No. 14, 11-15.
[4] Westell, M. (2006) Institutional Repositories: Proposed Indexes of Success. Library HiTech, No. 2, 211-226.
[5] Kim, Y.H. and Kim, H.H. (2007) Development and Validation of Evaluation Indexes for a Consortium of Institutional Repositories: Across Discipline. The Journal of Academy Librarian-Ship, No. 6, 647-654.
[6] Sun, T. and Tao, J. (2010) Review of Researches on Evaluation of Overseas Institutional Repositories from 2002 to 2009. Library Construction, No. 4, 6-9, 22.
[7] Berner-Lee, T. (2014) Is Your Linked Data 5 Star?
[8] Wang, J.Y., Li, C., et al. (2014) Summarization of Researches on Da-ta-Centered Smart City. Computer Research and Development, No. 2, 239-259.
[9] Ma, J.G. and Jiang, W. (2013) The Concept, Characteristics and Application of Big Data. Defence-Related Science and Technology, No. 2, 10-17.

Copyright © 2024 by authors and Scientific Research Publishing Inc.

Creative Commons License

This work and the related PDF file are licensed under a Creative Commons Attribution 4.0 International License.