Design and Application of Integrated Monitoring Model for Provincial Meteorological Observation Data Transmission

Abstract

With the development of meteorological services, there are more and more types of real-time observation data, and the timeliness requirements are getting higher and higher. The monitoring methods of existing meteorological observation data transmission can no longer meet the needs. This paper proposes a new monitoring model, namely the “integrated monitoring model” for provincial meteorological observation data transmission. The model can complete the whole network monitoring of meteorological observation data transmission process. Based on this model, the integrated monitoring system for meteorological observation data transmission in Guangdong Province is developed. The system uses Java as the programming language, and integrates J2EE, Hibernate, Quartz, Snmp4j and Slf4j frameworks, and uses Oracle database as the data storage carrier, following the MVC specification and agile development concept. The system development uses four key technologies, including simple network management protocol, network connectivity detection technology, remote host management technology and thread pool technology. The integrated monitoring system has been put into business application. As a highlight of Guangdong’s meteorological modernization, it has played an active role in many major meteorological services.

Share and Cite:

Zhang, Y. , Chen, X. , Bi, S. , Shen, P. , Ao, Z. and Hu, Z. (2019) Design and Application of Integrated Monitoring Model for Provincial Meteorological Observation Data Transmission. Journal of Geoscience and Environment Protection, 7, 94-104. doi: 10.4236/gep.2019.75010.

1. Introduction

Meteorological observation data is an important support for disaster weather monitoring, early warning, forecasting and service. Dynamic monitoring data processing timeliness timely discovers and resolves possible abnormalities in data processing, optimizes data flow, and ensures that meteorological observation data can be used in the shortest time. Entering the forecast business service platform (Xiao et al., 2009) is the main task of the regular operation monitoring service of meteorological information networks. For a long time, the National Meteorological Information Department has developed the “9210 System”, the new generation of domestic meteorological communication system and the China Integrated Meteorological Information Service System (CIMISS), including the provincial data transmission monitoring module, relying on these three major. In the system, provincial-level meteorological information departments are also rushing to develop data transmission monitoring systems, but most of the monitoring content is concentrated on the time-out rate of meteorological data transmission and the corresponding statistics (Zhang et al., 2009; Sun et al., 2012; Zhong et al., 2015) , alarms (Jia et al., 2013; Liang & Liu, 2014; Hua et al., 2015) , etc. The scope of surveillance is limited to provincial meteorological information systems (Li et al., 2016) . With the development of meteorological services, there are more and more types of real-time observation data, and the timeliness requirements are getting higher and higher. The monitoring methods of existing meteorological observation data transmission can no longer meet the needs.

To make the research background seen, we list some references with personal comments regarding this research area below. Lin et al. (2018) find that it is of great significance to actively adapt to the characteristics of information dissemination in the all-media era, and propose the construction of meteorological information dissemination mechanism. Chen et al. (2017) present a set of best practices for workflow design and implementation for numerical weather prediction models and meteorological data service, which have been in operation in China Meteorological Administration for years and have proven to be effective in the managing of the complexities of large-scale meteorological related work flows. Lin et al. (2017) propose a set of software classification method based on software operating characteristics with principal component analysis used to reduce the dimension of software running feature data and to interpret software characteristic information, and a modified K-means algorithm used to classify the meteorological data processing. Datta et al. (2016) aim to design optimal monitoring network to locate the plume boundary with a linked simulation optimization based methodology for optimal monitoring network design.

This proposed model solves lots of question within existing models. We illustrate why and how below. Why there are lots of questions within existing models? First, through the system, the user can only know the data transmission status (timely, over-limit or missing). If the transmission is not normal, the reason cannot be known. Knowing the reason, you need to pass the cumbersome “check”; Second, through the system, the user can only know the data status, cannot get the network status (on, off) to transmit the corresponding data; third, for the second point, even through the network The monitoring system obtains the network status, and the network device (interface) status for transmitting the corresponding data cannot be obtained. Fourthly, it is impossible to know whether the detecting device generates the observation data normally.

To explain how we solve lots of questions within existing models, this paper proposes a new monitoring model, namely the “integrated monitoring model” for provincial meteorological observation data transmission. The model can complete the whole network monitoring of the meteorological observation data transmission process, and realize the monitoring of the end (probe end) to the end (user end) by collecting all the information of the host, network and data involved in the data transmission process, and can realize data transmission. Quick positioning of fault points. Based on this model, an integrated monitoring system for meteorological observation data transmission in Guangdong Province was developed. The system has been used in business applications. As a bright spot in meteorological modernization, it has played an active role in many major meteorological services and has been recognized by front-line attendants and leaders at all levels.

2. Demand Analysis

2.1. Overall Goal

Combined with the requirements of provincial meteorological observation data transmission services and the existing monitoring systems, the integrated monitoring model needs to meet two requirements: First, the data transmission process is monitored throughout the network, and the observation data is from the detection device to the user interface. Transmission process monitoring enables network-wide monitoring of hosts, network devices, and data itself. As shown in Figure 1, the entire network is monitored from the data source

Figure 1. Flow chart of monitoring the entire network of detection data transmission of a radar station in Guangdong Province.

through five nodes to the user. Second, the fault point is quickly located. In the whole process of data transmission, if a data transmission abnormality occurs, the model can be used to initially determine the possible conditions, and the fault point can be quickly located to restore the normal state in time.

2.2. Business Functions

The “integrated monitoring model” for provincial-level meteorological observation data transmission proposed in this paper has some new functions in addition to the functions of the existing monitoring system. The specific business functions are as follows:

1) It not only includes all kinds of meteorological observation data to report, including timely, over-limit, lack of report, correction report, etc. It also includes information of various meteorological observation data flowing through each key node (host) during transmission.

2) Not only can the data transmission abnormality be discovered in time, but also the possible causes of the abnormality, including the network, the host, the transmission program, and the state of the observation device.

3) When the data transmission is abnormal, the status information of all the key network nodes in the transmission process, including network connectivity and network device interface status, can be obtained.

4) When it is found that the data transmission is abnormal, the status information of all the key host nodes in the transmission process can be obtained, including the CPU, memory, disk utilization, transmission program status, and FTP service status of the host.

5) The data transmission monitoring model also considers making a basic judgment on the data itself, such as whether it is an empty file, whether it is obviously too small, etc., and then determining whether the data is generated during the detection or during the transmission. It also provides reference information for the operation and maintenance of meteorological equipment.

3. Model Design

3.1. Transmission Process

According to the relevant documents of the China Meteorological Administration, the provincial meteorological observation data transmission process is basically station (county)-city-province. With the rapid development of meteorological modernization, there are more and more types of meteorological observation data, and the transmission process of different observation data is different. Take the short-term forecasting of the most important radar data as an example, Guangdong Province radar station to the city, provincial system has Yeston and mobile two links back to each other. After the radar detection data is generated, the files are first landed on the two hosts of the radar station; then transmitted to the city router through the Yeston or mobile network (radar station A (main) or B (main) router); then via Yeston or mobile network (city level) The A or B router is transmitted to the provincial router; then enters the provincial receiving directory, flows into the provincial system, and finally provides the user with the file, database or interface.

3.2. Model Design

In order to achieve the goal of monitoring the entire network of provincial meteorological observation data transmission process, an “integrated monitoring model” is proposed. Taking Figure 1 as an example, real-time monitoring data flows through three host nodes and three network node states to realize network-wide monitoring. The three host nodes are: radar station data transmission server, provincial data receiving server and data service node (including directory, database and interface); three network nodes are: radar station, city level, provincial level router (main and standby).

The idea is that the observation data corresponds to the network link, the network link is composed of network equipment and host, and the data-link model, link-network equipment-host model is established, and Figure 2 is the integrated monitoring of Zhaoqing radar data transmission in Guangdong.

As shown in Figure 2, the correspondence between the data and the network link is established by the Zhaoqing Radar Link ID (ZQRAD_LINKID) and the Zhaoqing Radar Data ID (ZQRAD_DATAID). Zhaoqing Radar Link ID (ZQRAD_LINKID) includes Zhaoqing Radar Station ID (ZQRAD_STAID), Provincial Zhaoqing Radar Data Service Virtual Host ID (ZQRAD_PVSHOSTID), Provincial Zhaoqing Radar Data Receiving Host (ZQRAD_XYDSRV), Provincial Zhaoqing Radar Virtual Router (ZQRAD_PVRID), city-level Zhaoqing radar virtual router (ZQRAD_CVRID), Zhaoqing radar station virtual router (ZQRAD_SVRID) and Zhaoqing radar data transmission host (ZQRAD_DHOSTID). Among them, the provincial Zhaoqing radar data service virtual host ID (ZQRAD_PVSHOSTID) and Zhaoqing radar data service host ID (ZQRAD_SHOSTID), Zhaoqing radar data service interface (ZQRAD_SIFID), Zhaoqing radar database service interface (ZQRAD_SDBID); provincial Zhaoqing radar virtual router (ZQRAD_PVRID), the municipal Zhaoqing radar virtual router (ZQRAD_CVRID), and the Zhaoqing radar station virtual router (ZQRAD_SVRID) are respectively associated with the main (A) and standby (B) routers. In this way, the data is associated with each key node in the whole transmission process, and then the host, network and data status information are collected separately to form a complete “integrated monitoring”.

As shown in Figure 2, the correspondence between the data and the network

Figure 2. Flow chart of Guangdong Zhaoqing radar data transmission integrated monitoring model.

link is established by the Zhaoqing Radar Link ID (ZQRAD_LINKID) and the Zhaoqing Radar Data ID (ZQRAD_DATAID). Zhaoqing Radar Link ID (ZQRAD_LINKID) includes Zhaoqing Radar Station ID (ZQRAD_STAID), Provincial Zhaoqing Radar Data Service Virtual Host ID (ZQRAD_PVSHOSTID), Provincial Zhaoqing Radar Data Receiving Host (ZQRAD_XYDSRV), Provincial Zhaoqing Radar Virtual Router (ZQRAD_PVRID), city-level Zhaoqing radar virtual router (ZQRAD_CVRID), Zhaoqing radar station virtual router (ZQRAD_SVRID) and Zhaoqing radar data transmission host (ZQRAD_DHOSTID). Among them, the provincial Zhaoqing radar data service virtual host ID (ZQRAD_PVSHOSTID) and Zhaoqing radar data service host ID (ZQRAD_SHOSTID), Zhaoqing radar data service interface (ZQRAD_SIFID), Zhaoqing radar database service interface (ZQRAD_SDBID); provincial Zhaoqing radar virtual router (ZQRAD_PVRID), the municipal Zhaoqing radar virtual router (ZQRAD_CVRID), and the Zhaoqing radar station virtual router (ZQRAD_SVRID) are respectively associated with the main (A) and standby (B) routers. In this way, the data is associated with each key node in the whole transmission process, and then the host, network and data status information are collected separately to form a complete “integrated monitoring”.

3.3. Technical Model

In order to achieve the goal of “integrated monitoring”, the following four technical models are designed to complete the collection and processing of monitoring information of various key nodes:

1) Host and file status information collection model

The Windows system host and file status information collection process collects status information for the development host local agent, including “host performance collection”, “file monitoring”, “file name regularization”, and “message sending”. For Linux and AIX system hosts, develop a remote host monitoring agent, and use SSH and Telnet technology to remotely collect host and file status information, as shown in Figure 3.

2) Network connectivity detection model

Network connectivity detection is implemented through the ping and tracert technologies. The process is as follows: First, obtain basic network parameters, then analyze the ping command to return information, and determine the network connectivity status. If not, use the tracert command to return information and compare the critical routing nodes. To identify possible network failure points.

3) Network device status information collection model

The network device status information collection is mainly implemented by the SNMP v2C protocol. The implementation process is as shown in the following Figure 4. First, according to the Object Identifier (OID) and interface (interface, IF) configuration, the configuration parameters are obtained, and the SNMP protocol program is developed to obtain the remote. Network device (open

Figure 3. Schematic diagram of host and file status information collection model.

Figure 4. Schematic diagram of network device status information collection model.

SNMP) status information, such as CPU, memory usage, interface status (normal or not, down/up) forwarding rate, packet loss rate, and so on.

4) Information processing and warehousing model

After the host, network, and data status information is collected, the information processing and storage are completed according to the following process, including judging the file expiration according to the program table, determining the file timely, over-limit or missing, and the result is inserted into the corresponding data table storage.

4. System Implementation

4.1. Technical Architecture

The overall architecture of the system is divided into three layers: data acquisition, data processing and data storage. The data acquisition layer collects monitoring information in the data transmission process through the host, the network, and the data agent; the message middle layer Active MQ captures the forwarding message, and the data processing layer is responsible for analyzing and processing various types of monitoring information; the data storage layer uses the ORACLE data table as The carrier inserts the result into the data table for presentation by the front end of the monitoring system.

The system is designed according to the “integrated monitoring model”, using Java as the programming language, integrating J2EE, Hibernate, Quartz, Snmp4j and Slf4j frameworks, using Oracle database as the data storage carrier, following the MVC specification and agile development concept.

According to the function and development needs, the system is divided into four modules: host agent, network connectivity detection, network device status information collection, information processing and storage. The technical flow of each module can be found in the corresponding part of the 3.3 Technical Model section.

4.2. Library Table Design

Using Oracle database storage, design 30 data tables, the main database table is shown in Table 1.

4.3. Key Technology

In this section, we will introduce clearly the technique barrier we break in this paper, like protocol. Firstly, we set up a protocol named Simple Network Management Protocol (SNMP). The network device is remotely managed by the SNMP to obtain network device status information. In development, the java version of the org.snmp4j.Snmp package is referenced, with OID as the parameter, and the corresponding information is returned. There are several manufacturers’ equipment in the Guangdong Meteorological Network System. When the OID is configured, the MG-SOFT MIB Browser software is used to verify the correctness one by one.

Secondly, we use the network connectivity detection technology. To determine the network status, use the network diagnostic tool ping (Packet Internet Groper) to determine network connectivity. Set the ping command 10 times in a row, the packet size is 1000 bytes, and if there are 3 times all packets are lost, the network is considered faulty. The tracer (Trace router) uses the IP Time to Live

Table 1. The main database table used by the integrated monitoring system.

(TTL) field and ICMP error messages to determine the route from one host to other hosts on the network. The development time command parameter is “Tracert-d -h5 target IP”, that is, the host name is not retrieved, and only the 5-hop route is tracked. Ensure tracert execution efficiency and avoid network infinite loops.

Thirdly, we also use the remote host management technology. One of the outstanding problems encountered in the development of surveillance systems is that they cannot affect the original business system. In order to do this, remote host protocol (telnet)/secure shell protocol (sh) is used to remotely log in to the host for the main service host, and related commands are generated to obtain related status information such as host, process and data.

Finally, we use thread pool technology. The monitoring system needs to monitor the host, network equipment, and data, and will gradually increase. And the monitoring system has higher requirements for aging. If you do not use multi-threading, thread pool technology, you cannot meet the actual business needs. The system uses the Quartz framework to implement scheduled multi-threaded monitoring information collection using the Scheduler, Job, Trigger, and JobExecutionContext objects.

5. Conclusion

According to the main work and achievements extracted from the aforementioned parts, we can get the following conclusions. At present, the provincial meteorological observation data transmission monitoring system cannot monitor the entire transmission process and cannot quickly locate the fault point, which cannot meet the needs of meteorological modernization development. Based on this, this paper proposes an “integrated monitoring model” for provincial meteorological observation data transmission. The model can realize the whole network monitoring of the meteorological observation data transmission process, realize the monitoring of the end (detection device end) to the end (application server end), and can quickly locate the data transmission fault point. The data transmission process of provincial meteorological observation data is introduced; the design ideas of the new model are expounded; and the four technical models involved are described. The important issues and key technologies in the overall framework, technical route, design and development of the main database table based on this model are discussed.

The “integrated monitoring model” for provincial meteorological observation data transmission introduced in this paper has a short time. Although the developed monitoring system has been running in the Guangdong Provincial Meteorological Bureau for about one year, the actual test time is not long enough and there is still room for improvement. At present, the meteorological observing equipment monitored is still incomplete. When new monitoring equipment is added one after another, there may still be some expansion problems, which need to be gradually improved. In addition, the model was not designed to control the host, network equipment and processes, which is an important issue to be studied in the next step.

Funding

This work was jointly funded by the Science and Technology Planning Project of Guangdong Province (grant number 2018B020207012,2019B020208016), Natural Science Foundation of Guangdong Province (grant number 2018A030313218), the 2018 Open Research Program of the State Key Laboratory of Severe Weather (grant number 2018LASW-B08), the Science and Technology Projects of Guangdong Meteorological Service (grant number GRMC2018M22) and the Graduate Student Scientific Innovation Projects in Jiangsu Province (grant number KYLX15_0870).

Conflicts of Interest

The authors declare no conflicts of interest.

References

[1] Chen, X., Wei, M., & Sun, J. (2017). Workflow-Based Platform Design and Implementation for Numerical Weather Prediction Models and Meteorological Data Service. Atmospheric and Climate Sciences, 7, 337-351.
https://doi.org/10.4236/acs.2017.73025
[2] Datta, B., Durand, F., Laforge, S., Prakash, O., Esfahani, H., Chadalavada, S., & Naidu, R. (2016). Preliminary Hydrogeologic Modeling and Optimal Monitoring Network Design for a Contaminated Abandoned Mine Site Area: Application of Developed Monitoring Network Design Software. Journal of Water Resource and Protection, 8, 46-64.
https://doi.org/10.4236/jwarp.2016.81005
[3] Hua, L., Wang, J., Jin, S. et al. (2015). Design and Implementation of Intelligent Voice Notification System Based on IMS for Meteorological Information Transmission. Meteorological Technology, 43, 1040-1045.
[4] Jia, X., Luo, Y., Lao, S. et al. (2013). Design and Implementation of Meteorological Information Transmission Monitoring Service System for Grassroots Stations. Meteorological Technology, 41, 1049-1052.
[5] Li, Z., Hu, J., Yang, L. et al. (2016). Design and Implementation of Quarantine Monitoring System for Meteorological Data Processing Based on CIMISS. Meteorology and Disaster Reduction Research, 39, 309-313.
[6] Liang, W., & Liu, J. (2014). Design and Implementation of Meteorological Information Transmission Monitoring BS System. Agricultural Science Research, 2, 94-96.
[7] Lin, L., Shao, B., Zhang, H., & Li, X. (2018). Research on Meteorological Information Transmission Mechanism in the Age of All Media. Journal of Geoscience and Environment Protection, 6, 60-65.
https://doi.org/10.4236/gep.2018.68006
[8] Lin, M., Zhao, X., Fan, C., Xie, L., Wei, L., & Guo, P. (2017). Polarimetric Meteorological Satellite Data Processing Software Classification Based on Principal Component Analysis and Improved K-Means Algorithm. Journal of Geoscience and Environment Protection, 5, 39-48.
https://doi.org/10.4236/gep.2017.57005
[9] Sun, Z., Xiao, W., Song Y. et al. (2012). Improved Design and Implementation of Real-Time Monitoring System for Meteorological Information. Journal of Chengdu University of Information Technology, 27, 168-173.
[10] Xiao, W., Wei, W., & Yang, G. (2009). Preliminary Analysis of Business Process Optimization of Meteorological Information Transmission. Journal of Applied Meteorology, 20, 497-503.
[11] Zhang, F., Wu, P., Huo, R. et al. (2009). Design and Implementation of Integrated Monitoring System for Meteorological Information Network Based on CS Structure. Meteorological and Environmental Sciences, 32, 279-282.
[12] Zhong, Y., Guo, H., Xia, Z. et al. (2015). Development of Mobile Weather Information Transmission Monitoring Software Based on Android. Meteorological Science, 43, 1065-1069.

Copyright © 2024 by authors and Scientific Research Publishing Inc.

Creative Commons License

This work and the related PDF file are licensed under a Creative Commons Attribution 4.0 International License.