Georeferencing 3D Tiles Generated from Photogrammetry-Derived Mesh Using Ground Control Points

Kyongil Woo, Adrian Onsen, WonSok Kim

Institute of Remote Sensing and Geoinformatics, Academy of Science, Pyongyang, DPRK.

DOI: 10.4236/jgis.2022.145023                

Abstract

In order to achieve efficient visualization of massive 3D photogrammetry models on the web-based virtual globe, original photogrammetry 3D models should be converted into 3D Tiles with accurate georeferencing information to render on the correct surface position. There are a few software tools and online platforms for generating 3D Tiles and various ways to generate 3D Tiles were reported, but none of them do not provide a way to georeference 3D Tiles with GCPs. This paper is intended to suggest an approach to georeference 3D Tiles with GCPs based on the Kabsch algorithm which is very similar to georeferencing raster data with GCPs in the traditional GIS. First, we mathematically describe how to apply the Kabsch algorithm for georeferencing 3D Tiles with GCPs. Following this description, we demonstrate our implementation allowing constructing GCPs and calculating the “transform” property of 3D Tiles. Finally, we tested the suggested method in Construkted Reality which is the platform to stream and visualize massive photogrammetry and point clouds datasets.

Keywords

3D Tiles, GCP, Kabsch, 3D WebGIS, Cesium

1. Introduction

In recent years, photogrammetry technology has made tremendous progress technically and automatic 3D model reconstruction technique is becoming more and more mature. However, the significant amount of data associated with 3D models poses a challenge in terms of efficiently transmitting and visualizing 3D models on the Web-based 3D model viewer. To accommodate the streaming of massive 3D models, various open 3D model format specifications such as 3D Tiles, Indexed 3D Scene Layer (I3S), and Spatial 3D Model (S3M) were developed, and among them, 3D Tiles exhibit greater flexibility, higher customizability, better portability, and openness [1].

3D Tiles [2] is an open specification for streaming and rendering massive multi-source heterogeneous 3D geospatial datasets. It defines a spatial data structure and a set of tile formats designed for 3D and optimized for streaming and rendering. Its primary purpose is to improve the streaming and rendering performance of massive heterogeneous datasets.

As 3D Tiles has gained widespread adoption as the preferred way for streaming massive 3D data on the web, more and more software platforms have added support for the 3D Tiles, and many photogrammetry software tools such as FME [3], Bently ContextCapture [4], Agisoft MetaShape [5], and RealityCapture [6] support exporting 3D Tiles.

In order to accurately render 3D Tiles in the real-world location, they should be georeferenced, and this can be done in various ways depending on how 3D Tiles are generated.

Many photogrammetry software tools use GCPs to ensure that a 3D model’s reconstruction has the correct scale, orientation, or absolute position information [7] but, the way to georeference 3D Tiles generated from the 3D model with used GCPs which is very desirable has not yet been reported.

Therefore, in this paper, we suggest and implement the method to georeference 3D Tiles with GCPs based on the Kabsch algorithm [8], which is very similar to georeferencing raster data with GCPs in traditional GIS.

The remainder of the paper is organized as follows: Section 2 provides the review of related works, and the suggested idea is presented in Section 3, in Section 4 we demonstrate the implementation of our system. In Section 5 the validity is tested through experiments for sample 3D Tiles. Finally, Section 6 concludes the paper.

2. Related Works

Converting different kinds of geospatial data such as photogrammetry-lidar derived mesh, BIM, and point cloud into 3D Tiles has drawn much attention since the strength of 3D Tiles was known, and a few commercial software tools and platforms like Cesium ion [9], FME, Bently Context Capture, RealityCapture for it were reported. However, there are some limitations in customizing 3D Tiles generation because the conversion workflow is still a blackbox for users. Moreover, using them is infeasible in case the data are sensitive to the restricted public access requirement.

It is generally agreed that so far, no standard, a wholesome open-source solution exists for converting geospatial data into 3D Tiles. Therefore, many researchers have studied different tiling methods according to the characteristics of their data and the purpose of their project.

Schilling et al. [10] described the data processing steps from CityGML to 3D Tiles, which used CESIUM_RTC [11] extension to georeference glTF. Gan et al. [12] suggested a method of generating 3D Tiles from DSM (digital surface model). Song et al. [13] proposed 3D Tiles dynamic loading and scheduling strategy based on terminal memory size and screen space error but did not explain how 3D Tiles used in their experimental was generated. Chen et al. [14] presented the workflow to convert IFC to 3D Tiles using multiple open source tools and libraries. Visuri et al. [15] used Cesium ion to convert their CityGML data to 3D Tiles. Kolarić et al. [16] implemented a set of routines for generating 3D Tiles that contain buildings with OpenStreetMap(OSM) data. Xu et al. [17] studied creating 3D Tiles from IFC using obj2gltf, and discussed how to calculate the transition matrix between the local coordinate system of IFC and EPSG:4978. Lu et al. [18] described the converting steps for generating 3D Tiles from WRCD (weather radar composing data) and they calculate a transform property of tileset.json to georeference 3D Tiles using the geographic coordinates of the WRCD center point. Mao et al. [19] proposed a workflow for generating 3D Tiles from CityGML. Li et al. [20] used objTo3dtiles to convert 3D electrical model to 3D Tiles but did not describe how their 3D Tiles was georeferenced. Jaillot et al. [21] suggested the temporal extension of 3D Tiles, and converted CityGML data of a 3DCityDB defined in a projected coordinate system to 3D Tiles using their own software, Py3DTiles for delivering time-evolving 3D city models on the web. Zhan et al. [1] presented a tiling algorithm of complex BIM models based on the R-tree.

In summary, although there were a few studies on generating 3D Tiles, to the best of our knowledge, no researchers have attempted to use GCPs to georeference 3D Tiles.

One of the easy and convenient ways to tile photogrammetry-derived 3D mesh is to use Cesium ion, which provides the “3D Tiles Location Editor” to georeference generated 3D Tiles. However, the “3D Tiles Location Editor” has a limitation in using because it is not easy to determine parameters like longitude, latitude, height, heading, pitch, and roll for georeferencing [22].

In the following section, technical specifications about our idea behind the georeferencing of 3D Tiles with GCPs are described.

3. Suggested Approach

We suggest a way to georeference 3D Tiles generated from a photogrammetry-derived mesh with GCPs using the Kabsch algorithm [8]. We do not mind how a mesh is converted to 3D Tiles and focus on only georeferencing generated 3D Tiles. And we assume coordinates of all vertices of 3D Tiles are defined in the local coordinate system and the given tileset does not contain any nested tileset.

3D Tiles can have georeferencing information in two ways: directly define vertex positions in WGS84, Earth-centered, Earth-fixed coordinate system (ECEF or EPSG:4978 [23]) using RTC_CENTER [24]; to define vertex positions in the local coordinate system and to specify the “transform” property of each tile so that all vertex positions are transformed into EPSG:4978.

The basic idea is to set up points corresponding to the given GCPs on the 3D Tiles and find the best transformation by which transformed corresponding points match GCPs, and set the “transform” property using calculated transformation. This is formalized to the problem to find the optimal/best rotation and translation between two sets of corresponding 3D points data (Figure 1).

To solve this problem, we use the Kabsch algorithm [8] which is the method for calculating the optimal translation and rotation that minimizes the Root Mean Squared Deviation (RMSD) between two paired sets of points. It is widely used in structural bioinformatics, molecular simulations, molecular modelling, structural biology research to superimpose pairs of molecules and physics simulation. Figure 2 shows the flowchart of Kabsch algorithm, and it is efficient and easy to implement because it only requires a few matrix operations.

However, the algorithm requires a predefined mapping between the sets of points, for example, in our case, the mapping of GCPs and corresponding points. We will describe how to setup such mapping in our system in Section 4.

Let $B_i,L_i,H_i\left(i=1,2,\cdots ,n\right)$ be latitude, longitude, and height of GCPs and $p_i\left(i=1,2,\cdots ,n\right)$ a point corresponding to ith GCP on the 3D Tiles.

So, $P$ of the Kabsch algorithm can be directly constructed with $p_i$

$P=\left(\begin{array}{ccc}{p}_{x1}& p_{y1}& p_{z1}\\ p_{x2}& p_{y2}& p_{z2}\\ ⋮& ⋮& ⋮\\ p_{xn}& p_{yn}& p_{zn}\end{array}\right)$ (1)

$Q$ of the Kabsch algorithm can be constructed by converting the geographic coordinate of GCPs into ECEF coordinate system.

$Q=\left(\begin{array}{ccc}{q}_{x1}& q_{y1}& q_{z1}\\ q_{x2}& q_{y2}& q_{z2}\\ ⋮& ⋮& ⋮\\ q_{xn}& q_{yn}& q_{zn}\end{array}\right)$ (2)

Figure 1. Georeferencing 3D Tiles by transformation.

Figure 2. Flowchart of Kabsch algorithm.

where $\left(q_{xi},q_{yi},q_{zi}\right)$ is a cartesian coordinate in ECEF for $B_i,L_i,H_i-H_{\text{offset}}\left(i=1,2,\cdots ,n\right)$. $H_{\text{offset}}$ is a constant called “altitudeOffset”, and its meaning is explained in the following section.

Let $C_p,C_q$ are centroids of GCPs and corresponding points.

$C_p=\frac{1}{n}\underset{i=1}{\overset{n}{{\displaystyle \sum }}}\left(p_{xi},p_{yi},p_{zi}\right)$ (3)

$C_q=\frac{1}{n}\underset{i=1}{\overset{n}{{\displaystyle \sum }}}\left(q_{xi},q_{yi},q_{zi}\right)$ (4)

Then the Translation step of the Kabsch algorithm is done using $C_p,C_q$, and P, Q are updated so that their centroid coincides with $C_p,C_q$.

$P=\left(\begin{array}{ccc}{p}_{x1}-C_{px}& p_{y1}-C_{py}& p_{z1}-C_{pz}\\ p_{x2}-C_{px}& p_{y2}-C_{py}& p_{z2}-C_{pz}\\ ⋮& ⋮& ⋮\\ p_{xn}-C_{px}& p_{yn}-C_{py}& p_{zn}-C_{pz}\end{array}\right)$ (5)

$Q=\left(\begin{array}{ccc}{q}_{x1}-C_{qx}& q_{y1}-C_{qy}& q_{z1}-C_{qz}\\ q_{x2}-C_{qx}& q_{y2}-C_{qy}& q_{z2}-C_{qz}\\ ⋮& ⋮& ⋮\\ q_{xn}-C_{qx}& q_{yn}-C_{qy}& q_{zn}-C_{qz}\end{array}\right)$ (6)

Next, the optimal rotation matrix R that turns P as close as possible to Q is calculated using Singular Value Decomposition [25] module as follows.

$H=Q^{\text{T}}P$ (7)

$U\Sigma V=SVD\left(H\right)$ (8)

$R=V{U}^{\text{T}}$ (9)

As a result, we get a 3 × 3 matrix R and then, we convert it to a 4 × 4 matrix:

$R=\left(\begin{array}{cccc}{R}_{00}& R_{10}& R_{20}& 0\\ R_{01}& R_{11}& R_{21}& 0\\ R_{02}& R_{12}& R_{22}& 0\\ 0& 0& 0& 0\end{array}\right)$ (10)

Finally, the transformation is determined:

$M=T_{q}R{T}_p$ (11)

where,

$T_p=\left(\begin{array}{cccc}1& 0& 0& -C_{px}\\ 0& 1& 0& -C_{py}\\ 0& 0& 1& -C_{pz}\\ 0& 0& 0& 1\end{array}\right)$ (12)

$T_q=\left(\begin{array}{cccc}1& 0& 0& C_{qx}\\ 0& 1& 0& C_{qy}\\ 0& 0& 1& C_{qz}\\ 0& 0& 0& 1\end{array}\right)$ (13)

Error for each GCP is estimated as a distance between the GCP and corresponding point transformed by the calculated M. Based on our assumption above we can simply georeference 3D Tiles by setting the transform property of the root tile of the given 3D Tiles with M.

4. System Demonstration

The suggested method in this paper has been adapted to the georeferencing module of Construkted Reality, which is the platform to stream and visualize massive photogrammetry and point cloud datasets with ease on the web. An online portal is available at https://construkted.com. The rendering and georeference module were developed with CesiumJS.

Once the photogrammetry-derived 3D mesh is uploaded, the backend of the Construkted Reality converts it to the 3D Tiles. The system detects whether or not the 3D Tiles are georeferenced and displays the 3D Tiles on the blank background if it does not have georeferencing information so that the user is notified that she/he needs to georeference 3D Tiles (Figure 3).

Then the user can start georeferencing on the “Multiple GCP Editor” panel (Figure 4), where it is possible to add necessary GCPs and input the latitude, longitude, and altitude of each GCP.

Next, by simply mouse clicking, users can specify points corresponding to all GCPs on the 3D Tiles (Figure 5).

The mapping info between GCPs and corresponding points on the 3D Tiles is stored in the “gcpData” metadata which is represented as a JSON object like Figure 6.

Figure 3. Not georeferenced 3D Tiles.

Figure 4. Multiple GCP Editor panel.

Figure 5. Specifying corresponding point.

Figure 6. JSON structure of “gcpData” metadata.

The “gcps” property in the “gcpData” metadata defines an array of ground control points defined in the geographic coordinates so that each element has the meaning of latitude, longitude, and altitude. The “correspondingPoints” property defines an array of cartesian points corresponding to ground control points. The “altitudeOffset” property is the offset value of altitude which is added to all ground control points’ altitude in calculating georeferencing information. The surveyed altitude of GCP may be different than the altitude of the terrain on the virtual globe, so this is very useful in case users hope to correctly clamp 3D Tiles to the scene’s terrain, for example, Cesium World Terrain [26].

At least 3 GCPs must be specified because the Kabsch algorithm requires a minimum of 3 unique points for a unique solution.

Once the user finishes the input of GCPs and corresponding points and clicks the “Update Asset Location” button, the system calculates M with GCPs and corresponding points as in described section 3, and displays an error for each GCP, and the camera of the scene flies to the calculated georeferenced location of the 3D Tiles (Figure 7).

5. Experiments

In order to evaluate the feasibility of the suggested method, several experiments were carried out with 3D models generated from sample datasets of OpenDroneMap[27], Pix4DMapper [28], and RealityCapture [6] that have GCPs (Appendix A).

5.1. Experiment 1

The dataset used in the experiment is an example dataset, “sheffield_cross” for OpenDroneMap, which has 5 GCPs defined in the WGS 84/UTM zone 17N. This dataset was processed through WebODM Lighting [29]. Points corresponding to GCPs were visually identified on the 3D Tiles using the GCP file and images included in the dataset (Figure 8).

Table 1 shows calculated errors for all GCPs, where Latitude, Longitude, and

Figure 7. The result of georeferencing.

Figure 8. Georeferenced “sheffield_cross”.

Table 1. Calculated errors for GCPs of sheffield_cross.

Altitude columns represent the geographic coordinates of a ground control point, and x, y, and z columns are the cartesian coordinate of a corresponding point defined in the 3D Tiles’ local coordinate system.

5.2. Experiment 2

In this experiment, we used an example project, “quarry” for PIX4DMapper. GCPs are defined in EPSG:21781 [30] and corresponding points were visually identified using GCP overview images of the dataset and 3D GCP Markers of the project (Figure 9).

Figure 10 and Table 2 show the result of georeferencing of 3D Tiles for this project.

5.3. Experiment 3

We used a sample dataset called “Drone Imagery + Ground control points” for RealityCapture, where all GCPs are defined in EPSG:4258 [31]. All GCPs are marked on the texture of the generated 3D model like a small section of a checkerboard (Figure 11) which leaves very little ambiguity about where the “point” of the ground control point is so we could identify all GCPs clearly on the texture of the 3D Tiles.

Figure 9. 3D GCP marker and overview image about first GCP.

Figure 10. Georeferenced “quarry”.

Figure 12 and Table 3 show the result of georeferencing 3D Tiles. As can be known in Table 3, the errors on all GCPs have a margin of a few centimeters.

Table 2. Calculated errors for GCPs of quarry.

Figure 11. The shape of GCPs on the texture of the 3D model.

Figure 12. The result of georeferencing.

Table 3. Calculated errors for all GCPs.

6. Conclusion

From experiments in this paper, it can be concluded that the Kabsch algorithm can be applied to georeferencing 3D Tiles using GCPs. Georeferencing errors were inevitably affected by the accuracy of “clicking on the 3D Tiles” for specifying points corresponding to GCPs. In experiment3 error was very small enough to accept the georeferencing result because it was easy to recognize GCP on the 3D Tiles. In order to clamp 3D Tiles to Cesium World Terrain, different values of “altitudeOffset” were used for the reason that Cesium World Terrain does not provide elevation relative to the WGS84 ellipsoid.

Data Availability Statement

Datasets used in this study are available at the following links.

https://github.com/pierotofy/drone_dataset_sheffield_cross/

https://support.pix4d.com/hc/en-us/articles/360000235126-Example-projects-real-photogrammetry-data#quarry

https://www.capturingreality.com/SampleDatasets

Appendix A

Results presented in this study can be found at the following links.

https://construkted.com/asset/a1fg3o0r69r/

https://construkted.com/asset/avajuri6r3g/

https://construkted.com/asset/avplbp844b3/

Conflicts of Interest

The authors declare no conflicts of interest regarding the publication of this paper.

References

[1]	Zhan, W., Chen, Y. and Chen, J. (2021) 3D Tiles-Based High-Efficiency Visualization Method for Complex BIM Models on the Web. ISPRS International Journal of Geo-Information, 10, Article No. 476. https://doi.org/10.3390/ijgi10070476
[2]	3D Tiles Format Specification. https://github.com/CesiumGS/3d-tiles/tree/main/specification#tile-format-specifications/
[3]	FME Desktop. https://www.safe.com/fme/fme-desktop/
[4]	ContextCapture Viewer. https://www.bentley.com/en/products/brands/contextcapture
[5]	Agisoft MetaShape. https://www.agisoft.com/
[6]	Reality Capture. https://www.capturingreality.com/
[7]	PIX4D (2019, December 4) Ground Control Points: Why Are They Important? https://www.pix4d.com/blog/why-ground-control-points-important
[8]	Wikipedia (n.d.) Kabsch Algorithm. https://en.m.wikipedia.org/wiki/Kabsch_algorithm
[9]	Cesium ion. https://cesium.com/platform/cesium-ion/
[10]	Schilling, A., Bolling, J. and Nagel, C. (2016) Using glTF for Streaming CityGML 3D City Models. Proceedings of the 21st International Conference on Web3D Technology, Anaheim, 22-24 July 2016, 109-116. https://doi.org/10.1145/2945292.2945312
[11]	CESIUM_RTC. https://github.com/KhronosGroup/glTF/blob/main/extensions/1.0/Vendor/CESIUM_RTC/README.md
[12]	Gan, L., Li, J. and Jing, N. (2017) Hybrid Organization and Visualization of the DSM Combined with 3D Building Model. Proceedings of the 2017 2nd International Conference on Image, Vision and Computing, Chengdu, 2-4 June 2017, 566-571. https://doi.org/10.1109/ICIVC.2017.7984619
[13]	Song, Z. and Li, J. (2018) A Dynamic Tiles Loading and Scheduling Strategy for Massive Oblique Photogrammetry Models. Proceedings of the 2018 3rd IEEE International Conference on Image, Vision and Computing, Chongqing, 27-29 June 2018, 648-652. https://doi.org/10.1109/ICIVC.2018.8492731
[14]	Chen, Y., Shooraj, E., Rajabifard, A. and Sabri, S. (2018) From IFC to 3D Tiles: An Integrated Open-Source Solution for Visualising BIMs on Cesium. ISPRS International Journal of Geo-Information, 7, Article No. 393. https://doi.org/10.3390/ijgi7100393
[15]	Visuri, H., Jokela, J., Mesterton, N., Latvala, P., Aarnio, T. (2019) Producing and Visualizing a Country-Wide 3D Data Repository in Finland. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-4/W15, 105-110. https://doi.org/10.5194/isprs-archives-XLII-4-W15-105-2019
[16]	Kolaric, S. and Shelden, D. (2019) DBL SmartCity: An Open-Source IoT Platform for Managing Large BIM and 3D Geo-Referenced Datasets. Proceedings of the 52nd Hawaii International Conference on System Sciences, Hawaii, 8 January 2019, 1965-1974. https://doi.org/10.24251/HICSS.2019.238
[17]	Xu, Z., Zhang, L., Li, H., Lin, Y. and Yin, S. (2020) Combining IFC and 3D Tiles to Create 3D Visualization for Building Information Modeling. Automation in Construction, 109, Article ID: 102995. https://doi.org/10.1016/j.autcon.2019.102995
[18]	Lu, M., Wang, X., Liu, X., Chen M, Bi, S., Zhang, Y. and Lao, T. (2020) Web-Based Real-Time Visualization of Large-Scale Weather Radar Data Using 3D Tiles. Transactions in GIS, 25, 25-43. https://doi.org/10.1111/tgis.12638
[19]	Mao, B., Ban, Y. and Laumert, B. (2020) Dynamic Online 3D Visualization Framework for Real-Time Energy Simulation Based on 3D Tiles. ISPRS International Journal of Geo-Information, 9, Article No. 166. https://doi.org/10.3390/ijgi9030166
[20]	Li, Y., Zhang, H. and Zhang, Q. (2021) A Framework for Interactive Online 3D Visualization of Electric Information. Journal of Physics: Conference Series, 1757, Article ID: 012170. https://doi.org/10.1088/1742-6596/1757/1/012170
[21]	Jaillot, V., Servigne, S. and Gesquière, G. (2020) Delivering Time-Evolving 3D City Models for Web Visualization. International Journal of Geographical Information Science, 34, 2030-2052. https://doi.org/10.1080/13658816.2020.1749637
[22]	Cesium (n.d.) Set Location for Data Uploaded to Cesium Ion. https://cesium.com/learn/3d-tiling/ion-tile-set-location/
[23]	EPSG:4978. https://spatialreference.org/ref/epsg/wgs-84-2/
[24]	3D Tiles Reference Card. https://github.com/CesiumGS/3d-tiles/blob/main/3d-tiles-reference-card.pdf
[25]	Wikipedia (n.d.) Singular Value Decomposition. https://en.wikipedia.org/wiki/Singular_value_decomposition
[26]	Cesium World Terrain. https://cesium.com/platform/cesium-ion/content/cesium-world-terrain/
[27]	OpenDroneMap. https://www.opendronemap.org/
[28]	PIX4Dmapper. https://www.pix4d.com/product/pix4dmapper-photogrammetry-software
[29]	WebODM Lightning. https://webodm.net/
[30]	EPSG:21781. https://spatialreference.org/ref/epsg/ch1903-lv03/
[31]	EPSG:4258. https://spatialreference.org/ref/epsg/etrs89/