Laboratory for Smart City and Spatial Big Data Analytics

AI-based Urban Object Monitoring

AI (artificial intelligence, also known as machine intelligence), inspired by biological systems, is capable of correctly interpreting data, learning knowledge from data, and achieving specific goals and tasks through its flexible adaptability. Focusing on the application of AI technology in urban object monitoring, the AI technology we studied includes the latest deep learning methods, new network structures, and more intelligent machine learning algorithms. Besides, we have developed image analysis software (ImageAnalytics) integrating a variety of deep learning frameworks and models.

ImageAnalytics

RS Image Classification

RS image classification technology refers to the process of analyzing the spectral information and spatial information of various types of ground features in RS images, and using different rules to divide different categories to obtain the correspondence between RS images and ground objects. The real-time accurate and efficient image classification algorithm is of great significance for the national surveying and mapping departments to carry out land resources surveys and normalized land use monitoring and evaluation. In recent years, with the rapid development of artificial intelligence, especially the emergence of deep learning technology, it has provided strong support for the application of remote sensing image classification methods. Our image analysis software implements a variety of remote sensing image classification tools, including a series of object-oriented RS image classification based on machine learning and the latest deep learning-based RS image classification methods.

[1] Ding, H., Xu, L., Wu, Y. & Shi, W. (2020). Classification of hyperspectral images by deep learning of spectral-spatial features. Arabian Journal of Geosciences, 13(12), 1-14.

[2] Miao, Z., Xiao, Y., Shi, W., et al. (2019), Integration of Satellite Images and Open Data for Impervious Surface Classification, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing. DOI: 10.1109/JSTARS.2019.2903585.

[3] Jia, T.*, Li, Y., Shi, W. and Zhu, L. (2019) Deriving a Forest Cover Map in Kyrgyzstan Using a Hybrid Fusion Strategy, Remote Sensing, 11(19), 2325. DOI: 10.3390/rs11192325.

[4] Liang, P., Shi, W.* and Zhang, X. (2018) Remote sensing image classification based on Stacked Denoising Autoencoder, Remote Sensing,10(1), 16 DOI: 10.3390/rs10010016.

[5] Zhang, H.*, Bruzzone, L.*, Shi, W.*, Hao, M. and Wang, Y. (2018) Enhanced Spatially Constrained Remotely Sensed Imagery Classification Using a Fuzzy Local Double Neighborhood Information C-Means Clustering Algorithm, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 11(8), 2896-2910 DOI: 10.1109/JSTARS.2018.2846603.

[6] Zhang, H., Shi, W.*, Hao, M., Li, Z., and Wang, Y. (2018) An adaptive spatially constrained fuzzy c-means algorithm for multispectral remotely sensed imagery clustering, International Journal of Remote Sensing, 39(8), 2207-2237 DOI: 10.1080/01431161.2017.1420934.

[7] Ruiz, I., and Shi, W.* (2018) A Random Forests classification method for urban land-use mapping integrating spatial metrics and texture analysis, International Journal of Remote Sensing, 39(4), 1175-1198 DOI: 10.1080/01431161.2017.1395968.

[8] Lv, Z.*, Shi, W.*, Benediktsson, J., and Gao, L., (2018) A modified mean filter for improving the classification performance of very high-resolution remote-sensing imagery, International Journal of Remote Sensing, 39(3), 770-785. (IF=1.724) DOI: 10.1080/01431161.2017.1390275.

[9] Zhang, H.*, Wang, Q.*, Shi, W. and Hao, M. (2017) A novel adaptive fuzzy local information C-means clustering algorithm for remotely sensed imagery classification, IEEE Transactions on Geoscience and Remote Sensing, 55(9), 5057-5068.

[10] Lv, Z., Shi, W.*, Benediktsson, J.A. and Ning, X. (2016) Novel Object-Based Filter for Improving Land-Cover Classification of Aerial Imagery with Very High Spatial Resolution, Remote Sensing, 8(12), 1023.

[11] Miao, Z., & Shi, W.* (2016). A new methodology for spectral-spatial classification of hyperspectral images, Journal of Sensors, vol. 2016, Article ID 104875. DOI: 10.1155/2016/1538973.

RS Object Extraction

The RS object extraction in our study mainly focuses on man-made objects, such as building extraction and road extraction. Due to the enormous changes in shape and the complexity of the surrounding environment, the detection of buildings is challenging. Airborne LiDAR can effectively distinguish between ground points and non-ground points through elevation information, which improves the automation of building detection, but the plane precision is not satisfying, and it is no easy to distinguish buildings from trees; high-resolution images have rich spectrum, texture and geometric information which can provide fine boundaries but simultaneously bring problems like spectral ambiguity, occlusion, shadows, etc. As can be seen from above, LiDAR and optical images are naturally complementary, so it is feasible to detect buildings automatically combining LiDAR and images. We propose an adaptive iterative segmentation automatic building extraction algorithm based on the fusion of LiDAR data and high spatial resolution images. The algorithm does not require training samples or depends on the segmentation parameters, and there is no need to set a specific threshold. The test accuracy on the ISPRS Vaihingen dataset reaches 85.5%.

The yellow area represents true positives, the red area represents false positives, and the green area represents false negatives

Most traditional RS image-based building detection methods extract objects by clustering, segmentation, edge detection, machine learning and other techniques. Those methods utilize image properties with manually designed features thus encountered many insurmountable difficulties in real-world application because of the complicated background. To address these problems, we introduced AI technology to automatically and accurately extract building areas in RS images through deep learning models.

Building detection based on Faster RCNN

Building detection based on Mask RCNN

Building detection based on residual multi-constrained FCN

Road network extraction

[1] Chen, S., Shi, W., Zhou, M., Zhang, M. & Chen, P. (2020). Automatic Building Extraction via Adaptive Iterative Segmentation With LiDAR Data and High Spatial Resolution Imagery Fusion. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 13, 2081-2095.

[2] Miao, Z., Gao, L., He, Y.*, Wu, L., Shi, W., Samat, A., Liu, S. and Li, J. (2019) Use of colour transformation and the geodesic method for road centreline extraction from VHR satellite images, International Journal of Remote Sensing. DOI: 10.1080/01431161.2018.1558374.

[3] Gao, L., Shi, W.*, Miao, Z. and Lv, Z. (2018) Method based on edge constraint and fast marching for road centerline extraction from very high-resolution remote sensing images. Remote Sensing, 10(6), Article number 900 DOI: 10.3390/rs10060900.

[4] Li, Z., and Shi, W.* (2016). Partial differential equation-based object extraction from remote sensing imagery, Journal of Infrared and Millimeter Waves, 35(3), 257-262. DOI: 10.11972/j.issn.1001-9014.2016.03.001.

[5] Miao, Z., Shi, W.*, Samat, A., and Lisini, G., and Gamba, P. (2016). Information Fusion for Urban Road Extraction From VHR Optical Satellite Images, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 9(5), 1817-1829. DOI: 10.1109/JSTARS.2015.2498663.

[6] Miao, Z., Shi, W.*, Gamba, P., & Li, Z. (2015) An Object-Based Method for Road Network Extraction in VHR Satellite Images, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 8(10), 4853-4862. DOI: 10.1109/JSTARS.2015.2443552.

[7] Li, Z., Shi, W.*, Wang, Q., & Miao, Z. (2015). Extracting man-made objects from high spatial resolution remote sensing images via fast level set evolutions, IEEE Transactions on Geoscience and Remote Sensing, 53(2), 883–899. DOI: 10.1109/TGRS.2014.2330341.

[8] Miao, Z., Wang, B., Shi, W.*, Wu, H., & Wan, Y. (2015). Use of GMM and SCMS for automatic extraction of road centerline from the classified image, Journal of Sensors, vol. 2015, Article ID 784504. DOI: 10.1155/2015/784504.

Landslide Mapping

Landslides are one of the most severe natural hazards in the world. The loss of human lives and properties and environmental damage caused by landslides are becoming more and more dangerous. Regional landslide mapping is currently mainly carried out through manual visual interpretation of RS images. There process is time-consuming, labor-intensive, and low extraction accuracy, which make it challenging to meet the needs of post-disaster emergency investigation and disaster assessment. Since AI has robust analysis and expression capabilities in RS image processing, we adopt a multi-temporal optical RS image landslide detection method based on a convolutional neural network. By combining with the terrain information provided by the digital elevation model (DEM), we can effectively identify the location, scope, trajectory, elevation and other properties of the landslide, to accurately identify the spatial distribution and attributes of the landslide, which can provide technical support for a post-disaster emergency investigation.

[1] Shi, W., Zhang, M., Ke, H., Fang, X., Zhan, Z. & Chen, S. (2020). Landslide recognition by deep convolutional neural network and change detection. IEEE Transactions on Geoscience and Remote Sensing. DOI: 10.1109/TGRS. 2020.3015826.

[2] Zhang, M., Shi, W., Chen, S., Zhan, Z. & Shi, Z. (2020). Deep multiple instance learning for landslide mapping. IEEE Geoscience Remote Sensing Letter. DOI: 10.1109/LGRS.2020.3007183.

[3] Shi, W., Deng, S., * and Xu, W. (2018) Extraction of multiscale landslide morphological features based on local Gi* using airborne LiDAR-derived DEM, Geomorphology, 303, 229-242. DOI:10.1016/j.geomorph.2017.12.005.

[4] Lv, Z.*, Shi, W.*, Zhang, X., and Benediktsson, J. (2018) Landslide Inventory Mapping From Bitemporal High-Resolution Remote Sensing Images Using Change Detection and Multiscale Segmentation, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 11(5), 1520. (IF=2.913) DOI: 10.1109/JSTARS.2018.2803784.

[5] Li, Z.*, Shi, W.*, Myint, S.W., Lu, P., & Wang, Q. (2016). Semi-automated landslide inventory mapping from bitemporal aerial photographs using change detection and level set method, Remote Sensing of Environment, 175, 215-230. DOI:10.1016/j.rse.2016.01.003.

[6] Li, Z.*, Shi, W., Lu, P., Yan, L., Wang, Q. and Miao, Z. (2016). Landslide mapping from aerial photographs using change detection-based Markov random field, Remote Sensing of Environment, 187, 76-90.

[7] Deng, S.*, & Shi, W. (2014). Semi-automatic approach for identifying locations of shallow debris slides/flows based on lidar-derived morphological features. International Journal of Remote Sensing, 35(10), 3741-3763. DOI: 10.1080/01431161.2014.915438.

RS Change Detection

RS image can directly or indirectly provide land cover and land use information, which is an essential method for obtaining information on the temporal changes of the earth’s surface. RS image change detection techniques have important applications in land cover and land use, geological disasters, urban change, environment, wetlands, agriculture, forestry and other fields. As an emerging intelligent data processing technology, deep learning technology can quickly and accurately detect changes from mass multi-temporal remote sensing data, which has become a significant research interest for us. Deep learning-based change detection software tools can promptly extract change information from multi-temporal RS images, providing technical support for smart city object monitoring and data update.

[1] Shi, W., Zhang, M., Zhang, R., Chen, S. & Zhan, Z. (2020). Change detection based on artificial intelligence: state-of-the-art and challenges. Remote Sensing, 12(10), pp. 1688.

[2] Zhang, M. & Shi, W. (2020). A feature difference convolutional neural network-based change detection method. IEEE Transactions on Geoscience and Remote Sensing. DOI: 10.1109/TGRS.2020.2981051.

[3] Zhang X, Shi W.*, Lv Z, et al. (2019). Land Cover Change Detection from High-Resolution Remote Sensing Imagery Using Multitemporal Deep Feature Collaborative Learning and a Semi-supervised Chan–Vese Model. Remote Sensing, 11(23): 2787.

[4] Choudhary, K.*, Shi, W., Boori, M-S. and Corgne, S. (2019) Agriculture Phenology Monitoring Using NDVI Time Series Based on Remote Sensing Satellites: A Case Study of Guangdong, China, Optical Memory and Neural Networks, 28(3), 204–214. DOI: 10.3103/S1060992X19030093.

[5] Hasan, S., Shi, W.*, Zhu, X. and Abbas S. (2019) Monitoring of Land Use/Land Cover and Socioeconomic Changes in South China over the Last Three Decades Using Landsat and Nighttime Light Data, Remote Sensing, 11(14), 1658. DOI: 10.3390/rs11141658.

[6] Hao, M., Shi, W.*, Ye, Y., Zhang, H., and Deng, K. (2019) A novel change detection approach for VHR remote sensing images by integrating multi-scale features, International Journal of Remote Sensing, 40(13), 4910-4933 DOI: 10.1080/01431161.2019.1577576.

[7] Shao, P., Shi, W.*, and Hao, M. (2018) Indicator-Kriging-Integrated Evidence Theory for Unsupervised Change Detection in Remotely Sensed Imagery, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 11(12), 4649-4663 DOI: 10.1109/JSTARS.2018.2878759.

[8] Cai, L., Shi, W.*, Hao, M., Zhang, H. and Gao, L. (2018) A Multi-Feature Fusion-Based Change Detection Method for Remote Sensing Images, Journal of the Indian Society of Remote Sensing, 46(12), 2015-2022 DOI: 10.1007/s12524-018-0864-1.

[9] He, P., Shi, W.*, and Zhang, H. (2018) Adaptive superpixel based Markov random field model for unsupervised change detection using remotely sensed images, Remote Sensing Letters, 9(8), 724-732. DOI: 10.1080/2150704X.2018.1470698.

[10] Lv, Z.*, Shi, W.*, Benediktsson, J., and Zhou, X. (2017) Semi-Automatic System for Land Cover Change Detection Using Bi-Temporal Remote Sensing Images. Remote Sensing, 9(11), 1112.

[11] Li Z.*, Shi W., Zhang H.*, and Hao M. (2017). Change Detection Based on Gabor Wavelet Features for Very High Resolution Remote Sensing Images. IEEE Geoscience and Remote Sensing Letters, 14(5): 783-787.

[12] Zhang, X., Shi, W.*, Liang, P., and Hao, M. (2017) Level Set Evolution with Local Uncertainty Constraints for Unsupervised Change Detection, Remote Sensing Letters, 8(8), 811-20.

[13] Zhang, X., Shi, W.*, Hao, M., Shao, P., and Lyu, X. (2017) Level set incorporated with an improved MRF model for unsupervised change detection for satellite images, European Journal of Remote Sensing, 50(1), 202-10.

[14] Cai, L., Shi, W.*, Zhang, H., and Hao, M. (2016). Object-oriented change detection method based on adaptive multi-method combination for remote-sensing images, International Journal of Remote Sensing, 37(22): 5457-5471.

[15] Hao, M., Shi, W.*, Zhang, H., Wang, Q., and Deng, K. (2016). A Scale-Driven Change Detection Method Incorporating Uncertainty Analysis for Remote Sensing Images. Remote Sensing, 8(9), 745, DOI: 10.3390/rs8090745.

[16] Hao, M., Shi, W.*, Deng, K., and Feng, Q. (2016). Superpixel-based active contour model for unsupervised change detection from satellite images, International Journal of Remote Sensing, 37(18), 4276-4295. DOI: 10.1080/01431161.2016.1210838.

[17] Shao, P., Shi, W.*, He, P., Hao, M., and Zhang, X. (2016). Novel Approach to Unsupervised Change Detection Based on a Robust Semi-Supervised FCM Clustering Algorithm, Remote Sensing, 8(3), DOI: 10.3390/rs8030264.

[18] Shi, W., Shao, P.*, Hao, M., He, P., and Wang, J. (2016). Fuzzy topology-based method for unsupervised change detection, Remote Sensing Letters, 7(1), 81-90. DOI: 10.1080/2150704X.2015.1109155.

[19] Hao, M., Shi, W.*, Deng, K., Zhang, H., and He, P. (2016). An Object-Based Change Detection Approach Using Uncertainty Analysis for VHR Images, Journal of Sensors, vol. 2016, Article ID 9078364, 17 pages. DOI: 10.1155/2016/9078364.

[20] Hao, M., Shi, W.*, Deng, K., & Zhang, H. (2015). Fusion-based approach to change detection to reduce the effect of the trade-off parameter in the active contour model, Remote Sensing Letters, 6(1), 39-48. DOI: 10.1080/2150704X.2014.1001078.

[21] Wang, Q., Atkinson, P. M., & Shi, W.* (2015). Fast subpixel mapping algorithms for subpixel resolution change detection, IEEE Transactions on Geoscience and Remote Sensing, 53(4), 1692–1706. DOI: 10.1109/TGRS.2014.2346535.

[22] Wang, Q., Shi, W.*, Atkinson, P. M., & Li, Z. (2015). Land cover change detection at subpixel resolution with a Hopfield neural network, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 8(3), 1339- 1352. DOI: 10.1109/JSTARS.2014.2355832.

[23] He, P., Shi, W.*, Miao, Z., Zhang, H., & Cai, L. (2015). Advanced Markov random field model based on local uncertainty for unsupervised change detection, Remote Sensing Letters, 6(9), 667-676. DOI: 10.1080/2150704X.2015.1054045.

[24] Hao, M., Shi, W.*, Deng, K., & Zhang, H. (2014). A contrast-sensitive Potts model custom-designed for change detection, European Journal of Remote Sensing, 47, 643-654. DOI: 10.5721/EuJRS20144736.

[25] He, P., Shi, W.*, Hua, Z., & Hao, M. (2014). A novel dynamic threshold method for unsupervised change detection from remotely sensed images, Remote Sensing Letters, 5(4), 396-403. DOI: 10.1080/2150704X.2014.912766.

Object Detection and Tracking

Object detection and tracking is one of the most popular directions in remote sensing (RS) image processing and even computer vision. It is widely used in RS image processing, robot navigation, industrial inspection, aerospace and other fields. Since it is of great practical significance to reduce the consumption of human capital through computer vision, which is an important branch of image processing, computer vision and a basic algorithm in the field of human detection, has become a research hotspot in recent years. We have integrated several popular deep learning-based object detection and tracking algorithms such as SSD, Faster RCNN, YOLO2/3 into our image analysis software. The algorithms are trained with data we collected and can be used for detection of any object of interest for data such as UAV RS images, which can detect and track specific objects in images or videos and provides a practical tool for the rapid monitoring of urban objects.

In addition, human detection has always been a research hotspot in the field of computer vision, and it has been widely used in public safety, autonomous driving, and rescue after disasters. Compared with visible spectral images that are commonly used in human detection, thermal infrared images have unparalleled advantages, such as strong environmental adaptability and better protection of privacy. We propose a human detection method based on thermal infrared images by combining the deep learning detection algorithm Faster RCNN with more advanced Inception V2, Resnet 50 and Resnet 101. The models are trained with labeled OTCBVS benchmark dataset. Experimental results demonstrate that the detection accuracy of these models is maintained at about 90%. It provides an effective method for flow information collection based on thermal infrared images.

[1] Wang, S. (2020). Human detection from thermal images by deep learning based on improved faster R-CNN. Thesis (Msc). The Hong Kong Polytechnic University.

• BACK