Deep-Learning-based Hyperspectral Pansharpening on large scale PRISMA dataset

Simone Zini a,b,*
Mirko Paolo Barbato a
Flavio Piccoli a
Paolo Napoletano a,b

a Imaging and Vision Laboratory, Department of Informatics, Systems and Communication, University of Milano-Bicocca
b Istituto Nazionale di Fisica Nucleare (INFN) - Milano

ArXiv Pre-Print
*Corresponding Author

[Paper ArXiv]

Abstract

In this work, we assess several Deep-Learning strategies for hyperspectral pansharpening. First, we present a new dataset with a greater extent than any other in the state of the art. This dataset, collected using the ASI PRISMA satellite, covers about 262200 km2, and its heterogeneity is granted by randomly sampling the Earth’s soil. Second, we adapted several state of the art approaches based on deep learning to fit PRISMA hyperspectral data and then assessed, quantitatively and qualitatively, the performance in this new scenario. The investigation has included two settings: Reduced Resolution (RR) to evaluate the techniques in a supervised environment and Full Resolution (FR) for a real-world evaluation. The main purpose is the evaluation of the reconstruction fidelity of the considered methods. In both scenarios, for the sake of completeness, we also included machine-learning-free approaches. From this extensive analysis has emerged that data-driven neural network methods outperform machine-learning-free approaches and adapt better to the task of hyperspectral pansharpening, both in RR and FR protocols.


Proposed PRISMA based Dataset

The main contribution of this work can be summarized in two main points:


Map of the patches acquired using the PRISMA satellite. On average, every patch covers about 1380 km2 of soil


Quantitative Comparison

Results of the methods for the Reduced Resolution (RR) protocol. Alongside the results are reported the dimensions (millions of parameters) of each model.

Method # of parameters (M) ERGAS ↓ SAM ↓ SCC ↑ Q2n ↑
PCA (Chavez et al., 1991) - 8.9545 4.8613 0.6414 0.6071
GSA (Aiazzi et al., 2007) - 7.9682 4.3499 0.6642 0.6686
HySure (Simoes et al., 2014) - 8.3699 4.8709 0.5832 0.561
PNN (Masi et al., 2016) 0.08 12.884 3.8465 0.8237 0.6702
PanNet (Yang et al., 2017) 0.19 6.7062 2.7951 0.8705 0.7659
MSDCNN (Yuan et al., 2018) 0.19 9.9105 3.0733 0.8727 0.7537
TFNet (Liu et al., 2020) 2.36 6.409 2.4644 0.8875 0.7897
SRPPNN (Cai and Huang, 2020) 1.83 6.4702 2.3823 0.889 0.7708
DIPNet(Xie et al., 2021) 2.95 5.183 2.3715 0.8721 0.7929


Results of the methods for the Full Resolution (FR) protocol. Alongside the results are reported the dimensions (millions of parameters) of each model.

Method # of parameters (M) Dkλ ↓ D∗s ↓ QNR ↑
PCA (Chavez et al., 1991) - 0.9411 1.5277 0.0558
GSA (Aiazzi et al., 2007) - 0.382 0.0016 0.617
HySure (Simoes et al., 2014) - 0.4151 0.0009 0.5843
PNN (Masi et al., 2016) 0.08 0.3801 0.0101 0.6136
PanNet (Yang et al., 2017) 0.19 0.3507 0.0203 0.636
MSDCNN (Yuan et al., 2018) 0.19 0.3915 0.0068 0.6044
TFNet (Liu et al., 2020) 2.36 0.3552 0.0066 0.6405
SRPPNN (Cai and Huang, 2020) 1.83 0.3948 0.0139 0.5965
DIPNet(Xie et al., 2021) 2.95 0.3681 0.0348 0.6098

Qualitative Comparison
Difference between spectral signatures of the fused images with respect to the input image. The difference is evaluated as the average of the differences for each pixel of the five images reported in the row below the graph. The graph on the left shows the average spectral difference, while the graph on the right shows the difference normalized for each band.

spectral signatures obtained in six different areas, labeled as Forest, Urban, Agriculture or Water areas. For each area are reported the spectral signatures of the input bands and the ones obtained by each pansharpening method. The area used to extract the signatures are the one in the red box highlighted in each image. The images thumbnails are in true colors (641 nm, 563 nm, 478 nm).



Methods results (click on image to open at full resolution)

0_Panchromatic

Panchromatic
1_Input

Input
2_PNN

PNN
3_SRPPNN

SRPPNN
4_PanNet

PanNet
5_TFNet

TFNet
6_DIPNet

DIPNet



0_Panchromatic

Panchromatic
1_Input

Input
2_PNN

PNN
3_SRPPNN

SRPPNN
4_PanNet

PanNet
5_TFNet

TFNet
6_DIPNet

DIPNet



0_Panchromatic

Panchromatic
1_Input

Input
2_PNN

PNN
3_SRPPNN

SRPPNN
4_PanNet

PanNet
5_TFNet

TFNet
6_DIPNet

DIPNet



0_Panchromatic

Panchromatic
1_Input

Input
2_PNN

PNN
3_SRPPNN

SRPPNN
4_PanNet

PanNet
5_TFNet

TFNet
6_DIPNet

DIPNet



0_Panchromatic

Panchromatic
1_Input

Input
3_PNN

PNN
3_SRPPNN

SRPPNN
4_PanNet

PanNet
5_TFNet

TFNet
6_DIPNet

DIPNet


BibTeX

If you use this work please cite us: 
      

        
          @misc{zini2023deep,
                title={Deep Learning Hyperspectral Pansharpening on large scale PRISMA dataset}, 
                author={Simone Zini and Mirko Paolo Barbato and Flavio Piccoli and Paolo Napoletano},
                year={2023},
                eprint={2307.11666},
                archivePrefix={arXiv},
                primaryClass={eess.IV}
          }