RadarSAT SWB and RAMP Product Upgrade Delivery
Quality Assurance SSP and PP Validation Report
Draft
September 26, 2003
Table of Contents
- Conclusion
Introduction
Synthetic aperture radar processing requires complex software algorithms to produce quality images that are of scientific value. In fact, the software processing is an integral part of the instrument itself. Therefore, any changes to the software code may have substantial ramifications to the quality of the products and impact scientific investigations negatively. To prevent this any substantial changes to the code must be validated to assure that, in addition to accomplishing the intended purpose, there are no unforeseen collateral impacts.
The validation process involves comparing the products of the initial, or operational, system, with the products from an identical hardware system, or test string, with the software upgrade, to verify that there are no unforeseen issues. For calibrated products all the metrics are examined to verify there are no unexpected statistical differences, for uncalibrated products a visual inspection is used to determine that the upgraded products are at least as good as the original operational products.
[return to top]Validation Data Set and Methods
The operational code for the precision processor (PP) and ScanSAR processor (SSP) were originally developed at the NASA Jet Propulsion Lab, which spun off many of the original developers to the Raytheon Corporation, which then got a support contract with ASF to maintain the code. A validation is done whenever a change is made to the code to verify that the change will accomplish the desired effect and cause no collateral anomalies. Due to budget cuts the Raytheon support contract was to be terminated and the code frozen, with this being the last upgrade validation that would be done for either the PP or SSP.
The PP validation data set consists of RadarSAT Antarctic Mapping Project (RAMP) images for RadarSAT beam modes Standard 2, 5, 6, 7 and Extended High Incidence Angle 4 (ST2, 5, 6, 7, EH4). The SSP validation data set consists of RadarSAT images of ScanSAR Wide A, ScanSAR Wide B, ScanSAR Narrow A and ScanSAR Narrow B (SWA, SWB, SNA SNB). Calibrated products will include an ascending and descending pass over both the Amazon rainforest and the ASF Delta Junction corner reflector calibration array; uncalibrated products will be validated with a visual comparison of ascending and descending like images. Calibrated products include all PP products, but only the SWB beam mode from the SSP, SNA, SNB, and SWA are uncalibrated. All analysis information was collected into Excel spreadsheets for comparison with the output from the previous version processor. The specifications are taken from the AMM Calibration Report and RadarSAT SWB Calibration Report. The differences between metrics of the old and new processors were compared with the standard deviation of the new processor and verified to be well below 2 standard deviations.
Images were ordered from the operational string and compared with images ordered from an identical hardware configuration known as the test string. To facilitate the validation process, a sample of two ascending and descending passes over Delta Junction and the Amazon were taken from the set of standard beam modes, with only a complete validation of EH4 and SWB. If no problems were found, the code was allowed to go operational, however, if problems were discovered an extended data set was ordered and the developers at Raytheon task to locate the bug. Image quality problems were discovered in low signal to noise ratio (SNR) areas around Delta Junction.
Interestingly, no statistical anomalies were found, the problem arose from visually apparent scalloping, light/dark banding across the range of the image in regions of low SNR. This was discovered one week before the Raytheon contract ended and a couple months after the person that had been working on the SWB upgrade had left to work at another facility. Incredibly, on the last day of the support contract, a developer examined the differences between the original code and modified code and found a potential bug. Happily, this solved the problem.
A list of the desired validation data set follows, one image was not available due to an older tape media, but this set was deemed sufficient for validation.
Table 1. Validation Data Set.
| Datatake ID | Processor | Calibrated | Application | Asc/Desc | Received |
| R1_09835_ST2_287 | PP | Yes | Image Quality | Descending | Yes |
| R1_09878_ST4_287 | PP | Yes | Image Quality | Descending | Yes |
| R1_09840_ST5_163 | PP | Yes | Image Quality | Ascending | Yes |
| R1_09797_ST6_163 | PP | Yes | Image Quality | Ascending | Yes |
| R1_09923_ST2_880 | PP | Yes | Radiometric | Ascending | Yes |
| R1_09980_ST7_880 | PP | Yes | Radiometric | Ascending | Yes |
| R1_10131_ST7_469 | PP | Yes | Radiometric | Descending | Yes |
| R1_09945_ST2_467 | PP | Yes | Radiometric | Descending | No |
| R1_10054_EH4_165 | PP | Yes | Image Quality | Ascending | Yes |
| R1_10150_EH4_284 | PP | Yes | Image Quality | Descending | Yes |
| R1_09837_EH4_880 | PP | Yes | Radiometric | Ascending | Yes |
| R1_09931_EH4_469 | PP | Yes | Radiometric | Descending | Yes |
| R1_31994_SWA_298 | SSP | No | General | Descending | Yes |
| R1_26375_SWA_152 | SSP | No | General | Ascending | Yes |
| R1_05582_SWB_157 | SSP | Yes | Image Quality | Ascending | Yes |
| R1_25598_SWB_287 | SSP | Yes | Image Quality | Descending | Yes |
| R1_16776_SWB_465 | SSP | Yes | Radiometric | Descending | Yes |
| R1_13682_SWB_881 | SSP | Yes | Radiometric | Ascending | Yes |
| R1_27778_SNA_295 | SSP | No | General | Descending | Yes |
| R1_18174_SNA_295 | SSP | No | General | Descending | Yes |
| R1_22733_SNB_295 | SSP | No | General | Descending | Yes |
| R1_20550_SNB_295 | SSP | No | General | Descending | Yes |
| Extended data | |||||
| R1_05604_SWB_291 | SSP | Yes | Image Quality | Descending | Yes |
| R1_07319_SWB_291 | SSP | Yes | Image Quality | Descending | Yes |
| R1_15265_SWB_292 | SSP | Yes | Image Quality | Descending | Yes |
| R1_15929_SWB_158 | SSP | Yes | Image Quality | Ascending | Yes |
| R1_17001_SWB_161 | SSP | Yes | Image Quality | Ascending | Yes |
| R1_22611_SWB_288 | SSP | Yes | Image Quality | Descending | Yes |
Results
The image quality and geolocation results are listed for each beam mode along with the difference between the 3.4 and 3.9 versions of the software; the absolute and relative radiometric results are listed for just the 3.9 version.
[return to top]RadarSAT Precision Processor, PP RAMP Products
Extended High 4, EH4
Image Quality
10054_165
| Average | Res(ra) | Res(az) | PSLR(ra) | PSLR(az) | ISLR(ra) | ISLR(az) | Peak Pwr |
| Difference | 0.14 | 0.24 | 0.55 | 1.97 | 0.03 | 0.55 | 0.02 |
| STD. Dev | 0.46 | 1.51 | 6.22 | 6.66 | 2.90 | 3.46 | 6.14 |
| Average | 17.39 | 31.35 | -26.89 | -24.59 | -15.03 | -17.06 | 84.26 |
| Spec. | 18 | 38 | -20 | -20 | -13 | -13 |
10150_284
| Average | Res(ra) | Res(az) | PSLR(ra) | PSLR(az) | ISLR(ra) | ISLR(az) | Peak Pwr |
| Difference | -0.36 | -0.78 | 0.46 | -2.70 | 1.13 | 1.45 | 0.19 |
| STD. Dev | 10.05 | 19.68 | 15.91 | 15.59 | 9.05 | 10.04 | 46.97 |
| Average | 24.10 | 32.03 | -25.91 | -26.25 | -16.07 | -14.21 | 80.09 |
| Spec. | 18 | 38 | -20 | -20 | -13 | -13 |
Geolocation
10054_165
| Average | Range (m) | Azimuth | Magnitude |
| Difference | -0.64 | 0.81 | -1.05 |
| Std. Dev. | 9.78 | 17.25 | 16.39 |
| Average | 27.08 | -34.79 | 44.17 |
| Spec. | 500 |
10150_284
| Average | Range (m) | Azimuth | Magnitude |
| Difference | 0.14 | -5.12 | 5.11 |
| Std. Dev | 6.63 | 79.74 | 79.90 |
| Average | -211.78 | *-6483.51 | *6486.97 |
| Spec. | 500 |
* Both processors showed the same anomaly
Radiometry
9837_880
| Average | IncAg (deg) | LookAg (deg) | Sigma0 (db) | StDSig0 (db) | Gama0 (db) |
| Difference | 0.0E+00 | 0.0E+00 | -1.7E-03 | -1.8E-03 | |
| Std. Dev. | 9.0E-01 | 6.6E-01 | 6.5E-02 | 4.8E-02 | 1.4E-01 |
Specification is the range of Gamma values on the chart
9931_469
| Average | IncAg (deg) | LookAg (deg) | Sigma0 (db) | StDSig0 (db) | Gama0 (db) |
| Difference | 0.0E+00 | 0.0E+00 | -1.1E-03 | -8.6E-04 | -1.1E-03 |
| Std. Dev. | 0.0E+00 | 0.0E+00 | -3.1E-.03 | -9.6E-04 | -8.0E-04 |
Specification is the range of Gamma values on the chart
Image Quality
Datatake ID# 9835_287
| Average | Res(ra) | Res(az) | PSLR(ra) | PSLR(az) | ISLR(ra) | ISLR(az) | Peak Pwr |
| Difference | -0.11 | -1.39 | 0.05 | -0.54 | -0.18 | 0.11 | 0.1 |
| Std. Dev. | 0.45 | 1.04 | 3.52 | 5.49 | 3.56 | 3.17 | 3.9 |
| Average | 10.94 | 31.60 | -23.20 | -22.51 | -13.68 | -12.47 | 79.2 |
| Spec. | 12 | 38 | -20 | -20 | -13 | -13 |
Geolocation
| Average | Range(m) | Azimuth | Magnitude |
| Difference | -0.26 | -0.52 | 0.57 |
| Std. Dev. | 4.67 | 21.94 | 18.21 |
| Average | -72.98 | -95.05 | 120.46 |
| Spec. | 500 |
Radiometry
Datatake ID #9923_880
| Average | IncAg(deg) | LookAg(deg) | Sigma0(db) | StDSig0(db) | Gama0(db) |
| Difference | 0 | 0 | -1.4E-04 | -1.4E-04 | -1.4E-04 |
| Spec. | 2.17 | 1.88 | 0.08 | 0.03 | 0.04 |
Standard Beam 5, ST5
Standard 5
Image QualityDatatake ID#9840_163
| Average | Res(ra) | Res(az) | PSLR(ra) | PSLR(az) | ISLR(ra) | ISLR(az) | Peak Pwr |
| Difference | -0.44 | 0.16 | -0.71 | -1.87 | 0.10 | 0.11 | -0.52 |
| Std. Dev. | 0.73 | 2.04 | 5.02 | 5.08 | 3.30 | 4.13 | 4.54 |
| Average | 16.83 | 32.45 | -21.81 | -23.15 | -10.67 | -11.50 | 78.63 |
| Spec. | 18 | 38 | -20 | -20 | -13 | -13 |
Geolocation
| Average | Range(m) | Azimuth | Magnitude |
| Difference | -6.63 | -3.16 | 5.09 | Std. Dev. | 24.21 | 15.52 | 20.48 |
| Average | 6.68 | -1.15 | 20.55 |
| Spec. | 500 |
Specification is the range of Gamma values on the chart
Standard Beam 6, ST6
Standard 6
Image Quality
Datatake ID# 9797_163
| Average | Res(ra) | Res(az) | PSLR(ra) | PSLR(az) | ISLR(ra) | ISLR(az) | Peak Pwr |
| Difference | 0.00 | 0.00 | 0.17 | 1.51 | 0.16 | 0.05 | 0.01 |
| Std. Dev. | 0.46 | 2.11 | 6.16 | 5.21 | 3.33 | 4.03 | 6.39 |
| Average | 17.54 | 32.19 | -21.62 | -20.77 | -11.60 | -12.60 | 76.25 |
| Spec. | 18 | 38 | -20 | -20 | -13 | -13 |
Geolocation
| Average | Range(m) | Azimuth | Magnitude |
| Difference | 0.23 | -0.22 | 0.47 |
| Std. Dev. | 20.96 | 18.48 | 21.85 |
| Average | 7.49 | -18.25 | 25.74 |
| Spec. | 500 |
Standard Beam 7, ST7
Radiometry10131_469
| Average | IncAg(deg) | LookAg(deg) | Sigma0(db) | StDSig0(db) | Gama0(db) |
| Difference | 10.27 | 8.30 | -0.84 | 0.01 | -0.09 |
| Std. Dev. | 4.61 | 3.56 | 0.37 | 0.04 | 0.08 |
Specification is the range of Gamma values on the chart
RadarSAT ScanSAR Processor, SSP
ScanSAR Wide B, SWB
Image Quality 5582_157 and 25598_287| Average | Res(ra) | Res(az) | PSLR(ra) | PSLR(az) | ISLR(ra) | ISLR(az) | Peak Pwr |
| Difference | -1.63 | -1.11 | -0.07 | 0.24 | 0.10 | 0.16 | -0.14 |
| Std. Dev. | 11.47 | 2.44 | 2.41 | 3.03 | 2.36 | 2.76 | 2.27 |
| Average | 98.75 | 63.75 | -15.43 | -16.82 | -8.08 | -8.33 | 23.68 |
No Image Quality Specifications
Geolocation
5582_157 and 25598_287| Average | Range(m) | Azimuth | Magnitude |
| Difference | 4.30 | -3.91 | 5.16 |
| Std. Dev. | 195.27 | 130.01 | 152.96 |
| Average | -112.10 | 32.45 | 216.17 |
| Spec. | 500.00 |
Radiometry
16776_465| Average | IncAg(deg) | LookAg(deg) | Sigma0(db) | StDSig0(db) | Gama0(db) |
| Difference | 0.10 | 0.09 | 0.04 | 0.28 | 0.04 |
| Std. Dev. | 7.53 | 6.40 | 0.52 | 1.46 | 0.21 |
Specification is the range of Gamma values on the chart
Scalloping Problem
Corrected Processor:
ScanSAR Wide A, SWA
ScanSAR Narrow A, SNB
ScanSAR Narrow B, SNB
Conclusion
The 3.9 versions of the PP and SSP processor is valid.