Data Formats in Depth

1. CEOS (Committee on Earth Observation Satellites)

A standard format published in 1988, used for radar data and originally expected to be used with tape media. The format does not specify a naming convention.

How to Open

CEOS data may be viewed using ASF’s MapReady. ESA’s Sentinel-1 Toolbox (S1TBX) is also available.

For users who do not wish to use MapReady or S1TBX, some useful information:

  • CEOS is a wrapper — image data are wrapped in an image file descriptor and record headers which must be discarded in order to work with the data.
  • Each file starts with a file descriptor record, which provides details on the format used to store the data. In addition, there is one record header for each line of data. Each line has a 12-byte record header (which contains the record count in the file, record type identification, and record length in bytes); data values, and possibly fill values
  • Note that the units for state vectors may be either meters or kilometers (not a worry if using MapReady).

CEOS File Format and Content Information

Data format information for ASF’s CEOS L1 and L1.5 (image) products:

File Type Description ERS-1, ERS-2, JERS-1 and RSAT-1 File Extension (L1) ALOS PALSAR File Prefix (L1.5)
SAR VOLUME file Stores the volume-management and file-management information. N/A VOL-
SAR LEADER file Contains detailed metadata. Useful records include the Dataset Summary, Platform Position, and the Facility-Related records. Contents can be read in plain text if the .L.txt file is available. .L LED-
SAR DATA file Contains image data .D IMG-
SAR TRAILER file .P contains basic metadata including orbit, beam mode, start/stop times and lat/long corners and center. Can be read with a text editor.
TRL contains a file descriptor, and for L1.1 and L1.5 data, low resolution image data.
.P TRL-

Product CEOS Naming Conventions

The CEOS format does not specify a naming convention, so naming is facility or agency-specific.

CEOS file format for RADARSAT-1, ERS-1, ERS-2, and JERS-1 Level 0 products.

Level 0 Processing Level

Examples:
R1_65186_ST3_L0_F287

E1_25499_STD_L0_F195

E2_84697_STD_L0_F293

J1_29564_STD_L0_F323

Options:

RADARSAT-1 Beam Mode: Extended High (EH3, 4, 6), Extending Low (EL1), Fine (FN1-5), Standard (ST1-7), Wide (WD1-3), ScanSAR — narrow (SNA, SNB), ScanSAR — wide (SWA, SWB).


CEOS file format for RADARSAT-1, ERS-1, ERS-2, and JERS-1 for Level 1 products.

Level 1 Processing Level

Examples:
R1_65186_ST3_F287

E1_25539_STD_F281

E2_82775_STD_F315

J1_28022_STD_F345

Options:

RADARSAT-1 Beam Mode: Extended High (EH3, 4, 6), Extending Low (EL1), Fine (FN1-5), Standard (ST1-7), Wide (WD1-3), ScanSAR — narrow (SNA, SNB), ScanSAR — wide (SWA, SWB).

CEOS file format for ALOS PALSAR granules.

Granule Name

Example:
ALPSRP273981290

Options:

Beam Mode: Wide Beam/ScanSAR (S); Fine single-pol, fine dual-pol, full-pol (P)


CEOS file format for ALOS PALSAR Level 1 products.

Level 1 Product Name

Example:
IMG-HH-ALPSRP273981290-H1.0__A

Options:

Beam Mode: Wide Beam/ScanSAR (S); Fine single-pol, fine dual-pol, full-pol (P)

Observation Mode: Fine mode (H); Scan SAR mode (W); Direct downlink mode (D); Polarimetry mode (P); Calibration mode (C)

Map Projection: Universal Transverse Mercator/UTM (U); Polar Stereo/PS (P); Mercator (M); Lambert conformal conic/LCC (L); “Not specified” (_ underscore)


CEOS file format for ALOS PALSAR Level 1.5 products.

Level 1.5 Product Name

Example:
IMG-HH-ALPSRP273981290-H1.5_UA

Options:

Beam Mode: Wide Beam/ScanSAR (S); Fine single-pol, fine dual-pol (P)

Observation Mode: Fine mode (H); ScanSAR mode (W); Direct downlink mode (D); Polarimetry mode (P); Calibration mode (C)

Map Projection: Universal Transverse Mercator/UTM (U); Polar Stereo/PS (P); Mercator (M); Lambert conformal conic/LCC (L); “Not specified” (_ underscore)

2. GeoTIFF (Tagged Information File Format)

Format for handling images and data within a single raster file, by including header tags such as size, definition, image-data arrangement, and applied image compression.

The GIS-friendly GeoTIFF format is an extension of TIFF that includes georeferencing or geocoding information embedded within a TIFF file (such as latitude, longitude, map projection, coordinate systems, ellipsoids, and datums) so an image can be positioned correctly on maps of Earth. It is a public domain metadata standard.

A georeferenced image is oriented in parallel with orbit direction:
Descending: The scene start is at the top of the image, and scene end is at the bottom.
Ascending: The scene end is at the top of the image, and the scene beginning is at the bottom
A geocoded image is projected on a map oriented in a north-south direction:
Descending
Ascending

Product GeoTIFF Naming Conventions

Data Name

Sentinel-1 products of all processing levels are delivered in the Standard Archive Format for Europe (SAFE) format. The SAFE format packages the data as a file structure which contains other format structures, such as XML, TIF, HTML, KML, PNG, and XSD formats. Measurement data are in GeoTIFF format for Level-1 products.

— Adapted from the European Space Agency

Measurement and Annotation Data Set Naming Convention.

GeoTIFF file format for ALOS PALSAR Radiometrically Terrain-Corrected products.

ALOS PALSAR Radiometrically Terrain Corrected (RTC) products are gamma naught, suitable for statistical analysis. For better viewing in GIS, convert to db or amplitude with QGIS or ArcGIS directions.

Granule Name

Example:
AP_26939_FBS_F3170_RT1

Options
Beam Mode: Wide Beam/ScanSAR (S); Fine, single-pol, fine dual-pol, full-pol (P)

Terrain Correction Resolution: high resolution (RT1); low resolution (RT2)

Naming Convention for Seasat products.

Granule Name

Example:
SS_01492_STD_F2829.tif

ALOS PALSAR RTC GeoTIFF File Extension and Description

File Extension Description Example
_HH.tif _HV.tif _VH.tif _VV.tif Terrain-corrected product stored in separate files for each available polarization in GeoTIFF format. AP_26939_PLR_F3170_RT1_HH.tif AP_26939_PLR_F3170_RT1_HV.tif AP_26939_PLR_F3170_RT1_VH.tif AP_26939_PLR_F3170_RT1_VV.tif
.iso.xml ISO-compliant metadata in XML format AP_26939_PLR_F3170_RT1.iso.xml
.inc_map.tif Incidence angle map in GeoTIFF format AP_26939_PLR_F3170_RT1.inc_map.tif
.ls_map.tif Layover/shadow mask in GeoTIFF format AP_26939_PLR_F3170_RT1.ls_map.tif
.dem.tif Digital elevation model used for terrain correction in GeoTIFF format AP_26939_PLR_F3170_RT1.dem.tif
.geo.jpg Browse image of the amplitude (including world and auxiliary file) in JPEG format AP_26939_PLR_F3170_RT1.geo.jpg
.kmz Browse image in Google Earth format AP_26939_PLR_F3170_RT1.kmz

3. SAFE (Standard Archive Format for Europe)

SAFE Product Folder Structure​

Sentinel data products use a Sentinel-specific variation of the SAFE format, an ESA folder structure containing data and information as follows:

The manifest.safe file is an XML file containing the mandatory product metadata common to all Sentinel-1 products.

Annotation datasets contain metadata describing the properties and characteristics of the measurement data or how they were generated. For each band of data there is a product annotation data set that contains metadata describing the main characteristics corresponding to that band such as the state of the platform during acquisition, image properties, polarization, Doppler information, swath merging and geographic location. Calibration annotations contain calibration information and the beta naught, sigma naught, gamma and digital number look-up tables that can be used for absolute product calibration. Noise data annotations contain the estimated thermal noise look-up tables. Annotated data sets are provided in XML format.

Measurement datasets contain the binary information of the actual acquired or processed data. For Level-0 this is the instrument data, for Level-1 it is processed data. Measurement datasets are provided in GeoTIFF format (georeferenced) for Level-1 products. There is one measurement data set per polarization and per swath. TOPSAR SLC products contain one complex measurement data set in GeoTIFF format (georeferenced) per swath per polarization. Level-1 GRD products contain one detected measurement data set in GeoTIFF format (georeferenced) per polarization.

In the Preview folder, quick-look datasets are power detected, averaged and decimated to produce a lower resolution version of the image. Single polarization products are represented with a grey scale image. Dual polarization products are represented by a single composite color image in RGB with the red channel (R) representing the first polarization, the green channel (G) represents the second polarization and the blue channel (B) represents an average of the absolute values of the two polarizations.

Representation datasets found in the Support folder contain information about the format or syntax of the measurement and annotated data sets and can be used to validate and exploit these data. Representation data sets are provided as XML schemas.

Sentinel-1 SAFE Naming Convention

The top-level Sentinel-1 product folder name is composed of upper-case alphanumeric characters separated by an underscore (_).

Product Name
Example:
S1A_IW_SLC__1SDV_20170413T131907_20170413T131934_016126_01AA29_8754.SAFE

Naming Notes:

  • In Resolution (R), underscore (_) is a valid input to mean “not applicable.”
  • Product Class — A (Annotation) is internal only and not distributed.
  • In Start/Stop Date-Time, date and time are separated by the character ‘T’
  • Absolute orbit (OOOOO) ranges from 000001 to 999999.
  • Mission data-take ID (DDDDDD) ranges from 000001 to FFFFFF.
  • The folder extension is always “SAFE”

4. HDF5

Hierarchical Data Format (HDF) is a set of file formats designed to store and organize large amounts of data.

HDF5 simplifies the file structure to include only two major types of object:

  • Datasets, which are multidimensional arrays of a homogeneous type
  • Groups, which are container structures which can hold datasets and other groups

HDF5 is a general purpose file format and programming library for storing scientific data. Use of the HDF library enables users to read HDF files on multiple platforms regardless of the architecture the platforms use to represent integer and floating point numbers.

ASF DAAC SAR datasets available in HDF5 format and tools

  • Seasat products are offered in either HDF5 or GeoTIFF formats. HDF5 products may be viewed with MapReady.
  • All SMAP standard products are in the Hierarchical Data Format version 5 (HDF5). SMAP products can be viewed with Panoply.

HDF5 Data Recipe

How to View Seasat HDF5 Files in ASF MapReady

Product HDF5 Naming Conventions

Naming Convention for Seasat products.

Granule Name

Example:
SS_01492_STD_F2829.h5

HDF5 file format for SMAP products.

Product Name

Example:
SMAP_L1A_RADAR_00934_A_20141225T074951_R04000_002.h5

Options

Processing Level: L1A_RADAR; L1B_S0_LoRes; L1C_S0_HiRes

Flight Direction: Ascending (A); Descending (D)

Composite Release ID (R is always used)

  • Launch indicator: calibration data (0), post-calibration data (1);
  • Major ID — incremented by major changes in algorithm or processing approach;
  • Minor ID — incremented by changes to data processing, such as algorithm, software, or parameters.

Data Format Extension: HDF5 (.h5); QA products (.qa)

5. UAVSAR

From JPL’s Polarimetric (PolSAR) Data Format and Interferometric (InSAR) Pair Data Format.

Polarimetric and RPI Product File Formats

SLC files (.slc): calibrated single look complex files; floating point format, little endian, 8 bytes per pixel, corresponding to the scattering matrix.

  • Polarimetric product: one SLC file for each polarization (HH, HV, VH, and VV)
  • Repeat-pass interferometric product: one SLC file for each flight track (track 1, T1; and track 2, T2). Files are available by request, but are not normally included in the data distribution.

Ground projected files (.grd): calibrated complex cross products (Polarimetric product) or interferometric products (RPI product) projected to the ground in simple geographic coordinates (latitude, longitude). There is a fixed number of looks for each pixel. Floating point or complex floating point, little endian, 8 or 4 bytes per pixel.

HGT file: the DEM that the imagery was projected to, in the same geographic coordinates as the ground projected files. Floating point (4 bytes per pixel), little endian, ground elevation in meters.

Annotation file (.ann): a text file with metadata.

Additional Polarimetric Product File Formats

MLC files (.mlc): calibrated multi-looked cross products, floating point format, three files 8 bytes per pixel, three files 4 bytes per pixel, little endian.

Compressed Stokes Matrix product (.dat): AIRSAR compressed stokes matrix format for software compatibility (AIRSAR Integrated Processor Documentation). 10 bytes per pixel.

Terrain slope file (.slope): The terrain slope file (.slope) contains the derivatives of the digital elevation model (DEM) in the East and North direction. The file is an array of two interleaved floating point numbers (2 x 4 bytes per pixel) with geometry identical to the other ground-projected data layers (.grd, .hgt, .inc). For each interleaved pixel, the first 4 byte value is the terrain slope in the east direction, and the second 4 byte value is the slope in the north direction. Floating point, little endian.

Incidence angle file (.inc): the local incidence angle, the angle between the surface normal and the radar line of site. The file consists of 4-byte floating point values, co-registered with the slope file. The floating point, little endian file contains values reported in radians, as shown in the image.

Terrain slope file (.slope): The terrain slope file (.slope) contains the derivatives of the digital elevation model (DEM) in the East and North direction. The file is an array of two interleaved floating point numbers (2 x 4 bytes per pixel) with geometry identical to the other ground-projected data layers (.grd, .hgt, .inc). For each interleaved pixel, the first 4 byte value is the terrain slope in the east direction, and the second 4 byte value is the slope in the north direction. Floating point, little endian.

Incidence angle file (.inc): the local incidence angle, the angle between the surface normal and the radar line of site. The file consists of 4-byte floating point values, co-registered with the slope file. The floating point, little endian file contains values reported in radians, as shown in the image.

Additional RPI Product File Formats

AMP files (.amp1 and .amp2): calibrated multi-looked amplitude products, one file per repeat track, floating point format 4 bytes per pixel, little endian.

INT files (.int): interferogram product, one file per pair of repeat tracks, complex floating point format 8 bytes per pixel, little endian.

UNW files (.unw): unwrapped interferometric phase product, one file per pair of repeat tracks, floating point format 4 bytes per pixel, little endian.

COR files (.cor): interferometric correlation product, one file per pair of repeat tracks, floating point format 4 bytes per pixel, little endian.

KML and KMZ files (.kml or .kmz): these files allow you to view a representation of their corresponding file type in Google Earth or similar software.

PNG files (.png): these are representations of the corresponding products in standard PNG Format.

Formation of Interferometric Products:

Prior to creating interferometric products, the SLC images from both tracks are co-registered to each other using GPS data and the data itself to estimate and compensate for the variable motion between the tracks.

The single look complex (SLC) data for each track is summed in range and azimuth by the number of looks specified in the annotation file (“Number of Looks in Range” and “Number of Looks in Azimuthz”) (typically 3 looks in range and 12 looks in azimuth), divided by the product of the number of looks in range and azimuth, and output as the amp1 and amp2 files.

The interferogram in the .int file is formed by multiplying the single look complex image from track 1 times the complex conjugate of the single look complex image from track 2. The resulting complex values are then summed in range and azimuth according to the desired number of looks in the range and azimuth direction, with each pixel then divided by the product of the the number of azimuth looks and the number of range looks.

The correlation file .cor is formed by dividing the interferogram values (the .int file) by the product of the multilooked amplitude values for track 1 and track 2 (the .amp1 and .amp2 files).

The unwrapped interferometric phase file UNW (the .unw file) is obtained by applying the Goldstein/Werner method on the interferogram: Goldstein, R. M. and Werner, C. L., 1998. Radar interferogram filtering for geophysical applications. Geophysical Research Letters, 25(21):4035-4038.

Calibration of the data:

Please see JPL’s calibration page for documentation on calibration of the data.

UAVSAR Product Naming Conventions

UAVSAR Polarimetric Product Naming Convention.

Example:
Dthvly_34501_08038_006_080731_L090HH_XX_01.slc

UAVSAR Repeat-pass Interferometric Product Naming Convention

Example:
SanAnd_26501_09083-010_10028-000_0174d_s01_L090HH_01.amp1.grd

6. AIRSAR (data format)

From JPL’s AirSAR website.

File Headers

All AIRSAR data files start with 3-4 header records.

  • First header: general information about data file including # of lines and samples and offset to the first data record
  • Parameter header: information specific to the scene
  • Calibration header: information on data calibration
  • DEM header: only present for TOPSAR data; contains the elevation offset and elevation increment needed to translate the integer*2 values to elevations in meters

Data Modes

POLSAR Data Mode

The POLSAR operating mode collects twelve channels of data, four in each of the three frequencies of AIRSAR: P-, L-, and C-band.  The four data channels are:

HH horizontally polarized transmit wave, horizontally polarized receive wave
HV horizontally polarized transmit wave, vertically polarized receive wave
VH vertically polarized receive wave, horizontally polarized receive wave
VV vertically polarized receive wave, vertically polarized receive wave

TOPSAR Data Modes

XTI1 – Generates a C-band DEM along with L- and P-band polarimetry
XTI2 – Generates a C-band and an L-band DEM, along with P-band polarimetry
(see exception below for when P-band data will not be present)

TOPSAR data are processed on the AIRSAR Integrated Processor (ver. 5.1 or ver. 6.1).

Data Files

POLSAR Data Files

Data Format — Compressed Stokes matrix data in slant range. File Size (10 km) is 15 Mbytes. CM data are oriented so that each pixel sample is decreasing azimuth (along track) and each pixel line is of increasing range (cross-track).

Each POLSAR set typically contains:

cm4212_c.dat
cm4212_l.dat
cm4212_p.dat
cm4212.gif
cm4212_meta.airsar
hdo4212.log

TOPSAR Data Files

Each TOPSAR scene typically contains:

  • 4 or 8 DEM (C-band or C-band and L-band) and related data files
  • 1 or 2 Polarimetric data files (see note below for exception)

They have the following files:

DEM Data (C-band, maybe L-band)

  • c.vvi2 – C-band VV polarization only
  • integer2 (signed 16 bit)
  • .demi2 – C-band DEM
  • byte file
  • .corgr – Correlation coefficient map
  • .incgr – Local incidence angle map

Polarimetric Data

  • l.datgr  – L-band polarimetry compressed Stokes matrix
  • p.datgr – P-band polarimetry compressed Stokes matrix

Note: Due to FCC restrictions, since 1994, P-band data are not included for TOPSAR datasets collected at 40 MHz bandwidth over sites in the United States.

AIRSAR Naming Conventions

Naming Convention for AIRSAR products.

Product Name

Example:
cm4212_l.dat

Options:

Data Format (FF): Compressed Stokes Matrix (cm), Synoptic (sy)

Band Frequency: C-band (c), L-band (l), P-band (p). (Due to FCC restrictions since 1994, P-band data are not included for POLSAR datasets collected at 40 MHz bandwidth over sites in the United States.)

File Extension: .dat, .gif, .airsar, .log

Naming Convention for AIRSAR products.

Product Name

Example:
ts1745_p.datgr

Options:

Data Format is TOPSAR (ts).

Band Frequency: C-band (c), L-band (l), P-band (p). (Due to FCC restrictions since 1994, P-band data are not included for POLSAR datasets collected at 40 MHz bandwidth over sites in the United States.)

File Extension: .datgr, .demi2, .log, .airsar, vvi2, .corgr, .incgr, .integer2. (All TOPSAR data, including any polarimetric data collected in a TOPSAR mode, will be projected in the ground range.)

TOPSAR Data Conversions

Data collected since 1993 are processed using Version 5.1 and Version 6.1.

Note that the integer*2 data will need to be converted using the following equations:

Convert demi*2 data to elevations in meters:
hs = (elevation increment) * DN + (elevation offset)

  • Elevation increment and offset are found in the DEM header record
  • DN is the integer*2 (signed) number from the .demi2 data file

Convert vvi*2 data to radar cross sections
sigma naught =(DN**2)/(General Scale Factor GFS)

  • DN is the integer*2 (signed) number as the amplitude (linear value) from the .vvi2 data file
  • GSF is in the second field of the Calibration Header.
  • Note that the GSF = 10**6

Polarimetric data collected in the TOPSAR mode are read the same way as POLSAR data.

Comments are closed.