Skip to content

WIP: Allow GDAL PDS format to be read #6023

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
chkim-usgs wants to merge 2 commits into
base: dev
Choose a base branch
from
Draft

WIP: Allow GDAL PDS format to be read #6023

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
70ce3ad
wip
chkim-usgs Apr 23, 2026
c2825bd
add CheckVsiIo() and clean up
chkim-usgs Apr 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
131 changes: 48 additions & 83 deletions isis/src/base/objs/ProcessImport/ProcessImport.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@ find files of those names at the top level of this repository. **/
#include "SpecialPixel.h"
#include "UserInterface.h"

#include "cpl_vsi.h"
Copy link
Copy Markdown
Collaborator

@acpaquette acpaquette Apr 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there documentation for this API? I'm not sure if this is something we want to commit ourselves to

Copy link
Copy Markdown
Collaborator

@acpaquette acpaquette Apr 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like enough of a documented API exists for this at:
https://gdal.org/en/stable/api/cpl.html#cpl-vsi-h

Copy link
Copy Markdown
Collaborator

@Kelvinrr Kelvinrr Apr 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that we do not want to use the VSI API directly, let GDAL handle it via /vsi prefixes.

Copy link
Copy Markdown
Collaborator

@acpaquette acpaquette Apr 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This may be the best way to handle this to be as minimally invasive as possible. The other option would be to add a new organiztion option to the importer specifically for vis. We would then have to add a custom member function to support ISIS functors

Copy link
Copy Markdown
Collaborator

@acpaquette acpaquette Apr 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thinking about this a bit more, I don't think the GDAL raster IO supports random byte access. Something to double check


#define EXPONENT_MASK ((char) 0x7F)


Expand Down Expand Up @@ -1370,33 +1372,28 @@ namespace Isis {
tok = tok.toUpper();
Isis::EndianSwapper swapper(tok);

ifstream fin;
// Open input file
Isis::FileName inFile(p_inFile);
QString inFileName(inFile.expanded());
fin.open(inFileName.toLatin1().data(), ios::in | ios::binary);
if (!fin.is_open()) {

size_t nRead = 0;

VSILFILE *fp = VSIFOpenL(inFileName.toLatin1().data(), "rb");
if (fp == nullptr) {
QString msg = "Cannot open input file [" + p_inFile + "]";
throw IException(IException::Io, msg, _FILEINFO_);
}

// Handle the file header
streampos pos = fin.tellg();
vsi_l_offset pos = VSIFTellL(fp);
if (p_saveFileHeader) {
p_fileHeader = new char[p_fileHeaderBytes];
fin.read(p_fileHeader, p_fileHeaderBytes);
fin.seekg(p_suffixData+p_fileHeaderBytes, ios_base::beg);
nRead = VSIFReadL(p_fileHeader, 1, p_fileHeaderBytes, fp);
CheckVsiIo(nRead, p_fileHeaderBytes, pos);
VSIFSeekL(fp, p_suffixData+p_fileHeaderBytes, SEEK_SET);
}
else {
fin.seekg(p_fileHeaderBytes+p_suffixData, ios_base::beg);
}

// Check the last io
if (!fin.good()) {
QString msg = "Cannot read file [" + p_inFile + "]. Position [" +
toString((int)pos) + "]. Byte count [" +
toString(p_fileHeaderBytes) + "]" ;
throw IException(IException::Io, msg, _FILEINFO_);
VSIFSeekL(fp, p_fileHeaderBytes+p_suffixData, SEEK_SET);
}

// Construct a line buffer manager
Expand Down Expand Up @@ -1427,56 +1424,37 @@ namespace Isis {
}

// Handle any data headers (e.g., the data at the beginning of each band)
pos = fin.tellg();
pos = VSIFTellL(fp);
if (p_saveDataHeader) {
p_dataHeader.push_back(new char[p_dataHeaderBytes]);
fin.read(p_dataHeader.back(), p_dataHeaderBytes);
nRead = VSIFReadL(p_dataHeader.back(), 1, p_dataHeaderBytes, fp);
CheckVsiIo(nRead, p_dataHeaderBytes, pos);
}
else {
fin.seekg(p_dataHeaderBytes, ios_base::cur);
VSIFSeekL(fp, p_dataHeaderBytes, SEEK_CUR);
}

// Check the last io
if (!fin.good()) {
QString msg = "Cannot read file [" + p_inFile + "]. Position [" +
toString((int)pos) + "]. Byte count [" +
toString(p_dataHeaderBytes) + "]" ;
throw IException(IException::Io, msg, _FILEINFO_);
}

// Space for storing prefix and suffix data pointers
vector<char *> tempPre, tempPost;

// Loop for each line in a band
for(int line = 0; line < p_nl; line++) {

// Handle any line prefix bytes
pos = fin.tellg();
pos = VSIFTellL(fp);
if (p_saveDataPre) {
tempPre.push_back(new char[p_dataPreBytes]);
fin.read(tempPre.back(), p_dataPreBytes);
nRead = VSIFReadL(tempPre.back(), 1, p_dataPreBytes, fp);
CheckVsiIo(nRead, p_dataPreBytes, pos);
}
else {
fin.seekg(p_dataPreBytes, ios_base::cur);
}

// Check the last io
if (!fin.good()) {
QString msg = "Cannot read file [" + p_inFile + "]. Position [" +
toString((int)pos) + "]. Byte count [" +
toString(p_dataPreBytes) + "]" ;
throw IException(IException::Io, msg, _FILEINFO_);
VSIFSeekL(fp, p_dataPreBytes, SEEK_CUR);
}

// Get a line of data from the input file
pos = fin.tellg();
fin.read(in, readBytes);
if (!fin.good()) {
QString msg = "Cannot read file [" + p_inFile + "]. Position [" +
toString((int)pos) + "]. Byte count [" +
toString(readBytes) + "]" ;
throw IException(IException::Io, msg, _FILEINFO_);
}
pos = VSIFTellL(fp);
nRead = VSIFReadL(in, 1, readBytes, fp);
CheckVsiIo(nRead, readBytes, pos);

// Swap the bytes if necessary and convert any out of bounds pixels
// to special pixels
Expand Down Expand Up @@ -1536,21 +1514,14 @@ namespace Isis {
p_progress->CheckStatus();

// Handle any line suffix bytes
pos = fin.tellg();
pos = VSIFTellL(fp);
if (p_saveDataPost) {
tempPost.push_back(new char[p_dataPostBytes]);
fin.read(tempPost.back(), p_dataPostBytes);
nRead = VSIFReadL(tempPost.back(), 1, p_dataPostBytes, fp);
CheckVsiIo(nRead, p_dataPostBytes, pos);
}
else {
fin.seekg(p_dataPostBytes, ios_base::cur);
}

// Check the last io
if (!fin.good()) {
QString msg = "Cannot read file [" + p_inFile + "]. Position [" +
toString((int)pos) + "]. Byte count [" +
toString(p_dataPreBytes) + "]" ;
throw IException(IException::Io, msg, _FILEINFO_);
VSIFSeekL(fp, p_dataPostBytes, SEEK_CUR);
}
} // End line loop

Expand All @@ -1567,47 +1538,32 @@ namespace Isis {
}

// Handle the band trailer
pos = fin.tellg();
pos = VSIFTellL(fp);
if (p_saveDataTrailer) {
p_dataTrailer.push_back(new char[p_dataTrailerBytes]);
fin.read(p_dataTrailer.back(), p_dataTrailerBytes);
nRead = VSIFReadL(p_dataTrailer.back(), 1, p_dataTrailerBytes, fp);
CheckVsiIo(nRead, p_dataTrailerBytes, pos);
}
else {
fin.seekg(p_dataTrailerBytes, ios_base::cur);
}

// Check the last io
if (!fin.good()) {
QString msg = "Cannot read file [" + p_inFile + "]. Position [" +
toString((int)pos) + "]. Byte count [" +
toString(p_fileHeaderBytes) + "]" ;
throw IException(IException::Io, msg, _FILEINFO_);
VSIFSeekL(fp, p_dataTrailerBytes, SEEK_CUR);
}

} // End band loop

// Handle the file trailer
pos = fin.tellg();
pos = VSIFTellL(fp);
if (p_saveFileTrailer) {
fin.seekg(0, ios_base::end);
streampos e = fin.tellg();
p_fileTrailerBytes = (int)(e - pos + (streampos)1);
VSIFSeekL(fp, 0, SEEK_END);
vsi_l_offset e = VSIFTellL(fp);
p_fileTrailerBytes = (int)(e - pos); //(int)(e - pos + (streampos)1);
p_fileTrailer = new char[p_fileTrailerBytes];
fin.seekg(pos);
fin.read(p_fileTrailer, p_fileTrailerBytes);

// Check the io
if (!fin.good()) {
QString msg = "Cannot read file [" + p_inFile + "]. Position [" +
toString((int)pos) + "]. Byte count [" +
toString(p_fileTrailerBytes) + "]" ;
throw IException(IException::Io, msg, _FILEINFO_);
}

VSIFSeekL(fp, pos, SEEK_SET);
nRead = VSIFReadL(p_fileTrailer, 1, p_fileTrailerBytes, fp);
CheckVsiIo(nRead, p_fileTrailerBytes, pos);
}

// Close the file and clean up
fin.close();
VSIFCloseL(fp);
delete [] in;
}

Expand Down Expand Up @@ -2323,4 +2279,13 @@ namespace Isis {
return p_inFile;
}

void ProcessImport::CheckVsiIo(size_t bytesRead, size_t expectedBytes, vsi_l_offset pos) {
if (bytesRead != expectedBytes) {
QString msg = "Cannot read file [" + p_inFile + "]." +
"Position [" + toString((int)pos) + "]." +
"Byte count [" + toString((long long)bytesRead) + "]." +
"Expected byte count [" + toString((long long)expectedBytes) + "]." ;
throw IException(IException::Io, msg, _FILEINFO_);
}
}
}
4 changes: 4 additions & 0 deletions isis/src/base/objs/ProcessImport/ProcessImport.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,9 @@ find files of those names at the top level of this repository. **/
#include "SpecialPixel.h"
#include "UserInterface.h"

#include "cpl_vsi.h"


namespace Isis {
/**
* @brief Byte swapper
Expand Down Expand Up @@ -177,6 +180,7 @@ namespace Isis {
virtual void StartProcess(void funct(Isis::Buffer &out));
void SetInputFile(const QString &file);
QString InputFile();
void CheckVsiIo(size_t bytesRead, size_t expectedBytes, vsi_l_offset pos);

// SetOutputCube() is not virtual in the Process class, so the following
// definitions for this method are the only ones that are allowed for
Expand Down
2 changes: 1 addition & 1 deletion isis/src/base/objs/ProcessImportPds/ProcessImportPds.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1399,7 +1399,7 @@ namespace Isis {

// Internalize the PDS label in the PVL that was passed in
try {
pdsLabel.read(pdsLabelFile);
pdsLabel = Pvl(pdsLabelFile);
}
catch (IException &e) {
throw IException(e, IException::User,
Expand Down
2 changes: 1 addition & 1 deletion isis/src/core/src/FileName.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -449,7 +449,7 @@ namespace Isis {
* @return Boolean
*/
bool FileName::fileExists() const {
if (toString().contains((QString)"https://")) {
if (toString().contains((QString)"https://") || toString().contains((QString)"/vsi")) {
return true;
}
return QFileInfo(expanded()).exists();
Expand Down
15 changes: 12 additions & 3 deletions isis/src/core/src/Pvl.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,9 +132,18 @@ namespace Isis {
throw IException(IException::Programmer, msg, _FILEINFO_);
}

CPLStringList metadata = CPLStringList(dataset->GetMetadata("json:ISIS3"), false);

if (metadata[0] != nullptr) {
CPLStringList metadataDomains = CPLStringList(dataset->GetMetadataDomainList(), false);
CPLStringList metadata;
const char* domainIsis = "json:ISIS3";
const char* domainPds = "json:PDS";
if (CSLFindString(metadataDomains.List(), domainIsis) != -1) {
metadata = CPLStringList(dataset->GetMetadata(domainIsis), false);
}
else if (CSLFindString(metadataDomains.List(), domainPds) != -1) {
metadata = CPLStringList(dataset->GetMetadata(domainPds), false);
}

if (metadata.Count() > 0 && metadata[0] != nullptr) {
const char *metadataJsonString = metadata[0];
nlohmann::ordered_json metadataAsJson = nlohmann::ordered_json::parse(metadataJsonString);
readObject(*this, metadataAsJson);
Expand Down
Loading