All software maintained by the U.S. Geological Survey (USGS) is subject to peer review. There are several different types of reviews required for software within the USGS depending on the software type and software stage.
Equally important to performing the peer review is documenting the review and reconciliation process. While similar at a high level, this process varies slightly depending on the current project state.
For new preliminary software...
This review occurs prior to migrating scratch software into the USGS organization on GitHub.
- Put the software in a version control repository accessible by the intended reviewer
- Create an issue in the software release reviews project requesting
a review on the software
- Include a link to where the software is accessible
- Add a label to the issue indiciating the type of review being requested
- Wait for reviewer to approve the changes
- This may involve iteratively fixing issues the reviewer points out
Note: Once the review is complete you typically want to finish the initial migration process as outlined in the releases section.
For existing preliminary software...
Every change to a repository in the USGS organization on GitHub must receive at least an administrative review. This can be done using pull requests. The following steps assume you are following the integration manager workflow.
- Submit a pull request from your user fork of the USGS repository back to
the primary USGS repository
- Assign (or request) a reviewer to the pull request
- Wait for reviewer to approve the changes
- This may involve iteratively fixing issues the reviewer points out
- Merge (or better, have the reviewer merge) the changes into the USGS repository
For approved software tags...
- Create a release candidate reconciliation branch
- Using semantic versioning this may be something like:
git checkout -b vMAJOR.MINOR.PATCH-rc1-reconciliation
- Using semantic versioning this may be something like:
- Create a release candidate tag
- Using semantic versioning this may be something like:
git tag -am 'vMAJOR.MINOR.PATCH-rc1' vMAJOR.MINOR.PATCH-rc1
- Using semantic versioning this may be something like:
- Create an issue on the project issues page requesting the appropriate level
of review be performed
- Provide a link to the release candidate tag
- Wait for reviewer(s) to approve the changes
- This may involve iteratively fixing issues the reviewer points out by making commits back to the release candidate branch
Note: Once the review is complete you typically want to finish the release process for approved software as outlined in the releases section.
Admiministrative reviews ensure personal, private, or otherwise sensitive information is not included in the repository. This includes:
- Personally identifiable information (PII)
- Absolute file system paths
- Internal server host names or IP addresses
- Usernames/Passwords
- ... etc ...
These reviews can be performed by any trusted person; the reviewer does not necessarily need a strong scientific or programming background. When migrating an existing project into the USGS organization on GitHub, it is important to remember that the entire project history needs to be reviewed if that history is to be maintained after migration.
All project types require this level of review.
Software in the following stages require this level of review:
Technical reviews ensure overall code quality. This can mean different things depending on the individual project/team but some typical quality checks include:
- Coding standards
- Unit tests passing
- User input cleansing
- Memory leaks
- Vulnerabilities
- Optimizations
- ... etc ...
These reviews should be performed by a trusted developer with applicable programming knowledge. This person need not have thorough subject-matter knowledge on the software topic.
All software types require this level of review.
Software in the following stages require this level of review:
Scientific reviews ensure the software generates output that align with published or otherwise well-known expected results. This may involve:
- Comparing output with external data sets
- Comparing algorithms with published scholarly articles about the algorithm
- Reviewing unit and integration test results
- ... etc ...
These reviews should be performed by a trusted scientific expert with applicable subject-matter knowledge as well as sufficient applicaple programming knowledge.
Software in the following stages require this level of review: