RPM building documentation overhaul for SVN to Git migration

docs/policy/packaging-svn-to-git-migration.md

@@ -0,0 +1,44 @@
+OSG Software Packaging SVN to Git Migration
+===========================================
+
+On Monday, February 16th, 2025, the OSG Software Team will be migrating the repository
+that hosts the source for OSG Software RPM packages from a Subversion repo
+at <https://vdt.cs.wisc.edu/svn> to GitHub at <https://github.com/osg-htc/software-packaging>.
+
+The new repo is available for preview and testing builds from but changes will not be
+preserved; you should keep using the SVN repo until the switchover.
+
+
+Requirements
+------------
+
+To use the new repo, you must have the latest OSG-Build, 2.2.1.
+Run `osg-build --version` to check which version you are using.
+If using it from Git, be sure you're on the `V2-branch`.
+
+Note the [OSG Build Tools page](https://osg-htc.org/technology/software/osg-build-tools/)
+has been updated with information about how to get the software via Pip or Apptainer.
+
+You must also have write permission to the
+[osg-htc/software-packaging repo](https://github.com/osg-htc/software-packaging);
+contact a software team member to request access.
+
+
+Workflow changes
+----------------
+
+The layout of the GitHub repo is the same as the SVN repo -- each set of targets,
+such as `24-main`, is a separate top-level directory, with package directories under it.
+
+Koji scratch builds are performed from the local file system as usual.
+To do a non-scratch build, you must push your changes to the `main` branch of the upstream repo.
+
+Feel free to push directly to the `main` branch -- no need to make a pull request,
+unless you want your changes reviewed.
+
+The [development process page](../../software/development-process/) has been updated with
+these new instructions as well as some advice for keeping track of changes when building packages
+for multiple release series.
+
+If you have Python 3.9 or newer, consider installing [pre-commit](https://pre-commit.com/#intro);
+the hook in the repo should prevent you from accidentally checking in large files.

docs/software/koji-workflow.md

@@ -8,43 +8,45 @@ Terminology
 
 Using and understanding the following terminology correctly will help in the reading of this document:
 
-**Package**  
-This refers to a named piece of software in the Koji database. An example would be "lcmaps".
+**Package**<br>
+This refers to a named piece of software in the Koji database. An example would be "xrootd".
 
-**Build**  
+**Build**<br>
 A specific version and release of a package, and an associated state. A build state may be successful (and contain RPMs), failed, or in-progress. A given build may be in one or more tags. The build is associated with the output of the latest build task with the same version and release of the package.
 
-**Tag**  
-A named set of packages and builds, parent tags, and reference to external repositories. An example would be the "osg-3.3-el6-development" tag, which contains (among others) the "lcmaps" package and the "lcmaps-1.6.6-1.1.osg33.el6" build. There is an inheritance structure to tags: by default, all packages/builds in a parent tag are added to the tag. A tag may contain a reference to (possibly inherited) external repositories; the RPMs in these repositories are added to repositories created from this tag. Examples of referenced external repositories include CentOS base, EPEL, or JPackage.
+**Tag**<br>
+A named set of packages and builds, parent tags, and reference to external repositories. An example would be the "osg-25-el9-development" tag, which contains (among others) the "xrootd" package and the "xrootd-5.9.1-1.2.osg25-el9" build. There is an inheritance structure to tags: by default, all packages/builds in a parent tag are added to the tag. A tag may contain a reference to (possibly inherited) external repositories; the RPMs in these repositories are added to repositories created from this tag. Examples of referenced external repositories include AlmaLinux base or EPEL.
 
 !!! note
     A tag is NOT a yum repository.
 
-**Target**  
-A target consists of a build tag and a destination tag. An example is "osg-3.3-el6", where the build tag is "osg-3.3-el6-build" and the destination tag is "osg-3.3-el6". A target is used by the build task to know what repository to build from and tag to build into.
+**Target**<br>
+A target consists of a build tag and a destination tag. An example is "osg-25-main-el9",
+where the build tag is "osg-25-main-el9" and the destination tag is "osg-25-main-el9-development".
+A target is used by the build task to know what repository to build from and tag to build into.
 
-**Task**  
+**Task**<br>
 A unit of work for Koji. Several common tasks are:
 
--   build  
+-   build<br>
     This task takes a SRPM and a target, and attempts to create a complete Build in the target's destination tag from the target's build repository. This task will launch one buildArch task for each architecture in the destination tag; if each subtask is successful, then it will launch a tagBuild subtask.
 
     !!! note
         If the build task is marked as "scratch", then it won't result in a saved Build.
 
--   buildArch  
+-   buildArch<br>
     This task takes a SRPM, architecture name, and a Koji repository as an input, and runs `mock` to create output RPMs for that arch. The build artifacts are added to the Build if all buildArch tasks are successful.
 
--   tagBuild  
+-   tagBuild<br>
     This adds a successful build to a given tag.
 
--   newRepo  
+-   newRepo<br>
     This creates a new repository from a given tag.
 
-**Build artifacts**  
+**Build artifacts**<br>
 The results of a buildArch task. Their metadata are recorded in the Koji database, and files are saved to disk. Metadata may include checksums, timestamps, and who initiated the task. Artifacts may include RPMs, SRPMs, and build logs.
 
-**Repository**  
+**Repository**<br>
 A yum repository created from the contents of a tag at a specific point in time. By default, the yum repository will contain all successful, non-blocked builds in the tag, plus all RPMs in the external repositories for the tag.
 
 
@@ -53,7 +55,7 @@ Obtaining Access
 
 Building OSG packages in Koji requires these privileges:
 
-- access to the OSG subversion repository at https://vdt.cs.wisc.edu/svn
+- access to the OSG Software Packaging repository at https://github.com/osg-htc/software-packaging
 - access to `osgsw-ap.chtc.wisc.edu` for uploading to the upstream sources directory
 - access to the Koji service via a Kerberos credential
 
@@ -84,22 +86,22 @@ Using Koji
 Before pushing package changes to the OSG Software Packaging repository, you should create a "scratch build".
 This builds an RPM from the current directory using Koji, but does not tag the resulting package.
 
-To make a scratch build, run:
+To make a scratch build and download the resulting files to your machine, run:
 
 ```console
-$ osg-build koji --scratch <PACKAGE DIRECTORY>
+$ osg-build koji --scratch --getfiles <PACKAGE DIRECTORY>
 ```
 
-To download all of the files from a scratch build, add the `--getfiles` flag;
-you may also visit the links that osg-build printed to download the files individually.
+If you do not want to download the files, omit the `--getfiles` flag.
+You may also visit the links that osg-build printed to download the files later.
 
 ### Creating a new build
 
-We create a new build in Koji from the package's directory in OSG Software subversion.
+We create a new build in Koji from the package's directory in [OSG Software Packaging repository](https://github.com/osg-htc/software-packaging).
 
 If a successful build already exists in Koji (regardless of whether it is in the tag you use), you cannot replace the build. Two builds are the same if they have the same NVR (Name-Version-Release). You *can* do a "scratch" build, which recompiles, but the results are not added to the tag. This is useful for experimenting with koji.
 
-To do a build, execute the following command from within the OSG Software subversion checkout:
+To do a build, execute the following command from within an up-to-date clone of the repository:
 
 ```console
 [you@host]$ osg-build koji <PACKAGE NAME>
@@ -116,39 +118,48 @@ The most recent build results are always shown on the home page of Koji:
 
 Clicking on a build result brings you to the build information page. A successful build will result in the build page having build logs, RPMs, and a SRPM.
 
-If your build isn't in the recent list, you can use the search box in the upper-right-hand corner. Type the exact package name (or use a wildcard), and it will bring up a list of all builds for that package. You can find your build from there. For example, the "lcmaps" package page is here:
+If your build isn't in the recent list, you can use the search box in the upper-right-hand corner.
+Type the exact package name (or use a glob, e.g. `osdf*`), and it will bring up a list of all builds for that package.
+You can find your build from there. For example, the `xrootd` package page is here:
 
-<https://koji.osg-htc.org/koji/packageinfo?packageID=56>
+<https://koji.osg-htc.org/koji/packageinfo?packageID=89>
 
-And the lcmaps-1.6.6-1.1.osg33.el6 build is here:
+And the `xrootd-5.9.1-1.2.osg25.el9` build is here:
 
-<https://koji.osg-htc.org/koji/buildinfo?buildID=7427>
+<https://koji.osg-htc.org/koji/buildinfo?buildID=18884>
 
 
 #### How to get the resulting RPM into a repository
 
 Once a package has been built, it will be signed and added to a tag, typically one of the `-development` tags.
 The build will eventually be copied and made available for installation at `repo.osg-htc.org`,
-in typically less than an hour.
-The build will be available for use as a build dependency in under five minutes.
+in typically less than half an hour.
+The build will be available for use as a build dependency in under five minutes;
+Koji will wait for the repo to be updated ("regenerated") before doing another build.
 
 The OSG repos with names ending in `-minefield` are built from the `-development` repos;
 installing RPMs from those repos pulls directly from the Koji build system.
 You may update the `-minefield` repos by running the `osg-koji regen-repo` command on the corresponding `-development` repo.
-For example, to update the `osg-minefield` repo for OSG series `24-main` on distro `el9`, run
+For example, to update the `osg-minefield` repo for OSG series `25-main` on distro `el9`, run
 ```console
-$ osg-koji regen-repo osg-24-main-el9-development
+$ osg-koji regen-repo osg-25-main-el9-development
 ```
-This typically only takes a few minutes.
+A regen-repo task will be created -- you can watch its progress in the web interface.
+
+!!!note
+  If the output you get is something like `Repo info: https://koji.osg-htc.org/koji/repoinfo?repoID=153995`,
+  it means that the build repo is already current.
+  If you still want to force repo creation, add the `--make-task` argument to the above `regen-repo` command.
+
 To do this for multiple distro versions, use a loop.
 For example:
 ```console
-$ for el in el8 el9 el10; do osg-koji regen-repo osg-24-main-${el}-development; done
+$ for el in el8 el9 el10; do osg-koji regen-repo osg-25-main-${el}-development; done
 ```
 
 !!!note
     Due to [SOFTWARE-6069](https://opensciencegrid.atlassian.net/browse/SOFTWARE-6069),
-    you must run `osg-koji regen-repo` by hand to update the `minefield` repos.
+    you may need to run `osg-koji regen-repo` by hand to update the `minefield` repos.
 
 You may check the status of regen-repo tasks in the
 [Koji web interface](https://koji.osg-htc.org/koji/tasks?method=newRepo&state=all&view=tree&order=-id).
@@ -158,7 +169,7 @@ You may check the status of regen-repo tasks in the
 
 -   Failed build tasks can be seen from the Koji homepage. The logs from the tasks are included. Relevant logs include:
 
-    -   `root.log`  
+    -   `root.log`<br>
         This is the log of mock trying to create an appropriate build root for your RPM. This will invoke yum twice: once to create a generic build root, once for all the dependencies in your BuildRequires. All RPMs in your build root will be logged here. If mock is unable to create the build root, the reason will show up here.
 
         -   404 Errors
@@ -167,7 +178,7 @@ You may check the status of regen-repo tasks in the
             this commonly indicates that an rpm repo mirror was updated and our build repo is out-of-date.
             This can be fixed by regenerating the relevant build repos for your builds.
 
-            This is usually something like `osg-3.4-el7-build` or `osg-upcoming-el7-build`;
+            This is usually something like `osg-25-main-el8-build` or `osg-24-upcoming-el9-build`;
             but you can find the exact build tag by clicking the Build Target link for the koji task,
             and whatever is listed for the Build Tag is the name of the repo to regen.
 
@@ -176,15 +187,9 @@ You may check the status of regen-repo tasks in the
                 :::console
                 $ osg-koji regen-repo <BUILD TAG>
 
-    -   `build.log`  
+    -   `build.log`<br>
         The output of the rpmbuild executable. If your package fails to compile, the reason will show up here.
 
--   One input to the buildArch task is a repository, which is based on a Koji tag. If the repository hasn't been recreated for a dependency you need (for example, when kojira isn't working), you may not have the right RPMs available in your build root.
-    -   One common issue is building a chain of dependencies. For example, suppose build B depends on the results of build A. If you build A then build B immediately, B will likely fail. This is because A is not in the repository that B uses. The proper string of events building A, starting the regeneration of the destination and build repo (which should happen in a few minutes of the build A task completing), then submitting build task B.
-
-        !!! note
-            if you submit build task B while the build repository task is open, it will not start until the build task has finished.
-
 - Other errors
 
     -   `package <PACKAGE NAME> not in list for tag <TAG>`<br/>
@@ -199,7 +204,7 @@ To promote from development to testing:
 
 #### Using *osg-promote*
 
-Before using `osg-promote`, [authenticate to Koji as above](#authenticating-to-koji).
+Before using `osg-promote`, authenticate to Koji using `kinit` and your Kerberos principal (same as for `osg-build`).
 
 If you want to promote the latest version:
 
@@ -217,11 +222,13 @@ If you want to promote a specific version:
 &lt;BUILD NAME&gt; is a full `name-version-revision.disttag` such as `gratia-probe-1.17.0-2.osg33.el6`.
 
 
-&lt;OSGVER&gt; is the OSG major version that you are promoting for (e.g. `3.4`).
+&lt;OSGVER&gt; is the OSG major version that you are promoting for (e.g. `24`).
 
-`osg-promote` will promote both the el6 and el7 builds of a package. After promoting, copy and paste the JIRA code `osg-promote` produces into the JIRA ticket that you are working on.
+`osg-promote` will promote the builds of a package for all distro versions (el8, el9, etc.).
+After promoting, copy and paste the Jira code `osg-promote` produces into the Jira ticket that you are working on.
 
- For `osg-promote`, you may omit the `.osg34.el6` or `.osg34.el7`; the script will add the appropriate disttag on.
+For `osg-promote`, you may omit the dist tag (e.g. `.osg25.el9`);
+the script will add the appropriate disttag on.
 
 See [OSG Build Tools](../software/osg-build-tools.md) for full details on `osg-promote`.