Adding the generic_package.sh script file by Shwetha-Acharya · Pull Request #44 · gluster/build-jobs

Shwetha-Acharya · 2020-10-07T14:58:14Z

This script file is used to build the rpms for
Debian and Ubuntu versions.

We can run this script on a ubuntu machine.
We create a chroot environment using pbuilder create.
This gives us the environment for both ubuntu and debian.
On this we build, sign and push the packages.

We update the changelogs and push them to the glusterfs-debian
repo as well.

Signed-off-by: Sheetal Pamecha spamecha@redhat.com
Signed-off-by: hari gowtham hgowtham@redhat.com
Signed-off-by: Shwetha K Acharya sacharya@redhat.com

This script file is used to build the rpms for Debian and Ubuntu versions. We can run this script on a ubuntu machine. We create a chroot environment using pbuilder create. This gives us the environment for both ubuntu and debian. On this we build, sign and push the packages. We update the changelogs and push them to the glusterfs-debian repo as well. Signed-off-by: Sheetal Pamecha <spamecha@redhat.com> Signed-off-by: hari gowtham <hgowtham@redhat.com> Signed-off-by: Shwetha K Acharya <sacharya@redhat.com>

This helps in automating the packaging task of Gluster for Debian and Ubuntu. Change-Id: Icdbc718f0a3f020715d959486d01e962f89fa80f Signed-off-by: hari gowtham <hgowtham@redhat.com> Signed-off-by: Shwetha K Acharya <sacharya@redhat.com>

mscherer · 2020-10-21T15:11:40Z

build-gluster-org/jobs/debian-ubuntu-package.yml

+    - string:
+        default:
+        description: Release number for the package to be built against.
+          Leave it empty if you are building above series 5.


Mhh, what is series 5, you mean version 5 branch ? is it still supported, or can we drp it ?

We can drop them off.

So, the comment is still here, what is series 5 ? Do you mean "release 5" ? (and if we can drop, should it be removed from the description ?)

build-gluster-org/scripts/debian-ubuntu-package.sh

mscherer · 2020-10-21T15:15:33Z

build-gluster-org/scripts/debian-ubuntu-package.sh

+        ../scripts/generic_package.sh ubuntu focal $SERIES $VERSION $RELEASE
+    elif [ "$OS" == "debian" ]; then
+        echo "packing debian alone"
+        if [ "$FLAVOR" == "stretch" ] || [ "$FLAVOR" == "9" ]; then


I think it would be cleaner to have 1 single if with the 6 possibles values than 3 separate ones.

can you please elaborate this comment

Well, here we have 3 lines with if, they all run the same script with the same argument, the only difference is the echo. This seems harder to read while :

case $FLAVOR in
"stretch" | "9" | "buster" | "10" | "bullseye" | "11")
../scripts/generic_package.sh $OS $FLAVOR $SERIES $VERSION $RELEASE $LATEST_SERIES $LATEST_VERSION

is shorter

mscherer · 2020-10-21T15:18:09Z

build-gluster-org/scripts/generic_package.sh

+tar czf ~/${os}-${flavor}-Glusterfs-${version}/${flavor}-apt-amd64-${version}.tgz pool/ dists/
+
+echo "Pushing Changelog changes.."
+git push origin ${flavor}-${series}-local:${flavor}-glusterfs-${series}


This is not going to work if the clone is done over HTTP

I am sorry, but I still see https for the clone. I was not precise enough, I want to point that we can't push over http, including https. And if we need to push a new changelog, we need a ssh key.

clone is now changed to ssh

build-gluster-org/scripts/generic_package.sh

mscherer · 2020-10-21T15:20:32Z

build-gluster-org/scripts/generic_package.sh

+sudo pbuilder build ~/${os}-${flavor}-Glusterfs-${version}/build/glusterfs_${version}-${release}.dsc | tee build.log
+
+#move the packages to packages directory.
+mv /var/cache/pbuilder/result/glusterfs*${version}-${release}*.deb ~/${os}-${flavor}-Glusterfs-${version}/packages/


I am not sure jenkins used can erase file in /var/cache , so mv would fail.

@kalebskeithley Any suggestions on this?

I think it can safely replaced by "cp", at least.

Seems to still be mv :/

But these files should be removed to avoid eventual filling up of space, How is such case handled generally with jenkins? @mscherer

mscherer · 2020-10-21T15:29:54Z

build-gluster-org/scripts/generic_package.sh

+echo "Uploading the packages.."
+if [ "$os" == "ubuntu" ]; then
+        cd ..
+        dput ppa:gluster/glusterfs-${series} glusterfs_${version}-${os}1~${flavor}1_source.changes


Does it requires some authentication ? I do not see how it is done.

No, AIFIK. @kalebskeithley any more insights?

Well, if could be without authentication, but then, we need some kind of signature, cause I do not think anyone can push debian package for us, no ?

yes, we would need to follow https://help.launchpad.net/Packaging/PPA/Uploading to get required authentiacation

The documentation do not tell much, but if that use ssh, then we need to have the key available to the builder, which is not declared in the job yaml file. There is example here: https://github.com/gluster/build-jobs/blob/master/build-gluster-org/jobs/centos7-regression.yml#L64 for adding a credential

The OpenPGP keys are used for signing as described here
The above link also shows how a new key can be created and added to launcpad account.

$ gpg --list-keys on the machine rhs-vm-17.storage-dev.lab.eng.bos.redhat.com under the user glusterpackager, shows the already existing keys. (documented the same in the mojo doc draft: let me know if any more information is needed to be documented)

But that's a internal VM, and Jenkins is hosted outside of the lan. I may miss something obvious, but Jenkins can't connect to that server, and I think Product Security would strongly dislike that a external server (build.gluster.org) is able to remotely execute any code on a internal system ( rhs-vm-17.storage-dev.lab.eng.bos.redhat.com ).

So we need to have the key as secret in the job, and store that in Jenkins. We can't use RH internal system.

yes, we can store the key as secret in the job. We can add it in jenkins machine, I can help in getting it from rhs-vm-17.storage-dev.lab.eng.bos.redhat.com

mscherer · 2020-10-21T15:36:55Z

build-gluster-org/scripts/generic_package.sh

+
+echo "Building source package.."
+cd ../glusterfs-${version}
+debuild -S -sa -k${debuild_key}


Where would the signing key stored ? (since -k requires that)

debuild_key variable contains the key.

I wasn't clear on my question. From what I understand, -k is the key identifier. But the actual private key is somewhere else (I think in ~/.gnupg), and so we need to discuss how that part is going to be managed.

we are initialising the values for debuild keys in https://github.com/gluster/build-jobs/pull/44/files/b32e0625d90b8c89432e753aab502d5e72b64b46#diff-02d2c32f5282eea2a7412f831420ae8c2f55ce0ed17671f6878e110c258b2e6bR29

The key creation is something we have maintained in a particular machine, whose access is not given to wider audience.

Then, where is that machine ?

Here is the draft of mojo doc that contains machine details: https://source.redhat.com/.motion/communitiesatredhat/crosscuttingco/pjmcop/prodev/study_grp/study_group_sign_up/A883E22D-0D6C-EB11-80F2-000D3A020FEB/.object

Page does not exist :/

Please checkout this link: https://source.redhat.com/.motion/communitiesatredhat/crosscuttingco/pjmcop/pjm_prodev/pjm_study_grp/pjm_study_group_sign_up/a883e22d-0d6c-eb11-80f2-000d3a020feb/.object

build-gluster-org/scripts/generic_package.sh

mscherer · 2020-10-21T15:51:20Z

build-gluster-org/scripts/generic_package.sh

+
+cd /var/www/repos/apt/debian/
+
+rm -rf pool/* dists/* db/*


Not sure jenkins can erase file there.

We can have a trail run to verify it.

Also, if we create file here, shouldn't it be cleaned with a trap, like the rest ?

yes, can be done

build-gluster-org/scripts/generic_package.sh

This helps in automating the packaging task of Gluster for Debian and Ubuntu. Change-Id: Icdbc718f0a3f020715d959486d01e962f89fa80f Signed-off-by: hari gowtham <hgowtham@redhat.com> Signed-off-by: Shwetha K Acharya <sacharya@redhat.com>

update mv to cp Signed-off-by: Shwetha K Acharya <sacharya@redhat.com>

Signed-off-by: Shwetha K Acharya <sacharya@redhat.com>

Shwetha-Acharya · 2021-02-04T13:14:42Z

Requesting next round of review @mscherer

mscherer · 2021-02-09T14:29:01Z

build-gluster-org/scripts/generic_package.sh

+  #removing folders created while packaging
+  rm -rf ~/${os}-${flavor}-Glusterfs-${version}
+}
+trap finish EXIT


Traps should be placed at the start of the script. Otherwise, it be used only when the script is over, which is not useful.

mscherer · 2021-02-09T14:29:31Z

build-gluster-org/scripts/generic_package.sh

+#copy the tar.gz file produced by the build to download.rht.gluster.org:/var/www/scratch
+scp $flavor-apt-amd64-$version.tgz gluster_ant@download.rht.gluster.org:/var/www/scratch
+
+ssh gluster_ant@download.rht.gluster.org /var/www/html/pub/gluster/unpacking-script.sh series version os flavor latest_version latest_series


Shouldn't it be "$version", etc ?

mscherer · 2021-02-09T14:30:21Z

build-gluster-org/scripts/generic_package.sh

+
+#move the packages to packages directory.
+cp /var/cache/pbuilder/result/glusterfs*${version}-${release}*.deb ~/${os}-${flavor}-Glusterfs-${version}/packages/
+rm -rf /var/cache/pbuilder/result/glusterfs*${version}-${release}*.deb


Why not mv instead of cp + rm ?

mscherer · 2021-02-09T15:12:25Z

build-gluster-org/scripts/generic_package.sh

+cp /var/cache/pbuilder/result/glusterfs*${version}-${release}*.deb ~/${os}-${flavor}-Glusterfs-${version}/packages/
+rm -rf /var/cache/pbuilder/result/glusterfs*${version}-${release}*.deb
+
+if [ "$flavor" != "stretch" ]; then


So, why is stretch special here ?

It is because /var/cache/pbuilder/result/libg*${version}-${release}*.deb are not created only in stretch. In buster and bullseye, they will be created

replace cp+rm with mv add misisng $ Signed-off-by: Shwetha K Acharya <sacharya@redhat.com>

Signed-off-by: Shwetha K Acharya <sacharya@redhat.com>

Shwetha-Acharya · 2021-02-11T06:38:10Z

@mscherer all the requested change sets are uploaded.

mscherer · 2021-03-09T10:22:54Z

So, since the review is getting a bit messy with comments, the blocking part is the gpg key secret integration for now.

mscherer · 2021-03-09T10:23:13Z

(and conflicts)

deepshikhaaa · 2021-05-20T11:18:00Z

build-gluster-org/jobs/debian-ubuntu-package.yml

+        name: ANNOUNCE_EMAIL
+
+    builders:
+    - shell: /opt/qa/debian-ubuntu-package.sh


Location is not correct. You have added the file in build-gluster-org/scripts/debian-ubuntu-package.sh

according to the actual loaction of the script Signed-off-by: Shwetha K Acharya <sacharya@redhat.com>

deepshikhaaa · 2021-05-20T12:23:42Z

build-gluster-org/scripts/debian-ubuntu-package.sh

+        echo "building everything"
+        echo "packing debian distribution"
+        for i in ${!deb_flavors[@]}; do
+        ~/build-gluster-org/scripts/generic_package.sh debian ${deb_flavors[$i]} $SERIES $VERSION $RELEASE $LATEST_SERIES $LATEST_VERSION


This location is wrong here. It should be ${WORKSPACE}/build-gluster-org/scripts/generic_package.sh

Signed-off-by: Shwetha K Acharya <sacharya@redhat.com>

Shwetha-Acharya · 2021-06-01T10:41:42Z

So, since the review is getting a bit messy with comments, the blocking part is the gpg key secret integration for now.

Action plan to resolve gpg key integration part: https://docs.google.com/document/d/1KRuzMC9zPD6K-5k03_5vUGjWrR0kUFiN2UhaYm18phQ/edit?usp=sharing

Also as agreed earlier, infra team is upposed to create the following users for these scripts to work:

user named gluster_ant in the build machine, who has permission to access download.rht.gluster.org , so that
built packages can be pushed to download.rht.gluster.org
github user so that script can push the changelogs to respective github reposotories.

deepshikhaaa · 2021-06-02T11:35:25Z

build-gluster-org/jobs/debian-ubuntu-package.yml

@@ -0,0 +1,57 @@
+- job:
+    name: debian-package-builder
+    node: master


Use the label debian10 for this job. We have a machine existing on jenkins https://build.gluster.org/computer/builder-deb10-1.int.rht.gluster.org/

@deepshikhaaa done!

@deepshikhaaa please validate the liburing-devel support as well.

https://build.gluster.org/computer/builder-deb10-1.int.rht.gluster.org/ Signed-off-by: Shwetha K Acharya <sacharya@redhat.com>

Shwetha-Acharya · 2021-07-08T07:51:45Z

We need a machaine with kernel 5.1 or above inorder to enable io-uring support with the builds. Please let me know which node can be used.

As mentioned earlier there is an Action plan to resolve gpg key integration part.

@mscherer @deepshikhaaa can we take this forward?
Going forward it is very much essential to keep this packaging and building part automated.

mscherer · 2021-07-15T07:43:33Z

We only have Debian 10 hosts, not Ubuntu one. I can try to spin a Ubuntu one but that would be the 1st we have, so I am unsure how long it will take.

mscherer · 2021-07-15T11:01:03Z

Ok so the Ubuntu installer do not work (and I do not understand why yet, there is no obvious error message). But so, do we need the kernel to be up to date for package building ? Given this will use a pbuilder, the headers would be there and nothing more should be needed, unless I am missing something ?

mscherer · 2021-07-15T15:03:12Z

So the Ubuntu installer didn't work because:

the firewall blocked the network (each time, i forget)
some weird ansible issue with variable (so it was installing centos), I fixed
then ubuntu changed the installer and so our RHEL 7 host is too old to detect that

So I am doing 18.04 and upgrade manually.

Shwetha-Acharya · 2021-07-16T10:07:36Z

Ok so the Ubuntu installer do not work (and I do not understand why yet, there is no obvious error message). But so, do we need the kernel to be up to date for package building ? Given this will use a pbuilder, the headers would be there and nothing more should be needed, unless I am missing something ?

Right, in this script the latest chroot will be created. We would require a ubuntu mechine as discussed offline as it is tested in ubuntu.

mscherer · 2021-07-20T12:30:32Z

But testing is done on a different job than the packaging one, no ? I still do not get the issue, at what point is the deb packages tested, and how ?

mscherer · 2021-07-20T18:53:14Z

So we have 20.04 builder now: https://build.gluster.org/computer/builder-ubuntu2004-1.int.rht.gluster.org/

(https://build.gluster.org/computer/builder-ubuntu2004-1.int.rht.gluster.org/) Signed-off-by: Shwetha K Acharya <sacharya@redhat.com>

Shwetha-Acharya added 2 commits October 7, 2020 20:03

Adding package job and script.

b32e062

This helps in automating the packaging task of Gluster for Debian and Ubuntu. Change-Id: Icdbc718f0a3f020715d959486d01e962f89fa80f Signed-off-by: hari gowtham <hgowtham@redhat.com> Signed-off-by: Shwetha K Acharya <sacharya@redhat.com>

mscherer requested changes Oct 21, 2020

View reviewed changes

mscherer reviewed Oct 21, 2020

View reviewed changes

build-gluster-org/scripts/generic_package.sh Outdated Show resolved Hide resolved

mscherer reviewed Oct 21, 2020

View reviewed changes

build-gluster-org/scripts/generic_package.sh Outdated Show resolved Hide resolved

mscherer reviewed Oct 21, 2020

View reviewed changes

build-gluster-org/scripts/generic_package.sh Show resolved Hide resolved

mscherer reviewed Oct 21, 2020

View reviewed changes

build-gluster-org/scripts/generic_package.sh Show resolved Hide resolved

Shwetha-Acharya added 7 commits November 2, 2020 14:03

Adding package job and script.

d99d9aa

This helps in automating the packaging task of Gluster for Debian and Ubuntu. Change-Id: Icdbc718f0a3f020715d959486d01e962f89fa80f Signed-off-by: hari gowtham <hgowtham@redhat.com> Signed-off-by: Shwetha K Acharya <sacharya@redhat.com>

Adding package job and script.

cf63851

This helps in automating the packaging task of Gluster for Debian and Ubuntu. Change-Id: Icdbc718f0a3f020715d959486d01e962f89fa80f Signed-off-by: hari gowtham <hgowtham@redhat.com> Signed-off-by: Shwetha K Acharya <sacharya@redhat.com>

Merge branch 'master' of https://github.com/Shwetha-Acharya/build-jobs

06dcc7e

Add code to execute script inside download.rht.gluster.org

1d0f57a

update mv to cp Signed-off-by: Shwetha K Acharya <sacharya@redhat.com>

Update the path for clean up

a71a40a

Signed-off-by: Shwetha K Acharya <sacharya@redhat.com>

Add loop instead of if

9d1a6e5

Signed-off-by: Shwetha K Acharya <sacharya@redhat.com>

Change http to https during clone

8286f13

Signed-off-by: Shwetha K Acharya <sacharya@redhat.com>

mscherer reviewed Feb 9, 2021

View reviewed changes

Shwetha-Acharya added 4 commits February 11, 2021 06:36

address issue with trap trap

3c10df6

replace cp+rm with mv add misisng $ Signed-off-by: Shwetha K Acharya <sacharya@redhat.com>

use ssh for cloning from github

4d1973a

Signed-off-by: Shwetha K Acharya <sacharya@redhat.com>

refactor loops

6550676

Signed-off-by: Shwetha K Acharya <sacharya@redhat.com>

remove comment with respect to series 5

0832088

Signed-off-by: Shwetha K Acharya <sacharya@redhat.com>

Base automatically changed from master to main February 17, 2021 09:51

deepshikhaaa reviewed May 20, 2021

View reviewed changes

modify shell path in debian-ubuntu-package.yml

1174edb

according to the actual loaction of the script Signed-off-by: Shwetha K Acharya <sacharya@redhat.com>

deepshikhaaa reviewed May 20, 2021

View reviewed changes

Alter path with respect to ${WORKSPACE}

33bb9d9

Signed-off-by: Shwetha K Acharya <sacharya@redhat.com>

Shwetha-Acharya force-pushed the master branch from dfcd075 to 33bb9d9 Compare May 24, 2021 05:20

Create packages inside the directory wrt ${WORKSPACE}

d9afc16

Signed-off-by: Shwetha K Acharya <sacharya@redhat.com>

deepshikhaaa reviewed Jun 2, 2021

View reviewed changes

Add debian10 as the label for the node inorder to use

8bd2f4f

https://build.gluster.org/computer/builder-deb10-1.int.rht.gluster.org/ Signed-off-by: Shwetha K Acharya <sacharya@redhat.com>

update the node label to ubuntu2004

9b2c747

(https://build.gluster.org/computer/builder-ubuntu2004-1.int.rht.gluster.org/) Signed-off-by: Shwetha K Acharya <sacharya@redhat.com>

rakshithakamath94 force-pushed the main branch from db6ed4c to 891c0b7 Compare July 24, 2023 08:17

Conversation

Shwetha-Acharya commented Oct 7, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!