Linux inbound agent support (#406)

Author: timja

README.md

@@ -7,7 +7,7 @@ A Jenkins Plugin to create Jenkins agents in Azure Virtual Machines (via Azure A
 
 Supported features:
 
-1. Windows Agents on Azure Cloud using SSH and JNLP
+1. Windows Agents on Azure Cloud using SSH and Inbound agent connection
    * For Windows images to launch via SSH, the image needs to be preconfigured with SSH.
    * For preparing custom windows image, refer to [Azure documentation](http://azure.microsoft.com/en-us/documentation/articles/virtual-machines-capture-image-windows-server/)
 2. Linux Agents on Azure Cloud using SSH
@@ -88,21 +88,20 @@ The built-in image only has a clean Windows or Ubuntu OS and some tools like Git
    * Use a custom user image (provide image URL and OS type - note, your custom image has to be available into the same storage account in which you are going to create agent nodes);
    * Using any marketplace image by specifying an image reference (provide image reference by publisher, offer, sku and version). You can get the publisher, offer and sku by looking at the ARM template of that image.
 3. For the launch method, select SSH or JNLP.
-   * Linux agents can be launched only using SSH.
-   * Windows agents can be launched using SSH or JNLP. For Windows agents, if the launch method is SSH then check Pre-Install SSH in Windows Agent or image needs to be custom-prepared with an SSH server pre-installed.
+   * For Windows agents, if the launch method is SSH then check Pre-Install SSH in Windows Agent or image needs to be custom-prepared with an SSH server pre-installed.
 
-   We recommend to use SSH rather than JNLP, for you need less init codes and get much clearer logs.
-   > When using the JNLP launch option, ensure the following:
+   We recommend to use SSH rather than connecting it to the controller, it's simpler to setup and get much clearer logs.
+   > When using the Inbound agent launch option, ensure the following:
    > * Jenkins URL (Manage Jenkins -> Configure System -> Jenkins Location)
    > * The URL needs to be reachable by the Azure agent, so make sure to configure any relevant firewall rules accordingly.
-   > * TCP port for JNLP agent agents (Manage Jenkins -> Configure Global Security -> Enable security -> TCP port for JNLP agents).
-   > * The TCP port needs to be reachable from the Azure agent launched using JNLP. It is recommended to use a fixed port so that any necessary firewall exceptions can be made.
+   > * TCP port for Inbound agents (Manage Jenkins -> Configure Global Security -> Enable security -> TCP port for inbound agents).
+   > * The TCP port needs to be reachable from the Azure agent launched with connecting to the controller. It is recommended to use a fixed port so that any necessary firewall exceptions can be made.
    >
-   > If the Jenkins controller is running on Azure, then open an endpoint for "TCP port for JNLP agent agents" and,
+   > If the Jenkins controller is running on Azure, then open an endpoint for "TCP port for Inbound agents" and,
    > in case of Windows, add the necessary firewall rules inside virtual machine (Run -> firewall.cpl).
 
-4. For the Initialization Script, you can provide a script that will be executed after the VM is provisioned. This allows to install any app/tool you need on the agent. Please be noted you need to at least install JRE if the image does not have Java pre-installed.
-   We prepared a sample script for Linux via SSH, Windows via SSH and Windows via JNLP. Please find details in help button.
+4. For the Initialization Script, you can provide a script that will be executed after the VM is provisioned. This allows to install any app/tool you need on the agent. Please be aware you need to at least install JRE if the image does not have Java pre-installed.
+   We prepared a sample script for Linux via SSH, Windows via SSH and Windows via connecting to the controller. Please find details in help button.
 
    If you hit the [storage scalability limits](https://docs.microsoft.com/en-us/azure/storage/storage-scalability-targets) for your custom images on the storage account where the VHD resides, you should consider using the agent's [temporary storage](https://blogs.msdn.microsoft.com/mast/2013/12/06/understanding-the-temporary-drive-on-windows-azure-virtual-machines) or copy your custom image in multiple storage accounts and use multiple VM templates with the same label within the same agent cloud.
 

docs/init-scripts/linux-inbound-agent.sh

@@ -0,0 +1,24 @@
+#!/usr/bin/env bash
+
+JENKINS_URL=$1
+AGENT_NAME=$2
+SECRET=$3
+
+mkdir /opt/jenkins
+
+(
+  cd /opt/jenkins
+  apt-get update
+  apt-get install -y default-jdk git
+
+  MAVEN_VERSION=3.9.1
+  curl -O "https://archive.apache.org/dist/maven/maven-3/${MAVEN_VERSION}/binaries/apache-maven-${MAVEN_VERSION}-bin.tar.gz"
+  tar zxvf apache-maven-${MAVEN_VERSION}-bin.tar.gz -C /opt/
+  ln -s /opt/apache-maven-${MAVEN_VERSION}/bin/mvn /usr/bin/mvn
+
+  mvn --version
+
+  curl -O "$JENKINS_URL/jnlpJars/agent.jar"
+
+  java -jar agent.jar -jnlpUrl "$JENKINS_URL/computer/$AGENT_NAME/jenkins-agent.jnlp" -secret "$SECRET" -workDir /opt/jenkins/workDir
+) |& tee /opt/jenkins/init-script.log

docs/init-scripts/windows-inbound-agent.ps1

@@ -32,27 +32,28 @@ $jenkinsserverurl = $args[0]
 $vmname = $args[1]
 $secret = $args[2]
 
-# Downloading jenkins agent jar
-Write-Output "Downloading jenkins agent jar "
-$slaveSource = $jenkinsserverurl + "jnlpJars/agent.jar"
-$destSource = "C:\agent.jar"
+# Download the service wrapper
+$wrapperExec = "c:\jenkins\jenkins-agent.exe"
+$configFile = "c:\jenkins\jenkins-agent.xml"
+$agentSource = $jenkinsserverurl + "jnlpJars/agent.jar"
+
+mkdir C:\jenkins
 $wc = New-Object System.Net.WebClient
-$wc.DownloadFile($slaveSource, $destSource)
+$wc.DownloadFile("https://github.com/winsw/winsw/releases/download/v2.12.0/WinSW.NET461.exe", $wrapperExec)
+$wc.DownloadFile("https://raw.githubusercontent.com/Azure/jenkins/master/agents_scripts/jenkins-slave.exe.config", "c:\jenkins\jenkins-agent.exe.config")
+$wc.DownloadFile("https://raw.githubusercontent.com/Azure/jenkins/master/agents_scripts/jenkins-slave.xml", $configFile)
 
-# execute agent
+# Prepare config
 Write-Output "Executing agent process "
-$java="java"
-$jar="-jar"
-$serverURL=$jenkinsserverurl + "computer/" + $vmname + '/jenkins-agent.jnlp'
-
-# TODO look at porting the run as service part of the old script from https://raw.githubusercontent.com/Azure/jenkins/master/agents_scripts/Jenkins-Windows-Init-Script-Jnlp.ps1
-while ($true) {
-  try {
-    # Launch
-    & $java -jar $destSource -secret $secret -jnlpUrl $serverURL
-  }
-  catch [System.Exception] {
-    Write-Output $_.Exception.ToString()
-  }
-  Start-Sleep 10
+$configExec = "java"
+$configArgs = "-jnlpUrl `"${jenkinsserverurl}/computer/${vmname}/jenkins-agent.jnlp`" -workDir C:\jenkins\workDir"
+if ($secret) {
+    $configArgs += " -secret `"$secret`""
 }
+(Get-Content $configFile).replace('@JAVA@', $configExec) | Set-Content $configFile
+(Get-Content $configFile).replace('@ARGS@', $configArgs) | Set-Content $configFile
+(Get-Content $configFile).replace('@SLAVE_JAR_URL', $agentSource) | Set-Content $configFile
+
+# Install the service
+& $wrapperExec install
+& $wrapperExec start

src/main/java/com/microsoft/azure/vmagent/AzureVMAgentTemplate.java

@@ -735,7 +735,9 @@ public static String getBasicInitScript(AzureVMAgentTemplate template) {
                         AzureVMManagementServiceDelegate.PRE_INSTALLED_TOOLS_SCRIPT
                                 .get(builtInImage).get(Constants.INSTALL_GIT));
             }
-            if ((builtInImage.equals(Constants.UBUNTU_1604_LTS) || builtInImage.equals(Constants.UBUNTU_2004_LTS))
+            if ((builtInImage.equals(Constants.UBUNTU_1604_LTS)
+                   || builtInImage.equals(Constants.UBUNTU_2004_LTS)
+                   || builtInImage.equals(Constants.UBUNTU_2204_LTS))
                     && template.isInstallDocker()) {
                 stringBuilder.append(getSeparator(template.getOsType()));
                 stringBuilder.append(
@@ -1545,8 +1547,8 @@ public ListBoxModel doFillUsageModeItems() throws IOException, ServletException
 
         public ListBoxModel doFillAgentLaunchMethodItems() {
             ListBoxModel model = new ListBoxModel();
-            model.add(Constants.LAUNCH_METHOD_SSH);
-            model.add(Constants.LAUNCH_METHOD_JNLP);
+            model.add(Messages.AzureVMAgentTemplate_SSH_Agent_Launch_Method(), Constants.LAUNCH_METHOD_SSH);
+            model.add(Messages.AzureVMAgentTemplate_Inbound_Agent_Launch_Method(), Constants.LAUNCH_METHOD_JNLP);
 
             return model;
         }
@@ -1584,6 +1586,7 @@ public ListBoxModel doFillExistingStorageAccountNameItems(
 
         public ListBoxModel doFillBuiltInImageItems() {
             ListBoxModel model = new ListBoxModel();
+            model.add(Constants.UBUNTU_2204_LTS);
             model.add(Constants.UBUNTU_2004_LTS);
             model.add(Constants.UBUNTU_1604_LTS);
             model.add(Constants.WINDOWS_SERVER_2022);
@@ -1848,8 +1851,8 @@ public FormValidation doVerifyConfiguration(
 
             // First validate the subscription info.  If it is not correct,
             // then we can't validate the
-            String result = AzureClientHolder.getDelegate(azureCredentialsId).verifyConfiguration(resourceGroupName,
-                    maxVirtualMachinesLimit, deploymentTimeout);
+            String result = AzureClientHolder.getDelegate(azureCredentialsId)
+                    .verifyConfiguration(resourceGroupName, deploymentTimeout);
             if (!result.equals(Constants.OP_SUCCESS)) {
                 return FormValidation.error(result);
             }

src/main/java/com/microsoft/azure/vmagent/AzureVMCloud.java

@@ -1108,7 +1108,6 @@ public String getDefaultResourceGroupName() {
         @RequirePOST
         public FormValidation doVerifyConfiguration(
                 @QueryParameter String azureCredentialsId,
-                @QueryParameter String maxVirtualMachinesLimit,
                 @QueryParameter String deploymentTimeout,
                 @QueryParameter String resourceGroupReferenceType,
                 @QueryParameter String newResourceGroupName,
@@ -1123,7 +1122,7 @@ public FormValidation doVerifyConfiguration(
             AzureResourceManager azureClient = AzureResourceManagerCache.get(azureCredentialsId);
             final String validationResult = AzureVMManagementServiceDelegate
                     .getInstance(azureClient, azureCredentialsId)
-                    .verifyConfiguration(resourceGroupName, maxVirtualMachinesLimit, deploymentTimeout);
+                    .verifyConfiguration(resourceGroupName, deploymentTimeout);
             if (!validationResult.equalsIgnoreCase(Constants.OP_SUCCESS)) {
                 return FormValidation.error(validationResult);
             }

src/main/java/com/microsoft/azure/vmagent/AzureVMCloudVerificationTask.java

@@ -170,7 +170,6 @@ public static boolean verifyConfiguration(AzureVMCloud cloud) {
         // Check the sub and off we go
         String result = cloud.getServiceDelegate().verifyConfiguration(
                 cloud.getResourceGroupName(),
-                Integer.toString(cloud.getMaxVirtualMachinesLimit()),
                 Integer.toString(cloud.getDeploymentTimeout()));
         if (!Constants.OP_SUCCESS.equals(result)) {
             LOGGER.log(getStaticNormalLoggingLevel(), "AzureVMCloudVerificationTask: verifyConfiguration: {0}", result);

src/main/java/com/microsoft/azure/vmagent/AzureVMManagementServiceDelegate.java

@@ -264,11 +264,17 @@ public AzureVMDeploymentInfo createDeployment(
                     && (isBasic || referenceType == ImageReferenceType.REFERENCE
                     || template.isPreInstallSsh());
 
-            final boolean useCustomScriptExtension
+            boolean windows = properties.get("osType").equals(Constants.OS_TYPE_WINDOWS);
+            boolean inboundAgent = properties.get("agentLaunchMethod").equals(Constants.LAUNCH_METHOD_JNLP);
+            boolean useCustomScriptExtension
                     = preInstallSshInWindows
-                    || properties.get("osType").equals(Constants.OS_TYPE_WINDOWS)
-                    && !StringUtils.isBlank((String) properties.get("initScript"))
-                    && properties.get("agentLaunchMethod").equals(Constants.LAUNCH_METHOD_JNLP);
+                    || windows
+                    && StringUtils.isNotBlank((String) properties.get("initScript"))
+                    && inboundAgent;
+
+            if (!useCustomScriptExtension && !windows && inboundAgent) {
+                useCustomScriptExtension = true;
+            }
 
             // check if a custom image id has been provided otherwise work with publisher and offer
             boolean useManagedDisk = diskType.equals(Constants.DISK_MANAGED);
@@ -316,6 +322,17 @@ public AzureVMDeploymentInfo createDeployment(
 
             final JsonNode tmp = MAPPER.readTree(embeddedTemplate);
 
+            if (useCustomScriptExtension) {
+                ArrayNode resources = (ArrayNode) tmp.get("resources");
+                for (JsonNode resource : resources) {
+                    String type = resource.get("type").asText();
+                    if (type.contains("virtualMachine")) {
+                        ArrayNode vmResources = (ArrayNode) resource.get("resources");
+                        vmResources.remove(windows ? 1 : 0);
+                    }
+                }
+            }
+
             // Add count variable for loop....
             final ObjectNode count = MAPPER.createObjectNode();
             count.put("type", "int");
@@ -1450,6 +1467,8 @@ private static Map<String, Map<String, String>> getDefaultImageProperties() {
                 imageProperties("Canonical", "UbuntuServer", "16.04-LTS", "16.04-LTS", Constants.OS_TYPE_LINUX));
         imageProperties.put(Constants.UBUNTU_2004_LTS,
                 imageProperties("canonical", "0001-com-ubuntu-server-focal", "20_04-lts-gen2", "20_04-lts-gen2", Constants.OS_TYPE_LINUX));
+        imageProperties.put(Constants.UBUNTU_2204_LTS,
+                imageProperties("canonical", "0001-com-ubuntu-server-jammy", "22_04-lts-gen2", "22_04-lts-gen2", Constants.OS_TYPE_LINUX));
 
         return imageProperties;
     }
@@ -1479,12 +1498,14 @@ private static Map<String, Map<String, String>> getPreInstalledToolsScript() {
         tools.put(Constants.WINDOWS_SERVER_2022, new HashMap<>());
         tools.put(Constants.UBUNTU_1604_LTS, new HashMap<>());
         tools.put(Constants.UBUNTU_2004_LTS, new HashMap<>());
+        tools.put(Constants.UBUNTU_2204_LTS, new HashMap<>());
         try {
             windows(Constants.WINDOWS_SERVER_2016, tools);
             windows(Constants.WINDOWS_SERVER_2019, tools);
             windows(Constants.WINDOWS_SERVER_2022, tools);
             ubuntu(Constants.UBUNTU_1604_LTS, tools);
             ubuntu(Constants.UBUNTU_2004_LTS, tools);
+            ubuntu(Constants.UBUNTU_2204_LTS, tools);
 
             return tools;
         } catch (IOException e) {
@@ -1575,13 +1596,11 @@ public Set<String> getVMSizes(String location) {
      * Validates certificate configuration.
      *
      * @param resourceGroupName Resource group name.
-     * @param maxVMLimit        Max limit of virtual machines.
      * @param timeOut           Timeout of the verification.
      * @return Verification result.
      */
     public String verifyConfiguration(
             String resourceGroupName,
-            String maxVMLimit,
             String timeOut) {
         try {
             if (!AzureUtil.isValidTimeOut(timeOut)) {
@@ -1593,11 +1612,7 @@ public String verifyConfiguration(
                 return "Error: " + Messages.Azure_GC_Template_ResourceGroupName_Err();
             }
 
-            if (!AzureUtil.isValidMAxVMLimit(maxVMLimit)) {
-                return "Invalid Limit, Should be a positive number, e.g. " + Constants.DEFAULT_MAX_VM_LIMIT;
-            }
-
-            if (!(AzureUtil.isValidTimeOut(timeOut) && AzureUtil.isValidMAxVMLimit(maxVMLimit)
+            if (!(AzureUtil.isValidTimeOut(timeOut)
                     && AzureUtil.isValidResourceGroupName(resourceGroupName))) {
                     return Messages.Azure_GC_Template_Val_Profile_Err();
             }

src/main/java/com/microsoft/azure/vmagent/util/AzureUtil.java

@@ -426,19 +426,6 @@ public static boolean isValidResourceGroupName(String resourceGroupName) {
         return false;
     }
 
-    /**
-     * Checks if the maximum virtual machines limit is valid.
-     *
-     * @param maxVMLimit Maximum Virtual Limit
-     * @return true if it is valid else return false
-     */
-    public static boolean isValidMAxVMLimit(String maxVMLimit) {
-        if (StringUtils.isBlank(maxVMLimit) || !maxVMLimit.matches(Constants.REG_EX_DIGIT)) {
-            return false;
-        }
-        return true;
-    }
-
     /**
      * Checks if the deployment Timeout is valid.
      *

src/main/java/com/microsoft/azure/vmagent/util/Constants.java

@@ -121,6 +121,7 @@ public final class Constants {
     public static final String WINDOWS_SERVER_2016 = "Windows Server 2016";
     public static final String UBUNTU_1604_LTS = "Ubuntu 16.04 LTS";
     public static final String UBUNTU_2004_LTS = "Ubuntu 20.04 LTS";
+    public static final String UBUNTU_2204_LTS = "Ubuntu 22.04 LTS";
 
     /**
      * ResourceGroup reference type.

src/main/resources/com/microsoft/azure/vmagent/Messages.properties

@@ -1,8 +1,7 @@
 # Global configuration - validations
 Azure_Config_Success=Successfully verified Azure configuration.
 Azure_GC_InitScript_Warn_Msg=Ensure image is pre-configured with a Java runtime or provide a script \
-  to install Java in headless (silent) mode. \
-                             \nIf using JNLP, see README.md for a sample script.
+  to install Java in headless (silent) mode.
 Azure_GC_LaunchMethod_Warn_Msg=Make sure the Azure agent can reach the controller via the Jenkins URL. \
   Refer to the help for details.
 Azure_GC_TemplateStatus_Warn_Msg=The template is marked as disabled. Check the template status details in \
@@ -65,7 +64,6 @@ Azure_GC_Template_ImageID_Not_Valid=The provided Image ID does not exist
 Azure_GC_Template_Gallery_Image_Not_Found=The target gallery image does not exist
 Azure_GC_Template_ImageURI_Not_In_Same_Account=The image URI is not located in the same storage account as the target \
   storage account for the VM
-Azure_GC_Template_JNLP_Not_Supported=The JNLP launch method is supported only for Windows.
 Azure_GC_Template_UN_Null_Or_Empty=Missing admin user name.
 Azure_GC_Template_PWD_Null_Or_Empty=Missing admin password.
 Azure_GC_Template_PWD_Not_Valid=Required: Not a valid password. The password length must be between 12 and 123 \
@@ -103,3 +101,7 @@ SA_Blank_Create_New=(Leave blank to create a new storage account)
 Azure_VM_Agent_Attach_Public_IP_Success=Successfully attached a public IP
 Azure_VM_Agent_Attach_Public_IP_Failure=Failed to attach a public IP
 Azure_GC_Template_NSG_NotFound=The Network Security Group {0} does not exist in the Resource Group.
+
+AzureVMAgentTemplate.Inbound_Agent_Launch_Method=Launch agent by connecting it to the controller
+AzureVMAgentTemplate.SSH_Agent_Launch_Method=Launch agents via SSH
+