Add joblog label to job

CONFIG.md

@@ -46,9 +46,38 @@ For Kubernetes, it is important to set resource limits.
 TODO: explain how to set limits, with default, project and spider specificity.
 
 ### [joblogs] section
+
+The joblogs section is used to configure the job logs feature. It is not enabled by default, but can be activated if you
+choose to use it. The job logs feature allows you to collect logs from the Kubernetes cluster and store them in a specified
+directory. The logs are collected from the pods running on the cluster and can be compressed using a specified method.
+If the log file was successfully uploaded to the remote storage, the pod is labeled with `org.scrapy.log_file_uploaded: true`.
+The job logs feature is implemented only for Kubernetes and is not available for Docker.
+
+The joblogs section contains the following parameters:
+
   * `logs_dir`     - a directory to store log files collected on k8s cluster (implemented only for Kubernetes). If you are using a Persistent Volume, keep in mind, that the provided path should be mounted in the deployment manifest. Read and write permissions should be granted to allow actions with log files in the provided directory, thus a [securityContext](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/) was added to the deployment manifest.
   * `compression_method` - a method to compress log files. Available options are `gzip` `bzip2`, `lzma`, `brotli` and `none`. If no compression_method is provided, it defaults to `none`.
 
+If you would like to provide a `compression_method` parameter, please, read the dedicated section below.
+
+### compression for joblogs
+
+When the log file was uploaded to the specified storage, the pod is labeled with org.scrapy.log_file_uploaded=true, it also
+contains labels to specify an end location of the log files in the storage.
+Since the label field has limitation in the number of symbols, an end location is split into several labels, each of them
+contains a part of the end location. The labels are named as follows:
+  * org.scrapy.project
+  * org.scrapy.spider
+  * org.scrapy.job_id
+  * org.scrapy.extension
+  * org.scrapy.compression
+
+How can you build the end location from these labels? Complete the following line
+  * logs/`org.scrapy.project`/`org.scrapy.spider`/`org.scrapy.job_id`.`org.scrapy.extension`.`org.scrapy.compression` 
+
+**Important**: if you did not use the `compression_method` parameter, the label `org.scrapy.compression` has `none` value
+so the final destination of the log file will be:
+  * logs/`org.scrapy.project`/`org.scrapy.spider`/`org.scrapy.job_id`.`org.scrapy.extension`
 
 ### Kubernetes API interaction
 

kubernetes.yaml

@@ -166,6 +166,7 @@ metadata:
 rules:
   - apiGroups: [""]
     resources: ["pods"]
+    # add "patch" if you use joblogs feature
     verbs: ["get", "list", "watch"]
   - apiGroups: [""]
     resources: ["pods/exec"]

scrapyd_k8s/joblogs/log_handler_k8s.py

@@ -84,6 +84,58 @@ def __init__(self, config):
             raise ValueError("Configuration error: 'logs_dir' is missing in joblogs configuration section.")
         self.object_storage_provider = LibcloudObjectStorage(self.config)
 
+    def _add_object_name_label(self, pod, object_name):
+        """
+        Adds file metadata as labels to the pod.
+
+        Parameters
+        ----------
+        pod : V1Pod
+            The Kubernetes pod object.
+        object_name : str
+            The object name in storage.
+
+        Returns
+        -------
+        bool
+            True if labels were added successfully, False otherwise.
+        """
+        try:
+            basename = os.path.basename(object_name)
+
+            extension = "none"
+            compression = "none"
+
+            if basename.endswith(".log"):
+                extension = "log"
+            elif ".log." in basename:
+                extension = "log"
+                compression = basename.rsplit(".log.", 1)[1]
+
+            patch_body = {
+                "metadata": {
+                    "labels": {
+                        "org.scrapy.extension": extension,
+                        "org.scrapy.compression": compression,
+                        "org.scrapy.log_file_uploaded": "true"
+                    }
+                }
+            }
+
+            v1 = client.CoreV1Api()
+            v1.patch_namespaced_pod(
+                name=pod.metadata.name,
+                namespace=self.namespace,
+                body=patch_body
+            )
+
+            logger.info(f"Added labels to pod '{pod.metadata.name}': extension='{extension}', compression='{compression}', uploaded='true'")
+            return True
+
+        except Exception as e:
+            logger.error(f"Failed to add labels to pod '{pod.metadata.name}': {e}")
+            return False
+
     def get_existing_log_filename(self, job_id):
         """
         Retrieves the existing temporary log file path for a job without creating a new one.
@@ -322,7 +374,12 @@ def handle_events(self, event):
                                 logger.info(
                                     f"Removed local log file '{log_filename}' since it already exists in storage.")
                         else:
-                            self.object_storage_provider.upload_file(project, spider, log_filename)
+                            object_name = self.object_storage_provider.upload_file(project, spider, log_filename)
+                            if object_name:
+                                try:
+                                    self._add_object_name_label(pod, object_name)
+                                except Exception as e:
+                                    logger.error(f"Exception while adding label for job '{job_id}': {e}")
                             os.remove(log_filename)
                             logger.info(f"Removed local log file '{log_filename}' after successful upload.")
                     else:

scrapyd_k8s/object_storage/libcloud_driver.py

@@ -133,7 +133,8 @@ def upload_file(self, project, spider, local_path):
 
         Returns
         -------
-        None
+        str or None
+            The object name in storage if upload is successful, None otherwise.
 
         Logs
         ----
@@ -172,6 +173,10 @@ def upload_file(self, project, spider, local_path):
                     f"Successfully uploaded compressed file '{object_name}' to container '{self._container_name}'.")
             else:
                 logger.info(f"Successfully uploaded file '{object_name}' to container '{self._container_name}'.")
+
+            # Return object_name on successful upload
+            return object_name
+
         except (ObjectError, ContainerDoesNotExistError, InvalidContainerNameError) as e:
             logger.exception(f"Error uploading the file '{object_name}': {e}")
         except Exception as e: