Skip to content

README.md

Latest commit

 

History

History
238 lines (169 loc) · 6.51 KB

README.md

File metadata and controls

238 lines (169 loc) · 6.51 KB
Raw

SSHPC

SSHPC (Secure Shell Processing Cluster) is a lightweight distributed processing system built entirely in Bash and based on the SSH protocol. It was developed at the LAMBDA Laboratory (UFPB) to increase the computational capacity available for large-scale bioinformatics workloads, particularly virtual screening and molecular docking experiments.

The system transforms idle Linux workstations into processing nodes that can be dynamically used by a central server, creating a scalable and secure distributed computing environment without requiring external dependencies or specialized cluster software. Manages client-server pairing

Motivation

Virtual screening and molecular docking are widely used in modern drug discovery pipelines. These approaches require the evaluation of thousands or millions of molecular interactions, demanding significant computational resources.

While dedicated HPC clusters provide high performance, they are often expensive and inaccessible to many research laboratories. SSHPC was designed as a simple alternative capable of leveraging underutilized laboratory computers to increase available processing power.

The project was originally developed during large-scale SARS-CoV-2 virtual screening studies and successfully increased the available computational throughput of the LAMBDA laboratory by approximately 40% through the utilization of otherwise idle machines.

Key Features

  • Pure Bash implementation
  • No external framework dependencies
  • SSH-based secure communication
  • Automatic client-server pairing
  • Distributed task execution
  • Dynamic worker availability detection
  • Automatic task recovery and monitoring
  • Installation through a single script
  • Uses idle computational resources without disrupting users
  • Scalable to multiple laboratories and institutions

Architecture

SSHPC follows a client-server architecture.

Server

The server is responsible for:

  • Managing the processing queue
  • Assigning jobs to available clients
  • Monitoring execution status
  • Receiving results
  • Maintaining execution history and reports

Client

The client:

  • Monitors local machine usage
  • Detects processing idleness
  • Requests jobs from the server when resources are available
  • Executes assigned workloads
  • Returns results to the server

All communication is performed through encrypted SSH connections.

               ┌───────────────┐
               │    SERVER     │
               │ Job Manager   │
               └───────┬───────┘
                       │ SSH
       ┌───────────────┼───────────────┐
       │               │               │
       ▼               ▼               ▼

  ┌─────────┐    ┌─────────┐    ┌─────────┐
  │ Client1 │    │ Client2 │    │ ClientN │
  │ Worker  │    │ Worker  │    │ Worker  │
  └─────────┘    └─────────┘    └─────────┘

Project Structure

.
├── client
│   └── .bin
│       ├── sshpc_worker.sh
│       └── sshpc_client_install.sh
│
├── server
│   ├── .bin
│   │   ├── sshpc_clean.sh
│   │   ├── sshpc_run.sh
│   │   ├── sshpc_split_vs.sh
│   │   ├── sshpc_summary.sh
│   │   ├── sshpc_verify.sh
│   │   └── sshpc_sc_pair.sh
│   │
│   └── sshpc_server_install.sh
│
└── README.md

Installation

Server

sudo ./server/sshpc_server_install.sh

The installation script:

  • Creates the SSHPC system user
  • Creates the required directory structure
  • Generates SSH keys
  • Configures cron jobs
  • Installs management scripts

Client

sudo ./client/.bin/sshpc_client_install.sh \
    -i <SERVER_IP> \
    -l <LAB_NAME>

Example:

sudo ./client/.bin/sshpc_client_install.sh \
    -i 192.168.1.100 \
    -l LAMBDA

The client installer:

  • Creates the SSHPC system user
  • Installs worker scripts
  • Creates configuration files
  • Configures cron jobs
  • Connects the machine to the cluster

Core Components

Client Scripts

Script Description
sshpc_client_install.sh Installs and configures a client node
sshpc_worker.sh Detects idle resources and requests jobs

Server Scripts

Script Description
sshpc_server_install.sh Installs and configures the server
sshpc_verify.sh Monitors worker requests
sshpc_run.sh Assigns and executes jobs
sshpc_summary.sh Generates cluster reports
sshpc_clean.sh Performs cleanup operations
sshpc_split_vs.sh Splits virtual screening workloads
sshpc_sc_pair.sh Manages client-server pairing

Workflow

  1. Client checks local resource availability.
  2. Client sends an availability request.
  3. Server receives the request.
  4. Server assigns a queued task.
  5. Client executes the workload.
  6. Results are returned to the server.
  7. Execution is recorded and summarized.

Security

SSHPC relies entirely on the Secure Shell (SSH) protocol.

Benefits include:

  • Authentication
  • Encrypted communication
  • Data integrity
  • Secure file transfer
  • Secure remote execution

Clients never receive direct access to the server filesystem. Communication is controlled entirely by the server.

Applications

Although originally developed for virtual screening and molecular docking, SSHPC can be used for any embarrassingly parallel workload, including:

  • Virtual screening
  • Molecular docking
  • Bioinformatics pipelines
  • Sequence analysis
  • Data processing
  • Batch simulations
  • Machine learning preprocessing
  • General scientific computing

Performance

During production tests performed at UFPB:

  • A dedicated server processed approximately 1000 molecular docking complexes/day.
  • By adding 10 laboratory workstations through SSHPC, processing capacity increased by approximately 40%.
  • Around 400 additional complexes/day were computed using machines that would otherwise remain idle.

Publication

This project was developed as part of the research:

Distributed Processing System Development Applied to Virtual Screening

Authors

  • Arthur Araújo de Lacerda
  • Edson Luiz Folador

License

This project is intended for academic, educational, and research purposes.