Cpu only for multi-user version#28
Open
kylincaster wants to merge 316 commits into
Open
Conversation
- Remove mail subsystem (mail.c/h, mymail.sh, ssmtp/sendmail, TS_MAIL_*) - TS_ONFINISH retained as default --on-finish cmd (override by --on-finish) - JSON output (-M json) expanded to match -i info fields - -M json -J <id> filters to single job (reuses Msg.jobid) - gen_topology.py: generates all strategies, MAX_GROUPS_PER_NODE - gen_topology.py: skips trivial (count==cores) and duplicate topologies - cpu_bind_defrag() two-phase: same-node first, cross-node fallback - cpu_owner[] tracks real jobid; cpu_bind_post_alloc in all paths - Project description: Task Spooler PLUS - multi-user job scheduler like slurm - Copyright: 2007-2026 Kylin JIANG - Lluís Batlle i Rossell - README: vs Slurm / vs original TS comparison tables - CLAUDE.md: hwloc topology detection commands + install guide - Fix: jobs[n] overwrite, skipped overwrite, mem leak in defrag - Fix: cgroups_set_cpuset create v2 dir before write
- init_jobids_DB(): startup validation scan against Jobs/Finished tables - get_jobids_DB() valid(>0) used directly; invalid triggers full scan - set_jobids_DB(): lightweight runtime write, no scan overhead - Global table INSERT OR IGNORE to avoid UNIQUE constraint - gen_topology.py: show recommended strategy in generated file list - jobids made extern for server.c access
Defrag code preserved in cpu_bind.c for future re-evaluation. Two callsites commented out: new_finished_job and s_remove_job.
- cpu_bind_defrag() restored in new_finished_job and s_remove_job, gated by cpu_bind_defrag_enabled() - --no-bind-defrag server startup option disables auto-defrag (use for MPI workloads where dynamic cpuset changes risk deadlock) - cpu_bind.c/h: cpu_bind_defrag_disabled flag + accessor functions - CLAUDE.md: defrag re-enabled, --no-bind-defrag documented
- README: add --no-bind-defrag option in long options - README_CN: same in Chinese - CHANGELOG: v2.6.4 entry for defrag re-enablement - version.h: 2.6.3 → 2.6.4
Co-Authored-By: deepseek-v4-pro[1m] <deepseek-ai@claude-code-best.win>
- --n-retry N: retry failed jobs up to N times inside run_child (exit error within 15s, exec failures 126/127 excluded) - Fix: s_list NULL user crash on -A, JSON null user deref Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- cpu_bind_defrag() moved to detached pthread; main thread preps allocs+findjob, thread does cgroup I/O (freeze→rebuild→cpuset→thaw) - findjob() stays in main thread — eliminates active_jobs race - defrag_in_progress flag gates CPU bind alloc/free and job freeze/thaw during defrag; conflicting operations are queued (deferred_ops vec), drained after defrag completes via poll - server_loop polls cpu_bind_defrag_poll() for completion+drain - make TS_CPU_BIND=1 links -lpthread; make (no TS_CPU_BIND) unchanged - docs: README, README_CN, CLAUDE.md, CHANGELOG updated Co-Authored-By: deepseek-v4-pro[1m] <deepseek-ai@claude-code-best.win>
…tion Co-Authored-By: deepseek-v4-pro[1m] <deepseek-ai@claude-code-best.win>
Co-Authored-By: deepseek-v4-pro[1m] <deepseek-ai@claude-code-best.win>
…lock -r on PAUSE Four interrelated fixes for PAUSE (timeout) job handling: 1. Block 'ts -r' on PAUSE jobs (like RUNNING) - s_remove_job() now rejects PAUSE state the same as RUNNING - Avoids cgroups_clean_job() on live processes 2. Free CPU binding on PAUSE (safe_pause_job) - cpu_bind_free() when pausing, so allocator can reassign CPUs - Re-allocate on PAUSE→RUNNING resume in config_running() 3. Fix double CPU allocation for QUEUED→RUNNING - Guard re-allocation in config_running() with p->pid != 0 - QUEUED jobs get allocated later in s_process_runjob_ok() 4. Fix cgroup cleanup order for v1 - Swap cleanup_freeze after cleanup_cpu in cgroups_clean_job() - CPU cleanup kills processes first, freeze rmdir succeeds immediately Also: cgroups v1/v2 init validation, reconnect improvements, start-service flag, non-fatal cpu.cfs_quota_us warnings.
- New ABNORMAL state in enum Jobstate — set when health check detects a RUNNING job is stuck (output empty AND no child processes) - s_check_running_health() — periodic check, runs every 10s in server loop - count_child_pids() — reads /proc/PID/children to detect zombie jobs - health_state field in struct Job — 0=UNCHECKED, 1=NORMAL, skip on rechecks - On ABNORMAL: free slots via free_cores(), release CPU binding, state changes to ABNORMAL so it's excluded from slot accounting - s_remove_job() rejects removing ABNORMAL jobs (use ts -k instead) - s_send_output() allows ABNORMAL state so ts -k can get the PID - Only adds new code — zero changes to existing logic paths
added 12 commits
July 1, 2026 17:18
s_add_job() now detects jobs that were paused before restart: - RUNNING state + pause_time > 0 in DB → restore as PAUSE - PAUSE state in DB → restore directly - If pause_time is missing (0), initialize to current monotonic time - The old pause_time from DB is still on the system monotonic timeline, so work_time calculation remains correct
Adds a backwards scan of active_jobs in s_update_slots_usage() that checks PID liveness via s_check_running_pid() for all RUNNING/PAUSE/ABNORMAL jobs. Dead jobs are moved to finished via job_finished() with errorlevel=-1, died_by_signal=1. This runs on every ts -l (list) and ts submission, ensuring zombie processes from crashes or external kills are cleaned up promptly.
Sets real_sec from get_work_time_by_job() so that paused jobs keep their accumulated runtime when moved to finished after process death. Previously real_sec=0 caused job_finished() to overwrite start_time, losing the runtime data.
Removed the 'if (total < 0) return -1' guard so --add-wtime accepts negative durations like '-30m' or '-18.5741d'. The downstream s_add_wtime() still rejects results <= 0.
- CHANGELOG: health check, ABNORMAL state, dead cleanup, add-wtime negative - CLAUDE: updated state machine with ABNORMAL, updated current branch work - README/README_CN: added health check and add-wtime features - main.c: --add-wtime help mentions negative values
Changes: 1. s_delete_job() DB leak (jobs.c:1016) - Added delete_DB(jobid, 'Jobs') before destroy_job() - Fixes: QUEUED job survives restart after RECONNECT + second disconnect 2. Orphan QUEUED auto-cleanup (jobs.c + server.c) - New s_cleanup_orphan_queued(): removes QUEUED + client_socket <= 0 - server_loop() 30-min one-time timer calls it after server start 3. Timeout order_id sync (sqlite.c/h + jobs.c) - New movebottom_DB(): sets order_id = max(order_id) + 1 - s_check_timeout() calls movebottom_DB() after moving job to queue back - Fixes: SELECT ... ORDER BY order_id matches runtime queue order after restart 4. CLAUDE.md: documented all three fixes in Restart Recovery §6 - DB leak scenario and fix - Orphan QUEUED auto-cleanup with flow diagram - Timeout order_id sync
get_order_id() used sqlite3_exec with a callback, which could not distinguish 'no row found' from 'order_id = 0'. For new jobs not yet in the DB, it returned order_id = 0 with err = 0, causing edit_DB() to skip the max_order_id + 1 fallback. All non-timeout jobs ended up with order_id = 0, making ORDER BY order_id return arbitrary order. Fix: rewrite with sqlite3_prepare_v2 / sqlite3_step so that SQLITE_ROW check correctly distinguishes 'no row' from 'value is 0'.
server.c:741: jstate2string(jp->state) replaces raw numeric state (e.g. 'for 0' → 'for queued')
- New --requeue <jobid> CLI option: manually requeue a running or paused job. Calls s_relegate_job() (pause + negative wall_time + PAUSE state), then moves job to end of active_jobs + movebottom_DB. - Extracted s_relegate_job() from check_timeout() — shared between automatic timeout and manual --requeue. - List state label: timeout → requeue (list.c) - Added REQUEUE_JOB message type, client/server handlers, help text. - Updated CHANGELOG.md and CLAUDE.md.
CLI option --hold is now --pause. Updated help text, README, and manual_test.sh. Internal function c_hold_job/s_hold_job unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
For multi-user, each user has the same opportunity to invoke a new job, if the user's slot and the total slot are large enough.
usage:
./${tmpdir}/socket-ts.rootor which could be specific byTS_SOCKETenvironment variable.(server_start.c)user.cwhich could be modified by the enivorment variableTS_USER_PATH. Moreover a log file is also controlled byuser.cNew features/Commands and the potential problem
--daemonRun the server as daemon by Root only.--holdand--restart[jobid] hold-on and restart a task.--lockand--unlockLock and unlock the task-spooler servers to avoid the potential conflict--stopand--cont[user], pause and continue all tasks, or lock/unlock all user by root-Ashow all user information and all tasks-Xrefresh the user configure on-the-fly-Kkill the task spooler server-rremove a job, even it is runningThe main problem of my work is that the root server cannot control the task run by the other normal user.
I found in my service I cannot stop/pause the task owner by the other normal user.
Could you have a look on the
c_remove_job()function in theclient.c