Skip to content

fix(signer): handle missing/behind KES period on registration #2952

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
leepl37 wants to merge 4 commits into
base: main
Choose a base branch
from
+67 −5
Open

fix(signer): handle missing/behind KES period on registration #2952

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
4c834f2
fix(signer): handle missing/behind KES period on registration
leepl37 Jan 24, 2026
c4280cc
style: fix rustfmt errors
leepl37 Jan 26, 2026
4cb1338
fix(signer): retry on KES period mismatch
leepl37 Jan 27, 2026
42e6ca8
chore: format runner after KES retry change
leepl37 Jan 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 67 additions & 5 deletions mithril-signer/src/runtime/runner.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -198,10 +198,72 @@ impl Runner for SignerRunner {
}
_ => (None, None),
};
let current_kes_period = self.services.chain_observer.get_current_kes_period().await?;
let kes_evolutions = operational_certificate.map(|operational_certificate| {
current_kes_period.unwrap_or_default() - operational_certificate.get_start_kes_period()
});
let mut attempt = 0;
let max_retries = 5;
let retry_delay = std::time::Duration::from_secs(1);

let (current_kes_period, kes_evolutions) = loop {
attempt += 1;

let kes_period = self.services.chain_observer.get_current_kes_period().await?;

let kes_period = match kes_period {
Some(kes_period) => kes_period,
None => {
if attempt >= max_retries {
return Err(
RunnerError::NoValueError("current_kes_period".to_string()).into()
);
}
warn!(
self.logger,
"Current KES period is not available yet. Retrying...";
"attempt" => attempt,
"max_retries" => max_retries
);
tokio::time::sleep(retry_delay).await;
continue;
}
};

let check_result = match &operational_certificate {
Some(op_cert) => {
let start_kes_period = op_cert.get_start_kes_period();
if kes_period < start_kes_period {
Err(start_kes_period)
} else {
Ok(Some(kes_period - start_kes_period))
}
}
None => Ok(None),
};

match check_result {
Ok(evolutions) => break (kes_period, evolutions),
Err(start_kes_period) => {
if attempt >= max_retries {
warn!(
self.logger,
"Current KES period is behind operational certificate start period.";
"current_kes_period" => u64::from(kes_period),
"start_kes_period" => u64::from(start_kes_period)
);
return Err(
RunnerError::NoValueError("kes_period_underflow".to_string()).into(),
);
}
warn!(
self.logger,
"KES period mismatch. Retrying...";
"current_kes_period" => u64::from(kes_period),
"start_kes_period" => u64::from(start_kes_period),
"attempt" => attempt,
"max_retries" => max_retries
);
tokio::time::sleep(retry_delay).await;
}
}
};
Comment on lines +201 to +266

@jpraynaud jpraynaud Jan 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that we could trigger an error if the Cardano node is not completely synced:

  • current_kes_period is None
  • or if current_kes_period is less than operational_certificate.get_start_kes_period()

However, this code is too complex and does not take advantage of the retry mechanism of the state machine (an error will restart the state machine which will eventually run this same register_signer_to_aggregator code later).

@leepl37 leepl37 Jan 28, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review @jpraynaud!
I agree that the retry loop adds complexity and that relying on the State Machine is usually the cleaner default.

The main reason I tried this workaround was to solve a race condition I kept hitting in the E2E Devnet tests:

  1. The Trigger: When the cardano-node is briefly syncing at an epoch boundary, it reports an outdated KES period.

  2. The Problem: This error causes the State Machine to trigger its standard run_interval sleep (default 5s).

  3. The Failure: Since Devnet epochs are extremely short, even a single backoff cycle creates a risk of missing the registration window.

Evidence:

Devnet Config: The epoch duration is only 75 seconds (0.75s slot length * 100 slots).

The Error: Before adding this loop, I consistently encountered this failure in E2E tests:

Error(Unretryable): Mithril End to End test failed
Caused by:
    0: Requesting aggregator `mithril-aggregator-1`
    1: Minimum expected mithril stake distribution epoch not reached: 20 < 21

This confirms the Signer missed the window for epoch 21 because it was sleeping during the transition time.

Mainnet & CI Verification:

  • Devnet: The CI build passed with this fix, confirming it stabilizes the race condition.

  • Mainnet: This logic is also safe for Mainnet. While the standard 5s backoff is negligible on Mainnet (due to 5-day epochs), adding this fast retry makes the Mainnet signer more robust against transient node glitches without introducing risks.

Question: Is there a preferred way to trigger an "immediate" retry (skipping the standard backoff) for this specific transient error? If not, I can remove the loop, but I am worried it will re-introduce these random CI failures.


let protocol_initializer = self
.services
Expand All @@ -217,7 +279,7 @@ impl Runner for SignerRunner {
stake,
&protocol_parameters,
self.services.kes_signer.clone(),
current_kes_period,
Some(current_kes_period),
)?;

let signer = Signer::new(
Expand Down