Merge branch 'main' into feat/remote-signer-keyring

Author: diorwave

README.md

@@ -1,16 +1,58 @@
 # Daemon
 
-**Note:** Daemon services code was adopted from dydx [](https://github.com/dydxprotocol/v4-chain/tree/main/protocol/daemons) and reconfigured.
+**Note:** Daemon services code was adopted from [dYdX](https://github.com/dydxprotocol/v4-chain/tree/main/protocol/daemons) and reconfigured.
+
+## Configuration
+
+The daemon loads environment variables from the current directory's `.env` file, or from `../.env` when run from a subdirectory. See [`env.example`](./env.example) for a complete starting point.
+
+Most CLI flags can also be provided as environment variables by uppercasing the flag name and replacing `-` or `.` with `_`, for example `--keyring-backend` becomes `KEYRING_BACKEND`. `LAYER_HOME` is preferred for the Layer home directory so the daemon does not accidentally use the shell's `HOME`.
+
+Layer endpoint configuration can be provided with comma-separated environment variables:
+
+```sh
+RPC_NODES=http://node_endpoint1:26657,http://node_endpoint2:26657
+GRPC_NODES=127.0.0.1:9090,node2:9090
+```
+
+The first endpoint in each list is treated as the primary endpoint. Later entries are used as ordered fallbacks.
+
+Both endpoint types are required when starting the reporter daemon:
+
+- `GRPC_NODES` / `--grpc` configures Cosmos gRPC query services.
+- `RPC_NODES` / `--node` configures CometBFT RPC. The reporter uses this for startup chain ID validation, block/status polling, transaction broadcast, and transaction lookup while waiting for inclusion.
+
+Endpoint env vars take precedence over the existing CLI flags:
+
+- `RPC_NODES` is preferred over `--node`.
+- `GRPC_NODES` is preferred over `--grpc`.
+- If an env var is unset, the daemon preserves the old behavior by using the matching flag value as a single endpoint.
+
+At startup, the daemon checks the configured gRPC and CometBFT RPC endpoints for a matching chain ID and starts with the first healthy matching endpoints. The reporter keeps both endpoint lists and falls back to later nodes for network/client failures. gRPC fallback is used for reporter chain queries, while RPC fallback is used for status checks, transaction lookup, and transaction broadcast. The reporter also periodically probes the primary endpoints and switches back when they are healthy again. It does not switch endpoints for semantic chain failures such as out-of-gas responses, non-zero tx result codes, or normal tx-not-found polling.
+
+The pricefeed client is started with the selected gRPC endpoint only. Endpoint-list fallback currently applies to the reporter client's chain query and transaction paths, not to the pricefeed client.
+
+Ethereum JSON-RPC configuration uses the same comma-separated primary/fallback pattern:
+
+```sh
+BRIDGE_CHAIN_RPC_NODES=https://mainnet.infura.io/v3/YOUR_INFURA_API_KEY,https://eth-mainnet.g.alchemy.com/v2/YOUR_ALCHEMY_API_KEY
+```
+
+`BRIDGE_CHAIN_RPC_NODES` is used by token bridge deposit monitoring. The first endpoint is tried first; later entries are ordered fallbacks.
+
+Ethereum mainnet custom query contract reads use the built-in mainnet endpoint templates by default. If `BRIDGE_CHAIN_RPC_NODES` points at a non-mainnet bridge chain such as Sepolia, set `ETH_MAINNET_RPC_NODES` to a comma-separated Ethereum mainnet endpoint list for custom queries.
+
+Custom query API keys are read from the generated `custom_query_config.toml` entries that reference environment placeholders. The current built-in templates use `CMC_PRO_API_KEY`, `CGPRO_API_KEY`, and `SUBGRAPH_API_KEY`.
 
 ## Task loops
 
 ## PriceFetcher
 
-- Will query exchanges for prices once or multiple times based on wether the api supports single vs multi markets; ie wether an api needs to be queried for each pair individually or can return multiple pairs at once, [See here for exchange details](./constants/static_exchange_details.go).
+- Will query exchanges for prices once or multiple times based on whether the API supports single vs multi markets; i.e. whether an API needs to be queried for each pair individually or can return multiple pairs at once. [See here for exchange details](./constants/static_exchange_details.go).
 
 ## PriceEncoder
 
-- Will update cache with the queried prices and encode appropriately also make adjustments as necessary based on if adjustByMarket is defined.
+- Will update the cache with queried prices, encode them appropriately, and make adjustments when `adjustByMarket` is defined.
 
 ### Configuration
 
@@ -34,7 +76,7 @@ example:
 
 ```go
 [[market_params]]
-ExchangeConfigJson = "{\"exchanges\":[{\"exchangeName\":\"Binance\",\"ticker\":\"\\\"ETHBTC\\\"\"},{\"exchangeName\":\"Bitfinex\",\"ticker\":\"tETHBTC\",\"adjustByMarket\":\"BTC-USD\"}]}" // this is just an example to show how to use adjustByMarket.  you can use ETH-USD without adjustbymarket
+ExchangeConfigJson = "{\"exchanges\":[{\"exchangeName\":\"Binance\",\"ticker\":\"\\\"ETHBTC\\\"\"},{\"exchangeName\":\"Bitfinex\",\"ticker\":\"tETHBTC\",\"adjustByMarket\":\"BTC-USD\"}]}" // This is an example showing how to use adjustByMarket. You can use ETH-USD without adjustByMarket.
 Exponent = -6
 Id = 2
 MinExchanges = 1
@@ -51,7 +93,7 @@ type MarketParam struct {
     Pair string
     // Static value. The exponent of the price.
     // For example if `Exponent == -5` then a `Value` of `1,000,000,000`
-    // represents “$10,000`. Therefore `10 ^ Exponent` represents the smallest
+    // represents "$10,000". Therefore `10 ^ Exponent` represents the smallest
     // price step (in dollars) that can be recorded.
     Exponent int32
     // The minimum number of exchanges that should be reporting a live price for
@@ -63,7 +105,7 @@ type MarketParam struct {
     // A string of json that encodes the configuration for resolving the price
     // of this market on various exchanges.
     ExchangeConfigJson string
-    // Query data is the market pair represention in layer
+    // Query data is the market pair representation in layer
     QueryData string
 }
 ```
@@ -74,10 +116,54 @@ A price is valid by default up to 30 seconds; to change this to a different defa
 **Also:** Config files are written to homedir/.layer/config/.
 To change/add exchange details or market pairs edit the files `pricefeed_exchange_config.toml` or `market_params.toml` respectively.
 
+## Keyring Password File
+
+When running the reporter daemon with `--keyring-backend file`, set `KEYRING_PASSWORD_FILE` to a file containing the keyring password. This lets the daemon unlock the account without requiring an interactive terminal prompt.
+
+For this systemd file-keyring setup, set `LAYER_HOME` to the same home directory that contains the daemon config and keyring files. This keeps the non-interactive service from resolving `home` from the service user's shell environment instead of the intended Layer home.
+
+Example systemd service snippet:
+
+```ini
+[Service]
+User=reporter
+Environment="KEYRING_PASSWORD_FILE=/etc/layer-daemons/reporter-keyring-password"
+Environment="LAYER_HOME=/home/reporter/.layer"
+Environment="GRPC_NODES=your-grpc-host:9090,your-fallback-grpc-host:9090"
+Environment="RPC_NODES=tcp://your-rpc-host:26657,tcp://your-fallback-rpc-host:26657"
+ExecStart=/usr/local/bin/reporterd \
+  --keyring-backend file \
+  --from your-key-name
+```
+
+Create the password file so only the service user can read it:
+
+```bash
+sudo install -d -m 700 -o reporter -g reporter /etc/layer-daemons
+sudo install -m 600 -o reporter -g reporter /dev/null /etc/layer-daemons/reporter-keyring-password
+sudo sh -c 'printf "%s\n" "YOUR_KEYRING_PASSWORD" > /etc/layer-daemons/reporter-keyring-password'
+```
+
+Make sure the service `User` can read the file. When `KEYRING_PASSWORD_FILE` is set, startup fails and the daemon exits if the file cannot be read, is empty, or cannot unlock the configured `--from` account. If `KEYRING_PASSWORD_FILE` is not set, the daemon falls back to reading the keyring password from stdin.
+
+## Reward Withdrawals And Auto-Unbonding
+
+The reporter periodically withdraws earned tips/rewards with `MsgWithdrawTip`. The interval is configured by `WITHDRAW_FREQUENCY` in seconds and defaults to `43200` (12 hours). By default, the validator operator address is derived from the reporter account address. If the reporter account is delegated to a different validator, set `REPORTERS_VALIDATOR_ADDRESS` to that validator's `tellorvaloper...` address.
+
+Auto-unbonding is optional and can be configured by CLI flags or equivalent environment variables:
+
+| Flag | Environment | Type | Default | Description |
+|------|-------------|------|---------|-------------|
+| `--auto-unbonding-frequency` | `AUTO_UNBONDING_FREQUENCY` | uint32 | `0` | Enables unbonding every N days (`0` = disabled, valid enabled range is 1-21). |
+| `--auto-unbonding-amount` | `AUTO_UNBONDING_AMOUNT` | uint32 | `0` | Amount of `loya` to unbond each time. Required when frequency is enabled. |
+| `--auto-unbonding-max-stake-percentage` | `AUTO_UNBONDING_MAX_STAKE_PERCENTAGE` | decimal string | `0.0` | Optional cap from `0.0` to `1.0`; if the configured amount exceeds this share of stake, the unbond is skipped. |
+
+Gas estimates are cached per transaction type. `--refresh-gas-estimates-interval` / `REFRESH_GAS_ESTIMATES_INTERVAL` resets cached estimates and gas-adjustment levels periodically; it defaults to `12h`, and values `<=0` disable the refresh loop.
+
 ### Median Server
 
-Median server was added for a way to query median values that were from an endpoint or cli. See usage [here](../x/oracle/client/cli/query_all_get_median.go).
-All median values or median value given query data using the following commands respectively.
+The median server can query median values from an endpoint or the CLI. See usage [here](../x/oracle/client/cli/query_all_get_median.go).
+Query all median values, or a median value for specific query data, using the following commands respectively.
 `layerd query oracle get-all-median-values`
 `layerd query oracle get-median-value <querydata>`
 
@@ -114,12 +200,12 @@ The Price Guard is a safety mechanism that prevents the reporter from submitting
 
 ### Flags
 
-| Flag | Type | Description | Required (if enabled) |
-|------|------|-------------|---------------------|
-| `--price-guard-enabled` | bool | Enables the price guard mechanism | No |
-| `--price-guard-threshold` | float64 | Maximum allowed percentage change (e.g., 0.5 = 50%). Submissions exceeding this change from the last reported price will be blocked. | Yes |
-| `--price-guard-max-age` | duration | Time after which a stored price is considered expired (e.g., "1h"). If the last price is expired, the new price is accepted regardless of deviation. | Yes |
-| `--price-guard-update-on-blocked` | bool | If true, updates the internal "last known price" to the new value even if submission was blocked. If false, keeps the old price as the baseline. | Yes |
+| Flag | Environment | Type | Description | Required (if enabled) |
+|------|-------------|------|-------------|-----------------------|
+| `--price-guard-enabled` | `PRICE_GUARD_ENABLED` | bool | Enables the price guard mechanism. | No |
+| `--price-guard-threshold` | `PRICE_GUARD_THRESHOLD` | float64 | Maximum allowed percentage change (e.g. `0.5` = 50%). Submissions exceeding this change from the last reported price are blocked. | Yes |
+| `--price-guard-max-age` | `PRICE_GUARD_MAX_AGE` | duration | Time after which a stored price is considered expired (e.g. `1h`). If the last price is expired, the new price is accepted regardless of deviation. | Yes |
+| `--price-guard-update-on-blocked` | `PRICE_GUARD_UPDATE_ON_BLOCKED` | bool | If true, updates the internal "last known price" to the new value even if submission was blocked. If false, keeps the old price as the baseline. | Yes |
 
 ### Notes
 
@@ -131,3 +217,46 @@ The Price Guard is a safety mechanism that prevents the reporter from submitting
 4. **Update on Blocked:**
    - If `true`: A blocked price becomes the new baseline for future checks.
    - If `false`: The old price remains the baseline; future submissions must be within threshold of the *old* price.
+
+## Auto balance-to-keep
+
+The reporter daemon can keep a target **loya** balance in the reporter wallet and automatically bridge any excess to Ethereum once per day. This uses Layer’s `MsgWithdrawTokens` bridge message (`isBridge` gas bucket, same tx pipeline as other bridge operations).
+
+### Flags
+
+| Flag | Environment | Type | Default | Description |
+|------|-------------|------|---------|-------------|
+| `--auto-balance-to-keep` | `AUTO_BALANCE_TO_KEEP` | uint64 | `0` | Target wallet balance in **loya** (`0` = disabled). Any amount above this, minus the gas reserve below, is bridged. |
+| `--auto-balance-execution-time` | `AUTO_BALANCE_EXECUTION_TIME` | string | `00:00` | UTC time to check balance and bridge, format **`HH:MM`** with hour 0-23 and minute 0-59 (e.g. `03:00`, `15:30`). |
+| `--auto-balance-bridge-to-eth-addr` | `AUTO_BALANCE_BRIDGE_TO_ETH_ADDR` | string | `""` | Ethereum recipient for bridged tokens. Required when `--auto-balance-to-keep > 0`. May include or omit the `0x` prefix. Validated with standard hex address checks at startup. |
+
+### Behavior
+
+1. **Schedule:** Once per UTC day at `--auto-balance-execution-time`, the daemon queries the reporter wallet's `loya` balance.
+2. **Amount:** `bridge_amount = wallet_balance - auto-balance-to-keep - 1_000_000` (a fixed **1 TRB** reserve in loya is left for future gas). If `bridge_amount <= 0`, nothing is sent.
+3. **Broadcast:** The transaction uses the shared broadcast path with RPC endpoint fallback and gas-adjustment retries for out-of-gas responses. Other failures are logged; the next balance check happens at the next scheduled UTC time.
+4. **Shutdown:** Bridge txs are enqueued with `trySend` so shutdown does not panic on a closed channel.
+
+### Startup validation
+
+When `--auto-balance-to-keep > 0`, the reporter **fails to start** if:
+
+- `--auto-balance-bridge-to-eth-addr` is missing or not a valid Ethereum address
+- `--auto-balance-execution-time` is not valid `HH:MM` (hour 0-23, minute 0-59)
+
+### Example
+
+Keep 5 TRB in the wallet (5_000_000 loya), run the check daily at 03:00 UTC, and bridge excess to an Ethereum address:
+
+```bash
+LAYER_HOME=/home/reporter/.layer \
+GRPC_NODES=your-grpc-host:9090 \
+RPC_NODES=tcp://your-rpc-host:26657 \
+reporterd \
+  --from your-key-name \
+  --auto-balance-to-keep=5000000 \
+  --auto-balance-execution-time=03:00 \
+  --auto-balance-bridge-to-eth-addr=0x0000000000000000000000000000000000000000
+```
+
+**Note:** Amounts are in **loya** (micro-denom), not whole TRB. `1 TRB = 1_000_000 loya`.

app.go

@@ -46,8 +46,9 @@ type App struct {
 func NewApp(
 	ctx context.Context,
 	logger log.Logger,
-	chainId,
-	grpcAddress,
+	chainId string,
+	grpcEndpoints []string,
+	rpcEndpoints []string,
 	homePath string,
 	prometheusPort int,
 ) *App {
@@ -140,7 +141,7 @@ func NewApp(
 		// Use cancellable context instead of context.Background
 		ctx,
 		daemonFlags,
-		grpcAddress,
+		grpcEndpoints[0],
 		logger,
 		&daemontypes.GrpcClientImpl{},
 		marketParamsConfig,
@@ -161,16 +162,18 @@ func NewApp(
 			// Use cancellable context instead of context.Background
 			ctx,
 			daemonFlags,
-			grpcAddress,
+			grpcEndpoints,
 			&daemontypes.GrpcClientImpl{},
 			marketParamsConfig,
 			indexPriceCache,
 			tokenDepositsCache,
 			tokenBridgeTipsCache,
 			queries,
 			chainId,
+			rpcEndpoints,
 		); err != nil {
 			logger.Error("Reporter client failed to start", "error", err)
+			os.Exit(1)
 		}
 	}()
 

cmd/chain_id.go

@@ -3,6 +3,7 @@ package main
 import (
 	"context"
 	"fmt"
+	"strings"
 	"time"
 
 	rpchttp "github.com/cometbft/cometbft/rpc/client/http"
@@ -12,29 +13,66 @@ import (
 	"github.com/cosmos/cosmos-sdk/client/grpc/cmtservice"
 )
 
-// detectChainID queries both the gRPC endpoint and the CometBFT RPC endpoint
-// for the chain ID, validates that they agree, and returns the chain ID.
-// Returns an error if either endpoint is unreachable or if they return
-// different chain IDs.
-func detectChainID(ctx context.Context, grpcAddr, nodeRPCAddr string) (string, error) {
-	detectCtx, cancel := context.WithTimeout(ctx, 15*time.Second)
-	defer cancel()
+type detectedEndpointChainID struct {
+	endpoint string
+	chainID  string
+}
+
+type chainIDDetector func(context.Context, string) (string, error)
+
+var chainIDEndpointDetectTimeout = 15 * time.Second
 
-	grpcChainID, err := chainIDFromGRPC(detectCtx, grpcAddr)
+func detectChainIDFromEndpoints(ctx context.Context, grpcAddrs, nodeRPCAddrs []string) (string, string, string, error) {
+	grpcChainIDs, err := detectEndpointChainIDs(ctx, "gRPC", grpcAddrs, chainIDFromGRPC)
 	if err != nil {
-		return "", fmt.Errorf("failed to detect chain ID via gRPC (%s): %w", grpcAddr, err)
+		return "", "", "", err
 	}
 
-	rpcChainID, err := chainIDFromRPC(detectCtx, nodeRPCAddr)
+	rpcChainIDs, err := detectEndpointChainIDs(ctx, "node RPC", nodeRPCAddrs, chainIDFromRPC)
 	if err != nil {
-		return "", fmt.Errorf("failed to detect chain ID via node RPC (%s): %w", nodeRPCAddr, err)
+		return "", "", "", err
+	}
+
+	if err := validateReachableChainIDs("gRPC", grpcChainIDs); err != nil {
+		return "", "", "", err
+	}
+	if err := validateReachableChainIDs("node RPC", rpcChainIDs); err != nil {
+		return "", "", "", err
 	}
+	if grpcChainIDs[0].chainID != rpcChainIDs[0].chainID {
+		return "", "", "", fmt.Errorf("chain ID mismatch: gRPC returned %q, node RPC returned %q", grpcChainIDs[0].chainID, rpcChainIDs[0].chainID)
+	}
+
+	return grpcChainIDs[0].chainID, grpcChainIDs[0].endpoint, rpcChainIDs[0].endpoint, nil
+}
 
-	if grpcChainID != rpcChainID {
-		return "", fmt.Errorf("chain ID mismatch: gRPC returned %q, node RPC returned %q", grpcChainID, rpcChainID)
+func detectEndpointChainIDs(ctx context.Context, endpointType string, endpoints []string, detector chainIDDetector) ([]detectedEndpointChainID, error) {
+	var detected []detectedEndpointChainID
+	var errs []string
+	for _, endpoint := range endpoints {
+		endpointCtx, cancel := context.WithTimeout(ctx, chainIDEndpointDetectTimeout)
+		chainID, err := detector(endpointCtx, endpoint)
+		cancel()
+		if err != nil {
+			errs = append(errs, fmt.Sprintf("%s: %v", endpoint, err))
+			continue
+		}
+		detected = append(detected, detectedEndpointChainID{endpoint: endpoint, chainID: chainID})
 	}
+	if len(detected) == 0 {
+		return nil, fmt.Errorf("failed to detect chain ID via any %s endpoint: %s", endpointType, strings.Join(errs, "; "))
+	}
+	return detected, nil
+}
 
-	return grpcChainID, nil
+func validateReachableChainIDs(endpointType string, detected []detectedEndpointChainID) error {
+	expected := detected[0].chainID
+	for _, item := range detected[1:] {
+		if item.chainID != expected {
+			return fmt.Errorf("%s endpoints disagree on chain ID: %s returned %q, %s returned %q", endpointType, detected[0].endpoint, expected, item.endpoint, item.chainID)
+		}
+	}
+	return nil
 }
 
 func chainIDFromGRPC(ctx context.Context, grpcAddr string) (string, error) {