Skip to content

Commit 87c3f8a

Browse files
ZhuLinsenAutoCode Bot
authored andcommitted
fix: 修复断点续传按自然日判断数据存在性的逻辑 (ZhuLinsen#880) (ZhuLinsen#900)
* fix(issue-880): [bug]-修复断点续传按自然日判断数据存在性的逻辑 * fix(review-feedback-900): address latest review comments * fix(review-feedback-900): address latest review comments * fix(review-feedback-900): address latest review comments * fix(review-feedback-900): address latest review comments * fix(review-feedback-900): address latest review comments * fix(review-feedback-900): address latest review comments * fix(review-feedback-900): address latest review comments * fix(review-feedback-900): address latest review comments * fix(review-feedback-900): address latest review comments * fix(review-feedback-900): address latest review comments * fix(review-feedback-900): address latest review comments * fix(review-feedback-900): address latest review comments * fix(review-feedback-900): address latest review comments * fix(review-feedback-900): address latest review comments * fix(review-feedback-900): address latest review comments * fix(review-feedback-900): address latest review comments * fix: remove transient pr draft from pr-900 * fix(review-feedback-900): address latest review comments * fix(review-feedback-900): address latest review comments * fix(issue-880): [bug]-修复断点续传按自然日判断数据存在性的逻辑 * fix(review-feedback-900): address latest review comments --------- Co-authored-by: AutoCode Bot <autocode@example.com>
1 parent 9114353 commit 87c3f8a

11 files changed

Lines changed: 504 additions & 27 deletions

README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -241,6 +241,7 @@
241241
>
242242
> - **环境变量方式**:在 `.env` 或 GitHub Secrets 中设置,影响所有运行方式(定时触发、手动触发、本地运行)
243243
> - **UI 勾选方式**:仅在 GitHub Actions 手动触发时可见,不影响定时任务,适合临时需求
244+
> - **断点续传与 `--dry-run` 的数据存在性判断**:会按股票所属市场的本地时区和交易日历解析“最新可复用交易日”;周末/节假日复用最近交易日,交易日盘中复用上一已完成交易日,盘后若当日数据已落库则可直接跳过。详细规则见 [完整指南](docs/full-guide.md)
244245
245246
### 方式二:本地运行 / Docker 部署
246247

docs/CHANGELOG.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,9 +19,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
1919
- [测试] 🧪 **补充前端变更验证命令** — 对应前端资源变更同步执行 `cd apps/dsa-web && npm ci && npm run lint && npm run build`,作为版本信息展示与 Docker 重建生效验证的最小验证闭环记录。
2020
- [修复] 内置定时调度器现在会在运行中感知 WebUI 保存后的 `SCHEDULE_TIME` 变化,并在下一轮检查时重绑 daily job,避免 `python main.py --serve --schedule` 仍固定按启动时的 `18:00` 触发;`.env.example` 也同步删除了重复的定时任务配置示例。
2121
- [修复] 🪟 **Windows Release 渠道编辑器保留 MiniMax 模型前缀** — 渠道模式下填写 `minimax/<模型名>` 时,后端归一化与 Web 设置页运行时模型列表都会保留该值原样,不再误改写成 `openai/minimax/<模型名>`,从而恢复 MiniMax 模型在 Win 客户端里的保存、选择与使用。
22+
- [修复] 🗓️ **断点续传与 `--dry-run` 改按市场时区和交易日历判断可复用数据**(fixes #880)— 股票数据存在性检查不再直接使用服务器自然日,而是按 A 股 / 港股 / 美股各自市场时区解析“最新可复用交易日”;周末、节假日、跨时区以及盘中 / 盘后场景会统一复用最近已完成交易日,避免误判导致重复抓取或错误跳过。
2223
- [修复] 🐳 **Docker WebUI 运行时优先复用预构建静态资源**`prepare_webui_frontend_assets()` 现在会先检查镜像内已有的 `static/index.html` 是否可直接复用;当容器运行时不包含 `apps/dsa-web` 源码目录且未安装 `npm` 时,也不会误报“未找到前端项目,无法自动构建”,从而恢复 Docker 部署后的 WebUI 打开能力。
2324
- [修复] 📨 **单股推送模式不再并发复用共享通知实例**`StockAnalysisPipeline.run()` 现在会保留个股分析并发,但把 `SINGLE_STOCK_NOTIFY=true` 下的即时通知挪到结果收集侧串行发送;同时 `_send_single_stock_notification()` 为同一个 pipeline 实例补上实例级临界区,避免直接调用 `process_single_stock(..., single_stock_notify=True)` 时多个线程继续共享同一个 `NotificationService` 进入报告生成与发送链路,导致通知乱序、重复发送或状态污染。
24-
- [修复] 📨 **单股推送模式不再并发复用共享通知实例**`StockAnalysisPipeline.run()` 现在会保留个股分析并发,但把 `SINGLE_STOCK_NOTIFY=true` 下的即时通知挪到结果收集侧串行发送;同时 `_send_single_stock_notification()` 为同一个 pipeline 实例补上实例级临界区,避免直接调用 `process_single_stock(..., single_stock_notify=True)` 时多个线程继续共享同一个 `NotificationService` 进入报告生成与发送链路,导致通知乱序、重复发送或状态污染。补充了 `tests/test_pipeline_single_stock_notify.py``tests/test_pipeline_single_notify_thread_safety.py` 的回归场景以覆盖并发单股推送与直接单股入口的串行化行为。
2525
- [修复] 🔇 **实时行情降级提示收口为单次告警** — 分析主流程获取股票名称时不再提前触发一次实时行情查询,避免每只股票重复命中 quote 链路;当某个前置实时数据源失败但后续 fallback 成功时,不再输出“实时行情获取失败”级别提示,只有在实时行情开关关闭或全部数据源都不可用时,才提示已降级为历史收盘价继续分析。
2626

2727
## [3.11.0] - 2026-03-27

docs/README_CHT.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -154,6 +154,8 @@
154154

155155
默認每個工作日 **18:00(北京時間)** 自動執行
156156

157+
> 斷點續傳與 `--dry-run` 的資料存在性判斷,現在會按股票所屬市場的本地時區與交易日曆解析「最新可復用交易日」;週末 / 節假日會復用最近交易日,交易日盤中會復用上一個已完成交易日,盤後若當日資料已落庫則可直接跳過。詳細規則見 [完整配置指南](full-guide.md)
158+
157159
### 方式二:本地運行 / Docker 部署
158160

159161
> 📖 本地運行、Docker 部署詳細步驟請參考 [完整配置指南](full-guide.md)

docs/README_EN.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -171,6 +171,8 @@ The system will:
171171
- Send analysis reports to all configured channels
172172
- Save reports locally
173173

174+
> Resume fetch and `--dry-run` data-existence checks now resolve the "latest reusable trading day" from each market's local timezone and trading calendar. Weekends and holidays reuse the most recent trading day, intraday runs reuse the last completed trading day, and after market close the run skips only if the current trading day's data is already stored. See [Full Guide](full-guide_EN.md) for the exact rules.
175+
174176
---
175177

176178
### Option 2: Local Deployment

docs/full-guide.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -522,6 +522,10 @@ docker run -e SCHEDULE_ENABLED=true -e SCHEDULE_RUN_IMMEDIATELY=false ...
522522
- 使用 `exchange-calendars` 区分 A 股 / 港股 / 美股各自的交易日历(含节假日)
523523
- 混合持仓时,每只股票只在其市场开市日分析,休市股票当日跳过
524524
- 全部相关市场均为非交易日时,整体跳过执行(不启动 pipeline、不发推送)
525+
- 断点续传和 `--dry-run` 的“数据已存在”判断共用同一套“最新可复用交易日”解析逻辑,不再直接使用服务器自然日
526+
- `最新可复用交易日` 会按股票所属市场的本地时区解析:A 股使用 `Asia/Shanghai`,港股使用 `Asia/Hong_Kong`,美股使用 `America/New_York`
527+
- 非交易日(周末 / 节假日)运行时,会回退到最近一个交易日检查本地数据;若该交易日数据已存在,则跳过重复抓取,否则继续补数
528+
- 交易日盘中或收盘前运行时,会以上一个已完成交易日作为复用目标;交易日收盘后运行时,当日数据已存在则可直接跳过,不存在则继续抓取
525529
- 覆盖方式:`TRADING_DAY_CHECK_ENABLED=false` 或 命令行 `--force-run`
526530

527531
#### 使用 Crontab

src/core/pipeline.py

Lines changed: 63 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@
1717
import uuid
1818
from collections import defaultdict
1919
from concurrent.futures import ThreadPoolExecutor, as_completed
20-
from datetime import date, timedelta
20+
from datetime import date, datetime, timedelta, timezone
2121
from typing import List, Dict, Any, Optional, Tuple
2222

2323
import pandas as pd
@@ -39,7 +39,12 @@
3939
from src.services.social_sentiment_service import SocialSentimentService
4040
from src.enums import ReportType
4141
from src.stock_analyzer import StockTrendAnalyzer, TrendAnalysisResult
42-
from src.core.trading_calendar import get_market_for_stock, is_market_open
42+
from src.core.trading_calendar import (
43+
get_effective_trading_date,
44+
get_market_for_stock,
45+
get_market_now,
46+
is_market_open,
47+
)
4348
from data_provider.us_index_mapping import is_us_stock_code
4449
from bot.models import BotMessage
4550

@@ -135,19 +140,21 @@ def __init__(
135140
def fetch_and_save_stock_data(
136141
self,
137142
code: str,
138-
force_refresh: bool = False
143+
force_refresh: bool = False,
144+
current_time: Optional[datetime] = None,
139145
) -> Tuple[bool, Optional[str]]:
140146
"""
141147
获取并保存单只股票数据
142148
143149
断点续传逻辑:
144-
1. 检查数据库是否已有今日数据
150+
1. 检查数据库是否已有最新可复用交易日数据
145151
2. 如果有且不强制刷新,则跳过网络请求
146152
3. 否则从数据源获取并保存
147153
148154
Args:
149155
code: 股票代码
150156
force_refresh: 是否强制刷新(忽略本地缓存)
157+
current_time: 本轮运行冻结的参考时间,用于统一断点续传目标交易日判断
151158
152159
Returns:
153160
Tuple[是否成功, 错误信息]
@@ -157,16 +164,15 @@ def fetch_and_save_stock_data(
157164
# 首先获取股票名称
158165
stock_name = self.fetcher_manager.get_stock_name(code, allow_realtime=False)
159166

160-
today = date.today()
161-
# 注意:这里用自然日 date.today() 做“断点续传”判断。
162-
# 若在周末/节假日/非交易日运行,或机器时区不在中国,可能出现:
163-
# - 数据库已有最新交易日数据但仍会重复拉取(has_today_data 返回 False)
164-
# - 或在跨日/时区偏移时误判“今日已有数据”
165-
# 该行为目前保留(按需求不改逻辑),但如需更严谨可改为“最新交易日/数据源最新日期”判断。
166-
167-
# 断点续传检查:如果今日数据已存在,跳过
168-
if not force_refresh and self.db.has_today_data(code, today):
169-
logger.info(f"{stock_name}({code}) 今日数据已存在,跳过获取(断点续传)")
167+
target_date = self._resolve_resume_target_date(
168+
code, current_time=current_time
169+
)
170+
171+
# 断点续传检查:如果最新可复用交易日的数据已存在,则跳过
172+
if not force_refresh and self.db.has_today_data(code, target_date):
173+
logger.info(
174+
f"{stock_name}({code}) {target_date} 数据已存在,跳过获取(断点续传)"
175+
)
170176
return True, None
171177

172178
# 从数据源获取数据
@@ -295,7 +301,8 @@ def analyze_stock(self, code: str, report_type: ReportType, query_id: str) -> Op
295301
# Step 3: 趋势分析(基于交易理念)— 在 Agent 分支之前执行,供两条路径共用
296302
trend_result: Optional[TrendAnalysisResult] = None
297303
try:
298-
end_date = date.today()
304+
_mkt = get_market_for_stock(normalize_stock_code(code))
305+
end_date = get_market_now(_mkt).date()
299306
start_date = end_date - timedelta(days=89) # ~60 trading days for MA60
300307
historical_bars = self.db.get_data_range(code, start_date, end_date)
301308
if historical_bars:
@@ -379,10 +386,13 @@ def analyze_stock(self, code: str, report_type: ReportType, query_id: str) -> Op
379386

380387
if context is None:
381388
logger.warning(f"{stock_name}({code}) 无法获取历史行情数据,将仅基于新闻和实时行情分析")
389+
_mkt_date = get_market_now(
390+
get_market_for_stock(normalize_stock_code(code))
391+
).date()
382392
context = {
383393
'code': code,
384394
'stock_name': stock_name,
385-
'date': date.today().isoformat(),
395+
'date': _mkt_date.isoformat(),
386396
'data_missing': True,
387397
'today': {},
388398
'yesterday': {}
@@ -566,7 +576,9 @@ def _enhance_context(
566576
enhanced['ma_status'] = self._compute_ma_status(
567577
price, trend_result.ma5, trend_result.ma10, trend_result.ma20
568578
)
569-
enhanced['date'] = date.today().isoformat()
579+
enhanced['date'] = get_market_now(
580+
get_market_for_stock(normalize_stock_code(enhanced.get('code', '')))
581+
).date().isoformat()
570582
if yesterday_close is not None:
571583
try:
572584
yc = float(yesterday_close)
@@ -950,7 +962,8 @@ def _augment_historical_with_realtime(
950962
if not enable_realtime_tech:
951963
return df
952964
market = get_market_for_stock(code)
953-
if market and not is_market_open(market, date.today()):
965+
market_today = get_market_now(market).date()
966+
if market and not is_market_open(market, market_today):
954967
return df
955968

956969
last_val = df['date'].max()
@@ -968,7 +981,7 @@ def _augment_historical_with_realtime(
968981
amt = getattr(realtime_quote, 'amount', None)
969982
pct = getattr(realtime_quote, 'change_pct', None)
970983

971-
if last_date >= date.today():
984+
if last_date >= market_today:
972985
# Update last row with realtime close (copy to avoid mutating caller's df)
973986
df = df.copy()
974987
idx = df.index[-1]
@@ -989,7 +1002,7 @@ def _augment_historical_with_realtime(
9891002
# Append virtual today row
9901003
new_row = {
9911004
'code': code,
992-
'date': date.today(),
1005+
'date': market_today,
9931006
'open': open_p,
9941007
'high': high_p,
9951008
'low': low_p,
@@ -1019,6 +1032,16 @@ def _build_context_snapshot(
10191032
"chip_distribution_raw": self._safe_to_dict(chip_data),
10201033
}
10211034

1035+
@staticmethod
1036+
def _resolve_resume_target_date(
1037+
code: str, current_time: Optional[datetime] = None
1038+
) -> date:
1039+
"""
1040+
Resolve the trading date used by checkpoint/resume checks.
1041+
"""
1042+
market = get_market_for_stock(normalize_stock_code(code))
1043+
return get_effective_trading_date(market, current_time=current_time)
1044+
10221045
@staticmethod
10231046
def _safe_to_dict(value: Any) -> Optional[Dict[str, Any]]:
10241047
"""
@@ -1092,6 +1115,7 @@ def process_single_stock(
10921115
single_stock_notify: bool = False,
10931116
report_type: ReportType = ReportType.SIMPLE,
10941117
analysis_query_id: Optional[str] = None,
1118+
current_time: Optional[datetime] = None,
10951119
) -> Optional[AnalysisResult]:
10961120
"""
10971121
处理单只股票的完整流程
@@ -1110,6 +1134,7 @@ def process_single_stock(
11101134
skip_analysis: 是否跳过 AI 分析
11111135
single_stock_notify: 是否启用单股推送模式(每分析完一只立即推送)
11121136
report_type: 报告类型枚举(从配置读取,Issue #119)
1137+
current_time: 本轮运行冻结的参考时间,用于统一断点续传目标交易日判断
11131138
11141139
Returns:
11151140
AnalysisResult 或 None
@@ -1118,7 +1143,9 @@ def process_single_stock(
11181143

11191144
try:
11201145
# Step 1: 获取并保存数据
1121-
success, error = self.fetch_and_save_stock_data(code)
1146+
success, error = self.fetch_and_save_stock_data(
1147+
code, current_time=current_time
1148+
)
11221149

11231150
if not success:
11241151
logger.warning(f"[{code}] 数据获取失败: {error}")
@@ -1197,6 +1224,9 @@ def run(
11971224
logger.info(f"===== 开始分析 {len(stock_codes)} 只股票 =====")
11981225
logger.info(f"股票列表: {', '.join(stock_codes)}")
11991226
logger.info(f"并发数: {self.max_workers}, 模式: {'仅获取数据' if dry_run else '完整分析'}")
1227+
1228+
# 冻结本轮运行的统一参考时间,避免跨市场收盘边界时同批股票使用不同目标交易日。
1229+
resume_reference_time = datetime.now(timezone.utc)
12001230

12011231
# === 批量预取实时行情(优化:避免每只股票都触发全量拉取)===
12021232
# 只有股票数量 >= 5 时才进行预取,少量股票直接逐个查询更高效
@@ -1243,6 +1273,7 @@ def run(
12431273
single_stock_notify=False,
12441274
report_type=report_type, # Issue #119: 传递报告类型
12451275
analysis_query_id=uuid.uuid4().hex,
1276+
current_time=resume_reference_time,
12461277
): code
12471278
for code in stock_codes
12481279
}
@@ -1278,8 +1309,17 @@ def run(
12781309

12791310
# dry-run 模式下,数据获取成功即视为成功
12801311
if dry_run:
1281-
# 检查哪些股票的数据今天已存在
1282-
success_count = sum(1 for code in stock_codes if self.db.has_today_data(code))
1312+
# 检查哪些股票的最新可复用交易日数据已存在
1313+
success_count = sum(
1314+
1
1315+
for code in stock_codes
1316+
if self.db.has_today_data(
1317+
code,
1318+
self._resolve_resume_target_date(
1319+
code, current_time=resume_reference_time
1320+
),
1321+
)
1322+
)
12831323
fail_count = len(stock_codes) - success_count
12841324
else:
12851325
success_count = len(results)

src/core/trading_calendar.py

Lines changed: 74 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@
1515
import logging
1616
from datetime import date, datetime
1717
from typing import Optional, Set
18+
from zoneinfo import ZoneInfo
1819

1920
logger = logging.getLogger(__name__)
2021

@@ -90,6 +91,79 @@ def is_market_open(market: str, check_date: date) -> bool:
9091
return True
9192

9293

94+
def get_market_now(
95+
market: Optional[str], current_time: Optional[datetime] = None
96+
) -> datetime:
97+
"""
98+
Return current time in the market's local timezone.
99+
100+
If current_time is naive, treat it as already expressed in the market timezone.
101+
Unknown markets fall back to the given datetime (or local system time).
102+
"""
103+
tz_name = MARKET_TIMEZONE.get(market or "")
104+
105+
if current_time is None:
106+
if tz_name:
107+
return datetime.now(ZoneInfo(tz_name))
108+
return datetime.now()
109+
110+
if not tz_name:
111+
return current_time
112+
113+
tz = ZoneInfo(tz_name)
114+
if current_time.tzinfo is None:
115+
return current_time.replace(tzinfo=tz)
116+
return current_time.astimezone(tz)
117+
118+
119+
def get_effective_trading_date(
120+
market: Optional[str], current_time: Optional[datetime] = None
121+
) -> date:
122+
"""
123+
Resolve the latest reusable daily-bar date for checkpoint/resume logic.
124+
125+
Rules:
126+
- Non-trading day / holiday: previous trading session
127+
- Trading day before market close: previous completed trading session
128+
- Trading day after market close: current trading session
129+
- Calendar lookup failure: fail-open to market-local natural date
130+
"""
131+
market_now = get_market_now(market, current_time=current_time)
132+
fallback_date = market_now.date()
133+
134+
if not _XCALS_AVAILABLE:
135+
return fallback_date
136+
137+
ex = MARKET_EXCHANGE.get(market or "")
138+
tz_name = MARKET_TIMEZONE.get(market or "")
139+
if not ex or not tz_name:
140+
return fallback_date
141+
142+
try:
143+
cal = xcals.get_calendar(ex)
144+
local_date = market_now.date()
145+
146+
if not cal.is_session(local_date):
147+
return cal.date_to_session(local_date, direction="previous").date()
148+
149+
session = cal.date_to_session(local_date, direction="previous")
150+
session_close = cal.session_close(session)
151+
if hasattr(session_close, "tz_convert"):
152+
close_local = session_close.tz_convert(tz_name).to_pydatetime()
153+
elif session_close.tzinfo is not None:
154+
close_local = session_close.astimezone(ZoneInfo(tz_name))
155+
else:
156+
close_local = session_close.replace(tzinfo=ZoneInfo(tz_name))
157+
158+
if market_now >= close_local:
159+
return session.date()
160+
161+
return cal.previous_session(session).date()
162+
except Exception as e:
163+
logger.warning("trading_calendar.get_effective_trading_date fail-open: %s", e)
164+
return fallback_date
165+
166+
93167
def get_open_markets_today() -> Set[str]:
94168
"""
95169
Get markets that are open today (by each market's local timezone).
@@ -100,7 +174,6 @@ def get_open_markets_today() -> Set[str]:
100174
if not _XCALS_AVAILABLE:
101175
return {"cn", "hk", "us"}
102176
result: Set[str] = set()
103-
from zoneinfo import ZoneInfo
104177
for mkt, tz_name in MARKET_TIMEZONE.items():
105178
try:
106179
tz = ZoneInfo(tz_name)

0 commit comments

Comments
 (0)