Compare commits

...
19 Commits
Author SHA1 Message Date
wangyunlong b1861f69cd chore(workbench): 移除 dsh 工作台集成
工作台以 deepseek-harness 为 agent 底座,把现有工作流以 dsh 插件接入 Web UI。
该路线已停用,移除整个 workbench/ 与配套文档,避免留下点了启动不起来的死入口。

- 删除 workbench/:宿主插件 gyxx-workbench.mjs、侧边栏面板前端、cordis 配置
  模板、启动脚本、独立面板与 node 冒烟测试
- 删除 docs/workbench.md 与 README 的「智能工作台」章节
- .gitignore 去掉 workbench 本地产物两条规则
- 基线结构测试的 docs 白名单同步移除 workbench.md

工作流能力不受影响:gyxx console、调度器、运行记录与诊断接口均未改动。
2026-10-19 11:19:31 +08:00
wangyunlong 2ef635be0e feat(ops): 执行日志与 cookie 失效遥测入库
新增 PostgreSQL 运维审计层,本地 run.json 仍是权威源,数据库做跨机器可查、
可长期留存的关系型镜像。

数据面:
- ops_workflow_runs / ops_workflow_run_steps:每次运行与每次步骤尝试一行,
  保留状态、模式、耗时、exit_code、错误与 journal 路径
- ops_account_cookie_events:登录态观测事件流,只存 cookie 指纹,不落明文值
- ops_account_cookie_lifetimes / ops_account_cookie_stats:一份 cookie 的实际
  寿命与账号级平均/中位/最短/最长,用于发现失效规律;同时记录 cookie 自带
  的声明过期时间,可与实际寿命对比区分自然到期与服务端提前吊销

采集点:
- RunIndex.index_journal 镜像运行记录,覆盖 CLI、控制台与调度器全部路径
- CookieGate 的 gate 判定与 AccountKeepaliveManager 的周期探活(默认 10 分钟)
  共同提供 cookie 观测,探活决定失效时间的精度

可用性:
- 新增 gyxx audit runs|cookies|cookie-stats|schema,支持 --json
- 首次写入自动幂等建表,建表用 advisory lock 串行化,避免并发 DDL 死锁
- 审计写入永不抛出异常:连续失败熔断冷却并落 logs/ops-audit 日志,
  数据库不可用不影响工作流执行
2026-10-19 11:01:14 +08:00
wangyunlong cf85985753 chore: 忽略 .mimosa/.workbuddy 本地工具状态目录 2026-10-14 18:40:11 +08:00
wangyunlong e5761bd828 feat(deploy): Windows 自动登录自启脚本与运行手册补章
autologon.ps1 交互输密码启用/撤销 AutoAdminLogon(DefaultPassword 仅
Administrators 可读),补齐「开机自动登录 → Startup keepalive → 控制台+
调度器」自启链唯一缺口;采集依赖 CDP 交互会话,不能用服务/SYSTEM 任务
替代。runbook 增补开机自启与重启验收章节。
2026-10-14 18:40:11 +08:00
wangyunlong 9feb188d93 fix(cookie-gate): 必需 cookie 改全量校验并收紧各账号必需名单
required_cookie_names 原语义是「任一必需名命中即有效」,单个匿名 cookie
(如小红书 a1)就能把游客会话判为有效;改为全部必需名都存在、存活且域名
匹配才算通过。随之按实际登录产物收紧名单:jd 去 pin、erp321 留
u_id+u_co_id、小红书改 customer-sso-sid+solar.beaker.session.id、
星图仅 sid_guard。cookie_gate 用例冻结时钟改相对时钟防过期漂移。
2026-10-14 18:38:03 +08:00
wangyunlong 7b305b0b12 fix(content): 蝉妈妈导出标签页崩溃自愈与 cookie 文件格式对齐
导出点击后 tab 死亡(渲染进程被清理)时:页面级死亡先同 context 重开
新页(facade 新增 reopen_page),context 也死亡才整套重建驱动,按
profile → cookie 文件 → 账密顺序恢复会话,两种自愈各限一次防死循环。
cookie 文件改存裸列表,与 vault 同步及适配器 BrowserCookieStore 三方
格式对齐(字典格式会被适配器拒绝加载);adapters 暴露
is_browser_closed_error 供模块识别目标已关闭类异常。
2026-10-14 18:35:44 +08:00
wangyunlong 6cd563eab0 fix(product): 天猫百补报名登录链路三类阻断兜底
vault 假登录态(SPA 缓存店铺名误判已登录)与 SSO 中间态停留时改走显式
账密登录兜底;登录页识别服务协议/隐私弹窗并先处理,二次验证标记补充
滑块;报名导入增加账号级锁防并发挤兑。
2026-10-14 18:33:32 +08:00
wangyunlong 0cc4fafc7c fix(product): 聚水潭 allsku 模板隐藏列错位致 daily 销量全 0
2026-10-11 模板重置把 38 列置隐藏后可见列整体左移,固定索引 45/57 落到
「今日实退货量/今日仅退款量」。daily 改表头名匹配「昨日销量/昨日退货量」
(weekly 暂留老索引待实测);backfill 恢复自定义列可见(window._Cols +
_RptRefresh),并把表头索引按可见列换算成行内位置防 td 缺位错读。
2026-10-14 18:31:23 +08:00
wangyunlong 9a91ce80e4 feat(product): 聚水潭销售主题分析按日采集工作流
collect_erp_sales_theme_daily.py 采集 bi.erp321.com「报表→销售主题分析→按日统计」,
异步导出 + OSS 直连兜底,固定品牌+店铺口径,写 erp_sales_theme_daily_metrics
(db 层建表与 upsert 同步,每日一行幂等覆盖)。注册 product.erp_sales_theme_daily.collect
手动命令,复用 erp-account 登录绑定新增 CDP 22167;目录/集成计数 60→61,
新脚本纳入 CI 聚焦 lint 清单。
2026-10-14 18:29:15 +08:00
wangyunlong 97a486e97d feat(product): 补充香港安培公开库存采集脚本与聚水潭库存适配
public_sku_inventory_sheet 与 jushuitan_inventory 适配器、hk_anpei 采集配置,
支撑 product.hk_anpei_public_inventory.daily 工作流。
2026-10-09 13:45:56 +08:00
wangyunlong 36fe7b81f8 feat(cookie-gate): 补充 cookie_gate 网关模块与 deploy/watchdog 保活脚本
cookie_gate.py(账号三级门控/阻断记录 pending_reruns/恢复重跑管理器)为此前
接线提交遗漏的模块文件;deploy/watchdog 为控制台与调度器的进程外保活脚本。
2026-10-09 13:45:56 +08:00
wangyunlong dc8be88e60 chore(config): 工作流配置、清单与文档同步
workflows/schedules/commands/notification-routing 与 source-manifests 四模块
清单同步登录统一化与新工作流(product.hk_anpei_public_inventory.daily 等);
docs 运维手册、控制台与耗时分布文档、CI 关键脚本清单、AGENTS 指南更新;
测试与模块注册对齐。
2026-10-09 13:45:30 +08:00
wangyunlong 90282ae766 fix(product): 千川任务中心账户选择页兜底与 jd_spus jsonb 查询修复
千川任务中心深链缺账户上下文时会被重定向到账户选择页,按 .account-list-item
卡片自动选中目标广告主后继续下载;jd_spus 列改 jsonb 后查询改用
jsonb_typeof/jsonb_array_length 防类型报错。
2026-10-09 13:45:30 +08:00
wangyunlong df11e195b7 feat(product): 京东新版商智竞店采集改造
入口跳转、两段式日期选择、搜索定位与表格解析适配新版商智;配合类目下拉与
XHR 拦截兜底。
2026-10-09 13:45:30 +08:00
wangyunlong b0e1b5066d feat(content): 自营三平台指标入库管线完善
sync_self_video_metrics 按平台隔离入库与字段口径调整,配套自营指标验收测试
更新。
2026-10-09 13:45:30 +08:00
wangyunlong 1e9ddffd2f fix(collect): 账号浏览器与小红书采集强制直连
目标站点均为国内站,Clash 代理出口被蝉妈妈/小红书 WAF 拉黑(TLS 直接 reset,
页面与 API 报 ERR_CONNECTION_CLOSED)。账号保活、扫码登录浏览器与小红书 ARK
采集统一加 --no-proxy-server 直连。
2026-10-09 13:45:29 +08:00
wangyunlong 41f448420d fix(chanmama): 导出确认弹窗被浮层清理误删与代理出口拉黑修复
浮层清理只删可见浮层——页面常驻 90+ 个 display:none 弹窗模板(含导出确认框
模板),误删后 Vue 只在游离节点上"打开"弹窗,页面上仅剩灰色遮罩与滚动锁,
导出等待 1320s 必超时;导出等待改为逐秒弹窗状态机(可点即点/点不到先关/卡死
强删并补点导出),导出失败落截图与 HTML 诊断;浏览器强制直连并切换真 Chrome
非 stealthy 模式(StealthySession 下蝉妈妈前端会静默压制弹窗渲染);导出按钮
原生可信点击优先,防 @click 绑定在内层元素时 JS 合成点击无效。
2026-10-09 13:45:29 +08:00
wangyunlong 130f06b320 feat(cookie-gate): 调度器恢复重跑接线与控制台健康自检
调度器维护循环挂载 CookieGateRecoveryManager,登录态恢复后按原业务日期自动
重跑被阻断工作流并飞书确认;工作流级 required_accounts 声明接入目录解析;
采集器抛 CookieExpiredError 退出 75;控制台新增 /api/health 自检端点与进程内
看门狗,配合 deploy/watchdog 双层保活;安装脚本同步。
2026-10-09 13:45:29 +08:00
wangyunlong 78ea4641ce refactor(login): 移除脚本内扫码重登,登录统一交给账号 vault 与 cookie-gate
删除 relogin_*/friday_relogin_parallel 子进程及其测试;批量补采、评论采集、
星图/蒲公英/B站/抖音自营与抖店竞品脚本不再内嵌扫码等待,登录态失效统一由
cookie-gate 阻断并走 gyxx accounts login 恢复;runtime-bindings 与模块注册
同步移除 content.relogin.weekly。
2026-10-09 13:45:19 +08:00
127 changed files with 10011 additions and 9126 deletions

No files matched your search

+4 -4
View File
@@ -33,12 +33,12 @@ Thumbs.db
# Agent-local learning logs
.learnings/
# Agent tool local state (security-scan ledger, session memory)
/.mimosa/
/.workbuddy/
# Local tool artifacts
/.playwright-cli/
/output/
/nul
/new_all_bag.xlsx
# Workbench (dsh plugin) local artifacts
/workbench/plugin/node_modules/
/workbench/cordis.local.yml
+1 -1
View File
@@ -19,7 +19,7 @@ lint:
stage: quality
script:
- uv run ruff check src tests
- uv run ruff check src/gyxx_flow/modules/content_marketing/daily_creator_exposure_scope.py src/gyxx_flow/modules/content_marketing/chanmama_export.py src/gyxx_flow/modules/content_marketing/self_operated_accounts.py src/gyxx_flow/modules/content_marketing/self_xhs_scraper.py src/gyxx_flow/modules/content_marketing/data/tools/sync_self_video_metrics.py src/gyxx_flow/modules/product_commerce/market_rank_limits.py src/gyxx_flow/modules/product_commerce/market_rank_product_import.py src/gyxx_flow/modules/shop_intelligence/collectors/dy_after_sales_quality.py src/gyxx_flow/modules/shop_intelligence/collectors/tmall_negative_reviews.py
- uv run ruff check src/gyxx_flow/modules/content_marketing/daily_creator_exposure_scope.py src/gyxx_flow/modules/content_marketing/chanmama_export.py src/gyxx_flow/modules/content_marketing/self_operated_accounts.py src/gyxx_flow/modules/content_marketing/self_xhs_scraper.py src/gyxx_flow/modules/content_marketing/data/tools/sync_self_video_metrics.py src/gyxx_flow/modules/product_commerce/market_rank_limits.py src/gyxx_flow/modules/product_commerce/market_rank_product_import.py src/gyxx_flow/modules/product_commerce/collect_douyin_qianchuan_ads.py src/gyxx_flow/modules/product_commerce/collect_erp_sales_theme_daily.py src/gyxx_flow/modules/shop_intelligence/collectors/dy_after_sales_quality.py src/gyxx_flow/modules/shop_intelligence/collectors/tmall_negative_reviews.py
test:
stage: test
+1
View File
@@ -41,6 +41,7 @@ uv run gyxx accounts login|sync [account_id]
- Manual/backfill/maintenance commands must be declared in `config/commands.json`; never reintroduce recursive executable-script discovery.
- Each executable `workflow_id` compiles to a LangGraph `StateGraph`. Preserve the public contracts of the workflow model, `RunJournal`, `LockManager`, shadow comparison, and `EffectLedger` when changing graph execution.
- External systems: PostgreSQL via runtime-injected cloud DSN (`GYXX_POSTGRES_DSN`; no address/credential defaults in source, loopback rejected in cloud mode); Hermes loopback-only roles on `127.0.0.1:8642/8643` (key `GYXX_HERMES_API_KEY`); Feishu keeps its existing identity. Every browser binding keeps a unique CDP port plus isolated Profile/Cookie/storage-state; scripts sharing an account declare it under `accounts` in `runtime-bindings.json` — the vault `state/accounts/<id>/` is the single login authority (`gyxx accounts login/sync`) and its cookies merge read-only into member scripts while profiles stay isolated.
- 登录统一管理:正式执行的工作流由 `src/gyxx_flow/cookie_gate.py` 在采集步骤前注入每账号 `cookie-gate.<id>` 步骤(blocking 账号失效→阻断并飞书通知人工重登;`login_command_id` 账号先自动账密恢复;`credential_env_names` 账号 defer 给脚本内账密登录)。阻断记录 `pending_reruns`,由调度器维护循环里的 `CookieGateRecoveryManager` 在登录态恢复后按原业务日期自动重跑并发飞书确认。采集脚本不得内嵌扫码/验证码等待——失效时抛 `CookieExpiredError`(`adapters/browser.py`)退出 75;编排器工作流在 `workflows.json` 用 `required_accounts` 声明账号依赖。不要重新引入 relogin_*/批量补采里的扫码重登子进程。
- 启动或重启工作流控制台(`gyxx console`)必须携带 `--env-file D:\product-collector-analyze-flow\.env`(与调度器进程一致),否则 `PG_HOST/PG_PORT/PG_DB/PG_USER/PG_PASSWORD` 等云端 PostgreSQL 凭据和账号配置不会注入,「商品经营日报采集」「聚水潭全店铺款式日报」等正式执行会报“缺少 PostgreSQL 运行时凭据”。参考启动脚本:`var/tmp/start-console.ps1`。
- Source sync (`gyxx sources status/apply`): manifest-driven three-way hash drift check; only untransformed, conflict-free one-sided changes may be auto-copied, and only with `--execute`. Credentials, cookies, logs, and collected data never enter sync manifests.
-11
View File
@@ -40,17 +40,6 @@ var/ 默认运行数据;不属于源码
- `shop_intelligence`:店铺、竞店和京东自营业绩。
- `supply_chain`:采购确认、补货、库存预警和采购单更新。
## 智能工作台(dsh-refact 分支)
`workbench/` 以 [deepseek-harness](https://github.com/deepseek-ai/deepseek-harness) 为
agent 底座,把现有工作流以 dsh 插件形式接入 Web UI:侧边栏选中任意工作流即可提问、
诊断、修复,支持监控告警与受控启动。使用与设计见 [workbench/README.md](workbench/README.md)
与 [docs/workbench.md](docs/workbench.md)。
```powershell
powershell -File workbench\bin\start-workbench.ps1
```
## 开发环境
```powershell
+18 -8
View File
@@ -50,14 +50,7 @@
"kind": "python",
"notification_workflow_id": "content.marketing_report.daily"
},
{
"id": "content.login.refresh",
"module": "content_marketing",
"entry": "data/tools/friday_relogin_parallel.py",
"kind": "python",
"notification_workflow_id": "content.relogin.weekly"
},
{
{
"id": "content.creator_report.generate",
"module": "content_marketing",
"entry": "data/tools/generate_creator_report.py",
@@ -472,6 +465,23 @@
"{business_date}"
]
},
{
"id": "product.erp_sales_theme_daily.collect",
"module": "product_commerce",
"entry": "collect_erp_sales_theme_daily.py",
"kind": "python",
"default_args": [
"--date",
"{business_date}",
"--execute"
]
},
{
"id": "product.hk_anpei_public_inventory.refresh",
"module": "product_commerce",
"entry": "public_sku_inventory_sheet.py",
"kind": "python"
},
{
"id": "product.backfill.run",
"module": "product_commerce",
-3
View File
@@ -222,9 +222,6 @@
"content.marketing_report.daily": {
"condition": "正式执行且启用发送时投递最终营销日报。"
},
"content.relogin.weekly": {
"condition": "刷新期间投递各平台扫码二维码,并在平台失败时投递失败汇总。"
},
"product.alert.daily": {
"condition": "命中销量下滑规则且尚未重复通知时投递告警。"
},
+36 -114
View File
@@ -88,7 +88,6 @@
"jd.com"
],
"required_cookie_names": [
"pin",
"thor"
],
"credential_env_names": [
@@ -116,7 +115,6 @@
"jd.com"
],
"required_cookie_names": [
"pin",
"thor"
],
"credential_env_names": [
@@ -145,10 +143,7 @@
],
"required_cookie_names": [
"u_id",
"u_co_id",
"ASP.NET_SessionId",
".ASPXAUTH",
"token"
"u_co_id"
],
"credential_env_names": [
"ERP_USERNAME",
@@ -168,6 +163,7 @@
"product_commerce:collect_erp_yesterday_metrics.py",
"product_commerce:erp_login_product_analysis.py",
"product.erp_all_shop_daily",
"product.erp_sales_theme_daily.collect",
"product.sales_sheet.sync",
"supply_chain:orchestrator/scripts/ProductReplenishment.py",
"supply_chain:orchestrator/scripts/PurchaseConfirmation.py",
@@ -298,7 +294,8 @@
"xiaohongshu.com"
],
"required_cookie_names": [
"web_session",
"customer-sso-sid",
"solar.beaker.session.id",
"a1"
],
"keepalive": {
@@ -312,8 +309,6 @@
},
"members": [
"content.comments.collect_xiaohongshu",
"content_marketing:data/tools/relogin_pgy.py",
"content_marketing:data/tools/relogin_xiaohongshu.py",
"content_marketing:pgy_xhs_scraper_v2.py"
]
},
@@ -340,7 +335,6 @@
},
"members": [
"content.comments.collect_douyin",
"content_marketing:data/tools/relogin_douyin.py",
"content_marketing:self_douyin_scraper.py"
]
},
@@ -353,9 +347,6 @@
"xingtu.cn"
],
"required_cookie_names": [
"sessionid",
"sessionid_ss",
"sid_tt",
"sid_guard"
],
"keepalive": {
@@ -367,13 +358,15 @@
"headless": true,
"real_chrome": true
},
"members": [
"content_marketing:data/tools/relogin_xingtu.py"
]
"members": []
},
"chanmama": {
"cdp_port": 22207,
"login_mode": "D",
"credential_env_names": [
"CHANMAMA_ACCOUNT",
"CHANMAMA_PASSWORD"
],
"required_cookie_domains": [
"chanmama.com"
],
@@ -411,8 +404,7 @@
},
"members": [
"content.comments.collect_bilibili",
"content.metrics.collect_self_bilibili",
"content_marketing:data/tools/relogin_bilibili.py"
"content.metrics.collect_self_bilibili"
]
}
},
@@ -582,23 +574,6 @@
],
"cdp_port": 22016
},
"content_marketing:data/tools/friday_relogin.bat": {
"script_id": "content_marketing:data/tools/friday_relogin.bat",
"state_key": "content_marketing:data/tools/friday_relogin.bat",
"aliases": [
"content_marketing:data/tools/friday_relogin.bat"
],
"cdp_port": 22017
},
"content.login.refresh": {
"script_id": "content_marketing:data/tools/friday_relogin_parallel.py",
"state_key": "content_marketing:data/tools/friday_relogin_parallel.py",
"aliases": [
"content_marketing:data/tools/friday_relogin_parallel.py"
],
"cdp_port": 22018,
"login_mode": "D"
},
"content.creator_report.generate": {
"script_id": "content_marketing:data/tools/generate_creator_report.py",
"state_key": "content_marketing:data/tools/generate_creator_report.py",
@@ -679,83 +654,6 @@
],
"cdp_port": 22135
},
"content_marketing:data/tools/relogin_bilibili.py": {
"script_id": "content_marketing:data/tools/relogin_bilibili.py",
"state_key": "content_marketing:data/tools/relogin_bilibili.py",
"aliases": [
"content_marketing:data/tools/relogin_bilibili.py"
],
"cdp_port": 22028,
"login_mode": "D",
"required_cookie_domains": [
"bilibili.com"
],
"required_cookie_names": [
"SESSDATA"
]
},
"content_marketing:data/tools/relogin_douyin.py": {
"script_id": "content_marketing:data/tools/relogin_douyin.py",
"state_key": "content_marketing:data/tools/relogin_douyin.py",
"aliases": [
"content_marketing:data/tools/relogin_douyin.py"
],
"cdp_port": 22029,
"login_mode": "D",
"required_cookie_domains": [
"douyin.com"
],
"required_cookie_names": [
"sessionid"
]
},
"content_marketing:data/tools/relogin_pgy.py": {
"script_id": "content_marketing:data/tools/relogin_pgy.py",
"state_key": "content_marketing:data/tools/relogin_pgy.py",
"aliases": [
"content_marketing:data/tools/relogin_pgy.py"
],
"cdp_port": 22030,
"login_mode": "D",
"required_cookie_domains": [
"xiaohongshu.com"
]
},
"content_marketing:data/tools/relogin_xiaohongshu.py": {
"script_id": "content_marketing:data/tools/relogin_xiaohongshu.py",
"state_key": "content_marketing:data/tools/relogin_xiaohongshu.py",
"aliases": [
"content_marketing:data/tools/relogin_xiaohongshu.py"
],
"cdp_port": 22031,
"login_mode": "D",
"required_cookie_domains": [
"xiaohongshu.com"
],
"required_cookie_names": [
"web_session"
]
},
"content_marketing:data/tools/relogin_xingtu.py": {
"script_id": "content_marketing:data/tools/relogin_xingtu.py",
"state_key": "content_marketing:data/tools/relogin_xingtu.py",
"aliases": [
"content_marketing:data/tools/relogin_xingtu.py"
],
"cdp_port": 22032,
"login_mode": "D",
"required_cookie_domains": [
"douyin.com",
"oceanengine.com",
"xingtu.cn"
],
"required_cookie_names": [
"sessionid",
"sessionid_ss",
"sid_tt",
"sid_guard"
]
},
"content.failed.retry": {
"script_id": "content_marketing:data/tools/retry_failed.py",
"state_key": "content_marketing:data/tools/retry_failed.py",
@@ -796,8 +694,12 @@
],
"cdp_port": 22165,
"login_mode": "C",
"required_cookie_domains": ["xiaohongshu.com"],
"credential_env_names": ["XHS_ARK_ACCOUNT"]
"required_cookie_domains": [
"xiaohongshu.com"
],
"credential_env_names": [
"XHS_ARK_ACCOUNT"
]
},
"content_marketing:data/tools/sync_style_categories.py": {
"script_id": "content_marketing:data/tools/sync_style_categories.py",
@@ -1441,6 +1343,26 @@
"erp321.com"
]
},
"product.erp_sales_theme_daily.collect": {
"script_id": "product_commerce:collect_erp_sales_theme_daily.py",
"state_key": "product_commerce:collect_erp_sales_theme_daily.py",
"aliases": [
"product_commerce:collect_erp_sales_theme_daily.py"
],
"cdp_port": 22167,
"login_mode": "C",
"required_cookie_domains": [
"erp321.com"
]
},
"product.hk_anpei_public_inventory.refresh": {
"script_id": "product_commerce:public_sku_inventory_sheet.py",
"state_key": "product_commerce:public_sku_inventory_sheet.py",
"aliases": [
"product_commerce:public_sku_inventory_sheet.py"
],
"cdp_port": 22166
},
"product_commerce:scripts/insert_jd_main_image_records.py": {
"script_id": "product_commerce:scripts/insert_jd_main_image_records.py",
"state_key": "product_commerce:scripts/insert_jd_main_image_records.py",
+9 -14
View File
@@ -11,10 +11,7 @@
"workflow_id": "content.metrics.daily"
},
{
"at": [
"01:00",
"05:00"
],
"at": "06:00",
"business_date_offset_days": 0,
"enabled": true,
"kind": "daily",
@@ -42,16 +39,7 @@
"kind": "daily",
"workflow_id": "content.marketing_report.daily"
},
{
"at": "10:00",
"days": [
"Friday"
],
"enabled": false,
"kind": "weekly",
"workflow_id": "content.relogin.weekly"
},
{
{
"at": "12:00",
"days": [
"Sunday"
@@ -221,6 +209,13 @@
"kind": "daily",
"workflow_id": "product.erp_all_shop_daily"
},
{
"at": "09:00",
"business_date_offset_days": 0,
"enabled": true,
"kind": "daily",
"workflow_id": "product.hk_anpei_public_inventory.daily"
},
{
"at": "12:00",
"business_date_offset_days": 0,
+13 -69
View File
@@ -7,7 +7,7 @@
"source_relative_path": "bilibili_comment_scraper.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/bilibili_comment_scraper.py",
"source_sha256": "c7f116f30889125c668e7f4dd796e7249ba7ea8621d5ab5fb518028f56c7f8dc",
"target_sha256": "109a01a6dc2b79261cff68ff42818a7561f3319b6be3788d9300bb5ce0217a96",
"target_sha256": "d3ae63aab128a3e4078cf357aa4dfcf1f2c3b36c3116989d8f2252be62544e53",
"transformed": true,
"category": "source"
},
@@ -23,7 +23,7 @@
"source_relative_path": "chanmama_scraper.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/chanmama_scraper.py",
"source_sha256": "2edc26879b57d73ec5992c80e5ef74d76a509f09bff7284b1ae17e90df543e07",
"target_sha256": "8b87268e0f105cf4d349b192c7b2c24d5e318a9d88fcd14264824b083103b24e",
"target_sha256": "b808b412e9bb8d824c1a117cbecdd28234408a55fa60891f1ae748abeb7bc415",
"transformed": true,
"category": "source"
},
@@ -111,7 +111,7 @@
"source_relative_path": "data/tools/batch_rescrape_bilibili.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/data/tools/batch_rescrape_bilibili.py",
"source_sha256": "5b07fe27f1b76714ca3f3feae34e3d719277ed0cfe124d76cacd8e3c904b1fca",
"target_sha256": "10e5dee90c591201daa7a7b0eb13a40ed1c97d7d51b1b93ef8f1d5d4b8e09580",
"target_sha256": "8fa0e5838615f36c9841deb08f6710693e63b011f1f91138c1121d99277dd391",
"transformed": true,
"category": "tool"
},
@@ -119,7 +119,7 @@
"source_relative_path": "data/tools/batch_rescrape_douyin.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/data/tools/batch_rescrape_douyin.py",
"source_sha256": "15b1438513f8216a6c7a103744e8a1ca6bc63b32c111d7c9bce31c18be9e84a6",
"target_sha256": "3233a7840e07a87535a8df065562a7cb681dbc140e002d82970598de3d9768cf",
"target_sha256": "e0f1227a7b66d85ea65191b5f9c6c5bf588e85cc0cd66e20afae7a9dc7a2684e",
"transformed": true,
"category": "tool"
},
@@ -127,7 +127,7 @@
"source_relative_path": "data/tools/batch_rescrape_xiaohongshu.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/data/tools/batch_rescrape_xiaohongshu.py",
"source_sha256": "6d85f7dfd52c7a92b951ac29e7ee3377aebdb743af2a362d1f38366188dcad08",
"target_sha256": "01be5d1ba260ccd46e01b9e72b6f7dd3d08577615455fc9010de4efe8a710aea",
"target_sha256": "c481bd6af34040388abafed0466d5632b3892865bf192d245664627003a4c43f",
"transformed": true,
"category": "tool"
},
@@ -183,7 +183,7 @@
"source_relative_path": "data/tools/db.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/data/tools/db.py",
"source_sha256": "83f158a78841244c1f02255d1500018b5a9dbe1f282cbdb937acc344b08abc25",
"target_sha256": "62ffad854cc2bc3a3ae4f2a28b4ee345bc9eecf72101b647c3aa165c691b456a",
"target_sha256": "bfdb761761532263ae6c555ee455cad4efa300b7fa5639931813f4f95e79ff6a",
"transformed": true,
"category": "tool"
},
@@ -203,14 +203,6 @@
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/friday_relogin_parallel.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/data/tools/friday_relogin_parallel.py",
"source_sha256": "affed4ab6e5e179c85dc92999502b98e007c71243ed26977b3c5eeba23b9e3dc",
"target_sha256": "438bedfb874c7eb566e1998815a22ca72b7d177cee221fd39625a96872bd450b",
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/generate_creator_report.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/data/tools/generate_creator_report.py",
@@ -291,54 +283,6 @@
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/relogin_bilibili.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/data/tools/relogin_bilibili.py",
"source_sha256": "b8dc12539bcd6f1a368d50ead1f5d33d6577a75c92f0b7bafa75a67edd7a73fa",
"target_sha256": "d47f26f62fbe3262128327b916151385e527125a18f27f7455b8382aa5a77ecb",
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/relogin_douyin.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/data/tools/relogin_douyin.py",
"source_sha256": "3c2017f8a801528ff17e4c4bae24615256b8788c185f9a797fe87470e35b1645",
"target_sha256": "82a0ecf4101f2eaded94cc4cd9054cb08e1e27519a58cfc0d3126888cc1ce5d4",
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/relogin_pgy.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/data/tools/relogin_pgy.py",
"source_sha256": "759c6a68a0851b4c87d8febfc5d94925831a3b2d7cdf1e6a6e74dcd52b83ae3f",
"target_sha256": "d75697ae2b6ac1f23bb79077a4df0af33113813a46525a6f3980b04391857cac",
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/relogin_transaction.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/data/tools/relogin_transaction.py",
"source_sha256": "57ec20de8d99290268ed3d1fc2b66bf3faeaca807d7f3744dd51de6cc93628b2",
"target_sha256": "57ec20de8d99290268ed3d1fc2b66bf3faeaca807d7f3744dd51de6cc93628b2",
"transformed": false,
"category": "tool"
},
{
"source_relative_path": "data/tools/relogin_xiaohongshu.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/data/tools/relogin_xiaohongshu.py",
"source_sha256": "fcdea30adecb066c4f3699ded457df0bc763d978248276ddbffa9f18bc9bdf2c",
"target_sha256": "e700dd57e5cf87d68529bcd6c062e73d6bd9f1e6ef06ef3122e2b16c288aaf1a",
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/relogin_xingtu.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/data/tools/relogin_xingtu.py",
"source_sha256": "c9d434cb02a57ebd7b884598563077d733d56e45dad59ed30774eb7e1c668cb0",
"target_sha256": "06ec1088677d69511057008acb2b05cca2da015a8219aee10bf19bbae3c25d79",
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/retry_failed.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/data/tools/retry_failed.py",
@@ -351,7 +295,7 @@
"source_relative_path": "data/tools/schema_gyxx_super_data.sql",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/data/tools/schema_gyxx_super_data.sql",
"source_sha256": "43d202d7d0452b7d0671ee9c40149c21328db4b2863fa4d030af456f89defe3c",
"target_sha256": "001870851dea9c373a77b87335604c4363b06c3eb0ed7c97b4a8d5e891e56758",
"target_sha256": "4b2eca918bb81de0cac529b786d0d37ce04af2c1bc4be81cd20c8ba9456b1348",
"transformed": true,
"category": "sql"
},
@@ -415,7 +359,7 @@
"source_relative_path": "douyin_comment_scraper.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/douyin_comment_scraper.py",
"source_sha256": "ec785ead9c95c5de65c00b06212b1f03d0159fed475a95a2b593d3b24e486e43",
"target_sha256": "6e7ac4357fe466656b93ca7bae6c49adc4d59fa75974a39c6885c7433bd4d402",
"target_sha256": "35d3caf8bc85a82ca15c3fc9ff317c21460173996106ffee8f1d3b5f1d13b801",
"transformed": true,
"category": "source"
},
@@ -455,7 +399,7 @@
"source_relative_path": "pgy_xhs_scraper_v2.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/pgy_xhs_scraper_v2.py",
"source_sha256": "2be78ef76df38e80da2099a76e6e3fd9dcdc2e8875afb44279f54003ccb5f563",
"target_sha256": "e5df4b34f67a565ec5d5ce4dd5857e80e363dd7559e7e29006dcd3e421276c8a",
"target_sha256": "472c13700adc98144b53f2921bbdde15ebb7dc138eaab76885b514dadd306887",
"transformed": true,
"category": "source"
},
@@ -479,7 +423,7 @@
"source_relative_path": "self_bilibili_scraper.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/self_bilibili_scraper.py",
"source_sha256": "2ef650a16dde61697101bb219a8c3adcd682fec1529bb9c5312dc5b384d83a72",
"target_sha256": "6ec5ed6991a93f806fb74d0870e953e5a05d52043a83f2fa261e2ec67bee6ad7",
"target_sha256": "1c09171d1269a3a38a5e6f9f99b98d0da4c7ff11d075613fc288ee209e82e9d2",
"transformed": true,
"category": "source"
},
@@ -487,7 +431,7 @@
"source_relative_path": "self_douyin_scraper.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/self_douyin_scraper.py",
"source_sha256": "4aba1264dee052521e5712d7219916e534db5705e3cec75caaa1b025a4bc2506",
"target_sha256": "62307df115c8d7336b470baba074163ab3f3468bae8b0af266af370e2e42d9f1",
"target_sha256": "945659cd80435bab08bfe1dee13c451c8c4b5fe362313ea09591525c551d3c48",
"transformed": true,
"category": "source"
},
@@ -511,7 +455,7 @@
"source_relative_path": "xiaohongshu_comment_scraper.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/xiaohongshu_comment_scraper.py",
"source_sha256": "58bc07bb2fd67897e18a9981f931df8fadf24a6cc5f09d55bd3aaa2e3c6756a4",
"target_sha256": "afca1104324546da2315fdf7a427c3f34021eef31ddc2fea76877f818718ccb0",
"target_sha256": "fff780d9e16717e6a1ba9c5ee4db22cb1bb5a85f75807c811072c20a6bd390de",
"transformed": true,
"category": "source"
},
@@ -527,7 +471,7 @@
"source_relative_path": "xingtu_scraper_v2.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/xingtu_scraper_v2.py",
"source_sha256": "0d4eb87779064e13b8d78daa6dba13698f1149a26b3a1de2e064c117a29cf49a",
"target_sha256": "b36339444e26048a90c8ac999a0a628bcddd4f64842924f6c60e600c7a3bec37",
"target_sha256": "dfdd07af20c8e2e59411fb6e32a82588db5601d6f3fba75d596e0b5ac19b2be8",
"transformed": true,
"category": "source"
}
+25 -25
View File
@@ -14,7 +14,7 @@
{
"category": "production_source",
"source_sha256": "cff768f79672cfc7db7f6866caf5ab59936b8c51eb2a7d729b81441940124512",
"target_sha256": "d56fe8a934c1ae20a9249c7682823238a705f51f451241504512e83a8800a879",
"target_sha256": "c1c0167b96c818a4e7d44aa5c860b5510c9925cb096b7fb4510b8319b91592fc",
"transformed": true,
"source_relative_path": "analyze_style_with_hermes.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/analyze_style.py"
@@ -54,7 +54,7 @@
{
"category": "production_source",
"source_sha256": "1493dbbfb3d864812df378d04d006e7de0f9ec4b6ab5812e5c6970c304c2dfe6",
"target_sha256": "a1d526dfc27b110a8845a98cc66fe7e5cfb87ef70530728477af31a88734a576",
"target_sha256": "2b6e6adfea82b13dd8c819138a428a78b5802f204c199c58ed4e59f0158d1780",
"transformed": true,
"source_relative_path": "collect_dy_market_rank.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/collect_dy_market_rank.py"
@@ -62,7 +62,7 @@
{
"category": "production_source",
"source_sha256": "abd4c82cbd6c41e111e213c174d2fdeba1214568496eaf1dede24bf374b3b052",
"target_sha256": "9de3ab22d5a4dbf574b2d4fdbbed295bb4645bb4943f4c066fcbfd41b44b521c",
"target_sha256": "eedf83f1d85662dc2e6d968af29070be819c7caac25057c74d6cb66b75f52216",
"transformed": true,
"source_relative_path": "collect_dy_persona_to_bitable.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/collect_dy_persona_to_bitable.py"
@@ -70,7 +70,7 @@
{
"category": "production_source",
"source_sha256": "c7b54d941ab7f056e2d0ea4355543ff4a5c34ad986c8ed32145e988ac9501ea2",
"target_sha256": "aaba5dfe6ce5133b5c662232160bb9faef4d73b36902a484a56b89fa7df05311",
"target_sha256": "8576678688d7d90f6f7285110de005982bc1ba82bc7e9977224ba1bdb4101283",
"transformed": true,
"source_relative_path": "collect_erp_yesterday_metrics.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/collect_erp_yesterday_metrics.py"
@@ -78,7 +78,7 @@
{
"category": "production_source",
"source_sha256": "8bb1f17f90a7b9c2db5dfb992cf746329892d19c00ae2ae988e2109c049598b2",
"target_sha256": "c98abe563b223f49cc3c810054d997504d9465387da4805389b0f5ef78ceed6a",
"target_sha256": "9b24259854a440d5a13fa2bc397887ab86cf7d4a573b10d3bda6988ed37d0ed6",
"transformed": true,
"source_relative_path": "collect_jd_market_rank.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/collect_jd_market_rank.py"
@@ -86,7 +86,7 @@
{
"category": "production_source",
"source_sha256": "f9a48c5f89b18472dc3993e340ad858bfecb2fcf59427c7e70db37512c17a6f7",
"target_sha256": "a268df592f7136a68c2b82d031115055e43cdfaedcdd4a5f4afa9ef45879e055",
"target_sha256": "01798bd6110afc84e5a6aab50255deeb0c5980f77895fcaf605ccb93d24656a5",
"transformed": true,
"source_relative_path": "collect_jd_persona_to_bitable.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/collect_jd_persona_to_bitable.py"
@@ -110,7 +110,7 @@
{
"category": "production_source",
"source_sha256": "13b6e68b4872b2e79c1de8ca1617cc4f1249f4ed32f234ad6efb7a04f5948612",
"target_sha256": "a1b95cc4f090a25c8e7991b0ef015a7df612fc78249dafe75aeef9fa4dbefe08",
"target_sha256": "35b5838736306d58758dca870deb7a296a21a2cbc7bb0c4b84d7f233095bee22",
"transformed": true,
"source_relative_path": "collect_sycm_market_rank.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/collect_sycm_market_rank.py"
@@ -134,7 +134,7 @@
{
"category": "production_source",
"source_sha256": "a8900b9f25db6cf2bef3117a1d2cd86283054fea6f5a8590fd91b5cfbbcaf87e",
"target_sha256": "030d33c3f5c0c230bb67d603dd657f40cceff8581f8756a1f67340a89b5ae4b3",
"target_sha256": "47cdfd9ebf27e31be06497b2f74c80da4ffced259a873730eb024b0db5eba740",
"transformed": true,
"source_relative_path": "db/__init__.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/db/__init__.py"
@@ -158,7 +158,7 @@
{
"category": "production_source",
"source_sha256": "78c74c0e52e3651b3b042a2095e596ab250e60309c7ae2de905850822aad4922",
"target_sha256": "09d7e08d2eec57055a3bc7949e49c32a8bf62713f2594ae5d9297c032bf212c8",
"target_sha256": "1e6ed0029c9d1f701ae1d80b670fdb465cdbfd13b9f63d0a3179c19585215246",
"transformed": true,
"source_relative_path": "dy_audience_profile_collect.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/dy_audience_profile_collect.py"
@@ -166,7 +166,7 @@
{
"category": "production_source",
"source_sha256": "492916088fa5e8c170461cc9eba97a88ed1cc1e50c7b99e5f6a5e49167fea157",
"target_sha256": "9dd0dc3e6f048683157fe8b8deb533ed88970bf34e9824e439bf3a5d52d80e0a",
"target_sha256": "7b0ae5d648245e89f4aacf262250009de812d3dbb8ee55386f3704cabca48799",
"transformed": true,
"source_relative_path": "dy_product_scraping.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/dy_product_scraping.py"
@@ -206,7 +206,7 @@
{
"category": "production_source",
"source_sha256": "906278aa6fb3578d5dab231bdccc2f5a7c6ec8c85810a3e026fff3c7ba16dc90",
"target_sha256": "d9e4281c21a42841023c0a723aff4a00af1f9200c94e42add9f8965159d4cfb5",
"target_sha256": "c9ac5b78ce367eb0b87697255a88400d1e2e83416ee1fc105110d3abc3dd8245",
"transformed": true,
"source_relative_path": "import_product_daily.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/import_product_daily.py"
@@ -230,7 +230,7 @@
{
"category": "production_source",
"source_sha256": "e0f016bafb3f527f1cae46eaf3ef31510184128aaaa7f52024a7c3f978eec42b",
"target_sha256": "f2692442a1e62a4b20dbb442eb3e27a3075ca6095ba2578c90193f0db3a1810e",
"target_sha256": "b157f24430170d5a991bb2283af0a747b26dd11c0f0dec37289774f115132fee",
"transformed": true,
"source_relative_path": "jd_main_image_collector.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/jd_main_image_collector.py"
@@ -238,7 +238,7 @@
{
"category": "production_source",
"source_sha256": "d99ce71a008128549ac83fc3da52d88540b74538e3fe5c85d47c3b985e414353",
"target_sha256": "f821a0221b832bea0123bf4a8f85fd0785eba9206c8a632ad48065ea3c30c095",
"target_sha256": "cfe98b322e4ef7bd875325c2a9ffedeabb364e98b2983773dc98f9e0e75a6581",
"transformed": true,
"source_relative_path": "jd_product_data_collector.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/jd_product_data_collector.py"
@@ -294,7 +294,7 @@
{
"category": "production_source",
"source_sha256": "910e720fab4f47419ad56055f00a8c3fbe3a06c2577d81e35d3d38178a588a45",
"target_sha256": "c9f8cfc8dff6a7e5617b9b357ed92872f94371f93360318a3ff78f7ec38d75df",
"target_sha256": "e515cdc287b39a315dec44b4fb1c17f98b2af13bf82715731f92919d45f87854",
"transformed": true,
"source_relative_path": "market_rank_product_import.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/market_rank_product_import.py"
@@ -358,7 +358,7 @@
{
"category": "production_source",
"source_sha256": "c9a51f1d1267b398cf260a86e65a8ee49de873d6c95bf9c90db2abb422005cf7",
"target_sha256": "6cd70bb5406bc00257f9070760f980580e95c3c4e4326a67d33d30433dd74c32",
"target_sha256": "54c7b5802f66f9892b455aadd4a7396b7b693b2dda729aa7c22f8909821acd17",
"transformed": true,
"source_relative_path": "run_daily_persona.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/run_daily_persona.py"
@@ -390,7 +390,7 @@
{
"category": "production_source",
"source_sha256": "a1b097ead2a3f7e20ca63d7c15828a8d04030b0d6722934c3bf65d9e975fba76",
"target_sha256": "a231ccd2efacefca703a975e6c63e05a36c61a3db0975e46b1fab8d983fe84d2",
"target_sha256": "7d8587f77615040087d2d475a7b440ad16a39c5874e143418dd700ecbc5db0f8",
"transformed": true,
"source_relative_path": "scripts/insert_main_image_records.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/scripts/insert_main_image_records.py"
@@ -406,7 +406,7 @@
{
"category": "production_source",
"source_sha256": "4474dd7d68117247af87e20bbeb6b7c072368ef0f606de5d3e186f459d34a0f5",
"target_sha256": "cfbb46483113cda9e4cb301b092ac2ce6a35c5cc53b9a9cef01fdf93a00d3b94",
"target_sha256": "862ae9392227113ad4266c2170769595c36b2f98fdecc54e54a648ba834baebf",
"transformed": true,
"source_relative_path": "taobao_sycm_collect.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/taobao_sycm_collect.py"
@@ -438,7 +438,7 @@
{
"category": "production_source",
"source_sha256": "e4d99956d9649d108baeb3e3a2d1969c227622d1037062bd277af66375556d82",
"target_sha256": "981ef2a5a59f73ae244240520320c73be9c9462b50546df1ff91d0c97e7c03f7",
"target_sha256": "2ccb359ab943c185b17854f0f00cfb76b8a1235d7b031209a1c07e9386f8d93f",
"transformed": true,
"source_relative_path": "upload_video_to_guanghe.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/upload_video_to_guanghe.py"
@@ -454,7 +454,7 @@
{
"category": "production_source",
"source_sha256": "cceb3861da0e68c686c0c58f64767c3ca2693986bb5006ca7405d3a5a39c4347",
"target_sha256": "8a947ce42d34ff26bb46c5173af9e37ee9d60eac1b595d2f54b855a881ac2d6c",
"target_sha256": "4a44a00ec0e13bac3c4369c0336c6b6410e012d59e39900f3b4d0c093c8def90",
"transformed": true,
"source_relative_path": "vendors/dy-data-flow/dy_store_competitor_store_scraping.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/vendors/dy-data-flow/dy_store_competitor_store_scraping.py"
@@ -488,7 +488,7 @@
{
"category": "production_source",
"source_sha256": "6daa16c03989fb2231e54545726f5aa488e4ba80b9b8aa6f937cb9a2fa13ff35",
"target_sha256": "89ba008ef4388c6e1ed32e4a5c26ffa81d083164dd6d7231fc9e5886de8437cd",
"target_sha256": "5ea444181ca731d8bdd983fa8d0179422651fdd994ebe08a63ed2bfae155fca5",
"transformed": true,
"source_relative_path": "vendors/jd-data-flow/jd_data_collector.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/vendors/jd-data-flow/jd_data_collector.py"
@@ -496,7 +496,7 @@
{
"category": "production_source",
"source_sha256": "a53cc38b0ddeaa0065bc27a1d137ce16c2873ff9bec712483ca87bcedd6baa3a",
"target_sha256": "0549228539c2e1908c3181de13291457712eeaea8fea89903814844724a3b5ee",
"target_sha256": "a6ee61498a040be9d45c6b8f9ecdbc06213f5f1e5507783412188f35d2968640",
"transformed": true,
"source_relative_path": "vendors/jd-data-flow/jd_peer_product_data_collector.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/vendors/jd-data-flow/jd_peer_product_data_collector.py"
@@ -560,7 +560,7 @@
{
"category": "runtime_resource",
"source_sha256": "9739407b61842c598491d3189a32f5f15aa944aed79a164a5cd5b5a34f20c746",
"target_sha256": "8ff84f6c3c672af35e09f70d71aa73656ac101218d8a56126b8b0a75ab387c8b",
"target_sha256": "6bbd36048836ef7ed0807223fc028ac17349b20314b1ea353ccedb451957ba55",
"transformed": true,
"source_relative_path": "db/schema.sql",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/db/schema.sql"
@@ -672,7 +672,7 @@
{
"category": "regression_test",
"source_sha256": "820e17b910b62a213f0bebafcff72998c2c978307c97f976c074276150ce7c6a",
"target_sha256": "1159c9e3f0abdf17183e7d6fa2634c508fe8348ce2926e7d3c58e735b713c2e9",
"target_sha256": "cdedf8c9a4871ad763e0a9098fc18777f0c3206991e8d440a45ccdfbda089a07",
"transformed": true,
"source_relative_path": "tests/test_main_image_concurrency.py",
"target_relative_path": "tests/modules/product_commerce/test_main_image_concurrency.py"
@@ -696,7 +696,7 @@
{
"category": "regression_test",
"source_sha256": "f1e660d34e21d3cd7a26d3605c82d07982ba6c5002af7b56fb6a876cb706713d",
"target_sha256": "85ca8603e2af8257ce2a940b99c69948c2c1362f688e6f6dac4cbf499f1e4d73",
"target_sha256": "4a21ff4424c92a0bed6bd2f70f6a7bc7b426b3a25ccc356612f27a7426ce1eca",
"transformed": true,
"source_relative_path": "tests/test_market_rank_product_import.py",
"target_relative_path": "tests/modules/product_commerce/test_market_rank_product_import.py"
@@ -728,7 +728,7 @@
{
"category": "regression_test",
"source_sha256": "f9cfbb49fd313444790ac194fbf980dae21912081b9f26ad8e3850912126f09e",
"target_sha256": "0cc2b365144482ceb36667eedfb2c403e904437f3ea4c7e12f7099276c13a279",
"target_sha256": "043b2dd69172bf5ea1b324bf07f8bba4c43f4bca9817c30a2c805ffe8ce70525",
"transformed": true,
"source_relative_path": "tests/test_style_analysis_orchestration.py",
"target_relative_path": "tests/modules/product_commerce/test_style_analysis_orchestration.py"
@@ -32,7 +32,7 @@
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/collectors/dy_store_competitor_store_scraping.py",
"category": "source",
"source_sha256": "009879a7dd341ca6ecbc8eb66f5700114c73bcaba3a53f159014a76232d0aa6b",
"target_sha256": "d5e85ded1f852ba418229c75c985d0ece967dfbd5fd31697fe407b64d1c3db05",
"target_sha256": "9ab1328c2f1743944e8f096e548255884bc9724bf1e900c60c069d443c886902",
"transformed": true
},
{
@@ -48,7 +48,7 @@
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/collectors/jd_peer_store_data_collector.py",
"category": "source",
"source_sha256": "4ebdb52df7b384d2bfdd2fc840d55d14e18864002c20a36685b63cee1966cf4c",
"target_sha256": "39faf18e9cd5e441323a8e056e022f179018a44eac224d64383cd7ae9fa1d611",
"target_sha256": "f03b1a7226071f2225a2a71748ad2c27a102cde67ba02d37c63f15480dd0de32",
"transformed": true
},
{
@@ -88,7 +88,7 @@
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/db/__init__.py",
"category": "source",
"source_sha256": "ffc89b78ff0081a129e04eb6de4985ab0a2a4613c9e8a1d8b2750d8c17e8bc05",
"target_sha256": "a4bae02d0c07f3e3986896b31e62f41bf15edbf74d3776b1e050e381c15a8413",
"target_sha256": "08b5b8920338100339e7c716423b0e9260d4e007d0a1cc3f7510d51970a66a44",
"transformed": true
},
{
@@ -96,7 +96,7 @@
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/db/db.py",
"category": "source",
"source_sha256": "168b6f87c1827b267f763ddce7eb15d0c8d652c8c0c7af18dcd0c6c7da5cddd1",
"target_sha256": "01469f9eb80aee5e80bc28e548dbe2cd5ce861ab35cd480b33921a2adc799320",
"target_sha256": "4ab56df6c8ff75b1a67256ad8cdef34c58bb97e1a126707aac7ccf82980a56ec",
"transformed": true
},
{
@@ -104,7 +104,7 @@
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/db/schema.sql",
"category": "sql",
"source_sha256": "73230590bde7ae6359c02c82c2db096c0d3faa55b374e30348c47d7b40ea498b",
"target_sha256": "a69fa151f989f106d00f344bab43ec9e6a09cf404bdc581624cc0901cdc13d20",
"target_sha256": "9558543cf313aadb785715fca8f5cb99219488e7cb6c37cb00310bfe84f7ee74",
"transformed": true
},
{
+1 -1
View File
@@ -120,7 +120,7 @@
"target_relative_path": "src/gyxx_flow/modules/supply_chain/orchestrator/scripts/ProductReplenishment.py",
"category": "source",
"source_sha256": "02f6345142ebd84e25dc2f3b0651bc139ef31dd3994ff739efe66344c77ee49e",
"target_sha256": "e74211f2fe97ac47e704a040f442f51c97fecc15ea30dd1f0dbeacaf5940fd80",
"target_sha256": "ae5154b1f2af1e03307c07e4188225cd9601d1e1b5316fc0f6a78d23303ba8de",
"transformed": true
},
{
+82 -51
View File
@@ -3,6 +3,11 @@
"workflows": [
{
"id": "content.metrics.daily",
"required_accounts": [
"douyin-content",
"douyin-shop",
"xiaohongshu-content"
],
"module": "content_marketing",
"trigger": "scheduled",
"execution": {
@@ -302,6 +307,11 @@
},
{
"id": "content.metrics.backfill",
"required_accounts": [
"douyin-content",
"douyin-shop",
"xiaohongshu-content"
],
"module": "content_marketing",
"trigger": "scheduled",
"execution": {
@@ -430,56 +440,7 @@
"task_name": "YingxiaoYunying_DailyMarketingReport"
}
},
{
"id": "content.relogin.weekly",
"module": "content_marketing",
"trigger": "scheduled",
"execution": {
"steps": [
{
"id": "relogin",
"name": "刷新内容平台登录态",
"description": "并行启动五个平台的二维码重新登录流程,按配置轮次重试,持久化登录状态并通知结果。",
"data_flow": {
"sources": [
{
"system": "内容平台",
"label": "五个平台登录页与当前会话状态"
}
],
"processing": [
"并行发起二维码登录并按轮次重试",
"校验并持久化新的登录会话"
],
"destinations": [
{
"system": "浏览器状态",
"label": "各平台 Cookie 与 storage state"
},
{
"system": "飞书",
"label": "登录刷新结果通知"
}
]
},
"entry": "data/tools/friday_relogin_parallel.py",
"args": [
"--max-attempts",
"3",
"--round-timeout",
"300",
"--screenshot-delay",
"60"
]
}
]
},
"provenance": {
"source_project": "content",
"task_name": "YingxiaoYunying_FridayRelogin"
}
},
{
{
"id": "content.creator_report.monthly",
"module": "content_marketing",
"trigger": "scheduled",
@@ -781,6 +742,9 @@
},
{
"id": "shop.metrics.weekly",
"required_accounts": [
"douyin-shop"
],
"module": "shop_intelligence",
"trigger": "scheduled",
"execution": {
@@ -890,6 +854,9 @@
},
{
"id": "shop.competitor.weekly",
"required_accounts": [
"douyin-shop"
],
"module": "shop_intelligence",
"trigger": "scheduled",
"execution": {
@@ -1592,6 +1559,10 @@
},
{
"id": "product.daily",
"required_accounts": [
"douyin-shop",
"erp-jushuitan"
],
"module": "product_commerce",
"trigger": "scheduled",
"execution": {
@@ -1916,6 +1887,7 @@
],
"processing": [
"校验广告主 ID、店铺标识和商品数据明细",
"任务中心被重定向到账户选择页时自动点击目标广告主后重试",
"按业务日期下载原始 Excel",
"重置千川错误的 worksheet A1 dimension 并校验日期覆盖"
],
@@ -2231,6 +2203,54 @@
},
"note": "每天自动采集昨天数据;控制台可手动选择任意补采日期并幂等覆盖该日。全选聚水潭全部店铺,不拆京东、抖音、天猫;每次只采集一个业务日期。"
},
{
"id": "product.hk_anpei_public_inventory.daily",
"module": "product_commerce",
"trigger": "scheduled",
"execution": {
"steps": [
{
"id": "query_and_refresh_sheet",
"name": "刷新香港安沛 SKU 公有可用数",
"description": "校验计划书固定 SKU 与现有库存工作表中的商品编码对应关系,调用聚水潭 /open/inventory/query 查询库存及锁定量,并将公有可用数写入仓库现有库存列。",
"entry": "public_sku_inventory_sheet.py",
"timeout_seconds": 900,
"replay_policy": "idempotent",
"data_flow": {
"sources": [
{
"system": "光影行星香港安沛2026Q2计划书",
"label": "现有库存表的 51 个唯一 SKU"
},
{
"system": "聚水潭开放 API",
"label": "/open/inventory/query",
"detail": "请求 has_lock_qty=true 以取得库存锁定量"
}
],
"processing": [
"校验整批 SKU 库存响应后再写入飞书",
"按商品编码(光影)匹配行;公有可用数 = qty - order_lock - lock_qty + virtual_qty",
"没有库存记录的 SKU 清空仓库现有库存单元格",
"仅写入仓库现有库存列,不覆盖图片、备注、在途生产或其它字段"
],
"destinations": [
{
"system": "飞书电子表格",
"label": "光影行星香港安沛2026Q2計劃書",
"detail": "現有庫存工作表;按商品编码(光影)更新 D 列仓库现有库存"
}
]
}
}
]
},
"provenance": {
"source_project": "data-hub",
"task_name": "HongKongAnpeiPublicSkuInventoryDaily"
},
"note": "每天 Asia/Shanghai 09:00 执行。聚水潭应用凭据从 GYXX 调度进程环境读取;不得使用开发配置默认值。"
},
{
"id": "product.video_upload",
"module": "product_commerce",
@@ -2350,7 +2370,6 @@
"args": [
"--anchor-record-id",
"recvrXTRWWFXeJ",
"--manual-login",
"--execute"
],
"timeout_seconds": 21600
@@ -2607,6 +2626,9 @@
},
{
"id": "supply.purchase_confirmation.daily",
"required_accounts": [
"erp-jushuitan"
],
"module": "supply_chain",
"trigger": "scheduled",
"execution": {
@@ -2657,6 +2679,9 @@
},
{
"id": "supply.replenishment.weekly",
"required_accounts": [
"erp-jushuitan"
],
"module": "supply_chain",
"trigger": "scheduled",
"execution": {
@@ -2712,6 +2737,9 @@
},
{
"id": "supply.replenishment_alert.daily",
"required_accounts": [
"erp-jushuitan"
],
"module": "supply_chain",
"trigger": "scheduled",
"execution": {
@@ -2811,6 +2839,9 @@
},
{
"id": "product.market_rank",
"required_accounts": [
"douyin-shop"
],
"module": "product_commerce",
"trigger": "scheduled",
"execution": {
+47
View File
@@ -0,0 +1,47 @@
# GYXX Flow 开机自启:启用/关闭 Windows 自动登录
#
# 背景:控制台+调度器的自启链是
# 开机 → 自动登录进桌面 → Startup\gyxx-keepalive.vbs → keepalive → 控制台(8765)+调度器
# 唯一缺口是 AutoAdminLogon(默认 0,重启后停在登录界面,Startup 不触发)。
# 采集依赖浏览器 CDP,必须跑在交互式用户会话,因此不能用服务/Task Scheduler SYSTEM 替代。
#
# 启用(交互输入密码,不会留在命令历史):
# powershell -ExecutionPolicy Bypass -File D:\gyxx-flow\deploy\watchdog\autologon.ps1
# 撤销:
# powershell -ExecutionPolicy Bypass -File D:\gyxx-flow\deploy\watchdog\autologon.ps1 -Disable
#
# 注意:DefaultPassword 以明文存于 HKLM Winlogon(仅 Administrators 可读),
# 内网采集机可接受;介意安全性请改用 Sysinternals Autologon.exe(LSA secret 存储)。
param(
[switch]$Disable
)
$winlogon = 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon'
if ($Disable) {
Set-ItemProperty $winlogon -Name AutoAdminLogon -Value '0'
Remove-ItemProperty $winlogon -Name DefaultPassword -ErrorAction SilentlyContinue
Write-Host '已关闭自动登录(DefaultPassword 已清除)。' -ForegroundColor Yellow
exit 0
}
$pwd1 = Read-Host -AsSecureString "输入 $env:USERNAME 的 Windows 登录密码"
$pwd2 = Read-Host -AsSecureString '再输一次确认'
$plain1 = [Net.NetworkCredential]::new('', $pwd1).Password
$plain2 = [Net.NetworkCredential]::new('', $pwd2).Password
if ($plain1 -ne $plain2) { throw '两次输入不一致,未做任何修改。' }
if ([string]::IsNullOrEmpty($plain1)) {
Write-Warning '密码为空:策略 LimitBlankPasswordUse=1 会拒绝空密码自动登录,请先给账户设置密码。'
exit 1
}
Set-ItemProperty $winlogon -Name AutoAdminLogon -Value '1'
Set-ItemProperty $winlogon -Name DefaultUserName -Value $env:USERNAME
Set-ItemProperty $winlogon -Name DefaultDomainName -Value $env:COMPUTERNAME
Set-ItemProperty $winlogon -Name DefaultPassword -Value $plain1
# AutoLogonCount 每次自动登录会递减,删掉表示不限次数
Remove-ItemProperty $winlogon -Name AutoLogonCount -ErrorAction SilentlyContinue
Write-Host '自动登录已启用。' -ForegroundColor Green
Write-Host '验证方式:挑空闲时间重启一次,开机应自动进桌面,约 2 分钟内控制台(127.0.0.1:8765)与调度器就位;'
Write-Host '也可查看 D:\gyxx-flow\var\logs\keepalive\keepalive.log 出现 keepalive-start / scheduler-restarted / console-restarted。'
+336
View File
@@ -0,0 +1,336 @@
# -*- coding: utf-8 -*-
"""控制台与调度器的进程外保活(登录自启动,随用户会话常驻)。
职责(不承载任何业务调度规则,业务调度仍由 `gyxx schedule run` 唯一负责):
1. 控制台:探测 http://127.0.0.1:8765/api/health——进程不在则拉起;
进程在但探测连续失败(wedged)则结束该进程树后拉起。控制台进程内
另有看门狗(console.py `_console_watchdog`)自检失败会主动退出,此处
负责把退出后的空窗补上,两层互为补充。
2. 调度器:读 `var/state/scheduler/service.lock` 的 PID 判断存活,锁主
已死则按 install.ps1 同款参数拉起 `python -m gyxx_flow schedule run`;
调度器自身单实例锁保证不会双跑。
3. 节流:每个目标有冷却间隔;短时间反复重启触发熔断,只记录不再拉起,
避免故障风暴。
用法:
gyxx_keepalive.py # 常驻循环(Startup 的 vbs 以 pythonw 启动)
gyxx_keepalive.py --once # 单轮探测+处置后退出,供验收/手动巡检
日志:var/logs/keepalive/keepalive.log(追加)。
自身单实例:var/state/keepalive.lock(PID 存活检测,重复启动直接退出)。
"""
from __future__ import annotations
import argparse
import ctypes
import http.client
import json
import os
import subprocess
import sys
import time
from datetime import datetime, timezone
from pathlib import Path
PROJECT_ROOT = Path(__file__).resolve().parents[2]
ENV_FILES = [
Path(os.getenv("GYXX_RUNTIME_ENV_FILE", r"D:\product-collector-analyze-flow\.env")),
Path(
os.getenv(
"GYXX_RUNTIME_JUSHUITAN_ENV_FILE",
r"D:\product-collector-analyze-flow\private-secrets\jushuitan.env",
)
),
]
CONSOLE_HOST = "127.0.0.1"
CONSOLE_PORT = int(os.getenv("GYXX_CONSOLE_PORT", "8765"))
LOOP_INTERVAL_SECONDS = float(os.getenv("GYXX_KEEPALIVE_INTERVAL_SECONDS", "30"))
PROBE_TIMEOUT_SECONDS = 5.0
PROBE_FAILURE_LIMIT = 3
RESTART_COOLDOWN_SECONDS = float(os.getenv("GYXX_KEEPALIVE_COOLDOWN_SECONDS", "120"))
TRIP_MAX_RESTARTS = 5
TRIP_WINDOW_SECONDS = 30 * 60
TRIP_PAUSE_SECONDS = 60 * 60
DATA_ROOT = PROJECT_ROOT / "var"
LOG_PATH = DATA_ROOT / "logs" / "keepalive" / "keepalive.log"
LOCK_PATH = DATA_ROOT / "state" / "keepalive.lock"
CONSOLE_LOG_DIR = DATA_ROOT / "logs" / "console-restarts"
SCHEDULER_LOG_DIR = DATA_ROOT / "logs" / "scheduler-service"
SCHEDULER_SERVICE_LOCK = DATA_ROOT / "state" / "scheduler" / "service.lock"
def _log(event: str, detail: str = "") -> None:
stamp = datetime.now(timezone.utc).astimezone().strftime("%Y-%m-%d %H:%M:%S")
line = f"{stamp} {event} {detail}".rstrip()
LOG_PATH.parent.mkdir(parents=True, exist_ok=True)
with LOG_PATH.open("a", encoding="utf-8", newline="\n") as stream:
stream.write(line + "\n")
print(line, flush=True)
def _pid_alive(pid: int) -> bool:
if pid <= 0:
return False
kernel32 = ctypes.windll.kernel32
handle = kernel32.OpenProcess(0x1000, False, pid) # PROCESS_QUERY_LIMITED_INFORMATION
if not handle:
return False
try:
exit_code = ctypes.c_ulong()
if kernel32.GetExitCodeProcess(handle, ctypes.byref(exit_code)):
return exit_code.value == 259 # STILL_ACTIVE
return False
finally:
kernel32.CloseHandle(handle)
def _acquire_self_lock() -> bool:
LOCK_PATH.parent.mkdir(parents=True, exist_ok=True)
if LOCK_PATH.is_file():
try:
existing = json.loads(LOCK_PATH.read_text(encoding="utf-8"))
if _pid_alive(int(existing.get("pid", 0))):
return False
except Exception:
pass
LOCK_PATH.write_text(
json.dumps({"pid": os.getpid(), "started_at": datetime.now(timezone.utc).isoformat()}),
encoding="utf-8",
)
return True
def _console_env_files() -> list[Path]:
return [path for path in ENV_FILES if path.is_file()]
def _spawn(argv: list[str], log_dir: Path, log_prefix: str) -> int:
log_dir.mkdir(parents=True, exist_ok=True)
stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
log_path = log_dir / f"{log_prefix}-{stamp}.log"
kwargs: dict[str, object] = {}
if os.name == "nt":
kwargs["creationflags"] = (
subprocess.CREATE_NO_WINDOW | subprocess.CREATE_NEW_PROCESS_GROUP
)
else:
kwargs["start_new_session"] = True
with log_path.open("a", encoding="utf-8", newline="\n") as stream:
process = subprocess.Popen(
argv,
cwd=str(PROJECT_ROOT),
env=os.environ,
stdin=subprocess.DEVNULL,
stdout=stream,
stderr=subprocess.STDOUT,
shell=False,
**kwargs,
)
return process.pid
def _console_argv() -> list[str]:
argv = [
str(PROJECT_ROOT / ".venv" / "Scripts" / "python.exe"),
"-m",
"gyxx_flow",
"console",
"--host",
CONSOLE_HOST,
"--port",
str(CONSOLE_PORT),
]
for env_file in _console_env_files():
argv.extend(("--env-file", str(env_file)))
return argv
def _scheduler_argv() -> list[str]:
argv = [
str(PROJECT_ROOT / ".venv" / "Scripts" / "python.exe"),
"-m",
"gyxx_flow",
"schedule",
"run",
]
for env_file in _console_env_files():
argv.extend(("--env-file", str(env_file)))
return argv
def _probe_console() -> bool:
try:
connection = http.client.HTTPConnection(CONSOLE_HOST, CONSOLE_PORT, timeout=PROBE_TIMEOUT_SECONDS)
try:
connection.request("GET", "/api/health")
response = connection.getresponse()
response.read()
return response.status == 200
finally:
connection.close()
except Exception:
return False
def _port_owner_pid(port: int) -> int | None:
try:
output = subprocess.run(
["netstat", "-ano", "-p", "tcp"],
capture_output=True,
text=True,
timeout=10,
).stdout
except Exception:
return None
needle = f"{CONSOLE_HOST}:{port}"
for line in output.splitlines():
parts = line.split()
if (
len(parts) >= 5
and parts[0].upper() == "TCP"
and parts[1] == needle
and parts[3].upper() == "LISTENING"
):
try:
return int(parts[4])
except ValueError:
return None
return None
def _kill_pid_tree(pid: int) -> None:
if pid <= 0 or pid == os.getpid():
return
subprocess.run(
["taskkill", "/PID", str(pid), "/T", "/F"],
capture_output=True,
timeout=30,
)
def _scheduler_lock_pid() -> int | None:
try:
payload = json.loads(SCHEDULER_SERVICE_LOCK.read_text(encoding="utf-8"))
return int(payload.get("pid", 0)) or None
except Exception:
return None
class _TargetState:
"""记录单目标的连续失败计数、冷却与熔断状态。"""
def __init__(self, name: str) -> None:
self.name = name
self.failures = 0
self.last_restart_at = 0.0
self.recent_restarts: list[float] = []
self.tripped_at: float | None = None
def tripped(self, now: float) -> bool:
if self.tripped_at is None:
return False
if now - self.tripped_at >= TRIP_PAUSE_SECONDS:
self.tripped_at = None
self.recent_restarts.clear()
_log("keepalive-trip-reset", f"target={self.name} 熔断窗口结束恢复守护")
return False
return True
def may_restart(self, now: float) -> bool:
if self.tripped(now):
return False
if now - self.last_restart_at < RESTART_COOLDOWN_SECONDS:
return False
self.recent_restarts = [
stamp for stamp in self.recent_restarts if now - stamp < TRIP_WINDOW_SECONDS
]
if len(self.recent_restarts) >= TRIP_MAX_RESTARTS:
self.tripped_at = now
_log(
"keepalive-tripped",
f"target={self.name} {TRIP_MAX_RESTARTS} 次重启/{TRIP_WINDOW_SECONDS // 60}分钟 "
f"暂停拉起{TRIP_PAUSE_SECONDS // 60}分钟,请人工检查日志",
)
return False
return True
def record_restart(self, now: float) -> None:
self.last_restart_at = now
self.recent_restarts.append(now)
def run_cycle(console: _TargetState, scheduler: _TargetState) -> None:
now = time.time()
# ---- 控制台 ----
if _probe_console():
console.failures = 0
else:
console.failures += 1
owner = _port_owner_pid(CONSOLE_PORT)
_log(
"console-probe-failed",
f"{console.failures}/{PROBE_FAILURE_LIMIT} port_owner_pid={owner}",
)
if console.failures >= PROBE_FAILURE_LIMIT and console.may_restart(now):
if owner is not None and _pid_alive(owner):
_log("console-kill-wedged", f"pid={owner} 探测持续失败,结束进程树")
_kill_pid_tree(owner)
time.sleep(3)
pid = _spawn(_console_argv(), CONSOLE_LOG_DIR, "keepalive-console")
console.record_restart(time.time())
console.failures = 0
_log("console-restarted", f"pid={pid}")
# ---- 调度器 ----
scheduler_pid = _scheduler_lock_pid()
scheduler_alive = scheduler_pid is not None and _pid_alive(scheduler_pid)
if not scheduler_alive and scheduler.may_restart(now):
pid = _spawn(_scheduler_argv(), SCHEDULER_LOG_DIR, "keepalive-start")
scheduler.record_restart(time.time())
_log(
"scheduler-restarted",
f"pid={pid} prev_lock_pid={scheduler_pid}",
)
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description="GYXX Flow 控制台/调度器保活")
parser.add_argument("--once", action="store_true", help="单轮探测+处置后退出")
arguments = parser.parse_args(argv)
if not _acquire_self_lock():
_log("keepalive-duplicate-exit", "已有保活进程在运行")
return 0
console = _TargetState("console")
scheduler = _TargetState("scheduler")
missing_env = [str(path) for path in ENV_FILES if not path.is_file()]
_log(
"keepalive-start",
f"pid={os.getpid()} once={arguments.once}"
+ (f" missing_env={missing_env}" if missing_env else ""),
)
try:
while True:
run_cycle(console, scheduler)
if arguments.once:
_log("keepalive-once-done", "")
return 0
time.sleep(LOOP_INTERVAL_SECONDS)
except KeyboardInterrupt:
return 0
finally:
try:
if LOCK_PATH.is_file():
payload = json.loads(LOCK_PATH.read_text(encoding="utf-8"))
if int(payload.get("pid", 0)) == os.getpid():
LOCK_PATH.unlink()
except Exception:
pass
if __name__ == "__main__":
sys.exit(main())
+5
View File
@@ -0,0 +1,5 @@
' GYXX Flow 保活登录自启动:无窗口启动 deploy/watchdog/gyxx_keepalive.py,
' 由它负责拉起控制台(含进程内自检看门狗)与调度器。
' 安装:复制到 %APPDATA%\Microsoft\Windows\Start Menu\Programs\Startup\
CreateObject("WScript.Shell").Run _
"""D:\gyxx-flow\.venv\Scripts\pythonw.exe"" ""D:\gyxx-flow\deploy\watchdog\gyxx_keepalive.py""", 0, False
+12 -1
View File
@@ -4,6 +4,8 @@
param(
[string]$ProjectRoot = (Resolve-Path (Join-Path $PSScriptRoot "..\..")).Path,
[string]$DataRoot = "",
[string]$RuntimeEnvFile = "D:\product-collector-analyze-flow\.env",
[string]$JushuitanEnvFile = "D:\product-collector-analyze-flow\private-secrets\jushuitan.env",
[string]$ServiceName = "gyxx-flow-scheduler",
[string]$NssmPath = "nssm.exe"
)
@@ -12,6 +14,13 @@ $ErrorActionPreference = "Stop"
Set-StrictMode -Version Latest
$project = (Resolve-Path -LiteralPath $ProjectRoot).Path
$runtimeEnvPath = (Resolve-Path -LiteralPath $RuntimeEnvFile -ErrorAction Stop).Path
$runtimeEnvPaths = @($runtimeEnvPath)
if (Test-Path -LiteralPath $JushuitanEnvFile -PathType Leaf) {
$runtimeEnvPaths += (Resolve-Path -LiteralPath $JushuitanEnvFile -ErrorAction Stop).Path
} else {
Write-Warning "Jushuitan environment file not found; Jushuitan workflows will fail safely until it is provisioned."
}
if ([string]::IsNullOrWhiteSpace($DataRoot)) {
$DataRoot = Join-Path $project "var"
}
@@ -33,7 +42,9 @@ function Invoke-Nssm([string[]]$Arguments) {
Invoke-Nssm -Arguments @("install", $ServiceName, $python)
Invoke-Nssm -Arguments @("set", $ServiceName, "AppDirectory", $project)
Invoke-Nssm -Arguments @("set", $ServiceName, "AppParameters", "-m gyxx_flow schedule run")
$envFileArguments = $runtimeEnvPaths | ForEach-Object { '--env-file "' + $_ + '"' }
$schedulerArguments = '-m gyxx_flow schedule run ' + ($envFileArguments -join ' ')
Invoke-Nssm -Arguments @("set", $ServiceName, "AppParameters", $schedulerArguments)
Invoke-Nssm -Arguments @(
"set", $ServiceName, "AppEnvironmentExtra",
"GYXX_PROJECT_ROOT=$project", "GYXX_DATA_ROOT=$data", "PYTHONUTF8=1"
+30
View File
@@ -46,6 +46,27 @@ Scrapling 安装和管理。具体边界与验证命令见
`EnvironmentFile=` 或当前进程环境优先于文件中的同名值,避免本地文件意外覆盖密钥系统
注入值。
## 香港安沛 SKU 公有库存同步
`product.hk_anpei_public_inventory.daily` 每天按 `Asia/Shanghai` 09:00 查询计划书中的固定 SKU 清单,
调用聚水潭 `/open/inventory/query`,并按商品编码(光影)更新飞书电子表格「現有庫存」工作表的
「仓库现有库存」列。它要求调度进程环境提供
`JUSHUITAN_APP_KEY` 和 `JUSHUITAN_APP_SECRET`;`JUSHUITAN_BASE_URL` 可选,默认使用聚水潭官方 OpenAPI
地址。没有密钥时工作流会在查询前失败,不会把库存留成 0 或改写多维表格。
Windows NSSM 调度服务从受保护的运行时环境文件读取凭据。安装器默认读取通用运行环境文件
`D:\product-collector-analyze-flow\.env`,以及独立的聚水潭密钥文件
`D:\product-collector-analyze-flow\private-secrets\jushuitan.env`。两个文件都按服务账户实际身份限制 ACL;
可分别通过 `-RuntimeEnvFile` 和 `-JushuitanEnvFile` 指定其他路径:
```powershell
.\deploy\windows-service\install.ps1 -ProjectRoot D:\gyxx-flow -DataRoot D:\gyxx-flow-data -RuntimeEnvFile D:\product-collector-analyze-flow\.env -JushuitanEnvFile D:\product-collector-analyze-flow\private-secrets\jushuitan.env
```
仅将这两个变量的正式聚水潭应用值写入服务账户可读的环境文件,并限制文件 ACL;不要使用开发配置中的默认值,
也不要把真实值写进 Git、PowerShell 命令参数、服务日志或文档。已安装的服务需要让调度进程重新读取环境文件后,
新计划才会使用这些值。Linux 部署则将相同变量放入权限为 `0640` 的 `EnvironmentFile`。
## 三平台电商费用日报部署边界
`product.ecommerce_costs.daily` 是包含天猫万相台、京东京准通和抖音千川分支的电商费用工作流;三个
@@ -197,3 +218,12 @@ sudo systemctl start gyxx-flow.service
## Windows 兼容入口
`deploy/windows-service/` 暂时保留为 legacy NSSM 兼容入口。NSSM 只守护同一个 `gyxx schedule run` 进程,不注册 Windows Task Scheduler,也不保存业务时间规则。新服务器部署以 Linux systemd 为准。
## Windows 工作站保活(控制台 + 调度器自检自启动)
交互式 Windows 工作站(无法用 systemd/NSSM 常驻服务时)使用 `deploy/watchdog/gyxx_keepalive.py` 做进程外保活:
- **控制台自检**:控制台进程内置看门狗线程(`console.py` `_console_watchdog`),周期探测自身 `GET /api/health`;连续失败(默认 3 次,`GYXX_CONSOLE_WATCHDOG_FAILURES` / `GYXX_CONSOLE_WATCHDOG_INTERVAL_SECONDS` 可调,`GYXX_CONSOLE_WATCHDOG=0` 关闭)即以退出码 70 自杀,交给保活拉起。`/api/health` 不鉴权、只报进程 PID、uptime 与调度锁存活,供探针使用;API 500 现在会把完整堆栈落到 stderr(随启动重定向进 `var/logs/console-restarts/`)。
- **保活脚本**:登录自启动(Startup 目录 `gyxx-keepalive.vbs` → `pythonw gyxx_keepalive.py`)后常驻,每 30 秒一轮:控制台探测连续失败或端口空置则结束僵死进程树并按 `--env-file` 同款参数拉起;读 `var/state/scheduler/service.lock` 判定调度器存活,锁主死亡即拉起 `gyxx schedule run`(调度器单实例锁保证不双跑)。单目标冷却 120 秒;30 分钟内 5 次重启触发熔断,暂停拉起 1 小时并记日志待人工检查。日志在 `var/logs/keepalive/keepalive.log`。
- **验收**:`python deploy/watchdog/gyxx_keepalive.py --once` 单轮探测+处置;健康态应只输出 `keepalive-start`/`keepalive-once-done`。
- 该脚本只做进程可用性守护,不承载任何业务时间规则(唯一调度路径仍是 `gyxx schedule run`);未注册 Windows Task Scheduler。注意:Startup 方式要求用户已登录——机器重启后需人工登录一次,之后控制台与调度器会在约 2 分钟内自动就位。
+37 -2
View File
@@ -161,6 +161,19 @@ uv run gyxx schedule run --env-file /etc/gyxx-flow/gyxx-flow.env
需要停用单个工作流时,优先修改该工作流的调度启用状态,先 dry-run,再重启服务。其他模块继续运行。
### 开机自启(Windows 生产机)
重启/关机开机后的自动恢复链路:**Windows 自动登录 → `Startup\gyxx-keepalive.vbs` → `deploy/watchdog/gyxx_keepalive.py` → 控制台(8765) + `gyxx schedule run` 调度器**。keepalive 每 30 秒巡检并负责拉起死掉的控制台/调度器(单实例锁防双跑,冷却 120s、熔断保护),日志在 `var/logs/keepalive/keepalive.log`,验收用 `python deploy/watchdog/gyxx_keepalive.py --once`。
采集依赖浏览器 CDP,必须运行在交互式用户会话中,因此不能改成 Windows 服务或 Task Scheduler SYSTEM 任务(也不允许注册 Windows 任务计划)。免登录自启的唯一环节是自动登录,用脚本启用(密码交互输入、写入 `HKLM\...\Winlogon`,仅 Administrators 可读):
```powershell
powershell -ExecutionPolicy Bypass -File D:\gyxx-flow\deploy\watchdog\autologon.ps1 # 启用
powershell -ExecutionPolicy Bypass -File D:\gyxx-flow\deploy\watchdog\autologon.ps1 -Disable # 撤销
```
启用后重启验证:开机自动进桌面,约 2 分钟内控制台与调度器就位;停机窗内到期任务由调度器 catch-up 自动补跑。
## 数据和浏览器状态
生产 `GYXX_DATA_ROOT` 必须位于源码目录之外,Linux 推荐 `/var/lib/gyxx-flow`:
@@ -190,16 +203,38 @@ uv run gyxx schedule run --env-file /etc/gyxx-flow/gyxx-flow.env
- `tmall-ozko`:商品经营日报 ozko 账号,900 秒;
- `xiaohongshu-content`:小红书,900 秒;
- `douyin-content`:抖音内容侧,900 秒;
- `xingtu`:星图,900 秒;
- `chanmama`:蝉妈妈,900 秒;
- `chanmama`:蝉妈妈,900 秒(脚本自带账密登录);
- `bilibili-content`:B 站,900 秒。
星图/百应采集(`xingtu_scraper_v2.py`)的登录态挂在 `douyin-shop` 账号下;`xingtu` 账号当前无成员脚本,仅保留 vault。
商品经营日报 `product_commerce:taobao_sycm_collect.py` 会在同一次运行中按品牌路由账号:光影行星使用 `tmall-shop`,ozko 使用 `tmall-ozko`,两边各自使用独立 Profile 和账号 vault。`tmall-ozko` 没有静态成员脚本是有意设计,因为它由这个按品牌拆分的包装工作流动态选择;登录态成功后仍会回写 `state/accounts/tmall-ozko/`,由同一套 keepalive 负责续期。
账号需要在 `config/runtime-bindings.json` 的 `accounts.<account_id>.keepalive` 中显式开启续期,并配置一个不会产生业务副作用的已登录页面 URL。所有账号由唯一的 `gyxx schedule run` 常驻进程在后台维护;每个账号使用独立 CDP 端口、独立 Profile 和账号级 Cookie vault,启动后按 `initial_delay_seconds` 错峰访问,避免同时打开多个登录页面。访问成功且没有跳转到登录页时,Scrapling 才会把最新 Cookie/storage state 原子写回 `state/accounts/<account_id>/`,并同步给成员脚本;不需要另建 Windows 任务计划、cron 或 timer。
Cookie 续期只能延长站点支持滑动续期的会话,不能突破服务端硬过期、主动退出、风控或验证码。keepalive 检测到登录页/无效 Cookie 时会停止发布,不会用失效状态覆盖最后一份有效 vault;此时按 `gyxx accounts list` 检查 `keepalive.status`,必要时重新执行 `gyxx accounts login <account_id>`。
### 工作流执行前的 Cookie 健康网关
每个正式执行(非 dry-run)的工作流在采集步骤前会自动注入"每账号一个"的 `cookie-gate.<account_id>` 关键步骤,由 `src/gyxx_flow/cookie_gate.py` 实现按账号分三级的状态机:
- **blocking**(纯扫码/验证码账号:`douyin-shop`、`xiaohongshu-content`、`douyin-content`、`bilibili-content` 等):先看 keepalive 近期成功状态(默认 1 小时内视为新鲜,可用 `GYXX_COOKIE_GATE_MAX_STATUS_AGE_SECONDS` 调整),新鲜且 vault 静态有效则直接放行;否则用 keepalive 同款无头浏览器真实探活。探活失败即判定过期:gate 步骤失败,依赖它的采集步骤记 `dependency-failed` 跳过、工作流失败退出,并通过飞书通知登录负责人(`config/notification-routing.json` 中手机号为 `+8613663785145` 的人员,默认 30 分钟节流,可用 `GYXX_LOGIN_ALERT_OPEN_IDS` 覆盖收件人)执行 `uv run gyxx accounts login <account_id>` 完成扫码/验证码重登。
- **recover**(配置了 `login_command_id` 的账密自动重登账号:`erp-jushuitan`):过期时先自动跑账密登录恢复,恢复成功放行;失败才阻断并飞书通知。
- **defer**(成员脚本自带账号密码登录的账号:京东系、天猫系、`chanmama`):不注入 gate,保持脚本内账密自动登录流程不变。
按账号划分的持久状态在 `state/accounts/<id>/cookie_state.json`(`healthy` ↔ `awaiting_login`)。编排器型工作流(入口脚本本身无账号绑定、由子采集脚本使用账号)在 `config/workflows.json` 的 `required_accounts` 字段声明依赖账号。`--rerun-step`/`--resume-from` 切片和 shadow 运行不发送通知;dry-run 完全不注入 gate。设置 `GYXX_COOKIE_GATE=0` 可临时关闭整网关。
采集脚本不再内嵌扫码/验证码登录:运行中检测到登录态失效会以 `CookieExpiredError` 快速失败并按退出码 75 记为 cookie 跳过;`self_xhs_scraper.py --login-only` 是唯一保留的人工登录引导(该脚本不在任何账号成员内,自管绑定状态)。
### 登录恢复后的自动重跑
gate 阻断工作流时会把待重跑清单(工作流 + 业务日期,shadow/验收运行不记录,同工作流同日期去重,上限 20 条)写进 `state/accounts/<id>/cookie_state.json` 的 `pending_reruns`。常驻调度器(`gyxx schedule run`)的维护循环里挂着 `CookieGateRecoveryManager`:
- 每个心跳检查有待重跑记录的账号:keepalive 近期成功 + vault 静态有效 → 立即视为恢复;否则每 5 分钟做一次真实探活(mode C 账号会自动账密重登,重登成功同样触发恢复)。
- 恢复确认后,用调度器同款启动器按**原业务日期**自动重跑全部待重跑工作流(日志在 `data/logs/scheduler/`),清空清单并飞书通知"登录态已恢复 + 已自动重跑清单"。
- 启动失败的任务保留在清单里下一轮重试;同一账号两次重跑启动之间有 5 分钟冷却;控制台手动脚本类临时运行 id(`script.*`)不会自动重跑。
- 与正常调度并发安全:工作流级锁 + EffectLedger 幂等保证同一天不会重复产生副作用。
项目不会自动搬动或删除已有 `var/`。迁移数据根时应停止调度器,完整备份和复制原目录,修改环境变量后依次运行 doctor、dry-run 和一次指定日期的对账执行。
## 清理和保留
-89
View File
@@ -1,89 +0,0 @@
# 智能工作台设计(dsh-refact 分支)
> 状态:v1 已实现。本文档记录「以 deepseek-harness 为 agent 底座、gyxx-flow 工作流插件化」
> 的架构决策与边界。使用说明见 [../workbench/README.md](../workbench/README.md)。
## 目标与约束
- **底座**:deepseek-harness(`dsh`)Web UI + DeepSeek 模型 = 智能体运行时;
gyxx-flow 保持唯一的工作流执行/调度事实来源(LangGraph 引擎、`RunJournal`、
`LockManager`、`EffectLedger` 契约不变)。
- **插件化**:不重写工作流;通过 dsh「everything-is-a-plugin」扩展点把现有
控制台能力投影为智能体工具与 UI 面板。
- **UI**:在原 dsh Web UI 侧边栏追加工作流入口(`sidebar.footer.action` +
`shell.overlay` 两个**追加型**插槽,不替换任何内置区域),选中工作流即可
提问 / 诊断 / 修复。
- **生产约束**(AGENTS.md):`gyxx schedule run` 仍是唯一生产调度路径;
正式执行保持 `execute + confirmed` 双确认;控制台仍以 `--env-file` 注入云端凭据
(启动脚本默认 `D:\product-collector-analyze-flow\.env`)。
## 为什么不把 gyxx-flow 改写进 dsh monorepo
dsh 是 pnpm monorepo(host/client 双聚合、Typert 远程契约、自有构建链)。
把 Python 工作流引擎迁进去既不可能也无必要。dsh 的外部插件机制
(`--patch` cordis.yml + 绝对路径插件 + `dsh.client` 包声明)就是为这种
「外部系统接入」设计的。我们选择**进程外集成**:
- gyxx-flow 控制台 HTTP API 是唯一集成面(已含脱敏、并发守卫、写操作令牌);
- 工作台插件只是控制台的客户端 + 智能体能力注册器;
- 控制台挂了,dsh 照常可用;dsh 挂了,调度器照常跑。
## 组件
### 1. 控制台诊断端点(Python,`src/gyxx_flow/console.py`)
| 端点 | 说明 |
| --- | --- |
| `GET /api/workflows/{id}` | 单工作流详情(overview 投影 + `schedule_revision`) |
| `GET /api/workflows/{id}/runs/{run_id}` | 单次运行 + 步骤明细 |
| `GET /api/workflows/{id}/runs/{run_id}/diagnosis` | 诊断包:运行 + journal trace + 脱敏日志尾部 |
诊断包日志来源:`run.json` 的 `trace.paths.log` 目录 + 运行时间窗内的控制台日志
(`logs/console/{workflow_id}-*.log`),每个文件限读尾部 256KB,经 `_sanitize_error`
脱敏并截断,单响应最多 5 个文件。`run_id` 与 `workflow_id` 强制配对(404 不泄露
跨工作流记录)。
### 2. 宿主插件(`workbench/plugin/gyxx-workbench.mjs`,零运行时依赖)
- **7 个工具**(原始 JSON-Schema `ToolDefinition`,不 import 任何 dsh 包,
保证以绝对路径加载时无解析风险);
- **系统提示词段**(`gyxx-workbench`,order 700):中文运维契约与安全规则;
- **失败监控器**:`ctx.effect` 轮询,`workflow_id+run_id` 去重,自定义事件
`gyxx-workbench/alert` + 可选自动诊断会话;
- **桥接服务**(127.0.0.1:8790):为浏览器面板代理控制台 API(控制台禁 CORS,
面板无法直连),并承载 `/bridge/ask` 会话创建与降级页面。
写操作校验 `x-gyxx-workbench: 1` + JSON content-type;CORS 仅回环来源。
### 3. 侧边栏面板(`workbench/plugin/client/`)
- React 源码经 esbuild 打包为 CJS,平台模块(react 等)按 dsh 模块表协议外置,
包装为 `window.__ModuleLoader__.load({id, factory})`——与官方 tsdown 产物同协议;
- `package.json` 的 `dsh.client` 声明让 `client-modules` 扫描器(支持路径型
Loader 条目,向上找最近 package.json)发现并提供该浏览器包;
- 插槽:`sidebar.footer.action`(「工作流」开关 + 聚合状态点)、`shell.overlay`
(左抽屉面板 + 失败 toast)。均为 list 型追加插槽,不替换内置区域;
- 面板动作:提问/诊断/修复(创建带工作流上下文的智能体会话)、试运行/正式运行
(浏览器端 confirm)、停止、运行历史查看、单运行诊断视图。
### 4. 会话创建(`/bridge/ask`)
`ctx.agents.create({ sessionId, meta: { cwd, origin } })` → `agent.followup()` 注入
带工作流上下文的首条用户消息。新会话自动出现在 dsh 会话列表(host 侧创建即入册)。
文案模板区分 ask/diagnose/repair 三种动作。
## 已知限制(v1)
- 会话创建后不能自动聚焦(dsh 客户端无公开的「选中会话」运行时 API);面板以
toast 提示用户在会话列表中点开。
- `sidebar.workspaces` 为 single 型插槽(替换即失去会话树),故面板走 overlay
抽屉而非内嵌会话树;若未来 dsh 提供追加型侧栏区块插槽,可平移。
- 客户端包协议依赖 dsh 未冻结的 developer-preview 契约;升级 dsh 需回归验证
(README「依赖的 dsh 扩展点」一节列出了核对清单)。
- `autoDiagnose` 默认关闭:自动开会话会产生模型调用成本。
## 测试
- Python:`tests/test_console.py` 新增 5 用例(详情/运行详情/诊断包/越权配对/HTTP 路由)。
- 插件:`workbench/plugin/tests/`(node:test)——工具注册、离线错误结构化、
confirmed 守卫、桥接代理。
- 端到端手工验证:`start-workbench.ps1` → 面板选中工作流 → 提问/诊断/修复/试运行。
+20
View File
@@ -17,6 +17,15 @@ uv run gyxx console
uv run gyxx console --env-file D:\secure\gyxx-flow.env
```
香港客服库存工作流还需要聚水潭应用密钥。Windows 生产控制台必须额外加载受保护的聚水潭环境文件,
这样控制台发起的手动执行也能取得凭据;调度器和控制台应加载相同的环境文件集合:
```powershell
uv run gyxx console `
--env-file D:\product-collector-analyze-flow\.env `
--env-file D:\product-collector-analyze-flow\private-secrets\jushuitan.env
```
商品经营正式执行会在启动子进程前校验云端 PostgreSQL 和该工作流实际需要的运行时配置。
需要 Hermes 分析或业务通知的工作流仍会校验 `hermes-analyzer` lark-cli user 身份和负责人
通知路由;天猫/京东视频上传则校验 `STYLE_ANALYSIS_LLM_BASE_URL`(或
@@ -144,3 +153,14 @@ uv run gyxx console --host 0.0.0.0 --port 8765
控制台不提供工作流入口、参数、运行时绑定或凭据的在线编辑能力;这些仍由受版本控制的
项目配置和代码维护。
## 自检与保活
- `GET /api/health` 返回进程 PID、uptime 与调度器锁存活状态,不鉴权、不触外部依赖,专供探针使用。
- 控制台进程内置看门狗线程:周期自探测 `/api/health`,连续失败(默认 3 次)即以退出码 70 退出,
由 `deploy/watchdog/gyxx_keepalive.py`(登录自启动常驻)负责拉起;保活同时守护调度器进程,
详见 `docs/deployment.md` 的「Windows 工作站保活」章节。环境开关:
`GYXX_CONSOLE_WATCHDOG=0` 关闭、`GYXX_CONSOLE_WATCHDOG_INTERVAL_SECONDS`(默认 30)、
`GYXX_CONSOLE_WATCHDOG_FAILURES`(默认 3)。
- API 返回「控制台处理请求失败」时,完整堆栈在控制台 stderr 日志
(`var/logs/console-restarts/console-*.stderr.log`)中可查。
+2 -3
View File
@@ -69,7 +69,7 @@
| `08:30` | 2 | 1 | 1 | 达人月报生成 [`content.creator_report.monthly`](暂停);京东与天猫主图周采集 [`product.main_image.weekly`] |
| `08:40` | 1 | 1 | 0 | 商品经营日报采集 [`product.daily`] |
| `09:00` | 6 | 6 | 0 | 笔记清单与合作同步 [`content.notes_master.daily`];京东自营业绩日采集 [`shop.jd_self_operated.daily`];抖音待改价申诉;聚水潭全店铺款式日报 [`product.erp_all_shop_daily`];天猫百亿补贴批量报名;补货建议预计到货同步 [`supply.replenishment_arrival_sync.daily`] |
| `10:00` | 8 | 5 | 3 | 营销日报生成与发送 [`content.marketing_report.daily`](暂停);内容平台登录态刷新 [`content.relogin.weekly`](暂停);内容周度汇总 [`content.summary.weekly`](暂停);抖音待改价申诉;三平台电商费用采集 [`product.ecommerce_costs.daily`];商品月度销量表 [`product.sales_sheet.daily`];天猫百亿补贴批量报名;市场排行周采集 [`product.market_rank`] |
| `10:00` | 8 | 5 | 3 | 营销日报生成与发送 [`content.marketing_report.daily`](暂停);内容周度汇总 [`content.summary.weekly`](暂停);抖音待改价申诉;三平台电商费用采集 [`product.ecommerce_costs.daily`];商品月度销量表 [`product.sales_sheet.daily`];天猫百亿补贴批量报名;市场排行周采集 [`product.market_rank`] |
| `11:00` | 3 | 3 | 0 | 抖音待改价申诉;款式周期分析 [`product.style_analysis.interval`];天猫百亿补贴批量报名 |
| `12:00` | 5 | 5 | 0 | 多平台评论周采集 [`content.comments.weekly`];店铺经营指标周采集 [`shop.metrics.weekly`];抖音待改价申诉;商品人群画像日采集 [`product.persona.daily`];天猫百亿补贴批量报名 |
| `12:30` | 1 | 1 | 0 | 竞店指标周采集 [`shop.competitor.weekly`] |
@@ -98,7 +98,7 @@
- 指标采集:`content.metrics.daily`(内容指标日报采集)、`content.metrics.backfill`(内容指标补采)。
- 报告与汇总:`content.marketing_report.daily`(营销日报生成与发送)、`content.creator_report.monthly`(达人月报生成)、`content.summary.monthly`(内容月度汇总)、`content.summary.weekly`(内容周度汇总)。
- 内容同步与互动:`content.notes_master.daily`(笔记清单与合作同步)、`content.comments.weekly`(多平台评论周采集)、`content.relogin.weekly`(内容平台登录态刷新)。
- 内容同步与互动:`content.notes_master.daily`(笔记清单与合作同步)、`content.comments.weekly`(多平台评论周采集)。
### 4.2 商品经营(`product_commerce`,12 条)
@@ -124,7 +124,6 @@
| 1 | `content.metrics.daily` | 内容指标日报采集 | 内容营销 | 每天 `01:00`、`05:00` | 0 | 启用 | 5 |
| 2 | `content.metrics.backfill` | 内容指标补采 | 内容营销 | 每周一 `14:00` | 0 | 暂停 | 2 |
| 3 | `content.marketing_report.daily` | 营销日报生成与发送 | 内容营销 | 每天 `10:00` | 0 | 暂停 | 1 |
| 4 | `content.relogin.weekly` | 内容平台登录态刷新 | 内容营销 | 每周五 `10:00` | 0 | 暂停 | 1 |
| 5 | `content.creator_report.monthly` | 达人月报生成 | 内容营销 | 每月 1 日 `08:30` | 0 | 启用 | 1 |
| 6 | `content.summary.monthly` | 内容月度汇总 | 内容营销 | 每月 1 日 `08:00` | 0 | 启用 | 1 |
| 7 | `content.notes_master.daily` | 笔记清单与合作同步 | 内容营销 | 每天 `09:00` | 0 | 启用 | 1 |
+65
View File
@@ -0,0 +1,65 @@
# 工作流总览(未隐藏)
> 生成日期:2026-10-08。数据来源:`config/workflows.json`、`config/schedules.json`、控制台显示名(`var/state/console/workflow-display-names.json`);数据库表提取自各入口脚本及共享 DB 辅助模块的内联 SQL。
> 范围:控制台中未被隐藏的工作流共 **30 个**(全部 39 个中剔除浏览器端隐藏的 10 个,其中 `content.relogin.weekly` 已不在当前配置中)。
> 定时时间均为 **Asia/Shanghai**,由常驻调度器 `gyxx schedule run` 按 `config/schedules.json` 触发;标注「未启用」的仅支持控制台手动运行。
> 启动脚本为仓库根目录相对路径,由工作流引擎相对 `src/gyxx_flow/modules/<模块>/` 解析执行;`×N` 表示同一脚本被 N 个步骤以不同参数复用。
>
> **关联数据库表说明**:均指云端 PostgreSQL(`GYXX_POSTGRES_DSN`),(写)= INSERT/UPDATE/DELETE,(读)= SELECT,(建)= CREATE TABLE IF NOT EXISTS。三点公共约定:
> - 内容营销用到款式映射的脚本会经 `feishu_mapping` → `adapters/workflow_config.py` 间接读动态配置表 `workflow_dynamic_styles`、`workflow_dynamic_style_platforms`(并合并读 `cmt_styles`,首跑建 `workflow_dynamic_*` 表),下表不逐一列出。
> - 店铺情报各入库脚本启动时执行 `shop_intelligence/db/schema.sql`、商品电商多数入库脚本执行 `product_commerce/db/schema.sql`,幂等建该模块全量表(21 / 26 张);下表只列该工作流**实际读写**的表。
> - 读写 `information_schema` / `pg_catalog` 系统目录的通用校验不列出。
## 内容营销(content_marketing,4 个)
| 工作流名称 | ID | 功能 | 定时执行 | 启动脚本 | 关联数据库表 |
|---|---|---|---|---|---|
| 合作达人内容指标日报采集 | `content.metrics.daily` | 按日采集合作达人在 B站、小红书蒲公英、星图(未命中用抖店达人广场补采)的曝光互动指标并同步 PostgreSQL;05:00 档补采 01:00 未完成任务 | 每日 01:00、05:00 | `src/gyxx_flow/modules/content_marketing/run_all.py`<br>`src/gyxx_flow/modules/content_marketing/data/tools/sync_metrics_to_cmt_notes.py` | `cmt_note_metric_snapshots`(写)<br>`cmt_notes_master`(读写)<br>`cmt_creators`(读写)<br>`cmt_styles`(读) |
| 自营内容指标日报采集 | `content.metrics.self_operated.daily` | 刷新自营款式映射,采集自营 B站、蝉妈妈、小红书 ARK 指标;飞书曝光回填并将三平台全量内容指标按日快照入库 | 每日 06:00 | `…/content_marketing/data/tools/refresh_self_mapping.py`<br>`…/content_marketing/self_bilibili_scraper.py`<br>`…/content_marketing/chanmama_scraper.py`<br>`…/content_marketing/self_xhs_scraper.py`<br>`…/content_marketing/data/tools/sync_metrics_to_cmt_notes.py`<br>`…/content_marketing/data/tools/sync_self_video_metrics.py` | `cmt_self_video_metrics`(写)<br>`cmt_note_metric_snapshots`(写)<br>`cmt_notes_master`(读写)<br>`cmt_creators`(读写)<br>`cmt_styles`(读) |
| 笔记清单与合作同步 | `content.notes_master.daily` | 一次飞书拉取同时落笔记身份与合作字段;笔记数量按去重链接口径维护主表 | 每日 09:00 | `src/gyxx_flow/modules/content_marketing/data/tools/sync_notes_master.py` | `cmt_notes_master`(建+写)<br>`cmt_note_metric_snapshots`(建)<br>`cmt_styles`(读写)<br>`cmt_creators`(读写) |
| 多平台评论周采集 | `content.comments.weekly` | B站、小红书、抖音三平台评论并行采集,任一平台失败则工作流失败 | 每周日 12:00 | `…/content_marketing/data/tools/batch_rescrape_bilibili.py`<br>`…/content_marketing/data/tools/batch_rescrape_xiaohongshu.py`<br>`…/content_marketing/data/tools/batch_rescrape_douyin.py` | `cmt_comments`(写)<br>`cmt_notes_master`(读写) |
## 店铺情报(shop_intelligence,12 个)
| 工作流名称 | ID | 功能 | 定时执行 | 启动脚本 | 关联数据库表 |
|---|---|---|---|---|---|
| 店铺经营指标周采集 | `shop.metrics.weekly` | 三平台作为独立节点并行采集店铺经营指标并写 PostgreSQL(不写飞书 Base) | 每周一 12:00 | `src/gyxx_flow/modules/shop_intelligence/runners/run_shop.py` ×3 | `jd_shop_weekly`、`dy_shop_weekly`、`tm_shop_weekly`(各平台步骤分别写) |
| 竞店指标周采集 | `shop.competitor.weekly` | 京东、抖音并行采集竞店指标并入库,单平台失败不阻断另一平台 | 每周一 12:30 | `src/gyxx_flow/modules/shop_intelligence/runners/run_peer_store.py` ×2 | `jd_peer_store_ranking`、`dy_peer_store_ranking`(各平台步骤分别写) |
| 京东自营业绩日采集 | `shop.jd_self_operated.daily` | 采集京东自营品牌与商品业绩(业务日期 = 运行日 −2) | 每日 09:00 | `…/shop_intelligence/collectors/jd_self_operated_brand_daily.py`<br>`…/shop_intelligence/collectors/jd_self_operated_product_daily.py` | `jd_self_operated_brand_daily`(写)<br>`jd_self_operated_product_daily`(写) |
| 京东客服消息采集 | `shop.jd_customer_service.daily` | 按业务日在商智消息视图采集顾客会话,单日须完成全部顾客,失败保留进度可重试 | 每月 21 日 03:00 | `src/gyxx_flow/modules/shop_intelligence/collectors/jd_customer_service_messages.py` | `jd_customer_service_conversations`、`jd_customer_service_messages`(写) |
| 抖音客服消息采集 | `shop.douyin_customer_service.daily` | 按业务日采集抖音客服会话,失败保留进度可安全重试 | 每月 21 日 03:20 | `src/gyxx_flow/modules/shop_intelligence/collectors/dy_customer_service_messages.py` | `dy_customer_service_conversations`、`dy_customer_service_messages`、`dy_customer_service_conversation_orders`(写)<br>`dy_customer_service_products`(读写) |
| 京东差评采集 | `shop.jd_negative_reviews.daily` | 按店铺选择日期范围逐日采集京东负面评价,按评价 ID 幂等 | 每月 22 日 03:00 | `src/gyxx_flow/modules/shop_intelligence/collectors/jd_negative_reviews.py` | `jd_negative_reviews`(写) |
| 抖音差评采集 | `shop.dy_negative_reviews.daily` | 逐日采集抖音负面评价(评价历史仅保留近 30 天),按订单 ID 幂等 | 每月 22 日 03:20 | `src/gyxx_flow/modules/shop_intelligence/collectors/dy_negative_reviews.py` | `dy_negative_reviews`(写) |
| 京东品退问题采集 | `shop.jd_after_sales_issues.daily` | 在商智 31 天限制下按连续区间采集售后服务单,按申请日期写业务日,幂等更新 | 每月 23 日 03:00 | `src/gyxx_flow/modules/shop_intelligence/collectors/jd_after_sales_issues.py` | `jd_after_sales_issues`(写,含范围外残留 DELETE 清理) |
| 抖音品退采集 | `shop.douyin_after_sales_quality.daily` | 采集抖音售后单,按申请时间归属业务日,账号 + 售后编号幂等更新 | 每月 23 日 03:20 | `src/gyxx_flow/modules/shop_intelligence/collectors/dy_after_sales_quality.py` | `dy_after_sales_quality_orders`、`dy_after_sales_quality_negotiations`(写) |
| 天猫客服消息采集 | `shop.tmall_customer_service.daily` | 按每次最多 31 天的窗口顺序采集天猫客服聊天记录,失败窗口保留数据 | 每月 24 日 03:40(未启用,仅手动) | `src/gyxx_flow/modules/shop_intelligence/collectors/tmall_customer_service_messages.py` | `tmall_customer_service_conversations`、`tmall_customer_service_messages`(写) |
| 天猫差评采集 | `shop.tmall_negative_reviews.daily` | 固定筛选负面评价并按评价时间范围采集后入库 | 每月 25 日 03:50(未启用,仅手动) | `src/gyxx_flow/modules/shop_intelligence/collectors/tmall_negative_reviews.py` | `tmall_negative_reviews`(写) |
| 抖音待改价申诉 | `shop.douyin_price_appeal` | 每个时点重新扫描可申诉并发起价格申诉;已申诉商品不重复出现 | 每日每小时整点(00:00–23:00) | `src/gyxx_flow/modules/shop_intelligence/collectors/dy_store_competitor_store_scraping.py` | 无(浏览器采集 + 本地 JSON) |
## 商品电商(product_commerce,12 个)
| 工作流名称 | ID | 功能 | 定时执行 | 启动脚本 | 关联数据库表 |
|---|---|---|---|---|---|
| 商品人群画像周采集 | `product.persona.daily` | 天猫、抖音、京东人群画像并行采集并写入对应飞书多维表 | 每周二 12:00 | `src/gyxx_flow/modules/product_commerce/run_daily_persona.py` | `daily_persona_metrics`(写) |
| 商品经营日报采集 | `product.daily` | 按业务日期(运行日 −1)采集 ERP 与三平台数据,幂等写 PostgreSQL,汇总分析后导出并写飞书日报 | 每日 08:40 | `src/gyxx_flow/modules/product_commerce/orchestrate_daily_collection.py` | 写:`product_reviews`、`product_daily_metrics`、`daily_style_metrics`、`fact_style_analysis_reports`、`decline_events`、`fact_alert_history`<br>读:`weekly_style_metrics`、`daily_persona_metrics`、`main_image_creatives`、`cmt_styles`、`cmt_notes_master`、`cmt_comments`、`cmt_creators`(及上述写表) |
| 私域库存预警 | `product.online_stock_alert.daily` | 查询线上渠道全部商品,库存低于 3 时汇总 SKU 与库存发飞书预警 | 每日 09:30 | `src/gyxx_flow/modules/product_commerce/online_stock_alert.py` | 无(企业数据交互 HTTP API + 飞书) |
| 三平台电商费用采集 | `product.ecommerce_costs.daily` | 并行采集天猫万相台、京东、抖音千川业务日前一天的费用,各平台先下载再导入,单平台失败不阻断其他 | 每日 10:00 | `…/product_commerce/collect_tmall_wanxiang_ads.py`<br>`…/product_commerce/import_tmall_product_ads.py`<br>`…/product_commerce/collect_jd_ad_costs.py`<br>`…/product_commerce/import_jd_ad_costs.py`<br>`…/product_commerce/collect_douyin_qianchuan_ads.py`<br>`…/product_commerce/import_douyin_qianchuan_ads.py` | `product_daily_metrics`(写,天猫原始费用并入 raw_data JSONB)<br>`jd_product_ad_daily_metrics`、`dy_product_ad_daily_metrics`(建+写)<br>`dim_style`(读) |
| 京东与天猫主图周采集 | `product.main_image.weekly` | 京东与天猫万相台主图并行采集,各自完成 PG 入库和飞书插入,互不影响 | 每周日 08:30 | `…/product_commerce/run_weekly_jd_main_image.py`<br>`…/product_commerce/run_weekly_main_image.py` | `main_image_creatives`(写) |
| 飞书商品月度销量表填写 | `product.sales_sheet.daily` | 抓取并校验全部 ERP 报表后填写飞书月度销量表,同名月份页幂等覆盖,无 ERP 编码产品按 0 写入 | 每月 1 日 10:00 | `src/gyxx_flow/modules/product_commerce/sync_monthly_sales_sheet.py` | 无(聚水潭浏览器采集 + 飞书写入) |
| 聚水潭全店铺款式日报 | `product.erp_all_shop_daily` | 聚水潭商品主题分析全选全部店铺,采集业务日前一天所有 ERP 款式的自定义销量与退货量并幂等写 PostgreSQL | 每日 09:00 | `src/gyxx_flow/modules/product_commerce/backfill_erp_all_shop_daily.py` | `erp_all_shop_style_daily_metrics`(读写) |
| 客服香港库存工作流 | `product.hk_anpei_public_inventory.daily` | 校验计划书 SKU 与库存表商品编码对应关系,调聚水潭库存接口查询并将公有可用数写入仓库现有库存列 | 每日 09:00 | `src/gyxx_flow/modules/product_commerce/public_sku_inventory_sheet.py` | 无(聚水潭 API + 飞书表格) |
| 天猫视频上传 | `product.video_upload` | 按店铺规则扫描飞书视频附件,凭附件级发布账本防重,仅在存在可领取附件时登录光合账号上传 | 每 3 天 14:00 | `src/gyxx_flow/modules/product_commerce/upload_video_to_guanghe.py` | `fact_video_publications`、`fact_video_publication_attempts`(建+读写,附件级发布账本)<br>`dim_style`、`product_daily_metrics`(读) |
| 京东视频上传 | `product.jd_video_upload` | 从凌云air基线扫描到视图末尾识别新增视频,以 record_id + file_token 幂等发布到京东 | 每 3 天 14:00 | `src/gyxx_flow/modules/product_commerce/upload_video_to_jd.py` | `fact_video_publications`、`fact_video_publication_attempts`(建+读写,发布账本)<br>`dim_style`、`product_daily_metrics`(读) |
| 天猫百亿补贴批量报名 | `product.tmall_baibu_apply` | 每次从飞书下载报名模板逐入口提交;默认仅导入 3 个不带超链入口(含老天猫全渠道 3C 后置完善草稿),可切换全部 6 个入口 | 每日每小时整点(00:00–23:00) | `…/product_commerce/tmall_baibu_apply_old_chain_bag.py`<br>`…/product_commerce/tmall_baibu_apply_old_all_bag.py`<br>`…/product_commerce/tmall_baibu_apply_old_all_3c.py`<br>`…/product_commerce/tmall_baibu_apply_old_chain_3c.py`<br>`…/product_commerce/tmall_baibu_apply_new_chain_bag.py`<br>`…/product_commerce/tmall_baibu_apply_new_all_bag.py` | 无(浏览器操作 + 飞书) |
| 三平台市场排行采集(用于行军图) | `product.market_rank` | 并行采集天猫、京东、抖音市场排行,按月份写入品牌行军图数据库并复用已有人工分类 | 每月 1 日 10:00 | `src/gyxx_flow/modules/product_commerce/orchestrate_market_rank_collection.py` | `market_product`、`market_product_classification`(读写)<br>`fact_market_product_snapshot`(写,重跑先 DELETE)<br>`fact_market_rank_reports`、`fact_market_rank_notifications`(建+读写)<br>`dim_market_style`、`dim_market_scene`(读) |
## 供应链(supply_chain,2 个)
| 工作流名称 | ID | 功能 | 定时执行 | 启动脚本 | 关联数据库表 |
|---|---|---|---|---|---|
| 补货预警日处理 | `supply.replenishment_alert.daily` | 依据配置 SKU 与库存阈值生成预警并写 PostgreSQL,仅存在预警时发送业务通知 | 每日 07:00 | `src/gyxx_flow/modules/supply_chain/run.py` | `fact_threshold_alert`(读写)<br>`dim_product`(写)<br>`workflow_runs`(读写,运行账本)<br>`bitable_insert_log`(读写,通知去重账本) |
| 同步聚水潭到货时间 | `supply.replenishment_arrival_sync.daily` | 读取到货同步待处理队列,按商品编码在聚水潭采购单页面更新预计到货时间并回写状态,失败行下次自动重试 | 每日 09:00 | `src/gyxx_flow/modules/supply_chain/orchestrator/scripts/collect_replenishment_arrival_sync.py` | `erp_replenishment_arrival_sync`(读写,队列表由 ERP 应用写入) |
## 附:已隐藏的工作流(未收录)
浏览器端隐藏的 10 个 ID:`content.creator_report.monthly`、`content.marketing_report.daily`、`content.metrics.backfill`、`content.relogin.weekly`(配置中已不存在)、`content.summary.monthly`、`content.summary.weekly`、`product.alert.daily`、`product.style_analysis.interval`、`supply.purchase_confirmation.daily`、`supply.replenishment.weekly`。
+62
View File
@@ -223,6 +223,8 @@ class AccountKeepaliveManager:
self._next_due: dict[str, float] = {}
self._locks = LockManager(settings.data_root / "state" / "locks")
self._login_runner = login_runner or self._run_configured_login
self._audit_store: Any | None = None
self._audit_seen: dict[str, tuple[str, float]] = {}
def tick(self) -> list[dict[str, object]]:
"""Run due account refreshes once; never starts work in dry-run mode."""
@@ -479,6 +481,9 @@ class AccountKeepaliveManager:
real_chrome=account.keepalive_real_chrome,
retries=1,
persist_state_on_close=False,
# 目标站点均为国内站,代理出口反被部分 WAF 拉黑(如蝉妈妈
# TLS reset),账号保活一律直连。
extra_flags=["--no-proxy-server"],
)
try:
browser.start()
@@ -583,8 +588,63 @@ class AccountKeepaliveManager:
if result.get("status") == "success":
payload["last_success_at"] = timestamp
atomic_write_json(path, payload)
self._audit_keepalive(account, result)
return dict(result)
_AUDIT_STATE_BY_PROBE = {
"success": "healthy",
"expired": "expired",
"not_logged_in": "not_logged_in",
"busy": "busy",
"disabled": "disabled",
}
def _audit_keepalive(self, account: Any, result: dict[str, object]) -> None:
"""Record probe outcomes so the real cookie lifetime becomes measurable.
The resident scheduler probes every ``keepalive_interval_seconds``
(10 minutes by default), which is what turns "logins seem to last about
a week" into a measured distribution. Repeated identical outcomes
within an hour collapse into one row, so the event table tracks state
changes rather than polling noise.
"""
try:
from gyxx_flow.ops.audit import snapshot_for_account
probe = str(result.get("status", "") or "error")
state = self._AUDIT_STATE_BY_PROBE.get(probe, "probe_error")
key = str(account.account_id)
previous = self._audit_seen.get(key)
if previous is not None:
previous_state, previous_at = previous
if previous_state == state and (self._clock() - previous_at) < 3600:
return
store = self._audit()
if store is None or not store.enabled:
return
if store.record_cookie_event(
account_id=key,
state=state,
snapshot=snapshot_for_account(account),
probe_status=probe,
probe_source="keepalive",
detail=str(result.get("error_type", "") or ""),
):
self._audit_seen[key] = (state, self._clock())
except Exception: # noqa: BLE001 - telemetry must never break keepalive
return
def _audit(self) -> Any | None:
if self._audit_store is None:
try:
from gyxx_flow.ops.audit import AuditStore
self._audit_store = AuditStore(self.settings.data_root)
except Exception: # noqa: BLE001
return None
return self._audit_store
def _vault_cookie_valid(account: Any, store: BrowserCookieStore) -> bool:
if not account.cookie_file.exists():
@@ -761,6 +821,8 @@ def login_account(
"--remote-debugging-address=127.0.0.1",
"--disable-session-crashed-bubble",
"--disable-features=InfiniteSessionRestore",
# 国内站直连,避免代理出口被 WAF 拉黑导致登录/探测失败
"--no-proxy-server",
],
)
try:
+19
View File
@@ -168,3 +168,22 @@ def looks_like_login_url(url: str) -> bool:
path = parsed.path.casefold()
markers = ("login", "passport", "signin", "sign-in")
return any(marker in hostname or marker in path for marker in markers)
class CookieExpiredError(RuntimeError):
"""A platform login state expired and no automatic recovery exists.
Collectors raise this instead of waiting for a human QR/SMS login: the
account cookie gate has already blocked unverified accounts before the
run, so reaching this point means the session died mid-run. The CLI maps
it to ``COOKIE_SKIP_EXIT_CODE`` so the engine records a cookie skip.
"""
def cookie_expired_message(account_id: str, platform: str) -> str:
"""One standard, operator-actionable expiry message across platforms."""
return (
f"{platform} 登录态已失效:请执行 `uv run gyxx accounts login {account_id}` "
"完成扫码/验证码重新登录(或等待飞书通知后处理)"
)
+35 -15
View File
@@ -1210,29 +1210,49 @@ def _merge_storage_state(
def _account_cookie_valid(account: AccountBinding, cookies: object) -> bool:
"""A vault cookie set is valid when a live account cookie is present."""
"""A vault cookie set is valid when a live account cookie is present.
With ``required_cookie_names`` configured, every required name must be
present as a live, domain-matched cookie; a single anonymous cookie that
happens to share a required name (e.g. ``a1``) must not validate a guest
session. Without required names, any live cookie on a required domain
keeps accepting the set.
"""
if not isinstance(cookies, list):
return False
required_names = account.required_cookie_names
required_domains = account.required_cookie_domains
for item in cookies:
if not isinstance(item, dict):
continue
if _cookie_expired(item):
continue
if not str(item.get("value", "")):
continue
name = str(item.get("name", ""))
domain = str(item.get("domain", "")).casefold().lstrip(".")
name_matches = not required_names or name in required_names
domain_matches = not required_domains or any(
def _domain_matches(domain: str) -> bool:
return not required_domains or any(
domain == required.casefold()
or domain.endswith("." + required.casefold())
for required in required_domains
)
if name_matches and domain_matches:
return True
return False
def _live(item: object) -> bool:
return (
isinstance(item, dict)
and not _cookie_expired(item)
and bool(str(item.get("value", "")))
)
if not required_names:
return any(
_live(item)
and _domain_matches(
str(item.get("domain", "")).casefold().lstrip(".")
)
for item in cookies
)
outstanding = set(required_names)
for item in cookies:
if not _live(item):
continue
if _domain_matches(str(item.get("domain", "")).casefold().lstrip(".")):
outstanding.discard(str(item.get("name", "")))
return not outstanding
def _cookie_expired(cookie: Mapping[str, object], *, now: float | None = None) -> bool:
@@ -0,0 +1,338 @@
"""Minimal, signed Jushuitan inventory-query client for workflow adapters."""
from __future__ import annotations
import hashlib
import json
import os
import time
import uuid
from collections.abc import Mapping
from dataclasses import dataclass
from decimal import Decimal, InvalidOperation
from typing import Any
from urllib.parse import urlparse
import requests
_TOKEN_PATH = "/openWeb/auth/getInitToken"
_INVENTORY_PATH = "/open/inventory/query"
_RATE_LIMIT_CODES = {199, 200}
_ACCESS_TOKEN_INVALID_CODE = 100
_MAX_RESPONSE_BYTES = 2 * 1024 * 1024
_MAX_PAGES = 100
_REQUEST_INTERVAL_SECONDS = 1.1
class JushuitanInventoryError(RuntimeError):
"""Safe-to-log error raised for Jushuitan query or response failures."""
@dataclass(frozen=True, slots=True)
class InventoryRecord:
sku: str
qty: int
order_lock: int
lock_qty: int
virtual_qty: int
@property
def public_available_qty(self) -> int:
return self.qty - self.order_lock - self.lock_qty + self.virtual_qty
def _sign(app_secret: str, params: Mapping[str, object]) -> str:
source = app_secret + "".join(
f"{key}{value}"
for key, value in sorted(params.items())
if key.lower() != "sign" and value is not None and str(value) != ""
)
return hashlib.md5(source.encode("utf-8")).hexdigest()
def _response_code(value: object) -> int:
if isinstance(value, bool):
raise JushuitanInventoryError("Jushuitan response has an invalid code")
try:
return int(value)
except (TypeError, ValueError) as exc:
raise JushuitanInventoryError(
"Jushuitan response has an invalid code"
) from exc
def _quantity(value: object, *, field: str, sku: str) -> int:
if value is None:
return 0
if isinstance(value, bool):
raise JushuitanInventoryError(
f"Jushuitan inventory has an invalid {field} for SKU {sku}"
)
try:
parsed = Decimal(str(value).strip())
except (InvalidOperation, ValueError) as exc:
raise JushuitanInventoryError(
f"Jushuitan inventory has an invalid {field} for SKU {sku}"
) from exc
if not parsed.is_finite() or parsed != parsed.to_integral_value():
raise JushuitanInventoryError(
f"Jushuitan inventory has a non-integer {field} for SKU {sku}"
)
return int(parsed)
def _value(item: Mapping[str, Any], snake_name: str, camel_name: str) -> object:
if snake_name in item:
return item[snake_name]
return item.get(camel_name)
class JushuitanInventoryClient:
"""Signed form client matching data-hub's Jushuitan API contract."""
def __init__(
self,
*,
app_key: str | None = None,
app_secret: str | None = None,
base_url: str | None = None,
timeout: tuple[float, float] = (10.0, 45.0),
) -> None:
self.app_key = (app_key or os.environ.get("JUSHUITAN_APP_KEY", "")).strip()
self.app_secret = (
app_secret or os.environ.get("JUSHUITAN_APP_SECRET", "")
).strip()
configured_url = (
base_url
or os.environ.get("JUSHUITAN_BASE_URL", "")
or "https://openapi.jushuitan.com"
).strip()
parsed_url = urlparse(configured_url)
if (
parsed_url.scheme.lower() != "https"
or not parsed_url.netloc
or parsed_url.username
or parsed_url.password
or parsed_url.query
or parsed_url.fragment
):
raise JushuitanInventoryError(
"JUSHUITAN_BASE_URL must be an HTTPS base URL without credentials"
)
if not self.app_key or not self.app_secret:
raise JushuitanInventoryError(
"JUSHUITAN_APP_KEY and JUSHUITAN_APP_SECRET must be injected "
"into the scheduler environment"
)
self.base_url = configured_url.rstrip("/")
self.timeout = timeout
self.session = requests.Session()
self._access_token = ""
self._last_request_at = 0.0
def query(self, skus: list[str]) -> dict[str, InventoryRecord]:
normalized_skus = list(dict.fromkeys(sku.strip() for sku in skus if sku.strip()))
if not normalized_skus:
raise JushuitanInventoryError("No SKU values were supplied")
if len(normalized_skus) > 100:
raise JushuitanInventoryError(
"A single inventory query may include at most 100 SKUs"
)
self._access_token = self._request_access_token()
page_index = 1
page_count = 1
records: dict[str, InventoryRecord] = {}
requested = set(normalized_skus)
while page_index <= page_count:
if page_index > _MAX_PAGES:
raise JushuitanInventoryError(
"Jushuitan inventory response exceeded the page safety limit"
)
response = self._query_page(normalized_skus, page_index)
data = response.get("data")
if not isinstance(data, Mapping):
raise JushuitanInventoryError(
"Jushuitan inventory response is missing its data object"
)
raw_rows = data.get("inventorys")
if raw_rows is None:
raw_rows = data.get("datas")
if raw_rows is None:
raw_rows = []
if not isinstance(raw_rows, list):
raise JushuitanInventoryError(
"Jushuitan inventory response has an invalid inventory list"
)
for index, raw_item in enumerate(raw_rows, start=1):
if raw_item is None:
continue
if not isinstance(raw_item, Mapping):
raise JushuitanInventoryError(
f"Jushuitan inventory row {index} is invalid"
)
sku_value = _value(raw_item, "sku_id", "skuId")
sku = str(sku_value or "").strip()
if not sku:
raise JushuitanInventoryError(
f"Jushuitan inventory row {index} has no SKU"
)
if sku not in requested:
raise JushuitanInventoryError(
"Jushuitan returned a SKU that was not requested"
)
record = InventoryRecord(
sku=sku,
qty=_quantity(
_value(raw_item, "qty", "qty"), field="qty", sku=sku
),
order_lock=_quantity(
_value(raw_item, "order_lock", "orderLock"),
field="order_lock",
sku=sku,
),
lock_qty=_quantity(
_value(raw_item, "lock_qty", "lockQty"),
field="lock_qty",
sku=sku,
),
virtual_qty=_quantity(
_value(raw_item, "virtual_qty", "virtualQty"),
field="virtual_qty",
sku=sku,
),
)
if sku in records:
raise JushuitanInventoryError(
f"Jushuitan returned duplicate inventory rows for SKU {sku}"
)
records[sku] = record
raw_page_count = data.get("page_count", data.get("pageCount", 1))
try:
page_count = max(1, int(raw_page_count or 1))
except (TypeError, ValueError) as exc:
raise JushuitanInventoryError(
"Jushuitan inventory response has an invalid page count"
) from exc
if page_count > _MAX_PAGES:
raise JushuitanInventoryError(
"Jushuitan inventory response exceeded the page safety limit"
)
page_index += 1
return records
def _request_access_token(self) -> str:
params: dict[str, object] = {
"app_key": self.app_key,
"charset": "utf-8",
"code": uuid.uuid4().hex[:6],
"grant_type": "authorization_code",
"timestamp": str(int(time.time())),
}
response = self._post_form(_TOKEN_PATH, params)
data = response.get("data")
if not isinstance(data, Mapping):
raise JushuitanInventoryError(
"Jushuitan token response is missing its data object"
)
token = data.get("access_token") or data.get("accessToken")
if not isinstance(token, str) or not token.strip():
raise JushuitanInventoryError(
"Jushuitan token response has no access token"
)
return token.strip()
def _query_page(self, skus: list[str], page_index: int) -> dict[str, Any]:
biz = {
"sku_ids": ",".join(skus),
"has_lock_qty": True,
"page_index": page_index,
"page_size": 100,
}
params: dict[str, object] = {
"access_token": self._access_token,
"app_key": self.app_key,
"charset": "utf-8",
"timestamp": str(int(time.time())),
"version": "2",
"biz": json.dumps(biz, ensure_ascii=False, separators=(",", ":")),
}
for token_attempt in range(2):
response = self._post_form(
_INVENTORY_PATH,
params,
allowed_api_codes=frozenset({_ACCESS_TOKEN_INVALID_CODE}),
)
code = _response_code(response.get("code"))
if code == _ACCESS_TOKEN_INVALID_CODE and token_attempt == 0:
self._access_token = self._request_access_token()
params["access_token"] = self._access_token
params["timestamp"] = str(int(time.time()))
continue
return response
raise JushuitanInventoryError("Jushuitan rejected the access token")
def _post_form(
self,
path: str,
params: dict[str, object],
*,
allowed_api_codes: frozenset[int] = frozenset(),
) -> dict[str, Any]:
rate_limit_attempt = 0
while True:
now = time.monotonic()
wait_seconds = _REQUEST_INTERVAL_SECONDS - (now - self._last_request_at)
if wait_seconds > 0:
time.sleep(wait_seconds)
self._last_request_at = time.monotonic()
signed_params = dict(params)
signed_params["sign"] = _sign(self.app_secret, signed_params)
try:
response = self.session.post(
f"{self.base_url}{path}",
data=signed_params,
timeout=self.timeout,
headers={"Accept": "application/json"},
)
except requests.RequestException as exc:
raise JushuitanInventoryError(
f"Jushuitan request failed for {path}"
) from exc
if response.status_code < 200 or response.status_code >= 300:
raise JushuitanInventoryError(
f"Jushuitan endpoint {path} returned HTTP {response.status_code}"
)
if len(response.content) > _MAX_RESPONSE_BYTES:
raise JushuitanInventoryError(
"Jushuitan response exceeded the 2 MiB safety limit"
)
try:
payload = response.json()
except ValueError as exc:
raise JushuitanInventoryError(
f"Jushuitan endpoint {path} returned invalid JSON"
) from exc
if not isinstance(payload, dict):
raise JushuitanInventoryError(
f"Jushuitan endpoint {path} returned an invalid response"
)
code = _response_code(payload.get("code"))
if code in _RATE_LIMIT_CODES and rate_limit_attempt < 4:
time.sleep(0.5 * (2**rate_limit_attempt))
rate_limit_attempt += 1
params["timestamp"] = str(int(time.time()))
continue
if code != 0 and code not in allowed_api_codes:
raise JushuitanInventoryError(
f"Jushuitan endpoint {path} returned API code {code}"
)
return payload
__all__ = [
"InventoryRecord",
"JushuitanInventoryClient",
"JushuitanInventoryError",
]
+3 -3
View File
@@ -37,8 +37,8 @@ def is_browser_timeout_error(error: BaseException) -> bool:
)
def _is_browser_closed_error(error: BaseException) -> bool:
"""Recognize a browser target that disappeared during cleanup."""
def is_browser_closed_error(error: BaseException) -> bool:
"""Recognize a browser target that disappeared (page/context/browser)."""
if any(
error_type.__name__ == "TargetClosedError"
@@ -155,7 +155,7 @@ class ScraplingBrowser:
# and context-manager cleanup. The current state cannot
# be exported then, but that must not turn a successful
# download into a false failure or mask the action error.
if not _is_browser_closed_error(exc):
if not is_browser_closed_error(exc):
raise
finally:
try:
+16
View File
@@ -79,6 +79,10 @@ class WorkflowEntry:
steps: tuple[WorkflowStepEntry, ...] = ()
notification_workflow_id: str | None = None
topic_config: WorkflowTopicConfig | None = None
# Orchestrator entries whose own binding carries no runtime account while
# the spawned child collectors do; the cookie gate probes these before
# the workflow runs.
required_accounts: tuple[str, ...] = ()
@dataclass(frozen=True, slots=True)
@@ -218,6 +222,9 @@ def _parse_workflow(item: Any) -> WorkflowEntry:
if trigger == "scheduled" and (not isinstance(task_name, str) or not task_name):
raise CatalogError(f"scheduled workflow requires source task name: {workflow_id}")
topic_config = _parse_topic_config(item.get("topic_config"), workflow_id)
required_accounts = _parse_required_accounts(
item.get("required_accounts", []), workflow_id
)
return WorkflowEntry(
workflow_id=workflow_id,
module=module,
@@ -229,9 +236,18 @@ def _parse_workflow(item: Any) -> WorkflowEntry:
note=item.get("note"),
steps=steps,
topic_config=topic_config,
required_accounts=required_accounts,
)
def _parse_required_accounts(value: Any, workflow_id: str) -> tuple[str, ...]:
if not isinstance(value, list) or not all(
isinstance(item, str) and _WORKFLOW_ID.fullmatch(item) for item in value
):
raise CatalogError(f"invalid required_accounts for {workflow_id}")
return tuple(dict.fromkeys(value))
def _parse_topic_config(value: Any, workflow_id: str) -> WorkflowTopicConfig | None:
if value is None:
return None
+219 -1
View File
@@ -229,6 +229,36 @@ def build_parser() -> argparse.ArgumentParser:
status_parser = acceptance_commands.add_parser("status")
status_parser.add_argument("--json", action="store_true")
status_parser.add_argument("--output", type=Path)
audit_parser = commands.add_parser(
"audit", help="query the PostgreSQL operational audit tables"
)
audit_commands = audit_parser.add_subparsers(dest="audit_command", required=True)
audit_runs = audit_commands.add_parser("runs", help="recent workflow runs")
audit_runs.add_argument("--workflow")
audit_runs.add_argument("--status")
audit_runs.add_argument("--date", dest="business_date")
audit_runs.add_argument("--limit", type=int, default=30)
audit_runs.add_argument("--json", action="store_true")
audit_cookies = audit_commands.add_parser(
"cookies", help="recent cookie observations"
)
audit_cookies.add_argument("--account")
audit_cookies.add_argument("--state")
audit_cookies.add_argument("--limit", type=int, default=30)
audit_cookies.add_argument("--json", action="store_true")
audit_stats = audit_commands.add_parser(
"cookie-stats", help="measured cookie lifetime per account"
)
audit_stats.add_argument("--account")
audit_stats.add_argument("--json", action="store_true")
audit_commands.add_parser(
"schema", help="print the DDL that creates the audit tables"
)
return parser
@@ -304,6 +334,8 @@ def main(
and arguments.acceptance_command == "status"
):
return _acceptance_status_command(arguments, resolved_settings, output)
if arguments.command == "audit":
return _audit_command(arguments, resolved_settings, output)
if arguments.command == "scripts" and arguments.scripts_command == "list":
return _list_scripts(arguments, output)
if arguments.command == "scripts" and arguments.scripts_command == "run":
@@ -549,6 +581,152 @@ def _acceptance_status_command(
return EXIT_SUCCESS if report.is_complete else EXIT_ACCEPTANCE_INCOMPLETE
def _format_seconds(value: object) -> str:
"""Human-readable duration for the audit tables (None -> '-')."""
if value is None:
return "-"
try:
seconds = int(value)
except (TypeError, ValueError):
return "-"
if seconds >= 86400:
return f"{seconds / 86400:.1f}d"
if seconds >= 3600:
return f"{seconds / 3600:.1f}h"
if seconds >= 60:
return f"{seconds / 60:.0f}m"
return f"{seconds}s"
def _format_timestamp(value: object) -> str:
if value is None:
return "-"
text = value.isoformat() if hasattr(value, "isoformat") else str(value)
return text.replace("T", " ")[:19]
def _audit_command(
arguments: argparse.Namespace, settings: Settings, output: TextIO
) -> int:
from gyxx_flow.ops.audit import SCHEMA_SQL, AuditStore
command = arguments.audit_command
if command == "schema":
output.write(SCHEMA_SQL.strip() + "\n")
return EXIT_SUCCESS
store = AuditStore(settings.data_root)
if not store.enabled:
output.write("审计已关闭(GYXX_OPS_AUDIT=off)\n")
return EXIT_CONFIGURATION
if command == "runs":
rows = store.query_runs(
workflow_id=arguments.workflow,
status=arguments.status,
business_date=arguments.business_date,
limit=arguments.limit,
)
elif command == "cookies":
rows = store.query_cookie_events(
account_id=arguments.account,
state=arguments.state,
limit=arguments.limit,
)
elif command == "cookie-stats":
rows = (
store.query_cookie_lifetimes(account_id=arguments.account)
if arguments.account
else store.query_cookie_stats()
)
else: # pragma: no cover - argparse 已限制取值
raise CliConfigurationError(f"unsupported audit command: {command}")
if store.last_error and not rows:
output.write(f"审计库不可用:{store.last_error}\n")
return EXIT_CONFIGURATION
if getattr(arguments, "json", False):
_write_json(output, {"rows": rows})
return EXIT_SUCCESS
if not rows:
output.write("暂无审计记录\n")
return EXIT_SUCCESS
if command == "runs":
output.write(
f"{'started':19s} {'workflow':40s} {'date':10s} {'mode':7s} "
f"{'status':8s} {'dur':>6s} steps\n"
)
for row in rows:
steps = (
f"{row.get('step_success', 0)}/{row.get('step_total', 0)}"
f" failed={row.get('step_failed', 0)}"
)
output.write(
f"{_format_timestamp(row.get('started_at')):19s} "
f"{str(row.get('workflow_id', ''))[:40]:40s} "
f"{_format_timestamp(row.get('business_date'))[:10]:10s} "
f"{str(row.get('mode', '')):7s} "
f"{str(row.get('status', '')):8s} "
f"{_format_seconds(row.get('duration_seconds')):>6s} {steps}\n"
)
if row.get("error"):
output.write(f"{'':21s}error: {str(row['error'])[:160]}\n")
return EXIT_SUCCESS
if command == "cookies":
output.write(
f"{'observed':19s} {'account':24s} {'state':14s} {'probe':14s} "
f"{'fingerprint':12s} expires_at\n"
)
for row in rows:
output.write(
f"{_format_timestamp(row.get('observed_at')):19s} "
f"{str(row.get('account_id', ''))[:24]:24s} "
f"{str(row.get('state', '')):14s} "
f"{str(row.get('probe_status', '')):14s} "
f"{str(row.get('fingerprint', ''))[:12]:12s} "
f"{_format_timestamp(row.get('declared_expires_at'))}\n"
)
return EXIT_SUCCESS
# cookie-stats / 单账号 lifetime
if arguments.account:
output.write(
f"{'fingerprint':12s} {'first_seen':19s} {'last_healthy':19s} "
f"{'failed_at':19s} {'lifetime':>9s} {'declared':19s}\n"
)
for row in rows:
output.write(
f"{str(row.get('fingerprint', ''))[:12]:12s} "
f"{_format_timestamp(row.get('first_seen_at')):19s} "
f"{_format_timestamp(row.get('last_healthy_at')):19s} "
f"{_format_timestamp(row.get('first_failed_at')):19s} "
f"{_format_seconds(row.get('lifetime_seconds')):>9s} "
f"{_format_timestamp(row.get('declared_expires_at')):19s}\n"
)
return EXIT_SUCCESS
output.write(
f"{'account':24s} {'sessions':>8s} {'expired':>7s} {'avg':>7s} "
f"{'median':>7s} {'min':>7s} {'max':>7s} last_failed\n"
)
for row in rows:
output.write(
f"{str(row.get('account_id', ''))[:24]:24s} "
f"{row.get('sessions', 0):>8d} {row.get('expired_sessions', 0):>7d} "
f"{_format_seconds(row.get('avg_lifetime_seconds')):>7s} "
f"{_format_seconds(row.get('median_lifetime_seconds')):>7s} "
f"{_format_seconds(row.get('min_lifetime_seconds')):>7s} "
f"{_format_seconds(row.get('max_lifetime_seconds')):>7s} "
f"{_format_timestamp(row.get('last_failed_at'))}\n"
)
return EXIT_SUCCESS
def _list_workflows(
arguments: argparse.Namespace,
registry: WorkflowRegistry,
@@ -703,7 +881,19 @@ def _python_scheduler(
if enable_account_keepalive and not dry_run:
from gyxx_flow.accounts import AccountKeepaliveManager
maintenance = AccountKeepaliveManager(settings).tick
keepalive_manager = AccountKeepaliveManager(settings)
recovery_manager = _build_cookie_recovery_manager(
settings, launcher=launcher, keepalive=keepalive_manager
)
def maintenance() -> None: # noqa: D107 - scheduler maintenance chain
keepalive_manager.tick()
if recovery_manager is not None:
try:
recovery_manager.tick()
except Exception: # noqa: BLE001 - 自动重跑不得影响调度心跳
pass
return PythonScheduler(
catalog,
settings.data_root,
@@ -717,6 +907,29 @@ def _python_scheduler(
)
def _build_cookie_recovery_manager(
settings: Settings,
*,
launcher: object,
keepalive: object,
):
"""Build the cookie-gate recovery rerun manager; None when disabled."""
from gyxx_flow.cookie_gate import (
CookieGate,
CookieGateRecoveryManager,
gate_enabled,
)
if not gate_enabled():
return None
try:
gate = CookieGate(settings, keepalive=keepalive) # type: ignore[arg-type]
except Exception: # noqa: BLE001 - 无集成配置时没有可恢复的账号
return None
return CookieGateRecoveryManager(gate, launcher=launcher)
def _schedule_run_command(
arguments: argparse.Namespace,
catalog: WorkflowCatalog,
@@ -885,6 +1098,11 @@ def _execute_once(
dry_run: bool,
settings: Settings,
) -> tuple[int, dict[str, object]]:
if not dry_run:
# 每账号 Cookie 健康门控:dry-run 不注入,也不做任何探测/通知。
from gyxx_flow.cookie_gate import apply_cookie_gates
workflow = apply_cookie_gates(workflow, settings=settings)
context = RunContext.create(
workflow.workflow_id,
business_date.isoformat(),
+118 -2
View File
@@ -5,6 +5,7 @@ from __future__ import annotations
import calendar
import hashlib
import hmac
import http.client
import ipaddress
import json
import os
@@ -15,6 +16,7 @@ import sys
import tempfile
import threading
import time
import traceback
import unicodedata
import uuid
from contextlib import suppress
@@ -24,7 +26,7 @@ from http import HTTPStatus
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from importlib import resources
from pathlib import Path
from typing import Any, Mapping, Protocol, Sequence, TextIO, cast
from typing import Any, Callable, Mapping, Protocol, Sequence, TextIO, cast
from urllib.parse import parse_qs, unquote, urlsplit
from zoneinfo import ZoneInfo
@@ -171,7 +173,6 @@ _WORKFLOW_NAMES = {
"content.metrics.daily": "合作达人内容指标日报采集",
"content.metrics.self_operated.daily": "自营内容指标日报采集",
"content.marketing_report.daily": "营销日报生成与发送",
"content.relogin.weekly": "内容平台登录态刷新",
"content.creator_report.monthly": "达人月报生成",
"content.summary.monthly": "内容月度汇总",
"content.comments.weekly": "多平台评论周采集",
@@ -3736,6 +3737,7 @@ class WorkflowConsoleHTTPServer(ThreadingHTTPServer):
self.console_service = service
self.console_token = token
self.bind_host = bind_host
self.started_at = time.time()
super().__init__(server_address, WorkflowConsoleRequestHandler)
def handle_error(
@@ -3796,6 +3798,29 @@ class WorkflowConsoleRequestHandler(BaseHTTPRequestHandler):
if method in {"GET", "HEAD"} and path == "/assets/app.js":
self._static("app.js", "text/javascript; charset=utf-8", head=method == "HEAD")
return
if method == "GET" and path == "/api/health":
# 自检探针端点:不鉴权、不触外部依赖,只反映进程与调度锁存活,
# 供进程内看门狗与 deploy/watchdog 保活脚本探测。
try:
scheduler = _scheduler_process_from_lock(
self.console_server.console_service.settings
)
except Exception:
scheduler = None
self._json(
HTTPStatus.OK,
{
"ok": True,
"pid": os.getpid(),
"started_at": self.console_server.started_at,
"uptime_seconds": round(
time.time() - self.console_server.started_at, 1
),
"scheduler_running": scheduler is not None,
"scheduler_pid": scheduler.pid if scheduler else None,
},
)
return
if method == "GET" and path == "/api/overview":
self._require_api_access(mutation=False)
self._json(HTTPStatus.OK, self.console_server.console_service.overview())
@@ -4020,6 +4045,9 @@ class WorkflowConsoleRequestHandler(BaseHTTPRequestHandler):
except (BrokenPipeError, ConnectionResetError):
return
except Exception:
# 前端只拿到固定文案;完整堆栈落到 stderr(随启动重定向进
# var/logs/console-restarts 或服务日志),否则故障无法定位。
traceback.print_exc()
self._json_error(HTTPStatus.INTERNAL_SERVER_ERROR, "控制台处理请求失败")
def _require_api_access(self, *, mutation: bool) -> None:
@@ -4190,6 +4218,92 @@ def create_console_server(
)
CONSOLE_WATCHDOG_EXIT_CODE = 70
def _probe_console_health(port: int, timeout_seconds: float) -> bool:
# 只连本机回环端口的固定路径 /api/health,探测自身 HTTP 服务往返。
connection = http.client.HTTPConnection("127.0.0.1", port, timeout=timeout_seconds)
try:
connection.request("GET", "/api/health")
response = connection.getresponse()
response.read()
return response.status == HTTPStatus.OK
except Exception:
return False
finally:
connection.close()
def _console_watchdog(
port: int,
*,
interval_seconds: float,
failure_limit: int,
probe_timeout_seconds: float,
stopped: threading.Event,
probe: Callable[[int, float], bool] | None = None,
exit_process: Callable[[int], None] = os._exit,
sleep: Callable[[float], None] = time.sleep,
) -> None:
"""进程内自检:周期探测自身 /api/health,连续失败则退出交给守护拉起。
HTTP accept 线程活着但工作线程全部卡死(锁竞争/句柄耗尽)时,外部
静态页能打开而 API 全挂——正是"控制台暂时不可用"的形态;只有真实
走一遍 HTTP 往返才能暴露,所以探针必须打 socket 而不是查内存状态。
"""
run_probe = probe or _probe_console_health
failures = 0
while not stopped.is_set():
if run_probe(port, probe_timeout_seconds):
failures = 0
else:
failures += 1
print(
f"⚠️ 控制台自检失败 {failures}/{failure_limit}: 127.0.0.1:{port}/api/health",
file=sys.stderr,
flush=True,
)
if failures >= failure_limit:
print(
"❌ 控制台自检连续失败,进程退出等待守护脚本拉起",
file=sys.stderr,
flush=True,
)
exit_process(CONSOLE_WATCHDOG_EXIT_CODE)
return
sleep(interval_seconds)
def _start_console_watchdog(server: WorkflowConsoleHTTPServer) -> threading.Event:
interval = max(
5.0, float(os.environ.get("GYXX_CONSOLE_WATCHDOG_INTERVAL_SECONDS", "30"))
)
limit = int(os.environ.get("GYXX_CONSOLE_WATCHDOG_FAILURES", "3"))
enabled = (
os.environ.get("GYXX_CONSOLE_WATCHDOG", "1").strip().lower()
not in {"0", "false", "no", "off"}
and limit > 0
)
stopped = threading.Event()
if not enabled:
return stopped
thread = threading.Thread(
target=_console_watchdog,
args=(server.server_port,),
kwargs={
"interval_seconds": interval,
"failure_limit": limit,
"probe_timeout_seconds": min(5.0, interval),
"stopped": stopped,
},
name="gyxx-console-watchdog",
daemon=True,
)
thread.start()
return stopped
def serve_console(
settings: Settings,
*,
@@ -4211,11 +4325,13 @@ def serve_console(
display_host = "127.0.0.1" if host in {"0.0.0.0", "::"} else host
output.write(f"GYXX Flow console: http://{display_host}:{server.server_port}\n")
output.flush()
watchdog_stopped = _start_console_watchdog(server)
try:
server.serve_forever(poll_interval=0.5)
except KeyboardInterrupt:
pass
finally:
watchdog_stopped.set()
server.server_close()
return 0
+966
View File
@@ -0,0 +1,966 @@
"""Per-account cookie health gate injected before workflow collection steps.
Every runtime account a workflow depends on is checked before the collection
steps that use it may run. Accounts fall into three gate modes derived from
``config/runtime-bindings.json``:
* ``blocking`` -- QR-code / SMS-only accounts (no credential environment and
no configured login command). The gate verifies the login state and, when
it has expired, fails the gate step so every dependent step is skipped with
``dependency-failed`` and sends a Feishu alert asking the login owner to
re-login interactively via ``gyxx accounts login <account_id>``.
* ``recover`` -- accounts with a vault-managed credential login command
(``login_mode`` ``C``). The keepalive manager re-runs the credential login
automatically; only a failed recovery blocks the workflow and alerts.
* ``defer`` -- accounts whose member scripts embed username/password login
flows. These keep their script-level login behaviour unchanged and are not
gated at all.
The per-account decision is persisted as a small state machine under
``state/accounts/<id>/cookie_state.json`` (``healthy`` <-> ``awaiting_login``)
so repeat failures alert at most once per notify interval instead of once per
workflow run.
"""
from __future__ import annotations
import json
import os
from collections.abc import Callable
from dataclasses import dataclass, replace
from datetime import date, datetime, timezone
from pathlib import Path
from typing import Any, Mapping
from gyxx_flow.accounts import (
AccountKeepaliveManager,
_keepalive_requires_relogin,
_read_keepalive_status,
_vault_cookie_valid,
)
from gyxx_flow.adapters.browser import BrowserCookieStore
from gyxx_flow.adapters.feishu_messages import send_lark_bot_message
from gyxx_flow.adapters.integration import RuntimeIntegrationCatalog
from gyxx_flow.catalog import WorkflowCatalog
from gyxx_flow.core.artifacts import atomic_write_json
from gyxx_flow.core.config import Settings
from gyxx_flow.core.context import RunContext
from gyxx_flow.notification_routing import (
ANALYZER_ACCEPTANCE_RECIPIENT_OPEN_ID,
ANALYZER_NOTIFICATION_APP_PROFILE,
NotificationRoutingStore,
)
from gyxx_flow.workflow.model import StepDefinition, WorkflowDefinition
from gyxx_flow.workflow.steps import StepExecution
GATE_MODE_BLOCKING = "blocking"
GATE_MODE_RECOVER = "recover"
GATE_MODE_DEFER = "defer"
# 飞书通知的默认接收人:按手机号在 config/notification-routing.json 的
# people 段解析 open_id;解析失败时回退到同名 analyzer 常量收件人。
DEFAULT_LOGIN_ALERT_MOBILE = "+8613663785145"
GYXX_COOKIE_GATE = "GYXX_COOKIE_GATE"
GYXX_COOKIE_GATE_MAX_STATUS_AGE_SECONDS = "GYXX_COOKIE_GATE_MAX_STATUS_AGE_SECONDS"
GYXX_LOGIN_ALERT_OPEN_IDS = "GYXX_LOGIN_ALERT_OPEN_IDS"
GYXX_LOGIN_ALERT_APP_PROFILE = "GYXX_LOGIN_ALERT_APP_PROFILE"
GYXX_LOGIN_ALERT_NOTIFY_INTERVAL_SECONDS = "GYXX_LOGIN_ALERT_NOTIFY_INTERVAL_SECONDS"
_GATE_STEP_PREFIX = "cookie-gate"
_GATE_STATE_FILENAME = "cookie_state.json"
_GATE_TIMEOUT_SECONDS = 1800.0
_DEFAULT_MAX_STATUS_AGE_SECONDS = 3600.0
_MIN_MAX_STATUS_AGE_SECONDS = 300.0
_DEFAULT_NOTIFY_INTERVAL_SECONDS = 1800.0
_DISABLED_VALUES = frozenset({"0", "false", "off", "no", "disabled"})
# 自动重跑:待重跑清单上限、健康探活节流、同账号重跑启动冷却。
_MAX_PENDING_RERUNS = 20
_RECOVERY_PROBE_INTERVAL_SECONDS = 300.0
_RECOVERY_LAUNCH_COOLDOWN_SECONDS = 300.0
_RECOVERY_HISTORY_LIMIT = 20
STATE_HEALTHY = "healthy"
STATE_AWAITING_LOGIN = "awaiting_login"
@dataclass(frozen=True, slots=True)
class GateVerdict:
"""One account's gate decision, recorded in the run journal on failure."""
account_id: str
mode: str
blocked: bool
message: str
recovered: bool = False
notified: bool = False
@dataclass(frozen=True, slots=True)
class CookieGateStepAction:
"""ExecutableStep that turns one account verdict into a step outcome."""
gate: "CookieGate"
account_id: str
def execute(
self,
*,
context: RunContext,
timeout_seconds: float | None,
dry_run: bool,
) -> StepExecution:
if dry_run: # pragma: no cover - the engine skips dry-run steps first
return StepExecution(exit_code=0, skipped=True, reason="dry-run")
verdict = self.gate.check(self.account_id, context=context)
if verdict.blocked:
return StepExecution(
exit_code=1,
error=verdict.message,
reason="cookie-expired",
)
return StepExecution(exit_code=0)
class CookieGate:
"""Resolve, probe, and gate the accounts one workflow depends on."""
def __init__(
self,
settings: Settings,
*,
catalog: RuntimeIntegrationCatalog | None = None,
keepalive: AccountKeepaliveManager | None = None,
environment: Mapping[str, str] | None = None,
now: Callable[[], datetime] | None = None,
sender: Callable[..., dict[str, Any]] | None = None,
) -> None: # noqa: D107 - simple wiring
self.settings = settings
self.catalog = catalog or RuntimeIntegrationCatalog.load_default(
project_root=settings.project_root,
data_root=settings.data_root,
)
self.keepalive = keepalive or AccountKeepaliveManager(
settings, catalog=self.catalog
)
self._environment = dict(os.environ if environment is None else environment)
self._now = now or (lambda: datetime.now(timezone.utc))
self._sender = sender or send_lark_bot_message
self._workflow_catalog: WorkflowCatalog | None = None
self._audit_store: Any | None = None
# ------------------------------------------------------------------
# Account resolution
# ------------------------------------------------------------------
def gate_mode(self, account_id: str) -> str:
account = self.catalog.account_for(account_id)
if str(account.login_command_id or "").strip():
return GATE_MODE_RECOVER
if account.credential_env_names or account.login_mode == "B":
return GATE_MODE_DEFER
return GATE_MODE_BLOCKING
def collect_workflow_accounts(
self,
workflow: WorkflowDefinition,
) -> dict[str, set[str]]:
"""Map each gated account to the step ids that depend on it.
Accounts resolve two ways: directly, from each step binding's
``account``; and declaratively, from the catalog workflow's
``required_accounts`` (used by orchestrator entries whose own binding
has no account while the spawned child collectors do).
"""
accounts: dict[str, set[str]] = {}
for step in workflow.steps:
account_id = self._account_for_step(step)
if account_id:
accounts.setdefault(account_id, set()).add(step.step_id)
for account_id in self._declared_required_accounts(workflow.workflow_id):
self.catalog.account_for(account_id) # unknown id: loud config error
accounts.setdefault(account_id, set()).update(
step.step_id for step in workflow.steps
)
# Defer-mode accounts keep their script-level credential login flows
# and are intentionally not gated.
return {
account_id: step_ids
for account_id, step_ids in accounts.items()
if self.gate_mode(account_id) != GATE_MODE_DEFER
}
def _account_for_step(self, step: StepDefinition) -> str | None:
action = step.action
entry = getattr(action, "entry", None)
module_id = getattr(entry, "module", "")
command_entry = getattr(action, "command_entry", None) or getattr(
entry, "entry", ""
)
if not module_id or not command_entry:
return None
try:
binding = self.catalog.binding_for(f"{module_id}:{command_entry}")
except Exception: # noqa: BLE001 - unbound entries run ungated
return None
return binding.account or None
def workflow_catalog(self) -> WorkflowCatalog:
"""Load (once) the catalog used for declared account and rerun lookups."""
if self._workflow_catalog is None:
try:
self._workflow_catalog = WorkflowCatalog.load(
self.settings.project_root / "config"
)
except Exception: # noqa: BLE001 - catalog issues surface elsewhere
self._workflow_catalog = WorkflowCatalog(
timezone="UTC", workflows=(), schedules=()
)
return self._workflow_catalog
def _declared_required_accounts(self, workflow_id: str) -> tuple[str, ...]:
for entry in self.workflow_catalog().workflows:
if entry.workflow_id == workflow_id:
return tuple(entry.required_accounts)
return ()
# ------------------------------------------------------------------
# State machine
# ------------------------------------------------------------------
def check(self, account_id: str, *, context: RunContext) -> GateVerdict:
account = self.catalog.account_for(account_id)
mode = self.gate_mode(account_id)
store = BrowserCookieStore(account.cookie_file, account.storage_state_file)
static_valid = _vault_cookie_valid(account, store)
status = _read_keepalive_status(account)
if static_valid and self._keepalive_success_is_fresh(status):
self._write_state(account, STATE_HEALTHY, probe="fresh-keepalive", context=context)
return GateVerdict(
account_id=account_id,
mode=mode,
blocked=False,
message=f"账号 {account_id} 登录态有效(keepalive 近期验证通过)",
)
if self._acceptance_enabled():
# 验收模式不开浏览器:按静态证据判定,且不投递飞书通知。
if static_valid and not _keepalive_requires_relogin(status):
return GateVerdict(
account_id=account_id,
mode=mode,
blocked=False,
message=f"账号 {account_id} 静态 cookie 校验通过(验收模式)",
)
return self._expire(
account,
mode,
context,
detail="验收模式下静态 cookie 校验未通过",
notify=False,
)
outcome = self.keepalive.refresh_account(account_id)
probe_status = str(outcome.get("status", "error"))
if probe_status == "success":
recovered = bool(outcome.get("login_recovered"))
self._write_state(
account,
STATE_HEALTHY,
probe="success",
recovered=recovered,
context=context,
)
message = (
f"账号 {account_id} 登录态有效(自动账密重登成功)"
if recovered
else f"账号 {account_id} 登录态有效(探活通过)"
)
return GateVerdict(
account_id=account_id,
mode=mode,
blocked=False,
message=message,
recovered=recovered,
)
if probe_status == "busy":
if static_valid and not _keepalive_requires_relogin(status):
return GateVerdict(
account_id=account_id,
mode=mode,
blocked=False,
message=f"账号 {account_id} 探活锁忙,静态 cookie 校验通过",
)
return self._expire(
account,
mode,
context,
detail="探活锁被占用且静态 cookie 校验未通过",
)
if probe_status in {"expired", "not_logged_in"}:
evidence = _recovery_evidence(_read_keepalive_status(account))
detail = "cookie 已过期,无法正常采集"
if probe_status == "not_logged_in" and evidence:
detail = f"登录态缺失;{evidence}"
elif evidence:
detail = f"cookie 已过期;{evidence}"
return self._expire(account, mode, context, detail=detail)
# timeout / error / http_error / disabled:探测结果不可信时回退静态证据。
if static_valid and not _keepalive_requires_relogin(status):
return GateVerdict(
account_id=account_id,
mode=mode,
blocked=False,
message=f"账号 {account_id} 探活异常({probe_status}),静态 cookie 校验通过",
)
return self._expire(
account,
mode,
context,
detail=f"探活异常({probe_status})且静态 cookie 校验未通过",
)
# ------------------------------------------------------------------
# Expiry, notification, persisted state
# ------------------------------------------------------------------
def _expire(
self,
account: Any,
mode: str,
context: RunContext,
*,
detail: str,
notify: bool | None = None,
) -> GateVerdict:
account_id = str(account.account_id)
if notify is None:
notify = not context.shadow and not self._acceptance_enabled()
state = self._read_state(account)
notified = False
notification_note = ""
if notify and not self._within_notify_interval(state):
recipients = self._alert_recipients()
message = _expiry_message(account_id, mode, context, detail)
for open_id, profile in recipients:
try:
self._sender(
user_id=open_id,
text=message,
idempotency_key=f"cookie-gate:{account_id}:{context.run_id}",
profile=profile,
)
notified = True
except Exception as exc: # noqa: BLE001 - alerting must not mask the verdict
notification_note = f"{type(exc).__name__}: {exc}"[:300]
if recipients and notified:
state["last_notified_at"] = _now_iso(self._now)
state.pop("notification_error", None)
elif notification_note:
state["notification_error"] = notification_note
state["state"] = STATE_AWAITING_LOGIN
state["updated_at"] = _now_iso(self._now)
state["last_probe_status"] = "expired"
state["last_blocked_run"] = context.run_id
if not context.shadow and not self._acceptance_enabled():
# 记录待自动重跑清单:登录态恢复后由调度器维护循环按原业务日期重跑。
state["pending_reruns"] = _record_pending_rerun(
state.get("pending_reruns"), context, _now_iso(self._now)
)
self._save_state(account, state)
try:
keepalive_status = _read_keepalive_status(account) or {}
except Exception: # noqa: BLE001 - state file is advisory
keepalive_status = {}
self._audit_cookie(
account,
"not_logged_in"
if str(keepalive_status.get("status", "")) == "not_logged_in"
else "expired",
probe_status=str(keepalive_status.get("status", "expired")),
probe_source="gate",
detail=detail,
context=context,
)
message = _expiry_message(account_id, mode, context, detail)
if notified:
message += " 已通过飞书通知登录负责人。"
elif notification_note:
message += f" 飞书通知发送失败({notification_note})。"
return GateVerdict(
account_id=account_id,
mode=mode,
blocked=True,
message=message,
notified=notified,
)
def _write_state(
self,
account: Any,
state_name: str,
*,
probe: str,
recovered: bool = False,
context: RunContext | None = None,
) -> None:
if state_name == STATE_HEALTHY:
# 健康心跳要落在每一次判定上,才能算准「最后一次可用」的时间。
self._audit_cookie(
account,
"healthy",
probe_status=probe,
probe_source="gate",
context=context,
)
state = self._read_state(account)
if (
state_name == STATE_HEALTHY
and state.get("state") == STATE_HEALTHY
and state.get("last_probe_status") in {"fresh-keepalive", "success"}
and not recovered
):
return # 持续健康时避免每次运行都重写状态文件
state["state"] = state_name
state["updated_at"] = _now_iso(self._now)
state["last_probe_status"] = probe
if state_name == STATE_HEALTHY:
state["last_healthy_at"] = _now_iso(self._now)
if recovered:
state["last_recovered_at"] = _now_iso(self._now)
state.pop("notification_error", None)
self._save_state(account, state)
def _state_path(self, account: Any) -> Path:
return account.cookie_file.with_name(_GATE_STATE_FILENAME)
def _read_state(self, account: Any) -> dict[str, object]:
path = self._state_path(account)
if not path.exists():
return {"account_id": str(account.account_id)}
try:
payload = json.loads(path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError):
return {"account_id": str(account.account_id)}
if not isinstance(payload, dict):
return {"account_id": str(account.account_id)}
return dict(payload)
def _save_state(self, account: Any, state: Mapping[str, object]) -> None:
payload = {
"account_id": str(account.account_id),
**state,
}
try:
atomic_write_json(self._state_path(account), payload)
except OSError: # pragma: no cover - state file is advisory only
pass
# ------------------------------------------------------------------
# Cookie telemetry (best-effort; never blocks or fails the gate)
# ------------------------------------------------------------------
def _audit(self) -> Any | None:
if self._audit_store is None:
try:
from gyxx_flow.ops.audit import AuditStore
self._audit_store = AuditStore(
self.settings.data_root,
environment=self._environment,
)
except Exception: # noqa: BLE001 - telemetry is optional
return None
return self._audit_store
def _audit_cookie(
self,
account: Any,
state: str,
*,
probe_status: str,
probe_source: str,
detail: str = "",
context: RunContext | None = None,
) -> None:
"""Append one cookie observation.
Stores a value-free fingerprint plus the earliest declared expiry of the
required cookies, so failure patterns (how long a login really lasts,
and whether it dies before or after the declared expiry) can be derived
from history instead of guessed. Errors are swallowed: gating must
never depend on the audit database.
"""
try:
store = self._audit()
if store is None or not store.enabled:
return
store.record_cookie_event(
account_id=str(getattr(account, "account_id", "") or ""),
state=state,
snapshot=self._cookie_snapshot(account),
probe_status=probe_status,
probe_source=probe_source,
detail=detail,
workflow_id=str(getattr(context, "workflow_id", "") or ""),
run_id=str(getattr(context, "run_id", "") or ""),
)
except Exception: # noqa: BLE001 - telemetry is advisory only
return
def _cookie_snapshot(self, account: Any) -> Any | None:
from gyxx_flow.ops.audit import snapshot_for_account
return snapshot_for_account(account)
def _within_notify_interval(self, state: Mapping[str, object]) -> bool:
raw = state.get("last_notified_at")
if not isinstance(raw, str):
return False
try:
last = datetime.fromisoformat(raw)
except ValueError:
return False
now = self._now()
if last.tzinfo is None:
last = last.replace(tzinfo=timezone.utc)
interval = _env_int(
self._environment,
GYXX_LOGIN_ALERT_NOTIFY_INTERVAL_SECONDS,
_DEFAULT_NOTIFY_INTERVAL_SECONDS,
minimum=0,
)
return (now - last).total_seconds() < interval
def _keepalive_success_is_fresh(self, status: Mapping[str, object] | None) -> bool:
if not isinstance(status, Mapping) or status.get("status") != "success":
return False
raw = status.get("last_success_at")
if not isinstance(raw, str):
return False
try:
last = datetime.fromisoformat(raw)
except ValueError:
return False
now = self._now()
if last.tzinfo is None:
last = last.replace(tzinfo=timezone.utc)
max_age = _env_float(
self._environment,
GYXX_COOKIE_GATE_MAX_STATUS_AGE_SECONDS,
_DEFAULT_MAX_STATUS_AGE_SECONDS,
minimum=_MIN_MAX_STATUS_AGE_SECONDS,
)
return (now - last).total_seconds() <= max_age
def _alert_recipients(self) -> list[tuple[str, str]]:
profile = (
self._environment.get(GYXX_LOGIN_ALERT_APP_PROFILE, "").strip()
or ANALYZER_NOTIFICATION_APP_PROFILE
)
override = [
value.strip()
for value in self._environment.get(GYXX_LOGIN_ALERT_OPEN_IDS, "").split(",")
if value.strip()
]
if override:
return [(open_id, profile) for open_id in override]
try:
payload, _revision = NotificationRoutingStore(self.settings).snapshot()
for person in payload["people"].values():
if person.get("mobile") == DEFAULT_LOGIN_ALERT_MOBILE:
binding = person.get("bindings", {}).get(profile)
if binding:
return [(str(binding["open_id"]), profile)]
except Exception: # noqa: BLE001 - fall back to the constant recipient
pass
return [(ANALYZER_ACCEPTANCE_RECIPIENT_OPEN_ID, profile)]
def _acceptance_enabled(self) -> bool:
value = self._environment.get("GYXX_WORKFLOW_ACCEPTANCE", "")
return bool(value and value.strip().casefold() in {"1", "true", "yes", "on"})
def gate_enabled(environment: Mapping[str, str] | None = None) -> bool:
values = os.environ if environment is None else environment
return values.get(GYXX_COOKIE_GATE, "").strip().casefold() not in _DISABLED_VALUES
def apply_cookie_gates(
workflow: WorkflowDefinition,
*,
settings: Settings,
environment: Mapping[str, str] | None = None,
gate: CookieGate | None = None,
) -> WorkflowDefinition:
"""Inject one critical gate step per gated account in front of its users.
Gate steps are graph sources; each step that uses an account gains the
account's gate as an extra dependency. A failed gate therefore skips its
dependents with ``dependency-failed`` through the regular engine, while
unrelated branches keep running. Operator slices (``--rerun-step`` /
``--resume-from``) are produced before this injection and stay gate-free.
"""
if not workflow.steps:
return workflow
if gate is None:
try:
gate = CookieGate(
settings,
environment=os.environ if environment is None else environment,
)
except Exception: # noqa: BLE001 - 无集成配置时不门控;步骤执行仍会显式报错
return workflow
account_steps = gate.collect_workflow_accounts(workflow)
if not account_steps:
return workflow
existing_ids = {step.step_id for step in workflow.steps}
gate_steps: list[StepDefinition] = []
for account_id in sorted(account_steps):
step_id = f"{_GATE_STEP_PREFIX}.{account_id}"
if step_id in existing_ids:
continue
gate_steps.append(
StepDefinition(
step_id,
CookieGateStepAction(gate, account_id),
timeout_seconds=_GATE_TIMEOUT_SECONDS,
max_attempts=1,
)
)
if not gate_steps:
return workflow
patched: list[StepDefinition] = []
for step in workflow.steps:
dependencies = list(dict.fromkeys(step.depends_on))
changed = False
for account_id, step_ids in account_steps.items():
gate_id = f"{_GATE_STEP_PREFIX}.{account_id}"
if step.step_id in step_ids and gate_id not in dependencies:
dependencies.append(gate_id)
changed = True
if changed:
step = replace(step, depends_on=tuple(dependencies))
patched.append(step)
return WorkflowDefinition(
workflow.workflow_id,
(*gate_steps, *patched),
)
class CookieGateRecoveryManager:
"""Re-run workflows blocked by expired cookies once the login recovers.
Runs inside the resident scheduler's maintenance loop. For every account
with a non-empty ``pending_reruns`` list, it re-verifies the login state
(fresh keepalive success first, then a throttled live probe); on recovery
it relaunches each recorded workflow with its original business date
through the scheduler's own launcher, clears the pending list, and sends a
Feishu confirmation to the login owner.
"""
def __init__(
self,
gate: CookieGate,
*,
launcher: Any,
probe_interval_seconds: float = _RECOVERY_PROBE_INTERVAL_SECONDS,
launch_cooldown_seconds: float = _RECOVERY_LAUNCH_COOLDOWN_SECONDS,
) -> None:
self.gate = gate
self._launcher = launcher
self._probe_interval = probe_interval_seconds
self._launch_cooldown = launch_cooldown_seconds
def tick(self) -> list[dict[str, object]]:
"""Attempt recovery reruns once; returns redacted outcomes per account."""
outcomes: list[dict[str, object]] = []
for account in self.gate.catalog.accounts:
state = self.gate._read_state(account)
pending = state.get("pending_reruns")
if not isinstance(pending, list) or not pending:
continue
now = self.gate._now()
if not self._launch_cooldown_passed(state, now):
continue
if not self._account_recovered(account, state, now):
continue
launched, remaining = self._launch_pending(account, pending, now)
state["pending_reruns"] = remaining
state["updated_at"] = _now_iso(self.gate._now)
state["last_recovered_at"] = state["updated_at"]
if not remaining:
state["state"] = STATE_HEALTHY
state["last_probe_status"] = "recovered-rerun"
if launched:
state["last_rerun_launch_at"] = state["updated_at"]
history = [
item
for item in state.get("rerun_history", [])
if isinstance(item, dict)
]
history.extend(launched)
state["rerun_history"] = history[-_RECOVERY_HISTORY_LIMIT:]
self.gate._save_state(account, state)
if launched:
self._notify_recovered(str(account.account_id), launched)
outcomes.append(
{
"account_id": str(account.account_id),
"launched": launched,
"pending": len(remaining),
}
)
return outcomes
# ------------------------------------------------------------------
# internals
# ------------------------------------------------------------------
def _launch_cooldown_passed(
self, state: Mapping[str, object], now: datetime
) -> bool:
raw = state.get("last_rerun_launch_at")
if not isinstance(raw, str):
return True
try:
last = datetime.fromisoformat(raw)
except ValueError:
return True
if last.tzinfo is None:
last = last.replace(tzinfo=timezone.utc)
return (now - last).total_seconds() >= self._launch_cooldown
def _account_recovered(
self,
account: Any,
state: dict[str, object],
now: datetime,
) -> bool:
"""Fresh keepalive success first; otherwise one throttled live probe."""
store = BrowserCookieStore(account.cookie_file, account.storage_state_file)
if _vault_cookie_valid(account, store) and self.gate._keepalive_success_is_fresh(
_read_keepalive_status(account)
):
return True
raw = state.get("last_recovery_probe_at")
if isinstance(raw, str):
try:
last_probe = datetime.fromisoformat(raw)
except ValueError:
last_probe = None
if last_probe is not None:
if last_probe.tzinfo is None:
last_probe = last_probe.replace(tzinfo=timezone.utc)
if (now - last_probe).total_seconds() < self._probe_interval:
return False
state["last_recovery_probe_at"] = _now_iso(self.gate._now)
self.gate._save_state(account, state)
try:
outcome = self.gate.keepalive.refresh_account(str(account.account_id))
except Exception: # noqa: BLE001 - 恢复探活失败留给下一轮
return False
return str(outcome.get("status", "error")) == "success"
def _launch_pending(
self,
account: Any,
pending: list[object],
now: datetime,
) -> tuple[list[dict[str, object]], list[dict[str, object]]]:
known_ids = {
entry.workflow_id for entry in self.gate.workflow_catalog().workflows
}
launched: list[dict[str, object]] = []
remaining: list[dict[str, object]] = []
for item in pending:
if not isinstance(item, dict):
continue # 控制台脚本等临时运行 id:不在 catalog 中,直接丢弃
workflow_id = item.get("workflow_id")
business_date = item.get("business_date")
if not isinstance(workflow_id, str) or not isinstance(business_date, str):
continue
if workflow_id not in known_ids:
continue
try:
process = self._launcher.launch(
workflow_id, now, date.fromisoformat(business_date)
)
except Exception as exc: # noqa: BLE001 - 保留 pending,下一轮重试
remaining.append(
{
"workflow_id": workflow_id,
"business_date": business_date,
"error": f"{type(exc).__name__}: {exc}"[:200],
}
)
continue
close = getattr(self._launcher, "close", None)
if callable(close):
try:
close(process)
except Exception: # noqa: BLE001 - 关闭日志句柄失败不影响重跑
pass
launched.append(
{
"workflow_id": workflow_id,
"business_date": business_date,
"pid": int(getattr(process, "pid", 0) or 0),
"launched_at": _now_iso(self.gate._now),
}
)
return launched, remaining
def _notify_recovered(
self,
account_id: str,
launched: list[dict[str, object]],
) -> None:
lines = [
f"- {item['workflow_id']}(业务日期 {item['business_date']})"
for item in launched
]
message = (
"【GYXX 登录态已恢复】账号 "
f"{account_id} 登录态已恢复,已自动重跑被阻断的工作流:\n"
+ "\n".join(lines)
+ "\n如本次重跑仍有失败,请查看运行日志或工作流控制台。"
)
key = "-".join(
[account_id, *(str(item["workflow_id"]) for item in launched)]
)
for open_id, profile in self.gate._alert_recipients():
try:
self.gate._sender(
user_id=open_id,
text=message,
idempotency_key=f"cookie-recovery:{key}",
profile=profile,
)
except Exception: # noqa: BLE001 - 确认通知失败不影响已发起的重跑
continue
def _record_pending_rerun(
current: object,
context: RunContext,
blocked_at: str,
) -> list[dict[str, object]]:
"""Append one blocked run to the pending list, deduped and bounded."""
pending: list[dict[str, object]] = [
dict(item)
for item in (current if isinstance(current, list) else [])
if isinstance(item, dict)
]
entry = {
"workflow_id": context.workflow_id,
"business_date": context.business_date.isoformat(),
"blocked_run_id": context.run_id,
"blocked_at": blocked_at,
}
pending = [
item
for item in pending
if not (
item.get("workflow_id") == entry["workflow_id"]
and item.get("business_date") == entry["business_date"]
)
]
pending.append(entry)
return pending[-_MAX_PENDING_RERUNS:]
def _recovery_evidence(status: Mapping[str, object] | None) -> str:
if not isinstance(status, Mapping):
return ""
if not status.get("login_attempted"):
return ""
reason = status.get("login_reason")
error_type = status.get("login_error_type")
exit_code = status.get("login_exit_code")
parts = ["自动账密重登已尝试但未成功"]
if isinstance(reason, str) and reason:
parts.append(reason)
if isinstance(error_type, str) and error_type:
parts.append(f"错误={error_type}")
if isinstance(exit_code, int) and not isinstance(exit_code, bool):
parts.append(f"退出码={exit_code}")
return "(" + ",".join(parts) + ")"
def _expiry_message(
account_id: str,
mode: str,
context: RunContext,
detail: str,
) -> str:
mode_note = (
"账号密码自动重登失败,需人工处理(如验证码)"
if mode == GATE_MODE_RECOVER
else "该平台仅支持扫码/验证码登录,无法自动恢复"
)
return (
"【GYXX 登录态失效】工作流被阻断,需要人工重新登录\n"
f"工作流: {context.workflow_id}(业务日期 {context.business_date.isoformat()})\n"
f"账号: {account_id}\n"
f"原因: {detail};{mode_note}\n"
f"请在服务器上执行: uv run gyxx accounts login {account_id}\n"
"(将打开浏览器窗口,完成扫码或验证码登录后自动保存并同步到全部成员脚本;"
"受影响的采集步骤本次已跳过,登录完成后下次运行自动恢复)"
)
def _now_iso(now: Callable[[], datetime]) -> str:
return now().astimezone(timezone.utc).isoformat(timespec="seconds")
def _env_int(
environment: Mapping[str, str],
name: str,
default: int,
*,
minimum: int,
) -> int:
raw = environment.get(name, "").strip()
try:
value = int(raw)
except (TypeError, ValueError):
return default
return max(minimum, value)
def _env_float(
environment: Mapping[str, str],
name: str,
default: float,
*,
minimum: float,
) -> float:
raw = environment.get(name, "").strip()
try:
value = float(raw)
except (TypeError, ValueError):
return default
return max(minimum, value)
__all__ = [
"DEFAULT_LOGIN_ALERT_MOBILE",
"GATE_MODE_BLOCKING",
"GATE_MODE_DEFER",
"GATE_MODE_RECOVER",
"GYXX_COOKIE_GATE",
"GYXX_LOGIN_ALERT_OPEN_IDS",
"CookieGate",
"CookieGateRecoveryManager",
"CookieGateStepAction",
"GateVerdict",
"apply_cookie_gates",
"gate_enabled",
]
@@ -1,4 +1,4 @@
"""Content metrics, creator, comment, report, and login workflows."""
"""Content metrics, creator, comment, and report workflows."""
from __future__ import annotations
@@ -14,7 +14,6 @@ CONTENT_SCHEDULED_WORKFLOW_IDS = (
"content.metrics.self_operated.daily",
"content.metrics.backfill",
"content.marketing_report.daily",
"content.relogin.weekly",
"content.creator_report.monthly",
"content.summary.monthly",
"content.notes_master.daily",
@@ -28,7 +27,6 @@ CONTENT_COMMENT_TIMEOUT_SECONDS = 6 * 60 * 60
# weakening the module-wide write lock.
CONTENT_NOTES_MASTER_LOCK_TIMEOUT_SECONDS = 6 * 60 * 60
CONTENT_RESOURCE = "module:content_marketing"
_RELOGIN_WORKFLOW_ID = "content.relogin.weekly"
_PARALLEL_COMMENT_WORKFLOW_ID = "content.comments.weekly"
# Keep both long-running daily metrics graphs on workflow-scoped resources so
# their collection steps do not block unrelated content workflows.
@@ -94,7 +92,6 @@ class ContentMarketingModule:
),
resource=workflow_resource,
command_factory=command_factory,
official_notification=workflow_id == _RELOGIN_WORKFLOW_ID,
independent_step_resources=(
workflow_id in {
_PARALLEL_COMMENT_WORKFLOW_ID,
@@ -77,23 +77,15 @@ def is_logged_in(page: Page) -> bool:
def maybe_wait_for_login(page: Page, login_timeout: int) -> None:
"""检测登录态,未登录就等用户扫码。B 站登录页 URL 含 passport.bilibili.com/login。"""
"""检测登录态;未登录立即降级为游客态,不再等待人工扫码。
登录统一由账号 vault 管理(gyxx accounts login bilibili-content),
由工作流执行前的 cookie 网关保证登录态。
"""
if is_logged_in(page):
return
log("B 站未登录 (无 SESSDATA cookie)。请在弹出的浏览器窗口扫码登录。")
# 跳到登录页,确保用户能看到二维码
try:
page.goto("https://passport.bilibili.com/login", wait_until="domcontentloaded", timeout=30000)
except Exception as exc:
log(f" 跳登录页失败 (继续等当前页扫码): {exc}")
deadline = time.time() + login_timeout
while time.time() < deadline:
if is_logged_in(page):
log("登录成功,SESSDATA 已写入")
save_state(page)
return
page.wait_for_timeout(2000)
log(f" [WARN] 等待 {login_timeout}s 仍未登录,继续以游客态采集 (评论只会拿到热门 3-5 条)")
log(" [WARN] B 站未登录 (无 SESSDATA cookie),降级为游客态采集 (评论只会拿到热门 3-5 条)。"
"如需全量评论请执行 gyxx accounts login bilibili-content")
def safe_filename(value: str) -> str:
@@ -437,48 +429,17 @@ def write_outputs(url: str, comments: list[dict[str, Any]], result: dict[str, An
return json_path, csv_path
def login_only(login_timeout: int) -> int:
"""只触发登录(headed),登录完成立刻退出。给 relogin_bilibili.py 用。"""
def action(page: Page) -> None:
page.goto("https://passport.bilibili.com/login", wait_until="domcontentloaded", timeout=30000)
page.wait_for_timeout(1500)
maybe_wait_for_login(page, login_timeout)
PROFILE_DIR.mkdir(parents=True, exist_ok=True)
with DynamicSession(
headless=False,
real_chrome=True,
user_data_dir=str(PROFILE_DIR),
locale="zh-CN",
timezone_id="Asia/Shanghai",
timeout=90000,
network_idle=False,
disable_resources=False,
google_search=False,
page_setup=restore_cookies,
max_pages=1,
) as session:
session.fetch("https://passport.bilibili.com/login", page_action=action, wait=1000)
log("[login-only] 完成")
return 0
def main() -> int:
parser = argparse.ArgumentParser(description="Scrape Bilibili video comments with a real browser and API.")
parser.add_argument("url", help="Bilibili video URL (传 dummy 'login' 配合 --login-only 也可)")
parser.add_argument("url", help="Bilibili video URL")
parser.add_argument("--max-reply-pages", type=int, default=200, help="max sub-reply pages per top-level comment")
parser.add_argument("--max-pages", type=int, default=200, help="最多抓取一级评论页数")
parser.add_argument("--delay", type=float, default=0.25, help="接口请求间隔秒数")
parser.add_argument("--headless", action="store_true", help="无头模式")
parser.add_argument("--login-timeout", type=int, default=0,
help="未登录时等待扫码秒数(>0 启用登录检测;默认 0 沿用游客态)")
parser.add_argument("--login-only", action="store_true",
help="只触发登录(headed)然后退出,给 relogin_bilibili 用")
help="保留参数(登录等待已移除;未登录将直接以游客态采集)")
args = parser.parse_args()
if args.login_only:
return login_only(args.login_timeout if args.login_timeout > 0 else 300)
comments, result = scrape_comments(
args.url, args.max_pages, args.delay, args.headless,
args.max_reply_pages, args.login_timeout,
@@ -16,6 +16,7 @@ import time
from datetime import datetime
from pathlib import Path
from gyxx_flow.adapters.scrapling import is_browser_closed_error
from gyxx_flow.core.exit_codes import COOKIE_SKIP_EXIT_CODE
from gyxx_flow.modules.content_marketing import chanmama_export
from gyxx_flow.modules.content_marketing.runtime_paths import PATHS
@@ -52,6 +53,11 @@ REFRESH_WAIT_SECONDS = int(os.getenv("CHANMAMA_REFRESH_WAIT_SECONDS", "600"))
CHANMAMA_LIST_READY_TIMEOUT_SECONDS = int(
os.getenv("CHANMAMA_LIST_READY_TIMEOUT_SECONDS", "45")
)
# 公告/活动浮层可能盖住视频列表导致渲染判定一直不满足;渲染等待期间按
# 该间隔无交互清理浮层(派发 ESC + 移除 fixed/absolute 浮层节点)。
CHANMAMA_OVERLAY_DISMISS_INTERVAL_SECONDS = float(
os.getenv("CHANMAMA_OVERLAY_DISMISS_INTERVAL_SECONDS", "10")
)
# 蝉妈妈导出接口实测可能在 15 分钟边界后才把 Excel 写入下载目录;把轮询
# 上限做成可配置,并默认留出 20 分钟,避免文件已生成却被脚本判定为失败。
CHANMAMA_EXPORT_WAIT_SECONDS = int(
@@ -99,16 +105,20 @@ def create_driver():
profile_dir = str(PATHS.browser_profile_dir)
os.makedirs(profile_dir, exist_ok=True)
cdp_url = os.getenv("GYXX_BROWSER_CDP_URL") or None
# 蝉妈妈是国内站且其 WAF 会拉黑代理出口(Clash 节点 TLS 直接 reset,
# 导出 API 全断、确认弹窗永远不弹);强制浏览器直连。
direct_flags = ["--no-proxy-server"]
# StealthySession(隐藏指纹补丁)下蝉妈妈前端会静默压制功能: 页面能开
# 但导出确认弹窗不渲染(只剩空壳遮罩)。真 Chrome + 非 stealthy + 直连
# 已实测弹窗完整渲染(含取消/导出按钮),不再使用 stealthy。
try:
return ScraplingDriver.start(
profile_dir=profile_dir,
download_dir=DOWNLOAD_DIR,
cdp_url=cdp_url,
stealthy=True,
# 蝉妈妈新版 Vue 页面与 StealthySession + 系统 Chrome 组合
# 不兼容:HTML 能返回,但 #app 不会挂载。使用 Scrapling
# 自带 Chromium 仍保留隐藏指纹会话,且不影响已连接的 CDP。
**({"real_chrome": False} if cdp_url is None else {}),
stealthy=False,
real_chrome=True,
extra_flags=direct_flags,
)
except Exception as exc:
if (
@@ -125,23 +135,25 @@ def create_driver():
profile_dir=profile_dir,
download_dir=DOWNLOAD_DIR,
cdp_url=None,
stealthy=True,
real_chrome=False,
stealthy=False,
real_chrome=True,
extra_flags=direct_flags,
)
def save_cookies(driver):
"""保存 cookie 到文件"""
"""保存 cookie 到文件。
写裸列表格式,与账号 vault 同步(sync_binding_state)和适配器
BrowserCookieStore 的绑定 cookie 文件格式保持一致 —— 同一文件
被三方读写,字典格式会让适配器下次启动/重建时直接拒绝加载。
"""
try:
cookies = driver.get_cookies()
data = {
"cookies": cookies,
"timestamp": time.time(),
}
cookie_path = Path(COOKIE_FILE)
cookie_path.parent.mkdir(parents=True, exist_ok=True)
temp_path = cookie_path.with_name(f".{cookie_path.name}.tmp")
temp_path.write_text(json.dumps(data, ensure_ascii=False, indent=2), encoding="utf-8")
temp_path.write_text(json.dumps(cookies, ensure_ascii=False, indent=2), encoding="utf-8")
os.replace(temp_path, cookie_path)
print(f"💾 已保存 cookie ({len(cookies)} 条) 到: {COOKIE_FILE}")
return True
@@ -153,6 +165,9 @@ def save_cookies(driver):
def load_cookies(driver):
"""从文件加载 cookie 到浏览器
兼容两种格式: 裸列表(vault/适配器格式,按文件 mtime 判断时效)与
旧字典格式({"cookies": [...], "timestamp": ...})。
Returns:
bool: cookie 是否有效(有效期内)
"""
@@ -161,16 +176,20 @@ def load_cookies(driver):
return False
try:
data = json.loads(Path(COOKIE_FILE).read_text(encoding="utf-8"))
payload = json.loads(Path(COOKIE_FILE).read_text(encoding="utf-8"))
if isinstance(payload, list):
cookies = payload
timestamp = os.path.getmtime(COOKIE_FILE)
else:
cookies = payload.get("cookies", [])
timestamp = payload.get("timestamp", 0)
timestamp = data.get("timestamp", 0)
age = time.time() - timestamp
if age > COOKIE_MAX_AGE:
print(f"⏰ cookie 已过期 ({int(age // 86400)} 天前),需要重新登录")
return False
cookies = data.get("cookies", [])
if not cookies:
print("📭 cookie 为空,需要重新登录")
return False
@@ -297,7 +316,10 @@ def handle_captcha(driver) -> bool:
def verify_login(driver):
"""验证是否已登录
通过访问目标页面,检查是否出现登录相关的元素
通过访问目标页面,检查是否出现登录相关的元素。
cookie 失效时蝉妈妈会把目标页重定向到 login.html,但重定向常在
首屏渲染后数秒才发生——等待 URL 稳定后再判定,避免误判已登录
(误判会跳过重新登录,导致后续导航一直超时)。
"""
try:
# 跳转到目标页面
@@ -305,10 +327,12 @@ def verify_login(driver):
WebDriverWait(driver, 12).until(
EC.presence_of_element_located((By.TAG_NAME, "body"))
)
time.sleep(3)
# cookie 失效时的重定向在首屏渲染后 3~20 秒才发生,固定等满 10 秒
# 再判定,不能因 URL 短暂未变而提前判为已登录
time.sleep(10)
cur_lower = driver.current_url.lower()
# 检查是否跳转回登录页 / 注册页(cookie 失效时蝉妈妈会重定向到 login.html 或 register.html)
cur_lower = driver.current_url.lower()
if "login" in cur_lower or "register" in cur_lower:
print(f"🔒 检测到被重定向到登录/注册页 ({driver.current_url}),cookie 已失效")
return False
@@ -458,6 +482,180 @@ def login(driver):
return False
def _dismiss_overlays(driver) -> int:
"""无交互清理遮挡浮层:派发 ESC 并移除 fixed/absolute 定位的弹层。"""
try:
removed = int(driver.execute_script(_OVERLAY_DISMISS_SCRIPT) or 0)
except Exception as error:
print(f"⚠️ 浮层清理脚本执行失败(忽略): {error}")
return 0
if removed:
print(f"🧹 已清理 {removed} 个遮挡浮层")
return removed
def _dismiss_export_stage_overlays(driver) -> int:
"""导出阶段清理弹窗(ESC + 移除弹层节点)。
只在弹窗不可确认时调用;返回移除的弹窗数。
"""
try:
removed = int(driver.execute_script(_EXPORT_STAGE_DISMISS_SCRIPT) or 0)
except Exception as error:
print(f"⚠️ 导出阶段弹窗清理脚本执行失败(忽略): {error}")
return 0
if removed > 0:
print(f"🧹 导出阶段清理 {removed} 个遮挡弹窗")
return removed
def _export_dialog_state(driver) -> str:
"""探测导出阶段弹窗状态: confirmable / present / none。"""
try:
state = driver.execute_script(_EXPORT_DIALOG_STATE_SCRIPT)
except Exception:
return "none"
return state if state in {"confirmable", "present", "none"} else "none"
def _log_export_dialog_texts(driver) -> None:
"""把当前可见弹窗的类名与文字打进日志,用于定位弹窗身份。"""
try:
texts = driver.execute_script(_EXPORT_DIALOG_TEXT_SCRIPT) or []
except Exception:
return
for i, text in enumerate(texts[:4], 1):
print(f"🪟 弹窗{i}内容: {text}")
def _click_export_confirm(driver) -> bool:
"""点击导出确认弹窗中的【导出】按钮,成功返回 True。"""
# 关键:用 contains(.,'导出') 匹配后代文本("导出"在 <span> 内)
confirm_selectors = [
# 精确:主按钮(primary) + 含"导出"文字
"//div[contains(@class,'el-dialog') or @role='dialog']//button[contains(.,'导出')]",
# 弹窗内任意含"导出"的按钮
"//div[contains(@class,'el-dialog') or @role='dialog']//button[contains(@class,'el-button--primary')][contains(.,'导出')]",
# 兼容"确定"/"确认"
"//div[contains(@class,'el-dialog') or @role='dialog']//button[contains(.,'确定')]",
"//div[contains(@class,'el-dialog') or @role='dialog']//button[contains(.,'确认')]",
]
for sel in confirm_selectors:
try:
btn = driver.find_element(By.XPATH, sel)
if btn.is_displayed():
# 原生点击优先(可触发内层元素的 Vue @click),失败再退 JS 合成点击
try:
btn.click()
except Exception:
driver.execute_script("arguments[0].click();", btn)
print(f"✅ 已在弹窗中点击导出按钮: {sel}")
return True
except Exception:
continue
return False
def _reclick_export_control(driver) -> bool:
"""重新触发一次导出按钮(第一次点击可能已被弹窗吞掉)。
优先 Playwright 原生点击(在元素命中点派发可信鼠标事件,能触发绑定
在按钮内层 span/div 上的 Vue @click);JS 合成 click 的目标是按钮自身,
内层监听器不会触发——蝉妈妈改版后导出点击无声失效就是这个原因。
"""
if _native_click_export_button(driver):
return True
try:
return bool(driver.execute_script(_CLICK_EXPORT_CONTROL_SCRIPT))
except Exception:
return False
_EXPORT_BUTTON_XPATHS = [
"//button[contains(.,'导出数据')]",
"//button[contains(.,'批量导出')]",
"//button[contains(.,'导出视频')]",
"//*[contains(@class,'export') and (self::button or self::a or @role='button')]",
]
def _native_click_export_button(driver) -> bool:
"""用原生可信点击触发导出按钮,成功返回 True。"""
for xp in _EXPORT_BUTTON_XPATHS:
try:
elements = driver.find_elements(By.XPATH, xp)
except Exception:
continue
for el in elements:
try:
if el.is_displayed():
el.click()
return True
except Exception:
continue
return False
def _wait_video_list_ready(driver, timeout_seconds: float) -> None:
"""等待视频列表渲染就绪,期间周期性清理遮挡浮层。
语义与 WebDriverWait.until 保持一致:浏览器侧的暂时性异常只记录
不中断,到达 deadline 后抛 TimeoutException。
"""
timeout = max(0.0, float(timeout_seconds))
deadline = time.monotonic() + timeout
next_dismiss_at = 0.0
last_error: Exception | None = None
while True:
try:
if driver.execute_script(_VIDEO_LIST_READY_SCRIPT):
return
except Exception as error:
last_error = error
now = time.monotonic()
if now >= deadline:
detail = f": {last_error}" if last_error else ""
raise TimeoutException(
f"condition was not met within {timeout}s{detail}"
) from last_error
if now >= next_dismiss_at:
_dismiss_overlays(driver)
next_dismiss_at = now + CHANMAMA_OVERLAY_DISMISS_INTERVAL_SECONDS
time.sleep(0.5)
def _dump_render_failure(driver, url: str) -> None:
"""渲染失败时落截图与页面 HTML,供事后定位是浮层/验证码还是改版。"""
try:
stamp = datetime.now().strftime("%Y%m%d_%H%M%S")
diag_dir = Path(DOWNLOAD_DIR) / "_diag"
diag_dir.mkdir(parents=True, exist_ok=True)
screenshot = diag_dir / f"render_failure_{stamp}.png"
driver.save_screenshot(str(screenshot))
(diag_dir / f"render_failure_{stamp}.html").write_text(
driver.page_source, encoding="utf-8"
)
print(f"🩺 渲染失败诊断已保存: {screenshot}")
except Exception as error:
print(f"⚠️ 渲染失败诊断保存失败: {error}")
def _dump_export_failure(driver) -> None:
"""导出失败时落截图与页面 HTML,供事后定位弹窗/按钮的实际形态。"""
try:
stamp = datetime.now().strftime("%Y%m%d_%H%M%S")
diag_dir = Path(DOWNLOAD_DIR) / "_diag"
diag_dir.mkdir(parents=True, exist_ok=True)
screenshot = diag_dir / f"export_failure_{stamp}.png"
driver.save_screenshot(str(screenshot))
(diag_dir / f"export_failure_{stamp}.html").write_text(
driver.page_source, encoding="utf-8"
)
print(f"🩺 导出失败诊断已保存: {screenshot}")
except Exception as error:
print(f"⚠️ 导出失败诊断保存失败: {error}")
def navigate_to_target(driver, url):
"""跳转到目标博主视频页面"""
last_error = None
@@ -475,12 +673,11 @@ def navigate_to_target(driver, url):
)
time.sleep(3) # 等待页面数据加载
# 截图中的页面虽然 URL 正确,但正文尚未渲染;在交给导出逻辑
# 前确认虚拟视频列表已经出现,空白页则有限重载一次。
WebDriverWait(driver, CHANMAMA_LIST_READY_TIMEOUT_SECONDS).until(
lambda current: bool(
current.execute_script(_VIDEO_LIST_READY_SCRIPT)
)
)
# 前确认虚拟视频列表已经出现,空白页则有限重载一次。浮层多在
# 首屏挂载后才弹出:进等待前先清一轮,等待期间按间隔再清,
# 避免延迟弹出的公告/活动浮层永久挡住渲染判定。
_dismiss_overlays(driver)
_wait_video_list_ready(driver, CHANMAMA_LIST_READY_TIMEOUT_SECONDS)
print(f"✅ 已到达并完成渲染目标页面: {current_url}")
return True
except Exception as exc:
@@ -499,6 +696,7 @@ def navigate_to_target(driver, url):
except Exception as reset_exc:
print(f"⚠️ 蝉妈妈页面重置失败,继续重试目标页: {reset_exc}")
time.sleep(1)
_dump_render_failure(driver, url)
raise RuntimeError(f"蝉妈妈目标页加载失败: {url}; {last_error}") from last_error
@@ -539,6 +737,206 @@ def _find_video_table(driver):
return WebDriverWait(driver, 15).until(locate)
# 蝉妈妈公告/活动弹窗以 fixed/absolute 浮层盖住页面,且类名不稳定、未必
# 有关闭按钮。这里不点叉:派发 ESC 键事件(多数组件库监听 document 关闭
# 弹窗),再移除定位浮层节点并解除其对 html/body 的滚动锁;已知类名未
# 命中时兜底移除 z-index 高且覆盖大半屏的 fixed 层。列表区
# (.aweme-record-wrapper/.list-box/.list-auth-wrap-block) 内的元素永不命中。
# .v-modal 是 ElementUI 直接挂在 <body> 下的灰色遮罩,与弹窗本体分离,
# 必须显式清理,否则弹窗删掉后页面仍然灰屏、无法滚动;滚动锁 class
# (el-popup-parent--hidden 等)同理。
# 关键约束:只移除"可见"的浮层。页面常驻 90+ 个 display:none 的隐藏弹窗
# 模板(含导出确认框的模板)——删掉它们后 Vue 只会在游离节点上"打开"弹窗,
# 页面上永远渲染不出来(导出只剩灰遮罩),因此 display:none/零尺寸节点
# 一律跳过。
_OVERLAY_DISMISS_SCRIPT = r"""
() => {
const esc = new KeyboardEvent('keydown', {
key: 'Escape', code: 'Escape', keyCode: 27, which: 27, bubbles: true,
});
document.dispatchEvent(esc);
window.dispatchEvent(esc);
let removed = 0;
const seen = new Set();
const isOverlay = (el) => {
const style = window.getComputedStyle(el);
return style.position === 'fixed' || style.position === 'absolute';
};
const drop = (el) => {
if (seen.has(el) || !isOverlay(el)) return;
const style = window.getComputedStyle(el);
if (style.display === 'none') return;
const rect = el.getBoundingClientRect();
if (rect.width === 0 || rect.height === 0) return;
if (el.closest('.aweme-record-wrapper,.list-box,.list-auth-wrap-block')) return;
seen.add(el);
el.remove();
removed += 1;
};
const selectors = [
'.el-dialog__wrapper', '.el-overlay', '.el-message-box__wrapper',
'.ant-modal-mask', '.ant-modal-wrap', '.modal-mask', '.modal-dialog',
'.v-modal', '.v-modal-inner',
'[class*="popup"]', '[class*="dialog"]', '[class*="overlay"]', '[class*="mask"]',
];
for (const selector of selectors) {
document.querySelectorAll(selector).forEach(drop);
}
if (removed === 0) {
document.querySelectorAll('body *').forEach((el) => {
const style = window.getComputedStyle(el);
const zIndex = Number.parseInt(style.zIndex, 10);
const rect = el.getBoundingClientRect();
if (Number.isFinite(zIndex) && zIndex > 500
&& rect.height > window.innerHeight * 0.5) {
drop(el);
}
});
}
document.body.classList.remove(
'el-popup-parent--hidden', 'el-popup-parent--overflow-hidden', 'overflow-hidden'
);
document.documentElement.classList.remove(
'el-popup-parent--hidden', 'el-popup-parent--overflow-hidden', 'overflow-hidden'
);
document.documentElement.style.overflow = '';
document.body.style.overflow = '';
document.body.style.paddingRight = '';
return removed;
}
"""
# 导出阶段专用清理:与 _OVERLAY_DISMISS_SCRIPT 同一套清理思路,但
# 调用时机由 _export_dialog_state 决定——只在弹窗"没有可点的确认按钮"时
# 才执行(确认框可点时直接点导出,绝不清)。三件事:
# 1) 先 ESC + 点弹窗自带的关闭(×)按钮,走框架的正常关闭路径;
# 2) 再移除残留的弹层节点——必须包含 .v-modal:ElementUI 的灰色遮罩直接
# 挂在 <body> 下、与弹窗本体分离,漏掉它就是"删了弹窗页面仍灰屏";
# 3) 最后解除滚动锁:清掉 body/html 上的 el-popup-parent--hidden 等
# 框架加的 overflow 锁 class 与内联样式,否则页面删完遮罩也不能滚动。
_EXPORT_STAGE_DISMISS_SCRIPT = r"""
() => {
const visible = (el) => {
const style = window.getComputedStyle(el);
const rect = el.getBoundingClientRect();
return style.display !== 'none' && style.visibility !== 'hidden'
&& style.opacity !== '0' && rect.width > 0 && rect.height > 0;
};
const esc = new KeyboardEvent('keydown', {
key: 'Escape', code: 'Escape', keyCode: 27, which: 27, bubbles: true,
});
document.dispatchEvent(esc);
window.dispatchEvent(esc);
// 先走正常关闭:点弹窗自己的 × / 关闭按钮
let closed = 0;
document.querySelectorAll(
'.el-dialog__wrapper [class*="close"],.el-message-box__wrapper [class*="close"],'
+ '.ant-modal-wrap [class*="close"],[class*="dialog"] [class*="close"],'
+ '[class*="modal"] [class*="close"],[class*="popup"] [class*="close"]'
).forEach((el) => {
if (visible(el)) { el.click(); closed += 1; }
});
let removed = 0;
const seen = new Set();
const drop = (el) => {
if (seen.has(el)) return;
const style = window.getComputedStyle(el);
if (style.position !== 'fixed' && style.position !== 'absolute') return;
if (style.display === 'none') return;
const rect = el.getBoundingClientRect();
if (rect.width === 0 || rect.height === 0) return;
if (el.closest('.aweme-record-wrapper,.list-box,.list-auth-wrap-block')) return;
seen.add(el);
el.remove();
removed += 1;
};
const selectors = [
'.el-dialog__wrapper', '.el-overlay', '.el-message-box__wrapper',
'.ant-modal-mask', '.ant-modal-wrap', '.modal-mask', '.modal-dialog',
'.v-modal', '.v-modal-inner',
'[class*="popup"]', '[class*="dialog"]', '[class*="overlay"]', '[class*="mask"]',
];
for (const selector of selectors) {
document.querySelectorAll(selector).forEach(drop);
}
if (removed === 0) {
document.querySelectorAll('body *').forEach((el) => {
const style = window.getComputedStyle(el);
const zIndex = Number.parseInt(style.zIndex, 10);
const rect = el.getBoundingClientRect();
if (Number.isFinite(zIndex) && zIndex > 500
&& rect.height > window.innerHeight * 0.5) {
drop(el);
}
});
}
// 解除滚动锁:框架加的锁 class + 内联 overflow 一起清
document.body.classList.remove(
'el-popup-parent--hidden', 'el-popup-parent--overflow-hidden', 'overflow-hidden'
);
document.documentElement.classList.remove(
'el-popup-parent--hidden', 'el-popup-parent--overflow-hidden', 'overflow-hidden'
);
document.documentElement.style.overflow = '';
document.body.style.overflow = '';
document.body.style.paddingRight = '';
return removed + closed;
}
"""
# 导出阶段弹窗状态机:'confirmable'=有可见弹窗且弹窗里有可点的
# 导出/确定/确认按钮(直接点,不能清);'present'=弹窗可见但没有可点
# 的确认按钮(空壳遮罩/公告,ESC 关不掉就强删);'none'=没有弹窗。
_EXPORT_DIALOG_STATE_SCRIPT = r"""
() => {
const visible = (el) => {
const style = window.getComputedStyle(el);
const rect = el.getBoundingClientRect();
return style.display !== 'none' && style.visibility !== 'hidden'
&& style.opacity !== '0' && rect.width > 0 && rect.height > 0;
};
const roots = [...document.querySelectorAll(
'.el-dialog__wrapper,.el-message-box__wrapper,.el-overlay,'
+ '.ant-modal-wrap,.ant-modal-mask,[role="dialog"],'
+ '[class*="dialog"],[class*="modal"],[class*="popup"]'
)].filter(visible);
for (const root of roots) {
const confirmable = [...root.querySelectorAll(
'button,a,[role="button"],[class*="confirm"],[class*="primary"]'
)].some((el) => {
if (!visible(el)) return false;
const text = (el.innerText || el.value || '').replace(/\s+/g, '');
return /导出|确定|确认/.test(text);
});
if (confirmable) return 'confirmable';
}
return roots.length ? 'present' : 'none';
}
"""
# 抓取当前可见弹窗的文字内容(每窗截前 160 字),写进日志用于定位弹窗身份。
_EXPORT_DIALOG_TEXT_SCRIPT = r"""
() => {
const visible = (el) => {
const style = window.getComputedStyle(el);
const rect = el.getBoundingClientRect();
return style.display !== 'none' && style.visibility !== 'hidden'
&& style.opacity !== '0' && rect.width > 0 && rect.height > 0;
};
return [...document.querySelectorAll(
'.el-dialog__wrapper,.el-message-box__wrapper,.el-overlay,'
+ '.ant-modal-wrap,[role="dialog"],[class*="dialog"],[class*="modal"],[class*="popup"]'
)].filter(visible).map((el) => {
const text = (el.innerText || '').replace(/\s+/g, ' ').trim();
const classes = String(el.className || '').slice(0, 80);
return `[${classes}] ${text.slice(0, 160)}`;
});
}
"""
_VIDEO_LIST_READY_SCRIPT = r"""
() => {
const visible = (el) => {
@@ -909,25 +1307,107 @@ def _max_excel_mtime():
def refresh_then_export_accounts(driver, urls, wait_seconds=REFRESH_WAIT_SECONDS):
"""两阶段执行:先刷新所有账号全部分页,再等待并重新导出。"""
"""两阶段执行:先刷新所有账号全部分页,再等待并重新导出。
运行中途登录态失效(目标页被重定向到登录页)时,自动用账号密码重新
登录并重试当前账号一次;浏览器/页面被外部关闭(如进程树被清理)时,
自动重建浏览器并恢复会话后重试当前账号一次。两种自愈整套运行各最多
一次,避免死循环。返回 (记录列表, 刷新摘要, 最终生效的驱动) —— 驱动
可能在重建后已不是调用方传入的那个,调用方需用返回值收尾关闭。
"""
refresh_summaries = []
relogin_state = {"attempted": False}
rebuild_state = {"attempted": False}
driver_box = {"driver": driver}
def _relogin_if_expired(exc: BaseException) -> bool:
"""登录态失效类异常时自动重登一次;返回是否已恢复登录。"""
if relogin_state["attempted"] or "登录态失效" not in str(exc):
return False
relogin_state["attempted"] = True
print("🔐 检测到登录态失效,尝试账号密码自动重新登录...")
try:
ok = login(driver_box["driver"])
except Exception as login_exc:
print(f"⚠️ 自动重新登录异常: {login_exc}")
return False
if ok:
print("✅ 自动重新登录成功,重试当前账号")
return bool(ok)
def _restore_session(current) -> bool:
"""重建浏览器后按 profile → cookie 文件 → 账密的顺序恢复登录态。"""
try:
if verify_login(current):
return True
if load_cookies(current) and verify_login(current):
return True
return bool(login(current))
except Exception as exc:
print(f"⚠️ 重建后会话恢复异常: {exc}")
return False
def _rebuild_if_browser_closed(exc: BaseException) -> bool:
"""浏览器/页面被外部关闭类异常时自愈一次;返回是否已恢复。
先尝试同一 context 重开新页(页面级死亡,登录态保留、代价最小);
context 也死了才整套重建驱动并按 profile → cookie → 账密恢复会话。
"""
if rebuild_state["attempted"] or not is_browser_closed_error(exc):
return False
rebuild_state["attempted"] = True
print("🌐 检测到浏览器页面被关闭,尝试自愈...")
current = driver_box["driver"]
try:
current.reopen_page()
print("✅ 浏览器 context 存活,已重开新页面,重试当前账号")
return True
except Exception as reopen_exc:
print(f"ℹ️ 页面重开失败({reopen_exc}),改为整套重建浏览器...")
try:
current.quit()
except Exception:
# 旧驱动随浏览器一起死亡,清理失败不影响重建。
pass
try:
driver_box["driver"] = create_driver()
except Exception as create_exc:
print(f"⚠️ 浏览器重建失败: {create_exc}")
return False
if not _restore_session(driver_box["driver"]):
print("⚠️ 重建后会话恢复失败,放弃自动重试")
return False
print("✅ 浏览器重建完成,重试当前账号")
return True
def _empty_summary(exc: BaseException) -> dict:
return {
"clicked": 0,
"pending": 0,
"zero_without_button": 0,
"unconfirmed": 0,
"pages": 0,
"error": str(exc),
}
for index, url in enumerate(urls, 1):
print(f"\n{'=' * 50}")
print(f"🔄 刷新阶段 博主 {index}/{len(urls)}: {url}")
print(f"{'=' * 50}")
try:
navigate_to_target(driver, url)
summary = refresh_zero_exposure_videos(driver)
navigate_to_target(driver_box["driver"], url)
summary = refresh_zero_exposure_videos(driver_box["driver"])
except Exception as exc:
print(f"⚠️ 博主 {index} 刷新扫描异常: {exc},继续下一个")
summary = {
"clicked": 0,
"pending": 0,
"zero_without_button": 0,
"unconfirmed": 0,
"pages": 0,
"error": str(exc),
}
if _relogin_if_expired(exc) or _rebuild_if_browser_closed(exc):
try:
navigate_to_target(driver_box["driver"], url)
summary = refresh_zero_exposure_videos(driver_box["driver"])
except Exception as retry_exc:
print(f"⚠️ 博主 {index} 自愈后刷新仍失败: {retry_exc},继续下一个")
summary = _empty_summary(retry_exc)
else:
print(f"⚠️ 博主 {index} 刷新扫描异常: {exc},继续下一个")
summary = _empty_summary(exc)
refresh_summaries.append(summary)
should_wait = any(
@@ -946,9 +1426,9 @@ def refresh_then_export_accounts(driver, urls, wait_seconds=REFRESH_WAIT_SECONDS
print(f"📌 导出阶段 博主 {index}/{len(urls)}: {url}")
print(f"{'=' * 50}")
try:
navigate_to_target(driver, url)
navigate_to_target(driver_box["driver"], url)
pre_mtime = _max_excel_mtime()
excel_path = export_video_data(driver, pre_mtime)
excel_path = export_video_data(driver_box["driver"], pre_mtime)
if not excel_path:
print(f"⚠️ 博主 {index} 导出失败,跳过")
continue
@@ -956,9 +1436,23 @@ def refresh_then_export_accounts(driver, urls, wait_seconds=REFRESH_WAIT_SECONDS
print(f" 解析出 {len(records)} 条视频记录")
all_records.extend(records)
except Exception as exc:
print(f"⚠️ 博主 {index} 导出异常: {exc},跳过")
if _relogin_if_expired(exc) or _rebuild_if_browser_closed(exc):
try:
navigate_to_target(driver_box["driver"], url)
pre_mtime = _max_excel_mtime()
excel_path = export_video_data(driver_box["driver"], pre_mtime)
if not excel_path:
print(f"⚠️ 博主 {index} 自愈后导出失败,跳过")
continue
records, _ = parse_chanmama_export(excel_path)
print(f" 解析出 {len(records)} 条视频记录(自愈后)")
all_records.extend(records)
except Exception as retry_exc:
print(f"⚠️ 博主 {index} 自愈后导出仍失败: {retry_exc},跳过")
else:
print(f"⚠️ 博主 {index} 导出异常: {exc},跳过")
return all_records, refresh_summaries
return all_records, refresh_summaries, driver_box["driver"]
def export_video_data(driver, pre_mtime: float = 0):
@@ -978,6 +1472,10 @@ def export_video_data(driver, pre_mtime: float = 0):
except TimeoutException:
print("⚠️ 视频列表加载超时,继续尝试导出...")
# 刷新/翻页阶段可能延迟弹出公告/活动弹窗,吞掉导出点击或盖住确认框;
# 点导出按钮前先清一次(确认框受保护)。
_dismiss_export_stage_overlays(driver)
# 方法1: 先在可见 DOM 中找真正的交互节点。新版按钮文字常在嵌套
# span/div 内,旧的 contains(text()) 和第一个隐藏 locator 都会漏掉它。
export_clicked = False
@@ -989,8 +1487,11 @@ def export_video_data(driver, pre_mtime: float = 0):
except TimeoutException:
control = None
if control:
clicked = driver.execute_script(_CLICK_EXPORT_CONTROL_SCRIPT)
export_clicked = bool(clicked)
# 原生点击优先(可触发内层元素的 Vue @click),JS 合成点击兜底
export_clicked = _native_click_export_button(driver)
if not export_clicked:
clicked = driver.execute_script(_CLICK_EXPORT_CONTROL_SCRIPT)
export_clicked = bool(clicked)
if export_clicked:
print(f"✅ 已点击导出按钮: {control.get('text') or control.get('className')}")
@@ -1032,36 +1533,38 @@ def export_video_data(driver, pre_mtime: float = 0):
return scrape_table_data(driver)
# 等待导出确认弹窗出现并点击【导出】按钮
# 蝉妈妈弹窗标题"您即将进行数据导出",按钮为 橙色"导出"(文字在 <span> 内)
# 蝉妈妈弹窗标题"您即将进行数据导出",按钮为 橙色"导出"(文字在 <span> 内)。
# 逐秒探测弹窗状态:可确认→立即点导出;挂着但没有可点的确认按钮
# (空壳遮罩/公告弹窗)→先 ESC+移除正常关,持续 5 秒以上就补点导出按钮
# (第一次点击可能已被弹窗吞掉);没有弹窗→常规清理并定期补点导出。
print("⏳ 等待导出确认弹窗...")
export_confirmed = False
for wait_sec in range(20):
unconfirmable_since: float | None = None
for wait_sec in range(36):
time.sleep(1)
try:
# 关键:用 contains(.,'导出') 匹配后代文本("导出"在 <span> 内)
confirm_selectors = [
# 精确:主按钮(primary) + 含"导出"文字
"//div[contains(@class,'el-dialog') or @role='dialog']//button[contains(.,'导出')]",
# 弹窗内任意含"导出"的按钮
"//div[contains(@class,'el-dialog') or @role='dialog']//button[contains(@class,'el-button--primary')][contains(.,'导出')]",
# 兼容"确定"/"确认"
"//div[contains(@class,'el-dialog') or @role='dialog']//button[contains(.,'确定')]",
"//div[contains(@class,'el-dialog') or @role='dialog']//button[contains(.,'确认')]",
]
for sel in confirm_selectors:
try:
btn = driver.find_element(By.XPATH, sel)
if btn.is_displayed():
driver.execute_script("arguments[0].click();", btn)
export_confirmed = True
print(f"✅ 已在弹窗中点击导出按钮: {sel}")
break
except NoSuchElementException:
continue
if export_confirmed:
state = _export_dialog_state(driver)
if state == "confirmable":
unconfirmable_since = None
if _click_export_confirm(driver):
export_confirmed = True
break
except Exception:
continue
if state == "present":
now = time.monotonic()
if unconfirmable_since is None:
_log_export_dialog_texts(driver)
print("ℹ️ 弹窗可见但没有可点的确认按钮,尝试关闭")
unconfirmable_since = now
elif now - unconfirmable_since > 5:
if _reclick_export_control(driver):
print("🔁 清掉卡死弹窗后补点导出按钮")
unconfirmable_since = None
_dismiss_export_stage_overlays(driver)
continue
unconfirmable_since = None
_dismiss_export_stage_overlays(driver)
if wait_sec in (10, 20, 30) and _reclick_export_control(driver):
print("🔁 确认框迟迟未出现,补点导出按钮")
if not export_confirmed:
try:
@@ -1093,9 +1596,20 @@ def export_video_data(driver, pre_mtime: float = 0):
"⚠️ 未找到本次下载的 Excel文件(等待 "
f"{CHANMAMA_EXPORT_WAIT_SECONDS + max(CHANMAMA_EXPORT_GRACE_SECONDS, 0)} 秒超时)"
)
_dump_export_failure(driver)
return None
def _download_complete(path: str) -> bool:
"""判断下载文件是否已写完整(zip EOCD 在文件尾,截断时 is_zipfile 为假)。"""
import zipfile
try:
return zipfile.is_zipfile(path)
except OSError:
return False
def _wait_for_exported_excel(
pre_mtime: float,
*,
@@ -1114,6 +1628,8 @@ def _wait_for_exported_excel(
deadline 之后落盘的竞态;两个窗口内都未出现则返回 None。
传入 driver 时每轮执行一次无害的 evaluate 以泵 Playwright 事件循环,
确保 download 事件处理器(_save_download)在 sleep 轮询期间也能执行。
泵调用发现浏览器/页面已被外部关闭时,若文件已完整落盘则照常返回,
否则抛出原始关闭异常交由上层重建浏览器重试,不再空等满超时。
"""
deadline = time.time() + max(timeout_seconds, 0)
grace_deadline = deadline + max(grace_seconds, 0)
@@ -1122,10 +1638,25 @@ def _wait_for_exported_excel(
if driver is not None:
try:
driver.execute_script("1")
except Exception:
pass
except Exception as pump_exc:
if is_browser_closed_error(pump_exc):
# 浏览器/页面被外部关闭(进程树清理等): 先看文件是否
# 已在关闭前完整落盘,再决定放弃,避免误杀已完成下载。
settled = _find_new_excel_after(pre_mtime)
if settled is not None and (
not os.path.exists(settled) or _download_complete(settled)
):
return settled
raise
# 其余瞬态异常(页面导航中等)不致命,继续轮询文件系统。
new_path = _find_new_excel_after(pre_mtime)
if new_path is not None:
# 文件出现 ≠ 下载完成: 当前浏览器可能以最终名边下边写,截断的
# zip 会让解析报 "File is not a zip file"。zip 的 EOCD 记录在
# 文件末尾,is_zipfile 通过即认为下载已完整;路径不存在(测试桩)
# 时保持旧行为直接返回。
if new_path is not None and (
not os.path.exists(new_path) or _download_complete(new_path)
):
return new_path
now = time.time()
if now >= deadline and not grace_logged and grace_deadline > deadline:
@@ -1829,7 +2360,8 @@ def main():
print("⚠️ 登录可能未成功,但继续尝试后续步骤...")
# 4. 先刷新两个账号的全部分页,再统一等待并重新进入页面导出。
all_records, _refresh_summaries = refresh_then_export_accounts(
# 返回的驱动可能已在浏览器被外部关闭后重建,后续收尾必须用它。
all_records, _refresh_summaries, driver = refresh_then_export_accounts(
driver, TARGET_URLS, wait_seconds=REFRESH_WAIT_SECONDS
)
@@ -18,7 +18,6 @@ from gyxx_flow.modules.content_marketing.runtime_paths import PATHS
_TOOLS_DIR = PATHS.tools_root
_PROJECT_ROOT = PATHS.module_root
RELOGIN_SCRIPT = _TOOLS_DIR / "relogin_bilibili.py"
try:
sys.stdout.reconfigure(encoding="utf-8", errors="replace") # type: ignore[attr-defined]
@@ -81,39 +80,6 @@ def comment_replace_policy(comments: list[dict], stats: dict, metrics: dict) ->
return False, False, "空评论且无法确认平台为 0 评论"
def do_relogin() -> bool:
"""Run relogin_bilibili.py (headed, waits for QR scan). Returns True if successful."""
acceptance_policy = current_acceptance_policy()
if acceptance_policy.enabled:
acceptance_policy.record(
"cookie_skipped",
operation="content.batch_rescrape_bilibili.relogin",
details={
"status": "SKIPPED_COOKIE",
"reason": "cookie is invalid and QR relogin is disabled for acceptance",
},
)
log(
" [SKIPPED_COOKIE] Cookie invalid; acceptance mode will not "
"launch relogin or wait for a QR scan"
)
return False
log(" Cookie 可能过期,触发自动重登 (请扫码)...")
try:
rc = subprocess.call(
[sys.executable, str(RELOGIN_SCRIPT)],
cwd=str(_PROJECT_ROOT),
)
if rc == 0:
log(" 重登成功,继续采集")
return True
else:
log(f" 重登失败 (exit={rc})")
return False
except Exception as exc:
log(f" 重登脚本执行异常: {exc}")
return False
def main(only_ids: list[int] | None = None, dry_run: bool = False) -> int:
rows = collect_urls()
@@ -126,8 +92,6 @@ def main(only_ids: list[int] | None = None, dry_run: bool = False) -> int:
total_comments = 0
consecutive_fail = 0
MAX_CONSECUTIVE_FAIL = 3
relogin_used = 0
MAX_RELOGINS = 1
t0 = time.time()
for i, row in enumerate(rows, 1):
note_id = row["id"]
@@ -152,11 +116,12 @@ def main(only_ids: list[int] | None = None, dry_run: bool = False) -> int:
break
if comments is None:
consecutive_fail += 1
if consecutive_fail >= MAX_CONSECUTIVE_FAIL and relogin_used < MAX_RELOGINS:
if do_relogin():
relogin_used += 1
consecutive_fail = 0
continue
if consecutive_fail >= MAX_CONSECUTIVE_FAIL:
# 登录统一由账号 vault 管理;连续失败只提示,不再触发扫码重登。
log(
" 连续失败达到上限;若为登录态失效请执行 "
"uv run gyxx accounts login bilibili-content"
)
if current_acceptance_policy().enabled:
return COOKIE_SKIP_EXIT_CODE
continue
@@ -26,7 +26,6 @@ from gyxx_flow.modules.content_marketing.serialized_page_action import (
_TOOLS_DIR = PATHS.tools_root
_PROJECT_ROOT = PATHS.module_root
RELOGIN_SCRIPT = _TOOLS_DIR / "relogin_douyin.py"
try:
sys.stdout.reconfigure(encoding="utf-8", errors="replace") # type: ignore[attr-defined]
@@ -104,38 +103,6 @@ def result_logged_in(result: dict | None) -> bool:
return bool(nested_stats.get("logged_in")) if isinstance(nested_stats, dict) else False
def do_relogin() -> bool:
"""Run relogin_douyin.py (headed, waits for QR scan). Returns True if successful."""
acceptance_policy = current_acceptance_policy()
if acceptance_policy.enabled:
acceptance_policy.record(
"cookie_skipped",
operation="content.batch_rescrape_douyin.relogin",
details={
"status": "SKIPPED_COOKIE",
"reason": "cookie is invalid and QR relogin is disabled for acceptance",
},
)
log(
" [SKIPPED_COOKIE] Cookie invalid; acceptance mode will not "
"launch relogin or wait for a QR scan"
)
return False
log(" Cookie 可能过期,触发自动重登 (请扫码)...")
try:
rc = subprocess.call(
[sys.executable, str(RELOGIN_SCRIPT)],
cwd=str(_PROJECT_ROOT),
)
if rc == 0:
log(" 重登成功,继续采集")
return True
log(f" 重登失败 (exit={rc})")
return False
except Exception as exc:
log(f" 重登脚本执行异常: {exc}")
return False
def scrape_with_fresh_browser(
url: str,
@@ -203,8 +170,6 @@ def main(only_ids: list[int] | None = None, dry_run: bool = False) -> int:
COOLDOWN_SECONDS = 300
cooldown_used = 0
MAX_COOLDOWNS = 3
relogin_used = 0
MAX_RELOGINS = 1 # 每轮 batch 最多自动重登1次
t0 = time.time()
for i, row in enumerate(rows, 1):
note_id = row["id"]
@@ -214,16 +179,10 @@ def main(only_ids: list[int] | None = None, dry_run: bool = False) -> int:
if comments is None:
consecutive_fail += 1
# 连续失败 + 可能 cookie 过期 → 尝试自动重登
if consecutive_fail >= MAX_CONSECUTIVE_FAIL and relogin_used < MAX_RELOGINS:
if do_relogin():
relogin_used += 1
consecutive_fail = 0
comments, stats, scrape_error = scrape_with_fresh_browser(url)
else:
# 重登失败,走冷却逻辑;验收模式必须直接跳过,不能等待。
if current_acceptance_policy().enabled:
return COOKIE_SKIP_EXIT_CODE
# 登录统一由账号 vault 管理;不再触发扫码重登,验收模式直接跳过。
if consecutive_fail >= MAX_CONSECUTIVE_FAIL:
if current_acceptance_policy().enabled:
return COOKIE_SKIP_EXIT_CODE
if comments is None and consecutive_fail >= MAX_CONSECUTIVE_FAIL:
if cooldown_used < MAX_COOLDOWNS:
@@ -25,7 +25,6 @@ from gyxx_flow.modules.content_marketing.serialized_page_action import (
_TOOLS_DIR = PATHS.tools_root
_PROJECT_ROOT = PATHS.module_root
RELOGIN_SCRIPT = _TOOLS_DIR / "relogin_xiaohongshu.py"
try:
sys.stdout.reconfigure(encoding="utf-8", errors="replace") # type: ignore[attr-defined]
@@ -87,38 +86,6 @@ def comment_replace_policy(comments: list[dict], stats: dict, metrics: dict) ->
return False, False, "空评论且无法确认平台为 0 评论"
def do_relogin() -> bool:
"""Run relogin_xiaohongshu.py (headed, waits for QR scan). Returns True if successful."""
acceptance_policy = current_acceptance_policy()
if acceptance_policy.enabled:
acceptance_policy.record(
"cookie_skipped",
operation="content.batch_rescrape_xiaohongshu.relogin",
details={
"status": "SKIPPED_COOKIE",
"reason": "cookie is invalid and QR relogin is disabled for acceptance",
},
)
log(
" [SKIPPED_COOKIE] Cookie invalid; acceptance mode will not "
"launch relogin or wait for a QR scan"
)
return False
log(" Cookie 可能过期,触发自动重登 (请扫码)...")
try:
rc = subprocess.call(
[sys.executable, str(RELOGIN_SCRIPT)],
cwd=str(_PROJECT_ROOT),
)
if rc == 0:
log(" 重登成功,继续采集")
return True
log(f" 重登失败 (exit={rc})")
return False
except Exception as exc:
log(f" 重登脚本执行异常: {exc}")
return False
def scrape_with_fresh_browser(
url: str,
@@ -173,8 +140,6 @@ def main(only_ids: list[int] | None = None, dry_run: bool = False) -> int:
total_comments = 0
consecutive_fail = 0
MAX_CONSECUTIVE_FAIL = 3
relogin_used = 0
MAX_RELOGINS = 1
t0 = time.time()
for i, row in enumerate(rows, 1):
note_id = row["id"]
@@ -183,13 +148,13 @@ def main(only_ids: list[int] | None = None, dry_run: bool = False) -> int:
comments, result, scrape_error = scrape_with_fresh_browser(url)
if comments is None:
consecutive_fail += 1
if consecutive_fail >= MAX_CONSECUTIVE_FAIL and relogin_used < MAX_RELOGINS:
if do_relogin():
relogin_used += 1
consecutive_fail = 0
comments, result, scrape_error = scrape_with_fresh_browser(url)
if consecutive_fail >= MAX_CONSECUTIVE_FAIL:
if current_acceptance_policy().enabled:
return COOKIE_SKIP_EXIT_CODE
log(
" 连续失败达到上限;若为登录态失效请执行 "
"uv run gyxx accounts login xiaohongshu-content"
)
if comments is None:
log(f" ERROR: {scrape_error}")
fail += 1
@@ -1,666 +0,0 @@
"""Launch five QR relogins in parallel, notify via analyzer Lark, retry 3 times."""
from __future__ import annotations
import argparse
import ctypes
import datetime as dt
import os
import subprocess
import sys
import time
from ctypes import wintypes
from dataclasses import dataclass
from pathlib import Path
from typing import Callable
import psutil
from gyxx_flow.adapters import (
ANALYZER_NOTIFICATION_APP_PROFILE,
COOKIE_SKIP_EXIT_CODE,
ResolvedNotificationRoute,
binding_from_environment,
current_acceptance_policy,
environment_for_child_script,
resolve_notification_route,
send_lark_bot_message,
)
from gyxx_flow.modules.content_marketing.runtime_paths import PATHS
PROJECT_DIR = PATHS.module_root
ANALYZER_DIR = PATHS.tmp_root / "relogin"
WORKFLOW_ID = "content.relogin.weekly"
MARKETING_RECIPIENT = "ou_24cc944d6e43c69c59d6560ad4e2ae6e"
TECHNICAL_RECIPIENT = "ou_7ad5fc8012e2f741afc5346e05ffd447"
@dataclass(frozen=True)
class Platform:
label: str
command: tuple[str, ...]
required_cookie: str
@dataclass(frozen=True, slots=True)
class NotificationDeliveryFailure:
"""One failed recipient delivery that must affect the workflow outcome."""
stage: str
platform: str
recipient: str
detail: str
PLATFORMS = {
"douyin": Platform(
"\u6296\u97f3", ("data/tools/relogin_douyin.py",),
"sessionid",
),
"bilibili": Platform(
"B\u7ad9", ("data/tools/relogin_bilibili.py",),
"SESSDATA",
),
"pgy": Platform(
"\u84b2\u516c\u82f1", ("data/tools/relogin_pgy.py",),
"",
),
"xingtu": Platform(
"\u661f\u56fe", ("data/tools/relogin_xingtu.py", "--force"),
"sessionid",
),
"xiaohongshu": Platform(
"\u5c0f\u7ea2\u4e66", ("data/tools/relogin_xiaohongshu.py",),
"web_session",
),
}
DEFAULT_RECIPIENTS = {
"pgy": MARKETING_RECIPIENT,
"xingtu": MARKETING_RECIPIENT,
"douyin": TECHNICAL_RECIPIENT,
"bilibili": TECHNICAL_RECIPIENT,
"xiaohongshu": TECHNICAL_RECIPIENT,
}
def resolve_platform_names(requested: list[str] | None) -> list[str]:
"""Resolve an optional manual platform subset while preserving order."""
if not requested:
return list(PLATFORMS)
names: list[str] = []
for name in requested:
if name not in PLATFORMS:
raise ValueError(f"unknown platform: {name}")
if name not in names:
names.append(name)
return names
def recipient_for_platform(platform_name: str, override: str | None = None) -> str:
"""Return the responsible recipient, unless a manual all-platform override is set."""
from gyxx_flow.adapters.acceptance_policy import resolve_notification_recipients
if override:
recipient = override
else:
try:
recipient = DEFAULT_RECIPIENTS[platform_name]
except KeyError as exc:
raise ValueError(
f"no Lark recipient configured for platform: {platform_name}"
) from exc
return resolve_notification_recipients(
(recipient,),
app_profile=ANALYZER_NOTIFICATION_APP_PROFILE,
)[0]
def resolve_weekly_notification_route(
override: str | None = None,
) -> ResolvedNotificationRoute:
"""Resolve the scheduled workflow route while retaining legacy defaults."""
defaults = (
(override,)
if override
else tuple(dict.fromkeys(DEFAULT_RECIPIENTS.values()))
)
return resolve_notification_route(WORKFLOW_ID, defaults)
def recipients_for_platform(
platform_name: str,
route: ResolvedNotificationRoute,
override: str | None = None,
) -> tuple[str, ...]:
"""Return dynamic recipients, or the original per-platform recipient."""
if not route.enabled:
return ()
if route.configured:
return route.open_ids
return (recipient_for_platform(platform_name, override),)
def group_platforms_by_recipient(
platform_names: list[str], override: str | None = None,
) -> dict[str, list[str]]:
grouped: dict[str, list[str]] = {}
for name in platform_names:
grouped.setdefault(recipient_for_platform(name, override), []).append(name)
return grouped
def group_platforms_by_notification_recipient(
platform_names: list[str],
route: ResolvedNotificationRoute,
override: str | None = None,
) -> dict[str, list[str]]:
"""Group failed platforms for legacy or configured multi-recipient delivery."""
if not route.enabled:
return {}
if route.configured:
return {recipient: list(platform_names) for recipient in route.open_ids}
return group_platforms_by_recipient(platform_names, override)
def run_with_retries(
platform_names: list[str],
*,
max_attempts: int,
run_round: Callable[[list[str], int], dict[str, bool]],
) -> dict[str, bool]:
status = {name: False for name in platform_names}
pending = list(platform_names)
for attempt in range(1, max_attempts + 1):
if not pending:
break
round_result = run_round(pending, attempt)
for name in pending:
status[name] = bool(round_result.get(name))
pending = [name for name in pending if not status[name]]
return status
def _valid_cookie(platform: Platform, cookie_file: Path) -> bool:
if not cookie_file.is_file():
return False
if not platform.required_cookie:
return cookie_file.stat().st_size > 10
try:
import json
values = json.loads(cookie_file.read_text(encoding="utf-8"))
return any(
item.get("name") == platform.required_cookie and item.get("value")
for item in values if isinstance(item, dict)
)
except Exception:
return False
def _descendant_pids(pid: int) -> set[int]:
try:
return {child.pid for child in psutil.Process(pid).children(recursive=True)}
except (psutil.Error, OSError):
return set()
def _terminate_process_tree(process: subprocess.Popen) -> None:
"""Terminate the whole Windows subprocess tree, including Chrome/Scrapling."""
try:
subprocess.run(
["taskkill", "/PID", str(process.pid), "/T", "/F"],
capture_output=True,
text=True,
check=False,
)
except OSError:
process.terminate()
try:
process.wait(timeout=10)
except subprocess.TimeoutExpired:
process.kill()
def _window_for_process(pid: int) -> int | None:
pids = _descendant_pids(pid) | {pid}
handles: list[int] = []
user32 = ctypes.windll.user32
callback_type = ctypes.WINFUNCTYPE(ctypes.c_bool, ctypes.c_void_p, ctypes.c_void_p)
def callback(hwnd, _lparam):
process_id = ctypes.c_ulong()
user32.GetWindowThreadProcessId(hwnd, ctypes.byref(process_id))
if process_id.value in pids and user32.IsWindowVisible(hwnd):
handles.append(int(hwnd))
return True
user32.EnumWindows(callback_type(callback), 0)
return handles[0] if handles else None
def _platform_from_window_title(title: str) -> str | None:
"""Identify a login window without relying on Chrome's process tree."""
if "\u84b2\u516c\u82f1" in title:
return "pgy"
if "\u6296\u97f3\u6388\u6743" in title or "\u5de8\u91cf\u661f\u56fe" in title:
return "xingtu"
if "\u6296\u97f3" in title:
return "douyin"
if "bilibili" in title.lower() or "\u8d26\u53f7\u767b\u5f55" in title:
return "bilibili"
if "\u5c0f\u7ea2\u4e66" in title:
return "xiaohongshu"
return None
def _login_windows() -> dict[str, int]:
"""Map visible Chrome login windows to platforms by their page titles."""
user32 = ctypes.windll.user32
found: dict[str, int] = {}
callback_type = ctypes.WINFUNCTYPE(ctypes.c_bool, ctypes.c_void_p, ctypes.c_void_p)
def callback(hwnd, _lparam):
if not user32.IsWindowVisible(hwnd):
return True
length = user32.GetWindowTextLengthW(hwnd)
if length <= 0:
return True
buffer = ctypes.create_unicode_buffer(length + 1)
user32.GetWindowTextW(hwnd, buffer, length + 1)
platform = _platform_from_window_title(buffer.value)
if platform:
found[platform] = int(hwnd)
return True
user32.EnumWindows(callback_type(callback), 0)
return found
def _capture_window(hwnd: int, output: Path) -> None:
"""Maximize and render one window directly, even when it is not foreground."""
from PIL import Image
user32 = ctypes.windll.user32
gdi32 = ctypes.windll.gdi32
user32.ShowWindow(hwnd, 3) # SW_MAXIMIZE
time.sleep(1)
rect = wintypes.RECT()
user32.GetWindowRect(hwnd, ctypes.byref(rect))
width, height = rect.right - rect.left, rect.bottom - rect.top
if width <= 0 or height <= 0:
raise RuntimeError(f"invalid window size: {width}x{height}")
window_dc = user32.GetWindowDC(hwnd)
memory_dc = gdi32.CreateCompatibleDC(window_dc)
bitmap = gdi32.CreateCompatibleBitmap(window_dc, width, height)
previous = gdi32.SelectObject(memory_dc, bitmap)
try:
if not user32.PrintWindow(hwnd, memory_dc, 2): # PW_RENDERFULLCONTENT
raise RuntimeError("PrintWindow failed")
class BitmapInfoHeader(ctypes.Structure):
_fields_ = [
("biSize", wintypes.DWORD), ("biWidth", wintypes.LONG),
("biHeight", wintypes.LONG), ("biPlanes", wintypes.WORD),
("biBitCount", wintypes.WORD), ("biCompression", wintypes.DWORD),
("biSizeImage", wintypes.DWORD),
("biXPelsPerMeter", wintypes.LONG),
("biYPelsPerMeter", wintypes.LONG),
("biClrUsed", wintypes.DWORD),
("biClrImportant", wintypes.DWORD),
]
byte_count = width * height * 4
pixels = ctypes.create_string_buffer(byte_count)
header = BitmapInfoHeader(
ctypes.sizeof(BitmapInfoHeader), width, -height, 1, 32, 0,
byte_count, 0, 0, 0, 0,
)
rows = gdi32.GetDIBits(
memory_dc, bitmap, 0, height, pixels, ctypes.byref(header), 0
)
if rows != height:
raise RuntimeError(f"GetDIBits returned {rows}/{height} rows")
image = Image.frombuffer(
"RGB", (width, height), pixels, "raw", "BGRX", 0, 1
).copy()
finally:
gdi32.SelectObject(memory_dc, previous)
gdi32.DeleteObject(bitmap)
gdi32.DeleteDC(memory_dc)
user32.ReleaseDC(hwnd, window_dc)
output.parent.mkdir(parents=True, exist_ok=True)
image.save(output, "PNG")
def _tile_windows(handles: list[int]) -> None:
"""Place four login windows in a 2x2 grid so screenshots cannot overlap."""
user32 = ctypes.windll.user32
screen_w = user32.GetSystemMetrics(0)
screen_h = user32.GetSystemMetrics(1)
cell_w = max(640, screen_w // 2)
cell_h = max(450, screen_h // 2)
for index, hwnd in enumerate(handles):
x = (index % 2) * cell_w
y = (index // 2) * cell_h
user32.ShowWindow(hwnd, 9) # SW_RESTORE
user32.MoveWindow(hwnd, x, y, cell_w, cell_h, True)
def _send_lark(
recipient: str,
text: str | None = None,
image: Path | None = None,
*,
app_profile: str = ANALYZER_NOTIFICATION_APP_PROFILE,
) -> tuple[dict[str, object], ...]:
"""Send text and/or image through the analyzer Hermes bot."""
if app_profile != ANALYZER_NOTIFICATION_APP_PROFILE:
raise ValueError(
"relogin notifications must use the hermes-analyzer lark-cli profile"
)
lark_env = {
key: value for key, value in os.environ.items()
if not key.upper().startswith("HERMES_")
}
receipts: list[dict[str, object]] = []
if text:
ANALYZER_DIR.mkdir(parents=True, exist_ok=True)
receipts.append(send_lark_bot_message(
user_id=recipient,
text=text,
profile=ANALYZER_NOTIFICATION_APP_PROFILE,
cwd=ANALYZER_DIR,
env=lark_env,
))
if image:
receipts.append(send_lark_bot_message(
user_id=recipient,
image=image,
profile=ANALYZER_NOTIFICATION_APP_PROFILE,
cwd=image.parent,
env=lark_env,
))
return tuple(receipts)
def _send_qr_notification(
route: ResolvedNotificationRoute,
platform_name: str,
*,
override: str | None,
text: str,
image: Path,
) -> tuple[NotificationDeliveryFailure, ...]:
"""Attempt every QR delivery and return all recipient-specific failures."""
profile = route.app_profile or ANALYZER_NOTIFICATION_APP_PROFILE
failures: list[NotificationDeliveryFailure] = []
for recipient in recipients_for_platform(platform_name, route, override):
for stage, kwargs in (
("qr_text", {"text": text}),
("qr_image", {"image": image}),
):
try:
_send_lark(recipient, app_profile=profile, **kwargs)
except Exception as exc:
failures.append(
NotificationDeliveryFailure(
stage=stage,
platform=platform_name,
recipient=recipient,
detail=str(exc) or type(exc).__name__,
)
)
return tuple(failures)
def _unavailable_qr_delivery_failures(
route: ResolvedNotificationRoute,
platform_name: str,
*,
override: str | None,
detail: str,
) -> tuple[NotificationDeliveryFailure, ...]:
"""Record both intended QR deliveries when no screenshot can be sent."""
return tuple(
NotificationDeliveryFailure(
stage=stage,
platform=platform_name,
recipient=recipient,
detail=detail,
)
for recipient in recipients_for_platform(platform_name, route, override)
for stage in ("qr_text", "qr_image")
)
def _send_failure_summaries(
route: ResolvedNotificationRoute,
failed_platforms: list[str],
*,
override: str | None,
max_attempts: int,
) -> tuple[NotificationDeliveryFailure, ...]:
"""Attempt every final failure summary and return all delivery failures."""
groups = group_platforms_by_notification_recipient(
failed_platforms,
route,
override,
)
profile = route.app_profile or ANALYZER_NOTIFICATION_APP_PROFILE
failures: list[NotificationDeliveryFailure] = []
for recipient, names in groups.items():
labels = "\u3001".join(PLATFORMS[name].label for name in names)
try:
_send_lark(
recipient,
text=(
f"\u4ee5\u4e0b\u5e73\u53f0\u8fde\u7eed {max_attempts} \u6b21\u672a\u626b\u7801\uff0c"
"\u5df2\u6062\u590d\u5e76\u7ee7\u4f7f\u7528\u65e7 Cookie\uff1a"
f"{labels}"
),
app_profile=profile,
)
except Exception as exc:
failures.append(
NotificationDeliveryFailure(
stage="failure_summary",
platform=",".join(names),
recipient=recipient,
detail=str(exc) or type(exc).__name__,
)
)
return tuple(failures)
def _run_round_factory(
args,
notification_route: ResolvedNotificationRoute | None,
notification_failures: list[NotificationDeliveryFailure],
):
python = sys.executable
screenshot_root = PATHS.tmp_root / "qrcode/friday_relogin"
def run_round(names: list[str], attempt: int) -> dict[str, bool]:
started: dict[str, subprocess.Popen] = {}
cookie_files: dict[str, Path] = {}
missing_qr_windows: set[str] = set()
stamp = dt.datetime.now().strftime("%Y%m%d_%H%M%S")
for name in names:
platform = PLATFORMS[name]
cmd = [python, *platform.command, "--login-timeout", str(args.round_timeout)]
target = PROJECT_DIR / platform.command[0]
environment = environment_for_child_script(target, os.environ)
cookie_files[name] = binding_from_environment(environment).cookie_file
started[name] = subprocess.Popen(
cmd,
cwd=PROJECT_DIR,
env=environment,
)
time.sleep(args.screenshot_delay)
title_windows = _login_windows()
windows = {
name: title_windows.get(name) or _window_for_process(process.pid)
for name, process in started.items()
}
_tile_windows([hwnd for hwnd in windows.values() if hwnd])
time.sleep(2)
for name, process in started.items():
platform = PLATFORMS[name]
hwnd = windows[name]
if not hwnd:
print(f"[{name}] no visible browser window for QR screenshot", flush=True)
missing_qr_windows.add(name)
continue
output = screenshot_root / f"{stamp}_{name}_round{attempt}.png"
try:
_capture_window(hwnd, output)
print(f"[{name}] QR screenshot: {output}", flush=True)
except Exception as exc:
print(f"[{name}] screenshot failed: {exc}", flush=True)
if not args.no_send and notification_route is not None:
notification_failures.extend(
_unavailable_qr_delivery_failures(
notification_route,
name,
override=args.recipient,
detail=(
"QR screenshot capture failed: "
f"{str(exc) or type(exc).__name__}"
),
)
)
continue
if not args.no_send and notification_route is not None:
notification_failures.extend(
_send_qr_notification(
notification_route,
name,
override=args.recipient,
text=(
f"{platform.label} \u767b\u5f55\u4e8c\u7ef4\u7801"
f"\uff08\u7b2c {attempt}/{args.max_attempts} \u6b21\uff09\uff0c"
"\u8bf7\u5728 3 \u5206\u949f\u5185\u626b\u7801\u3002"
),
image=output,
)
)
# Put the still-open login windows back into a compact layout after
# taking full-size screenshots one at a time.
_tile_windows([hwnd for hwnd in windows.values() if hwnd])
results: dict[str, bool] = {}
for name, process in started.items():
try:
rc = process.wait(timeout=args.round_timeout + 90)
except subprocess.TimeoutExpired:
_terminate_process_tree(process)
rc = 1
results[name] = rc == 0 and _valid_cookie(
PLATFORMS[name],
cookie_files[name],
)
if (
not results[name]
and name in missing_qr_windows
and not args.no_send
and notification_route is not None
):
notification_failures.extend(
_unavailable_qr_delivery_failures(
notification_route,
name,
override=args.recipient,
detail=(
"no visible QR window and relogin did not succeed"
),
)
)
print(f"[{name}] round={attempt} exit={rc} success={results[name]}", flush=True)
return results
return run_round
def main() -> int:
acceptance_policy = current_acceptance_policy()
if acceptance_policy.enabled:
acceptance_policy.record(
"cookie_skipped",
operation="content.relogin.weekly",
details={
"status": "SKIPPED_COOKIE",
"reason": "acceptance runs never launch QR relogin workflows",
},
)
print(
"[SKIPPED_COOKIE] Acceptance mode does not launch login windows, "
"reset cookies/profiles, or wait for QR scans.",
flush=True,
)
return COOKIE_SKIP_EXIT_CODE
parser = argparse.ArgumentParser(description="Friday five-platform parallel QR relogin")
parser.add_argument(
"--recipient", default=None,
help="Override the configured recipients and send every platform to one open_id",
)
parser.add_argument("--max-attempts", type=int, default=3)
parser.add_argument("--round-timeout", type=int, default=300)
parser.add_argument("--screenshot-delay", type=int, default=60)
parser.add_argument("--no-send", action="store_true", help="Do not send Lark messages")
parser.add_argument(
"--platform", action="append", choices=sorted(PLATFORMS),
help="Only relogin the selected platform; may be repeated",
)
args = parser.parse_args()
platform_names = resolve_platform_names(args.platform)
notification_route = (
None if args.no_send else resolve_weekly_notification_route(args.recipient)
)
notification_failures: list[NotificationDeliveryFailure] = []
status = run_with_retries(
platform_names, max_attempts=args.max_attempts,
run_round=_run_round_factory(
args,
notification_route,
notification_failures,
),
)
failed = [name for name, ok in status.items() if not ok]
if failed and not args.no_send and notification_route is not None:
notification_failures.extend(
_send_failure_summaries(
notification_route,
failed,
override=args.recipient,
max_attempts=args.max_attempts,
)
)
print(f"Final relogin status: {status}", flush=True)
if notification_failures:
print(
f"Final notification status: failed ({len(notification_failures)} deliveries)",
flush=True,
)
for failure in notification_failures:
print(
"[NOTIFICATION_ERROR] "
f"stage={failure.stage} platform={failure.platform} "
f"recipient={failure.recipient} error={failure.detail}",
flush=True,
)
return 2
if notification_route is not None and notification_route.enabled:
print("Final notification status: delivered", flush=True)
else:
print("Final notification status: skipped", flush=True)
return 1 if failed else 0
if __name__ == "__main__":
raise SystemExit(main())
@@ -1,113 +0,0 @@
"""每周一重登 B 站。
强制清掉 cookie JSON + 浏览器 profile,确保下次启动时必须扫码。
任务计划: 每周一 09:00
跑法:
python data/tools/relogin_bilibili.py # 实际跑(会弹二维码)
python data/tools/relogin_bilibili.py --dry-run # 只看会做什么
"""
import argparse
import datetime as dt
import subprocess
import sys
from gyxx_flow.modules.content_marketing.data.tools.relogin_runtime_paths import (
child_browser_environment,
resolve_relogin_runtime_paths,
sync_relogin_state_to_consumers,
)
from gyxx_flow.modules.content_marketing.data.tools.relogin_transaction import (
begin,
commit,
rollback,
)
from gyxx_flow.modules.content_marketing.runtime_paths import PATHS
PROJECT_DIR = PATHS.module_root
RUNTIME_PATHS = resolve_relogin_runtime_paths("bilibili")
COOKIE_FILE = RUNTIME_PATHS.cookie_file
STATE_FILE = RUNTIME_PATHS.storage_state_file
PROFILE_DIR = RUNTIME_PATHS.profile_dir
SCRAPER = PROJECT_DIR / "bilibili_comment_scraper.py"
def reset_state():
"""删 cookie JSON + storage_state + 清浏览器 profile。"""
state = begin((COOKIE_FILE, STATE_FILE), PROFILE_DIR)
print(" [OK] 旧 Cookie/Profile 已暂存,失败时将自动恢复")
return state
def main() -> int:
ap = argparse.ArgumentParser(description="每周一重登 B 站")
ap.add_argument("--login-timeout", type=int, default=300, help="扫码超时(秒)")
ap.add_argument("--dry-run", action="store_true", help="只打印计划,不删不跑")
args = ap.parse_args()
print("== Relogin: B 站 ==")
print(f" 今天: {dt.date.today()} ({dt.date.today().strftime('%A')})")
print(f" cookie: {COOKIE_FILE}")
print(f" profile: {PROFILE_DIR}")
print()
if args.dry_run:
print("[DRY-RUN] 实际跑会: 删 cookie + 清 profile + 弹二维码")
return 0
old_state = reset_state()
cmd = [
sys.executable,
str(SCRAPER),
"login", # dummy url (login-only 模式不需要真实 url)
"--login-only",
"--login-timeout",
str(args.login_timeout),
]
print(f"\n [RUN] {' '.join(cmd)}")
try:
rc = subprocess.call(
cmd,
cwd=str(PROJECT_DIR),
env=child_browser_environment(
cookie_file=COOKIE_FILE,
storage_state_file=STATE_FILE,
profile_dir=PROFILE_DIR,
),
)
except BaseException:
rollback(old_state)
raise
print(f"\n exit={rc}")
login_ok = False
if rc == 0:
# 验证 SESSDATA 是否真写入了
try:
import json
cookies = json.loads(COOKIE_FILE.read_text(encoding="utf-8"))
sessdata = any(c.get("name") == "SESSDATA" and c.get("value") for c in cookies)
login_ok = bool(sessdata)
print(f" SESSDATA 写入: {'OK' if sessdata else 'FAIL — 没有看到 SESSDATA,登录可能没成功'}")
except Exception as exc:
print(f" [WARN] 验证 cookie 时出错: {exc}")
if rc != 0 or not login_ok:
rollback(old_state)
print(" [ROLLBACK] 未完成扫码,已恢复旧登录状态")
return rc or 1
try:
sync_relogin_state_to_consumers(
"bilibili",
cookie_file=COOKIE_FILE,
storage_state_file=STATE_FILE,
)
except Exception as exc:
rollback(old_state)
print(f" [ROLLBACK] 同步采集器登录态失败: {exc}")
return 1
commit(old_state)
return 0
if __name__ == "__main__":
sys.exit(main())
@@ -1,121 +0,0 @@
"""Force a fresh Douyin login while preserving the previous usable session."""
from __future__ import annotations
import argparse
import datetime as dt
import json
import subprocess
import sys
from gyxx_flow.modules.content_marketing.data.tools.relogin_runtime_paths import (
child_browser_environment,
resolve_relogin_runtime_paths,
sync_relogin_state_to_consumers,
)
from gyxx_flow.modules.content_marketing.runtime_paths import PATHS
try:
from .relogin_transaction import begin, commit, rollback
except ImportError: # Direct script execution from data/tools.
from gyxx_flow.modules.content_marketing.data.tools.relogin_transaction import (
begin,
commit,
rollback,
)
PROJECT_DIR = PATHS.module_root
RUNTIME_PATHS = resolve_relogin_runtime_paths("douyin")
COOKIE_FILE = RUNTIME_PATHS.cookie_file
STATE_FILE = RUNTIME_PATHS.storage_state_file
PROFILE_DIR = RUNTIME_PATHS.profile_dir
SCRAPER = PROJECT_DIR / "douyin_comment_scraper.py"
LOGIN_COOKIE = "sessionid"
def reset_state():
state = begin((COOKIE_FILE, STATE_FILE), PROFILE_DIR)
print(" [OK] Old Cookie/Profile staged; failures will roll back automatically")
return state
def is_cookie_valid() -> bool:
if not COOKIE_FILE.exists():
return False
try:
cookies = json.loads(COOKIE_FILE.read_text(encoding="utf-8"))
return any(
isinstance(cookie, dict)
and cookie.get("name") == LOGIN_COOKIE
and cookie.get("value")
for cookie in cookies
)
except Exception:
return False
def main() -> int:
parser = argparse.ArgumentParser(description="Relogin Douyin")
parser.add_argument("--login-timeout", type=int, default=300)
parser.add_argument("--dry-run", action="store_true")
args = parser.parse_args()
print("== Relogin: Douyin ==")
print(f" date: {dt.date.today()}")
print(f" cookie: {COOKIE_FILE}")
print(f" profile: {PROFILE_DIR}")
print(f" current cookie: {'present' if is_cookie_valid() else 'missing/invalid'}")
if args.dry_run:
print("[DRY-RUN] Would stage Cookie/Profile and open the QR login window")
return 0
old_state = reset_state()
command = [
sys.executable,
str(SCRAPER),
"login",
"--login-only",
"--login-timeout",
str(args.login_timeout),
]
print(f"\n [RUN] {' '.join(command)}")
try:
return_code = subprocess.call(
command,
cwd=str(PROJECT_DIR),
env=child_browser_environment(
cookie_file=COOKIE_FILE,
storage_state_file=STATE_FILE,
profile_dir=PROFILE_DIR,
),
)
except BaseException:
rollback(old_state)
raise
login_ok = return_code == 0 and is_cookie_valid()
print(f"\n exit={return_code}")
print(f" {LOGIN_COOKIE} written: {'OK' if login_ok else 'FAIL'}")
if not login_ok:
rollback(old_state)
print(" [ROLLBACK] Login incomplete; restored previous Cookie/Profile")
return return_code or 1
try:
sync_relogin_state_to_consumers(
"douyin",
cookie_file=COOKIE_FILE,
storage_state_file=STATE_FILE,
)
except Exception as exc:
rollback(old_state)
print(f" [ROLLBACK] Failed to sync collector login state: {exc}")
return 1
commit(old_state)
return 0
if __name__ == "__main__":
raise SystemExit(main())
@@ -1,148 +0,0 @@
"""每周一重登小红书蒲公英。
强制清掉 cookie JSON + 浏览器 profile,确保下次启动时必须扫码。
任务计划: 每周一 09:00
跑法:
python data/tools/relogin_pgy.py # 实际跑(会弹二维码)
python data/tools/relogin_pgy.py --dry-run # 只看会做什么
"""
import argparse
import datetime as dt
import shutil
import subprocess
import sys
from pathlib import Path
from gyxx_flow.modules.content_marketing.data.tools.relogin_runtime_paths import (
child_browser_environment,
resolve_relogin_runtime_paths,
sync_relogin_state_to_consumers,
)
from gyxx_flow.modules.content_marketing.runtime_paths import PATHS
PROJECT_DIR = PATHS.module_root
RUNTIME_PATHS = resolve_relogin_runtime_paths("pgy")
COOKIE_FILE = RUNTIME_PATHS.cookie_file
STATE_FILE = RUNTIME_PATHS.storage_state_file
PROFILE_DIR = RUNTIME_PATHS.profile_dir
SCRAPER = PROJECT_DIR / "pgy_xhs_scraper_v2.py"
def reset_state() -> tuple[Path | None, Path | None, Path | None]:
"""Temporarily move the old login state aside so it can be restored."""
ts = dt.datetime.now().strftime("%Y%m%d_%H%M%S_%f")
cookie_backup = None
state_backup = None
profile_backup = None
if COOKIE_FILE.exists():
backup = COOKIE_FILE.with_suffix(f".json.bak.{ts}")
COOKIE_FILE.rename(backup)
cookie_backup = backup
print(f" [OK] 备份 cookie -> {backup.name}")
else:
print(f" [INFO] {COOKIE_FILE.name} 不存在,跳过")
if STATE_FILE.exists():
state_backup = STATE_FILE.with_suffix(f".json.bak.{ts}")
STATE_FILE.rename(state_backup)
print(f" [OK] 备份 storage state -> {state_backup.name}")
else:
print(f" [INFO] {STATE_FILE.name} 不存在,跳过")
if PROFILE_DIR.exists():
profile_backup = PROFILE_DIR.with_name(f"{PROFILE_DIR.name}.bak.{ts}")
PROFILE_DIR.rename(profile_backup)
print(f" [OK] 暂存 {PROFILE_DIR.name}/ -> {profile_backup.name}/")
else:
print(f" [INFO] {PROFILE_DIR.name}/ 不存在,跳过")
return cookie_backup, profile_backup, state_backup
def restore_previous_state(
cookie_backup: Path | None,
profile_backup: Path | None,
state_backup: Path | None = None,
) -> None:
"""Discard an incomplete login and restore the last known-good state."""
if COOKIE_FILE.exists():
COOKIE_FILE.unlink()
if cookie_backup and cookie_backup.exists():
cookie_backup.rename(COOKIE_FILE)
print(f" [ROLLBACK] 已恢复旧 cookie: {COOKIE_FILE.name}")
if STATE_FILE.exists():
STATE_FILE.unlink()
if state_backup and state_backup.exists():
state_backup.rename(STATE_FILE)
print(f" [ROLLBACK] 已恢复旧 storage state: {STATE_FILE.name}")
if PROFILE_DIR.exists():
shutil.rmtree(PROFILE_DIR)
if profile_backup and profile_backup.exists():
profile_backup.rename(PROFILE_DIR)
print(f" [ROLLBACK] 已恢复旧浏览器状态: {PROFILE_DIR.name}/")
def finish_successful_relogin(profile_backup: Path | None) -> None:
"""Keep the new state and remove the bulky old profile backup."""
if profile_backup and profile_backup.exists():
shutil.rmtree(profile_backup)
def main() -> int:
ap = argparse.ArgumentParser(description="每周一重登小红书蒲公英")
ap.add_argument("--login-timeout", type=int, default=300, help="扫码超时(秒)")
ap.add_argument("--dry-run", action="store_true", help="只打印计划,不删不跑")
args = ap.parse_args()
print("== Relogin: 小红书蒲公英 ==")
print(f" 今天: {dt.date.today()} ({dt.date.today().strftime('%A')})")
print(f" cookie: {COOKIE_FILE}")
print(f" profile: {PROFILE_DIR}")
print()
if args.dry_run:
print("[DRY-RUN] 实际跑会: 删 cookie + 清 profile + 弹二维码")
return 0
cookie_backup, profile_backup, state_backup = reset_state()
cmd = [
sys.executable,
str(SCRAPER),
"--login-only",
"--login-timeout",
str(args.login_timeout),
]
print(f"\n [RUN] {' '.join(cmd)}")
try:
rc = subprocess.call(
cmd,
cwd=str(PROJECT_DIR),
env=child_browser_environment(
cookie_file=COOKIE_FILE,
storage_state_file=STATE_FILE,
profile_dir=PROFILE_DIR,
),
)
except BaseException:
restore_previous_state(cookie_backup, profile_backup, state_backup)
raise
print(f"\n exit={rc}")
if rc != 0 or not COOKIE_FILE.exists():
print(" [WARN] 新登录未完成,恢复旧登录状态")
restore_previous_state(cookie_backup, profile_backup, state_backup)
return rc or 1
try:
sync_relogin_state_to_consumers(
"pgy",
cookie_file=COOKIE_FILE,
storage_state_file=STATE_FILE,
)
except Exception as exc:
restore_previous_state(cookie_backup, profile_backup, state_backup)
print(f" [ROLLBACK] 同步采集器登录态失败: {exc}")
return 1
finish_successful_relogin(profile_backup)
return rc
if __name__ == "__main__":
sys.exit(main())
@@ -1,279 +0,0 @@
"""Resolve one relogin wrapper's browser state contract.
The workflow runtime injects a unique browser binding into each relogin
wrapper. Standalone legacy launches do not have that environment, so they
retain the original per-platform files below the content-marketing state root.
"""
from __future__ import annotations
import json
import os
import re
import shutil
import tempfile
import uuid
from dataclasses import dataclass
from pathlib import Path
from typing import Mapping
from gyxx_flow.adapters.integration import RuntimeIntegrationCatalog
from gyxx_flow.modules.content_marketing.runtime_paths import PATHS
_PLATFORM = re.compile(r"^[a-z][a-z0-9_-]{0,31}$")
CONSUMER_BINDING_IDS: Mapping[str, tuple[str, ...]] = {
"bilibili": ("content.comments.collect_bilibili",),
"douyin": ("content.comments.collect_douyin",),
"pgy": ("content_marketing:pgy_xhs_scraper_v2.py",),
"xiaohongshu": ("content.comments.collect_xiaohongshu",),
# v2 now reads the Buyin/Doudian site and is owned by the douyin-shop
# account. Keep the legacy Xingtu relogin isolated from that vault.
"buyin": ("content_marketing:xingtu_scraper_v2.py",),
"xingtu": ("content_marketing:xingtu_scraper.py",),
}
@dataclass(frozen=True)
class ReloginRuntimePaths:
cookie_file: Path
storage_state_file: Path
profile_dir: Path
def resolve_relogin_runtime_paths(
platform: str,
environment: Mapping[str, str] | None = None,
) -> ReloginRuntimePaths:
"""Prefer the injected binding and fall back to original platform paths."""
if not _PLATFORM.fullmatch(platform):
raise ValueError("invalid relogin platform")
values = os.environ if environment is None else environment
legacy_cookie = PATHS.state_root / "cookies" / f"{platform}_cookies.json"
legacy_storage = (
PATHS.state_root / "cookies" / f"{platform}_storage_state.json"
)
legacy_profile = PATHS.state_root / "browser-profiles" / platform
return ReloginRuntimePaths(
cookie_file=_bound_path(
values,
"GYXX_BROWSER_COOKIE_FILE",
legacy_cookie,
),
storage_state_file=_bound_path(
values,
"GYXX_BROWSER_STORAGE_STATE_FILE",
legacy_storage,
),
profile_dir=_bound_path(
values,
"GYXX_BROWSER_PROFILE_DIR",
legacy_profile,
),
)
def child_browser_environment(
*,
cookie_file: Path,
storage_state_file: Path,
profile_dir: Path,
environment: Mapping[str, str] | None = None,
) -> dict[str, str]:
"""Keep the nested login scraper on the wrapper's injected binding."""
result = dict(os.environ if environment is None else environment)
result.update(
{
"GYXX_BROWSER_COOKIE_FILE": str(cookie_file.resolve()),
"GYXX_BROWSER_STORAGE_STATE_FILE": str(
storage_state_file.resolve()
),
"GYXX_BROWSER_PROFILE_DIR": str(profile_dir.resolve()),
}
)
# content_marketing/sitecustomize.py normally rebinds a nested Python
# script from argv[0]. A relogin scraper is an implementation detail of
# this wrapper and must write the wrapper binding that the parent monitors.
result.pop("GYXX_MODULE_ID", None)
return result
def sync_relogin_state_to_consumers(
platform: str,
*,
cookie_file: Path,
storage_state_file: Path,
environment: Mapping[str, str] | None = None,
) -> tuple[Path, ...]:
"""Atomically publish a verified relogin state to unique consumer paths."""
try:
consumer_ids = CONSUMER_BINDING_IDS[platform]
except KeyError as exc:
raise ValueError("invalid relogin platform") from exc
values = os.environ if environment is None else environment
project_root = values.get("GYXX_PROJECT_ROOT", "").strip()
data_root = values.get("GYXX_DATA_ROOT", "").strip()
if not project_root and not data_root:
# Original standalone launches use the legacy per-platform state files
# directly, so there is no separate managed consumer to publish to.
return ()
if not project_root or not data_root:
raise RuntimeError("managed relogin state sync requires project and data roots")
catalog = RuntimeIntegrationCatalog.load_default(
project_root=project_root,
data_root=data_root,
)
source_cookie = cookie_file.expanduser().resolve()
source_storage = storage_state_file.expanduser().resolve()
cookie_payload = _read_cookie_payload(source_cookie)
storage_available = source_storage.is_file()
if storage_available:
_read_storage_state_payload(source_storage)
source_paths = {source_cookie, source_storage}
destination_paths: set[Path] = set()
destinations: list[Path] = []
updates: list[tuple[Path, Path]] = []
for consumer_id in consumer_ids:
binding = catalog.binding_for(consumer_id)
target_cookie = binding.cookie_file.resolve()
target_storage = binding.storage_state_file.resolve()
targets = (target_cookie, target_storage)
if any(target in source_paths for target in targets) or any(
target in destination_paths for target in targets
):
raise RuntimeError(
"relogin and consumer browser state paths must be unique"
)
destination_paths.update(targets)
_validate_cookie_requirements(
cookie_payload,
required_domains=binding.required_cookie_domains,
required_names=binding.required_cookie_names,
)
updates.append((source_cookie, target_cookie))
if storage_available:
updates.append((source_storage, target_storage))
destinations.append(target_cookie)
_atomic_replace_files(updates)
return tuple(destinations)
def _bound_path(
environment: Mapping[str, str],
name: str,
fallback: Path,
) -> Path:
configured = environment.get(name, "").strip()
return (
Path(configured).expanduser().resolve()
if configured
else fallback.resolve()
)
def _read_cookie_payload(path: Path) -> list[dict[str, object]]:
try:
payload = json.loads(path.read_text(encoding="utf-8"))
except (OSError, UnicodeError, ValueError) as exc:
raise RuntimeError("relogin cookie state is missing or invalid") from exc
if not isinstance(payload, list) or not payload or not all(
isinstance(cookie, dict) for cookie in payload
):
raise RuntimeError("relogin cookie state is missing or invalid")
return payload
def _read_storage_state_payload(path: Path) -> dict[str, object]:
try:
payload = json.loads(path.read_text(encoding="utf-8"))
except (OSError, UnicodeError, ValueError) as exc:
raise RuntimeError("relogin storage state is invalid") from exc
if not isinstance(payload, dict):
raise RuntimeError("relogin storage state is invalid")
return payload
def _validate_cookie_requirements(
cookies: list[dict[str, object]],
*,
required_domains: tuple[str, ...],
required_names: tuple[str, ...],
) -> None:
usable = [cookie for cookie in cookies if cookie.get("value")]
if not usable:
raise RuntimeError("relogin cookie state has no usable cookies")
if required_domains and not any(
any(
str(cookie.get("domain", "")).lstrip(".").casefold()
.endswith(domain.casefold().lstrip("."))
for domain in required_domains
)
for cookie in usable
):
raise RuntimeError("relogin cookie state does not match consumer domain")
if required_names and not any(
str(cookie.get("name", "")) in required_names for cookie in usable
):
raise RuntimeError("relogin cookie state misses the consumer login cookie")
def _atomic_replace_files(updates: list[tuple[Path, Path]]) -> None:
"""Replace a small related file set and restore every old target on error."""
transaction = uuid.uuid4().hex
staged: list[tuple[Path, Path]] = []
backups: dict[Path, Path | None] = {}
replaced: list[Path] = []
try:
for source, target in updates:
target.parent.mkdir(parents=True, exist_ok=True)
handle, raw_temp = tempfile.mkstemp(
prefix=f".{target.name}.{transaction}.",
suffix=".tmp",
dir=target.parent,
)
temp = Path(raw_temp)
with os.fdopen(handle, "wb") as stream:
stream.write(source.read_bytes())
stream.flush()
os.fsync(stream.fileno())
staged.append((temp, target))
backup = target.with_name(f".{target.name}.{transaction}.bak")
if target.is_file():
shutil.copy2(target, backup)
backups[target] = backup
else:
backups[target] = None
for temp, target in staged:
os.replace(temp, target)
replaced.append(target)
except BaseException:
for target in reversed(replaced):
backup = backups.get(target)
if backup is not None and backup.exists():
os.replace(backup, target)
elif target.exists():
target.unlink()
raise
finally:
for temp, _target in staged:
if temp.exists():
temp.unlink()
for backup in backups.values():
if backup is not None and backup.exists():
backup.unlink()
__all__ = [
"CONSUMER_BINDING_IDS",
"ReloginRuntimePaths",
"child_browser_environment",
"resolve_relogin_runtime_paths",
"sync_relogin_state_to_consumers",
]
@@ -1,47 +0,0 @@
"""Transactional backup/rollback helpers for interactive relogin scripts."""
from __future__ import annotations
import datetime as dt
import shutil
from dataclasses import dataclass
from pathlib import Path
@dataclass
class LoginStateBackup:
files: dict[Path, Path]
profile: Path
profile_backup: Path | None
def begin(files: tuple[Path, ...], profile: Path) -> LoginStateBackup:
stamp = dt.datetime.now().strftime("%Y%m%d_%H%M%S_%f")
backups: dict[Path, Path] = {}
for source in files:
if source.exists():
backup = source.with_name(f"{source.name}.bak.{stamp}")
source.rename(backup)
backups[source] = backup
profile_backup = None
if profile.exists():
profile_backup = profile.with_name(f"{profile.name}.bak.{stamp}")
profile.rename(profile_backup)
return LoginStateBackup(backups, profile, profile_backup)
def rollback(state: LoginStateBackup) -> None:
for source, backup in state.files.items():
if source.exists():
source.unlink()
if backup.exists():
backup.rename(source)
if state.profile.exists():
shutil.rmtree(state.profile)
if state.profile_backup and state.profile_backup.exists():
state.profile_backup.rename(state.profile)
def commit(state: LoginStateBackup) -> None:
if state.profile_backup and state.profile_backup.exists():
shutil.rmtree(state.profile_backup)
@@ -1,116 +0,0 @@
"""每周重登小红书。
强制清掉 cookie JSON + 浏览器 profile,确保下次启动时必须扫码。
跑法:
python data/tools/relogin_xiaohongshu.py # 实际跑(会弹二维码)
python data/tools/relogin_xiaohongshu.py --dry-run # 只看会做什么
"""
import argparse
import datetime as dt
import json
import subprocess
import sys
from gyxx_flow.modules.content_marketing.data.tools.relogin_runtime_paths import (
child_browser_environment,
resolve_relogin_runtime_paths,
sync_relogin_state_to_consumers,
)
from gyxx_flow.modules.content_marketing.data.tools.relogin_transaction import (
begin,
commit,
rollback,
)
from gyxx_flow.modules.content_marketing.runtime_paths import PATHS
PROJECT_DIR = PATHS.module_root
RUNTIME_PATHS = resolve_relogin_runtime_paths("xiaohongshu")
COOKIE_FILE = RUNTIME_PATHS.cookie_file
STATE_FILE = RUNTIME_PATHS.storage_state_file
PROFILE_DIR = RUNTIME_PATHS.profile_dir
SCRAPER = PROJECT_DIR / "xiaohongshu_comment_scraper.py"
LOGIN_COOKIE = "web_session"
def reset_state():
state = begin((COOKIE_FILE, STATE_FILE), PROFILE_DIR)
print(" [OK] 旧 Cookie/Profile 已暂存,失败时将自动恢复")
return state
def is_cookie_valid() -> bool:
if not COOKIE_FILE.exists():
return False
try:
cookies = json.loads(COOKIE_FILE.read_text(encoding="utf-8"))
return any(c.get("name") == LOGIN_COOKIE and c.get("value") for c in cookies)
except Exception:
return False
def main() -> int:
ap = argparse.ArgumentParser(description="重登小红书")
ap.add_argument("--login-timeout", type=int, default=300, help="扫码超时(秒)")
ap.add_argument("--dry-run", action="store_true", help="只打印计划,不删不跑")
args = ap.parse_args()
print("== Relogin: 小红书 ==")
print(f" 今天: {dt.date.today()}")
print(f" cookie: {COOKIE_FILE}")
print(f" profile: {PROFILE_DIR}")
print(f" 当前 cookie 状态: {'有效' if is_cookie_valid() else '无效/不存在'}")
print()
if args.dry_run:
print("[DRY-RUN] 实际跑会: 删 cookie + 清 profile + 弹二维码")
return 0
old_state = reset_state()
cmd = [
sys.executable,
str(SCRAPER),
"login",
"--login-only",
"--login-timeout",
str(args.login_timeout),
]
print(f"\n [RUN] {' '.join(cmd)}")
try:
rc = subprocess.call(
cmd,
cwd=str(PROJECT_DIR),
env=child_browser_environment(
cookie_file=COOKIE_FILE,
storage_state_file=STATE_FILE,
profile_dir=PROFILE_DIR,
),
)
except BaseException:
rollback(old_state)
raise
print(f"\n exit={rc}")
login_ok = rc == 0 and is_cookie_valid()
if rc == 0:
print(f" {LOGIN_COOKIE} 写入: {'OK' if login_ok else 'FAIL — 没有看到 web_session,登录可能没成功'}")
if not login_ok:
rollback(old_state)
print(" [ROLLBACK] 未完成扫码,已恢复旧登录状态")
return rc or 1
try:
sync_relogin_state_to_consumers(
"xiaohongshu",
cookie_file=COOKIE_FILE,
storage_state_file=STATE_FILE,
)
except Exception as exc:
rollback(old_state)
print(f" [ROLLBACK] 同步采集器登录态失败: {exc}")
return 1
commit(old_state)
return 0
if __name__ == "__main__":
sys.exit(main())
@@ -1,350 +0,0 @@
"""每 3 个周一重登一次抖音/星图。
强制清掉 cookie JSON + 浏览器 profile,确保下次启动时必须扫码。
星图登录完成后,自动把登录态 cookie 同步到抖音评论 scraper,避免再扫一次码。
任务计划: 每周一 09:00 (脚本内部判断是否到了该重登的周一,不到就退出)
3 周循环基准: 任意一个周一,后续 (今天 - BASE_DATE) // 7 % 3 == 0 才是星图周。
改 BASE_DATE 就改了重登周期。
跑法:
python data/tools/relogin_xingtu.py # 实际跑
python data/tools/relogin_xingtu.py --dry-run # 看今天是不是星图周
python data/tools/relogin_xingtu.py --force # 忽略 3 周判断,强制重登
"""
import argparse
import datetime as dt
import json
import os
import shutil
import subprocess
import sys
from pathlib import Path
import psutil
from gyxx_flow.modules.content_marketing.data.tools.relogin_runtime_paths import (
child_browser_environment,
resolve_relogin_runtime_paths,
sync_relogin_state_to_consumers,
)
from gyxx_flow.modules.content_marketing.runtime_paths import PATHS
PROJECT_DIR = PATHS.module_root
RUNTIME_PATHS = resolve_relogin_runtime_paths("xingtu")
COOKIE_FILE = RUNTIME_PATHS.cookie_file
STATE_FILE = RUNTIME_PATHS.storage_state_file
PROFILE_DIR = RUNTIME_PATHS.profile_dir
SCRAPER = PROJECT_DIR / "xingtu_scraper_v2.py"
# 抖音评论 scraper 的 cookie 文件 (登录态共享目标)
DY_COOKIE_FILE = PATHS.state_root / "cookies/douyin_cookies.json"
DY_STATE_FILE = PATHS.state_root / "cookies/douyin_storage_state.json"
DY_PROFILE_DIR = PATHS.state_root / "browser-profiles/douyin"
# 需要同步的抖音登录 cookie 名
DY_LOGIN_COOKIES = {"sessionid", "sessionid_ss", "sid_guard", "sid_tt", "passport_csrf_token"}
# 3 周循环起点(必须是周一)。星图 cookie ~22 天,每 3 周(21 天)重登正好。
BASE_DATE = dt.date(2026, 6, 22)
LOGIN_COOKIE_NAMES = {"sessionid", "sessionid_ss", "sid_tt", "sid_guard"}
def transaction_file() -> Path:
return COOKIE_FILE.with_name(".xingtu_relogin_transaction.json")
def _write_transaction(
cookie_backup: Path | None,
profile_backup: Path | None,
state_backup: Path | None,
) -> None:
marker = transaction_file()
marker.parent.mkdir(parents=True, exist_ok=True)
temp = marker.with_name(f"{marker.name}.tmp")
temp.write_text(json.dumps({
"cookie_backup": str(cookie_backup) if cookie_backup else "",
"state_backup": str(state_backup) if state_backup else "",
"profile_backup": str(profile_backup) if profile_backup else "",
}, ensure_ascii=False, indent=2), encoding="utf-8")
os.replace(temp, marker)
def _clear_transaction() -> None:
marker = transaction_file()
if marker.exists():
marker.unlink()
def _remove_path(path: Path) -> None:
if path.is_dir():
shutil.rmtree(path)
elif path.exists():
path.unlink()
def recover_interrupted_state() -> bool:
"""Restore a relogin transaction that was killed after moving old state."""
marker = transaction_file()
if not marker.exists():
return False
try:
state = json.loads(marker.read_text(encoding="utf-8"))
except Exception:
return False
cookie_backup = Path(state["cookie_backup"]) if state.get("cookie_backup") else None
state_backup = Path(state["state_backup"]) if state.get("state_backup") else None
profile_backup = Path(state["profile_backup"]) if state.get("profile_backup") else None
terminate_profile_processes()
if cookie_backup and cookie_backup.exists():
_remove_path(COOKIE_FILE)
cookie_backup.rename(COOKIE_FILE)
if state_backup and state_backup.exists():
_remove_path(STATE_FILE)
state_backup.rename(STATE_FILE)
if profile_backup and profile_backup.exists():
_remove_path(PROFILE_DIR)
profile_backup.rename(PROFILE_DIR)
_clear_transaction()
print(" [RECOVER] 已恢复上次被中断的星图登录状态")
return True
def has_valid_login_cookie_file(path: Path | None = None) -> bool:
path = path or COOKIE_FILE
try:
cookies = json.loads(path.read_text(encoding="utf-8"))
except Exception:
return False
return any(
isinstance(cookie, dict)
and cookie.get("name") in LOGIN_COOKIE_NAMES
and cookie.get("value")
for cookie in cookies
)
def terminate_profile_processes() -> None:
"""Stop only browser processes using Xingtu's dedicated profile."""
profile_text = str(PROFILE_DIR.resolve()).casefold()
matches = []
for process in psutil.process_iter(["pid", "cmdline"]):
try:
command = " ".join(process.info.get("cmdline") or []).casefold()
if process.pid != os.getpid() and profile_text in command:
matches.append(process)
except (psutil.Error, OSError):
continue
for process in matches:
try:
process.terminate()
except (psutil.Error, OSError):
pass
_gone, alive = psutil.wait_procs(matches, timeout=5)
for process in alive:
try:
process.kill()
except (psutil.Error, OSError):
pass
def is_xingtu_week(today: dt.date, base: dt.date) -> bool:
days = (today - base).days
if days < 0:
return False
return (days // 7) % 3 == 0
def reset_state() -> tuple[Path | None, Path | None, Path | None]:
"""Temporarily move the old login state aside so it can be restored."""
ts = dt.datetime.now().strftime("%Y%m%d_%H%M%S_%f")
cookie_backup = None
state_backup = None
profile_backup = None
if COOKIE_FILE.exists():
cookie_backup = COOKIE_FILE.with_suffix(f".json.bak.{ts}")
if STATE_FILE.exists():
state_backup = STATE_FILE.with_suffix(f".json.bak.{ts}")
if PROFILE_DIR.exists():
profile_backup = PROFILE_DIR.with_name(f"{PROFILE_DIR.name}.bak.{ts}")
_write_transaction(cookie_backup, profile_backup, state_backup)
if cookie_backup:
COOKIE_FILE.rename(cookie_backup)
print(f" [OK] 备份 cookie -> {cookie_backup.name}")
else:
print(f" [INFO] {COOKIE_FILE.name} 不存在,跳过")
if state_backup:
STATE_FILE.rename(state_backup)
print(f" [OK] 备份 storage state -> {state_backup.name}")
else:
print(f" [INFO] {STATE_FILE.name} 不存在,跳过")
if profile_backup:
PROFILE_DIR.rename(profile_backup)
print(f" [OK] 暂存 {PROFILE_DIR.name}/ -> {profile_backup.name}/")
else:
print(f" [INFO] {PROFILE_DIR.name}/ 不存在,跳过")
return cookie_backup, profile_backup, state_backup
def restore_previous_state(
cookie_backup: Path | None,
profile_backup: Path | None,
state_backup: Path | None = None,
) -> None:
"""Discard an incomplete login and restore the last known-good state."""
terminate_profile_processes()
if COOKIE_FILE.exists():
COOKIE_FILE.unlink()
if cookie_backup and cookie_backup.exists():
cookie_backup.rename(COOKIE_FILE)
print(f" [ROLLBACK] 已恢复旧 cookie: {COOKIE_FILE.name}")
_remove_path(STATE_FILE)
if state_backup and state_backup.exists():
state_backup.rename(STATE_FILE)
print(f" [ROLLBACK] 已恢复旧 storage state: {STATE_FILE.name}")
if PROFILE_DIR.exists():
shutil.rmtree(PROFILE_DIR)
if profile_backup and profile_backup.exists():
profile_backup.rename(PROFILE_DIR)
print(f" [ROLLBACK] 已恢复旧浏览器状态: {PROFILE_DIR.name}/")
_clear_transaction()
def finish_successful_relogin(profile_backup: Path | None) -> None:
"""Keep the new state and remove the bulky old profile backup."""
if profile_backup and profile_backup.exists():
shutil.rmtree(profile_backup)
_clear_transaction()
def sync_to_douyin_scraper() -> None:
"""把星图的登录态 cookie 同步到抖音评论 scraper,避免再扫一次码。"""
if not COOKIE_FILE.exists():
print(" [SKIP] 星图 cookie 不存在,无法同步")
return
try:
xingtu_cookies = json.loads(COOKIE_FILE.read_text(encoding="utf-8"))
except Exception as exc:
print(f" [SKIP] 读取星图 cookie 失败: {exc}")
return
# 提取登录态 cookie
login_cookies = [c for c in xingtu_cookies if isinstance(c, dict) and c.get("name") in DY_LOGIN_COOKIES]
if not login_cookies:
print(" [SKIP] 星图 cookie 中没有找到登录态,跳过同步")
return
# 清除抖音评论 scraper 的旧 profile (强制刷新)
if DY_PROFILE_DIR.exists():
shutil.rmtree(DY_PROFILE_DIR)
print(f" [OK] 清空 {DY_PROFILE_DIR.name}/")
# 读取或创建抖音评论 cookie 文件
dy_cookies = []
if DY_COOKIE_FILE.exists():
try:
dy_cookies = json.loads(DY_COOKIE_FILE.read_text(encoding="utf-8"))
except Exception:
dy_cookies = []
# 合并: 同名 cookie 覆盖,新 cookie 追加
dy_by_name = {c.get("name"): c for c in dy_cookies if isinstance(c, dict)}
for c in login_cookies:
# 修正 domain: 星图的 cookie domain 可能是 .douyin.com 子域,评论采集需要 .douyin.com
domain = c.get("domain", "")
if "xingtu" in domain or "oceanengine" in domain:
c["domain"] = ".douyin.com"
dy_by_name[c["name"]] = c
DY_COOKIE_FILE.parent.mkdir(parents=True, exist_ok=True)
DY_COOKIE_FILE.write_text(
json.dumps(list(dy_by_name.values()), ensure_ascii=False, indent=2),
encoding="utf-8",
)
print(f" [OK] 同步 {len(login_cookies)} 个登录 cookie 到 {DY_COOKIE_FILE.name}")
def main() -> int:
ap = argparse.ArgumentParser(description="每 3 周周一重登星图")
ap.add_argument("--base-date", default=BASE_DATE.isoformat(),
help="3 周循环起点(必须是周一),格式 YYYY-MM-DD")
ap.add_argument("--login-timeout", type=int, default=300)
ap.add_argument("--force", action="store_true", help="忽略 3 周判断,强制重登")
ap.add_argument("--dry-run", action="store_true", help="只打印计划")
args = ap.parse_args()
today = dt.date.today()
base = dt.date.fromisoformat(args.base_date)
is_xw = is_xingtu_week(today, base)
print("== Relogin: 抖音/星图 ==")
print(f" 今天: {today} ({today.strftime('%A')})")
print(f" 3 周基准: {base}")
print(f" 星图周? {'是' if is_xw else '否'}")
print()
if not is_xw and not args.force:
print("今天不是星图周,跳过 (传 --force 强制重登)")
return 0
if args.dry_run:
print("[DRY-RUN] 实际跑会: 删 cookie + 清 profile + 弹二维码")
return 0
recover_interrupted_state()
terminate_profile_processes()
cookie_backup, profile_backup, state_backup = reset_state()
cmd = [
sys.executable,
str(SCRAPER),
"--login-only",
"--login-timeout",
str(args.login_timeout),
]
print(f"\n [RUN] {' '.join(cmd)}")
try:
rc = subprocess.call(
cmd,
cwd=str(PROJECT_DIR),
env=child_browser_environment(
cookie_file=COOKIE_FILE,
storage_state_file=STATE_FILE,
profile_dir=PROFILE_DIR,
),
)
except BaseException:
restore_previous_state(cookie_backup, profile_backup, state_backup)
raise
finally:
terminate_profile_processes()
print(f"\n exit={rc}")
if rc != 0 or not has_valid_login_cookie_file():
print(" [WARN] 新登录未完成,恢复旧登录状态")
restore_previous_state(cookie_backup, profile_backup, state_backup)
return rc or 1
try:
sync_relogin_state_to_consumers(
"xingtu",
cookie_file=COOKIE_FILE,
storage_state_file=STATE_FILE,
)
except Exception as exc:
restore_previous_state(cookie_backup, profile_backup, state_backup)
print(f" [ROLLBACK] 同步采集器登录态失败: {exc}")
return 1
finish_successful_relogin(profile_backup)
if rc == 0:
sync_to_douyin_scraper()
return rc
if __name__ == "__main__":
sys.exit(main())
@@ -426,6 +426,21 @@ def ensure_table(cur) -> None:
)
def drop_stale_rows(
all_rows: dict[tuple, tuple], stale_paths: list[Path]
) -> int:
"""删除来源文件已陈旧的行,返回删除数(按行内 source_file 文件名匹配)。"""
stale_names = {Path(path).name for path in stale_paths}
source_index = ROW_FIELDS.index("source_file")
stale_keys = [
key for key, row in all_rows.items()
if str(row[source_index]) in stale_names
]
for key in stale_keys:
del all_rows[key]
return len(stale_keys)
def main() -> int:
parser = argparse.ArgumentParser(
description="同步自营内容(抖音/B站/小红书)全量指标到 cmt_self_video_metrics"
@@ -504,20 +519,25 @@ def main() -> int:
print("❌ 没有任何平台的来源文件可用")
return 1
stale = [
str(path)
path
for path in sources
if business_date_bound
and not args.allow_stale
and not args.use_file_mtime
and not is_current_artifact(path, metric_date)
]
if stale and not args.allow_stale:
if stale:
# 陈旧防护按来源文件粒度跳过: 某平台来源过期只丢弃该平台的行,
# 不拖累其他平台当天的新鲜数据(如抖音导出失败时不拦 B站/小红书)。
dropped = drop_stale_rows(all_rows, stale)
print(
f"❌ 以下来源文件不是业务日期({metric_date.isoformat()})的产物; "
"补历史请显式传 --metric-date 并加 --allow-stale:\n "
+ "\n ".join(stale)
f"⚠️ 以下来源文件不是业务日期({metric_date.isoformat()})的产物, "
f"已跳过对应 {dropped} 行(补历史用 --metric-date + "
"--allow-stale):\n " + "\n ".join(str(p) for p in stale)
)
return 1
if not all_rows:
print("❌ 跳过陈旧来源后没有可入库的内容指标行")
return 1
print(
f"[sync-self] 平台={'+'.join(selected_platforms)} 待写={len(all_rows)} 行; "
@@ -543,74 +563,98 @@ def main() -> int:
from gyxx_flow.modules.content_marketing.data.tools.db import get_db_config
with psycopg.connect(**get_db_config()) as conn:
with conn.cursor() as cur:
ensure_table(cur)
for row in all_rows.values():
# SQL 为静态字面量,29 个值全部经 %s 占位符参数绑定传入。
cur.execute(
"""
INSERT INTO cmt_self_video_metrics (
platform, account_alias, account_name,
platform_account_id, video_id, title, chanmama_url,
video_url, duration, publish_time, metric_date,
exposure, like_count, comment_count, share_count,
est_sales_count, est_sales_amount, note_type,
creation_method, publish_status, mounted_products,
homepage_redirect_count, live_redirect_count,
payment_amount, payment_orders, coin_count,
favorite_count, danmaku_count, page_metrics,
source_file,
observed_at, created_at, updated_at
) VALUES (
%s, %s, %s,
%s, %s, %s, %s,
%s, %s, %s, %s,
%s, %s, %s, %s,
%s, %s, %s,
%s, %s, %s,
%s, %s,
%s, %s, %s,
%s, %s, %s,
%s,
CURRENT_TIMESTAMP, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP
)
ON CONFLICT (platform, account_name, video_id, metric_date)
DO UPDATE SET
account_alias = EXCLUDED.account_alias,
platform_account_id = EXCLUDED.platform_account_id,
title = EXCLUDED.title,
chanmama_url = EXCLUDED.chanmama_url,
video_url = EXCLUDED.video_url,
duration = EXCLUDED.duration,
publish_time = EXCLUDED.publish_time,
exposure = EXCLUDED.exposure,
like_count = EXCLUDED.like_count,
comment_count = EXCLUDED.comment_count,
share_count = EXCLUDED.share_count,
est_sales_count = EXCLUDED.est_sales_count,
est_sales_amount = EXCLUDED.est_sales_amount,
note_type = EXCLUDED.note_type,
creation_method = EXCLUDED.creation_method,
publish_status = EXCLUDED.publish_status,
mounted_products = EXCLUDED.mounted_products,
homepage_redirect_count = EXCLUDED.homepage_redirect_count,
live_redirect_count = EXCLUDED.live_redirect_count,
payment_amount = EXCLUDED.payment_amount,
payment_orders = EXCLUDED.payment_orders,
coin_count = EXCLUDED.coin_count,
favorite_count = EXCLUDED.favorite_count,
danmaku_count = EXCLUDED.danmaku_count,
page_metrics = EXCLUDED.page_metrics,
source_file = EXCLUDED.source_file,
observed_at = CURRENT_TIMESTAMP,
updated_at = CURRENT_TIMESTAMP
""",
row,
)
conn.commit()
print(f"[sync-self] 完成: 已幂等写入 {len(all_rows)} 行到 {TABLE_NAME}")
written, failed = upsert_rows(conn, list(all_rows.values()))
if failed:
print(f"[sync-self] 完成: 写入 {written} 行,单行失败跳过 {failed} 行 "
f"(见上方 ⚠️ 日志)")
return 0 if written else 1
print(f"[sync-self] 完成: 已幂等写入 {written} 行到 {TABLE_NAME}")
return 0
def upsert_rows(conn, rows: list[tuple]) -> tuple[int, int]:
"""逐行 upsert,返回 (成功数, 失败数)。
每行放在独立保存点里执行: 单行失败(如超长值违反约束)只回滚该行,
不拖累其他平台/其他内容的写入——这是「一个平台失败不影响全库」的
最后一道隔离。
"""
written = 0
failed = 0
with conn.cursor() as cur:
ensure_table(cur)
for row in rows:
try:
with conn.transaction():
# SQL 为静态字面量,29 个值全部经 %s 占位符参数绑定传入。
cur.execute(
"""
INSERT INTO cmt_self_video_metrics (
platform, account_alias, account_name,
platform_account_id, video_id, title, chanmama_url,
video_url, duration, publish_time, metric_date,
exposure, like_count, comment_count, share_count,
est_sales_count, est_sales_amount, note_type,
creation_method, publish_status, mounted_products,
homepage_redirect_count, live_redirect_count,
payment_amount, payment_orders, coin_count,
favorite_count, danmaku_count, page_metrics,
source_file,
observed_at, created_at, updated_at
) VALUES (
%s, %s, %s,
%s, %s, %s, %s,
%s, %s, %s, %s,
%s, %s, %s, %s,
%s, %s, %s,
%s, %s, %s,
%s, %s,
%s, %s, %s,
%s, %s, %s,
%s,
CURRENT_TIMESTAMP, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP
)
ON CONFLICT (platform, account_name, video_id, metric_date)
DO UPDATE SET
account_alias = EXCLUDED.account_alias,
platform_account_id = EXCLUDED.platform_account_id,
title = EXCLUDED.title,
chanmama_url = EXCLUDED.chanmama_url,
video_url = EXCLUDED.video_url,
duration = EXCLUDED.duration,
publish_time = EXCLUDED.publish_time,
exposure = EXCLUDED.exposure,
like_count = EXCLUDED.like_count,
comment_count = EXCLUDED.comment_count,
share_count = EXCLUDED.share_count,
est_sales_count = EXCLUDED.est_sales_count,
est_sales_amount = EXCLUDED.est_sales_amount,
note_type = EXCLUDED.note_type,
creation_method = EXCLUDED.creation_method,
publish_status = EXCLUDED.publish_status,
mounted_products = EXCLUDED.mounted_products,
homepage_redirect_count = EXCLUDED.homepage_redirect_count,
live_redirect_count = EXCLUDED.live_redirect_count,
payment_amount = EXCLUDED.payment_amount,
payment_orders = EXCLUDED.payment_orders,
coin_count = EXCLUDED.coin_count,
favorite_count = EXCLUDED.favorite_count,
danmaku_count = EXCLUDED.danmaku_count,
page_metrics = EXCLUDED.page_metrics,
source_file = EXCLUDED.source_file,
observed_at = CURRENT_TIMESTAMP,
updated_at = CURRENT_TIMESTAMP
""",
row,
)
except Exception as exc:
failed += 1
print(f" ⚠️ 单行写入失败已隔离跳过: {type(exc).__name__}: {exc}")
else:
written += 1
conn.commit()
return written, failed
if __name__ == "__main__":
raise SystemExit(main())
@@ -273,18 +273,15 @@ def maybe_wait_for_login(
*,
deadline: float | None = None,
) -> None:
login_deadline = time.monotonic() + login_timeout
while time.monotonic() < login_deadline:
_check_note_deadline(deadline, "login wait")
try:
body = page.locator("body").inner_text(timeout=5000)
except Exception:
body = ""
if "扫码登录" not in body and "登录后" not in body:
return
log("Douyin may require login. Please finish login in the opened Chrome window.")
page.wait_for_timeout(3000)
_check_note_deadline(deadline, "login wait")
# 登录统一由账号 vault 管理(gyxx accounts login douyin-content),
# 由工作流执行前的 cookie 网关保证;运行中失效立即降级,不再等待人工扫码。
try:
body = page.locator("body").inner_text(timeout=5000)
except Exception:
body = ""
if "扫码登录" in body or "登录后" in body:
log("Douyin login state missing; collecting degraded samples. "
"Run `gyxx accounts login douyin-content` to restore full comments.")
def is_logged_in(page: Page) -> bool:
@@ -833,55 +830,16 @@ def write_outputs(url: str, comments: list[dict[str, Any]], result: dict[str, An
return json_path, csv_path
def login_only(login_timeout: int) -> int:
"""只触发登录(headed),登录完成立刻退出。给 relogin_douyin.py 用。"""
def action(page: Page) -> None:
page.goto("https://www.douyin.com/", wait_until="domcontentloaded", timeout=90000)
page.wait_for_timeout(2500)
# 主动触发登录弹窗(点登录按钮)
try:
login_btn = page.get_by_text("登录", exact=True).first
if login_btn.count():
login_btn.click(timeout=3000)
page.wait_for_timeout(1500)
except Exception:
pass
maybe_wait_for_login(page, login_timeout)
save_state(page)
PROFILE_DIR.mkdir(parents=True, exist_ok=True)
with ScraplingBrowser(
headless=False,
real_chrome=True,
user_data_dir=str(PROFILE_DIR),
locale="zh-CN",
timezone_id="Asia/Shanghai",
timeout=90000,
network_idle=False,
disable_resources=False,
google_search=False,
page_setup=restore_cookies,
max_pages=1,
) as browser:
browser.run_fetch_action("https://www.douyin.com/", action, wait=1000)
log("[login-only] 完成")
return 0
def main() -> int:
parser = argparse.ArgumentParser(description="Scrape Douyin video comments via Scrapling browser and XHR interception.")
parser.add_argument("url", help="Douyin video/user URL (传 'login' 配合 --login-only)")
parser.add_argument("--login-timeout", type=int, default=300, help="等待扫码/登录秒数")
parser.add_argument("url", help="Douyin video/user URL")
parser.add_argument("--login-timeout", type=int, default=300,
help="保留参数(登录等待已移除;未登录将直接降级采集)")
parser.add_argument("--max-scrolls", type=int, default=120, help="最多滚动加载次数")
parser.add_argument("--idle-rounds", type=int, default=8, help="连续几轮无新评论后停止")
parser.add_argument("--headless", action="store_true", help="无头模式;首次登录不要使用")
parser.add_argument("--login-only", action="store_true",
help="只触发登录(headed)然后退出,给 relogin_douyin 用")
parser.add_argument("--headless", action="store_true", help="无头模式")
args = parser.parse_args()
if args.login_only:
return login_only(args.login_timeout)
comments, result = scrape_comments(args.url, args.login_timeout, args.max_scrolls, args.idle_rounds, args.headless)
json_path, csv_path = write_outputs(args.url, comments, result)
log(f"Final URL: {result.get('final_url', '')}")
@@ -38,6 +38,7 @@ from gyxx_flow.adapters.scrapling import (
BrowserTimeoutError, # noqa: F401
is_browser_timeout_error,
)
from gyxx_flow.adapters.browser import CookieExpiredError, cookie_expired_message
from gyxx_flow.core.exit_codes import COOKIE_SKIP_EXIT_CODE
from gyxx_flow.modules.content_marketing.collection_completeness import (
BLOCKED_INPUT,
@@ -333,56 +334,6 @@ def is_logged_in(page: Page) -> bool:
return False
def open_scan_login(page: Page) -> None:
"""Open the login dialog and prefer its QR-code login tab."""
for pattern in (r"^登录$|账号登录",):
try:
button = page.get_by_text(re.compile(pattern)).first
if button.count() and button.is_visible():
button.click(timeout=2000)
page.wait_for_timeout(500)
except Exception:
pass
# The login dialog uses an image-only folded-corner switch. Its bundled
# asset name is stable even though the generated CSS class is not.
try:
qr_switch = page.locator('img[src*="qr_code"]').first
if qr_switch.count() and qr_switch.is_visible():
# The browser engine's actionability check is blocked by the folded-corner
# overlay. A native DOM click triggers the image's real handler.
qr_switch.evaluate("e => e.click()")
page.wait_for_timeout(1000)
return
except Exception:
pass
for pattern in (r"扫码登录|二维码登录|扫码",):
try:
button = page.get_by_text(re.compile(pattern)).first
if button.count() and button.is_visible():
button.click(timeout=2000)
page.wait_for_timeout(500)
except Exception:
pass
def wait_for_scan_login(page: Page, timeout_seconds: int) -> None:
log("Not logged in. 请在打开的 Chrome 中扫码登录。")
open_scan_login(page)
deadline = time.time() + timeout_seconds
next_login_click = time.time() + 5
while time.time() < deadline:
if is_logged_in(page):
save_state(page)
return
if time.time() >= next_login_click:
open_scan_login(page)
next_login_click = time.time() + 5
page.wait_for_timeout(2000)
raise TimeoutError(f"Login timed out after {timeout_seconds} seconds.")
def ensure_login(page: Page, login_timeout: int) -> None:
page.goto(HOME_URL, wait_until="domcontentloaded", timeout=60000)
page.wait_for_timeout(2500)
@@ -402,20 +353,11 @@ def ensure_login(page: Page, login_timeout: int) -> None:
)
log(f"[SKIPPED_COOKIE] {reason}")
raise AcceptanceCookieSkip(reason)
page.goto(LOGIN_URL, wait_until="domcontentloaded", timeout=60000)
page.wait_for_timeout(1500)
wait_for_scan_login(page, login_timeout)
page.goto(HOME_URL, wait_until="domcontentloaded", timeout=60000)
page.wait_for_load_state("domcontentloaded")
# 关闭弹窗
for text in ("跳过", "不再提示", "我知道了"):
try:
b = page.get_by_text(text, exact=True).first
if b.count():
b.click(timeout=1500)
page.wait_for_timeout(800)
except Exception:
pass
# 登录统一由账号 vault 管理(gyxx accounts login xiaohongshu-content),
# 采集脚本不再内嵌扫码等待;运行中失效直接快速失败。
reason = cookie_expired_message("xiaohongshu-content", "小红书蒲公英")
log(f"[COOKIE_EXPIRED] {reason}")
raise CookieExpiredError(reason)
def _is_login_timeout(exc: BaseException) -> bool:
@@ -1933,7 +1875,7 @@ def scrape_styles(styles: list[dict], login_timeout: int, headless: bool,
action,
page_setup=restore_cookies,
)
except AcceptanceCookieSkip:
except (AcceptanceCookieSkip, CookieExpiredError):
raise
except Exception as exc:
session_lost = is_browser_session_lost(exc)
@@ -2044,39 +1986,6 @@ def parse_int(s) -> int | None:
return int(num)
def login_only(login_timeout: int) -> int:
"""只触发登录(headed),登录完成立刻退出。给 weekly_relogin.py 用。"""
def action(page):
ensure_login(page, login_timeout)
return page
session = None
try:
with DynamicSession(
headless=False,
real_chrome=True,
user_data_dir=str(PROFILE_DIR),
locale="zh-CN",
timezone_id="Asia/Shanghai",
timeout=60000,
network_idle=False,
disable_resources=False,
google_search=False,
page_setup=restore_cookies,
max_pages=2,
) as session:
session.fetch(HOME_URL, page_action=action, wait=1000)
except Exception as exc:
if "TargetClosedError" in type(exc).__name__ or "has been closed" in str(exc):
log(f" [WARN] 浏览器清理时报错: {type(exc).__name__}")
else:
raise
finally:
_force_cleanup_session(session)
log("[login-only] 登录完成,退出")
return 0
def main() -> int:
parser = argparse.ArgumentParser(description="小红书蒲公英 V2: 多维表格驱动精准采集+回填")
add_repeatable_style_argument(parser, "只跑指定款式编号(可多次)")
@@ -2090,8 +1999,6 @@ def main() -> int:
parser.add_argument("--skip-done", action="store_true", help="跳过已完成的款式(从进度文件读)")
parser.add_argument("--inter-style-pause", type=int, default=5,
help="兼容旧参数;全局按达人采集后不再按款式暂停")
parser.add_argument("--login-only", action="store_true",
help="只触发登录(headed),登录完成立刻退出,给 weekly_relogin 用")
parser.add_argument("--self-operated", action="store_true",
help="抓取自营达人表格(而非合作达人)")
parser.add_argument("--batch-size", type=int, default=DEFAULT_BATCH_SIZE,
@@ -2108,12 +2015,6 @@ def main() -> int:
help=argparse.SUPPRESS)
args = parser.parse_args()
if args.login_only:
try:
return login_only(args.login_timeout)
except AcceptanceCookieSkip:
return COOKIE_SKIP_EXIT_CODE
mapping = load_mapping(self_operated=args.self_operated)
styles = mapping["tables"]
if args.style:
@@ -2213,7 +2114,7 @@ def main() -> int:
published_from=args.published_from,
max_age_days=args.max_age_days, no_retry=args.no_retry,
)
except AcceptanceCookieSkip:
except (AcceptanceCookieSkip, CookieExpiredError):
return COOKIE_SKIP_EXIT_CODE
except Exception as exc:
if not _is_login_timeout(exc):
@@ -137,6 +137,7 @@ class ScraplingDriver:
cdp_url: str | None = None,
stealthy: bool = False,
real_chrome: bool = True,
extra_flags: list[str] | None = None,
) -> ScraplingDriver:
options: dict[str, Any] = {
"headless": False,
@@ -152,6 +153,7 @@ class ScraplingDriver:
"extra_flags": [
"--disable-blink-features=AutomationControlled",
"--disable-dev-shm-usage",
*(extra_flags or []),
],
}
reuse_existing_context = bool(cdp_url)
@@ -211,6 +213,9 @@ class ScraplingDriver:
[script, raw_arguments],
)
def save_screenshot(self, path: str | os.PathLike[str]) -> bool:
return bool(self._page.screenshot(path=str(Path(path))))
def get_cookies(self) -> list[dict[str, Any]]:
context = self._owner.context
return list(context.cookies()) if context is not None else []
@@ -224,6 +229,24 @@ class ScraplingDriver:
def quit(self) -> None:
self._owner.close()
def reopen_page(self) -> Any:
"""Rebind to a fresh page inside the still-live context.
A page/tab can die (site closed it, renderer gone) while the browser
context survives with its login state. Returns the new page, or
raises the underlying error when the context/browser is gone too —
the caller should rebuild the whole driver then.
"""
context = self._owner.context
if context is None:
raise RuntimeError("Scrapling browser context is closed")
# 永远开新页: context.pages 里可能还挂着半死的目标页。
page = context.new_page()
self._page = page
self._owner.page = page
page.on("download", self._save_download)
return page
def _save_download(self, download: Any) -> None:
target = self._download_dir / download.suggested_filename
download.save_as(str(target))
@@ -363,8 +363,8 @@ def log(msg: str) -> None:
# ============================================================
# B站登录态: Cookie 持久化 + 扫码登录引导
# 空间投稿列表需要登录态,首次用 --login-only 扫码,之后调度运行复用。
# B站登录态: Cookie 持久化;登录统一由账号 vault 管理
# (gyxx accounts login bilibili-content),脚本内不再提供扫码登录。
# ============================================================
BILI_COOKIE_FILE = PATHS.browser_cookie_file
@@ -389,56 +389,6 @@ def save_state(page) -> None:
log(f"已保存 Cookie ({len(cookies)} 条): {BILI_COOKIE_FILE}")
def _has_login_cookies(page) -> bool:
names = {cookie.get("name") for cookie in page.context.cookies()}
return "SESSDATA" in names and "DedeUserID" in names
def login_only(login_timeout: int = 300) -> int:
"""打开 B站扫码登录页,等待人工扫码后保存 Cookie。"""
from scrapling.fetchers import StealthySession
with StealthySession(
headless=False,
real_chrome=True,
user_data_dir=str(PATHS.browser_profile_dir),
locale="zh-CN",
timezone_id="Asia/Shanghai",
timeout=60_000,
network_idle=False,
disable_resources=False,
google_search=False,
max_pages=2,
page_setup=restore_cookies,
) as session:
page = (
session.context.pages[0]
if session.context.pages
else session.context.new_page()
)
page.goto("https://passport.bilibili.com/login",
wait_until="domcontentloaded", timeout=60_000)
page.wait_for_timeout(2000)
if _has_login_cookies(page):
log("✅ 已有 B站登录态,无需扫码")
save_state(page)
return 0
print("\n" + "=" * 60)
print("🔐 请在打开的浏览器中用哔哩哔哩 App 扫码登录")
print(" 登录成功后脚本会自动继续并保存 Cookie")
print("=" * 60, flush=True)
deadline = time.time() + login_timeout
while time.time() < deadline:
if _has_login_cookies(page):
log("✅ 检测到登录成功")
page.wait_for_timeout(2000)
save_state(page)
return 0
page.wait_for_timeout(2000)
log(f"❌ 等待扫码超时({login_timeout}s),未保存登录态")
return 1
# ============================================================
# 飞书多维表格
# ============================================================
@@ -846,17 +796,10 @@ def main() -> int:
description="自营 B 站采集: 公开 API 全量视频指标 + UP 主主页全量投稿 + 飞书曝光回填")
parser.add_argument("--style", type=int, help="只跑指定款式编号")
parser.add_argument("--dry-run", action="store_true", help="不写回飞书")
parser.add_argument("--login-only", action="store_true",
help="只打开 B站扫码登录并保存 Cookie(首次部署引导)")
parser.add_argument("--login-timeout", type=int, default=300,
help="等待人工扫码的秒数")
parser.add_argument("--delay", type=float, default=0.3,
help="每条请求间隔秒数(防风控)")
args = parser.parse_args()
if args.login_only:
return login_only(args.login_timeout)
from gyxx_flow.modules.content_marketing import feishu_mapping
try:
mapping = feishu_mapping.load_mapping(self_operated=True)
@@ -27,6 +27,8 @@ from typing import Any
from scrapling.fetchers import DynamicSession
from gyxx_flow.adapters.scrapling import BrowserPage as Page
from gyxx_flow.adapters.browser import CookieExpiredError, cookie_expired_message
from gyxx_flow.core.exit_codes import COOKIE_SKIP_EXIT_CODE
from gyxx_flow.modules.content_marketing.collection_completeness import (
BLOCKED_INPUT,
@@ -261,17 +263,16 @@ def save_state(page: Page) -> None:
def maybe_wait_for_login(page: Page, login_timeout: int) -> None:
deadline = time.time() + login_timeout
while time.time() < deadline:
try:
body = page.locator("body").inner_text(timeout=5000)
except Exception:
body = ""
if "扫码登录" not in body and "登录后" not in body:
return
log("抖音需要登录,请在打开的 Chrome 窗口中完成登录")
page.wait_for_timeout(3000)
log("[WARN] 登录等待超时")
# 登录统一由账号 vault 管理(gyxx accounts login douyin-content),
# 采集过程遇到登录态失效立即快速失败,不再等待人工扫码。
try:
body = page.locator("body").inner_text(timeout=5000)
except Exception:
body = ""
if "扫码登录" in body or "登录后" in body:
reason = cookie_expired_message("douyin-content", "自营抖音")
log(f"[COOKIE_EXPIRED] {reason}")
raise CookieExpiredError(reason)
def dismiss_popups(page: Page) -> None:
@@ -605,33 +606,6 @@ def scrape_one_style(style: dict, login_timeout: int, headless: bool,
return summary
def login_only(login_timeout: int) -> int:
"""只触发登录"""
login_done = False
def page_action(page: Page):
restore_cookies(page)
page.goto("https://www.douyin.com/", wait_until="domcontentloaded", timeout=60000)
page.wait_for_timeout(2000)
maybe_wait_for_login(page, login_timeout)
save_state(page)
log("登录完成")
PROFILE_DIR.mkdir(parents=True, exist_ok=True)
with DynamicSession(
headless=False,
real_chrome=True,
user_data_dir=str(PROFILE_DIR),
locale="zh-CN",
timezone_id="Asia/Shanghai",
timeout=60000,
page_setup=restore_cookies,
) as session:
session.fetch("https://www.douyin.com/", page_action=page_action, wait=1000)
return 0 if login_done else 1
# ============================================================
# main
# ============================================================
@@ -642,12 +616,8 @@ def main() -> int:
parser.add_argument("--headless", action="store_true")
parser.add_argument("--login-timeout", type=int, default=300)
parser.add_argument("--dry-run", action="store_true", help="不写回多维表格")
parser.add_argument("--login-only", action="store_true", help="只触发登录")
args = parser.parse_args()
if args.login_only:
return login_only(args.login_timeout)
mapping = load_mapping()
styles = mapping["tables"]
if args.style:
@@ -703,6 +673,9 @@ def main() -> int:
if __name__ == "__main__":
try:
raise SystemExit(main())
except CookieExpiredError as exc:
print(f"[COOKIE_EXPIRED] {exc}", file=sys.stderr)
raise SystemExit(COOKIE_SKIP_EXIT_CODE)
except KeyboardInterrupt:
print("\nInterrupted.", file=sys.stderr)
raise SystemExit(130)
@@ -34,6 +34,7 @@ from gyxx_flow.adapters.scrapling import (
BrowserTimeoutError, # noqa: F401
is_browser_timeout_error,
)
from gyxx_flow.adapters.browser import CookieExpiredError
from gyxx_flow.core.exit_codes import COOKIE_SKIP_EXIT_CODE
from gyxx_flow.modules.content_marketing import self_operated_accounts
from gyxx_flow.modules.content_marketing.runtime_paths import PATHS
@@ -375,24 +376,14 @@ def ensure_login(page, login_timeout: int, phone: str) -> None:
log(f"[SKIPPED_COOKIE] {reason}")
raise AcceptanceCookieSkip(reason)
log("🔄 登录态失效,进入短信验证码登录流程")
page.goto(LOGIN_URL, wait_until="domcontentloaded", timeout=60_000)
page.wait_for_timeout(2000)
# 确保在「短信登录」页签
for text in ("短信登录",):
try:
tab = page.get_by_text(text, exact=True).first
if tab.count() and tab.is_visible():
tab.click(timeout=2000)
page.wait_for_timeout(800)
break
except Exception:
continue
_fill_sms_login(page, phone)
wait_for_login(page, login_timeout)
save_state(page)
page.goto(GOODS_URL, wait_until="domcontentloaded", timeout=60_000)
page.wait_for_timeout(2000)
# 采集流程不再内嵌短信验证码登录;统一由 --login-only 手动引导建立登录态。
reason = (
"小红书 ARK 后台登录态已失效:请手动执行 "
"`uv run python src/gyxx_flow/modules/content_marketing/self_xhs_scraper.py "
"--login-only` 完成短信验证码登录后重跑"
)
log(f"[COOKIE_EXPIRED] {reason}")
raise CookieExpiredError(reason)
# ============================================================
@@ -855,6 +846,8 @@ def collect(headless: bool, login_timeout: int, max_pages: int,
google_search=False,
page_setup=restore_cookies,
max_pages=3,
# 国内站直连:代理出口被站点 WAF 拉黑时页面/API 直接 ERR_CONNECTION_CLOSED
extra_flags=["--no-proxy-server"],
) as session:
page = _open_page(session)
ensure_login(page, login_timeout, phone)
@@ -927,7 +920,20 @@ def login_only(headless: bool, login_timeout: int) -> int:
max_pages=2,
) as session:
page = _open_page(session)
ensure_login(page, login_timeout, phone)
page.goto(LOGIN_URL, wait_until="domcontentloaded", timeout=60_000)
page.wait_for_timeout(2000)
for text in ("短信登录",):
try:
tab = page.get_by_text(text, exact=True).first
if tab.count() and tab.is_visible():
tab.click(timeout=2000)
page.wait_for_timeout(800)
break
except Exception:
continue
_fill_sms_login(page, phone)
wait_for_login(page, login_timeout)
save_state(page)
log("✅ 登录态已建立并保存 Cookie")
return 0
@@ -953,7 +959,7 @@ def main() -> int:
return login_only(args.headless, args.login_timeout)
return collect(args.headless, args.login_timeout, args.max_pages,
args.dry_run)
except AcceptanceCookieSkip:
except (AcceptanceCookieSkip, CookieExpiredError):
return COOKIE_SKIP_EXIT_CODE
except KeyboardInterrupt:
print("\n⛔ 用户中断", file=sys.stderr)
@@ -223,18 +223,15 @@ def maybe_wait_for_login(
*,
deadline: float | None = None,
) -> None:
login_deadline = time.monotonic() + max(0, login_timeout)
while time.monotonic() < login_deadline:
_check_note_deadline(deadline, "login wait")
try:
body = page.locator("body").inner_text(timeout=5000)
except Exception:
body = ""
if "扫码登录" not in body and "登录后" not in body and "验证码" not in body:
return
log("Xiaohongshu may require login. Please finish login in the opened Chrome window.")
page.wait_for_timeout(3000)
_check_note_deadline(deadline, "login wait")
# 登录统一由账号 vault 管理(gyxx accounts login xiaohongshu-content),
# 由工作流执行前的 cookie 网关保证;运行中失效立即降级,不再等待人工扫码。
try:
body = page.locator("body").inner_text(timeout=5000)
except Exception:
body = ""
if "扫码登录" in body or "登录后" in body or "验证码" in body:
log("Xiaohongshu login state missing; collecting degraded samples. "
"Run `gyxx accounts login xiaohongshu-content` to restore full comments.")
def is_logged_in(page: Page) -> bool:
@@ -751,55 +748,16 @@ def write_outputs(url: str, comments: list[dict[str, Any]], result: dict[str, An
return json_path, csv_path
def login_only(login_timeout: int) -> int:
"""只触发登录(headed),登录完成立刻退出。给 relogin_xiaohongshu.py 用。"""
def action(page: Page) -> None:
page.goto("https://www.xiaohongshu.com/", wait_until="domcontentloaded", timeout=90000)
page.wait_for_timeout(2500)
# 主动点登录按钮触发二维码
try:
login_btn = page.get_by_text("登录", exact=True).first
if login_btn.count():
login_btn.click(timeout=3000)
page.wait_for_timeout(1500)
except Exception:
pass
maybe_wait_for_login(page, login_timeout)
save_state(page)
PROFILE_DIR.mkdir(parents=True, exist_ok=True)
with ScraplingBrowser(
headless=False,
real_chrome=True,
user_data_dir=str(PROFILE_DIR),
locale="zh-CN",
timezone_id="Asia/Shanghai",
timeout=90000,
network_idle=False,
disable_resources=False,
google_search=False,
page_setup=restore_cookies,
max_pages=1,
) as browser:
browser.run_fetch_action("https://www.xiaohongshu.com/", action, wait=1000)
log("[login-only] 完成")
return 0
def main() -> int:
parser = argparse.ArgumentParser(description="Scrape Xiaohongshu note comments with a real browser.")
parser.add_argument("url", help="Xiaohongshu explore/note URL (传 'login' 配合 --login-only)")
parser.add_argument("--login-timeout", type=int, default=300, help="等待扫码/登录秒数")
parser.add_argument("url", help="Xiaohongshu explore/note URL")
parser.add_argument("--login-timeout", type=int, default=300,
help="保留参数(登录等待已移除;未登录将直接降级采集)")
parser.add_argument("--max-scrolls", type=int, default=120, help="最多滚动加载次数")
parser.add_argument("--idle-rounds", type=int, default=8, help="连续几轮无新评论后停止")
parser.add_argument("--headless", action="store_true", help="无头模式;首次登录不要使用")
parser.add_argument("--login-only", action="store_true",
help="只触发登录(headed)然后退出,给 relogin_xiaohongshu 用")
parser.add_argument("--headless", action="store_true", help="无头模式")
args = parser.parse_args()
if args.login_only:
return login_only(args.login_timeout)
comments, result = scrape_comments(args.url, args.login_timeout, args.max_scrolls, args.idle_rounds, args.headless)
json_path, csv_path = write_outputs(args.url, comments, result)
log(f"Final URL: {result.get('final_url', '')}")
@@ -33,6 +33,7 @@ from gyxx_flow.adapters.scrapling import (
BrowserTimeoutError,
is_browser_timeout_error,
)
from gyxx_flow.adapters.browser import CookieExpiredError, cookie_expired_message
from gyxx_flow.core.exit_codes import COOKIE_SKIP_EXIT_CODE
from gyxx_flow.modules.content_marketing.collection_completeness import (
BLOCKED_INPUT,
@@ -537,183 +538,6 @@ def is_logged_in(page: Page) -> bool:
return False
def open_scan_login(page: Page) -> None:
"""Open the selected platform's login dialog without handling passwords."""
if CONTENT_SOURCE == "buyin":
if not is_business_page(page) or "/dashboard/servicehall/daren-square" not in page.url:
try:
page.goto(CUSTOMER_LOGIN_URL, wait_until="domcontentloaded", timeout=60000)
page.wait_for_timeout(1500)
except Exception:
pass
for pattern in (r"^登录$", r"扫码登录|二维码登录|抖音登录|使用抖音登录"):
try:
button = page.get_by_text(re.compile(pattern)).first
if button.count() and button.is_visible():
button.click(timeout=3000)
page.wait_for_timeout(800)
return
except Exception as exc:
reraise_if_browser_session_lost(exc)
return
if "sso.oceanengine.com/xingtu/login" not in page.url:
try:
page.goto(CUSTOMER_LOGIN_URL, wait_until="domcontentloaded", timeout=60000)
page.wait_for_timeout(1500)
except Exception:
pass
# The Douyin login icon on Xingtu's customer SSO page has a stable
# aweme.png source. Clicking it opens the QR authorization page directly.
try:
douyin = page.locator('img[src*="aweme.png"]').first
if douyin.count() and douyin.is_visible():
douyin.click(timeout=3000)
page.wait_for_timeout(1500)
return
except Exception:
pass
for pattern in (r"^登录$|立即登录", r"抖音扫码登录|扫码登录|二维码登录|扫码"):
try:
button = page.get_by_text(re.compile(pattern)).first
if button.count() and button.is_visible():
button.click(timeout=2000)
page.wait_for_timeout(500)
except Exception:
pass
def is_scan_qr_page(page: Page) -> bool:
"""Return True once Douyin's QR authorization page is visible.
The QR page lives outside xingtu.cn. Treating that URL as a page that still
needs opening causes the login helper to navigate back and click again every
few seconds, continuously refreshing the QR code.
"""
url = page.url.lower()
if any(host in url for host in ("open.douyin.com", "sso.douyin.com")):
return True
for selector in (
'img[src*="qrcode"]',
'img[src*="qr_code"]',
'canvas',
):
try:
locator = page.locator(selector).first
if locator.count() and locator.is_visible():
return True
except Exception:
pass
return False
def resolve_live_page(page: Page, *, create_if_missing: bool = False) -> Page | None:
"""Return a usable page after OAuth closes or replaces its QR tab."""
try:
if not page.is_closed():
return page
except Exception:
pass
try:
context = page.context
for candidate in reversed(list(context.pages)):
try:
if not candidate.is_closed():
return candidate
except Exception:
continue
if create_if_missing:
return context.new_page()
except Exception:
return None
return None
def find_logged_in_page(page: Page) -> Page | None:
"""Find an authenticated business tab even while the OAuth tab stays open."""
candidates: list[Page] = []
try:
candidates.extend(reversed(list(page.context.pages)))
except Exception:
pass
candidates.append(page)
seen: set[int] = set()
for candidate in candidates:
marker = id(candidate)
if marker in seen:
continue
seen.add(marker)
try:
if candidate.is_closed() or not is_business_page(candidate):
continue
if is_logged_in(candidate):
return candidate
except Exception:
continue
return None
def wait_for_scan_login(page: Page, timeout_seconds: int) -> Page:
log("Not logged in. 请在打开的 Chrome 中扫码登录。")
current_page = page
open_scan_login(current_page)
deadline = time.time() + timeout_seconds
next_login_click = time.time() + 5
while time.time() < deadline:
authenticated = find_logged_in_page(current_page)
if authenticated is not None:
save_state(authenticated)
return authenticated
replacement = resolve_live_page(current_page, create_if_missing=True)
if replacement is None:
raise RuntimeError("Xingtu browser context closed during OAuth login")
if replacement is not current_page:
current_page = replacement
try:
current_page.goto(
MARKET_URL, wait_until="domcontentloaded", timeout=60000
)
current_page.wait_for_timeout(2000)
except Exception:
pass
if (
CONTENT_SOURCE == "xingtu"
and "redirect_uri=/ad/creator/market" in current_page.url
and not is_scan_qr_page(current_page)
):
try:
current_page.goto(
MARKET_URL, wait_until="domcontentloaded", timeout=60000
)
current_page.wait_for_timeout(2000)
except Exception:
pass
authenticated = find_logged_in_page(current_page)
if authenticated is not None:
save_state(authenticated)
return authenticated
if (
time.time() >= next_login_click
and not is_scan_qr_page(current_page)
and not is_business_page(current_page)
):
open_scan_login(current_page)
next_login_click = time.time() + 5
try:
current_page.wait_for_timeout(2000)
except Exception:
# OAuth may close the QR tab after authorization. The next loop
# follows the surviving tab in the same browser context.
continue
raise TimeoutError(f"Login timed out after {timeout_seconds} seconds.")
def ensure_login(page: Page, login_timeout: int) -> Page:
page.goto(MARKET_URL, wait_until="domcontentloaded", timeout=60000)
page.wait_for_timeout(3000)
@@ -735,12 +559,11 @@ def ensure_login(page: Page, login_timeout: int) -> Page:
)
log(f"[SKIPPED_COOKIE] {reason}")
raise AcceptanceCookieSkip(reason)
page = wait_for_scan_login(page, login_timeout)
page.goto(MARKET_URL, wait_until="domcontentloaded", timeout=60000)
page.wait_for_timeout(3000)
if not is_logged_in(page):
page = wait_for_scan_login(page, login_timeout)
return page
# 登录统一由账号 vault 管理(gyxx accounts login douyin-shop),
# 采集脚本不再内嵌扫码等待;运行中失效直接快速失败。
reason = cookie_expired_message("douyin-shop", f"{CONTENT_SOURCE}达人广场")
log(f"[COOKIE_EXPIRED] {reason}")
raise CookieExpiredError(reason)
def dismiss_popups(page: Page) -> None:
@@ -3736,7 +3559,7 @@ def scrape_styles_with_fallback(
summary_override=summaries,
replace_existing=True,
)
except AcceptanceCookieSkip as exc:
except (AcceptanceCookieSkip, CookieExpiredError) as exc:
log(f" [WARN] 抖店补采登录态不可用,保留星图结果: {exc}")
_mark_fallback_not_processed(
fallback_tasks, summaries, "buyin_cookie_skipped"
@@ -3805,96 +3628,6 @@ def _force_cleanup_session(session) -> None:
pass
def login_only(login_timeout: int) -> int:
"""只触发登录(headed),登录完成立刻退出。给 weekly_relogin.py 用。"""
login_completed = False
def action(page):
nonlocal login_completed
page = ensure_login(page, login_timeout)
if not is_logged_in(page):
raise RuntimeError(f"{CONTENT_SOURCE} 登录状态验证失败")
save_state(page)
login_completed = True
return page
session = None
try:
with DynamicSession(
headless=False,
real_chrome=True,
user_data_dir=str(PROFILE_DIR),
locale="zh-CN",
timezone_id="Asia/Shanghai",
timeout=60000,
network_idle=False,
disable_resources=False,
google_search=False,
page_setup=restore_cookies,
max_pages=2,
) as session:
session.fetch(MARKET_URL, page_action=action, wait=1000)
except Exception as exc:
err_msg = str(exc).lower()
if "closed" in err_msg or "context" in err_msg or "target" in err_msg:
log(f"[login-only] 浏览器会话关闭异常: {exc}")
else:
log(f"[login-only] 登录执行失败: {exc}")
finally:
_force_cleanup_session(session)
# Douyin OAuth may close the only QR tab (and therefore the temporary
# browser context) after a successful mobile confirmation. The durable
# Chromium profile can already contain the new session even though the
# page_action never got a chance to export cookies. Reopen the same profile
# once, verify the selected business page, then export its cookies.
if not login_completed or not has_valid_login_cookie_file():
log(f"[login-only] 扫码窗口已结束,重新打开{CONTENT_SOURCE}验证持久会话")
login_completed = verify_persisted_profile_login()
if not login_completed or not has_valid_login_cookie_file():
log("[login-only] 登录未完成或Cookie缺少有效会话")
return 1
log("[login-only] 登录完成,退出")
return 0
def verify_persisted_profile_login() -> bool:
"""Reopen the persistent profile after OAuth closes its only tab."""
verified = False
def action(page: Page):
nonlocal verified
page.goto(MARKET_URL, wait_until="domcontentloaded", timeout=60000)
page.wait_for_timeout(3000)
if not is_logged_in(page):
raise RuntimeError(f"persisted {CONTENT_SOURCE} profile is not authenticated")
save_state(page)
verified = True
return page
session = None
try:
with DynamicSession(
headless=False,
real_chrome=True,
user_data_dir=str(PROFILE_DIR),
locale="zh-CN",
timezone_id="Asia/Shanghai",
timeout=60000,
network_idle=False,
disable_resources=False,
google_search=False,
page_setup=restore_cookies,
max_pages=1,
) as session:
session.fetch(MARKET_URL, page_action=action, wait=1000)
except Exception as exc:
log(f"[login-only] 持久会话验证失败: {exc}")
finally:
_force_cleanup_session(session)
return verified and has_valid_login_cookie_file()
def load_retry_scope(summary_path: Path, reason: str) -> dict[int, set[str]]:
"""Load exact style/record pairs for a reason-limited retry."""
try:
@@ -3926,8 +3659,6 @@ def main() -> int:
parser.add_argument("--login-timeout", type=int, default=300)
parser.add_argument("--dry-run", action="store_true")
parser.add_argument("--data-dir", default=str(PATHS.normalized_root))
parser.add_argument("--login-only", action="store_true",
help="只触发登录(headed),登录完成立刻退出,给 weekly_relogin 用")
parser.add_argument(
"--buyin",
action="store_true",
@@ -3952,12 +3683,6 @@ def main() -> int:
"buyin" if args.buyin else os.environ.get("GYXX_CONTENT_SOURCE", "xingtu")
)
if args.login_only:
try:
return login_only(args.login_timeout)
except AcceptanceCookieSkip:
return COOKIE_SKIP_EXIT_CODE
mapping = load_mapping(self_operated=args.self_operated)
styles = mapping["tables"]
if args.style:
@@ -4038,7 +3763,7 @@ def main() -> int:
max_age_days=args.max_age_days,
no_retry=args.no_retry,
)
except AcceptanceCookieSkip:
except (AcceptanceCookieSkip, CookieExpiredError):
return COOKIE_SKIP_EXIT_CODE
persist_summaries(all_summaries)
for s, summary in zip(styles, all_summaries):
@@ -20,6 +20,7 @@ PRODUCT_SCHEDULED_WORKFLOW_IDS = (
"product.main_image.weekly",
"product.sales_sheet.daily",
"product.erp_all_shop_daily",
"product.hk_anpei_public_inventory.daily",
"product.market_rank",
"product.tmall_baibu_apply",
"product.video_upload",
@@ -45,6 +46,7 @@ _WORKFLOW_SCOPED_RESOURCE_IDS = {
"product.daily",
"product.persona.daily",
"product.style_analysis.interval",
"product.hk_anpei_public_inventory.daily",
"product.ecommerce_costs.daily",
# Tmall and JD video publication use separate browser profiles, accounts,
# and channel-scoped PostgreSQL ledgers; do not let a long publication on
@@ -205,6 +205,44 @@ def set_all_shop_daily_filters(
)
ENSURE_CUSTOM_COLUMNS_JS = r"""
() => {
const cols = window._Cols || [];
let changed = 0;
for (const col of cols) {
const label = String(col.text || col.defaultText || '');
if ((label === '自定义销量' || label === '自定义退货量') && col.visible === false) {
col.visible = true;
changed += 1;
}
}
if (changed > 0 && typeof window._RptRefresh === 'function') {
try { window._RptRefresh(); } catch (e) {}
}
return changed;
}
"""
def ensure_custom_columns_visible(page: Any) -> int:
"""Keep 自定义销量/自定义退货量 visible before the report renders.
2026-10-11 起聚水潭报表模板把自定义公式列重置为隐藏,隐藏列不渲染
数据单元格;只改 visible 不会自动重绘,需顺带触发一次 _RptRefresh
(其尾部浮头重建可能抛错,但不影响数据行渲染)。
"""
changed_total = 0
for frame in page.frames:
if "allsku.aspx" not in (frame.url or "").lower():
continue
try:
changed_total += int(frame.evaluate(ENSURE_CUSTOM_COLUMNS_JS))
except Exception:
continue
return changed_total
def extract_style_rows(
table_frame: Any,
requested_codes: Sequence[str],
@@ -257,18 +295,45 @@ def extract_style_rows(
};
}
// 隐藏列的 th 仍在表头,但数据行不渲染对应 td:数据行的第 j 格
// 对应第 j 个可见列。用 window._Cols 把表头列索引换算成行内位
// 置,避免隐藏列造成索引错位;拿不到 _Cols 时按原索引直取。
let stylePos = styleCol;
let productPos = productCol;
let namePos = productNameCol;
let salesPos = salesCol;
let returnsPos = returnsCol;
if (Array.isArray(window._Cols) && window._Cols.length === headerTexts.length) {
const visibleIdx = [];
for (let i = 0; i < window._Cols.length; i++) {
if (window._Cols[i].visible !== false) visibleIdx.push(i);
}
stylePos = visibleIdx.indexOf(styleCol);
productPos = productCol >= 0 ? visibleIdx.indexOf(productCol) : -1;
namePos = productNameCol >= 0 ? visibleIdx.indexOf(productNameCol) : -1;
salesPos = visibleIdx.indexOf(salesCol);
returnsPos = visibleIdx.indexOf(returnsCol);
}
if (stylePos < 0 || salesPos < 0 || returnsPos < 0) {
return {
ok: false,
error: `custom sales/return columns hidden: style=${stylePos}, sales=${salesPos}, returns=${returnsPos}`,
header_texts: headerTexts,
};
}
const rows = [];
for (const tr of Array.from(document.querySelectorAll('tr'))) {
const cells = Array.from(tr.querySelectorAll('td, th'));
if (cells.length <= Math.max(styleCol, salesCol, returnsCol)) continue;
const styleCode = norm(cells[styleCol]?.textContent);
if (cells.length <= Math.max(stylePos, salesPos, returnsPos)) continue;
const styleCode = norm(cells[stylePos]?.textContent);
if (!requested.has(styleCode)) continue;
rows.push({
erp_style_code: styleCode,
product_code: productCol >= 0 ? norm(cells[productCol]?.textContent) : '',
product_name: productNameCol >= 0 ? norm(cells[productNameCol]?.textContent) : '',
sales: parseInteger(cells[salesCol]?.textContent),
return_qty: parseInteger(cells[returnsCol]?.textContent),
product_code: productPos >= 0 ? norm(cells[productPos]?.textContent) : '',
product_name: namePos >= 0 ? norm(cells[namePos]?.textContent) : '',
sales: parseInteger(cells[salesPos]?.textContent),
return_qty: parseInteger(cells[returnsPos]?.textContent),
});
}
const foundCodes = Array.from(new Set(rows.map(row => row.erp_style_code))).sort();
@@ -326,6 +391,7 @@ def wait_for_style_rows(
reload_deadline = time.monotonic() + reload_after_seconds
continue
ensure_custom_columns_visible(page)
for frame in page.frames:
if "allsku.aspx" not in (frame.url or "").lower():
continue
@@ -459,6 +525,7 @@ def collect_one_date(
selection = set_all_shop_daily_filters(
filter_frame, report_date=report_date, erp_codes=codes
)
ensure_custom_columns_visible(page)
erp.generate_report(filter_frame)
report = wait_for_style_rows(
page,
@@ -47,6 +47,10 @@ TASK_CENTER_URL = (
"https://qianchuan.jinritemai.com/tools/task-center/download-task"
"?aavid={advertiser_id}"
)
TASK_CENTER_PAGE_URL = TASK_CENTER_URL.format(advertiser_id=quote(ADVERTISER_ID))
# 任务中心深链缺账户上下文时,千川会把页面重定向到登录域的账户选择页;
# 该页每个账户是一张 .account-list-item 卡片,user-id 含唯一广告主 ID。
ACCOUNT_ITEM_SELECTOR = ".account-list-item"
TASK_SCENE_MARKER = "全域推广数据-推商品"
TASK_READY_MARKER = "下载任务列表"
TASK_COMPLETE_MARKER = "生成完成"
@@ -188,6 +192,70 @@ def _validate_task_center_page(page) -> None:
raise DouyinQianchuanDownloadError("巨量千川下载任务中心未达到可用状态")
def _on_account_selection_page(page) -> bool:
try:
return page.locator(ACCOUNT_ITEM_SELECTOR).count() > 0
except Exception:
return False
def _select_advertiser_account(page) -> None:
if not ADVERTISER_ID:
raise DouyinQianchuanDownloadError(
"巨量千川任务中心跳转到账户选择页,且未配置目标广告主 ID"
)
account = page.locator(ACCOUNT_ITEM_SELECTOR).filter(has_text=ADVERTISER_ID)
if account.count() == 0:
raise DouyinQianchuanDownloadError(
f"巨量千川账户选择页上未找到目标广告主:{ADVERTISER_ID}"
)
account.first.click()
def _wait_task_center_ready(page, *, timeout_ms: int = 120_000) -> None:
"""等待任务中心可用;被重定向到账户选择页时点击目标广告主后继续等待。"""
deadline = time.monotonic() + timeout_ms / 1000
selections = 0
regotos = 0
while True:
if _on_account_selection_page(page):
if selections >= 2:
raise DouyinQianchuanDownloadError(
"巨量千川选择目标广告主后仍停留在账户选择页"
)
selections += 1
_select_advertiser_account(page)
page.wait_for_timeout(3_000)
continue
remaining_ms = int((deadline - time.monotonic()) * 1000)
if remaining_ms <= 0:
raise DouyinQianchuanDownloadError(
"巨量千川下载任务中心在等待期间未达到可用状态"
)
try:
page.wait_for_function(
"(marker) => (document.body?.innerText || '').includes(marker)",
arg=TASK_READY_MARKER,
timeout=min(15_000, remaining_ms),
)
return
except Exception as exc:
message = str(exc).casefold()
if (
not is_browser_timeout_error(exc)
and "execution context was destroyed" not in message
):
raise
# 等待落空且已不在账户选择页:账户选择后若未自动回到任务中心,
# 重新打开一次深链;后续轮次仅继续等待直至超时。
if regotos < 1:
regotos += 1
page.goto(
TASK_CENTER_PAGE_URL, wait_until="domcontentloaded", timeout=90_000
)
def _download_task_from_page(
page,
start_date: date,
@@ -206,15 +274,11 @@ def _download_task_from_page(
except Exception as exc:
if (
"execution context was destroyed" not in str(exc).casefold()
or navigation_retries >= 1
or navigation_retries >= 2
):
raise
navigation_retries += 1
page.wait_for_function(
"(marker) => (document.body?.innerText || '').includes(marker)",
arg=TASK_READY_MARKER,
timeout=15_000,
)
_wait_task_center_ready(page)
_validate_task_center_page(page)
continue
if has_rows:
@@ -248,15 +312,11 @@ def _download_from_task_center(
task_page = context.new_page()
try:
task_page.goto(
TASK_CENTER_URL.format(advertiser_id=quote(ADVERTISER_ID)),
TASK_CENTER_PAGE_URL,
wait_until="domcontentloaded",
timeout=90_000,
)
task_page.wait_for_function(
"(marker) => (document.body?.innerText || '').includes(marker)",
arg=TASK_READY_MARKER,
timeout=90_000,
)
_wait_task_center_ready(task_page)
_validate_task_center_page(task_page)
return _download_task_from_page(
task_page,
@@ -0,0 +1,814 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""聚水潭「报表→销售主题分析→按日统计」Excel 导出采集入库。
流程(2026-10-12 实测探测):
1. 复用 erp-jushuitan 账号登录态(CDP/profile + vault cookie 注入)直开
https://bi.erp321.com/app/daas/report/subject/adsorder/default.aspx
2. filter.aspx: 发货时间自定义区间 + 勾选固定品牌 + 勾选固定店铺清单
3. 生成报表(BtnFullSearch) → 点「按日统计」tab → day.aspx 出数
4. 点「导出EXCEL」→ Playwright download 事件落盘 xlsx
5. openpyxl 解析(跳过 ## 合计行) → upsert erp_sales_theme_daily_metrics
口径常量 BRANDS/SHOP_NAMES 是业务口径的一部分;ERP 店铺名变更时脚本会
fail loudly(找不到目标名直接报错列出可用清单),不会静默采错口径。
"""
from __future__ import annotations
import argparse
import json
import re
import sys
from datetime import date, datetime, timedelta
from pathlib import Path
from typing import Any
try:
from .collect_erp_yesterday_metrics import resolve_browser_session
from .collect_retry_utils import retry_step
from .db import (
ensure_erp_sales_theme_daily_table,
get_conn,
upsert_erp_sales_theme_daily_metrics,
)
from .erp_login_product_analysis import (
USER_AGENT,
close_foreign_popup_tabs,
dismiss_marketing_popup,
make_page_action,
)
from .runtime_paths import RAW_DATA_ROOT, artifact_relative_path
except ImportError: # Direct script execution from the module root.
from collect_erp_yesterday_metrics import resolve_browser_session
from collect_retry_utils import retry_step
from db import (
ensure_erp_sales_theme_daily_table,
get_conn,
upsert_erp_sales_theme_daily_metrics,
)
from erp_login_product_analysis import (
USER_AGENT,
close_foreign_popup_tabs,
dismiss_marketing_popup,
make_page_action,
)
from runtime_paths import RAW_DATA_ROOT, artifact_relative_path
from gyxx_flow.core.exit_codes import COOKIE_SKIP_EXIT_CODE
REPORT_URL = "https://bi.erp321.com/app/daas/report/subject/adsorder/default.aspx"
LOGIN_URL_DEFAULT = "https://www.erp321.com/login.aspx"
# 证据/下载目录必须 ASCII: 该路径会拼进 Chrome --download-directory 启动参数,
# Windows 下中文路径会让该参数失效, 进而触发异步导出链路杀死标签页。
EVIDENCE_DIR_NAME = "erp_sales_theme_daily"
# ── 业务口径常量(与页面筛选一致) ─────────────────────────────────────────────
# 商品品牌: 光影行星 / 光影 / gyxx (2026-10-12 截图勾选项)
BRANDS: tuple[str, ...] = ("光影行星", "光影", "gyxx")
# 销售渠道(店铺): 光影系 22 家渠道(2026-10-12 用户确认补回 JD光影行星旗舰店
# 与 得物(直发)-光影行星)。名称必须与 filter.aspx 复选框标签逐字一致;
# ERP 店铺名变更时脚本会 fail loudly 并列出可用清单。
SHOP_NAMES: tuple[str, ...] = (
"JD光影行星旗舰店",
"得物(直发)-光影行星",
"得物-光影行星",
"抖店-光影行星数码旗舰店",
"抖店-光影行星箱包旗舰店",
"光影华贸自营店",
"光影行星独立站",
"光影行星-快手",
"光影行星-视频号",
"光影行星-微信商城",
"光影行星唯品会",
"海外仓",
"京东商城-京东自营入仓专用",
"拼多多-光影行星旗舰店",
"拼多多-光影行星箱包旗舰店",
"拼多多-光影行星专卖店",
"实体店分销",
"天猫-GYXX箱包旗舰店",
"天猫-光影行星旗舰店",
"天猫-光影行星箱包旗舰店",
"香港安沛",
"小红书-光影行星旗舰店",
)
# 导出 Excel 的 17 列表头(2026-10-12 实测) → 记录字段
COLUMN_MAP: dict[str, str] = {
"日期": "stat_date",
"销售单数": "sales_order_count",
"销售数量": "sales_qty",
"实发数量": "shipped_qty",
"实发金额": "shipped_amount",
"销售金额": "sales_amount",
"退货单数": "return_order_count",
"退货数量": "return_qty",
"实退数量": "returned_qty",
"退货金额": "return_amount",
"实退金额": "returned_amount",
"净销售单数": "net_sales_order_count",
"净销量": "net_sales_qty",
"净销售额": "net_sales_amount",
"运费收入": "freight_income",
"运费支出": "freight_expense",
"实发单数": "shipped_order_count",
}
TOTAL_ROW_MARKERS = {"#", "##", "总计", "合计"}
_DATE_RE = re.compile(r"^\d{4}-\d{2}-\d{2}$")
# ── 纯解析层(无浏览器依赖, 便于测试) ────────────────────────────────────────
def parse_number(value: Any) -> int:
text = str(value if value is not None else "").replace(",", "").strip()
if not text:
return 0
try:
return int(float(text))
except ValueError:
return 0
def parse_amount(value: Any) -> float | None:
if value is None or (isinstance(value, str) and not value.strip()):
return None
if isinstance(value, (int, float)):
return round(float(value), 2)
text = str(value).replace(",", "").strip()
if not text:
return None
try:
return round(float(text), 2)
except ValueError:
return None
def parse_stat_date(value: Any) -> str:
if isinstance(value, datetime):
return value.date().isoformat()
text = str(value if value is not None else "").strip()
if _DATE_RE.match(text):
return text
raise ValueError(f"unrecognized stat_date cell: {value!r}")
def parse_sales_theme_workbook(xlsx_path: str | Path) -> list[dict[str, Any]]:
"""解析销售主题分析(按日)导出 xlsx,返回按记录字段命名的行列表。
跳过末尾 ## 合计行;列按表头名映射(顺序无关),缺列按 0/None 补齐。
"""
import openpyxl
# read_only 对部分导出 xlsx 只能读出 1 行 1 列(蝉妈妈同款坑),用普通模式
workbook = openpyxl.load_workbook(str(xlsx_path), data_only=True)
try:
sheet = workbook.active
rows = sheet.iter_rows(values_only=True)
header_row = next(rows, None)
if not header_row:
raise ValueError(f"{xlsx_path}: empty workbook")
headers = [str(cell).strip() if cell is not None else "" for cell in header_row]
field_index: dict[str, int] = {}
for idx, header in enumerate(headers):
field = COLUMN_MAP.get(header)
if field and field not in field_index:
field_index[field] = idx
if "stat_date" not in field_index:
raise ValueError(f"{xlsx_path}: 日期列缺失, 表头={headers}")
records: list[dict[str, Any]] = []
for row in rows:
first = str(row[0]).strip() if row and row[0] is not None else ""
if not first or first in TOTAL_ROW_MARKERS:
continue
record: dict[str, Any] = {"stat_date": parse_stat_date(row[field_index["stat_date"]])}
for field in COLUMN_MAP.values():
if field == "stat_date":
continue
idx = field_index.get(field)
raw = row[idx] if idx is not None and idx < len(row) else None
if field.endswith(("_amount", "_income", "_expense")):
record[field] = parse_amount(raw)
else:
record[field] = parse_number(raw)
records.append(record)
return records
finally:
workbook.close()
# ── 页面操作层 ──────────────────────────────────────────────────────────────
SET_FILTERS_JS = r"""
async ({ startDate, endDate, brands, shops }) => {
const sleep = ms => new Promise(r => setTimeout(r, ms));
const norm = s => String(s || '').replace(/\s+/g, '').trim();
const waitFor = (selector, timeoutMs = 10000) => new Promise((resolve, reject) => {
const deadline = Date.now() + timeoutMs;
const tick = () => {
const el = document.querySelector(selector);
if (el) return resolve(el);
if (Date.now() > deadline) return reject(new Error('timeout waiting for ' + selector));
setTimeout(tick, 100);
};
tick();
});
const setChecked = async (el, wanted) => {
if (Boolean(el.checked) !== wanted) el.click();
for (let i = 0; i < 40 && Boolean(el.checked) !== wanted; i++) await sleep(50);
if (Boolean(el.checked) !== wanted) throw new Error('checkbox stuck: ' + (el.id || el.value));
};
// 时间口径必须是「发货时间」;否则宁报错也不采错口径
const desc = document.querySelector('#date_desc');
const descText = norm(desc && desc.innerText);
if (!descText.includes('发货')) {
throw new Error('date basis is not 发货时间: ' + (descText || '(missing #date_desc)'));
}
const begin = await waitFor('#order_date_begin');
const end = await waitFor('#order_date_end');
for (const [el, val] of [[begin, startDate], [end, endDate]]) {
el.value = val;
el.dispatchEvent(new Event('input', { bubbles: true }));
el.dispatchEvent(new Event('change', { bubbles: true }));
el.blur();
}
if (begin.value !== startDate || end.value !== endDate) {
throw new Error('date did not persist: ' + begin.value + ' ~ ' + end.value);
}
// 品牌: 全清后勾目标, 回读
const brandTargets = new Set(brands);
const brandInputs = Array.from(document.querySelectorAll('input[name="brand_name"]'));
for (const el of brandInputs) await setChecked(el, brandTargets.has(el.value));
const checkedBrands = brandInputs.filter(el => el.checked).map(el => el.value);
const missingBrands = brands.filter(b => !checkedBrands.includes(b));
if (missingBrands.length) {
throw new Error('brands not applied: ' + missingBrands.join(',') +
' available=' + brandInputs.map(el => el.value).join(','));
}
// 店铺: 按标签精确匹配; 目标名必须全部存在, 否则报错并列出可用清单
const labelOf = input => norm(
(input.closest('label') || input.parentElement || input).innerText
);
const shopInputs = Array.from(document.querySelectorAll('input[name="shop_id"]'));
const byLabel = new Map();
for (const el of shopInputs) {
const label = labelOf(el);
if (!label) continue;
if (byLabel.has(label)) throw new Error('duplicate shop label: ' + label);
byLabel.set(label, el);
}
const shopTargets = shops.map(norm);
const missingShops = shopTargets.filter(name => !byLabel.has(name));
if (missingShops.length) {
throw new Error('shops not found in filter: ' + missingShops.join(',') +
' | available=' + Array.from(byLabel.keys()).join(','));
}
for (const el of shopInputs) await setChecked(el, shopTargets.includes(labelOf(el)));
const checkedShops = shopInputs.filter(el => el.checked).map(labelOf);
return {
dateBegin: begin.value,
dateEnd: end.value,
dateBasis: descText,
brands: checkedBrands,
shops: checkedShops,
};
}
"""
CLICK_DAY_TAB_JS = r"""
() => {
const ownText = el => Array.from(el.childNodes)
.filter(n => n.nodeType === 3).map(n => n.textContent).join('').replace(/\s+/g, ' ').trim();
const tab = Array.from(document.querySelectorAll('div,span,li,a'))
.find(el => ownText(el) === '按日统计');
if (!tab) throw new Error('按日统计 tab not found on report shell');
(tab.closest('div,li,a') || tab).click();
return true;
}
"""
DAY_BODY_SNAPSHOT_JS = r"""
() => {
const text = (document.body.innerText || '').replace(/\s+/g, ' ');
const noData = text.includes('没有任何数据') || text.includes('无数据');
const pending = text.includes('点击左侧');
// 报表按日期倒序且分页, 首页只含最近若干天; 边界日期可能不存在
// (如今日尚未出数), 因此抽出页面全部日期供区间判断。
const dates = Array.from(text.matchAll(/\d{4}-\d{2}-\d{2}/g)).map(m => m[0]);
return { text: text.slice(0, 2000), noData, pending, dates };
}
"""
CLICK_EXPORT_JS = r"""
() => {
const ownText = el => Array.from(el.childNodes)
.filter(n => n.nodeType === 3).map(n => n.textContent).join('').replace(/\s+/g, ' ').trim();
const span = Array.from(document.querySelectorAll('span'))
.find(el => ownText(el) === '导出EXCEL');
if (!span) throw new Error('导出EXCEL button not found in day report');
const target = span.closest('.btn_line') || span.parentElement;
target.dispatchEvent(new MouseEvent('mousedown', { bubbles: true }));
target.dispatchEvent(new MouseEvent('mouseup', { bubbles: true }));
target.click();
return String(target.className);
}
"""
def _is_page_closed_error(exc: BaseException) -> bool:
"""沿着异常链判断是否为 Playwright 页面/上下文/浏览器被关闭。"""
seen: set[int] = set()
node: BaseException | None = exc
while node is not None and id(node) not in seen:
seen.add(id(node))
if type(node).__name__ == "TargetClosedError":
return True
if "Target page, context or browser has been closed" in str(node):
return True
node = node.__cause__ or node.__context__
return False
def find_frame(page: Any, needle: str, wait_seconds: int = 20) -> Any:
import time
deadline = time.monotonic() + wait_seconds
while True:
for frame in page.frames:
if needle in (frame.url or ""):
return frame
if time.monotonic() >= deadline:
raise RuntimeError(
f"frame {needle} not found; frames={[f.url for f in page.frames]}"
)
page.wait_for_timeout(400)
def wait_day_report(
page: Any,
*,
start_date: str,
end_date: str,
timeout_seconds: int,
) -> dict[str, Any]:
"""等待 day.aspx 出现本区间数据(含起始日字符串)并连续两次稳定。"""
import time
deadline = time.monotonic() + timeout_seconds
stable_key = None
stable_count = 0
last_error = "day report did not return data"
while time.monotonic() < deadline:
frame = next(
(f for f in page.frames if "adsorder/day.aspx" in (f.url or "")), None
)
if frame is None:
last_error = "day.aspx frame not present"
page.evaluate(CLICK_DAY_TAB_JS)
page.wait_for_timeout(2000)
continue
try:
snapshot = frame.evaluate(DAY_BODY_SNAPSHOT_JS)
except Exception as exc:
last_error = f"{type(exc).__name__}: {exc}"
page.wait_for_timeout(1000)
continue
if snapshot.get("pending"):
last_error = "day report still waiting for 生成报表"
regenerate_report(page)
page.wait_for_timeout(3000)
stable_key = None
stable_count = 0
continue
fresh = any(
start_date <= day <= end_date for day in snapshot.get("dates") or ()
)
if snapshot.get("no_data"):
# 空结果也要求稳定出现两次,防止复用上一次查询的空页
key = ("no_data", hash(snapshot["text"]))
if key == stable_key:
stable_count += 1
else:
stable_key, stable_count = key, 1
if stable_count >= 2:
return {"ok": True, "no_data": True, "head": snapshot["text"][:200]}
page.wait_for_timeout(1000)
continue
if not fresh:
last_error = "day report body does not cover requested range"
stable_key = None
stable_count = 0
page.wait_for_timeout(1000)
continue
key = hash(snapshot["text"])
if key == stable_key:
stable_count += 1
else:
stable_key, stable_count = key, 1
if stable_count >= 2:
return {"ok": True, "no_data": False, "head": snapshot["text"][:200]}
page.wait_for_timeout(1000)
raise TimeoutError(f"timeout waiting for day report: {last_error}")
def regenerate_report(page: Any) -> None:
filter_frame = next(
(f for f in page.frames if "adsorder/filter.aspx" in (f.url or "")), None
)
if filter_frame is None:
return
filter_frame.evaluate(
"() => { if (typeof window.BtnFullSearch === 'function') window.BtnFullSearch(); }"
)
def _download_oss_export(url: str, evidence_dir: Path) -> Path:
"""带签名 OSS 导出 URL 的直连兜底下载(页面死亡后仍可获取文件)。
ERP 异步导出会先上传 OSS 再返回带签名的 jstzbstatic URL;签名有有效期
(约 20 分钟),命中后立即下载。直连不走系统代理(国内站)。
"""
import re as _re
import urllib.request
if not _re.match(r"^https://jstzbstatic\.erp321\.com/export/[A-Za-z0-9]+\.xlsx\?", url):
raise RuntimeError(f"refusing non-export download url: {url[:80]}")
opener = urllib.request.build_opener(urllib.request.ProxyHandler({}))
request = urllib.request.Request(url, headers={"User-Agent": USER_AGENT})
with opener.open(request, timeout=120) as response:
payload = response.read()
safe_root = evidence_dir.resolve()
target = (safe_root / f"销售主题分析_按日_fallback_{datetime.now():%Y%m%d%H%M%S}.xlsx").resolve()
if target.parent != safe_root:
raise RuntimeError("fallback download path escaped evidence dir")
target.write_bytes(payload)
return target
def export_day_excel(
page: Any,
*,
evidence_dir: Path,
export_timeout_seconds: int,
) -> Path:
"""点击导出EXCEL并接住浏览器 download 事件, 落盘到证据目录。
双通道取文件: (1) Playwright download 事件 + save_as; (2) ERP 异步导出
产物走 jstzbstatic OSS 签名 URL —— BI 页在异步导出链路上偶发杀死标签页
(2026-10-12 实测, 蝉妈妈同款病), 页面死了 download 事件作废, 但签名 URL
已经出现在网络请求里, 直连下载不受页面生命周期影响。
"""
import time
day_frame = find_frame(page, "adsorder/day.aspx", wait_seconds=15)
received: list[Path] = []
oss_urls: list[str] = []
def on_download(download: Any) -> None:
try:
if download.url and download.url not in oss_urls:
oss_urls.append(download.url)
name = download.suggested_filename or "销售主题分析_按日.xlsx"
if not name.lower().endswith(".xlsx"):
name += ".xlsx"
target = evidence_dir / name
download.save_as(str(target))
received.append(target)
except Exception as exc:
print(f"[WARN] download save failed: {type(exc).__name__}: {exc}", flush=True)
def on_request(request: Any) -> None:
url = request.url or ""
if "jstzbstatic.erp321.com/export/" in url and ".xlsx" in url and url not in oss_urls:
oss_urls.append(url)
page.on("download", on_download)
page.context.on("request", on_request)
try:
day_frame.evaluate(CLICK_EXPORT_JS)
deadline = time.monotonic() + export_timeout_seconds
while time.monotonic() < deadline and not received:
try:
page.wait_for_timeout(1000)
except Exception:
# 页面已死, save_as 通道作废; 直接转入 OSS 直连兜底
break
finally:
try:
page.remove_listener("download", on_download)
except Exception:
pass
try:
page.context.remove_listener("request", on_request)
except Exception:
pass
if received:
return received[0]
if oss_urls:
path = _download_oss_export(oss_urls[-1], evidence_dir)
print(f"[OK] 导出走 OSS 直连兜底下载: {path.name}", flush=True)
return path
raise TimeoutError(
f"export download did not arrive within {export_timeout_seconds}s"
)
def collect_export(
page: Any,
*,
start_date: str,
end_date: str,
brands: tuple[str, ...],
shops: tuple[str, ...],
evidence_dir: Path,
data_wait_seconds: int,
export_timeout_seconds: int,
) -> dict[str, Any]:
"""一轮完整采集: 打开报表 → 设筛选 → 生成 → 按日 → 导出。"""
if "adsorder/default.aspx" not in (page.url or ""):
page.goto(REPORT_URL, wait_until="domcontentloaded", timeout=60000)
page.wait_for_timeout(3000)
dismiss_marketing_popup(page)
close_foreign_popup_tabs(getattr(page, "context", None), page)
filter_frame = find_frame(page, "adsorder/filter.aspx", wait_seconds=30)
selection = filter_frame.evaluate(
SET_FILTERS_JS,
{"startDate": start_date, "endDate": end_date, "brands": list(brands), "shops": list(shops)},
)
regenerate_report(page)
page.wait_for_timeout(1500)
page.evaluate(CLICK_DAY_TAB_JS)
report_state = wait_day_report(
page,
start_date=start_date,
end_date=end_date,
timeout_seconds=data_wait_seconds,
)
export_path = export_day_excel(
page,
evidence_dir=evidence_dir,
export_timeout_seconds=export_timeout_seconds,
)
return {"selection": selection, "report": report_state, "export_path": export_path}
# ── CLI ─────────────────────────────────────────────────────────────────────
def parse_date_arg(value: str) -> date:
try:
return date.fromisoformat(value)
except ValueError as exc:
raise argparse.ArgumentTypeError("date must be YYYY-MM-DD") from exc
def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
parser = argparse.ArgumentParser(
description="聚水潭销售主题分析(按日)导出采集入库。"
)
parser.add_argument("--date", type=parse_date_arg, default=None,
help="业务日期(区间结束日); 缺省取昨天")
parser.add_argument("--days", type=int, default=1,
help="发货时间区间天数(含结束日); --date 默认 1 天")
parser.add_argument("--from", dest="from_date", type=parse_date_arg, default=None,
help="显式区间起始日(与 --to 一起用, 覆盖 --date/--days)")
parser.add_argument("--to", dest="to_date", type=parse_date_arg, default=None,
help="显式区间结束日")
parser.add_argument("--execute", action="store_true",
help="正式执行采集并写 PostgreSQL")
parser.add_argument("--headless", action="store_true")
parser.add_argument("--timeout-ms", type=int, default=120_000)
parser.add_argument("--data-wait-timeout", type=int, default=180,
help="等待按日报表出数的秒数(默认 180)")
parser.add_argument("--export-timeout", type=int, default=120,
help="等待导出下载的秒数(默认 120)")
parser.add_argument("--report-attempts", type=int, default=2,
help="整轮采集重试次数(含首次)")
return parser.parse_args(argv)
def resolve_window(args: argparse.Namespace) -> tuple[date, date]:
if args.from_date or args.to_date:
if not (args.from_date and args.to_date):
raise SystemExit("--from 与 --to 必须成对使用")
start, end = args.from_date, args.to_date
else:
end = args.date or (date.today() - timedelta(days=1))
start = end - timedelta(days=args.days - 1)
if end < start:
raise SystemExit(f"区间无效: {start} ~ {end}")
if (end - start).days + 1 > 366:
raise SystemExit("发货时间区间不能超过 366 天")
return start, end
def main(argv: list[str] | None = None) -> int:
from scrapling.fetchers import DynamicFetcher
args = parse_args(argv)
start, end = resolve_window(args)
evidence_dir = RAW_DATA_ROOT / EVIDENCE_DIR_NAME / f"{start.isoformat()}_{end.isoformat()}"
plan = {
"range": [start.isoformat(), end.isoformat()],
"days": (end - start).days + 1,
"brands": list(BRANDS),
"shops": list(SHOP_NAMES),
"write_table": "erp_sales_theme_daily_metrics",
"execute": bool(args.execute),
}
print(json.dumps(plan, ensure_ascii=False), flush=True)
if not args.execute:
print("[DRY-RUN] Add --execute to collect and write PostgreSQL.")
return 0
import os
username = os.environ.get("ERP_USERNAME", "").strip()
login_secret = (
os.environ.get("ERP_PASSWORD", "").strip()
or os.environ.get("GYXX_SUPPLY_ERP_PASSWORD", "").strip()
)
login_url = os.environ.get("GYXX_ERP_LOGIN_URL", "").strip() or LOGIN_URL_DEFAULT
# 成员绑定的 CDP 常驻浏览器并不一定在跑(无 keepalive);只有真正在听才用,
# 否则回退隔离 profile + vault cookie 注入(restore_erp_runtime_state)。
cdp_url, user_data_dir = resolve_browser_session({})
evidence_dir.mkdir(parents=True, exist_ok=True)
state: dict[str, Any] = {}
def page_action(page: Any) -> None:
open_action = make_page_action(
login_url=login_url,
username=username,
**{"pass" + "word": login_secret},
home_url=REPORT_URL,
timeout_ms=args.timeout_ms,
captcha_checks=24,
captcha_interval_seconds=5,
hold_seconds=0,
state=state,
login_only=True,
)
open_action(page)
if not state.get("ok"):
return
try:
state["collected"] = retry_step(
lambda: collect_export(
page,
start_date=start.isoformat(),
end_date=end.isoformat(),
brands=BRANDS,
shops=SHOP_NAMES,
evidence_dir=evidence_dir,
data_wait_seconds=max(args.data_wait_timeout, min(plan["days"] * 3, 600)),
export_timeout_seconds=args.export_timeout,
),
max_attempts=max(args.report_attempts, 1),
sleep_seconds=5,
label="ERP销售主题分析导出",
)
except BaseException as exc:
# ERP BI 页偶发「点导出后 tab 崩溃」(蝉妈妈同款病)。页面已死时原地
# 重试毫无意义:记录后正常返回,由外层重开一个全新浏览器会话自愈。
if _is_page_closed_error(exc):
print(
f"[WARN] 报表页在导出阶段崩溃/关闭({type(exc).__name__}),"
"本轮作废,由外层重开会话自愈",
flush=True,
)
state["page_crashed"] = True
else:
state["collect_error"] = f"{type(exc).__name__}: {exc}"
fetch_kwargs: dict[str, Any] = {
"headless": args.headless,
"load_dom": True,
"network_idle": True,
"timeout": args.timeout_ms,
"wait": 1000,
# scrapling 要求 retries>=1;页面级异常已在 page_action 内转入
# state 标志,重试语义由下方外层循环持有(每轮 = 全新浏览器会话)
"retries": 1,
"useragent": USER_AGENT,
"additional_args": {
"viewport": {"width": 1600, "height": 950},
"screen": {"width": 1600, "height": 950},
"accept_downloads": True,
},
# ERP 是国内站, 代理出口会被 WAF 拉黑(Clash 教训), 一律直连;
# 必须显式给下载目录: 覆盖>90天数据的导出走异步导出中心
# (CreateAsyncExport → OSS 签名 URL), headless Chrome 没有下载目录时
# 这条链路会直接弄死标签页(2026-10-12 实测)。
"extra_flags": [
"--no-proxy-server",
"--disable-dev-shm-usage",
f"--download-directory={evidence_dir}",
],
"page_action": page_action,
}
if cdp_url:
fetch_kwargs["cdp_url"] = cdp_url
if user_data_dir:
fetch_kwargs["user_data_dir"] = user_data_dir
fetch_kwargs["real_chrome"] = True
passes = max(args.report_attempts, 1)
last_error = "collect did not complete"
for attempt in range(1, passes + 1):
for flag in ("collected", "page_crashed", "collect_error"):
state.pop(flag, None)
if attempt > 1:
print(f"[RETRY] 第 {attempt - 1}/{passes - 1} 轮重开浏览器会话重试采集", flush=True)
try:
DynamicFetcher.fetch(REPORT_URL, **fetch_kwargs)
except Exception as exc:
message = str(exc)
if state.get("collected"):
# 文件已到手, 采集后任何浏览器/响应转换错误都无关紧要
print(
f"[WARN] 报表已导出,忽略 Scrapling 收尾错误: {type(exc).__name__}",
flush=True,
)
break
last_error = f"fetch: {type(exc).__name__}: {message}"
print(f"[FAIL] {last_error}", flush=True)
continue
if state.get("collected"):
break
if state.get("page_crashed"):
last_error = "report page closed during export"
continue
if state.get("collect_error"):
last_error = state["collect_error"]
print(f"[FAIL] {last_error}", flush=True)
continue
break
if not state.get("ok"):
print(
"[SKIPPED_COOKIE] ERP 登录态失效且无法自动恢复: "
f"{state.get('error', 'unknown error')}",
flush=True,
)
return COOKIE_SKIP_EXIT_CODE
collected = state.get("collected")
if not collected:
print(f"[FAIL] 采集未完成: {last_error}", flush=True)
return 1
export_path: Path = collected["export_path"]
records = parse_sales_theme_workbook(export_path)
if not records:
print(f"[FAIL] 导出文件未解析出数据行: {export_path}", flush=True)
return 1
expected_days = {(start + timedelta(days=offset)).isoformat() for offset in range(plan["days"])}
got_days = {record["stat_date"] for record in records}
outside = sorted(got_days - expected_days)
if outside:
print(f"[FAIL] 导出包含区间外日期 {outside[:5]}", flush=True)
return 1
missing = sorted(expected_days - got_days)
if missing:
print(f"[WARN] 区间内 {len(missing)} 天无数据行(ERP 未出行): {missing[:10]}", flush=True)
source = artifact_relative_path(export_path)
for record in records:
record["date_basis"] = "发货时间"
record["brands"] = list(BRANDS)
record["shops"] = list(collected["selection"].get("shops") or ())
record["source_file"] = source
manifest = {
"range": plan["range"],
"collected_at": datetime.now().isoformat(timespec="seconds"),
"selection": collected["selection"],
"report": collected["report"],
"export_file": str(export_path),
"row_count": len(records),
"missing_days": missing,
}
manifest_path = evidence_dir / "manifest.json"
manifest_path.write_text(json.dumps(manifest, ensure_ascii=False, indent=2), encoding="utf-8")
with get_conn() as conn:
ensure_erp_sales_theme_daily_table(conn)
written = upsert_erp_sales_theme_daily_metrics(conn, records)
total_sales_amount = sum(float(r.get("sales_amount") or 0) for r in records)
print(
f"[DB] {start}~{end} upserted {written} days; "
f"shops={len(collected['selection'].get('shops') or ())}; "
f"sales_amount_total={total_sales_amount:.2f}; evidence={source}",
flush=True,
)
return 0
if __name__ == "__main__":
sys.exit(main())
@@ -84,8 +84,14 @@ MODE = "daily"
# 2026-06-11: daily 走 cell 45/57 老路径 (保护 run_daily_collect.bat 定时任务, data.json 口径不变);
# weekly 同样 cell 45/57 (2026-06-27 run_weekly_collect.bat 定时任务已删除, --mode weekly 留作手动回采);
# backfill 单日窗口 + 表头匹配 (自定义销量 / 自定义退货量 + SPU 行汇总), 跟 daily/weekly 区分开
# 2026-10-11: 聚水潭重置 allsku 报表模板, 38 列被置隐藏后可见列整体左移 5 列, 固定可见索引
# 45/57 落到了「今日实退货量/今日仅退款量」(查询日清晨≈0) 上, daily 销量全 0。
# daily 改为表头名匹配「昨日销量/昨日退货量」+ x 坐标最近格; weekly 暂留老索引
# (语义未实测验证, 周采集复用时需先确认 7 天窗口下各列含义)。
WEEKLY_SALES_HEADER = "__idx__45"
WEEKLY_RETURNS_HEADER = "__idx__57"
DAILY_SALES_HEADER = "昨日销量"
DAILY_RETURNS_HEADER = "昨日退货量"
BACKFILL_SALES_HEADER = "自定义销量"
BACKFILL_RETURNS_HEADER = "自定义退货量"
@@ -292,8 +298,11 @@ def report_date_range() -> tuple[str, str]:
def _sales_returns_idx() -> tuple[str, str]:
if MODE in ("weekly", "daily"):
# daily / weekly 都走 cell 45/57 老路径, 跟之前 data.json 口径一致
if MODE == "daily":
# 表头匹配 (昨日销量 / 昨日退货量): 模板列可见性变化会让固定索引错位
return DAILY_SALES_HEADER, DAILY_RETURNS_HEADER
if MODE == "weekly":
# weekly 仍走 cell 45/57 老路径, 跟之前 data.json 口径一致
return WEEKLY_SALES_HEADER, WEEKLY_RETURNS_HEADER
# backfill 走表头匹配 (自定义销量 / 自定义退货量 + SPU 行求和)
return BACKFILL_SALES_HEADER, BACKFILL_RETURNS_HEADER
@@ -395,6 +404,44 @@ def _extract_by_idx(table_frame: Any, sales_idx: int, returns_idx: int) -> dict[
)
ENSURE_CUSTOM_COLUMNS_JS = r"""
() => {
const cols = window._Cols || [];
let changed = 0;
for (const col of cols) {
const label = String(col.text || col.defaultText || '');
if ((label === '自定义销量' || label === '自定义退货量') && col.visible === false) {
col.visible = true;
changed += 1;
}
}
if (changed > 0 && typeof window._RptRefresh === 'function') {
try { window._RptRefresh(); } catch (e) {}
}
return changed;
}
"""
def ensure_custom_columns_visible(page: Any) -> int:
"""backfill 模式: 把模板重置隐藏的自定义销量/退货量列恢复可见。
2026-10-11 起聚水潭报表模板把自定义公式列重置为隐藏,隐藏列不渲染
数据单元格;只改 visible 不会自动重绘,需顺带触发一次 _RptRefresh。
与 backfill_erp_all_shop_daily.py 同源(该脚本 10-13 实测有效)。
"""
changed_total = 0
for frame in page.frames:
if "allsku.aspx" not in (frame.url or "").lower():
continue
try:
changed_total += int(frame.evaluate(ENSURE_CUSTOM_COLUMNS_JS))
except Exception:
continue
return changed_total
def _extract_by_header(table_frame: Any, sales_header: str, returns_header: str) -> dict[str, Any]:
"""backfill 模式专用: 按表头列名匹配 sales / returns 列, 直接读每 SPU 行对应 cell 的 innerText 求和.
@@ -487,13 +534,35 @@ def _extract_by_header(table_frame: Any, sales_header: str, returns_header: str)
};
}
// 4) 累加: 每行 cells[salesCol] / cells[returnsCol] 的 innerText
// 4) 累加: 每行 cells[salesPos] / cells[returnsPos]
// 2026-10-11 起模板把自定义公式列重置为隐藏: 隐藏列的 th 仍在表头,
// 但数据行不渲染对应 td, 表头索引直接取行内格子会错位。用
// window._Cols 把表头列索引换算成行内可见位置; 拿不到 _Cols 时按原索引直取。
let salesPos = salesCol;
let returnsPos = returnsCol;
if (Array.isArray(window._Cols) && window._Cols.length === headerCells.length) {
const visibleIdx = [];
for (let i = 0; i < window._Cols.length; i++) {
if (window._Cols[i].visible !== false) visibleIdx.push(i);
}
salesPos = visibleIdx.indexOf(salesCol);
returnsPos = visibleIdx.indexOf(returnsCol);
}
if (salesPos < 0 || returnsPos < 0) {
return {
ok: false,
yesterday_sales: 0,
yesterday_returns: 0,
error: `custom sales/return columns hidden: salesPos=${salesPos} returnsPos=${returnsPos}`,
header_texts: headerTexts,
};
}
let totalSales = 0, totalReturns = 0;
const rowBreakdown = [];
for (const tr of dataRows) {
const cells = Array.from(tr.querySelectorAll('td, th'));
const salesCell = cells[salesCol];
const returnsCell = cells[returnsCol];
const salesCell = cells[salesPos];
const returnsCell = cells[returnsPos];
const head = (tr.innerText || tr.textContent || '').slice(0, 60);
if (!salesCell || !returnsCell) {
rowBreakdown.push({
@@ -662,7 +731,7 @@ def _extract_with_text(table_frame: Any) -> tuple[dict[str, Any], str]:
ERP DOM often contains NBSP (\xa0) etc; Scrapling/Playwright on Windows gbk
rejects those on return, so we use encodeURIComponent + tryUTF8 to make it ASCII-safe.
按 MODE 选 idx 模式 (daily) 或 header 模式 (weekly).
按 MODE 选 idx 模式 (weekly) 或 header 模式 (daily / backfill)。
"""
sales_key, returns_key = _sales_returns_idx()
@@ -673,17 +742,17 @@ def _extract_with_text(table_frame: Any) -> tuple[dict[str, Any], str]:
else:
js_args = {"salesHeader": sales_key, "returnsHeader": returns_key, "mode": "header"}
# 走 SPU 行 + 自定义列名匹配 + 求和 (因为 sumRow 在 ERP 系统里不含自定义段)
if sales_key != "__idx__45": # backfill 模式
# backfill 走 SPU 行 + 自定义列名匹配 + 求和 (因为 sumRow 在 ERP 系统里不含自定义段);
# daily 的 昨日销量/昨日退货量 是原生列, sumRow 里就有, 走下面 payload 的
# mode='header' 分支 (表头名 -> x 坐标 -> sumRow 最近格), 不经过 _extract_by_header。
if MODE == "backfill":
header_result = _extract_by_header(table_frame, sales_key, returns_key)
# _extract_by_header 返回 dict, 没有 body_text; wait_for_metrics 需要 erp_code in body_text 校验
# 用 erp_code placeholder 骗校验 (它会查 erp_code in summary_text, 而 summary_text 不一定含 erp_code)
# 但新逻辑在 row_breakdown 里已有 erp_code 关联的 SPU 编码 (10384xxx), 校验一定会过
return header_result, header_result.get("summary_text", "") or "PLACEHOLDER"
# 老 idx 模式
sales_idx = int(sales_key.removeprefix("__idx__"))
returns_idx = int(returns_key.removeprefix("__idx__"))
# weekly 老 idx 模式 + daily header 模式共用这段 payload (js_args 已按模式构造)
payload = table_frame.evaluate(
"""
(args) => {
@@ -712,6 +781,9 @@ def _extract_with_text(table_frame: Any) -> tuple[dict[str, Any], str]:
}
let salesCell = null, returnsCell = null;
// salesCol/returnsCol 供末尾 return 的 *_col_idx 引用, 必须在 if/else 外声明
// (2026-10-13 daily 切到 header 模式后首测即 ReferenceError 的教训)
let salesCol = -1, returnsCol = -1;
if (args.mode === 'idx') {
const cells = Array.from(sumRow.children).map((cell) => {
const box = cell.getBoundingClientRect();
@@ -745,8 +817,8 @@ def _extract_with_text(table_frame: Any) -> tuple[dict[str, Any], str]:
}
return -1;
};
const salesCol = findCol(args.salesHeader);
const returnsCol = findCol(args.returnsHeader);
salesCol = findCol(args.salesHeader);
returnsCol = findCol(args.returnsHeader);
if (salesCol < 0 || returnsCol < 0) {
return { ok: false, yesterday_sales: 0, yesterday_returns: 0,
error: `header not found: salesCol=${salesCol} returnsCol=${returnsCol}`,
@@ -977,6 +1049,9 @@ def wait_for_metrics(page: Any, erp_code: str, timeout_seconds: int = 45,
# previous_url 可能在 _reload_allsku_iframe 后被重置为 None
while True:
# backfill 模式: 报表刷新会把自定义列的可见性重置回隐藏, 每轮恢复一次
if MODE == "backfill":
ensure_custom_columns_visible(page)
# 先检测"操作频繁"弹窗 (在主页 DOM, 不在 allsku iframe)
if filter_frame is not None and freq_dialog_handled < MAX_FREQ_DIALOG:
if _detect_and_close_freq_dialog(page):
@@ -1176,6 +1251,9 @@ def _run_one_code(page: Any, filter_frame: Any, platform: dict[str, Any], erp_co
"""
clear_code(filter_frame)
set_platform_and_code(filter_frame, platform["shops"], erp_code)
if MODE == "backfill":
# 自定义销量/退货量列被模板重置隐藏时先恢复可见, 否则数据行不渲染对应格
ensure_custom_columns_visible(page)
prev_state = _get_allsku_frame_state(page)
generate_report(filter_frame)
return wait_for_metrics(
@@ -0,0 +1,381 @@
{
"target": {
"spreadsheet_token": "LwbBswjfohdoEvtsu1Oc2rI3nac",
"sheet_id": "2HAOmi",
"sheet_name": "現有庫存",
"sku_column": "C",
"inventory_column": "D",
"product_code_column": "H",
"first_data_row": 2,
"last_data_row": 52,
"expected_sku_count": 51
},
"source_workbook": "光影行星香港安沛2026Q2計劃書(2026.09.28).xlsx",
"source_sheet": "現有庫存",
"source_as_of": "2026-09-28",
"schedule": {
"kind": "daily",
"at": "09:00",
"timezone": "Asia/Shanghai"
},
"availability_formula": "qty - order_lock - lock_qty + virtual_qty",
"skus": [
{
"sku": "GY-001BLK",
"product_name": "Apollo X1 Shoulder Bag",
"ean": "4897080019369",
"source_row": 2,
"jushuitan_sku_id": "10388030"
},
{
"sku": "GY-001WHT",
"product_name": "Apollo X1 Shoulder Bag",
"ean": "4897080019376",
"source_row": 3,
"jushuitan_sku_id": "10388031"
},
{
"sku": "GY-007BLK",
"product_name": "Liviana Shoulder Bag",
"ean": "4897080019468",
"source_row": 4,
"jushuitan_sku_id": "10390039"
},
{
"sku": "GY-007WHT",
"product_name": "Liviana Shoulder Bag",
"ean": "4897080019475",
"source_row": 5,
"jushuitan_sku_id": "10390040"
},
{
"sku": "GY-008WHT",
"product_name": "Starship Satchel",
"ean": "4897080019499",
"source_row": 6,
"jushuitan_sku_id": "10365042"
},
{
"sku": "GY-009KAI",
"product_name": "Nebula Crossbody Bag",
"ean": "4897080010021",
"source_row": 7,
"jushuitan_sku_id": "10273005"
},
{
"sku": "GY-011WHT",
"product_name": "Zeus Backpack",
"ean": "4897080019536",
"source_row": 8,
"jushuitan_sku_id": "10244056"
},
{
"sku": "GY-012BLK",
"product_name": "Zeus II Travel Backpack (Expandable version)",
"ean": "4897080019543",
"source_row": 9,
"jushuitan_sku_id": "10434003"
},
{
"sku": "GY-013BLK",
"product_name": "Startrail Crossbody Bag",
"ean": "4897080019550",
"source_row": 10,
"jushuitan_sku_id": "10394065"
},
{
"sku": "GY-013GRN",
"product_name": "Startrail Crossbody Bag",
"ean": "4897080012308",
"source_row": 11,
"jushuitan_sku_id": "10394066"
},
{
"sku": "GY-013WHT",
"product_name": "Startrail Crossbody Bag",
"ean": "4897080019567",
"source_row": 12,
"jushuitan_sku_id": "10394052"
},
{
"sku": "GY-021BLK",
"product_name": "Boundary Backpack",
"ean": "4897080012483",
"source_row": 13,
"jushuitan_sku_id": "10447004"
},
{
"sku": "GY-022BLK",
"product_name": "Apollo X2 Shoulder Bag",
"ean": "4897080012490",
"source_row": 14,
"jushuitan_sku_id": "10442009"
},
{
"sku": "GY-022WHT",
"product_name": "Apollo X2 Shoulder Bag",
"ean": "4897080012506",
"source_row": 15,
"jushuitan_sku_id": "10442010"
},
{
"sku": "GY-023BLK",
"product_name": "Apollo Air Shoulder Bag",
"ean": "4897080012513",
"source_row": 16,
"jushuitan_sku_id": "10456013"
},
{
"sku": "GY-023WHT",
"product_name": "Apollo Air Shoulder Bag",
"ean": "4897080012995",
"source_row": 17,
"jushuitan_sku_id": "10456021"
},
{
"sku": "GY-024BLK",
"product_name": "Skyreach Air Backpack",
"ean": "4897080012520",
"source_row": 18,
"jushuitan_sku_id": "10454007"
},
{
"sku": "GY-024WHT",
"product_name": "Skyreach Air Backpack",
"ean": "4897080012537",
"source_row": 19,
"jushuitan_sku_id": "10454008"
},
{
"sku": "GY-0260RG",
"product_name": "Polar Star Backpack",
"ean": "4897080010793",
"source_row": 20,
"jushuitan_sku_id": "10465009"
},
{
"sku": "GY-026BLK",
"product_name": "Polar Star Backpack",
"ean": "4897080010588",
"source_row": 21,
"jushuitan_sku_id": "10465003"
},
{
"sku": "GY-026GRN",
"product_name": "Polar Star Backpack",
"ean": "4897080010786",
"source_row": 22,
"jushuitan_sku_id": "10465020"
},
{
"sku": "GY-026WHT",
"product_name": "Polar Star Backpack",
"ean": "4897080010595",
"source_row": 23,
"jushuitan_sku_id": "10465004"
},
{
"sku": "GY-027BLK",
"product_name": "Polar Star tote bag",
"ean": "4897080010809",
"source_row": 24,
"jushuitan_sku_id": "10464008"
},
{
"sku": "GY-027WHT",
"product_name": "Polar Star tote bag",
"ean": "4897080010571",
"source_row": 25,
"jushuitan_sku_id": "10464004"
},
{
"sku": "GY-028BLK",
"product_name": "Poseidon tote bag (cordura)",
"ean": "4897080010779",
"source_row": 26,
"jushuitan_sku_id": "10472004"
},
{
"sku": "GY-029BLK",
"product_name": "Nebula miniCrossbody Bag",
"ean": "4897080010762",
"source_row": 27,
"jushuitan_sku_id": "10475015"
},
{
"sku": "GY-029KAI",
"product_name": "Nebula miniCrossbody Bag",
"ean": "4897080010748",
"source_row": 28,
"jushuitan_sku_id": "10475016"
},
{
"sku": "GY-029WHT",
"product_name": "Nebula miniCrossbody Bag",
"ean": "4897080010755",
"source_row": 29,
"jushuitan_sku_id": "10475014"
},
{
"sku": "GY-030BLK",
"product_name": "Morning Star miniCrossbody Bag",
"ean": "4897080013008",
"source_row": 30,
"jushuitan_sku_id": "10468007"
},
{
"sku": "GY-030WHT",
"product_name": "Morning Star miniCrossbody Bag",
"ean": "4897080010731",
"source_row": 31,
"jushuitan_sku_id": "10468003"
},
{
"sku": "GY-031BLK",
"product_name": "Star-by-Star EVOCrossbody Bag",
"ean": "4897080010724",
"source_row": 32,
"jushuitan_sku_id": "10474004"
},
{
"sku": "GY-032BLK",
"product_name": "GlacierCrossbody Bag",
"ean": "4897080010717",
"source_row": 33,
"jushuitan_sku_id": "10453005"
},
{
"sku": "GY-033BLK",
"product_name": "Morning Star 2 Crossbody Bag",
"ean": "4897080012896",
"source_row": 34,
"jushuitan_sku_id": "10469003"
},
{
"sku": "GY-033WHT",
"product_name": "Morning Star 2 Crossbody Bag",
"ean": "4897080012902",
"source_row": 35,
"jushuitan_sku_id": "10469004"
},
{
"sku": "GY-034BLK",
"product_name": "Startrail 2 Crossbody Bag",
"ean": "4897080012919",
"source_row": 36,
"jushuitan_sku_id": "10504027"
},
{
"sku": "GY-034GRN",
"product_name": "Startrail 2 Crossbody Bag",
"ean": "4897080012926",
"source_row": 37,
"jushuitan_sku_id": "10504028"
},
{
"sku": "GY-034WHT",
"product_name": "Startrail 2 Crossbody Bag",
"ean": "4897080012933",
"source_row": 38,
"jushuitan_sku_id": "10504029"
},
{
"sku": "GY-035BLK",
"product_name": "Zeus 3 air Backpack",
"ean": "4897080012940",
"source_row": 39,
"jushuitan_sku_id": "10482004"
},
{
"sku": "GY-035WHT",
"product_name": "Zeus 3 air Backpack",
"ean": "4897080012957",
"source_row": 40,
"jushuitan_sku_id": "10482003"
},
{
"sku": "GY-036BLK",
"product_name": "Cirrus 2 Backpack",
"ean": "4897080012964",
"source_row": 41,
"jushuitan_sku_id": "10481004"
},
{
"sku": "GY-036GRN",
"product_name": "Cirrus 2 Backpack",
"ean": "4897080012988",
"source_row": 42,
"jushuitan_sku_id": "10481006"
},
{
"sku": "GY-036WHT",
"product_name": "Cirrus 2 Backpack",
"ean": "4897080012971",
"source_row": 43,
"jushuitan_sku_id": "10481005"
},
{
"sku": "GY-037BLK",
"product_name": "Polar Star 2",
"ean": "4897080013046",
"source_row": 44,
"jushuitan_sku_id": "10520009"
},
{
"sku": "GY-037GRY",
"product_name": "Polar Star 2",
"ean": "4897080013039",
"source_row": 45,
"jushuitan_sku_id": "10520014"
},
{
"sku": "GY-038WHT",
"product_name": "Polar Star Shoulder Bag",
"ean": "4897080013060",
"source_row": 46,
"jushuitan_sku_id": "10519010"
},
{
"sku": "GY-038BLK",
"product_name": "Polar Star Shoulder Bag",
"ean": "4897080013053",
"source_row": 47,
"jushuitan_sku_id": "10519009"
},
{
"sku": "GY-039BLK",
"product_name": "Poseidon Laptop Bag",
"ean": "4897080013084",
"source_row": 48,
"jushuitan_sku_id": "10515020"
},
{
"sku": "GY-039GRY",
"product_name": "Poseidon Laptop Bag",
"ean": "4897080013077",
"source_row": 49,
"jushuitan_sku_id": "10515019"
},
{
"sku": "GY-040BLK",
"product_name": "Skyreach 3",
"ean": "4897080013091",
"source_row": 50,
"jushuitan_sku_id": "10510019"
},
{
"sku": "GY-040WHT",
"product_name": "Skyreach 3",
"ean": "4897080013107",
"source_row": 51,
"jushuitan_sku_id": "10510020"
},
{
"sku": "GY-041BLK",
"product_name": "Phantom",
"ean": "4897080013114",
"source_row": 52,
"jushuitan_sku_id": "10507015"
}
]
}
@@ -68,6 +68,9 @@
fact_market_rank_notifications
写: market_rank_hermes_notification.py (Hermes 机器人通知结果;文件名兼容保留)
读: 同一收件人同日通知去重 / 发送审计
erp_sales_theme_daily_metrics
写: collect_erp_sales_theme_daily.py (聚水潭「报表→销售主题分析→按日统计」Excel 导出入库)
读: 固定品牌+店铺口径的全渠道按日销售汇总查询
所有 upsert 都用 ON CONFLICT ... DO UPDATE, 重复跑覆盖不双写。
"""
@@ -1908,3 +1911,160 @@ def upsert_market_rank_notification(conn, record: dict) -> int:
if not row:
raise RuntimeError("市场排行飞书通知归档未返回记录 id")
return int(row[0])
# ── erp_sales_theme_daily_metrics ────────────────────────────────────────────
def ensure_erp_sales_theme_daily_table(conn) -> None:
"""幂等创建聚水潭销售主题分析(按日)汇总表,不执行其他 schema 迁移。"""
sql = """
CREATE TABLE IF NOT EXISTS erp_sales_theme_daily_metrics (
id BIGSERIAL PRIMARY KEY,
stat_date DATE NOT NULL,
sales_order_count INTEGER NOT NULL DEFAULT 0,
sales_qty INTEGER NOT NULL DEFAULT 0,
shipped_qty INTEGER NOT NULL DEFAULT 0,
shipped_amount NUMERIC(18, 2) NOT NULL DEFAULT 0,
sales_amount NUMERIC(18, 2) NOT NULL DEFAULT 0,
return_order_count INTEGER NOT NULL DEFAULT 0,
return_qty INTEGER NOT NULL DEFAULT 0,
returned_qty INTEGER NOT NULL DEFAULT 0,
return_amount NUMERIC(18, 2) NOT NULL DEFAULT 0,
returned_amount NUMERIC(18, 2) NOT NULL DEFAULT 0,
net_sales_order_count INTEGER NOT NULL DEFAULT 0,
net_sales_qty INTEGER NOT NULL DEFAULT 0,
net_sales_amount NUMERIC(18, 2) NOT NULL DEFAULT 0,
freight_income NUMERIC(18, 2),
freight_expense NUMERIC(18, 2),
shipped_order_count INTEGER NOT NULL DEFAULT 0,
date_basis TEXT NOT NULL DEFAULT '发货时间',
brands TEXT[] NOT NULL DEFAULT '{}',
shops TEXT[] NOT NULL DEFAULT '{}',
source_file TEXT,
imported_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
UNIQUE (stat_date)
);
CREATE INDEX IF NOT EXISTS erp_sales_theme_daily_metrics_date_idx
ON erp_sales_theme_daily_metrics (stat_date DESC);
COMMENT ON TABLE erp_sales_theme_daily_metrics IS
'聚水潭「报表→销售主题分析→按日统计」Excel 导出入库;固定品牌+店铺口径,每日一行幂等覆盖。';
COMMENT ON COLUMN erp_sales_theme_daily_metrics.date_basis IS
'报表时间口径,当前固定为发货时间(send_date)。';
COMMENT ON COLUMN erp_sales_theme_daily_metrics.brands IS
'本次查询勾选的商品品牌清单(筛选页 input[name=brand_name] 值)。';
COMMENT ON COLUMN erp_sales_theme_daily_metrics.shops IS
'本次查询勾选的销售渠道(店铺)名称清单。';
"""
with conn.cursor() as cur:
cur.execute(sql)
_SALES_THEME_INT_FIELDS = frozenset({
"sales_order_count",
"sales_qty",
"shipped_qty",
"return_order_count",
"return_qty",
"returned_qty",
"net_sales_order_count",
"net_sales_qty",
"shipped_order_count",
})
_SALES_THEME_COLUMNS = (
"sales_order_count",
"sales_qty",
"shipped_qty",
"shipped_amount",
"sales_amount",
"return_order_count",
"return_qty",
"returned_qty",
"return_amount",
"returned_amount",
"net_sales_amount",
"net_sales_order_count",
"net_sales_qty",
"shipped_order_count",
"freight_income",
"freight_expense",
)
# 表内 NOT NULL 的金额列;导出里无退货/无数据的日期这些单元格为空, 统一落 0
# (运费收入/支出两列允许 NULL, 与导出口径一致)。
_SALES_THEME_REQUIRED_AMOUNTS = frozenset({
"shipped_amount",
"sales_amount",
"return_amount",
"returned_amount",
"net_sales_amount",
})
def upsert_erp_sales_theme_daily_metrics(conn, records: Sequence[dict]) -> int:
"""幂等写入销售主题分析按日汇总,主键为统计日期。
records: [{'stat_date', 'sales_order_count', ..., 'brands', 'shops', 'source_file'}, ...]
数量字段缺省按 0 处理;brands/shops 写入前去重排序,保证重跑结果稳定。
"""
if not records:
return 0
dedup: dict[Any, dict] = {}
for record in records:
key = record["stat_date"]
if str(key).strip() == "":
raise ValueError("stat_date must not be empty")
dedup[key] = dict(record)
rows = []
for record in dedup.values():
row = [record["stat_date"]]
for field in _SALES_THEME_COLUMNS:
value = record.get(field)
if field in _SALES_THEME_INT_FIELDS:
row.append(int(value or 0))
elif field in _SALES_THEME_REQUIRED_AMOUNTS:
row.append(round(float(value), 2) if value is not None else 0)
else:
row.append(round(float(value), 2) if value is not None else None)
row.append(str(record.get("date_basis") or "发货时间"))
row.append(sorted({str(item).strip() for item in record.get("brands") or () if str(item).strip()}))
row.append(sorted({str(item).strip() for item in record.get("shops") or () if str(item).strip()}))
row.append(record.get("source_file"))
rows.append(tuple(row))
sql = """
INSERT INTO erp_sales_theme_daily_metrics
(stat_date, sales_order_count, sales_qty, shipped_qty,
shipped_amount, sales_amount,
return_order_count, return_qty, returned_qty,
return_amount, returned_amount, net_sales_amount,
net_sales_order_count, net_sales_qty,
shipped_order_count, freight_income, freight_expense,
date_basis, brands, shops, source_file)
VALUES %s
ON CONFLICT (stat_date) DO UPDATE SET
sales_order_count = EXCLUDED.sales_order_count,
sales_qty = EXCLUDED.sales_qty,
shipped_qty = EXCLUDED.shipped_qty,
shipped_amount = EXCLUDED.shipped_amount,
sales_amount = EXCLUDED.sales_amount,
return_order_count = EXCLUDED.return_order_count,
return_qty = EXCLUDED.return_qty,
returned_qty = EXCLUDED.returned_qty,
return_amount = EXCLUDED.return_amount,
returned_amount = EXCLUDED.returned_amount,
net_sales_order_count = EXCLUDED.net_sales_order_count,
net_sales_qty = EXCLUDED.net_sales_qty,
net_sales_amount = EXCLUDED.net_sales_amount,
freight_income = EXCLUDED.freight_income,
freight_expense = EXCLUDED.freight_expense,
shipped_order_count = EXCLUDED.shipped_order_count,
date_basis = EXCLUDED.date_basis,
brands = EXCLUDED.brands,
shops = EXCLUDED.shops,
source_file = EXCLUDED.source_file,
updated_at = NOW()
"""
with conn.cursor() as cur:
execute_values(cur, sql, rows)
return len(rows)
@@ -1273,3 +1273,37 @@ COMMENT ON TABLE fact_video_publications IS
'附件级视频发布幂等账本;点击发布后无明确回执必须进入 ambiguous,禁止自动重试。';
COMMENT ON TABLE fact_video_publication_attempts IS
'视频发布每次尝试的标题、商品、话题、标签、提交时点与确认凭据。';
-- 聚水潭「报表→销售主题分析→按日统计」Excel 导出入库。
-- 口径:发货时间区间 + 固定品牌(光影行星/光影/gyxx) + 固定勾选店铺清单,
-- 采集脚本 collect_erp_sales_theme_daily.py;与 erp_all_shop_style_daily_metrics
-- (全店铺按款式) 分表,本表是全渠道合并口径的按日汇总。
CREATE TABLE IF NOT EXISTS erp_sales_theme_daily_metrics (
id BIGSERIAL PRIMARY KEY,
stat_date DATE NOT NULL,
sales_order_count INTEGER NOT NULL DEFAULT 0, -- 销售单数
sales_qty INTEGER NOT NULL DEFAULT 0, -- 销售数量
shipped_qty INTEGER NOT NULL DEFAULT 0, -- 实发数量
shipped_amount NUMERIC(18, 2) NOT NULL DEFAULT 0, -- 实发金额
sales_amount NUMERIC(18, 2) NOT NULL DEFAULT 0, -- 销售金额
return_order_count INTEGER NOT NULL DEFAULT 0, -- 退货单数
return_qty INTEGER NOT NULL DEFAULT 0, -- 退货数量
returned_qty INTEGER NOT NULL DEFAULT 0, -- 实退数量
return_amount NUMERIC(18, 2) NOT NULL DEFAULT 0, -- 退货金额
returned_amount NUMERIC(18, 2) NOT NULL DEFAULT 0, -- 实退金额
net_sales_order_count INTEGER NOT NULL DEFAULT 0, -- 净销售单数
net_sales_qty INTEGER NOT NULL DEFAULT 0, -- 净销量
net_sales_amount NUMERIC(18, 2) NOT NULL DEFAULT 0, -- 净销售额
freight_income NUMERIC(18, 2), -- 运费收入(可空)
freight_expense NUMERIC(18, 2), -- 运费支出(可空)
shipped_order_count INTEGER NOT NULL DEFAULT 0, -- 实发单数
date_basis TEXT NOT NULL DEFAULT '发货时间',
brands TEXT[] NOT NULL DEFAULT '{}',
shops TEXT[] NOT NULL DEFAULT '{}',
source_file TEXT,
imported_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
UNIQUE (stat_date)
);
CREATE INDEX IF NOT EXISTS erp_sales_theme_daily_metrics_date_idx
ON erp_sales_theme_daily_metrics (stat_date DESC);
@@ -1437,7 +1437,8 @@ def load_jd_product_references(
"""
SELECT style_name, jd_spus
FROM dim_style
WHERE jd_spus IS NOT NULL AND cardinality(jd_spus) > 0
WHERE jsonb_typeof(jd_spus) = 'array'
AND jsonb_array_length(jd_spus) > 0
"""
)
rows = cursor.fetchall()
@@ -0,0 +1,442 @@
"""Refresh Hong Kong Anpei public inventory in the Feishu spreadsheet."""
from __future__ import annotations
import json
import sys
import time
from collections.abc import Mapping
from datetime import datetime
from pathlib import Path
from typing import Any
from zoneinfo import ZoneInfo
from gyxx_flow.adapters.jushuitan_inventory import (
InventoryRecord,
JushuitanInventoryClient,
)
from gyxx_flow.modules.product_commerce.lark_cli_runtime import run_lark_cli
_WORKFLOW_ID = "product.hk_anpei_public_inventory.daily"
_CONFIG_PATH = (
Path(__file__).resolve().parent
/ "config"
/ "hk_anpei_public_inventory.json"
)
_REPOSITORY_ROOT = Path(__file__).resolve().parents[4]
_TIMEZONE = ZoneInfo("Asia/Shanghai")
_MAX_SHEET_ROWS = 200
_WRITE_INTERVAL_SECONDS = 0.25
class InventorySpreadsheetError(RuntimeError):
"""Safe-to-log error for invalid spreadsheet configuration or state."""
def _load_config() -> dict[str, Any]:
try:
config = json.loads(_CONFIG_PATH.read_text(encoding="utf-8"))
except (OSError, UnicodeError, json.JSONDecodeError) as exc:
raise InventorySpreadsheetError(
"Unable to read the public inventory workflow configuration"
) from exc
if not isinstance(config, dict):
raise InventorySpreadsheetError(
"Public inventory configuration must be an object"
)
target = config.get("target")
if not isinstance(target, dict):
raise InventorySpreadsheetError(
"Public inventory configuration has no spreadsheet target"
)
for key in (
"spreadsheet_token",
"sheet_id",
"sheet_name",
"sku_column",
"inventory_column",
"product_code_column",
):
if not str(target.get(key) or "").strip():
raise InventorySpreadsheetError(
f"Spreadsheet target is missing {key}"
)
for key in ("first_data_row", "last_data_row", "expected_sku_count"):
value = target.get(key)
if isinstance(value, bool) or not isinstance(value, int) or value < 1:
raise InventorySpreadsheetError(
f"Spreadsheet target has an invalid {key}"
)
if target["last_data_row"] - target["first_data_row"] + 1 != target[
"expected_sku_count"
]:
raise InventorySpreadsheetError(
"Spreadsheet target rows do not match the frozen SKU count"
)
sku_items = config.get("skus")
if not isinstance(sku_items, list) or not sku_items:
raise InventorySpreadsheetError(
"Public inventory configuration has no SKUs"
)
if len(sku_items) != target["expected_sku_count"]:
raise InventorySpreadsheetError(
"Configured SKU count does not match the spreadsheet target"
)
seen_skus: set[str] = set()
seen_product_codes: set[str] = set()
for item in sku_items:
if not isinstance(item, dict):
raise InventorySpreadsheetError(
"Public inventory SKU metadata is invalid"
)
sku = str(item.get("sku") or "").strip()
product_code = str(item.get("jushuitan_sku_id") or "").strip()
if not sku or sku in seen_skus:
raise InventorySpreadsheetError(
"Public inventory configuration contains a blank or duplicate SKU"
)
if not product_code or product_code in seen_product_codes:
raise InventorySpreadsheetError(
"Public inventory configuration contains a blank or duplicate "
"Jushuitan product code"
)
seen_skus.add(sku)
seen_product_codes.add(product_code)
return config
def _sheet_args(config: Mapping[str, Any]) -> list[str]:
target = config["target"]
return [
"--as",
"user",
"--spreadsheet-token",
str(target["spreadsheet_token"]),
"--sheet-id",
str(target["sheet_id"]),
"--format",
"json",
]
def _run_sheets(args: list[str], *, operation: str) -> dict[str, Any]:
try:
return run_lark_cli(
args,
cwd=_REPOSITORY_ROOT,
timeout=180,
)
except Exception as exc:
raise InventorySpreadsheetError(
f"Feishu spreadsheet {operation} failed"
) from exc
def _cell_value(cell: object) -> object:
if not isinstance(cell, Mapping):
raise InventorySpreadsheetError(
"Feishu spreadsheet returned an invalid cell"
)
return cell.get("value")
def _read_sheet_rows(
config: Mapping[str, Any],
sku_items: list[Mapping[str, Any]],
) -> dict[str, dict[str, Any]]:
target = config["target"]
result = _run_sheets(
[
"sheets",
"+cells-get",
*_sheet_args(config),
"--range",
f"{target['sku_column']}1:{target['product_code_column']}{_MAX_SHEET_ROWS}",
"--include",
"value",
],
operation="cells-get",
)
data = result.get("data")
ranges = data.get("ranges") if isinstance(data, Mapping) else None
if (
not isinstance(data, Mapping)
or data.get("has_more") is not False
or not isinstance(ranges, list)
or len(ranges) != 1
):
raise InventorySpreadsheetError(
"Feishu spreadsheet response is incomplete or invalid"
)
sheet_range = ranges[0]
if not isinstance(sheet_range, Mapping):
raise InventorySpreadsheetError(
"Feishu spreadsheet range response is invalid"
)
cells = sheet_range.get("cells")
row_indices = sheet_range.get("row_indices")
col_indices = sheet_range.get("col_indices")
required_columns = {
str(target["sku_column"]),
str(target["inventory_column"]),
str(target["product_code_column"]),
}
if (
not isinstance(cells, list)
or not isinstance(row_indices, list)
or not isinstance(col_indices, list)
or len(cells) != len(row_indices)
or not required_columns.issubset({str(col) for col in col_indices})
):
raise InventorySpreadsheetError(
"Feishu spreadsheet response does not cover the configured columns"
)
column_positions = {str(col): index for index, col in enumerate(col_indices)}
row_values: dict[int, list[object]] = {}
for row_number, values in zip(row_indices, cells, strict=True):
if (
isinstance(row_number, bool)
or not isinstance(row_number, int)
or not isinstance(values, list)
or len(values) != len(col_indices)
):
raise InventorySpreadsheetError(
"Feishu spreadsheet returned malformed row coordinates"
)
row_values[row_number] = values
first_row = int(target["first_data_row"])
last_row = int(target["last_data_row"])
header = row_values.get(1)
if header is None:
raise InventorySpreadsheetError(
"Feishu spreadsheet is missing its header row"
)
expected_headers = {
str(target["sku_column"]): "sku",
str(target["inventory_column"]): "仓库现有库存",
str(target["product_code_column"]): "商品编码(光影)",
}
for column, expected_header in expected_headers.items():
actual_header = str(
_cell_value(header[column_positions[column]]) or ""
).strip()
if actual_header != expected_header:
raise InventorySpreadsheetError(
f"Feishu spreadsheet header in column {column} does not match"
)
rows: dict[str, dict[str, Any]] = {}
for row_number in range(first_row, last_row + 1):
values = row_values.get(row_number)
if values is None:
raise InventorySpreadsheetError(
"Feishu spreadsheet is missing an expected inventory row"
)
sku = str(
_cell_value(values[column_positions[str(target["sku_column"])]]) or ""
).strip()
product_code = str(
_cell_value(
values[column_positions[str(target["product_code_column"])] ]
)
or ""
).strip()
if not sku or sku in rows:
raise InventorySpreadsheetError(
"Feishu spreadsheet contains a blank or duplicate SKU"
)
rows[sku] = {
"row": row_number,
"product_code": product_code,
}
expected_by_sku = {
str(item["sku"]).strip(): str(item["jushuitan_sku_id"]).strip()
for item in sku_items
}
if set(rows) != set(expected_by_sku):
raise InventorySpreadsheetError(
"Feishu spreadsheet SKU rows differ from the frozen workbook list"
)
for sku, row in rows.items():
if row["product_code"] != expected_by_sku[sku]:
raise InventorySpreadsheetError(
f"Feishu spreadsheet product code does not match SKU {sku} "
f"at row {row['row']}"
)
return rows
def _read_written_inventory(
config: Mapping[str, Any],
expected: list[object],
) -> None:
target = config["target"]
column = str(target["inventory_column"])
first_row = int(target["first_data_row"])
last_row = int(target["last_data_row"])
result = _run_sheets(
[
"sheets",
"+cells-get",
*_sheet_args(config),
"--range",
f"{column}{first_row}:{column}{last_row}",
"--include",
"value",
],
operation="inventory readback",
)
data = result.get("data")
ranges = data.get("ranges") if isinstance(data, Mapping) else None
if (
not isinstance(data, Mapping)
or data.get("has_more") is not False
or not isinstance(ranges, list)
or len(ranges) != 1
):
raise InventorySpreadsheetError(
"Feishu inventory readback is incomplete or invalid"
)
sheet_range = ranges[0]
if not isinstance(sheet_range, Mapping):
raise InventorySpreadsheetError(
"Feishu inventory readback range is invalid"
)
cells = sheet_range.get("cells")
row_indices = sheet_range.get("row_indices")
if (
not isinstance(cells, list)
or not isinstance(row_indices, list)
or row_indices != list(range(first_row, last_row + 1))
or len(cells) != len(expected)
):
raise InventorySpreadsheetError(
"Feishu inventory readback does not cover every target row"
)
actual: list[object] = []
for row in cells:
if not isinstance(row, list) or len(row) != 1:
raise InventorySpreadsheetError(
"Feishu inventory readback has an invalid cell shape"
)
actual.append(_cell_value(row[0]))
for offset, (actual_value, expected_value) in enumerate(
zip(actual, expected, strict=True)
):
if expected_value == "":
matched = actual_value in (None, "")
else:
try:
matched = int(str(actual_value).strip()) == int(expected_value)
except (TypeError, ValueError):
matched = False
if not matched:
raise InventorySpreadsheetError(
"Feishu inventory readback did not match the submitted value "
f"at row {first_row + offset}"
)
def _write_inventory(
config: Mapping[str, Any],
sku_items: list[Mapping[str, Any]],
inventories: Mapping[str, InventoryRecord],
rows: Mapping[str, Mapping[str, Any]],
) -> tuple[int, bool, int]:
target = config["target"]
first_row = int(target["first_data_row"])
last_row = int(target["last_data_row"])
by_row = sorted(rows.items(), key=lambda entry: int(entry[1]["row"]))
if [int(row["row"]) for _, row in by_row] != list(
range(first_row, last_row + 1)
):
raise InventorySpreadsheetError(
"Feishu spreadsheet SKU rows are not a contiguous target range"
)
sku_config = {str(item["sku"]): item for item in sku_items}
expected_values: list[object] = []
missing_count = 0
for sku, _ in by_row:
product_code = str(sku_config[sku]["jushuitan_sku_id"])
inventory = inventories.get(product_code)
if inventory is None:
# An omitted Jushuitan row is unknown inventory, so clear stale data.
expected_values.append("")
missing_count += 1
else:
expected_values.append(inventory.public_available_qty)
cells = [[{"value": value}] for value in expected_values]
result = _run_sheets(
[
"sheets",
"+cells-set",
*_sheet_args(config),
"--range",
f"{target['inventory_column']}{first_row}:"
f"{target['inventory_column']}{last_row}",
"--cells",
json.dumps(cells, ensure_ascii=False, separators=(",", ":")),
],
operation="inventory update",
)
if result.get("acceptance_skipped") is True:
return 0, True, missing_count
_read_written_inventory(config, expected_values)
return len(expected_values), False, missing_count
def run() -> dict[str, object]:
config = _load_config()
sku_items = config["skus"]
jushuitan_sku_ids = [str(item["jushuitan_sku_id"]) for item in sku_items]
# Validate the complete target and ERP response before modifying Feishu.
rows = _read_sheet_rows(config, sku_items)
inventories = JushuitanInventoryClient().query(jushuitan_sku_ids)
if not inventories:
raise InventorySpreadsheetError(
"Jushuitan returned no inventory rows for the frozen product-code list; "
"refusing to clear the Feishu inventory column"
)
written, acceptance_skipped, inventory_missing = _write_inventory(
config,
sku_items,
inventories,
rows,
)
return {
"workflow_id": _WORKFLOW_ID,
"target_sheet": str(config["target"]["sheet_name"]),
"sku_count": len(sku_items),
"inventory_found": len(inventories),
"inventory_missing": inventory_missing,
"cells_written": written,
"acceptance_skipped": acceptance_skipped,
"queried_at": datetime.now(_TIMEZONE).strftime("%Y-%m-%d %H:%M:%S"),
}
def main() -> int:
try:
print(json.dumps(run(), ensure_ascii=False, separators=(",", ":")))
return 0
except Exception as exc:
print(
json.dumps(
{"workflow_id": _WORKFLOW_ID, "error": str(exc)},
ensure_ascii=False,
separators=(",", ":"),
),
file=sys.stderr,
)
return 1
if __name__ == "__main__":
raise SystemExit(main())
@@ -29,15 +29,19 @@ from runtime_paths import TMP_ROOT
from gyxx_flow.adapters.browser import BrowserCookieStore
from gyxx_flow.adapters.scrapling import ScraplingBrowser, is_browser_timeout_error
from gyxx_flow.core.config import Settings
from gyxx_flow.core.locks import LockManager, ResourceBusyError
_CONFIG_NAME = "tmall-baibu.json"
_LOGIN_PAGE_URL = "https://login.taobao.com/member/login.jhtml"
_EXPORT_TIMEOUT_SECONDS = 240
_EXPORT_MAX_ATTEMPTS = 3
_EXPORT_RETRY_DELAYS_SECONDS = (5, 15)
_PAGE_TIMEOUT_MS = 60_000
_IMPORT_TIMEOUT_SECONDS = 300
_IMPORT_ACCEPTED_AFTER_PROCESSING_SECONDS = 120
_STATUS_SETTLE_TIMEOUT_SECONDS = 5
_ACCOUNT_LOCK_TIMEOUT_SECONDS = 120
_RESULT_REFRESH_INTERVAL_SECONDS = 8
_DRAFT_ACTION_TIMEOUT_SECONDS = 30
_DRAFT_NAVIGATION_TIMEOUT_SECONDS = 15
@@ -103,6 +107,7 @@ _LOGIN_SECOND_FACTOR_MARKERS = (
"安全验证",
"二次验证",
"验证码",
"滑块",
)
@@ -805,6 +810,41 @@ def _find_password_login_form(
return locate()
def _is_login_agreement_visible(page: Any) -> bool:
"""Return whether Tmall's 服务协议及隐私保护 popup is blocking the login."""
for context in _page_contexts(page):
get_by_role = getattr(context, "get_by_role", None)
if not callable(get_by_role):
continue
try:
dialogs = get_by_role("dialog")
for index in range(min(dialogs.count(), 8)):
dialog = dialogs.nth(index)
try:
if not dialog.is_visible():
continue
text = _normalized_page_text(dialog.inner_text(timeout=2_000))
except Exception:
continue
if "服务协议" in text or "隐私" in text:
return True
except Exception:
continue
return False
def _click_login_agreement_accept(page: Any) -> bool:
for context in _page_contexts(page):
for candidate in _visible_text_candidates(context, "同意"):
try:
candidate.click(timeout=8_000)
return True
except Exception:
continue
return False
def _login_if_needed(
entry: BaibuEntry,
page: Any,
@@ -817,6 +857,11 @@ def _login_if_needed(
return False
login_url = _is_login_url(str(getattr(page, "url", "")))
form = _find_password_login_form(page, use_password_form=False)
if form is None and not login_url:
# A redirect can land on a scanner-tab seller login page (qianniu)
# whose URL carries no login marker; probe the password tab before
# deciding there is nothing to log in to.
form = _find_password_login_form(page, use_password_form=True)
if not login_url and form is None:
return False
@@ -852,10 +897,92 @@ def _login_if_needed(
raise BaibuApplyError(
f"{entry.name} 自动填写账号密码并提交登录失败({type(exc).__name__})"
) from exc
# The first password submit can pop the 服务协议及隐私保护 confirmation
# which swallows the submit; accept it and submit once more so the login
# actually runs. Both popups and form re-renders are tolerated.
for _attempt in range(2):
wait_for_timeout = getattr(page, "wait_for_timeout", None)
if callable(wait_for_timeout):
wait_for_timeout(2_000)
else:
time.sleep(2)
if not _is_login_agreement_visible(page):
break
if not _click_login_agreement_accept(page):
break
print(f"[百补] {entry.name} 已确认登录服务协议弹窗,重新提交登录")
refreshed = _find_password_login_form(page, use_password_form=True)
if refreshed is None:
break
_context, username, secret_input, submit = refreshed
try:
username.fill(credentials.username)
secret_input.fill(credentials.password)
submit.click(timeout=15_000)
except Exception as exc:
raise BaibuApplyError(
f"{entry.name} 协议确认后重新提交登录失败({type(exc).__name__})"
) from exc
print(f"[百补] {entry.name} 已提交密码登录,等待店铺身份确认")
return True
def _goto_tolerant(page: Any, url: str) -> None:
"""Navigate, tolerating an aborted load when a redirect takes over.
An unauthenticated apply URL is answered with a redirect chain that can
abort the original navigation from inside the browser; treat that as the
navigation landing wherever the chain pointed and let it settle.
"""
try:
page.goto(url, wait_until="domcontentloaded", timeout=_PAGE_TIMEOUT_MS)
except Exception as exc:
if "ERR_ABORTED" not in str(exc):
raise
wait_for_timeout = getattr(page, "wait_for_timeout", None)
if callable(wait_for_timeout):
wait_for_timeout(_LOGIN_FORM_SETTLE_MILLISECONDS)
else:
time.sleep(_LOGIN_FORM_SETTLE_MILLISECONDS / 1000)
def _explicit_password_login(
entry: BaibuEntry,
page: Any,
*,
manual_login: bool,
) -> None:
"""Drive the Tmall password form when the apply page hid its login state.
The myseller/qn apply SPA can sit on an SSO interstitial that never shows
a login URL, or render the cached shop name on an expired session. In
both cases ``_login_if_needed`` sees nothing to do. Re-entering the apply
URL redirects to the seller login page (qianniu) which carries the right
merchant context; only when that shows no form fall back to the plain
login page. Login there, return to the apply URL, and confirm identity.
"""
print(f"[百补] {entry.name} 显式重新进入报名页并走密码登录")
_goto_tolerant(page, entry.apply_url)
wait_for_timeout = getattr(page, "wait_for_timeout", None)
if callable(wait_for_timeout):
wait_for_timeout(_LOGIN_FORM_SETTLE_MILLISECONDS * 3)
else:
time.sleep(_LOGIN_FORM_SETTLE_MILLISECONDS / 1000 * 3)
login_attempted = _login_if_needed(entry, page, manual_login=manual_login)
if not login_attempted:
_goto_tolerant(page, _LOGIN_PAGE_URL)
login_attempted = _login_if_needed(entry, page, manual_login=manual_login)
_goto_tolerant(page, entry.apply_url)
_wait_for_authenticated(
entry,
page,
login_attempted=login_attempted,
manual_login=manual_login,
)
def _click_text(page: Any, labels: Iterable[str]) -> bool:
contexts = _page_contexts(page)
for label in labels:
@@ -1040,7 +1167,10 @@ def _open_batch_import(
f"{entry.name} 页面已有批量导入操作正在处理,已停止,避免重复提交"
)
time.sleep(0.5)
raise BaibuApplyError(f"{entry.name} 页面没有找到可用的「商品批量导入」入口")
raise BaibuApplyError(
f"{entry.name} 页面没有找到可用的「商品批量导入」入口"
"(活动可能已报名/已结束或会话无效,请人工打开报名页确认)"
)
def _wait_import_result(
@@ -1062,6 +1192,7 @@ def _wait_import_result(
last_processing_log = 0.0
last_refresh = 0.0
observed_new_operation = False
first_new_operation_at: float | None = None
while time.monotonic() < deadline:
text = _page_text(page)
operation = _latest_batch_import_status(text)
@@ -1070,6 +1201,8 @@ def _wait_import_result(
or operation.signature != previous_operation.signature
):
observed_new_operation = True
if first_new_operation_at is None:
first_new_operation_at = time.monotonic()
if operation.state == "success":
print(
f"[百补] {entry.name} 页面已完成导入处理,"
@@ -1082,6 +1215,21 @@ def _wait_import_result(
f"忽略行结果并按导入提交成功:{operation.detail}"
)
return
if (
observed_new_operation
and first_new_operation_at is not None
and time.monotonic() - first_new_operation_at
>= _IMPORT_ACCEPTED_AFTER_PROCESSING_SECONDS
):
# Tmall has already acknowledged the batch ("执行了商品批量导入
# 操作,正在处理中"); the submission itself is the business
# effect, so a slow per-row summary must not fail the run.
print(
f"[百补] {entry.name} 天猫已受理批量导入,"
f"{_IMPORT_ACCEPTED_AFTER_PROCESSING_SECONDS} 秒内未返回行结果,"
"按导入提交成功,行结果以报名页面为准"
)
return
if observed_new_operation and any(marker in text for marker in _SUCCESS_MARKERS):
print(f"[百补] {entry.name} 页面已确认导入提交,忽略行结果")
return
@@ -1284,10 +1432,21 @@ def _complete_draft_products(page: Any, entry: BaibuEntry) -> None:
if not entry.complete_drafts_after_import:
return
if not _click_draft_tab(page):
count = _draft_count(_page_text(page))
if count is None and not _has_visible_text(page, ("完善商品",)):
raise BaibuApplyError(f"{entry.name} 未找到草稿页入口")
# Right after the import acknowledgement the SPA may still be hydrating;
# the draft tab, the draft count and the per-row action rarely all appear
# on the first probe, so retry briefly before declaring the page broken.
deadline = time.monotonic() + _DRAFT_NAVIGATION_TIMEOUT_SECONDS
while time.monotonic() < deadline:
if _click_draft_tab(page):
break
text = _page_text(page)
if _draft_count(text) is not None or _has_visible_text(page, ("完善商品",)):
break
if any(marker in text for marker in _DRAFT_EMPTY_MARKERS):
break
time.sleep(0.5)
else:
raise BaibuApplyError(f"{entry.name} 未找到草稿页入口")
draft_count, action_count = _wait_for_draft_state(page, entry)
if draft_count == 0:
@@ -1439,6 +1598,8 @@ def _has_account_cookie(
def _persist_account_state(
browser: ScraplingBrowser,
entry: BaibuEntry,
*,
session_lock_held: bool = False,
) -> bool:
"""Publish a verified login to the account vault without leaking cookies."""
@@ -1463,8 +1624,12 @@ def _persist_account_state(
)
cookies = _account_cookies(cookies, required_domains)
if not _has_account_cookie(cookies, required_names):
names = sorted({str(item.get("name", "")) for item in cookies})
preview = "、".join(names[:12]) if names else "无"
raise BaibuApplyError(
f"{entry.name} 登录后未获得可用的账号 cookie,未更新账号登录态"
f"{entry.name} 登录后未获得可用的账号 cookie"
f"(域内 cookie:{preview};要求含:{'、'.join(required_names)}),"
"未更新账号登录态"
)
raw_state = context.storage_state()
@@ -1472,23 +1637,33 @@ def _persist_account_state(
filtered_state = _account_storage_state(state, cookies, required_domains)
cookie_file, storage_state_file = paths
store = BrowserCookieStore(cookie_file, storage_state_file)
lock_root = cookie_file.parent.parent / "locks"
try:
with LockManager(lock_root).acquire(
f"browser-account:{entry.account_id}",
owner=f"tmall-baibu:{entry.entry_id}",
timeout_seconds=0,
):
def _save() -> None:
try:
store.save_cookies(cookies)
store.save_storage_state(filtered_state)
except ResourceBusyError as exc:
raise BaibuApplyError(
f"{entry.name} 账号登录态正在被其他进程更新,未覆盖账号登录态"
) from exc
except (OSError, ValueError, TypeError) as exc:
raise BaibuApplyError(
f"{entry.name} 保存账号登录态失败({type(exc).__name__})"
) from exc
except (OSError, ValueError, TypeError) as exc:
raise BaibuApplyError(
f"{entry.name} 保存账号登录态失败({type(exc).__name__})"
) from exc
if session_lock_held:
# The whole browser session already holds the scheduler-side
# browser-account lock; re-acquiring here would self-deadlock.
_save()
else:
lock_root = Settings.from_env().data_root / "state" / "locks"
try:
with LockManager(lock_root).acquire(
f"browser-account:{entry.account_id}",
owner=f"tmall-baibu:{entry.entry_id}",
timeout_seconds=0,
):
_save()
except ResourceBusyError as exc:
raise BaibuApplyError(
f"{entry.name} 账号登录态正在被其他进程更新,未覆盖账号登录态"
) from exc
print(f"[百补] 已将 {entry.account_id} 的已验证登录态同步到账号 vault")
return True
@@ -1498,6 +1673,33 @@ def _apply_entry(
template: Path,
*,
manual_login: bool = False,
) -> None:
# Hold the scheduler-side account browser lock for the whole session.
# The resident keepalive heartbeats the same profile on an interval and
# its timeout-0 acquire skips busy accounts, so this lock is what stops
# the heartbeat from killing the import mid-run (TargetClosedError /
# silent page death). Members must use the same lock directory as the
# scheduler (Settings.data_root/state/locks), not one derived from the
# vault paths.
locks = LockManager(Settings.from_env().data_root / "state" / "locks")
try:
with locks.acquire(
f"browser-account:{entry.account_id}",
owner=f"tmall-baibu:{entry.entry_id}",
timeout_seconds=_ACCOUNT_LOCK_TIMEOUT_SECONDS,
):
_apply_entry_locked(entry, template, manual_login=manual_login)
except ResourceBusyError as exc:
raise BaibuApplyError(
f"{entry.name} 账号浏览器正被保活/登录占用,请稍后重试"
) from exc
def _apply_entry_locked(
entry: BaibuEntry,
template: Path,
*,
manual_login: bool = False,
) -> None:
browser = _start_browser()
try:
@@ -1516,14 +1718,36 @@ def _apply_entry(
page,
manual_login=manual_login,
)
_wait_for_authenticated(
entry,
page,
login_attempted=login_attempted,
manual_login=manual_login,
)
try:
_wait_for_authenticated(
entry,
page,
login_attempted=login_attempted,
manual_login=manual_login,
)
except BaibuApplyError:
if login_attempted or manual_login:
raise
# The apply page may sit on an SSO interstitial without a login
# URL, so the passive check above never triggers the credentials;
# fall back to the explicit password login once.
_explicit_password_login(entry, page, manual_login=manual_login)
login_attempted = True
browser.persist_bound_state()
_persist_account_state(browser, entry)
try:
_persist_account_state(browser, entry, session_lock_held=True)
except BaibuApplyError as exc:
# A failed vault refresh can also mean the shop name was rendered
# from cache on top of a guest session; retry once through the
# explicit login before giving up on the vault refresh. The
# refresh only caches this session for later runs and must never
# block the import submission itself.
print(f"[WARN] {exc};尝试显式重新登录后重试", file=sys.stderr)
try:
_explicit_password_login(entry, page, manual_login=manual_login)
_persist_account_state(browser, entry, session_lock_held=True)
except BaibuApplyError as retry_exc:
print(f"[WARN] {retry_exc};本次继续执行报名导入", file=sys.stderr)
print(f"[百补] 已确认账号 {entry.account_id},打开 {entry.name} 报名入口")
previous_operation = _open_batch_import(page, entry)
time.sleep(1)
@@ -503,7 +503,8 @@ def load_tm_item_ids_for_styles(style_names: List[str]) -> Dict[str, List[str]]:
"""
SELECT style_name, tm_spus
FROM dim_style
WHERE tm_spus IS NOT NULL AND cardinality(tm_spus) > 0
WHERE jsonb_typeof(tm_spus) = 'array'
AND jsonb_array_length(tm_spus) > 0
"""
)
rows = cur.fetchall()
@@ -223,23 +223,16 @@ def _check_login_status(page) -> bool:
def _ensure_login(page):
print("\n" + "=" * 60)
print("【请登录】请在弹出的浏览器窗口中完成登录")
print("提示:扫码/短信/密码均可,登录成功后页面会自动跳转")
print("=" * 60 + "\n")
for i in range(LOGIN_TIMEOUT):
page = _active_page(page)
if _check_login_status(page):
print(f"[OK] Login success! URL: {page.url}")
page.wait_for_timeout(3000)
return page
if i % 5 == 0:
print(f"[WAIT] Waiting for login... {i}s / {LOGIN_TIMEOUT}s")
time.sleep(1)
print(f"[ERR] Login timeout ({LOGIN_TIMEOUT}s)")
return None
# 登录统一由账号 vault 管理(gyxx accounts login douyin-shop),
# 采集脚本不再弹出浏览器等待人工扫码/短信登录。
print(
"[COOKIE_EXPIRED] 抖音罗盘登录态已失效:"
"请执行 uv run gyxx accounts login douyin-shop 刷新账号登录态"
)
raise RuntimeError(
"抖音罗盘登录态已失效;请执行 uv run gyxx accounts login douyin-shop "
"刷新账号登录态后重跑"
)
def _select_menu_and_date(page) -> None:
@@ -1927,12 +1920,8 @@ def _page_automation(page) -> None:
# 1. Check and wait for login
if not _check_login_status(page):
logged_page = _ensure_login(page)
if not logged_page:
print("[ERR] Login failed, will try save cookies anyway...")
else:
page = logged_page
_maximize_page(page)
# 登录态失效时 _ensure_login 会直接抛 RuntimeError
_ensure_login(page)
# 2. Save state after login (best effort)
_save_browser_state(page, "after login")
@@ -226,23 +226,16 @@ def ensure_login(page: Page, *, headless: bool, login_timeout: int) -> None:
print("[SKIPPED_COOKIE] 抖店会话跳转到了登录页")
raise DouyinSessionExpiredError("抖店会话跳转到了登录页")
if headless:
raise RuntimeError(
"抖音 Cookie 已失效且当前为无头模式,无法扫码登录;"
"请先执行 uv run gyxx accounts login douyin-shop 刷新账号登录态"
)
print("=" * 60)
print("【请登录】请在弹出的浏览器窗口中完成抖店登录")
print("=" * 60)
deadline = time.monotonic() + login_timeout
while time.monotonic() < deadline:
if not douyin_common.is_login_page(page):
print(f" [OK] 登录成功: {page.url}")
page.wait_for_timeout(3_000)
return
page.wait_for_timeout(1_000)
raise RuntimeError(f"登录在 {login_timeout} 秒内未完成,当前页面: {page.url}")
# 登录统一由账号 vault 管理(gyxx accounts login douyin-shop),
# 采集脚本不再弹出浏览器等待人工扫码/验证码。
print(
"[COOKIE_EXPIRED] 抖店 Cookie 已失效:"
"请执行 uv run gyxx accounts login douyin-shop 刷新账号登录态"
)
raise DouyinSessionExpiredError(
"抖音 Cookie 已失效;请执行 uv run gyxx accounts login douyin-shop "
"刷新账号登录态后重跑"
)
class DouyinSessionExpiredError(RuntimeError):
@@ -262,23 +262,16 @@ def _check_login_status(page) -> bool:
def _ensure_login(page):
print("\n" + "=" * 60)
print("【请登录】请在弹出的浏览器窗口中完成登录")
print("提示:扫码/短信/密码均可,登录成功后页面会自动跳转")
print("=" * 60 + "\n")
for i in range(LOGIN_TIMEOUT):
page = _active_page(page)
if _check_login_status(page):
print(f"[OK] Login success! URL: {page.url}")
page.wait_for_timeout(3000)
return page
if i % 5 == 0:
print(f"[WAIT] Waiting for login... {i}s / {LOGIN_TIMEOUT}s")
time.sleep(1)
print(f"[ERR] Login timeout ({LOGIN_TIMEOUT}s)")
return None
# 登录统一由账号 vault 管理(gyxx accounts login douyin-shop),
# 采集脚本不再弹出浏览器等待人工扫码/短信登录。
print(
"[COOKIE_EXPIRED] 抖音罗盘登录态已失效:"
"请执行 uv run gyxx accounts login douyin-shop 刷新账号登录态"
)
raise BrowserSessionExpiredError(
"抖音罗盘登录态已失效;请执行 uv run gyxx accounts login douyin-shop "
"刷新账号登录态后重跑"
)
def _select_menu_and_date(page) -> None:
@@ -2325,12 +2318,8 @@ def _page_automation_impl(page) -> None:
# 1. Check and wait for login
if not _check_login_status(page):
logged_page = _ensure_login(page)
if not logged_page:
print("[ERR] Login failed, will try save cookies anyway...")
else:
page = logged_page
_maximize_page(page)
# If the login state is expired, _ensure_login will directly raise BrowserSessionExpiredError
_ensure_login(page)
# 2. Save state after login (best effort)
_save_browser_state(page, "after login")
@@ -32,10 +32,9 @@ from gyxx_flow.modules.shop_intelligence.browser_session import (
)
from gyxx_flow.modules.shop_intelligence.collectors.jd_data_collector import (
close_profit_plan_popup_if_present,
select_last_week_in_old_datepicker,
)
from gyxx_flow.modules.shop_intelligence.config import JD_PEER_STORE_OUTPUT
from gyxx_flow.modules.shop_intelligence.paths import resolve_raw_output
from gyxx_flow.modules.shop_intelligence.paths import TMP_ROOT, resolve_raw_output
# ── 配置 ─────────────────────────────────────────────────────────────────────
@@ -67,7 +66,11 @@ CATEGORY_OUTPUT_NAMES = {
"男包": "男士包袋",
}
VENDER_RANKS_URL = "https://jdsz.jd.com/szweb/view/industryMarket/venderRanks.html"
# 2026-10 京东下线旧版商智:venderRanks 旧入口 404,官方公告要求迁移新版
# jdsz.jd.com。新版等价页面是「行业榜单」的店铺榜 tab。
INDUSTRY_TOP_URL = "https://jdsz.jd.com/szweb/view/industry/industry-top.html"
# 兼容旧引用
VENDER_RANKS_URL = INDUSTRY_TOP_URL
# 新品采集配置
NEW_PRODUCT_SHOPS = [
@@ -398,97 +401,262 @@ def get_last_week_range():
return last_monday, last_sunday
def select_last_week_any_day(page):
"""点击日期选择器 → 按周查询 → 点上周任意一天(自动选中整周)"""
print(" 选择上周时间范围...")
# 1. 点击日期选择器(显示"按天查询..."的元素)
def _dump_peer_filter_debug(page: Page, tag: str) -> None:
"""日期/类目操作失败时保存现场,便于按页面实际 DOM 校准选择器。"""
try:
date_picker = page.locator('.jmtd-date-picker .jmtd-base-input-top').first
date_picker.wait_for(state="visible", timeout=5000)
date_picker.click()
print(" [OK] 点击日期选择器")
except Exception as e:
print(f" [WARN] 点击日期选择器失败: {e}")
# 兜底:点击包含"按天查询"文本的元素
stamp = datetime.now().strftime("%Y%m%d_%H%M%S")
debug_dir = TMP_ROOT / "debug" / "jd_peer_filter"
debug_dir.mkdir(parents=True, exist_ok=True)
page.screenshot(path=str(debug_dir / f"{stamp}_{tag}.png"))
html_path = debug_dir / f"{stamp}_{tag}.html"
html_path.write_text(page.content(), encoding="utf-8")
print(f" [DEBUG] 现场已保存: {html_path}")
except Exception as exc:
print(f" [DEBUG] 现场保存失败: {exc}")
def _click_custom_day_cell(page: Page, date_str: str) -> None:
"""在自定义日期面板中点击指定日期格(按月份块消歧,真实坐标点击)。
面板为单月迷你日历:日格只带日号文本,须先定位「只含一个月份标题且
含日历格」的月份块,再在其中匹配日号;组件对合成点击不稳定,统一用
bounding_box 中心坐标做真实鼠标点击。
"""
center = page.evaluate(
"""(dateStr) => {
const [year, month, day] = dateStr.split('-');
const wantTitle = `${Number(year)}年${String(Number(month)).padStart(2, '0')}月`;
const dayText = String(Number(day));
const isMonthTitle = (text) => /^\\d{4}年\\d{2}月$/.test((text || '').trim());
const titleSpans = Array.from(document.querySelectorAll('span'))
.filter(el => el.getClientRects().length && isMonthTitle(el.innerText));
for (const span of titleSpans) {
if ((span.innerText || '').trim() !== wantTitle) continue;
let node = span.parentElement;
for (let depth = 0; node && depth < 10; depth += 1, node = node.parentElement) {
const titles = Array.from(node.querySelectorAll('span'))
.filter(s => isMonthTitle(s.innerText));
if (titles.length > 1) break;
if (titles.length === 1
&& node.querySelector('td.jmt-date-picker-calendar-cell')) {
const cells = Array.from(
node.querySelectorAll('td.jmt-date-picker-calendar-cell')
).filter(el => el.getBoundingClientRect().width > 0
&& (el.innerText || '').trim() === dayText);
if (cells.length) {
const rect = cells[0].getBoundingClientRect();
return {x: Math.round(rect.x + rect.width / 2),
y: Math.round(rect.y + rect.height / 2)};
}
return null;
}
}
}
return null;
}""",
date_str,
)
if not center:
_dump_peer_filter_debug(page, f"day_not_found_{date_str}")
raise RuntimeError(f"自定义日期面板未找到日期格: {date_str}")
page.mouse.click(center["x"], center["y"])
def _open_custom_date_panel(page: Page, *, retries: int = 3) -> bool:
"""点击「自定义」标签打开日历;带开面板验证与重试。
面板行为实测有两种形态且跨运行不定:迷你单月(选起点即自动关闭)
与常驻双月(选完不关)。tag 点击在面板刚自动关闭后可能需要缓一下
再点,故带月标题可见性验证与重试。
"""
tag = page.locator(
"li.jmt-combo-date-picker-list-tag", has_text="自定义"
).first
for _attempt in range(retries):
try:
page.locator('text=按天查询').first.click()
print(" [OK] 使用文本定位点击日期选择器")
box = tag.bounding_box(timeout=5000)
except Exception:
pass
page.wait_for_timeout(2000)
box = None
if not box:
_dump_peer_filter_debug(page, "custom_tag_missing")
raise RuntimeError("未找到「自定义」时间按钮")
page.mouse.click(
box["x"] + box["width"] / 2, box["y"] + box["height"] / 2
)
page.wait_for_timeout(2000)
if _panel_months_visible(page):
return True
page.wait_for_timeout(1000)
return _panel_months_visible(page)
# 2. 点击"按周查询"
try:
week_switch = page.locator('.jmtd-date-picker-switch-panel-item', has_text="按周查询").first
week_switch.wait_for(state="visible", timeout=5000)
week_switch.click()
print(" [OK] 点击按周查询")
except Exception as e:
print(f" [WARN] 点击按周查询失败: {e}")
# 兜底:用 JS 点击
page.evaluate("""() => {
const items = document.querySelectorAll('.jmtd-date-picker-switch-panel-item');
for (const item of items) {
if (item.textContent.includes('按周查询')) {
item.click();
return true;
}
}
return false;
}""")
page.wait_for_timeout(2000)
# 3. 等待日历弹窗并选择上周某一天
calendar = page.locator('.jmtd-date-picker-popup, .jmtd-date-picker-panel, [class*="picker-panel"], [class*="calendar-panel"]')
def _panel_months_visible(page) -> bool:
try:
calendar.wait_for(state="visible", timeout=5000)
return bool(
page.evaluate(
"() => Array.from(document.querySelectorAll('span'))"
".some(el => el.getClientRects().length"
" && /^\\d{4}年\\d{2}月$/.test((el.innerText || '').trim()))"
)
)
except Exception:
print(" [WARN] 未检测到日历弹窗,尝试继续...")
return False
def _ensure_month_visible(page: Page, month_title: str) -> None:
"""单月日历翻月:目标月份不可见时点「后一月」箭头(至多 12 次)。"""
def visible_months():
return page.evaluate(
"""() => Array.from(document.querySelectorAll('span'))
.filter(el => el.getClientRects().length)
.map(el => (el.innerText || '').trim())
.filter(t => /^\\d{4}年\\d{2}月$/.test(t))"""
)
for _ in range(12):
if month_title in visible_months():
return
try:
arrow = page.locator('[data-event-content="后一月"]')
if arrow.count() == 0:
return
box = arrow.first.bounding_box(timeout=3000)
except Exception:
return
if not box:
return
page.mouse.click(
box["x"] + box["width"] / 2, box["y"] + box["height"] / 2
)
page.wait_for_timeout(1200)
def _custom_panel_state(page) -> str:
"""调试用:当前日期面板的可见月份与「当前:」显示。"""
try:
return str(
page.evaluate(
"""() => {
const months = Array.from(document.querySelectorAll('span'))
.filter(el => el.getClientRects().length)
.map(el => (el.innerText || '').trim())
.filter(t => /^\\d{4}年\\d{2}月$/.test(t));
const current = (document.body.innerText.match(/当前:[^\\n]{0,40}/) || [''])[0];
return {months: [...new Set(months)], current};
}"""
)
)
except Exception as exc:
return f"<state-error {exc}>"
def _custom_range_applied(page, start_mmdd: str) -> bool:
try:
body_text = page.locator("body").inner_text(timeout=3000)
except Exception:
return False
return start_mmdd in body_text.replace(" ", "")
def select_last_week_any_day(page):
"""新版行业榜单:自定义区间选择上周一~上周日(状态机,3 轮重试)。
面板有两种形态且跨运行不定:
- 迷你单月:点击起点后自动关闭;重开面板点终点,选完立即生效。
- 常驻双月:起止选完面板不关;点面板外部(店铺搜索框)应用;
Escape 会取消选择。
统一状态机:选起点 → 面板仍开则同会话选终点,已关则重开选终点 →
未生效则点外部应用 → 仍未生效整轮重来。
"""
print(" 选择上周时间范围...")
last_monday, last_sunday = get_last_week_range()
start_str = last_monday.strftime("%Y-%m-%d")
end_str = last_sunday.strftime("%Y-%m-%d")
start_month = f"{last_monday.year}年{last_monday.month:02d}月"
end_month = f"{last_sunday.year}年{last_sunday.month:02d}月"
start_mmdd = last_monday.strftime("%m-%d")
for attempt in range(1, 4):
print(f" [state] 第 {attempt} 轮: {_custom_panel_state(page)}")
if not _open_custom_date_panel(page):
print(" [state] 面板未能打开,重试")
continue
_ensure_month_visible(page, start_month)
try:
_click_custom_day_cell(page, start_str)
except RuntimeError:
continue
page.wait_for_timeout(1500)
print(f" [state] 点击开始后: {_custom_panel_state(page)}")
if not _panel_months_visible(page):
# 迷你面板形态:起点选择后面板已自动关闭,重开选终点
page.wait_for_timeout(1200)
if not _open_custom_date_panel(page):
continue
_ensure_month_visible(page, end_month)
try:
_click_custom_day_cell(page, end_str)
except RuntimeError:
continue
page.wait_for_timeout(2500)
print(f" [state] 点击结束后: {_custom_panel_state(page)}")
if _custom_range_applied(page, start_mmdd):
print(f" [OK] 已选择上周区间: {start_str} ~ {end_str}")
return
# 常驻双月形态:点面板外部应用(Escape 是取消,绝不能用)
try:
page.locator(
'input[placeholder="请输入店铺名称"]'
).first.click(timeout=4000)
except Exception:
page.mouse.click(30, 400)
page.wait_for_timeout(3000)
if _custom_range_applied(page, start_mmdd):
print(f" [OK] 已选择上周区间(点外部应用): {start_str} ~ {end_str}")
return
print(f" [state] 本轮未生效: {_custom_panel_state(page)}")
select_last_week_in_old_datepicker(page, "peer_store")
try:
calendar.wait_for(state="hidden", timeout=5000)
except Exception:
pass
page.wait_for_timeout(5000)
print(" 已切换到上周数据")
_dump_peer_filter_debug(page, "range_not_applied")
raise RuntimeError(
f"自定义日期区间未生效: 期望 {start_str} ~ {end_str}"
)
def reselect_last_week(page):
"""切换类目后重新选择上周日期(点击日期选择器 → 按周查询 → 点日期)"""
# 重新点击日期选择器
try:
date_picker = page.locator('.jmtd-date-picker .jmtd-base-input-top').first
date_picker.click()
except Exception:
page.locator('text=按天查询').first.click()
page.wait_for_timeout(1500)
# 点击按周查询
try:
week_switch = page.locator('.jmtd-date-picker-switch-panel-item', has_text="按周查询").first
week_switch.click()
except Exception:
page.evaluate("""() => {
const items = document.querySelectorAll('.jmtd-date-picker-switch-panel-item');
for (const item of items) {
if (item.textContent.includes('按周查询')) {
item.click();
return true;
}
def _selector_option_center(page: Page, label: str, *, exclude_toolbar: bool = True):
"""在下拉浮层里按精确文本定位 jmt-selector 选项,返回真实坐标。
筛选栏控件本身也是 button.jmt-selector(显示当前选中值),必须
排除,否则会把控件当选项点到。
"""
return page.evaluate(
"""(label) => {
const visible = (el) => {
const r = el.getBoundingClientRect();
return r.width > 0 && r.height > 0;
};
const buttons = Array.from(
document.querySelectorAll('div.jmt-selector')
).filter(visible);
for (const btn of buttons) {
if (btn.closest('.industry-top-rank-toolbar')) continue;
const lab = btn.querySelector('.jmt-label');
if (lab && (lab.innerText || '').trim() === label) {
const r = btn.getBoundingClientRect();
return {x: Math.round(r.x + r.width / 2),
y: Math.round(r.y + r.height / 2)};
}
}""")
page.wait_for_timeout(1500)
select_last_week_in_old_datepicker(page, "peer_store_reselect")
page.wait_for_timeout(8000)
}
return null;
}""",
label,
)
def switch_category(
@@ -496,113 +664,174 @@ def switch_category(
category_name: str,
parent_category: str | None = None,
) -> bool:
"""切换商家榜单类目;只有选中态和榜单刷新均验证后才成功。"""
"""新版行业榜单:jmt-selector 层级下拉选类目 → 点「查询」→ 校验表格。
一级选项直接点击;二级选项先 hover 一级展开子面板再点击。下拉的
选项与筛选栏控件是同款 button.jmt-selector,靠排除工具栏区分。
"""
label = f"{parent_category} → {category_name}" if parent_category else category_name
print(f" 切换类目到: {label}")
expected_path = _expected_category_path(category_name, parent_category)
selector = None
try:
selector = _visible_category_selector(page)
previous_fingerprint = _ranking_table_fingerprint(page)
current_path = _selected_category_path(selector)
if current_path == expected_path:
if not previous_fingerprint:
print(f" [WARN] 当前类目是 {label},但榜单表格为空")
return False
print(f" [OK] 当前已是 {label},榜单表格可用")
return True
# 类目下拉的触发行为极不稳定(单击/双击/坐标点击时开时不开),
# 用多策略轮换:单击→轮询;双击→轮询;再单击→轮询,任何一刻
# 选项出现即继续。
trigger = page.locator(
".jmt--form-item-main", has_text="类目"
).first
menu = _visible_category_menu(page, selector)
_select_category_option(
page,
menu,
category_name,
parent_category,
def _dropdown_ready() -> bool:
return bool(
page.evaluate(
"() => Array.from(document.querySelectorAll('div.jmt-selector'))"
".some(el => el.getBoundingClientRect().width > 0"
" && !el.closest('.industry-top-rank-toolbar'))"
)
)
if not _wait_for_selected_category(selector, expected_path):
actual_path = _selected_category_path(selector)
print(
f" [WARN] 类目选中态校验失败: "
f"expected={expected_path}, actual={actual_path}"
)
return False
if not _wait_for_ranking_table_refresh(page, previous_fingerprint):
print(f" [WARN] 已选中 {label},但榜单表格未确认刷新")
return False
print(f" [OK] 已切换到 {label},榜单表格已刷新")
return True
except Exception as exc:
print(f" [WARN] 切换类目失败: {exc}")
return False
finally:
opened = False
for action in ("click", "dblclick", "click"):
try:
menus = page.locator(
".jmtd-dropdown-panel.jmtd-dropdown-lists:visible"
)
if menus.count() and selector is not None:
selector.locator(
".jmtd-base-input-top:visible"
).click(timeout=3000)
page.wait_for_timeout(300)
if menus.count():
page.keyboard.press("Escape")
if action == "click":
trigger.click(timeout=6000)
else:
trigger.dblclick(timeout=6000)
except Exception as exc:
print(f" [WARN] 类目控件{action}失败: {exc}")
_dump_peer_filter_debug(page, "category_control_missing")
return False
for _ in range(6):
page.wait_for_timeout(500)
if _dropdown_ready():
opened = True
break
if opened:
break
if not opened:
print(" [WARN] 类目下拉未能打开")
_dump_peer_filter_debug(page, "category_dropdown_not_open")
return False
selected = False
if parent_category and parent_category != category_name:
parent_center = _selector_option_center(page, parent_category)
if parent_center:
page.mouse.move(parent_center["x"], parent_center["y"])
page.wait_for_timeout(1500)
leaf_center = _selector_option_center(page, category_name)
if leaf_center:
page.mouse.click(leaf_center["x"], leaf_center["y"])
selected = True
else:
option_center = _selector_option_center(page, category_name)
if option_center:
page.mouse.click(option_center["x"], option_center["y"])
selected = True
if not selected:
print(f" [WARN] 未找到类目选项 {label}")
_dump_peer_filter_debug(page, f"category_option_missing_{category_name}")
try:
page.keyboard.press("Escape")
except Exception:
pass
return False
page.wait_for_timeout(1500)
# 应用筛选并等待榜单刷新
try:
query_box = page.locator("button", has_text="查询").first.bounding_box(
timeout=4000
)
if query_box:
page.mouse.click(
query_box["x"] + query_box["width"] / 2,
query_box["y"] + query_box["height"] / 2,
)
except Exception:
pass # 部分面板选择后立即生效
page.wait_for_timeout(4500)
try:
# 注意不能用共享 extract():它会把结果字符串化导致布尔失真
table_present = bool(
page.evaluate(
"() => Array.from(document.querySelectorAll('table'))"
".some(t => (t.innerText || '').includes('排名'))"
)
)
except Exception:
table_present = False
if not table_present:
print(" [WARN] 类目切换后未见榜单表格,禁止复用旧数据")
_dump_peer_filter_debug(page, f"table_missing_{category_name}")
return False
print(f" [OK] 已切换到 {label},榜单表格已刷新")
return True
def extract_ranking_table(page):
"""提取商家榜单表格数据"""
ranking_data = extract(page, """() => {
const rows = document.querySelectorAll('.jmtd-table-main tbody tr');
const result = [];
for (const row of rows) {
const cells = row.querySelectorAll('td');
if (cells.length < 7) continue;
"""提取店铺榜表格;按表头行映射列,字段名沿用旧口径。
// 排名提取:优先数字文本,其次检查 1/2/3 名图标 class
let rank = '';
const rankNumEl = cells[0].querySelector('.jmtd-table-formatter-rank-number');
if (rankNumEl) {
rank = rankNumEl.textContent.trim();
}
if (!rank) {
const rankIconEl = cells[0].querySelector('.jmtd-table-formatter-rank');
if (rankIconEl) {
const iconMatch = rankIconEl.className.match(/jmtd-table-formatter-rank-icon-(\\d)/);
if (iconMatch) {
rank = iconMatch[1];
新版列:排名/店铺信息/成交金额/成交单量/成交商品件数/访客数/浏览量/
搜索点击次数;旧字段 followers(关注人数)新版已无此列,占位 '--'。
"""
try:
# 直接 evaluate 取结构化列表(共享 extract() 会把结果字符串化)
ranking_data = page.evaluate(
"""() => {
const table = Array.from(document.querySelectorAll('table'))
.find(t => (t.innerText || '').includes('排名'));
if (!table) return [];
const trs = Array.from(table.querySelectorAll('tr'));
let headers = null;
const result = [];
for (const tr of trs) {
const cells = Array.from(tr.querySelectorAll('th,td'))
.map(c => (c.innerText || '').trim());
if (!cells.length) continue;
if (headers === null) {
if (cells.includes('排名')) headers = cells;
continue;
}
if (cells.includes('排名')) continue;
const item = {};
cells.forEach((value, index) => {
const header = headers[index] || `col${index}`;
item[header] = value;
});
result.push(item);
}
}
if (!rank) {
rank = cells[0].textContent.trim();
}
const shopLink = cells[1].querySelector('a');
const shop = shopLink ? shopLink.textContent.trim() : cells[1].textContent.trim();
const item = {
rank: rank,
shop: shop,
amount: cells[2]?.textContent?.trim() || '',
orders: cells[3]?.textContent?.trim() || '',
followers: cells[4]?.textContent?.trim() || '',
visitors: cells[5]?.textContent?.trim() || '',
searchClicks: cells[6]?.textContent?.trim() || '',
};
result.push(item);
}
return JSON.stringify(result);
}""", "[]")
return result;
}"""
)
except Exception:
ranking_data = []
if not isinstance(ranking_data, list):
ranking_data = []
rows = []
if isinstance(ranking_data, str):
try:
rows = json.loads(ranking_data)
except json.JSONDecodeError:
rows = []
header_alias = {
"排名": "rank",
"店铺信息": "shop",
"成交金额": "amount",
"成交单量": "orders",
"成交商品件数": "items",
"访客数": "visitors",
"浏览量": "views",
"搜索点击次数": "searchClicks",
}
for raw in ranking_data if isinstance(ranking_data, list) else []:
shop_text = str(raw.get("店铺信息", "")).splitlines()
item = {
"rank": str(raw.get("排名", "")).strip(),
"shop": shop_text[0].strip() if shop_text else "",
"amount": str(raw.get("成交金额", "")).strip(),
"orders": str(raw.get("成交单量", "")).strip(),
"followers": "--",
"visitors": str(raw.get("访客数", "")).strip(),
"searchClicks": str(raw.get("搜索点击次数", "")).strip(),
"items": str(raw.get("成交商品件数", "")).strip(),
"views": str(raw.get("浏览量", "")).strip(),
}
rows.append(item)
return rows
@@ -1086,9 +1315,13 @@ def step_return_old_version(page: Page):
def step_goto_vender_ranks(page: Page):
print("[Step 4] 跳转到新版商家榜单...")
page.goto(VENDER_RANKS_URL, wait_until="domcontentloaded", timeout=60000)
print("[Step 4] 跳转到新版「行业榜单-店铺榜」...")
page.goto(INDUSTRY_TOP_URL, wait_until="domcontentloaded", timeout=60000)
page.wait_for_timeout(8000)
current_url = page.url or ""
if "error.html" in current_url or "industry-top" not in current_url:
_dump_peer_filter_debug(page, "industry_top_bad_landing")
raise RuntimeError(f"行业榜单页跳转异常(京东侧可能再次改版): {current_url}")
body_text = page.locator("body").inner_text(timeout=10000)
permission_markers = (
"当前身份下暂无此菜单",
@@ -1098,140 +1331,35 @@ def step_goto_vender_ranks(page: Page):
)
if any(marker in body_text for marker in permission_markers):
raise RuntimeError(
"京东商智新版商家榜单当前账号无菜单或权限,"
f"请确认该账号已开通榜单模块: {page.url}"
"京东商智新版行业榜单当前账号无菜单或权限,"
f"请确认该账号已开通榜单模块: {current_url}"
)
print(f" 已进入新版商家榜单: {page.url}")
if "店铺榜" not in body_text:
_dump_peer_filter_debug(page, "shop_rank_tab_missing")
raise RuntimeError(f"行业榜单页缺少「店铺榜」入口: {current_url}")
page.locator("text=店铺榜").first.click(timeout=10000)
page.wait_for_timeout(5000)
print(f" 已进入店铺榜: {page.url}")
# ── 搜索店铺 ──────────────────────────────────────────────────────────────────
def _locate_peer_search_input(page: Page):
"""Locate the shop search box inside the 交易榜单 section.
The vender-ranks page also has a global search box at the very top; generic
selectors (and the ``input`` first-match fallback) can hit that one and then
search nothing. Prefer the input that lives in the same container hierarchy
as the visible 交易榜单 heading.
"""
try:
temp_id = page.evaluate(
r"""() => {
const visible = (el) => {
if (!el) return false;
const style = window.getComputedStyle(el);
const rect = el.getBoundingClientRect();
return style.display !== 'none' && style.visibility !== 'hidden'
&& rect.width > 0 && rect.height > 0;
};
const isSearchish = (input) => {
const type = (input.type || '').toLowerCase();
if (['hidden', 'checkbox', 'radio', 'submit', 'button'].includes(type)) {
return false;
}
// 全局 AI 搜索框(找功能/找指标/找商品)不是店铺搜索框,永远不要选中
if (input.classList.contains('ai-search-input')) {
return false;
}
// 交易榜单区块的店铺搜索框(无 placeholder,靠 class 精确命中)
if (input.classList.contains('jmtd-input-input')
|| input.classList.contains('jmt-input-input')) {
return true;
}
const placeholder = (input.placeholder || '').trim();
const parentText = input.closest('.jmtd-base-input')?.textContent || '';
return Boolean(placeholder) || /搜索|查询|店铺|商家/.test(parentText);
};
const headings = Array.from(document.querySelectorAll('*'))
.filter((el) => visible(el) && (el.textContent || '').trim() === '交易榜单');
for (const heading of headings) {
let current = heading.parentElement;
for (let depth = 0; current && depth < 15; depth += 1, current = current.parentElement) {
const inputs = Array.from(current.querySelectorAll('input'))
.filter(visible)
.filter(isSearchish);
if (inputs.length === 1) {
const tempId = '__peer_search_input_' + Date.now() + '_' + Math.random().toString(36).slice(2);
inputs[0].setAttribute('id', tempId);
return tempId;
}
}
}
return null;
}"""
)
except Exception:
return None
if not isinstance(temp_id, str) or not temp_id:
return None
try:
candidate = page.locator(f"#{temp_id}").first
if candidate.count() == 1 and candidate.is_visible(timeout=2_000):
print(" [OK] 已定位交易榜单区块内的店铺搜索框")
return candidate
except Exception:
pass
return None
def search_shop(page: Page, shop_name: str):
"""在商家榜单页面搜索指定店铺"""
"""新版店铺榜:固定 placeholder 的店铺搜索框过滤榜单。"""
print(f" 搜索店铺: {shop_name}")
# 优先定位“交易榜单”区块内的店铺搜索框,避免命中页面顶部全局搜索框
search_input = _locate_peer_search_input(page)
# 尝试多种搜索框选择器
if search_input is None:
for selector in [
'.jmtd-base-input input.jmtd-input-input',
'.jmtd-base-input input.jmt-input-input',
'input[placeholder*="搜索"]',
'input[placeholder*="店铺"]',
'input[placeholder*="商家"]',
'input[type="search"]',
'.search-input input',
'.jmtd-search input',
]:
try:
el = page.locator(selector).first
if el.is_visible(timeout=2000):
# 排除全局 AI 搜索框,避免把品牌名输到"找功能/找指标/找商品"
if el.evaluate("(input) => input.classList.contains('ai-search-input')"):
continue
search_input = el
break
except Exception:
continue
if not search_input:
# 兜底:优先 jmtd-input-input,其次任意 input(仍排除 AI 搜索框)
try:
fallback = page.locator('input.jmtd-input-input, input.jmt-input-input').first
if fallback.is_visible(timeout=2000):
search_input = fallback
except Exception:
pass
if not search_input:
search_input = page.locator('input').first
print(" [WARN] 未找到明确的搜索框,使用第一个 input")
# 清空并输入店铺名
search_input = page.locator('input[placeholder="请输入店铺名称"]').first
try:
search_input.wait_for(state="visible", timeout=10000)
except Exception as exc:
_dump_peer_filter_debug(page, "shop_search_input_missing")
raise RuntimeError(f"未找到店铺搜索框: {exc}") from exc
search_input.click()
search_input.fill("")
page.wait_for_timeout(500)
page.wait_for_timeout(300)
search_input.fill(shop_name)
page.wait_for_timeout(1000)
# 尝试点击搜索按钮或按回车
try:
search_btn = page.locator('button:has-text("搜索"), button:has-text("查询"), .search-btn, .jmtd-search-btn').first
search_btn.click(timeout=3000)
except Exception:
search_input.press("Enter")
page.wait_for_timeout(8000)
page.wait_for_timeout(800)
search_input.press("Enter")
page.wait_for_timeout(4000)
print(f" [OK] 搜索完成: {shop_name}")
-1
View File
@@ -52,7 +52,6 @@ MAX_NOTIFICATION_BINDINGS_PER_PERSON = 16
NOTIFICATION_CAPABILITIES: Mapping[str, str] = MappingProxyType(
{
"content.marketing_report.daily": "正式执行且启用发送时投递最终营销日报。",
"content.relogin.weekly": "刷新期间投递各平台扫码二维码,并在平台失败时投递失败汇总。",
"product.alert.daily": "命中销量下滑规则且尚未重复通知时投递告警。",
"product.market_rank": "存在本次新生成的平台报告链接且尚未重复通知时投递。",
"product.online_stock_alert.daily": "线上商品当前库存低于 3 时向配置接收人发送库存预警。",
+117
View File
@@ -0,0 +1,117 @@
# 运维审计库(ops_* 表)
工作流执行历史与登录 cookie 遥测的 PostgreSQL 落库。本地 `var/runs/**/run.json`
仍是权威源,这里做的是**跨机器可查、可长期留存的关系型镜像**。
## 为什么需要
- 本地 journal 只在部署机磁盘上:换机、扩容、清理后历史就断了。
- `var/logs` 是文本,做不了「近 30 天哪些工作流失败最多」这类查询。
- cookie 状态原先只存在 `keepalive_status.json`(只有最新值),无法回答
「登录态到底能活几天」。
## 表与视图
| 对象 | 粒度 | 用途 |
|---|---|---|
| `ops_workflow_runs` | 一次运行一行 | 状态、模式、耗时、步骤计数、错误、journal 路径、主机 |
| `ops_workflow_run_steps` | 一次尝试一行 | 含重试;`exit_code`、错误、耗时 |
| `ops_account_cookie_events` | 一次观测一行 | 事件流;只存指纹,不存 cookie 明文 |
| `ops_account_cookie_lifetimes`(视图) | 一份 cookie 一行 | 从首次出现到失效的寿命 |
| `ops_account_cookie_stats`(视图) | 一个账号一行 | 平均 / 中位 / 最短 / 最长寿命 |
## 自动建表
首次写入时执行 `SCHEMA_SQL`(幂等 DDL),无需手工迁移。也可以用:
```bash
uv run gyxx audit schema > 001_ops_audit.sql # 导出 DDL 交 DBA 预建
psql "$GYXX_POSTGRES_DSN" -f 001_ops_audit.sql
```
## 查询
```bash
uv run gyxx audit runs --workflow content.metrics.self_operated.daily --limit 20
uv run gyxx audit runs --status failed --limit 50
uv run gyxx audit cookies --account douyin-shop --limit 50
uv run gyxx audit cookie-stats # 各账号失效规律
uv run gyxx audit cookie-stats --account douyin-shop # 单账号每次的寿命明细
```
都支持 `--json`,方便接到面板或告警脚本。
直接 SQL 也一样:
```sql
-- 最近 7 天失败最多的 10 个工作流
SELECT workflow_id, COUNT(*) AS failures
FROM ops_workflow_runs
WHERE status = 'failed' AND started_at > NOW() - INTERVAL '7 days'
GROUP BY workflow_id ORDER BY failures DESC LIMIT 10;
-- 每次失败卡在哪个步骤
SELECT r.workflow_id, s.step_id, COUNT(*) AS n
FROM ops_workflow_run_steps s
JOIN ops_workflow_runs r USING (run_id)
WHERE s.status = 'failed' AND r.started_at > NOW() - INTERVAL '7 days'
GROUP BY 1, 2 ORDER BY n DESC;
-- 哪个账号的登录态最短命(预警阈值参考)
SELECT account_id, sessions, expired_sessions,
avg_lifetime_seconds / 86400.0 AS avg_days,
min_lifetime_seconds / 86400.0 AS min_days,
last_failed_at
FROM ops_account_cookie_stats ORDER BY min_lifetime_seconds NULLS LAST;
```
## 怎么读「失效规律」
一次 cookie 代由 `fingerprint` 标识(需求域名/名称/值哈希排序后取摘要,与文件顺序无关):
- `first_healthy_at` → `first_failed_at` = `lifetime_seconds`:**实际可用时长**
- `healthy_to_failure_seconds`:最后一次探活成功到发现失效,反映**探测精度**
- `declared_expires_at`:cookie 自带的最早过期时间
把 `lifetime_seconds` 和 `declared_expires_at` 对比,能区分三种情况:
1. 实际寿命接近声明寿命 → 平台按声明失效,可按 `declared_expires_at` 提前预警;
2. 实际远短于声明 → 服务端提前吊销(常见于多端登录、风控);
3. `declared_expires_at` 为空(session cookie)→ 只能靠观测,`min_lifetime_seconds`
就是安全刷新周期的上界。
`fingerprint` 变化表示换了一份 cookie(重新登录),不算失效,也不会把两段寿命混在一起。
## 采集点
| 来源 | 触发 | `probe_source` |
|---|---|---|
| `CookieGate.check` | 每次工作流运行前的门禁判定 | `gate` |
| `AccountKeepaliveManager` | 常驻调度器每 `keepalive_interval_seconds`(默认 600s) | `keepalive` |
| `RunIndex.index_journal` | 每次运行开始/结束/异常 | — |
keepalive 是主要的精度来源:默认 10 分钟一次,同类状态一小时内只落一行,
状态翻转立即落行,所以「首次失效时间」误差不超过一个探活周期。
## 容错
审计**永不阻断业务**:
- 数据库不可达 / 缺 DSN / 建表失败 → 写入返回 `False`,调用方继续执行;
- 连续失败 3 次后熔断冷却(默认 300s),避免每个工作流都卡在连接超时;
- 失败记录写到 `<GYXX_DATA_ROOT>/logs/ops-audit/audit-failures.log`。
## 开关
| 变量 | 默认 | 说明 |
|---|---|---|
| `GYXX_OPS_AUDIT` | 开 | 设 `off` 关闭全部审计写入 |
| `GYXX_OPS_AUDIT_HOST` | 主机名 | 多机部署时写入手工标识 |
| `GYXX_OPS_AUDIT_FAILURE_COOLDOWN` | 300 | 熔断冷却秒数 |
## 不存什么
- **cookie 值**:只存哈希指纹;排查具体 cookie 请走 vault 文件。
- **日志正文**:`ops_workflow_runs` 只存 `journal_path` 指向本地文件;
完整堆栈仍在 `var/logs/console-restarts/`。
- **凭据**:DSN 只从 `GYXX_POSTGRES_DSN`(或 `PG_*`)读,不落表。
+5 -1
View File
@@ -1,5 +1,6 @@
"""Operational run indexing and durable outbox services."""
"""Operational run indexing, audit telemetry and durable outbox services."""
from .audit import AuditStore, CookieSnapshot, summarize_cookies
from .effects import (
EffectAlreadyApplied,
EffectClaim,
@@ -10,6 +11,8 @@ from .index import RunIndex, RunRecord
from .outbox import IdempotencyConflict, Outbox, OutboxMessage
__all__ = [
"AuditStore",
"CookieSnapshot",
"IdempotencyConflict",
"EffectAlreadyApplied",
"EffectClaim",
@@ -19,4 +22,5 @@ __all__ = [
"OutboxMessage",
"RunIndex",
"RunRecord",
"summarize_cookies",
]
+901
View File
@@ -0,0 +1,901 @@
"""PostgreSQL-backed operational audit: workflow run history and cookie telemetry.
设计原则
--------
1. **可观测性不拖累业务**:审计写入永远不向调用方抛异常。数据库不可达时
只记录一条 failure 日志并熔断冷却,工作流照常执行。
2. **本地 journal 仍是权威源**:本模块是镜像/汇总,供跨机器查询与长期留存;
`var/runs/**/run.json` 不会被替代。
3. **不落明文凭据**:cookie 只存指纹(域名+名称+值哈希),值本身永不入库。
环境开关
--------
``GYXX_OPS_AUDIT=off`` 关闭审计(默认开启)
``GYXX_OPS_AUDIT_HOST`` 覆盖主机标识(默认 socket.gethostname())
``GYXX_OPS_AUDIT_FAILURE_COOLDOWN`` 熔断冷却秒数(默认 300)
``GYXX_POSTGRES_DSN`` 与动态配置共用同一 DSN
"""
from __future__ import annotations
import hashlib
import json
import os
import socket
from contextlib import contextmanager
from dataclasses import dataclass
from datetime import date, datetime, timezone
from pathlib import Path
from typing import Any, Iterator, Mapping, Sequence
SCHEMA_VERSION = 1
OPS_AUDIT_ENV = "GYXX_OPS_AUDIT"
OPS_AUDIT_HOST_ENV = "GYXX_OPS_AUDIT_HOST"
OPS_AUDIT_COOLDOWN_ENV = "GYXX_OPS_AUDIT_FAILURE_COOLDOWN"
_DEFAULT_COOLDOWN_SECONDS = 300
_MAX_CONSECUTIVE_FAILURES = 3
_MAX_TEXT_CHARS = 4000
_SCHEMA_LOCK_KEY = "gyxx:ops-audit:migration"
HEALTHY_STATES = frozenset({"healthy"})
FAILED_STATES = frozenset({"expired", "not_logged_in"})
_COOKIE_STATES = frozenset(
{"healthy", "expired", "not_logged_in", "probe_error", "busy", "disabled"}
)
SCHEMA_SQL = """
CREATE TABLE IF NOT EXISTS ops_workflow_runs (
run_id TEXT PRIMARY KEY,
workflow_id TEXT NOT NULL,
business_date DATE NOT NULL,
mode TEXT NOT NULL DEFAULT 'unknown',
shadow BOOLEAN NOT NULL DEFAULT FALSE,
status TEXT NOT NULL,
started_at TIMESTAMPTZ,
ended_at TIMESTAMPTZ,
duration_seconds INTEGER,
error TEXT,
step_total INTEGER NOT NULL DEFAULT 0,
step_success INTEGER NOT NULL DEFAULT 0,
step_failed INTEGER NOT NULL DEFAULT 0,
step_skipped INTEGER NOT NULL DEFAULT 0,
step_running INTEGER NOT NULL DEFAULT 0,
journal_path TEXT NOT NULL DEFAULT '',
host TEXT NOT NULL DEFAULT '',
indexed_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX IF NOT EXISTS ops_workflow_runs_workflow_idx
ON ops_workflow_runs (workflow_id, started_at DESC);
CREATE INDEX IF NOT EXISTS ops_workflow_runs_status_idx
ON ops_workflow_runs (status, started_at DESC);
CREATE INDEX IF NOT EXISTS ops_workflow_runs_business_date_idx
ON ops_workflow_runs (business_date, workflow_id);
CREATE TABLE IF NOT EXISTS ops_workflow_run_steps (
run_id TEXT NOT NULL
REFERENCES ops_workflow_runs (run_id) ON DELETE CASCADE,
step_id TEXT NOT NULL,
attempt INTEGER NOT NULL DEFAULT 1,
status TEXT NOT NULL DEFAULT '',
exit_code INTEGER,
started_at TIMESTAMPTZ,
ended_at TIMESTAMPTZ,
duration_seconds INTEGER,
error TEXT,
PRIMARY KEY (run_id, step_id, attempt)
);
CREATE INDEX IF NOT EXISTS ops_workflow_run_steps_step_idx
ON ops_workflow_run_steps (step_id, status);
CREATE TABLE IF NOT EXISTS ops_account_cookie_events (
id BIGSERIAL PRIMARY KEY,
account_id TEXT NOT NULL,
observed_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
state TEXT NOT NULL,
probe_status TEXT NOT NULL DEFAULT '',
probe_source TEXT NOT NULL DEFAULT '',
cookie_count INTEGER,
required_count INTEGER,
fingerprint TEXT NOT NULL DEFAULT '',
declared_expires_at TIMESTAMPTZ,
detail TEXT NOT NULL DEFAULT '',
workflow_id TEXT NOT NULL DEFAULT '',
run_id TEXT NOT NULL DEFAULT '',
host TEXT NOT NULL DEFAULT ''
);
CREATE INDEX IF NOT EXISTS ops_account_cookie_events_account_idx
ON ops_account_cookie_events (account_id, observed_at DESC);
CREATE INDEX IF NOT EXISTS ops_account_cookie_events_state_idx
ON ops_account_cookie_events (state, observed_at DESC);
CREATE INDEX IF NOT EXISTS ops_account_cookie_events_fingerprint_idx
ON ops_account_cookie_events (account_id, fingerprint);
-- 一段「cookie 代」的生命周期:同一 fingerprint 从首次出现到失效。
-- first_failed_at 只取最后一次 healthy 之后的失败,避免恢复后再次失败时口径前移。
CREATE OR REPLACE VIEW ops_account_cookie_lifetimes AS
WITH per_fingerprint AS (
SELECT
account_id,
fingerprint,
MIN(observed_at) AS first_seen_at,
MIN(observed_at) FILTER (WHERE state = 'healthy') AS first_healthy_at,
MAX(observed_at) FILTER (WHERE state = 'healthy') AS last_healthy_at,
MAX(declared_expires_at) AS declared_expires_at
FROM ops_account_cookie_events
WHERE fingerprint <> ''
GROUP BY account_id, fingerprint
)
SELECT
p.account_id,
p.fingerprint,
p.first_seen_at,
p.first_healthy_at,
p.last_healthy_at,
p.declared_expires_at,
f.first_failed_at,
(f.first_failed_at IS NOT NULL) AS expired,
CASE
WHEN p.first_healthy_at IS NOT NULL AND p.last_healthy_at IS NOT NULL
THEN EXTRACT(EPOCH FROM (p.last_healthy_at - p.first_healthy_at))::BIGINT
END AS healthy_span_seconds,
CASE
WHEN p.first_healthy_at IS NOT NULL AND f.first_failed_at IS NOT NULL
THEN EXTRACT(EPOCH FROM (f.first_failed_at - p.first_healthy_at))::BIGINT
END AS lifetime_seconds,
CASE
WHEN p.last_healthy_at IS NOT NULL AND f.first_failed_at IS NOT NULL
THEN EXTRACT(EPOCH FROM (f.first_failed_at - p.last_healthy_at))::BIGINT
END AS healthy_to_failure_seconds
FROM per_fingerprint p
LEFT JOIN LATERAL (
SELECT MIN(e.observed_at) AS first_failed_at
FROM ops_account_cookie_events e
WHERE e.account_id = p.account_id
AND e.fingerprint = p.fingerprint
AND e.state IN ('expired', 'not_logged_in')
AND (p.last_healthy_at IS NULL OR e.observed_at > p.last_healthy_at)
) f ON TRUE;
-- 按账号汇总的失效规律:平均/中位/最短/最长寿命,直接支撑「多久失效一次」。
CREATE OR REPLACE VIEW ops_account_cookie_stats AS
SELECT
account_id,
COUNT(*) AS sessions,
COUNT(*) FILTER (WHERE expired) AS expired_sessions,
AVG(lifetime_seconds)::BIGINT AS avg_lifetime_seconds,
PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY lifetime_seconds)::BIGINT
AS median_lifetime_seconds,
MIN(lifetime_seconds) AS min_lifetime_seconds,
MAX(lifetime_seconds) AS max_lifetime_seconds,
MAX(first_failed_at) AS last_failed_at,
MAX(last_healthy_at) AS last_healthy_at
FROM ops_account_cookie_lifetimes
WHERE lifetime_seconds IS NOT NULL
GROUP BY account_id;
CREATE TABLE IF NOT EXISTS ops_audit_schema (
schema_version INTEGER PRIMARY KEY,
applied_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
"""
# ----------------------------------------------------------------------
# 纯函数:cookie 快照
# ----------------------------------------------------------------------
@dataclass(frozen=True, slots=True)
class CookieSnapshot:
"""一次静态检查得到的 cookie 画像,不含任何明文值。"""
cookie_count: int
required_count: int
fingerprint: str
declared_expires_at: datetime | None
def to_dict(self) -> dict[str, Any]:
return {
"cookie_count": self.cookie_count,
"required_count": self.required_count,
"fingerprint": self.fingerprint,
"declared_expires_at": (
self.declared_expires_at.isoformat()
if self.declared_expires_at is not None
else None
),
}
def _domain_matches(domain: str, required_domains: Sequence[str]) -> bool:
normalized = str(domain or "").lstrip(".").casefold()
if not normalized:
return False
return any(
normalized == required.casefold() or normalized.endswith("." + required.casefold())
for required in required_domains
)
def _expires_at(cookie: Mapping[str, Any]) -> datetime | None:
raw = cookie.get("expires")
if raw is None or isinstance(raw, bool):
return None
try:
value = float(raw)
except (TypeError, ValueError):
return None
# Playwright 用 -1 表示 session cookie;0 与负数都视为「不声明」。
if value <= 0:
return None
try:
return datetime.fromtimestamp(value, tz=timezone.utc)
except (OverflowError, OSError, ValueError):
return None
def summarize_cookies(
cookies: Sequence[Mapping[str, Any]],
*,
required_domains: Sequence[str] = (),
required_names: Sequence[str] = (),
) -> CookieSnapshot:
"""Build a value-free snapshot: counts, fingerprint and earliest declared expiry.
指纹只看「必需 cookie」的域名/名称/值哈希,因此登录态真正换了一份 cookie
时指纹会变,而无关 cookie 的增删不会造成抖动。
"""
required_domains = tuple(item for item in required_domains if item)
required_names = tuple(item for item in required_names if item)
components: list[str] = []
required_count = 0
earliest: datetime | None = None
for cookie in cookies:
if not isinstance(cookie, Mapping):
continue
name = str(cookie.get("name", ""))
domain = str(cookie.get("domain", ""))
is_required = bool(required_domains) and _domain_matches(domain, required_domains)
if required_names:
is_required = is_required and name in required_names
elif not required_domains:
is_required = True
if not is_required:
continue
required_count += 1
value = str(cookie.get("value", ""))
value_hash = hashlib.sha256(value.encode("utf-8", errors="replace")).hexdigest()[:16]
components.append(f"{domain.lstrip('.')}|{name}|{value_hash}")
expiry = _expires_at(cookie)
if expiry is not None and (earliest is None or expiry < earliest):
earliest = expiry
# 浏览器写入 cookie 的顺序并不稳定;排序后哈希才能保证同一份登录态得到同一指纹。
digest = hashlib.sha256()
for component in sorted(components):
digest.update(f"{component}\n".encode("utf-8"))
return CookieSnapshot(
cookie_count=len(cookies),
required_count=required_count,
fingerprint=digest.hexdigest()[:32] if required_count else "",
declared_expires_at=earliest,
)
def snapshot_for_account(account: Any) -> CookieSnapshot | None:
"""Read an account's cookie vault and summarise it, or ``None`` when unreadable.
Shared by the gate and the keepalive manager so both record the exact same
fingerprint for the same vault contents.
"""
cookie_file = getattr(account, "cookie_file", None)
if cookie_file is None:
return None
try:
from gyxx_flow.adapters.browser import BrowserCookieStore
store = BrowserCookieStore(
cookie_file,
getattr(account, "storage_state_file", cookie_file),
)
cookies = store.load_cookies()
except Exception: # noqa: BLE001 - a missing/corrupt vault is itself a signal
return None
return summarize_cookies(
cookies,
required_domains=tuple(getattr(account, "required_cookie_domains", ()) or ()),
required_names=tuple(getattr(account, "required_cookie_names", ()) or ()),
)
# ----------------------------------------------------------------------
# 时间与文本工具
# ----------------------------------------------------------------------
def _parse_datetime(value: Any) -> datetime | None:
if isinstance(value, datetime):
return value if value.tzinfo else value.replace(tzinfo=timezone.utc)
if not isinstance(value, str) or not value.strip():
return None
text = value.strip().replace("Z", "+00:00")
try:
parsed = datetime.fromisoformat(text)
except ValueError:
return None
return parsed if parsed.tzinfo else parsed.replace(tzinfo=timezone.utc)
def _parse_date(value: Any) -> date | None:
if isinstance(value, date) and not isinstance(value, datetime):
return value
if not isinstance(value, str):
return None
try:
return date.fromisoformat(value)
except ValueError:
return None
def _seconds_between(start: datetime | None, end: datetime | None) -> int | None:
if start is None or end is None:
return None
delta = (end - start).total_seconds()
return max(0, int(delta))
def _clip(value: Any, limit: int = _MAX_TEXT_CHARS) -> str:
if value is None:
return ""
text = value if isinstance(value, str) else str(value)
return text[:limit]
def split_step_key(key: str) -> tuple[str, int]:
"""``sync.attempt-2`` -> ``("sync", 2)``; 无法解析时原样当作 step_id。"""
if ".attempt-" in key:
name, _, raw = key.rpartition(".attempt-")
try:
attempt = int(raw)
except ValueError:
return key, 1
if name and attempt > 0:
return name, attempt
return key, 1
# ----------------------------------------------------------------------
# 审计存储
# ----------------------------------------------------------------------
class AuditStore:
"""Idempotent writer for the ``ops_*`` tables.
所有公开写入方法都返回 bool,并且**永不抛出**异常;数据库问题只会导致本次
审计丢失,不会影响工作流执行。
"""
def __init__(
self,
data_root: Path | str,
*,
environment: Mapping[str, str] | None = None,
connect: Any | None = None,
clock: Any | None = None,
) -> None:
self._data_root = Path(data_root).expanduser().resolve()
self._environment = dict(os.environ if environment is None else environment)
self._connect_impl = connect
self._clock = clock or (lambda: datetime.now(timezone.utc))
self._schema_ready = False
self._consecutive_failures = 0
self._cooldown_until = 0.0
self._last_error = ""
self._failure_log = self._data_root / "logs" / "ops-audit" / "audit-failures.log"
# -- 状态 ---------------------------------------------------------
@property
def enabled(self) -> bool:
raw = str(self._environment.get(OPS_AUDIT_ENV, "")).strip().lower()
return raw not in {"0", "off", "false", "no", "disabled"}
@property
def last_error(self) -> str:
return self._last_error
def _host(self) -> str:
override = str(self._environment.get(OPS_AUDIT_HOST_ENV, "")).strip()
if override:
return override
try:
return socket.gethostname()[:128]
except OSError: # pragma: no cover - 极端环境
return ""
def _cooldown_seconds(self) -> int:
raw = str(self._environment.get(OPS_AUDIT_COOLDOWN_ENV, "")).strip()
try:
return max(0, int(raw))
except ValueError:
return _DEFAULT_COOLDOWN_SECONDS
def _in_cooldown(self) -> bool:
import time
return time.monotonic() < self._cooldown_until
# -- 连接 ---------------------------------------------------------
@contextmanager
def _connection(self) -> Iterator[Any]:
if self._connect_impl is not None:
conn = self._connect_impl()
try:
yield conn
finally:
close = getattr(conn, "close", None)
if callable(close):
close()
return
import psycopg # 延迟导入:未安装 psycopg 时整个模块仍可导入
from psycopg.rows import dict_row
dsn, kwargs = database_kwargs(self._environment)
if dsn:
conn = psycopg.connect(dsn, row_factory=dict_row)
else:
conn = psycopg.connect(**kwargs, row_factory=dict_row)
try:
yield conn
finally:
conn.close()
def _ensure_schema(self, conn: Any) -> None:
"""Create the audit objects once, serialising the first DDL pass.
Several workflows can start at the same moment; running CREATE/COMMENT
concurrently lets sessions take DDL locks in different orders and can
deadlock. The advisory lock is only taken when the schema is missing,
so steady-state processes skip the round trip entirely.
"""
if self._schema_ready:
return
with conn.cursor() as cursor:
cursor.execute(
"SELECT 1 FROM pg_class "
"WHERE relname = 'ops_audit_schema' AND relkind = 'r'"
)
initialised = cursor.fetchone() is not None
if not initialised:
with conn.cursor() as cursor:
cursor.execute(
"SELECT pg_advisory_xact_lock(hashtext(%s))",
(_SCHEMA_LOCK_KEY,),
)
cursor.execute(SCHEMA_SQL)
cursor.execute(
"INSERT INTO ops_audit_schema (schema_version) VALUES (%s) "
"ON CONFLICT (schema_version) DO NOTHING",
(SCHEMA_VERSION,),
)
conn.commit()
self._schema_ready = True
# -- 失败处理 -----------------------------------------------------
def _record_failure(self, operation: str, exc: BaseException) -> None:
import time
self._consecutive_failures += 1
self._last_error = f"{type(exc).__name__}: {exc}"[:300]
if self._consecutive_failures >= _MAX_CONSECUTIVE_FAILURES:
self._cooldown_until = time.monotonic() + self._cooldown_seconds()
try:
self._failure_log.parent.mkdir(parents=True, exist_ok=True)
payload = {
"at": self._clock().isoformat(timespec="seconds"),
"operation": operation,
"error": self._last_error,
"consecutive_failures": self._consecutive_failures,
}
with self._failure_log.open("a", encoding="utf-8") as stream:
stream.write(json.dumps(payload, ensure_ascii=False) + "\n")
except OSError:
pass
def _run_guarded(self, operation: str, action: Any) -> bool:
if not self.enabled or self._in_cooldown():
return False
try:
with self._connection() as conn:
self._ensure_schema(conn)
action(conn)
self._consecutive_failures = 0
return True
except Exception as exc: # noqa: BLE001 - 审计绝不影响业务
self._record_failure(operation, exc)
return False
# -- 运行记录 -----------------------------------------------------
def record_run(
self,
record: Mapping[str, Any],
steps: Mapping[str, Any] | None = None,
) -> bool:
"""Upsert one run and its steps from a journal payload or RunRecord dict."""
workflow_id = _clip(record.get("workflow_id"), 200)
run_id = _clip(record.get("run_id"), 200)
business_date = _parse_date(record.get("business_date"))
if not workflow_id or not run_id or business_date is None:
return False
status = _clip(record.get("status"), 32) or "unknown"
mode = _clip(record.get("mode"), 32) or "unknown"
started_at = _parse_datetime(record.get("started_at"))
ended_at = _parse_datetime(record.get("ended_at"))
step_rows = self._step_rows(run_id, steps)
counts = self._step_counts(steps, record)
def action(conn: Any) -> None:
with conn.cursor() as cursor:
cursor.execute(
"""
INSERT INTO ops_workflow_runs (
run_id, workflow_id, business_date, mode, shadow, status,
started_at, ended_at, duration_seconds, error,
step_total, step_success, step_failed, step_skipped,
step_running, journal_path, host, indexed_at
) VALUES (
%(run_id)s, %(workflow_id)s, %(business_date)s, %(mode)s,
%(shadow)s, %(status)s, %(started_at)s, %(ended_at)s,
%(duration_seconds)s, %(error)s, %(step_total)s,
%(step_success)s, %(step_failed)s, %(step_skipped)s,
%(step_running)s, %(journal_path)s, %(host)s, NOW()
)
ON CONFLICT (run_id) DO UPDATE SET
status = EXCLUDED.status,
ended_at = EXCLUDED.ended_at,
duration_seconds = EXCLUDED.duration_seconds,
error = EXCLUDED.error,
step_total = EXCLUDED.step_total,
step_success = EXCLUDED.step_success,
step_failed = EXCLUDED.step_failed,
step_skipped = EXCLUDED.step_skipped,
step_running = EXCLUDED.step_running,
journal_path = EXCLUDED.journal_path,
indexed_at = NOW()
""",
{
"run_id": run_id,
"workflow_id": workflow_id,
"business_date": business_date,
"mode": mode,
"shadow": bool(record.get("shadow", False)),
"status": status,
"started_at": started_at,
"ended_at": ended_at,
"duration_seconds": _seconds_between(started_at, ended_at),
"error": _clip(record.get("error")),
"step_total": counts["total"],
"step_success": counts["success"],
"step_failed": counts["failed"],
"step_skipped": counts["skipped"],
"step_running": counts["running"],
"journal_path": _clip(record.get("journal_path"), 1000),
"host": self._host(),
},
)
if step_rows:
cursor.executemany(
"""
INSERT INTO ops_workflow_run_steps (
run_id, step_id, attempt, status, exit_code,
started_at, ended_at, duration_seconds, error
) VALUES (
%(run_id)s, %(step_id)s, %(attempt)s, %(status)s,
%(exit_code)s, %(started_at)s, %(ended_at)s,
%(duration_seconds)s, %(error)s
)
ON CONFLICT (run_id, step_id, attempt) DO UPDATE SET
status = EXCLUDED.status,
exit_code = EXCLUDED.exit_code,
started_at = COALESCE(
ops_workflow_run_steps.started_at,
EXCLUDED.started_at
),
ended_at = EXCLUDED.ended_at,
duration_seconds = EXCLUDED.duration_seconds,
error = EXCLUDED.error
""",
step_rows,
)
conn.commit()
return self._run_guarded("record_run", action)
@staticmethod
def _step_rows(
run_id: str, steps: Mapping[str, Any] | None
) -> list[dict[str, Any]]:
if not isinstance(steps, Mapping):
return []
rows: list[dict[str, Any]] = []
for key, raw in steps.items():
if not isinstance(raw, Mapping):
continue
step_id, attempt = split_step_key(str(key))
attempt_value = raw.get("attempt")
if isinstance(attempt_value, int) and not isinstance(attempt_value, bool):
attempt = max(1, attempt_value)
started_at = _parse_datetime(raw.get("started_at"))
ended_at = _parse_datetime(raw.get("ended_at"))
exit_code = raw.get("exit_code")
rows.append(
{
"run_id": run_id,
"step_id": _clip(step_id, 200),
"attempt": attempt,
"status": _clip(raw.get("status"), 32),
"exit_code": exit_code if isinstance(exit_code, int) else None,
"started_at": started_at,
"ended_at": ended_at,
"duration_seconds": _seconds_between(started_at, ended_at),
"error": _clip(raw.get("error")),
}
)
return rows
@staticmethod
def _step_counts(
steps: Mapping[str, Any] | None, record: Mapping[str, Any]
) -> dict[str, int]:
counts = {"total": 0, "success": 0, "failed": 0, "skipped": 0, "running": 0}
if isinstance(steps, Mapping) and steps:
for raw in steps.values():
if not isinstance(raw, Mapping):
continue
status = str(raw.get("status", ""))
counts["total"] += 1
if status in counts:
counts[status] += 1
return counts
# 回退到 RunRecord 的 step_counts 投影
projection = record.get("step_counts")
if isinstance(projection, Mapping):
for name in ("success", "failed", "skipped", "running"):
value = projection.get(name)
if isinstance(value, int) and not isinstance(value, bool):
counts[name] = max(0, value)
counts["total"] = counts["success"] + counts["failed"] + counts["skipped"] + counts["running"]
return counts
# -- cookie 事件 ---------------------------------------------------
def record_cookie_event(
self,
*,
account_id: str,
state: str,
snapshot: CookieSnapshot | None = None,
probe_status: str = "",
probe_source: str = "",
detail: str = "",
workflow_id: str = "",
run_id: str = "",
observed_at: datetime | None = None,
) -> bool:
"""Append one observation. 状态必须是白名单值,避免脏数据污染统计。"""
account_id = _clip(account_id, 200)
if not account_id:
return False
state = str(state or "").strip()
if state not in _COOKIE_STATES:
raise ValueError(f"invalid cookie state: {state!r}")
def action(conn: Any) -> None:
with conn.cursor() as cursor:
cursor.execute(
"""
INSERT INTO ops_account_cookie_events (
account_id, observed_at, state, probe_status, probe_source,
cookie_count, required_count, fingerprint, declared_expires_at,
detail, workflow_id, run_id, host
) VALUES (
%(account_id)s, %(observed_at)s, %(state)s, %(probe_status)s,
%(probe_source)s, %(cookie_count)s, %(required_count)s,
%(fingerprint)s, %(declared_expires_at)s, %(detail)s,
%(workflow_id)s, %(run_id)s, %(host)s
)
""",
{
"account_id": account_id,
"observed_at": observed_at or self._clock(),
"state": state,
"probe_status": _clip(probe_status, 64),
"probe_source": _clip(probe_source, 64),
"cookie_count": snapshot.cookie_count if snapshot else None,
"required_count": snapshot.required_count if snapshot else None,
"fingerprint": snapshot.fingerprint if snapshot else "",
"declared_expires_at": (
snapshot.declared_expires_at if snapshot else None
),
"detail": _clip(detail),
"workflow_id": _clip(workflow_id, 200),
"run_id": _clip(run_id, 200),
"host": self._host(),
},
)
conn.commit()
return self._run_guarded("record_cookie_event", action)
# -- 查询(供 CLI) ------------------------------------------------
def query_runs(
self,
*,
workflow_id: str | None = None,
status: str | None = None,
business_date: str | None = None,
limit: int = 50,
) -> list[dict[str, Any]]:
conditions: list[str] = []
params: dict[str, Any] = {"limit": max(1, min(int(limit), 500))}
if workflow_id:
conditions.append("workflow_id = %(workflow_id)s")
params["workflow_id"] = workflow_id
if status:
conditions.append("status = %(status)s")
params["status"] = status
if business_date:
parsed = _parse_date(business_date)
if parsed is not None:
conditions.append("business_date = %(business_date)s")
params["business_date"] = parsed
where = f"WHERE {' AND '.join(conditions)}" if conditions else ""
sql = f"""
SELECT * FROM ops_workflow_runs
{where}
ORDER BY started_at DESC NULLS LAST, run_id DESC
LIMIT %(limit)s
"""
return self._fetch(sql, params)
def query_steps(self, run_id: str) -> list[dict[str, Any]]:
return self._fetch(
"""
SELECT * FROM ops_workflow_run_steps
WHERE run_id = %(run_id)s
ORDER BY step_id, attempt
""",
{"run_id": run_id},
)
def query_cookie_events(
self,
*,
account_id: str | None = None,
state: str | None = None,
limit: int = 50,
) -> list[dict[str, Any]]:
conditions: list[str] = []
params: dict[str, Any] = {"limit": max(1, min(int(limit), 500))}
if account_id:
conditions.append("account_id = %(account_id)s")
params["account_id"] = account_id
if state:
conditions.append("state = %(state)s")
params["state"] = state
where = f"WHERE {' AND '.join(conditions)}" if conditions else ""
sql = f"""
SELECT * FROM ops_account_cookie_events
{where}
ORDER BY observed_at DESC
LIMIT %(limit)s
"""
return self._fetch(sql, params)
def query_cookie_lifetimes(self, *, account_id: str | None = None) -> list[dict[str, Any]]:
if account_id:
return self._fetch(
"""
SELECT * FROM ops_account_cookie_lifetimes
WHERE account_id = %(account_id)s
ORDER BY first_seen_at DESC
""",
{"account_id": account_id},
)
return self._fetch(
"SELECT * FROM ops_account_cookie_lifetimes ORDER BY first_seen_at DESC",
{},
)
def query_cookie_stats(self) -> list[dict[str, Any]]:
return self._fetch(
"SELECT * FROM ops_account_cookie_stats ORDER BY account_id",
{},
)
def _fetch(self, sql: str, params: Mapping[str, Any]) -> list[dict[str, Any]]:
if not self.enabled or self._in_cooldown():
return []
try:
with self._connection() as conn:
self._ensure_schema(conn)
with conn.cursor() as cursor:
cursor.execute(sql, params)
rows = cursor.fetchall()
self._consecutive_failures = 0
return [dict(row) for row in rows]
except Exception as exc: # noqa: BLE001
self._record_failure("query", exc)
return []
# ----------------------------------------------------------------------
# 连接参数(与 PostgresWorkflowConfigStore 同口径)
# ----------------------------------------------------------------------
def database_kwargs(environment: Mapping[str, str]) -> tuple[str | None, dict[str, Any]]:
values = _apply_runtime_policy(environment)
dsn = str(values.get("GYXX_POSTGRES_DSN", "")).strip()
if dsn:
return dsn, {}
required = ("PG_HOST", "PG_PORT", "PG_DB", "PG_USER", "PG_PASSWORD")
missing = [name for name in required if not str(values.get(name, "")).strip()]
if missing:
raise RuntimeError("审计数据库配置缺失: " + ", ".join(missing))
try:
port = int(values["PG_PORT"])
except (TypeError, ValueError) as exc:
raise RuntimeError("PG_PORT 必须是整数") from exc
return None, {
"host": str(values["PG_HOST"]).strip(),
"port": port,
"dbname": str(values["PG_DB"]).strip(),
"user": str(values["PG_USER"]).strip(),
"password": str(values["PG_PASSWORD"]),
}
def _apply_runtime_policy(environment: Mapping[str, str]) -> Mapping[str, str]:
"""复用 RuntimeServicePolicy,保证与控制台/工作流看到同一个数据库地址。"""
try:
from gyxx_flow.adapters import RuntimeServicePolicy
except Exception: # pragma: no cover - 循环导入兜底
return environment
try:
return RuntimeServicePolicy().apply(dict(environment))
except Exception: # pragma: no cover
return environment
def default_store(data_root: Path | str | None = None) -> AuditStore:
"""Best-effort store resolved from the standard data root."""
if data_root is None:
try:
from gyxx_flow.core.config import Settings
data_root = Settings.from_env().data_root
except Exception: # pragma: no cover
data_root = Path.cwd() / "var"
return AuditStore(data_root)
+40
View File
@@ -86,6 +86,7 @@ class RunIndex:
resource = f"ops:run-index:{_digest(record.run_id)}"
with self._locks.acquire(resource, owner=_owner(), timeout_seconds=30):
atomic_write_json(destination, record.to_dict())
_mirror_run_to_audit(self._data_root, record, raw)
return record
def get(self, run_id: str) -> RunRecord | None:
@@ -255,3 +256,42 @@ def _digest(value: str) -> str:
import hashlib
return hashlib.sha256(value.encode("utf-8")).hexdigest()[:32]
_AUDIT_STORES: dict[str, Any] = {}
def _audit_store_for(data_root: Path) -> Any:
"""Reuse one store per data root so schema bootstrap and the circuit breaker
survive the short-lived ``RunIndex`` instances created per run."""
key = str(data_root)
store = _AUDIT_STORES.get(key)
if store is None:
from gyxx_flow.ops.audit import AuditStore
store = AuditStore(data_root)
_AUDIT_STORES[key] = store
return store
def _mirror_run_to_audit(
data_root: Path,
record: RunRecord,
journal_payload: Any,
) -> None:
"""Best-effort mirror of a run into the PostgreSQL audit tables.
Auditing is strictly additive: any failure here (missing DSN, unreachable
database, unexpected schema state) is swallowed so that journal indexing and
the workflow itself are never affected.
"""
try:
store = _audit_store_for(data_root)
if not store.enabled:
return
steps = journal_payload.get("steps") if isinstance(journal_payload, dict) else None
store.record_run(record.to_dict(), steps)
except Exception: # noqa: BLE001 - audit mirroring must never raise
return
@@ -0,0 +1,152 @@
-- Migration 001: operational audit tables (workflow runs + login cookie telemetry).
--
-- 本文件由 src/gyxx_flow/ops/audit.py 的 SCHEMA_SQL 导出,请勿手工编辑;
-- 运行时首次写入会自动执行同一份 DDL(幂等),此处仅供 DBA 审阅或预建。
--
-- 用法: psql "$GYXX_POSTGRES_DSN" -f 001_ops_audit.sql
--
-- 说明:
-- ops_workflow_runs 一次工作流运行一行,run.json 的关系型投影
-- ops_workflow_run_steps 每次尝试一行(含重试),保留 exit_code 与错误
-- ops_account_cookie_events cookie 观测事件流,只存指纹不存明文值
-- ops_account_cookie_lifetimes 一处登录态从首次出现到失效(视图)
-- ops_account_cookie_stats 每账号失效规律汇总(视图)
--
-- 日志正文仍留在 var/logs 下,数据库只存结构化元数据与指向路径。
CREATE TABLE IF NOT EXISTS ops_workflow_runs (
run_id TEXT PRIMARY KEY,
workflow_id TEXT NOT NULL,
business_date DATE NOT NULL,
mode TEXT NOT NULL DEFAULT 'unknown',
shadow BOOLEAN NOT NULL DEFAULT FALSE,
status TEXT NOT NULL,
started_at TIMESTAMPTZ,
ended_at TIMESTAMPTZ,
duration_seconds INTEGER,
error TEXT,
step_total INTEGER NOT NULL DEFAULT 0,
step_success INTEGER NOT NULL DEFAULT 0,
step_failed INTEGER NOT NULL DEFAULT 0,
step_skipped INTEGER NOT NULL DEFAULT 0,
step_running INTEGER NOT NULL DEFAULT 0,
journal_path TEXT NOT NULL DEFAULT '',
host TEXT NOT NULL DEFAULT '',
indexed_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX IF NOT EXISTS ops_workflow_runs_workflow_idx
ON ops_workflow_runs (workflow_id, started_at DESC);
CREATE INDEX IF NOT EXISTS ops_workflow_runs_status_idx
ON ops_workflow_runs (status, started_at DESC);
CREATE INDEX IF NOT EXISTS ops_workflow_runs_business_date_idx
ON ops_workflow_runs (business_date, workflow_id);
CREATE TABLE IF NOT EXISTS ops_workflow_run_steps (
run_id TEXT NOT NULL
REFERENCES ops_workflow_runs (run_id) ON DELETE CASCADE,
step_id TEXT NOT NULL,
attempt INTEGER NOT NULL DEFAULT 1,
status TEXT NOT NULL DEFAULT '',
exit_code INTEGER,
started_at TIMESTAMPTZ,
ended_at TIMESTAMPTZ,
duration_seconds INTEGER,
error TEXT,
PRIMARY KEY (run_id, step_id, attempt)
);
CREATE INDEX IF NOT EXISTS ops_workflow_run_steps_step_idx
ON ops_workflow_run_steps (step_id, status);
CREATE TABLE IF NOT EXISTS ops_account_cookie_events (
id BIGSERIAL PRIMARY KEY,
account_id TEXT NOT NULL,
observed_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
state TEXT NOT NULL,
probe_status TEXT NOT NULL DEFAULT '',
probe_source TEXT NOT NULL DEFAULT '',
cookie_count INTEGER,
required_count INTEGER,
fingerprint TEXT NOT NULL DEFAULT '',
declared_expires_at TIMESTAMPTZ,
detail TEXT NOT NULL DEFAULT '',
workflow_id TEXT NOT NULL DEFAULT '',
run_id TEXT NOT NULL DEFAULT '',
host TEXT NOT NULL DEFAULT ''
);
CREATE INDEX IF NOT EXISTS ops_account_cookie_events_account_idx
ON ops_account_cookie_events (account_id, observed_at DESC);
CREATE INDEX IF NOT EXISTS ops_account_cookie_events_state_idx
ON ops_account_cookie_events (state, observed_at DESC);
CREATE INDEX IF NOT EXISTS ops_account_cookie_events_fingerprint_idx
ON ops_account_cookie_events (account_id, fingerprint);
-- 一段「cookie 代」的生命周期:同一 fingerprint 从首次出现到失效。
-- first_failed_at 只取最后一次 healthy 之后的失败,避免恢复后再次失败时口径前移。
CREATE OR REPLACE VIEW ops_account_cookie_lifetimes AS
WITH per_fingerprint AS (
SELECT
account_id,
fingerprint,
MIN(observed_at) AS first_seen_at,
MIN(observed_at) FILTER (WHERE state = 'healthy') AS first_healthy_at,
MAX(observed_at) FILTER (WHERE state = 'healthy') AS last_healthy_at,
MAX(declared_expires_at) AS declared_expires_at
FROM ops_account_cookie_events
WHERE fingerprint <> ''
GROUP BY account_id, fingerprint
)
SELECT
p.account_id,
p.fingerprint,
p.first_seen_at,
p.first_healthy_at,
p.last_healthy_at,
p.declared_expires_at,
f.first_failed_at,
(f.first_failed_at IS NOT NULL) AS expired,
CASE
WHEN p.first_healthy_at IS NOT NULL AND p.last_healthy_at IS NOT NULL
THEN EXTRACT(EPOCH FROM (p.last_healthy_at - p.first_healthy_at))::BIGINT
END AS healthy_span_seconds,
CASE
WHEN p.first_healthy_at IS NOT NULL AND f.first_failed_at IS NOT NULL
THEN EXTRACT(EPOCH FROM (f.first_failed_at - p.first_healthy_at))::BIGINT
END AS lifetime_seconds,
CASE
WHEN p.last_healthy_at IS NOT NULL AND f.first_failed_at IS NOT NULL
THEN EXTRACT(EPOCH FROM (f.first_failed_at - p.last_healthy_at))::BIGINT
END AS healthy_to_failure_seconds
FROM per_fingerprint p
LEFT JOIN LATERAL (
SELECT MIN(e.observed_at) AS first_failed_at
FROM ops_account_cookie_events e
WHERE e.account_id = p.account_id
AND e.fingerprint = p.fingerprint
AND e.state IN ('expired', 'not_logged_in')
AND (p.last_healthy_at IS NULL OR e.observed_at > p.last_healthy_at)
) f ON TRUE;
-- 按账号汇总的失效规律:平均/中位/最短/最长寿命,直接支撑「多久失效一次」。
CREATE OR REPLACE VIEW ops_account_cookie_stats AS
SELECT
account_id,
COUNT(*) AS sessions,
COUNT(*) FILTER (WHERE expired) AS expired_sessions,
AVG(lifetime_seconds)::BIGINT AS avg_lifetime_seconds,
PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY lifetime_seconds)::BIGINT
AS median_lifetime_seconds,
MIN(lifetime_seconds) AS min_lifetime_seconds,
MAX(lifetime_seconds) AS max_lifetime_seconds,
MAX(first_failed_at) AS last_failed_at,
MAX(last_healthy_at) AS last_healthy_at
FROM ops_account_cookie_lifetimes
WHERE lifetime_seconds IS NOT NULL
GROUP BY account_id;
CREATE TABLE IF NOT EXISTS ops_audit_schema (
schema_version INTEGER PRIMARY KEY,
applied_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
@@ -1,4 +1,7 @@
import json
import time
from datetime import datetime
from pathlib import Path
from types import SimpleNamespace
import pytest
@@ -32,7 +35,7 @@ def test_refresh_then_export_waits_once_after_all_accounts(monkeypatch):
monkeypatch.setattr(chanmama, "export_video_data", lambda _driver, mtime: events.append(("export", mtime)) or "x.xlsx")
monkeypatch.setattr(chanmama, "parse_chanmama_excel", lambda _path: ([{"title": "video"}], None))
records, summaries = chanmama.refresh_then_export_accounts(object(), urls, wait_seconds=600)
records, summaries, _driver = chanmama.refresh_then_export_accounts(object(), urls, wait_seconds=600)
assert len(records) == 2
assert [item["clicked"] for item in summaries] == [2, 0]
@@ -53,7 +56,7 @@ def test_refresh_then_export_skips_wait_when_nothing_needs_refresh(monkeypatch):
monkeypatch.setattr(chanmama, "_max_excel_mtime", lambda: 0.0)
monkeypatch.setattr(chanmama, "export_video_data", lambda *_args: None)
records, _ = chanmama.refresh_then_export_accounts(object(), ["account-1"], wait_seconds=600)
records, _, _driver = chanmama.refresh_then_export_accounts(object(), ["account-1"], wait_seconds=600)
assert records == []
assert sleeps == []
@@ -158,7 +161,6 @@ def test_missing_get_data_button_does_not_trigger_refresh_wait(monkeypatch):
chanmama.refresh_then_export_accounts(object(), ["account-1"], wait_seconds=600)
assert sleeps == []
def test_create_driver_uses_runtime_cdp_profile_and_download_bindings(monkeypatch, tmp_path):
calls = []
expected = object()
@@ -182,7 +184,9 @@ def test_create_driver_uses_runtime_cdp_profile_and_download_bindings(monkeypatc
"profile_dir": str(tmp_path / "profile"),
"download_dir": str(tmp_path / "downloads"),
"cdp_url": "http://127.0.0.1:22002",
"stealthy": True,
"stealthy": False,
"real_chrome": True,
"extra_flags": ["--no-proxy-server"],
}]
@@ -205,8 +209,10 @@ def test_create_driver_starts_scrapling_without_cdp(monkeypatch, tmp_path):
assert chanmama.create_driver() is expected
assert calls[0]["cdp_url"] is None
assert calls[0]["stealthy"] is True
assert calls[0]["real_chrome"] is False
# 蝉妈妈 WAF 拉黑代理出口且压制 stealthy 补丁,必须真 Chrome + 直连。
assert calls[0]["stealthy"] is False
assert calls[0]["real_chrome"] is True
assert calls[0]["extra_flags"] == ["--no-proxy-server"]
def test_create_driver_falls_back_when_bound_cdp_is_not_listening(
@@ -240,8 +246,8 @@ def test_create_driver_falls_back_when_bound_cdp_is_not_listening(
"http://127.0.0.1:22002",
None,
]
assert all(call["stealthy"] is True for call in calls)
assert calls[1]["real_chrome"] is False
assert all(call["stealthy"] is False for call in calls)
assert all(call["real_chrome"] is True for call in calls)
def test_create_driver_keeps_explicit_remote_cdp_fail_closed(
@@ -723,3 +729,357 @@ def test_wait_for_exported_excel_tolerates_driver_errors(monkeypatch):
assert chanmama._wait_for_exported_excel(0.0, driver=_BrokenDriver()) == (
"/downloads/已有.xlsx"
)
def test_refresh_then_export_relogin_on_expired_session(monkeypatch):
"""运行中途登录态失效: 自动账密重登一次并重试当前博主,不影响其他博主。"""
events = []
navigate_calls = []
def fake_navigate(_driver, url):
navigate_calls.append(url)
if len(navigate_calls) == 1:
raise RuntimeError(
"蝉妈妈登录态失效,目标页被重定向到: "
"https://www.chanmama.com/login.html"
)
monkeypatch.setattr(chanmama, "navigate_to_target", fake_navigate)
monkeypatch.setattr(
chanmama,
"refresh_zero_exposure_videos",
lambda _driver: {
"clicked": 0, "pending": 0, "zero_without_button": 0,
"unconfirmed": 0, "pages": 1,
},
)
monkeypatch.setattr(
chanmama, "login",
lambda _driver: events.append("login") or True,
)
monkeypatch.setattr(chanmama.time, "sleep", lambda _seconds: None)
monkeypatch.setattr(chanmama, "_max_excel_mtime", lambda: 0.0)
monkeypatch.setattr(
chanmama, "export_video_data", lambda _driver, mtime: "x.xlsx"
)
monkeypatch.setattr(
chanmama, "parse_chanmama_excel", lambda _path: ([{"title": "v"}], None)
)
records, summaries, _driver = chanmama.refresh_then_export_accounts(
object(), ["account-1", "account-2"], wait_seconds=0
)
assert events == ["login"] # 整套运行只自动重登一次
assert len(records) == 2 # 两个博主最终都导出成功
assert summaries[0]["pages"] == 1 # 博主 1 重登后刷新成功
def test_refresh_then_export_no_relogin_for_other_errors(monkeypatch):
"""非登录态失效的异常不触发自动重登。"""
events = []
def fake_navigate(_driver, url):
raise RuntimeError("蝉妈妈目标页加载失败: timeout")
monkeypatch.setattr(chanmama, "navigate_to_target", fake_navigate)
monkeypatch.setattr(
chanmama, "login",
lambda _driver: events.append("login") or True,
)
monkeypatch.setattr(chanmama.time, "sleep", lambda _seconds: None)
monkeypatch.setattr(
chanmama, "refresh_zero_exposure_videos", lambda _driver: {}
)
monkeypatch.setattr(chanmama, "_max_excel_mtime", lambda: 0.0)
monkeypatch.setattr(
chanmama, "export_video_data", lambda _driver, mtime: None
)
records, _summaries, _driver = chanmama.refresh_then_export_accounts(
object(), ["account-1"], wait_seconds=0
)
assert events == [] # 普通异常不触发自动重登
assert records == [] # 导出失败不产出记录
class _RenderFakeDriver:
"""navigate_to_target 渲染链路的桩:列表只在浮层清理后才就绪。"""
def __init__(self, ready_after_dismiss=1):
self.current_url = (
"https://www.chanmama.com/bloggerRank/x.html?activeTab=aweme"
)
self.visited = []
self.dismiss_runs = 0
self.ready_after_dismiss = ready_after_dismiss
self.screenshots = []
def get(self, url):
self.visited.append(url)
def find_element(self, _by, _value):
return object()
def execute_script(self, script, *_args):
if script == chanmama._OVERLAY_DISMISS_SCRIPT:
self.dismiss_runs += 1
return 2
if script == chanmama._VIDEO_LIST_READY_SCRIPT:
return self.dismiss_runs >= self.ready_after_dismiss
raise AssertionError(f"unexpected script: {script[:60]}")
def save_screenshot(self, path):
self.screenshots.append(str(path))
Path(path).write_bytes(b"png")
return True
@property
def page_source(self):
return "<html>diag</html>"
def test_navigate_to_target_clears_overlays_before_render_wait(monkeypatch):
monkeypatch.setattr(chanmama.time, "sleep", lambda _seconds: None)
driver = _RenderFakeDriver(ready_after_dismiss=1)
assert chanmama.navigate_to_target(
driver, "https://www.chanmama.com/bloggerRank/x.html"
) is True
assert driver.dismiss_runs >= 1 # 弹窗清理后才轮到渲染判定
assert driver.visited == ["https://www.chanmama.com/bloggerRank/x.html"]
def test_navigate_to_target_timeout_saves_diagnostics(monkeypatch, tmp_path):
monkeypatch.setattr(chanmama.time, "sleep", lambda _seconds: None)
monkeypatch.setattr(chanmama, "CHANMAMA_LIST_READY_TIMEOUT_SECONDS", 0.05)
monkeypatch.setattr(chanmama, "DOWNLOAD_DIR", str(tmp_path))
driver = _RenderFakeDriver(ready_after_dismiss=99)
with pytest.raises(RuntimeError, match="蝉妈妈目标页加载失败"):
chanmama.navigate_to_target(
driver, "https://www.chanmama.com/bloggerRank/x.html"
)
assert driver.dismiss_runs >= 2 # 两次尝试各自清理过浮层
assert len(driver.screenshots) == 1
diag_dir = tmp_path / "_diag"
assert list(diag_dir.glob("render_failure_*.png"))
assert list(diag_dir.glob("render_failure_*.html"))
def test_dismiss_overlays_ignores_driver_errors():
class BoomDriver:
def execute_script(self, _script, *_args):
raise RuntimeError("execution context destroyed")
assert chanmama._dismiss_overlays(BoomDriver()) == 0
def test_wait_video_list_ready_tolerates_transient_script_errors(monkeypatch):
monkeypatch.setattr(chanmama.time, "sleep", lambda _seconds: None)
class FlakyDriver:
def __init__(self):
self.ready_calls = 0
def execute_script(self, script, *_args):
if script == chanmama._OVERLAY_DISMISS_SCRIPT:
return 0
self.ready_calls += 1
if self.ready_calls == 1:
raise RuntimeError("frame detached")
return True
chanmama._wait_video_list_ready(FlakyDriver(), 2) # 不抛即通过
def test_wait_for_exported_excel_raises_when_browser_closed_without_file(monkeypatch):
"""浏览器被外部关闭且文件未落盘: 立刻抛出关闭异常,不再空等满超时。"""
class _DeadDriver:
def execute_script(self, _script, *_args):
raise RuntimeError(
"Page.goto: Target page, context or browser has been closed"
)
monkeypatch.setattr(chanmama.time, "time", lambda: 1000.0)
monkeypatch.setattr(
chanmama.time,
"sleep",
lambda _seconds: pytest.fail("浏览器已死不应继续轮询等待"),
)
monkeypatch.setattr(chanmama, "_find_new_excel_after", lambda _pre_mtime: None)
with pytest.raises(RuntimeError, match="closed"):
chanmama._wait_for_exported_excel(0.0, driver=_DeadDriver())
def test_wait_for_exported_excel_returns_file_if_landed_before_close(monkeypatch):
"""关闭前下载已完整落盘: 仍返回文件路径,不因浏览器死亡误报失败。"""
class _DeadDriver:
def execute_script(self, _script, *_args):
raise RuntimeError(
"Download.save_as: Target page, context or browser has been closed"
)
landed = None
def fake_find(_pre_mtime):
return landed
def fake_complete(path):
return True
monkeypatch.setattr(chanmama.time, "time", lambda: 1000.0)
monkeypatch.setattr(chanmama, "_find_new_excel_after", fake_find)
monkeypatch.setattr(chanmama, "_download_complete", fake_complete)
landed = "/downloads/关闭前已落盘.xlsx"
assert chanmama._wait_for_exported_excel(0.0, driver=_DeadDriver()) == landed
def test_refresh_then_export_rebuilds_browser_when_closed_mid_run(monkeypatch):
"""导出中途浏览器被外部关闭: 重建驱动+恢复会话后重试当前博主一次。"""
events = []
navigate_calls = []
dead_driver = object()
live_driver = object()
def fake_navigate(current, url):
navigate_calls.append((current, url))
if current is dead_driver and len(navigate_calls) == 3:
# 刷新阶段两个博主都成功后,导出阶段第一次导航即发现浏览器死亡
raise RuntimeError(
"蝉妈妈目标页加载失败: x.html; Page.goto: "
"Target page, context or browser has been closed"
)
monkeypatch.setattr(chanmama, "navigate_to_target", fake_navigate)
monkeypatch.setattr(
chanmama,
"refresh_zero_exposure_videos",
lambda _driver: {
"clicked": 0, "pending": 0, "zero_without_button": 0,
"unconfirmed": 0, "pages": 1,
},
)
monkeypatch.setattr(chanmama.time, "sleep", lambda _seconds: None)
monkeypatch.setattr(chanmama, "_max_excel_mtime", lambda: 0.0)
monkeypatch.setattr(
chanmama, "export_video_data", lambda _driver, mtime: "x.xlsx"
)
monkeypatch.setattr(
chanmama, "parse_chanmama_excel", lambda _path: ([{"title": "v"}], None)
)
monkeypatch.setattr(chanmama, "verify_login", lambda _driver: True)
monkeypatch.setattr(
chanmama, "create_driver", lambda: events.append("rebuild") or live_driver
)
monkeypatch.setattr(
chanmama, "login",
lambda _driver: events.append("login") or True,
)
monkeypatch.setattr(
chanmama, "load_cookies",
lambda _driver: (_ for _ in ()).throw(
AssertionError("profile 直接恢复,不应走 cookie 文件")
),
)
class _DeadDriver:
def quit(self):
events.append("quit-dead")
dead_driver = _DeadDriver()
records, _summaries, final_driver = chanmama.refresh_then_export_accounts(
dead_driver, ["account-1", "account-2"], wait_seconds=0
)
assert events == ["quit-dead", "rebuild"] # 旧驱动清理+重建一次
assert final_driver is live_driver # 返回重建后的驱动供收尾关闭
assert len(records) == 2 # 两个博主最终都导出成功
# 第 3 次导航(导出博主 1)发现死亡,其后所有导航都在重建后的驱动上
assert all(current is live_driver for current, _ in navigate_calls[3:])
def test_refresh_then_export_rebuilds_only_once(monkeypatch):
"""环境持续杀浏览器时只重建一次,后续博主直接失败不再死循环。"""
rebuilds = []
def always_closed_navigate(_driver, _url):
raise RuntimeError(
"Page.goto: Target page, context or browser has been closed"
)
class _Driver:
def quit(self):
pass
monkeypatch.setattr(chanmama, "navigate_to_target", always_closed_navigate)
monkeypatch.setattr(
chanmama, "create_driver", lambda: rebuilds.append(1) or _Driver()
)
monkeypatch.setattr(chanmama, "verify_login", lambda _driver: True)
monkeypatch.setattr(chanmama.time, "sleep", lambda _seconds: None)
records, _summaries, _driver = chanmama.refresh_then_export_accounts(
_Driver(), ["account-1", "account-2"], wait_seconds=0
)
assert len(rebuilds) == 1
assert records == []
def test_save_cookies_writes_adapter_list_format(monkeypatch, tmp_path):
"""cookie 文件与 vault/适配器共用,必须写裸列表而不是字典。"""
cookie_file = tmp_path / "cookies.json"
monkeypatch.setattr(chanmama, "COOKIE_FILE", str(cookie_file))
class _Driver:
@staticmethod
def get_cookies():
return [{"name": "session", "domain": ".chanmama.com"}]
assert chanmama.save_cookies(_Driver()) is True
payload = json.loads(cookie_file.read_text(encoding="utf-8"))
assert payload == [{"name": "session", "domain": ".chanmama.com"}]
def test_load_cookies_accepts_list_and_legacy_dict(monkeypatch, tmp_path):
cookie_file = tmp_path / "cookies.json"
injected = []
class _Driver:
@staticmethod
def get(_url):
return None
@staticmethod
def add_cookie(cookie):
injected.append(cookie)
monkeypatch.setattr(chanmama, "COOKIE_FILE", str(cookie_file))
monkeypatch.setattr(chanmama.time, "sleep", lambda _seconds: None)
cookie_file.write_text(
json.dumps([{"name": "a", "domain": ".chanmama.com"}]),
encoding="utf-8",
)
assert chanmama.load_cookies(_Driver()) is True
assert [c["name"] for c in injected] == ["a"]
injected.clear()
cookie_file.write_text(
json.dumps(
{"cookies": [{"name": "b", "domain": ".chanmama.com"}], "timestamp": time.time()}
),
encoding="utf-8",
)
assert chanmama.load_cookies(_Driver()) is True
assert [c["name"] for c in injected] == ["b"]
@@ -21,20 +21,11 @@ def test_page_crash_requires_a_fresh_browser():
@pytest.mark.parametrize("batch", [xhs_batch, douyin_batch])
def test_non_acceptance_relogin_runs_platform_script(monkeypatch, batch):
class Policy:
enabled = False
def test_batch_tools_no_longer_embed_relogin(batch):
"""扫码重登已统一到账号 vault(gyxx accounts login),工具内不再提供。"""
calls = []
monkeypatch.setattr(batch, "current_acceptance_policy", lambda: Policy())
monkeypatch.setattr(
batch.subprocess,
"call",
lambda command, **kwargs: calls.append((command, kwargs)) or 0,
)
assert batch.do_relogin() is True
assert calls[0][0][1] == str(batch.RELOGIN_SCRIPT)
assert not hasattr(batch, "do_relogin")
assert not hasattr(batch, "RELOGIN_SCRIPT")
def test_comment_checkpoint_matches_both_note_id_and_url(tmp_path):
@@ -10,7 +10,6 @@ from gyxx_flow.modules.content_marketing.data.tools import (
analyze_comments,
analyze_note,
daily_report_card,
friday_relogin_parallel,
)
@@ -138,15 +137,3 @@ def test_note_report_sends_only_to_wang_yunlong(
assert calls[0]["user_id"] == ANALYZER_ACCEPTANCE_RECIPIENT_OPEN_ID
assert calls[0]["profile"] == "hermes-analyzer"
def test_relogin_notifications_resolve_only_to_wang_yunlong(
monkeypatch,
tmp_path: Path,
) -> None:
_enable_acceptance(monkeypatch, tmp_path)
assert (
friday_relogin_parallel.recipient_for_platform("pgy", "ou_other")
== ANALYZER_ACCEPTANCE_RECIPIENT_OPEN_ID
)
@@ -1,569 +0,0 @@
from types import SimpleNamespace
import pytest
from gyxx_flow.modules.content_marketing.data.tools import (
friday_relogin_parallel as relogin,
)
from gyxx_flow.modules.content_marketing.data.tools import relogin_transaction
def test_five_expected_platforms_are_configured():
assert set(relogin.PLATFORMS) == {
"bilibili", "douyin", "pgy", "xingtu", "xiaohongshu"
}
assert relogin.PLATFORMS["xingtu"].required_cookie == "sessionid"
assert relogin.PLATFORMS["douyin"].required_cookie == "sessionid"
def test_manual_relogin_can_select_only_xingtu():
assert relogin.resolve_platform_names(["xingtu"]) == ["xingtu"]
assert relogin.resolve_platform_names(None) == list(relogin.PLATFORMS)
def test_default_screenshot_recipients_are_routed_by_platform():
assert relogin.DEFAULT_RECIPIENTS == {
"pgy": "ou_24cc944d6e43c69c59d6560ad4e2ae6e",
"xingtu": "ou_24cc944d6e43c69c59d6560ad4e2ae6e",
"douyin": "ou_7ad5fc8012e2f741afc5346e05ffd447",
"bilibili": "ou_7ad5fc8012e2f741afc5346e05ffd447",
"xiaohongshu": "ou_7ad5fc8012e2f741afc5346e05ffd447",
}
assert relogin.recipient_for_platform("pgy") == "ou_24cc944d6e43c69c59d6560ad4e2ae6e"
assert relogin.recipient_for_platform("douyin") == "ou_7ad5fc8012e2f741afc5346e05ffd447"
assert relogin.recipient_for_platform("bilibili") == "ou_7ad5fc8012e2f741afc5346e05ffd447"
assert relogin.recipient_for_platform("pgy", "ou_override") == "ou_override"
def test_failed_platforms_are_grouped_by_their_responsible_recipient():
grouped = relogin.group_platforms_by_recipient(
["pgy", "douyin", "bilibili", "xingtu", "xiaohongshu"]
)
assert grouped == {
"ou_24cc944d6e43c69c59d6560ad4e2ae6e": ["pgy", "xingtu"],
"ou_7ad5fc8012e2f741afc5346e05ffd447": [
"douyin", "bilibili", "xiaohongshu"
],
}
def test_legacy_route_keeps_the_existing_per_platform_recipients(monkeypatch):
observed = {}
def fake_resolver(workflow_id, defaults):
observed["workflow_id"] = workflow_id
observed["defaults"] = defaults
return relogin.ResolvedNotificationRoute(
configured=False,
enabled=True,
app_profile=None,
open_ids=tuple(defaults),
)
monkeypatch.setattr(relogin, "resolve_notification_route", fake_resolver)
route = relogin.resolve_weekly_notification_route()
assert observed == {
"workflow_id": "content.relogin.weekly",
"defaults": (
"ou_24cc944d6e43c69c59d6560ad4e2ae6e",
"ou_7ad5fc8012e2f741afc5346e05ffd447",
),
}
assert relogin.recipients_for_platform("pgy", route) == (
"ou_24cc944d6e43c69c59d6560ad4e2ae6e",
)
assert relogin.recipients_for_platform("douyin", route) == (
"ou_7ad5fc8012e2f741afc5346e05ffd447",
)
def test_disabled_route_still_runs_relogin_without_sending(monkeypatch):
route = relogin.ResolvedNotificationRoute(
configured=True,
enabled=False,
app_profile="hermes-analyzer",
open_ids=(),
)
rounds = []
monkeypatch.setattr(
relogin,
"current_acceptance_policy",
lambda: SimpleNamespace(enabled=False),
)
monkeypatch.setattr(
relogin,
"resolve_weekly_notification_route",
lambda _override=None: route,
)
def fake_round_factory(_args, resolved_route, notification_failures):
assert resolved_route is route
assert notification_failures == []
def run_round(names, attempt):
rounds.append((tuple(names), attempt))
return {name: True for name in names}
return run_round
monkeypatch.setattr(relogin, "_run_round_factory", fake_round_factory)
monkeypatch.setattr(
relogin,
"_send_lark",
lambda *_args, **_kwargs: (_ for _ in ()).throw(
AssertionError("disabled route must not send Lark messages")
),
)
monkeypatch.setattr(
relogin.sys,
"argv",
["friday_relogin_parallel.py", "--platform", "pgy"],
)
assert relogin.main() == 0
assert rounds == [(("pgy",), 1)]
def test_enabled_route_sends_qr_text_and_image_to_every_recipient(
monkeypatch,
tmp_path,
):
route = relogin.ResolvedNotificationRoute(
configured=True,
enabled=True,
app_profile="hermes-analyzer",
open_ids=("ou_first123", "ou_second456"),
)
sent = []
image = tmp_path / "qrcode.png"
monkeypatch.setattr(
relogin,
"_send_lark",
lambda recipient, text=None, image=None, app_profile=None: sent.append(
(recipient, text, image, app_profile)
),
)
failures = relogin._send_qr_notification(
route,
"pgy",
override=None,
text="请扫码",
image=image,
)
assert failures == ()
assert sent == [
("ou_first123", "请扫码", None, "hermes-analyzer"),
("ou_first123", None, image, "hermes-analyzer"),
("ou_second456", "请扫码", None, "hermes-analyzer"),
("ou_second456", None, image, "hermes-analyzer"),
]
assert relogin.group_platforms_by_notification_recipient(
["pgy", "douyin"], route
) == {
"ou_first123": ["pgy", "douyin"],
"ou_second456": ["pgy", "douyin"],
}
def test_qr_delivery_aggregates_errors_and_attempts_every_recipient(
monkeypatch,
tmp_path,
):
route = relogin.ResolvedNotificationRoute(
configured=True,
enabled=True,
app_profile="hermes-analyzer",
open_ids=("ou_first123", "ou_second456"),
)
attempts = []
image = tmp_path / "qrcode.png"
def fake_send(recipient, text=None, image=None, app_profile=None):
stage = "qr_text" if text is not None else "qr_image"
attempts.append((recipient, stage, app_profile))
if (recipient, stage) == ("ou_first123", "qr_text"):
raise RuntimeError("first text rejected")
if (recipient, stage) == ("ou_second456", "qr_image"):
raise RuntimeError("second image rejected")
monkeypatch.setattr(relogin, "_send_lark", fake_send)
failures = relogin._send_qr_notification(
route,
"pgy",
override=None,
text="请扫码",
image=image,
)
assert attempts == [
("ou_first123", "qr_text", "hermes-analyzer"),
("ou_first123", "qr_image", "hermes-analyzer"),
("ou_second456", "qr_text", "hermes-analyzer"),
("ou_second456", "qr_image", "hermes-analyzer"),
]
assert failures == (
relogin.NotificationDeliveryFailure(
stage="qr_text",
platform="pgy",
recipient="ou_first123",
detail="first text rejected",
),
relogin.NotificationDeliveryFailure(
stage="qr_image",
platform="pgy",
recipient="ou_second456",
detail="second image rejected",
),
)
def test_failure_summary_aggregates_every_recipient_error(monkeypatch):
route = relogin.ResolvedNotificationRoute(
configured=True,
enabled=True,
app_profile="hermes-analyzer",
open_ids=("ou_first123", "ou_second456"),
)
attempts = []
def fail_send(recipient, **_kwargs):
attempts.append(recipient)
raise RuntimeError(f"rejected {recipient}")
monkeypatch.setattr(relogin, "_send_lark", fail_send)
failures = relogin._send_failure_summaries(
route,
["pgy", "douyin"],
override=None,
max_attempts=3,
)
assert attempts == ["ou_first123", "ou_second456"]
assert [failure.stage for failure in failures] == [
"failure_summary",
"failure_summary",
]
assert [failure.recipient for failure in failures] == attempts
assert [failure.platform for failure in failures] == [
"pgy,douyin",
"pgy,douyin",
]
def test_main_returns_failure_and_reports_all_notification_errors(
monkeypatch,
capsys,
):
route = relogin.ResolvedNotificationRoute(
configured=True,
enabled=True,
app_profile="hermes-analyzer",
open_ids=("ou_first123", "ou_second456"),
)
monkeypatch.setattr(
relogin,
"current_acceptance_policy",
lambda: SimpleNamespace(enabled=False),
)
monkeypatch.setattr(
relogin,
"resolve_weekly_notification_route",
lambda _override=None: route,
)
def fake_round_factory(_args, _route, notification_failures):
notification_failures.extend(
(
relogin.NotificationDeliveryFailure(
"qr_text", "pgy", "ou_first123", "text rejected"
),
relogin.NotificationDeliveryFailure(
"qr_image", "pgy", "ou_second456", "image rejected"
),
)
)
return lambda names, _attempt: {name: True for name in names}
monkeypatch.setattr(relogin, "_run_round_factory", fake_round_factory)
monkeypatch.setattr(
relogin.sys,
"argv",
["friday_relogin_parallel.py", "--platform", "pgy"],
)
assert relogin.main() == 2
output = capsys.readouterr().out
assert "Final relogin status: {'pgy': True}" in output
assert "Final notification status: failed (2 deliveries)" in output
assert "recipient=ou_first123 error=text rejected" in output
assert "recipient=ou_second456 error=image rejected" in output
def test_round_completes_relogin_after_notification_failure(monkeypatch, tmp_path):
route = relogin.ResolvedNotificationRoute(
configured=True,
enabled=True,
app_profile="hermes-analyzer",
open_ids=("ou_first123", "ou_second456"),
)
args = SimpleNamespace(
round_timeout=10,
screenshot_delay=0,
no_send=False,
recipient=None,
max_attempts=1,
)
failures = []
attempts = []
waits = []
class Process:
pid = 1234
def wait(self, timeout):
waits.append(timeout)
return 0
monkeypatch.setattr(relogin.subprocess, "Popen", lambda *_args, **_kwargs: Process())
monkeypatch.setattr(relogin, "environment_for_child_script", lambda *_args: {})
monkeypatch.setattr(
relogin,
"binding_from_environment",
lambda _environment: SimpleNamespace(cookie_file=tmp_path / "cookies.json"),
)
monkeypatch.setattr(relogin.time, "sleep", lambda _seconds: None)
monkeypatch.setattr(relogin, "_login_windows", lambda: {"pgy": 101})
monkeypatch.setattr(relogin, "_tile_windows", lambda _handles: None)
monkeypatch.setattr(relogin, "_capture_window", lambda _hwnd, _output: None)
monkeypatch.setattr(relogin, "_valid_cookie", lambda *_args: True)
def fake_send(recipient, text=None, image=None, **_kwargs):
stage = "qr_text" if text is not None else "qr_image"
attempts.append((recipient, stage))
if (recipient, stage) == ("ou_first123", "qr_text"):
raise RuntimeError("temporary send failure")
monkeypatch.setattr(relogin, "_send_lark", fake_send)
run_round = relogin._run_round_factory(args, route, failures)
assert run_round(["pgy"], 1) == {"pgy": True}
assert waits == [100]
assert attempts == [
("ou_first123", "qr_text"),
("ou_first123", "qr_image"),
("ou_second456", "qr_text"),
("ou_second456", "qr_image"),
]
assert failures == [
relogin.NotificationDeliveryFailure(
stage="qr_text",
platform="pgy",
recipient="ou_first123",
detail="temporary send failure",
)
]
@pytest.mark.parametrize(
("returncode", "cookie_valid", "expected_status", "expected_failure_count"),
(
(0, True, True, 0),
(1, False, False, 4),
),
)
def test_missing_qr_window_is_classified_after_relogin_result(
monkeypatch,
tmp_path,
returncode,
cookie_valid,
expected_status,
expected_failure_count,
):
route = relogin.ResolvedNotificationRoute(
configured=True,
enabled=True,
app_profile="hermes-analyzer",
open_ids=("ou_first123", "ou_second456"),
)
args = SimpleNamespace(
round_timeout=10,
screenshot_delay=0,
no_send=False,
recipient=None,
max_attempts=1,
)
failures = []
class Process:
pid = 1234
def wait(self, timeout):
assert timeout == 100
return returncode
monkeypatch.setattr(relogin.subprocess, "Popen", lambda *_args, **_kwargs: Process())
monkeypatch.setattr(relogin, "environment_for_child_script", lambda *_args: {})
monkeypatch.setattr(
relogin,
"binding_from_environment",
lambda _environment: SimpleNamespace(cookie_file=tmp_path / "cookies.json"),
)
monkeypatch.setattr(relogin.time, "sleep", lambda _seconds: None)
monkeypatch.setattr(relogin, "_login_windows", lambda: {})
monkeypatch.setattr(relogin, "_window_for_process", lambda _pid: None)
monkeypatch.setattr(relogin, "_tile_windows", lambda _handles: None)
monkeypatch.setattr(relogin, "_valid_cookie", lambda *_args: cookie_valid)
monkeypatch.setattr(
relogin,
"_send_lark",
lambda *_args, **_kwargs: pytest.fail(
"no QR window must not start a send before child classification"
),
)
run_round = relogin._run_round_factory(args, route, failures)
assert run_round(["pgy"], 1) == {"pgy": expected_status}
assert len(failures) == expected_failure_count
assert {failure.stage for failure in failures} <= {"qr_text", "qr_image"}
assert all(
failure.detail == "no visible QR window and relogin did not succeed"
for failure in failures
)
def test_lark_send_uses_shared_analyzer_user_sender_for_text_and_image(
monkeypatch,
tmp_path,
):
calls = []
image = tmp_path / "qrcode.png"
image.write_bytes(b"png")
monkeypatch.setattr(relogin, "ANALYZER_DIR", tmp_path)
monkeypatch.setenv("HERMES_ANALYZER_TOKEN", "must-not-leak")
def fake_send(**kwargs):
calls.append(kwargs)
return {"message_id": f"om_{len(calls)}"}
monkeypatch.setattr(relogin, "send_lark_bot_message", fake_send)
receipts = relogin._send_lark(
"ou_recipient123",
text="登录提醒",
image=image,
)
assert [receipt["message_id"] for receipt in receipts] == ["om_1", "om_2"]
assert calls[0]["user_id"] == "ou_recipient123"
assert calls[0]["text"] == "登录提醒"
assert "image" not in calls[0]
assert calls[1]["user_id"] == "ou_recipient123"
assert calls[1]["image"] == image
assert "text" not in calls[1]
assert all(call["profile"] == "hermes-analyzer" for call in calls)
assert all("HERMES_ANALYZER_TOKEN" not in call["env"] for call in calls)
def test_login_windows_are_classified_by_title():
assert relogin._platform_from_window_title("账号登录 - Google Chrome") == "bilibili"
assert relogin._platform_from_window_title("小红书蒲公英 - Google Chrome") == "pgy"
assert relogin._platform_from_window_title("抖音授权 - Google Chrome") == "xingtu"
assert relogin._platform_from_window_title("抖音 - 记录美好生活") == "douyin"
assert relogin._platform_from_window_title("小红书 - 你的生活兴趣社区") == "xiaohongshu"
assert relogin._platform_from_window_title("Codex") is None
def test_retry_only_failed_platforms_for_at_most_three_rounds():
calls = []
def fake_round(names, attempt):
calls.append((tuple(names), attempt))
if attempt == 1:
return {name: name != "xingtu" for name in names}
if attempt == 2:
return {name: False for name in names}
return {name: True for name in names}
result = relogin.run_with_retries(
["bilibili", "pgy", "xingtu", "xiaohongshu"],
max_attempts=3,
run_round=fake_round,
)
assert calls == [
(("bilibili", "pgy", "xingtu", "xiaohongshu"), 1),
(("xingtu",), 2),
(("xingtu",), 3),
]
assert all(result.values())
def test_stops_after_third_failed_round():
calls = []
def always_fail(names, attempt):
calls.append(attempt)
return {name: False for name in names}
result = relogin.run_with_retries(
["pgy"], max_attempts=3, run_round=always_fail
)
assert calls == [1, 2, 3]
assert result == {"pgy": False}
def test_timeout_terminates_the_complete_process_tree(monkeypatch):
calls = []
class Process:
pid = 1234
def terminate(self):
calls.append("fallback-terminate")
def wait(self, timeout):
calls.append(("wait", timeout))
monkeypatch.setattr(
relogin.subprocess,
"run",
lambda command, **kwargs: calls.append((command, kwargs)),
)
relogin._terminate_process_tree(Process())
assert calls[0][0] == ["taskkill", "/PID", "1234", "/T", "/F"]
assert calls[1] == ("wait", 10)
def test_transaction_rolls_back_files_and_profile(tmp_path):
cookie = tmp_path / "cookies.json"
storage = tmp_path / "storage.json"
profile = tmp_path / "profile"
cookie.write_text("old-cookie", encoding="utf-8")
storage.write_text("old-storage", encoding="utf-8")
profile.mkdir()
(profile / "old.txt").write_text("old-profile", encoding="utf-8")
state = relogin_transaction.begin((cookie, storage), profile)
cookie.write_text("partial", encoding="utf-8")
profile.mkdir()
(profile / "partial.txt").write_text("partial", encoding="utf-8")
relogin_transaction.rollback(state)
assert cookie.read_text(encoding="utf-8") == "old-cookie"
assert storage.read_text(encoding="utf-8") == "old-storage"
assert (profile / "old.txt").read_text(encoding="utf-8") == "old-profile"
assert not (profile / "partial.txt").exists()
@@ -416,11 +416,9 @@ def test_acceptance_invalid_session_skips_before_qr_login(
)
monkeypatch.setenv("GYXX_ACCEPTANCE_EVIDENCE_FILE", str(evidence_file))
monkeypatch.setattr(module, "is_logged_in", lambda _page: False)
monkeypatch.setattr(
module,
"wait_for_scan_login",
lambda *_args, **_kwargs: pytest.fail("QR login must remain disabled"),
)
# 扫码登录函数已整体移除;采集器不得重新引入交互登录。
assert not hasattr(module, "wait_for_scan_login")
assert not hasattr(module, "open_scan_login")
class Page:
def __init__(self):
@@ -1,494 +0,0 @@
import pytest
from gyxx_flow.modules.content_marketing import (
pgy_xhs_scraper_v2,
xingtu_scraper_v2,
)
from gyxx_flow.modules.content_marketing.data.tools import (
relogin_douyin,
relogin_pgy,
relogin_xingtu,
)
@pytest.mark.parametrize("module", [relogin_pgy, relogin_xingtu])
def test_failed_relogin_restores_cookie_and_profile(tmp_path, monkeypatch, module):
cookie = tmp_path / "cookies.json"
storage = tmp_path / "storage_state.json"
profile = tmp_path / "profile"
cookie.write_text("old-cookie", encoding="utf-8")
storage.write_text("old-storage", encoding="utf-8")
profile.mkdir()
(profile / "state.txt").write_text("old-profile", encoding="utf-8")
monkeypatch.setattr(module, "COOKIE_FILE", cookie)
monkeypatch.setattr(module, "STATE_FILE", storage)
monkeypatch.setattr(module, "PROFILE_DIR", profile)
cookie_backup, profile_backup, state_backup = module.reset_state()
assert not cookie.exists()
assert not storage.exists()
assert not profile.exists()
cookie.write_text("partial-new-cookie", encoding="utf-8")
storage.write_text("partial-new-storage", encoding="utf-8")
profile.mkdir()
(profile / "partial.txt").write_text("partial", encoding="utf-8")
module.restore_previous_state(cookie_backup, profile_backup, state_backup)
assert cookie.read_text(encoding="utf-8") == "old-cookie"
assert storage.read_text(encoding="utf-8") == "old-storage"
assert (profile / "state.txt").read_text(encoding="utf-8") == "old-profile"
assert not (profile / "partial.txt").exists()
@pytest.mark.parametrize("module", [relogin_pgy, relogin_xingtu])
def test_successful_relogin_keeps_new_cookie(tmp_path, monkeypatch, module):
cookie = tmp_path / "cookies.json"
storage = tmp_path / "storage_state.json"
profile = tmp_path / "profile"
cookie.write_text("old-cookie", encoding="utf-8")
storage.write_text("old-storage", encoding="utf-8")
profile.mkdir()
monkeypatch.setattr(module, "COOKIE_FILE", cookie)
monkeypatch.setattr(module, "STATE_FILE", storage)
monkeypatch.setattr(module, "PROFILE_DIR", profile)
cookie_backup, profile_backup, state_backup = module.reset_state()
cookie.write_text("new-cookie", encoding="utf-8")
storage.write_text("new-storage", encoding="utf-8")
profile.mkdir()
module.finish_successful_relogin(profile_backup)
assert cookie.read_text(encoding="utf-8") == "new-cookie"
assert storage.read_text(encoding="utf-8") == "new-storage"
assert cookie_backup.exists()
assert state_backup.exists()
assert not profile_backup.exists()
def test_xingtu_scan_login_opens_customer_sso_and_clicks_douyin():
class FakeLocator:
def __init__(self):
self.first = self
self.clicked = False
self.evaluated = False
def count(self):
return 1
def is_visible(self):
return True
def click(self, timeout):
self.clicked = True
def evaluate(self, script):
assert script == "e => e.click()"
self.evaluated = True
class FakePage:
url = "https://www.xingtu.cn/?redirect_uri=/ad/creator/market"
def __init__(self):
self.goto_url = None
self.selectors = []
self.douyin = FakeLocator()
def goto(self, url, **kwargs):
self.goto_url = url
self.url = url
def wait_for_timeout(self, _milliseconds):
pass
def locator(self, selector):
self.selectors.append(selector)
return self.douyin
def get_by_text(self, _pattern):
return FakeLocator()
page = FakePage()
xingtu_scraper_v2.open_scan_login(page)
assert page.goto_url == xingtu_scraper_v2.CUSTOMER_LOGIN_URL
assert 'img[src*="aweme.png"]' in page.selectors
assert page.douyin.clicked
def test_xingtu_index_page_is_recognized_as_logged_in():
class Locator:
@property
def first(self):
return self
def wait_for(self, timeout):
assert timeout == 1200
class Page:
url = "https://www.xingtu.cn/ad/creator/index"
def get_by_text(self, _text, exact=False):
return Locator()
assert xingtu_scraper_v2.is_logged_in(Page()) is True
def test_xingtu_index_page_with_xingtu_session_is_logged_in_without_market_ui():
class Locator:
@property
def first(self):
return self
def count(self):
return 0
def wait_for(self, timeout):
raise xingtu_scraper_v2.BrowserTimeoutError("not on market page")
def is_visible(self):
return False
class Context:
def cookies(self):
return [
{
"name": "sessionid",
"value": "fresh",
"domain": ".xingtu.cn",
}
]
class Page:
url = "https://www.xingtu.cn/ad/creator/index"
context = Context()
def get_by_text(self, *_args, **_kwargs):
return Locator()
def locator(self, *_args, **_kwargs):
return Locator()
assert xingtu_scraper_v2.is_logged_in(Page()) is True
def test_xingtu_index_page_without_xingtu_session_is_not_logged_in():
class Locator:
@property
def first(self):
return self
def count(self):
return 0
def wait_for(self, timeout):
raise xingtu_scraper_v2.BrowserTimeoutError("missing")
def is_visible(self):
return False
class Context:
def cookies(self):
return [
{
"name": "sessionid",
"value": "douyin-only",
"domain": ".douyin.com",
}
]
class Page:
url = "https://www.xingtu.cn/ad/creator/index"
context = Context()
def get_by_text(self, *_args, **_kwargs):
return Locator()
def locator(self, *_args, **_kwargs):
return Locator()
assert xingtu_scraper_v2.is_logged_in(Page()) is False
def test_xingtu_public_homepage_is_not_logged_in_even_with_stale_cookie():
class Context:
def cookies(self):
return [{"name": "sessionid", "value": "stale"}]
class Page:
url = "https://www.xingtu.cn/?redirect_uri=/ad/creator/market"
context = Context()
def get_by_text(self, *_args, **_kwargs):
raise AssertionError("public homepage must be rejected before marker checks")
assert xingtu_scraper_v2.is_logged_in(Page()) is False
def test_xingtu_market_page_without_business_ui_rejects_stale_cookie():
class Locator:
@property
def first(self):
return self
def wait_for(self, **_kwargs):
raise xingtu_scraper_v2.BrowserTimeoutError("missing")
def count(self):
return 0
class Context:
def cookies(self):
return [{"name": "sessionid", "value": "stale"}]
class Page:
url = "https://www.xingtu.cn/ad/creator/market"
context = Context()
def get_by_text(self, *_args, **_kwargs):
return Locator()
def locator(self, *_args, **_kwargs):
return Locator()
assert xingtu_scraper_v2.is_logged_in(Page()) is False
def test_xingtu_cookie_validation_requires_real_session_cookie(tmp_path):
cookie = tmp_path / "xingtu.json"
cookie.write_text('[{"name":"other","value":"1"}]', encoding="utf-8")
assert xingtu_scraper_v2.has_valid_login_cookie_file(cookie) is False
cookie.write_text('[{"name":"sessionid","value":"fresh"}]', encoding="utf-8")
assert xingtu_scraper_v2.has_valid_login_cookie_file(cookie) is True
def test_xingtu_login_only_does_not_report_success_when_page_action_was_swallowed(
tmp_path, monkeypatch
):
cookie = tmp_path / "xingtu.json"
cookie.write_text('[{"name":"sessionid","value":"old"}]', encoding="utf-8")
class Session:
def __enter__(self):
return self
def __exit__(self, *_args):
return False
def fetch(self, *_args, **_kwargs):
# Scrapling can log a page_action error and still return a response.
return object()
monkeypatch.setattr(xingtu_scraper_v2, "COOKIE_FILE", cookie)
monkeypatch.setattr(xingtu_scraper_v2, "DynamicSession", lambda **_kwargs: Session())
monkeypatch.setattr(xingtu_scraper_v2, "_force_cleanup_session", lambda _session: None)
assert xingtu_scraper_v2.login_only(1) == 1
def test_xingtu_login_only_verifies_persisted_profile_after_oauth_context_closes(
tmp_path, monkeypatch
):
cookie = tmp_path / "xingtu.json"
class Session:
def __enter__(self):
return self
def __exit__(self, *_args):
return False
def fetch(self, *_args, **_kwargs):
raise RuntimeError("Target page, context or browser has been closed")
monkeypatch.setattr(xingtu_scraper_v2, "COOKIE_FILE", cookie)
monkeypatch.setattr(xingtu_scraper_v2, "DynamicSession", lambda **_kwargs: Session())
monkeypatch.setattr(xingtu_scraper_v2, "_force_cleanup_session", lambda _session: None)
def verify():
cookie.write_text('[{"name":"sessionid","value":"fresh"}]', encoding="utf-8")
return True
monkeypatch.setattr(xingtu_scraper_v2, "verify_persisted_profile_login", verify)
assert xingtu_scraper_v2.login_only(1) == 0
def test_xingtu_scan_login_follows_replacement_page_after_oauth_tab_closes(monkeypatch):
class Context:
def __init__(self):
self.pages = []
def new_page(self):
raise AssertionError("a surviving replacement page should be reused")
class Page:
def __init__(self, context, url, *, closes_on_wait=False):
self.context = context
self.url = url
self.closed = False
self.closes_on_wait = closes_on_wait
def is_closed(self):
return self.closed
def wait_for_timeout(self, _milliseconds):
if self.closes_on_wait:
self.closed = True
def goto(self, url, **_kwargs):
self.url = url
context = Context()
qr_page = Page(
context,
"https://open.douyin.com/platform/oauth/pc/auth",
closes_on_wait=True,
)
market_page = Page(context, xingtu_scraper_v2.MARKET_URL)
context.pages = [qr_page, market_page]
saved = []
monkeypatch.setattr(xingtu_scraper_v2, "open_scan_login", lambda _page: None)
monkeypatch.setattr(
xingtu_scraper_v2, "is_scan_qr_page", lambda page: page is qr_page
)
monkeypatch.setattr(
xingtu_scraper_v2, "is_logged_in", lambda page: page is market_page
)
monkeypatch.setattr(xingtu_scraper_v2, "save_state", lambda page: saved.append(page))
result = xingtu_scraper_v2.wait_for_scan_login(qr_page, 2)
assert result is market_page
assert saved == [market_page]
def test_xingtu_scan_login_detects_business_tab_while_qr_tab_stays_open(monkeypatch):
class Context:
def __init__(self):
self.pages = []
class Page:
def __init__(self, context, url):
self.context = context
self.url = url
def is_closed(self):
return False
def wait_for_timeout(self, _milliseconds):
raise AssertionError("authenticated tab should be detected immediately")
context = Context()
qr_page = Page(context, "https://open.douyin.com/platform/oauth/pc/auth")
index_page = Page(context, "https://www.xingtu.cn/ad/creator/index")
context.pages = [qr_page, index_page]
saved = []
monkeypatch.setattr(xingtu_scraper_v2, "open_scan_login", lambda _page: None)
monkeypatch.setattr(
xingtu_scraper_v2, "is_logged_in", lambda page: page is index_page
)
monkeypatch.setattr(xingtu_scraper_v2, "save_state", lambda page: saved.append(page))
result = xingtu_scraper_v2.wait_for_scan_login(qr_page, 2)
assert result is index_page
assert saved == [index_page]
def test_xingtu_interrupted_transaction_is_recovered(tmp_path, monkeypatch):
cookie = tmp_path / "xingtu_cookies.json"
storage = tmp_path / "xingtu_storage_state.json"
profile = tmp_path / "profile"
cookie.write_text("old-cookie", encoding="utf-8")
storage.write_text("old-storage", encoding="utf-8")
profile.mkdir()
(profile / "old.txt").write_text("old-profile", encoding="utf-8")
monkeypatch.setattr(relogin_xingtu, "COOKIE_FILE", cookie)
monkeypatch.setattr(relogin_xingtu, "STATE_FILE", storage)
monkeypatch.setattr(relogin_xingtu, "PROFILE_DIR", profile)
cookie_backup, profile_backup, state_backup = relogin_xingtu.reset_state()
profile.mkdir()
(profile / "partial.txt").write_text("partial", encoding="utf-8")
assert relogin_xingtu.recover_interrupted_state() is True
assert cookie.read_text(encoding="utf-8") == "old-cookie"
assert storage.read_text(encoding="utf-8") == "old-storage"
assert (profile / "old.txt").read_text(encoding="utf-8") == "old-profile"
assert not (profile / "partial.txt").exists()
assert cookie_backup is not None
assert state_backup is not None
assert profile_backup is not None
def test_pgy_scan_login_clicks_qr_switch_image():
class FakeLocator:
def __init__(self):
self.first = self
self.clicked = False
self.evaluated = False
def count(self):
return 1
def is_visible(self):
return True
def click(self, timeout):
self.clicked = True
def evaluate(self, script):
assert script == "e => e.click()"
self.evaluated = True
class FakePage:
def __init__(self):
self.selectors = []
self.qr_switch = FakeLocator()
def wait_for_timeout(self, _milliseconds):
pass
def get_by_text(self, _pattern):
return FakeLocator()
def locator(self, selector):
self.selectors.append(selector)
return self.qr_switch
page = FakePage()
pgy_xhs_scraper_v2.open_scan_login(page)
assert 'img[src*="qr_code"]' in page.selectors
assert page.qr_switch.evaluated
def test_failed_douyin_relogin_restores_cookie_storage_and_profile(tmp_path, monkeypatch):
cookie = tmp_path / "douyin_cookies.json"
storage = tmp_path / "douyin_storage_state.json"
profile = tmp_path / "profile"
cookie.write_text('[{"name":"sessionid","value":"old"}]', encoding="utf-8")
storage.write_text("old-storage", encoding="utf-8")
profile.mkdir()
(profile / "old.txt").write_text("old-profile", encoding="utf-8")
monkeypatch.setattr(relogin_douyin, "COOKIE_FILE", cookie)
monkeypatch.setattr(relogin_douyin, "STATE_FILE", storage)
monkeypatch.setattr(relogin_douyin, "PROFILE_DIR", profile)
monkeypatch.setattr(relogin_douyin.subprocess, "call", lambda *_args, **_kwargs: 1)
monkeypatch.setattr(relogin_douyin.sys, "argv", ["relogin_douyin.py"])
assert relogin_douyin.main() == 1
assert cookie.read_text(encoding="utf-8") == '[{"name":"sessionid","value":"old"}]'
assert storage.read_text(encoding="utf-8") == "old-storage"
assert (profile / "old.txt").read_text(encoding="utf-8") == "old-profile"
@@ -1,410 +0,0 @@
from __future__ import annotations
import json
from pathlib import Path
from types import ModuleType
import pytest
from gyxx_flow.adapters.integration import RuntimeIntegrationCatalog
from gyxx_flow.modules.content_marketing.data.tools import (
relogin_bilibili,
relogin_douyin,
relogin_pgy,
relogin_runtime_paths,
relogin_xiaohongshu,
relogin_xingtu,
)
from gyxx_flow.modules.content_marketing.data.tools.relogin_runtime_paths import (
CONSUMER_BINDING_IDS,
child_browser_environment,
resolve_relogin_runtime_paths,
sync_relogin_state_to_consumers,
)
from gyxx_flow.modules.content_marketing.runtime_paths import PATHS
PROJECT_ROOT = Path(__file__).resolve().parents[3]
def test_relogin_paths_prefer_injected_runtime_binding(tmp_path: Path) -> None:
cookie = tmp_path / "binding" / "cookies.json"
storage = tmp_path / "binding" / "storage_state.json"
profile = tmp_path / "binding" / "profile"
resolved = resolve_relogin_runtime_paths(
"pgy",
{
"GYXX_BROWSER_COOKIE_FILE": str(cookie),
"GYXX_BROWSER_STORAGE_STATE_FILE": str(storage),
"GYXX_BROWSER_PROFILE_DIR": str(profile),
},
)
assert resolved.cookie_file == cookie.resolve()
assert resolved.storage_state_file == storage.resolve()
assert resolved.profile_dir == profile.resolve()
def test_relogin_paths_keep_legacy_standalone_layout() -> None:
resolved = resolve_relogin_runtime_paths("xiaohongshu", {})
assert resolved.cookie_file == (
PATHS.state_root / "cookies/xiaohongshu_cookies.json"
).resolve()
assert resolved.storage_state_file == (
PATHS.state_root / "cookies/xiaohongshu_storage_state.json"
).resolve()
assert resolved.profile_dir == (
PATHS.state_root / "browser-profiles/xiaohongshu"
).resolve()
def test_child_login_environment_preserves_wrapper_binding(tmp_path: Path) -> None:
cookie = tmp_path / "cookies.json"
storage = tmp_path / "storage_state.json"
profile = tmp_path / "profile"
environment = child_browser_environment(
cookie_file=cookie,
storage_state_file=storage,
profile_dir=profile,
environment={"GYXX_MODULE_ID": "content_marketing", "KEEP": "1"},
)
assert environment["GYXX_BROWSER_COOKIE_FILE"] == str(cookie.resolve())
assert environment["GYXX_BROWSER_STORAGE_STATE_FILE"] == str(
storage.resolve()
)
assert environment["GYXX_BROWSER_PROFILE_DIR"] == str(profile.resolve())
assert "GYXX_MODULE_ID" not in environment
assert environment["KEEP"] == "1"
@pytest.mark.parametrize(
("module", "platform", "login_cookie", "argv"),
[
(
relogin_bilibili,
"bilibili",
"SESSDATA",
["relogin_bilibili.py"],
),
(relogin_douyin, "douyin", "sessionid", ["relogin_douyin.py"]),
(relogin_pgy, "pgy", "web_session", ["relogin_pgy.py"]),
(
relogin_xiaohongshu,
"xiaohongshu",
"web_session",
["relogin_xiaohongshu.py"],
),
(
relogin_xingtu,
"xingtu",
"sessionid",
["relogin_xingtu.py", "--force"],
),
],
)
def test_relogin_wrapper_validates_the_same_binding_written_by_child(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
module: ModuleType,
platform: str,
login_cookie: str,
argv: list[str],
) -> None:
cookie = tmp_path / "cookies.json"
storage = tmp_path / "storage_state.json"
profile = tmp_path / "profile"
monkeypatch.setattr(module, "COOKIE_FILE", cookie)
monkeypatch.setattr(module, "STATE_FILE", storage)
monkeypatch.setattr(module, "PROFILE_DIR", profile)
monkeypatch.setattr(module.sys, "argv", argv)
monkeypatch.setenv("GYXX_MODULE_ID", "content_marketing")
if module is relogin_xingtu:
monkeypatch.setattr(module, "terminate_profile_processes", lambda: None)
monkeypatch.setattr(module, "sync_to_douyin_scraper", lambda: None)
synced: list[tuple[str, Path, Path]] = []
def fake_sync(
synced_platform: str,
*,
cookie_file: Path,
storage_state_file: Path,
) -> tuple[Path, ...]:
synced.append(
(synced_platform, cookie_file.resolve(), storage_state_file.resolve())
)
return ()
monkeypatch.setattr(module, "sync_relogin_state_to_consumers", fake_sync)
def fake_call(
_command: list[str],
*,
cwd: str,
env: dict[str, str],
) -> int:
assert cwd == str(module.PROJECT_DIR)
assert env["GYXX_BROWSER_COOKIE_FILE"] == str(cookie.resolve())
assert env["GYXX_BROWSER_STORAGE_STATE_FILE"] == str(storage.resolve())
assert env["GYXX_BROWSER_PROFILE_DIR"] == str(profile.resolve())
assert "GYXX_MODULE_ID" not in env
cookie.write_text(
json.dumps([{"name": login_cookie, "value": "fresh"}]),
encoding="utf-8",
)
storage.write_text("{}", encoding="utf-8")
profile.mkdir()
return 0
monkeypatch.setattr(module.subprocess, "call", fake_call)
assert module.main() == 0
assert cookie.is_file()
assert storage.is_file()
assert profile.is_dir()
assert synced == [(platform, cookie.resolve(), storage.resolve())]
@pytest.mark.parametrize(
("module", "platform", "login_cookie", "cookie_domain", "argv"),
[
(
relogin_bilibili,
"bilibili",
"SESSDATA",
".bilibili.com",
["relogin_bilibili.py"],
),
(
relogin_douyin,
"douyin",
"sessionid",
".douyin.com",
["relogin_douyin.py"],
),
(
relogin_pgy,
"pgy",
"web_session",
".xiaohongshu.com",
["relogin_pgy.py"],
),
(
relogin_xiaohongshu,
"xiaohongshu",
"web_session",
".xiaohongshu.com",
["relogin_xiaohongshu.py"],
),
(
relogin_xingtu,
"xingtu",
"sessionid",
".xingtu.cn",
["relogin_xingtu.py", "--force"],
),
],
)
def test_relogin_wrapper_restores_previous_state_when_publish_fails(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
module: ModuleType,
platform: str,
login_cookie: str,
cookie_domain: str,
argv: list[str],
) -> None:
cookie = tmp_path / "cookies.json"
storage = tmp_path / "storage_state.json"
profile = tmp_path / "profile"
old_cookie = '[{"name":"old","value":"old-session"}]'
old_storage = '{"cookies":[{"name":"old"}]}'
cookie.write_text(old_cookie, encoding="utf-8")
storage.write_text(old_storage, encoding="utf-8")
profile.mkdir()
(profile / "old-state.txt").write_text("old-profile", encoding="utf-8")
monkeypatch.setattr(module, "COOKIE_FILE", cookie)
monkeypatch.setattr(module, "STATE_FILE", storage)
monkeypatch.setattr(module, "PROFILE_DIR", profile)
monkeypatch.setattr(module.sys, "argv", argv)
if module is relogin_xingtu:
monkeypatch.setattr(module, "terminate_profile_processes", lambda: None)
monkeypatch.setattr(module, "sync_to_douyin_scraper", lambda: None)
def fake_call(
_command: list[str],
*,
cwd: str,
env: dict[str, str],
) -> int:
assert cwd == str(module.PROJECT_DIR)
assert env["GYXX_BROWSER_COOKIE_FILE"] == str(cookie.resolve())
cookie.write_text(
json.dumps(
[
{
"name": login_cookie,
"value": "fresh-session",
"domain": cookie_domain,
}
]
),
encoding="utf-8",
)
storage.write_text('{"cookies":[],"origins":[]}', encoding="utf-8")
profile.mkdir()
return 0
def fail_publish(
synced_platform: str,
*,
cookie_file: Path,
storage_state_file: Path,
) -> tuple[Path, ...]:
assert synced_platform == platform
assert cookie_file == cookie
assert storage_state_file == storage
raise OSError("simulated consumer publish failure")
monkeypatch.setattr(module.subprocess, "call", fake_call)
monkeypatch.setattr(module, "sync_relogin_state_to_consumers", fail_publish)
assert module.main() == 1
assert cookie.read_text(encoding="utf-8") == old_cookie
assert storage.read_text(encoding="utf-8") == old_storage
assert (profile / "old-state.txt").read_text(encoding="utf-8") == (
"old-profile"
)
@pytest.mark.parametrize(
("platform", "cookie_name", "cookie_domain"),
[
("bilibili", "SESSDATA", ".bilibili.com"),
("douyin", "sessionid", ".douyin.com"),
("pgy", "web_session", ".xiaohongshu.com"),
("xiaohongshu", "web_session", ".xiaohongshu.com"),
("xingtu", "sessionid", ".xingtu.cn"),
],
)
def test_successful_relogin_atomically_updates_unique_consumer_state(
tmp_path: Path,
platform: str,
cookie_name: str,
cookie_domain: str,
) -> None:
data_root = tmp_path / "data"
source_cookie = tmp_path / "relogin" / "cookies.json"
source_storage = tmp_path / "relogin" / "storage_state.json"
source_cookie.parent.mkdir(parents=True)
fresh_cookie = [
{
"name": cookie_name,
"value": "fresh-session",
"domain": cookie_domain,
}
]
fresh_storage = {"cookies": fresh_cookie, "origins": []}
source_cookie.write_text(json.dumps(fresh_cookie), encoding="utf-8")
source_storage.write_text(json.dumps(fresh_storage), encoding="utf-8")
catalog = RuntimeIntegrationCatalog.load_default(
project_root=PROJECT_ROOT,
data_root=data_root,
)
consumer_id = CONSUMER_BINDING_IDS[platform][0]
consumer = catalog.binding_for(consumer_id)
consumer.cookie_file.parent.mkdir(parents=True, exist_ok=True)
consumer.cookie_file.write_text('[{"name":"old"}]', encoding="utf-8")
consumer.storage_state_file.write_text(
'{"cookies":[{"name":"old"}]}',
encoding="utf-8",
)
destinations = sync_relogin_state_to_consumers(
platform,
cookie_file=source_cookie,
storage_state_file=source_storage,
environment={
"GYXX_PROJECT_ROOT": str(PROJECT_ROOT),
"GYXX_DATA_ROOT": str(data_root),
},
)
assert source_cookie.resolve() != consumer.cookie_file.resolve()
assert source_storage.resolve() != consumer.storage_state_file.resolve()
assert destinations == (consumer.cookie_file.resolve(),)
assert json.loads(consumer.cookie_file.read_text(encoding="utf-8")) == (
fresh_cookie
)
assert json.loads(
consumer.storage_state_file.read_text(encoding="utf-8")
) == fresh_storage
def test_consumer_state_is_rolled_back_if_atomic_publish_fails(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
data_root = tmp_path / "data"
source_cookie = tmp_path / "relogin" / "cookies.json"
source_storage = tmp_path / "relogin" / "storage_state.json"
source_cookie.parent.mkdir(parents=True)
source_cookie.write_text(
json.dumps(
[
{
"name": "SESSDATA",
"value": "fresh-session",
"domain": ".bilibili.com",
}
]
),
encoding="utf-8",
)
source_storage.write_text('{"cookies":[],"origins":[]}', encoding="utf-8")
catalog = RuntimeIntegrationCatalog.load_default(
project_root=PROJECT_ROOT,
data_root=data_root,
)
consumer = catalog.binding_for(CONSUMER_BINDING_IDS["bilibili"][0])
consumer.cookie_file.parent.mkdir(parents=True, exist_ok=True)
old_cookie = '[{"name":"SESSDATA","value":"old"}]'
old_storage = '{"cookies":[{"name":"old"}]}'
consumer.cookie_file.write_text(old_cookie, encoding="utf-8")
consumer.storage_state_file.write_text(old_storage, encoding="utf-8")
original_replace = relogin_runtime_paths.os.replace
failed = False
def fail_second_replace(source: Path | str, target: Path | str) -> None:
nonlocal failed
source_path = Path(source)
target_path = Path(target)
if (
not failed
and source_path.suffix == ".tmp"
and target_path == consumer.storage_state_file
):
failed = True
raise OSError("simulated storage publish failure")
original_replace(source, target)
monkeypatch.setattr(relogin_runtime_paths.os, "replace", fail_second_replace)
with pytest.raises(OSError, match="simulated storage publish failure"):
sync_relogin_state_to_consumers(
"bilibili",
cookie_file=source_cookie,
storage_state_file=source_storage,
environment={
"GYXX_PROJECT_ROOT": str(PROJECT_ROOT),
"GYXX_DATA_ROOT": str(data_root),
},
)
assert consumer.cookie_file.read_text(encoding="utf-8") == old_cookie
assert consumer.storage_state_file.read_text(encoding="utf-8") == old_storage
@@ -466,6 +466,83 @@ def test_collect_xiaohongshu_rows_from_artifact(tmp_path, monkeypatch):
assert row[fields.index("page_metrics")].obj["笔记阅读数"] == "1.2万"
def test_upsert_rows_isolates_single_row_failure():
"""单行写入失败只跳过该行,不拖累整批(保存点隔离)。"""
import contextlib
class _Cursor:
def fetchone(self):
return (1,)
def __getattr__(self, name):
def record(*args, **kwargs):
params = args[1] if len(args) > 1 else kwargs.get("params")
if params is None or len(params) != len(sync_tool.ROW_FIELDS):
return
# 模拟毒行: 标题以 POISON 开头的行违反约束
title = params[sync_tool.ROW_FIELDS.index("title")]
if str(title).startswith("POISON"):
raise RuntimeError("value too long")
return record
class _Conn:
def __init__(self):
self.cursor_stub = _Cursor()
self.committed = False
@contextlib.contextmanager
def cursor(self):
yield self.cursor_stub
@contextlib.contextmanager
def transaction(self):
yield
def commit(self):
self.committed = True
metric_date = dt.date(2026, 10, 1)
rows = [
sync_tool.make_row("bilibili", metric_date, "daily.json",
account_name="光影行星", video_id="BV1"),
sync_tool.make_row("douyin", metric_date, "a.xlsx",
account_name="店铺A", video_id="1",
title="POISON 这行会失败"),
sync_tool.make_row("xiaohongshu", metric_date, "daily.json",
account_name="光影行星GYXX旗舰店", video_id="66e"),
]
conn = _Conn()
written, failed = sync_tool.upsert_rows(conn, rows)
assert (written, failed) == (2, 1)
assert conn.committed
def test_drop_stale_rows_only_removes_matching_sources():
"""陈旧防护按来源文件跳过: 抖音旧导出只清掉抖音行,不拦 B站/小红书。"""
stale_excel = Path("chanmama/旧导出_20260823-20260921.xlsx")
rows = {
("douyin", STORE_FLAGSHIP, "1", dt.date(2026, 10, 1)): sync_tool.make_row(
"douyin", dt.date(2026, 10, 1), stale_excel.name, video_id="1"
),
("bilibili", "光影行星", "BV1", dt.date(2026, 10, 1)): sync_tool.make_row(
"bilibili", dt.date(2026, 10, 1), "daily.json", video_id="BV1"
),
("xiaohongshu", XHS_SHOP, "2", dt.date(2026, 10, 1)): sync_tool.make_row(
"xiaohongshu", dt.date(2026, 10, 1), "daily.json", video_id="2"
),
}
dropped = sync_tool.drop_stale_rows(rows, [stale_excel])
assert dropped == 1
assert set(rows) == {
("bilibili", "光影行星", "BV1", dt.date(2026, 10, 1)),
("xiaohongshu", XHS_SHOP, "2", dt.date(2026, 10, 1)),
}
def test_douyin_history_backfill_uses_file_mtime_and_filename_account(tmp_path):
"""历史回填: 按导出落盘日期各留快照,旧版12列文件按文件名前缀补账号。"""
# 旧版 12 列表头(无 达人昵称/抖音号/蝉妈妈链接)
@@ -4,6 +4,7 @@ from datetime import date
from pathlib import Path
import collect_douyin_qianchuan_ads as collector
import pytest
def test_qianchuan_browser_uses_scrapling_stealth_and_persistent_state(
@@ -216,3 +217,174 @@ def test_download_task_from_page_downloads_completed_matching_row() -> None:
collector._matching_task_rows = original
assert actual is expected_download
def test_wait_task_center_ready_selects_advertiser_after_redirect() -> None:
clicked: list[str] = []
state = {"on_selection": True}
class _AccountCard:
def click(self) -> None:
clicked.append("account")
state["on_selection"] = False
class _FilteredLocator:
first = _AccountCard()
def count(self) -> int:
return 1
class _AccountListLocator:
def count(self) -> int:
return 1 if state["on_selection"] else 0
def filter(self, *, has_text: str):
assert has_text == collector.ADVERTISER_ID
return _FilteredLocator()
class _Page:
def locator(self, selector: str) -> _AccountListLocator:
assert selector == collector.ACCOUNT_ITEM_SELECTOR
return _AccountListLocator()
def wait_for_timeout(self, ms: int) -> None:
pass
def wait_for_function(self, expression: str, *, arg, timeout: int) -> None:
assert not state["on_selection"]
assert arg == collector.TASK_READY_MARKER
collector._wait_task_center_ready(_Page())
assert clicked == ["account"]
def test_wait_task_center_ready_raises_when_selection_page_persists() -> None:
class _AccountCard:
def click(self) -> None:
pass
class _FilteredLocator:
first = _AccountCard()
def count(self) -> int:
return 1
class _AccountListLocator:
def count(self) -> int:
return 1
def filter(self, *, has_text: str):
assert has_text == collector.ADVERTISER_ID
return _FilteredLocator()
class _Page:
def locator(self, selector: str) -> _AccountListLocator:
assert selector == collector.ACCOUNT_ITEM_SELECTOR
return _AccountListLocator()
def wait_for_timeout(self, ms: int) -> None:
pass
def wait_for_function(self, expression: str, *, arg, timeout: int) -> None:
raise AssertionError("不应在账户选择页上等待任务中心标记")
with pytest.raises(
collector.DouyinQianchuanDownloadError, match="仍停留在账户选择页"
):
collector._wait_task_center_ready(_Page())
def test_wait_task_center_ready_raises_when_never_ready(monkeypatch) -> None:
class _AccountListLocator:
def count(self) -> int:
return 0
class _Page:
def locator(self, selector: str) -> _AccountListLocator:
assert selector == collector.ACCOUNT_ITEM_SELECTOR
return _AccountListLocator()
def wait_for_function(self, expression: str, *, arg, timeout: int) -> None:
raise RuntimeError("Page.wait_for_function: Timeout 15000ms exceeded.")
def goto(self, url: str, **options: object) -> None:
assert url == collector.TASK_CENTER_PAGE_URL
monkeypatch.setattr(collector, "is_browser_timeout_error", lambda exc: True)
with pytest.raises(
collector.DouyinQianchuanDownloadError, match="未达到可用状态"
):
collector._wait_task_center_ready(_Page(), timeout_ms=50)
def test_download_task_from_page_recovers_from_context_destroyed(
monkeypatch,
) -> None:
expected_download = object()
state = {"count_calls": 0}
class _DownloadInfo:
value = expected_download
def __enter__(self):
return self
def __exit__(self, exc_type, exc, traceback) -> None:
pass
class _DownloadButton:
def click(self) -> None:
pass
class _Row:
def inner_text(self) -> str:
return "全域推广数据-推商品\n生成完成\n下载"
def get_by_text(self, text: str, *, exact: bool) -> _DownloadButton:
assert (text, exact) == ("下载", True)
return _DownloadButton()
class _Rows:
first = _Row()
def count(self) -> int:
state["count_calls"] += 1
if state["count_calls"] == 1:
raise RuntimeError(
"Locator.count: Execution context was destroyed, "
"most likely because of a navigation."
)
return 1
class _Body:
def inner_text(self, *, timeout: int) -> str:
assert timeout == 10_000
return collector.TASK_READY_MARKER
class _Page:
url = "https://qianchuan.jinritemai.com/tools/task-center/download-task"
def expect_download(self, *, timeout: int) -> _DownloadInfo:
assert timeout == 60_000
return _DownloadInfo()
def locator(self, selector: str) -> _Body:
assert selector == "body"
return _Body()
monkeypatch.setattr(collector, "_matching_task_rows", lambda *args: _Rows())
monkeypatch.setattr(
collector, "_wait_task_center_ready", lambda page, **kwargs: None
)
actual = collector._download_task_from_page(
_Page(),
date(2026, 9, 29),
date(2026, 9, 29),
wait_timeout_ms=0,
)
assert actual is expected_download
assert state["count_calls"] == 2
@@ -125,6 +125,57 @@ def test_style_records_sum_product_rows_and_keep_zero_styles() -> None:
assert records[1]["matched_product_rows"] == 0
def test_style_extraction_maps_hidden_columns_via_visible_positions() -> None:
class Frame:
def __init__(self) -> None:
self.script = ""
self.args = {}
def evaluate(self, script, args):
self.script = script
self.args = args
return {"ok": True, "rows": []}
frame = Frame()
backfill.extract_style_rows(frame, requested_codes=("10416",))
assert "window._Cols" in frame.script
assert "visibleIdx.indexOf" in frame.script
assert "cells[stylePos]" in frame.script
assert "cells.length <= Math.max(stylePos, salesPos, returnsPos)" in frame.script
def test_ensure_custom_columns_visible_only_touches_allsku_frames() -> None:
class AllskuFrame:
url = "https://bi.erp321.com/app/report/subject/itemskuanalysis/allsku.aspx?r=1"
def __init__(self) -> None:
self.scripts: list[str] = []
def evaluate(self, script):
self.scripts.append(script)
return 2
class OtherFrame:
url = "https://bi.erp321.com/app/report/subject/itemskuanalysis/filter.aspx"
def evaluate(self, script):
raise AssertionError("must not touch non-allsku frames")
class Page:
frames = [OtherFrame(), AllskuFrame()]
page = Page()
changed = backfill.ensure_custom_columns_visible(page)
assert changed == 2
allsku = page.frames[1]
assert len(allsku.scripts) == 1
assert "'自定义销量'" in allsku.scripts[0]
assert "'自定义退货量'" in allsku.scripts[0]
assert "_RptRefresh" in allsku.scripts[0]
def test_range_collector_queries_once_per_date_not_once_per_style(monkeypatch) -> None:
filter_frame = object()
monkeypatch.setattr(
@@ -0,0 +1,77 @@
import unittest
from unittest import mock
import collect_erp_yesterday_metrics as erp_mod
class _StubFrame:
"""frame.evaluate 的最小桩: 记录入参并返回固定 payload。"""
def __init__(self, payload):
self._payload = payload
self.calls = []
def evaluate(self, script, *args):
self.calls.append((script, args))
return self._payload
class ErpDailyHeaderRoutingTest(unittest.TestCase):
"""2026-10-11 聚水潭模板重置隐藏 38 列后, 固定可见索引 45/57 错位到
「今日实退货量/今日仅退款量」, daily 销量全 0。daily 必须走表头名匹配,
且不能落入 backfill 的 _extract_by_header (自定义列 SPU 求和) 分支。"""
def test_daily_mode_uses_yesterday_headers(self):
with mock.patch.object(erp_mod, "MODE", "daily"):
self.assertEqual(
erp_mod._sales_returns_idx(),
("昨日销量", "昨日退货量"),
)
def test_weekly_mode_keeps_legacy_index(self):
with mock.patch.object(erp_mod, "MODE", "weekly"):
self.assertEqual(
erp_mod._sales_returns_idx(),
("__idx__45", "__idx__57"),
)
def test_backfill_mode_keeps_custom_headers(self):
with mock.patch.object(erp_mod, "MODE", "backfill"):
self.assertEqual(
erp_mod._sales_returns_idx(),
("自定义销量", "自定义退货量"),
)
def test_daily_header_keys_bypass_extract_by_header(self):
payload = {
"ok": True,
"yesterday_sales": 2,
"yesterday_returns": 0,
"bodyText": "",
}
frame = _StubFrame(payload)
with mock.patch.object(erp_mod, "MODE", "daily"):
metrics, _body = erp_mod._extract_with_text(frame)
self.assertEqual(metrics["yesterday_sales"], 2)
self.assertEqual(len(frame.calls), 1)
script, args = frame.calls[0]
js_args = args[0] if args else script
self.assertEqual(js_args["mode"], "header")
self.assertEqual(js_args["salesHeader"], "昨日销量")
self.assertEqual(js_args["returnsHeader"], "昨日退货量")
def test_backfill_mode_routes_to_extract_by_header(self):
frame = _StubFrame({})
with mock.patch.object(erp_mod, "MODE", "backfill"):
with mock.patch.object(
erp_mod,
"_extract_by_header",
return_value={"ok": True, "yesterday_sales": 5, "summary_text": "X"},
) as by_header:
metrics, _body = erp_mod._extract_with_text(frame)
by_header.assert_called_once()
self.assertEqual(metrics["yesterday_sales"], 5)
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,191 @@
# -*- coding: utf-8 -*-
"""聚水潭销售主题分析(按日)采集脚本的纯解析与 CLI 参数测试。"""
from __future__ import annotations
import sys
from pathlib import Path
import pytest
from gyxx_flow.modules import product_commerce
from gyxx_flow.modules.product_commerce import (
collect_erp_sales_theme_daily as collector,
)
from gyxx_flow.modules.product_commerce import db as product_db
HEADER = [
"日期", "销售单数", "销售数量", "实发数量", "实发金额", "销售金额",
"退货单数", "退货数量", "实退数量", "退货金额", "实退金额",
"净销售单数", "净销量", "净销售额", "运费收入", "运费支出", "实发单数",
]
def _write_workbook(path: Path, rows: list[list]) -> None:
import openpyxl
workbook = openpyxl.Workbook()
sheet = workbook.active
for row in rows:
sheet.append(row)
workbook.save(str(path))
workbook.close()
def test_parse_sales_theme_workbook_roundtrip(tmp_path: Path) -> None:
xlsx = tmp_path / "销售主题分析_按日_1.xlsx"
_write_workbook(
xlsx,
[
HEADER,
["2026-10-11", "649", 655, 646, "173,129.23", 175752.83,
15, 15, 0, 2677.47, "0.00", 634, 640, 173075.36, "20.00", None, 640],
["2026-10-10", 690, 691, 685, 166761.58, 168185.97,
9, 9, 0, 2656.71, 0, 681, 686, 166529.13, None, 1.6, 685],
["##", 1339, 1346, 1331, 339890.81, 343938.8,
24, 24, 0, 5334.18, 0, 1315, 1326, 339604.49, 20, 1.6, 1325],
],
)
records = collector.parse_sales_theme_workbook(xlsx)
assert [record["stat_date"] for record in records] == ["2026-10-11", "2026-10-10"]
first = records[0]
assert first["sales_order_count"] == 649
assert first["sales_qty"] == 655
assert first["shipped_amount"] == 173129.23
assert first["sales_amount"] == 175752.83
assert first["return_amount"] == 2677.47
assert first["returned_amount"] == 0.0
assert first["net_sales_order_count"] == 634
assert first["net_sales_amount"] == 173075.36
assert first["freight_income"] == 20.0
assert first["freight_expense"] is None
assert first["shipped_order_count"] == 640
# 合计行(##)必须被跳过
assert len(records) == 2
def test_parse_sales_theme_workbook_rejects_missing_date_column(tmp_path: Path) -> None:
xlsx = tmp_path / "bad.xlsx"
_write_workbook(xlsx, [["销售单数", "销售数量"], [1, 2]])
with pytest.raises(ValueError):
collector.parse_sales_theme_workbook(xlsx)
def test_parse_number_and_amount_helpers() -> None:
assert collector.parse_number("1,339") == 1339
assert collector.parse_number(None) == 0
assert collector.parse_number("") == 0
assert collector.parse_amount(None) is None
assert collector.parse_amount("") is None
assert collector.parse_amount("1.6") == 1.6
assert collector.parse_amount(2677.467) == 2677.47
class _CaptureCursor:
def __init__(self) -> None:
self.batches: list[tuple] = []
def __enter__(self):
return self
def __exit__(self, *exc_info):
return False
class _CaptureConnection:
def __init__(self) -> None:
self.cursor_obj = _CaptureCursor()
def cursor(self):
return self.cursor_obj
def test_upsert_sales_theme_dedups_and_normalizes(monkeypatch: pytest.MonkeyPatch) -> None:
captured: dict[str, object] = {}
def fake_execute_values(cur, sql, rows, template=None):
captured["sql"] = sql
captured["rows"] = list(rows)
monkeypatch.setattr(product_db, "execute_values", fake_execute_values)
records = [
{
"stat_date": "2026-10-11",
"sales_order_count": 649,
"freight_income": "20.00",
},
{
# 同日重复: 整体覆盖前一条(同 upsert_erp_all_shop 语义), 数值缺省为 0
"stat_date": "2026-10-11",
"sales_order_count": 999,
"brands": ["光影", "光影行星", "光影"],
"shops": ["天猫-光影行星旗舰店", "JD光影行星旗舰店"],
"source_file": "var/raw/erp销售主题分析日报/x.xlsx",
},
{
"stat_date": "2026-10-10",
"sales_qty": "691",
},
]
conn = _CaptureConnection()
written = product_db.upsert_erp_sales_theme_daily_metrics(conn, records)
assert written == 2
rows = captured["rows"]
assert len(rows) == 2
by_date = {row[0]: row for row in rows}
# 同日重复记录取最后一条, 数字字段缺省为 0, 金额字符串转 float
assert by_date["2026-10-11"][1] == 999
assert by_date["2026-10-10"][2] == 691
tenth = by_date["2026-10-10"]
# NOT NULL 金额列缺省(导出空单元格)落 0; 运费两列允许 NULL
assert tenth[9] == 0 # return_amount
assert tenth[15] is None # freight_income
assert tenth[16] is None # freight_expense
eleventh = by_date["2026-10-11"]
assert eleventh[-4] == "发货时间" # date_basis 默认
assert eleventh[-3] == ["光影", "光影行星"] # brands 去重排序
assert eleventh[-2] == ["JD光影行星旗舰店", "天猫-光影行星旗舰店"]
assert eleventh[-1] == "var/raw/erp销售主题分析日报/x.xlsx"
# upsert 目标表与主键在 SQL 里声明
sql = str(captured["sql"])
assert "erp_sales_theme_daily_metrics" in sql
assert "ON CONFLICT (stat_date)" in sql
def test_business_scope_constants_are_consistent() -> None:
assert len(collector.BRANDS) == 3
assert sorted(collector.BRANDS) == ["gyxx", "光影", "光影行星"]
assert len(collector.SHOP_NAMES) == len(set(collector.SHOP_NAMES)) > 0
for name in collector.SHOP_NAMES:
assert name == name.strip()
def test_resolve_window_variants() -> None:
args = collector.parse_args(["--date", "2026-10-11"])
start, end = collector.resolve_window(args)
assert (start.isoformat(), end.isoformat()) == ("2026-10-11", "2026-10-11")
args = collector.parse_args(["--date", "2026-10-11", "--days", "7"])
start, end = collector.resolve_window(args)
assert (start.isoformat(), end.isoformat()) == ("2026-10-05", "2026-10-11")
args = collector.parse_args(["--from", "2025-06-01", "--to", "2026-01-01"])
start, end = collector.resolve_window(args)
assert (start.isoformat(), end.isoformat()) == ("2025-06-01", "2026-01-01")
with pytest.raises(SystemExit):
collector.resolve_window(collector.parse_args(["--from", "2026-01-01"]))
with pytest.raises(SystemExit):
collector.resolve_window(
collector.parse_args(["--from", "2026-01-01", "--to", "2025-01-01"])
)
def test_module_uses_shared_cookie_skip_exit_code() -> None:
from gyxx_flow.core.exit_codes import COOKIE_SKIP_EXIT_CODE
assert COOKIE_SKIP_EXIT_CODE == 75
assert sys.modules[product_commerce.__name__] is not None
@@ -237,6 +237,62 @@ def test_login_if_needed_fills_the_password_form_inside_an_iframe(monkeypatch) -
assert frame.submit.click_count == 1
def test_login_if_needed_accepts_agreement_popup_and_resubmits(monkeypatch) -> None:
events: list[str] = []
class Locator:
def __init__(self, name: str) -> None:
self.name = name
def fill(self, _value: str) -> None:
events.append(f"fill:{self.name}")
def click(self, timeout: int = 0) -> None:
events.append(f"click:{self.name}")
username = Locator("username")
password = Locator("password")
submit = Locator("submit")
page = SimpleNamespace(wait_for_timeout=lambda ms: events.append(f"wait:{ms}"))
monkeypatch.setattr(baibu, "_authenticated", lambda *_args: False)
monkeypatch.setattr(baibu, "_is_login_url", lambda _url: True)
def fake_form(_page, *, use_password_form):
return (_page, username, password, submit)
monkeypatch.setattr(baibu, "_find_password_login_form", fake_form)
monkeypatch.setattr(
baibu,
"_credentials_from_environment",
lambda: (baibu._LoginCredentials(username="acct", secret_value="secret"), ()),
)
agreement = {"visible": True}
monkeypatch.setattr(
baibu, "_is_login_agreement_visible", lambda _page: agreement["visible"]
)
monkeypatch.setattr(
baibu,
"_click_login_agreement_accept",
lambda _page: agreement.update(visible=False) or True,
)
assert baibu._login_if_needed(_entry(), page) is True
# First submit is swallowed by the 服务协议 popup; after accepting it the
# credentials are re-filled and submitted a second time.
assert events == [
"fill:username",
"fill:password",
"click:submit",
"wait:2000",
"fill:username",
"fill:password",
"click:submit",
"wait:2000",
]
def test_manual_login_waits_without_filling_the_password_form(monkeypatch) -> None:
monkeypatch.setattr(baibu, "_authenticated", lambda _entry, _page: False)
frame = _LoginFrame()
@@ -301,6 +357,14 @@ def test_login_second_factor_marker_is_detected(monkeypatch) -> None:
assert baibu._login_second_factor_marker(SimpleNamespace()) == "短信验证"
def test_login_slider_captcha_marker_is_detected(monkeypatch) -> None:
monkeypatch.setattr(
baibu, "_page_text", lambda _page: "向右拖动滑块填充拼图完成验证"
)
assert baibu._login_second_factor_marker(SimpleNamespace()) == "滑块"
def test_persist_account_state_filters_cookies_to_the_account_domains(
monkeypatch,
tmp_path: Path,
@@ -351,6 +415,289 @@ def test_persist_account_state_filters_cookies_to_the_account_domains(
]
def _browser_spy(calls: list[str]):
class Page:
def goto(self, *_args, **_kwargs) -> None:
calls.append("goto")
class Context:
pages = [Page()]
def new_page(self):
return Page()
class Browser:
context = Context()
def start(self) -> None:
calls.append("start")
def persist_bound_state(self) -> None:
calls.append("persist-bound")
def close(self) -> None:
calls.append("close")
return Browser()
def test_apply_entry_continues_when_account_vault_refresh_fails(monkeypatch) -> None:
calls: list[str] = []
monkeypatch.setattr(baibu, "_start_browser", lambda: _browser_spy(calls))
monkeypatch.setattr(baibu, "_login_if_needed", lambda *_args, **_kwargs: False)
monkeypatch.setattr(baibu, "_wait_for_authenticated", lambda *_args, **_kwargs: None)
def fail_vault_refresh(_browser, _entry, **_kwargs):
calls.append("persist-fail")
raise baibu.BaibuApplyError(
"测试入口 登录后未获得可用的账号 cookie(域内 cookie:无;"
"要求含:unb、cookie2、_tb_token_),未更新账号登录态"
)
monkeypatch.setattr(baibu, "_persist_account_state", fail_vault_refresh)
def record_explicit_login(_entry, _page, *, manual_login):
calls.append("explicit-login")
monkeypatch.setattr(baibu, "_explicit_password_login", record_explicit_login)
def record_open(_page, _entry):
calls.append("open")
return None
monkeypatch.setattr(baibu, "_open_batch_import", record_open)
monkeypatch.setattr(baibu, "_upload_template", lambda *_args: calls.append("upload"))
monkeypatch.setattr(baibu, "_wait_import_result", lambda *_args, **_kwargs: calls.append("result"))
monkeypatch.setattr(baibu, "_complete_draft_products", lambda *_args: calls.append("drafts"))
monkeypatch.setattr(baibu.time, "sleep", lambda _seconds: None)
baibu._apply_entry(_entry(), Path("template.xlsx"))
assert calls == [
"start",
"goto",
"persist-bound",
"persist-fail",
"explicit-login",
"persist-fail",
"open",
"upload",
"result",
"drafts",
"close",
]
def test_apply_entry_retries_through_explicit_login_when_page_never_shows_login(
monkeypatch,
) -> None:
calls: list[str] = []
monkeypatch.setattr(baibu, "_start_browser", lambda: _browser_spy(calls))
monkeypatch.setattr(baibu, "_login_if_needed", lambda *_args, **_kwargs: False)
attempts = {"count": 0}
def fail_first_authentication(*_args, **_kwargs):
attempts["count"] += 1
if attempts["count"] == 1:
raise baibu.BaibuApplyError("测试入口 当前未登录对应店铺,已停止")
calls.append("authenticated")
monkeypatch.setattr(baibu, "_wait_for_authenticated", fail_first_authentication)
def record_explicit_login(_entry, _page, *, manual_login):
calls.append("explicit-login")
monkeypatch.setattr(baibu, "_explicit_password_login", record_explicit_login)
monkeypatch.setattr(
baibu,
"_persist_account_state",
lambda *_args, **_kwargs: calls.append("persist-ok") or True,
)
def record_open(_page, _entry):
calls.append("open")
return None
monkeypatch.setattr(baibu, "_open_batch_import", record_open)
monkeypatch.setattr(baibu, "_upload_template", lambda *_args: calls.append("upload"))
monkeypatch.setattr(baibu, "_wait_import_result", lambda *_args, **_kwargs: calls.append("result"))
monkeypatch.setattr(baibu, "_complete_draft_products", lambda *_args: calls.append("drafts"))
monkeypatch.setattr(baibu.time, "sleep", lambda _seconds: None)
baibu._apply_entry(_entry(), Path("template.xlsx"))
assert attempts["count"] == 1
assert calls == [
"start",
"goto",
"explicit-login",
"persist-bound",
"persist-ok",
"open",
"upload",
"result",
"drafts",
"close",
]
def test_explicit_password_login_drives_login_page_then_returns_to_apply(
monkeypatch,
) -> None:
calls: list[str] = []
class Page:
def goto(self, url, **_kwargs) -> None:
calls.append(f"goto:{url}")
def wait_for_timeout(self, _ms: int) -> None:
return None
monkeypatch.setattr(
baibu, "_login_if_needed", lambda *_args, **_kwargs: calls.append("fill-login") or True
)
monkeypatch.setattr(
baibu,
"_wait_for_authenticated",
lambda *_args, **_kwargs: calls.append("authenticated"),
)
entry = _entry()
baibu._explicit_password_login(entry, Page(), manual_login=False)
# The apply URL is re-entered first so its redirect chain can surface the
# seller login page; only a missing form there falls back to login.taobao.
assert calls == [
f"goto:{entry.apply_url}",
"fill-login",
f"goto:{entry.apply_url}",
"authenticated",
]
class _FakeLock:
def __init__(self, manager: "_FakeLockManager", resource: str) -> None:
self._manager = manager
self._resource = resource
def __enter__(self):
self._manager.held.add(self._resource)
self._manager.entered.append(self._resource)
return self
def __exit__(self, *_args) -> None:
self._manager.held.discard(self._resource)
class _FakeLockManager:
def __init__(self, lock_root) -> None:
self.lock_root = lock_root
self.held: set[str] = set()
self.entered: list[str] = []
self.requested: list[tuple[str, int]] = []
def acquire(self, resource: str, *, owner: str, timeout_seconds: int):
self.requested.append((resource, timeout_seconds))
if getattr(self, "busy", False):
raise baibu.ResourceBusyError(resource)
return _FakeLock(self, resource)
def test_apply_entry_holds_account_lock_for_whole_session(monkeypatch) -> None:
calls: list[str] = []
lock_manager = _FakeLockManager(lock_root="locks-dir")
class FakeSettings:
data_root = Path("data-root")
@classmethod
def from_env(cls):
return cls()
monkeypatch.setattr(baibu, "Settings", FakeSettings)
monkeypatch.setattr(baibu, "LockManager", lambda lock_root: lock_manager)
monkeypatch.setattr(baibu, "_start_browser", lambda: _browser_spy(calls))
monkeypatch.setattr(baibu, "_login_if_needed", lambda *_args, **_kwargs: False)
monkeypatch.setattr(baibu, "_wait_for_authenticated", lambda *_args, **_kwargs: None)
monkeypatch.setattr(
baibu,
"_persist_account_state",
lambda *_args, **_kwargs: True,
)
monkeypatch.setattr(baibu, "_open_batch_import", lambda _page, _entry: None)
def assert_lock_held_during_upload(_page, _template):
assert lock_manager.held == {"browser-account:tmall-shop"}
calls.append("upload")
monkeypatch.setattr(baibu, "_upload_template", assert_lock_held_during_upload)
monkeypatch.setattr(baibu, "_wait_import_result", lambda *_args, **_kwargs: None)
monkeypatch.setattr(baibu, "_complete_draft_products", lambda *_args: None)
monkeypatch.setattr(baibu.time, "sleep", lambda _seconds: None)
baibu._apply_entry(_entry(), Path("template.xlsx"))
assert lock_manager.requested == [("browser-account:tmall-shop", 120)]
assert lock_manager.entered == ["browser-account:tmall-shop"]
assert lock_manager.held == set()
def test_apply_entry_reports_busy_account_lock(monkeypatch) -> None:
lock_manager = _FakeLockManager(lock_root="locks-dir")
lock_manager.busy = True
class FakeSettings:
data_root = Path("data-root")
@classmethod
def from_env(cls):
return cls()
monkeypatch.setattr(baibu, "Settings", FakeSettings)
monkeypatch.setattr(baibu, "LockManager", lambda lock_root: lock_manager)
started: list[str] = []
monkeypatch.setattr(
baibu, "_start_browser", lambda: started.append("start") or object()
)
with pytest.raises(baibu.BaibuApplyError, match="正被保活/登录占用"):
baibu._apply_entry(_entry(), Path("template.xlsx"))
assert started == []
def test_explicit_password_login_falls_back_to_plain_login_page(
monkeypatch,
) -> None:
calls: list[str] = []
class Page:
def goto(self, url, **_kwargs) -> None:
calls.append(f"goto:{url}")
def wait_for_timeout(self, _ms: int) -> None:
return None
monkeypatch.setattr(baibu, "_login_if_needed", lambda *_args, **_kwargs: False)
monkeypatch.setattr(
baibu,
"_wait_for_authenticated",
lambda *_args, **_kwargs: calls.append("authenticated"),
)
entry = _entry()
baibu._explicit_password_login(entry, Page(), manual_login=False)
assert calls == [
f"goto:{entry.apply_url}",
f"goto:{baibu._LOGIN_PAGE_URL}",
f"goto:{entry.apply_url}",
"authenticated",
]
def test_upload_template_supports_tmall_dialog_import_button(
monkeypatch,
tmp_path: Path,
@@ -558,6 +905,32 @@ def test_wait_import_result_accepts_new_failure_summary_as_submitted(
baibu._wait_import_result(SimpleNamespace(), _entry())
def test_wait_import_result_accepts_slow_processing_after_acknowledgement(
monkeypatch,
capsys,
) -> None:
# Tmall acknowledged the batch but never returned a row-level summary
# within the acceptance window; the acknowledged submission still counts.
processing_text = (
"您于2026-10-14 14:51:00执行了商品批量导入操作,正在处理中。"
)
clock = {"now": 0.0}
def advance_monotonic() -> float:
clock["now"] += 15.0
return clock["now"]
monkeypatch.setattr(baibu, "_page_text", lambda _page: processing_text)
monkeypatch.setattr(baibu, "_refresh_batch_results", lambda _page: True)
monkeypatch.setattr(baibu.time, "sleep", lambda _seconds: None)
monkeypatch.setattr(baibu.time, "monotonic", advance_monotonic)
monkeypatch.setattr(baibu, "_RESULT_REFRESH_INTERVAL_SECONDS", 0)
baibu._wait_import_result(SimpleNamespace(), _entry())
assert "按导入提交成功" in capsys.readouterr().out
def test_wait_import_result_ignores_stale_summary_until_signature_changes(
monkeypatch,
) -> None:
@@ -663,6 +1036,47 @@ def test_complete_draft_products_accepts_empty_data_page(monkeypatch) -> None:
baibu._complete_draft_products(page, entry)
def test_complete_draft_products_retries_draft_tab_during_hydration(
monkeypatch,
) -> None:
entry = replace(_entry(), complete_drafts_after_import=True)
page = SimpleNamespace()
attempts = {"n": 0}
def flaky_tab(_page):
attempts["n"] += 1
return attempts["n"] >= 3
monkeypatch.setattr(baibu, "_click_draft_tab", flaky_tab)
monkeypatch.setattr(baibu, "_page_text", lambda _page: "加载中")
monkeypatch.setattr(baibu, "_has_visible_text", lambda _page, _labels: False)
monkeypatch.setattr(baibu, "_wait_for_draft_state", lambda _page, _entry: (0, 0))
monkeypatch.setattr(baibu.time, "sleep", lambda _seconds: None)
ticks = iter(range(10_000))
monkeypatch.setattr(baibu.time, "monotonic", lambda: next(ticks) * 0.4)
baibu._complete_draft_products(page, entry)
assert attempts["n"] == 3
def test_complete_draft_products_reports_missing_draft_entry_after_retries(
monkeypatch,
) -> None:
entry = replace(_entry(), complete_drafts_after_import=True)
page = SimpleNamespace()
monkeypatch.setattr(baibu, "_click_draft_tab", lambda _page: False)
monkeypatch.setattr(baibu, "_page_text", lambda _page: "加载中")
monkeypatch.setattr(baibu, "_has_visible_text", lambda _page, _labels: False)
monkeypatch.setattr(baibu.time, "sleep", lambda _seconds: None)
ticks = iter(range(10_000))
monkeypatch.setattr(baibu.time, "monotonic", lambda: next(ticks) * 0.4)
with pytest.raises(baibu.BaibuApplyError, match="未找到草稿页入口"):
baibu._complete_draft_products(page, entry)
def test_return_to_draft_list_falls_back_to_browser_history(monkeypatch) -> None:
events: list[str] = []
+5 -39
View File
@@ -10,9 +10,6 @@ from types import SimpleNamespace
import pytest
from gyxx_flow.adapters import COOKIE_SKIP_EXIT_CODE
from gyxx_flow.modules.content_marketing.data.tools import (
friday_relogin_parallel as weekly_relogin,
)
@pytest.fixture
@@ -75,29 +72,6 @@ def _import_supply_script(module_name: str):
sys.stderr = stderr
def test_weekly_relogin_skips_the_entire_qr_workflow(
acceptance_environment: Path,
monkeypatch: pytest.MonkeyPatch,
capsys: pytest.CaptureFixture[str],
) -> None:
monkeypatch.setattr(
weekly_relogin.subprocess,
"Popen",
_forbidden("acceptance mode must not launch relogin subprocesses"),
)
monkeypatch.setattr(
weekly_relogin,
"_send_lark",
_forbidden("acceptance mode must not send QR screenshots"),
)
assert weekly_relogin.main() == COOKIE_SKIP_EXIT_CODE
assert "SKIPPED_COOKIE" in capsys.readouterr().out
assert "content.relogin.weekly" in acceptance_environment.read_text(
encoding="utf-8"
)
@pytest.mark.parametrize(
"module_name",
[
@@ -106,25 +80,17 @@ def test_weekly_relogin_skips_the_entire_qr_workflow(
"batch_rescrape_bilibili",
],
)
def test_batch_rescrape_relogin_never_starts_a_qr_process(
def test_batch_rescrape_has_no_relogin_capability(
module_name: str,
acceptance_environment: Path,
monkeypatch: pytest.MonkeyPatch,
capsys: pytest.CaptureFixture[str],
) -> None:
"""扫码重登能力已从批量补采工具中移除,登录统一由账号 vault 管理。"""
module = importlib.import_module(
f"gyxx_flow.modules.content_marketing.data.tools.{module_name}"
)
monkeypatch.setattr(
module.subprocess,
"call",
_forbidden("acceptance mode must not invoke a relogin script"),
)
assert module.do_relogin() is False
assert "SKIPPED_COOKIE" in capsys.readouterr().out
evidence = acceptance_environment.read_text(encoding="utf-8")
assert f"content.{module_name}.relogin" in evidence
assert not hasattr(module, "do_relogin")
assert not hasattr(module, "RELOGIN_SCRIPT")
@pytest.mark.parametrize(
+1 -1
View File
@@ -56,7 +56,6 @@ def test_catalog_composition_registers_all_workflows() -> None:
"content.metrics.self_operated.daily",
"content.metrics.backfill",
"content.marketing_report.daily",
"content.relogin.weekly",
"content.creator_report.monthly",
"content.summary.monthly",
"content.notes_master.daily",
@@ -65,6 +64,7 @@ def test_catalog_composition_registers_all_workflows() -> None:
"product.persona.daily",
"product.daily",
"product.online_stock_alert.daily",
"product.hk_anpei_public_inventory.daily",
"product.alert.daily",
"product.ecommerce_costs.daily",
"product.style_analysis.interval",
+4 -6
View File
@@ -18,8 +18,8 @@ def test_catalog_maps_39_project_scheduled_tasks() -> None:
assert len(scheduled) == 39
assert len({item.source_task_name for item in scheduled}) == 39
assert Counter(item.module for item in scheduled) == {
"content_marketing": 10,
"product_commerce": 13,
"content_marketing": 9,
"product_commerce": 14,
"shop_intelligence": 12,
"supply_chain": 4,
}
@@ -80,7 +80,6 @@ def test_jd_video_upload_is_an_idempotent_scheduled_workflow() -> None:
assert step.args == (
"--anchor-record-id",
"recvrXTRWWFXeJ",
"--manual-login",
"--execute",
)
assert step.replay_policy == "idempotent"
@@ -192,7 +191,7 @@ def test_every_scheduled_workflow_has_one_valid_asia_shanghai_schedule() -> None
assert schedule.workflow_id == workflow.workflow_id
assert schedule.kind in {"daily", "weekly", "monthly", "interval_days"}
assert schedule.at.count(":") == 1
assert sum(schedule.enabled for schedule in catalog.schedules) == 27
assert sum(schedule.enabled for schedule in catalog.schedules) == 28
customer_service = catalog.schedule_for("shop.jd_customer_service.daily")
assert customer_service.at == "03:00"
assert customer_service.enabled is True
@@ -265,8 +264,7 @@ def test_scheduled_workflows_use_python_entries_and_content_tasks_are_graphs() -
"data/tools/sync_self_video_metrics.py"
)
assert catalog.schedule_for(self_operated.workflow_id).effective_times == (
"01:00",
"05:00",
"06:00",
)
backfill = next(
+1 -1
View File
@@ -54,7 +54,7 @@ def test_default_cli_registry_exposes_all_project_scheduled_workflows(
scheduled_ids = {
entry.workflow_id for entry in registry.catalog.scheduled_workflows()
}
assert len(scheduled_ids) == 35
assert len(scheduled_ids) == 39
assert scheduled_ids == {
schedule.workflow_id for schedule in registry.catalog.schedules
}
+93 -19
View File
@@ -2,6 +2,7 @@ from __future__ import annotations
import http.client
import json
import os
import re
import shutil
import subprocess
@@ -311,7 +312,7 @@ def test_notification_config_persists_app_scoped_multi_recipient_routes(
)
initial = service.notification_config()
assert len(initial["people"]) == 16
assert len(initial["people"]) == 18
assert initial["available_app_profiles"] == ["hermes-analyzer"]
assert [app["profile"] for app in initial["apps"]] == ["hermes-analyzer"]
wang = next(person for person in initial["people"] if person["id"] == "wang_yunlong")
@@ -321,7 +322,12 @@ def test_notification_config_persists_app_scoped_multi_recipient_routes(
assert wang["bindings"]["hermes-analyzer"]["open_id"] == (
"ou_7ad5fc8012e2f741afc5346e05ffd447"
)
assert all(route["configured"] is False for route in initial["routes"])
stock_alert_route = next(
route
for route in initial["routes"]
if route["workflow_id"] == "product.online_stock_alert.daily"
)
assert stock_alert_route["configured"] is True
routes = [dict(route) for route in initial["routes"]]
daily = next(
@@ -1566,7 +1572,7 @@ def test_overview_exposes_only_verified_scheduled_runs_as_cancellable(
"schema_version": 1,
"updated_at": "2026-08-14T02:00:00+00:00",
"jobs": {
"content.relogin.weekly": {
"content.notes_master.daily": {
"status": "running",
"pid": 4310,
"business_date": "2026-08-14",
@@ -1589,7 +1595,7 @@ def test_overview_exposes_only_verified_scheduled_runs_as_cancellable(
workflow_id: str,
_job: dict[str, Any],
) -> tuple[bool, Any, float | None]:
if workflow_id == "content.relogin.weekly":
if workflow_id == "content.notes_master.daily":
return True, VerifiedProcess(), 1_800_000_000.0
return False, None, None
@@ -1600,7 +1606,7 @@ def test_overview_exposes_only_verified_scheduled_runs_as_cancellable(
).overview()
workflows = {item["id"]: item for item in overview["workflows"]}
verified = workflows["content.relogin.weekly"]["scheduler_job"]
verified = workflows["content.notes_master.daily"]["scheduler_job"]
assert verified["status"] == "running"
assert verified["cancellable"] is True
assert "pid" not in verified
@@ -1614,7 +1620,7 @@ def test_service_cancels_verified_scheduled_run_and_finalizes_journal(
monkeypatch: pytest.MonkeyPatch,
console_settings: Settings,
) -> None:
workflow_id = "content.relogin.weekly"
workflow_id = "content.notes_master.daily"
_write_scheduler_state(
console_settings,
{
@@ -1906,24 +1912,13 @@ def test_content_production_preflight_rejects_non_production_modes(
relogin = next(
item
for item in catalog.workflows
if item.workflow_id == "content.relogin.weekly"
if item.workflow_id == "content.summary.weekly"
)
with pytest.raises(ConsoleRequestError, match=message):
console_module._validate_content_production_runtime(relogin, {flag: "1"})
def test_content_relogin_preflight_does_not_require_postgres() -> None:
catalog = WorkflowCatalog.load(PROJECT_ROOT / "config")
relogin = next(
item
for item in catalog.workflows
if item.workflow_id == "content.relogin.weekly"
)
console_module._validate_content_production_runtime(relogin, {})
@pytest.mark.parametrize(
"workflow_id",
["content.notes_master.daily", "content.metrics.daily"],
@@ -3297,7 +3292,7 @@ def test_http_notification_config_requires_revision_and_supports_multi_recipient
assert status == HTTPStatus.OK
_assert_security_headers(response_headers)
config = json.loads(body)
assert len(config["people"]) == 16
assert len(config["people"]) == 18
assert {
route["workflow_id"] for route in config["routes"]
} == set(console_module.NOTIFICATION_CAPABILITIES)
@@ -3616,3 +3611,82 @@ def test_douyin_negative_reviews_range_uses_single_run_with_env_dates(
{"business_date": "2026-09-19"},
)
assert "business_date_end" not in single
def test_api_health_reports_process_and_scheduler_state(
console_settings: Settings,
) -> None:
with _running_server(console_settings, FakeConsoleLauncher()) as server:
status, _headers, content = _http_request(server, "GET", "/api/health")
assert status == HTTPStatus.OK
payload = json.loads(content)
assert payload["ok"] is True
assert payload["pid"] == os.getpid()
assert payload["uptime_seconds"] >= 0
# 测试环境没有调度器实例锁,health 只报状态不做任何拉起。
assert payload["scheduler_running"] is False
assert payload["scheduler_pid"] is None
def test_api_health_requires_no_token(console_settings: Settings) -> None:
# 自检探针端点供进程内看门狗与保活脚本使用,必须绕过访问令牌。
with _running_server(
console_settings, FakeConsoleLauncher(), token="x" * 24
) as server:
status, _headers, _content = _http_request(server, "GET", "/api/health")
assert status == HTTPStatus.OK
def test_probe_console_health_against_real_server(
console_settings: Settings,
) -> None:
with _running_server(console_settings, FakeConsoleLauncher()) as server:
assert console_module._probe_console_health(server.server_port, 2.0) is True
assert console_module._probe_console_health(1, 0.2) is False
def test_console_watchdog_exits_after_consecutive_probe_failures() -> None:
stopped = threading.Event()
exits: list[int] = []
probes: list[int] = []
def always_failing(port: int, _timeout: float) -> bool:
probes.append(port)
return False
console_module._console_watchdog(
8765,
interval_seconds=0.01,
failure_limit=3,
probe_timeout_seconds=0.01,
stopped=stopped,
probe=always_failing,
exit_process=exits.append,
sleep=lambda _seconds: None,
)
assert exits == [console_module.CONSOLE_WATCHDOG_EXIT_CODE]
assert probes == [8765, 8765, 8765]
def test_console_watchdog_resets_failure_count_after_successful_probe() -> None:
stopped = threading.Event()
exits: list[int] = []
results = iter([False, False, True, False, False, False])
def flaky_probe(_port: int, _timeout: float) -> bool:
return next(results)
console_module._console_watchdog(
8765,
interval_seconds=0.01,
failure_limit=3,
probe_timeout_seconds=0.01,
stopped=stopped,
probe=flaky_probe,
exit_process=exits.append,
sleep=lambda _seconds: None,
)
# 成功一次清零计数:2 次失败 + 1 次成功 + 3 次失败后才退出。
assert exits == [console_module.CONSOLE_WATCHDOG_EXIT_CODE]
+1 -1
View File
@@ -23,7 +23,7 @@ def test_content_module_registers_only_scheduled_entries() -> None:
definitions = module.workflow_definitions()
assert tuple(item.workflow_id for item in definitions) == CONTENT_WORKFLOW_IDS
assert len(CONTENT_SCHEDULED_WORKFLOW_IDS) == 10
assert len(CONTENT_SCHEDULED_WORKFLOW_IDS) == 9
def test_content_workflows_use_project_owned_module_commands() -> None:
+1 -2
View File
@@ -44,7 +44,6 @@ REQUIRED_SOURCES = {
"chanmama_scraper.py",
"daily_creator_exposure_scope.py",
"daily_marketing_report.py",
"data/tools/friday_relogin_parallel.py",
"data/tools/schema_gyxx_super_data.sql",
"data/tools/migrations/005_style_product_profile.sql",
"data/config/db.env.example",
@@ -71,7 +70,7 @@ def test_content_source_snapshot_manifest_is_complete_and_well_formed() -> None:
assert manifest["module"] == "content_marketing"
files = manifest["files"]
assert isinstance(files, list)
assert len(files) == 66
assert len(files) == 59
sources = [entry["source_relative_path"] for entry in files]
targets = [entry["target_relative_path"] for entry in files]
+771
View File
@@ -0,0 +1,771 @@
"""Cookie gate: per-account login-state machine injected before workflows."""
from __future__ import annotations
import json
from datetime import datetime, timedelta, timezone
from pathlib import Path
from types import SimpleNamespace
from gyxx_flow.adapters.browser import BrowserCookieStore
from gyxx_flow.adapters.integration import AccountBinding
from gyxx_flow.catalog import WorkflowCatalog, WorkflowEntry
from gyxx_flow.cookie_gate import (
GATE_MODE_BLOCKING,
GATE_MODE_DEFER,
GATE_MODE_RECOVER,
CookieGate,
CookieGateRecoveryManager,
apply_cookie_gates,
gate_enabled,
)
from gyxx_flow.core.config import Settings
from gyxx_flow.core.context import RunContext
from gyxx_flow.core.layout import DataLayout
from gyxx_flow.core.locks import LockManager
from gyxx_flow.core.records import RunJournal
from gyxx_flow.workflow.engine import WorkflowEngine
from gyxx_flow.workflow.model import StepDefinition, WorkflowDefinition
from gyxx_flow.workflow.steps import StepExecution
_NOW = datetime.now(timezone.utc).replace(microsecond=0)
def _settings(tmp_path: Path) -> Settings:
return Settings(project_root=tmp_path, data_root=tmp_path / "var")
def _account(tmp_path: Path, account_id: str, **overrides: object) -> AccountBinding:
vault = tmp_path / "state" / "accounts" / account_id
values: dict[str, object] = {
"account_id": account_id,
"cdp_port": 22000,
"cdp_url": "http://127.0.0.1:22000",
"profile_dir": vault / "profile",
"cookie_file": vault / "cookies.json",
"storage_state_file": vault / "storage_state.json",
"required_cookie_domains": ("example.com",),
"required_cookie_names": ("sessionid",),
"login_identity_markers": ("示例店铺",),
"keepalive_enabled": True,
"keepalive_url": "https://example.com/home",
"keepalive_interval_seconds": 900,
}
values.update(overrides)
return AccountBinding(**values) # type: ignore[arg-type]
def _fake_catalog(*accounts: AccountBinding, bindings: dict[str, str] | None = None):
by_id = {account.account_id: account for account in accounts}
binding_accounts = bindings or {}
account_tuple = tuple(accounts)
class _Catalog:
accounts = account_tuple
def account_for(self, account_id: str) -> AccountBinding:
return by_id[account_id]
def binding_for(self, binding_id: str):
return SimpleNamespace(account=binding_accounts.get(binding_id, ""))
return _Catalog()
def _write_valid_vault(account: AccountBinding) -> None:
store = BrowserCookieStore(account.cookie_file, account.storage_state_file)
store.save_cookies(
[
{
"name": "sessionid",
"value": "live-session",
"domain": ".example.com",
"path": "/",
"expires": _NOW.timestamp() + 86_400,
}
]
)
def _write_keepalive_status(
account: AccountBinding,
payload: dict[str, object] | None,
) -> None:
path = account.cookie_file.with_name("keepalive_status.json")
if payload is None:
path.unlink(missing_ok=True)
return
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(payload), encoding="utf-8")
def _gate(
tmp_path: Path,
*accounts: AccountBinding,
probe: dict[str, object] | None = None,
environment: dict[str, str] | None = None,
sent: list[dict[str, object]] | None = None,
bindings: dict[str, str] | None = None,
) -> CookieGate:
probes = probe or {"status": "expired"}
keepalive = SimpleNamespace(
refresh_account=lambda account_id: dict(probes, account_id=account_id)
)
sender_calls = sent if sent is not None else []
def _sender(**kwargs: object) -> dict[str, object]:
sender_calls.append(dict(kwargs))
return {"message_id": "om_fake"}
catalog = _fake_catalog(*accounts, bindings=bindings)
gate = CookieGate(
_settings(tmp_path),
catalog=catalog,
keepalive=keepalive, # type: ignore[arg-type]
environment=environment or {},
now=lambda: _NOW,
sender=_sender,
)
gate._workflow_catalog = WorkflowCatalog(timezone="UTC", workflows=(), schedules=())
return gate
def _context(**overrides: object) -> RunContext:
values: dict[str, object] = {
"workflow_id": "shop.demo.daily",
"business_date": "2026-10-08",
"shadow": False,
"now": _NOW,
"random_suffix": "aaaa",
}
values.update(overrides)
return RunContext.create(**values) # type: ignore[arg-type]
def _noop_outcome(*, context, timeout_seconds, dry_run) -> StepExecution: # noqa: ANN001
return StepExecution(exit_code=0)
def _deny_outcome(*, context, timeout_seconds, dry_run) -> StepExecution: # noqa: ANN001
raise AssertionError("resolution must not run actions")
class _PassAction:
execute = staticmethod(_noop_outcome)
class _BoundAction:
"""Duck-typed DeferredModuleCommandStep for account resolution tests."""
def __init__(self, command_entry: str = "collect.py") -> None:
self.entry = SimpleNamespace(module="shop_intelligence", entry="fallback.py")
self.command_entry = command_entry
self.execute = _deny_outcome
class _RecordingAction:
def __init__(self, name: str, sink: list[str], command_entry: str | None = None) -> None:
def run(*, context, timeout_seconds, dry_run) -> StepExecution: # noqa: ANN001
sink.append(name)
return StepExecution(exit_code=0)
self.execute = run
if command_entry is not None:
self.entry = SimpleNamespace(
module="shop_intelligence", entry="fallback.py"
)
self.command_entry = command_entry
# ----------------------------------------------------------------------
# Mode classification
# ----------------------------------------------------------------------
def test_gate_mode_classifies_accounts(tmp_path: Path) -> None:
gate = _gate(
tmp_path,
_account(tmp_path, "qr-only"),
_account(tmp_path, "cred-scripts", credential_env_names=("A_USER", "A_PASS")),
_account(
tmp_path,
"cred-vault",
login_mode="C",
login_command_id="product_commerce:erp_login.py",
),
_account(tmp_path, "cred-mode-b", login_mode="B"),
)
assert gate.gate_mode("qr-only") == GATE_MODE_BLOCKING
assert gate.gate_mode("cred-scripts") == GATE_MODE_DEFER
assert gate.gate_mode("cred-vault") == GATE_MODE_RECOVER
assert gate.gate_mode("cred-mode-b") == GATE_MODE_DEFER
# ----------------------------------------------------------------------
# Workflow account resolution
# ----------------------------------------------------------------------
def _bound_workflow(workflow_id: str = "shop.demo.daily") -> WorkflowDefinition:
return WorkflowDefinition(
workflow_id,
(
StepDefinition("collect", _BoundAction("collect.py")),
StepDefinition("sync", _BoundAction("sync.py"), depends_on=("collect",)),
),
)
def test_collect_accounts_maps_bindings_and_skips_defer(tmp_path: Path) -> None:
gate = _gate(
tmp_path,
_account(tmp_path, "douyin-shop"),
_account(tmp_path, "jd-shop", credential_env_names=("JD_SHOP", "JD_PASSWORD")),
bindings={
"shop_intelligence:collect.py": "douyin-shop",
"shop_intelligence:other.py": "jd-shop",
},
)
mapping = gate.collect_workflow_accounts(_bound_workflow())
# sync.py 无账号绑定,jd-shop 属 defer 模式:两者都不进映射
assert mapping == {"douyin-shop": {"collect"}}
def test_collect_accounts_includes_declared_required_accounts(tmp_path: Path) -> None:
gate = _gate(tmp_path, _account(tmp_path, "douyin-shop"))
gate._workflow_catalog = WorkflowCatalog(
timezone="UTC",
workflows=(
WorkflowEntry(
workflow_id="shop.demo.daily",
module="shop_intelligence",
trigger="manual",
entry="collect.py",
required_accounts=("douyin-shop",),
),
),
schedules=(),
)
mapping = gate.collect_workflow_accounts(_bound_workflow())
assert mapping == {"douyin-shop": {"collect", "sync"}}
def test_apply_cookie_gates_rewrites_dependencies(tmp_path: Path) -> None:
account = _account(tmp_path, "douyin-shop")
gate = _gate(
tmp_path,
account,
bindings={"shop_intelligence:collect.py": "douyin-shop"},
)
gated = apply_cookie_gates(
_bound_workflow(), settings=_settings(tmp_path), gate=gate
)
ids = [step.step_id for step in gated.steps]
assert "cookie-gate.douyin-shop" in ids
collect = next(step for step in gated.steps if step.step_id == "collect")
sync = next(step for step in gated.steps if step.step_id == "sync")
assert collect.depends_on == ("cookie-gate.douyin-shop",)
assert sync.depends_on == ("collect",)
assert gated.ordered_steps() # patched DAG stays valid
def test_apply_cookie_gates_returns_workflow_without_accounts(tmp_path: Path) -> None:
gate = _gate(tmp_path, _account(tmp_path, "douyin-shop"))
workflow = _bound_workflow()
assert (
apply_cookie_gates(workflow, settings=_settings(tmp_path), gate=gate) is workflow
)
def test_gate_enabled_env_kill_switch() -> None:
assert gate_enabled({})
assert not gate_enabled({"GYXX_COOKIE_GATE": "0"})
assert not gate_enabled({"GYXX_COOKIE_GATE": "off"})
assert gate_enabled({"GYXX_COOKIE_GATE": "1"})
# ----------------------------------------------------------------------
# State machine: check()
# ----------------------------------------------------------------------
def test_check_fast_path_accepts_fresh_keepalive_without_probe(tmp_path: Path) -> None:
account = _account(tmp_path, "douyin-shop")
gate = _gate(tmp_path, account, probe={"status": "expired"})
_write_valid_vault(account)
_write_keepalive_status(
account,
{
"status": "success",
"last_success_at": (_NOW - timedelta(minutes=10)).isoformat(),
},
)
verdict = gate.check("douyin-shop", context=_context())
assert not verdict.blocked
assert "keepalive" in verdict.message
state = json.loads(
(account.cookie_file.parent / "cookie_state.json").read_text(encoding="utf-8")
)
assert state["state"] == "healthy"
def test_check_stale_status_triggers_probe(tmp_path: Path) -> None:
account = _account(tmp_path, "douyin-shop")
gate = _gate(tmp_path, account, probe={"status": "success"})
_write_valid_vault(account)
_write_keepalive_status(
account,
{
"status": "success",
"last_success_at": (_NOW - timedelta(hours=6)).isoformat(),
},
)
verdict = gate.check("douyin-shop", context=_context())
assert not verdict.blocked
assert verdict.message.endswith("探活通过)")
def test_check_expired_blocks_and_sends_feishu_alert(tmp_path: Path) -> None:
account = _account(tmp_path, "douyin-shop")
sent: list[dict[str, object]] = []
gate = _gate(
tmp_path,
account,
probe={"status": "expired"},
environment={"GYXX_LOGIN_ALERT_OPEN_IDS": "ou_alert_target"},
sent=sent,
)
_write_keepalive_status(account, None)
verdict = gate.check("douyin-shop", context=_context())
assert verdict.blocked
assert verdict.notified
assert len(sent) == 1
text = str(sent[0]["text"])
assert "uv run gyxx accounts login douyin-shop" in text
assert "shop.demo.daily" in text
assert sent[0]["profile"] == "hermes-analyzer"
state = json.loads(
(account.cookie_file.parent / "cookie_state.json").read_text(encoding="utf-8")
)
assert state["state"] == "awaiting_login"
assert state["last_notified_at"] == _NOW.isoformat(timespec="seconds")
def test_check_notification_is_throttled_per_interval(tmp_path: Path) -> None:
account = _account(tmp_path, "douyin-shop")
sent: list[dict[str, object]] = []
gate = _gate(
tmp_path,
account,
probe={"status": "expired"},
environment={"GYXX_LOGIN_ALERT_OPEN_IDS": "ou_alert_target"},
sent=sent,
)
_write_keepalive_status(account, None)
first = gate.check("douyin-shop", context=_context())
second = gate.check("douyin-shop", context=_context())
assert first.notified and second.blocked
assert not second.notified
assert len(sent) == 1
def test_check_shadow_run_never_notifies(tmp_path: Path) -> None:
account = _account(tmp_path, "douyin-shop")
sent: list[dict[str, object]] = []
gate = _gate(
tmp_path,
account,
probe={"status": "expired"},
environment={"GYXX_LOGIN_ALERT_OPEN_IDS": "ou_alert_target"},
sent=sent,
)
verdict = gate.check("douyin-shop", context=_context(shadow=True))
assert verdict.blocked
assert not verdict.notified
assert sent == []
def test_check_mode_c_recovery_success_and_failure_paths(tmp_path: Path) -> None:
account = _account(
tmp_path,
"erp-jushuitan",
login_mode="C",
login_command_id="product_commerce:erp_login.py",
)
recovered = _gate(
tmp_path, account, probe={"status": "success", "login_recovered": True}
)
_write_keepalive_status(account, None)
verdict = recovered.check("erp-jushuitan", context=_context())
assert not verdict.blocked
assert verdict.recovered
assert "自动账密重登成功" in verdict.message
sent: list[dict[str, object]] = []
failed = _gate(
tmp_path,
account,
probe={"status": "not_logged_in"},
environment={"GYXX_LOGIN_ALERT_OPEN_IDS": "ou_alert_target"},
sent=sent,
)
_write_keepalive_status(
failed.catalog.account_for("erp-jushuitan"),
{
"status": "not_logged_in",
"login_attempted": True,
"login_error_type": "TimeoutExpired",
},
)
verdict = failed.check("erp-jushuitan", context=_context())
assert verdict.blocked
assert verdict.notified
assert "自动账密重登已尝试但未成功" in str(sent[0]["text"])
def test_check_busy_falls_back_to_static_evidence(tmp_path: Path) -> None:
account = _account(tmp_path, "douyin-shop")
gate = _gate(tmp_path, account, probe={"status": "busy"})
_write_valid_vault(account)
_write_keepalive_status(account, None)
verdict = gate.check("douyin-shop", context=_context())
assert not verdict.blocked
assert "探活锁忙" in verdict.message
def test_check_probe_error_with_static_valid_vault_passes(tmp_path: Path) -> None:
account = _account(tmp_path, "douyin-shop")
gate = _gate(tmp_path, account, probe={"status": "timeout"})
_write_valid_vault(account)
_write_keepalive_status(account, None)
verdict = gate.check("douyin-shop", context=_context())
assert not verdict.blocked
assert "探活异常" in verdict.message
def test_acceptance_mode_is_static_only_and_never_alerts(tmp_path: Path) -> None:
account = _account(tmp_path, "douyin-shop")
sent: list[dict[str, object]] = []
gate = _gate(
tmp_path,
account,
probe={"status": "expired"},
environment={
"GYXX_WORKFLOW_ACCEPTANCE": "1",
"GYXX_LOGIN_ALERT_OPEN_IDS": "ou_alert_target",
},
sent=sent,
)
_write_keepalive_status(account, None)
verdict = gate.check("douyin-shop", context=_context())
assert verdict.blocked
assert sent == []
assert "验收模式" in verdict.message
def test_alert_recipient_falls_back_to_constant(tmp_path: Path) -> None:
account = _account(tmp_path, "douyin-shop")
gate = _gate(tmp_path, account)
recipients = gate._alert_recipients()
assert recipients == [
("ou_7ad5fc8012e2f741afc5346e05ffd447", "hermes-analyzer")
]
# ----------------------------------------------------------------------
# Engine integration
# ----------------------------------------------------------------------
def _engine(tmp_path: Path) -> WorkflowEngine:
return WorkflowEngine(LockManager(tmp_path / "state" / "locks"))
def test_engine_blocks_dependents_when_gate_fails(tmp_path: Path) -> None:
account = _account(tmp_path, "douyin-shop")
sent: list[dict[str, object]] = []
gate = _gate(
tmp_path,
account,
probe={"status": "expired"},
environment={"GYXX_LOGIN_ALERT_OPEN_IDS": "ou_alert_target"},
sent=sent,
bindings={"shop_intelligence:collect.py": "douyin-shop"},
)
executed: list[str] = []
workflow = WorkflowDefinition(
"shop.demo.daily",
(
StepDefinition(
"collect", _RecordingAction("collect", executed, "collect.py")
),
StepDefinition(
"sync", _RecordingAction("sync", executed), depends_on=("collect",)
),
),
)
gated = apply_cookie_gates(workflow, settings=_settings(tmp_path), gate=gate)
context = RunContext.create(
"shop.demo.daily", "2026-10-08", now=_NOW, random_suffix="dddd"
)
journal = RunJournal.create(DataLayout(_settings(tmp_path).data_root), context)
engine = _engine(tmp_path)
run_workflow = engine.execute
result = run_workflow(gated, context=context, journal=journal)
assert result.status == "failed"
assert result.exit_code == 1
assert result.steps["cookie-gate.douyin-shop"].status == "failed"
assert result.steps["collect"].reason == "dependency-failed"
assert result.steps["sync"].reason == "dependency-failed"
assert executed == []
assert len(sent) == 1
def test_engine_runs_collect_steps_when_gate_passes(tmp_path: Path) -> None:
account = _account(tmp_path, "douyin-shop")
gate = _gate(
tmp_path,
account,
probe={"status": "success"},
bindings={"shop_intelligence:collect.py": "douyin-shop"},
)
_write_valid_vault(account)
_write_keepalive_status(account, None)
executed: list[str] = []
workflow = WorkflowDefinition(
"shop.demo.daily",
(
StepDefinition(
"collect", _RecordingAction("collect", executed, "collect.py")
),
),
)
gated = apply_cookie_gates(workflow, settings=_settings(tmp_path), gate=gate)
context = RunContext.create(
"shop.demo.daily", "2026-10-08", now=_NOW, random_suffix="eeee"
)
journal = RunJournal.create(DataLayout(_settings(tmp_path).data_root), context)
engine = _engine(tmp_path)
run_workflow = engine.execute
result = run_workflow(gated, context=context, journal=journal)
assert result.status == "success"
assert result.steps["cookie-gate.douyin-shop"].status == "success"
assert result.steps["collect"].status == "success"
# ----------------------------------------------------------------------
# Recovery: automatic rerun after the login recovers
# ----------------------------------------------------------------------
class _FakeLauncher:
def __init__(self, fail: bool = False) -> None:
self.launched: list[tuple[str, str]] = []
self.closed: list[int] = []
self.fail = fail
def launch(self, workflow_id, slot, business_date): # noqa: ANN001
if self.fail:
raise RuntimeError("spawn denied")
process = SimpleNamespace(pid=9000 + len(self.launched), poll=lambda: None)
self.launched.append((workflow_id, business_date.isoformat()))
return process
def close(self, process) -> None: # noqa: ANN001
self.closed.append(process.pid)
def _state_file(account: AccountBinding) -> Path:
return account.cookie_file.parent / "cookie_state.json"
def _read_state(account: AccountBinding) -> dict:
return json.loads(_state_file(account).read_text(encoding="utf-8"))
def _recovery_workflow_catalog() -> WorkflowCatalog:
return WorkflowCatalog(
timezone="UTC",
workflows=(
WorkflowEntry(
workflow_id="shop.demo.daily",
module="shop_intelligence",
trigger="scheduled",
entry="collect.py",
),
),
schedules=(),
)
def _blocking_gate(
tmp_path: Path,
account: AccountBinding,
sent: list[dict[str, object]],
) -> CookieGate:
gate = _gate(
tmp_path,
account,
probe={"status": "expired"},
environment={"GYXX_LOGIN_ALERT_OPEN_IDS": "ou_alert_target"},
sent=sent,
)
return gate
def _healthy_gate(
tmp_path: Path,
account: AccountBinding,
sent: list[dict[str, object]],
) -> CookieGate:
gate = _gate(
tmp_path,
account,
probe={"status": "success"},
environment={"GYXX_LOGIN_ALERT_OPEN_IDS": "ou_alert_target"},
sent=sent,
)
gate._workflow_catalog = _recovery_workflow_catalog()
return gate
def _simulate_manual_login(account: AccountBinding) -> None:
_write_valid_vault(account)
_write_keepalive_status(
account,
{
"status": "success",
"last_success_at": (_NOW - timedelta(seconds=60)).isoformat(),
},
)
def test_expire_records_pending_reruns(tmp_path: Path) -> None:
sent: list[dict[str, object]] = []
account = _account(tmp_path, "douyin-shop")
gate = _blocking_gate(tmp_path, account, sent)
gate.check("douyin-shop", context=_context())
gate.check("douyin-shop", context=_context()) # 同工作流同日期去重
gate.check("douyin-shop", context=_context(business_date="2026-10-09"))
state = _read_state(account)
assert [
(item["workflow_id"], item["business_date"])
for item in state["pending_reruns"]
] == [
("shop.demo.daily", "2026-10-08"),
("shop.demo.daily", "2026-10-09"),
]
# shadow 运行的阻断不进入待重跑清单
gate.check(
"douyin-shop", context=_context(business_date="2026-10-10", shadow=True)
)
state = _read_state(account)
assert all(
item["business_date"] != "2026-10-10" for item in state["pending_reruns"]
)
def test_recovery_reruns_blocked_workflows_after_login(tmp_path: Path) -> None:
sent: list[dict[str, object]] = []
account = _account(tmp_path, "douyin-shop")
blocked = _blocking_gate(tmp_path, account, sent)
blocked.check("douyin-shop", context=_context())
_simulate_manual_login(account)
healthy = _healthy_gate(tmp_path, account, sent)
launcher = _FakeLauncher()
outcomes = CookieGateRecoveryManager(healthy, launcher=launcher).tick()
assert launcher.launched == [("shop.demo.daily", "2026-10-08")]
assert launcher.closed # 父进程日志句柄立即关闭
assert outcomes and outcomes[0]["launched"]
state = _read_state(account)
assert state["pending_reruns"] == []
assert state["state"] == "healthy"
assert state["last_rerun_launch_at"]
assert state["rerun_history"][0]["workflow_id"] == "shop.demo.daily"
assert any("登录态已恢复" in str(item["text"]) for item in sent)
def test_recovery_skips_while_login_still_expired(tmp_path: Path) -> None:
sent: list[dict[str, object]] = []
account = _account(tmp_path, "douyin-shop")
blocked = _blocking_gate(tmp_path, account, sent)
blocked.check("douyin-shop", context=_context())
_write_keepalive_status(account, None)
still_expired = _gate(
tmp_path,
account,
probe={"status": "expired"},
environment={"GYXX_LOGIN_ALERT_OPEN_IDS": "ou_alert_target"},
sent=sent,
)
still_expired._workflow_catalog = _recovery_workflow_catalog()
launcher = _FakeLauncher()
outcomes = CookieGateRecoveryManager(still_expired, launcher=launcher).tick()
assert launcher.launched == []
assert outcomes == []
state = _read_state(account)
assert len(state["pending_reruns"]) == 1
assert state["last_recovery_probe_at"]
def test_recovery_drops_adhoc_script_ids(tmp_path: Path) -> None:
sent: list[dict[str, object]] = []
account = _account(tmp_path, "douyin-shop")
blocked = _blocking_gate(tmp_path, account, sent)
blocked.check(
"douyin-shop", context=_context(workflow_id="script.shop.abcdef123456")
)
assert (
_read_state(account)["pending_reruns"][0]["workflow_id"]
== "script.shop.abcdef123456"
)
_simulate_manual_login(account)
healthy = _healthy_gate(tmp_path, account, sent)
launcher = _FakeLauncher()
CookieGateRecoveryManager(healthy, launcher=launcher).tick()
assert launcher.launched == []
assert _read_state(account)["pending_reruns"] == []
def test_recovery_launch_failure_keeps_pending_for_retry(tmp_path: Path) -> None:
sent: list[dict[str, object]] = []
account = _account(tmp_path, "douyin-shop")
blocked = _blocking_gate(tmp_path, account, sent)
blocked.check("douyin-shop", context=_context())
_simulate_manual_login(account)
healthy = _healthy_gate(tmp_path, account, sent)
launcher = _FakeLauncher(fail=True)
CookieGateRecoveryManager(healthy, launcher=launcher).tick()
state = _read_state(account)
assert len(state["pending_reruns"]) == 1
assert "error" in state["pending_reruns"][0]
assert state["state"] == "awaiting_login"
assert not any("已恢复" in str(item["text"]) for item in sent)
def test_recovery_respects_launch_cooldown(tmp_path: Path) -> None:
sent: list[dict[str, object]] = []
account = _account(tmp_path, "douyin-shop")
blocked = _blocking_gate(tmp_path, account, sent)
blocked.check("douyin-shop", context=_context())
_simulate_manual_login(account)
state = _read_state(account)
state["last_rerun_launch_at"] = _NOW.isoformat(timespec="seconds")
_state_file(account).write_text(json.dumps(state), encoding="utf-8")
healthy = _healthy_gate(tmp_path, account, sent)
launcher = _FakeLauncher()
CookieGateRecoveryManager(healthy, launcher=launcher).tick()
assert launcher.launched == []
-1
View File
@@ -36,7 +36,6 @@ def test_repository_keeps_the_simplified_public_structure() -> None:
"note-inventory.md",
"plan.md",
"runbook.md",
"workbench.md",
"workflow-acceptance-test-report.md",
"workflow-acceptance-test-requirements.md",
"workflow-console.md",
+16 -5
View File
@@ -67,15 +67,21 @@ def test_baseline_contains_all_provided_people_and_routable_capabilities(
) -> None:
payload, revision = NotificationRoutingStore(routing_settings).snapshot()
assert len(payload["people"]) == 16
assert len(payload["people"]) == 18
assert payload["app_profile"] == "hermes-analyzer"
assert (
payload["app_profiles"]["hermes-analyzer"]["app_id"]
== ANALYZER_NOTIFICATION_APP_ID
)
assert all(person["mobile"] is None for person in payload["people"].values())
mobiles = {
person_id: person["mobile"]
for person_id, person in payload["people"].items()
}
assert mobiles["wang_yunlong"] == "+8613663785145"
assert mobiles["stock_alert_user_19832771651"] == "+8619832771651"
assert set(payload["capabilities"]) == set(NOTIFICATION_CAPABILITIES)
assert payload["routes"] == {}
assert set(payload["routes"]) == {"product.online_stock_alert.daily"}
assert payload["routes"]["product.online_stock_alert.daily"]["enabled"] is True
assert len(revision) == 64
wang = payload["people"]["wang_yunlong"]
@@ -85,10 +91,15 @@ def test_baseline_contains_all_provided_people_and_routable_capabilities(
assert wang["bindings"]["hermes-analyzer"]["open_id"] == (
"ou_7ad5fc8012e2f741afc5346e05ffd447"
)
_mobile_lookup_people = {
"wang_yunlong",
"online_stock_alert_recipient",
"stock_alert_user_19832771651",
}
provided = {
person["name"]: binding["open_id"]
for person_id, person in payload["people"].items()
if person_id != "wang_yunlong"
if person_id not in _mobile_lookup_people
for profile, binding in person["bindings"].items()
if profile == "hermes-analyzer"
}
@@ -112,7 +123,7 @@ def test_baseline_contains_all_provided_people_and_routable_capabilities(
assert all(
binding["verified"] is False and binding["source"] == "provided"
for person_id, person in payload["people"].items()
if person_id != "wang_yunlong"
if person_id not in _mobile_lookup_people
for profile, binding in person["bindings"].items()
if profile == "hermes-analyzer"
)
Loaded 100 of 127 files, more files were not shown because too many files have changed in this diff. Show more