Compare commits

...
10 Commits
Author SHA1 Message Date
wangyunlong 97a486e97d feat(product): 补充香港安培公开库存采集脚本与聚水潭库存适配
public_sku_inventory_sheet 与 jushuitan_inventory 适配器、hk_anpei 采集配置,
支撑 product.hk_anpei_public_inventory.daily 工作流。
2026-10-09 13:45:56 +08:00
wangyunlong 36fe7b81f8 feat(cookie-gate): 补充 cookie_gate 网关模块与 deploy/watchdog 保活脚本
cookie_gate.py(账号三级门控/阻断记录 pending_reruns/恢复重跑管理器)为此前
接线提交遗漏的模块文件;deploy/watchdog 为控制台与调度器的进程外保活脚本。
2026-10-09 13:45:56 +08:00
wangyunlong dc8be88e60 chore(config): 工作流配置、清单与文档同步
workflows/schedules/commands/notification-routing 与 source-manifests 四模块
清单同步登录统一化与新工作流(product.hk_anpei_public_inventory.daily 等);
docs 运维手册、控制台与耗时分布文档、CI 关键脚本清单、AGENTS 指南更新;
测试与模块注册对齐。
2026-10-09 13:45:30 +08:00
wangyunlong 90282ae766 fix(product): 千川任务中心账户选择页兜底与 jd_spus jsonb 查询修复
千川任务中心深链缺账户上下文时会被重定向到账户选择页,按 .account-list-item
卡片自动选中目标广告主后继续下载;jd_spus 列改 jsonb 后查询改用
jsonb_typeof/jsonb_array_length 防类型报错。
2026-10-09 13:45:30 +08:00
wangyunlong df11e195b7 feat(product): 京东新版商智竞店采集改造
入口跳转、两段式日期选择、搜索定位与表格解析适配新版商智;配合类目下拉与
XHR 拦截兜底。
2026-10-09 13:45:30 +08:00
wangyunlong b0e1b5066d feat(content): 自营三平台指标入库管线完善
sync_self_video_metrics 按平台隔离入库与字段口径调整,配套自营指标验收测试
更新。
2026-10-09 13:45:30 +08:00
wangyunlong 1e9ddffd2f fix(collect): 账号浏览器与小红书采集强制直连
目标站点均为国内站,Clash 代理出口被蝉妈妈/小红书 WAF 拉黑(TLS 直接 reset,
页面与 API 报 ERR_CONNECTION_CLOSED)。账号保活、扫码登录浏览器与小红书 ARK
采集统一加 --no-proxy-server 直连。
2026-10-09 13:45:29 +08:00
wangyunlong 41f448420d fix(chanmama): 导出确认弹窗被浮层清理误删与代理出口拉黑修复
浮层清理只删可见浮层——页面常驻 90+ 个 display:none 弹窗模板(含导出确认框
模板),误删后 Vue 只在游离节点上"打开"弹窗,页面上仅剩灰色遮罩与滚动锁,
导出等待 1320s 必超时;导出等待改为逐秒弹窗状态机(可点即点/点不到先关/卡死
强删并补点导出),导出失败落截图与 HTML 诊断;浏览器强制直连并切换真 Chrome
非 stealthy 模式(StealthySession 下蝉妈妈前端会静默压制弹窗渲染);导出按钮
原生可信点击优先,防 @click 绑定在内层元素时 JS 合成点击无效。
2026-10-09 13:45:29 +08:00
wangyunlong 130f06b320 feat(cookie-gate): 调度器恢复重跑接线与控制台健康自检
调度器维护循环挂载 CookieGateRecoveryManager,登录态恢复后按原业务日期自动
重跑被阻断工作流并飞书确认;工作流级 required_accounts 声明接入目录解析;
采集器抛 CookieExpiredError 退出 75;控制台新增 /api/health 自检端点与进程内
看门狗,配合 deploy/watchdog 双层保活;安装脚本同步。
2026-10-09 13:45:29 +08:00
wangyunlong 78ea4641ce refactor(login): 移除脚本内扫码重登,登录统一交给账号 vault 与 cookie-gate
删除 relogin_*/friday_relogin_parallel 子进程及其测试;批量补采、评论采集、
星图/蒲公英/B站/抖音自营与抖店竞品脚本不再内嵌扫码等待,登录态失效统一由
cookie-gate 阻断并走 gyxx accounts login 恢复;runtime-bindings 与模块注册
同步移除 content.relogin.weekly。
2026-10-09 13:45:19 +08:00
86 changed files with 5424 additions and 4937 deletions

No files matched your search

+1 -1
View File
@@ -19,7 +19,7 @@ lint:
stage: quality
script:
- uv run ruff check src tests
- uv run ruff check src/gyxx_flow/modules/content_marketing/daily_creator_exposure_scope.py src/gyxx_flow/modules/content_marketing/chanmama_export.py src/gyxx_flow/modules/content_marketing/self_operated_accounts.py src/gyxx_flow/modules/content_marketing/self_xhs_scraper.py src/gyxx_flow/modules/content_marketing/data/tools/sync_self_video_metrics.py src/gyxx_flow/modules/product_commerce/market_rank_limits.py src/gyxx_flow/modules/product_commerce/market_rank_product_import.py src/gyxx_flow/modules/shop_intelligence/collectors/dy_after_sales_quality.py src/gyxx_flow/modules/shop_intelligence/collectors/tmall_negative_reviews.py
- uv run ruff check src/gyxx_flow/modules/content_marketing/daily_creator_exposure_scope.py src/gyxx_flow/modules/content_marketing/chanmama_export.py src/gyxx_flow/modules/content_marketing/self_operated_accounts.py src/gyxx_flow/modules/content_marketing/self_xhs_scraper.py src/gyxx_flow/modules/content_marketing/data/tools/sync_self_video_metrics.py src/gyxx_flow/modules/product_commerce/market_rank_limits.py src/gyxx_flow/modules/product_commerce/market_rank_product_import.py src/gyxx_flow/modules/product_commerce/collect_douyin_qianchuan_ads.py src/gyxx_flow/modules/shop_intelligence/collectors/dy_after_sales_quality.py src/gyxx_flow/modules/shop_intelligence/collectors/tmall_negative_reviews.py
test:
stage: test
+1
View File
@@ -41,6 +41,7 @@ uv run gyxx accounts login|sync [account_id]
- Manual/backfill/maintenance commands must be declared in `config/commands.json`; never reintroduce recursive executable-script discovery.
- Each executable `workflow_id` compiles to a LangGraph `StateGraph`. Preserve the public contracts of the workflow model, `RunJournal`, `LockManager`, shadow comparison, and `EffectLedger` when changing graph execution.
- External systems: PostgreSQL via runtime-injected cloud DSN (`GYXX_POSTGRES_DSN`; no address/credential defaults in source, loopback rejected in cloud mode); Hermes loopback-only roles on `127.0.0.1:8642/8643` (key `GYXX_HERMES_API_KEY`); Feishu keeps its existing identity. Every browser binding keeps a unique CDP port plus isolated Profile/Cookie/storage-state; scripts sharing an account declare it under `accounts` in `runtime-bindings.json` — the vault `state/accounts/<id>/` is the single login authority (`gyxx accounts login/sync`) and its cookies merge read-only into member scripts while profiles stay isolated.
- 登录统一管理:正式执行的工作流由 `src/gyxx_flow/cookie_gate.py` 在采集步骤前注入每账号 `cookie-gate.<id>` 步骤(blocking 账号失效→阻断并飞书通知人工重登;`login_command_id` 账号先自动账密恢复;`credential_env_names` 账号 defer 给脚本内账密登录)。阻断记录 `pending_reruns`,由调度器维护循环里的 `CookieGateRecoveryManager` 在登录态恢复后按原业务日期自动重跑并发飞书确认。采集脚本不得内嵌扫码/验证码等待——失效时抛 `CookieExpiredError`(`adapters/browser.py`)退出 75;编排器工作流在 `workflows.json` 用 `required_accounts` 声明账号依赖。不要重新引入 relogin_*/批量补采里的扫码重登子进程。
- 启动或重启工作流控制台(`gyxx console`)必须携带 `--env-file D:\product-collector-analyze-flow\.env`(与调度器进程一致),否则 `PG_HOST/PG_PORT/PG_DB/PG_USER/PG_PASSWORD` 等云端 PostgreSQL 凭据和账号配置不会注入,「商品经营日报采集」「聚水潭全店铺款式日报」等正式执行会报“缺少 PostgreSQL 运行时凭据”。参考启动脚本:`var/tmp/start-console.ps1`。
- Source sync (`gyxx sources status/apply`): manifest-driven three-way hash drift check; only untransformed, conflict-free one-sided changes may be auto-copied, and only with `--execute`. Credentials, cookies, logs, and collected data never enter sync manifests.
+6 -7
View File
@@ -50,13 +50,6 @@
"kind": "python",
"notification_workflow_id": "content.marketing_report.daily"
},
{
"id": "content.login.refresh",
"module": "content_marketing",
"entry": "data/tools/friday_relogin_parallel.py",
"kind": "python",
"notification_workflow_id": "content.relogin.weekly"
},
{
"id": "content.creator_report.generate",
"module": "content_marketing",
@@ -472,6 +465,12 @@
"{business_date}"
]
},
{
"id": "product.hk_anpei_public_inventory.refresh",
"module": "product_commerce",
"entry": "public_sku_inventory_sheet.py",
"kind": "python"
},
{
"id": "product.backfill.run",
"module": "product_commerce",
-3
View File
@@ -222,9 +222,6 @@
"content.marketing_report.daily": {
"condition": "正式执行且启用发送时投递最终营销日报。"
},
"content.relogin.weekly": {
"condition": "刷新期间投递各平台扫码二维码,并在平台失败时投递失败汇总。"
},
"product.alert.daily": {
"condition": "命中销量下滑规则且尚未重复通知时投递告警。"
},
+13 -100
View File
@@ -312,8 +312,6 @@
},
"members": [
"content.comments.collect_xiaohongshu",
"content_marketing:data/tools/relogin_pgy.py",
"content_marketing:data/tools/relogin_xiaohongshu.py",
"content_marketing:pgy_xhs_scraper_v2.py"
]
},
@@ -340,7 +338,6 @@
},
"members": [
"content.comments.collect_douyin",
"content_marketing:data/tools/relogin_douyin.py",
"content_marketing:self_douyin_scraper.py"
]
},
@@ -368,12 +365,15 @@
"real_chrome": true
},
"members": [
"content_marketing:data/tools/relogin_xingtu.py"
]
},
"chanmama": {
"cdp_port": 22207,
"login_mode": "D",
"credential_env_names": [
"CHANMAMA_ACCOUNT",
"CHANMAMA_PASSWORD"
],
"required_cookie_domains": [
"chanmama.com"
],
@@ -411,8 +411,7 @@
},
"members": [
"content.comments.collect_bilibili",
"content.metrics.collect_self_bilibili",
"content_marketing:data/tools/relogin_bilibili.py"
"content.metrics.collect_self_bilibili"
]
}
},
@@ -582,23 +581,6 @@
],
"cdp_port": 22016
},
"content_marketing:data/tools/friday_relogin.bat": {
"script_id": "content_marketing:data/tools/friday_relogin.bat",
"state_key": "content_marketing:data/tools/friday_relogin.bat",
"aliases": [
"content_marketing:data/tools/friday_relogin.bat"
],
"cdp_port": 22017
},
"content.login.refresh": {
"script_id": "content_marketing:data/tools/friday_relogin_parallel.py",
"state_key": "content_marketing:data/tools/friday_relogin_parallel.py",
"aliases": [
"content_marketing:data/tools/friday_relogin_parallel.py"
],
"cdp_port": 22018,
"login_mode": "D"
},
"content.creator_report.generate": {
"script_id": "content_marketing:data/tools/generate_creator_report.py",
"state_key": "content_marketing:data/tools/generate_creator_report.py",
@@ -679,83 +661,6 @@
],
"cdp_port": 22135
},
"content_marketing:data/tools/relogin_bilibili.py": {
"script_id": "content_marketing:data/tools/relogin_bilibili.py",
"state_key": "content_marketing:data/tools/relogin_bilibili.py",
"aliases": [
"content_marketing:data/tools/relogin_bilibili.py"
],
"cdp_port": 22028,
"login_mode": "D",
"required_cookie_domains": [
"bilibili.com"
],
"required_cookie_names": [
"SESSDATA"
]
},
"content_marketing:data/tools/relogin_douyin.py": {
"script_id": "content_marketing:data/tools/relogin_douyin.py",
"state_key": "content_marketing:data/tools/relogin_douyin.py",
"aliases": [
"content_marketing:data/tools/relogin_douyin.py"
],
"cdp_port": 22029,
"login_mode": "D",
"required_cookie_domains": [
"douyin.com"
],
"required_cookie_names": [
"sessionid"
]
},
"content_marketing:data/tools/relogin_pgy.py": {
"script_id": "content_marketing:data/tools/relogin_pgy.py",
"state_key": "content_marketing:data/tools/relogin_pgy.py",
"aliases": [
"content_marketing:data/tools/relogin_pgy.py"
],
"cdp_port": 22030,
"login_mode": "D",
"required_cookie_domains": [
"xiaohongshu.com"
]
},
"content_marketing:data/tools/relogin_xiaohongshu.py": {
"script_id": "content_marketing:data/tools/relogin_xiaohongshu.py",
"state_key": "content_marketing:data/tools/relogin_xiaohongshu.py",
"aliases": [
"content_marketing:data/tools/relogin_xiaohongshu.py"
],
"cdp_port": 22031,
"login_mode": "D",
"required_cookie_domains": [
"xiaohongshu.com"
],
"required_cookie_names": [
"web_session"
]
},
"content_marketing:data/tools/relogin_xingtu.py": {
"script_id": "content_marketing:data/tools/relogin_xingtu.py",
"state_key": "content_marketing:data/tools/relogin_xingtu.py",
"aliases": [
"content_marketing:data/tools/relogin_xingtu.py"
],
"cdp_port": 22032,
"login_mode": "D",
"required_cookie_domains": [
"douyin.com",
"oceanengine.com",
"xingtu.cn"
],
"required_cookie_names": [
"sessionid",
"sessionid_ss",
"sid_tt",
"sid_guard"
]
},
"content.failed.retry": {
"script_id": "content_marketing:data/tools/retry_failed.py",
"state_key": "content_marketing:data/tools/retry_failed.py",
@@ -1441,6 +1346,14 @@
"erp321.com"
]
},
"product.hk_anpei_public_inventory.refresh": {
"script_id": "product_commerce:public_sku_inventory_sheet.py",
"state_key": "product_commerce:public_sku_inventory_sheet.py",
"aliases": [
"product_commerce:public_sku_inventory_sheet.py"
],
"cdp_port": 22166
},
"product_commerce:scripts/insert_jd_main_image_records.py": {
"script_id": "product_commerce:scripts/insert_jd_main_image_records.py",
"state_key": "product_commerce:scripts/insert_jd_main_image_records.py",
+8 -13
View File
@@ -11,10 +11,7 @@
"workflow_id": "content.metrics.daily"
},
{
"at": [
"01:00",
"05:00"
],
"at": "06:00",
"business_date_offset_days": 0,
"enabled": true,
"kind": "daily",
@@ -42,15 +39,6 @@
"kind": "daily",
"workflow_id": "content.marketing_report.daily"
},
{
"at": "10:00",
"days": [
"Friday"
],
"enabled": false,
"kind": "weekly",
"workflow_id": "content.relogin.weekly"
},
{
"at": "12:00",
"days": [
@@ -221,6 +209,13 @@
"kind": "daily",
"workflow_id": "product.erp_all_shop_daily"
},
{
"at": "09:00",
"business_date_offset_days": 0,
"enabled": true,
"kind": "daily",
"workflow_id": "product.hk_anpei_public_inventory.daily"
},
{
"at": "12:00",
"business_date_offset_days": 0,
+13 -69
View File
@@ -7,7 +7,7 @@
"source_relative_path": "bilibili_comment_scraper.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/bilibili_comment_scraper.py",
"source_sha256": "c7f116f30889125c668e7f4dd796e7249ba7ea8621d5ab5fb518028f56c7f8dc",
"target_sha256": "109a01a6dc2b79261cff68ff42818a7561f3319b6be3788d9300bb5ce0217a96",
"target_sha256": "d3ae63aab128a3e4078cf357aa4dfcf1f2c3b36c3116989d8f2252be62544e53",
"transformed": true,
"category": "source"
},
@@ -23,7 +23,7 @@
"source_relative_path": "chanmama_scraper.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/chanmama_scraper.py",
"source_sha256": "2edc26879b57d73ec5992c80e5ef74d76a509f09bff7284b1ae17e90df543e07",
"target_sha256": "8b87268e0f105cf4d349b192c7b2c24d5e318a9d88fcd14264824b083103b24e",
"target_sha256": "b808b412e9bb8d824c1a117cbecdd28234408a55fa60891f1ae748abeb7bc415",
"transformed": true,
"category": "source"
},
@@ -111,7 +111,7 @@
"source_relative_path": "data/tools/batch_rescrape_bilibili.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/data/tools/batch_rescrape_bilibili.py",
"source_sha256": "5b07fe27f1b76714ca3f3feae34e3d719277ed0cfe124d76cacd8e3c904b1fca",
"target_sha256": "10e5dee90c591201daa7a7b0eb13a40ed1c97d7d51b1b93ef8f1d5d4b8e09580",
"target_sha256": "8fa0e5838615f36c9841deb08f6710693e63b011f1f91138c1121d99277dd391",
"transformed": true,
"category": "tool"
},
@@ -119,7 +119,7 @@
"source_relative_path": "data/tools/batch_rescrape_douyin.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/data/tools/batch_rescrape_douyin.py",
"source_sha256": "15b1438513f8216a6c7a103744e8a1ca6bc63b32c111d7c9bce31c18be9e84a6",
"target_sha256": "3233a7840e07a87535a8df065562a7cb681dbc140e002d82970598de3d9768cf",
"target_sha256": "e0f1227a7b66d85ea65191b5f9c6c5bf588e85cc0cd66e20afae7a9dc7a2684e",
"transformed": true,
"category": "tool"
},
@@ -127,7 +127,7 @@
"source_relative_path": "data/tools/batch_rescrape_xiaohongshu.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/data/tools/batch_rescrape_xiaohongshu.py",
"source_sha256": "6d85f7dfd52c7a92b951ac29e7ee3377aebdb743af2a362d1f38366188dcad08",
"target_sha256": "01be5d1ba260ccd46e01b9e72b6f7dd3d08577615455fc9010de4efe8a710aea",
"target_sha256": "c481bd6af34040388abafed0466d5632b3892865bf192d245664627003a4c43f",
"transformed": true,
"category": "tool"
},
@@ -183,7 +183,7 @@
"source_relative_path": "data/tools/db.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/data/tools/db.py",
"source_sha256": "83f158a78841244c1f02255d1500018b5a9dbe1f282cbdb937acc344b08abc25",
"target_sha256": "62ffad854cc2bc3a3ae4f2a28b4ee345bc9eecf72101b647c3aa165c691b456a",
"target_sha256": "bfdb761761532263ae6c555ee455cad4efa300b7fa5639931813f4f95e79ff6a",
"transformed": true,
"category": "tool"
},
@@ -203,14 +203,6 @@
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/friday_relogin_parallel.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/data/tools/friday_relogin_parallel.py",
"source_sha256": "affed4ab6e5e179c85dc92999502b98e007c71243ed26977b3c5eeba23b9e3dc",
"target_sha256": "438bedfb874c7eb566e1998815a22ca72b7d177cee221fd39625a96872bd450b",
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/generate_creator_report.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/data/tools/generate_creator_report.py",
@@ -291,54 +283,6 @@
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/relogin_bilibili.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/data/tools/relogin_bilibili.py",
"source_sha256": "b8dc12539bcd6f1a368d50ead1f5d33d6577a75c92f0b7bafa75a67edd7a73fa",
"target_sha256": "d47f26f62fbe3262128327b916151385e527125a18f27f7455b8382aa5a77ecb",
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/relogin_douyin.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/data/tools/relogin_douyin.py",
"source_sha256": "3c2017f8a801528ff17e4c4bae24615256b8788c185f9a797fe87470e35b1645",
"target_sha256": "82a0ecf4101f2eaded94cc4cd9054cb08e1e27519a58cfc0d3126888cc1ce5d4",
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/relogin_pgy.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/data/tools/relogin_pgy.py",
"source_sha256": "759c6a68a0851b4c87d8febfc5d94925831a3b2d7cdf1e6a6e74dcd52b83ae3f",
"target_sha256": "d75697ae2b6ac1f23bb79077a4df0af33113813a46525a6f3980b04391857cac",
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/relogin_transaction.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/data/tools/relogin_transaction.py",
"source_sha256": "57ec20de8d99290268ed3d1fc2b66bf3faeaca807d7f3744dd51de6cc93628b2",
"target_sha256": "57ec20de8d99290268ed3d1fc2b66bf3faeaca807d7f3744dd51de6cc93628b2",
"transformed": false,
"category": "tool"
},
{
"source_relative_path": "data/tools/relogin_xiaohongshu.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/data/tools/relogin_xiaohongshu.py",
"source_sha256": "fcdea30adecb066c4f3699ded457df0bc763d978248276ddbffa9f18bc9bdf2c",
"target_sha256": "e700dd57e5cf87d68529bcd6c062e73d6bd9f1e6ef06ef3122e2b16c288aaf1a",
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/relogin_xingtu.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/data/tools/relogin_xingtu.py",
"source_sha256": "c9d434cb02a57ebd7b884598563077d733d56e45dad59ed30774eb7e1c668cb0",
"target_sha256": "06ec1088677d69511057008acb2b05cca2da015a8219aee10bf19bbae3c25d79",
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/retry_failed.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/data/tools/retry_failed.py",
@@ -351,7 +295,7 @@
"source_relative_path": "data/tools/schema_gyxx_super_data.sql",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/data/tools/schema_gyxx_super_data.sql",
"source_sha256": "43d202d7d0452b7d0671ee9c40149c21328db4b2863fa4d030af456f89defe3c",
"target_sha256": "001870851dea9c373a77b87335604c4363b06c3eb0ed7c97b4a8d5e891e56758",
"target_sha256": "4b2eca918bb81de0cac529b786d0d37ce04af2c1bc4be81cd20c8ba9456b1348",
"transformed": true,
"category": "sql"
},
@@ -415,7 +359,7 @@
"source_relative_path": "douyin_comment_scraper.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/douyin_comment_scraper.py",
"source_sha256": "ec785ead9c95c5de65c00b06212b1f03d0159fed475a95a2b593d3b24e486e43",
"target_sha256": "6e7ac4357fe466656b93ca7bae6c49adc4d59fa75974a39c6885c7433bd4d402",
"target_sha256": "35d3caf8bc85a82ca15c3fc9ff317c21460173996106ffee8f1d3b5f1d13b801",
"transformed": true,
"category": "source"
},
@@ -455,7 +399,7 @@
"source_relative_path": "pgy_xhs_scraper_v2.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/pgy_xhs_scraper_v2.py",
"source_sha256": "2be78ef76df38e80da2099a76e6e3fd9dcdc2e8875afb44279f54003ccb5f563",
"target_sha256": "e5df4b34f67a565ec5d5ce4dd5857e80e363dd7559e7e29006dcd3e421276c8a",
"target_sha256": "472c13700adc98144b53f2921bbdde15ebb7dc138eaab76885b514dadd306887",
"transformed": true,
"category": "source"
},
@@ -479,7 +423,7 @@
"source_relative_path": "self_bilibili_scraper.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/self_bilibili_scraper.py",
"source_sha256": "2ef650a16dde61697101bb219a8c3adcd682fec1529bb9c5312dc5b384d83a72",
"target_sha256": "6ec5ed6991a93f806fb74d0870e953e5a05d52043a83f2fa261e2ec67bee6ad7",
"target_sha256": "1c09171d1269a3a38a5e6f9f99b98d0da4c7ff11d075613fc288ee209e82e9d2",
"transformed": true,
"category": "source"
},
@@ -487,7 +431,7 @@
"source_relative_path": "self_douyin_scraper.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/self_douyin_scraper.py",
"source_sha256": "4aba1264dee052521e5712d7219916e534db5705e3cec75caaa1b025a4bc2506",
"target_sha256": "62307df115c8d7336b470baba074163ab3f3468bae8b0af266af370e2e42d9f1",
"target_sha256": "945659cd80435bab08bfe1dee13c451c8c4b5fe362313ea09591525c551d3c48",
"transformed": true,
"category": "source"
},
@@ -511,7 +455,7 @@
"source_relative_path": "xiaohongshu_comment_scraper.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/xiaohongshu_comment_scraper.py",
"source_sha256": "58bc07bb2fd67897e18a9981f931df8fadf24a6cc5f09d55bd3aaa2e3c6756a4",
"target_sha256": "afca1104324546da2315fdf7a427c3f34021eef31ddc2fea76877f818718ccb0",
"target_sha256": "fff780d9e16717e6a1ba9c5ee4db22cb1bb5a85f75807c811072c20a6bd390de",
"transformed": true,
"category": "source"
},
@@ -527,7 +471,7 @@
"source_relative_path": "xingtu_scraper_v2.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/xingtu_scraper_v2.py",
"source_sha256": "0d4eb87779064e13b8d78daa6dba13698f1149a26b3a1de2e064c117a29cf49a",
"target_sha256": "b36339444e26048a90c8ac999a0a628bcddd4f64842924f6c60e600c7a3bec37",
"target_sha256": "dfdd07af20c8e2e59411fb6e32a82588db5601d6f3fba75d596e0b5ac19b2be8",
"transformed": true,
"category": "source"
}
+25 -25
View File
@@ -14,7 +14,7 @@
{
"category": "production_source",
"source_sha256": "cff768f79672cfc7db7f6866caf5ab59936b8c51eb2a7d729b81441940124512",
"target_sha256": "d56fe8a934c1ae20a9249c7682823238a705f51f451241504512e83a8800a879",
"target_sha256": "c1c0167b96c818a4e7d44aa5c860b5510c9925cb096b7fb4510b8319b91592fc",
"transformed": true,
"source_relative_path": "analyze_style_with_hermes.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/analyze_style.py"
@@ -54,7 +54,7 @@
{
"category": "production_source",
"source_sha256": "1493dbbfb3d864812df378d04d006e7de0f9ec4b6ab5812e5c6970c304c2dfe6",
"target_sha256": "a1d526dfc27b110a8845a98cc66fe7e5cfb87ef70530728477af31a88734a576",
"target_sha256": "2b6e6adfea82b13dd8c819138a428a78b5802f204c199c58ed4e59f0158d1780",
"transformed": true,
"source_relative_path": "collect_dy_market_rank.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/collect_dy_market_rank.py"
@@ -62,7 +62,7 @@
{
"category": "production_source",
"source_sha256": "abd4c82cbd6c41e111e213c174d2fdeba1214568496eaf1dede24bf374b3b052",
"target_sha256": "9de3ab22d5a4dbf574b2d4fdbbed295bb4645bb4943f4c066fcbfd41b44b521c",
"target_sha256": "eedf83f1d85662dc2e6d968af29070be819c7caac25057c74d6cb66b75f52216",
"transformed": true,
"source_relative_path": "collect_dy_persona_to_bitable.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/collect_dy_persona_to_bitable.py"
@@ -70,7 +70,7 @@
{
"category": "production_source",
"source_sha256": "c7b54d941ab7f056e2d0ea4355543ff4a5c34ad986c8ed32145e988ac9501ea2",
"target_sha256": "aaba5dfe6ce5133b5c662232160bb9faef4d73b36902a484a56b89fa7df05311",
"target_sha256": "8576678688d7d90f6f7285110de005982bc1ba82bc7e9977224ba1bdb4101283",
"transformed": true,
"source_relative_path": "collect_erp_yesterday_metrics.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/collect_erp_yesterday_metrics.py"
@@ -78,7 +78,7 @@
{
"category": "production_source",
"source_sha256": "8bb1f17f90a7b9c2db5dfb992cf746329892d19c00ae2ae988e2109c049598b2",
"target_sha256": "c98abe563b223f49cc3c810054d997504d9465387da4805389b0f5ef78ceed6a",
"target_sha256": "9b24259854a440d5a13fa2bc397887ab86cf7d4a573b10d3bda6988ed37d0ed6",
"transformed": true,
"source_relative_path": "collect_jd_market_rank.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/collect_jd_market_rank.py"
@@ -86,7 +86,7 @@
{
"category": "production_source",
"source_sha256": "f9a48c5f89b18472dc3993e340ad858bfecb2fcf59427c7e70db37512c17a6f7",
"target_sha256": "a268df592f7136a68c2b82d031115055e43cdfaedcdd4a5f4afa9ef45879e055",
"target_sha256": "01798bd6110afc84e5a6aab50255deeb0c5980f77895fcaf605ccb93d24656a5",
"transformed": true,
"source_relative_path": "collect_jd_persona_to_bitable.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/collect_jd_persona_to_bitable.py"
@@ -110,7 +110,7 @@
{
"category": "production_source",
"source_sha256": "13b6e68b4872b2e79c1de8ca1617cc4f1249f4ed32f234ad6efb7a04f5948612",
"target_sha256": "a1b95cc4f090a25c8e7991b0ef015a7df612fc78249dafe75aeef9fa4dbefe08",
"target_sha256": "35b5838736306d58758dca870deb7a296a21a2cbc7bb0c4b84d7f233095bee22",
"transformed": true,
"source_relative_path": "collect_sycm_market_rank.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/collect_sycm_market_rank.py"
@@ -134,7 +134,7 @@
{
"category": "production_source",
"source_sha256": "a8900b9f25db6cf2bef3117a1d2cd86283054fea6f5a8590fd91b5cfbbcaf87e",
"target_sha256": "030d33c3f5c0c230bb67d603dd657f40cceff8581f8756a1f67340a89b5ae4b3",
"target_sha256": "47cdfd9ebf27e31be06497b2f74c80da4ffced259a873730eb024b0db5eba740",
"transformed": true,
"source_relative_path": "db/__init__.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/db/__init__.py"
@@ -158,7 +158,7 @@
{
"category": "production_source",
"source_sha256": "78c74c0e52e3651b3b042a2095e596ab250e60309c7ae2de905850822aad4922",
"target_sha256": "09d7e08d2eec57055a3bc7949e49c32a8bf62713f2594ae5d9297c032bf212c8",
"target_sha256": "1e6ed0029c9d1f701ae1d80b670fdb465cdbfd13b9f63d0a3179c19585215246",
"transformed": true,
"source_relative_path": "dy_audience_profile_collect.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/dy_audience_profile_collect.py"
@@ -166,7 +166,7 @@
{
"category": "production_source",
"source_sha256": "492916088fa5e8c170461cc9eba97a88ed1cc1e50c7b99e5f6a5e49167fea157",
"target_sha256": "9dd0dc3e6f048683157fe8b8deb533ed88970bf34e9824e439bf3a5d52d80e0a",
"target_sha256": "7b0ae5d648245e89f4aacf262250009de812d3dbb8ee55386f3704cabca48799",
"transformed": true,
"source_relative_path": "dy_product_scraping.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/dy_product_scraping.py"
@@ -206,7 +206,7 @@
{
"category": "production_source",
"source_sha256": "906278aa6fb3578d5dab231bdccc2f5a7c6ec8c85810a3e026fff3c7ba16dc90",
"target_sha256": "d9e4281c21a42841023c0a723aff4a00af1f9200c94e42add9f8965159d4cfb5",
"target_sha256": "c9ac5b78ce367eb0b87697255a88400d1e2e83416ee1fc105110d3abc3dd8245",
"transformed": true,
"source_relative_path": "import_product_daily.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/import_product_daily.py"
@@ -230,7 +230,7 @@
{
"category": "production_source",
"source_sha256": "e0f016bafb3f527f1cae46eaf3ef31510184128aaaa7f52024a7c3f978eec42b",
"target_sha256": "f2692442a1e62a4b20dbb442eb3e27a3075ca6095ba2578c90193f0db3a1810e",
"target_sha256": "b157f24430170d5a991bb2283af0a747b26dd11c0f0dec37289774f115132fee",
"transformed": true,
"source_relative_path": "jd_main_image_collector.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/jd_main_image_collector.py"
@@ -238,7 +238,7 @@
{
"category": "production_source",
"source_sha256": "d99ce71a008128549ac83fc3da52d88540b74538e3fe5c85d47c3b985e414353",
"target_sha256": "f821a0221b832bea0123bf4a8f85fd0785eba9206c8a632ad48065ea3c30c095",
"target_sha256": "cfe98b322e4ef7bd875325c2a9ffedeabb364e98b2983773dc98f9e0e75a6581",
"transformed": true,
"source_relative_path": "jd_product_data_collector.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/jd_product_data_collector.py"
@@ -294,7 +294,7 @@
{
"category": "production_source",
"source_sha256": "910e720fab4f47419ad56055f00a8c3fbe3a06c2577d81e35d3d38178a588a45",
"target_sha256": "c9f8cfc8dff6a7e5617b9b357ed92872f94371f93360318a3ff78f7ec38d75df",
"target_sha256": "e515cdc287b39a315dec44b4fb1c17f98b2af13bf82715731f92919d45f87854",
"transformed": true,
"source_relative_path": "market_rank_product_import.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/market_rank_product_import.py"
@@ -358,7 +358,7 @@
{
"category": "production_source",
"source_sha256": "c9a51f1d1267b398cf260a86e65a8ee49de873d6c95bf9c90db2abb422005cf7",
"target_sha256": "6cd70bb5406bc00257f9070760f980580e95c3c4e4326a67d33d30433dd74c32",
"target_sha256": "54c7b5802f66f9892b455aadd4a7396b7b693b2dda729aa7c22f8909821acd17",
"transformed": true,
"source_relative_path": "run_daily_persona.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/run_daily_persona.py"
@@ -390,7 +390,7 @@
{
"category": "production_source",
"source_sha256": "a1b097ead2a3f7e20ca63d7c15828a8d04030b0d6722934c3bf65d9e975fba76",
"target_sha256": "a231ccd2efacefca703a975e6c63e05a36c61a3db0975e46b1fab8d983fe84d2",
"target_sha256": "7d8587f77615040087d2d475a7b440ad16a39c5874e143418dd700ecbc5db0f8",
"transformed": true,
"source_relative_path": "scripts/insert_main_image_records.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/scripts/insert_main_image_records.py"
@@ -406,7 +406,7 @@
{
"category": "production_source",
"source_sha256": "4474dd7d68117247af87e20bbeb6b7c072368ef0f606de5d3e186f459d34a0f5",
"target_sha256": "cfbb46483113cda9e4cb301b092ac2ce6a35c5cc53b9a9cef01fdf93a00d3b94",
"target_sha256": "862ae9392227113ad4266c2170769595c36b2f98fdecc54e54a648ba834baebf",
"transformed": true,
"source_relative_path": "taobao_sycm_collect.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/taobao_sycm_collect.py"
@@ -438,7 +438,7 @@
{
"category": "production_source",
"source_sha256": "e4d99956d9649d108baeb3e3a2d1969c227622d1037062bd277af66375556d82",
"target_sha256": "981ef2a5a59f73ae244240520320c73be9c9462b50546df1ff91d0c97e7c03f7",
"target_sha256": "2ccb359ab943c185b17854f0f00cfb76b8a1235d7b031209a1c07e9386f8d93f",
"transformed": true,
"source_relative_path": "upload_video_to_guanghe.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/upload_video_to_guanghe.py"
@@ -454,7 +454,7 @@
{
"category": "production_source",
"source_sha256": "cceb3861da0e68c686c0c58f64767c3ca2693986bb5006ca7405d3a5a39c4347",
"target_sha256": "8a947ce42d34ff26bb46c5173af9e37ee9d60eac1b595d2f54b855a881ac2d6c",
"target_sha256": "4a44a00ec0e13bac3c4369c0336c6b6410e012d59e39900f3b4d0c093c8def90",
"transformed": true,
"source_relative_path": "vendors/dy-data-flow/dy_store_competitor_store_scraping.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/vendors/dy-data-flow/dy_store_competitor_store_scraping.py"
@@ -488,7 +488,7 @@
{
"category": "production_source",
"source_sha256": "6daa16c03989fb2231e54545726f5aa488e4ba80b9b8aa6f937cb9a2fa13ff35",
"target_sha256": "89ba008ef4388c6e1ed32e4a5c26ffa81d083164dd6d7231fc9e5886de8437cd",
"target_sha256": "5ea444181ca731d8bdd983fa8d0179422651fdd994ebe08a63ed2bfae155fca5",
"transformed": true,
"source_relative_path": "vendors/jd-data-flow/jd_data_collector.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/vendors/jd-data-flow/jd_data_collector.py"
@@ -496,7 +496,7 @@
{
"category": "production_source",
"source_sha256": "a53cc38b0ddeaa0065bc27a1d137ce16c2873ff9bec712483ca87bcedd6baa3a",
"target_sha256": "0549228539c2e1908c3181de13291457712eeaea8fea89903814844724a3b5ee",
"target_sha256": "a6ee61498a040be9d45c6b8f9ecdbc06213f5f1e5507783412188f35d2968640",
"transformed": true,
"source_relative_path": "vendors/jd-data-flow/jd_peer_product_data_collector.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/vendors/jd-data-flow/jd_peer_product_data_collector.py"
@@ -560,7 +560,7 @@
{
"category": "runtime_resource",
"source_sha256": "9739407b61842c598491d3189a32f5f15aa944aed79a164a5cd5b5a34f20c746",
"target_sha256": "8ff84f6c3c672af35e09f70d71aa73656ac101218d8a56126b8b0a75ab387c8b",
"target_sha256": "6bbd36048836ef7ed0807223fc028ac17349b20314b1ea353ccedb451957ba55",
"transformed": true,
"source_relative_path": "db/schema.sql",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/db/schema.sql"
@@ -672,7 +672,7 @@
{
"category": "regression_test",
"source_sha256": "820e17b910b62a213f0bebafcff72998c2c978307c97f976c074276150ce7c6a",
"target_sha256": "1159c9e3f0abdf17183e7d6fa2634c508fe8348ce2926e7d3c58e735b713c2e9",
"target_sha256": "cdedf8c9a4871ad763e0a9098fc18777f0c3206991e8d440a45ccdfbda089a07",
"transformed": true,
"source_relative_path": "tests/test_main_image_concurrency.py",
"target_relative_path": "tests/modules/product_commerce/test_main_image_concurrency.py"
@@ -696,7 +696,7 @@
{
"category": "regression_test",
"source_sha256": "f1e660d34e21d3cd7a26d3605c82d07982ba6c5002af7b56fb6a876cb706713d",
"target_sha256": "85ca8603e2af8257ce2a940b99c69948c2c1362f688e6f6dac4cbf499f1e4d73",
"target_sha256": "4a21ff4424c92a0bed6bd2f70f6a7bc7b426b3a25ccc356612f27a7426ce1eca",
"transformed": true,
"source_relative_path": "tests/test_market_rank_product_import.py",
"target_relative_path": "tests/modules/product_commerce/test_market_rank_product_import.py"
@@ -728,7 +728,7 @@
{
"category": "regression_test",
"source_sha256": "f9cfbb49fd313444790ac194fbf980dae21912081b9f26ad8e3850912126f09e",
"target_sha256": "0cc2b365144482ceb36667eedfb2c403e904437f3ea4c7e12f7099276c13a279",
"target_sha256": "043b2dd69172bf5ea1b324bf07f8bba4c43f4bca9817c30a2c805ffe8ce70525",
"transformed": true,
"source_relative_path": "tests/test_style_analysis_orchestration.py",
"target_relative_path": "tests/modules/product_commerce/test_style_analysis_orchestration.py"
@@ -32,7 +32,7 @@
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/collectors/dy_store_competitor_store_scraping.py",
"category": "source",
"source_sha256": "009879a7dd341ca6ecbc8eb66f5700114c73bcaba3a53f159014a76232d0aa6b",
"target_sha256": "d5e85ded1f852ba418229c75c985d0ece967dfbd5fd31697fe407b64d1c3db05",
"target_sha256": "9ab1328c2f1743944e8f096e548255884bc9724bf1e900c60c069d443c886902",
"transformed": true
},
{
@@ -48,7 +48,7 @@
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/collectors/jd_peer_store_data_collector.py",
"category": "source",
"source_sha256": "4ebdb52df7b384d2bfdd2fc840d55d14e18864002c20a36685b63cee1966cf4c",
"target_sha256": "39faf18e9cd5e441323a8e056e022f179018a44eac224d64383cd7ae9fa1d611",
"target_sha256": "f03b1a7226071f2225a2a71748ad2c27a102cde67ba02d37c63f15480dd0de32",
"transformed": true
},
{
@@ -88,7 +88,7 @@
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/db/__init__.py",
"category": "source",
"source_sha256": "ffc89b78ff0081a129e04eb6de4985ab0a2a4613c9e8a1d8b2750d8c17e8bc05",
"target_sha256": "a4bae02d0c07f3e3986896b31e62f41bf15edbf74d3776b1e050e381c15a8413",
"target_sha256": "08b5b8920338100339e7c716423b0e9260d4e007d0a1cc3f7510d51970a66a44",
"transformed": true
},
{
@@ -96,7 +96,7 @@
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/db/db.py",
"category": "source",
"source_sha256": "168b6f87c1827b267f763ddce7eb15d0c8d652c8c0c7af18dcd0c6c7da5cddd1",
"target_sha256": "01469f9eb80aee5e80bc28e548dbe2cd5ce861ab35cd480b33921a2adc799320",
"target_sha256": "4ab56df6c8ff75b1a67256ad8cdef34c58bb97e1a126707aac7ccf82980a56ec",
"transformed": true
},
{
@@ -104,7 +104,7 @@
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/db/schema.sql",
"category": "sql",
"source_sha256": "73230590bde7ae6359c02c82c2db096c0d3faa55b374e30348c47d7b40ea498b",
"target_sha256": "a69fa151f989f106d00f344bab43ec9e6a09cf404bdc581624cc0901cdc13d20",
"target_sha256": "9558543cf313aadb785715fca8f5cb99219488e7cb6c37cb00310bfe84f7ee74",
"transformed": true
},
{
+1 -1
View File
@@ -120,7 +120,7 @@
"target_relative_path": "src/gyxx_flow/modules/supply_chain/orchestrator/scripts/ProductReplenishment.py",
"category": "source",
"source_sha256": "02f6345142ebd84e25dc2f3b0651bc139ef31dd3994ff739efe66344c77ee49e",
"target_sha256": "e74211f2fe97ac47e704a040f442f51c97fecc15ea30dd1f0dbeacaf5940fd80",
"target_sha256": "ae5154b1f2af1e03307c07e4188225cd9601d1e1b5316fc0f6a78d23303ba8de",
"transformed": true
},
{
+81 -50
View File
@@ -3,6 +3,11 @@
"workflows": [
{
"id": "content.metrics.daily",
"required_accounts": [
"douyin-content",
"douyin-shop",
"xiaohongshu-content"
],
"module": "content_marketing",
"trigger": "scheduled",
"execution": {
@@ -302,6 +307,11 @@
},
{
"id": "content.metrics.backfill",
"required_accounts": [
"douyin-content",
"douyin-shop",
"xiaohongshu-content"
],
"module": "content_marketing",
"trigger": "scheduled",
"execution": {
@@ -430,55 +440,6 @@
"task_name": "YingxiaoYunying_DailyMarketingReport"
}
},
{
"id": "content.relogin.weekly",
"module": "content_marketing",
"trigger": "scheduled",
"execution": {
"steps": [
{
"id": "relogin",
"name": "刷新内容平台登录态",
"description": "并行启动五个平台的二维码重新登录流程,按配置轮次重试,持久化登录状态并通知结果。",
"data_flow": {
"sources": [
{
"system": "内容平台",
"label": "五个平台登录页与当前会话状态"
}
],
"processing": [
"并行发起二维码登录并按轮次重试",
"校验并持久化新的登录会话"
],
"destinations": [
{
"system": "浏览器状态",
"label": "各平台 Cookie 与 storage state"
},
{
"system": "飞书",
"label": "登录刷新结果通知"
}
]
},
"entry": "data/tools/friday_relogin_parallel.py",
"args": [
"--max-attempts",
"3",
"--round-timeout",
"300",
"--screenshot-delay",
"60"
]
}
]
},
"provenance": {
"source_project": "content",
"task_name": "YingxiaoYunying_FridayRelogin"
}
},
{
"id": "content.creator_report.monthly",
"module": "content_marketing",
@@ -781,6 +742,9 @@
},
{
"id": "shop.metrics.weekly",
"required_accounts": [
"douyin-shop"
],
"module": "shop_intelligence",
"trigger": "scheduled",
"execution": {
@@ -890,6 +854,9 @@
},
{
"id": "shop.competitor.weekly",
"required_accounts": [
"douyin-shop"
],
"module": "shop_intelligence",
"trigger": "scheduled",
"execution": {
@@ -1592,6 +1559,10 @@
},
{
"id": "product.daily",
"required_accounts": [
"douyin-shop",
"erp-jushuitan"
],
"module": "product_commerce",
"trigger": "scheduled",
"execution": {
@@ -1916,6 +1887,7 @@
],
"processing": [
"校验广告主 ID、店铺标识和商品数据明细",
"任务中心被重定向到账户选择页时自动点击目标广告主后重试",
"按业务日期下载原始 Excel",
"重置千川错误的 worksheet A1 dimension 并校验日期覆盖"
],
@@ -2231,6 +2203,54 @@
},
"note": "每天自动采集昨天数据;控制台可手动选择任意补采日期并幂等覆盖该日。全选聚水潭全部店铺,不拆京东、抖音、天猫;每次只采集一个业务日期。"
},
{
"id": "product.hk_anpei_public_inventory.daily",
"module": "product_commerce",
"trigger": "scheduled",
"execution": {
"steps": [
{
"id": "query_and_refresh_sheet",
"name": "刷新香港安沛 SKU 公有可用数",
"description": "校验计划书固定 SKU 与现有库存工作表中的商品编码对应关系,调用聚水潭 /open/inventory/query 查询库存及锁定量,并将公有可用数写入仓库现有库存列。",
"entry": "public_sku_inventory_sheet.py",
"timeout_seconds": 900,
"replay_policy": "idempotent",
"data_flow": {
"sources": [
{
"system": "光影行星香港安沛2026Q2计划书",
"label": "现有库存表的 51 个唯一 SKU"
},
{
"system": "聚水潭开放 API",
"label": "/open/inventory/query",
"detail": "请求 has_lock_qty=true 以取得库存锁定量"
}
],
"processing": [
"校验整批 SKU 库存响应后再写入飞书",
"按商品编码(光影)匹配行;公有可用数 = qty - order_lock - lock_qty + virtual_qty",
"没有库存记录的 SKU 清空仓库现有库存单元格",
"仅写入仓库现有库存列,不覆盖图片、备注、在途生产或其它字段"
],
"destinations": [
{
"system": "飞书电子表格",
"label": "光影行星香港安沛2026Q2計劃書",
"detail": "現有庫存工作表;按商品编码(光影)更新 D 列仓库现有库存"
}
]
}
}
]
},
"provenance": {
"source_project": "data-hub",
"task_name": "HongKongAnpeiPublicSkuInventoryDaily"
},
"note": "每天 Asia/Shanghai 09:00 执行。聚水潭应用凭据从 GYXX 调度进程环境读取;不得使用开发配置默认值。"
},
{
"id": "product.video_upload",
"module": "product_commerce",
@@ -2350,7 +2370,6 @@
"args": [
"--anchor-record-id",
"recvrXTRWWFXeJ",
"--manual-login",
"--execute"
],
"timeout_seconds": 21600
@@ -2607,6 +2626,9 @@
},
{
"id": "supply.purchase_confirmation.daily",
"required_accounts": [
"erp-jushuitan"
],
"module": "supply_chain",
"trigger": "scheduled",
"execution": {
@@ -2657,6 +2679,9 @@
},
{
"id": "supply.replenishment.weekly",
"required_accounts": [
"erp-jushuitan"
],
"module": "supply_chain",
"trigger": "scheduled",
"execution": {
@@ -2712,6 +2737,9 @@
},
{
"id": "supply.replenishment_alert.daily",
"required_accounts": [
"erp-jushuitan"
],
"module": "supply_chain",
"trigger": "scheduled",
"execution": {
@@ -2811,6 +2839,9 @@
},
{
"id": "product.market_rank",
"required_accounts": [
"douyin-shop"
],
"module": "product_commerce",
"trigger": "scheduled",
"execution": {
+336
View File
@@ -0,0 +1,336 @@
# -*- coding: utf-8 -*-
"""控制台与调度器的进程外保活(登录自启动,随用户会话常驻)。
职责(不承载任何业务调度规则,业务调度仍由 `gyxx schedule run` 唯一负责):
1. 控制台:探测 http://127.0.0.1:8765/api/health——进程不在则拉起;
进程在但探测连续失败(wedged)则结束该进程树后拉起。控制台进程内
另有看门狗(console.py `_console_watchdog`)自检失败会主动退出,此处
负责把退出后的空窗补上,两层互为补充。
2. 调度器:读 `var/state/scheduler/service.lock` 的 PID 判断存活,锁主
已死则按 install.ps1 同款参数拉起 `python -m gyxx_flow schedule run`;
调度器自身单实例锁保证不会双跑。
3. 节流:每个目标有冷却间隔;短时间反复重启触发熔断,只记录不再拉起,
避免故障风暴。
用法:
gyxx_keepalive.py # 常驻循环(Startup 的 vbs 以 pythonw 启动)
gyxx_keepalive.py --once # 单轮探测+处置后退出,供验收/手动巡检
日志:var/logs/keepalive/keepalive.log(追加)。
自身单实例:var/state/keepalive.lock(PID 存活检测,重复启动直接退出)。
"""
from __future__ import annotations
import argparse
import ctypes
import http.client
import json
import os
import subprocess
import sys
import time
from datetime import datetime, timezone
from pathlib import Path
PROJECT_ROOT = Path(__file__).resolve().parents[2]
ENV_FILES = [
Path(os.getenv("GYXX_RUNTIME_ENV_FILE", r"D:\product-collector-analyze-flow\.env")),
Path(
os.getenv(
"GYXX_RUNTIME_JUSHUITAN_ENV_FILE",
r"D:\product-collector-analyze-flow\private-secrets\jushuitan.env",
)
),
]
CONSOLE_HOST = "127.0.0.1"
CONSOLE_PORT = int(os.getenv("GYXX_CONSOLE_PORT", "8765"))
LOOP_INTERVAL_SECONDS = float(os.getenv("GYXX_KEEPALIVE_INTERVAL_SECONDS", "30"))
PROBE_TIMEOUT_SECONDS = 5.0
PROBE_FAILURE_LIMIT = 3
RESTART_COOLDOWN_SECONDS = float(os.getenv("GYXX_KEEPALIVE_COOLDOWN_SECONDS", "120"))
TRIP_MAX_RESTARTS = 5
TRIP_WINDOW_SECONDS = 30 * 60
TRIP_PAUSE_SECONDS = 60 * 60
DATA_ROOT = PROJECT_ROOT / "var"
LOG_PATH = DATA_ROOT / "logs" / "keepalive" / "keepalive.log"
LOCK_PATH = DATA_ROOT / "state" / "keepalive.lock"
CONSOLE_LOG_DIR = DATA_ROOT / "logs" / "console-restarts"
SCHEDULER_LOG_DIR = DATA_ROOT / "logs" / "scheduler-service"
SCHEDULER_SERVICE_LOCK = DATA_ROOT / "state" / "scheduler" / "service.lock"
def _log(event: str, detail: str = "") -> None:
stamp = datetime.now(timezone.utc).astimezone().strftime("%Y-%m-%d %H:%M:%S")
line = f"{stamp} {event} {detail}".rstrip()
LOG_PATH.parent.mkdir(parents=True, exist_ok=True)
with LOG_PATH.open("a", encoding="utf-8", newline="\n") as stream:
stream.write(line + "\n")
print(line, flush=True)
def _pid_alive(pid: int) -> bool:
if pid <= 0:
return False
kernel32 = ctypes.windll.kernel32
handle = kernel32.OpenProcess(0x1000, False, pid) # PROCESS_QUERY_LIMITED_INFORMATION
if not handle:
return False
try:
exit_code = ctypes.c_ulong()
if kernel32.GetExitCodeProcess(handle, ctypes.byref(exit_code)):
return exit_code.value == 259 # STILL_ACTIVE
return False
finally:
kernel32.CloseHandle(handle)
def _acquire_self_lock() -> bool:
LOCK_PATH.parent.mkdir(parents=True, exist_ok=True)
if LOCK_PATH.is_file():
try:
existing = json.loads(LOCK_PATH.read_text(encoding="utf-8"))
if _pid_alive(int(existing.get("pid", 0))):
return False
except Exception:
pass
LOCK_PATH.write_text(
json.dumps({"pid": os.getpid(), "started_at": datetime.now(timezone.utc).isoformat()}),
encoding="utf-8",
)
return True
def _console_env_files() -> list[Path]:
return [path for path in ENV_FILES if path.is_file()]
def _spawn(argv: list[str], log_dir: Path, log_prefix: str) -> int:
log_dir.mkdir(parents=True, exist_ok=True)
stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
log_path = log_dir / f"{log_prefix}-{stamp}.log"
kwargs: dict[str, object] = {}
if os.name == "nt":
kwargs["creationflags"] = (
subprocess.CREATE_NO_WINDOW | subprocess.CREATE_NEW_PROCESS_GROUP
)
else:
kwargs["start_new_session"] = True
with log_path.open("a", encoding="utf-8", newline="\n") as stream:
process = subprocess.Popen(
argv,
cwd=str(PROJECT_ROOT),
env=os.environ,
stdin=subprocess.DEVNULL,
stdout=stream,
stderr=subprocess.STDOUT,
shell=False,
**kwargs,
)
return process.pid
def _console_argv() -> list[str]:
argv = [
str(PROJECT_ROOT / ".venv" / "Scripts" / "python.exe"),
"-m",
"gyxx_flow",
"console",
"--host",
CONSOLE_HOST,
"--port",
str(CONSOLE_PORT),
]
for env_file in _console_env_files():
argv.extend(("--env-file", str(env_file)))
return argv
def _scheduler_argv() -> list[str]:
argv = [
str(PROJECT_ROOT / ".venv" / "Scripts" / "python.exe"),
"-m",
"gyxx_flow",
"schedule",
"run",
]
for env_file in _console_env_files():
argv.extend(("--env-file", str(env_file)))
return argv
def _probe_console() -> bool:
try:
connection = http.client.HTTPConnection(CONSOLE_HOST, CONSOLE_PORT, timeout=PROBE_TIMEOUT_SECONDS)
try:
connection.request("GET", "/api/health")
response = connection.getresponse()
response.read()
return response.status == 200
finally:
connection.close()
except Exception:
return False
def _port_owner_pid(port: int) -> int | None:
try:
output = subprocess.run(
["netstat", "-ano", "-p", "tcp"],
capture_output=True,
text=True,
timeout=10,
).stdout
except Exception:
return None
needle = f"{CONSOLE_HOST}:{port}"
for line in output.splitlines():
parts = line.split()
if (
len(parts) >= 5
and parts[0].upper() == "TCP"
and parts[1] == needle
and parts[3].upper() == "LISTENING"
):
try:
return int(parts[4])
except ValueError:
return None
return None
def _kill_pid_tree(pid: int) -> None:
if pid <= 0 or pid == os.getpid():
return
subprocess.run(
["taskkill", "/PID", str(pid), "/T", "/F"],
capture_output=True,
timeout=30,
)
def _scheduler_lock_pid() -> int | None:
try:
payload = json.loads(SCHEDULER_SERVICE_LOCK.read_text(encoding="utf-8"))
return int(payload.get("pid", 0)) or None
except Exception:
return None
class _TargetState:
"""记录单目标的连续失败计数、冷却与熔断状态。"""
def __init__(self, name: str) -> None:
self.name = name
self.failures = 0
self.last_restart_at = 0.0
self.recent_restarts: list[float] = []
self.tripped_at: float | None = None
def tripped(self, now: float) -> bool:
if self.tripped_at is None:
return False
if now - self.tripped_at >= TRIP_PAUSE_SECONDS:
self.tripped_at = None
self.recent_restarts.clear()
_log("keepalive-trip-reset", f"target={self.name} 熔断窗口结束恢复守护")
return False
return True
def may_restart(self, now: float) -> bool:
if self.tripped(now):
return False
if now - self.last_restart_at < RESTART_COOLDOWN_SECONDS:
return False
self.recent_restarts = [
stamp for stamp in self.recent_restarts if now - stamp < TRIP_WINDOW_SECONDS
]
if len(self.recent_restarts) >= TRIP_MAX_RESTARTS:
self.tripped_at = now
_log(
"keepalive-tripped",
f"target={self.name} {TRIP_MAX_RESTARTS} 次重启/{TRIP_WINDOW_SECONDS // 60}分钟 "
f"暂停拉起{TRIP_PAUSE_SECONDS // 60}分钟,请人工检查日志",
)
return False
return True
def record_restart(self, now: float) -> None:
self.last_restart_at = now
self.recent_restarts.append(now)
def run_cycle(console: _TargetState, scheduler: _TargetState) -> None:
now = time.time()
# ---- 控制台 ----
if _probe_console():
console.failures = 0
else:
console.failures += 1
owner = _port_owner_pid(CONSOLE_PORT)
_log(
"console-probe-failed",
f"{console.failures}/{PROBE_FAILURE_LIMIT} port_owner_pid={owner}",
)
if console.failures >= PROBE_FAILURE_LIMIT and console.may_restart(now):
if owner is not None and _pid_alive(owner):
_log("console-kill-wedged", f"pid={owner} 探测持续失败,结束进程树")
_kill_pid_tree(owner)
time.sleep(3)
pid = _spawn(_console_argv(), CONSOLE_LOG_DIR, "keepalive-console")
console.record_restart(time.time())
console.failures = 0
_log("console-restarted", f"pid={pid}")
# ---- 调度器 ----
scheduler_pid = _scheduler_lock_pid()
scheduler_alive = scheduler_pid is not None and _pid_alive(scheduler_pid)
if not scheduler_alive and scheduler.may_restart(now):
pid = _spawn(_scheduler_argv(), SCHEDULER_LOG_DIR, "keepalive-start")
scheduler.record_restart(time.time())
_log(
"scheduler-restarted",
f"pid={pid} prev_lock_pid={scheduler_pid}",
)
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description="GYXX Flow 控制台/调度器保活")
parser.add_argument("--once", action="store_true", help="单轮探测+处置后退出")
arguments = parser.parse_args(argv)
if not _acquire_self_lock():
_log("keepalive-duplicate-exit", "已有保活进程在运行")
return 0
console = _TargetState("console")
scheduler = _TargetState("scheduler")
missing_env = [str(path) for path in ENV_FILES if not path.is_file()]
_log(
"keepalive-start",
f"pid={os.getpid()} once={arguments.once}"
+ (f" missing_env={missing_env}" if missing_env else ""),
)
try:
while True:
run_cycle(console, scheduler)
if arguments.once:
_log("keepalive-once-done", "")
return 0
time.sleep(LOOP_INTERVAL_SECONDS)
except KeyboardInterrupt:
return 0
finally:
try:
if LOCK_PATH.is_file():
payload = json.loads(LOCK_PATH.read_text(encoding="utf-8"))
if int(payload.get("pid", 0)) == os.getpid():
LOCK_PATH.unlink()
except Exception:
pass
if __name__ == "__main__":
sys.exit(main())
+5
View File
@@ -0,0 +1,5 @@
' GYXX Flow 保活登录自启动:无窗口启动 deploy/watchdog/gyxx_keepalive.py,
' 由它负责拉起控制台(含进程内自检看门狗)与调度器。
' 安装:复制到 %APPDATA%\Microsoft\Windows\Start Menu\Programs\Startup\
CreateObject("WScript.Shell").Run _
"""D:\gyxx-flow\.venv\Scripts\pythonw.exe"" ""D:\gyxx-flow\deploy\watchdog\gyxx_keepalive.py""", 0, False
+12 -1
View File
@@ -4,6 +4,8 @@
param(
[string]$ProjectRoot = (Resolve-Path (Join-Path $PSScriptRoot "..\..")).Path,
[string]$DataRoot = "",
[string]$RuntimeEnvFile = "D:\product-collector-analyze-flow\.env",
[string]$JushuitanEnvFile = "D:\product-collector-analyze-flow\private-secrets\jushuitan.env",
[string]$ServiceName = "gyxx-flow-scheduler",
[string]$NssmPath = "nssm.exe"
)
@@ -12,6 +14,13 @@ $ErrorActionPreference = "Stop"
Set-StrictMode -Version Latest
$project = (Resolve-Path -LiteralPath $ProjectRoot).Path
$runtimeEnvPath = (Resolve-Path -LiteralPath $RuntimeEnvFile -ErrorAction Stop).Path
$runtimeEnvPaths = @($runtimeEnvPath)
if (Test-Path -LiteralPath $JushuitanEnvFile -PathType Leaf) {
$runtimeEnvPaths += (Resolve-Path -LiteralPath $JushuitanEnvFile -ErrorAction Stop).Path
} else {
Write-Warning "Jushuitan environment file not found; Jushuitan workflows will fail safely until it is provisioned."
}
if ([string]::IsNullOrWhiteSpace($DataRoot)) {
$DataRoot = Join-Path $project "var"
}
@@ -33,7 +42,9 @@ function Invoke-Nssm([string[]]$Arguments) {
Invoke-Nssm -Arguments @("install", $ServiceName, $python)
Invoke-Nssm -Arguments @("set", $ServiceName, "AppDirectory", $project)
Invoke-Nssm -Arguments @("set", $ServiceName, "AppParameters", "-m gyxx_flow schedule run")
$envFileArguments = $runtimeEnvPaths | ForEach-Object { '--env-file "' + $_ + '"' }
$schedulerArguments = '-m gyxx_flow schedule run ' + ($envFileArguments -join ' ')
Invoke-Nssm -Arguments @("set", $ServiceName, "AppParameters", $schedulerArguments)
Invoke-Nssm -Arguments @(
"set", $ServiceName, "AppEnvironmentExtra",
"GYXX_PROJECT_ROOT=$project", "GYXX_DATA_ROOT=$data", "PYTHONUTF8=1"
+30
View File
@@ -46,6 +46,27 @@ Scrapling 安装和管理。具体边界与验证命令见
`EnvironmentFile=` 或当前进程环境优先于文件中的同名值,避免本地文件意外覆盖密钥系统
注入值。
## 香港安沛 SKU 公有库存同步
`product.hk_anpei_public_inventory.daily` 每天按 `Asia/Shanghai` 09:00 查询计划书中的固定 SKU 清单,
调用聚水潭 `/open/inventory/query`,并按商品编码(光影)更新飞书电子表格「現有庫存」工作表的
「仓库现有库存」列。它要求调度进程环境提供
`JUSHUITAN_APP_KEY` 和 `JUSHUITAN_APP_SECRET`;`JUSHUITAN_BASE_URL` 可选,默认使用聚水潭官方 OpenAPI
地址。没有密钥时工作流会在查询前失败,不会把库存留成 0 或改写多维表格。
Windows NSSM 调度服务从受保护的运行时环境文件读取凭据。安装器默认读取通用运行环境文件
`D:\product-collector-analyze-flow\.env`,以及独立的聚水潭密钥文件
`D:\product-collector-analyze-flow\private-secrets\jushuitan.env`。两个文件都按服务账户实际身份限制 ACL;
可分别通过 `-RuntimeEnvFile` 和 `-JushuitanEnvFile` 指定其他路径:
```powershell
.\deploy\windows-service\install.ps1 -ProjectRoot D:\gyxx-flow -DataRoot D:\gyxx-flow-data -RuntimeEnvFile D:\product-collector-analyze-flow\.env -JushuitanEnvFile D:\product-collector-analyze-flow\private-secrets\jushuitan.env
```
仅将这两个变量的正式聚水潭应用值写入服务账户可读的环境文件,并限制文件 ACL;不要使用开发配置中的默认值,
也不要把真实值写进 Git、PowerShell 命令参数、服务日志或文档。已安装的服务需要让调度进程重新读取环境文件后,
新计划才会使用这些值。Linux 部署则将相同变量放入权限为 `0640` 的 `EnvironmentFile`。
## 三平台电商费用日报部署边界
`product.ecommerce_costs.daily` 是包含天猫万相台、京东京准通和抖音千川分支的电商费用工作流;三个
@@ -197,3 +218,12 @@ sudo systemctl start gyxx-flow.service
## Windows 兼容入口
`deploy/windows-service/` 暂时保留为 legacy NSSM 兼容入口。NSSM 只守护同一个 `gyxx schedule run` 进程,不注册 Windows Task Scheduler,也不保存业务时间规则。新服务器部署以 Linux systemd 为准。
## Windows 工作站保活(控制台 + 调度器自检自启动)
交互式 Windows 工作站(无法用 systemd/NSSM 常驻服务时)使用 `deploy/watchdog/gyxx_keepalive.py` 做进程外保活:
- **控制台自检**:控制台进程内置看门狗线程(`console.py` `_console_watchdog`),周期探测自身 `GET /api/health`;连续失败(默认 3 次,`GYXX_CONSOLE_WATCHDOG_FAILURES` / `GYXX_CONSOLE_WATCHDOG_INTERVAL_SECONDS` 可调,`GYXX_CONSOLE_WATCHDOG=0` 关闭)即以退出码 70 自杀,交给保活拉起。`/api/health` 不鉴权、只报进程 PID、uptime 与调度锁存活,供探针使用;API 500 现在会把完整堆栈落到 stderr(随启动重定向进 `var/logs/console-restarts/`)。
- **保活脚本**:登录自启动(Startup 目录 `gyxx-keepalive.vbs` → `pythonw gyxx_keepalive.py`)后常驻,每 30 秒一轮:控制台探测连续失败或端口空置则结束僵死进程树并按 `--env-file` 同款参数拉起;读 `var/state/scheduler/service.lock` 判定调度器存活,锁主死亡即拉起 `gyxx schedule run`(调度器单实例锁保证不双跑)。单目标冷却 120 秒;30 分钟内 5 次重启触发熔断,暂停拉起 1 小时并记日志待人工检查。日志在 `var/logs/keepalive/keepalive.log`。
- **验收**:`python deploy/watchdog/gyxx_keepalive.py --once` 单轮探测+处置;健康态应只输出 `keepalive-start`/`keepalive-once-done`。
- 该脚本只做进程可用性守护,不承载任何业务时间规则(唯一调度路径仍是 `gyxx schedule run`);未注册 Windows Task Scheduler。注意:Startup 方式要求用户已登录——机器重启后需人工登录一次,之后控制台与调度器会在约 2 分钟内自动就位。
+24 -2
View File
@@ -190,16 +190,38 @@ uv run gyxx schedule run --env-file /etc/gyxx-flow/gyxx-flow.env
- `tmall-ozko`:商品经营日报 ozko 账号,900 秒;
- `xiaohongshu-content`:小红书,900 秒;
- `douyin-content`:抖音内容侧,900 秒;
- `xingtu`:星图,900 秒;
- `chanmama`:蝉妈妈,900 秒;
- `chanmama`:蝉妈妈,900 秒(脚本自带账密登录);
- `bilibili-content`:B 站,900 秒。
星图/百应采集(`xingtu_scraper_v2.py`)的登录态挂在 `douyin-shop` 账号下;`xingtu` 账号当前无成员脚本,仅保留 vault。
商品经营日报 `product_commerce:taobao_sycm_collect.py` 会在同一次运行中按品牌路由账号:光影行星使用 `tmall-shop`,ozko 使用 `tmall-ozko`,两边各自使用独立 Profile 和账号 vault。`tmall-ozko` 没有静态成员脚本是有意设计,因为它由这个按品牌拆分的包装工作流动态选择;登录态成功后仍会回写 `state/accounts/tmall-ozko/`,由同一套 keepalive 负责续期。
账号需要在 `config/runtime-bindings.json` 的 `accounts.<account_id>.keepalive` 中显式开启续期,并配置一个不会产生业务副作用的已登录页面 URL。所有账号由唯一的 `gyxx schedule run` 常驻进程在后台维护;每个账号使用独立 CDP 端口、独立 Profile 和账号级 Cookie vault,启动后按 `initial_delay_seconds` 错峰访问,避免同时打开多个登录页面。访问成功且没有跳转到登录页时,Scrapling 才会把最新 Cookie/storage state 原子写回 `state/accounts/<account_id>/`,并同步给成员脚本;不需要另建 Windows 任务计划、cron 或 timer。
Cookie 续期只能延长站点支持滑动续期的会话,不能突破服务端硬过期、主动退出、风控或验证码。keepalive 检测到登录页/无效 Cookie 时会停止发布,不会用失效状态覆盖最后一份有效 vault;此时按 `gyxx accounts list` 检查 `keepalive.status`,必要时重新执行 `gyxx accounts login <account_id>`。
### 工作流执行前的 Cookie 健康网关
每个正式执行(非 dry-run)的工作流在采集步骤前会自动注入"每账号一个"的 `cookie-gate.<account_id>` 关键步骤,由 `src/gyxx_flow/cookie_gate.py` 实现按账号分三级的状态机:
- **blocking**(纯扫码/验证码账号:`douyin-shop`、`xiaohongshu-content`、`douyin-content`、`bilibili-content` 等):先看 keepalive 近期成功状态(默认 1 小时内视为新鲜,可用 `GYXX_COOKIE_GATE_MAX_STATUS_AGE_SECONDS` 调整),新鲜且 vault 静态有效则直接放行;否则用 keepalive 同款无头浏览器真实探活。探活失败即判定过期:gate 步骤失败,依赖它的采集步骤记 `dependency-failed` 跳过、工作流失败退出,并通过飞书通知登录负责人(`config/notification-routing.json` 中手机号为 `+8613663785145` 的人员,默认 30 分钟节流,可用 `GYXX_LOGIN_ALERT_OPEN_IDS` 覆盖收件人)执行 `uv run gyxx accounts login <account_id>` 完成扫码/验证码重登。
- **recover**(配置了 `login_command_id` 的账密自动重登账号:`erp-jushuitan`):过期时先自动跑账密登录恢复,恢复成功放行;失败才阻断并飞书通知。
- **defer**(成员脚本自带账号密码登录的账号:京东系、天猫系、`chanmama`):不注入 gate,保持脚本内账密自动登录流程不变。
按账号划分的持久状态在 `state/accounts/<id>/cookie_state.json`(`healthy` ↔ `awaiting_login`)。编排器型工作流(入口脚本本身无账号绑定、由子采集脚本使用账号)在 `config/workflows.json` 的 `required_accounts` 字段声明依赖账号。`--rerun-step`/`--resume-from` 切片和 shadow 运行不发送通知;dry-run 完全不注入 gate。设置 `GYXX_COOKIE_GATE=0` 可临时关闭整网关。
采集脚本不再内嵌扫码/验证码登录:运行中检测到登录态失效会以 `CookieExpiredError` 快速失败并按退出码 75 记为 cookie 跳过;`self_xhs_scraper.py --login-only` 是唯一保留的人工登录引导(该脚本不在任何账号成员内,自管绑定状态)。
### 登录恢复后的自动重跑
gate 阻断工作流时会把待重跑清单(工作流 + 业务日期,shadow/验收运行不记录,同工作流同日期去重,上限 20 条)写进 `state/accounts/<id>/cookie_state.json` 的 `pending_reruns`。常驻调度器(`gyxx schedule run`)的维护循环里挂着 `CookieGateRecoveryManager`:
- 每个心跳检查有待重跑记录的账号:keepalive 近期成功 + vault 静态有效 → 立即视为恢复;否则每 5 分钟做一次真实探活(mode C 账号会自动账密重登,重登成功同样触发恢复)。
- 恢复确认后,用调度器同款启动器按**原业务日期**自动重跑全部待重跑工作流(日志在 `data/logs/scheduler/`),清空清单并飞书通知"登录态已恢复 + 已自动重跑清单"。
- 启动失败的任务保留在清单里下一轮重试;同一账号两次重跑启动之间有 5 分钟冷却;控制台手动脚本类临时运行 id(`script.*`)不会自动重跑。
- 与正常调度并发安全:工作流级锁 + EffectLedger 幂等保证同一天不会重复产生副作用。
项目不会自动搬动或删除已有 `var/`。迁移数据根时应停止调度器,完整备份和复制原目录,修改环境变量后依次运行 doctor、dry-run 和一次指定日期的对账执行。
## 清理和保留
+20
View File
@@ -17,6 +17,15 @@ uv run gyxx console
uv run gyxx console --env-file D:\secure\gyxx-flow.env
```
香港客服库存工作流还需要聚水潭应用密钥。Windows 生产控制台必须额外加载受保护的聚水潭环境文件,
这样控制台发起的手动执行也能取得凭据;调度器和控制台应加载相同的环境文件集合:
```powershell
uv run gyxx console `
--env-file D:\product-collector-analyze-flow\.env `
--env-file D:\product-collector-analyze-flow\private-secrets\jushuitan.env
```
商品经营正式执行会在启动子进程前校验云端 PostgreSQL 和该工作流实际需要的运行时配置。
需要 Hermes 分析或业务通知的工作流仍会校验 `hermes-analyzer` lark-cli user 身份和负责人
通知路由;天猫/京东视频上传则校验 `STYLE_ANALYSIS_LLM_BASE_URL`(或
@@ -144,3 +153,14 @@ uv run gyxx console --host 0.0.0.0 --port 8765
控制台不提供工作流入口、参数、运行时绑定或凭据的在线编辑能力;这些仍由受版本控制的
项目配置和代码维护。
## 自检与保活
- `GET /api/health` 返回进程 PID、uptime 与调度器锁存活状态,不鉴权、不触外部依赖,专供探针使用。
- 控制台进程内置看门狗线程:周期自探测 `/api/health`,连续失败(默认 3 次)即以退出码 70 退出,
由 `deploy/watchdog/gyxx_keepalive.py`(登录自启动常驻)负责拉起;保活同时守护调度器进程,
详见 `docs/deployment.md` 的「Windows 工作站保活」章节。环境开关:
`GYXX_CONSOLE_WATCHDOG=0` 关闭、`GYXX_CONSOLE_WATCHDOG_INTERVAL_SECONDS`(默认 30)、
`GYXX_CONSOLE_WATCHDOG_FAILURES`(默认 3)。
- API 返回「控制台处理请求失败」时,完整堆栈在控制台 stderr 日志
(`var/logs/console-restarts/console-*.stderr.log`)中可查。
+2 -3
View File
@@ -69,7 +69,7 @@
| `08:30` | 2 | 1 | 1 | 达人月报生成 [`content.creator_report.monthly`](暂停);京东与天猫主图周采集 [`product.main_image.weekly`] |
| `08:40` | 1 | 1 | 0 | 商品经营日报采集 [`product.daily`] |
| `09:00` | 6 | 6 | 0 | 笔记清单与合作同步 [`content.notes_master.daily`];京东自营业绩日采集 [`shop.jd_self_operated.daily`];抖音待改价申诉;聚水潭全店铺款式日报 [`product.erp_all_shop_daily`];天猫百亿补贴批量报名;补货建议预计到货同步 [`supply.replenishment_arrival_sync.daily`] |
| `10:00` | 8 | 5 | 3 | 营销日报生成与发送 [`content.marketing_report.daily`](暂停);内容平台登录态刷新 [`content.relogin.weekly`](暂停);内容周度汇总 [`content.summary.weekly`](暂停);抖音待改价申诉;三平台电商费用采集 [`product.ecommerce_costs.daily`];商品月度销量表 [`product.sales_sheet.daily`];天猫百亿补贴批量报名;市场排行周采集 [`product.market_rank`] |
| `10:00` | 8 | 5 | 3 | 营销日报生成与发送 [`content.marketing_report.daily`](暂停);内容周度汇总 [`content.summary.weekly`](暂停);抖音待改价申诉;三平台电商费用采集 [`product.ecommerce_costs.daily`];商品月度销量表 [`product.sales_sheet.daily`];天猫百亿补贴批量报名;市场排行周采集 [`product.market_rank`] |
| `11:00` | 3 | 3 | 0 | 抖音待改价申诉;款式周期分析 [`product.style_analysis.interval`];天猫百亿补贴批量报名 |
| `12:00` | 5 | 5 | 0 | 多平台评论周采集 [`content.comments.weekly`];店铺经营指标周采集 [`shop.metrics.weekly`];抖音待改价申诉;商品人群画像日采集 [`product.persona.daily`];天猫百亿补贴批量报名 |
| `12:30` | 1 | 1 | 0 | 竞店指标周采集 [`shop.competitor.weekly`] |
@@ -98,7 +98,7 @@
- 指标采集:`content.metrics.daily`(内容指标日报采集)、`content.metrics.backfill`(内容指标补采)。
- 报告与汇总:`content.marketing_report.daily`(营销日报生成与发送)、`content.creator_report.monthly`(达人月报生成)、`content.summary.monthly`(内容月度汇总)、`content.summary.weekly`(内容周度汇总)。
- 内容同步与互动:`content.notes_master.daily`(笔记清单与合作同步)、`content.comments.weekly`(多平台评论周采集)、`content.relogin.weekly`(内容平台登录态刷新)。
- 内容同步与互动:`content.notes_master.daily`(笔记清单与合作同步)、`content.comments.weekly`(多平台评论周采集)。
### 4.2 商品经营(`product_commerce`,12 条)
@@ -124,7 +124,6 @@
| 1 | `content.metrics.daily` | 内容指标日报采集 | 内容营销 | 每天 `01:00`、`05:00` | 0 | 启用 | 5 |
| 2 | `content.metrics.backfill` | 内容指标补采 | 内容营销 | 每周一 `14:00` | 0 | 暂停 | 2 |
| 3 | `content.marketing_report.daily` | 营销日报生成与发送 | 内容营销 | 每天 `10:00` | 0 | 暂停 | 1 |
| 4 | `content.relogin.weekly` | 内容平台登录态刷新 | 内容营销 | 每周五 `10:00` | 0 | 暂停 | 1 |
| 5 | `content.creator_report.monthly` | 达人月报生成 | 内容营销 | 每月 1 日 `08:30` | 0 | 启用 | 1 |
| 6 | `content.summary.monthly` | 内容月度汇总 | 内容营销 | 每月 1 日 `08:00` | 0 | 启用 | 1 |
| 7 | `content.notes_master.daily` | 笔记清单与合作同步 | 内容营销 | 每天 `09:00` | 0 | 启用 | 1 |
+65
View File
@@ -0,0 +1,65 @@
# 工作流总览(未隐藏)
> 生成日期:2026-10-08。数据来源:`config/workflows.json`、`config/schedules.json`、控制台显示名(`var/state/console/workflow-display-names.json`);数据库表提取自各入口脚本及共享 DB 辅助模块的内联 SQL。
> 范围:控制台中未被隐藏的工作流共 **30 个**(全部 39 个中剔除浏览器端隐藏的 10 个,其中 `content.relogin.weekly` 已不在当前配置中)。
> 定时时间均为 **Asia/Shanghai**,由常驻调度器 `gyxx schedule run` 按 `config/schedules.json` 触发;标注「未启用」的仅支持控制台手动运行。
> 启动脚本为仓库根目录相对路径,由工作流引擎相对 `src/gyxx_flow/modules/<模块>/` 解析执行;`×N` 表示同一脚本被 N 个步骤以不同参数复用。
>
> **关联数据库表说明**:均指云端 PostgreSQL(`GYXX_POSTGRES_DSN`),(写)= INSERT/UPDATE/DELETE,(读)= SELECT,(建)= CREATE TABLE IF NOT EXISTS。三点公共约定:
> - 内容营销用到款式映射的脚本会经 `feishu_mapping` → `adapters/workflow_config.py` 间接读动态配置表 `workflow_dynamic_styles`、`workflow_dynamic_style_platforms`(并合并读 `cmt_styles`,首跑建 `workflow_dynamic_*` 表),下表不逐一列出。
> - 店铺情报各入库脚本启动时执行 `shop_intelligence/db/schema.sql`、商品电商多数入库脚本执行 `product_commerce/db/schema.sql`,幂等建该模块全量表(21 / 26 张);下表只列该工作流**实际读写**的表。
> - 读写 `information_schema` / `pg_catalog` 系统目录的通用校验不列出。
## 内容营销(content_marketing,4 个)
| 工作流名称 | ID | 功能 | 定时执行 | 启动脚本 | 关联数据库表 |
|---|---|---|---|---|---|
| 合作达人内容指标日报采集 | `content.metrics.daily` | 按日采集合作达人在 B站、小红书蒲公英、星图(未命中用抖店达人广场补采)的曝光互动指标并同步 PostgreSQL;05:00 档补采 01:00 未完成任务 | 每日 01:00、05:00 | `src/gyxx_flow/modules/content_marketing/run_all.py`<br>`src/gyxx_flow/modules/content_marketing/data/tools/sync_metrics_to_cmt_notes.py` | `cmt_note_metric_snapshots`(写)<br>`cmt_notes_master`(读写)<br>`cmt_creators`(读写)<br>`cmt_styles`(读) |
| 自营内容指标日报采集 | `content.metrics.self_operated.daily` | 刷新自营款式映射,采集自营 B站、蝉妈妈、小红书 ARK 指标;飞书曝光回填并将三平台全量内容指标按日快照入库 | 每日 06:00 | `…/content_marketing/data/tools/refresh_self_mapping.py`<br>`…/content_marketing/self_bilibili_scraper.py`<br>`…/content_marketing/chanmama_scraper.py`<br>`…/content_marketing/self_xhs_scraper.py`<br>`…/content_marketing/data/tools/sync_metrics_to_cmt_notes.py`<br>`…/content_marketing/data/tools/sync_self_video_metrics.py` | `cmt_self_video_metrics`(写)<br>`cmt_note_metric_snapshots`(写)<br>`cmt_notes_master`(读写)<br>`cmt_creators`(读写)<br>`cmt_styles`(读) |
| 笔记清单与合作同步 | `content.notes_master.daily` | 一次飞书拉取同时落笔记身份与合作字段;笔记数量按去重链接口径维护主表 | 每日 09:00 | `src/gyxx_flow/modules/content_marketing/data/tools/sync_notes_master.py` | `cmt_notes_master`(建+写)<br>`cmt_note_metric_snapshots`(建)<br>`cmt_styles`(读写)<br>`cmt_creators`(读写) |
| 多平台评论周采集 | `content.comments.weekly` | B站、小红书、抖音三平台评论并行采集,任一平台失败则工作流失败 | 每周日 12:00 | `…/content_marketing/data/tools/batch_rescrape_bilibili.py`<br>`…/content_marketing/data/tools/batch_rescrape_xiaohongshu.py`<br>`…/content_marketing/data/tools/batch_rescrape_douyin.py` | `cmt_comments`(写)<br>`cmt_notes_master`(读写) |
## 店铺情报(shop_intelligence,12 个)
| 工作流名称 | ID | 功能 | 定时执行 | 启动脚本 | 关联数据库表 |
|---|---|---|---|---|---|
| 店铺经营指标周采集 | `shop.metrics.weekly` | 三平台作为独立节点并行采集店铺经营指标并写 PostgreSQL(不写飞书 Base) | 每周一 12:00 | `src/gyxx_flow/modules/shop_intelligence/runners/run_shop.py` ×3 | `jd_shop_weekly`、`dy_shop_weekly`、`tm_shop_weekly`(各平台步骤分别写) |
| 竞店指标周采集 | `shop.competitor.weekly` | 京东、抖音并行采集竞店指标并入库,单平台失败不阻断另一平台 | 每周一 12:30 | `src/gyxx_flow/modules/shop_intelligence/runners/run_peer_store.py` ×2 | `jd_peer_store_ranking`、`dy_peer_store_ranking`(各平台步骤分别写) |
| 京东自营业绩日采集 | `shop.jd_self_operated.daily` | 采集京东自营品牌与商品业绩(业务日期 = 运行日 −2) | 每日 09:00 | `…/shop_intelligence/collectors/jd_self_operated_brand_daily.py`<br>`…/shop_intelligence/collectors/jd_self_operated_product_daily.py` | `jd_self_operated_brand_daily`(写)<br>`jd_self_operated_product_daily`(写) |
| 京东客服消息采集 | `shop.jd_customer_service.daily` | 按业务日在商智消息视图采集顾客会话,单日须完成全部顾客,失败保留进度可重试 | 每月 21 日 03:00 | `src/gyxx_flow/modules/shop_intelligence/collectors/jd_customer_service_messages.py` | `jd_customer_service_conversations`、`jd_customer_service_messages`(写) |
| 抖音客服消息采集 | `shop.douyin_customer_service.daily` | 按业务日采集抖音客服会话,失败保留进度可安全重试 | 每月 21 日 03:20 | `src/gyxx_flow/modules/shop_intelligence/collectors/dy_customer_service_messages.py` | `dy_customer_service_conversations`、`dy_customer_service_messages`、`dy_customer_service_conversation_orders`(写)<br>`dy_customer_service_products`(读写) |
| 京东差评采集 | `shop.jd_negative_reviews.daily` | 按店铺选择日期范围逐日采集京东负面评价,按评价 ID 幂等 | 每月 22 日 03:00 | `src/gyxx_flow/modules/shop_intelligence/collectors/jd_negative_reviews.py` | `jd_negative_reviews`(写) |
| 抖音差评采集 | `shop.dy_negative_reviews.daily` | 逐日采集抖音负面评价(评价历史仅保留近 30 天),按订单 ID 幂等 | 每月 22 日 03:20 | `src/gyxx_flow/modules/shop_intelligence/collectors/dy_negative_reviews.py` | `dy_negative_reviews`(写) |
| 京东品退问题采集 | `shop.jd_after_sales_issues.daily` | 在商智 31 天限制下按连续区间采集售后服务单,按申请日期写业务日,幂等更新 | 每月 23 日 03:00 | `src/gyxx_flow/modules/shop_intelligence/collectors/jd_after_sales_issues.py` | `jd_after_sales_issues`(写,含范围外残留 DELETE 清理) |
| 抖音品退采集 | `shop.douyin_after_sales_quality.daily` | 采集抖音售后单,按申请时间归属业务日,账号 + 售后编号幂等更新 | 每月 23 日 03:20 | `src/gyxx_flow/modules/shop_intelligence/collectors/dy_after_sales_quality.py` | `dy_after_sales_quality_orders`、`dy_after_sales_quality_negotiations`(写) |
| 天猫客服消息采集 | `shop.tmall_customer_service.daily` | 按每次最多 31 天的窗口顺序采集天猫客服聊天记录,失败窗口保留数据 | 每月 24 日 03:40(未启用,仅手动) | `src/gyxx_flow/modules/shop_intelligence/collectors/tmall_customer_service_messages.py` | `tmall_customer_service_conversations`、`tmall_customer_service_messages`(写) |
| 天猫差评采集 | `shop.tmall_negative_reviews.daily` | 固定筛选负面评价并按评价时间范围采集后入库 | 每月 25 日 03:50(未启用,仅手动) | `src/gyxx_flow/modules/shop_intelligence/collectors/tmall_negative_reviews.py` | `tmall_negative_reviews`(写) |
| 抖音待改价申诉 | `shop.douyin_price_appeal` | 每个时点重新扫描可申诉并发起价格申诉;已申诉商品不重复出现 | 每日每小时整点(00:00–23:00) | `src/gyxx_flow/modules/shop_intelligence/collectors/dy_store_competitor_store_scraping.py` | 无(浏览器采集 + 本地 JSON) |
## 商品电商(product_commerce,12 个)
| 工作流名称 | ID | 功能 | 定时执行 | 启动脚本 | 关联数据库表 |
|---|---|---|---|---|---|
| 商品人群画像周采集 | `product.persona.daily` | 天猫、抖音、京东人群画像并行采集并写入对应飞书多维表 | 每周二 12:00 | `src/gyxx_flow/modules/product_commerce/run_daily_persona.py` | `daily_persona_metrics`(写) |
| 商品经营日报采集 | `product.daily` | 按业务日期(运行日 −1)采集 ERP 与三平台数据,幂等写 PostgreSQL,汇总分析后导出并写飞书日报 | 每日 08:40 | `src/gyxx_flow/modules/product_commerce/orchestrate_daily_collection.py` | 写:`product_reviews`、`product_daily_metrics`、`daily_style_metrics`、`fact_style_analysis_reports`、`decline_events`、`fact_alert_history`<br>读:`weekly_style_metrics`、`daily_persona_metrics`、`main_image_creatives`、`cmt_styles`、`cmt_notes_master`、`cmt_comments`、`cmt_creators`(及上述写表) |
| 私域库存预警 | `product.online_stock_alert.daily` | 查询线上渠道全部商品,库存低于 3 时汇总 SKU 与库存发飞书预警 | 每日 09:30 | `src/gyxx_flow/modules/product_commerce/online_stock_alert.py` | 无(企业数据交互 HTTP API + 飞书) |
| 三平台电商费用采集 | `product.ecommerce_costs.daily` | 并行采集天猫万相台、京东、抖音千川业务日前一天的费用,各平台先下载再导入,单平台失败不阻断其他 | 每日 10:00 | `…/product_commerce/collect_tmall_wanxiang_ads.py`<br>`…/product_commerce/import_tmall_product_ads.py`<br>`…/product_commerce/collect_jd_ad_costs.py`<br>`…/product_commerce/import_jd_ad_costs.py`<br>`…/product_commerce/collect_douyin_qianchuan_ads.py`<br>`…/product_commerce/import_douyin_qianchuan_ads.py` | `product_daily_metrics`(写,天猫原始费用并入 raw_data JSONB)<br>`jd_product_ad_daily_metrics`、`dy_product_ad_daily_metrics`(建+写)<br>`dim_style`(读) |
| 京东与天猫主图周采集 | `product.main_image.weekly` | 京东与天猫万相台主图并行采集,各自完成 PG 入库和飞书插入,互不影响 | 每周日 08:30 | `…/product_commerce/run_weekly_jd_main_image.py`<br>`…/product_commerce/run_weekly_main_image.py` | `main_image_creatives`(写) |
| 飞书商品月度销量表填写 | `product.sales_sheet.daily` | 抓取并校验全部 ERP 报表后填写飞书月度销量表,同名月份页幂等覆盖,无 ERP 编码产品按 0 写入 | 每月 1 日 10:00 | `src/gyxx_flow/modules/product_commerce/sync_monthly_sales_sheet.py` | 无(聚水潭浏览器采集 + 飞书写入) |
| 聚水潭全店铺款式日报 | `product.erp_all_shop_daily` | 聚水潭商品主题分析全选全部店铺,采集业务日前一天所有 ERP 款式的自定义销量与退货量并幂等写 PostgreSQL | 每日 09:00 | `src/gyxx_flow/modules/product_commerce/backfill_erp_all_shop_daily.py` | `erp_all_shop_style_daily_metrics`(读写) |
| 客服香港库存工作流 | `product.hk_anpei_public_inventory.daily` | 校验计划书 SKU 与库存表商品编码对应关系,调聚水潭库存接口查询并将公有可用数写入仓库现有库存列 | 每日 09:00 | `src/gyxx_flow/modules/product_commerce/public_sku_inventory_sheet.py` | 无(聚水潭 API + 飞书表格) |
| 天猫视频上传 | `product.video_upload` | 按店铺规则扫描飞书视频附件,凭附件级发布账本防重,仅在存在可领取附件时登录光合账号上传 | 每 3 天 14:00 | `src/gyxx_flow/modules/product_commerce/upload_video_to_guanghe.py` | `fact_video_publications`、`fact_video_publication_attempts`(建+读写,附件级发布账本)<br>`dim_style`、`product_daily_metrics`(读) |
| 京东视频上传 | `product.jd_video_upload` | 从凌云air基线扫描到视图末尾识别新增视频,以 record_id + file_token 幂等发布到京东 | 每 3 天 14:00 | `src/gyxx_flow/modules/product_commerce/upload_video_to_jd.py` | `fact_video_publications`、`fact_video_publication_attempts`(建+读写,发布账本)<br>`dim_style`、`product_daily_metrics`(读) |
| 天猫百亿补贴批量报名 | `product.tmall_baibu_apply` | 每次从飞书下载报名模板逐入口提交;默认仅导入 3 个不带超链入口(含老天猫全渠道 3C 后置完善草稿),可切换全部 6 个入口 | 每日每小时整点(00:00–23:00) | `…/product_commerce/tmall_baibu_apply_old_chain_bag.py`<br>`…/product_commerce/tmall_baibu_apply_old_all_bag.py`<br>`…/product_commerce/tmall_baibu_apply_old_all_3c.py`<br>`…/product_commerce/tmall_baibu_apply_old_chain_3c.py`<br>`…/product_commerce/tmall_baibu_apply_new_chain_bag.py`<br>`…/product_commerce/tmall_baibu_apply_new_all_bag.py` | 无(浏览器操作 + 飞书) |
| 三平台市场排行采集(用于行军图) | `product.market_rank` | 并行采集天猫、京东、抖音市场排行,按月份写入品牌行军图数据库并复用已有人工分类 | 每月 1 日 10:00 | `src/gyxx_flow/modules/product_commerce/orchestrate_market_rank_collection.py` | `market_product`、`market_product_classification`(读写)<br>`fact_market_product_snapshot`(写,重跑先 DELETE)<br>`fact_market_rank_reports`、`fact_market_rank_notifications`(建+读写)<br>`dim_market_style`、`dim_market_scene`(读) |
## 供应链(supply_chain,2 个)
| 工作流名称 | ID | 功能 | 定时执行 | 启动脚本 | 关联数据库表 |
|---|---|---|---|---|---|
| 补货预警日处理 | `supply.replenishment_alert.daily` | 依据配置 SKU 与库存阈值生成预警并写 PostgreSQL,仅存在预警时发送业务通知 | 每日 07:00 | `src/gyxx_flow/modules/supply_chain/run.py` | `fact_threshold_alert`(读写)<br>`dim_product`(写)<br>`workflow_runs`(读写,运行账本)<br>`bitable_insert_log`(读写,通知去重账本) |
| 同步聚水潭到货时间 | `supply.replenishment_arrival_sync.daily` | 读取到货同步待处理队列,按商品编码在聚水潭采购单页面更新预计到货时间并回写状态,失败行下次自动重试 | 每日 09:00 | `src/gyxx_flow/modules/supply_chain/orchestrator/scripts/collect_replenishment_arrival_sync.py` | `erp_replenishment_arrival_sync`(读写,队列表由 ERP 应用写入) |
## 附:已隐藏的工作流(未收录)
浏览器端隐藏的 10 个 ID:`content.creator_report.monthly`、`content.marketing_report.daily`、`content.metrics.backfill`、`content.relogin.weekly`(配置中已不存在)、`content.summary.monthly`、`content.summary.weekly`、`product.alert.daily`、`product.style_analysis.interval`、`supply.purchase_confirmation.daily`、`supply.replenishment.weekly`。
+5
View File
@@ -479,6 +479,9 @@ class AccountKeepaliveManager:
real_chrome=account.keepalive_real_chrome,
retries=1,
persist_state_on_close=False,
# 目标站点均为国内站,代理出口反被部分 WAF 拉黑(如蝉妈妈
# TLS reset),账号保活一律直连。
extra_flags=["--no-proxy-server"],
)
try:
browser.start()
@@ -761,6 +764,8 @@ def login_account(
"--remote-debugging-address=127.0.0.1",
"--disable-session-crashed-bubble",
"--disable-features=InfiniteSessionRestore",
# 国内站直连,避免代理出口被 WAF 拉黑导致登录/探测失败
"--no-proxy-server",
],
)
try:
+19
View File
@@ -168,3 +168,22 @@ def looks_like_login_url(url: str) -> bool:
path = parsed.path.casefold()
markers = ("login", "passport", "signin", "sign-in")
return any(marker in hostname or marker in path for marker in markers)
class CookieExpiredError(RuntimeError):
"""A platform login state expired and no automatic recovery exists.
Collectors raise this instead of waiting for a human QR/SMS login: the
account cookie gate has already blocked unverified accounts before the
run, so reaching this point means the session died mid-run. The CLI maps
it to ``COOKIE_SKIP_EXIT_CODE`` so the engine records a cookie skip.
"""
def cookie_expired_message(account_id: str, platform: str) -> str:
"""One standard, operator-actionable expiry message across platforms."""
return (
f"{platform} 登录态已失效:请执行 `uv run gyxx accounts login {account_id}` "
"完成扫码/验证码重新登录(或等待飞书通知后处理)"
)
@@ -0,0 +1,338 @@
"""Minimal, signed Jushuitan inventory-query client for workflow adapters."""
from __future__ import annotations
import hashlib
import json
import os
import time
import uuid
from collections.abc import Mapping
from dataclasses import dataclass
from decimal import Decimal, InvalidOperation
from typing import Any
from urllib.parse import urlparse
import requests
_TOKEN_PATH = "/openWeb/auth/getInitToken"
_INVENTORY_PATH = "/open/inventory/query"
_RATE_LIMIT_CODES = {199, 200}
_ACCESS_TOKEN_INVALID_CODE = 100
_MAX_RESPONSE_BYTES = 2 * 1024 * 1024
_MAX_PAGES = 100
_REQUEST_INTERVAL_SECONDS = 1.1
class JushuitanInventoryError(RuntimeError):
"""Safe-to-log error raised for Jushuitan query or response failures."""
@dataclass(frozen=True, slots=True)
class InventoryRecord:
sku: str
qty: int
order_lock: int
lock_qty: int
virtual_qty: int
@property
def public_available_qty(self) -> int:
return self.qty - self.order_lock - self.lock_qty + self.virtual_qty
def _sign(app_secret: str, params: Mapping[str, object]) -> str:
source = app_secret + "".join(
f"{key}{value}"
for key, value in sorted(params.items())
if key.lower() != "sign" and value is not None and str(value) != ""
)
return hashlib.md5(source.encode("utf-8")).hexdigest()
def _response_code(value: object) -> int:
if isinstance(value, bool):
raise JushuitanInventoryError("Jushuitan response has an invalid code")
try:
return int(value)
except (TypeError, ValueError) as exc:
raise JushuitanInventoryError(
"Jushuitan response has an invalid code"
) from exc
def _quantity(value: object, *, field: str, sku: str) -> int:
if value is None:
return 0
if isinstance(value, bool):
raise JushuitanInventoryError(
f"Jushuitan inventory has an invalid {field} for SKU {sku}"
)
try:
parsed = Decimal(str(value).strip())
except (InvalidOperation, ValueError) as exc:
raise JushuitanInventoryError(
f"Jushuitan inventory has an invalid {field} for SKU {sku}"
) from exc
if not parsed.is_finite() or parsed != parsed.to_integral_value():
raise JushuitanInventoryError(
f"Jushuitan inventory has a non-integer {field} for SKU {sku}"
)
return int(parsed)
def _value(item: Mapping[str, Any], snake_name: str, camel_name: str) -> object:
if snake_name in item:
return item[snake_name]
return item.get(camel_name)
class JushuitanInventoryClient:
"""Signed form client matching data-hub's Jushuitan API contract."""
def __init__(
self,
*,
app_key: str | None = None,
app_secret: str | None = None,
base_url: str | None = None,
timeout: tuple[float, float] = (10.0, 45.0),
) -> None:
self.app_key = (app_key or os.environ.get("JUSHUITAN_APP_KEY", "")).strip()
self.app_secret = (
app_secret or os.environ.get("JUSHUITAN_APP_SECRET", "")
).strip()
configured_url = (
base_url
or os.environ.get("JUSHUITAN_BASE_URL", "")
or "https://openapi.jushuitan.com"
).strip()
parsed_url = urlparse(configured_url)
if (
parsed_url.scheme.lower() != "https"
or not parsed_url.netloc
or parsed_url.username
or parsed_url.password
or parsed_url.query
or parsed_url.fragment
):
raise JushuitanInventoryError(
"JUSHUITAN_BASE_URL must be an HTTPS base URL without credentials"
)
if not self.app_key or not self.app_secret:
raise JushuitanInventoryError(
"JUSHUITAN_APP_KEY and JUSHUITAN_APP_SECRET must be injected "
"into the scheduler environment"
)
self.base_url = configured_url.rstrip("/")
self.timeout = timeout
self.session = requests.Session()
self._access_token = ""
self._last_request_at = 0.0
def query(self, skus: list[str]) -> dict[str, InventoryRecord]:
normalized_skus = list(dict.fromkeys(sku.strip() for sku in skus if sku.strip()))
if not normalized_skus:
raise JushuitanInventoryError("No SKU values were supplied")
if len(normalized_skus) > 100:
raise JushuitanInventoryError(
"A single inventory query may include at most 100 SKUs"
)
self._access_token = self._request_access_token()
page_index = 1
page_count = 1
records: dict[str, InventoryRecord] = {}
requested = set(normalized_skus)
while page_index <= page_count:
if page_index > _MAX_PAGES:
raise JushuitanInventoryError(
"Jushuitan inventory response exceeded the page safety limit"
)
response = self._query_page(normalized_skus, page_index)
data = response.get("data")
if not isinstance(data, Mapping):
raise JushuitanInventoryError(
"Jushuitan inventory response is missing its data object"
)
raw_rows = data.get("inventorys")
if raw_rows is None:
raw_rows = data.get("datas")
if raw_rows is None:
raw_rows = []
if not isinstance(raw_rows, list):
raise JushuitanInventoryError(
"Jushuitan inventory response has an invalid inventory list"
)
for index, raw_item in enumerate(raw_rows, start=1):
if raw_item is None:
continue
if not isinstance(raw_item, Mapping):
raise JushuitanInventoryError(
f"Jushuitan inventory row {index} is invalid"
)
sku_value = _value(raw_item, "sku_id", "skuId")
sku = str(sku_value or "").strip()
if not sku:
raise JushuitanInventoryError(
f"Jushuitan inventory row {index} has no SKU"
)
if sku not in requested:
raise JushuitanInventoryError(
"Jushuitan returned a SKU that was not requested"
)
record = InventoryRecord(
sku=sku,
qty=_quantity(
_value(raw_item, "qty", "qty"), field="qty", sku=sku
),
order_lock=_quantity(
_value(raw_item, "order_lock", "orderLock"),
field="order_lock",
sku=sku,
),
lock_qty=_quantity(
_value(raw_item, "lock_qty", "lockQty"),
field="lock_qty",
sku=sku,
),
virtual_qty=_quantity(
_value(raw_item, "virtual_qty", "virtualQty"),
field="virtual_qty",
sku=sku,
),
)
if sku in records:
raise JushuitanInventoryError(
f"Jushuitan returned duplicate inventory rows for SKU {sku}"
)
records[sku] = record
raw_page_count = data.get("page_count", data.get("pageCount", 1))
try:
page_count = max(1, int(raw_page_count or 1))
except (TypeError, ValueError) as exc:
raise JushuitanInventoryError(
"Jushuitan inventory response has an invalid page count"
) from exc
if page_count > _MAX_PAGES:
raise JushuitanInventoryError(
"Jushuitan inventory response exceeded the page safety limit"
)
page_index += 1
return records
def _request_access_token(self) -> str:
params: dict[str, object] = {
"app_key": self.app_key,
"charset": "utf-8",
"code": uuid.uuid4().hex[:6],
"grant_type": "authorization_code",
"timestamp": str(int(time.time())),
}
response = self._post_form(_TOKEN_PATH, params)
data = response.get("data")
if not isinstance(data, Mapping):
raise JushuitanInventoryError(
"Jushuitan token response is missing its data object"
)
token = data.get("access_token") or data.get("accessToken")
if not isinstance(token, str) or not token.strip():
raise JushuitanInventoryError(
"Jushuitan token response has no access token"
)
return token.strip()
def _query_page(self, skus: list[str], page_index: int) -> dict[str, Any]:
biz = {
"sku_ids": ",".join(skus),
"has_lock_qty": True,
"page_index": page_index,
"page_size": 100,
}
params: dict[str, object] = {
"access_token": self._access_token,
"app_key": self.app_key,
"charset": "utf-8",
"timestamp": str(int(time.time())),
"version": "2",
"biz": json.dumps(biz, ensure_ascii=False, separators=(",", ":")),
}
for token_attempt in range(2):
response = self._post_form(
_INVENTORY_PATH,
params,
allowed_api_codes=frozenset({_ACCESS_TOKEN_INVALID_CODE}),
)
code = _response_code(response.get("code"))
if code == _ACCESS_TOKEN_INVALID_CODE and token_attempt == 0:
self._access_token = self._request_access_token()
params["access_token"] = self._access_token
params["timestamp"] = str(int(time.time()))
continue
return response
raise JushuitanInventoryError("Jushuitan rejected the access token")
def _post_form(
self,
path: str,
params: dict[str, object],
*,
allowed_api_codes: frozenset[int] = frozenset(),
) -> dict[str, Any]:
rate_limit_attempt = 0
while True:
now = time.monotonic()
wait_seconds = _REQUEST_INTERVAL_SECONDS - (now - self._last_request_at)
if wait_seconds > 0:
time.sleep(wait_seconds)
self._last_request_at = time.monotonic()
signed_params = dict(params)
signed_params["sign"] = _sign(self.app_secret, signed_params)
try:
response = self.session.post(
f"{self.base_url}{path}",
data=signed_params,
timeout=self.timeout,
headers={"Accept": "application/json"},
)
except requests.RequestException as exc:
raise JushuitanInventoryError(
f"Jushuitan request failed for {path}"
) from exc
if response.status_code < 200 or response.status_code >= 300:
raise JushuitanInventoryError(
f"Jushuitan endpoint {path} returned HTTP {response.status_code}"
)
if len(response.content) > _MAX_RESPONSE_BYTES:
raise JushuitanInventoryError(
"Jushuitan response exceeded the 2 MiB safety limit"
)
try:
payload = response.json()
except ValueError as exc:
raise JushuitanInventoryError(
f"Jushuitan endpoint {path} returned invalid JSON"
) from exc
if not isinstance(payload, dict):
raise JushuitanInventoryError(
f"Jushuitan endpoint {path} returned an invalid response"
)
code = _response_code(payload.get("code"))
if code in _RATE_LIMIT_CODES and rate_limit_attempt < 4:
time.sleep(0.5 * (2**rate_limit_attempt))
rate_limit_attempt += 1
params["timestamp"] = str(int(time.time()))
continue
if code != 0 and code not in allowed_api_codes:
raise JushuitanInventoryError(
f"Jushuitan endpoint {path} returned API code {code}"
)
return payload
__all__ = [
"InventoryRecord",
"JushuitanInventoryClient",
"JushuitanInventoryError",
]
+16
View File
@@ -79,6 +79,10 @@ class WorkflowEntry:
steps: tuple[WorkflowStepEntry, ...] = ()
notification_workflow_id: str | None = None
topic_config: WorkflowTopicConfig | None = None
# Orchestrator entries whose own binding carries no runtime account while
# the spawned child collectors do; the cookie gate probes these before
# the workflow runs.
required_accounts: tuple[str, ...] = ()
@dataclass(frozen=True, slots=True)
@@ -218,6 +222,9 @@ def _parse_workflow(item: Any) -> WorkflowEntry:
if trigger == "scheduled" and (not isinstance(task_name, str) or not task_name):
raise CatalogError(f"scheduled workflow requires source task name: {workflow_id}")
topic_config = _parse_topic_config(item.get("topic_config"), workflow_id)
required_accounts = _parse_required_accounts(
item.get("required_accounts", []), workflow_id
)
return WorkflowEntry(
workflow_id=workflow_id,
module=module,
@@ -229,9 +236,18 @@ def _parse_workflow(item: Any) -> WorkflowEntry:
note=item.get("note"),
steps=steps,
topic_config=topic_config,
required_accounts=required_accounts,
)
def _parse_required_accounts(value: Any, workflow_id: str) -> tuple[str, ...]:
if not isinstance(value, list) or not all(
isinstance(item, str) and _WORKFLOW_ID.fullmatch(item) for item in value
):
raise CatalogError(f"invalid required_accounts for {workflow_id}")
return tuple(dict.fromkeys(value))
def _parse_topic_config(value: Any, workflow_id: str) -> WorkflowTopicConfig | None:
if value is None:
return None
+41 -1
View File
@@ -703,7 +703,19 @@ def _python_scheduler(
if enable_account_keepalive and not dry_run:
from gyxx_flow.accounts import AccountKeepaliveManager
maintenance = AccountKeepaliveManager(settings).tick
keepalive_manager = AccountKeepaliveManager(settings)
recovery_manager = _build_cookie_recovery_manager(
settings, launcher=launcher, keepalive=keepalive_manager
)
def maintenance() -> None: # noqa: D107 - scheduler maintenance chain
keepalive_manager.tick()
if recovery_manager is not None:
try:
recovery_manager.tick()
except Exception: # noqa: BLE001 - 自动重跑不得影响调度心跳
pass
return PythonScheduler(
catalog,
settings.data_root,
@@ -717,6 +729,29 @@ def _python_scheduler(
)
def _build_cookie_recovery_manager(
settings: Settings,
*,
launcher: object,
keepalive: object,
):
"""Build the cookie-gate recovery rerun manager; None when disabled."""
from gyxx_flow.cookie_gate import (
CookieGate,
CookieGateRecoveryManager,
gate_enabled,
)
if not gate_enabled():
return None
try:
gate = CookieGate(settings, keepalive=keepalive) # type: ignore[arg-type]
except Exception: # noqa: BLE001 - 无集成配置时没有可恢复的账号
return None
return CookieGateRecoveryManager(gate, launcher=launcher)
def _schedule_run_command(
arguments: argparse.Namespace,
catalog: WorkflowCatalog,
@@ -885,6 +920,11 @@ def _execute_once(
dry_run: bool,
settings: Settings,
) -> tuple[int, dict[str, object]]:
if not dry_run:
# 每账号 Cookie 健康门控:dry-run 不注入,也不做任何探测/通知。
from gyxx_flow.cookie_gate import apply_cookie_gates
workflow = apply_cookie_gates(workflow, settings=settings)
context = RunContext.create(
workflow.workflow_id,
business_date.isoformat(),
+118 -2
View File
@@ -5,6 +5,7 @@ from __future__ import annotations
import calendar
import hashlib
import hmac
import http.client
import ipaddress
import json
import os
@@ -15,6 +16,7 @@ import sys
import tempfile
import threading
import time
import traceback
import unicodedata
import uuid
from contextlib import suppress
@@ -24,7 +26,7 @@ from http import HTTPStatus
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from importlib import resources
from pathlib import Path
from typing import Any, Mapping, Protocol, Sequence, TextIO, cast
from typing import Any, Callable, Mapping, Protocol, Sequence, TextIO, cast
from urllib.parse import parse_qs, unquote, urlsplit
from zoneinfo import ZoneInfo
@@ -171,7 +173,6 @@ _WORKFLOW_NAMES = {
"content.metrics.daily": "合作达人内容指标日报采集",
"content.metrics.self_operated.daily": "自营内容指标日报采集",
"content.marketing_report.daily": "营销日报生成与发送",
"content.relogin.weekly": "内容平台登录态刷新",
"content.creator_report.monthly": "达人月报生成",
"content.summary.monthly": "内容月度汇总",
"content.comments.weekly": "多平台评论周采集",
@@ -3736,6 +3737,7 @@ class WorkflowConsoleHTTPServer(ThreadingHTTPServer):
self.console_service = service
self.console_token = token
self.bind_host = bind_host
self.started_at = time.time()
super().__init__(server_address, WorkflowConsoleRequestHandler)
def handle_error(
@@ -3796,6 +3798,29 @@ class WorkflowConsoleRequestHandler(BaseHTTPRequestHandler):
if method in {"GET", "HEAD"} and path == "/assets/app.js":
self._static("app.js", "text/javascript; charset=utf-8", head=method == "HEAD")
return
if method == "GET" and path == "/api/health":
# 自检探针端点:不鉴权、不触外部依赖,只反映进程与调度锁存活,
# 供进程内看门狗与 deploy/watchdog 保活脚本探测。
try:
scheduler = _scheduler_process_from_lock(
self.console_server.console_service.settings
)
except Exception:
scheduler = None
self._json(
HTTPStatus.OK,
{
"ok": True,
"pid": os.getpid(),
"started_at": self.console_server.started_at,
"uptime_seconds": round(
time.time() - self.console_server.started_at, 1
),
"scheduler_running": scheduler is not None,
"scheduler_pid": scheduler.pid if scheduler else None,
},
)
return
if method == "GET" and path == "/api/overview":
self._require_api_access(mutation=False)
self._json(HTTPStatus.OK, self.console_server.console_service.overview())
@@ -4020,6 +4045,9 @@ class WorkflowConsoleRequestHandler(BaseHTTPRequestHandler):
except (BrokenPipeError, ConnectionResetError):
return
except Exception:
# 前端只拿到固定文案;完整堆栈落到 stderr(随启动重定向进
# var/logs/console-restarts 或服务日志),否则故障无法定位。
traceback.print_exc()
self._json_error(HTTPStatus.INTERNAL_SERVER_ERROR, "控制台处理请求失败")
def _require_api_access(self, *, mutation: bool) -> None:
@@ -4190,6 +4218,92 @@ def create_console_server(
)
CONSOLE_WATCHDOG_EXIT_CODE = 70
def _probe_console_health(port: int, timeout_seconds: float) -> bool:
# 只连本机回环端口的固定路径 /api/health,探测自身 HTTP 服务往返。
connection = http.client.HTTPConnection("127.0.0.1", port, timeout=timeout_seconds)
try:
connection.request("GET", "/api/health")
response = connection.getresponse()
response.read()
return response.status == HTTPStatus.OK
except Exception:
return False
finally:
connection.close()
def _console_watchdog(
port: int,
*,
interval_seconds: float,
failure_limit: int,
probe_timeout_seconds: float,
stopped: threading.Event,
probe: Callable[[int, float], bool] | None = None,
exit_process: Callable[[int], None] = os._exit,
sleep: Callable[[float], None] = time.sleep,
) -> None:
"""进程内自检:周期探测自身 /api/health,连续失败则退出交给守护拉起。
HTTP accept 线程活着但工作线程全部卡死(锁竞争/句柄耗尽)时,外部
静态页能打开而 API 全挂——正是"控制台暂时不可用"的形态;只有真实
走一遍 HTTP 往返才能暴露,所以探针必须打 socket 而不是查内存状态。
"""
run_probe = probe or _probe_console_health
failures = 0
while not stopped.is_set():
if run_probe(port, probe_timeout_seconds):
failures = 0
else:
failures += 1
print(
f"⚠️ 控制台自检失败 {failures}/{failure_limit}: 127.0.0.1:{port}/api/health",
file=sys.stderr,
flush=True,
)
if failures >= failure_limit:
print(
"❌ 控制台自检连续失败,进程退出等待守护脚本拉起",
file=sys.stderr,
flush=True,
)
exit_process(CONSOLE_WATCHDOG_EXIT_CODE)
return
sleep(interval_seconds)
def _start_console_watchdog(server: WorkflowConsoleHTTPServer) -> threading.Event:
interval = max(
5.0, float(os.environ.get("GYXX_CONSOLE_WATCHDOG_INTERVAL_SECONDS", "30"))
)
limit = int(os.environ.get("GYXX_CONSOLE_WATCHDOG_FAILURES", "3"))
enabled = (
os.environ.get("GYXX_CONSOLE_WATCHDOG", "1").strip().lower()
not in {"0", "false", "no", "off"}
and limit > 0
)
stopped = threading.Event()
if not enabled:
return stopped
thread = threading.Thread(
target=_console_watchdog,
args=(server.server_port,),
kwargs={
"interval_seconds": interval,
"failure_limit": limit,
"probe_timeout_seconds": min(5.0, interval),
"stopped": stopped,
},
name="gyxx-console-watchdog",
daemon=True,
)
thread.start()
return stopped
def serve_console(
settings: Settings,
*,
@@ -4211,11 +4325,13 @@ def serve_console(
display_host = "127.0.0.1" if host in {"0.0.0.0", "::"} else host
output.write(f"GYXX Flow console: http://{display_host}:{server.server_port}\n")
output.flush()
watchdog_stopped = _start_console_watchdog(server)
try:
server.serve_forever(poll_interval=0.5)
except KeyboardInterrupt:
pass
finally:
watchdog_stopped.set()
server.server_close()
return 0
+882
View File
@@ -0,0 +1,882 @@
"""Per-account cookie health gate injected before workflow collection steps.
Every runtime account a workflow depends on is checked before the collection
steps that use it may run. Accounts fall into three gate modes derived from
``config/runtime-bindings.json``:
* ``blocking`` -- QR-code / SMS-only accounts (no credential environment and
no configured login command). The gate verifies the login state and, when
it has expired, fails the gate step so every dependent step is skipped with
``dependency-failed`` and sends a Feishu alert asking the login owner to
re-login interactively via ``gyxx accounts login <account_id>``.
* ``recover`` -- accounts with a vault-managed credential login command
(``login_mode`` ``C``). The keepalive manager re-runs the credential login
automatically; only a failed recovery blocks the workflow and alerts.
* ``defer`` -- accounts whose member scripts embed username/password login
flows. These keep their script-level login behaviour unchanged and are not
gated at all.
The per-account decision is persisted as a small state machine under
``state/accounts/<id>/cookie_state.json`` (``healthy`` <-> ``awaiting_login``)
so repeat failures alert at most once per notify interval instead of once per
workflow run.
"""
from __future__ import annotations
import json
import os
from collections.abc import Callable
from dataclasses import dataclass, replace
from datetime import date, datetime, timezone
from pathlib import Path
from typing import Any, Mapping
from gyxx_flow.accounts import (
AccountKeepaliveManager,
_keepalive_requires_relogin,
_read_keepalive_status,
_vault_cookie_valid,
)
from gyxx_flow.adapters.browser import BrowserCookieStore
from gyxx_flow.adapters.feishu_messages import send_lark_bot_message
from gyxx_flow.adapters.integration import RuntimeIntegrationCatalog
from gyxx_flow.catalog import WorkflowCatalog
from gyxx_flow.core.artifacts import atomic_write_json
from gyxx_flow.core.config import Settings
from gyxx_flow.core.context import RunContext
from gyxx_flow.notification_routing import (
ANALYZER_ACCEPTANCE_RECIPIENT_OPEN_ID,
ANALYZER_NOTIFICATION_APP_PROFILE,
NotificationRoutingStore,
)
from gyxx_flow.workflow.model import StepDefinition, WorkflowDefinition
from gyxx_flow.workflow.steps import StepExecution
GATE_MODE_BLOCKING = "blocking"
GATE_MODE_RECOVER = "recover"
GATE_MODE_DEFER = "defer"
# 飞书通知的默认接收人:按手机号在 config/notification-routing.json 的
# people 段解析 open_id;解析失败时回退到同名 analyzer 常量收件人。
DEFAULT_LOGIN_ALERT_MOBILE = "+8613663785145"
GYXX_COOKIE_GATE = "GYXX_COOKIE_GATE"
GYXX_COOKIE_GATE_MAX_STATUS_AGE_SECONDS = "GYXX_COOKIE_GATE_MAX_STATUS_AGE_SECONDS"
GYXX_LOGIN_ALERT_OPEN_IDS = "GYXX_LOGIN_ALERT_OPEN_IDS"
GYXX_LOGIN_ALERT_APP_PROFILE = "GYXX_LOGIN_ALERT_APP_PROFILE"
GYXX_LOGIN_ALERT_NOTIFY_INTERVAL_SECONDS = "GYXX_LOGIN_ALERT_NOTIFY_INTERVAL_SECONDS"
_GATE_STEP_PREFIX = "cookie-gate"
_GATE_STATE_FILENAME = "cookie_state.json"
_GATE_TIMEOUT_SECONDS = 1800.0
_DEFAULT_MAX_STATUS_AGE_SECONDS = 3600.0
_MIN_MAX_STATUS_AGE_SECONDS = 300.0
_DEFAULT_NOTIFY_INTERVAL_SECONDS = 1800.0
_DISABLED_VALUES = frozenset({"0", "false", "off", "no", "disabled"})
# 自动重跑:待重跑清单上限、健康探活节流、同账号重跑启动冷却。
_MAX_PENDING_RERUNS = 20
_RECOVERY_PROBE_INTERVAL_SECONDS = 300.0
_RECOVERY_LAUNCH_COOLDOWN_SECONDS = 300.0
_RECOVERY_HISTORY_LIMIT = 20
STATE_HEALTHY = "healthy"
STATE_AWAITING_LOGIN = "awaiting_login"
@dataclass(frozen=True, slots=True)
class GateVerdict:
"""One account's gate decision, recorded in the run journal on failure."""
account_id: str
mode: str
blocked: bool
message: str
recovered: bool = False
notified: bool = False
@dataclass(frozen=True, slots=True)
class CookieGateStepAction:
"""ExecutableStep that turns one account verdict into a step outcome."""
gate: "CookieGate"
account_id: str
def execute(
self,
*,
context: RunContext,
timeout_seconds: float | None,
dry_run: bool,
) -> StepExecution:
if dry_run: # pragma: no cover - the engine skips dry-run steps first
return StepExecution(exit_code=0, skipped=True, reason="dry-run")
verdict = self.gate.check(self.account_id, context=context)
if verdict.blocked:
return StepExecution(
exit_code=1,
error=verdict.message,
reason="cookie-expired",
)
return StepExecution(exit_code=0)
class CookieGate:
"""Resolve, probe, and gate the accounts one workflow depends on."""
def __init__(
self,
settings: Settings,
*,
catalog: RuntimeIntegrationCatalog | None = None,
keepalive: AccountKeepaliveManager | None = None,
environment: Mapping[str, str] | None = None,
now: Callable[[], datetime] | None = None,
sender: Callable[..., dict[str, Any]] | None = None,
) -> None: # noqa: D107 - simple wiring
self.settings = settings
self.catalog = catalog or RuntimeIntegrationCatalog.load_default(
project_root=settings.project_root,
data_root=settings.data_root,
)
self.keepalive = keepalive or AccountKeepaliveManager(
settings, catalog=self.catalog
)
self._environment = dict(os.environ if environment is None else environment)
self._now = now or (lambda: datetime.now(timezone.utc))
self._sender = sender or send_lark_bot_message
self._workflow_catalog: WorkflowCatalog | None = None
# ------------------------------------------------------------------
# Account resolution
# ------------------------------------------------------------------
def gate_mode(self, account_id: str) -> str:
account = self.catalog.account_for(account_id)
if str(account.login_command_id or "").strip():
return GATE_MODE_RECOVER
if account.credential_env_names or account.login_mode == "B":
return GATE_MODE_DEFER
return GATE_MODE_BLOCKING
def collect_workflow_accounts(
self,
workflow: WorkflowDefinition,
) -> dict[str, set[str]]:
"""Map each gated account to the step ids that depend on it.
Accounts resolve two ways: directly, from each step binding's
``account``; and declaratively, from the catalog workflow's
``required_accounts`` (used by orchestrator entries whose own binding
has no account while the spawned child collectors do).
"""
accounts: dict[str, set[str]] = {}
for step in workflow.steps:
account_id = self._account_for_step(step)
if account_id:
accounts.setdefault(account_id, set()).add(step.step_id)
for account_id in self._declared_required_accounts(workflow.workflow_id):
self.catalog.account_for(account_id) # unknown id: loud config error
accounts.setdefault(account_id, set()).update(
step.step_id for step in workflow.steps
)
# Defer-mode accounts keep their script-level credential login flows
# and are intentionally not gated.
return {
account_id: step_ids
for account_id, step_ids in accounts.items()
if self.gate_mode(account_id) != GATE_MODE_DEFER
}
def _account_for_step(self, step: StepDefinition) -> str | None:
action = step.action
entry = getattr(action, "entry", None)
module_id = getattr(entry, "module", "")
command_entry = getattr(action, "command_entry", None) or getattr(
entry, "entry", ""
)
if not module_id or not command_entry:
return None
try:
binding = self.catalog.binding_for(f"{module_id}:{command_entry}")
except Exception: # noqa: BLE001 - unbound entries run ungated
return None
return binding.account or None
def workflow_catalog(self) -> WorkflowCatalog:
"""Load (once) the catalog used for declared account and rerun lookups."""
if self._workflow_catalog is None:
try:
self._workflow_catalog = WorkflowCatalog.load(
self.settings.project_root / "config"
)
except Exception: # noqa: BLE001 - catalog issues surface elsewhere
self._workflow_catalog = WorkflowCatalog(
timezone="UTC", workflows=(), schedules=()
)
return self._workflow_catalog
def _declared_required_accounts(self, workflow_id: str) -> tuple[str, ...]:
for entry in self.workflow_catalog().workflows:
if entry.workflow_id == workflow_id:
return tuple(entry.required_accounts)
return ()
# ------------------------------------------------------------------
# State machine
# ------------------------------------------------------------------
def check(self, account_id: str, *, context: RunContext) -> GateVerdict:
account = self.catalog.account_for(account_id)
mode = self.gate_mode(account_id)
store = BrowserCookieStore(account.cookie_file, account.storage_state_file)
static_valid = _vault_cookie_valid(account, store)
status = _read_keepalive_status(account)
if static_valid and self._keepalive_success_is_fresh(status):
self._write_state(account, STATE_HEALTHY, probe="fresh-keepalive")
return GateVerdict(
account_id=account_id,
mode=mode,
blocked=False,
message=f"账号 {account_id} 登录态有效(keepalive 近期验证通过)",
)
if self._acceptance_enabled():
# 验收模式不开浏览器:按静态证据判定,且不投递飞书通知。
if static_valid and not _keepalive_requires_relogin(status):
return GateVerdict(
account_id=account_id,
mode=mode,
blocked=False,
message=f"账号 {account_id} 静态 cookie 校验通过(验收模式)",
)
return self._expire(
account,
mode,
context,
detail="验收模式下静态 cookie 校验未通过",
notify=False,
)
outcome = self.keepalive.refresh_account(account_id)
probe_status = str(outcome.get("status", "error"))
if probe_status == "success":
recovered = bool(outcome.get("login_recovered"))
self._write_state(
account,
STATE_HEALTHY,
probe="success",
recovered=recovered,
)
message = (
f"账号 {account_id} 登录态有效(自动账密重登成功)"
if recovered
else f"账号 {account_id} 登录态有效(探活通过)"
)
return GateVerdict(
account_id=account_id,
mode=mode,
blocked=False,
message=message,
recovered=recovered,
)
if probe_status == "busy":
if static_valid and not _keepalive_requires_relogin(status):
return GateVerdict(
account_id=account_id,
mode=mode,
blocked=False,
message=f"账号 {account_id} 探活锁忙,静态 cookie 校验通过",
)
return self._expire(
account,
mode,
context,
detail="探活锁被占用且静态 cookie 校验未通过",
)
if probe_status in {"expired", "not_logged_in"}:
evidence = _recovery_evidence(_read_keepalive_status(account))
detail = "cookie 已过期,无法正常采集"
if probe_status == "not_logged_in" and evidence:
detail = f"登录态缺失;{evidence}"
elif evidence:
detail = f"cookie 已过期;{evidence}"
return self._expire(account, mode, context, detail=detail)
# timeout / error / http_error / disabled:探测结果不可信时回退静态证据。
if static_valid and not _keepalive_requires_relogin(status):
return GateVerdict(
account_id=account_id,
mode=mode,
blocked=False,
message=f"账号 {account_id} 探活异常({probe_status}),静态 cookie 校验通过",
)
return self._expire(
account,
mode,
context,
detail=f"探活异常({probe_status})且静态 cookie 校验未通过",
)
# ------------------------------------------------------------------
# Expiry, notification, persisted state
# ------------------------------------------------------------------
def _expire(
self,
account: Any,
mode: str,
context: RunContext,
*,
detail: str,
notify: bool | None = None,
) -> GateVerdict:
account_id = str(account.account_id)
if notify is None:
notify = not context.shadow and not self._acceptance_enabled()
state = self._read_state(account)
notified = False
notification_note = ""
if notify and not self._within_notify_interval(state):
recipients = self._alert_recipients()
message = _expiry_message(account_id, mode, context, detail)
for open_id, profile in recipients:
try:
self._sender(
user_id=open_id,
text=message,
idempotency_key=f"cookie-gate:{account_id}:{context.run_id}",
profile=profile,
)
notified = True
except Exception as exc: # noqa: BLE001 - alerting must not mask the verdict
notification_note = f"{type(exc).__name__}: {exc}"[:300]
if recipients and notified:
state["last_notified_at"] = _now_iso(self._now)
state.pop("notification_error", None)
elif notification_note:
state["notification_error"] = notification_note
state["state"] = STATE_AWAITING_LOGIN
state["updated_at"] = _now_iso(self._now)
state["last_probe_status"] = "expired"
state["last_blocked_run"] = context.run_id
if not context.shadow and not self._acceptance_enabled():
# 记录待自动重跑清单:登录态恢复后由调度器维护循环按原业务日期重跑。
state["pending_reruns"] = _record_pending_rerun(
state.get("pending_reruns"), context, _now_iso(self._now)
)
self._save_state(account, state)
message = _expiry_message(account_id, mode, context, detail)
if notified:
message += " 已通过飞书通知登录负责人。"
elif notification_note:
message += f" 飞书通知发送失败({notification_note})。"
return GateVerdict(
account_id=account_id,
mode=mode,
blocked=True,
message=message,
notified=notified,
)
def _write_state(
self,
account: Any,
state_name: str,
*,
probe: str,
recovered: bool = False,
) -> None:
state = self._read_state(account)
if (
state_name == STATE_HEALTHY
and state.get("state") == STATE_HEALTHY
and state.get("last_probe_status") in {"fresh-keepalive", "success"}
and not recovered
):
return # 持续健康时避免每次运行都重写状态文件
state["state"] = state_name
state["updated_at"] = _now_iso(self._now)
state["last_probe_status"] = probe
if state_name == STATE_HEALTHY:
state["last_healthy_at"] = _now_iso(self._now)
if recovered:
state["last_recovered_at"] = _now_iso(self._now)
state.pop("notification_error", None)
self._save_state(account, state)
def _state_path(self, account: Any) -> Path:
return account.cookie_file.with_name(_GATE_STATE_FILENAME)
def _read_state(self, account: Any) -> dict[str, object]:
path = self._state_path(account)
if not path.exists():
return {"account_id": str(account.account_id)}
try:
payload = json.loads(path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError):
return {"account_id": str(account.account_id)}
if not isinstance(payload, dict):
return {"account_id": str(account.account_id)}
return dict(payload)
def _save_state(self, account: Any, state: Mapping[str, object]) -> None:
payload = {
"account_id": str(account.account_id),
**state,
}
try:
atomic_write_json(self._state_path(account), payload)
except OSError: # pragma: no cover - state file is advisory only
pass
def _within_notify_interval(self, state: Mapping[str, object]) -> bool:
raw = state.get("last_notified_at")
if not isinstance(raw, str):
return False
try:
last = datetime.fromisoformat(raw)
except ValueError:
return False
now = self._now()
if last.tzinfo is None:
last = last.replace(tzinfo=timezone.utc)
interval = _env_int(
self._environment,
GYXX_LOGIN_ALERT_NOTIFY_INTERVAL_SECONDS,
_DEFAULT_NOTIFY_INTERVAL_SECONDS,
minimum=0,
)
return (now - last).total_seconds() < interval
def _keepalive_success_is_fresh(self, status: Mapping[str, object] | None) -> bool:
if not isinstance(status, Mapping) or status.get("status") != "success":
return False
raw = status.get("last_success_at")
if not isinstance(raw, str):
return False
try:
last = datetime.fromisoformat(raw)
except ValueError:
return False
now = self._now()
if last.tzinfo is None:
last = last.replace(tzinfo=timezone.utc)
max_age = _env_float(
self._environment,
GYXX_COOKIE_GATE_MAX_STATUS_AGE_SECONDS,
_DEFAULT_MAX_STATUS_AGE_SECONDS,
minimum=_MIN_MAX_STATUS_AGE_SECONDS,
)
return (now - last).total_seconds() <= max_age
def _alert_recipients(self) -> list[tuple[str, str]]:
profile = (
self._environment.get(GYXX_LOGIN_ALERT_APP_PROFILE, "").strip()
or ANALYZER_NOTIFICATION_APP_PROFILE
)
override = [
value.strip()
for value in self._environment.get(GYXX_LOGIN_ALERT_OPEN_IDS, "").split(",")
if value.strip()
]
if override:
return [(open_id, profile) for open_id in override]
try:
payload, _revision = NotificationRoutingStore(self.settings).snapshot()
for person in payload["people"].values():
if person.get("mobile") == DEFAULT_LOGIN_ALERT_MOBILE:
binding = person.get("bindings", {}).get(profile)
if binding:
return [(str(binding["open_id"]), profile)]
except Exception: # noqa: BLE001 - fall back to the constant recipient
pass
return [(ANALYZER_ACCEPTANCE_RECIPIENT_OPEN_ID, profile)]
def _acceptance_enabled(self) -> bool:
value = self._environment.get("GYXX_WORKFLOW_ACCEPTANCE", "")
return bool(value and value.strip().casefold() in {"1", "true", "yes", "on"})
def gate_enabled(environment: Mapping[str, str] | None = None) -> bool:
values = os.environ if environment is None else environment
return values.get(GYXX_COOKIE_GATE, "").strip().casefold() not in _DISABLED_VALUES
def apply_cookie_gates(
workflow: WorkflowDefinition,
*,
settings: Settings,
environment: Mapping[str, str] | None = None,
gate: CookieGate | None = None,
) -> WorkflowDefinition:
"""Inject one critical gate step per gated account in front of its users.
Gate steps are graph sources; each step that uses an account gains the
account's gate as an extra dependency. A failed gate therefore skips its
dependents with ``dependency-failed`` through the regular engine, while
unrelated branches keep running. Operator slices (``--rerun-step`` /
``--resume-from``) are produced before this injection and stay gate-free.
"""
if not workflow.steps:
return workflow
if gate is None:
try:
gate = CookieGate(
settings,
environment=os.environ if environment is None else environment,
)
except Exception: # noqa: BLE001 - 无集成配置时不门控;步骤执行仍会显式报错
return workflow
account_steps = gate.collect_workflow_accounts(workflow)
if not account_steps:
return workflow
existing_ids = {step.step_id for step in workflow.steps}
gate_steps: list[StepDefinition] = []
for account_id in sorted(account_steps):
step_id = f"{_GATE_STEP_PREFIX}.{account_id}"
if step_id in existing_ids:
continue
gate_steps.append(
StepDefinition(
step_id,
CookieGateStepAction(gate, account_id),
timeout_seconds=_GATE_TIMEOUT_SECONDS,
max_attempts=1,
)
)
if not gate_steps:
return workflow
patched: list[StepDefinition] = []
for step in workflow.steps:
dependencies = list(dict.fromkeys(step.depends_on))
changed = False
for account_id, step_ids in account_steps.items():
gate_id = f"{_GATE_STEP_PREFIX}.{account_id}"
if step.step_id in step_ids and gate_id not in dependencies:
dependencies.append(gate_id)
changed = True
if changed:
step = replace(step, depends_on=tuple(dependencies))
patched.append(step)
return WorkflowDefinition(
workflow.workflow_id,
(*gate_steps, *patched),
)
class CookieGateRecoveryManager:
"""Re-run workflows blocked by expired cookies once the login recovers.
Runs inside the resident scheduler's maintenance loop. For every account
with a non-empty ``pending_reruns`` list, it re-verifies the login state
(fresh keepalive success first, then a throttled live probe); on recovery
it relaunches each recorded workflow with its original business date
through the scheduler's own launcher, clears the pending list, and sends a
Feishu confirmation to the login owner.
"""
def __init__(
self,
gate: CookieGate,
*,
launcher: Any,
probe_interval_seconds: float = _RECOVERY_PROBE_INTERVAL_SECONDS,
launch_cooldown_seconds: float = _RECOVERY_LAUNCH_COOLDOWN_SECONDS,
) -> None:
self.gate = gate
self._launcher = launcher
self._probe_interval = probe_interval_seconds
self._launch_cooldown = launch_cooldown_seconds
def tick(self) -> list[dict[str, object]]:
"""Attempt recovery reruns once; returns redacted outcomes per account."""
outcomes: list[dict[str, object]] = []
for account in self.gate.catalog.accounts:
state = self.gate._read_state(account)
pending = state.get("pending_reruns")
if not isinstance(pending, list) or not pending:
continue
now = self.gate._now()
if not self._launch_cooldown_passed(state, now):
continue
if not self._account_recovered(account, state, now):
continue
launched, remaining = self._launch_pending(account, pending, now)
state["pending_reruns"] = remaining
state["updated_at"] = _now_iso(self.gate._now)
state["last_recovered_at"] = state["updated_at"]
if not remaining:
state["state"] = STATE_HEALTHY
state["last_probe_status"] = "recovered-rerun"
if launched:
state["last_rerun_launch_at"] = state["updated_at"]
history = [
item
for item in state.get("rerun_history", [])
if isinstance(item, dict)
]
history.extend(launched)
state["rerun_history"] = history[-_RECOVERY_HISTORY_LIMIT:]
self.gate._save_state(account, state)
if launched:
self._notify_recovered(str(account.account_id), launched)
outcomes.append(
{
"account_id": str(account.account_id),
"launched": launched,
"pending": len(remaining),
}
)
return outcomes
# ------------------------------------------------------------------
# internals
# ------------------------------------------------------------------
def _launch_cooldown_passed(
self, state: Mapping[str, object], now: datetime
) -> bool:
raw = state.get("last_rerun_launch_at")
if not isinstance(raw, str):
return True
try:
last = datetime.fromisoformat(raw)
except ValueError:
return True
if last.tzinfo is None:
last = last.replace(tzinfo=timezone.utc)
return (now - last).total_seconds() >= self._launch_cooldown
def _account_recovered(
self,
account: Any,
state: dict[str, object],
now: datetime,
) -> bool:
"""Fresh keepalive success first; otherwise one throttled live probe."""
store = BrowserCookieStore(account.cookie_file, account.storage_state_file)
if _vault_cookie_valid(account, store) and self.gate._keepalive_success_is_fresh(
_read_keepalive_status(account)
):
return True
raw = state.get("last_recovery_probe_at")
if isinstance(raw, str):
try:
last_probe = datetime.fromisoformat(raw)
except ValueError:
last_probe = None
if last_probe is not None:
if last_probe.tzinfo is None:
last_probe = last_probe.replace(tzinfo=timezone.utc)
if (now - last_probe).total_seconds() < self._probe_interval:
return False
state["last_recovery_probe_at"] = _now_iso(self.gate._now)
self.gate._save_state(account, state)
try:
outcome = self.gate.keepalive.refresh_account(str(account.account_id))
except Exception: # noqa: BLE001 - 恢复探活失败留给下一轮
return False
return str(outcome.get("status", "error")) == "success"
def _launch_pending(
self,
account: Any,
pending: list[object],
now: datetime,
) -> tuple[list[dict[str, object]], list[dict[str, object]]]:
known_ids = {
entry.workflow_id for entry in self.gate.workflow_catalog().workflows
}
launched: list[dict[str, object]] = []
remaining: list[dict[str, object]] = []
for item in pending:
if not isinstance(item, dict):
continue # 控制台脚本等临时运行 id:不在 catalog 中,直接丢弃
workflow_id = item.get("workflow_id")
business_date = item.get("business_date")
if not isinstance(workflow_id, str) or not isinstance(business_date, str):
continue
if workflow_id not in known_ids:
continue
try:
process = self._launcher.launch(
workflow_id, now, date.fromisoformat(business_date)
)
except Exception as exc: # noqa: BLE001 - 保留 pending,下一轮重试
remaining.append(
{
"workflow_id": workflow_id,
"business_date": business_date,
"error": f"{type(exc).__name__}: {exc}"[:200],
}
)
continue
close = getattr(self._launcher, "close", None)
if callable(close):
try:
close(process)
except Exception: # noqa: BLE001 - 关闭日志句柄失败不影响重跑
pass
launched.append(
{
"workflow_id": workflow_id,
"business_date": business_date,
"pid": int(getattr(process, "pid", 0) or 0),
"launched_at": _now_iso(self.gate._now),
}
)
return launched, remaining
def _notify_recovered(
self,
account_id: str,
launched: list[dict[str, object]],
) -> None:
lines = [
f"- {item['workflow_id']}(业务日期 {item['business_date']})"
for item in launched
]
message = (
"【GYXX 登录态已恢复】账号 "
f"{account_id} 登录态已恢复,已自动重跑被阻断的工作流:\n"
+ "\n".join(lines)
+ "\n如本次重跑仍有失败,请查看运行日志或工作流控制台。"
)
key = "-".join(
[account_id, *(str(item["workflow_id"]) for item in launched)]
)
for open_id, profile in self.gate._alert_recipients():
try:
self.gate._sender(
user_id=open_id,
text=message,
idempotency_key=f"cookie-recovery:{key}",
profile=profile,
)
except Exception: # noqa: BLE001 - 确认通知失败不影响已发起的重跑
continue
def _record_pending_rerun(
current: object,
context: RunContext,
blocked_at: str,
) -> list[dict[str, object]]:
"""Append one blocked run to the pending list, deduped and bounded."""
pending: list[dict[str, object]] = [
dict(item)
for item in (current if isinstance(current, list) else [])
if isinstance(item, dict)
]
entry = {
"workflow_id": context.workflow_id,
"business_date": context.business_date.isoformat(),
"blocked_run_id": context.run_id,
"blocked_at": blocked_at,
}
pending = [
item
for item in pending
if not (
item.get("workflow_id") == entry["workflow_id"]
and item.get("business_date") == entry["business_date"]
)
]
pending.append(entry)
return pending[-_MAX_PENDING_RERUNS:]
def _recovery_evidence(status: Mapping[str, object] | None) -> str:
if not isinstance(status, Mapping):
return ""
if not status.get("login_attempted"):
return ""
reason = status.get("login_reason")
error_type = status.get("login_error_type")
exit_code = status.get("login_exit_code")
parts = ["自动账密重登已尝试但未成功"]
if isinstance(reason, str) and reason:
parts.append(reason)
if isinstance(error_type, str) and error_type:
parts.append(f"错误={error_type}")
if isinstance(exit_code, int) and not isinstance(exit_code, bool):
parts.append(f"退出码={exit_code}")
return "(" + ",".join(parts) + ")"
def _expiry_message(
account_id: str,
mode: str,
context: RunContext,
detail: str,
) -> str:
mode_note = (
"账号密码自动重登失败,需人工处理(如验证码)"
if mode == GATE_MODE_RECOVER
else "该平台仅支持扫码/验证码登录,无法自动恢复"
)
return (
"【GYXX 登录态失效】工作流被阻断,需要人工重新登录\n"
f"工作流: {context.workflow_id}(业务日期 {context.business_date.isoformat()})\n"
f"账号: {account_id}\n"
f"原因: {detail};{mode_note}\n"
f"请在服务器上执行: uv run gyxx accounts login {account_id}\n"
"(将打开浏览器窗口,完成扫码或验证码登录后自动保存并同步到全部成员脚本;"
"受影响的采集步骤本次已跳过,登录完成后下次运行自动恢复)"
)
def _now_iso(now: Callable[[], datetime]) -> str:
return now().astimezone(timezone.utc).isoformat(timespec="seconds")
def _env_int(
environment: Mapping[str, str],
name: str,
default: int,
*,
minimum: int,
) -> int:
raw = environment.get(name, "").strip()
try:
value = int(raw)
except (TypeError, ValueError):
return default
return max(minimum, value)
def _env_float(
environment: Mapping[str, str],
name: str,
default: float,
*,
minimum: float,
) -> float:
raw = environment.get(name, "").strip()
try:
value = float(raw)
except (TypeError, ValueError):
return default
return max(minimum, value)
__all__ = [
"DEFAULT_LOGIN_ALERT_MOBILE",
"GATE_MODE_BLOCKING",
"GATE_MODE_DEFER",
"GATE_MODE_RECOVER",
"GYXX_COOKIE_GATE",
"GYXX_LOGIN_ALERT_OPEN_IDS",
"CookieGate",
"CookieGateRecoveryManager",
"CookieGateStepAction",
"GateVerdict",
"apply_cookie_gates",
"gate_enabled",
]
@@ -1,4 +1,4 @@
"""Content metrics, creator, comment, report, and login workflows."""
"""Content metrics, creator, comment, and report workflows."""
from __future__ import annotations
@@ -14,7 +14,6 @@ CONTENT_SCHEDULED_WORKFLOW_IDS = (
"content.metrics.self_operated.daily",
"content.metrics.backfill",
"content.marketing_report.daily",
"content.relogin.weekly",
"content.creator_report.monthly",
"content.summary.monthly",
"content.notes_master.daily",
@@ -28,7 +27,6 @@ CONTENT_COMMENT_TIMEOUT_SECONDS = 6 * 60 * 60
# weakening the module-wide write lock.
CONTENT_NOTES_MASTER_LOCK_TIMEOUT_SECONDS = 6 * 60 * 60
CONTENT_RESOURCE = "module:content_marketing"
_RELOGIN_WORKFLOW_ID = "content.relogin.weekly"
_PARALLEL_COMMENT_WORKFLOW_ID = "content.comments.weekly"
# Keep both long-running daily metrics graphs on workflow-scoped resources so
# their collection steps do not block unrelated content workflows.
@@ -94,7 +92,6 @@ class ContentMarketingModule:
),
resource=workflow_resource,
command_factory=command_factory,
official_notification=workflow_id == _RELOGIN_WORKFLOW_ID,
independent_step_resources=(
workflow_id in {
_PARALLEL_COMMENT_WORKFLOW_ID,
@@ -77,23 +77,15 @@ def is_logged_in(page: Page) -> bool:
def maybe_wait_for_login(page: Page, login_timeout: int) -> None:
"""检测登录态,未登录就等用户扫码。B 站登录页 URL 含 passport.bilibili.com/login。"""
"""检测登录态;未登录立即降级为游客态,不再等待人工扫码。
登录统一由账号 vault 管理(gyxx accounts login bilibili-content),
由工作流执行前的 cookie 网关保证登录态。
"""
if is_logged_in(page):
return
log("B 站未登录 (无 SESSDATA cookie)。请在弹出的浏览器窗口扫码登录。")
# 跳到登录页,确保用户能看到二维码
try:
page.goto("https://passport.bilibili.com/login", wait_until="domcontentloaded", timeout=30000)
except Exception as exc:
log(f" 跳登录页失败 (继续等当前页扫码): {exc}")
deadline = time.time() + login_timeout
while time.time() < deadline:
if is_logged_in(page):
log("登录成功,SESSDATA 已写入")
save_state(page)
return
page.wait_for_timeout(2000)
log(f" [WARN] 等待 {login_timeout}s 仍未登录,继续以游客态采集 (评论只会拿到热门 3-5 条)")
log(" [WARN] B 站未登录 (无 SESSDATA cookie),降级为游客态采集 (评论只会拿到热门 3-5 条)。"
"如需全量评论请执行 gyxx accounts login bilibili-content")
def safe_filename(value: str) -> str:
@@ -437,48 +429,17 @@ def write_outputs(url: str, comments: list[dict[str, Any]], result: dict[str, An
return json_path, csv_path
def login_only(login_timeout: int) -> int:
"""只触发登录(headed),登录完成立刻退出。给 relogin_bilibili.py 用。"""
def action(page: Page) -> None:
page.goto("https://passport.bilibili.com/login", wait_until="domcontentloaded", timeout=30000)
page.wait_for_timeout(1500)
maybe_wait_for_login(page, login_timeout)
PROFILE_DIR.mkdir(parents=True, exist_ok=True)
with DynamicSession(
headless=False,
real_chrome=True,
user_data_dir=str(PROFILE_DIR),
locale="zh-CN",
timezone_id="Asia/Shanghai",
timeout=90000,
network_idle=False,
disable_resources=False,
google_search=False,
page_setup=restore_cookies,
max_pages=1,
) as session:
session.fetch("https://passport.bilibili.com/login", page_action=action, wait=1000)
log("[login-only] 完成")
return 0
def main() -> int:
parser = argparse.ArgumentParser(description="Scrape Bilibili video comments with a real browser and API.")
parser.add_argument("url", help="Bilibili video URL (传 dummy 'login' 配合 --login-only 也可)")
parser.add_argument("url", help="Bilibili video URL")
parser.add_argument("--max-reply-pages", type=int, default=200, help="max sub-reply pages per top-level comment")
parser.add_argument("--max-pages", type=int, default=200, help="最多抓取一级评论页数")
parser.add_argument("--delay", type=float, default=0.25, help="接口请求间隔秒数")
parser.add_argument("--headless", action="store_true", help="无头模式")
parser.add_argument("--login-timeout", type=int, default=0,
help="未登录时等待扫码秒数(>0 启用登录检测;默认 0 沿用游客态)")
parser.add_argument("--login-only", action="store_true",
help="只触发登录(headed)然后退出,给 relogin_bilibili 用")
help="保留参数(登录等待已移除;未登录将直接以游客态采集)")
args = parser.parse_args()
if args.login_only:
return login_only(args.login_timeout if args.login_timeout > 0 else 300)
comments, result = scrape_comments(
args.url, args.max_pages, args.delay, args.headless,
args.max_reply_pages, args.login_timeout,
@@ -52,6 +52,11 @@ REFRESH_WAIT_SECONDS = int(os.getenv("CHANMAMA_REFRESH_WAIT_SECONDS", "600"))
CHANMAMA_LIST_READY_TIMEOUT_SECONDS = int(
os.getenv("CHANMAMA_LIST_READY_TIMEOUT_SECONDS", "45")
)
# 公告/活动浮层可能盖住视频列表导致渲染判定一直不满足;渲染等待期间按
# 该间隔无交互清理浮层(派发 ESC + 移除 fixed/absolute 浮层节点)。
CHANMAMA_OVERLAY_DISMISS_INTERVAL_SECONDS = float(
os.getenv("CHANMAMA_OVERLAY_DISMISS_INTERVAL_SECONDS", "10")
)
# 蝉妈妈导出接口实测可能在 15 分钟边界后才把 Excel 写入下载目录;把轮询
# 上限做成可配置,并默认留出 20 分钟,避免文件已生成却被脚本判定为失败。
CHANMAMA_EXPORT_WAIT_SECONDS = int(
@@ -99,16 +104,20 @@ def create_driver():
profile_dir = str(PATHS.browser_profile_dir)
os.makedirs(profile_dir, exist_ok=True)
cdp_url = os.getenv("GYXX_BROWSER_CDP_URL") or None
# 蝉妈妈是国内站且其 WAF 会拉黑代理出口(Clash 节点 TLS 直接 reset,
# 导出 API 全断、确认弹窗永远不弹);强制浏览器直连。
direct_flags = ["--no-proxy-server"]
# StealthySession(隐藏指纹补丁)下蝉妈妈前端会静默压制功能: 页面能开
# 但导出确认弹窗不渲染(只剩空壳遮罩)。真 Chrome + 非 stealthy + 直连
# 已实测弹窗完整渲染(含取消/导出按钮),不再使用 stealthy。
try:
return ScraplingDriver.start(
profile_dir=profile_dir,
download_dir=DOWNLOAD_DIR,
cdp_url=cdp_url,
stealthy=True,
# 蝉妈妈新版 Vue 页面与 StealthySession + 系统 Chrome 组合
# 不兼容:HTML 能返回,但 #app 不会挂载。使用 Scrapling
# 自带 Chromium 仍保留隐藏指纹会话,且不影响已连接的 CDP。
**({"real_chrome": False} if cdp_url is None else {}),
stealthy=False,
real_chrome=True,
extra_flags=direct_flags,
)
except Exception as exc:
if (
@@ -125,8 +134,9 @@ def create_driver():
profile_dir=profile_dir,
download_dir=DOWNLOAD_DIR,
cdp_url=None,
stealthy=True,
real_chrome=False,
stealthy=False,
real_chrome=True,
extra_flags=direct_flags,
)
@@ -297,7 +307,10 @@ def handle_captcha(driver) -> bool:
def verify_login(driver):
"""验证是否已登录
通过访问目标页面,检查是否出现登录相关的元素
通过访问目标页面,检查是否出现登录相关的元素。
cookie 失效时蝉妈妈会把目标页重定向到 login.html,但重定向常在
首屏渲染后数秒才发生——等待 URL 稳定后再判定,避免误判已登录
(误判会跳过重新登录,导致后续导航一直超时)。
"""
try:
# 跳转到目标页面
@@ -305,10 +318,12 @@ def verify_login(driver):
WebDriverWait(driver, 12).until(
EC.presence_of_element_located((By.TAG_NAME, "body"))
)
time.sleep(3)
# cookie 失效时的重定向在首屏渲染后 3~20 秒才发生,固定等满 10 秒
# 再判定,不能因 URL 短暂未变而提前判为已登录
time.sleep(10)
cur_lower = driver.current_url.lower()
# 检查是否跳转回登录页 / 注册页(cookie 失效时蝉妈妈会重定向到 login.html 或 register.html)
cur_lower = driver.current_url.lower()
if "login" in cur_lower or "register" in cur_lower:
print(f"🔒 检测到被重定向到登录/注册页 ({driver.current_url}),cookie 已失效")
return False
@@ -458,6 +473,180 @@ def login(driver):
return False
def _dismiss_overlays(driver) -> int:
"""无交互清理遮挡浮层:派发 ESC 并移除 fixed/absolute 定位的弹层。"""
try:
removed = int(driver.execute_script(_OVERLAY_DISMISS_SCRIPT) or 0)
except Exception as error:
print(f"⚠️ 浮层清理脚本执行失败(忽略): {error}")
return 0
if removed:
print(f"🧹 已清理 {removed} 个遮挡浮层")
return removed
def _dismiss_export_stage_overlays(driver) -> int:
"""导出阶段清理弹窗(ESC + 移除弹层节点)。
只在弹窗不可确认时调用;返回移除的弹窗数。
"""
try:
removed = int(driver.execute_script(_EXPORT_STAGE_DISMISS_SCRIPT) or 0)
except Exception as error:
print(f"⚠️ 导出阶段弹窗清理脚本执行失败(忽略): {error}")
return 0
if removed > 0:
print(f"🧹 导出阶段清理 {removed} 个遮挡弹窗")
return removed
def _export_dialog_state(driver) -> str:
"""探测导出阶段弹窗状态: confirmable / present / none。"""
try:
state = driver.execute_script(_EXPORT_DIALOG_STATE_SCRIPT)
except Exception:
return "none"
return state if state in {"confirmable", "present", "none"} else "none"
def _log_export_dialog_texts(driver) -> None:
"""把当前可见弹窗的类名与文字打进日志,用于定位弹窗身份。"""
try:
texts = driver.execute_script(_EXPORT_DIALOG_TEXT_SCRIPT) or []
except Exception:
return
for i, text in enumerate(texts[:4], 1):
print(f"🪟 弹窗{i}内容: {text}")
def _click_export_confirm(driver) -> bool:
"""点击导出确认弹窗中的【导出】按钮,成功返回 True。"""
# 关键:用 contains(.,'导出') 匹配后代文本("导出"在 <span> 内)
confirm_selectors = [
# 精确:主按钮(primary) + 含"导出"文字
"//div[contains(@class,'el-dialog') or @role='dialog']//button[contains(.,'导出')]",
# 弹窗内任意含"导出"的按钮
"//div[contains(@class,'el-dialog') or @role='dialog']//button[contains(@class,'el-button--primary')][contains(.,'导出')]",
# 兼容"确定"/"确认"
"//div[contains(@class,'el-dialog') or @role='dialog']//button[contains(.,'确定')]",
"//div[contains(@class,'el-dialog') or @role='dialog']//button[contains(.,'确认')]",
]
for sel in confirm_selectors:
try:
btn = driver.find_element(By.XPATH, sel)
if btn.is_displayed():
# 原生点击优先(可触发内层元素的 Vue @click),失败再退 JS 合成点击
try:
btn.click()
except Exception:
driver.execute_script("arguments[0].click();", btn)
print(f"✅ 已在弹窗中点击导出按钮: {sel}")
return True
except Exception:
continue
return False
def _reclick_export_control(driver) -> bool:
"""重新触发一次导出按钮(第一次点击可能已被弹窗吞掉)。
优先 Playwright 原生点击(在元素命中点派发可信鼠标事件,能触发绑定
在按钮内层 span/div 上的 Vue @click);JS 合成 click 的目标是按钮自身,
内层监听器不会触发——蝉妈妈改版后导出点击无声失效就是这个原因。
"""
if _native_click_export_button(driver):
return True
try:
return bool(driver.execute_script(_CLICK_EXPORT_CONTROL_SCRIPT))
except Exception:
return False
_EXPORT_BUTTON_XPATHS = [
"//button[contains(.,'导出数据')]",
"//button[contains(.,'批量导出')]",
"//button[contains(.,'导出视频')]",
"//*[contains(@class,'export') and (self::button or self::a or @role='button')]",
]
def _native_click_export_button(driver) -> bool:
"""用原生可信点击触发导出按钮,成功返回 True。"""
for xp in _EXPORT_BUTTON_XPATHS:
try:
elements = driver.find_elements(By.XPATH, xp)
except Exception:
continue
for el in elements:
try:
if el.is_displayed():
el.click()
return True
except Exception:
continue
return False
def _wait_video_list_ready(driver, timeout_seconds: float) -> None:
"""等待视频列表渲染就绪,期间周期性清理遮挡浮层。
语义与 WebDriverWait.until 保持一致:浏览器侧的暂时性异常只记录
不中断,到达 deadline 后抛 TimeoutException。
"""
timeout = max(0.0, float(timeout_seconds))
deadline = time.monotonic() + timeout
next_dismiss_at = 0.0
last_error: Exception | None = None
while True:
try:
if driver.execute_script(_VIDEO_LIST_READY_SCRIPT):
return
except Exception as error:
last_error = error
now = time.monotonic()
if now >= deadline:
detail = f": {last_error}" if last_error else ""
raise TimeoutException(
f"condition was not met within {timeout}s{detail}"
) from last_error
if now >= next_dismiss_at:
_dismiss_overlays(driver)
next_dismiss_at = now + CHANMAMA_OVERLAY_DISMISS_INTERVAL_SECONDS
time.sleep(0.5)
def _dump_render_failure(driver, url: str) -> None:
"""渲染失败时落截图与页面 HTML,供事后定位是浮层/验证码还是改版。"""
try:
stamp = datetime.now().strftime("%Y%m%d_%H%M%S")
diag_dir = Path(DOWNLOAD_DIR) / "_diag"
diag_dir.mkdir(parents=True, exist_ok=True)
screenshot = diag_dir / f"render_failure_{stamp}.png"
driver.save_screenshot(str(screenshot))
(diag_dir / f"render_failure_{stamp}.html").write_text(
driver.page_source, encoding="utf-8"
)
print(f"🩺 渲染失败诊断已保存: {screenshot}")
except Exception as error:
print(f"⚠️ 渲染失败诊断保存失败: {error}")
def _dump_export_failure(driver) -> None:
"""导出失败时落截图与页面 HTML,供事后定位弹窗/按钮的实际形态。"""
try:
stamp = datetime.now().strftime("%Y%m%d_%H%M%S")
diag_dir = Path(DOWNLOAD_DIR) / "_diag"
diag_dir.mkdir(parents=True, exist_ok=True)
screenshot = diag_dir / f"export_failure_{stamp}.png"
driver.save_screenshot(str(screenshot))
(diag_dir / f"export_failure_{stamp}.html").write_text(
driver.page_source, encoding="utf-8"
)
print(f"🩺 导出失败诊断已保存: {screenshot}")
except Exception as error:
print(f"⚠️ 导出失败诊断保存失败: {error}")
def navigate_to_target(driver, url):
"""跳转到目标博主视频页面"""
last_error = None
@@ -475,12 +664,11 @@ def navigate_to_target(driver, url):
)
time.sleep(3) # 等待页面数据加载
# 截图中的页面虽然 URL 正确,但正文尚未渲染;在交给导出逻辑
# 前确认虚拟视频列表已经出现,空白页则有限重载一次。
WebDriverWait(driver, CHANMAMA_LIST_READY_TIMEOUT_SECONDS).until(
lambda current: bool(
current.execute_script(_VIDEO_LIST_READY_SCRIPT)
)
)
# 前确认虚拟视频列表已经出现,空白页则有限重载一次。浮层多在
# 首屏挂载后才弹出:进等待前先清一轮,等待期间按间隔再清,
# 避免延迟弹出的公告/活动浮层永久挡住渲染判定。
_dismiss_overlays(driver)
_wait_video_list_ready(driver, CHANMAMA_LIST_READY_TIMEOUT_SECONDS)
print(f"✅ 已到达并完成渲染目标页面: {current_url}")
return True
except Exception as exc:
@@ -499,6 +687,7 @@ def navigate_to_target(driver, url):
except Exception as reset_exc:
print(f"⚠️ 蝉妈妈页面重置失败,继续重试目标页: {reset_exc}")
time.sleep(1)
_dump_render_failure(driver, url)
raise RuntimeError(f"蝉妈妈目标页加载失败: {url}; {last_error}") from last_error
@@ -539,6 +728,206 @@ def _find_video_table(driver):
return WebDriverWait(driver, 15).until(locate)
# 蝉妈妈公告/活动弹窗以 fixed/absolute 浮层盖住页面,且类名不稳定、未必
# 有关闭按钮。这里不点叉:派发 ESC 键事件(多数组件库监听 document 关闭
# 弹窗),再移除定位浮层节点并解除其对 html/body 的滚动锁;已知类名未
# 命中时兜底移除 z-index 高且覆盖大半屏的 fixed 层。列表区
# (.aweme-record-wrapper/.list-box/.list-auth-wrap-block) 内的元素永不命中。
# .v-modal 是 ElementUI 直接挂在 <body> 下的灰色遮罩,与弹窗本体分离,
# 必须显式清理,否则弹窗删掉后页面仍然灰屏、无法滚动;滚动锁 class
# (el-popup-parent--hidden 等)同理。
# 关键约束:只移除"可见"的浮层。页面常驻 90+ 个 display:none 的隐藏弹窗
# 模板(含导出确认框的模板)——删掉它们后 Vue 只会在游离节点上"打开"弹窗,
# 页面上永远渲染不出来(导出只剩灰遮罩),因此 display:none/零尺寸节点
# 一律跳过。
_OVERLAY_DISMISS_SCRIPT = r"""
() => {
const esc = new KeyboardEvent('keydown', {
key: 'Escape', code: 'Escape', keyCode: 27, which: 27, bubbles: true,
});
document.dispatchEvent(esc);
window.dispatchEvent(esc);
let removed = 0;
const seen = new Set();
const isOverlay = (el) => {
const style = window.getComputedStyle(el);
return style.position === 'fixed' || style.position === 'absolute';
};
const drop = (el) => {
if (seen.has(el) || !isOverlay(el)) return;
const style = window.getComputedStyle(el);
if (style.display === 'none') return;
const rect = el.getBoundingClientRect();
if (rect.width === 0 || rect.height === 0) return;
if (el.closest('.aweme-record-wrapper,.list-box,.list-auth-wrap-block')) return;
seen.add(el);
el.remove();
removed += 1;
};
const selectors = [
'.el-dialog__wrapper', '.el-overlay', '.el-message-box__wrapper',
'.ant-modal-mask', '.ant-modal-wrap', '.modal-mask', '.modal-dialog',
'.v-modal', '.v-modal-inner',
'[class*="popup"]', '[class*="dialog"]', '[class*="overlay"]', '[class*="mask"]',
];
for (const selector of selectors) {
document.querySelectorAll(selector).forEach(drop);
}
if (removed === 0) {
document.querySelectorAll('body *').forEach((el) => {
const style = window.getComputedStyle(el);
const zIndex = Number.parseInt(style.zIndex, 10);
const rect = el.getBoundingClientRect();
if (Number.isFinite(zIndex) && zIndex > 500
&& rect.height > window.innerHeight * 0.5) {
drop(el);
}
});
}
document.body.classList.remove(
'el-popup-parent--hidden', 'el-popup-parent--overflow-hidden', 'overflow-hidden'
);
document.documentElement.classList.remove(
'el-popup-parent--hidden', 'el-popup-parent--overflow-hidden', 'overflow-hidden'
);
document.documentElement.style.overflow = '';
document.body.style.overflow = '';
document.body.style.paddingRight = '';
return removed;
}
"""
# 导出阶段专用清理:与 _OVERLAY_DISMISS_SCRIPT 同一套清理思路,但
# 调用时机由 _export_dialog_state 决定——只在弹窗"没有可点的确认按钮"时
# 才执行(确认框可点时直接点导出,绝不清)。三件事:
# 1) 先 ESC + 点弹窗自带的关闭(×)按钮,走框架的正常关闭路径;
# 2) 再移除残留的弹层节点——必须包含 .v-modal:ElementUI 的灰色遮罩直接
# 挂在 <body> 下、与弹窗本体分离,漏掉它就是"删了弹窗页面仍灰屏";
# 3) 最后解除滚动锁:清掉 body/html 上的 el-popup-parent--hidden 等
# 框架加的 overflow 锁 class 与内联样式,否则页面删完遮罩也不能滚动。
_EXPORT_STAGE_DISMISS_SCRIPT = r"""
() => {
const visible = (el) => {
const style = window.getComputedStyle(el);
const rect = el.getBoundingClientRect();
return style.display !== 'none' && style.visibility !== 'hidden'
&& style.opacity !== '0' && rect.width > 0 && rect.height > 0;
};
const esc = new KeyboardEvent('keydown', {
key: 'Escape', code: 'Escape', keyCode: 27, which: 27, bubbles: true,
});
document.dispatchEvent(esc);
window.dispatchEvent(esc);
// 先走正常关闭:点弹窗自己的 × / 关闭按钮
let closed = 0;
document.querySelectorAll(
'.el-dialog__wrapper [class*="close"],.el-message-box__wrapper [class*="close"],'
+ '.ant-modal-wrap [class*="close"],[class*="dialog"] [class*="close"],'
+ '[class*="modal"] [class*="close"],[class*="popup"] [class*="close"]'
).forEach((el) => {
if (visible(el)) { el.click(); closed += 1; }
});
let removed = 0;
const seen = new Set();
const drop = (el) => {
if (seen.has(el)) return;
const style = window.getComputedStyle(el);
if (style.position !== 'fixed' && style.position !== 'absolute') return;
if (style.display === 'none') return;
const rect = el.getBoundingClientRect();
if (rect.width === 0 || rect.height === 0) return;
if (el.closest('.aweme-record-wrapper,.list-box,.list-auth-wrap-block')) return;
seen.add(el);
el.remove();
removed += 1;
};
const selectors = [
'.el-dialog__wrapper', '.el-overlay', '.el-message-box__wrapper',
'.ant-modal-mask', '.ant-modal-wrap', '.modal-mask', '.modal-dialog',
'.v-modal', '.v-modal-inner',
'[class*="popup"]', '[class*="dialog"]', '[class*="overlay"]', '[class*="mask"]',
];
for (const selector of selectors) {
document.querySelectorAll(selector).forEach(drop);
}
if (removed === 0) {
document.querySelectorAll('body *').forEach((el) => {
const style = window.getComputedStyle(el);
const zIndex = Number.parseInt(style.zIndex, 10);
const rect = el.getBoundingClientRect();
if (Number.isFinite(zIndex) && zIndex > 500
&& rect.height > window.innerHeight * 0.5) {
drop(el);
}
});
}
// 解除滚动锁:框架加的锁 class + 内联 overflow 一起清
document.body.classList.remove(
'el-popup-parent--hidden', 'el-popup-parent--overflow-hidden', 'overflow-hidden'
);
document.documentElement.classList.remove(
'el-popup-parent--hidden', 'el-popup-parent--overflow-hidden', 'overflow-hidden'
);
document.documentElement.style.overflow = '';
document.body.style.overflow = '';
document.body.style.paddingRight = '';
return removed + closed;
}
"""
# 导出阶段弹窗状态机:'confirmable'=有可见弹窗且弹窗里有可点的
# 导出/确定/确认按钮(直接点,不能清);'present'=弹窗可见但没有可点
# 的确认按钮(空壳遮罩/公告,ESC 关不掉就强删);'none'=没有弹窗。
_EXPORT_DIALOG_STATE_SCRIPT = r"""
() => {
const visible = (el) => {
const style = window.getComputedStyle(el);
const rect = el.getBoundingClientRect();
return style.display !== 'none' && style.visibility !== 'hidden'
&& style.opacity !== '0' && rect.width > 0 && rect.height > 0;
};
const roots = [...document.querySelectorAll(
'.el-dialog__wrapper,.el-message-box__wrapper,.el-overlay,'
+ '.ant-modal-wrap,.ant-modal-mask,[role="dialog"],'
+ '[class*="dialog"],[class*="modal"],[class*="popup"]'
)].filter(visible);
for (const root of roots) {
const confirmable = [...root.querySelectorAll(
'button,a,[role="button"],[class*="confirm"],[class*="primary"]'
)].some((el) => {
if (!visible(el)) return false;
const text = (el.innerText || el.value || '').replace(/\s+/g, '');
return /导出|确定|确认/.test(text);
});
if (confirmable) return 'confirmable';
}
return roots.length ? 'present' : 'none';
}
"""
# 抓取当前可见弹窗的文字内容(每窗截前 160 字),写进日志用于定位弹窗身份。
_EXPORT_DIALOG_TEXT_SCRIPT = r"""
() => {
const visible = (el) => {
const style = window.getComputedStyle(el);
const rect = el.getBoundingClientRect();
return style.display !== 'none' && style.visibility !== 'hidden'
&& style.opacity !== '0' && rect.width > 0 && rect.height > 0;
};
return [...document.querySelectorAll(
'.el-dialog__wrapper,.el-message-box__wrapper,.el-overlay,'
+ '.ant-modal-wrap,[role="dialog"],[class*="dialog"],[class*="modal"],[class*="popup"]'
)].filter(visible).map((el) => {
const text = (el.innerText || '').replace(/\s+/g, ' ').trim();
const classes = String(el.className || '').slice(0, 80);
return `[${classes}] ${text.slice(0, 160)}`;
});
}
"""
_VIDEO_LIST_READY_SCRIPT = r"""
() => {
const visible = (el) => {
@@ -909,8 +1298,39 @@ def _max_excel_mtime():
def refresh_then_export_accounts(driver, urls, wait_seconds=REFRESH_WAIT_SECONDS):
"""两阶段执行:先刷新所有账号全部分页,再等待并重新导出。"""
"""两阶段执行:先刷新所有账号全部分页,再等待并重新导出。
运行中途登录态失效(目标页被重定向到登录页)时,自动用账号密码重新
登录并重试当前账号一次;整套运行最多自动重登一次,避免死循环。
"""
refresh_summaries = []
relogin_state = {"attempted": False}
def _relogin_if_expired(exc: BaseException) -> bool:
"""登录态失效类异常时自动重登一次;返回是否已恢复登录。"""
if relogin_state["attempted"] or "登录态失效" not in str(exc):
return False
relogin_state["attempted"] = True
print("🔐 检测到登录态失效,尝试账号密码自动重新登录...")
try:
ok = login(driver)
except Exception as login_exc:
print(f"⚠️ 自动重新登录异常: {login_exc}")
return False
if ok:
print("✅ 自动重新登录成功,重试当前账号")
return bool(ok)
def _empty_summary(exc: BaseException) -> dict:
return {
"clicked": 0,
"pending": 0,
"zero_without_button": 0,
"unconfirmed": 0,
"pages": 0,
"error": str(exc),
}
for index, url in enumerate(urls, 1):
print(f"\n{'=' * 50}")
print(f"🔄 刷新阶段 博主 {index}/{len(urls)}: {url}")
@@ -919,15 +1339,16 @@ def refresh_then_export_accounts(driver, urls, wait_seconds=REFRESH_WAIT_SECONDS
navigate_to_target(driver, url)
summary = refresh_zero_exposure_videos(driver)
except Exception as exc:
if _relogin_if_expired(exc):
try:
navigate_to_target(driver, url)
summary = refresh_zero_exposure_videos(driver)
except Exception as retry_exc:
print(f"⚠️ 博主 {index} 重登后刷新仍失败: {retry_exc},继续下一个")
summary = _empty_summary(retry_exc)
else:
print(f"⚠️ 博主 {index} 刷新扫描异常: {exc},继续下一个")
summary = {
"clicked": 0,
"pending": 0,
"zero_without_button": 0,
"unconfirmed": 0,
"pages": 0,
"error": str(exc),
}
summary = _empty_summary(exc)
refresh_summaries.append(summary)
should_wait = any(
@@ -956,6 +1377,20 @@ def refresh_then_export_accounts(driver, urls, wait_seconds=REFRESH_WAIT_SECONDS
print(f" 解析出 {len(records)} 条视频记录")
all_records.extend(records)
except Exception as exc:
if _relogin_if_expired(exc):
try:
navigate_to_target(driver, url)
pre_mtime = _max_excel_mtime()
excel_path = export_video_data(driver, pre_mtime)
if not excel_path:
print(f"⚠️ 博主 {index} 重登后导出失败,跳过")
continue
records, _ = parse_chanmama_export(excel_path)
print(f" 解析出 {len(records)} 条视频记录(重登后)")
all_records.extend(records)
except Exception as retry_exc:
print(f"⚠️ 博主 {index} 重登后导出仍失败: {retry_exc},跳过")
else:
print(f"⚠️ 博主 {index} 导出异常: {exc},跳过")
return all_records, refresh_summaries
@@ -978,6 +1413,10 @@ def export_video_data(driver, pre_mtime: float = 0):
except TimeoutException:
print("⚠️ 视频列表加载超时,继续尝试导出...")
# 刷新/翻页阶段可能延迟弹出公告/活动弹窗,吞掉导出点击或盖住确认框;
# 点导出按钮前先清一次(确认框受保护)。
_dismiss_export_stage_overlays(driver)
# 方法1: 先在可见 DOM 中找真正的交互节点。新版按钮文字常在嵌套
# span/div 内,旧的 contains(text()) 和第一个隐藏 locator 都会漏掉它。
export_clicked = False
@@ -989,6 +1428,9 @@ def export_video_data(driver, pre_mtime: float = 0):
except TimeoutException:
control = None
if control:
# 原生点击优先(可触发内层元素的 Vue @click),JS 合成点击兜底
export_clicked = _native_click_export_button(driver)
if not export_clicked:
clicked = driver.execute_script(_CLICK_EXPORT_CONTROL_SCRIPT)
export_clicked = bool(clicked)
if export_clicked:
@@ -1032,36 +1474,38 @@ def export_video_data(driver, pre_mtime: float = 0):
return scrape_table_data(driver)
# 等待导出确认弹窗出现并点击【导出】按钮
# 蝉妈妈弹窗标题"您即将进行数据导出",按钮为 橙色"导出"(文字在 <span> 内)
# 蝉妈妈弹窗标题"您即将进行数据导出",按钮为 橙色"导出"(文字在 <span> 内)。
# 逐秒探测弹窗状态:可确认→立即点导出;挂着但没有可点的确认按钮
# (空壳遮罩/公告弹窗)→先 ESC+移除正常关,持续 5 秒以上就补点导出按钮
# (第一次点击可能已被弹窗吞掉);没有弹窗→常规清理并定期补点导出。
print("⏳ 等待导出确认弹窗...")
export_confirmed = False
for wait_sec in range(20):
unconfirmable_since: float | None = None
for wait_sec in range(36):
time.sleep(1)
try:
# 关键:用 contains(.,'导出') 匹配后代文本("导出"在 <span> 内)
confirm_selectors = [
# 精确:主按钮(primary) + 含"导出"文字
"//div[contains(@class,'el-dialog') or @role='dialog']//button[contains(.,'导出')]",
# 弹窗内任意含"导出"的按钮
"//div[contains(@class,'el-dialog') or @role='dialog']//button[contains(@class,'el-button--primary')][contains(.,'导出')]",
# 兼容"确定"/"确认"
"//div[contains(@class,'el-dialog') or @role='dialog']//button[contains(.,'确定')]",
"//div[contains(@class,'el-dialog') or @role='dialog']//button[contains(.,'确认')]",
]
for sel in confirm_selectors:
try:
btn = driver.find_element(By.XPATH, sel)
if btn.is_displayed():
driver.execute_script("arguments[0].click();", btn)
state = _export_dialog_state(driver)
if state == "confirmable":
unconfirmable_since = None
if _click_export_confirm(driver):
export_confirmed = True
print(f"✅ 已在弹窗中点击导出按钮: {sel}")
break
except NoSuchElementException:
continue
if export_confirmed:
break
except Exception:
if state == "present":
now = time.monotonic()
if unconfirmable_since is None:
_log_export_dialog_texts(driver)
print("ℹ️ 弹窗可见但没有可点的确认按钮,尝试关闭")
unconfirmable_since = now
elif now - unconfirmable_since > 5:
if _reclick_export_control(driver):
print("🔁 清掉卡死弹窗后补点导出按钮")
unconfirmable_since = None
_dismiss_export_stage_overlays(driver)
continue
unconfirmable_since = None
_dismiss_export_stage_overlays(driver)
if wait_sec in (10, 20, 30) and _reclick_export_control(driver):
print("🔁 确认框迟迟未出现,补点导出按钮")
if not export_confirmed:
try:
@@ -1093,9 +1537,20 @@ def export_video_data(driver, pre_mtime: float = 0):
"⚠️ 未找到本次下载的 Excel文件(等待 "
f"{CHANMAMA_EXPORT_WAIT_SECONDS + max(CHANMAMA_EXPORT_GRACE_SECONDS, 0)} 秒超时)"
)
_dump_export_failure(driver)
return None
def _download_complete(path: str) -> bool:
"""判断下载文件是否已写完整(zip EOCD 在文件尾,截断时 is_zipfile 为假)。"""
import zipfile
try:
return zipfile.is_zipfile(path)
except OSError:
return False
def _wait_for_exported_excel(
pre_mtime: float,
*,
@@ -1125,7 +1580,13 @@ def _wait_for_exported_excel(
except Exception:
pass
new_path = _find_new_excel_after(pre_mtime)
if new_path is not None:
# 文件出现 ≠ 下载完成: 当前浏览器可能以最终名边下边写,截断的
# zip 会让解析报 "File is not a zip file"。zip 的 EOCD 记录在
# 文件末尾,is_zipfile 通过即认为下载已完整;路径不存在(测试桩)
# 时保持旧行为直接返回。
if new_path is not None and (
not os.path.exists(new_path) or _download_complete(new_path)
):
return new_path
now = time.time()
if now >= deadline and not grace_logged and grace_deadline > deadline:
@@ -18,7 +18,6 @@ from gyxx_flow.modules.content_marketing.runtime_paths import PATHS
_TOOLS_DIR = PATHS.tools_root
_PROJECT_ROOT = PATHS.module_root
RELOGIN_SCRIPT = _TOOLS_DIR / "relogin_bilibili.py"
try:
sys.stdout.reconfigure(encoding="utf-8", errors="replace") # type: ignore[attr-defined]
@@ -81,39 +80,6 @@ def comment_replace_policy(comments: list[dict], stats: dict, metrics: dict) ->
return False, False, "空评论且无法确认平台为 0 评论"
def do_relogin() -> bool:
"""Run relogin_bilibili.py (headed, waits for QR scan). Returns True if successful."""
acceptance_policy = current_acceptance_policy()
if acceptance_policy.enabled:
acceptance_policy.record(
"cookie_skipped",
operation="content.batch_rescrape_bilibili.relogin",
details={
"status": "SKIPPED_COOKIE",
"reason": "cookie is invalid and QR relogin is disabled for acceptance",
},
)
log(
" [SKIPPED_COOKIE] Cookie invalid; acceptance mode will not "
"launch relogin or wait for a QR scan"
)
return False
log(" Cookie 可能过期,触发自动重登 (请扫码)...")
try:
rc = subprocess.call(
[sys.executable, str(RELOGIN_SCRIPT)],
cwd=str(_PROJECT_ROOT),
)
if rc == 0:
log(" 重登成功,继续采集")
return True
else:
log(f" 重登失败 (exit={rc})")
return False
except Exception as exc:
log(f" 重登脚本执行异常: {exc}")
return False
def main(only_ids: list[int] | None = None, dry_run: bool = False) -> int:
rows = collect_urls()
@@ -126,8 +92,6 @@ def main(only_ids: list[int] | None = None, dry_run: bool = False) -> int:
total_comments = 0
consecutive_fail = 0
MAX_CONSECUTIVE_FAIL = 3
relogin_used = 0
MAX_RELOGINS = 1
t0 = time.time()
for i, row in enumerate(rows, 1):
note_id = row["id"]
@@ -152,11 +116,12 @@ def main(only_ids: list[int] | None = None, dry_run: bool = False) -> int:
break
if comments is None:
consecutive_fail += 1
if consecutive_fail >= MAX_CONSECUTIVE_FAIL and relogin_used < MAX_RELOGINS:
if do_relogin():
relogin_used += 1
consecutive_fail = 0
continue
if consecutive_fail >= MAX_CONSECUTIVE_FAIL:
# 登录统一由账号 vault 管理;连续失败只提示,不再触发扫码重登。
log(
" 连续失败达到上限;若为登录态失效请执行 "
"uv run gyxx accounts login bilibili-content"
)
if current_acceptance_policy().enabled:
return COOKIE_SKIP_EXIT_CODE
continue
@@ -26,7 +26,6 @@ from gyxx_flow.modules.content_marketing.serialized_page_action import (
_TOOLS_DIR = PATHS.tools_root
_PROJECT_ROOT = PATHS.module_root
RELOGIN_SCRIPT = _TOOLS_DIR / "relogin_douyin.py"
try:
sys.stdout.reconfigure(encoding="utf-8", errors="replace") # type: ignore[attr-defined]
@@ -104,38 +103,6 @@ def result_logged_in(result: dict | None) -> bool:
return bool(nested_stats.get("logged_in")) if isinstance(nested_stats, dict) else False
def do_relogin() -> bool:
"""Run relogin_douyin.py (headed, waits for QR scan). Returns True if successful."""
acceptance_policy = current_acceptance_policy()
if acceptance_policy.enabled:
acceptance_policy.record(
"cookie_skipped",
operation="content.batch_rescrape_douyin.relogin",
details={
"status": "SKIPPED_COOKIE",
"reason": "cookie is invalid and QR relogin is disabled for acceptance",
},
)
log(
" [SKIPPED_COOKIE] Cookie invalid; acceptance mode will not "
"launch relogin or wait for a QR scan"
)
return False
log(" Cookie 可能过期,触发自动重登 (请扫码)...")
try:
rc = subprocess.call(
[sys.executable, str(RELOGIN_SCRIPT)],
cwd=str(_PROJECT_ROOT),
)
if rc == 0:
log(" 重登成功,继续采集")
return True
log(f" 重登失败 (exit={rc})")
return False
except Exception as exc:
log(f" 重登脚本执行异常: {exc}")
return False
def scrape_with_fresh_browser(
url: str,
@@ -203,8 +170,6 @@ def main(only_ids: list[int] | None = None, dry_run: bool = False) -> int:
COOLDOWN_SECONDS = 300
cooldown_used = 0
MAX_COOLDOWNS = 3
relogin_used = 0
MAX_RELOGINS = 1 # 每轮 batch 最多自动重登1次
t0 = time.time()
for i, row in enumerate(rows, 1):
note_id = row["id"]
@@ -214,14 +179,8 @@ def main(only_ids: list[int] | None = None, dry_run: bool = False) -> int:
if comments is None:
consecutive_fail += 1
# 连续失败 + 可能 cookie 过期 → 尝试自动重登
if consecutive_fail >= MAX_CONSECUTIVE_FAIL and relogin_used < MAX_RELOGINS:
if do_relogin():
relogin_used += 1
consecutive_fail = 0
comments, stats, scrape_error = scrape_with_fresh_browser(url)
else:
# 重登失败,走冷却逻辑;验收模式必须直接跳过,不能等待。
# 登录统一由账号 vault 管理;不再触发扫码重登,验收模式直接跳过。
if consecutive_fail >= MAX_CONSECUTIVE_FAIL:
if current_acceptance_policy().enabled:
return COOKIE_SKIP_EXIT_CODE
@@ -25,7 +25,6 @@ from gyxx_flow.modules.content_marketing.serialized_page_action import (
_TOOLS_DIR = PATHS.tools_root
_PROJECT_ROOT = PATHS.module_root
RELOGIN_SCRIPT = _TOOLS_DIR / "relogin_xiaohongshu.py"
try:
sys.stdout.reconfigure(encoding="utf-8", errors="replace") # type: ignore[attr-defined]
@@ -87,38 +86,6 @@ def comment_replace_policy(comments: list[dict], stats: dict, metrics: dict) ->
return False, False, "空评论且无法确认平台为 0 评论"
def do_relogin() -> bool:
"""Run relogin_xiaohongshu.py (headed, waits for QR scan). Returns True if successful."""
acceptance_policy = current_acceptance_policy()
if acceptance_policy.enabled:
acceptance_policy.record(
"cookie_skipped",
operation="content.batch_rescrape_xiaohongshu.relogin",
details={
"status": "SKIPPED_COOKIE",
"reason": "cookie is invalid and QR relogin is disabled for acceptance",
},
)
log(
" [SKIPPED_COOKIE] Cookie invalid; acceptance mode will not "
"launch relogin or wait for a QR scan"
)
return False
log(" Cookie 可能过期,触发自动重登 (请扫码)...")
try:
rc = subprocess.call(
[sys.executable, str(RELOGIN_SCRIPT)],
cwd=str(_PROJECT_ROOT),
)
if rc == 0:
log(" 重登成功,继续采集")
return True
log(f" 重登失败 (exit={rc})")
return False
except Exception as exc:
log(f" 重登脚本执行异常: {exc}")
return False
def scrape_with_fresh_browser(
url: str,
@@ -173,8 +140,6 @@ def main(only_ids: list[int] | None = None, dry_run: bool = False) -> int:
total_comments = 0
consecutive_fail = 0
MAX_CONSECUTIVE_FAIL = 3
relogin_used = 0
MAX_RELOGINS = 1
t0 = time.time()
for i, row in enumerate(rows, 1):
note_id = row["id"]
@@ -183,13 +148,13 @@ def main(only_ids: list[int] | None = None, dry_run: bool = False) -> int:
comments, result, scrape_error = scrape_with_fresh_browser(url)
if comments is None:
consecutive_fail += 1
if consecutive_fail >= MAX_CONSECUTIVE_FAIL and relogin_used < MAX_RELOGINS:
if do_relogin():
relogin_used += 1
consecutive_fail = 0
comments, result, scrape_error = scrape_with_fresh_browser(url)
if consecutive_fail >= MAX_CONSECUTIVE_FAIL:
if current_acceptance_policy().enabled:
return COOKIE_SKIP_EXIT_CODE
log(
" 连续失败达到上限;若为登录态失效请执行 "
"uv run gyxx accounts login xiaohongshu-content"
)
if comments is None:
log(f" ERROR: {scrape_error}")
fail += 1
@@ -1,666 +0,0 @@
"""Launch five QR relogins in parallel, notify via analyzer Lark, retry 3 times."""
from __future__ import annotations
import argparse
import ctypes
import datetime as dt
import os
import subprocess
import sys
import time
from ctypes import wintypes
from dataclasses import dataclass
from pathlib import Path
from typing import Callable
import psutil
from gyxx_flow.adapters import (
ANALYZER_NOTIFICATION_APP_PROFILE,
COOKIE_SKIP_EXIT_CODE,
ResolvedNotificationRoute,
binding_from_environment,
current_acceptance_policy,
environment_for_child_script,
resolve_notification_route,
send_lark_bot_message,
)
from gyxx_flow.modules.content_marketing.runtime_paths import PATHS
PROJECT_DIR = PATHS.module_root
ANALYZER_DIR = PATHS.tmp_root / "relogin"
WORKFLOW_ID = "content.relogin.weekly"
MARKETING_RECIPIENT = "ou_24cc944d6e43c69c59d6560ad4e2ae6e"
TECHNICAL_RECIPIENT = "ou_7ad5fc8012e2f741afc5346e05ffd447"
@dataclass(frozen=True)
class Platform:
label: str
command: tuple[str, ...]
required_cookie: str
@dataclass(frozen=True, slots=True)
class NotificationDeliveryFailure:
"""One failed recipient delivery that must affect the workflow outcome."""
stage: str
platform: str
recipient: str
detail: str
PLATFORMS = {
"douyin": Platform(
"\u6296\u97f3", ("data/tools/relogin_douyin.py",),
"sessionid",
),
"bilibili": Platform(
"B\u7ad9", ("data/tools/relogin_bilibili.py",),
"SESSDATA",
),
"pgy": Platform(
"\u84b2\u516c\u82f1", ("data/tools/relogin_pgy.py",),
"",
),
"xingtu": Platform(
"\u661f\u56fe", ("data/tools/relogin_xingtu.py", "--force"),
"sessionid",
),
"xiaohongshu": Platform(
"\u5c0f\u7ea2\u4e66", ("data/tools/relogin_xiaohongshu.py",),
"web_session",
),
}
DEFAULT_RECIPIENTS = {
"pgy": MARKETING_RECIPIENT,
"xingtu": MARKETING_RECIPIENT,
"douyin": TECHNICAL_RECIPIENT,
"bilibili": TECHNICAL_RECIPIENT,
"xiaohongshu": TECHNICAL_RECIPIENT,
}
def resolve_platform_names(requested: list[str] | None) -> list[str]:
"""Resolve an optional manual platform subset while preserving order."""
if not requested:
return list(PLATFORMS)
names: list[str] = []
for name in requested:
if name not in PLATFORMS:
raise ValueError(f"unknown platform: {name}")
if name not in names:
names.append(name)
return names
def recipient_for_platform(platform_name: str, override: str | None = None) -> str:
"""Return the responsible recipient, unless a manual all-platform override is set."""
from gyxx_flow.adapters.acceptance_policy import resolve_notification_recipients
if override:
recipient = override
else:
try:
recipient = DEFAULT_RECIPIENTS[platform_name]
except KeyError as exc:
raise ValueError(
f"no Lark recipient configured for platform: {platform_name}"
) from exc
return resolve_notification_recipients(
(recipient,),
app_profile=ANALYZER_NOTIFICATION_APP_PROFILE,
)[0]
def resolve_weekly_notification_route(
override: str | None = None,
) -> ResolvedNotificationRoute:
"""Resolve the scheduled workflow route while retaining legacy defaults."""
defaults = (
(override,)
if override
else tuple(dict.fromkeys(DEFAULT_RECIPIENTS.values()))
)
return resolve_notification_route(WORKFLOW_ID, defaults)
def recipients_for_platform(
platform_name: str,
route: ResolvedNotificationRoute,
override: str | None = None,
) -> tuple[str, ...]:
"""Return dynamic recipients, or the original per-platform recipient."""
if not route.enabled:
return ()
if route.configured:
return route.open_ids
return (recipient_for_platform(platform_name, override),)
def group_platforms_by_recipient(
platform_names: list[str], override: str | None = None,
) -> dict[str, list[str]]:
grouped: dict[str, list[str]] = {}
for name in platform_names:
grouped.setdefault(recipient_for_platform(name, override), []).append(name)
return grouped
def group_platforms_by_notification_recipient(
platform_names: list[str],
route: ResolvedNotificationRoute,
override: str | None = None,
) -> dict[str, list[str]]:
"""Group failed platforms for legacy or configured multi-recipient delivery."""
if not route.enabled:
return {}
if route.configured:
return {recipient: list(platform_names) for recipient in route.open_ids}
return group_platforms_by_recipient(platform_names, override)
def run_with_retries(
platform_names: list[str],
*,
max_attempts: int,
run_round: Callable[[list[str], int], dict[str, bool]],
) -> dict[str, bool]:
status = {name: False for name in platform_names}
pending = list(platform_names)
for attempt in range(1, max_attempts + 1):
if not pending:
break
round_result = run_round(pending, attempt)
for name in pending:
status[name] = bool(round_result.get(name))
pending = [name for name in pending if not status[name]]
return status
def _valid_cookie(platform: Platform, cookie_file: Path) -> bool:
if not cookie_file.is_file():
return False
if not platform.required_cookie:
return cookie_file.stat().st_size > 10
try:
import json
values = json.loads(cookie_file.read_text(encoding="utf-8"))
return any(
item.get("name") == platform.required_cookie and item.get("value")
for item in values if isinstance(item, dict)
)
except Exception:
return False
def _descendant_pids(pid: int) -> set[int]:
try:
return {child.pid for child in psutil.Process(pid).children(recursive=True)}
except (psutil.Error, OSError):
return set()
def _terminate_process_tree(process: subprocess.Popen) -> None:
"""Terminate the whole Windows subprocess tree, including Chrome/Scrapling."""
try:
subprocess.run(
["taskkill", "/PID", str(process.pid), "/T", "/F"],
capture_output=True,
text=True,
check=False,
)
except OSError:
process.terminate()
try:
process.wait(timeout=10)
except subprocess.TimeoutExpired:
process.kill()
def _window_for_process(pid: int) -> int | None:
pids = _descendant_pids(pid) | {pid}
handles: list[int] = []
user32 = ctypes.windll.user32
callback_type = ctypes.WINFUNCTYPE(ctypes.c_bool, ctypes.c_void_p, ctypes.c_void_p)
def callback(hwnd, _lparam):
process_id = ctypes.c_ulong()
user32.GetWindowThreadProcessId(hwnd, ctypes.byref(process_id))
if process_id.value in pids and user32.IsWindowVisible(hwnd):
handles.append(int(hwnd))
return True
user32.EnumWindows(callback_type(callback), 0)
return handles[0] if handles else None
def _platform_from_window_title(title: str) -> str | None:
"""Identify a login window without relying on Chrome's process tree."""
if "\u84b2\u516c\u82f1" in title:
return "pgy"
if "\u6296\u97f3\u6388\u6743" in title or "\u5de8\u91cf\u661f\u56fe" in title:
return "xingtu"
if "\u6296\u97f3" in title:
return "douyin"
if "bilibili" in title.lower() or "\u8d26\u53f7\u767b\u5f55" in title:
return "bilibili"
if "\u5c0f\u7ea2\u4e66" in title:
return "xiaohongshu"
return None
def _login_windows() -> dict[str, int]:
"""Map visible Chrome login windows to platforms by their page titles."""
user32 = ctypes.windll.user32
found: dict[str, int] = {}
callback_type = ctypes.WINFUNCTYPE(ctypes.c_bool, ctypes.c_void_p, ctypes.c_void_p)
def callback(hwnd, _lparam):
if not user32.IsWindowVisible(hwnd):
return True
length = user32.GetWindowTextLengthW(hwnd)
if length <= 0:
return True
buffer = ctypes.create_unicode_buffer(length + 1)
user32.GetWindowTextW(hwnd, buffer, length + 1)
platform = _platform_from_window_title(buffer.value)
if platform:
found[platform] = int(hwnd)
return True
user32.EnumWindows(callback_type(callback), 0)
return found
def _capture_window(hwnd: int, output: Path) -> None:
"""Maximize and render one window directly, even when it is not foreground."""
from PIL import Image
user32 = ctypes.windll.user32
gdi32 = ctypes.windll.gdi32
user32.ShowWindow(hwnd, 3) # SW_MAXIMIZE
time.sleep(1)
rect = wintypes.RECT()
user32.GetWindowRect(hwnd, ctypes.byref(rect))
width, height = rect.right - rect.left, rect.bottom - rect.top
if width <= 0 or height <= 0:
raise RuntimeError(f"invalid window size: {width}x{height}")
window_dc = user32.GetWindowDC(hwnd)
memory_dc = gdi32.CreateCompatibleDC(window_dc)
bitmap = gdi32.CreateCompatibleBitmap(window_dc, width, height)
previous = gdi32.SelectObject(memory_dc, bitmap)
try:
if not user32.PrintWindow(hwnd, memory_dc, 2): # PW_RENDERFULLCONTENT
raise RuntimeError("PrintWindow failed")
class BitmapInfoHeader(ctypes.Structure):
_fields_ = [
("biSize", wintypes.DWORD), ("biWidth", wintypes.LONG),
("biHeight", wintypes.LONG), ("biPlanes", wintypes.WORD),
("biBitCount", wintypes.WORD), ("biCompression", wintypes.DWORD),
("biSizeImage", wintypes.DWORD),
("biXPelsPerMeter", wintypes.LONG),
("biYPelsPerMeter", wintypes.LONG),
("biClrUsed", wintypes.DWORD),
("biClrImportant", wintypes.DWORD),
]
byte_count = width * height * 4
pixels = ctypes.create_string_buffer(byte_count)
header = BitmapInfoHeader(
ctypes.sizeof(BitmapInfoHeader), width, -height, 1, 32, 0,
byte_count, 0, 0, 0, 0,
)
rows = gdi32.GetDIBits(
memory_dc, bitmap, 0, height, pixels, ctypes.byref(header), 0
)
if rows != height:
raise RuntimeError(f"GetDIBits returned {rows}/{height} rows")
image = Image.frombuffer(
"RGB", (width, height), pixels, "raw", "BGRX", 0, 1
).copy()
finally:
gdi32.SelectObject(memory_dc, previous)
gdi32.DeleteObject(bitmap)
gdi32.DeleteDC(memory_dc)
user32.ReleaseDC(hwnd, window_dc)
output.parent.mkdir(parents=True, exist_ok=True)
image.save(output, "PNG")
def _tile_windows(handles: list[int]) -> None:
"""Place four login windows in a 2x2 grid so screenshots cannot overlap."""
user32 = ctypes.windll.user32
screen_w = user32.GetSystemMetrics(0)
screen_h = user32.GetSystemMetrics(1)
cell_w = max(640, screen_w // 2)
cell_h = max(450, screen_h // 2)
for index, hwnd in enumerate(handles):
x = (index % 2) * cell_w
y = (index // 2) * cell_h
user32.ShowWindow(hwnd, 9) # SW_RESTORE
user32.MoveWindow(hwnd, x, y, cell_w, cell_h, True)
def _send_lark(
recipient: str,
text: str | None = None,
image: Path | None = None,
*,
app_profile: str = ANALYZER_NOTIFICATION_APP_PROFILE,
) -> tuple[dict[str, object], ...]:
"""Send text and/or image through the analyzer Hermes bot."""
if app_profile != ANALYZER_NOTIFICATION_APP_PROFILE:
raise ValueError(
"relogin notifications must use the hermes-analyzer lark-cli profile"
)
lark_env = {
key: value for key, value in os.environ.items()
if not key.upper().startswith("HERMES_")
}
receipts: list[dict[str, object]] = []
if text:
ANALYZER_DIR.mkdir(parents=True, exist_ok=True)
receipts.append(send_lark_bot_message(
user_id=recipient,
text=text,
profile=ANALYZER_NOTIFICATION_APP_PROFILE,
cwd=ANALYZER_DIR,
env=lark_env,
))
if image:
receipts.append(send_lark_bot_message(
user_id=recipient,
image=image,
profile=ANALYZER_NOTIFICATION_APP_PROFILE,
cwd=image.parent,
env=lark_env,
))
return tuple(receipts)
def _send_qr_notification(
route: ResolvedNotificationRoute,
platform_name: str,
*,
override: str | None,
text: str,
image: Path,
) -> tuple[NotificationDeliveryFailure, ...]:
"""Attempt every QR delivery and return all recipient-specific failures."""
profile = route.app_profile or ANALYZER_NOTIFICATION_APP_PROFILE
failures: list[NotificationDeliveryFailure] = []
for recipient in recipients_for_platform(platform_name, route, override):
for stage, kwargs in (
("qr_text", {"text": text}),
("qr_image", {"image": image}),
):
try:
_send_lark(recipient, app_profile=profile, **kwargs)
except Exception as exc:
failures.append(
NotificationDeliveryFailure(
stage=stage,
platform=platform_name,
recipient=recipient,
detail=str(exc) or type(exc).__name__,
)
)
return tuple(failures)
def _unavailable_qr_delivery_failures(
route: ResolvedNotificationRoute,
platform_name: str,
*,
override: str | None,
detail: str,
) -> tuple[NotificationDeliveryFailure, ...]:
"""Record both intended QR deliveries when no screenshot can be sent."""
return tuple(
NotificationDeliveryFailure(
stage=stage,
platform=platform_name,
recipient=recipient,
detail=detail,
)
for recipient in recipients_for_platform(platform_name, route, override)
for stage in ("qr_text", "qr_image")
)
def _send_failure_summaries(
route: ResolvedNotificationRoute,
failed_platforms: list[str],
*,
override: str | None,
max_attempts: int,
) -> tuple[NotificationDeliveryFailure, ...]:
"""Attempt every final failure summary and return all delivery failures."""
groups = group_platforms_by_notification_recipient(
failed_platforms,
route,
override,
)
profile = route.app_profile or ANALYZER_NOTIFICATION_APP_PROFILE
failures: list[NotificationDeliveryFailure] = []
for recipient, names in groups.items():
labels = "\u3001".join(PLATFORMS[name].label for name in names)
try:
_send_lark(
recipient,
text=(
f"\u4ee5\u4e0b\u5e73\u53f0\u8fde\u7eed {max_attempts} \u6b21\u672a\u626b\u7801\uff0c"
"\u5df2\u6062\u590d\u5e76\u7ee7\u4f7f\u7528\u65e7 Cookie\uff1a"
f"{labels}"
),
app_profile=profile,
)
except Exception as exc:
failures.append(
NotificationDeliveryFailure(
stage="failure_summary",
platform=",".join(names),
recipient=recipient,
detail=str(exc) or type(exc).__name__,
)
)
return tuple(failures)
def _run_round_factory(
args,
notification_route: ResolvedNotificationRoute | None,
notification_failures: list[NotificationDeliveryFailure],
):
python = sys.executable
screenshot_root = PATHS.tmp_root / "qrcode/friday_relogin"
def run_round(names: list[str], attempt: int) -> dict[str, bool]:
started: dict[str, subprocess.Popen] = {}
cookie_files: dict[str, Path] = {}
missing_qr_windows: set[str] = set()
stamp = dt.datetime.now().strftime("%Y%m%d_%H%M%S")
for name in names:
platform = PLATFORMS[name]
cmd = [python, *platform.command, "--login-timeout", str(args.round_timeout)]
target = PROJECT_DIR / platform.command[0]
environment = environment_for_child_script(target, os.environ)
cookie_files[name] = binding_from_environment(environment).cookie_file
started[name] = subprocess.Popen(
cmd,
cwd=PROJECT_DIR,
env=environment,
)
time.sleep(args.screenshot_delay)
title_windows = _login_windows()
windows = {
name: title_windows.get(name) or _window_for_process(process.pid)
for name, process in started.items()
}
_tile_windows([hwnd for hwnd in windows.values() if hwnd])
time.sleep(2)
for name, process in started.items():
platform = PLATFORMS[name]
hwnd = windows[name]
if not hwnd:
print(f"[{name}] no visible browser window for QR screenshot", flush=True)
missing_qr_windows.add(name)
continue
output = screenshot_root / f"{stamp}_{name}_round{attempt}.png"
try:
_capture_window(hwnd, output)
print(f"[{name}] QR screenshot: {output}", flush=True)
except Exception as exc:
print(f"[{name}] screenshot failed: {exc}", flush=True)
if not args.no_send and notification_route is not None:
notification_failures.extend(
_unavailable_qr_delivery_failures(
notification_route,
name,
override=args.recipient,
detail=(
"QR screenshot capture failed: "
f"{str(exc) or type(exc).__name__}"
),
)
)
continue
if not args.no_send and notification_route is not None:
notification_failures.extend(
_send_qr_notification(
notification_route,
name,
override=args.recipient,
text=(
f"{platform.label} \u767b\u5f55\u4e8c\u7ef4\u7801"
f"\uff08\u7b2c {attempt}/{args.max_attempts} \u6b21\uff09\uff0c"
"\u8bf7\u5728 3 \u5206\u949f\u5185\u626b\u7801\u3002"
),
image=output,
)
)
# Put the still-open login windows back into a compact layout after
# taking full-size screenshots one at a time.
_tile_windows([hwnd for hwnd in windows.values() if hwnd])
results: dict[str, bool] = {}
for name, process in started.items():
try:
rc = process.wait(timeout=args.round_timeout + 90)
except subprocess.TimeoutExpired:
_terminate_process_tree(process)
rc = 1
results[name] = rc == 0 and _valid_cookie(
PLATFORMS[name],
cookie_files[name],
)
if (
not results[name]
and name in missing_qr_windows
and not args.no_send
and notification_route is not None
):
notification_failures.extend(
_unavailable_qr_delivery_failures(
notification_route,
name,
override=args.recipient,
detail=(
"no visible QR window and relogin did not succeed"
),
)
)
print(f"[{name}] round={attempt} exit={rc} success={results[name]}", flush=True)
return results
return run_round
def main() -> int:
acceptance_policy = current_acceptance_policy()
if acceptance_policy.enabled:
acceptance_policy.record(
"cookie_skipped",
operation="content.relogin.weekly",
details={
"status": "SKIPPED_COOKIE",
"reason": "acceptance runs never launch QR relogin workflows",
},
)
print(
"[SKIPPED_COOKIE] Acceptance mode does not launch login windows, "
"reset cookies/profiles, or wait for QR scans.",
flush=True,
)
return COOKIE_SKIP_EXIT_CODE
parser = argparse.ArgumentParser(description="Friday five-platform parallel QR relogin")
parser.add_argument(
"--recipient", default=None,
help="Override the configured recipients and send every platform to one open_id",
)
parser.add_argument("--max-attempts", type=int, default=3)
parser.add_argument("--round-timeout", type=int, default=300)
parser.add_argument("--screenshot-delay", type=int, default=60)
parser.add_argument("--no-send", action="store_true", help="Do not send Lark messages")
parser.add_argument(
"--platform", action="append", choices=sorted(PLATFORMS),
help="Only relogin the selected platform; may be repeated",
)
args = parser.parse_args()
platform_names = resolve_platform_names(args.platform)
notification_route = (
None if args.no_send else resolve_weekly_notification_route(args.recipient)
)
notification_failures: list[NotificationDeliveryFailure] = []
status = run_with_retries(
platform_names, max_attempts=args.max_attempts,
run_round=_run_round_factory(
args,
notification_route,
notification_failures,
),
)
failed = [name for name, ok in status.items() if not ok]
if failed and not args.no_send and notification_route is not None:
notification_failures.extend(
_send_failure_summaries(
notification_route,
failed,
override=args.recipient,
max_attempts=args.max_attempts,
)
)
print(f"Final relogin status: {status}", flush=True)
if notification_failures:
print(
f"Final notification status: failed ({len(notification_failures)} deliveries)",
flush=True,
)
for failure in notification_failures:
print(
"[NOTIFICATION_ERROR] "
f"stage={failure.stage} platform={failure.platform} "
f"recipient={failure.recipient} error={failure.detail}",
flush=True,
)
return 2
if notification_route is not None and notification_route.enabled:
print("Final notification status: delivered", flush=True)
else:
print("Final notification status: skipped", flush=True)
return 1 if failed else 0
if __name__ == "__main__":
raise SystemExit(main())
@@ -1,113 +0,0 @@
"""每周一重登 B 站。
强制清掉 cookie JSON + 浏览器 profile,确保下次启动时必须扫码。
任务计划: 每周一 09:00
跑法:
python data/tools/relogin_bilibili.py # 实际跑(会弹二维码)
python data/tools/relogin_bilibili.py --dry-run # 只看会做什么
"""
import argparse
import datetime as dt
import subprocess
import sys
from gyxx_flow.modules.content_marketing.data.tools.relogin_runtime_paths import (
child_browser_environment,
resolve_relogin_runtime_paths,
sync_relogin_state_to_consumers,
)
from gyxx_flow.modules.content_marketing.data.tools.relogin_transaction import (
begin,
commit,
rollback,
)
from gyxx_flow.modules.content_marketing.runtime_paths import PATHS
PROJECT_DIR = PATHS.module_root
RUNTIME_PATHS = resolve_relogin_runtime_paths("bilibili")
COOKIE_FILE = RUNTIME_PATHS.cookie_file
STATE_FILE = RUNTIME_PATHS.storage_state_file
PROFILE_DIR = RUNTIME_PATHS.profile_dir
SCRAPER = PROJECT_DIR / "bilibili_comment_scraper.py"
def reset_state():
"""删 cookie JSON + storage_state + 清浏览器 profile。"""
state = begin((COOKIE_FILE, STATE_FILE), PROFILE_DIR)
print(" [OK] 旧 Cookie/Profile 已暂存,失败时将自动恢复")
return state
def main() -> int:
ap = argparse.ArgumentParser(description="每周一重登 B 站")
ap.add_argument("--login-timeout", type=int, default=300, help="扫码超时(秒)")
ap.add_argument("--dry-run", action="store_true", help="只打印计划,不删不跑")
args = ap.parse_args()
print("== Relogin: B 站 ==")
print(f" 今天: {dt.date.today()} ({dt.date.today().strftime('%A')})")
print(f" cookie: {COOKIE_FILE}")
print(f" profile: {PROFILE_DIR}")
print()
if args.dry_run:
print("[DRY-RUN] 实际跑会: 删 cookie + 清 profile + 弹二维码")
return 0
old_state = reset_state()
cmd = [
sys.executable,
str(SCRAPER),
"login", # dummy url (login-only 模式不需要真实 url)
"--login-only",
"--login-timeout",
str(args.login_timeout),
]
print(f"\n [RUN] {' '.join(cmd)}")
try:
rc = subprocess.call(
cmd,
cwd=str(PROJECT_DIR),
env=child_browser_environment(
cookie_file=COOKIE_FILE,
storage_state_file=STATE_FILE,
profile_dir=PROFILE_DIR,
),
)
except BaseException:
rollback(old_state)
raise
print(f"\n exit={rc}")
login_ok = False
if rc == 0:
# 验证 SESSDATA 是否真写入了
try:
import json
cookies = json.loads(COOKIE_FILE.read_text(encoding="utf-8"))
sessdata = any(c.get("name") == "SESSDATA" and c.get("value") for c in cookies)
login_ok = bool(sessdata)
print(f" SESSDATA 写入: {'OK' if sessdata else 'FAIL — 没有看到 SESSDATA,登录可能没成功'}")
except Exception as exc:
print(f" [WARN] 验证 cookie 时出错: {exc}")
if rc != 0 or not login_ok:
rollback(old_state)
print(" [ROLLBACK] 未完成扫码,已恢复旧登录状态")
return rc or 1
try:
sync_relogin_state_to_consumers(
"bilibili",
cookie_file=COOKIE_FILE,
storage_state_file=STATE_FILE,
)
except Exception as exc:
rollback(old_state)
print(f" [ROLLBACK] 同步采集器登录态失败: {exc}")
return 1
commit(old_state)
return 0
if __name__ == "__main__":
sys.exit(main())
@@ -1,121 +0,0 @@
"""Force a fresh Douyin login while preserving the previous usable session."""
from __future__ import annotations
import argparse
import datetime as dt
import json
import subprocess
import sys
from gyxx_flow.modules.content_marketing.data.tools.relogin_runtime_paths import (
child_browser_environment,
resolve_relogin_runtime_paths,
sync_relogin_state_to_consumers,
)
from gyxx_flow.modules.content_marketing.runtime_paths import PATHS
try:
from .relogin_transaction import begin, commit, rollback
except ImportError: # Direct script execution from data/tools.
from gyxx_flow.modules.content_marketing.data.tools.relogin_transaction import (
begin,
commit,
rollback,
)
PROJECT_DIR = PATHS.module_root
RUNTIME_PATHS = resolve_relogin_runtime_paths("douyin")
COOKIE_FILE = RUNTIME_PATHS.cookie_file
STATE_FILE = RUNTIME_PATHS.storage_state_file
PROFILE_DIR = RUNTIME_PATHS.profile_dir
SCRAPER = PROJECT_DIR / "douyin_comment_scraper.py"
LOGIN_COOKIE = "sessionid"
def reset_state():
state = begin((COOKIE_FILE, STATE_FILE), PROFILE_DIR)
print(" [OK] Old Cookie/Profile staged; failures will roll back automatically")
return state
def is_cookie_valid() -> bool:
if not COOKIE_FILE.exists():
return False
try:
cookies = json.loads(COOKIE_FILE.read_text(encoding="utf-8"))
return any(
isinstance(cookie, dict)
and cookie.get("name") == LOGIN_COOKIE
and cookie.get("value")
for cookie in cookies
)
except Exception:
return False
def main() -> int:
parser = argparse.ArgumentParser(description="Relogin Douyin")
parser.add_argument("--login-timeout", type=int, default=300)
parser.add_argument("--dry-run", action="store_true")
args = parser.parse_args()
print("== Relogin: Douyin ==")
print(f" date: {dt.date.today()}")
print(f" cookie: {COOKIE_FILE}")
print(f" profile: {PROFILE_DIR}")
print(f" current cookie: {'present' if is_cookie_valid() else 'missing/invalid'}")
if args.dry_run:
print("[DRY-RUN] Would stage Cookie/Profile and open the QR login window")
return 0
old_state = reset_state()
command = [
sys.executable,
str(SCRAPER),
"login",
"--login-only",
"--login-timeout",
str(args.login_timeout),
]
print(f"\n [RUN] {' '.join(command)}")
try:
return_code = subprocess.call(
command,
cwd=str(PROJECT_DIR),
env=child_browser_environment(
cookie_file=COOKIE_FILE,
storage_state_file=STATE_FILE,
profile_dir=PROFILE_DIR,
),
)
except BaseException:
rollback(old_state)
raise
login_ok = return_code == 0 and is_cookie_valid()
print(f"\n exit={return_code}")
print(f" {LOGIN_COOKIE} written: {'OK' if login_ok else 'FAIL'}")
if not login_ok:
rollback(old_state)
print(" [ROLLBACK] Login incomplete; restored previous Cookie/Profile")
return return_code or 1
try:
sync_relogin_state_to_consumers(
"douyin",
cookie_file=COOKIE_FILE,
storage_state_file=STATE_FILE,
)
except Exception as exc:
rollback(old_state)
print(f" [ROLLBACK] Failed to sync collector login state: {exc}")
return 1
commit(old_state)
return 0
if __name__ == "__main__":
raise SystemExit(main())
@@ -1,148 +0,0 @@
"""每周一重登小红书蒲公英。
强制清掉 cookie JSON + 浏览器 profile,确保下次启动时必须扫码。
任务计划: 每周一 09:00
跑法:
python data/tools/relogin_pgy.py # 实际跑(会弹二维码)
python data/tools/relogin_pgy.py --dry-run # 只看会做什么
"""
import argparse
import datetime as dt
import shutil
import subprocess
import sys
from pathlib import Path
from gyxx_flow.modules.content_marketing.data.tools.relogin_runtime_paths import (
child_browser_environment,
resolve_relogin_runtime_paths,
sync_relogin_state_to_consumers,
)
from gyxx_flow.modules.content_marketing.runtime_paths import PATHS
PROJECT_DIR = PATHS.module_root
RUNTIME_PATHS = resolve_relogin_runtime_paths("pgy")
COOKIE_FILE = RUNTIME_PATHS.cookie_file
STATE_FILE = RUNTIME_PATHS.storage_state_file
PROFILE_DIR = RUNTIME_PATHS.profile_dir
SCRAPER = PROJECT_DIR / "pgy_xhs_scraper_v2.py"
def reset_state() -> tuple[Path | None, Path | None, Path | None]:
"""Temporarily move the old login state aside so it can be restored."""
ts = dt.datetime.now().strftime("%Y%m%d_%H%M%S_%f")
cookie_backup = None
state_backup = None
profile_backup = None
if COOKIE_FILE.exists():
backup = COOKIE_FILE.with_suffix(f".json.bak.{ts}")
COOKIE_FILE.rename(backup)
cookie_backup = backup
print(f" [OK] 备份 cookie -> {backup.name}")
else:
print(f" [INFO] {COOKIE_FILE.name} 不存在,跳过")
if STATE_FILE.exists():
state_backup = STATE_FILE.with_suffix(f".json.bak.{ts}")
STATE_FILE.rename(state_backup)
print(f" [OK] 备份 storage state -> {state_backup.name}")
else:
print(f" [INFO] {STATE_FILE.name} 不存在,跳过")
if PROFILE_DIR.exists():
profile_backup = PROFILE_DIR.with_name(f"{PROFILE_DIR.name}.bak.{ts}")
PROFILE_DIR.rename(profile_backup)
print(f" [OK] 暂存 {PROFILE_DIR.name}/ -> {profile_backup.name}/")
else:
print(f" [INFO] {PROFILE_DIR.name}/ 不存在,跳过")
return cookie_backup, profile_backup, state_backup
def restore_previous_state(
cookie_backup: Path | None,
profile_backup: Path | None,
state_backup: Path | None = None,
) -> None:
"""Discard an incomplete login and restore the last known-good state."""
if COOKIE_FILE.exists():
COOKIE_FILE.unlink()
if cookie_backup and cookie_backup.exists():
cookie_backup.rename(COOKIE_FILE)
print(f" [ROLLBACK] 已恢复旧 cookie: {COOKIE_FILE.name}")
if STATE_FILE.exists():
STATE_FILE.unlink()
if state_backup and state_backup.exists():
state_backup.rename(STATE_FILE)
print(f" [ROLLBACK] 已恢复旧 storage state: {STATE_FILE.name}")
if PROFILE_DIR.exists():
shutil.rmtree(PROFILE_DIR)
if profile_backup and profile_backup.exists():
profile_backup.rename(PROFILE_DIR)
print(f" [ROLLBACK] 已恢复旧浏览器状态: {PROFILE_DIR.name}/")
def finish_successful_relogin(profile_backup: Path | None) -> None:
"""Keep the new state and remove the bulky old profile backup."""
if profile_backup and profile_backup.exists():
shutil.rmtree(profile_backup)
def main() -> int:
ap = argparse.ArgumentParser(description="每周一重登小红书蒲公英")
ap.add_argument("--login-timeout", type=int, default=300, help="扫码超时(秒)")
ap.add_argument("--dry-run", action="store_true", help="只打印计划,不删不跑")
args = ap.parse_args()
print("== Relogin: 小红书蒲公英 ==")
print(f" 今天: {dt.date.today()} ({dt.date.today().strftime('%A')})")
print(f" cookie: {COOKIE_FILE}")
print(f" profile: {PROFILE_DIR}")
print()
if args.dry_run:
print("[DRY-RUN] 实际跑会: 删 cookie + 清 profile + 弹二维码")
return 0
cookie_backup, profile_backup, state_backup = reset_state()
cmd = [
sys.executable,
str(SCRAPER),
"--login-only",
"--login-timeout",
str(args.login_timeout),
]
print(f"\n [RUN] {' '.join(cmd)}")
try:
rc = subprocess.call(
cmd,
cwd=str(PROJECT_DIR),
env=child_browser_environment(
cookie_file=COOKIE_FILE,
storage_state_file=STATE_FILE,
profile_dir=PROFILE_DIR,
),
)
except BaseException:
restore_previous_state(cookie_backup, profile_backup, state_backup)
raise
print(f"\n exit={rc}")
if rc != 0 or not COOKIE_FILE.exists():
print(" [WARN] 新登录未完成,恢复旧登录状态")
restore_previous_state(cookie_backup, profile_backup, state_backup)
return rc or 1
try:
sync_relogin_state_to_consumers(
"pgy",
cookie_file=COOKIE_FILE,
storage_state_file=STATE_FILE,
)
except Exception as exc:
restore_previous_state(cookie_backup, profile_backup, state_backup)
print(f" [ROLLBACK] 同步采集器登录态失败: {exc}")
return 1
finish_successful_relogin(profile_backup)
return rc
if __name__ == "__main__":
sys.exit(main())
@@ -1,279 +0,0 @@
"""Resolve one relogin wrapper's browser state contract.
The workflow runtime injects a unique browser binding into each relogin
wrapper. Standalone legacy launches do not have that environment, so they
retain the original per-platform files below the content-marketing state root.
"""
from __future__ import annotations
import json
import os
import re
import shutil
import tempfile
import uuid
from dataclasses import dataclass
from pathlib import Path
from typing import Mapping
from gyxx_flow.adapters.integration import RuntimeIntegrationCatalog
from gyxx_flow.modules.content_marketing.runtime_paths import PATHS
_PLATFORM = re.compile(r"^[a-z][a-z0-9_-]{0,31}$")
CONSUMER_BINDING_IDS: Mapping[str, tuple[str, ...]] = {
"bilibili": ("content.comments.collect_bilibili",),
"douyin": ("content.comments.collect_douyin",),
"pgy": ("content_marketing:pgy_xhs_scraper_v2.py",),
"xiaohongshu": ("content.comments.collect_xiaohongshu",),
# v2 now reads the Buyin/Doudian site and is owned by the douyin-shop
# account. Keep the legacy Xingtu relogin isolated from that vault.
"buyin": ("content_marketing:xingtu_scraper_v2.py",),
"xingtu": ("content_marketing:xingtu_scraper.py",),
}
@dataclass(frozen=True)
class ReloginRuntimePaths:
cookie_file: Path
storage_state_file: Path
profile_dir: Path
def resolve_relogin_runtime_paths(
platform: str,
environment: Mapping[str, str] | None = None,
) -> ReloginRuntimePaths:
"""Prefer the injected binding and fall back to original platform paths."""
if not _PLATFORM.fullmatch(platform):
raise ValueError("invalid relogin platform")
values = os.environ if environment is None else environment
legacy_cookie = PATHS.state_root / "cookies" / f"{platform}_cookies.json"
legacy_storage = (
PATHS.state_root / "cookies" / f"{platform}_storage_state.json"
)
legacy_profile = PATHS.state_root / "browser-profiles" / platform
return ReloginRuntimePaths(
cookie_file=_bound_path(
values,
"GYXX_BROWSER_COOKIE_FILE",
legacy_cookie,
),
storage_state_file=_bound_path(
values,
"GYXX_BROWSER_STORAGE_STATE_FILE",
legacy_storage,
),
profile_dir=_bound_path(
values,
"GYXX_BROWSER_PROFILE_DIR",
legacy_profile,
),
)
def child_browser_environment(
*,
cookie_file: Path,
storage_state_file: Path,
profile_dir: Path,
environment: Mapping[str, str] | None = None,
) -> dict[str, str]:
"""Keep the nested login scraper on the wrapper's injected binding."""
result = dict(os.environ if environment is None else environment)
result.update(
{
"GYXX_BROWSER_COOKIE_FILE": str(cookie_file.resolve()),
"GYXX_BROWSER_STORAGE_STATE_FILE": str(
storage_state_file.resolve()
),
"GYXX_BROWSER_PROFILE_DIR": str(profile_dir.resolve()),
}
)
# content_marketing/sitecustomize.py normally rebinds a nested Python
# script from argv[0]. A relogin scraper is an implementation detail of
# this wrapper and must write the wrapper binding that the parent monitors.
result.pop("GYXX_MODULE_ID", None)
return result
def sync_relogin_state_to_consumers(
platform: str,
*,
cookie_file: Path,
storage_state_file: Path,
environment: Mapping[str, str] | None = None,
) -> tuple[Path, ...]:
"""Atomically publish a verified relogin state to unique consumer paths."""
try:
consumer_ids = CONSUMER_BINDING_IDS[platform]
except KeyError as exc:
raise ValueError("invalid relogin platform") from exc
values = os.environ if environment is None else environment
project_root = values.get("GYXX_PROJECT_ROOT", "").strip()
data_root = values.get("GYXX_DATA_ROOT", "").strip()
if not project_root and not data_root:
# Original standalone launches use the legacy per-platform state files
# directly, so there is no separate managed consumer to publish to.
return ()
if not project_root or not data_root:
raise RuntimeError("managed relogin state sync requires project and data roots")
catalog = RuntimeIntegrationCatalog.load_default(
project_root=project_root,
data_root=data_root,
)
source_cookie = cookie_file.expanduser().resolve()
source_storage = storage_state_file.expanduser().resolve()
cookie_payload = _read_cookie_payload(source_cookie)
storage_available = source_storage.is_file()
if storage_available:
_read_storage_state_payload(source_storage)
source_paths = {source_cookie, source_storage}
destination_paths: set[Path] = set()
destinations: list[Path] = []
updates: list[tuple[Path, Path]] = []
for consumer_id in consumer_ids:
binding = catalog.binding_for(consumer_id)
target_cookie = binding.cookie_file.resolve()
target_storage = binding.storage_state_file.resolve()
targets = (target_cookie, target_storage)
if any(target in source_paths for target in targets) or any(
target in destination_paths for target in targets
):
raise RuntimeError(
"relogin and consumer browser state paths must be unique"
)
destination_paths.update(targets)
_validate_cookie_requirements(
cookie_payload,
required_domains=binding.required_cookie_domains,
required_names=binding.required_cookie_names,
)
updates.append((source_cookie, target_cookie))
if storage_available:
updates.append((source_storage, target_storage))
destinations.append(target_cookie)
_atomic_replace_files(updates)
return tuple(destinations)
def _bound_path(
environment: Mapping[str, str],
name: str,
fallback: Path,
) -> Path:
configured = environment.get(name, "").strip()
return (
Path(configured).expanduser().resolve()
if configured
else fallback.resolve()
)
def _read_cookie_payload(path: Path) -> list[dict[str, object]]:
try:
payload = json.loads(path.read_text(encoding="utf-8"))
except (OSError, UnicodeError, ValueError) as exc:
raise RuntimeError("relogin cookie state is missing or invalid") from exc
if not isinstance(payload, list) or not payload or not all(
isinstance(cookie, dict) for cookie in payload
):
raise RuntimeError("relogin cookie state is missing or invalid")
return payload
def _read_storage_state_payload(path: Path) -> dict[str, object]:
try:
payload = json.loads(path.read_text(encoding="utf-8"))
except (OSError, UnicodeError, ValueError) as exc:
raise RuntimeError("relogin storage state is invalid") from exc
if not isinstance(payload, dict):
raise RuntimeError("relogin storage state is invalid")
return payload
def _validate_cookie_requirements(
cookies: list[dict[str, object]],
*,
required_domains: tuple[str, ...],
required_names: tuple[str, ...],
) -> None:
usable = [cookie for cookie in cookies if cookie.get("value")]
if not usable:
raise RuntimeError("relogin cookie state has no usable cookies")
if required_domains and not any(
any(
str(cookie.get("domain", "")).lstrip(".").casefold()
.endswith(domain.casefold().lstrip("."))
for domain in required_domains
)
for cookie in usable
):
raise RuntimeError("relogin cookie state does not match consumer domain")
if required_names and not any(
str(cookie.get("name", "")) in required_names for cookie in usable
):
raise RuntimeError("relogin cookie state misses the consumer login cookie")
def _atomic_replace_files(updates: list[tuple[Path, Path]]) -> None:
"""Replace a small related file set and restore every old target on error."""
transaction = uuid.uuid4().hex
staged: list[tuple[Path, Path]] = []
backups: dict[Path, Path | None] = {}
replaced: list[Path] = []
try:
for source, target in updates:
target.parent.mkdir(parents=True, exist_ok=True)
handle, raw_temp = tempfile.mkstemp(
prefix=f".{target.name}.{transaction}.",
suffix=".tmp",
dir=target.parent,
)
temp = Path(raw_temp)
with os.fdopen(handle, "wb") as stream:
stream.write(source.read_bytes())
stream.flush()
os.fsync(stream.fileno())
staged.append((temp, target))
backup = target.with_name(f".{target.name}.{transaction}.bak")
if target.is_file():
shutil.copy2(target, backup)
backups[target] = backup
else:
backups[target] = None
for temp, target in staged:
os.replace(temp, target)
replaced.append(target)
except BaseException:
for target in reversed(replaced):
backup = backups.get(target)
if backup is not None and backup.exists():
os.replace(backup, target)
elif target.exists():
target.unlink()
raise
finally:
for temp, _target in staged:
if temp.exists():
temp.unlink()
for backup in backups.values():
if backup is not None and backup.exists():
backup.unlink()
__all__ = [
"CONSUMER_BINDING_IDS",
"ReloginRuntimePaths",
"child_browser_environment",
"resolve_relogin_runtime_paths",
"sync_relogin_state_to_consumers",
]
@@ -1,47 +0,0 @@
"""Transactional backup/rollback helpers for interactive relogin scripts."""
from __future__ import annotations
import datetime as dt
import shutil
from dataclasses import dataclass
from pathlib import Path
@dataclass
class LoginStateBackup:
files: dict[Path, Path]
profile: Path
profile_backup: Path | None
def begin(files: tuple[Path, ...], profile: Path) -> LoginStateBackup:
stamp = dt.datetime.now().strftime("%Y%m%d_%H%M%S_%f")
backups: dict[Path, Path] = {}
for source in files:
if source.exists():
backup = source.with_name(f"{source.name}.bak.{stamp}")
source.rename(backup)
backups[source] = backup
profile_backup = None
if profile.exists():
profile_backup = profile.with_name(f"{profile.name}.bak.{stamp}")
profile.rename(profile_backup)
return LoginStateBackup(backups, profile, profile_backup)
def rollback(state: LoginStateBackup) -> None:
for source, backup in state.files.items():
if source.exists():
source.unlink()
if backup.exists():
backup.rename(source)
if state.profile.exists():
shutil.rmtree(state.profile)
if state.profile_backup and state.profile_backup.exists():
state.profile_backup.rename(state.profile)
def commit(state: LoginStateBackup) -> None:
if state.profile_backup and state.profile_backup.exists():
shutil.rmtree(state.profile_backup)
@@ -1,116 +0,0 @@
"""每周重登小红书。
强制清掉 cookie JSON + 浏览器 profile,确保下次启动时必须扫码。
跑法:
python data/tools/relogin_xiaohongshu.py # 实际跑(会弹二维码)
python data/tools/relogin_xiaohongshu.py --dry-run # 只看会做什么
"""
import argparse
import datetime as dt
import json
import subprocess
import sys
from gyxx_flow.modules.content_marketing.data.tools.relogin_runtime_paths import (
child_browser_environment,
resolve_relogin_runtime_paths,
sync_relogin_state_to_consumers,
)
from gyxx_flow.modules.content_marketing.data.tools.relogin_transaction import (
begin,
commit,
rollback,
)
from gyxx_flow.modules.content_marketing.runtime_paths import PATHS
PROJECT_DIR = PATHS.module_root
RUNTIME_PATHS = resolve_relogin_runtime_paths("xiaohongshu")
COOKIE_FILE = RUNTIME_PATHS.cookie_file
STATE_FILE = RUNTIME_PATHS.storage_state_file
PROFILE_DIR = RUNTIME_PATHS.profile_dir
SCRAPER = PROJECT_DIR / "xiaohongshu_comment_scraper.py"
LOGIN_COOKIE = "web_session"
def reset_state():
state = begin((COOKIE_FILE, STATE_FILE), PROFILE_DIR)
print(" [OK] 旧 Cookie/Profile 已暂存,失败时将自动恢复")
return state
def is_cookie_valid() -> bool:
if not COOKIE_FILE.exists():
return False
try:
cookies = json.loads(COOKIE_FILE.read_text(encoding="utf-8"))
return any(c.get("name") == LOGIN_COOKIE and c.get("value") for c in cookies)
except Exception:
return False
def main() -> int:
ap = argparse.ArgumentParser(description="重登小红书")
ap.add_argument("--login-timeout", type=int, default=300, help="扫码超时(秒)")
ap.add_argument("--dry-run", action="store_true", help="只打印计划,不删不跑")
args = ap.parse_args()
print("== Relogin: 小红书 ==")
print(f" 今天: {dt.date.today()}")
print(f" cookie: {COOKIE_FILE}")
print(f" profile: {PROFILE_DIR}")
print(f" 当前 cookie 状态: {'有效' if is_cookie_valid() else '无效/不存在'}")
print()
if args.dry_run:
print("[DRY-RUN] 实际跑会: 删 cookie + 清 profile + 弹二维码")
return 0
old_state = reset_state()
cmd = [
sys.executable,
str(SCRAPER),
"login",
"--login-only",
"--login-timeout",
str(args.login_timeout),
]
print(f"\n [RUN] {' '.join(cmd)}")
try:
rc = subprocess.call(
cmd,
cwd=str(PROJECT_DIR),
env=child_browser_environment(
cookie_file=COOKIE_FILE,
storage_state_file=STATE_FILE,
profile_dir=PROFILE_DIR,
),
)
except BaseException:
rollback(old_state)
raise
print(f"\n exit={rc}")
login_ok = rc == 0 and is_cookie_valid()
if rc == 0:
print(f" {LOGIN_COOKIE} 写入: {'OK' if login_ok else 'FAIL — 没有看到 web_session,登录可能没成功'}")
if not login_ok:
rollback(old_state)
print(" [ROLLBACK] 未完成扫码,已恢复旧登录状态")
return rc or 1
try:
sync_relogin_state_to_consumers(
"xiaohongshu",
cookie_file=COOKIE_FILE,
storage_state_file=STATE_FILE,
)
except Exception as exc:
rollback(old_state)
print(f" [ROLLBACK] 同步采集器登录态失败: {exc}")
return 1
commit(old_state)
return 0
if __name__ == "__main__":
sys.exit(main())
@@ -1,350 +0,0 @@
"""每 3 个周一重登一次抖音/星图。
强制清掉 cookie JSON + 浏览器 profile,确保下次启动时必须扫码。
星图登录完成后,自动把登录态 cookie 同步到抖音评论 scraper,避免再扫一次码。
任务计划: 每周一 09:00 (脚本内部判断是否到了该重登的周一,不到就退出)
3 周循环基准: 任意一个周一,后续 (今天 - BASE_DATE) // 7 % 3 == 0 才是星图周。
改 BASE_DATE 就改了重登周期。
跑法:
python data/tools/relogin_xingtu.py # 实际跑
python data/tools/relogin_xingtu.py --dry-run # 看今天是不是星图周
python data/tools/relogin_xingtu.py --force # 忽略 3 周判断,强制重登
"""
import argparse
import datetime as dt
import json
import os
import shutil
import subprocess
import sys
from pathlib import Path
import psutil
from gyxx_flow.modules.content_marketing.data.tools.relogin_runtime_paths import (
child_browser_environment,
resolve_relogin_runtime_paths,
sync_relogin_state_to_consumers,
)
from gyxx_flow.modules.content_marketing.runtime_paths import PATHS
PROJECT_DIR = PATHS.module_root
RUNTIME_PATHS = resolve_relogin_runtime_paths("xingtu")
COOKIE_FILE = RUNTIME_PATHS.cookie_file
STATE_FILE = RUNTIME_PATHS.storage_state_file
PROFILE_DIR = RUNTIME_PATHS.profile_dir
SCRAPER = PROJECT_DIR / "xingtu_scraper_v2.py"
# 抖音评论 scraper 的 cookie 文件 (登录态共享目标)
DY_COOKIE_FILE = PATHS.state_root / "cookies/douyin_cookies.json"
DY_STATE_FILE = PATHS.state_root / "cookies/douyin_storage_state.json"
DY_PROFILE_DIR = PATHS.state_root / "browser-profiles/douyin"
# 需要同步的抖音登录 cookie 名
DY_LOGIN_COOKIES = {"sessionid", "sessionid_ss", "sid_guard", "sid_tt", "passport_csrf_token"}
# 3 周循环起点(必须是周一)。星图 cookie ~22 天,每 3 周(21 天)重登正好。
BASE_DATE = dt.date(2026, 6, 22)
LOGIN_COOKIE_NAMES = {"sessionid", "sessionid_ss", "sid_tt", "sid_guard"}
def transaction_file() -> Path:
return COOKIE_FILE.with_name(".xingtu_relogin_transaction.json")
def _write_transaction(
cookie_backup: Path | None,
profile_backup: Path | None,
state_backup: Path | None,
) -> None:
marker = transaction_file()
marker.parent.mkdir(parents=True, exist_ok=True)
temp = marker.with_name(f"{marker.name}.tmp")
temp.write_text(json.dumps({
"cookie_backup": str(cookie_backup) if cookie_backup else "",
"state_backup": str(state_backup) if state_backup else "",
"profile_backup": str(profile_backup) if profile_backup else "",
}, ensure_ascii=False, indent=2), encoding="utf-8")
os.replace(temp, marker)
def _clear_transaction() -> None:
marker = transaction_file()
if marker.exists():
marker.unlink()
def _remove_path(path: Path) -> None:
if path.is_dir():
shutil.rmtree(path)
elif path.exists():
path.unlink()
def recover_interrupted_state() -> bool:
"""Restore a relogin transaction that was killed after moving old state."""
marker = transaction_file()
if not marker.exists():
return False
try:
state = json.loads(marker.read_text(encoding="utf-8"))
except Exception:
return False
cookie_backup = Path(state["cookie_backup"]) if state.get("cookie_backup") else None
state_backup = Path(state["state_backup"]) if state.get("state_backup") else None
profile_backup = Path(state["profile_backup"]) if state.get("profile_backup") else None
terminate_profile_processes()
if cookie_backup and cookie_backup.exists():
_remove_path(COOKIE_FILE)
cookie_backup.rename(COOKIE_FILE)
if state_backup and state_backup.exists():
_remove_path(STATE_FILE)
state_backup.rename(STATE_FILE)
if profile_backup and profile_backup.exists():
_remove_path(PROFILE_DIR)
profile_backup.rename(PROFILE_DIR)
_clear_transaction()
print(" [RECOVER] 已恢复上次被中断的星图登录状态")
return True
def has_valid_login_cookie_file(path: Path | None = None) -> bool:
path = path or COOKIE_FILE
try:
cookies = json.loads(path.read_text(encoding="utf-8"))
except Exception:
return False
return any(
isinstance(cookie, dict)
and cookie.get("name") in LOGIN_COOKIE_NAMES
and cookie.get("value")
for cookie in cookies
)
def terminate_profile_processes() -> None:
"""Stop only browser processes using Xingtu's dedicated profile."""
profile_text = str(PROFILE_DIR.resolve()).casefold()
matches = []
for process in psutil.process_iter(["pid", "cmdline"]):
try:
command = " ".join(process.info.get("cmdline") or []).casefold()
if process.pid != os.getpid() and profile_text in command:
matches.append(process)
except (psutil.Error, OSError):
continue
for process in matches:
try:
process.terminate()
except (psutil.Error, OSError):
pass
_gone, alive = psutil.wait_procs(matches, timeout=5)
for process in alive:
try:
process.kill()
except (psutil.Error, OSError):
pass
def is_xingtu_week(today: dt.date, base: dt.date) -> bool:
days = (today - base).days
if days < 0:
return False
return (days // 7) % 3 == 0
def reset_state() -> tuple[Path | None, Path | None, Path | None]:
"""Temporarily move the old login state aside so it can be restored."""
ts = dt.datetime.now().strftime("%Y%m%d_%H%M%S_%f")
cookie_backup = None
state_backup = None
profile_backup = None
if COOKIE_FILE.exists():
cookie_backup = COOKIE_FILE.with_suffix(f".json.bak.{ts}")
if STATE_FILE.exists():
state_backup = STATE_FILE.with_suffix(f".json.bak.{ts}")
if PROFILE_DIR.exists():
profile_backup = PROFILE_DIR.with_name(f"{PROFILE_DIR.name}.bak.{ts}")
_write_transaction(cookie_backup, profile_backup, state_backup)
if cookie_backup:
COOKIE_FILE.rename(cookie_backup)
print(f" [OK] 备份 cookie -> {cookie_backup.name}")
else:
print(f" [INFO] {COOKIE_FILE.name} 不存在,跳过")
if state_backup:
STATE_FILE.rename(state_backup)
print(f" [OK] 备份 storage state -> {state_backup.name}")
else:
print(f" [INFO] {STATE_FILE.name} 不存在,跳过")
if profile_backup:
PROFILE_DIR.rename(profile_backup)
print(f" [OK] 暂存 {PROFILE_DIR.name}/ -> {profile_backup.name}/")
else:
print(f" [INFO] {PROFILE_DIR.name}/ 不存在,跳过")
return cookie_backup, profile_backup, state_backup
def restore_previous_state(
cookie_backup: Path | None,
profile_backup: Path | None,
state_backup: Path | None = None,
) -> None:
"""Discard an incomplete login and restore the last known-good state."""
terminate_profile_processes()
if COOKIE_FILE.exists():
COOKIE_FILE.unlink()
if cookie_backup and cookie_backup.exists():
cookie_backup.rename(COOKIE_FILE)
print(f" [ROLLBACK] 已恢复旧 cookie: {COOKIE_FILE.name}")
_remove_path(STATE_FILE)
if state_backup and state_backup.exists():
state_backup.rename(STATE_FILE)
print(f" [ROLLBACK] 已恢复旧 storage state: {STATE_FILE.name}")
if PROFILE_DIR.exists():
shutil.rmtree(PROFILE_DIR)
if profile_backup and profile_backup.exists():
profile_backup.rename(PROFILE_DIR)
print(f" [ROLLBACK] 已恢复旧浏览器状态: {PROFILE_DIR.name}/")
_clear_transaction()
def finish_successful_relogin(profile_backup: Path | None) -> None:
"""Keep the new state and remove the bulky old profile backup."""
if profile_backup and profile_backup.exists():
shutil.rmtree(profile_backup)
_clear_transaction()
def sync_to_douyin_scraper() -> None:
"""把星图的登录态 cookie 同步到抖音评论 scraper,避免再扫一次码。"""
if not COOKIE_FILE.exists():
print(" [SKIP] 星图 cookie 不存在,无法同步")
return
try:
xingtu_cookies = json.loads(COOKIE_FILE.read_text(encoding="utf-8"))
except Exception as exc:
print(f" [SKIP] 读取星图 cookie 失败: {exc}")
return
# 提取登录态 cookie
login_cookies = [c for c in xingtu_cookies if isinstance(c, dict) and c.get("name") in DY_LOGIN_COOKIES]
if not login_cookies:
print(" [SKIP] 星图 cookie 中没有找到登录态,跳过同步")
return
# 清除抖音评论 scraper 的旧 profile (强制刷新)
if DY_PROFILE_DIR.exists():
shutil.rmtree(DY_PROFILE_DIR)
print(f" [OK] 清空 {DY_PROFILE_DIR.name}/")
# 读取或创建抖音评论 cookie 文件
dy_cookies = []
if DY_COOKIE_FILE.exists():
try:
dy_cookies = json.loads(DY_COOKIE_FILE.read_text(encoding="utf-8"))
except Exception:
dy_cookies = []
# 合并: 同名 cookie 覆盖,新 cookie 追加
dy_by_name = {c.get("name"): c for c in dy_cookies if isinstance(c, dict)}
for c in login_cookies:
# 修正 domain: 星图的 cookie domain 可能是 .douyin.com 子域,评论采集需要 .douyin.com
domain = c.get("domain", "")
if "xingtu" in domain or "oceanengine" in domain:
c["domain"] = ".douyin.com"
dy_by_name[c["name"]] = c
DY_COOKIE_FILE.parent.mkdir(parents=True, exist_ok=True)
DY_COOKIE_FILE.write_text(
json.dumps(list(dy_by_name.values()), ensure_ascii=False, indent=2),
encoding="utf-8",
)
print(f" [OK] 同步 {len(login_cookies)} 个登录 cookie 到 {DY_COOKIE_FILE.name}")
def main() -> int:
ap = argparse.ArgumentParser(description="每 3 周周一重登星图")
ap.add_argument("--base-date", default=BASE_DATE.isoformat(),
help="3 周循环起点(必须是周一),格式 YYYY-MM-DD")
ap.add_argument("--login-timeout", type=int, default=300)
ap.add_argument("--force", action="store_true", help="忽略 3 周判断,强制重登")
ap.add_argument("--dry-run", action="store_true", help="只打印计划")
args = ap.parse_args()
today = dt.date.today()
base = dt.date.fromisoformat(args.base_date)
is_xw = is_xingtu_week(today, base)
print("== Relogin: 抖音/星图 ==")
print(f" 今天: {today} ({today.strftime('%A')})")
print(f" 3 周基准: {base}")
print(f" 星图周? {'是' if is_xw else '否'}")
print()
if not is_xw and not args.force:
print("今天不是星图周,跳过 (传 --force 强制重登)")
return 0
if args.dry_run:
print("[DRY-RUN] 实际跑会: 删 cookie + 清 profile + 弹二维码")
return 0
recover_interrupted_state()
terminate_profile_processes()
cookie_backup, profile_backup, state_backup = reset_state()
cmd = [
sys.executable,
str(SCRAPER),
"--login-only",
"--login-timeout",
str(args.login_timeout),
]
print(f"\n [RUN] {' '.join(cmd)}")
try:
rc = subprocess.call(
cmd,
cwd=str(PROJECT_DIR),
env=child_browser_environment(
cookie_file=COOKIE_FILE,
storage_state_file=STATE_FILE,
profile_dir=PROFILE_DIR,
),
)
except BaseException:
restore_previous_state(cookie_backup, profile_backup, state_backup)
raise
finally:
terminate_profile_processes()
print(f"\n exit={rc}")
if rc != 0 or not has_valid_login_cookie_file():
print(" [WARN] 新登录未完成,恢复旧登录状态")
restore_previous_state(cookie_backup, profile_backup, state_backup)
return rc or 1
try:
sync_relogin_state_to_consumers(
"xingtu",
cookie_file=COOKIE_FILE,
storage_state_file=STATE_FILE,
)
except Exception as exc:
restore_previous_state(cookie_backup, profile_backup, state_backup)
print(f" [ROLLBACK] 同步采集器登录态失败: {exc}")
return 1
finish_successful_relogin(profile_backup)
if rc == 0:
sync_to_douyin_scraper()
return rc
if __name__ == "__main__":
sys.exit(main())
@@ -426,6 +426,21 @@ def ensure_table(cur) -> None:
)
def drop_stale_rows(
all_rows: dict[tuple, tuple], stale_paths: list[Path]
) -> int:
"""删除来源文件已陈旧的行,返回删除数(按行内 source_file 文件名匹配)。"""
stale_names = {Path(path).name for path in stale_paths}
source_index = ROW_FIELDS.index("source_file")
stale_keys = [
key for key, row in all_rows.items()
if str(row[source_index]) in stale_names
]
for key in stale_keys:
del all_rows[key]
return len(stale_keys)
def main() -> int:
parser = argparse.ArgumentParser(
description="同步自营内容(抖音/B站/小红书)全量指标到 cmt_self_video_metrics"
@@ -504,19 +519,24 @@ def main() -> int:
print("❌ 没有任何平台的来源文件可用")
return 1
stale = [
str(path)
path
for path in sources
if business_date_bound
and not args.allow_stale
and not args.use_file_mtime
and not is_current_artifact(path, metric_date)
]
if stale and not args.allow_stale:
if stale:
# 陈旧防护按来源文件粒度跳过: 某平台来源过期只丢弃该平台的行,
# 不拖累其他平台当天的新鲜数据(如抖音导出失败时不拦 B站/小红书)。
dropped = drop_stale_rows(all_rows, stale)
print(
f"❌ 以下来源文件不是业务日期({metric_date.isoformat()})的产物; "
"补历史请显式传 --metric-date 并加 --allow-stale:\n "
+ "\n ".join(stale)
f"⚠️ 以下来源文件不是业务日期({metric_date.isoformat()})的产物, "
f"已跳过对应 {dropped} 行(补历史用 --metric-date + "
"--allow-stale):\n " + "\n ".join(str(p) for p in stale)
)
if not all_rows:
print("❌ 跳过陈旧来源后没有可入库的内容指标行")
return 1
print(
@@ -543,9 +563,29 @@ def main() -> int:
from gyxx_flow.modules.content_marketing.data.tools.db import get_db_config
with psycopg.connect(**get_db_config()) as conn:
written, failed = upsert_rows(conn, list(all_rows.values()))
if failed:
print(f"[sync-self] 完成: 写入 {written} 行,单行失败跳过 {failed} 行 "
f"(见上方 ⚠️ 日志)")
return 0 if written else 1
print(f"[sync-self] 完成: 已幂等写入 {written} 行到 {TABLE_NAME}")
return 0
def upsert_rows(conn, rows: list[tuple]) -> tuple[int, int]:
"""逐行 upsert,返回 (成功数, 失败数)。
每行放在独立保存点里执行: 单行失败(如超长值违反约束)只回滚该行,
不拖累其他平台/其他内容的写入——这是「一个平台失败不影响全库」的
最后一道隔离。
"""
written = 0
failed = 0
with conn.cursor() as cur:
ensure_table(cur)
for row in all_rows.values():
for row in rows:
try:
with conn.transaction():
# SQL 为静态字面量,29 个值全部经 %s 占位符参数绑定传入。
cur.execute(
"""
@@ -607,9 +647,13 @@ def main() -> int:
""",
row,
)
except Exception as exc:
failed += 1
print(f" ⚠️ 单行写入失败已隔离跳过: {type(exc).__name__}: {exc}")
else:
written += 1
conn.commit()
print(f"[sync-self] 完成: 已幂等写入 {len(all_rows)} 行到 {TABLE_NAME}")
return 0
return written, failed
if __name__ == "__main__":
@@ -273,18 +273,15 @@ def maybe_wait_for_login(
*,
deadline: float | None = None,
) -> None:
login_deadline = time.monotonic() + login_timeout
while time.monotonic() < login_deadline:
_check_note_deadline(deadline, "login wait")
# 登录统一由账号 vault 管理(gyxx accounts login douyin-content),
# 由工作流执行前的 cookie 网关保证;运行中失效立即降级,不再等待人工扫码。
try:
body = page.locator("body").inner_text(timeout=5000)
except Exception:
body = ""
if "扫码登录" not in body and "登录后" not in body:
return
log("Douyin may require login. Please finish login in the opened Chrome window.")
page.wait_for_timeout(3000)
_check_note_deadline(deadline, "login wait")
if "扫码登录" in body or "登录后" in body:
log("Douyin login state missing; collecting degraded samples. "
"Run `gyxx accounts login douyin-content` to restore full comments.")
def is_logged_in(page: Page) -> bool:
@@ -833,55 +830,16 @@ def write_outputs(url: str, comments: list[dict[str, Any]], result: dict[str, An
return json_path, csv_path
def login_only(login_timeout: int) -> int:
"""只触发登录(headed),登录完成立刻退出。给 relogin_douyin.py 用。"""
def action(page: Page) -> None:
page.goto("https://www.douyin.com/", wait_until="domcontentloaded", timeout=90000)
page.wait_for_timeout(2500)
# 主动触发登录弹窗(点登录按钮)
try:
login_btn = page.get_by_text("登录", exact=True).first
if login_btn.count():
login_btn.click(timeout=3000)
page.wait_for_timeout(1500)
except Exception:
pass
maybe_wait_for_login(page, login_timeout)
save_state(page)
PROFILE_DIR.mkdir(parents=True, exist_ok=True)
with ScraplingBrowser(
headless=False,
real_chrome=True,
user_data_dir=str(PROFILE_DIR),
locale="zh-CN",
timezone_id="Asia/Shanghai",
timeout=90000,
network_idle=False,
disable_resources=False,
google_search=False,
page_setup=restore_cookies,
max_pages=1,
) as browser:
browser.run_fetch_action("https://www.douyin.com/", action, wait=1000)
log("[login-only] 完成")
return 0
def main() -> int:
parser = argparse.ArgumentParser(description="Scrape Douyin video comments via Scrapling browser and XHR interception.")
parser.add_argument("url", help="Douyin video/user URL (传 'login' 配合 --login-only)")
parser.add_argument("--login-timeout", type=int, default=300, help="等待扫码/登录秒数")
parser.add_argument("url", help="Douyin video/user URL")
parser.add_argument("--login-timeout", type=int, default=300,
help="保留参数(登录等待已移除;未登录将直接降级采集)")
parser.add_argument("--max-scrolls", type=int, default=120, help="最多滚动加载次数")
parser.add_argument("--idle-rounds", type=int, default=8, help="连续几轮无新评论后停止")
parser.add_argument("--headless", action="store_true", help="无头模式;首次登录不要使用")
parser.add_argument("--login-only", action="store_true",
help="只触发登录(headed)然后退出,给 relogin_douyin 用")
parser.add_argument("--headless", action="store_true", help="无头模式")
args = parser.parse_args()
if args.login_only:
return login_only(args.login_timeout)
comments, result = scrape_comments(args.url, args.login_timeout, args.max_scrolls, args.idle_rounds, args.headless)
json_path, csv_path = write_outputs(args.url, comments, result)
log(f"Final URL: {result.get('final_url', '')}")
@@ -38,6 +38,7 @@ from gyxx_flow.adapters.scrapling import (
BrowserTimeoutError, # noqa: F401
is_browser_timeout_error,
)
from gyxx_flow.adapters.browser import CookieExpiredError, cookie_expired_message
from gyxx_flow.core.exit_codes import COOKIE_SKIP_EXIT_CODE
from gyxx_flow.modules.content_marketing.collection_completeness import (
BLOCKED_INPUT,
@@ -333,56 +334,6 @@ def is_logged_in(page: Page) -> bool:
return False
def open_scan_login(page: Page) -> None:
"""Open the login dialog and prefer its QR-code login tab."""
for pattern in (r"^登录$|账号登录",):
try:
button = page.get_by_text(re.compile(pattern)).first
if button.count() and button.is_visible():
button.click(timeout=2000)
page.wait_for_timeout(500)
except Exception:
pass
# The login dialog uses an image-only folded-corner switch. Its bundled
# asset name is stable even though the generated CSS class is not.
try:
qr_switch = page.locator('img[src*="qr_code"]').first
if qr_switch.count() and qr_switch.is_visible():
# The browser engine's actionability check is blocked by the folded-corner
# overlay. A native DOM click triggers the image's real handler.
qr_switch.evaluate("e => e.click()")
page.wait_for_timeout(1000)
return
except Exception:
pass
for pattern in (r"扫码登录|二维码登录|扫码",):
try:
button = page.get_by_text(re.compile(pattern)).first
if button.count() and button.is_visible():
button.click(timeout=2000)
page.wait_for_timeout(500)
except Exception:
pass
def wait_for_scan_login(page: Page, timeout_seconds: int) -> None:
log("Not logged in. 请在打开的 Chrome 中扫码登录。")
open_scan_login(page)
deadline = time.time() + timeout_seconds
next_login_click = time.time() + 5
while time.time() < deadline:
if is_logged_in(page):
save_state(page)
return
if time.time() >= next_login_click:
open_scan_login(page)
next_login_click = time.time() + 5
page.wait_for_timeout(2000)
raise TimeoutError(f"Login timed out after {timeout_seconds} seconds.")
def ensure_login(page: Page, login_timeout: int) -> None:
page.goto(HOME_URL, wait_until="domcontentloaded", timeout=60000)
page.wait_for_timeout(2500)
@@ -402,20 +353,11 @@ def ensure_login(page: Page, login_timeout: int) -> None:
)
log(f"[SKIPPED_COOKIE] {reason}")
raise AcceptanceCookieSkip(reason)
page.goto(LOGIN_URL, wait_until="domcontentloaded", timeout=60000)
page.wait_for_timeout(1500)
wait_for_scan_login(page, login_timeout)
page.goto(HOME_URL, wait_until="domcontentloaded", timeout=60000)
page.wait_for_load_state("domcontentloaded")
# 关闭弹窗
for text in ("跳过", "不再提示", "我知道了"):
try:
b = page.get_by_text(text, exact=True).first
if b.count():
b.click(timeout=1500)
page.wait_for_timeout(800)
except Exception:
pass
# 登录统一由账号 vault 管理(gyxx accounts login xiaohongshu-content),
# 采集脚本不再内嵌扫码等待;运行中失效直接快速失败。
reason = cookie_expired_message("xiaohongshu-content", "小红书蒲公英")
log(f"[COOKIE_EXPIRED] {reason}")
raise CookieExpiredError(reason)
def _is_login_timeout(exc: BaseException) -> bool:
@@ -1933,7 +1875,7 @@ def scrape_styles(styles: list[dict], login_timeout: int, headless: bool,
action,
page_setup=restore_cookies,
)
except AcceptanceCookieSkip:
except (AcceptanceCookieSkip, CookieExpiredError):
raise
except Exception as exc:
session_lost = is_browser_session_lost(exc)
@@ -2044,39 +1986,6 @@ def parse_int(s) -> int | None:
return int(num)
def login_only(login_timeout: int) -> int:
"""只触发登录(headed),登录完成立刻退出。给 weekly_relogin.py 用。"""
def action(page):
ensure_login(page, login_timeout)
return page
session = None
try:
with DynamicSession(
headless=False,
real_chrome=True,
user_data_dir=str(PROFILE_DIR),
locale="zh-CN",
timezone_id="Asia/Shanghai",
timeout=60000,
network_idle=False,
disable_resources=False,
google_search=False,
page_setup=restore_cookies,
max_pages=2,
) as session:
session.fetch(HOME_URL, page_action=action, wait=1000)
except Exception as exc:
if "TargetClosedError" in type(exc).__name__ or "has been closed" in str(exc):
log(f" [WARN] 浏览器清理时报错: {type(exc).__name__}")
else:
raise
finally:
_force_cleanup_session(session)
log("[login-only] 登录完成,退出")
return 0
def main() -> int:
parser = argparse.ArgumentParser(description="小红书蒲公英 V2: 多维表格驱动精准采集+回填")
add_repeatable_style_argument(parser, "只跑指定款式编号(可多次)")
@@ -2090,8 +1999,6 @@ def main() -> int:
parser.add_argument("--skip-done", action="store_true", help="跳过已完成的款式(从进度文件读)")
parser.add_argument("--inter-style-pause", type=int, default=5,
help="兼容旧参数;全局按达人采集后不再按款式暂停")
parser.add_argument("--login-only", action="store_true",
help="只触发登录(headed),登录完成立刻退出,给 weekly_relogin 用")
parser.add_argument("--self-operated", action="store_true",
help="抓取自营达人表格(而非合作达人)")
parser.add_argument("--batch-size", type=int, default=DEFAULT_BATCH_SIZE,
@@ -2108,12 +2015,6 @@ def main() -> int:
help=argparse.SUPPRESS)
args = parser.parse_args()
if args.login_only:
try:
return login_only(args.login_timeout)
except AcceptanceCookieSkip:
return COOKIE_SKIP_EXIT_CODE
mapping = load_mapping(self_operated=args.self_operated)
styles = mapping["tables"]
if args.style:
@@ -2213,7 +2114,7 @@ def main() -> int:
published_from=args.published_from,
max_age_days=args.max_age_days, no_retry=args.no_retry,
)
except AcceptanceCookieSkip:
except (AcceptanceCookieSkip, CookieExpiredError):
return COOKIE_SKIP_EXIT_CODE
except Exception as exc:
if not _is_login_timeout(exc):
@@ -137,6 +137,7 @@ class ScraplingDriver:
cdp_url: str | None = None,
stealthy: bool = False,
real_chrome: bool = True,
extra_flags: list[str] | None = None,
) -> ScraplingDriver:
options: dict[str, Any] = {
"headless": False,
@@ -152,6 +153,7 @@ class ScraplingDriver:
"extra_flags": [
"--disable-blink-features=AutomationControlled",
"--disable-dev-shm-usage",
*(extra_flags or []),
],
}
reuse_existing_context = bool(cdp_url)
@@ -211,6 +213,9 @@ class ScraplingDriver:
[script, raw_arguments],
)
def save_screenshot(self, path: str | os.PathLike[str]) -> bool:
return bool(self._page.screenshot(path=str(Path(path))))
def get_cookies(self) -> list[dict[str, Any]]:
context = self._owner.context
return list(context.cookies()) if context is not None else []
@@ -363,8 +363,8 @@ def log(msg: str) -> None:
# ============================================================
# B站登录态: Cookie 持久化 + 扫码登录引导
# 空间投稿列表需要登录态,首次用 --login-only 扫码,之后调度运行复用。
# B站登录态: Cookie 持久化;登录统一由账号 vault 管理
# (gyxx accounts login bilibili-content),脚本内不再提供扫码登录。
# ============================================================
BILI_COOKIE_FILE = PATHS.browser_cookie_file
@@ -389,56 +389,6 @@ def save_state(page) -> None:
log(f"已保存 Cookie ({len(cookies)} 条): {BILI_COOKIE_FILE}")
def _has_login_cookies(page) -> bool:
names = {cookie.get("name") for cookie in page.context.cookies()}
return "SESSDATA" in names and "DedeUserID" in names
def login_only(login_timeout: int = 300) -> int:
"""打开 B站扫码登录页,等待人工扫码后保存 Cookie。"""
from scrapling.fetchers import StealthySession
with StealthySession(
headless=False,
real_chrome=True,
user_data_dir=str(PATHS.browser_profile_dir),
locale="zh-CN",
timezone_id="Asia/Shanghai",
timeout=60_000,
network_idle=False,
disable_resources=False,
google_search=False,
max_pages=2,
page_setup=restore_cookies,
) as session:
page = (
session.context.pages[0]
if session.context.pages
else session.context.new_page()
)
page.goto("https://passport.bilibili.com/login",
wait_until="domcontentloaded", timeout=60_000)
page.wait_for_timeout(2000)
if _has_login_cookies(page):
log("✅ 已有 B站登录态,无需扫码")
save_state(page)
return 0
print("\n" + "=" * 60)
print("🔐 请在打开的浏览器中用哔哩哔哩 App 扫码登录")
print(" 登录成功后脚本会自动继续并保存 Cookie")
print("=" * 60, flush=True)
deadline = time.time() + login_timeout
while time.time() < deadline:
if _has_login_cookies(page):
log("✅ 检测到登录成功")
page.wait_for_timeout(2000)
save_state(page)
return 0
page.wait_for_timeout(2000)
log(f"❌ 等待扫码超时({login_timeout}s),未保存登录态")
return 1
# ============================================================
# 飞书多维表格
# ============================================================
@@ -846,17 +796,10 @@ def main() -> int:
description="自营 B 站采集: 公开 API 全量视频指标 + UP 主主页全量投稿 + 飞书曝光回填")
parser.add_argument("--style", type=int, help="只跑指定款式编号")
parser.add_argument("--dry-run", action="store_true", help="不写回飞书")
parser.add_argument("--login-only", action="store_true",
help="只打开 B站扫码登录并保存 Cookie(首次部署引导)")
parser.add_argument("--login-timeout", type=int, default=300,
help="等待人工扫码的秒数")
parser.add_argument("--delay", type=float, default=0.3,
help="每条请求间隔秒数(防风控)")
args = parser.parse_args()
if args.login_only:
return login_only(args.login_timeout)
from gyxx_flow.modules.content_marketing import feishu_mapping
try:
mapping = feishu_mapping.load_mapping(self_operated=True)
@@ -27,6 +27,8 @@ from typing import Any
from scrapling.fetchers import DynamicSession
from gyxx_flow.adapters.scrapling import BrowserPage as Page
from gyxx_flow.adapters.browser import CookieExpiredError, cookie_expired_message
from gyxx_flow.core.exit_codes import COOKIE_SKIP_EXIT_CODE
from gyxx_flow.modules.content_marketing.collection_completeness import (
BLOCKED_INPUT,
@@ -261,17 +263,16 @@ def save_state(page: Page) -> None:
def maybe_wait_for_login(page: Page, login_timeout: int) -> None:
deadline = time.time() + login_timeout
while time.time() < deadline:
# 登录统一由账号 vault 管理(gyxx accounts login douyin-content),
# 采集过程遇到登录态失效立即快速失败,不再等待人工扫码。
try:
body = page.locator("body").inner_text(timeout=5000)
except Exception:
body = ""
if "扫码登录" not in body and "登录后" not in body:
return
log("抖音需要登录,请在打开的 Chrome 窗口中完成登录")
page.wait_for_timeout(3000)
log("[WARN] 登录等待超时")
if "扫码登录" in body or "登录后" in body:
reason = cookie_expired_message("douyin-content", "自营抖音")
log(f"[COOKIE_EXPIRED] {reason}")
raise CookieExpiredError(reason)
def dismiss_popups(page: Page) -> None:
@@ -605,33 +606,6 @@ def scrape_one_style(style: dict, login_timeout: int, headless: bool,
return summary
def login_only(login_timeout: int) -> int:
"""只触发登录"""
login_done = False
def page_action(page: Page):
restore_cookies(page)
page.goto("https://www.douyin.com/", wait_until="domcontentloaded", timeout=60000)
page.wait_for_timeout(2000)
maybe_wait_for_login(page, login_timeout)
save_state(page)
log("登录完成")
PROFILE_DIR.mkdir(parents=True, exist_ok=True)
with DynamicSession(
headless=False,
real_chrome=True,
user_data_dir=str(PROFILE_DIR),
locale="zh-CN",
timezone_id="Asia/Shanghai",
timeout=60000,
page_setup=restore_cookies,
) as session:
session.fetch("https://www.douyin.com/", page_action=page_action, wait=1000)
return 0 if login_done else 1
# ============================================================
# main
# ============================================================
@@ -642,12 +616,8 @@ def main() -> int:
parser.add_argument("--headless", action="store_true")
parser.add_argument("--login-timeout", type=int, default=300)
parser.add_argument("--dry-run", action="store_true", help="不写回多维表格")
parser.add_argument("--login-only", action="store_true", help="只触发登录")
args = parser.parse_args()
if args.login_only:
return login_only(args.login_timeout)
mapping = load_mapping()
styles = mapping["tables"]
if args.style:
@@ -703,6 +673,9 @@ def main() -> int:
if __name__ == "__main__":
try:
raise SystemExit(main())
except CookieExpiredError as exc:
print(f"[COOKIE_EXPIRED] {exc}", file=sys.stderr)
raise SystemExit(COOKIE_SKIP_EXIT_CODE)
except KeyboardInterrupt:
print("\nInterrupted.", file=sys.stderr)
raise SystemExit(130)
@@ -34,6 +34,7 @@ from gyxx_flow.adapters.scrapling import (
BrowserTimeoutError, # noqa: F401
is_browser_timeout_error,
)
from gyxx_flow.adapters.browser import CookieExpiredError
from gyxx_flow.core.exit_codes import COOKIE_SKIP_EXIT_CODE
from gyxx_flow.modules.content_marketing import self_operated_accounts
from gyxx_flow.modules.content_marketing.runtime_paths import PATHS
@@ -375,24 +376,14 @@ def ensure_login(page, login_timeout: int, phone: str) -> None:
log(f"[SKIPPED_COOKIE] {reason}")
raise AcceptanceCookieSkip(reason)
log("🔄 登录态失效,进入短信验证码登录流程")
page.goto(LOGIN_URL, wait_until="domcontentloaded", timeout=60_000)
page.wait_for_timeout(2000)
# 确保在「短信登录」页签
for text in ("短信登录",):
try:
tab = page.get_by_text(text, exact=True).first
if tab.count() and tab.is_visible():
tab.click(timeout=2000)
page.wait_for_timeout(800)
break
except Exception:
continue
_fill_sms_login(page, phone)
wait_for_login(page, login_timeout)
save_state(page)
page.goto(GOODS_URL, wait_until="domcontentloaded", timeout=60_000)
page.wait_for_timeout(2000)
# 采集流程不再内嵌短信验证码登录;统一由 --login-only 手动引导建立登录态。
reason = (
"小红书 ARK 后台登录态已失效:请手动执行 "
"`uv run python src/gyxx_flow/modules/content_marketing/self_xhs_scraper.py "
"--login-only` 完成短信验证码登录后重跑"
)
log(f"[COOKIE_EXPIRED] {reason}")
raise CookieExpiredError(reason)
# ============================================================
@@ -855,6 +846,8 @@ def collect(headless: bool, login_timeout: int, max_pages: int,
google_search=False,
page_setup=restore_cookies,
max_pages=3,
# 国内站直连:代理出口被站点 WAF 拉黑时页面/API 直接 ERR_CONNECTION_CLOSED
extra_flags=["--no-proxy-server"],
) as session:
page = _open_page(session)
ensure_login(page, login_timeout, phone)
@@ -927,7 +920,20 @@ def login_only(headless: bool, login_timeout: int) -> int:
max_pages=2,
) as session:
page = _open_page(session)
ensure_login(page, login_timeout, phone)
page.goto(LOGIN_URL, wait_until="domcontentloaded", timeout=60_000)
page.wait_for_timeout(2000)
for text in ("短信登录",):
try:
tab = page.get_by_text(text, exact=True).first
if tab.count() and tab.is_visible():
tab.click(timeout=2000)
page.wait_for_timeout(800)
break
except Exception:
continue
_fill_sms_login(page, phone)
wait_for_login(page, login_timeout)
save_state(page)
log("✅ 登录态已建立并保存 Cookie")
return 0
@@ -953,7 +959,7 @@ def main() -> int:
return login_only(args.headless, args.login_timeout)
return collect(args.headless, args.login_timeout, args.max_pages,
args.dry_run)
except AcceptanceCookieSkip:
except (AcceptanceCookieSkip, CookieExpiredError):
return COOKIE_SKIP_EXIT_CODE
except KeyboardInterrupt:
print("\n⛔ 用户中断", file=sys.stderr)
@@ -223,18 +223,15 @@ def maybe_wait_for_login(
*,
deadline: float | None = None,
) -> None:
login_deadline = time.monotonic() + max(0, login_timeout)
while time.monotonic() < login_deadline:
_check_note_deadline(deadline, "login wait")
# 登录统一由账号 vault 管理(gyxx accounts login xiaohongshu-content),
# 由工作流执行前的 cookie 网关保证;运行中失效立即降级,不再等待人工扫码。
try:
body = page.locator("body").inner_text(timeout=5000)
except Exception:
body = ""
if "扫码登录" not in body and "登录后" not in body and "验证码" not in body:
return
log("Xiaohongshu may require login. Please finish login in the opened Chrome window.")
page.wait_for_timeout(3000)
_check_note_deadline(deadline, "login wait")
if "扫码登录" in body or "登录后" in body or "验证码" in body:
log("Xiaohongshu login state missing; collecting degraded samples. "
"Run `gyxx accounts login xiaohongshu-content` to restore full comments.")
def is_logged_in(page: Page) -> bool:
@@ -751,55 +748,16 @@ def write_outputs(url: str, comments: list[dict[str, Any]], result: dict[str, An
return json_path, csv_path
def login_only(login_timeout: int) -> int:
"""只触发登录(headed),登录完成立刻退出。给 relogin_xiaohongshu.py 用。"""
def action(page: Page) -> None:
page.goto("https://www.xiaohongshu.com/", wait_until="domcontentloaded", timeout=90000)
page.wait_for_timeout(2500)
# 主动点登录按钮触发二维码
try:
login_btn = page.get_by_text("登录", exact=True).first
if login_btn.count():
login_btn.click(timeout=3000)
page.wait_for_timeout(1500)
except Exception:
pass
maybe_wait_for_login(page, login_timeout)
save_state(page)
PROFILE_DIR.mkdir(parents=True, exist_ok=True)
with ScraplingBrowser(
headless=False,
real_chrome=True,
user_data_dir=str(PROFILE_DIR),
locale="zh-CN",
timezone_id="Asia/Shanghai",
timeout=90000,
network_idle=False,
disable_resources=False,
google_search=False,
page_setup=restore_cookies,
max_pages=1,
) as browser:
browser.run_fetch_action("https://www.xiaohongshu.com/", action, wait=1000)
log("[login-only] 完成")
return 0
def main() -> int:
parser = argparse.ArgumentParser(description="Scrape Xiaohongshu note comments with a real browser.")
parser.add_argument("url", help="Xiaohongshu explore/note URL (传 'login' 配合 --login-only)")
parser.add_argument("--login-timeout", type=int, default=300, help="等待扫码/登录秒数")
parser.add_argument("url", help="Xiaohongshu explore/note URL")
parser.add_argument("--login-timeout", type=int, default=300,
help="保留参数(登录等待已移除;未登录将直接降级采集)")
parser.add_argument("--max-scrolls", type=int, default=120, help="最多滚动加载次数")
parser.add_argument("--idle-rounds", type=int, default=8, help="连续几轮无新评论后停止")
parser.add_argument("--headless", action="store_true", help="无头模式;首次登录不要使用")
parser.add_argument("--login-only", action="store_true",
help="只触发登录(headed)然后退出,给 relogin_xiaohongshu 用")
parser.add_argument("--headless", action="store_true", help="无头模式")
args = parser.parse_args()
if args.login_only:
return login_only(args.login_timeout)
comments, result = scrape_comments(args.url, args.login_timeout, args.max_scrolls, args.idle_rounds, args.headless)
json_path, csv_path = write_outputs(args.url, comments, result)
log(f"Final URL: {result.get('final_url', '')}")
@@ -33,6 +33,7 @@ from gyxx_flow.adapters.scrapling import (
BrowserTimeoutError,
is_browser_timeout_error,
)
from gyxx_flow.adapters.browser import CookieExpiredError, cookie_expired_message
from gyxx_flow.core.exit_codes import COOKIE_SKIP_EXIT_CODE
from gyxx_flow.modules.content_marketing.collection_completeness import (
BLOCKED_INPUT,
@@ -537,183 +538,6 @@ def is_logged_in(page: Page) -> bool:
return False
def open_scan_login(page: Page) -> None:
"""Open the selected platform's login dialog without handling passwords."""
if CONTENT_SOURCE == "buyin":
if not is_business_page(page) or "/dashboard/servicehall/daren-square" not in page.url:
try:
page.goto(CUSTOMER_LOGIN_URL, wait_until="domcontentloaded", timeout=60000)
page.wait_for_timeout(1500)
except Exception:
pass
for pattern in (r"^登录$", r"扫码登录|二维码登录|抖音登录|使用抖音登录"):
try:
button = page.get_by_text(re.compile(pattern)).first
if button.count() and button.is_visible():
button.click(timeout=3000)
page.wait_for_timeout(800)
return
except Exception as exc:
reraise_if_browser_session_lost(exc)
return
if "sso.oceanengine.com/xingtu/login" not in page.url:
try:
page.goto(CUSTOMER_LOGIN_URL, wait_until="domcontentloaded", timeout=60000)
page.wait_for_timeout(1500)
except Exception:
pass
# The Douyin login icon on Xingtu's customer SSO page has a stable
# aweme.png source. Clicking it opens the QR authorization page directly.
try:
douyin = page.locator('img[src*="aweme.png"]').first
if douyin.count() and douyin.is_visible():
douyin.click(timeout=3000)
page.wait_for_timeout(1500)
return
except Exception:
pass
for pattern in (r"^登录$|立即登录", r"抖音扫码登录|扫码登录|二维码登录|扫码"):
try:
button = page.get_by_text(re.compile(pattern)).first
if button.count() and button.is_visible():
button.click(timeout=2000)
page.wait_for_timeout(500)
except Exception:
pass
def is_scan_qr_page(page: Page) -> bool:
"""Return True once Douyin's QR authorization page is visible.
The QR page lives outside xingtu.cn. Treating that URL as a page that still
needs opening causes the login helper to navigate back and click again every
few seconds, continuously refreshing the QR code.
"""
url = page.url.lower()
if any(host in url for host in ("open.douyin.com", "sso.douyin.com")):
return True
for selector in (
'img[src*="qrcode"]',
'img[src*="qr_code"]',
'canvas',
):
try:
locator = page.locator(selector).first
if locator.count() and locator.is_visible():
return True
except Exception:
pass
return False
def resolve_live_page(page: Page, *, create_if_missing: bool = False) -> Page | None:
"""Return a usable page after OAuth closes or replaces its QR tab."""
try:
if not page.is_closed():
return page
except Exception:
pass
try:
context = page.context
for candidate in reversed(list(context.pages)):
try:
if not candidate.is_closed():
return candidate
except Exception:
continue
if create_if_missing:
return context.new_page()
except Exception:
return None
return None
def find_logged_in_page(page: Page) -> Page | None:
"""Find an authenticated business tab even while the OAuth tab stays open."""
candidates: list[Page] = []
try:
candidates.extend(reversed(list(page.context.pages)))
except Exception:
pass
candidates.append(page)
seen: set[int] = set()
for candidate in candidates:
marker = id(candidate)
if marker in seen:
continue
seen.add(marker)
try:
if candidate.is_closed() or not is_business_page(candidate):
continue
if is_logged_in(candidate):
return candidate
except Exception:
continue
return None
def wait_for_scan_login(page: Page, timeout_seconds: int) -> Page:
log("Not logged in. 请在打开的 Chrome 中扫码登录。")
current_page = page
open_scan_login(current_page)
deadline = time.time() + timeout_seconds
next_login_click = time.time() + 5
while time.time() < deadline:
authenticated = find_logged_in_page(current_page)
if authenticated is not None:
save_state(authenticated)
return authenticated
replacement = resolve_live_page(current_page, create_if_missing=True)
if replacement is None:
raise RuntimeError("Xingtu browser context closed during OAuth login")
if replacement is not current_page:
current_page = replacement
try:
current_page.goto(
MARKET_URL, wait_until="domcontentloaded", timeout=60000
)
current_page.wait_for_timeout(2000)
except Exception:
pass
if (
CONTENT_SOURCE == "xingtu"
and "redirect_uri=/ad/creator/market" in current_page.url
and not is_scan_qr_page(current_page)
):
try:
current_page.goto(
MARKET_URL, wait_until="domcontentloaded", timeout=60000
)
current_page.wait_for_timeout(2000)
except Exception:
pass
authenticated = find_logged_in_page(current_page)
if authenticated is not None:
save_state(authenticated)
return authenticated
if (
time.time() >= next_login_click
and not is_scan_qr_page(current_page)
and not is_business_page(current_page)
):
open_scan_login(current_page)
next_login_click = time.time() + 5
try:
current_page.wait_for_timeout(2000)
except Exception:
# OAuth may close the QR tab after authorization. The next loop
# follows the surviving tab in the same browser context.
continue
raise TimeoutError(f"Login timed out after {timeout_seconds} seconds.")
def ensure_login(page: Page, login_timeout: int) -> Page:
page.goto(MARKET_URL, wait_until="domcontentloaded", timeout=60000)
page.wait_for_timeout(3000)
@@ -735,12 +559,11 @@ def ensure_login(page: Page, login_timeout: int) -> Page:
)
log(f"[SKIPPED_COOKIE] {reason}")
raise AcceptanceCookieSkip(reason)
page = wait_for_scan_login(page, login_timeout)
page.goto(MARKET_URL, wait_until="domcontentloaded", timeout=60000)
page.wait_for_timeout(3000)
if not is_logged_in(page):
page = wait_for_scan_login(page, login_timeout)
return page
# 登录统一由账号 vault 管理(gyxx accounts login douyin-shop),
# 采集脚本不再内嵌扫码等待;运行中失效直接快速失败。
reason = cookie_expired_message("douyin-shop", f"{CONTENT_SOURCE}达人广场")
log(f"[COOKIE_EXPIRED] {reason}")
raise CookieExpiredError(reason)
def dismiss_popups(page: Page) -> None:
@@ -3736,7 +3559,7 @@ def scrape_styles_with_fallback(
summary_override=summaries,
replace_existing=True,
)
except AcceptanceCookieSkip as exc:
except (AcceptanceCookieSkip, CookieExpiredError) as exc:
log(f" [WARN] 抖店补采登录态不可用,保留星图结果: {exc}")
_mark_fallback_not_processed(
fallback_tasks, summaries, "buyin_cookie_skipped"
@@ -3805,96 +3628,6 @@ def _force_cleanup_session(session) -> None:
pass
def login_only(login_timeout: int) -> int:
"""只触发登录(headed),登录完成立刻退出。给 weekly_relogin.py 用。"""
login_completed = False
def action(page):
nonlocal login_completed
page = ensure_login(page, login_timeout)
if not is_logged_in(page):
raise RuntimeError(f"{CONTENT_SOURCE} 登录状态验证失败")
save_state(page)
login_completed = True
return page
session = None
try:
with DynamicSession(
headless=False,
real_chrome=True,
user_data_dir=str(PROFILE_DIR),
locale="zh-CN",
timezone_id="Asia/Shanghai",
timeout=60000,
network_idle=False,
disable_resources=False,
google_search=False,
page_setup=restore_cookies,
max_pages=2,
) as session:
session.fetch(MARKET_URL, page_action=action, wait=1000)
except Exception as exc:
err_msg = str(exc).lower()
if "closed" in err_msg or "context" in err_msg or "target" in err_msg:
log(f"[login-only] 浏览器会话关闭异常: {exc}")
else:
log(f"[login-only] 登录执行失败: {exc}")
finally:
_force_cleanup_session(session)
# Douyin OAuth may close the only QR tab (and therefore the temporary
# browser context) after a successful mobile confirmation. The durable
# Chromium profile can already contain the new session even though the
# page_action never got a chance to export cookies. Reopen the same profile
# once, verify the selected business page, then export its cookies.
if not login_completed or not has_valid_login_cookie_file():
log(f"[login-only] 扫码窗口已结束,重新打开{CONTENT_SOURCE}验证持久会话")
login_completed = verify_persisted_profile_login()
if not login_completed or not has_valid_login_cookie_file():
log("[login-only] 登录未完成或Cookie缺少有效会话")
return 1
log("[login-only] 登录完成,退出")
return 0
def verify_persisted_profile_login() -> bool:
"""Reopen the persistent profile after OAuth closes its only tab."""
verified = False
def action(page: Page):
nonlocal verified
page.goto(MARKET_URL, wait_until="domcontentloaded", timeout=60000)
page.wait_for_timeout(3000)
if not is_logged_in(page):
raise RuntimeError(f"persisted {CONTENT_SOURCE} profile is not authenticated")
save_state(page)
verified = True
return page
session = None
try:
with DynamicSession(
headless=False,
real_chrome=True,
user_data_dir=str(PROFILE_DIR),
locale="zh-CN",
timezone_id="Asia/Shanghai",
timeout=60000,
network_idle=False,
disable_resources=False,
google_search=False,
page_setup=restore_cookies,
max_pages=1,
) as session:
session.fetch(MARKET_URL, page_action=action, wait=1000)
except Exception as exc:
log(f"[login-only] 持久会话验证失败: {exc}")
finally:
_force_cleanup_session(session)
return verified and has_valid_login_cookie_file()
def load_retry_scope(summary_path: Path, reason: str) -> dict[int, set[str]]:
"""Load exact style/record pairs for a reason-limited retry."""
try:
@@ -3926,8 +3659,6 @@ def main() -> int:
parser.add_argument("--login-timeout", type=int, default=300)
parser.add_argument("--dry-run", action="store_true")
parser.add_argument("--data-dir", default=str(PATHS.normalized_root))
parser.add_argument("--login-only", action="store_true",
help="只触发登录(headed),登录完成立刻退出,给 weekly_relogin 用")
parser.add_argument(
"--buyin",
action="store_true",
@@ -3952,12 +3683,6 @@ def main() -> int:
"buyin" if args.buyin else os.environ.get("GYXX_CONTENT_SOURCE", "xingtu")
)
if args.login_only:
try:
return login_only(args.login_timeout)
except AcceptanceCookieSkip:
return COOKIE_SKIP_EXIT_CODE
mapping = load_mapping(self_operated=args.self_operated)
styles = mapping["tables"]
if args.style:
@@ -4038,7 +3763,7 @@ def main() -> int:
max_age_days=args.max_age_days,
no_retry=args.no_retry,
)
except AcceptanceCookieSkip:
except (AcceptanceCookieSkip, CookieExpiredError):
return COOKIE_SKIP_EXIT_CODE
persist_summaries(all_summaries)
for s, summary in zip(styles, all_summaries):
@@ -20,6 +20,7 @@ PRODUCT_SCHEDULED_WORKFLOW_IDS = (
"product.main_image.weekly",
"product.sales_sheet.daily",
"product.erp_all_shop_daily",
"product.hk_anpei_public_inventory.daily",
"product.market_rank",
"product.tmall_baibu_apply",
"product.video_upload",
@@ -45,6 +46,7 @@ _WORKFLOW_SCOPED_RESOURCE_IDS = {
"product.daily",
"product.persona.daily",
"product.style_analysis.interval",
"product.hk_anpei_public_inventory.daily",
"product.ecommerce_costs.daily",
# Tmall and JD video publication use separate browser profiles, accounts,
# and channel-scoped PostgreSQL ledgers; do not let a long publication on
@@ -47,6 +47,10 @@ TASK_CENTER_URL = (
"https://qianchuan.jinritemai.com/tools/task-center/download-task"
"?aavid={advertiser_id}"
)
TASK_CENTER_PAGE_URL = TASK_CENTER_URL.format(advertiser_id=quote(ADVERTISER_ID))
# 任务中心深链缺账户上下文时,千川会把页面重定向到登录域的账户选择页;
# 该页每个账户是一张 .account-list-item 卡片,user-id 含唯一广告主 ID。
ACCOUNT_ITEM_SELECTOR = ".account-list-item"
TASK_SCENE_MARKER = "全域推广数据-推商品"
TASK_READY_MARKER = "下载任务列表"
TASK_COMPLETE_MARKER = "生成完成"
@@ -188,6 +192,70 @@ def _validate_task_center_page(page) -> None:
raise DouyinQianchuanDownloadError("巨量千川下载任务中心未达到可用状态")
def _on_account_selection_page(page) -> bool:
try:
return page.locator(ACCOUNT_ITEM_SELECTOR).count() > 0
except Exception:
return False
def _select_advertiser_account(page) -> None:
if not ADVERTISER_ID:
raise DouyinQianchuanDownloadError(
"巨量千川任务中心跳转到账户选择页,且未配置目标广告主 ID"
)
account = page.locator(ACCOUNT_ITEM_SELECTOR).filter(has_text=ADVERTISER_ID)
if account.count() == 0:
raise DouyinQianchuanDownloadError(
f"巨量千川账户选择页上未找到目标广告主:{ADVERTISER_ID}"
)
account.first.click()
def _wait_task_center_ready(page, *, timeout_ms: int = 120_000) -> None:
"""等待任务中心可用;被重定向到账户选择页时点击目标广告主后继续等待。"""
deadline = time.monotonic() + timeout_ms / 1000
selections = 0
regotos = 0
while True:
if _on_account_selection_page(page):
if selections >= 2:
raise DouyinQianchuanDownloadError(
"巨量千川选择目标广告主后仍停留在账户选择页"
)
selections += 1
_select_advertiser_account(page)
page.wait_for_timeout(3_000)
continue
remaining_ms = int((deadline - time.monotonic()) * 1000)
if remaining_ms <= 0:
raise DouyinQianchuanDownloadError(
"巨量千川下载任务中心在等待期间未达到可用状态"
)
try:
page.wait_for_function(
"(marker) => (document.body?.innerText || '').includes(marker)",
arg=TASK_READY_MARKER,
timeout=min(15_000, remaining_ms),
)
return
except Exception as exc:
message = str(exc).casefold()
if (
not is_browser_timeout_error(exc)
and "execution context was destroyed" not in message
):
raise
# 等待落空且已不在账户选择页:账户选择后若未自动回到任务中心,
# 重新打开一次深链;后续轮次仅继续等待直至超时。
if regotos < 1:
regotos += 1
page.goto(
TASK_CENTER_PAGE_URL, wait_until="domcontentloaded", timeout=90_000
)
def _download_task_from_page(
page,
start_date: date,
@@ -206,15 +274,11 @@ def _download_task_from_page(
except Exception as exc:
if (
"execution context was destroyed" not in str(exc).casefold()
or navigation_retries >= 1
or navigation_retries >= 2
):
raise
navigation_retries += 1
page.wait_for_function(
"(marker) => (document.body?.innerText || '').includes(marker)",
arg=TASK_READY_MARKER,
timeout=15_000,
)
_wait_task_center_ready(page)
_validate_task_center_page(page)
continue
if has_rows:
@@ -248,15 +312,11 @@ def _download_from_task_center(
task_page = context.new_page()
try:
task_page.goto(
TASK_CENTER_URL.format(advertiser_id=quote(ADVERTISER_ID)),
TASK_CENTER_PAGE_URL,
wait_until="domcontentloaded",
timeout=90_000,
)
task_page.wait_for_function(
"(marker) => (document.body?.innerText || '').includes(marker)",
arg=TASK_READY_MARKER,
timeout=90_000,
)
_wait_task_center_ready(task_page)
_validate_task_center_page(task_page)
return _download_task_from_page(
task_page,
@@ -0,0 +1,381 @@
{
"target": {
"spreadsheet_token": "LwbBswjfohdoEvtsu1Oc2rI3nac",
"sheet_id": "2HAOmi",
"sheet_name": "現有庫存",
"sku_column": "C",
"inventory_column": "D",
"product_code_column": "H",
"first_data_row": 2,
"last_data_row": 52,
"expected_sku_count": 51
},
"source_workbook": "光影行星香港安沛2026Q2計劃書(2026.09.28).xlsx",
"source_sheet": "現有庫存",
"source_as_of": "2026-09-28",
"schedule": {
"kind": "daily",
"at": "09:00",
"timezone": "Asia/Shanghai"
},
"availability_formula": "qty - order_lock - lock_qty + virtual_qty",
"skus": [
{
"sku": "GY-001BLK",
"product_name": "Apollo X1 Shoulder Bag",
"ean": "4897080019369",
"source_row": 2,
"jushuitan_sku_id": "10388030"
},
{
"sku": "GY-001WHT",
"product_name": "Apollo X1 Shoulder Bag",
"ean": "4897080019376",
"source_row": 3,
"jushuitan_sku_id": "10388031"
},
{
"sku": "GY-007BLK",
"product_name": "Liviana Shoulder Bag",
"ean": "4897080019468",
"source_row": 4,
"jushuitan_sku_id": "10390039"
},
{
"sku": "GY-007WHT",
"product_name": "Liviana Shoulder Bag",
"ean": "4897080019475",
"source_row": 5,
"jushuitan_sku_id": "10390040"
},
{
"sku": "GY-008WHT",
"product_name": "Starship Satchel",
"ean": "4897080019499",
"source_row": 6,
"jushuitan_sku_id": "10365042"
},
{
"sku": "GY-009KAI",
"product_name": "Nebula Crossbody Bag",
"ean": "4897080010021",
"source_row": 7,
"jushuitan_sku_id": "10273005"
},
{
"sku": "GY-011WHT",
"product_name": "Zeus Backpack",
"ean": "4897080019536",
"source_row": 8,
"jushuitan_sku_id": "10244056"
},
{
"sku": "GY-012BLK",
"product_name": "Zeus II Travel Backpack (Expandable version)",
"ean": "4897080019543",
"source_row": 9,
"jushuitan_sku_id": "10434003"
},
{
"sku": "GY-013BLK",
"product_name": "Startrail Crossbody Bag",
"ean": "4897080019550",
"source_row": 10,
"jushuitan_sku_id": "10394065"
},
{
"sku": "GY-013GRN",
"product_name": "Startrail Crossbody Bag",
"ean": "4897080012308",
"source_row": 11,
"jushuitan_sku_id": "10394066"
},
{
"sku": "GY-013WHT",
"product_name": "Startrail Crossbody Bag",
"ean": "4897080019567",
"source_row": 12,
"jushuitan_sku_id": "10394052"
},
{
"sku": "GY-021BLK",
"product_name": "Boundary Backpack",
"ean": "4897080012483",
"source_row": 13,
"jushuitan_sku_id": "10447004"
},
{
"sku": "GY-022BLK",
"product_name": "Apollo X2 Shoulder Bag",
"ean": "4897080012490",
"source_row": 14,
"jushuitan_sku_id": "10442009"
},
{
"sku": "GY-022WHT",
"product_name": "Apollo X2 Shoulder Bag",
"ean": "4897080012506",
"source_row": 15,
"jushuitan_sku_id": "10442010"
},
{
"sku": "GY-023BLK",
"product_name": "Apollo Air Shoulder Bag",
"ean": "4897080012513",
"source_row": 16,
"jushuitan_sku_id": "10456013"
},
{
"sku": "GY-023WHT",
"product_name": "Apollo Air Shoulder Bag",
"ean": "4897080012995",
"source_row": 17,
"jushuitan_sku_id": "10456021"
},
{
"sku": "GY-024BLK",
"product_name": "Skyreach Air Backpack",
"ean": "4897080012520",
"source_row": 18,
"jushuitan_sku_id": "10454007"
},
{
"sku": "GY-024WHT",
"product_name": "Skyreach Air Backpack",
"ean": "4897080012537",
"source_row": 19,
"jushuitan_sku_id": "10454008"
},
{
"sku": "GY-0260RG",
"product_name": "Polar Star Backpack",
"ean": "4897080010793",
"source_row": 20,
"jushuitan_sku_id": "10465009"
},
{
"sku": "GY-026BLK",
"product_name": "Polar Star Backpack",
"ean": "4897080010588",
"source_row": 21,
"jushuitan_sku_id": "10465003"
},
{
"sku": "GY-026GRN",
"product_name": "Polar Star Backpack",
"ean": "4897080010786",
"source_row": 22,
"jushuitan_sku_id": "10465020"
},
{
"sku": "GY-026WHT",
"product_name": "Polar Star Backpack",
"ean": "4897080010595",
"source_row": 23,
"jushuitan_sku_id": "10465004"
},
{
"sku": "GY-027BLK",
"product_name": "Polar Star tote bag",
"ean": "4897080010809",
"source_row": 24,
"jushuitan_sku_id": "10464008"
},
{
"sku": "GY-027WHT",
"product_name": "Polar Star tote bag",
"ean": "4897080010571",
"source_row": 25,
"jushuitan_sku_id": "10464004"
},
{
"sku": "GY-028BLK",
"product_name": "Poseidon tote bag (cordura)",
"ean": "4897080010779",
"source_row": 26,
"jushuitan_sku_id": "10472004"
},
{
"sku": "GY-029BLK",
"product_name": "Nebula miniCrossbody Bag",
"ean": "4897080010762",
"source_row": 27,
"jushuitan_sku_id": "10475015"
},
{
"sku": "GY-029KAI",
"product_name": "Nebula miniCrossbody Bag",
"ean": "4897080010748",
"source_row": 28,
"jushuitan_sku_id": "10475016"
},
{
"sku": "GY-029WHT",
"product_name": "Nebula miniCrossbody Bag",
"ean": "4897080010755",
"source_row": 29,
"jushuitan_sku_id": "10475014"
},
{
"sku": "GY-030BLK",
"product_name": "Morning Star miniCrossbody Bag",
"ean": "4897080013008",
"source_row": 30,
"jushuitan_sku_id": "10468007"
},
{
"sku": "GY-030WHT",
"product_name": "Morning Star miniCrossbody Bag",
"ean": "4897080010731",
"source_row": 31,
"jushuitan_sku_id": "10468003"
},
{
"sku": "GY-031BLK",
"product_name": "Star-by-Star EVOCrossbody Bag",
"ean": "4897080010724",
"source_row": 32,
"jushuitan_sku_id": "10474004"
},
{
"sku": "GY-032BLK",
"product_name": "GlacierCrossbody Bag",
"ean": "4897080010717",
"source_row": 33,
"jushuitan_sku_id": "10453005"
},
{
"sku": "GY-033BLK",
"product_name": "Morning Star 2 Crossbody Bag",
"ean": "4897080012896",
"source_row": 34,
"jushuitan_sku_id": "10469003"
},
{
"sku": "GY-033WHT",
"product_name": "Morning Star 2 Crossbody Bag",
"ean": "4897080012902",
"source_row": 35,
"jushuitan_sku_id": "10469004"
},
{
"sku": "GY-034BLK",
"product_name": "Startrail 2 Crossbody Bag",
"ean": "4897080012919",
"source_row": 36,
"jushuitan_sku_id": "10504027"
},
{
"sku": "GY-034GRN",
"product_name": "Startrail 2 Crossbody Bag",
"ean": "4897080012926",
"source_row": 37,
"jushuitan_sku_id": "10504028"
},
{
"sku": "GY-034WHT",
"product_name": "Startrail 2 Crossbody Bag",
"ean": "4897080012933",
"source_row": 38,
"jushuitan_sku_id": "10504029"
},
{
"sku": "GY-035BLK",
"product_name": "Zeus 3 air Backpack",
"ean": "4897080012940",
"source_row": 39,
"jushuitan_sku_id": "10482004"
},
{
"sku": "GY-035WHT",
"product_name": "Zeus 3 air Backpack",
"ean": "4897080012957",
"source_row": 40,
"jushuitan_sku_id": "10482003"
},
{
"sku": "GY-036BLK",
"product_name": "Cirrus 2 Backpack",
"ean": "4897080012964",
"source_row": 41,
"jushuitan_sku_id": "10481004"
},
{
"sku": "GY-036GRN",
"product_name": "Cirrus 2 Backpack",
"ean": "4897080012988",
"source_row": 42,
"jushuitan_sku_id": "10481006"
},
{
"sku": "GY-036WHT",
"product_name": "Cirrus 2 Backpack",
"ean": "4897080012971",
"source_row": 43,
"jushuitan_sku_id": "10481005"
},
{
"sku": "GY-037BLK",
"product_name": "Polar Star 2",
"ean": "4897080013046",
"source_row": 44,
"jushuitan_sku_id": "10520009"
},
{
"sku": "GY-037GRY",
"product_name": "Polar Star 2",
"ean": "4897080013039",
"source_row": 45,
"jushuitan_sku_id": "10520014"
},
{
"sku": "GY-038WHT",
"product_name": "Polar Star Shoulder Bag",
"ean": "4897080013060",
"source_row": 46,
"jushuitan_sku_id": "10519010"
},
{
"sku": "GY-038BLK",
"product_name": "Polar Star Shoulder Bag",
"ean": "4897080013053",
"source_row": 47,
"jushuitan_sku_id": "10519009"
},
{
"sku": "GY-039BLK",
"product_name": "Poseidon Laptop Bag",
"ean": "4897080013084",
"source_row": 48,
"jushuitan_sku_id": "10515020"
},
{
"sku": "GY-039GRY",
"product_name": "Poseidon Laptop Bag",
"ean": "4897080013077",
"source_row": 49,
"jushuitan_sku_id": "10515019"
},
{
"sku": "GY-040BLK",
"product_name": "Skyreach 3",
"ean": "4897080013091",
"source_row": 50,
"jushuitan_sku_id": "10510019"
},
{
"sku": "GY-040WHT",
"product_name": "Skyreach 3",
"ean": "4897080013107",
"source_row": 51,
"jushuitan_sku_id": "10510020"
},
{
"sku": "GY-041BLK",
"product_name": "Phantom",
"ean": "4897080013114",
"source_row": 52,
"jushuitan_sku_id": "10507015"
}
]
}
@@ -1437,7 +1437,8 @@ def load_jd_product_references(
"""
SELECT style_name, jd_spus
FROM dim_style
WHERE jd_spus IS NOT NULL AND cardinality(jd_spus) > 0
WHERE jsonb_typeof(jd_spus) = 'array'
AND jsonb_array_length(jd_spus) > 0
"""
)
rows = cursor.fetchall()
@@ -0,0 +1,442 @@
"""Refresh Hong Kong Anpei public inventory in the Feishu spreadsheet."""
from __future__ import annotations
import json
import sys
import time
from collections.abc import Mapping
from datetime import datetime
from pathlib import Path
from typing import Any
from zoneinfo import ZoneInfo
from gyxx_flow.adapters.jushuitan_inventory import (
InventoryRecord,
JushuitanInventoryClient,
)
from gyxx_flow.modules.product_commerce.lark_cli_runtime import run_lark_cli
_WORKFLOW_ID = "product.hk_anpei_public_inventory.daily"
_CONFIG_PATH = (
Path(__file__).resolve().parent
/ "config"
/ "hk_anpei_public_inventory.json"
)
_REPOSITORY_ROOT = Path(__file__).resolve().parents[4]
_TIMEZONE = ZoneInfo("Asia/Shanghai")
_MAX_SHEET_ROWS = 200
_WRITE_INTERVAL_SECONDS = 0.25
class InventorySpreadsheetError(RuntimeError):
"""Safe-to-log error for invalid spreadsheet configuration or state."""
def _load_config() -> dict[str, Any]:
try:
config = json.loads(_CONFIG_PATH.read_text(encoding="utf-8"))
except (OSError, UnicodeError, json.JSONDecodeError) as exc:
raise InventorySpreadsheetError(
"Unable to read the public inventory workflow configuration"
) from exc
if not isinstance(config, dict):
raise InventorySpreadsheetError(
"Public inventory configuration must be an object"
)
target = config.get("target")
if not isinstance(target, dict):
raise InventorySpreadsheetError(
"Public inventory configuration has no spreadsheet target"
)
for key in (
"spreadsheet_token",
"sheet_id",
"sheet_name",
"sku_column",
"inventory_column",
"product_code_column",
):
if not str(target.get(key) or "").strip():
raise InventorySpreadsheetError(
f"Spreadsheet target is missing {key}"
)
for key in ("first_data_row", "last_data_row", "expected_sku_count"):
value = target.get(key)
if isinstance(value, bool) or not isinstance(value, int) or value < 1:
raise InventorySpreadsheetError(
f"Spreadsheet target has an invalid {key}"
)
if target["last_data_row"] - target["first_data_row"] + 1 != target[
"expected_sku_count"
]:
raise InventorySpreadsheetError(
"Spreadsheet target rows do not match the frozen SKU count"
)
sku_items = config.get("skus")
if not isinstance(sku_items, list) or not sku_items:
raise InventorySpreadsheetError(
"Public inventory configuration has no SKUs"
)
if len(sku_items) != target["expected_sku_count"]:
raise InventorySpreadsheetError(
"Configured SKU count does not match the spreadsheet target"
)
seen_skus: set[str] = set()
seen_product_codes: set[str] = set()
for item in sku_items:
if not isinstance(item, dict):
raise InventorySpreadsheetError(
"Public inventory SKU metadata is invalid"
)
sku = str(item.get("sku") or "").strip()
product_code = str(item.get("jushuitan_sku_id") or "").strip()
if not sku or sku in seen_skus:
raise InventorySpreadsheetError(
"Public inventory configuration contains a blank or duplicate SKU"
)
if not product_code or product_code in seen_product_codes:
raise InventorySpreadsheetError(
"Public inventory configuration contains a blank or duplicate "
"Jushuitan product code"
)
seen_skus.add(sku)
seen_product_codes.add(product_code)
return config
def _sheet_args(config: Mapping[str, Any]) -> list[str]:
target = config["target"]
return [
"--as",
"user",
"--spreadsheet-token",
str(target["spreadsheet_token"]),
"--sheet-id",
str(target["sheet_id"]),
"--format",
"json",
]
def _run_sheets(args: list[str], *, operation: str) -> dict[str, Any]:
try:
return run_lark_cli(
args,
cwd=_REPOSITORY_ROOT,
timeout=180,
)
except Exception as exc:
raise InventorySpreadsheetError(
f"Feishu spreadsheet {operation} failed"
) from exc
def _cell_value(cell: object) -> object:
if not isinstance(cell, Mapping):
raise InventorySpreadsheetError(
"Feishu spreadsheet returned an invalid cell"
)
return cell.get("value")
def _read_sheet_rows(
config: Mapping[str, Any],
sku_items: list[Mapping[str, Any]],
) -> dict[str, dict[str, Any]]:
target = config["target"]
result = _run_sheets(
[
"sheets",
"+cells-get",
*_sheet_args(config),
"--range",
f"{target['sku_column']}1:{target['product_code_column']}{_MAX_SHEET_ROWS}",
"--include",
"value",
],
operation="cells-get",
)
data = result.get("data")
ranges = data.get("ranges") if isinstance(data, Mapping) else None
if (
not isinstance(data, Mapping)
or data.get("has_more") is not False
or not isinstance(ranges, list)
or len(ranges) != 1
):
raise InventorySpreadsheetError(
"Feishu spreadsheet response is incomplete or invalid"
)
sheet_range = ranges[0]
if not isinstance(sheet_range, Mapping):
raise InventorySpreadsheetError(
"Feishu spreadsheet range response is invalid"
)
cells = sheet_range.get("cells")
row_indices = sheet_range.get("row_indices")
col_indices = sheet_range.get("col_indices")
required_columns = {
str(target["sku_column"]),
str(target["inventory_column"]),
str(target["product_code_column"]),
}
if (
not isinstance(cells, list)
or not isinstance(row_indices, list)
or not isinstance(col_indices, list)
or len(cells) != len(row_indices)
or not required_columns.issubset({str(col) for col in col_indices})
):
raise InventorySpreadsheetError(
"Feishu spreadsheet response does not cover the configured columns"
)
column_positions = {str(col): index for index, col in enumerate(col_indices)}
row_values: dict[int, list[object]] = {}
for row_number, values in zip(row_indices, cells, strict=True):
if (
isinstance(row_number, bool)
or not isinstance(row_number, int)
or not isinstance(values, list)
or len(values) != len(col_indices)
):
raise InventorySpreadsheetError(
"Feishu spreadsheet returned malformed row coordinates"
)
row_values[row_number] = values
first_row = int(target["first_data_row"])
last_row = int(target["last_data_row"])
header = row_values.get(1)
if header is None:
raise InventorySpreadsheetError(
"Feishu spreadsheet is missing its header row"
)
expected_headers = {
str(target["sku_column"]): "sku",
str(target["inventory_column"]): "仓库现有库存",
str(target["product_code_column"]): "商品编码(光影)",
}
for column, expected_header in expected_headers.items():
actual_header = str(
_cell_value(header[column_positions[column]]) or ""
).strip()
if actual_header != expected_header:
raise InventorySpreadsheetError(
f"Feishu spreadsheet header in column {column} does not match"
)
rows: dict[str, dict[str, Any]] = {}
for row_number in range(first_row, last_row + 1):
values = row_values.get(row_number)
if values is None:
raise InventorySpreadsheetError(
"Feishu spreadsheet is missing an expected inventory row"
)
sku = str(
_cell_value(values[column_positions[str(target["sku_column"])]]) or ""
).strip()
product_code = str(
_cell_value(
values[column_positions[str(target["product_code_column"])] ]
)
or ""
).strip()
if not sku or sku in rows:
raise InventorySpreadsheetError(
"Feishu spreadsheet contains a blank or duplicate SKU"
)
rows[sku] = {
"row": row_number,
"product_code": product_code,
}
expected_by_sku = {
str(item["sku"]).strip(): str(item["jushuitan_sku_id"]).strip()
for item in sku_items
}
if set(rows) != set(expected_by_sku):
raise InventorySpreadsheetError(
"Feishu spreadsheet SKU rows differ from the frozen workbook list"
)
for sku, row in rows.items():
if row["product_code"] != expected_by_sku[sku]:
raise InventorySpreadsheetError(
f"Feishu spreadsheet product code does not match SKU {sku} "
f"at row {row['row']}"
)
return rows
def _read_written_inventory(
config: Mapping[str, Any],
expected: list[object],
) -> None:
target = config["target"]
column = str(target["inventory_column"])
first_row = int(target["first_data_row"])
last_row = int(target["last_data_row"])
result = _run_sheets(
[
"sheets",
"+cells-get",
*_sheet_args(config),
"--range",
f"{column}{first_row}:{column}{last_row}",
"--include",
"value",
],
operation="inventory readback",
)
data = result.get("data")
ranges = data.get("ranges") if isinstance(data, Mapping) else None
if (
not isinstance(data, Mapping)
or data.get("has_more") is not False
or not isinstance(ranges, list)
or len(ranges) != 1
):
raise InventorySpreadsheetError(
"Feishu inventory readback is incomplete or invalid"
)
sheet_range = ranges[0]
if not isinstance(sheet_range, Mapping):
raise InventorySpreadsheetError(
"Feishu inventory readback range is invalid"
)
cells = sheet_range.get("cells")
row_indices = sheet_range.get("row_indices")
if (
not isinstance(cells, list)
or not isinstance(row_indices, list)
or row_indices != list(range(first_row, last_row + 1))
or len(cells) != len(expected)
):
raise InventorySpreadsheetError(
"Feishu inventory readback does not cover every target row"
)
actual: list[object] = []
for row in cells:
if not isinstance(row, list) or len(row) != 1:
raise InventorySpreadsheetError(
"Feishu inventory readback has an invalid cell shape"
)
actual.append(_cell_value(row[0]))
for offset, (actual_value, expected_value) in enumerate(
zip(actual, expected, strict=True)
):
if expected_value == "":
matched = actual_value in (None, "")
else:
try:
matched = int(str(actual_value).strip()) == int(expected_value)
except (TypeError, ValueError):
matched = False
if not matched:
raise InventorySpreadsheetError(
"Feishu inventory readback did not match the submitted value "
f"at row {first_row + offset}"
)
def _write_inventory(
config: Mapping[str, Any],
sku_items: list[Mapping[str, Any]],
inventories: Mapping[str, InventoryRecord],
rows: Mapping[str, Mapping[str, Any]],
) -> tuple[int, bool, int]:
target = config["target"]
first_row = int(target["first_data_row"])
last_row = int(target["last_data_row"])
by_row = sorted(rows.items(), key=lambda entry: int(entry[1]["row"]))
if [int(row["row"]) for _, row in by_row] != list(
range(first_row, last_row + 1)
):
raise InventorySpreadsheetError(
"Feishu spreadsheet SKU rows are not a contiguous target range"
)
sku_config = {str(item["sku"]): item for item in sku_items}
expected_values: list[object] = []
missing_count = 0
for sku, _ in by_row:
product_code = str(sku_config[sku]["jushuitan_sku_id"])
inventory = inventories.get(product_code)
if inventory is None:
# An omitted Jushuitan row is unknown inventory, so clear stale data.
expected_values.append("")
missing_count += 1
else:
expected_values.append(inventory.public_available_qty)
cells = [[{"value": value}] for value in expected_values]
result = _run_sheets(
[
"sheets",
"+cells-set",
*_sheet_args(config),
"--range",
f"{target['inventory_column']}{first_row}:"
f"{target['inventory_column']}{last_row}",
"--cells",
json.dumps(cells, ensure_ascii=False, separators=(",", ":")),
],
operation="inventory update",
)
if result.get("acceptance_skipped") is True:
return 0, True, missing_count
_read_written_inventory(config, expected_values)
return len(expected_values), False, missing_count
def run() -> dict[str, object]:
config = _load_config()
sku_items = config["skus"]
jushuitan_sku_ids = [str(item["jushuitan_sku_id"]) for item in sku_items]
# Validate the complete target and ERP response before modifying Feishu.
rows = _read_sheet_rows(config, sku_items)
inventories = JushuitanInventoryClient().query(jushuitan_sku_ids)
if not inventories:
raise InventorySpreadsheetError(
"Jushuitan returned no inventory rows for the frozen product-code list; "
"refusing to clear the Feishu inventory column"
)
written, acceptance_skipped, inventory_missing = _write_inventory(
config,
sku_items,
inventories,
rows,
)
return {
"workflow_id": _WORKFLOW_ID,
"target_sheet": str(config["target"]["sheet_name"]),
"sku_count": len(sku_items),
"inventory_found": len(inventories),
"inventory_missing": inventory_missing,
"cells_written": written,
"acceptance_skipped": acceptance_skipped,
"queried_at": datetime.now(_TIMEZONE).strftime("%Y-%m-%d %H:%M:%S"),
}
def main() -> int:
try:
print(json.dumps(run(), ensure_ascii=False, separators=(",", ":")))
return 0
except Exception as exc:
print(
json.dumps(
{"workflow_id": _WORKFLOW_ID, "error": str(exc)},
ensure_ascii=False,
separators=(",", ":"),
),
file=sys.stderr,
)
return 1
if __name__ == "__main__":
raise SystemExit(main())
@@ -503,7 +503,8 @@ def load_tm_item_ids_for_styles(style_names: List[str]) -> Dict[str, List[str]]:
"""
SELECT style_name, tm_spus
FROM dim_style
WHERE tm_spus IS NOT NULL AND cardinality(tm_spus) > 0
WHERE jsonb_typeof(tm_spus) = 'array'
AND jsonb_array_length(tm_spus) > 0
"""
)
rows = cur.fetchall()
@@ -223,23 +223,16 @@ def _check_login_status(page) -> bool:
def _ensure_login(page):
print("\n" + "=" * 60)
print("【请登录】请在弹出的浏览器窗口中完成登录")
print("提示:扫码/短信/密码均可,登录成功后页面会自动跳转")
print("=" * 60 + "\n")
for i in range(LOGIN_TIMEOUT):
page = _active_page(page)
if _check_login_status(page):
print(f"[OK] Login success! URL: {page.url}")
page.wait_for_timeout(3000)
return page
if i % 5 == 0:
print(f"[WAIT] Waiting for login... {i}s / {LOGIN_TIMEOUT}s")
time.sleep(1)
print(f"[ERR] Login timeout ({LOGIN_TIMEOUT}s)")
return None
# 登录统一由账号 vault 管理(gyxx accounts login douyin-shop),
# 采集脚本不再弹出浏览器等待人工扫码/短信登录。
print(
"[COOKIE_EXPIRED] 抖音罗盘登录态已失效:"
"请执行 uv run gyxx accounts login douyin-shop 刷新账号登录态"
)
raise RuntimeError(
"抖音罗盘登录态已失效;请执行 uv run gyxx accounts login douyin-shop "
"刷新账号登录态后重跑"
)
def _select_menu_and_date(page) -> None:
@@ -1927,12 +1920,8 @@ def _page_automation(page) -> None:
# 1. Check and wait for login
if not _check_login_status(page):
logged_page = _ensure_login(page)
if not logged_page:
print("[ERR] Login failed, will try save cookies anyway...")
else:
page = logged_page
_maximize_page(page)
# 登录态失效时 _ensure_login 会直接抛 RuntimeError
_ensure_login(page)
# 2. Save state after login (best effort)
_save_browser_state(page, "after login")
@@ -226,23 +226,16 @@ def ensure_login(page: Page, *, headless: bool, login_timeout: int) -> None:
print("[SKIPPED_COOKIE] 抖店会话跳转到了登录页")
raise DouyinSessionExpiredError("抖店会话跳转到了登录页")
if headless:
raise RuntimeError(
"抖音 Cookie 已失效且当前为无头模式,无法扫码登录;"
"请先执行 uv run gyxx accounts login douyin-shop 刷新账号登录态"
# 登录统一由账号 vault 管理(gyxx accounts login douyin-shop),
# 采集脚本不再弹出浏览器等待人工扫码/验证码。
print(
"[COOKIE_EXPIRED] 抖店 Cookie 已失效:"
"请执行 uv run gyxx accounts login douyin-shop 刷新账号登录态"
)
raise DouyinSessionExpiredError(
"抖音 Cookie 已失效;请执行 uv run gyxx accounts login douyin-shop "
"刷新账号登录态后重跑"
)
print("=" * 60)
print("【请登录】请在弹出的浏览器窗口中完成抖店登录")
print("=" * 60)
deadline = time.monotonic() + login_timeout
while time.monotonic() < deadline:
if not douyin_common.is_login_page(page):
print(f" [OK] 登录成功: {page.url}")
page.wait_for_timeout(3_000)
return
page.wait_for_timeout(1_000)
raise RuntimeError(f"登录在 {login_timeout} 秒内未完成,当前页面: {page.url}")
class DouyinSessionExpiredError(RuntimeError):
@@ -262,23 +262,16 @@ def _check_login_status(page) -> bool:
def _ensure_login(page):
print("\n" + "=" * 60)
print("【请登录】请在弹出的浏览器窗口中完成登录")
print("提示:扫码/短信/密码均可,登录成功后页面会自动跳转")
print("=" * 60 + "\n")
for i in range(LOGIN_TIMEOUT):
page = _active_page(page)
if _check_login_status(page):
print(f"[OK] Login success! URL: {page.url}")
page.wait_for_timeout(3000)
return page
if i % 5 == 0:
print(f"[WAIT] Waiting for login... {i}s / {LOGIN_TIMEOUT}s")
time.sleep(1)
print(f"[ERR] Login timeout ({LOGIN_TIMEOUT}s)")
return None
# 登录统一由账号 vault 管理(gyxx accounts login douyin-shop),
# 采集脚本不再弹出浏览器等待人工扫码/短信登录。
print(
"[COOKIE_EXPIRED] 抖音罗盘登录态已失效:"
"请执行 uv run gyxx accounts login douyin-shop 刷新账号登录态"
)
raise BrowserSessionExpiredError(
"抖音罗盘登录态已失效;请执行 uv run gyxx accounts login douyin-shop "
"刷新账号登录态后重跑"
)
def _select_menu_and_date(page) -> None:
@@ -2325,12 +2318,8 @@ def _page_automation_impl(page) -> None:
# 1. Check and wait for login
if not _check_login_status(page):
logged_page = _ensure_login(page)
if not logged_page:
print("[ERR] Login failed, will try save cookies anyway...")
else:
page = logged_page
_maximize_page(page)
# If the login state is expired, _ensure_login will directly raise BrowserSessionExpiredError
_ensure_login(page)
# 2. Save state after login (best effort)
_save_browser_state(page, "after login")
@@ -32,10 +32,9 @@ from gyxx_flow.modules.shop_intelligence.browser_session import (
)
from gyxx_flow.modules.shop_intelligence.collectors.jd_data_collector import (
close_profit_plan_popup_if_present,
select_last_week_in_old_datepicker,
)
from gyxx_flow.modules.shop_intelligence.config import JD_PEER_STORE_OUTPUT
from gyxx_flow.modules.shop_intelligence.paths import resolve_raw_output
from gyxx_flow.modules.shop_intelligence.paths import TMP_ROOT, resolve_raw_output
# ── 配置 ─────────────────────────────────────────────────────────────────────
@@ -67,7 +66,11 @@ CATEGORY_OUTPUT_NAMES = {
"男包": "男士包袋",
}
VENDER_RANKS_URL = "https://jdsz.jd.com/szweb/view/industryMarket/venderRanks.html"
# 2026-10 京东下线旧版商智:venderRanks 旧入口 404,官方公告要求迁移新版
# jdsz.jd.com。新版等价页面是「行业榜单」的店铺榜 tab。
INDUSTRY_TOP_URL = "https://jdsz.jd.com/szweb/view/industry/industry-top.html"
# 兼容旧引用
VENDER_RANKS_URL = INDUSTRY_TOP_URL
# 新品采集配置
NEW_PRODUCT_SHOPS = [
@@ -398,97 +401,262 @@ def get_last_week_range():
return last_monday, last_sunday
def _dump_peer_filter_debug(page: Page, tag: str) -> None:
"""日期/类目操作失败时保存现场,便于按页面实际 DOM 校准选择器。"""
try:
stamp = datetime.now().strftime("%Y%m%d_%H%M%S")
debug_dir = TMP_ROOT / "debug" / "jd_peer_filter"
debug_dir.mkdir(parents=True, exist_ok=True)
page.screenshot(path=str(debug_dir / f"{stamp}_{tag}.png"))
html_path = debug_dir / f"{stamp}_{tag}.html"
html_path.write_text(page.content(), encoding="utf-8")
print(f" [DEBUG] 现场已保存: {html_path}")
except Exception as exc:
print(f" [DEBUG] 现场保存失败: {exc}")
def _click_custom_day_cell(page: Page, date_str: str) -> None:
"""在自定义日期面板中点击指定日期格(按月份块消歧,真实坐标点击)。
面板为单月迷你日历:日格只带日号文本,须先定位「只含一个月份标题且
含日历格」的月份块,再在其中匹配日号;组件对合成点击不稳定,统一用
bounding_box 中心坐标做真实鼠标点击。
"""
center = page.evaluate(
"""(dateStr) => {
const [year, month, day] = dateStr.split('-');
const wantTitle = `${Number(year)}年${String(Number(month)).padStart(2, '0')}月`;
const dayText = String(Number(day));
const isMonthTitle = (text) => /^\\d{4}年\\d{2}月$/.test((text || '').trim());
const titleSpans = Array.from(document.querySelectorAll('span'))
.filter(el => el.getClientRects().length && isMonthTitle(el.innerText));
for (const span of titleSpans) {
if ((span.innerText || '').trim() !== wantTitle) continue;
let node = span.parentElement;
for (let depth = 0; node && depth < 10; depth += 1, node = node.parentElement) {
const titles = Array.from(node.querySelectorAll('span'))
.filter(s => isMonthTitle(s.innerText));
if (titles.length > 1) break;
if (titles.length === 1
&& node.querySelector('td.jmt-date-picker-calendar-cell')) {
const cells = Array.from(
node.querySelectorAll('td.jmt-date-picker-calendar-cell')
).filter(el => el.getBoundingClientRect().width > 0
&& (el.innerText || '').trim() === dayText);
if (cells.length) {
const rect = cells[0].getBoundingClientRect();
return {x: Math.round(rect.x + rect.width / 2),
y: Math.round(rect.y + rect.height / 2)};
}
return null;
}
}
}
return null;
}""",
date_str,
)
if not center:
_dump_peer_filter_debug(page, f"day_not_found_{date_str}")
raise RuntimeError(f"自定义日期面板未找到日期格: {date_str}")
page.mouse.click(center["x"], center["y"])
def _open_custom_date_panel(page: Page, *, retries: int = 3) -> bool:
"""点击「自定义」标签打开日历;带开面板验证与重试。
面板行为实测有两种形态且跨运行不定:迷你单月(选起点即自动关闭)
与常驻双月(选完不关)。tag 点击在面板刚自动关闭后可能需要缓一下
再点,故带月标题可见性验证与重试。
"""
tag = page.locator(
"li.jmt-combo-date-picker-list-tag", has_text="自定义"
).first
for _attempt in range(retries):
try:
box = tag.bounding_box(timeout=5000)
except Exception:
box = None
if not box:
_dump_peer_filter_debug(page, "custom_tag_missing")
raise RuntimeError("未找到「自定义」时间按钮")
page.mouse.click(
box["x"] + box["width"] / 2, box["y"] + box["height"] / 2
)
page.wait_for_timeout(2000)
if _panel_months_visible(page):
return True
page.wait_for_timeout(1000)
return _panel_months_visible(page)
def _panel_months_visible(page) -> bool:
try:
return bool(
page.evaluate(
"() => Array.from(document.querySelectorAll('span'))"
".some(el => el.getClientRects().length"
" && /^\\d{4}年\\d{2}月$/.test((el.innerText || '').trim()))"
)
)
except Exception:
return False
def _ensure_month_visible(page: Page, month_title: str) -> None:
"""单月日历翻月:目标月份不可见时点「后一月」箭头(至多 12 次)。"""
def visible_months():
return page.evaluate(
"""() => Array.from(document.querySelectorAll('span'))
.filter(el => el.getClientRects().length)
.map(el => (el.innerText || '').trim())
.filter(t => /^\\d{4}年\\d{2}月$/.test(t))"""
)
for _ in range(12):
if month_title in visible_months():
return
try:
arrow = page.locator('[data-event-content="后一月"]')
if arrow.count() == 0:
return
box = arrow.first.bounding_box(timeout=3000)
except Exception:
return
if not box:
return
page.mouse.click(
box["x"] + box["width"] / 2, box["y"] + box["height"] / 2
)
page.wait_for_timeout(1200)
def _custom_panel_state(page) -> str:
"""调试用:当前日期面板的可见月份与「当前:」显示。"""
try:
return str(
page.evaluate(
"""() => {
const months = Array.from(document.querySelectorAll('span'))
.filter(el => el.getClientRects().length)
.map(el => (el.innerText || '').trim())
.filter(t => /^\\d{4}年\\d{2}月$/.test(t));
const current = (document.body.innerText.match(/当前:[^\\n]{0,40}/) || [''])[0];
return {months: [...new Set(months)], current};
}"""
)
)
except Exception as exc:
return f"<state-error {exc}>"
def _custom_range_applied(page, start_mmdd: str) -> bool:
try:
body_text = page.locator("body").inner_text(timeout=3000)
except Exception:
return False
return start_mmdd in body_text.replace(" ", "")
def select_last_week_any_day(page):
"""点击日期选择器 → 按周查询 → 点上周任意一天(自动选中整周)"""
"""新版行业榜单:自定义区间选择上周一~上周日(状态机,3 轮重试)。
面板有两种形态且跨运行不定:
- 迷你单月:点击起点后自动关闭;重开面板点终点,选完立即生效。
- 常驻双月:起止选完面板不关;点面板外部(店铺搜索框)应用;
Escape 会取消选择。
统一状态机:选起点 → 面板仍开则同会话选终点,已关则重开选终点 →
未生效则点外部应用 → 仍未生效整轮重来。
"""
print(" 选择上周时间范围...")
last_monday, last_sunday = get_last_week_range()
start_str = last_monday.strftime("%Y-%m-%d")
end_str = last_sunday.strftime("%Y-%m-%d")
start_month = f"{last_monday.year}年{last_monday.month:02d}月"
end_month = f"{last_sunday.year}年{last_sunday.month:02d}月"
start_mmdd = last_monday.strftime("%m-%d")
# 1. 点击日期选择器(显示"按天查询..."的元素)
for attempt in range(1, 4):
print(f" [state] 第 {attempt} 轮: {_custom_panel_state(page)}")
if not _open_custom_date_panel(page):
print(" [state] 面板未能打开,重试")
continue
_ensure_month_visible(page, start_month)
try:
date_picker = page.locator('.jmtd-date-picker .jmtd-base-input-top').first
date_picker.wait_for(state="visible", timeout=5000)
date_picker.click()
print(" [OK] 点击日期选择器")
except Exception as e:
print(f" [WARN] 点击日期选择器失败: {e}")
# 兜底:点击包含"按天查询"文本的元素
_click_custom_day_cell(page, start_str)
except RuntimeError:
continue
page.wait_for_timeout(1500)
print(f" [state] 点击开始后: {_custom_panel_state(page)}")
if not _panel_months_visible(page):
# 迷你面板形态:起点选择后面板已自动关闭,重开选终点
page.wait_for_timeout(1200)
if not _open_custom_date_panel(page):
continue
_ensure_month_visible(page, end_month)
try:
page.locator('text=按天查询').first.click()
print(" [OK] 使用文本定位点击日期选择器")
_click_custom_day_cell(page, end_str)
except RuntimeError:
continue
page.wait_for_timeout(2500)
print(f" [state] 点击结束后: {_custom_panel_state(page)}")
if _custom_range_applied(page, start_mmdd):
print(f" [OK] 已选择上周区间: {start_str} ~ {end_str}")
return
# 常驻双月形态:点面板外部应用(Escape 是取消,绝不能用)
try:
page.locator(
'input[placeholder="请输入店铺名称"]'
).first.click(timeout=4000)
except Exception:
pass
page.wait_for_timeout(2000)
# 2. 点击"按周查询"
try:
week_switch = page.locator('.jmtd-date-picker-switch-panel-item', has_text="按周查询").first
week_switch.wait_for(state="visible", timeout=5000)
week_switch.click()
print(" [OK] 点击按周查询")
except Exception as e:
print(f" [WARN] 点击按周查询失败: {e}")
# 兜底:用 JS 点击
page.evaluate("""() => {
const items = document.querySelectorAll('.jmtd-date-picker-switch-panel-item');
for (const item of items) {
if (item.textContent.includes('按周查询')) {
item.click();
return true;
}
}
return false;
}""")
page.wait_for_timeout(2000)
# 3. 等待日历弹窗并选择上周某一天
calendar = page.locator('.jmtd-date-picker-popup, .jmtd-date-picker-panel, [class*="picker-panel"], [class*="calendar-panel"]')
try:
calendar.wait_for(state="visible", timeout=5000)
except Exception:
print(" [WARN] 未检测到日历弹窗,尝试继续...")
page.mouse.click(30, 400)
page.wait_for_timeout(3000)
if _custom_range_applied(page, start_mmdd):
print(f" [OK] 已选择上周区间(点外部应用): {start_str} ~ {end_str}")
return
print(f" [state] 本轮未生效: {_custom_panel_state(page)}")
select_last_week_in_old_datepicker(page, "peer_store")
try:
calendar.wait_for(state="hidden", timeout=5000)
except Exception:
pass
page.wait_for_timeout(5000)
print(" 已切换到上周数据")
_dump_peer_filter_debug(page, "range_not_applied")
raise RuntimeError(
f"自定义日期区间未生效: 期望 {start_str} ~ {end_str}"
)
def reselect_last_week(page):
"""切换类目后重新选择上周日期(点击日期选择器 → 按周查询 → 点日期)"""
# 重新点击日期选择器
try:
date_picker = page.locator('.jmtd-date-picker .jmtd-base-input-top').first
date_picker.click()
except Exception:
page.locator('text=按天查询').first.click()
page.wait_for_timeout(1500)
# 点击按周查询
try:
week_switch = page.locator('.jmtd-date-picker-switch-panel-item', has_text="按周查询").first
week_switch.click()
except Exception:
page.evaluate("""() => {
const items = document.querySelectorAll('.jmtd-date-picker-switch-panel-item');
for (const item of items) {
if (item.textContent.includes('按周查询')) {
item.click();
return true;
def _selector_option_center(page: Page, label: str, *, exclude_toolbar: bool = True):
"""在下拉浮层里按精确文本定位 jmt-selector 选项,返回真实坐标。
筛选栏控件本身也是 button.jmt-selector(显示当前选中值),必须
排除,否则会把控件当选项点到。
"""
return page.evaluate(
"""(label) => {
const visible = (el) => {
const r = el.getBoundingClientRect();
return r.width > 0 && r.height > 0;
};
const buttons = Array.from(
document.querySelectorAll('div.jmt-selector')
).filter(visible);
for (const btn of buttons) {
if (btn.closest('.industry-top-rank-toolbar')) continue;
const lab = btn.querySelector('.jmt-label');
if (lab && (lab.innerText || '').trim() === label) {
const r = btn.getBoundingClientRect();
return {x: Math.round(r.x + r.width / 2),
y: Math.round(r.y + r.height / 2)};
}
}
}""")
page.wait_for_timeout(1500)
select_last_week_in_old_datepicker(page, "peer_store_reselect")
page.wait_for_timeout(8000)
return null;
}""",
label,
)
def switch_category(
@@ -496,113 +664,174 @@ def switch_category(
category_name: str,
parent_category: str | None = None,
) -> bool:
"""切换商家榜单类目;只有选中态和榜单刷新均验证后才成功。"""
"""新版行业榜单:jmt-selector 层级下拉选类目 → 点「查询」→ 校验表格。
一级选项直接点击;二级选项先 hover 一级展开子面板再点击。下拉的
选项与筛选栏控件是同款 button.jmt-selector,靠排除工具栏区分。
"""
label = f"{parent_category} → {category_name}" if parent_category else category_name
print(f" 切换类目到: {label}")
expected_path = _expected_category_path(category_name, parent_category)
selector = None
# 类目下拉的触发行为极不稳定(单击/双击/坐标点击时开时不开),
# 用多策略轮换:单击→轮询;双击→轮询;再单击→轮询,任何一刻
# 选项出现即继续。
trigger = page.locator(
".jmt--form-item-main", has_text="类目"
).first
def _dropdown_ready() -> bool:
return bool(
page.evaluate(
"() => Array.from(document.querySelectorAll('div.jmt-selector'))"
".some(el => el.getBoundingClientRect().width > 0"
" && !el.closest('.industry-top-rank-toolbar'))"
)
)
opened = False
for action in ("click", "dblclick", "click"):
try:
selector = _visible_category_selector(page)
previous_fingerprint = _ranking_table_fingerprint(page)
current_path = _selected_category_path(selector)
if current_path == expected_path:
if not previous_fingerprint:
print(f" [WARN] 当前类目是 {label},但榜单表格为空")
return False
print(f" [OK] 当前已是 {label},榜单表格可用")
return True
menu = _visible_category_menu(page, selector)
_select_category_option(
page,
menu,
category_name,
parent_category,
)
if not _wait_for_selected_category(selector, expected_path):
actual_path = _selected_category_path(selector)
print(
f" [WARN] 类目选中态校验失败: "
f"expected={expected_path}, actual={actual_path}"
)
return False
if not _wait_for_ranking_table_refresh(page, previous_fingerprint):
print(f" [WARN] 已选中 {label},但榜单表格未确认刷新")
return False
print(f" [OK] 已切换到 {label},榜单表格已刷新")
return True
if action == "click":
trigger.click(timeout=6000)
else:
trigger.dblclick(timeout=6000)
except Exception as exc:
print(f" [WARN] 切换类目失败: {exc}")
print(f" [WARN] 类目控件{action}失败: {exc}")
_dump_peer_filter_debug(page, "category_control_missing")
return False
finally:
for _ in range(6):
page.wait_for_timeout(500)
if _dropdown_ready():
opened = True
break
if opened:
break
if not opened:
print(" [WARN] 类目下拉未能打开")
_dump_peer_filter_debug(page, "category_dropdown_not_open")
return False
selected = False
if parent_category and parent_category != category_name:
parent_center = _selector_option_center(page, parent_category)
if parent_center:
page.mouse.move(parent_center["x"], parent_center["y"])
page.wait_for_timeout(1500)
leaf_center = _selector_option_center(page, category_name)
if leaf_center:
page.mouse.click(leaf_center["x"], leaf_center["y"])
selected = True
else:
option_center = _selector_option_center(page, category_name)
if option_center:
page.mouse.click(option_center["x"], option_center["y"])
selected = True
if not selected:
print(f" [WARN] 未找到类目选项 {label}")
_dump_peer_filter_debug(page, f"category_option_missing_{category_name}")
try:
menus = page.locator(
".jmtd-dropdown-panel.jmtd-dropdown-lists:visible"
)
if menus.count() and selector is not None:
selector.locator(
".jmtd-base-input-top:visible"
).click(timeout=3000)
page.wait_for_timeout(300)
if menus.count():
page.keyboard.press("Escape")
except Exception:
pass
return False
page.wait_for_timeout(1500)
# 应用筛选并等待榜单刷新
try:
query_box = page.locator("button", has_text="查询").first.bounding_box(
timeout=4000
)
if query_box:
page.mouse.click(
query_box["x"] + query_box["width"] / 2,
query_box["y"] + query_box["height"] / 2,
)
except Exception:
pass # 部分面板选择后立即生效
page.wait_for_timeout(4500)
try:
# 注意不能用共享 extract():它会把结果字符串化导致布尔失真
table_present = bool(
page.evaluate(
"() => Array.from(document.querySelectorAll('table'))"
".some(t => (t.innerText || '').includes('排名'))"
)
)
except Exception:
table_present = False
if not table_present:
print(" [WARN] 类目切换后未见榜单表格,禁止复用旧数据")
_dump_peer_filter_debug(page, f"table_missing_{category_name}")
return False
print(f" [OK] 已切换到 {label},榜单表格已刷新")
return True
def extract_ranking_table(page):
"""提取商家榜单表格数据"""
ranking_data = extract(page, """() => {
const rows = document.querySelectorAll('.jmtd-table-main tbody tr');
"""提取店铺榜表格;按表头行映射列,字段名沿用旧口径。
新版列:排名/店铺信息/成交金额/成交单量/成交商品件数/访客数/浏览量/
搜索点击次数;旧字段 followers(关注人数)新版已无此列,占位 '--'。
"""
try:
# 直接 evaluate 取结构化列表(共享 extract() 会把结果字符串化)
ranking_data = page.evaluate(
"""() => {
const table = Array.from(document.querySelectorAll('table'))
.find(t => (t.innerText || '').includes('排名'));
if (!table) return [];
const trs = Array.from(table.querySelectorAll('tr'));
let headers = null;
const result = [];
for (const row of rows) {
const cells = row.querySelectorAll('td');
if (cells.length < 7) continue;
// 排名提取:优先数字文本,其次检查 1/2/3 名图标 class
let rank = '';
const rankNumEl = cells[0].querySelector('.jmtd-table-formatter-rank-number');
if (rankNumEl) {
rank = rankNumEl.textContent.trim();
for (const tr of trs) {
const cells = Array.from(tr.querySelectorAll('th,td'))
.map(c => (c.innerText || '').trim());
if (!cells.length) continue;
if (headers === null) {
if (cells.includes('排名')) headers = cells;
continue;
}
if (!rank) {
const rankIconEl = cells[0].querySelector('.jmtd-table-formatter-rank');
if (rankIconEl) {
const iconMatch = rankIconEl.className.match(/jmtd-table-formatter-rank-icon-(\\d)/);
if (iconMatch) {
rank = iconMatch[1];
}
}
}
if (!rank) {
rank = cells[0].textContent.trim();
}
const shopLink = cells[1].querySelector('a');
const shop = shopLink ? shopLink.textContent.trim() : cells[1].textContent.trim();
const item = {
rank: rank,
shop: shop,
amount: cells[2]?.textContent?.trim() || '',
orders: cells[3]?.textContent?.trim() || '',
followers: cells[4]?.textContent?.trim() || '',
visitors: cells[5]?.textContent?.trim() || '',
searchClicks: cells[6]?.textContent?.trim() || '',
};
if (cells.includes('排名')) continue;
const item = {};
cells.forEach((value, index) => {
const header = headers[index] || `col${index}`;
item[header] = value;
});
result.push(item);
}
return JSON.stringify(result);
}""", "[]")
return result;
}"""
)
except Exception:
ranking_data = []
if not isinstance(ranking_data, list):
ranking_data = []
rows = []
if isinstance(ranking_data, str):
try:
rows = json.loads(ranking_data)
except json.JSONDecodeError:
rows = []
header_alias = {
"排名": "rank",
"店铺信息": "shop",
"成交金额": "amount",
"成交单量": "orders",
"成交商品件数": "items",
"访客数": "visitors",
"浏览量": "views",
"搜索点击次数": "searchClicks",
}
for raw in ranking_data if isinstance(ranking_data, list) else []:
shop_text = str(raw.get("店铺信息", "")).splitlines()
item = {
"rank": str(raw.get("排名", "")).strip(),
"shop": shop_text[0].strip() if shop_text else "",
"amount": str(raw.get("成交金额", "")).strip(),
"orders": str(raw.get("成交单量", "")).strip(),
"followers": "--",
"visitors": str(raw.get("访客数", "")).strip(),
"searchClicks": str(raw.get("搜索点击次数", "")).strip(),
"items": str(raw.get("成交商品件数", "")).strip(),
"views": str(raw.get("浏览量", "")).strip(),
}
rows.append(item)
return rows
@@ -1086,9 +1315,13 @@ def step_return_old_version(page: Page):
def step_goto_vender_ranks(page: Page):
print("[Step 4] 跳转到新版商家榜单...")
page.goto(VENDER_RANKS_URL, wait_until="domcontentloaded", timeout=60000)
print("[Step 4] 跳转到新版「行业榜单-店铺榜」...")
page.goto(INDUSTRY_TOP_URL, wait_until="domcontentloaded", timeout=60000)
page.wait_for_timeout(8000)
current_url = page.url or ""
if "error.html" in current_url or "industry-top" not in current_url:
_dump_peer_filter_debug(page, "industry_top_bad_landing")
raise RuntimeError(f"行业榜单页跳转异常(京东侧可能再次改版): {current_url}")
body_text = page.locator("body").inner_text(timeout=10000)
permission_markers = (
"当前身份下暂无此菜单",
@@ -1098,140 +1331,35 @@ def step_goto_vender_ranks(page: Page):
)
if any(marker in body_text for marker in permission_markers):
raise RuntimeError(
"京东商智新版商家榜单当前账号无菜单或权限,"
f"请确认该账号已开通榜单模块: {page.url}"
"京东商智新版行业榜单当前账号无菜单或权限,"
f"请确认该账号已开通榜单模块: {current_url}"
)
print(f" 已进入新版商家榜单: {page.url}")
if "店铺榜" not in body_text:
_dump_peer_filter_debug(page, "shop_rank_tab_missing")
raise RuntimeError(f"行业榜单页缺少「店铺榜」入口: {current_url}")
page.locator("text=店铺榜").first.click(timeout=10000)
page.wait_for_timeout(5000)
print(f" 已进入店铺榜: {page.url}")
# ── 搜索店铺 ──────────────────────────────────────────────────────────────────
def _locate_peer_search_input(page: Page):
"""Locate the shop search box inside the 交易榜单 section.
The vender-ranks page also has a global search box at the very top; generic
selectors (and the ``input`` first-match fallback) can hit that one and then
search nothing. Prefer the input that lives in the same container hierarchy
as the visible 交易榜单 heading.
"""
try:
temp_id = page.evaluate(
r"""() => {
const visible = (el) => {
if (!el) return false;
const style = window.getComputedStyle(el);
const rect = el.getBoundingClientRect();
return style.display !== 'none' && style.visibility !== 'hidden'
&& rect.width > 0 && rect.height > 0;
};
const isSearchish = (input) => {
const type = (input.type || '').toLowerCase();
if (['hidden', 'checkbox', 'radio', 'submit', 'button'].includes(type)) {
return false;
}
// 全局 AI 搜索框(找功能/找指标/找商品)不是店铺搜索框,永远不要选中
if (input.classList.contains('ai-search-input')) {
return false;
}
// 交易榜单区块的店铺搜索框(无 placeholder,靠 class 精确命中)
if (input.classList.contains('jmtd-input-input')
|| input.classList.contains('jmt-input-input')) {
return true;
}
const placeholder = (input.placeholder || '').trim();
const parentText = input.closest('.jmtd-base-input')?.textContent || '';
return Boolean(placeholder) || /搜索|查询|店铺|商家/.test(parentText);
};
const headings = Array.from(document.querySelectorAll('*'))
.filter((el) => visible(el) && (el.textContent || '').trim() === '交易榜单');
for (const heading of headings) {
let current = heading.parentElement;
for (let depth = 0; current && depth < 15; depth += 1, current = current.parentElement) {
const inputs = Array.from(current.querySelectorAll('input'))
.filter(visible)
.filter(isSearchish);
if (inputs.length === 1) {
const tempId = '__peer_search_input_' + Date.now() + '_' + Math.random().toString(36).slice(2);
inputs[0].setAttribute('id', tempId);
return tempId;
}
}
}
return null;
}"""
)
except Exception:
return None
if not isinstance(temp_id, str) or not temp_id:
return None
try:
candidate = page.locator(f"#{temp_id}").first
if candidate.count() == 1 and candidate.is_visible(timeout=2_000):
print(" [OK] 已定位交易榜单区块内的店铺搜索框")
return candidate
except Exception:
pass
return None
def search_shop(page: Page, shop_name: str):
"""在商家榜单页面搜索指定店铺"""
"""新版店铺榜:固定 placeholder 的店铺搜索框过滤榜单。"""
print(f" 搜索店铺: {shop_name}")
# 优先定位“交易榜单”区块内的店铺搜索框,避免命中页面顶部全局搜索框
search_input = _locate_peer_search_input(page)
# 尝试多种搜索框选择器
if search_input is None:
for selector in [
'.jmtd-base-input input.jmtd-input-input',
'.jmtd-base-input input.jmt-input-input',
'input[placeholder*="搜索"]',
'input[placeholder*="店铺"]',
'input[placeholder*="商家"]',
'input[type="search"]',
'.search-input input',
'.jmtd-search input',
]:
search_input = page.locator('input[placeholder="请输入店铺名称"]').first
try:
el = page.locator(selector).first
if el.is_visible(timeout=2000):
# 排除全局 AI 搜索框,避免把品牌名输到"找功能/找指标/找商品"
if el.evaluate("(input) => input.classList.contains('ai-search-input')"):
continue
search_input = el
break
except Exception:
continue
if not search_input:
# 兜底:优先 jmtd-input-input,其次任意 input(仍排除 AI 搜索框)
try:
fallback = page.locator('input.jmtd-input-input, input.jmt-input-input').first
if fallback.is_visible(timeout=2000):
search_input = fallback
except Exception:
pass
if not search_input:
search_input = page.locator('input').first
print(" [WARN] 未找到明确的搜索框,使用第一个 input")
# 清空并输入店铺名
search_input.wait_for(state="visible", timeout=10000)
except Exception as exc:
_dump_peer_filter_debug(page, "shop_search_input_missing")
raise RuntimeError(f"未找到店铺搜索框: {exc}") from exc
search_input.click()
search_input.fill("")
page.wait_for_timeout(500)
page.wait_for_timeout(300)
search_input.fill(shop_name)
page.wait_for_timeout(1000)
# 尝试点击搜索按钮或按回车
try:
search_btn = page.locator('button:has-text("搜索"), button:has-text("查询"), .search-btn, .jmtd-search-btn').first
search_btn.click(timeout=3000)
except Exception:
page.wait_for_timeout(800)
search_input.press("Enter")
page.wait_for_timeout(8000)
page.wait_for_timeout(4000)
print(f" [OK] 搜索完成: {shop_name}")
-1
View File
@@ -52,7 +52,6 @@ MAX_NOTIFICATION_BINDINGS_PER_PERSON = 16
NOTIFICATION_CAPABILITIES: Mapping[str, str] = MappingProxyType(
{
"content.marketing_report.daily": "正式执行且启用发送时投递最终营销日报。",
"content.relogin.weekly": "刷新期间投递各平台扫码二维码,并在平台失败时投递失败汇总。",
"product.alert.daily": "命中销量下滑规则且尚未重复通知时投递告警。",
"product.market_rank": "存在本次新生成的平台报告链接且尚未重复通知时投递。",
"product.online_stock_alert.daily": "线上商品当前库存低于 3 时向配置接收人发送库存预警。",
@@ -1,4 +1,5 @@
from datetime import datetime
from pathlib import Path
from types import SimpleNamespace
import pytest
@@ -723,3 +724,168 @@ def test_wait_for_exported_excel_tolerates_driver_errors(monkeypatch):
assert chanmama._wait_for_exported_excel(0.0, driver=_BrokenDriver()) == (
"/downloads/已有.xlsx"
)
def test_refresh_then_export_relogin_on_expired_session(monkeypatch):
"""运行中途登录态失效: 自动账密重登一次并重试当前博主,不影响其他博主。"""
events = []
navigate_calls = []
def fake_navigate(_driver, url):
navigate_calls.append(url)
if len(navigate_calls) == 1:
raise RuntimeError(
"蝉妈妈登录态失效,目标页被重定向到: "
"https://www.chanmama.com/login.html"
)
monkeypatch.setattr(chanmama, "navigate_to_target", fake_navigate)
monkeypatch.setattr(
chanmama,
"refresh_zero_exposure_videos",
lambda _driver: {
"clicked": 0, "pending": 0, "zero_without_button": 0,
"unconfirmed": 0, "pages": 1,
},
)
monkeypatch.setattr(
chanmama, "login",
lambda _driver: events.append("login") or True,
)
monkeypatch.setattr(chanmama.time, "sleep", lambda _seconds: None)
monkeypatch.setattr(chanmama, "_max_excel_mtime", lambda: 0.0)
monkeypatch.setattr(
chanmama, "export_video_data", lambda _driver, mtime: "x.xlsx"
)
monkeypatch.setattr(
chanmama, "parse_chanmama_excel", lambda _path: ([{"title": "v"}], None)
)
records, summaries = chanmama.refresh_then_export_accounts(
object(), ["account-1", "account-2"], wait_seconds=0
)
assert events == ["login"] # 整套运行只自动重登一次
assert len(records) == 2 # 两个博主最终都导出成功
assert summaries[0]["pages"] == 1 # 博主 1 重登后刷新成功
def test_refresh_then_export_no_relogin_for_other_errors(monkeypatch):
"""非登录态失效的异常不触发自动重登。"""
events = []
def fake_navigate(_driver, url):
raise RuntimeError("蝉妈妈目标页加载失败: timeout")
monkeypatch.setattr(chanmama, "navigate_to_target", fake_navigate)
monkeypatch.setattr(
chanmama, "login",
lambda _driver: events.append("login") or True,
)
monkeypatch.setattr(chanmama.time, "sleep", lambda _seconds: None)
monkeypatch.setattr(
chanmama, "refresh_zero_exposure_videos", lambda _driver: {}
)
monkeypatch.setattr(chanmama, "_max_excel_mtime", lambda: 0.0)
monkeypatch.setattr(
chanmama, "export_video_data", lambda _driver, mtime: None
)
records, _summaries = chanmama.refresh_then_export_accounts(
object(), ["account-1"], wait_seconds=0
)
assert events == [] # 普通异常不触发自动重登
assert records == [] # 导出失败不产出记录
class _RenderFakeDriver:
"""navigate_to_target 渲染链路的桩:列表只在浮层清理后才就绪。"""
def __init__(self, ready_after_dismiss=1):
self.current_url = (
"https://www.chanmama.com/bloggerRank/x.html?activeTab=aweme"
)
self.visited = []
self.dismiss_runs = 0
self.ready_after_dismiss = ready_after_dismiss
self.screenshots = []
def get(self, url):
self.visited.append(url)
def find_element(self, _by, _value):
return object()
def execute_script(self, script, *_args):
if script == chanmama._OVERLAY_DISMISS_SCRIPT:
self.dismiss_runs += 1
return 2
if script == chanmama._VIDEO_LIST_READY_SCRIPT:
return self.dismiss_runs >= self.ready_after_dismiss
raise AssertionError(f"unexpected script: {script[:60]}")
def save_screenshot(self, path):
self.screenshots.append(str(path))
Path(path).write_bytes(b"png")
return True
@property
def page_source(self):
return "<html>diag</html>"
def test_navigate_to_target_clears_overlays_before_render_wait(monkeypatch):
monkeypatch.setattr(chanmama.time, "sleep", lambda _seconds: None)
driver = _RenderFakeDriver(ready_after_dismiss=1)
assert chanmama.navigate_to_target(
driver, "https://www.chanmama.com/bloggerRank/x.html"
) is True
assert driver.dismiss_runs >= 1 # 弹窗清理后才轮到渲染判定
assert driver.visited == ["https://www.chanmama.com/bloggerRank/x.html"]
def test_navigate_to_target_timeout_saves_diagnostics(monkeypatch, tmp_path):
monkeypatch.setattr(chanmama.time, "sleep", lambda _seconds: None)
monkeypatch.setattr(chanmama, "CHANMAMA_LIST_READY_TIMEOUT_SECONDS", 0.05)
monkeypatch.setattr(chanmama, "DOWNLOAD_DIR", str(tmp_path))
driver = _RenderFakeDriver(ready_after_dismiss=99)
with pytest.raises(RuntimeError, match="蝉妈妈目标页加载失败"):
chanmama.navigate_to_target(
driver, "https://www.chanmama.com/bloggerRank/x.html"
)
assert driver.dismiss_runs >= 2 # 两次尝试各自清理过浮层
assert len(driver.screenshots) == 1
diag_dir = tmp_path / "_diag"
assert list(diag_dir.glob("render_failure_*.png"))
assert list(diag_dir.glob("render_failure_*.html"))
def test_dismiss_overlays_ignores_driver_errors():
class BoomDriver:
def execute_script(self, _script, *_args):
raise RuntimeError("execution context destroyed")
assert chanmama._dismiss_overlays(BoomDriver()) == 0
def test_wait_video_list_ready_tolerates_transient_script_errors(monkeypatch):
monkeypatch.setattr(chanmama.time, "sleep", lambda _seconds: None)
class FlakyDriver:
def __init__(self):
self.ready_calls = 0
def execute_script(self, script, *_args):
if script == chanmama._OVERLAY_DISMISS_SCRIPT:
return 0
self.ready_calls += 1
if self.ready_calls == 1:
raise RuntimeError("frame detached")
return True
chanmama._wait_video_list_ready(FlakyDriver(), 2) # 不抛即通过
@@ -21,20 +21,11 @@ def test_page_crash_requires_a_fresh_browser():
@pytest.mark.parametrize("batch", [xhs_batch, douyin_batch])
def test_non_acceptance_relogin_runs_platform_script(monkeypatch, batch):
class Policy:
enabled = False
def test_batch_tools_no_longer_embed_relogin(batch):
"""扫码重登已统一到账号 vault(gyxx accounts login),工具内不再提供。"""
calls = []
monkeypatch.setattr(batch, "current_acceptance_policy", lambda: Policy())
monkeypatch.setattr(
batch.subprocess,
"call",
lambda command, **kwargs: calls.append((command, kwargs)) or 0,
)
assert batch.do_relogin() is True
assert calls[0][0][1] == str(batch.RELOGIN_SCRIPT)
assert not hasattr(batch, "do_relogin")
assert not hasattr(batch, "RELOGIN_SCRIPT")
def test_comment_checkpoint_matches_both_note_id_and_url(tmp_path):
@@ -10,7 +10,6 @@ from gyxx_flow.modules.content_marketing.data.tools import (
analyze_comments,
analyze_note,
daily_report_card,
friday_relogin_parallel,
)
@@ -138,15 +137,3 @@ def test_note_report_sends_only_to_wang_yunlong(
assert calls[0]["user_id"] == ANALYZER_ACCEPTANCE_RECIPIENT_OPEN_ID
assert calls[0]["profile"] == "hermes-analyzer"
def test_relogin_notifications_resolve_only_to_wang_yunlong(
monkeypatch,
tmp_path: Path,
) -> None:
_enable_acceptance(monkeypatch, tmp_path)
assert (
friday_relogin_parallel.recipient_for_platform("pgy", "ou_other")
== ANALYZER_ACCEPTANCE_RECIPIENT_OPEN_ID
)
@@ -1,569 +0,0 @@
from types import SimpleNamespace
import pytest
from gyxx_flow.modules.content_marketing.data.tools import (
friday_relogin_parallel as relogin,
)
from gyxx_flow.modules.content_marketing.data.tools import relogin_transaction
def test_five_expected_platforms_are_configured():
assert set(relogin.PLATFORMS) == {
"bilibili", "douyin", "pgy", "xingtu", "xiaohongshu"
}
assert relogin.PLATFORMS["xingtu"].required_cookie == "sessionid"
assert relogin.PLATFORMS["douyin"].required_cookie == "sessionid"
def test_manual_relogin_can_select_only_xingtu():
assert relogin.resolve_platform_names(["xingtu"]) == ["xingtu"]
assert relogin.resolve_platform_names(None) == list(relogin.PLATFORMS)
def test_default_screenshot_recipients_are_routed_by_platform():
assert relogin.DEFAULT_RECIPIENTS == {
"pgy": "ou_24cc944d6e43c69c59d6560ad4e2ae6e",
"xingtu": "ou_24cc944d6e43c69c59d6560ad4e2ae6e",
"douyin": "ou_7ad5fc8012e2f741afc5346e05ffd447",
"bilibili": "ou_7ad5fc8012e2f741afc5346e05ffd447",
"xiaohongshu": "ou_7ad5fc8012e2f741afc5346e05ffd447",
}
assert relogin.recipient_for_platform("pgy") == "ou_24cc944d6e43c69c59d6560ad4e2ae6e"
assert relogin.recipient_for_platform("douyin") == "ou_7ad5fc8012e2f741afc5346e05ffd447"
assert relogin.recipient_for_platform("bilibili") == "ou_7ad5fc8012e2f741afc5346e05ffd447"
assert relogin.recipient_for_platform("pgy", "ou_override") == "ou_override"
def test_failed_platforms_are_grouped_by_their_responsible_recipient():
grouped = relogin.group_platforms_by_recipient(
["pgy", "douyin", "bilibili", "xingtu", "xiaohongshu"]
)
assert grouped == {
"ou_24cc944d6e43c69c59d6560ad4e2ae6e": ["pgy", "xingtu"],
"ou_7ad5fc8012e2f741afc5346e05ffd447": [
"douyin", "bilibili", "xiaohongshu"
],
}
def test_legacy_route_keeps_the_existing_per_platform_recipients(monkeypatch):
observed = {}
def fake_resolver(workflow_id, defaults):
observed["workflow_id"] = workflow_id
observed["defaults"] = defaults
return relogin.ResolvedNotificationRoute(
configured=False,
enabled=True,
app_profile=None,
open_ids=tuple(defaults),
)
monkeypatch.setattr(relogin, "resolve_notification_route", fake_resolver)
route = relogin.resolve_weekly_notification_route()
assert observed == {
"workflow_id": "content.relogin.weekly",
"defaults": (
"ou_24cc944d6e43c69c59d6560ad4e2ae6e",
"ou_7ad5fc8012e2f741afc5346e05ffd447",
),
}
assert relogin.recipients_for_platform("pgy", route) == (
"ou_24cc944d6e43c69c59d6560ad4e2ae6e",
)
assert relogin.recipients_for_platform("douyin", route) == (
"ou_7ad5fc8012e2f741afc5346e05ffd447",
)
def test_disabled_route_still_runs_relogin_without_sending(monkeypatch):
route = relogin.ResolvedNotificationRoute(
configured=True,
enabled=False,
app_profile="hermes-analyzer",
open_ids=(),
)
rounds = []
monkeypatch.setattr(
relogin,
"current_acceptance_policy",
lambda: SimpleNamespace(enabled=False),
)
monkeypatch.setattr(
relogin,
"resolve_weekly_notification_route",
lambda _override=None: route,
)
def fake_round_factory(_args, resolved_route, notification_failures):
assert resolved_route is route
assert notification_failures == []
def run_round(names, attempt):
rounds.append((tuple(names), attempt))
return {name: True for name in names}
return run_round
monkeypatch.setattr(relogin, "_run_round_factory", fake_round_factory)
monkeypatch.setattr(
relogin,
"_send_lark",
lambda *_args, **_kwargs: (_ for _ in ()).throw(
AssertionError("disabled route must not send Lark messages")
),
)
monkeypatch.setattr(
relogin.sys,
"argv",
["friday_relogin_parallel.py", "--platform", "pgy"],
)
assert relogin.main() == 0
assert rounds == [(("pgy",), 1)]
def test_enabled_route_sends_qr_text_and_image_to_every_recipient(
monkeypatch,
tmp_path,
):
route = relogin.ResolvedNotificationRoute(
configured=True,
enabled=True,
app_profile="hermes-analyzer",
open_ids=("ou_first123", "ou_second456"),
)
sent = []
image = tmp_path / "qrcode.png"
monkeypatch.setattr(
relogin,
"_send_lark",
lambda recipient, text=None, image=None, app_profile=None: sent.append(
(recipient, text, image, app_profile)
),
)
failures = relogin._send_qr_notification(
route,
"pgy",
override=None,
text="请扫码",
image=image,
)
assert failures == ()
assert sent == [
("ou_first123", "请扫码", None, "hermes-analyzer"),
("ou_first123", None, image, "hermes-analyzer"),
("ou_second456", "请扫码", None, "hermes-analyzer"),
("ou_second456", None, image, "hermes-analyzer"),
]
assert relogin.group_platforms_by_notification_recipient(
["pgy", "douyin"], route
) == {
"ou_first123": ["pgy", "douyin"],
"ou_second456": ["pgy", "douyin"],
}
def test_qr_delivery_aggregates_errors_and_attempts_every_recipient(
monkeypatch,
tmp_path,
):
route = relogin.ResolvedNotificationRoute(
configured=True,
enabled=True,
app_profile="hermes-analyzer",
open_ids=("ou_first123", "ou_second456"),
)
attempts = []
image = tmp_path / "qrcode.png"
def fake_send(recipient, text=None, image=None, app_profile=None):
stage = "qr_text" if text is not None else "qr_image"
attempts.append((recipient, stage, app_profile))
if (recipient, stage) == ("ou_first123", "qr_text"):
raise RuntimeError("first text rejected")
if (recipient, stage) == ("ou_second456", "qr_image"):
raise RuntimeError("second image rejected")
monkeypatch.setattr(relogin, "_send_lark", fake_send)
failures = relogin._send_qr_notification(
route,
"pgy",
override=None,
text="请扫码",
image=image,
)
assert attempts == [
("ou_first123", "qr_text", "hermes-analyzer"),
("ou_first123", "qr_image", "hermes-analyzer"),
("ou_second456", "qr_text", "hermes-analyzer"),
("ou_second456", "qr_image", "hermes-analyzer"),
]
assert failures == (
relogin.NotificationDeliveryFailure(
stage="qr_text",
platform="pgy",
recipient="ou_first123",
detail="first text rejected",
),
relogin.NotificationDeliveryFailure(
stage="qr_image",
platform="pgy",
recipient="ou_second456",
detail="second image rejected",
),
)
def test_failure_summary_aggregates_every_recipient_error(monkeypatch):
route = relogin.ResolvedNotificationRoute(
configured=True,
enabled=True,
app_profile="hermes-analyzer",
open_ids=("ou_first123", "ou_second456"),
)
attempts = []
def fail_send(recipient, **_kwargs):
attempts.append(recipient)
raise RuntimeError(f"rejected {recipient}")
monkeypatch.setattr(relogin, "_send_lark", fail_send)
failures = relogin._send_failure_summaries(
route,
["pgy", "douyin"],
override=None,
max_attempts=3,
)
assert attempts == ["ou_first123", "ou_second456"]
assert [failure.stage for failure in failures] == [
"failure_summary",
"failure_summary",
]
assert [failure.recipient for failure in failures] == attempts
assert [failure.platform for failure in failures] == [
"pgy,douyin",
"pgy,douyin",
]
def test_main_returns_failure_and_reports_all_notification_errors(
monkeypatch,
capsys,
):
route = relogin.ResolvedNotificationRoute(
configured=True,
enabled=True,
app_profile="hermes-analyzer",
open_ids=("ou_first123", "ou_second456"),
)
monkeypatch.setattr(
relogin,
"current_acceptance_policy",
lambda: SimpleNamespace(enabled=False),
)
monkeypatch.setattr(
relogin,
"resolve_weekly_notification_route",
lambda _override=None: route,
)
def fake_round_factory(_args, _route, notification_failures):
notification_failures.extend(
(
relogin.NotificationDeliveryFailure(
"qr_text", "pgy", "ou_first123", "text rejected"
),
relogin.NotificationDeliveryFailure(
"qr_image", "pgy", "ou_second456", "image rejected"
),
)
)
return lambda names, _attempt: {name: True for name in names}
monkeypatch.setattr(relogin, "_run_round_factory", fake_round_factory)
monkeypatch.setattr(
relogin.sys,
"argv",
["friday_relogin_parallel.py", "--platform", "pgy"],
)
assert relogin.main() == 2
output = capsys.readouterr().out
assert "Final relogin status: {'pgy': True}" in output
assert "Final notification status: failed (2 deliveries)" in output
assert "recipient=ou_first123 error=text rejected" in output
assert "recipient=ou_second456 error=image rejected" in output
def test_round_completes_relogin_after_notification_failure(monkeypatch, tmp_path):
route = relogin.ResolvedNotificationRoute(
configured=True,
enabled=True,
app_profile="hermes-analyzer",
open_ids=("ou_first123", "ou_second456"),
)
args = SimpleNamespace(
round_timeout=10,
screenshot_delay=0,
no_send=False,
recipient=None,
max_attempts=1,
)
failures = []
attempts = []
waits = []
class Process:
pid = 1234
def wait(self, timeout):
waits.append(timeout)
return 0
monkeypatch.setattr(relogin.subprocess, "Popen", lambda *_args, **_kwargs: Process())
monkeypatch.setattr(relogin, "environment_for_child_script", lambda *_args: {})
monkeypatch.setattr(
relogin,
"binding_from_environment",
lambda _environment: SimpleNamespace(cookie_file=tmp_path / "cookies.json"),
)
monkeypatch.setattr(relogin.time, "sleep", lambda _seconds: None)
monkeypatch.setattr(relogin, "_login_windows", lambda: {"pgy": 101})
monkeypatch.setattr(relogin, "_tile_windows", lambda _handles: None)
monkeypatch.setattr(relogin, "_capture_window", lambda _hwnd, _output: None)
monkeypatch.setattr(relogin, "_valid_cookie", lambda *_args: True)
def fake_send(recipient, text=None, image=None, **_kwargs):
stage = "qr_text" if text is not None else "qr_image"
attempts.append((recipient, stage))
if (recipient, stage) == ("ou_first123", "qr_text"):
raise RuntimeError("temporary send failure")
monkeypatch.setattr(relogin, "_send_lark", fake_send)
run_round = relogin._run_round_factory(args, route, failures)
assert run_round(["pgy"], 1) == {"pgy": True}
assert waits == [100]
assert attempts == [
("ou_first123", "qr_text"),
("ou_first123", "qr_image"),
("ou_second456", "qr_text"),
("ou_second456", "qr_image"),
]
assert failures == [
relogin.NotificationDeliveryFailure(
stage="qr_text",
platform="pgy",
recipient="ou_first123",
detail="temporary send failure",
)
]
@pytest.mark.parametrize(
("returncode", "cookie_valid", "expected_status", "expected_failure_count"),
(
(0, True, True, 0),
(1, False, False, 4),
),
)
def test_missing_qr_window_is_classified_after_relogin_result(
monkeypatch,
tmp_path,
returncode,
cookie_valid,
expected_status,
expected_failure_count,
):
route = relogin.ResolvedNotificationRoute(
configured=True,
enabled=True,
app_profile="hermes-analyzer",
open_ids=("ou_first123", "ou_second456"),
)
args = SimpleNamespace(
round_timeout=10,
screenshot_delay=0,
no_send=False,
recipient=None,
max_attempts=1,
)
failures = []
class Process:
pid = 1234
def wait(self, timeout):
assert timeout == 100
return returncode
monkeypatch.setattr(relogin.subprocess, "Popen", lambda *_args, **_kwargs: Process())
monkeypatch.setattr(relogin, "environment_for_child_script", lambda *_args: {})
monkeypatch.setattr(
relogin,
"binding_from_environment",
lambda _environment: SimpleNamespace(cookie_file=tmp_path / "cookies.json"),
)
monkeypatch.setattr(relogin.time, "sleep", lambda _seconds: None)
monkeypatch.setattr(relogin, "_login_windows", lambda: {})
monkeypatch.setattr(relogin, "_window_for_process", lambda _pid: None)
monkeypatch.setattr(relogin, "_tile_windows", lambda _handles: None)
monkeypatch.setattr(relogin, "_valid_cookie", lambda *_args: cookie_valid)
monkeypatch.setattr(
relogin,
"_send_lark",
lambda *_args, **_kwargs: pytest.fail(
"no QR window must not start a send before child classification"
),
)
run_round = relogin._run_round_factory(args, route, failures)
assert run_round(["pgy"], 1) == {"pgy": expected_status}
assert len(failures) == expected_failure_count
assert {failure.stage for failure in failures} <= {"qr_text", "qr_image"}
assert all(
failure.detail == "no visible QR window and relogin did not succeed"
for failure in failures
)
def test_lark_send_uses_shared_analyzer_user_sender_for_text_and_image(
monkeypatch,
tmp_path,
):
calls = []
image = tmp_path / "qrcode.png"
image.write_bytes(b"png")
monkeypatch.setattr(relogin, "ANALYZER_DIR", tmp_path)
monkeypatch.setenv("HERMES_ANALYZER_TOKEN", "must-not-leak")
def fake_send(**kwargs):
calls.append(kwargs)
return {"message_id": f"om_{len(calls)}"}
monkeypatch.setattr(relogin, "send_lark_bot_message", fake_send)
receipts = relogin._send_lark(
"ou_recipient123",
text="登录提醒",
image=image,
)
assert [receipt["message_id"] for receipt in receipts] == ["om_1", "om_2"]
assert calls[0]["user_id"] == "ou_recipient123"
assert calls[0]["text"] == "登录提醒"
assert "image" not in calls[0]
assert calls[1]["user_id"] == "ou_recipient123"
assert calls[1]["image"] == image
assert "text" not in calls[1]
assert all(call["profile"] == "hermes-analyzer" for call in calls)
assert all("HERMES_ANALYZER_TOKEN" not in call["env"] for call in calls)
def test_login_windows_are_classified_by_title():
assert relogin._platform_from_window_title("账号登录 - Google Chrome") == "bilibili"
assert relogin._platform_from_window_title("小红书蒲公英 - Google Chrome") == "pgy"
assert relogin._platform_from_window_title("抖音授权 - Google Chrome") == "xingtu"
assert relogin._platform_from_window_title("抖音 - 记录美好生活") == "douyin"
assert relogin._platform_from_window_title("小红书 - 你的生活兴趣社区") == "xiaohongshu"
assert relogin._platform_from_window_title("Codex") is None
def test_retry_only_failed_platforms_for_at_most_three_rounds():
calls = []
def fake_round(names, attempt):
calls.append((tuple(names), attempt))
if attempt == 1:
return {name: name != "xingtu" for name in names}
if attempt == 2:
return {name: False for name in names}
return {name: True for name in names}
result = relogin.run_with_retries(
["bilibili", "pgy", "xingtu", "xiaohongshu"],
max_attempts=3,
run_round=fake_round,
)
assert calls == [
(("bilibili", "pgy", "xingtu", "xiaohongshu"), 1),
(("xingtu",), 2),
(("xingtu",), 3),
]
assert all(result.values())
def test_stops_after_third_failed_round():
calls = []
def always_fail(names, attempt):
calls.append(attempt)
return {name: False for name in names}
result = relogin.run_with_retries(
["pgy"], max_attempts=3, run_round=always_fail
)
assert calls == [1, 2, 3]
assert result == {"pgy": False}
def test_timeout_terminates_the_complete_process_tree(monkeypatch):
calls = []
class Process:
pid = 1234
def terminate(self):
calls.append("fallback-terminate")
def wait(self, timeout):
calls.append(("wait", timeout))
monkeypatch.setattr(
relogin.subprocess,
"run",
lambda command, **kwargs: calls.append((command, kwargs)),
)
relogin._terminate_process_tree(Process())
assert calls[0][0] == ["taskkill", "/PID", "1234", "/T", "/F"]
assert calls[1] == ("wait", 10)
def test_transaction_rolls_back_files_and_profile(tmp_path):
cookie = tmp_path / "cookies.json"
storage = tmp_path / "storage.json"
profile = tmp_path / "profile"
cookie.write_text("old-cookie", encoding="utf-8")
storage.write_text("old-storage", encoding="utf-8")
profile.mkdir()
(profile / "old.txt").write_text("old-profile", encoding="utf-8")
state = relogin_transaction.begin((cookie, storage), profile)
cookie.write_text("partial", encoding="utf-8")
profile.mkdir()
(profile / "partial.txt").write_text("partial", encoding="utf-8")
relogin_transaction.rollback(state)
assert cookie.read_text(encoding="utf-8") == "old-cookie"
assert storage.read_text(encoding="utf-8") == "old-storage"
assert (profile / "old.txt").read_text(encoding="utf-8") == "old-profile"
assert not (profile / "partial.txt").exists()
@@ -416,11 +416,9 @@ def test_acceptance_invalid_session_skips_before_qr_login(
)
monkeypatch.setenv("GYXX_ACCEPTANCE_EVIDENCE_FILE", str(evidence_file))
monkeypatch.setattr(module, "is_logged_in", lambda _page: False)
monkeypatch.setattr(
module,
"wait_for_scan_login",
lambda *_args, **_kwargs: pytest.fail("QR login must remain disabled"),
)
# 扫码登录函数已整体移除;采集器不得重新引入交互登录。
assert not hasattr(module, "wait_for_scan_login")
assert not hasattr(module, "open_scan_login")
class Page:
def __init__(self):
@@ -1,494 +0,0 @@
import pytest
from gyxx_flow.modules.content_marketing import (
pgy_xhs_scraper_v2,
xingtu_scraper_v2,
)
from gyxx_flow.modules.content_marketing.data.tools import (
relogin_douyin,
relogin_pgy,
relogin_xingtu,
)
@pytest.mark.parametrize("module", [relogin_pgy, relogin_xingtu])
def test_failed_relogin_restores_cookie_and_profile(tmp_path, monkeypatch, module):
cookie = tmp_path / "cookies.json"
storage = tmp_path / "storage_state.json"
profile = tmp_path / "profile"
cookie.write_text("old-cookie", encoding="utf-8")
storage.write_text("old-storage", encoding="utf-8")
profile.mkdir()
(profile / "state.txt").write_text("old-profile", encoding="utf-8")
monkeypatch.setattr(module, "COOKIE_FILE", cookie)
monkeypatch.setattr(module, "STATE_FILE", storage)
monkeypatch.setattr(module, "PROFILE_DIR", profile)
cookie_backup, profile_backup, state_backup = module.reset_state()
assert not cookie.exists()
assert not storage.exists()
assert not profile.exists()
cookie.write_text("partial-new-cookie", encoding="utf-8")
storage.write_text("partial-new-storage", encoding="utf-8")
profile.mkdir()
(profile / "partial.txt").write_text("partial", encoding="utf-8")
module.restore_previous_state(cookie_backup, profile_backup, state_backup)
assert cookie.read_text(encoding="utf-8") == "old-cookie"
assert storage.read_text(encoding="utf-8") == "old-storage"
assert (profile / "state.txt").read_text(encoding="utf-8") == "old-profile"
assert not (profile / "partial.txt").exists()
@pytest.mark.parametrize("module", [relogin_pgy, relogin_xingtu])
def test_successful_relogin_keeps_new_cookie(tmp_path, monkeypatch, module):
cookie = tmp_path / "cookies.json"
storage = tmp_path / "storage_state.json"
profile = tmp_path / "profile"
cookie.write_text("old-cookie", encoding="utf-8")
storage.write_text("old-storage", encoding="utf-8")
profile.mkdir()
monkeypatch.setattr(module, "COOKIE_FILE", cookie)
monkeypatch.setattr(module, "STATE_FILE", storage)
monkeypatch.setattr(module, "PROFILE_DIR", profile)
cookie_backup, profile_backup, state_backup = module.reset_state()
cookie.write_text("new-cookie", encoding="utf-8")
storage.write_text("new-storage", encoding="utf-8")
profile.mkdir()
module.finish_successful_relogin(profile_backup)
assert cookie.read_text(encoding="utf-8") == "new-cookie"
assert storage.read_text(encoding="utf-8") == "new-storage"
assert cookie_backup.exists()
assert state_backup.exists()
assert not profile_backup.exists()
def test_xingtu_scan_login_opens_customer_sso_and_clicks_douyin():
class FakeLocator:
def __init__(self):
self.first = self
self.clicked = False
self.evaluated = False
def count(self):
return 1
def is_visible(self):
return True
def click(self, timeout):
self.clicked = True
def evaluate(self, script):
assert script == "e => e.click()"
self.evaluated = True
class FakePage:
url = "https://www.xingtu.cn/?redirect_uri=/ad/creator/market"
def __init__(self):
self.goto_url = None
self.selectors = []
self.douyin = FakeLocator()
def goto(self, url, **kwargs):
self.goto_url = url
self.url = url
def wait_for_timeout(self, _milliseconds):
pass
def locator(self, selector):
self.selectors.append(selector)
return self.douyin
def get_by_text(self, _pattern):
return FakeLocator()
page = FakePage()
xingtu_scraper_v2.open_scan_login(page)
assert page.goto_url == xingtu_scraper_v2.CUSTOMER_LOGIN_URL
assert 'img[src*="aweme.png"]' in page.selectors
assert page.douyin.clicked
def test_xingtu_index_page_is_recognized_as_logged_in():
class Locator:
@property
def first(self):
return self
def wait_for(self, timeout):
assert timeout == 1200
class Page:
url = "https://www.xingtu.cn/ad/creator/index"
def get_by_text(self, _text, exact=False):
return Locator()
assert xingtu_scraper_v2.is_logged_in(Page()) is True
def test_xingtu_index_page_with_xingtu_session_is_logged_in_without_market_ui():
class Locator:
@property
def first(self):
return self
def count(self):
return 0
def wait_for(self, timeout):
raise xingtu_scraper_v2.BrowserTimeoutError("not on market page")
def is_visible(self):
return False
class Context:
def cookies(self):
return [
{
"name": "sessionid",
"value": "fresh",
"domain": ".xingtu.cn",
}
]
class Page:
url = "https://www.xingtu.cn/ad/creator/index"
context = Context()
def get_by_text(self, *_args, **_kwargs):
return Locator()
def locator(self, *_args, **_kwargs):
return Locator()
assert xingtu_scraper_v2.is_logged_in(Page()) is True
def test_xingtu_index_page_without_xingtu_session_is_not_logged_in():
class Locator:
@property
def first(self):
return self
def count(self):
return 0
def wait_for(self, timeout):
raise xingtu_scraper_v2.BrowserTimeoutError("missing")
def is_visible(self):
return False
class Context:
def cookies(self):
return [
{
"name": "sessionid",
"value": "douyin-only",
"domain": ".douyin.com",
}
]
class Page:
url = "https://www.xingtu.cn/ad/creator/index"
context = Context()
def get_by_text(self, *_args, **_kwargs):
return Locator()
def locator(self, *_args, **_kwargs):
return Locator()
assert xingtu_scraper_v2.is_logged_in(Page()) is False
def test_xingtu_public_homepage_is_not_logged_in_even_with_stale_cookie():
class Context:
def cookies(self):
return [{"name": "sessionid", "value": "stale"}]
class Page:
url = "https://www.xingtu.cn/?redirect_uri=/ad/creator/market"
context = Context()
def get_by_text(self, *_args, **_kwargs):
raise AssertionError("public homepage must be rejected before marker checks")
assert xingtu_scraper_v2.is_logged_in(Page()) is False
def test_xingtu_market_page_without_business_ui_rejects_stale_cookie():
class Locator:
@property
def first(self):
return self
def wait_for(self, **_kwargs):
raise xingtu_scraper_v2.BrowserTimeoutError("missing")
def count(self):
return 0
class Context:
def cookies(self):
return [{"name": "sessionid", "value": "stale"}]
class Page:
url = "https://www.xingtu.cn/ad/creator/market"
context = Context()
def get_by_text(self, *_args, **_kwargs):
return Locator()
def locator(self, *_args, **_kwargs):
return Locator()
assert xingtu_scraper_v2.is_logged_in(Page()) is False
def test_xingtu_cookie_validation_requires_real_session_cookie(tmp_path):
cookie = tmp_path / "xingtu.json"
cookie.write_text('[{"name":"other","value":"1"}]', encoding="utf-8")
assert xingtu_scraper_v2.has_valid_login_cookie_file(cookie) is False
cookie.write_text('[{"name":"sessionid","value":"fresh"}]', encoding="utf-8")
assert xingtu_scraper_v2.has_valid_login_cookie_file(cookie) is True
def test_xingtu_login_only_does_not_report_success_when_page_action_was_swallowed(
tmp_path, monkeypatch
):
cookie = tmp_path / "xingtu.json"
cookie.write_text('[{"name":"sessionid","value":"old"}]', encoding="utf-8")
class Session:
def __enter__(self):
return self
def __exit__(self, *_args):
return False
def fetch(self, *_args, **_kwargs):
# Scrapling can log a page_action error and still return a response.
return object()
monkeypatch.setattr(xingtu_scraper_v2, "COOKIE_FILE", cookie)
monkeypatch.setattr(xingtu_scraper_v2, "DynamicSession", lambda **_kwargs: Session())
monkeypatch.setattr(xingtu_scraper_v2, "_force_cleanup_session", lambda _session: None)
assert xingtu_scraper_v2.login_only(1) == 1
def test_xingtu_login_only_verifies_persisted_profile_after_oauth_context_closes(
tmp_path, monkeypatch
):
cookie = tmp_path / "xingtu.json"
class Session:
def __enter__(self):
return self
def __exit__(self, *_args):
return False
def fetch(self, *_args, **_kwargs):
raise RuntimeError("Target page, context or browser has been closed")
monkeypatch.setattr(xingtu_scraper_v2, "COOKIE_FILE", cookie)
monkeypatch.setattr(xingtu_scraper_v2, "DynamicSession", lambda **_kwargs: Session())
monkeypatch.setattr(xingtu_scraper_v2, "_force_cleanup_session", lambda _session: None)
def verify():
cookie.write_text('[{"name":"sessionid","value":"fresh"}]', encoding="utf-8")
return True
monkeypatch.setattr(xingtu_scraper_v2, "verify_persisted_profile_login", verify)
assert xingtu_scraper_v2.login_only(1) == 0
def test_xingtu_scan_login_follows_replacement_page_after_oauth_tab_closes(monkeypatch):
class Context:
def __init__(self):
self.pages = []
def new_page(self):
raise AssertionError("a surviving replacement page should be reused")
class Page:
def __init__(self, context, url, *, closes_on_wait=False):
self.context = context
self.url = url
self.closed = False
self.closes_on_wait = closes_on_wait
def is_closed(self):
return self.closed
def wait_for_timeout(self, _milliseconds):
if self.closes_on_wait:
self.closed = True
def goto(self, url, **_kwargs):
self.url = url
context = Context()
qr_page = Page(
context,
"https://open.douyin.com/platform/oauth/pc/auth",
closes_on_wait=True,
)
market_page = Page(context, xingtu_scraper_v2.MARKET_URL)
context.pages = [qr_page, market_page]
saved = []
monkeypatch.setattr(xingtu_scraper_v2, "open_scan_login", lambda _page: None)
monkeypatch.setattr(
xingtu_scraper_v2, "is_scan_qr_page", lambda page: page is qr_page
)
monkeypatch.setattr(
xingtu_scraper_v2, "is_logged_in", lambda page: page is market_page
)
monkeypatch.setattr(xingtu_scraper_v2, "save_state", lambda page: saved.append(page))
result = xingtu_scraper_v2.wait_for_scan_login(qr_page, 2)
assert result is market_page
assert saved == [market_page]
def test_xingtu_scan_login_detects_business_tab_while_qr_tab_stays_open(monkeypatch):
class Context:
def __init__(self):
self.pages = []
class Page:
def __init__(self, context, url):
self.context = context
self.url = url
def is_closed(self):
return False
def wait_for_timeout(self, _milliseconds):
raise AssertionError("authenticated tab should be detected immediately")
context = Context()
qr_page = Page(context, "https://open.douyin.com/platform/oauth/pc/auth")
index_page = Page(context, "https://www.xingtu.cn/ad/creator/index")
context.pages = [qr_page, index_page]
saved = []
monkeypatch.setattr(xingtu_scraper_v2, "open_scan_login", lambda _page: None)
monkeypatch.setattr(
xingtu_scraper_v2, "is_logged_in", lambda page: page is index_page
)
monkeypatch.setattr(xingtu_scraper_v2, "save_state", lambda page: saved.append(page))
result = xingtu_scraper_v2.wait_for_scan_login(qr_page, 2)
assert result is index_page
assert saved == [index_page]
def test_xingtu_interrupted_transaction_is_recovered(tmp_path, monkeypatch):
cookie = tmp_path / "xingtu_cookies.json"
storage = tmp_path / "xingtu_storage_state.json"
profile = tmp_path / "profile"
cookie.write_text("old-cookie", encoding="utf-8")
storage.write_text("old-storage", encoding="utf-8")
profile.mkdir()
(profile / "old.txt").write_text("old-profile", encoding="utf-8")
monkeypatch.setattr(relogin_xingtu, "COOKIE_FILE", cookie)
monkeypatch.setattr(relogin_xingtu, "STATE_FILE", storage)
monkeypatch.setattr(relogin_xingtu, "PROFILE_DIR", profile)
cookie_backup, profile_backup, state_backup = relogin_xingtu.reset_state()
profile.mkdir()
(profile / "partial.txt").write_text("partial", encoding="utf-8")
assert relogin_xingtu.recover_interrupted_state() is True
assert cookie.read_text(encoding="utf-8") == "old-cookie"
assert storage.read_text(encoding="utf-8") == "old-storage"
assert (profile / "old.txt").read_text(encoding="utf-8") == "old-profile"
assert not (profile / "partial.txt").exists()
assert cookie_backup is not None
assert state_backup is not None
assert profile_backup is not None
def test_pgy_scan_login_clicks_qr_switch_image():
class FakeLocator:
def __init__(self):
self.first = self
self.clicked = False
self.evaluated = False
def count(self):
return 1
def is_visible(self):
return True
def click(self, timeout):
self.clicked = True
def evaluate(self, script):
assert script == "e => e.click()"
self.evaluated = True
class FakePage:
def __init__(self):
self.selectors = []
self.qr_switch = FakeLocator()
def wait_for_timeout(self, _milliseconds):
pass
def get_by_text(self, _pattern):
return FakeLocator()
def locator(self, selector):
self.selectors.append(selector)
return self.qr_switch
page = FakePage()
pgy_xhs_scraper_v2.open_scan_login(page)
assert 'img[src*="qr_code"]' in page.selectors
assert page.qr_switch.evaluated
def test_failed_douyin_relogin_restores_cookie_storage_and_profile(tmp_path, monkeypatch):
cookie = tmp_path / "douyin_cookies.json"
storage = tmp_path / "douyin_storage_state.json"
profile = tmp_path / "profile"
cookie.write_text('[{"name":"sessionid","value":"old"}]', encoding="utf-8")
storage.write_text("old-storage", encoding="utf-8")
profile.mkdir()
(profile / "old.txt").write_text("old-profile", encoding="utf-8")
monkeypatch.setattr(relogin_douyin, "COOKIE_FILE", cookie)
monkeypatch.setattr(relogin_douyin, "STATE_FILE", storage)
monkeypatch.setattr(relogin_douyin, "PROFILE_DIR", profile)
monkeypatch.setattr(relogin_douyin.subprocess, "call", lambda *_args, **_kwargs: 1)
monkeypatch.setattr(relogin_douyin.sys, "argv", ["relogin_douyin.py"])
assert relogin_douyin.main() == 1
assert cookie.read_text(encoding="utf-8") == '[{"name":"sessionid","value":"old"}]'
assert storage.read_text(encoding="utf-8") == "old-storage"
assert (profile / "old.txt").read_text(encoding="utf-8") == "old-profile"
@@ -1,410 +0,0 @@
from __future__ import annotations
import json
from pathlib import Path
from types import ModuleType
import pytest
from gyxx_flow.adapters.integration import RuntimeIntegrationCatalog
from gyxx_flow.modules.content_marketing.data.tools import (
relogin_bilibili,
relogin_douyin,
relogin_pgy,
relogin_runtime_paths,
relogin_xiaohongshu,
relogin_xingtu,
)
from gyxx_flow.modules.content_marketing.data.tools.relogin_runtime_paths import (
CONSUMER_BINDING_IDS,
child_browser_environment,
resolve_relogin_runtime_paths,
sync_relogin_state_to_consumers,
)
from gyxx_flow.modules.content_marketing.runtime_paths import PATHS
PROJECT_ROOT = Path(__file__).resolve().parents[3]
def test_relogin_paths_prefer_injected_runtime_binding(tmp_path: Path) -> None:
cookie = tmp_path / "binding" / "cookies.json"
storage = tmp_path / "binding" / "storage_state.json"
profile = tmp_path / "binding" / "profile"
resolved = resolve_relogin_runtime_paths(
"pgy",
{
"GYXX_BROWSER_COOKIE_FILE": str(cookie),
"GYXX_BROWSER_STORAGE_STATE_FILE": str(storage),
"GYXX_BROWSER_PROFILE_DIR": str(profile),
},
)
assert resolved.cookie_file == cookie.resolve()
assert resolved.storage_state_file == storage.resolve()
assert resolved.profile_dir == profile.resolve()
def test_relogin_paths_keep_legacy_standalone_layout() -> None:
resolved = resolve_relogin_runtime_paths("xiaohongshu", {})
assert resolved.cookie_file == (
PATHS.state_root / "cookies/xiaohongshu_cookies.json"
).resolve()
assert resolved.storage_state_file == (
PATHS.state_root / "cookies/xiaohongshu_storage_state.json"
).resolve()
assert resolved.profile_dir == (
PATHS.state_root / "browser-profiles/xiaohongshu"
).resolve()
def test_child_login_environment_preserves_wrapper_binding(tmp_path: Path) -> None:
cookie = tmp_path / "cookies.json"
storage = tmp_path / "storage_state.json"
profile = tmp_path / "profile"
environment = child_browser_environment(
cookie_file=cookie,
storage_state_file=storage,
profile_dir=profile,
environment={"GYXX_MODULE_ID": "content_marketing", "KEEP": "1"},
)
assert environment["GYXX_BROWSER_COOKIE_FILE"] == str(cookie.resolve())
assert environment["GYXX_BROWSER_STORAGE_STATE_FILE"] == str(
storage.resolve()
)
assert environment["GYXX_BROWSER_PROFILE_DIR"] == str(profile.resolve())
assert "GYXX_MODULE_ID" not in environment
assert environment["KEEP"] == "1"
@pytest.mark.parametrize(
("module", "platform", "login_cookie", "argv"),
[
(
relogin_bilibili,
"bilibili",
"SESSDATA",
["relogin_bilibili.py"],
),
(relogin_douyin, "douyin", "sessionid", ["relogin_douyin.py"]),
(relogin_pgy, "pgy", "web_session", ["relogin_pgy.py"]),
(
relogin_xiaohongshu,
"xiaohongshu",
"web_session",
["relogin_xiaohongshu.py"],
),
(
relogin_xingtu,
"xingtu",
"sessionid",
["relogin_xingtu.py", "--force"],
),
],
)
def test_relogin_wrapper_validates_the_same_binding_written_by_child(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
module: ModuleType,
platform: str,
login_cookie: str,
argv: list[str],
) -> None:
cookie = tmp_path / "cookies.json"
storage = tmp_path / "storage_state.json"
profile = tmp_path / "profile"
monkeypatch.setattr(module, "COOKIE_FILE", cookie)
monkeypatch.setattr(module, "STATE_FILE", storage)
monkeypatch.setattr(module, "PROFILE_DIR", profile)
monkeypatch.setattr(module.sys, "argv", argv)
monkeypatch.setenv("GYXX_MODULE_ID", "content_marketing")
if module is relogin_xingtu:
monkeypatch.setattr(module, "terminate_profile_processes", lambda: None)
monkeypatch.setattr(module, "sync_to_douyin_scraper", lambda: None)
synced: list[tuple[str, Path, Path]] = []
def fake_sync(
synced_platform: str,
*,
cookie_file: Path,
storage_state_file: Path,
) -> tuple[Path, ...]:
synced.append(
(synced_platform, cookie_file.resolve(), storage_state_file.resolve())
)
return ()
monkeypatch.setattr(module, "sync_relogin_state_to_consumers", fake_sync)
def fake_call(
_command: list[str],
*,
cwd: str,
env: dict[str, str],
) -> int:
assert cwd == str(module.PROJECT_DIR)
assert env["GYXX_BROWSER_COOKIE_FILE"] == str(cookie.resolve())
assert env["GYXX_BROWSER_STORAGE_STATE_FILE"] == str(storage.resolve())
assert env["GYXX_BROWSER_PROFILE_DIR"] == str(profile.resolve())
assert "GYXX_MODULE_ID" not in env
cookie.write_text(
json.dumps([{"name": login_cookie, "value": "fresh"}]),
encoding="utf-8",
)
storage.write_text("{}", encoding="utf-8")
profile.mkdir()
return 0
monkeypatch.setattr(module.subprocess, "call", fake_call)
assert module.main() == 0
assert cookie.is_file()
assert storage.is_file()
assert profile.is_dir()
assert synced == [(platform, cookie.resolve(), storage.resolve())]
@pytest.mark.parametrize(
("module", "platform", "login_cookie", "cookie_domain", "argv"),
[
(
relogin_bilibili,
"bilibili",
"SESSDATA",
".bilibili.com",
["relogin_bilibili.py"],
),
(
relogin_douyin,
"douyin",
"sessionid",
".douyin.com",
["relogin_douyin.py"],
),
(
relogin_pgy,
"pgy",
"web_session",
".xiaohongshu.com",
["relogin_pgy.py"],
),
(
relogin_xiaohongshu,
"xiaohongshu",
"web_session",
".xiaohongshu.com",
["relogin_xiaohongshu.py"],
),
(
relogin_xingtu,
"xingtu",
"sessionid",
".xingtu.cn",
["relogin_xingtu.py", "--force"],
),
],
)
def test_relogin_wrapper_restores_previous_state_when_publish_fails(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
module: ModuleType,
platform: str,
login_cookie: str,
cookie_domain: str,
argv: list[str],
) -> None:
cookie = tmp_path / "cookies.json"
storage = tmp_path / "storage_state.json"
profile = tmp_path / "profile"
old_cookie = '[{"name":"old","value":"old-session"}]'
old_storage = '{"cookies":[{"name":"old"}]}'
cookie.write_text(old_cookie, encoding="utf-8")
storage.write_text(old_storage, encoding="utf-8")
profile.mkdir()
(profile / "old-state.txt").write_text("old-profile", encoding="utf-8")
monkeypatch.setattr(module, "COOKIE_FILE", cookie)
monkeypatch.setattr(module, "STATE_FILE", storage)
monkeypatch.setattr(module, "PROFILE_DIR", profile)
monkeypatch.setattr(module.sys, "argv", argv)
if module is relogin_xingtu:
monkeypatch.setattr(module, "terminate_profile_processes", lambda: None)
monkeypatch.setattr(module, "sync_to_douyin_scraper", lambda: None)
def fake_call(
_command: list[str],
*,
cwd: str,
env: dict[str, str],
) -> int:
assert cwd == str(module.PROJECT_DIR)
assert env["GYXX_BROWSER_COOKIE_FILE"] == str(cookie.resolve())
cookie.write_text(
json.dumps(
[
{
"name": login_cookie,
"value": "fresh-session",
"domain": cookie_domain,
}
]
),
encoding="utf-8",
)
storage.write_text('{"cookies":[],"origins":[]}', encoding="utf-8")
profile.mkdir()
return 0
def fail_publish(
synced_platform: str,
*,
cookie_file: Path,
storage_state_file: Path,
) -> tuple[Path, ...]:
assert synced_platform == platform
assert cookie_file == cookie
assert storage_state_file == storage
raise OSError("simulated consumer publish failure")
monkeypatch.setattr(module.subprocess, "call", fake_call)
monkeypatch.setattr(module, "sync_relogin_state_to_consumers", fail_publish)
assert module.main() == 1
assert cookie.read_text(encoding="utf-8") == old_cookie
assert storage.read_text(encoding="utf-8") == old_storage
assert (profile / "old-state.txt").read_text(encoding="utf-8") == (
"old-profile"
)
@pytest.mark.parametrize(
("platform", "cookie_name", "cookie_domain"),
[
("bilibili", "SESSDATA", ".bilibili.com"),
("douyin", "sessionid", ".douyin.com"),
("pgy", "web_session", ".xiaohongshu.com"),
("xiaohongshu", "web_session", ".xiaohongshu.com"),
("xingtu", "sessionid", ".xingtu.cn"),
],
)
def test_successful_relogin_atomically_updates_unique_consumer_state(
tmp_path: Path,
platform: str,
cookie_name: str,
cookie_domain: str,
) -> None:
data_root = tmp_path / "data"
source_cookie = tmp_path / "relogin" / "cookies.json"
source_storage = tmp_path / "relogin" / "storage_state.json"
source_cookie.parent.mkdir(parents=True)
fresh_cookie = [
{
"name": cookie_name,
"value": "fresh-session",
"domain": cookie_domain,
}
]
fresh_storage = {"cookies": fresh_cookie, "origins": []}
source_cookie.write_text(json.dumps(fresh_cookie), encoding="utf-8")
source_storage.write_text(json.dumps(fresh_storage), encoding="utf-8")
catalog = RuntimeIntegrationCatalog.load_default(
project_root=PROJECT_ROOT,
data_root=data_root,
)
consumer_id = CONSUMER_BINDING_IDS[platform][0]
consumer = catalog.binding_for(consumer_id)
consumer.cookie_file.parent.mkdir(parents=True, exist_ok=True)
consumer.cookie_file.write_text('[{"name":"old"}]', encoding="utf-8")
consumer.storage_state_file.write_text(
'{"cookies":[{"name":"old"}]}',
encoding="utf-8",
)
destinations = sync_relogin_state_to_consumers(
platform,
cookie_file=source_cookie,
storage_state_file=source_storage,
environment={
"GYXX_PROJECT_ROOT": str(PROJECT_ROOT),
"GYXX_DATA_ROOT": str(data_root),
},
)
assert source_cookie.resolve() != consumer.cookie_file.resolve()
assert source_storage.resolve() != consumer.storage_state_file.resolve()
assert destinations == (consumer.cookie_file.resolve(),)
assert json.loads(consumer.cookie_file.read_text(encoding="utf-8")) == (
fresh_cookie
)
assert json.loads(
consumer.storage_state_file.read_text(encoding="utf-8")
) == fresh_storage
def test_consumer_state_is_rolled_back_if_atomic_publish_fails(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
data_root = tmp_path / "data"
source_cookie = tmp_path / "relogin" / "cookies.json"
source_storage = tmp_path / "relogin" / "storage_state.json"
source_cookie.parent.mkdir(parents=True)
source_cookie.write_text(
json.dumps(
[
{
"name": "SESSDATA",
"value": "fresh-session",
"domain": ".bilibili.com",
}
]
),
encoding="utf-8",
)
source_storage.write_text('{"cookies":[],"origins":[]}', encoding="utf-8")
catalog = RuntimeIntegrationCatalog.load_default(
project_root=PROJECT_ROOT,
data_root=data_root,
)
consumer = catalog.binding_for(CONSUMER_BINDING_IDS["bilibili"][0])
consumer.cookie_file.parent.mkdir(parents=True, exist_ok=True)
old_cookie = '[{"name":"SESSDATA","value":"old"}]'
old_storage = '{"cookies":[{"name":"old"}]}'
consumer.cookie_file.write_text(old_cookie, encoding="utf-8")
consumer.storage_state_file.write_text(old_storage, encoding="utf-8")
original_replace = relogin_runtime_paths.os.replace
failed = False
def fail_second_replace(source: Path | str, target: Path | str) -> None:
nonlocal failed
source_path = Path(source)
target_path = Path(target)
if (
not failed
and source_path.suffix == ".tmp"
and target_path == consumer.storage_state_file
):
failed = True
raise OSError("simulated storage publish failure")
original_replace(source, target)
monkeypatch.setattr(relogin_runtime_paths.os, "replace", fail_second_replace)
with pytest.raises(OSError, match="simulated storage publish failure"):
sync_relogin_state_to_consumers(
"bilibili",
cookie_file=source_cookie,
storage_state_file=source_storage,
environment={
"GYXX_PROJECT_ROOT": str(PROJECT_ROOT),
"GYXX_DATA_ROOT": str(data_root),
},
)
assert consumer.cookie_file.read_text(encoding="utf-8") == old_cookie
assert consumer.storage_state_file.read_text(encoding="utf-8") == old_storage
@@ -466,6 +466,83 @@ def test_collect_xiaohongshu_rows_from_artifact(tmp_path, monkeypatch):
assert row[fields.index("page_metrics")].obj["笔记阅读数"] == "1.2万"
def test_upsert_rows_isolates_single_row_failure():
"""单行写入失败只跳过该行,不拖累整批(保存点隔离)。"""
import contextlib
class _Cursor:
def fetchone(self):
return (1,)
def __getattr__(self, name):
def record(*args, **kwargs):
params = args[1] if len(args) > 1 else kwargs.get("params")
if params is None or len(params) != len(sync_tool.ROW_FIELDS):
return
# 模拟毒行: 标题以 POISON 开头的行违反约束
title = params[sync_tool.ROW_FIELDS.index("title")]
if str(title).startswith("POISON"):
raise RuntimeError("value too long")
return record
class _Conn:
def __init__(self):
self.cursor_stub = _Cursor()
self.committed = False
@contextlib.contextmanager
def cursor(self):
yield self.cursor_stub
@contextlib.contextmanager
def transaction(self):
yield
def commit(self):
self.committed = True
metric_date = dt.date(2026, 10, 1)
rows = [
sync_tool.make_row("bilibili", metric_date, "daily.json",
account_name="光影行星", video_id="BV1"),
sync_tool.make_row("douyin", metric_date, "a.xlsx",
account_name="店铺A", video_id="1",
title="POISON 这行会失败"),
sync_tool.make_row("xiaohongshu", metric_date, "daily.json",
account_name="光影行星GYXX旗舰店", video_id="66e"),
]
conn = _Conn()
written, failed = sync_tool.upsert_rows(conn, rows)
assert (written, failed) == (2, 1)
assert conn.committed
def test_drop_stale_rows_only_removes_matching_sources():
"""陈旧防护按来源文件跳过: 抖音旧导出只清掉抖音行,不拦 B站/小红书。"""
stale_excel = Path("chanmama/旧导出_20260823-20260921.xlsx")
rows = {
("douyin", STORE_FLAGSHIP, "1", dt.date(2026, 10, 1)): sync_tool.make_row(
"douyin", dt.date(2026, 10, 1), stale_excel.name, video_id="1"
),
("bilibili", "光影行星", "BV1", dt.date(2026, 10, 1)): sync_tool.make_row(
"bilibili", dt.date(2026, 10, 1), "daily.json", video_id="BV1"
),
("xiaohongshu", XHS_SHOP, "2", dt.date(2026, 10, 1)): sync_tool.make_row(
"xiaohongshu", dt.date(2026, 10, 1), "daily.json", video_id="2"
),
}
dropped = sync_tool.drop_stale_rows(rows, [stale_excel])
assert dropped == 1
assert set(rows) == {
("bilibili", "光影行星", "BV1", dt.date(2026, 10, 1)),
("xiaohongshu", XHS_SHOP, "2", dt.date(2026, 10, 1)),
}
def test_douyin_history_backfill_uses_file_mtime_and_filename_account(tmp_path):
"""历史回填: 按导出落盘日期各留快照,旧版12列文件按文件名前缀补账号。"""
# 旧版 12 列表头(无 达人昵称/抖音号/蝉妈妈链接)
@@ -4,6 +4,7 @@ from datetime import date
from pathlib import Path
import collect_douyin_qianchuan_ads as collector
import pytest
def test_qianchuan_browser_uses_scrapling_stealth_and_persistent_state(
@@ -216,3 +217,174 @@ def test_download_task_from_page_downloads_completed_matching_row() -> None:
collector._matching_task_rows = original
assert actual is expected_download
def test_wait_task_center_ready_selects_advertiser_after_redirect() -> None:
clicked: list[str] = []
state = {"on_selection": True}
class _AccountCard:
def click(self) -> None:
clicked.append("account")
state["on_selection"] = False
class _FilteredLocator:
first = _AccountCard()
def count(self) -> int:
return 1
class _AccountListLocator:
def count(self) -> int:
return 1 if state["on_selection"] else 0
def filter(self, *, has_text: str):
assert has_text == collector.ADVERTISER_ID
return _FilteredLocator()
class _Page:
def locator(self, selector: str) -> _AccountListLocator:
assert selector == collector.ACCOUNT_ITEM_SELECTOR
return _AccountListLocator()
def wait_for_timeout(self, ms: int) -> None:
pass
def wait_for_function(self, expression: str, *, arg, timeout: int) -> None:
assert not state["on_selection"]
assert arg == collector.TASK_READY_MARKER
collector._wait_task_center_ready(_Page())
assert clicked == ["account"]
def test_wait_task_center_ready_raises_when_selection_page_persists() -> None:
class _AccountCard:
def click(self) -> None:
pass
class _FilteredLocator:
first = _AccountCard()
def count(self) -> int:
return 1
class _AccountListLocator:
def count(self) -> int:
return 1
def filter(self, *, has_text: str):
assert has_text == collector.ADVERTISER_ID
return _FilteredLocator()
class _Page:
def locator(self, selector: str) -> _AccountListLocator:
assert selector == collector.ACCOUNT_ITEM_SELECTOR
return _AccountListLocator()
def wait_for_timeout(self, ms: int) -> None:
pass
def wait_for_function(self, expression: str, *, arg, timeout: int) -> None:
raise AssertionError("不应在账户选择页上等待任务中心标记")
with pytest.raises(
collector.DouyinQianchuanDownloadError, match="仍停留在账户选择页"
):
collector._wait_task_center_ready(_Page())
def test_wait_task_center_ready_raises_when_never_ready(monkeypatch) -> None:
class _AccountListLocator:
def count(self) -> int:
return 0
class _Page:
def locator(self, selector: str) -> _AccountListLocator:
assert selector == collector.ACCOUNT_ITEM_SELECTOR
return _AccountListLocator()
def wait_for_function(self, expression: str, *, arg, timeout: int) -> None:
raise RuntimeError("Page.wait_for_function: Timeout 15000ms exceeded.")
def goto(self, url: str, **options: object) -> None:
assert url == collector.TASK_CENTER_PAGE_URL
monkeypatch.setattr(collector, "is_browser_timeout_error", lambda exc: True)
with pytest.raises(
collector.DouyinQianchuanDownloadError, match="未达到可用状态"
):
collector._wait_task_center_ready(_Page(), timeout_ms=50)
def test_download_task_from_page_recovers_from_context_destroyed(
monkeypatch,
) -> None:
expected_download = object()
state = {"count_calls": 0}
class _DownloadInfo:
value = expected_download
def __enter__(self):
return self
def __exit__(self, exc_type, exc, traceback) -> None:
pass
class _DownloadButton:
def click(self) -> None:
pass
class _Row:
def inner_text(self) -> str:
return "全域推广数据-推商品\n生成完成\n下载"
def get_by_text(self, text: str, *, exact: bool) -> _DownloadButton:
assert (text, exact) == ("下载", True)
return _DownloadButton()
class _Rows:
first = _Row()
def count(self) -> int:
state["count_calls"] += 1
if state["count_calls"] == 1:
raise RuntimeError(
"Locator.count: Execution context was destroyed, "
"most likely because of a navigation."
)
return 1
class _Body:
def inner_text(self, *, timeout: int) -> str:
assert timeout == 10_000
return collector.TASK_READY_MARKER
class _Page:
url = "https://qianchuan.jinritemai.com/tools/task-center/download-task"
def expect_download(self, *, timeout: int) -> _DownloadInfo:
assert timeout == 60_000
return _DownloadInfo()
def locator(self, selector: str) -> _Body:
assert selector == "body"
return _Body()
monkeypatch.setattr(collector, "_matching_task_rows", lambda *args: _Rows())
monkeypatch.setattr(
collector, "_wait_task_center_ready", lambda page, **kwargs: None
)
actual = collector._download_task_from_page(
_Page(),
date(2026, 9, 29),
date(2026, 9, 29),
wait_timeout_ms=0,
)
assert actual is expected_download
assert state["count_calls"] == 2
+5 -39
View File
@@ -10,9 +10,6 @@ from types import SimpleNamespace
import pytest
from gyxx_flow.adapters import COOKIE_SKIP_EXIT_CODE
from gyxx_flow.modules.content_marketing.data.tools import (
friday_relogin_parallel as weekly_relogin,
)
@pytest.fixture
@@ -75,29 +72,6 @@ def _import_supply_script(module_name: str):
sys.stderr = stderr
def test_weekly_relogin_skips_the_entire_qr_workflow(
acceptance_environment: Path,
monkeypatch: pytest.MonkeyPatch,
capsys: pytest.CaptureFixture[str],
) -> None:
monkeypatch.setattr(
weekly_relogin.subprocess,
"Popen",
_forbidden("acceptance mode must not launch relogin subprocesses"),
)
monkeypatch.setattr(
weekly_relogin,
"_send_lark",
_forbidden("acceptance mode must not send QR screenshots"),
)
assert weekly_relogin.main() == COOKIE_SKIP_EXIT_CODE
assert "SKIPPED_COOKIE" in capsys.readouterr().out
assert "content.relogin.weekly" in acceptance_environment.read_text(
encoding="utf-8"
)
@pytest.mark.parametrize(
"module_name",
[
@@ -106,25 +80,17 @@ def test_weekly_relogin_skips_the_entire_qr_workflow(
"batch_rescrape_bilibili",
],
)
def test_batch_rescrape_relogin_never_starts_a_qr_process(
def test_batch_rescrape_has_no_relogin_capability(
module_name: str,
acceptance_environment: Path,
monkeypatch: pytest.MonkeyPatch,
capsys: pytest.CaptureFixture[str],
) -> None:
"""扫码重登能力已从批量补采工具中移除,登录统一由账号 vault 管理。"""
module = importlib.import_module(
f"gyxx_flow.modules.content_marketing.data.tools.{module_name}"
)
monkeypatch.setattr(
module.subprocess,
"call",
_forbidden("acceptance mode must not invoke a relogin script"),
)
assert module.do_relogin() is False
assert "SKIPPED_COOKIE" in capsys.readouterr().out
evidence = acceptance_environment.read_text(encoding="utf-8")
assert f"content.{module_name}.relogin" in evidence
assert not hasattr(module, "do_relogin")
assert not hasattr(module, "RELOGIN_SCRIPT")
@pytest.mark.parametrize(
+1 -1
View File
@@ -56,7 +56,6 @@ def test_catalog_composition_registers_all_workflows() -> None:
"content.metrics.self_operated.daily",
"content.metrics.backfill",
"content.marketing_report.daily",
"content.relogin.weekly",
"content.creator_report.monthly",
"content.summary.monthly",
"content.notes_master.daily",
@@ -65,6 +64,7 @@ def test_catalog_composition_registers_all_workflows() -> None:
"product.persona.daily",
"product.daily",
"product.online_stock_alert.daily",
"product.hk_anpei_public_inventory.daily",
"product.alert.daily",
"product.ecommerce_costs.daily",
"product.style_analysis.interval",
+4 -6
View File
@@ -18,8 +18,8 @@ def test_catalog_maps_39_project_scheduled_tasks() -> None:
assert len(scheduled) == 39
assert len({item.source_task_name for item in scheduled}) == 39
assert Counter(item.module for item in scheduled) == {
"content_marketing": 10,
"product_commerce": 13,
"content_marketing": 9,
"product_commerce": 14,
"shop_intelligence": 12,
"supply_chain": 4,
}
@@ -80,7 +80,6 @@ def test_jd_video_upload_is_an_idempotent_scheduled_workflow() -> None:
assert step.args == (
"--anchor-record-id",
"recvrXTRWWFXeJ",
"--manual-login",
"--execute",
)
assert step.replay_policy == "idempotent"
@@ -192,7 +191,7 @@ def test_every_scheduled_workflow_has_one_valid_asia_shanghai_schedule() -> None
assert schedule.workflow_id == workflow.workflow_id
assert schedule.kind in {"daily", "weekly", "monthly", "interval_days"}
assert schedule.at.count(":") == 1
assert sum(schedule.enabled for schedule in catalog.schedules) == 27
assert sum(schedule.enabled for schedule in catalog.schedules) == 28
customer_service = catalog.schedule_for("shop.jd_customer_service.daily")
assert customer_service.at == "03:00"
assert customer_service.enabled is True
@@ -265,8 +264,7 @@ def test_scheduled_workflows_use_python_entries_and_content_tasks_are_graphs() -
"data/tools/sync_self_video_metrics.py"
)
assert catalog.schedule_for(self_operated.workflow_id).effective_times == (
"01:00",
"05:00",
"06:00",
)
backfill = next(
+1 -1
View File
@@ -54,7 +54,7 @@ def test_default_cli_registry_exposes_all_project_scheduled_workflows(
scheduled_ids = {
entry.workflow_id for entry in registry.catalog.scheduled_workflows()
}
assert len(scheduled_ids) == 35
assert len(scheduled_ids) == 39
assert scheduled_ids == {
schedule.workflow_id for schedule in registry.catalog.schedules
}
+93 -19
View File
@@ -2,6 +2,7 @@ from __future__ import annotations
import http.client
import json
import os
import re
import shutil
import subprocess
@@ -311,7 +312,7 @@ def test_notification_config_persists_app_scoped_multi_recipient_routes(
)
initial = service.notification_config()
assert len(initial["people"]) == 16
assert len(initial["people"]) == 18
assert initial["available_app_profiles"] == ["hermes-analyzer"]
assert [app["profile"] for app in initial["apps"]] == ["hermes-analyzer"]
wang = next(person for person in initial["people"] if person["id"] == "wang_yunlong")
@@ -321,7 +322,12 @@ def test_notification_config_persists_app_scoped_multi_recipient_routes(
assert wang["bindings"]["hermes-analyzer"]["open_id"] == (
"ou_7ad5fc8012e2f741afc5346e05ffd447"
)
assert all(route["configured"] is False for route in initial["routes"])
stock_alert_route = next(
route
for route in initial["routes"]
if route["workflow_id"] == "product.online_stock_alert.daily"
)
assert stock_alert_route["configured"] is True
routes = [dict(route) for route in initial["routes"]]
daily = next(
@@ -1566,7 +1572,7 @@ def test_overview_exposes_only_verified_scheduled_runs_as_cancellable(
"schema_version": 1,
"updated_at": "2026-08-14T02:00:00+00:00",
"jobs": {
"content.relogin.weekly": {
"content.notes_master.daily": {
"status": "running",
"pid": 4310,
"business_date": "2026-08-14",
@@ -1589,7 +1595,7 @@ def test_overview_exposes_only_verified_scheduled_runs_as_cancellable(
workflow_id: str,
_job: dict[str, Any],
) -> tuple[bool, Any, float | None]:
if workflow_id == "content.relogin.weekly":
if workflow_id == "content.notes_master.daily":
return True, VerifiedProcess(), 1_800_000_000.0
return False, None, None
@@ -1600,7 +1606,7 @@ def test_overview_exposes_only_verified_scheduled_runs_as_cancellable(
).overview()
workflows = {item["id"]: item for item in overview["workflows"]}
verified = workflows["content.relogin.weekly"]["scheduler_job"]
verified = workflows["content.notes_master.daily"]["scheduler_job"]
assert verified["status"] == "running"
assert verified["cancellable"] is True
assert "pid" not in verified
@@ -1614,7 +1620,7 @@ def test_service_cancels_verified_scheduled_run_and_finalizes_journal(
monkeypatch: pytest.MonkeyPatch,
console_settings: Settings,
) -> None:
workflow_id = "content.relogin.weekly"
workflow_id = "content.notes_master.daily"
_write_scheduler_state(
console_settings,
{
@@ -1906,24 +1912,13 @@ def test_content_production_preflight_rejects_non_production_modes(
relogin = next(
item
for item in catalog.workflows
if item.workflow_id == "content.relogin.weekly"
if item.workflow_id == "content.summary.weekly"
)
with pytest.raises(ConsoleRequestError, match=message):
console_module._validate_content_production_runtime(relogin, {flag: "1"})
def test_content_relogin_preflight_does_not_require_postgres() -> None:
catalog = WorkflowCatalog.load(PROJECT_ROOT / "config")
relogin = next(
item
for item in catalog.workflows
if item.workflow_id == "content.relogin.weekly"
)
console_module._validate_content_production_runtime(relogin, {})
@pytest.mark.parametrize(
"workflow_id",
["content.notes_master.daily", "content.metrics.daily"],
@@ -3297,7 +3292,7 @@ def test_http_notification_config_requires_revision_and_supports_multi_recipient
assert status == HTTPStatus.OK
_assert_security_headers(response_headers)
config = json.loads(body)
assert len(config["people"]) == 16
assert len(config["people"]) == 18
assert {
route["workflow_id"] for route in config["routes"]
} == set(console_module.NOTIFICATION_CAPABILITIES)
@@ -3616,3 +3611,82 @@ def test_douyin_negative_reviews_range_uses_single_run_with_env_dates(
{"business_date": "2026-09-19"},
)
assert "business_date_end" not in single
def test_api_health_reports_process_and_scheduler_state(
console_settings: Settings,
) -> None:
with _running_server(console_settings, FakeConsoleLauncher()) as server:
status, _headers, content = _http_request(server, "GET", "/api/health")
assert status == HTTPStatus.OK
payload = json.loads(content)
assert payload["ok"] is True
assert payload["pid"] == os.getpid()
assert payload["uptime_seconds"] >= 0
# 测试环境没有调度器实例锁,health 只报状态不做任何拉起。
assert payload["scheduler_running"] is False
assert payload["scheduler_pid"] is None
def test_api_health_requires_no_token(console_settings: Settings) -> None:
# 自检探针端点供进程内看门狗与保活脚本使用,必须绕过访问令牌。
with _running_server(
console_settings, FakeConsoleLauncher(), token="x" * 24
) as server:
status, _headers, _content = _http_request(server, "GET", "/api/health")
assert status == HTTPStatus.OK
def test_probe_console_health_against_real_server(
console_settings: Settings,
) -> None:
with _running_server(console_settings, FakeConsoleLauncher()) as server:
assert console_module._probe_console_health(server.server_port, 2.0) is True
assert console_module._probe_console_health(1, 0.2) is False
def test_console_watchdog_exits_after_consecutive_probe_failures() -> None:
stopped = threading.Event()
exits: list[int] = []
probes: list[int] = []
def always_failing(port: int, _timeout: float) -> bool:
probes.append(port)
return False
console_module._console_watchdog(
8765,
interval_seconds=0.01,
failure_limit=3,
probe_timeout_seconds=0.01,
stopped=stopped,
probe=always_failing,
exit_process=exits.append,
sleep=lambda _seconds: None,
)
assert exits == [console_module.CONSOLE_WATCHDOG_EXIT_CODE]
assert probes == [8765, 8765, 8765]
def test_console_watchdog_resets_failure_count_after_successful_probe() -> None:
stopped = threading.Event()
exits: list[int] = []
results = iter([False, False, True, False, False, False])
def flaky_probe(_port: int, _timeout: float) -> bool:
return next(results)
console_module._console_watchdog(
8765,
interval_seconds=0.01,
failure_limit=3,
probe_timeout_seconds=0.01,
stopped=stopped,
probe=flaky_probe,
exit_process=exits.append,
sleep=lambda _seconds: None,
)
# 成功一次清零计数:2 次失败 + 1 次成功 + 3 次失败后才退出。
assert exits == [console_module.CONSOLE_WATCHDOG_EXIT_CODE]
+1 -1
View File
@@ -23,7 +23,7 @@ def test_content_module_registers_only_scheduled_entries() -> None:
definitions = module.workflow_definitions()
assert tuple(item.workflow_id for item in definitions) == CONTENT_WORKFLOW_IDS
assert len(CONTENT_SCHEDULED_WORKFLOW_IDS) == 10
assert len(CONTENT_SCHEDULED_WORKFLOW_IDS) == 9
def test_content_workflows_use_project_owned_module_commands() -> None:
+1 -2
View File
@@ -44,7 +44,6 @@ REQUIRED_SOURCES = {
"chanmama_scraper.py",
"daily_creator_exposure_scope.py",
"daily_marketing_report.py",
"data/tools/friday_relogin_parallel.py",
"data/tools/schema_gyxx_super_data.sql",
"data/tools/migrations/005_style_product_profile.sql",
"data/config/db.env.example",
@@ -71,7 +70,7 @@ def test_content_source_snapshot_manifest_is_complete_and_well_formed() -> None:
assert manifest["module"] == "content_marketing"
files = manifest["files"]
assert isinstance(files, list)
assert len(files) == 66
assert len(files) == 59
sources = [entry["source_relative_path"] for entry in files]
targets = [entry["target_relative_path"] for entry in files]
+771
View File
@@ -0,0 +1,771 @@
"""Cookie gate: per-account login-state machine injected before workflows."""
from __future__ import annotations
import json
from datetime import datetime, timedelta, timezone
from pathlib import Path
from types import SimpleNamespace
from gyxx_flow.adapters.browser import BrowserCookieStore
from gyxx_flow.adapters.integration import AccountBinding
from gyxx_flow.catalog import WorkflowCatalog, WorkflowEntry
from gyxx_flow.cookie_gate import (
GATE_MODE_BLOCKING,
GATE_MODE_DEFER,
GATE_MODE_RECOVER,
CookieGate,
CookieGateRecoveryManager,
apply_cookie_gates,
gate_enabled,
)
from gyxx_flow.core.config import Settings
from gyxx_flow.core.context import RunContext
from gyxx_flow.core.layout import DataLayout
from gyxx_flow.core.locks import LockManager
from gyxx_flow.core.records import RunJournal
from gyxx_flow.workflow.engine import WorkflowEngine
from gyxx_flow.workflow.model import StepDefinition, WorkflowDefinition
from gyxx_flow.workflow.steps import StepExecution
_NOW = datetime(2026, 10, 8, 12, 0, 0, tzinfo=timezone.utc)
def _settings(tmp_path: Path) -> Settings:
return Settings(project_root=tmp_path, data_root=tmp_path / "var")
def _account(tmp_path: Path, account_id: str, **overrides: object) -> AccountBinding:
vault = tmp_path / "state" / "accounts" / account_id
values: dict[str, object] = {
"account_id": account_id,
"cdp_port": 22000,
"cdp_url": "http://127.0.0.1:22000",
"profile_dir": vault / "profile",
"cookie_file": vault / "cookies.json",
"storage_state_file": vault / "storage_state.json",
"required_cookie_domains": ("example.com",),
"required_cookie_names": ("sessionid",),
"login_identity_markers": ("示例店铺",),
"keepalive_enabled": True,
"keepalive_url": "https://example.com/home",
"keepalive_interval_seconds": 900,
}
values.update(overrides)
return AccountBinding(**values) # type: ignore[arg-type]
def _fake_catalog(*accounts: AccountBinding, bindings: dict[str, str] | None = None):
by_id = {account.account_id: account for account in accounts}
binding_accounts = bindings or {}
account_tuple = tuple(accounts)
class _Catalog:
accounts = account_tuple
def account_for(self, account_id: str) -> AccountBinding:
return by_id[account_id]
def binding_for(self, binding_id: str):
return SimpleNamespace(account=binding_accounts.get(binding_id, ""))
return _Catalog()
def _write_valid_vault(account: AccountBinding) -> None:
store = BrowserCookieStore(account.cookie_file, account.storage_state_file)
store.save_cookies(
[
{
"name": "sessionid",
"value": "live-session",
"domain": ".example.com",
"path": "/",
"expires": _NOW.timestamp() + 86_400,
}
]
)
def _write_keepalive_status(
account: AccountBinding,
payload: dict[str, object] | None,
) -> None:
path = account.cookie_file.with_name("keepalive_status.json")
if payload is None:
path.unlink(missing_ok=True)
return
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(payload), encoding="utf-8")
def _gate(
tmp_path: Path,
*accounts: AccountBinding,
probe: dict[str, object] | None = None,
environment: dict[str, str] | None = None,
sent: list[dict[str, object]] | None = None,
bindings: dict[str, str] | None = None,
) -> CookieGate:
probes = probe or {"status": "expired"}
keepalive = SimpleNamespace(
refresh_account=lambda account_id: dict(probes, account_id=account_id)
)
sender_calls = sent if sent is not None else []
def _sender(**kwargs: object) -> dict[str, object]:
sender_calls.append(dict(kwargs))
return {"message_id": "om_fake"}
catalog = _fake_catalog(*accounts, bindings=bindings)
gate = CookieGate(
_settings(tmp_path),
catalog=catalog,
keepalive=keepalive, # type: ignore[arg-type]
environment=environment or {},
now=lambda: _NOW,
sender=_sender,
)
gate._workflow_catalog = WorkflowCatalog(timezone="UTC", workflows=(), schedules=())
return gate
def _context(**overrides: object) -> RunContext:
values: dict[str, object] = {
"workflow_id": "shop.demo.daily",
"business_date": "2026-10-08",
"shadow": False,
"now": _NOW,
"random_suffix": "aaaa",
}
values.update(overrides)
return RunContext.create(**values) # type: ignore[arg-type]
def _noop_outcome(*, context, timeout_seconds, dry_run) -> StepExecution: # noqa: ANN001
return StepExecution(exit_code=0)
def _deny_outcome(*, context, timeout_seconds, dry_run) -> StepExecution: # noqa: ANN001
raise AssertionError("resolution must not run actions")
class _PassAction:
execute = staticmethod(_noop_outcome)
class _BoundAction:
"""Duck-typed DeferredModuleCommandStep for account resolution tests."""
def __init__(self, command_entry: str = "collect.py") -> None:
self.entry = SimpleNamespace(module="shop_intelligence", entry="fallback.py")
self.command_entry = command_entry
self.execute = _deny_outcome
class _RecordingAction:
def __init__(self, name: str, sink: list[str], command_entry: str | None = None) -> None:
def run(*, context, timeout_seconds, dry_run) -> StepExecution: # noqa: ANN001
sink.append(name)
return StepExecution(exit_code=0)
self.execute = run
if command_entry is not None:
self.entry = SimpleNamespace(
module="shop_intelligence", entry="fallback.py"
)
self.command_entry = command_entry
# ----------------------------------------------------------------------
# Mode classification
# ----------------------------------------------------------------------
def test_gate_mode_classifies_accounts(tmp_path: Path) -> None:
gate = _gate(
tmp_path,
_account(tmp_path, "qr-only"),
_account(tmp_path, "cred-scripts", credential_env_names=("A_USER", "A_PASS")),
_account(
tmp_path,
"cred-vault",
login_mode="C",
login_command_id="product_commerce:erp_login.py",
),
_account(tmp_path, "cred-mode-b", login_mode="B"),
)
assert gate.gate_mode("qr-only") == GATE_MODE_BLOCKING
assert gate.gate_mode("cred-scripts") == GATE_MODE_DEFER
assert gate.gate_mode("cred-vault") == GATE_MODE_RECOVER
assert gate.gate_mode("cred-mode-b") == GATE_MODE_DEFER
# ----------------------------------------------------------------------
# Workflow account resolution
# ----------------------------------------------------------------------
def _bound_workflow(workflow_id: str = "shop.demo.daily") -> WorkflowDefinition:
return WorkflowDefinition(
workflow_id,
(
StepDefinition("collect", _BoundAction("collect.py")),
StepDefinition("sync", _BoundAction("sync.py"), depends_on=("collect",)),
),
)
def test_collect_accounts_maps_bindings_and_skips_defer(tmp_path: Path) -> None:
gate = _gate(
tmp_path,
_account(tmp_path, "douyin-shop"),
_account(tmp_path, "jd-shop", credential_env_names=("JD_SHOP", "JD_PASSWORD")),
bindings={
"shop_intelligence:collect.py": "douyin-shop",
"shop_intelligence:other.py": "jd-shop",
},
)
mapping = gate.collect_workflow_accounts(_bound_workflow())
# sync.py 无账号绑定,jd-shop 属 defer 模式:两者都不进映射
assert mapping == {"douyin-shop": {"collect"}}
def test_collect_accounts_includes_declared_required_accounts(tmp_path: Path) -> None:
gate = _gate(tmp_path, _account(tmp_path, "douyin-shop"))
gate._workflow_catalog = WorkflowCatalog(
timezone="UTC",
workflows=(
WorkflowEntry(
workflow_id="shop.demo.daily",
module="shop_intelligence",
trigger="manual",
entry="collect.py",
required_accounts=("douyin-shop",),
),
),
schedules=(),
)
mapping = gate.collect_workflow_accounts(_bound_workflow())
assert mapping == {"douyin-shop": {"collect", "sync"}}
def test_apply_cookie_gates_rewrites_dependencies(tmp_path: Path) -> None:
account = _account(tmp_path, "douyin-shop")
gate = _gate(
tmp_path,
account,
bindings={"shop_intelligence:collect.py": "douyin-shop"},
)
gated = apply_cookie_gates(
_bound_workflow(), settings=_settings(tmp_path), gate=gate
)
ids = [step.step_id for step in gated.steps]
assert "cookie-gate.douyin-shop" in ids
collect = next(step for step in gated.steps if step.step_id == "collect")
sync = next(step for step in gated.steps if step.step_id == "sync")
assert collect.depends_on == ("cookie-gate.douyin-shop",)
assert sync.depends_on == ("collect",)
assert gated.ordered_steps() # patched DAG stays valid
def test_apply_cookie_gates_returns_workflow_without_accounts(tmp_path: Path) -> None:
gate = _gate(tmp_path, _account(tmp_path, "douyin-shop"))
workflow = _bound_workflow()
assert (
apply_cookie_gates(workflow, settings=_settings(tmp_path), gate=gate) is workflow
)
def test_gate_enabled_env_kill_switch() -> None:
assert gate_enabled({})
assert not gate_enabled({"GYXX_COOKIE_GATE": "0"})
assert not gate_enabled({"GYXX_COOKIE_GATE": "off"})
assert gate_enabled({"GYXX_COOKIE_GATE": "1"})
# ----------------------------------------------------------------------
# State machine: check()
# ----------------------------------------------------------------------
def test_check_fast_path_accepts_fresh_keepalive_without_probe(tmp_path: Path) -> None:
account = _account(tmp_path, "douyin-shop")
gate = _gate(tmp_path, account, probe={"status": "expired"})
_write_valid_vault(account)
_write_keepalive_status(
account,
{
"status": "success",
"last_success_at": (_NOW - timedelta(minutes=10)).isoformat(),
},
)
verdict = gate.check("douyin-shop", context=_context())
assert not verdict.blocked
assert "keepalive" in verdict.message
state = json.loads(
(account.cookie_file.parent / "cookie_state.json").read_text(encoding="utf-8")
)
assert state["state"] == "healthy"
def test_check_stale_status_triggers_probe(tmp_path: Path) -> None:
account = _account(tmp_path, "douyin-shop")
gate = _gate(tmp_path, account, probe={"status": "success"})
_write_valid_vault(account)
_write_keepalive_status(
account,
{
"status": "success",
"last_success_at": (_NOW - timedelta(hours=6)).isoformat(),
},
)
verdict = gate.check("douyin-shop", context=_context())
assert not verdict.blocked
assert verdict.message.endswith("探活通过)")
def test_check_expired_blocks_and_sends_feishu_alert(tmp_path: Path) -> None:
account = _account(tmp_path, "douyin-shop")
sent: list[dict[str, object]] = []
gate = _gate(
tmp_path,
account,
probe={"status": "expired"},
environment={"GYXX_LOGIN_ALERT_OPEN_IDS": "ou_alert_target"},
sent=sent,
)
_write_keepalive_status(account, None)
verdict = gate.check("douyin-shop", context=_context())
assert verdict.blocked
assert verdict.notified
assert len(sent) == 1
text = str(sent[0]["text"])
assert "uv run gyxx accounts login douyin-shop" in text
assert "shop.demo.daily" in text
assert sent[0]["profile"] == "hermes-analyzer"
state = json.loads(
(account.cookie_file.parent / "cookie_state.json").read_text(encoding="utf-8")
)
assert state["state"] == "awaiting_login"
assert state["last_notified_at"] == _NOW.isoformat(timespec="seconds")
def test_check_notification_is_throttled_per_interval(tmp_path: Path) -> None:
account = _account(tmp_path, "douyin-shop")
sent: list[dict[str, object]] = []
gate = _gate(
tmp_path,
account,
probe={"status": "expired"},
environment={"GYXX_LOGIN_ALERT_OPEN_IDS": "ou_alert_target"},
sent=sent,
)
_write_keepalive_status(account, None)
first = gate.check("douyin-shop", context=_context())
second = gate.check("douyin-shop", context=_context())
assert first.notified and second.blocked
assert not second.notified
assert len(sent) == 1
def test_check_shadow_run_never_notifies(tmp_path: Path) -> None:
account = _account(tmp_path, "douyin-shop")
sent: list[dict[str, object]] = []
gate = _gate(
tmp_path,
account,
probe={"status": "expired"},
environment={"GYXX_LOGIN_ALERT_OPEN_IDS": "ou_alert_target"},
sent=sent,
)
verdict = gate.check("douyin-shop", context=_context(shadow=True))
assert verdict.blocked
assert not verdict.notified
assert sent == []
def test_check_mode_c_recovery_success_and_failure_paths(tmp_path: Path) -> None:
account = _account(
tmp_path,
"erp-jushuitan",
login_mode="C",
login_command_id="product_commerce:erp_login.py",
)
recovered = _gate(
tmp_path, account, probe={"status": "success", "login_recovered": True}
)
_write_keepalive_status(account, None)
verdict = recovered.check("erp-jushuitan", context=_context())
assert not verdict.blocked
assert verdict.recovered
assert "自动账密重登成功" in verdict.message
sent: list[dict[str, object]] = []
failed = _gate(
tmp_path,
account,
probe={"status": "not_logged_in"},
environment={"GYXX_LOGIN_ALERT_OPEN_IDS": "ou_alert_target"},
sent=sent,
)
_write_keepalive_status(
failed.catalog.account_for("erp-jushuitan"),
{
"status": "not_logged_in",
"login_attempted": True,
"login_error_type": "TimeoutExpired",
},
)
verdict = failed.check("erp-jushuitan", context=_context())
assert verdict.blocked
assert verdict.notified
assert "自动账密重登已尝试但未成功" in str(sent[0]["text"])
def test_check_busy_falls_back_to_static_evidence(tmp_path: Path) -> None:
account = _account(tmp_path, "douyin-shop")
gate = _gate(tmp_path, account, probe={"status": "busy"})
_write_valid_vault(account)
_write_keepalive_status(account, None)
verdict = gate.check("douyin-shop", context=_context())
assert not verdict.blocked
assert "探活锁忙" in verdict.message
def test_check_probe_error_with_static_valid_vault_passes(tmp_path: Path) -> None:
account = _account(tmp_path, "douyin-shop")
gate = _gate(tmp_path, account, probe={"status": "timeout"})
_write_valid_vault(account)
_write_keepalive_status(account, None)
verdict = gate.check("douyin-shop", context=_context())
assert not verdict.blocked
assert "探活异常" in verdict.message
def test_acceptance_mode_is_static_only_and_never_alerts(tmp_path: Path) -> None:
account = _account(tmp_path, "douyin-shop")
sent: list[dict[str, object]] = []
gate = _gate(
tmp_path,
account,
probe={"status": "expired"},
environment={
"GYXX_WORKFLOW_ACCEPTANCE": "1",
"GYXX_LOGIN_ALERT_OPEN_IDS": "ou_alert_target",
},
sent=sent,
)
_write_keepalive_status(account, None)
verdict = gate.check("douyin-shop", context=_context())
assert verdict.blocked
assert sent == []
assert "验收模式" in verdict.message
def test_alert_recipient_falls_back_to_constant(tmp_path: Path) -> None:
account = _account(tmp_path, "douyin-shop")
gate = _gate(tmp_path, account)
recipients = gate._alert_recipients()
assert recipients == [
("ou_7ad5fc8012e2f741afc5346e05ffd447", "hermes-analyzer")
]
# ----------------------------------------------------------------------
# Engine integration
# ----------------------------------------------------------------------
def _engine(tmp_path: Path) -> WorkflowEngine:
return WorkflowEngine(LockManager(tmp_path / "state" / "locks"))
def test_engine_blocks_dependents_when_gate_fails(tmp_path: Path) -> None:
account = _account(tmp_path, "douyin-shop")
sent: list[dict[str, object]] = []
gate = _gate(
tmp_path,
account,
probe={"status": "expired"},
environment={"GYXX_LOGIN_ALERT_OPEN_IDS": "ou_alert_target"},
sent=sent,
bindings={"shop_intelligence:collect.py": "douyin-shop"},
)
executed: list[str] = []
workflow = WorkflowDefinition(
"shop.demo.daily",
(
StepDefinition(
"collect", _RecordingAction("collect", executed, "collect.py")
),
StepDefinition(
"sync", _RecordingAction("sync", executed), depends_on=("collect",)
),
),
)
gated = apply_cookie_gates(workflow, settings=_settings(tmp_path), gate=gate)
context = RunContext.create(
"shop.demo.daily", "2026-10-08", now=_NOW, random_suffix="dddd"
)
journal = RunJournal.create(DataLayout(_settings(tmp_path).data_root), context)
engine = _engine(tmp_path)
run_workflow = engine.execute
result = run_workflow(gated, context=context, journal=journal)
assert result.status == "failed"
assert result.exit_code == 1
assert result.steps["cookie-gate.douyin-shop"].status == "failed"
assert result.steps["collect"].reason == "dependency-failed"
assert result.steps["sync"].reason == "dependency-failed"
assert executed == []
assert len(sent) == 1
def test_engine_runs_collect_steps_when_gate_passes(tmp_path: Path) -> None:
account = _account(tmp_path, "douyin-shop")
gate = _gate(
tmp_path,
account,
probe={"status": "success"},
bindings={"shop_intelligence:collect.py": "douyin-shop"},
)
_write_valid_vault(account)
_write_keepalive_status(account, None)
executed: list[str] = []
workflow = WorkflowDefinition(
"shop.demo.daily",
(
StepDefinition(
"collect", _RecordingAction("collect", executed, "collect.py")
),
),
)
gated = apply_cookie_gates(workflow, settings=_settings(tmp_path), gate=gate)
context = RunContext.create(
"shop.demo.daily", "2026-10-08", now=_NOW, random_suffix="eeee"
)
journal = RunJournal.create(DataLayout(_settings(tmp_path).data_root), context)
engine = _engine(tmp_path)
run_workflow = engine.execute
result = run_workflow(gated, context=context, journal=journal)
assert result.status == "success"
assert result.steps["cookie-gate.douyin-shop"].status == "success"
assert result.steps["collect"].status == "success"
# ----------------------------------------------------------------------
# Recovery: automatic rerun after the login recovers
# ----------------------------------------------------------------------
class _FakeLauncher:
def __init__(self, fail: bool = False) -> None:
self.launched: list[tuple[str, str]] = []
self.closed: list[int] = []
self.fail = fail
def launch(self, workflow_id, slot, business_date): # noqa: ANN001
if self.fail:
raise RuntimeError("spawn denied")
process = SimpleNamespace(pid=9000 + len(self.launched), poll=lambda: None)
self.launched.append((workflow_id, business_date.isoformat()))
return process
def close(self, process) -> None: # noqa: ANN001
self.closed.append(process.pid)
def _state_file(account: AccountBinding) -> Path:
return account.cookie_file.parent / "cookie_state.json"
def _read_state(account: AccountBinding) -> dict:
return json.loads(_state_file(account).read_text(encoding="utf-8"))
def _recovery_workflow_catalog() -> WorkflowCatalog:
return WorkflowCatalog(
timezone="UTC",
workflows=(
WorkflowEntry(
workflow_id="shop.demo.daily",
module="shop_intelligence",
trigger="scheduled",
entry="collect.py",
),
),
schedules=(),
)
def _blocking_gate(
tmp_path: Path,
account: AccountBinding,
sent: list[dict[str, object]],
) -> CookieGate:
gate = _gate(
tmp_path,
account,
probe={"status": "expired"},
environment={"GYXX_LOGIN_ALERT_OPEN_IDS": "ou_alert_target"},
sent=sent,
)
return gate
def _healthy_gate(
tmp_path: Path,
account: AccountBinding,
sent: list[dict[str, object]],
) -> CookieGate:
gate = _gate(
tmp_path,
account,
probe={"status": "success"},
environment={"GYXX_LOGIN_ALERT_OPEN_IDS": "ou_alert_target"},
sent=sent,
)
gate._workflow_catalog = _recovery_workflow_catalog()
return gate
def _simulate_manual_login(account: AccountBinding) -> None:
_write_valid_vault(account)
_write_keepalive_status(
account,
{
"status": "success",
"last_success_at": (_NOW - timedelta(seconds=60)).isoformat(),
},
)
def test_expire_records_pending_reruns(tmp_path: Path) -> None:
sent: list[dict[str, object]] = []
account = _account(tmp_path, "douyin-shop")
gate = _blocking_gate(tmp_path, account, sent)
gate.check("douyin-shop", context=_context())
gate.check("douyin-shop", context=_context()) # 同工作流同日期去重
gate.check("douyin-shop", context=_context(business_date="2026-10-09"))
state = _read_state(account)
assert [
(item["workflow_id"], item["business_date"])
for item in state["pending_reruns"]
] == [
("shop.demo.daily", "2026-10-08"),
("shop.demo.daily", "2026-10-09"),
]
# shadow 运行的阻断不进入待重跑清单
gate.check(
"douyin-shop", context=_context(business_date="2026-10-10", shadow=True)
)
state = _read_state(account)
assert all(
item["business_date"] != "2026-10-10" for item in state["pending_reruns"]
)
def test_recovery_reruns_blocked_workflows_after_login(tmp_path: Path) -> None:
sent: list[dict[str, object]] = []
account = _account(tmp_path, "douyin-shop")
blocked = _blocking_gate(tmp_path, account, sent)
blocked.check("douyin-shop", context=_context())
_simulate_manual_login(account)
healthy = _healthy_gate(tmp_path, account, sent)
launcher = _FakeLauncher()
outcomes = CookieGateRecoveryManager(healthy, launcher=launcher).tick()
assert launcher.launched == [("shop.demo.daily", "2026-10-08")]
assert launcher.closed # 父进程日志句柄立即关闭
assert outcomes and outcomes[0]["launched"]
state = _read_state(account)
assert state["pending_reruns"] == []
assert state["state"] == "healthy"
assert state["last_rerun_launch_at"]
assert state["rerun_history"][0]["workflow_id"] == "shop.demo.daily"
assert any("登录态已恢复" in str(item["text"]) for item in sent)
def test_recovery_skips_while_login_still_expired(tmp_path: Path) -> None:
sent: list[dict[str, object]] = []
account = _account(tmp_path, "douyin-shop")
blocked = _blocking_gate(tmp_path, account, sent)
blocked.check("douyin-shop", context=_context())
_write_keepalive_status(account, None)
still_expired = _gate(
tmp_path,
account,
probe={"status": "expired"},
environment={"GYXX_LOGIN_ALERT_OPEN_IDS": "ou_alert_target"},
sent=sent,
)
still_expired._workflow_catalog = _recovery_workflow_catalog()
launcher = _FakeLauncher()
outcomes = CookieGateRecoveryManager(still_expired, launcher=launcher).tick()
assert launcher.launched == []
assert outcomes == []
state = _read_state(account)
assert len(state["pending_reruns"]) == 1
assert state["last_recovery_probe_at"]
def test_recovery_drops_adhoc_script_ids(tmp_path: Path) -> None:
sent: list[dict[str, object]] = []
account = _account(tmp_path, "douyin-shop")
blocked = _blocking_gate(tmp_path, account, sent)
blocked.check(
"douyin-shop", context=_context(workflow_id="script.shop.abcdef123456")
)
assert (
_read_state(account)["pending_reruns"][0]["workflow_id"]
== "script.shop.abcdef123456"
)
_simulate_manual_login(account)
healthy = _healthy_gate(tmp_path, account, sent)
launcher = _FakeLauncher()
CookieGateRecoveryManager(healthy, launcher=launcher).tick()
assert launcher.launched == []
assert _read_state(account)["pending_reruns"] == []
def test_recovery_launch_failure_keeps_pending_for_retry(tmp_path: Path) -> None:
sent: list[dict[str, object]] = []
account = _account(tmp_path, "douyin-shop")
blocked = _blocking_gate(tmp_path, account, sent)
blocked.check("douyin-shop", context=_context())
_simulate_manual_login(account)
healthy = _healthy_gate(tmp_path, account, sent)
launcher = _FakeLauncher(fail=True)
CookieGateRecoveryManager(healthy, launcher=launcher).tick()
state = _read_state(account)
assert len(state["pending_reruns"]) == 1
assert "error" in state["pending_reruns"][0]
assert state["state"] == "awaiting_login"
assert not any("已恢复" in str(item["text"]) for item in sent)
def test_recovery_respects_launch_cooldown(tmp_path: Path) -> None:
sent: list[dict[str, object]] = []
account = _account(tmp_path, "douyin-shop")
blocked = _blocking_gate(tmp_path, account, sent)
blocked.check("douyin-shop", context=_context())
_simulate_manual_login(account)
state = _read_state(account)
state["last_rerun_launch_at"] = _NOW.isoformat(timespec="seconds")
_state_file(account).write_text(json.dumps(state), encoding="utf-8")
healthy = _healthy_gate(tmp_path, account, sent)
launcher = _FakeLauncher()
CookieGateRecoveryManager(healthy, launcher=launcher).tick()
assert launcher.launched == []
+16 -5
View File
@@ -67,15 +67,21 @@ def test_baseline_contains_all_provided_people_and_routable_capabilities(
) -> None:
payload, revision = NotificationRoutingStore(routing_settings).snapshot()
assert len(payload["people"]) == 16
assert len(payload["people"]) == 18
assert payload["app_profile"] == "hermes-analyzer"
assert (
payload["app_profiles"]["hermes-analyzer"]["app_id"]
== ANALYZER_NOTIFICATION_APP_ID
)
assert all(person["mobile"] is None for person in payload["people"].values())
mobiles = {
person_id: person["mobile"]
for person_id, person in payload["people"].items()
}
assert mobiles["wang_yunlong"] == "+8613663785145"
assert mobiles["stock_alert_user_19832771651"] == "+8619832771651"
assert set(payload["capabilities"]) == set(NOTIFICATION_CAPABILITIES)
assert payload["routes"] == {}
assert set(payload["routes"]) == {"product.online_stock_alert.daily"}
assert payload["routes"]["product.online_stock_alert.daily"]["enabled"] is True
assert len(revision) == 64
wang = payload["people"]["wang_yunlong"]
@@ -85,10 +91,15 @@ def test_baseline_contains_all_provided_people_and_routable_capabilities(
assert wang["bindings"]["hermes-analyzer"]["open_id"] == (
"ou_7ad5fc8012e2f741afc5346e05ffd447"
)
_mobile_lookup_people = {
"wang_yunlong",
"online_stock_alert_recipient",
"stock_alert_user_19832771651",
}
provided = {
person["name"]: binding["open_id"]
for person_id, person in payload["people"].items()
if person_id != "wang_yunlong"
if person_id not in _mobile_lookup_people
for profile, binding in person["bindings"].items()
if profile == "hermes-analyzer"
}
@@ -112,7 +123,7 @@ def test_baseline_contains_all_provided_people_and_routable_capabilities(
assert all(
binding["verified"] is False and binding["source"] == "provided"
for person_id, person in payload["people"].items()
if person_id != "wang_yunlong"
if person_id not in _mobile_lookup_people
for profile, binding in person["bindings"].items()
if profile == "hermes-analyzer"
)
+6 -3
View File
@@ -24,7 +24,7 @@ def test_product_module_registers_scheduled_and_manual_entries() -> None:
WorkflowCatalog.load(PROJECT_ROOT / "config")
)
assert tuple(item.workflow_id for item in module.workflow_definitions()) == PRODUCT_WORKFLOW_IDS
assert len(PRODUCT_SCHEDULED_WORKFLOW_IDS) == 12
assert len(PRODUCT_SCHEDULED_WORKFLOW_IDS) == 14
assert PRODUCT_MANUAL_WORKFLOW_IDS == ()
@@ -78,7 +78,10 @@ def test_product_workflows_use_project_owned_module_commands() -> None:
assert definition.steps
for step in definition.steps:
assert isinstance(step.action, DeferredModuleCommandStep)
assert step.timeout_seconds == PRODUCT_TIMEOUT_SECONDS == 6 * 60 * 60
assert step.timeout_seconds == PRODUCT_TIMEOUT_SECONDS == 6 * 60 * 60 or (
definition.workflow_id == "product.hk_anpei_public_inventory.daily"
and step.timeout_seconds == 900.0
)
assert step.max_attempts == 1
if definition.workflow_id == "product.ecommerce_costs.daily":
assert step.resources == (
@@ -93,6 +96,7 @@ def test_product_workflows_use_project_owned_module_commands() -> None:
"product.video_upload",
"product.jd_video_upload",
"product.tmall_baibu_apply",
"product.hk_anpei_public_inventory.daily",
}:
assert step.resources == (
f"{PRODUCT_RESOURCE}:{definition.workflow_id}",
@@ -219,7 +223,6 @@ def test_jd_video_upload_workflow_allows_idempotent_rescans() -> None:
assert step.action.command_args == (
"--anchor-record-id",
"recvrXTRWWFXeJ",
"--manual-login",
"--execute",
)
+3 -3
View File
@@ -53,10 +53,10 @@ def test_catalog_and_scripts_work_after_project_and_data_roots_move(
assert workflows.returncode == 0, workflows.stderr
assert scripts.returncode == 0, scripts.stderr
assert len(json.loads(workflows.stdout)) == 35
assert len(json.loads(workflows.stdout)) == 39
commands = json.loads(scripts.stdout)
assert len(commands) == 55
assert len({row["command_id"] for row in commands}) == 55
assert len(commands) == 60
assert len({row["command_id"] for row in commands}) == 60
assert all(row["legacy_script_id"].startswith(f"{row['module']}:") for row in commands)
+3 -3
View File
@@ -103,9 +103,9 @@ def test_project_registry_covers_every_script_with_unique_stable_port() -> None:
)
assert len(scripts.command_ids) == 60
assert len(first.command_ids) == 163
assert len(first.command_ids) == 157
bindings = [first.binding_for(command_id) for command_id in first.command_ids]
assert len({binding.cdp_port for binding in bindings}) == 163
assert len({binding.cdp_port for binding in bindings}) == 157
assert all(22000 <= binding.cdp_port <= 22999 for binding in bindings)
assert all(binding.cdp_url.startswith("http://127.0.0.1:") for binding in bindings)
assert bindings == [second.binding_for(item.script_id) for item in bindings]
@@ -450,7 +450,7 @@ def test_project_accounts_cover_all_configured_platform_login_identities() -> No
assert all(
catalog.bindings_for_account(account.account_id)
for account in catalog.accounts
if account.account_id != "tmall-ozko"
if account.account_id not in {"tmall-ozko", "xingtu"}
)
assert catalog.bindings_for_account("tmall-ozko") == ()
assert catalog.account_for("jd-shop").credential_env_names == (
+1 -1
View File
@@ -54,7 +54,7 @@ def test_python_scheduler_status_is_project_owned_and_cross_platform(tmp_path: P
assert exit_code == 0
payload = json.loads(output.getvalue())
assert payload["timezone"] == "Asia/Shanghai"
assert payload["scheduled_count"] == 35
assert payload["scheduled_count"] == 39
assert payload["state"]["schema_version"] == 1
+3 -2
View File
@@ -206,16 +206,17 @@ def test_misfire_older_than_grace_is_not_launched(tmp_path: Path) -> None:
def test_configured_schedules_disable_cookie_refresh_job() -> None:
catalog = WorkflowCatalog.load(PROJECT_ROOT / "config")
assert len(catalog.schedules) == 35
assert len(catalog.schedules) == 39
assert [
schedule.workflow_id for schedule in catalog.schedules if not schedule.enabled
] == [
"content.metrics.backfill",
"content.marketing_report.daily",
"content.relogin.weekly",
"content.summary.weekly",
"content.summary.monthly",
"content.creator_report.monthly",
"shop.tmall_negative_reviews.daily",
"shop.tmall_customer_service.daily",
"product.alert.daily",
"product.style_analysis.interval",
"supply.purchase_confirmation.daily",
+4 -8
View File
@@ -80,10 +80,10 @@ def test_default_registry_covers_every_executable_workflow_entry() -> None:
for entry in entries:
workflow_entries.add(f"{workflow['module']}:{entry}")
assert scheduled_workflows == 35
assert scheduled_workflows == 39
assert manual_workflows == 0
assert len(workflow_entries) == 50
assert len(catalog.scripts) == 55
assert len(workflow_entries) == 55
assert len(catalog.scripts) == 60
assert workflow_entries < set(catalog.script_ids)
assert {
"content.mapping.rebuild",
@@ -179,10 +179,6 @@ def test_default_registry_covers_every_executable_workflow_entry() -> None:
catalog.get("content.marketing_report.generate").notification_workflow_id
== "content.marketing_report.daily"
)
assert (
catalog.get("content.login.refresh").notification_workflow_id
== "content.relogin.weekly"
)
@pytest.mark.parametrize(
@@ -284,7 +280,7 @@ def test_cli_lists_and_dry_runs_registered_commands(tmp_path: Path) -> None:
assert main(["scripts", "list", "--json"], settings=settings, stdout=output) == 0
rows = json.loads(output.getvalue())
assert len(rows) == 55
assert len(rows) == 60
assert {row["module"] for row in rows} == {
"content_marketing",
"product_commerce",
+2 -2
View File
@@ -47,8 +47,8 @@ def test_deployment_documents_one_scheduler_and_external_data_root() -> None:
assert "GYXX_DATA_ROOT" in guide
assert "/var/lib/gyxx-flow" in guide
assert "Windows Task Scheduler" in guide
assert len(payload["schedules"]) == 35
assert sum(item.get("enabled", True) for item in payload["schedules"]) == 25
assert len(payload["schedules"]) == 39
assert sum(item.get("enabled", True) for item in payload["schedules"]) == 28
def test_three_canonical_documents_cover_architecture_deployment_and_operations() -> None:
+1 -1
View File
@@ -57,6 +57,6 @@ def test_every_workflow_and_manual_script_resolves_inside_new_project() -> None:
assert target.is_relative_to(PROJECT_ROOT)
commands = ScriptCatalog.load_default()
assert len(commands.scripts) == 55
assert len(commands.scripts) == 60
assert len(commands.command_ids) == len(set(commands.command_ids))
assert all(command.path.is_relative_to(PROJECT_ROOT) for command in commands.scripts)