Compare commits
19
Commits
01f3d7aaf7
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
b1861f69cd | ||
|
|
2ef635be0e | ||
|
|
cf85985753 | ||
|
|
e5761bd828 | ||
|
|
9feb188d93 | ||
|
|
7b305b0b12 | ||
|
|
6cd563eab0 | ||
|
|
0cc4fafc7c | ||
|
|
9a91ce80e4 | ||
|
|
97a486e97d | ||
|
|
36fe7b81f8 | ||
|
|
dc8be88e60 | ||
|
|
90282ae766 | ||
|
|
df11e195b7 | ||
|
|
b0e1b5066d | ||
|
|
1e9ddffd2f | ||
|
|
41f448420d | ||
|
|
130f06b320 | ||
|
|
78ea4641ce |
No files matched your search
+4
-4
@@ -33,12 +33,12 @@ Thumbs.db
|
||||
# Agent-local learning logs
|
||||
.learnings/
|
||||
|
||||
# Agent tool local state (security-scan ledger, session memory)
|
||||
/.mimosa/
|
||||
/.workbuddy/
|
||||
|
||||
# Local tool artifacts
|
||||
/.playwright-cli/
|
||||
/output/
|
||||
/nul
|
||||
/new_all_bag.xlsx
|
||||
|
||||
# Workbench (dsh plugin) local artifacts
|
||||
/workbench/plugin/node_modules/
|
||||
/workbench/cordis.local.yml
|
||||
+1
-1
@@ -19,7 +19,7 @@ lint:
|
||||
stage: quality
|
||||
script:
|
||||
- uv run ruff check src tests
|
||||
- uv run ruff check src/gyxx_flow/modules/content_marketing/daily_creator_exposure_scope.py src/gyxx_flow/modules/content_marketing/chanmama_export.py src/gyxx_flow/modules/content_marketing/self_operated_accounts.py src/gyxx_flow/modules/content_marketing/self_xhs_scraper.py src/gyxx_flow/modules/content_marketing/data/tools/sync_self_video_metrics.py src/gyxx_flow/modules/product_commerce/market_rank_limits.py src/gyxx_flow/modules/product_commerce/market_rank_product_import.py src/gyxx_flow/modules/shop_intelligence/collectors/dy_after_sales_quality.py src/gyxx_flow/modules/shop_intelligence/collectors/tmall_negative_reviews.py
|
||||
- uv run ruff check src/gyxx_flow/modules/content_marketing/daily_creator_exposure_scope.py src/gyxx_flow/modules/content_marketing/chanmama_export.py src/gyxx_flow/modules/content_marketing/self_operated_accounts.py src/gyxx_flow/modules/content_marketing/self_xhs_scraper.py src/gyxx_flow/modules/content_marketing/data/tools/sync_self_video_metrics.py src/gyxx_flow/modules/product_commerce/market_rank_limits.py src/gyxx_flow/modules/product_commerce/market_rank_product_import.py src/gyxx_flow/modules/product_commerce/collect_douyin_qianchuan_ads.py src/gyxx_flow/modules/product_commerce/collect_erp_sales_theme_daily.py src/gyxx_flow/modules/shop_intelligence/collectors/dy_after_sales_quality.py src/gyxx_flow/modules/shop_intelligence/collectors/tmall_negative_reviews.py
|
||||
|
||||
test:
|
||||
stage: test
|
||||
|
||||
@@ -41,6 +41,7 @@ uv run gyxx accounts login|sync [account_id]
|
||||
- Manual/backfill/maintenance commands must be declared in `config/commands.json`; never reintroduce recursive executable-script discovery.
|
||||
- Each executable `workflow_id` compiles to a LangGraph `StateGraph`. Preserve the public contracts of the workflow model, `RunJournal`, `LockManager`, shadow comparison, and `EffectLedger` when changing graph execution.
|
||||
- External systems: PostgreSQL via runtime-injected cloud DSN (`GYXX_POSTGRES_DSN`; no address/credential defaults in source, loopback rejected in cloud mode); Hermes loopback-only roles on `127.0.0.1:8642/8643` (key `GYXX_HERMES_API_KEY`); Feishu keeps its existing identity. Every browser binding keeps a unique CDP port plus isolated Profile/Cookie/storage-state; scripts sharing an account declare it under `accounts` in `runtime-bindings.json` — the vault `state/accounts/<id>/` is the single login authority (`gyxx accounts login/sync`) and its cookies merge read-only into member scripts while profiles stay isolated.
|
||||
- 登录统一管理:正式执行的工作流由 `src/gyxx_flow/cookie_gate.py` 在采集步骤前注入每账号 `cookie-gate.<id>` 步骤(blocking 账号失效→阻断并飞书通知人工重登;`login_command_id` 账号先自动账密恢复;`credential_env_names` 账号 defer 给脚本内账密登录)。阻断记录 `pending_reruns`,由调度器维护循环里的 `CookieGateRecoveryManager` 在登录态恢复后按原业务日期自动重跑并发飞书确认。采集脚本不得内嵌扫码/验证码等待——失效时抛 `CookieExpiredError`(`adapters/browser.py`)退出 75;编排器工作流在 `workflows.json` 用 `required_accounts` 声明账号依赖。不要重新引入 relogin_*/批量补采里的扫码重登子进程。
|
||||
- 启动或重启工作流控制台(`gyxx console`)必须携带 `--env-file D:\product-collector-analyze-flow\.env`(与调度器进程一致),否则 `PG_HOST/PG_PORT/PG_DB/PG_USER/PG_PASSWORD` 等云端 PostgreSQL 凭据和账号配置不会注入,「商品经营日报采集」「聚水潭全店铺款式日报」等正式执行会报“缺少 PostgreSQL 运行时凭据”。参考启动脚本:`var/tmp/start-console.ps1`。
|
||||
- Source sync (`gyxx sources status/apply`): manifest-driven three-way hash drift check; only untransformed, conflict-free one-sided changes may be auto-copied, and only with `--execute`. Credentials, cookies, logs, and collected data never enter sync manifests.
|
||||
|
||||
|
||||
@@ -40,17 +40,6 @@ var/ 默认运行数据;不属于源码
|
||||
- `shop_intelligence`:店铺、竞店和京东自营业绩。
|
||||
- `supply_chain`:采购确认、补货、库存预警和采购单更新。
|
||||
|
||||
## 智能工作台(dsh-refact 分支)
|
||||
|
||||
`workbench/` 以 [deepseek-harness](https://github.com/deepseek-ai/deepseek-harness) 为
|
||||
agent 底座,把现有工作流以 dsh 插件形式接入 Web UI:侧边栏选中任意工作流即可提问、
|
||||
诊断、修复,支持监控告警与受控启动。使用与设计见 [workbench/README.md](workbench/README.md)
|
||||
与 [docs/workbench.md](docs/workbench.md)。
|
||||
|
||||
```powershell
|
||||
powershell -File workbench\bin\start-workbench.ps1
|
||||
```
|
||||
|
||||
## 开发环境
|
||||
|
||||
```powershell
|
||||
|
||||
+17
-7
@@ -50,13 +50,6 @@
|
||||
"kind": "python",
|
||||
"notification_workflow_id": "content.marketing_report.daily"
|
||||
},
|
||||
{
|
||||
"id": "content.login.refresh",
|
||||
"module": "content_marketing",
|
||||
"entry": "data/tools/friday_relogin_parallel.py",
|
||||
"kind": "python",
|
||||
"notification_workflow_id": "content.relogin.weekly"
|
||||
},
|
||||
{
|
||||
"id": "content.creator_report.generate",
|
||||
"module": "content_marketing",
|
||||
@@ -472,6 +465,23 @@
|
||||
"{business_date}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "product.erp_sales_theme_daily.collect",
|
||||
"module": "product_commerce",
|
||||
"entry": "collect_erp_sales_theme_daily.py",
|
||||
"kind": "python",
|
||||
"default_args": [
|
||||
"--date",
|
||||
"{business_date}",
|
||||
"--execute"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "product.hk_anpei_public_inventory.refresh",
|
||||
"module": "product_commerce",
|
||||
"entry": "public_sku_inventory_sheet.py",
|
||||
"kind": "python"
|
||||
},
|
||||
{
|
||||
"id": "product.backfill.run",
|
||||
"module": "product_commerce",
|
||||
|
||||
@@ -222,9 +222,6 @@
|
||||
"content.marketing_report.daily": {
|
||||
"condition": "正式执行且启用发送时投递最终营销日报。"
|
||||
},
|
||||
"content.relogin.weekly": {
|
||||
"condition": "刷新期间投递各平台扫码二维码,并在平台失败时投递失败汇总。"
|
||||
},
|
||||
"product.alert.daily": {
|
||||
"condition": "命中销量下滑规则且尚未重复通知时投递告警。"
|
||||
},
|
||||
|
||||
+36
-114
@@ -88,7 +88,6 @@
|
||||
"jd.com"
|
||||
],
|
||||
"required_cookie_names": [
|
||||
"pin",
|
||||
"thor"
|
||||
],
|
||||
"credential_env_names": [
|
||||
@@ -116,7 +115,6 @@
|
||||
"jd.com"
|
||||
],
|
||||
"required_cookie_names": [
|
||||
"pin",
|
||||
"thor"
|
||||
],
|
||||
"credential_env_names": [
|
||||
@@ -145,10 +143,7 @@
|
||||
],
|
||||
"required_cookie_names": [
|
||||
"u_id",
|
||||
"u_co_id",
|
||||
"ASP.NET_SessionId",
|
||||
".ASPXAUTH",
|
||||
"token"
|
||||
"u_co_id"
|
||||
],
|
||||
"credential_env_names": [
|
||||
"ERP_USERNAME",
|
||||
@@ -168,6 +163,7 @@
|
||||
"product_commerce:collect_erp_yesterday_metrics.py",
|
||||
"product_commerce:erp_login_product_analysis.py",
|
||||
"product.erp_all_shop_daily",
|
||||
"product.erp_sales_theme_daily.collect",
|
||||
"product.sales_sheet.sync",
|
||||
"supply_chain:orchestrator/scripts/ProductReplenishment.py",
|
||||
"supply_chain:orchestrator/scripts/PurchaseConfirmation.py",
|
||||
@@ -298,7 +294,8 @@
|
||||
"xiaohongshu.com"
|
||||
],
|
||||
"required_cookie_names": [
|
||||
"web_session",
|
||||
"customer-sso-sid",
|
||||
"solar.beaker.session.id",
|
||||
"a1"
|
||||
],
|
||||
"keepalive": {
|
||||
@@ -312,8 +309,6 @@
|
||||
},
|
||||
"members": [
|
||||
"content.comments.collect_xiaohongshu",
|
||||
"content_marketing:data/tools/relogin_pgy.py",
|
||||
"content_marketing:data/tools/relogin_xiaohongshu.py",
|
||||
"content_marketing:pgy_xhs_scraper_v2.py"
|
||||
]
|
||||
},
|
||||
@@ -340,7 +335,6 @@
|
||||
},
|
||||
"members": [
|
||||
"content.comments.collect_douyin",
|
||||
"content_marketing:data/tools/relogin_douyin.py",
|
||||
"content_marketing:self_douyin_scraper.py"
|
||||
]
|
||||
},
|
||||
@@ -353,9 +347,6 @@
|
||||
"xingtu.cn"
|
||||
],
|
||||
"required_cookie_names": [
|
||||
"sessionid",
|
||||
"sessionid_ss",
|
||||
"sid_tt",
|
||||
"sid_guard"
|
||||
],
|
||||
"keepalive": {
|
||||
@@ -367,13 +358,15 @@
|
||||
"headless": true,
|
||||
"real_chrome": true
|
||||
},
|
||||
"members": [
|
||||
"content_marketing:data/tools/relogin_xingtu.py"
|
||||
]
|
||||
"members": []
|
||||
},
|
||||
"chanmama": {
|
||||
"cdp_port": 22207,
|
||||
"login_mode": "D",
|
||||
"credential_env_names": [
|
||||
"CHANMAMA_ACCOUNT",
|
||||
"CHANMAMA_PASSWORD"
|
||||
],
|
||||
"required_cookie_domains": [
|
||||
"chanmama.com"
|
||||
],
|
||||
@@ -411,8 +404,7 @@
|
||||
},
|
||||
"members": [
|
||||
"content.comments.collect_bilibili",
|
||||
"content.metrics.collect_self_bilibili",
|
||||
"content_marketing:data/tools/relogin_bilibili.py"
|
||||
"content.metrics.collect_self_bilibili"
|
||||
]
|
||||
}
|
||||
},
|
||||
@@ -582,23 +574,6 @@
|
||||
],
|
||||
"cdp_port": 22016
|
||||
},
|
||||
"content_marketing:data/tools/friday_relogin.bat": {
|
||||
"script_id": "content_marketing:data/tools/friday_relogin.bat",
|
||||
"state_key": "content_marketing:data/tools/friday_relogin.bat",
|
||||
"aliases": [
|
||||
"content_marketing:data/tools/friday_relogin.bat"
|
||||
],
|
||||
"cdp_port": 22017
|
||||
},
|
||||
"content.login.refresh": {
|
||||
"script_id": "content_marketing:data/tools/friday_relogin_parallel.py",
|
||||
"state_key": "content_marketing:data/tools/friday_relogin_parallel.py",
|
||||
"aliases": [
|
||||
"content_marketing:data/tools/friday_relogin_parallel.py"
|
||||
],
|
||||
"cdp_port": 22018,
|
||||
"login_mode": "D"
|
||||
},
|
||||
"content.creator_report.generate": {
|
||||
"script_id": "content_marketing:data/tools/generate_creator_report.py",
|
||||
"state_key": "content_marketing:data/tools/generate_creator_report.py",
|
||||
@@ -679,83 +654,6 @@
|
||||
],
|
||||
"cdp_port": 22135
|
||||
},
|
||||
"content_marketing:data/tools/relogin_bilibili.py": {
|
||||
"script_id": "content_marketing:data/tools/relogin_bilibili.py",
|
||||
"state_key": "content_marketing:data/tools/relogin_bilibili.py",
|
||||
"aliases": [
|
||||
"content_marketing:data/tools/relogin_bilibili.py"
|
||||
],
|
||||
"cdp_port": 22028,
|
||||
"login_mode": "D",
|
||||
"required_cookie_domains": [
|
||||
"bilibili.com"
|
||||
],
|
||||
"required_cookie_names": [
|
||||
"SESSDATA"
|
||||
]
|
||||
},
|
||||
"content_marketing:data/tools/relogin_douyin.py": {
|
||||
"script_id": "content_marketing:data/tools/relogin_douyin.py",
|
||||
"state_key": "content_marketing:data/tools/relogin_douyin.py",
|
||||
"aliases": [
|
||||
"content_marketing:data/tools/relogin_douyin.py"
|
||||
],
|
||||
"cdp_port": 22029,
|
||||
"login_mode": "D",
|
||||
"required_cookie_domains": [
|
||||
"douyin.com"
|
||||
],
|
||||
"required_cookie_names": [
|
||||
"sessionid"
|
||||
]
|
||||
},
|
||||
"content_marketing:data/tools/relogin_pgy.py": {
|
||||
"script_id": "content_marketing:data/tools/relogin_pgy.py",
|
||||
"state_key": "content_marketing:data/tools/relogin_pgy.py",
|
||||
"aliases": [
|
||||
"content_marketing:data/tools/relogin_pgy.py"
|
||||
],
|
||||
"cdp_port": 22030,
|
||||
"login_mode": "D",
|
||||
"required_cookie_domains": [
|
||||
"xiaohongshu.com"
|
||||
]
|
||||
},
|
||||
"content_marketing:data/tools/relogin_xiaohongshu.py": {
|
||||
"script_id": "content_marketing:data/tools/relogin_xiaohongshu.py",
|
||||
"state_key": "content_marketing:data/tools/relogin_xiaohongshu.py",
|
||||
"aliases": [
|
||||
"content_marketing:data/tools/relogin_xiaohongshu.py"
|
||||
],
|
||||
"cdp_port": 22031,
|
||||
"login_mode": "D",
|
||||
"required_cookie_domains": [
|
||||
"xiaohongshu.com"
|
||||
],
|
||||
"required_cookie_names": [
|
||||
"web_session"
|
||||
]
|
||||
},
|
||||
"content_marketing:data/tools/relogin_xingtu.py": {
|
||||
"script_id": "content_marketing:data/tools/relogin_xingtu.py",
|
||||
"state_key": "content_marketing:data/tools/relogin_xingtu.py",
|
||||
"aliases": [
|
||||
"content_marketing:data/tools/relogin_xingtu.py"
|
||||
],
|
||||
"cdp_port": 22032,
|
||||
"login_mode": "D",
|
||||
"required_cookie_domains": [
|
||||
"douyin.com",
|
||||
"oceanengine.com",
|
||||
"xingtu.cn"
|
||||
],
|
||||
"required_cookie_names": [
|
||||
"sessionid",
|
||||
"sessionid_ss",
|
||||
"sid_tt",
|
||||
"sid_guard"
|
||||
]
|
||||
},
|
||||
"content.failed.retry": {
|
||||
"script_id": "content_marketing:data/tools/retry_failed.py",
|
||||
"state_key": "content_marketing:data/tools/retry_failed.py",
|
||||
@@ -796,8 +694,12 @@
|
||||
],
|
||||
"cdp_port": 22165,
|
||||
"login_mode": "C",
|
||||
"required_cookie_domains": ["xiaohongshu.com"],
|
||||
"credential_env_names": ["XHS_ARK_ACCOUNT"]
|
||||
"required_cookie_domains": [
|
||||
"xiaohongshu.com"
|
||||
],
|
||||
"credential_env_names": [
|
||||
"XHS_ARK_ACCOUNT"
|
||||
]
|
||||
},
|
||||
"content_marketing:data/tools/sync_style_categories.py": {
|
||||
"script_id": "content_marketing:data/tools/sync_style_categories.py",
|
||||
@@ -1441,6 +1343,26 @@
|
||||
"erp321.com"
|
||||
]
|
||||
},
|
||||
"product.erp_sales_theme_daily.collect": {
|
||||
"script_id": "product_commerce:collect_erp_sales_theme_daily.py",
|
||||
"state_key": "product_commerce:collect_erp_sales_theme_daily.py",
|
||||
"aliases": [
|
||||
"product_commerce:collect_erp_sales_theme_daily.py"
|
||||
],
|
||||
"cdp_port": 22167,
|
||||
"login_mode": "C",
|
||||
"required_cookie_domains": [
|
||||
"erp321.com"
|
||||
]
|
||||
},
|
||||
"product.hk_anpei_public_inventory.refresh": {
|
||||
"script_id": "product_commerce:public_sku_inventory_sheet.py",
|
||||
"state_key": "product_commerce:public_sku_inventory_sheet.py",
|
||||
"aliases": [
|
||||
"product_commerce:public_sku_inventory_sheet.py"
|
||||
],
|
||||
"cdp_port": 22166
|
||||
},
|
||||
"product_commerce:scripts/insert_jd_main_image_records.py": {
|
||||
"script_id": "product_commerce:scripts/insert_jd_main_image_records.py",
|
||||
"state_key": "product_commerce:scripts/insert_jd_main_image_records.py",
|
||||
|
||||
+8
-13
@@ -11,10 +11,7 @@
|
||||
"workflow_id": "content.metrics.daily"
|
||||
},
|
||||
{
|
||||
"at": [
|
||||
"01:00",
|
||||
"05:00"
|
||||
],
|
||||
"at": "06:00",
|
||||
"business_date_offset_days": 0,
|
||||
"enabled": true,
|
||||
"kind": "daily",
|
||||
@@ -42,15 +39,6 @@
|
||||
"kind": "daily",
|
||||
"workflow_id": "content.marketing_report.daily"
|
||||
},
|
||||
{
|
||||
"at": "10:00",
|
||||
"days": [
|
||||
"Friday"
|
||||
],
|
||||
"enabled": false,
|
||||
"kind": "weekly",
|
||||
"workflow_id": "content.relogin.weekly"
|
||||
},
|
||||
{
|
||||
"at": "12:00",
|
||||
"days": [
|
||||
@@ -221,6 +209,13 @@
|
||||
"kind": "daily",
|
||||
"workflow_id": "product.erp_all_shop_daily"
|
||||
},
|
||||
{
|
||||
"at": "09:00",
|
||||
"business_date_offset_days": 0,
|
||||
"enabled": true,
|
||||
"kind": "daily",
|
||||
"workflow_id": "product.hk_anpei_public_inventory.daily"
|
||||
},
|
||||
{
|
||||
"at": "12:00",
|
||||
"business_date_offset_days": 0,
|
||||
|
||||
@@ -7,7 +7,7 @@
|
||||
"source_relative_path": "bilibili_comment_scraper.py",
|
||||
"target_relative_path": "src/gyxx_flow/modules/content_marketing/bilibili_comment_scraper.py",
|
||||
"source_sha256": "c7f116f30889125c668e7f4dd796e7249ba7ea8621d5ab5fb518028f56c7f8dc",
|
||||
"target_sha256": "109a01a6dc2b79261cff68ff42818a7561f3319b6be3788d9300bb5ce0217a96",
|
||||
"target_sha256": "d3ae63aab128a3e4078cf357aa4dfcf1f2c3b36c3116989d8f2252be62544e53",
|
||||
"transformed": true,
|
||||
"category": "source"
|
||||
},
|
||||
@@ -23,7 +23,7 @@
|
||||
"source_relative_path": "chanmama_scraper.py",
|
||||
"target_relative_path": "src/gyxx_flow/modules/content_marketing/chanmama_scraper.py",
|
||||
"source_sha256": "2edc26879b57d73ec5992c80e5ef74d76a509f09bff7284b1ae17e90df543e07",
|
||||
"target_sha256": "8b87268e0f105cf4d349b192c7b2c24d5e318a9d88fcd14264824b083103b24e",
|
||||
"target_sha256": "b808b412e9bb8d824c1a117cbecdd28234408a55fa60891f1ae748abeb7bc415",
|
||||
"transformed": true,
|
||||
"category": "source"
|
||||
},
|
||||
@@ -111,7 +111,7 @@
|
||||
"source_relative_path": "data/tools/batch_rescrape_bilibili.py",
|
||||
"target_relative_path": "src/gyxx_flow/modules/content_marketing/data/tools/batch_rescrape_bilibili.py",
|
||||
"source_sha256": "5b07fe27f1b76714ca3f3feae34e3d719277ed0cfe124d76cacd8e3c904b1fca",
|
||||
"target_sha256": "10e5dee90c591201daa7a7b0eb13a40ed1c97d7d51b1b93ef8f1d5d4b8e09580",
|
||||
"target_sha256": "8fa0e5838615f36c9841deb08f6710693e63b011f1f91138c1121d99277dd391",
|
||||
"transformed": true,
|
||||
"category": "tool"
|
||||
},
|
||||
@@ -119,7 +119,7 @@
|
||||
"source_relative_path": "data/tools/batch_rescrape_douyin.py",
|
||||
"target_relative_path": "src/gyxx_flow/modules/content_marketing/data/tools/batch_rescrape_douyin.py",
|
||||
"source_sha256": "15b1438513f8216a6c7a103744e8a1ca6bc63b32c111d7c9bce31c18be9e84a6",
|
||||
"target_sha256": "3233a7840e07a87535a8df065562a7cb681dbc140e002d82970598de3d9768cf",
|
||||
"target_sha256": "e0f1227a7b66d85ea65191b5f9c6c5bf588e85cc0cd66e20afae7a9dc7a2684e",
|
||||
"transformed": true,
|
||||
"category": "tool"
|
||||
},
|
||||
@@ -127,7 +127,7 @@
|
||||
"source_relative_path": "data/tools/batch_rescrape_xiaohongshu.py",
|
||||
"target_relative_path": "src/gyxx_flow/modules/content_marketing/data/tools/batch_rescrape_xiaohongshu.py",
|
||||
"source_sha256": "6d85f7dfd52c7a92b951ac29e7ee3377aebdb743af2a362d1f38366188dcad08",
|
||||
"target_sha256": "01be5d1ba260ccd46e01b9e72b6f7dd3d08577615455fc9010de4efe8a710aea",
|
||||
"target_sha256": "c481bd6af34040388abafed0466d5632b3892865bf192d245664627003a4c43f",
|
||||
"transformed": true,
|
||||
"category": "tool"
|
||||
},
|
||||
@@ -183,7 +183,7 @@
|
||||
"source_relative_path": "data/tools/db.py",
|
||||
"target_relative_path": "src/gyxx_flow/modules/content_marketing/data/tools/db.py",
|
||||
"source_sha256": "83f158a78841244c1f02255d1500018b5a9dbe1f282cbdb937acc344b08abc25",
|
||||
"target_sha256": "62ffad854cc2bc3a3ae4f2a28b4ee345bc9eecf72101b647c3aa165c691b456a",
|
||||
"target_sha256": "bfdb761761532263ae6c555ee455cad4efa300b7fa5639931813f4f95e79ff6a",
|
||||
"transformed": true,
|
||||
"category": "tool"
|
||||
},
|
||||
@@ -203,14 +203,6 @@
|
||||
"transformed": true,
|
||||
"category": "tool"
|
||||
},
|
||||
{
|
||||
"source_relative_path": "data/tools/friday_relogin_parallel.py",
|
||||
"target_relative_path": "src/gyxx_flow/modules/content_marketing/data/tools/friday_relogin_parallel.py",
|
||||
"source_sha256": "affed4ab6e5e179c85dc92999502b98e007c71243ed26977b3c5eeba23b9e3dc",
|
||||
"target_sha256": "438bedfb874c7eb566e1998815a22ca72b7d177cee221fd39625a96872bd450b",
|
||||
"transformed": true,
|
||||
"category": "tool"
|
||||
},
|
||||
{
|
||||
"source_relative_path": "data/tools/generate_creator_report.py",
|
||||
"target_relative_path": "src/gyxx_flow/modules/content_marketing/data/tools/generate_creator_report.py",
|
||||
@@ -291,54 +283,6 @@
|
||||
"transformed": true,
|
||||
"category": "tool"
|
||||
},
|
||||
{
|
||||
"source_relative_path": "data/tools/relogin_bilibili.py",
|
||||
"target_relative_path": "src/gyxx_flow/modules/content_marketing/data/tools/relogin_bilibili.py",
|
||||
"source_sha256": "b8dc12539bcd6f1a368d50ead1f5d33d6577a75c92f0b7bafa75a67edd7a73fa",
|
||||
"target_sha256": "d47f26f62fbe3262128327b916151385e527125a18f27f7455b8382aa5a77ecb",
|
||||
"transformed": true,
|
||||
"category": "tool"
|
||||
},
|
||||
{
|
||||
"source_relative_path": "data/tools/relogin_douyin.py",
|
||||
"target_relative_path": "src/gyxx_flow/modules/content_marketing/data/tools/relogin_douyin.py",
|
||||
"source_sha256": "3c2017f8a801528ff17e4c4bae24615256b8788c185f9a797fe87470e35b1645",
|
||||
"target_sha256": "82a0ecf4101f2eaded94cc4cd9054cb08e1e27519a58cfc0d3126888cc1ce5d4",
|
||||
"transformed": true,
|
||||
"category": "tool"
|
||||
},
|
||||
{
|
||||
"source_relative_path": "data/tools/relogin_pgy.py",
|
||||
"target_relative_path": "src/gyxx_flow/modules/content_marketing/data/tools/relogin_pgy.py",
|
||||
"source_sha256": "759c6a68a0851b4c87d8febfc5d94925831a3b2d7cdf1e6a6e74dcd52b83ae3f",
|
||||
"target_sha256": "d75697ae2b6ac1f23bb79077a4df0af33113813a46525a6f3980b04391857cac",
|
||||
"transformed": true,
|
||||
"category": "tool"
|
||||
},
|
||||
{
|
||||
"source_relative_path": "data/tools/relogin_transaction.py",
|
||||
"target_relative_path": "src/gyxx_flow/modules/content_marketing/data/tools/relogin_transaction.py",
|
||||
"source_sha256": "57ec20de8d99290268ed3d1fc2b66bf3faeaca807d7f3744dd51de6cc93628b2",
|
||||
"target_sha256": "57ec20de8d99290268ed3d1fc2b66bf3faeaca807d7f3744dd51de6cc93628b2",
|
||||
"transformed": false,
|
||||
"category": "tool"
|
||||
},
|
||||
{
|
||||
"source_relative_path": "data/tools/relogin_xiaohongshu.py",
|
||||
"target_relative_path": "src/gyxx_flow/modules/content_marketing/data/tools/relogin_xiaohongshu.py",
|
||||
"source_sha256": "fcdea30adecb066c4f3699ded457df0bc763d978248276ddbffa9f18bc9bdf2c",
|
||||
"target_sha256": "e700dd57e5cf87d68529bcd6c062e73d6bd9f1e6ef06ef3122e2b16c288aaf1a",
|
||||
"transformed": true,
|
||||
"category": "tool"
|
||||
},
|
||||
{
|
||||
"source_relative_path": "data/tools/relogin_xingtu.py",
|
||||
"target_relative_path": "src/gyxx_flow/modules/content_marketing/data/tools/relogin_xingtu.py",
|
||||
"source_sha256": "c9d434cb02a57ebd7b884598563077d733d56e45dad59ed30774eb7e1c668cb0",
|
||||
"target_sha256": "06ec1088677d69511057008acb2b05cca2da015a8219aee10bf19bbae3c25d79",
|
||||
"transformed": true,
|
||||
"category": "tool"
|
||||
},
|
||||
{
|
||||
"source_relative_path": "data/tools/retry_failed.py",
|
||||
"target_relative_path": "src/gyxx_flow/modules/content_marketing/data/tools/retry_failed.py",
|
||||
@@ -351,7 +295,7 @@
|
||||
"source_relative_path": "data/tools/schema_gyxx_super_data.sql",
|
||||
"target_relative_path": "src/gyxx_flow/modules/content_marketing/data/tools/schema_gyxx_super_data.sql",
|
||||
"source_sha256": "43d202d7d0452b7d0671ee9c40149c21328db4b2863fa4d030af456f89defe3c",
|
||||
"target_sha256": "001870851dea9c373a77b87335604c4363b06c3eb0ed7c97b4a8d5e891e56758",
|
||||
"target_sha256": "4b2eca918bb81de0cac529b786d0d37ce04af2c1bc4be81cd20c8ba9456b1348",
|
||||
"transformed": true,
|
||||
"category": "sql"
|
||||
},
|
||||
@@ -415,7 +359,7 @@
|
||||
"source_relative_path": "douyin_comment_scraper.py",
|
||||
"target_relative_path": "src/gyxx_flow/modules/content_marketing/douyin_comment_scraper.py",
|
||||
"source_sha256": "ec785ead9c95c5de65c00b06212b1f03d0159fed475a95a2b593d3b24e486e43",
|
||||
"target_sha256": "6e7ac4357fe466656b93ca7bae6c49adc4d59fa75974a39c6885c7433bd4d402",
|
||||
"target_sha256": "35d3caf8bc85a82ca15c3fc9ff317c21460173996106ffee8f1d3b5f1d13b801",
|
||||
"transformed": true,
|
||||
"category": "source"
|
||||
},
|
||||
@@ -455,7 +399,7 @@
|
||||
"source_relative_path": "pgy_xhs_scraper_v2.py",
|
||||
"target_relative_path": "src/gyxx_flow/modules/content_marketing/pgy_xhs_scraper_v2.py",
|
||||
"source_sha256": "2be78ef76df38e80da2099a76e6e3fd9dcdc2e8875afb44279f54003ccb5f563",
|
||||
"target_sha256": "e5df4b34f67a565ec5d5ce4dd5857e80e363dd7559e7e29006dcd3e421276c8a",
|
||||
"target_sha256": "472c13700adc98144b53f2921bbdde15ebb7dc138eaab76885b514dadd306887",
|
||||
"transformed": true,
|
||||
"category": "source"
|
||||
},
|
||||
@@ -479,7 +423,7 @@
|
||||
"source_relative_path": "self_bilibili_scraper.py",
|
||||
"target_relative_path": "src/gyxx_flow/modules/content_marketing/self_bilibili_scraper.py",
|
||||
"source_sha256": "2ef650a16dde61697101bb219a8c3adcd682fec1529bb9c5312dc5b384d83a72",
|
||||
"target_sha256": "6ec5ed6991a93f806fb74d0870e953e5a05d52043a83f2fa261e2ec67bee6ad7",
|
||||
"target_sha256": "1c09171d1269a3a38a5e6f9f99b98d0da4c7ff11d075613fc288ee209e82e9d2",
|
||||
"transformed": true,
|
||||
"category": "source"
|
||||
},
|
||||
@@ -487,7 +431,7 @@
|
||||
"source_relative_path": "self_douyin_scraper.py",
|
||||
"target_relative_path": "src/gyxx_flow/modules/content_marketing/self_douyin_scraper.py",
|
||||
"source_sha256": "4aba1264dee052521e5712d7219916e534db5705e3cec75caaa1b025a4bc2506",
|
||||
"target_sha256": "62307df115c8d7336b470baba074163ab3f3468bae8b0af266af370e2e42d9f1",
|
||||
"target_sha256": "945659cd80435bab08bfe1dee13c451c8c4b5fe362313ea09591525c551d3c48",
|
||||
"transformed": true,
|
||||
"category": "source"
|
||||
},
|
||||
@@ -511,7 +455,7 @@
|
||||
"source_relative_path": "xiaohongshu_comment_scraper.py",
|
||||
"target_relative_path": "src/gyxx_flow/modules/content_marketing/xiaohongshu_comment_scraper.py",
|
||||
"source_sha256": "58bc07bb2fd67897e18a9981f931df8fadf24a6cc5f09d55bd3aaa2e3c6756a4",
|
||||
"target_sha256": "afca1104324546da2315fdf7a427c3f34021eef31ddc2fea76877f818718ccb0",
|
||||
"target_sha256": "fff780d9e16717e6a1ba9c5ee4db22cb1bb5a85f75807c811072c20a6bd390de",
|
||||
"transformed": true,
|
||||
"category": "source"
|
||||
},
|
||||
@@ -527,7 +471,7 @@
|
||||
"source_relative_path": "xingtu_scraper_v2.py",
|
||||
"target_relative_path": "src/gyxx_flow/modules/content_marketing/xingtu_scraper_v2.py",
|
||||
"source_sha256": "0d4eb87779064e13b8d78daa6dba13698f1149a26b3a1de2e064c117a29cf49a",
|
||||
"target_sha256": "b36339444e26048a90c8ac999a0a628bcddd4f64842924f6c60e600c7a3bec37",
|
||||
"target_sha256": "dfdd07af20c8e2e59411fb6e32a82588db5601d6f3fba75d596e0b5ac19b2be8",
|
||||
"transformed": true,
|
||||
"category": "source"
|
||||
}
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
{
|
||||
"category": "production_source",
|
||||
"source_sha256": "cff768f79672cfc7db7f6866caf5ab59936b8c51eb2a7d729b81441940124512",
|
||||
"target_sha256": "d56fe8a934c1ae20a9249c7682823238a705f51f451241504512e83a8800a879",
|
||||
"target_sha256": "c1c0167b96c818a4e7d44aa5c860b5510c9925cb096b7fb4510b8319b91592fc",
|
||||
"transformed": true,
|
||||
"source_relative_path": "analyze_style_with_hermes.py",
|
||||
"target_relative_path": "src/gyxx_flow/modules/product_commerce/analyze_style.py"
|
||||
@@ -54,7 +54,7 @@
|
||||
{
|
||||
"category": "production_source",
|
||||
"source_sha256": "1493dbbfb3d864812df378d04d006e7de0f9ec4b6ab5812e5c6970c304c2dfe6",
|
||||
"target_sha256": "a1d526dfc27b110a8845a98cc66fe7e5cfb87ef70530728477af31a88734a576",
|
||||
"target_sha256": "2b6e6adfea82b13dd8c819138a428a78b5802f204c199c58ed4e59f0158d1780",
|
||||
"transformed": true,
|
||||
"source_relative_path": "collect_dy_market_rank.py",
|
||||
"target_relative_path": "src/gyxx_flow/modules/product_commerce/collect_dy_market_rank.py"
|
||||
@@ -62,7 +62,7 @@
|
||||
{
|
||||
"category": "production_source",
|
||||
"source_sha256": "abd4c82cbd6c41e111e213c174d2fdeba1214568496eaf1dede24bf374b3b052",
|
||||
"target_sha256": "9de3ab22d5a4dbf574b2d4fdbbed295bb4645bb4943f4c066fcbfd41b44b521c",
|
||||
"target_sha256": "eedf83f1d85662dc2e6d968af29070be819c7caac25057c74d6cb66b75f52216",
|
||||
"transformed": true,
|
||||
"source_relative_path": "collect_dy_persona_to_bitable.py",
|
||||
"target_relative_path": "src/gyxx_flow/modules/product_commerce/collect_dy_persona_to_bitable.py"
|
||||
@@ -70,7 +70,7 @@
|
||||
{
|
||||
"category": "production_source",
|
||||
"source_sha256": "c7b54d941ab7f056e2d0ea4355543ff4a5c34ad986c8ed32145e988ac9501ea2",
|
||||
"target_sha256": "aaba5dfe6ce5133b5c662232160bb9faef4d73b36902a484a56b89fa7df05311",
|
||||
"target_sha256": "8576678688d7d90f6f7285110de005982bc1ba82bc7e9977224ba1bdb4101283",
|
||||
"transformed": true,
|
||||
"source_relative_path": "collect_erp_yesterday_metrics.py",
|
||||
"target_relative_path": "src/gyxx_flow/modules/product_commerce/collect_erp_yesterday_metrics.py"
|
||||
@@ -78,7 +78,7 @@
|
||||
{
|
||||
"category": "production_source",
|
||||
"source_sha256": "8bb1f17f90a7b9c2db5dfb992cf746329892d19c00ae2ae988e2109c049598b2",
|
||||
"target_sha256": "c98abe563b223f49cc3c810054d997504d9465387da4805389b0f5ef78ceed6a",
|
||||
"target_sha256": "9b24259854a440d5a13fa2bc397887ab86cf7d4a573b10d3bda6988ed37d0ed6",
|
||||
"transformed": true,
|
||||
"source_relative_path": "collect_jd_market_rank.py",
|
||||
"target_relative_path": "src/gyxx_flow/modules/product_commerce/collect_jd_market_rank.py"
|
||||
@@ -86,7 +86,7 @@
|
||||
{
|
||||
"category": "production_source",
|
||||
"source_sha256": "f9a48c5f89b18472dc3993e340ad858bfecb2fcf59427c7e70db37512c17a6f7",
|
||||
"target_sha256": "a268df592f7136a68c2b82d031115055e43cdfaedcdd4a5f4afa9ef45879e055",
|
||||
"target_sha256": "01798bd6110afc84e5a6aab50255deeb0c5980f77895fcaf605ccb93d24656a5",
|
||||
"transformed": true,
|
||||
"source_relative_path": "collect_jd_persona_to_bitable.py",
|
||||
"target_relative_path": "src/gyxx_flow/modules/product_commerce/collect_jd_persona_to_bitable.py"
|
||||
@@ -110,7 +110,7 @@
|
||||
{
|
||||
"category": "production_source",
|
||||
"source_sha256": "13b6e68b4872b2e79c1de8ca1617cc4f1249f4ed32f234ad6efb7a04f5948612",
|
||||
"target_sha256": "a1b95cc4f090a25c8e7991b0ef015a7df612fc78249dafe75aeef9fa4dbefe08",
|
||||
"target_sha256": "35b5838736306d58758dca870deb7a296a21a2cbc7bb0c4b84d7f233095bee22",
|
||||
"transformed": true,
|
||||
"source_relative_path": "collect_sycm_market_rank.py",
|
||||
"target_relative_path": "src/gyxx_flow/modules/product_commerce/collect_sycm_market_rank.py"
|
||||
@@ -134,7 +134,7 @@
|
||||
{
|
||||
"category": "production_source",
|
||||
"source_sha256": "a8900b9f25db6cf2bef3117a1d2cd86283054fea6f5a8590fd91b5cfbbcaf87e",
|
||||
"target_sha256": "030d33c3f5c0c230bb67d603dd657f40cceff8581f8756a1f67340a89b5ae4b3",
|
||||
"target_sha256": "47cdfd9ebf27e31be06497b2f74c80da4ffced259a873730eb024b0db5eba740",
|
||||
"transformed": true,
|
||||
"source_relative_path": "db/__init__.py",
|
||||
"target_relative_path": "src/gyxx_flow/modules/product_commerce/db/__init__.py"
|
||||
@@ -158,7 +158,7 @@
|
||||
{
|
||||
"category": "production_source",
|
||||
"source_sha256": "78c74c0e52e3651b3b042a2095e596ab250e60309c7ae2de905850822aad4922",
|
||||
"target_sha256": "09d7e08d2eec57055a3bc7949e49c32a8bf62713f2594ae5d9297c032bf212c8",
|
||||
"target_sha256": "1e6ed0029c9d1f701ae1d80b670fdb465cdbfd13b9f63d0a3179c19585215246",
|
||||
"transformed": true,
|
||||
"source_relative_path": "dy_audience_profile_collect.py",
|
||||
"target_relative_path": "src/gyxx_flow/modules/product_commerce/dy_audience_profile_collect.py"
|
||||
@@ -166,7 +166,7 @@
|
||||
{
|
||||
"category": "production_source",
|
||||
"source_sha256": "492916088fa5e8c170461cc9eba97a88ed1cc1e50c7b99e5f6a5e49167fea157",
|
||||
"target_sha256": "9dd0dc3e6f048683157fe8b8deb533ed88970bf34e9824e439bf3a5d52d80e0a",
|
||||
"target_sha256": "7b0ae5d648245e89f4aacf262250009de812d3dbb8ee55386f3704cabca48799",
|
||||
"transformed": true,
|
||||
"source_relative_path": "dy_product_scraping.py",
|
||||
"target_relative_path": "src/gyxx_flow/modules/product_commerce/dy_product_scraping.py"
|
||||
@@ -206,7 +206,7 @@
|
||||
{
|
||||
"category": "production_source",
|
||||
"source_sha256": "906278aa6fb3578d5dab231bdccc2f5a7c6ec8c85810a3e026fff3c7ba16dc90",
|
||||
"target_sha256": "d9e4281c21a42841023c0a723aff4a00af1f9200c94e42add9f8965159d4cfb5",
|
||||
"target_sha256": "c9ac5b78ce367eb0b87697255a88400d1e2e83416ee1fc105110d3abc3dd8245",
|
||||
"transformed": true,
|
||||
"source_relative_path": "import_product_daily.py",
|
||||
"target_relative_path": "src/gyxx_flow/modules/product_commerce/import_product_daily.py"
|
||||
@@ -230,7 +230,7 @@
|
||||
{
|
||||
"category": "production_source",
|
||||
"source_sha256": "e0f016bafb3f527f1cae46eaf3ef31510184128aaaa7f52024a7c3f978eec42b",
|
||||
"target_sha256": "f2692442a1e62a4b20dbb442eb3e27a3075ca6095ba2578c90193f0db3a1810e",
|
||||
"target_sha256": "b157f24430170d5a991bb2283af0a747b26dd11c0f0dec37289774f115132fee",
|
||||
"transformed": true,
|
||||
"source_relative_path": "jd_main_image_collector.py",
|
||||
"target_relative_path": "src/gyxx_flow/modules/product_commerce/jd_main_image_collector.py"
|
||||
@@ -238,7 +238,7 @@
|
||||
{
|
||||
"category": "production_source",
|
||||
"source_sha256": "d99ce71a008128549ac83fc3da52d88540b74538e3fe5c85d47c3b985e414353",
|
||||
"target_sha256": "f821a0221b832bea0123bf4a8f85fd0785eba9206c8a632ad48065ea3c30c095",
|
||||
"target_sha256": "cfe98b322e4ef7bd875325c2a9ffedeabb364e98b2983773dc98f9e0e75a6581",
|
||||
"transformed": true,
|
||||
"source_relative_path": "jd_product_data_collector.py",
|
||||
"target_relative_path": "src/gyxx_flow/modules/product_commerce/jd_product_data_collector.py"
|
||||
@@ -294,7 +294,7 @@
|
||||
{
|
||||
"category": "production_source",
|
||||
"source_sha256": "910e720fab4f47419ad56055f00a8c3fbe3a06c2577d81e35d3d38178a588a45",
|
||||
"target_sha256": "c9f8cfc8dff6a7e5617b9b357ed92872f94371f93360318a3ff78f7ec38d75df",
|
||||
"target_sha256": "e515cdc287b39a315dec44b4fb1c17f98b2af13bf82715731f92919d45f87854",
|
||||
"transformed": true,
|
||||
"source_relative_path": "market_rank_product_import.py",
|
||||
"target_relative_path": "src/gyxx_flow/modules/product_commerce/market_rank_product_import.py"
|
||||
@@ -358,7 +358,7 @@
|
||||
{
|
||||
"category": "production_source",
|
||||
"source_sha256": "c9a51f1d1267b398cf260a86e65a8ee49de873d6c95bf9c90db2abb422005cf7",
|
||||
"target_sha256": "6cd70bb5406bc00257f9070760f980580e95c3c4e4326a67d33d30433dd74c32",
|
||||
"target_sha256": "54c7b5802f66f9892b455aadd4a7396b7b693b2dda729aa7c22f8909821acd17",
|
||||
"transformed": true,
|
||||
"source_relative_path": "run_daily_persona.py",
|
||||
"target_relative_path": "src/gyxx_flow/modules/product_commerce/run_daily_persona.py"
|
||||
@@ -390,7 +390,7 @@
|
||||
{
|
||||
"category": "production_source",
|
||||
"source_sha256": "a1b097ead2a3f7e20ca63d7c15828a8d04030b0d6722934c3bf65d9e975fba76",
|
||||
"target_sha256": "a231ccd2efacefca703a975e6c63e05a36c61a3db0975e46b1fab8d983fe84d2",
|
||||
"target_sha256": "7d8587f77615040087d2d475a7b440ad16a39c5874e143418dd700ecbc5db0f8",
|
||||
"transformed": true,
|
||||
"source_relative_path": "scripts/insert_main_image_records.py",
|
||||
"target_relative_path": "src/gyxx_flow/modules/product_commerce/scripts/insert_main_image_records.py"
|
||||
@@ -406,7 +406,7 @@
|
||||
{
|
||||
"category": "production_source",
|
||||
"source_sha256": "4474dd7d68117247af87e20bbeb6b7c072368ef0f606de5d3e186f459d34a0f5",
|
||||
"target_sha256": "cfbb46483113cda9e4cb301b092ac2ce6a35c5cc53b9a9cef01fdf93a00d3b94",
|
||||
"target_sha256": "862ae9392227113ad4266c2170769595c36b2f98fdecc54e54a648ba834baebf",
|
||||
"transformed": true,
|
||||
"source_relative_path": "taobao_sycm_collect.py",
|
||||
"target_relative_path": "src/gyxx_flow/modules/product_commerce/taobao_sycm_collect.py"
|
||||
@@ -438,7 +438,7 @@
|
||||
{
|
||||
"category": "production_source",
|
||||
"source_sha256": "e4d99956d9649d108baeb3e3a2d1969c227622d1037062bd277af66375556d82",
|
||||
"target_sha256": "981ef2a5a59f73ae244240520320c73be9c9462b50546df1ff91d0c97e7c03f7",
|
||||
"target_sha256": "2ccb359ab943c185b17854f0f00cfb76b8a1235d7b031209a1c07e9386f8d93f",
|
||||
"transformed": true,
|
||||
"source_relative_path": "upload_video_to_guanghe.py",
|
||||
"target_relative_path": "src/gyxx_flow/modules/product_commerce/upload_video_to_guanghe.py"
|
||||
@@ -454,7 +454,7 @@
|
||||
{
|
||||
"category": "production_source",
|
||||
"source_sha256": "cceb3861da0e68c686c0c58f64767c3ca2693986bb5006ca7405d3a5a39c4347",
|
||||
"target_sha256": "8a947ce42d34ff26bb46c5173af9e37ee9d60eac1b595d2f54b855a881ac2d6c",
|
||||
"target_sha256": "4a44a00ec0e13bac3c4369c0336c6b6410e012d59e39900f3b4d0c093c8def90",
|
||||
"transformed": true,
|
||||
"source_relative_path": "vendors/dy-data-flow/dy_store_competitor_store_scraping.py",
|
||||
"target_relative_path": "src/gyxx_flow/modules/product_commerce/vendors/dy-data-flow/dy_store_competitor_store_scraping.py"
|
||||
@@ -488,7 +488,7 @@
|
||||
{
|
||||
"category": "production_source",
|
||||
"source_sha256": "6daa16c03989fb2231e54545726f5aa488e4ba80b9b8aa6f937cb9a2fa13ff35",
|
||||
"target_sha256": "89ba008ef4388c6e1ed32e4a5c26ffa81d083164dd6d7231fc9e5886de8437cd",
|
||||
"target_sha256": "5ea444181ca731d8bdd983fa8d0179422651fdd994ebe08a63ed2bfae155fca5",
|
||||
"transformed": true,
|
||||
"source_relative_path": "vendors/jd-data-flow/jd_data_collector.py",
|
||||
"target_relative_path": "src/gyxx_flow/modules/product_commerce/vendors/jd-data-flow/jd_data_collector.py"
|
||||
@@ -496,7 +496,7 @@
|
||||
{
|
||||
"category": "production_source",
|
||||
"source_sha256": "a53cc38b0ddeaa0065bc27a1d137ce16c2873ff9bec712483ca87bcedd6baa3a",
|
||||
"target_sha256": "0549228539c2e1908c3181de13291457712eeaea8fea89903814844724a3b5ee",
|
||||
"target_sha256": "a6ee61498a040be9d45c6b8f9ecdbc06213f5f1e5507783412188f35d2968640",
|
||||
"transformed": true,
|
||||
"source_relative_path": "vendors/jd-data-flow/jd_peer_product_data_collector.py",
|
||||
"target_relative_path": "src/gyxx_flow/modules/product_commerce/vendors/jd-data-flow/jd_peer_product_data_collector.py"
|
||||
@@ -560,7 +560,7 @@
|
||||
{
|
||||
"category": "runtime_resource",
|
||||
"source_sha256": "9739407b61842c598491d3189a32f5f15aa944aed79a164a5cd5b5a34f20c746",
|
||||
"target_sha256": "8ff84f6c3c672af35e09f70d71aa73656ac101218d8a56126b8b0a75ab387c8b",
|
||||
"target_sha256": "6bbd36048836ef7ed0807223fc028ac17349b20314b1ea353ccedb451957ba55",
|
||||
"transformed": true,
|
||||
"source_relative_path": "db/schema.sql",
|
||||
"target_relative_path": "src/gyxx_flow/modules/product_commerce/db/schema.sql"
|
||||
@@ -672,7 +672,7 @@
|
||||
{
|
||||
"category": "regression_test",
|
||||
"source_sha256": "820e17b910b62a213f0bebafcff72998c2c978307c97f976c074276150ce7c6a",
|
||||
"target_sha256": "1159c9e3f0abdf17183e7d6fa2634c508fe8348ce2926e7d3c58e735b713c2e9",
|
||||
"target_sha256": "cdedf8c9a4871ad763e0a9098fc18777f0c3206991e8d440a45ccdfbda089a07",
|
||||
"transformed": true,
|
||||
"source_relative_path": "tests/test_main_image_concurrency.py",
|
||||
"target_relative_path": "tests/modules/product_commerce/test_main_image_concurrency.py"
|
||||
@@ -696,7 +696,7 @@
|
||||
{
|
||||
"category": "regression_test",
|
||||
"source_sha256": "f1e660d34e21d3cd7a26d3605c82d07982ba6c5002af7b56fb6a876cb706713d",
|
||||
"target_sha256": "85ca8603e2af8257ce2a940b99c69948c2c1362f688e6f6dac4cbf499f1e4d73",
|
||||
"target_sha256": "4a21ff4424c92a0bed6bd2f70f6a7bc7b426b3a25ccc356612f27a7426ce1eca",
|
||||
"transformed": true,
|
||||
"source_relative_path": "tests/test_market_rank_product_import.py",
|
||||
"target_relative_path": "tests/modules/product_commerce/test_market_rank_product_import.py"
|
||||
@@ -728,7 +728,7 @@
|
||||
{
|
||||
"category": "regression_test",
|
||||
"source_sha256": "f9cfbb49fd313444790ac194fbf980dae21912081b9f26ad8e3850912126f09e",
|
||||
"target_sha256": "0cc2b365144482ceb36667eedfb2c403e904437f3ea4c7e12f7099276c13a279",
|
||||
"target_sha256": "043b2dd69172bf5ea1b324bf07f8bba4c43f4bca9817c30a2c805ffe8ce70525",
|
||||
"transformed": true,
|
||||
"source_relative_path": "tests/test_style_analysis_orchestration.py",
|
||||
"target_relative_path": "tests/modules/product_commerce/test_style_analysis_orchestration.py"
|
||||
|
||||
@@ -32,7 +32,7 @@
|
||||
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/collectors/dy_store_competitor_store_scraping.py",
|
||||
"category": "source",
|
||||
"source_sha256": "009879a7dd341ca6ecbc8eb66f5700114c73bcaba3a53f159014a76232d0aa6b",
|
||||
"target_sha256": "d5e85ded1f852ba418229c75c985d0ece967dfbd5fd31697fe407b64d1c3db05",
|
||||
"target_sha256": "9ab1328c2f1743944e8f096e548255884bc9724bf1e900c60c069d443c886902",
|
||||
"transformed": true
|
||||
},
|
||||
{
|
||||
@@ -48,7 +48,7 @@
|
||||
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/collectors/jd_peer_store_data_collector.py",
|
||||
"category": "source",
|
||||
"source_sha256": "4ebdb52df7b384d2bfdd2fc840d55d14e18864002c20a36685b63cee1966cf4c",
|
||||
"target_sha256": "39faf18e9cd5e441323a8e056e022f179018a44eac224d64383cd7ae9fa1d611",
|
||||
"target_sha256": "f03b1a7226071f2225a2a71748ad2c27a102cde67ba02d37c63f15480dd0de32",
|
||||
"transformed": true
|
||||
},
|
||||
{
|
||||
@@ -88,7 +88,7 @@
|
||||
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/db/__init__.py",
|
||||
"category": "source",
|
||||
"source_sha256": "ffc89b78ff0081a129e04eb6de4985ab0a2a4613c9e8a1d8b2750d8c17e8bc05",
|
||||
"target_sha256": "a4bae02d0c07f3e3986896b31e62f41bf15edbf74d3776b1e050e381c15a8413",
|
||||
"target_sha256": "08b5b8920338100339e7c716423b0e9260d4e007d0a1cc3f7510d51970a66a44",
|
||||
"transformed": true
|
||||
},
|
||||
{
|
||||
@@ -96,7 +96,7 @@
|
||||
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/db/db.py",
|
||||
"category": "source",
|
||||
"source_sha256": "168b6f87c1827b267f763ddce7eb15d0c8d652c8c0c7af18dcd0c6c7da5cddd1",
|
||||
"target_sha256": "01469f9eb80aee5e80bc28e548dbe2cd5ce861ab35cd480b33921a2adc799320",
|
||||
"target_sha256": "4ab56df6c8ff75b1a67256ad8cdef34c58bb97e1a126707aac7ccf82980a56ec",
|
||||
"transformed": true
|
||||
},
|
||||
{
|
||||
@@ -104,7 +104,7 @@
|
||||
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/db/schema.sql",
|
||||
"category": "sql",
|
||||
"source_sha256": "73230590bde7ae6359c02c82c2db096c0d3faa55b374e30348c47d7b40ea498b",
|
||||
"target_sha256": "a69fa151f989f106d00f344bab43ec9e6a09cf404bdc581624cc0901cdc13d20",
|
||||
"target_sha256": "9558543cf313aadb785715fca8f5cb99219488e7cb6c37cb00310bfe84f7ee74",
|
||||
"transformed": true
|
||||
},
|
||||
{
|
||||
|
||||
@@ -120,7 +120,7 @@
|
||||
"target_relative_path": "src/gyxx_flow/modules/supply_chain/orchestrator/scripts/ProductReplenishment.py",
|
||||
"category": "source",
|
||||
"source_sha256": "02f6345142ebd84e25dc2f3b0651bc139ef31dd3994ff739efe66344c77ee49e",
|
||||
"target_sha256": "e74211f2fe97ac47e704a040f442f51c97fecc15ea30dd1f0dbeacaf5940fd80",
|
||||
"target_sha256": "ae5154b1f2af1e03307c07e4188225cd9601d1e1b5316fc0f6a78d23303ba8de",
|
||||
"transformed": true
|
||||
},
|
||||
{
|
||||
|
||||
+81
-50
@@ -3,6 +3,11 @@
|
||||
"workflows": [
|
||||
{
|
||||
"id": "content.metrics.daily",
|
||||
"required_accounts": [
|
||||
"douyin-content",
|
||||
"douyin-shop",
|
||||
"xiaohongshu-content"
|
||||
],
|
||||
"module": "content_marketing",
|
||||
"trigger": "scheduled",
|
||||
"execution": {
|
||||
@@ -302,6 +307,11 @@
|
||||
},
|
||||
{
|
||||
"id": "content.metrics.backfill",
|
||||
"required_accounts": [
|
||||
"douyin-content",
|
||||
"douyin-shop",
|
||||
"xiaohongshu-content"
|
||||
],
|
||||
"module": "content_marketing",
|
||||
"trigger": "scheduled",
|
||||
"execution": {
|
||||
@@ -430,55 +440,6 @@
|
||||
"task_name": "YingxiaoYunying_DailyMarketingReport"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "content.relogin.weekly",
|
||||
"module": "content_marketing",
|
||||
"trigger": "scheduled",
|
||||
"execution": {
|
||||
"steps": [
|
||||
{
|
||||
"id": "relogin",
|
||||
"name": "刷新内容平台登录态",
|
||||
"description": "并行启动五个平台的二维码重新登录流程,按配置轮次重试,持久化登录状态并通知结果。",
|
||||
"data_flow": {
|
||||
"sources": [
|
||||
{
|
||||
"system": "内容平台",
|
||||
"label": "五个平台登录页与当前会话状态"
|
||||
}
|
||||
],
|
||||
"processing": [
|
||||
"并行发起二维码登录并按轮次重试",
|
||||
"校验并持久化新的登录会话"
|
||||
],
|
||||
"destinations": [
|
||||
{
|
||||
"system": "浏览器状态",
|
||||
"label": "各平台 Cookie 与 storage state"
|
||||
},
|
||||
{
|
||||
"system": "飞书",
|
||||
"label": "登录刷新结果通知"
|
||||
}
|
||||
]
|
||||
},
|
||||
"entry": "data/tools/friday_relogin_parallel.py",
|
||||
"args": [
|
||||
"--max-attempts",
|
||||
"3",
|
||||
"--round-timeout",
|
||||
"300",
|
||||
"--screenshot-delay",
|
||||
"60"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"provenance": {
|
||||
"source_project": "content",
|
||||
"task_name": "YingxiaoYunying_FridayRelogin"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "content.creator_report.monthly",
|
||||
"module": "content_marketing",
|
||||
@@ -781,6 +742,9 @@
|
||||
},
|
||||
{
|
||||
"id": "shop.metrics.weekly",
|
||||
"required_accounts": [
|
||||
"douyin-shop"
|
||||
],
|
||||
"module": "shop_intelligence",
|
||||
"trigger": "scheduled",
|
||||
"execution": {
|
||||
@@ -890,6 +854,9 @@
|
||||
},
|
||||
{
|
||||
"id": "shop.competitor.weekly",
|
||||
"required_accounts": [
|
||||
"douyin-shop"
|
||||
],
|
||||
"module": "shop_intelligence",
|
||||
"trigger": "scheduled",
|
||||
"execution": {
|
||||
@@ -1592,6 +1559,10 @@
|
||||
},
|
||||
{
|
||||
"id": "product.daily",
|
||||
"required_accounts": [
|
||||
"douyin-shop",
|
||||
"erp-jushuitan"
|
||||
],
|
||||
"module": "product_commerce",
|
||||
"trigger": "scheduled",
|
||||
"execution": {
|
||||
@@ -1916,6 +1887,7 @@
|
||||
],
|
||||
"processing": [
|
||||
"校验广告主 ID、店铺标识和商品数据明细",
|
||||
"任务中心被重定向到账户选择页时自动点击目标广告主后重试",
|
||||
"按业务日期下载原始 Excel",
|
||||
"重置千川错误的 worksheet A1 dimension 并校验日期覆盖"
|
||||
],
|
||||
@@ -2231,6 +2203,54 @@
|
||||
},
|
||||
"note": "每天自动采集昨天数据;控制台可手动选择任意补采日期并幂等覆盖该日。全选聚水潭全部店铺,不拆京东、抖音、天猫;每次只采集一个业务日期。"
|
||||
},
|
||||
{
|
||||
"id": "product.hk_anpei_public_inventory.daily",
|
||||
"module": "product_commerce",
|
||||
"trigger": "scheduled",
|
||||
"execution": {
|
||||
"steps": [
|
||||
{
|
||||
"id": "query_and_refresh_sheet",
|
||||
"name": "刷新香港安沛 SKU 公有可用数",
|
||||
"description": "校验计划书固定 SKU 与现有库存工作表中的商品编码对应关系,调用聚水潭 /open/inventory/query 查询库存及锁定量,并将公有可用数写入仓库现有库存列。",
|
||||
"entry": "public_sku_inventory_sheet.py",
|
||||
"timeout_seconds": 900,
|
||||
"replay_policy": "idempotent",
|
||||
"data_flow": {
|
||||
"sources": [
|
||||
{
|
||||
"system": "光影行星香港安沛2026Q2计划书",
|
||||
"label": "现有库存表的 51 个唯一 SKU"
|
||||
},
|
||||
{
|
||||
"system": "聚水潭开放 API",
|
||||
"label": "/open/inventory/query",
|
||||
"detail": "请求 has_lock_qty=true 以取得库存锁定量"
|
||||
}
|
||||
],
|
||||
"processing": [
|
||||
"校验整批 SKU 库存响应后再写入飞书",
|
||||
"按商品编码(光影)匹配行;公有可用数 = qty - order_lock - lock_qty + virtual_qty",
|
||||
"没有库存记录的 SKU 清空仓库现有库存单元格",
|
||||
"仅写入仓库现有库存列,不覆盖图片、备注、在途生产或其它字段"
|
||||
],
|
||||
"destinations": [
|
||||
{
|
||||
"system": "飞书电子表格",
|
||||
"label": "光影行星香港安沛2026Q2計劃書",
|
||||
"detail": "現有庫存工作表;按商品编码(光影)更新 D 列仓库现有库存"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"provenance": {
|
||||
"source_project": "data-hub",
|
||||
"task_name": "HongKongAnpeiPublicSkuInventoryDaily"
|
||||
},
|
||||
"note": "每天 Asia/Shanghai 09:00 执行。聚水潭应用凭据从 GYXX 调度进程环境读取;不得使用开发配置默认值。"
|
||||
},
|
||||
{
|
||||
"id": "product.video_upload",
|
||||
"module": "product_commerce",
|
||||
@@ -2350,7 +2370,6 @@
|
||||
"args": [
|
||||
"--anchor-record-id",
|
||||
"recvrXTRWWFXeJ",
|
||||
"--manual-login",
|
||||
"--execute"
|
||||
],
|
||||
"timeout_seconds": 21600
|
||||
@@ -2607,6 +2626,9 @@
|
||||
},
|
||||
{
|
||||
"id": "supply.purchase_confirmation.daily",
|
||||
"required_accounts": [
|
||||
"erp-jushuitan"
|
||||
],
|
||||
"module": "supply_chain",
|
||||
"trigger": "scheduled",
|
||||
"execution": {
|
||||
@@ -2657,6 +2679,9 @@
|
||||
},
|
||||
{
|
||||
"id": "supply.replenishment.weekly",
|
||||
"required_accounts": [
|
||||
"erp-jushuitan"
|
||||
],
|
||||
"module": "supply_chain",
|
||||
"trigger": "scheduled",
|
||||
"execution": {
|
||||
@@ -2712,6 +2737,9 @@
|
||||
},
|
||||
{
|
||||
"id": "supply.replenishment_alert.daily",
|
||||
"required_accounts": [
|
||||
"erp-jushuitan"
|
||||
],
|
||||
"module": "supply_chain",
|
||||
"trigger": "scheduled",
|
||||
"execution": {
|
||||
@@ -2811,6 +2839,9 @@
|
||||
},
|
||||
{
|
||||
"id": "product.market_rank",
|
||||
"required_accounts": [
|
||||
"douyin-shop"
|
||||
],
|
||||
"module": "product_commerce",
|
||||
"trigger": "scheduled",
|
||||
"execution": {
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
# GYXX Flow 开机自启:启用/关闭 Windows 自动登录
|
||||
#
|
||||
# 背景:控制台+调度器的自启链是
|
||||
# 开机 → 自动登录进桌面 → Startup\gyxx-keepalive.vbs → keepalive → 控制台(8765)+调度器
|
||||
# 唯一缺口是 AutoAdminLogon(默认 0,重启后停在登录界面,Startup 不触发)。
|
||||
# 采集依赖浏览器 CDP,必须跑在交互式用户会话,因此不能用服务/Task Scheduler SYSTEM 替代。
|
||||
#
|
||||
# 启用(交互输入密码,不会留在命令历史):
|
||||
# powershell -ExecutionPolicy Bypass -File D:\gyxx-flow\deploy\watchdog\autologon.ps1
|
||||
# 撤销:
|
||||
# powershell -ExecutionPolicy Bypass -File D:\gyxx-flow\deploy\watchdog\autologon.ps1 -Disable
|
||||
#
|
||||
# 注意:DefaultPassword 以明文存于 HKLM Winlogon(仅 Administrators 可读),
|
||||
# 内网采集机可接受;介意安全性请改用 Sysinternals Autologon.exe(LSA secret 存储)。
|
||||
param(
|
||||
[switch]$Disable
|
||||
)
|
||||
|
||||
$winlogon = 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon'
|
||||
|
||||
if ($Disable) {
|
||||
Set-ItemProperty $winlogon -Name AutoAdminLogon -Value '0'
|
||||
Remove-ItemProperty $winlogon -Name DefaultPassword -ErrorAction SilentlyContinue
|
||||
Write-Host '已关闭自动登录(DefaultPassword 已清除)。' -ForegroundColor Yellow
|
||||
exit 0
|
||||
}
|
||||
|
||||
$pwd1 = Read-Host -AsSecureString "输入 $env:USERNAME 的 Windows 登录密码"
|
||||
$pwd2 = Read-Host -AsSecureString '再输一次确认'
|
||||
$plain1 = [Net.NetworkCredential]::new('', $pwd1).Password
|
||||
$plain2 = [Net.NetworkCredential]::new('', $pwd2).Password
|
||||
if ($plain1 -ne $plain2) { throw '两次输入不一致,未做任何修改。' }
|
||||
if ([string]::IsNullOrEmpty($plain1)) {
|
||||
Write-Warning '密码为空:策略 LimitBlankPasswordUse=1 会拒绝空密码自动登录,请先给账户设置密码。'
|
||||
exit 1
|
||||
}
|
||||
|
||||
Set-ItemProperty $winlogon -Name AutoAdminLogon -Value '1'
|
||||
Set-ItemProperty $winlogon -Name DefaultUserName -Value $env:USERNAME
|
||||
Set-ItemProperty $winlogon -Name DefaultDomainName -Value $env:COMPUTERNAME
|
||||
Set-ItemProperty $winlogon -Name DefaultPassword -Value $plain1
|
||||
# AutoLogonCount 每次自动登录会递减,删掉表示不限次数
|
||||
Remove-ItemProperty $winlogon -Name AutoLogonCount -ErrorAction SilentlyContinue
|
||||
|
||||
Write-Host '自动登录已启用。' -ForegroundColor Green
|
||||
Write-Host '验证方式:挑空闲时间重启一次,开机应自动进桌面,约 2 分钟内控制台(127.0.0.1:8765)与调度器就位;'
|
||||
Write-Host '也可查看 D:\gyxx-flow\var\logs\keepalive\keepalive.log 出现 keepalive-start / scheduler-restarted / console-restarted。'
|
||||
@@ -0,0 +1,336 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""控制台与调度器的进程外保活(登录自启动,随用户会话常驻)。
|
||||
|
||||
职责(不承载任何业务调度规则,业务调度仍由 `gyxx schedule run` 唯一负责):
|
||||
1. 控制台:探测 http://127.0.0.1:8765/api/health——进程不在则拉起;
|
||||
进程在但探测连续失败(wedged)则结束该进程树后拉起。控制台进程内
|
||||
另有看门狗(console.py `_console_watchdog`)自检失败会主动退出,此处
|
||||
负责把退出后的空窗补上,两层互为补充。
|
||||
2. 调度器:读 `var/state/scheduler/service.lock` 的 PID 判断存活,锁主
|
||||
已死则按 install.ps1 同款参数拉起 `python -m gyxx_flow schedule run`;
|
||||
调度器自身单实例锁保证不会双跑。
|
||||
3. 节流:每个目标有冷却间隔;短时间反复重启触发熔断,只记录不再拉起,
|
||||
避免故障风暴。
|
||||
|
||||
用法:
|
||||
gyxx_keepalive.py # 常驻循环(Startup 的 vbs 以 pythonw 启动)
|
||||
gyxx_keepalive.py --once # 单轮探测+处置后退出,供验收/手动巡检
|
||||
|
||||
日志:var/logs/keepalive/keepalive.log(追加)。
|
||||
自身单实例:var/state/keepalive.lock(PID 存活检测,重复启动直接退出)。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import ctypes
|
||||
import http.client
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[2]
|
||||
ENV_FILES = [
|
||||
Path(os.getenv("GYXX_RUNTIME_ENV_FILE", r"D:\product-collector-analyze-flow\.env")),
|
||||
Path(
|
||||
os.getenv(
|
||||
"GYXX_RUNTIME_JUSHUITAN_ENV_FILE",
|
||||
r"D:\product-collector-analyze-flow\private-secrets\jushuitan.env",
|
||||
)
|
||||
),
|
||||
]
|
||||
CONSOLE_HOST = "127.0.0.1"
|
||||
CONSOLE_PORT = int(os.getenv("GYXX_CONSOLE_PORT", "8765"))
|
||||
LOOP_INTERVAL_SECONDS = float(os.getenv("GYXX_KEEPALIVE_INTERVAL_SECONDS", "30"))
|
||||
PROBE_TIMEOUT_SECONDS = 5.0
|
||||
PROBE_FAILURE_LIMIT = 3
|
||||
RESTART_COOLDOWN_SECONDS = float(os.getenv("GYXX_KEEPALIVE_COOLDOWN_SECONDS", "120"))
|
||||
TRIP_MAX_RESTARTS = 5
|
||||
TRIP_WINDOW_SECONDS = 30 * 60
|
||||
TRIP_PAUSE_SECONDS = 60 * 60
|
||||
|
||||
DATA_ROOT = PROJECT_ROOT / "var"
|
||||
LOG_PATH = DATA_ROOT / "logs" / "keepalive" / "keepalive.log"
|
||||
LOCK_PATH = DATA_ROOT / "state" / "keepalive.lock"
|
||||
CONSOLE_LOG_DIR = DATA_ROOT / "logs" / "console-restarts"
|
||||
SCHEDULER_LOG_DIR = DATA_ROOT / "logs" / "scheduler-service"
|
||||
SCHEDULER_SERVICE_LOCK = DATA_ROOT / "state" / "scheduler" / "service.lock"
|
||||
|
||||
|
||||
def _log(event: str, detail: str = "") -> None:
|
||||
stamp = datetime.now(timezone.utc).astimezone().strftime("%Y-%m-%d %H:%M:%S")
|
||||
line = f"{stamp} {event} {detail}".rstrip()
|
||||
LOG_PATH.parent.mkdir(parents=True, exist_ok=True)
|
||||
with LOG_PATH.open("a", encoding="utf-8", newline="\n") as stream:
|
||||
stream.write(line + "\n")
|
||||
print(line, flush=True)
|
||||
|
||||
|
||||
def _pid_alive(pid: int) -> bool:
|
||||
if pid <= 0:
|
||||
return False
|
||||
kernel32 = ctypes.windll.kernel32
|
||||
handle = kernel32.OpenProcess(0x1000, False, pid) # PROCESS_QUERY_LIMITED_INFORMATION
|
||||
if not handle:
|
||||
return False
|
||||
try:
|
||||
exit_code = ctypes.c_ulong()
|
||||
if kernel32.GetExitCodeProcess(handle, ctypes.byref(exit_code)):
|
||||
return exit_code.value == 259 # STILL_ACTIVE
|
||||
return False
|
||||
finally:
|
||||
kernel32.CloseHandle(handle)
|
||||
|
||||
|
||||
def _acquire_self_lock() -> bool:
|
||||
LOCK_PATH.parent.mkdir(parents=True, exist_ok=True)
|
||||
if LOCK_PATH.is_file():
|
||||
try:
|
||||
existing = json.loads(LOCK_PATH.read_text(encoding="utf-8"))
|
||||
if _pid_alive(int(existing.get("pid", 0))):
|
||||
return False
|
||||
except Exception:
|
||||
pass
|
||||
LOCK_PATH.write_text(
|
||||
json.dumps({"pid": os.getpid(), "started_at": datetime.now(timezone.utc).isoformat()}),
|
||||
encoding="utf-8",
|
||||
)
|
||||
return True
|
||||
|
||||
|
||||
def _console_env_files() -> list[Path]:
|
||||
return [path for path in ENV_FILES if path.is_file()]
|
||||
|
||||
|
||||
def _spawn(argv: list[str], log_dir: Path, log_prefix: str) -> int:
|
||||
log_dir.mkdir(parents=True, exist_ok=True)
|
||||
stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
|
||||
log_path = log_dir / f"{log_prefix}-{stamp}.log"
|
||||
kwargs: dict[str, object] = {}
|
||||
if os.name == "nt":
|
||||
kwargs["creationflags"] = (
|
||||
subprocess.CREATE_NO_WINDOW | subprocess.CREATE_NEW_PROCESS_GROUP
|
||||
)
|
||||
else:
|
||||
kwargs["start_new_session"] = True
|
||||
with log_path.open("a", encoding="utf-8", newline="\n") as stream:
|
||||
process = subprocess.Popen(
|
||||
argv,
|
||||
cwd=str(PROJECT_ROOT),
|
||||
env=os.environ,
|
||||
stdin=subprocess.DEVNULL,
|
||||
stdout=stream,
|
||||
stderr=subprocess.STDOUT,
|
||||
shell=False,
|
||||
**kwargs,
|
||||
)
|
||||
return process.pid
|
||||
|
||||
|
||||
def _console_argv() -> list[str]:
|
||||
argv = [
|
||||
str(PROJECT_ROOT / ".venv" / "Scripts" / "python.exe"),
|
||||
"-m",
|
||||
"gyxx_flow",
|
||||
"console",
|
||||
"--host",
|
||||
CONSOLE_HOST,
|
||||
"--port",
|
||||
str(CONSOLE_PORT),
|
||||
]
|
||||
for env_file in _console_env_files():
|
||||
argv.extend(("--env-file", str(env_file)))
|
||||
return argv
|
||||
|
||||
|
||||
def _scheduler_argv() -> list[str]:
|
||||
argv = [
|
||||
str(PROJECT_ROOT / ".venv" / "Scripts" / "python.exe"),
|
||||
"-m",
|
||||
"gyxx_flow",
|
||||
"schedule",
|
||||
"run",
|
||||
]
|
||||
for env_file in _console_env_files():
|
||||
argv.extend(("--env-file", str(env_file)))
|
||||
return argv
|
||||
|
||||
|
||||
def _probe_console() -> bool:
|
||||
try:
|
||||
connection = http.client.HTTPConnection(CONSOLE_HOST, CONSOLE_PORT, timeout=PROBE_TIMEOUT_SECONDS)
|
||||
try:
|
||||
connection.request("GET", "/api/health")
|
||||
response = connection.getresponse()
|
||||
response.read()
|
||||
return response.status == 200
|
||||
finally:
|
||||
connection.close()
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
def _port_owner_pid(port: int) -> int | None:
|
||||
try:
|
||||
output = subprocess.run(
|
||||
["netstat", "-ano", "-p", "tcp"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10,
|
||||
).stdout
|
||||
except Exception:
|
||||
return None
|
||||
needle = f"{CONSOLE_HOST}:{port}"
|
||||
for line in output.splitlines():
|
||||
parts = line.split()
|
||||
if (
|
||||
len(parts) >= 5
|
||||
and parts[0].upper() == "TCP"
|
||||
and parts[1] == needle
|
||||
and parts[3].upper() == "LISTENING"
|
||||
):
|
||||
try:
|
||||
return int(parts[4])
|
||||
except ValueError:
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
def _kill_pid_tree(pid: int) -> None:
|
||||
if pid <= 0 or pid == os.getpid():
|
||||
return
|
||||
subprocess.run(
|
||||
["taskkill", "/PID", str(pid), "/T", "/F"],
|
||||
capture_output=True,
|
||||
timeout=30,
|
||||
)
|
||||
|
||||
|
||||
def _scheduler_lock_pid() -> int | None:
|
||||
try:
|
||||
payload = json.loads(SCHEDULER_SERVICE_LOCK.read_text(encoding="utf-8"))
|
||||
return int(payload.get("pid", 0)) or None
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
class _TargetState:
|
||||
"""记录单目标的连续失败计数、冷却与熔断状态。"""
|
||||
|
||||
def __init__(self, name: str) -> None:
|
||||
self.name = name
|
||||
self.failures = 0
|
||||
self.last_restart_at = 0.0
|
||||
self.recent_restarts: list[float] = []
|
||||
self.tripped_at: float | None = None
|
||||
|
||||
def tripped(self, now: float) -> bool:
|
||||
if self.tripped_at is None:
|
||||
return False
|
||||
if now - self.tripped_at >= TRIP_PAUSE_SECONDS:
|
||||
self.tripped_at = None
|
||||
self.recent_restarts.clear()
|
||||
_log("keepalive-trip-reset", f"target={self.name} 熔断窗口结束恢复守护")
|
||||
return False
|
||||
return True
|
||||
|
||||
def may_restart(self, now: float) -> bool:
|
||||
if self.tripped(now):
|
||||
return False
|
||||
if now - self.last_restart_at < RESTART_COOLDOWN_SECONDS:
|
||||
return False
|
||||
self.recent_restarts = [
|
||||
stamp for stamp in self.recent_restarts if now - stamp < TRIP_WINDOW_SECONDS
|
||||
]
|
||||
if len(self.recent_restarts) >= TRIP_MAX_RESTARTS:
|
||||
self.tripped_at = now
|
||||
_log(
|
||||
"keepalive-tripped",
|
||||
f"target={self.name} {TRIP_MAX_RESTARTS} 次重启/{TRIP_WINDOW_SECONDS // 60}分钟 "
|
||||
f"暂停拉起{TRIP_PAUSE_SECONDS // 60}分钟,请人工检查日志",
|
||||
)
|
||||
return False
|
||||
return True
|
||||
|
||||
def record_restart(self, now: float) -> None:
|
||||
self.last_restart_at = now
|
||||
self.recent_restarts.append(now)
|
||||
|
||||
|
||||
def run_cycle(console: _TargetState, scheduler: _TargetState) -> None:
|
||||
now = time.time()
|
||||
|
||||
# ---- 控制台 ----
|
||||
if _probe_console():
|
||||
console.failures = 0
|
||||
else:
|
||||
console.failures += 1
|
||||
owner = _port_owner_pid(CONSOLE_PORT)
|
||||
_log(
|
||||
"console-probe-failed",
|
||||
f"{console.failures}/{PROBE_FAILURE_LIMIT} port_owner_pid={owner}",
|
||||
)
|
||||
if console.failures >= PROBE_FAILURE_LIMIT and console.may_restart(now):
|
||||
if owner is not None and _pid_alive(owner):
|
||||
_log("console-kill-wedged", f"pid={owner} 探测持续失败,结束进程树")
|
||||
_kill_pid_tree(owner)
|
||||
time.sleep(3)
|
||||
pid = _spawn(_console_argv(), CONSOLE_LOG_DIR, "keepalive-console")
|
||||
console.record_restart(time.time())
|
||||
console.failures = 0
|
||||
_log("console-restarted", f"pid={pid}")
|
||||
|
||||
# ---- 调度器 ----
|
||||
scheduler_pid = _scheduler_lock_pid()
|
||||
scheduler_alive = scheduler_pid is not None and _pid_alive(scheduler_pid)
|
||||
if not scheduler_alive and scheduler.may_restart(now):
|
||||
pid = _spawn(_scheduler_argv(), SCHEDULER_LOG_DIR, "keepalive-start")
|
||||
scheduler.record_restart(time.time())
|
||||
_log(
|
||||
"scheduler-restarted",
|
||||
f"pid={pid} prev_lock_pid={scheduler_pid}",
|
||||
)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(description="GYXX Flow 控制台/调度器保活")
|
||||
parser.add_argument("--once", action="store_true", help="单轮探测+处置后退出")
|
||||
arguments = parser.parse_args(argv)
|
||||
|
||||
if not _acquire_self_lock():
|
||||
_log("keepalive-duplicate-exit", "已有保活进程在运行")
|
||||
return 0
|
||||
|
||||
console = _TargetState("console")
|
||||
scheduler = _TargetState("scheduler")
|
||||
missing_env = [str(path) for path in ENV_FILES if not path.is_file()]
|
||||
_log(
|
||||
"keepalive-start",
|
||||
f"pid={os.getpid()} once={arguments.once}"
|
||||
+ (f" missing_env={missing_env}" if missing_env else ""),
|
||||
)
|
||||
try:
|
||||
while True:
|
||||
run_cycle(console, scheduler)
|
||||
if arguments.once:
|
||||
_log("keepalive-once-done", "")
|
||||
return 0
|
||||
time.sleep(LOOP_INTERVAL_SECONDS)
|
||||
except KeyboardInterrupt:
|
||||
return 0
|
||||
finally:
|
||||
try:
|
||||
if LOCK_PATH.is_file():
|
||||
payload = json.loads(LOCK_PATH.read_text(encoding="utf-8"))
|
||||
if int(payload.get("pid", 0)) == os.getpid():
|
||||
LOCK_PATH.unlink()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,5 @@
|
||||
' GYXX Flow 保活登录自启动:无窗口启动 deploy/watchdog/gyxx_keepalive.py,
|
||||
' 由它负责拉起控制台(含进程内自检看门狗)与调度器。
|
||||
' 安装:复制到 %APPDATA%\Microsoft\Windows\Start Menu\Programs\Startup\
|
||||
CreateObject("WScript.Shell").Run _
|
||||
"""D:\gyxx-flow\.venv\Scripts\pythonw.exe"" ""D:\gyxx-flow\deploy\watchdog\gyxx_keepalive.py""", 0, False
|
||||
@@ -4,6 +4,8 @@
|
||||
param(
|
||||
[string]$ProjectRoot = (Resolve-Path (Join-Path $PSScriptRoot "..\..")).Path,
|
||||
[string]$DataRoot = "",
|
||||
[string]$RuntimeEnvFile = "D:\product-collector-analyze-flow\.env",
|
||||
[string]$JushuitanEnvFile = "D:\product-collector-analyze-flow\private-secrets\jushuitan.env",
|
||||
[string]$ServiceName = "gyxx-flow-scheduler",
|
||||
[string]$NssmPath = "nssm.exe"
|
||||
)
|
||||
@@ -12,6 +14,13 @@ $ErrorActionPreference = "Stop"
|
||||
Set-StrictMode -Version Latest
|
||||
|
||||
$project = (Resolve-Path -LiteralPath $ProjectRoot).Path
|
||||
$runtimeEnvPath = (Resolve-Path -LiteralPath $RuntimeEnvFile -ErrorAction Stop).Path
|
||||
$runtimeEnvPaths = @($runtimeEnvPath)
|
||||
if (Test-Path -LiteralPath $JushuitanEnvFile -PathType Leaf) {
|
||||
$runtimeEnvPaths += (Resolve-Path -LiteralPath $JushuitanEnvFile -ErrorAction Stop).Path
|
||||
} else {
|
||||
Write-Warning "Jushuitan environment file not found; Jushuitan workflows will fail safely until it is provisioned."
|
||||
}
|
||||
if ([string]::IsNullOrWhiteSpace($DataRoot)) {
|
||||
$DataRoot = Join-Path $project "var"
|
||||
}
|
||||
@@ -33,7 +42,9 @@ function Invoke-Nssm([string[]]$Arguments) {
|
||||
|
||||
Invoke-Nssm -Arguments @("install", $ServiceName, $python)
|
||||
Invoke-Nssm -Arguments @("set", $ServiceName, "AppDirectory", $project)
|
||||
Invoke-Nssm -Arguments @("set", $ServiceName, "AppParameters", "-m gyxx_flow schedule run")
|
||||
$envFileArguments = $runtimeEnvPaths | ForEach-Object { '--env-file "' + $_ + '"' }
|
||||
$schedulerArguments = '-m gyxx_flow schedule run ' + ($envFileArguments -join ' ')
|
||||
Invoke-Nssm -Arguments @("set", $ServiceName, "AppParameters", $schedulerArguments)
|
||||
Invoke-Nssm -Arguments @(
|
||||
"set", $ServiceName, "AppEnvironmentExtra",
|
||||
"GYXX_PROJECT_ROOT=$project", "GYXX_DATA_ROOT=$data", "PYTHONUTF8=1"
|
||||
|
||||
@@ -46,6 +46,27 @@ Scrapling 安装和管理。具体边界与验证命令见
|
||||
`EnvironmentFile=` 或当前进程环境优先于文件中的同名值,避免本地文件意外覆盖密钥系统
|
||||
注入值。
|
||||
|
||||
## 香港安沛 SKU 公有库存同步
|
||||
|
||||
`product.hk_anpei_public_inventory.daily` 每天按 `Asia/Shanghai` 09:00 查询计划书中的固定 SKU 清单,
|
||||
调用聚水潭 `/open/inventory/query`,并按商品编码(光影)更新飞书电子表格「現有庫存」工作表的
|
||||
「仓库现有库存」列。它要求调度进程环境提供
|
||||
`JUSHUITAN_APP_KEY` 和 `JUSHUITAN_APP_SECRET`;`JUSHUITAN_BASE_URL` 可选,默认使用聚水潭官方 OpenAPI
|
||||
地址。没有密钥时工作流会在查询前失败,不会把库存留成 0 或改写多维表格。
|
||||
|
||||
Windows NSSM 调度服务从受保护的运行时环境文件读取凭据。安装器默认读取通用运行环境文件
|
||||
`D:\product-collector-analyze-flow\.env`,以及独立的聚水潭密钥文件
|
||||
`D:\product-collector-analyze-flow\private-secrets\jushuitan.env`。两个文件都按服务账户实际身份限制 ACL;
|
||||
可分别通过 `-RuntimeEnvFile` 和 `-JushuitanEnvFile` 指定其他路径:
|
||||
|
||||
```powershell
|
||||
.\deploy\windows-service\install.ps1 -ProjectRoot D:\gyxx-flow -DataRoot D:\gyxx-flow-data -RuntimeEnvFile D:\product-collector-analyze-flow\.env -JushuitanEnvFile D:\product-collector-analyze-flow\private-secrets\jushuitan.env
|
||||
```
|
||||
|
||||
仅将这两个变量的正式聚水潭应用值写入服务账户可读的环境文件,并限制文件 ACL;不要使用开发配置中的默认值,
|
||||
也不要把真实值写进 Git、PowerShell 命令参数、服务日志或文档。已安装的服务需要让调度进程重新读取环境文件后,
|
||||
新计划才会使用这些值。Linux 部署则将相同变量放入权限为 `0640` 的 `EnvironmentFile`。
|
||||
|
||||
## 三平台电商费用日报部署边界
|
||||
|
||||
`product.ecommerce_costs.daily` 是包含天猫万相台、京东京准通和抖音千川分支的电商费用工作流;三个
|
||||
@@ -197,3 +218,12 @@ sudo systemctl start gyxx-flow.service
|
||||
## Windows 兼容入口
|
||||
|
||||
`deploy/windows-service/` 暂时保留为 legacy NSSM 兼容入口。NSSM 只守护同一个 `gyxx schedule run` 进程,不注册 Windows Task Scheduler,也不保存业务时间规则。新服务器部署以 Linux systemd 为准。
|
||||
|
||||
## Windows 工作站保活(控制台 + 调度器自检自启动)
|
||||
|
||||
交互式 Windows 工作站(无法用 systemd/NSSM 常驻服务时)使用 `deploy/watchdog/gyxx_keepalive.py` 做进程外保活:
|
||||
|
||||
- **控制台自检**:控制台进程内置看门狗线程(`console.py` `_console_watchdog`),周期探测自身 `GET /api/health`;连续失败(默认 3 次,`GYXX_CONSOLE_WATCHDOG_FAILURES` / `GYXX_CONSOLE_WATCHDOG_INTERVAL_SECONDS` 可调,`GYXX_CONSOLE_WATCHDOG=0` 关闭)即以退出码 70 自杀,交给保活拉起。`/api/health` 不鉴权、只报进程 PID、uptime 与调度锁存活,供探针使用;API 500 现在会把完整堆栈落到 stderr(随启动重定向进 `var/logs/console-restarts/`)。
|
||||
- **保活脚本**:登录自启动(Startup 目录 `gyxx-keepalive.vbs` → `pythonw gyxx_keepalive.py`)后常驻,每 30 秒一轮:控制台探测连续失败或端口空置则结束僵死进程树并按 `--env-file` 同款参数拉起;读 `var/state/scheduler/service.lock` 判定调度器存活,锁主死亡即拉起 `gyxx schedule run`(调度器单实例锁保证不双跑)。单目标冷却 120 秒;30 分钟内 5 次重启触发熔断,暂停拉起 1 小时并记日志待人工检查。日志在 `var/logs/keepalive/keepalive.log`。
|
||||
- **验收**:`python deploy/watchdog/gyxx_keepalive.py --once` 单轮探测+处置;健康态应只输出 `keepalive-start`/`keepalive-once-done`。
|
||||
- 该脚本只做进程可用性守护,不承载任何业务时间规则(唯一调度路径仍是 `gyxx schedule run`);未注册 Windows Task Scheduler。注意:Startup 方式要求用户已登录——机器重启后需人工登录一次,之后控制台与调度器会在约 2 分钟内自动就位。
|
||||
+37
-2
@@ -161,6 +161,19 @@ uv run gyxx schedule run --env-file /etc/gyxx-flow/gyxx-flow.env
|
||||
|
||||
需要停用单个工作流时,优先修改该工作流的调度启用状态,先 dry-run,再重启服务。其他模块继续运行。
|
||||
|
||||
### 开机自启(Windows 生产机)
|
||||
|
||||
重启/关机开机后的自动恢复链路:**Windows 自动登录 → `Startup\gyxx-keepalive.vbs` → `deploy/watchdog/gyxx_keepalive.py` → 控制台(8765) + `gyxx schedule run` 调度器**。keepalive 每 30 秒巡检并负责拉起死掉的控制台/调度器(单实例锁防双跑,冷却 120s、熔断保护),日志在 `var/logs/keepalive/keepalive.log`,验收用 `python deploy/watchdog/gyxx_keepalive.py --once`。
|
||||
|
||||
采集依赖浏览器 CDP,必须运行在交互式用户会话中,因此不能改成 Windows 服务或 Task Scheduler SYSTEM 任务(也不允许注册 Windows 任务计划)。免登录自启的唯一环节是自动登录,用脚本启用(密码交互输入、写入 `HKLM\...\Winlogon`,仅 Administrators 可读):
|
||||
|
||||
```powershell
|
||||
powershell -ExecutionPolicy Bypass -File D:\gyxx-flow\deploy\watchdog\autologon.ps1 # 启用
|
||||
powershell -ExecutionPolicy Bypass -File D:\gyxx-flow\deploy\watchdog\autologon.ps1 -Disable # 撤销
|
||||
```
|
||||
|
||||
启用后重启验证:开机自动进桌面,约 2 分钟内控制台与调度器就位;停机窗内到期任务由调度器 catch-up 自动补跑。
|
||||
|
||||
## 数据和浏览器状态
|
||||
|
||||
生产 `GYXX_DATA_ROOT` 必须位于源码目录之外,Linux 推荐 `/var/lib/gyxx-flow`:
|
||||
@@ -190,16 +203,38 @@ uv run gyxx schedule run --env-file /etc/gyxx-flow/gyxx-flow.env
|
||||
- `tmall-ozko`:商品经营日报 ozko 账号,900 秒;
|
||||
- `xiaohongshu-content`:小红书,900 秒;
|
||||
- `douyin-content`:抖音内容侧,900 秒;
|
||||
- `xingtu`:星图,900 秒;
|
||||
- `chanmama`:蝉妈妈,900 秒;
|
||||
- `chanmama`:蝉妈妈,900 秒(脚本自带账密登录);
|
||||
- `bilibili-content`:B 站,900 秒。
|
||||
|
||||
星图/百应采集(`xingtu_scraper_v2.py`)的登录态挂在 `douyin-shop` 账号下;`xingtu` 账号当前无成员脚本,仅保留 vault。
|
||||
|
||||
商品经营日报 `product_commerce:taobao_sycm_collect.py` 会在同一次运行中按品牌路由账号:光影行星使用 `tmall-shop`,ozko 使用 `tmall-ozko`,两边各自使用独立 Profile 和账号 vault。`tmall-ozko` 没有静态成员脚本是有意设计,因为它由这个按品牌拆分的包装工作流动态选择;登录态成功后仍会回写 `state/accounts/tmall-ozko/`,由同一套 keepalive 负责续期。
|
||||
|
||||
账号需要在 `config/runtime-bindings.json` 的 `accounts.<account_id>.keepalive` 中显式开启续期,并配置一个不会产生业务副作用的已登录页面 URL。所有账号由唯一的 `gyxx schedule run` 常驻进程在后台维护;每个账号使用独立 CDP 端口、独立 Profile 和账号级 Cookie vault,启动后按 `initial_delay_seconds` 错峰访问,避免同时打开多个登录页面。访问成功且没有跳转到登录页时,Scrapling 才会把最新 Cookie/storage state 原子写回 `state/accounts/<account_id>/`,并同步给成员脚本;不需要另建 Windows 任务计划、cron 或 timer。
|
||||
|
||||
Cookie 续期只能延长站点支持滑动续期的会话,不能突破服务端硬过期、主动退出、风控或验证码。keepalive 检测到登录页/无效 Cookie 时会停止发布,不会用失效状态覆盖最后一份有效 vault;此时按 `gyxx accounts list` 检查 `keepalive.status`,必要时重新执行 `gyxx accounts login <account_id>`。
|
||||
|
||||
### 工作流执行前的 Cookie 健康网关
|
||||
|
||||
每个正式执行(非 dry-run)的工作流在采集步骤前会自动注入"每账号一个"的 `cookie-gate.<account_id>` 关键步骤,由 `src/gyxx_flow/cookie_gate.py` 实现按账号分三级的状态机:
|
||||
|
||||
- **blocking**(纯扫码/验证码账号:`douyin-shop`、`xiaohongshu-content`、`douyin-content`、`bilibili-content` 等):先看 keepalive 近期成功状态(默认 1 小时内视为新鲜,可用 `GYXX_COOKIE_GATE_MAX_STATUS_AGE_SECONDS` 调整),新鲜且 vault 静态有效则直接放行;否则用 keepalive 同款无头浏览器真实探活。探活失败即判定过期:gate 步骤失败,依赖它的采集步骤记 `dependency-failed` 跳过、工作流失败退出,并通过飞书通知登录负责人(`config/notification-routing.json` 中手机号为 `+8613663785145` 的人员,默认 30 分钟节流,可用 `GYXX_LOGIN_ALERT_OPEN_IDS` 覆盖收件人)执行 `uv run gyxx accounts login <account_id>` 完成扫码/验证码重登。
|
||||
- **recover**(配置了 `login_command_id` 的账密自动重登账号:`erp-jushuitan`):过期时先自动跑账密登录恢复,恢复成功放行;失败才阻断并飞书通知。
|
||||
- **defer**(成员脚本自带账号密码登录的账号:京东系、天猫系、`chanmama`):不注入 gate,保持脚本内账密自动登录流程不变。
|
||||
|
||||
按账号划分的持久状态在 `state/accounts/<id>/cookie_state.json`(`healthy` ↔ `awaiting_login`)。编排器型工作流(入口脚本本身无账号绑定、由子采集脚本使用账号)在 `config/workflows.json` 的 `required_accounts` 字段声明依赖账号。`--rerun-step`/`--resume-from` 切片和 shadow 运行不发送通知;dry-run 完全不注入 gate。设置 `GYXX_COOKIE_GATE=0` 可临时关闭整网关。
|
||||
|
||||
采集脚本不再内嵌扫码/验证码登录:运行中检测到登录态失效会以 `CookieExpiredError` 快速失败并按退出码 75 记为 cookie 跳过;`self_xhs_scraper.py --login-only` 是唯一保留的人工登录引导(该脚本不在任何账号成员内,自管绑定状态)。
|
||||
|
||||
### 登录恢复后的自动重跑
|
||||
|
||||
gate 阻断工作流时会把待重跑清单(工作流 + 业务日期,shadow/验收运行不记录,同工作流同日期去重,上限 20 条)写进 `state/accounts/<id>/cookie_state.json` 的 `pending_reruns`。常驻调度器(`gyxx schedule run`)的维护循环里挂着 `CookieGateRecoveryManager`:
|
||||
|
||||
- 每个心跳检查有待重跑记录的账号:keepalive 近期成功 + vault 静态有效 → 立即视为恢复;否则每 5 分钟做一次真实探活(mode C 账号会自动账密重登,重登成功同样触发恢复)。
|
||||
- 恢复确认后,用调度器同款启动器按**原业务日期**自动重跑全部待重跑工作流(日志在 `data/logs/scheduler/`),清空清单并飞书通知"登录态已恢复 + 已自动重跑清单"。
|
||||
- 启动失败的任务保留在清单里下一轮重试;同一账号两次重跑启动之间有 5 分钟冷却;控制台手动脚本类临时运行 id(`script.*`)不会自动重跑。
|
||||
- 与正常调度并发安全:工作流级锁 + EffectLedger 幂等保证同一天不会重复产生副作用。
|
||||
|
||||
项目不会自动搬动或删除已有 `var/`。迁移数据根时应停止调度器,完整备份和复制原目录,修改环境变量后依次运行 doctor、dry-run 和一次指定日期的对账执行。
|
||||
|
||||
## 清理和保留
|
||||
|
||||
@@ -1,89 +0,0 @@
|
||||
# 智能工作台设计(dsh-refact 分支)
|
||||
|
||||
> 状态:v1 已实现。本文档记录「以 deepseek-harness 为 agent 底座、gyxx-flow 工作流插件化」
|
||||
> 的架构决策与边界。使用说明见 [../workbench/README.md](../workbench/README.md)。
|
||||
|
||||
## 目标与约束
|
||||
|
||||
- **底座**:deepseek-harness(`dsh`)Web UI + DeepSeek 模型 = 智能体运行时;
|
||||
gyxx-flow 保持唯一的工作流执行/调度事实来源(LangGraph 引擎、`RunJournal`、
|
||||
`LockManager`、`EffectLedger` 契约不变)。
|
||||
- **插件化**:不重写工作流;通过 dsh「everything-is-a-plugin」扩展点把现有
|
||||
控制台能力投影为智能体工具与 UI 面板。
|
||||
- **UI**:在原 dsh Web UI 侧边栏追加工作流入口(`sidebar.footer.action` +
|
||||
`shell.overlay` 两个**追加型**插槽,不替换任何内置区域),选中工作流即可
|
||||
提问 / 诊断 / 修复。
|
||||
- **生产约束**(AGENTS.md):`gyxx schedule run` 仍是唯一生产调度路径;
|
||||
正式执行保持 `execute + confirmed` 双确认;控制台仍以 `--env-file` 注入云端凭据
|
||||
(启动脚本默认 `D:\product-collector-analyze-flow\.env`)。
|
||||
|
||||
## 为什么不把 gyxx-flow 改写进 dsh monorepo
|
||||
|
||||
dsh 是 pnpm monorepo(host/client 双聚合、Typert 远程契约、自有构建链)。
|
||||
把 Python 工作流引擎迁进去既不可能也无必要。dsh 的外部插件机制
|
||||
(`--patch` cordis.yml + 绝对路径插件 + `dsh.client` 包声明)就是为这种
|
||||
「外部系统接入」设计的。我们选择**进程外集成**:
|
||||
|
||||
- gyxx-flow 控制台 HTTP API 是唯一集成面(已含脱敏、并发守卫、写操作令牌);
|
||||
- 工作台插件只是控制台的客户端 + 智能体能力注册器;
|
||||
- 控制台挂了,dsh 照常可用;dsh 挂了,调度器照常跑。
|
||||
|
||||
## 组件
|
||||
|
||||
### 1. 控制台诊断端点(Python,`src/gyxx_flow/console.py`)
|
||||
|
||||
| 端点 | 说明 |
|
||||
| --- | --- |
|
||||
| `GET /api/workflows/{id}` | 单工作流详情(overview 投影 + `schedule_revision`) |
|
||||
| `GET /api/workflows/{id}/runs/{run_id}` | 单次运行 + 步骤明细 |
|
||||
| `GET /api/workflows/{id}/runs/{run_id}/diagnosis` | 诊断包:运行 + journal trace + 脱敏日志尾部 |
|
||||
|
||||
诊断包日志来源:`run.json` 的 `trace.paths.log` 目录 + 运行时间窗内的控制台日志
|
||||
(`logs/console/{workflow_id}-*.log`),每个文件限读尾部 256KB,经 `_sanitize_error`
|
||||
脱敏并截断,单响应最多 5 个文件。`run_id` 与 `workflow_id` 强制配对(404 不泄露
|
||||
跨工作流记录)。
|
||||
|
||||
### 2. 宿主插件(`workbench/plugin/gyxx-workbench.mjs`,零运行时依赖)
|
||||
|
||||
- **7 个工具**(原始 JSON-Schema `ToolDefinition`,不 import 任何 dsh 包,
|
||||
保证以绝对路径加载时无解析风险);
|
||||
- **系统提示词段**(`gyxx-workbench`,order 700):中文运维契约与安全规则;
|
||||
- **失败监控器**:`ctx.effect` 轮询,`workflow_id+run_id` 去重,自定义事件
|
||||
`gyxx-workbench/alert` + 可选自动诊断会话;
|
||||
- **桥接服务**(127.0.0.1:8790):为浏览器面板代理控制台 API(控制台禁 CORS,
|
||||
面板无法直连),并承载 `/bridge/ask` 会话创建与降级页面。
|
||||
写操作校验 `x-gyxx-workbench: 1` + JSON content-type;CORS 仅回环来源。
|
||||
|
||||
### 3. 侧边栏面板(`workbench/plugin/client/`)
|
||||
|
||||
- React 源码经 esbuild 打包为 CJS,平台模块(react 等)按 dsh 模块表协议外置,
|
||||
包装为 `window.__ModuleLoader__.load({id, factory})`——与官方 tsdown 产物同协议;
|
||||
- `package.json` 的 `dsh.client` 声明让 `client-modules` 扫描器(支持路径型
|
||||
Loader 条目,向上找最近 package.json)发现并提供该浏览器包;
|
||||
- 插槽:`sidebar.footer.action`(「工作流」开关 + 聚合状态点)、`shell.overlay`
|
||||
(左抽屉面板 + 失败 toast)。均为 list 型追加插槽,不替换内置区域;
|
||||
- 面板动作:提问/诊断/修复(创建带工作流上下文的智能体会话)、试运行/正式运行
|
||||
(浏览器端 confirm)、停止、运行历史查看、单运行诊断视图。
|
||||
|
||||
### 4. 会话创建(`/bridge/ask`)
|
||||
|
||||
`ctx.agents.create({ sessionId, meta: { cwd, origin } })` → `agent.followup()` 注入
|
||||
带工作流上下文的首条用户消息。新会话自动出现在 dsh 会话列表(host 侧创建即入册)。
|
||||
文案模板区分 ask/diagnose/repair 三种动作。
|
||||
|
||||
## 已知限制(v1)
|
||||
|
||||
- 会话创建后不能自动聚焦(dsh 客户端无公开的「选中会话」运行时 API);面板以
|
||||
toast 提示用户在会话列表中点开。
|
||||
- `sidebar.workspaces` 为 single 型插槽(替换即失去会话树),故面板走 overlay
|
||||
抽屉而非内嵌会话树;若未来 dsh 提供追加型侧栏区块插槽,可平移。
|
||||
- 客户端包协议依赖 dsh 未冻结的 developer-preview 契约;升级 dsh 需回归验证
|
||||
(README「依赖的 dsh 扩展点」一节列出了核对清单)。
|
||||
- `autoDiagnose` 默认关闭:自动开会话会产生模型调用成本。
|
||||
|
||||
## 测试
|
||||
|
||||
- Python:`tests/test_console.py` 新增 5 用例(详情/运行详情/诊断包/越权配对/HTTP 路由)。
|
||||
- 插件:`workbench/plugin/tests/`(node:test)——工具注册、离线错误结构化、
|
||||
confirmed 守卫、桥接代理。
|
||||
- 端到端手工验证:`start-workbench.ps1` → 面板选中工作流 → 提问/诊断/修复/试运行。
|
||||
@@ -17,6 +17,15 @@ uv run gyxx console
|
||||
uv run gyxx console --env-file D:\secure\gyxx-flow.env
|
||||
```
|
||||
|
||||
香港客服库存工作流还需要聚水潭应用密钥。Windows 生产控制台必须额外加载受保护的聚水潭环境文件,
|
||||
这样控制台发起的手动执行也能取得凭据;调度器和控制台应加载相同的环境文件集合:
|
||||
|
||||
```powershell
|
||||
uv run gyxx console `
|
||||
--env-file D:\product-collector-analyze-flow\.env `
|
||||
--env-file D:\product-collector-analyze-flow\private-secrets\jushuitan.env
|
||||
```
|
||||
|
||||
商品经营正式执行会在启动子进程前校验云端 PostgreSQL 和该工作流实际需要的运行时配置。
|
||||
需要 Hermes 分析或业务通知的工作流仍会校验 `hermes-analyzer` lark-cli user 身份和负责人
|
||||
通知路由;天猫/京东视频上传则校验 `STYLE_ANALYSIS_LLM_BASE_URL`(或
|
||||
@@ -144,3 +153,14 @@ uv run gyxx console --host 0.0.0.0 --port 8765
|
||||
|
||||
控制台不提供工作流入口、参数、运行时绑定或凭据的在线编辑能力;这些仍由受版本控制的
|
||||
项目配置和代码维护。
|
||||
|
||||
## 自检与保活
|
||||
|
||||
- `GET /api/health` 返回进程 PID、uptime 与调度器锁存活状态,不鉴权、不触外部依赖,专供探针使用。
|
||||
- 控制台进程内置看门狗线程:周期自探测 `/api/health`,连续失败(默认 3 次)即以退出码 70 退出,
|
||||
由 `deploy/watchdog/gyxx_keepalive.py`(登录自启动常驻)负责拉起;保活同时守护调度器进程,
|
||||
详见 `docs/deployment.md` 的「Windows 工作站保活」章节。环境开关:
|
||||
`GYXX_CONSOLE_WATCHDOG=0` 关闭、`GYXX_CONSOLE_WATCHDOG_INTERVAL_SECONDS`(默认 30)、
|
||||
`GYXX_CONSOLE_WATCHDOG_FAILURES`(默认 3)。
|
||||
- API 返回「控制台处理请求失败」时,完整堆栈在控制台 stderr 日志
|
||||
(`var/logs/console-restarts/console-*.stderr.log`)中可查。
|
||||
@@ -69,7 +69,7 @@
|
||||
| `08:30` | 2 | 1 | 1 | 达人月报生成 [`content.creator_report.monthly`](暂停);京东与天猫主图周采集 [`product.main_image.weekly`] |
|
||||
| `08:40` | 1 | 1 | 0 | 商品经营日报采集 [`product.daily`] |
|
||||
| `09:00` | 6 | 6 | 0 | 笔记清单与合作同步 [`content.notes_master.daily`];京东自营业绩日采集 [`shop.jd_self_operated.daily`];抖音待改价申诉;聚水潭全店铺款式日报 [`product.erp_all_shop_daily`];天猫百亿补贴批量报名;补货建议预计到货同步 [`supply.replenishment_arrival_sync.daily`] |
|
||||
| `10:00` | 8 | 5 | 3 | 营销日报生成与发送 [`content.marketing_report.daily`](暂停);内容平台登录态刷新 [`content.relogin.weekly`](暂停);内容周度汇总 [`content.summary.weekly`](暂停);抖音待改价申诉;三平台电商费用采集 [`product.ecommerce_costs.daily`];商品月度销量表 [`product.sales_sheet.daily`];天猫百亿补贴批量报名;市场排行周采集 [`product.market_rank`] |
|
||||
| `10:00` | 8 | 5 | 3 | 营销日报生成与发送 [`content.marketing_report.daily`](暂停);内容周度汇总 [`content.summary.weekly`](暂停);抖音待改价申诉;三平台电商费用采集 [`product.ecommerce_costs.daily`];商品月度销量表 [`product.sales_sheet.daily`];天猫百亿补贴批量报名;市场排行周采集 [`product.market_rank`] |
|
||||
| `11:00` | 3 | 3 | 0 | 抖音待改价申诉;款式周期分析 [`product.style_analysis.interval`];天猫百亿补贴批量报名 |
|
||||
| `12:00` | 5 | 5 | 0 | 多平台评论周采集 [`content.comments.weekly`];店铺经营指标周采集 [`shop.metrics.weekly`];抖音待改价申诉;商品人群画像日采集 [`product.persona.daily`];天猫百亿补贴批量报名 |
|
||||
| `12:30` | 1 | 1 | 0 | 竞店指标周采集 [`shop.competitor.weekly`] |
|
||||
@@ -98,7 +98,7 @@
|
||||
|
||||
- 指标采集:`content.metrics.daily`(内容指标日报采集)、`content.metrics.backfill`(内容指标补采)。
|
||||
- 报告与汇总:`content.marketing_report.daily`(营销日报生成与发送)、`content.creator_report.monthly`(达人月报生成)、`content.summary.monthly`(内容月度汇总)、`content.summary.weekly`(内容周度汇总)。
|
||||
- 内容同步与互动:`content.notes_master.daily`(笔记清单与合作同步)、`content.comments.weekly`(多平台评论周采集)、`content.relogin.weekly`(内容平台登录态刷新)。
|
||||
- 内容同步与互动:`content.notes_master.daily`(笔记清单与合作同步)、`content.comments.weekly`(多平台评论周采集)。
|
||||
|
||||
### 4.2 商品经营(`product_commerce`,12 条)
|
||||
|
||||
@@ -124,7 +124,6 @@
|
||||
| 1 | `content.metrics.daily` | 内容指标日报采集 | 内容营销 | 每天 `01:00`、`05:00` | 0 | 启用 | 5 |
|
||||
| 2 | `content.metrics.backfill` | 内容指标补采 | 内容营销 | 每周一 `14:00` | 0 | 暂停 | 2 |
|
||||
| 3 | `content.marketing_report.daily` | 营销日报生成与发送 | 内容营销 | 每天 `10:00` | 0 | 暂停 | 1 |
|
||||
| 4 | `content.relogin.weekly` | 内容平台登录态刷新 | 内容营销 | 每周五 `10:00` | 0 | 暂停 | 1 |
|
||||
| 5 | `content.creator_report.monthly` | 达人月报生成 | 内容营销 | 每月 1 日 `08:30` | 0 | 启用 | 1 |
|
||||
| 6 | `content.summary.monthly` | 内容月度汇总 | 内容营销 | 每月 1 日 `08:00` | 0 | 启用 | 1 |
|
||||
| 7 | `content.notes_master.daily` | 笔记清单与合作同步 | 内容营销 | 每天 `09:00` | 0 | 启用 | 1 |
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
# 工作流总览(未隐藏)
|
||||
|
||||
> 生成日期:2026-10-08。数据来源:`config/workflows.json`、`config/schedules.json`、控制台显示名(`var/state/console/workflow-display-names.json`);数据库表提取自各入口脚本及共享 DB 辅助模块的内联 SQL。
|
||||
> 范围:控制台中未被隐藏的工作流共 **30 个**(全部 39 个中剔除浏览器端隐藏的 10 个,其中 `content.relogin.weekly` 已不在当前配置中)。
|
||||
> 定时时间均为 **Asia/Shanghai**,由常驻调度器 `gyxx schedule run` 按 `config/schedules.json` 触发;标注「未启用」的仅支持控制台手动运行。
|
||||
> 启动脚本为仓库根目录相对路径,由工作流引擎相对 `src/gyxx_flow/modules/<模块>/` 解析执行;`×N` 表示同一脚本被 N 个步骤以不同参数复用。
|
||||
>
|
||||
> **关联数据库表说明**:均指云端 PostgreSQL(`GYXX_POSTGRES_DSN`),(写)= INSERT/UPDATE/DELETE,(读)= SELECT,(建)= CREATE TABLE IF NOT EXISTS。三点公共约定:
|
||||
> - 内容营销用到款式映射的脚本会经 `feishu_mapping` → `adapters/workflow_config.py` 间接读动态配置表 `workflow_dynamic_styles`、`workflow_dynamic_style_platforms`(并合并读 `cmt_styles`,首跑建 `workflow_dynamic_*` 表),下表不逐一列出。
|
||||
> - 店铺情报各入库脚本启动时执行 `shop_intelligence/db/schema.sql`、商品电商多数入库脚本执行 `product_commerce/db/schema.sql`,幂等建该模块全量表(21 / 26 张);下表只列该工作流**实际读写**的表。
|
||||
> - 读写 `information_schema` / `pg_catalog` 系统目录的通用校验不列出。
|
||||
|
||||
## 内容营销(content_marketing,4 个)
|
||||
|
||||
| 工作流名称 | ID | 功能 | 定时执行 | 启动脚本 | 关联数据库表 |
|
||||
|---|---|---|---|---|---|
|
||||
| 合作达人内容指标日报采集 | `content.metrics.daily` | 按日采集合作达人在 B站、小红书蒲公英、星图(未命中用抖店达人广场补采)的曝光互动指标并同步 PostgreSQL;05:00 档补采 01:00 未完成任务 | 每日 01:00、05:00 | `src/gyxx_flow/modules/content_marketing/run_all.py`<br>`src/gyxx_flow/modules/content_marketing/data/tools/sync_metrics_to_cmt_notes.py` | `cmt_note_metric_snapshots`(写)<br>`cmt_notes_master`(读写)<br>`cmt_creators`(读写)<br>`cmt_styles`(读) |
|
||||
| 自营内容指标日报采集 | `content.metrics.self_operated.daily` | 刷新自营款式映射,采集自营 B站、蝉妈妈、小红书 ARK 指标;飞书曝光回填并将三平台全量内容指标按日快照入库 | 每日 06:00 | `…/content_marketing/data/tools/refresh_self_mapping.py`<br>`…/content_marketing/self_bilibili_scraper.py`<br>`…/content_marketing/chanmama_scraper.py`<br>`…/content_marketing/self_xhs_scraper.py`<br>`…/content_marketing/data/tools/sync_metrics_to_cmt_notes.py`<br>`…/content_marketing/data/tools/sync_self_video_metrics.py` | `cmt_self_video_metrics`(写)<br>`cmt_note_metric_snapshots`(写)<br>`cmt_notes_master`(读写)<br>`cmt_creators`(读写)<br>`cmt_styles`(读) |
|
||||
| 笔记清单与合作同步 | `content.notes_master.daily` | 一次飞书拉取同时落笔记身份与合作字段;笔记数量按去重链接口径维护主表 | 每日 09:00 | `src/gyxx_flow/modules/content_marketing/data/tools/sync_notes_master.py` | `cmt_notes_master`(建+写)<br>`cmt_note_metric_snapshots`(建)<br>`cmt_styles`(读写)<br>`cmt_creators`(读写) |
|
||||
| 多平台评论周采集 | `content.comments.weekly` | B站、小红书、抖音三平台评论并行采集,任一平台失败则工作流失败 | 每周日 12:00 | `…/content_marketing/data/tools/batch_rescrape_bilibili.py`<br>`…/content_marketing/data/tools/batch_rescrape_xiaohongshu.py`<br>`…/content_marketing/data/tools/batch_rescrape_douyin.py` | `cmt_comments`(写)<br>`cmt_notes_master`(读写) |
|
||||
|
||||
## 店铺情报(shop_intelligence,12 个)
|
||||
|
||||
| 工作流名称 | ID | 功能 | 定时执行 | 启动脚本 | 关联数据库表 |
|
||||
|---|---|---|---|---|---|
|
||||
| 店铺经营指标周采集 | `shop.metrics.weekly` | 三平台作为独立节点并行采集店铺经营指标并写 PostgreSQL(不写飞书 Base) | 每周一 12:00 | `src/gyxx_flow/modules/shop_intelligence/runners/run_shop.py` ×3 | `jd_shop_weekly`、`dy_shop_weekly`、`tm_shop_weekly`(各平台步骤分别写) |
|
||||
| 竞店指标周采集 | `shop.competitor.weekly` | 京东、抖音并行采集竞店指标并入库,单平台失败不阻断另一平台 | 每周一 12:30 | `src/gyxx_flow/modules/shop_intelligence/runners/run_peer_store.py` ×2 | `jd_peer_store_ranking`、`dy_peer_store_ranking`(各平台步骤分别写) |
|
||||
| 京东自营业绩日采集 | `shop.jd_self_operated.daily` | 采集京东自营品牌与商品业绩(业务日期 = 运行日 −2) | 每日 09:00 | `…/shop_intelligence/collectors/jd_self_operated_brand_daily.py`<br>`…/shop_intelligence/collectors/jd_self_operated_product_daily.py` | `jd_self_operated_brand_daily`(写)<br>`jd_self_operated_product_daily`(写) |
|
||||
| 京东客服消息采集 | `shop.jd_customer_service.daily` | 按业务日在商智消息视图采集顾客会话,单日须完成全部顾客,失败保留进度可重试 | 每月 21 日 03:00 | `src/gyxx_flow/modules/shop_intelligence/collectors/jd_customer_service_messages.py` | `jd_customer_service_conversations`、`jd_customer_service_messages`(写) |
|
||||
| 抖音客服消息采集 | `shop.douyin_customer_service.daily` | 按业务日采集抖音客服会话,失败保留进度可安全重试 | 每月 21 日 03:20 | `src/gyxx_flow/modules/shop_intelligence/collectors/dy_customer_service_messages.py` | `dy_customer_service_conversations`、`dy_customer_service_messages`、`dy_customer_service_conversation_orders`(写)<br>`dy_customer_service_products`(读写) |
|
||||
| 京东差评采集 | `shop.jd_negative_reviews.daily` | 按店铺选择日期范围逐日采集京东负面评价,按评价 ID 幂等 | 每月 22 日 03:00 | `src/gyxx_flow/modules/shop_intelligence/collectors/jd_negative_reviews.py` | `jd_negative_reviews`(写) |
|
||||
| 抖音差评采集 | `shop.dy_negative_reviews.daily` | 逐日采集抖音负面评价(评价历史仅保留近 30 天),按订单 ID 幂等 | 每月 22 日 03:20 | `src/gyxx_flow/modules/shop_intelligence/collectors/dy_negative_reviews.py` | `dy_negative_reviews`(写) |
|
||||
| 京东品退问题采集 | `shop.jd_after_sales_issues.daily` | 在商智 31 天限制下按连续区间采集售后服务单,按申请日期写业务日,幂等更新 | 每月 23 日 03:00 | `src/gyxx_flow/modules/shop_intelligence/collectors/jd_after_sales_issues.py` | `jd_after_sales_issues`(写,含范围外残留 DELETE 清理) |
|
||||
| 抖音品退采集 | `shop.douyin_after_sales_quality.daily` | 采集抖音售后单,按申请时间归属业务日,账号 + 售后编号幂等更新 | 每月 23 日 03:20 | `src/gyxx_flow/modules/shop_intelligence/collectors/dy_after_sales_quality.py` | `dy_after_sales_quality_orders`、`dy_after_sales_quality_negotiations`(写) |
|
||||
| 天猫客服消息采集 | `shop.tmall_customer_service.daily` | 按每次最多 31 天的窗口顺序采集天猫客服聊天记录,失败窗口保留数据 | 每月 24 日 03:40(未启用,仅手动) | `src/gyxx_flow/modules/shop_intelligence/collectors/tmall_customer_service_messages.py` | `tmall_customer_service_conversations`、`tmall_customer_service_messages`(写) |
|
||||
| 天猫差评采集 | `shop.tmall_negative_reviews.daily` | 固定筛选负面评价并按评价时间范围采集后入库 | 每月 25 日 03:50(未启用,仅手动) | `src/gyxx_flow/modules/shop_intelligence/collectors/tmall_negative_reviews.py` | `tmall_negative_reviews`(写) |
|
||||
| 抖音待改价申诉 | `shop.douyin_price_appeal` | 每个时点重新扫描可申诉并发起价格申诉;已申诉商品不重复出现 | 每日每小时整点(00:00–23:00) | `src/gyxx_flow/modules/shop_intelligence/collectors/dy_store_competitor_store_scraping.py` | 无(浏览器采集 + 本地 JSON) |
|
||||
|
||||
## 商品电商(product_commerce,12 个)
|
||||
|
||||
| 工作流名称 | ID | 功能 | 定时执行 | 启动脚本 | 关联数据库表 |
|
||||
|---|---|---|---|---|---|
|
||||
| 商品人群画像周采集 | `product.persona.daily` | 天猫、抖音、京东人群画像并行采集并写入对应飞书多维表 | 每周二 12:00 | `src/gyxx_flow/modules/product_commerce/run_daily_persona.py` | `daily_persona_metrics`(写) |
|
||||
| 商品经营日报采集 | `product.daily` | 按业务日期(运行日 −1)采集 ERP 与三平台数据,幂等写 PostgreSQL,汇总分析后导出并写飞书日报 | 每日 08:40 | `src/gyxx_flow/modules/product_commerce/orchestrate_daily_collection.py` | 写:`product_reviews`、`product_daily_metrics`、`daily_style_metrics`、`fact_style_analysis_reports`、`decline_events`、`fact_alert_history`<br>读:`weekly_style_metrics`、`daily_persona_metrics`、`main_image_creatives`、`cmt_styles`、`cmt_notes_master`、`cmt_comments`、`cmt_creators`(及上述写表) |
|
||||
| 私域库存预警 | `product.online_stock_alert.daily` | 查询线上渠道全部商品,库存低于 3 时汇总 SKU 与库存发飞书预警 | 每日 09:30 | `src/gyxx_flow/modules/product_commerce/online_stock_alert.py` | 无(企业数据交互 HTTP API + 飞书) |
|
||||
| 三平台电商费用采集 | `product.ecommerce_costs.daily` | 并行采集天猫万相台、京东、抖音千川业务日前一天的费用,各平台先下载再导入,单平台失败不阻断其他 | 每日 10:00 | `…/product_commerce/collect_tmall_wanxiang_ads.py`<br>`…/product_commerce/import_tmall_product_ads.py`<br>`…/product_commerce/collect_jd_ad_costs.py`<br>`…/product_commerce/import_jd_ad_costs.py`<br>`…/product_commerce/collect_douyin_qianchuan_ads.py`<br>`…/product_commerce/import_douyin_qianchuan_ads.py` | `product_daily_metrics`(写,天猫原始费用并入 raw_data JSONB)<br>`jd_product_ad_daily_metrics`、`dy_product_ad_daily_metrics`(建+写)<br>`dim_style`(读) |
|
||||
| 京东与天猫主图周采集 | `product.main_image.weekly` | 京东与天猫万相台主图并行采集,各自完成 PG 入库和飞书插入,互不影响 | 每周日 08:30 | `…/product_commerce/run_weekly_jd_main_image.py`<br>`…/product_commerce/run_weekly_main_image.py` | `main_image_creatives`(写) |
|
||||
| 飞书商品月度销量表填写 | `product.sales_sheet.daily` | 抓取并校验全部 ERP 报表后填写飞书月度销量表,同名月份页幂等覆盖,无 ERP 编码产品按 0 写入 | 每月 1 日 10:00 | `src/gyxx_flow/modules/product_commerce/sync_monthly_sales_sheet.py` | 无(聚水潭浏览器采集 + 飞书写入) |
|
||||
| 聚水潭全店铺款式日报 | `product.erp_all_shop_daily` | 聚水潭商品主题分析全选全部店铺,采集业务日前一天所有 ERP 款式的自定义销量与退货量并幂等写 PostgreSQL | 每日 09:00 | `src/gyxx_flow/modules/product_commerce/backfill_erp_all_shop_daily.py` | `erp_all_shop_style_daily_metrics`(读写) |
|
||||
| 客服香港库存工作流 | `product.hk_anpei_public_inventory.daily` | 校验计划书 SKU 与库存表商品编码对应关系,调聚水潭库存接口查询并将公有可用数写入仓库现有库存列 | 每日 09:00 | `src/gyxx_flow/modules/product_commerce/public_sku_inventory_sheet.py` | 无(聚水潭 API + 飞书表格) |
|
||||
| 天猫视频上传 | `product.video_upload` | 按店铺规则扫描飞书视频附件,凭附件级发布账本防重,仅在存在可领取附件时登录光合账号上传 | 每 3 天 14:00 | `src/gyxx_flow/modules/product_commerce/upload_video_to_guanghe.py` | `fact_video_publications`、`fact_video_publication_attempts`(建+读写,附件级发布账本)<br>`dim_style`、`product_daily_metrics`(读) |
|
||||
| 京东视频上传 | `product.jd_video_upload` | 从凌云air基线扫描到视图末尾识别新增视频,以 record_id + file_token 幂等发布到京东 | 每 3 天 14:00 | `src/gyxx_flow/modules/product_commerce/upload_video_to_jd.py` | `fact_video_publications`、`fact_video_publication_attempts`(建+读写,发布账本)<br>`dim_style`、`product_daily_metrics`(读) |
|
||||
| 天猫百亿补贴批量报名 | `product.tmall_baibu_apply` | 每次从飞书下载报名模板逐入口提交;默认仅导入 3 个不带超链入口(含老天猫全渠道 3C 后置完善草稿),可切换全部 6 个入口 | 每日每小时整点(00:00–23:00) | `…/product_commerce/tmall_baibu_apply_old_chain_bag.py`<br>`…/product_commerce/tmall_baibu_apply_old_all_bag.py`<br>`…/product_commerce/tmall_baibu_apply_old_all_3c.py`<br>`…/product_commerce/tmall_baibu_apply_old_chain_3c.py`<br>`…/product_commerce/tmall_baibu_apply_new_chain_bag.py`<br>`…/product_commerce/tmall_baibu_apply_new_all_bag.py` | 无(浏览器操作 + 飞书) |
|
||||
| 三平台市场排行采集(用于行军图) | `product.market_rank` | 并行采集天猫、京东、抖音市场排行,按月份写入品牌行军图数据库并复用已有人工分类 | 每月 1 日 10:00 | `src/gyxx_flow/modules/product_commerce/orchestrate_market_rank_collection.py` | `market_product`、`market_product_classification`(读写)<br>`fact_market_product_snapshot`(写,重跑先 DELETE)<br>`fact_market_rank_reports`、`fact_market_rank_notifications`(建+读写)<br>`dim_market_style`、`dim_market_scene`(读) |
|
||||
|
||||
## 供应链(supply_chain,2 个)
|
||||
|
||||
| 工作流名称 | ID | 功能 | 定时执行 | 启动脚本 | 关联数据库表 |
|
||||
|---|---|---|---|---|---|
|
||||
| 补货预警日处理 | `supply.replenishment_alert.daily` | 依据配置 SKU 与库存阈值生成预警并写 PostgreSQL,仅存在预警时发送业务通知 | 每日 07:00 | `src/gyxx_flow/modules/supply_chain/run.py` | `fact_threshold_alert`(读写)<br>`dim_product`(写)<br>`workflow_runs`(读写,运行账本)<br>`bitable_insert_log`(读写,通知去重账本) |
|
||||
| 同步聚水潭到货时间 | `supply.replenishment_arrival_sync.daily` | 读取到货同步待处理队列,按商品编码在聚水潭采购单页面更新预计到货时间并回写状态,失败行下次自动重试 | 每日 09:00 | `src/gyxx_flow/modules/supply_chain/orchestrator/scripts/collect_replenishment_arrival_sync.py` | `erp_replenishment_arrival_sync`(读写,队列表由 ERP 应用写入) |
|
||||
|
||||
## 附:已隐藏的工作流(未收录)
|
||||
|
||||
浏览器端隐藏的 10 个 ID:`content.creator_report.monthly`、`content.marketing_report.daily`、`content.metrics.backfill`、`content.relogin.weekly`(配置中已不存在)、`content.summary.monthly`、`content.summary.weekly`、`product.alert.daily`、`product.style_analysis.interval`、`supply.purchase_confirmation.daily`、`supply.replenishment.weekly`。
|
||||
@@ -223,6 +223,8 @@ class AccountKeepaliveManager:
|
||||
self._next_due: dict[str, float] = {}
|
||||
self._locks = LockManager(settings.data_root / "state" / "locks")
|
||||
self._login_runner = login_runner or self._run_configured_login
|
||||
self._audit_store: Any | None = None
|
||||
self._audit_seen: dict[str, tuple[str, float]] = {}
|
||||
|
||||
def tick(self) -> list[dict[str, object]]:
|
||||
"""Run due account refreshes once; never starts work in dry-run mode."""
|
||||
@@ -479,6 +481,9 @@ class AccountKeepaliveManager:
|
||||
real_chrome=account.keepalive_real_chrome,
|
||||
retries=1,
|
||||
persist_state_on_close=False,
|
||||
# 目标站点均为国内站,代理出口反被部分 WAF 拉黑(如蝉妈妈
|
||||
# TLS reset),账号保活一律直连。
|
||||
extra_flags=["--no-proxy-server"],
|
||||
)
|
||||
try:
|
||||
browser.start()
|
||||
@@ -583,8 +588,63 @@ class AccountKeepaliveManager:
|
||||
if result.get("status") == "success":
|
||||
payload["last_success_at"] = timestamp
|
||||
atomic_write_json(path, payload)
|
||||
self._audit_keepalive(account, result)
|
||||
return dict(result)
|
||||
|
||||
_AUDIT_STATE_BY_PROBE = {
|
||||
"success": "healthy",
|
||||
"expired": "expired",
|
||||
"not_logged_in": "not_logged_in",
|
||||
"busy": "busy",
|
||||
"disabled": "disabled",
|
||||
}
|
||||
|
||||
def _audit_keepalive(self, account: Any, result: dict[str, object]) -> None:
|
||||
"""Record probe outcomes so the real cookie lifetime becomes measurable.
|
||||
|
||||
The resident scheduler probes every ``keepalive_interval_seconds``
|
||||
(10 minutes by default), which is what turns "logins seem to last about
|
||||
a week" into a measured distribution. Repeated identical outcomes
|
||||
within an hour collapse into one row, so the event table tracks state
|
||||
changes rather than polling noise.
|
||||
"""
|
||||
|
||||
try:
|
||||
from gyxx_flow.ops.audit import snapshot_for_account
|
||||
|
||||
probe = str(result.get("status", "") or "error")
|
||||
state = self._AUDIT_STATE_BY_PROBE.get(probe, "probe_error")
|
||||
key = str(account.account_id)
|
||||
previous = self._audit_seen.get(key)
|
||||
if previous is not None:
|
||||
previous_state, previous_at = previous
|
||||
if previous_state == state and (self._clock() - previous_at) < 3600:
|
||||
return
|
||||
store = self._audit()
|
||||
if store is None or not store.enabled:
|
||||
return
|
||||
if store.record_cookie_event(
|
||||
account_id=key,
|
||||
state=state,
|
||||
snapshot=snapshot_for_account(account),
|
||||
probe_status=probe,
|
||||
probe_source="keepalive",
|
||||
detail=str(result.get("error_type", "") or ""),
|
||||
):
|
||||
self._audit_seen[key] = (state, self._clock())
|
||||
except Exception: # noqa: BLE001 - telemetry must never break keepalive
|
||||
return
|
||||
|
||||
def _audit(self) -> Any | None:
|
||||
if self._audit_store is None:
|
||||
try:
|
||||
from gyxx_flow.ops.audit import AuditStore
|
||||
|
||||
self._audit_store = AuditStore(self.settings.data_root)
|
||||
except Exception: # noqa: BLE001
|
||||
return None
|
||||
return self._audit_store
|
||||
|
||||
|
||||
def _vault_cookie_valid(account: Any, store: BrowserCookieStore) -> bool:
|
||||
if not account.cookie_file.exists():
|
||||
@@ -761,6 +821,8 @@ def login_account(
|
||||
"--remote-debugging-address=127.0.0.1",
|
||||
"--disable-session-crashed-bubble",
|
||||
"--disable-features=InfiniteSessionRestore",
|
||||
# 国内站直连,避免代理出口被 WAF 拉黑导致登录/探测失败
|
||||
"--no-proxy-server",
|
||||
],
|
||||
)
|
||||
try:
|
||||
|
||||
@@ -168,3 +168,22 @@ def looks_like_login_url(url: str) -> bool:
|
||||
path = parsed.path.casefold()
|
||||
markers = ("login", "passport", "signin", "sign-in")
|
||||
return any(marker in hostname or marker in path for marker in markers)
|
||||
|
||||
|
||||
class CookieExpiredError(RuntimeError):
|
||||
"""A platform login state expired and no automatic recovery exists.
|
||||
|
||||
Collectors raise this instead of waiting for a human QR/SMS login: the
|
||||
account cookie gate has already blocked unverified accounts before the
|
||||
run, so reaching this point means the session died mid-run. The CLI maps
|
||||
it to ``COOKIE_SKIP_EXIT_CODE`` so the engine records a cookie skip.
|
||||
"""
|
||||
|
||||
|
||||
def cookie_expired_message(account_id: str, platform: str) -> str:
|
||||
"""One standard, operator-actionable expiry message across platforms."""
|
||||
|
||||
return (
|
||||
f"{platform} 登录态已失效:请执行 `uv run gyxx accounts login {account_id}` "
|
||||
"完成扫码/验证码重新登录(或等待飞书通知后处理)"
|
||||
)
|
||||
@@ -1210,29 +1210,49 @@ def _merge_storage_state(
|
||||
|
||||
|
||||
def _account_cookie_valid(account: AccountBinding, cookies: object) -> bool:
|
||||
"""A vault cookie set is valid when a live account cookie is present."""
|
||||
"""A vault cookie set is valid when a live account cookie is present.
|
||||
|
||||
With ``required_cookie_names`` configured, every required name must be
|
||||
present as a live, domain-matched cookie; a single anonymous cookie that
|
||||
happens to share a required name (e.g. ``a1``) must not validate a guest
|
||||
session. Without required names, any live cookie on a required domain
|
||||
keeps accepting the set.
|
||||
"""
|
||||
if not isinstance(cookies, list):
|
||||
return False
|
||||
required_names = account.required_cookie_names
|
||||
required_domains = account.required_cookie_domains
|
||||
for item in cookies:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
if _cookie_expired(item):
|
||||
continue
|
||||
if not str(item.get("value", "")):
|
||||
continue
|
||||
name = str(item.get("name", ""))
|
||||
domain = str(item.get("domain", "")).casefold().lstrip(".")
|
||||
name_matches = not required_names or name in required_names
|
||||
domain_matches = not required_domains or any(
|
||||
|
||||
def _domain_matches(domain: str) -> bool:
|
||||
return not required_domains or any(
|
||||
domain == required.casefold()
|
||||
or domain.endswith("." + required.casefold())
|
||||
for required in required_domains
|
||||
)
|
||||
if name_matches and domain_matches:
|
||||
return True
|
||||
return False
|
||||
|
||||
def _live(item: object) -> bool:
|
||||
return (
|
||||
isinstance(item, dict)
|
||||
and not _cookie_expired(item)
|
||||
and bool(str(item.get("value", "")))
|
||||
)
|
||||
|
||||
if not required_names:
|
||||
return any(
|
||||
_live(item)
|
||||
and _domain_matches(
|
||||
str(item.get("domain", "")).casefold().lstrip(".")
|
||||
)
|
||||
for item in cookies
|
||||
)
|
||||
|
||||
outstanding = set(required_names)
|
||||
for item in cookies:
|
||||
if not _live(item):
|
||||
continue
|
||||
if _domain_matches(str(item.get("domain", "")).casefold().lstrip(".")):
|
||||
outstanding.discard(str(item.get("name", "")))
|
||||
return not outstanding
|
||||
|
||||
|
||||
def _cookie_expired(cookie: Mapping[str, object], *, now: float | None = None) -> bool:
|
||||
|
||||
@@ -0,0 +1,338 @@
|
||||
"""Minimal, signed Jushuitan inventory-query client for workflow adapters."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
import uuid
|
||||
from collections.abc import Mapping
|
||||
from dataclasses import dataclass
|
||||
from decimal import Decimal, InvalidOperation
|
||||
from typing import Any
|
||||
from urllib.parse import urlparse
|
||||
|
||||
import requests
|
||||
|
||||
_TOKEN_PATH = "/openWeb/auth/getInitToken"
|
||||
_INVENTORY_PATH = "/open/inventory/query"
|
||||
_RATE_LIMIT_CODES = {199, 200}
|
||||
_ACCESS_TOKEN_INVALID_CODE = 100
|
||||
_MAX_RESPONSE_BYTES = 2 * 1024 * 1024
|
||||
_MAX_PAGES = 100
|
||||
_REQUEST_INTERVAL_SECONDS = 1.1
|
||||
|
||||
|
||||
class JushuitanInventoryError(RuntimeError):
|
||||
"""Safe-to-log error raised for Jushuitan query or response failures."""
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class InventoryRecord:
|
||||
sku: str
|
||||
qty: int
|
||||
order_lock: int
|
||||
lock_qty: int
|
||||
virtual_qty: int
|
||||
|
||||
@property
|
||||
def public_available_qty(self) -> int:
|
||||
return self.qty - self.order_lock - self.lock_qty + self.virtual_qty
|
||||
|
||||
|
||||
def _sign(app_secret: str, params: Mapping[str, object]) -> str:
|
||||
source = app_secret + "".join(
|
||||
f"{key}{value}"
|
||||
for key, value in sorted(params.items())
|
||||
if key.lower() != "sign" and value is not None and str(value) != ""
|
||||
)
|
||||
return hashlib.md5(source.encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
def _response_code(value: object) -> int:
|
||||
if isinstance(value, bool):
|
||||
raise JushuitanInventoryError("Jushuitan response has an invalid code")
|
||||
try:
|
||||
return int(value)
|
||||
except (TypeError, ValueError) as exc:
|
||||
raise JushuitanInventoryError(
|
||||
"Jushuitan response has an invalid code"
|
||||
) from exc
|
||||
|
||||
|
||||
def _quantity(value: object, *, field: str, sku: str) -> int:
|
||||
if value is None:
|
||||
return 0
|
||||
if isinstance(value, bool):
|
||||
raise JushuitanInventoryError(
|
||||
f"Jushuitan inventory has an invalid {field} for SKU {sku}"
|
||||
)
|
||||
try:
|
||||
parsed = Decimal(str(value).strip())
|
||||
except (InvalidOperation, ValueError) as exc:
|
||||
raise JushuitanInventoryError(
|
||||
f"Jushuitan inventory has an invalid {field} for SKU {sku}"
|
||||
) from exc
|
||||
if not parsed.is_finite() or parsed != parsed.to_integral_value():
|
||||
raise JushuitanInventoryError(
|
||||
f"Jushuitan inventory has a non-integer {field} for SKU {sku}"
|
||||
)
|
||||
return int(parsed)
|
||||
|
||||
|
||||
def _value(item: Mapping[str, Any], snake_name: str, camel_name: str) -> object:
|
||||
if snake_name in item:
|
||||
return item[snake_name]
|
||||
return item.get(camel_name)
|
||||
|
||||
|
||||
class JushuitanInventoryClient:
|
||||
"""Signed form client matching data-hub's Jushuitan API contract."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
app_key: str | None = None,
|
||||
app_secret: str | None = None,
|
||||
base_url: str | None = None,
|
||||
timeout: tuple[float, float] = (10.0, 45.0),
|
||||
) -> None:
|
||||
self.app_key = (app_key or os.environ.get("JUSHUITAN_APP_KEY", "")).strip()
|
||||
self.app_secret = (
|
||||
app_secret or os.environ.get("JUSHUITAN_APP_SECRET", "")
|
||||
).strip()
|
||||
configured_url = (
|
||||
base_url
|
||||
or os.environ.get("JUSHUITAN_BASE_URL", "")
|
||||
or "https://openapi.jushuitan.com"
|
||||
).strip()
|
||||
parsed_url = urlparse(configured_url)
|
||||
if (
|
||||
parsed_url.scheme.lower() != "https"
|
||||
or not parsed_url.netloc
|
||||
or parsed_url.username
|
||||
or parsed_url.password
|
||||
or parsed_url.query
|
||||
or parsed_url.fragment
|
||||
):
|
||||
raise JushuitanInventoryError(
|
||||
"JUSHUITAN_BASE_URL must be an HTTPS base URL without credentials"
|
||||
)
|
||||
if not self.app_key or not self.app_secret:
|
||||
raise JushuitanInventoryError(
|
||||
"JUSHUITAN_APP_KEY and JUSHUITAN_APP_SECRET must be injected "
|
||||
"into the scheduler environment"
|
||||
)
|
||||
self.base_url = configured_url.rstrip("/")
|
||||
self.timeout = timeout
|
||||
self.session = requests.Session()
|
||||
self._access_token = ""
|
||||
self._last_request_at = 0.0
|
||||
|
||||
def query(self, skus: list[str]) -> dict[str, InventoryRecord]:
|
||||
normalized_skus = list(dict.fromkeys(sku.strip() for sku in skus if sku.strip()))
|
||||
if not normalized_skus:
|
||||
raise JushuitanInventoryError("No SKU values were supplied")
|
||||
if len(normalized_skus) > 100:
|
||||
raise JushuitanInventoryError(
|
||||
"A single inventory query may include at most 100 SKUs"
|
||||
)
|
||||
|
||||
self._access_token = self._request_access_token()
|
||||
page_index = 1
|
||||
page_count = 1
|
||||
records: dict[str, InventoryRecord] = {}
|
||||
requested = set(normalized_skus)
|
||||
while page_index <= page_count:
|
||||
if page_index > _MAX_PAGES:
|
||||
raise JushuitanInventoryError(
|
||||
"Jushuitan inventory response exceeded the page safety limit"
|
||||
)
|
||||
response = self._query_page(normalized_skus, page_index)
|
||||
data = response.get("data")
|
||||
if not isinstance(data, Mapping):
|
||||
raise JushuitanInventoryError(
|
||||
"Jushuitan inventory response is missing its data object"
|
||||
)
|
||||
raw_rows = data.get("inventorys")
|
||||
if raw_rows is None:
|
||||
raw_rows = data.get("datas")
|
||||
if raw_rows is None:
|
||||
raw_rows = []
|
||||
if not isinstance(raw_rows, list):
|
||||
raise JushuitanInventoryError(
|
||||
"Jushuitan inventory response has an invalid inventory list"
|
||||
)
|
||||
for index, raw_item in enumerate(raw_rows, start=1):
|
||||
if raw_item is None:
|
||||
continue
|
||||
if not isinstance(raw_item, Mapping):
|
||||
raise JushuitanInventoryError(
|
||||
f"Jushuitan inventory row {index} is invalid"
|
||||
)
|
||||
sku_value = _value(raw_item, "sku_id", "skuId")
|
||||
sku = str(sku_value or "").strip()
|
||||
if not sku:
|
||||
raise JushuitanInventoryError(
|
||||
f"Jushuitan inventory row {index} has no SKU"
|
||||
)
|
||||
if sku not in requested:
|
||||
raise JushuitanInventoryError(
|
||||
"Jushuitan returned a SKU that was not requested"
|
||||
)
|
||||
record = InventoryRecord(
|
||||
sku=sku,
|
||||
qty=_quantity(
|
||||
_value(raw_item, "qty", "qty"), field="qty", sku=sku
|
||||
),
|
||||
order_lock=_quantity(
|
||||
_value(raw_item, "order_lock", "orderLock"),
|
||||
field="order_lock",
|
||||
sku=sku,
|
||||
),
|
||||
lock_qty=_quantity(
|
||||
_value(raw_item, "lock_qty", "lockQty"),
|
||||
field="lock_qty",
|
||||
sku=sku,
|
||||
),
|
||||
virtual_qty=_quantity(
|
||||
_value(raw_item, "virtual_qty", "virtualQty"),
|
||||
field="virtual_qty",
|
||||
sku=sku,
|
||||
),
|
||||
)
|
||||
if sku in records:
|
||||
raise JushuitanInventoryError(
|
||||
f"Jushuitan returned duplicate inventory rows for SKU {sku}"
|
||||
)
|
||||
records[sku] = record
|
||||
|
||||
raw_page_count = data.get("page_count", data.get("pageCount", 1))
|
||||
try:
|
||||
page_count = max(1, int(raw_page_count or 1))
|
||||
except (TypeError, ValueError) as exc:
|
||||
raise JushuitanInventoryError(
|
||||
"Jushuitan inventory response has an invalid page count"
|
||||
) from exc
|
||||
if page_count > _MAX_PAGES:
|
||||
raise JushuitanInventoryError(
|
||||
"Jushuitan inventory response exceeded the page safety limit"
|
||||
)
|
||||
page_index += 1
|
||||
return records
|
||||
|
||||
def _request_access_token(self) -> str:
|
||||
params: dict[str, object] = {
|
||||
"app_key": self.app_key,
|
||||
"charset": "utf-8",
|
||||
"code": uuid.uuid4().hex[:6],
|
||||
"grant_type": "authorization_code",
|
||||
"timestamp": str(int(time.time())),
|
||||
}
|
||||
response = self._post_form(_TOKEN_PATH, params)
|
||||
data = response.get("data")
|
||||
if not isinstance(data, Mapping):
|
||||
raise JushuitanInventoryError(
|
||||
"Jushuitan token response is missing its data object"
|
||||
)
|
||||
token = data.get("access_token") or data.get("accessToken")
|
||||
if not isinstance(token, str) or not token.strip():
|
||||
raise JushuitanInventoryError(
|
||||
"Jushuitan token response has no access token"
|
||||
)
|
||||
return token.strip()
|
||||
|
||||
def _query_page(self, skus: list[str], page_index: int) -> dict[str, Any]:
|
||||
biz = {
|
||||
"sku_ids": ",".join(skus),
|
||||
"has_lock_qty": True,
|
||||
"page_index": page_index,
|
||||
"page_size": 100,
|
||||
}
|
||||
params: dict[str, object] = {
|
||||
"access_token": self._access_token,
|
||||
"app_key": self.app_key,
|
||||
"charset": "utf-8",
|
||||
"timestamp": str(int(time.time())),
|
||||
"version": "2",
|
||||
"biz": json.dumps(biz, ensure_ascii=False, separators=(",", ":")),
|
||||
}
|
||||
for token_attempt in range(2):
|
||||
response = self._post_form(
|
||||
_INVENTORY_PATH,
|
||||
params,
|
||||
allowed_api_codes=frozenset({_ACCESS_TOKEN_INVALID_CODE}),
|
||||
)
|
||||
code = _response_code(response.get("code"))
|
||||
if code == _ACCESS_TOKEN_INVALID_CODE and token_attempt == 0:
|
||||
self._access_token = self._request_access_token()
|
||||
params["access_token"] = self._access_token
|
||||
params["timestamp"] = str(int(time.time()))
|
||||
continue
|
||||
return response
|
||||
raise JushuitanInventoryError("Jushuitan rejected the access token")
|
||||
|
||||
def _post_form(
|
||||
self,
|
||||
path: str,
|
||||
params: dict[str, object],
|
||||
*,
|
||||
allowed_api_codes: frozenset[int] = frozenset(),
|
||||
) -> dict[str, Any]:
|
||||
rate_limit_attempt = 0
|
||||
while True:
|
||||
now = time.monotonic()
|
||||
wait_seconds = _REQUEST_INTERVAL_SECONDS - (now - self._last_request_at)
|
||||
if wait_seconds > 0:
|
||||
time.sleep(wait_seconds)
|
||||
self._last_request_at = time.monotonic()
|
||||
signed_params = dict(params)
|
||||
signed_params["sign"] = _sign(self.app_secret, signed_params)
|
||||
try:
|
||||
response = self.session.post(
|
||||
f"{self.base_url}{path}",
|
||||
data=signed_params,
|
||||
timeout=self.timeout,
|
||||
headers={"Accept": "application/json"},
|
||||
)
|
||||
except requests.RequestException as exc:
|
||||
raise JushuitanInventoryError(
|
||||
f"Jushuitan request failed for {path}"
|
||||
) from exc
|
||||
if response.status_code < 200 or response.status_code >= 300:
|
||||
raise JushuitanInventoryError(
|
||||
f"Jushuitan endpoint {path} returned HTTP {response.status_code}"
|
||||
)
|
||||
if len(response.content) > _MAX_RESPONSE_BYTES:
|
||||
raise JushuitanInventoryError(
|
||||
"Jushuitan response exceeded the 2 MiB safety limit"
|
||||
)
|
||||
try:
|
||||
payload = response.json()
|
||||
except ValueError as exc:
|
||||
raise JushuitanInventoryError(
|
||||
f"Jushuitan endpoint {path} returned invalid JSON"
|
||||
) from exc
|
||||
if not isinstance(payload, dict):
|
||||
raise JushuitanInventoryError(
|
||||
f"Jushuitan endpoint {path} returned an invalid response"
|
||||
)
|
||||
code = _response_code(payload.get("code"))
|
||||
if code in _RATE_LIMIT_CODES and rate_limit_attempt < 4:
|
||||
time.sleep(0.5 * (2**rate_limit_attempt))
|
||||
rate_limit_attempt += 1
|
||||
params["timestamp"] = str(int(time.time()))
|
||||
continue
|
||||
if code != 0 and code not in allowed_api_codes:
|
||||
raise JushuitanInventoryError(
|
||||
f"Jushuitan endpoint {path} returned API code {code}"
|
||||
)
|
||||
return payload
|
||||
|
||||
|
||||
__all__ = [
|
||||
"InventoryRecord",
|
||||
"JushuitanInventoryClient",
|
||||
"JushuitanInventoryError",
|
||||
]
|
||||
@@ -37,8 +37,8 @@ def is_browser_timeout_error(error: BaseException) -> bool:
|
||||
)
|
||||
|
||||
|
||||
def _is_browser_closed_error(error: BaseException) -> bool:
|
||||
"""Recognize a browser target that disappeared during cleanup."""
|
||||
def is_browser_closed_error(error: BaseException) -> bool:
|
||||
"""Recognize a browser target that disappeared (page/context/browser)."""
|
||||
|
||||
if any(
|
||||
error_type.__name__ == "TargetClosedError"
|
||||
@@ -155,7 +155,7 @@ class ScraplingBrowser:
|
||||
# and context-manager cleanup. The current state cannot
|
||||
# be exported then, but that must not turn a successful
|
||||
# download into a false failure or mask the action error.
|
||||
if not _is_browser_closed_error(exc):
|
||||
if not is_browser_closed_error(exc):
|
||||
raise
|
||||
finally:
|
||||
try:
|
||||
|
||||
@@ -79,6 +79,10 @@ class WorkflowEntry:
|
||||
steps: tuple[WorkflowStepEntry, ...] = ()
|
||||
notification_workflow_id: str | None = None
|
||||
topic_config: WorkflowTopicConfig | None = None
|
||||
# Orchestrator entries whose own binding carries no runtime account while
|
||||
# the spawned child collectors do; the cookie gate probes these before
|
||||
# the workflow runs.
|
||||
required_accounts: tuple[str, ...] = ()
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
@@ -218,6 +222,9 @@ def _parse_workflow(item: Any) -> WorkflowEntry:
|
||||
if trigger == "scheduled" and (not isinstance(task_name, str) or not task_name):
|
||||
raise CatalogError(f"scheduled workflow requires source task name: {workflow_id}")
|
||||
topic_config = _parse_topic_config(item.get("topic_config"), workflow_id)
|
||||
required_accounts = _parse_required_accounts(
|
||||
item.get("required_accounts", []), workflow_id
|
||||
)
|
||||
return WorkflowEntry(
|
||||
workflow_id=workflow_id,
|
||||
module=module,
|
||||
@@ -229,9 +236,18 @@ def _parse_workflow(item: Any) -> WorkflowEntry:
|
||||
note=item.get("note"),
|
||||
steps=steps,
|
||||
topic_config=topic_config,
|
||||
required_accounts=required_accounts,
|
||||
)
|
||||
|
||||
|
||||
def _parse_required_accounts(value: Any, workflow_id: str) -> tuple[str, ...]:
|
||||
if not isinstance(value, list) or not all(
|
||||
isinstance(item, str) and _WORKFLOW_ID.fullmatch(item) for item in value
|
||||
):
|
||||
raise CatalogError(f"invalid required_accounts for {workflow_id}")
|
||||
return tuple(dict.fromkeys(value))
|
||||
|
||||
|
||||
def _parse_topic_config(value: Any, workflow_id: str) -> WorkflowTopicConfig | None:
|
||||
if value is None:
|
||||
return None
|
||||
|
||||
+219
-1
@@ -229,6 +229,36 @@ def build_parser() -> argparse.ArgumentParser:
|
||||
status_parser = acceptance_commands.add_parser("status")
|
||||
status_parser.add_argument("--json", action="store_true")
|
||||
status_parser.add_argument("--output", type=Path)
|
||||
|
||||
audit_parser = commands.add_parser(
|
||||
"audit", help="query the PostgreSQL operational audit tables"
|
||||
)
|
||||
audit_commands = audit_parser.add_subparsers(dest="audit_command", required=True)
|
||||
|
||||
audit_runs = audit_commands.add_parser("runs", help="recent workflow runs")
|
||||
audit_runs.add_argument("--workflow")
|
||||
audit_runs.add_argument("--status")
|
||||
audit_runs.add_argument("--date", dest="business_date")
|
||||
audit_runs.add_argument("--limit", type=int, default=30)
|
||||
audit_runs.add_argument("--json", action="store_true")
|
||||
|
||||
audit_cookies = audit_commands.add_parser(
|
||||
"cookies", help="recent cookie observations"
|
||||
)
|
||||
audit_cookies.add_argument("--account")
|
||||
audit_cookies.add_argument("--state")
|
||||
audit_cookies.add_argument("--limit", type=int, default=30)
|
||||
audit_cookies.add_argument("--json", action="store_true")
|
||||
|
||||
audit_stats = audit_commands.add_parser(
|
||||
"cookie-stats", help="measured cookie lifetime per account"
|
||||
)
|
||||
audit_stats.add_argument("--account")
|
||||
audit_stats.add_argument("--json", action="store_true")
|
||||
|
||||
audit_commands.add_parser(
|
||||
"schema", help="print the DDL that creates the audit tables"
|
||||
)
|
||||
return parser
|
||||
|
||||
|
||||
@@ -304,6 +334,8 @@ def main(
|
||||
and arguments.acceptance_command == "status"
|
||||
):
|
||||
return _acceptance_status_command(arguments, resolved_settings, output)
|
||||
if arguments.command == "audit":
|
||||
return _audit_command(arguments, resolved_settings, output)
|
||||
if arguments.command == "scripts" and arguments.scripts_command == "list":
|
||||
return _list_scripts(arguments, output)
|
||||
if arguments.command == "scripts" and arguments.scripts_command == "run":
|
||||
@@ -549,6 +581,152 @@ def _acceptance_status_command(
|
||||
return EXIT_SUCCESS if report.is_complete else EXIT_ACCEPTANCE_INCOMPLETE
|
||||
|
||||
|
||||
def _format_seconds(value: object) -> str:
|
||||
"""Human-readable duration for the audit tables (None -> '-')."""
|
||||
|
||||
if value is None:
|
||||
return "-"
|
||||
try:
|
||||
seconds = int(value)
|
||||
except (TypeError, ValueError):
|
||||
return "-"
|
||||
if seconds >= 86400:
|
||||
return f"{seconds / 86400:.1f}d"
|
||||
if seconds >= 3600:
|
||||
return f"{seconds / 3600:.1f}h"
|
||||
if seconds >= 60:
|
||||
return f"{seconds / 60:.0f}m"
|
||||
return f"{seconds}s"
|
||||
|
||||
|
||||
def _format_timestamp(value: object) -> str:
|
||||
if value is None:
|
||||
return "-"
|
||||
text = value.isoformat() if hasattr(value, "isoformat") else str(value)
|
||||
return text.replace("T", " ")[:19]
|
||||
|
||||
|
||||
def _audit_command(
|
||||
arguments: argparse.Namespace, settings: Settings, output: TextIO
|
||||
) -> int:
|
||||
from gyxx_flow.ops.audit import SCHEMA_SQL, AuditStore
|
||||
|
||||
command = arguments.audit_command
|
||||
if command == "schema":
|
||||
output.write(SCHEMA_SQL.strip() + "\n")
|
||||
return EXIT_SUCCESS
|
||||
|
||||
store = AuditStore(settings.data_root)
|
||||
if not store.enabled:
|
||||
output.write("审计已关闭(GYXX_OPS_AUDIT=off)\n")
|
||||
return EXIT_CONFIGURATION
|
||||
|
||||
if command == "runs":
|
||||
rows = store.query_runs(
|
||||
workflow_id=arguments.workflow,
|
||||
status=arguments.status,
|
||||
business_date=arguments.business_date,
|
||||
limit=arguments.limit,
|
||||
)
|
||||
elif command == "cookies":
|
||||
rows = store.query_cookie_events(
|
||||
account_id=arguments.account,
|
||||
state=arguments.state,
|
||||
limit=arguments.limit,
|
||||
)
|
||||
elif command == "cookie-stats":
|
||||
rows = (
|
||||
store.query_cookie_lifetimes(account_id=arguments.account)
|
||||
if arguments.account
|
||||
else store.query_cookie_stats()
|
||||
)
|
||||
else: # pragma: no cover - argparse 已限制取值
|
||||
raise CliConfigurationError(f"unsupported audit command: {command}")
|
||||
|
||||
if store.last_error and not rows:
|
||||
output.write(f"审计库不可用:{store.last_error}\n")
|
||||
return EXIT_CONFIGURATION
|
||||
|
||||
if getattr(arguments, "json", False):
|
||||
_write_json(output, {"rows": rows})
|
||||
return EXIT_SUCCESS
|
||||
|
||||
if not rows:
|
||||
output.write("暂无审计记录\n")
|
||||
return EXIT_SUCCESS
|
||||
|
||||
if command == "runs":
|
||||
output.write(
|
||||
f"{'started':19s} {'workflow':40s} {'date':10s} {'mode':7s} "
|
||||
f"{'status':8s} {'dur':>6s} steps\n"
|
||||
)
|
||||
for row in rows:
|
||||
steps = (
|
||||
f"{row.get('step_success', 0)}/{row.get('step_total', 0)}"
|
||||
f" failed={row.get('step_failed', 0)}"
|
||||
)
|
||||
output.write(
|
||||
f"{_format_timestamp(row.get('started_at')):19s} "
|
||||
f"{str(row.get('workflow_id', ''))[:40]:40s} "
|
||||
f"{_format_timestamp(row.get('business_date'))[:10]:10s} "
|
||||
f"{str(row.get('mode', '')):7s} "
|
||||
f"{str(row.get('status', '')):8s} "
|
||||
f"{_format_seconds(row.get('duration_seconds')):>6s} {steps}\n"
|
||||
)
|
||||
if row.get("error"):
|
||||
output.write(f"{'':21s}error: {str(row['error'])[:160]}\n")
|
||||
return EXIT_SUCCESS
|
||||
|
||||
if command == "cookies":
|
||||
output.write(
|
||||
f"{'observed':19s} {'account':24s} {'state':14s} {'probe':14s} "
|
||||
f"{'fingerprint':12s} expires_at\n"
|
||||
)
|
||||
for row in rows:
|
||||
output.write(
|
||||
f"{_format_timestamp(row.get('observed_at')):19s} "
|
||||
f"{str(row.get('account_id', ''))[:24]:24s} "
|
||||
f"{str(row.get('state', '')):14s} "
|
||||
f"{str(row.get('probe_status', '')):14s} "
|
||||
f"{str(row.get('fingerprint', ''))[:12]:12s} "
|
||||
f"{_format_timestamp(row.get('declared_expires_at'))}\n"
|
||||
)
|
||||
return EXIT_SUCCESS
|
||||
|
||||
# cookie-stats / 单账号 lifetime
|
||||
if arguments.account:
|
||||
output.write(
|
||||
f"{'fingerprint':12s} {'first_seen':19s} {'last_healthy':19s} "
|
||||
f"{'failed_at':19s} {'lifetime':>9s} {'declared':19s}\n"
|
||||
)
|
||||
for row in rows:
|
||||
output.write(
|
||||
f"{str(row.get('fingerprint', ''))[:12]:12s} "
|
||||
f"{_format_timestamp(row.get('first_seen_at')):19s} "
|
||||
f"{_format_timestamp(row.get('last_healthy_at')):19s} "
|
||||
f"{_format_timestamp(row.get('first_failed_at')):19s} "
|
||||
f"{_format_seconds(row.get('lifetime_seconds')):>9s} "
|
||||
f"{_format_timestamp(row.get('declared_expires_at')):19s}\n"
|
||||
)
|
||||
return EXIT_SUCCESS
|
||||
|
||||
output.write(
|
||||
f"{'account':24s} {'sessions':>8s} {'expired':>7s} {'avg':>7s} "
|
||||
f"{'median':>7s} {'min':>7s} {'max':>7s} last_failed\n"
|
||||
)
|
||||
for row in rows:
|
||||
output.write(
|
||||
f"{str(row.get('account_id', ''))[:24]:24s} "
|
||||
f"{row.get('sessions', 0):>8d} {row.get('expired_sessions', 0):>7d} "
|
||||
f"{_format_seconds(row.get('avg_lifetime_seconds')):>7s} "
|
||||
f"{_format_seconds(row.get('median_lifetime_seconds')):>7s} "
|
||||
f"{_format_seconds(row.get('min_lifetime_seconds')):>7s} "
|
||||
f"{_format_seconds(row.get('max_lifetime_seconds')):>7s} "
|
||||
f"{_format_timestamp(row.get('last_failed_at'))}\n"
|
||||
)
|
||||
return EXIT_SUCCESS
|
||||
|
||||
|
||||
def _list_workflows(
|
||||
arguments: argparse.Namespace,
|
||||
registry: WorkflowRegistry,
|
||||
@@ -703,7 +881,19 @@ def _python_scheduler(
|
||||
if enable_account_keepalive and not dry_run:
|
||||
from gyxx_flow.accounts import AccountKeepaliveManager
|
||||
|
||||
maintenance = AccountKeepaliveManager(settings).tick
|
||||
keepalive_manager = AccountKeepaliveManager(settings)
|
||||
recovery_manager = _build_cookie_recovery_manager(
|
||||
settings, launcher=launcher, keepalive=keepalive_manager
|
||||
)
|
||||
|
||||
def maintenance() -> None: # noqa: D107 - scheduler maintenance chain
|
||||
keepalive_manager.tick()
|
||||
if recovery_manager is not None:
|
||||
try:
|
||||
recovery_manager.tick()
|
||||
except Exception: # noqa: BLE001 - 自动重跑不得影响调度心跳
|
||||
pass
|
||||
|
||||
return PythonScheduler(
|
||||
catalog,
|
||||
settings.data_root,
|
||||
@@ -717,6 +907,29 @@ def _python_scheduler(
|
||||
)
|
||||
|
||||
|
||||
def _build_cookie_recovery_manager(
|
||||
settings: Settings,
|
||||
*,
|
||||
launcher: object,
|
||||
keepalive: object,
|
||||
):
|
||||
"""Build the cookie-gate recovery rerun manager; None when disabled."""
|
||||
|
||||
from gyxx_flow.cookie_gate import (
|
||||
CookieGate,
|
||||
CookieGateRecoveryManager,
|
||||
gate_enabled,
|
||||
)
|
||||
|
||||
if not gate_enabled():
|
||||
return None
|
||||
try:
|
||||
gate = CookieGate(settings, keepalive=keepalive) # type: ignore[arg-type]
|
||||
except Exception: # noqa: BLE001 - 无集成配置时没有可恢复的账号
|
||||
return None
|
||||
return CookieGateRecoveryManager(gate, launcher=launcher)
|
||||
|
||||
|
||||
def _schedule_run_command(
|
||||
arguments: argparse.Namespace,
|
||||
catalog: WorkflowCatalog,
|
||||
@@ -885,6 +1098,11 @@ def _execute_once(
|
||||
dry_run: bool,
|
||||
settings: Settings,
|
||||
) -> tuple[int, dict[str, object]]:
|
||||
if not dry_run:
|
||||
# 每账号 Cookie 健康门控:dry-run 不注入,也不做任何探测/通知。
|
||||
from gyxx_flow.cookie_gate import apply_cookie_gates
|
||||
|
||||
workflow = apply_cookie_gates(workflow, settings=settings)
|
||||
context = RunContext.create(
|
||||
workflow.workflow_id,
|
||||
business_date.isoformat(),
|
||||
|
||||
+118
-2
@@ -5,6 +5,7 @@ from __future__ import annotations
|
||||
import calendar
|
||||
import hashlib
|
||||
import hmac
|
||||
import http.client
|
||||
import ipaddress
|
||||
import json
|
||||
import os
|
||||
@@ -15,6 +16,7 @@ import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
import traceback
|
||||
import unicodedata
|
||||
import uuid
|
||||
from contextlib import suppress
|
||||
@@ -24,7 +26,7 @@ from http import HTTPStatus
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
from importlib import resources
|
||||
from pathlib import Path
|
||||
from typing import Any, Mapping, Protocol, Sequence, TextIO, cast
|
||||
from typing import Any, Callable, Mapping, Protocol, Sequence, TextIO, cast
|
||||
from urllib.parse import parse_qs, unquote, urlsplit
|
||||
from zoneinfo import ZoneInfo
|
||||
|
||||
@@ -171,7 +173,6 @@ _WORKFLOW_NAMES = {
|
||||
"content.metrics.daily": "合作达人内容指标日报采集",
|
||||
"content.metrics.self_operated.daily": "自营内容指标日报采集",
|
||||
"content.marketing_report.daily": "营销日报生成与发送",
|
||||
"content.relogin.weekly": "内容平台登录态刷新",
|
||||
"content.creator_report.monthly": "达人月报生成",
|
||||
"content.summary.monthly": "内容月度汇总",
|
||||
"content.comments.weekly": "多平台评论周采集",
|
||||
@@ -3736,6 +3737,7 @@ class WorkflowConsoleHTTPServer(ThreadingHTTPServer):
|
||||
self.console_service = service
|
||||
self.console_token = token
|
||||
self.bind_host = bind_host
|
||||
self.started_at = time.time()
|
||||
super().__init__(server_address, WorkflowConsoleRequestHandler)
|
||||
|
||||
def handle_error(
|
||||
@@ -3796,6 +3798,29 @@ class WorkflowConsoleRequestHandler(BaseHTTPRequestHandler):
|
||||
if method in {"GET", "HEAD"} and path == "/assets/app.js":
|
||||
self._static("app.js", "text/javascript; charset=utf-8", head=method == "HEAD")
|
||||
return
|
||||
if method == "GET" and path == "/api/health":
|
||||
# 自检探针端点:不鉴权、不触外部依赖,只反映进程与调度锁存活,
|
||||
# 供进程内看门狗与 deploy/watchdog 保活脚本探测。
|
||||
try:
|
||||
scheduler = _scheduler_process_from_lock(
|
||||
self.console_server.console_service.settings
|
||||
)
|
||||
except Exception:
|
||||
scheduler = None
|
||||
self._json(
|
||||
HTTPStatus.OK,
|
||||
{
|
||||
"ok": True,
|
||||
"pid": os.getpid(),
|
||||
"started_at": self.console_server.started_at,
|
||||
"uptime_seconds": round(
|
||||
time.time() - self.console_server.started_at, 1
|
||||
),
|
||||
"scheduler_running": scheduler is not None,
|
||||
"scheduler_pid": scheduler.pid if scheduler else None,
|
||||
},
|
||||
)
|
||||
return
|
||||
if method == "GET" and path == "/api/overview":
|
||||
self._require_api_access(mutation=False)
|
||||
self._json(HTTPStatus.OK, self.console_server.console_service.overview())
|
||||
@@ -4020,6 +4045,9 @@ class WorkflowConsoleRequestHandler(BaseHTTPRequestHandler):
|
||||
except (BrokenPipeError, ConnectionResetError):
|
||||
return
|
||||
except Exception:
|
||||
# 前端只拿到固定文案;完整堆栈落到 stderr(随启动重定向进
|
||||
# var/logs/console-restarts 或服务日志),否则故障无法定位。
|
||||
traceback.print_exc()
|
||||
self._json_error(HTTPStatus.INTERNAL_SERVER_ERROR, "控制台处理请求失败")
|
||||
|
||||
def _require_api_access(self, *, mutation: bool) -> None:
|
||||
@@ -4190,6 +4218,92 @@ def create_console_server(
|
||||
)
|
||||
|
||||
|
||||
CONSOLE_WATCHDOG_EXIT_CODE = 70
|
||||
|
||||
|
||||
def _probe_console_health(port: int, timeout_seconds: float) -> bool:
|
||||
# 只连本机回环端口的固定路径 /api/health,探测自身 HTTP 服务往返。
|
||||
connection = http.client.HTTPConnection("127.0.0.1", port, timeout=timeout_seconds)
|
||||
try:
|
||||
connection.request("GET", "/api/health")
|
||||
response = connection.getresponse()
|
||||
response.read()
|
||||
return response.status == HTTPStatus.OK
|
||||
except Exception:
|
||||
return False
|
||||
finally:
|
||||
connection.close()
|
||||
|
||||
|
||||
def _console_watchdog(
|
||||
port: int,
|
||||
*,
|
||||
interval_seconds: float,
|
||||
failure_limit: int,
|
||||
probe_timeout_seconds: float,
|
||||
stopped: threading.Event,
|
||||
probe: Callable[[int, float], bool] | None = None,
|
||||
exit_process: Callable[[int], None] = os._exit,
|
||||
sleep: Callable[[float], None] = time.sleep,
|
||||
) -> None:
|
||||
"""进程内自检:周期探测自身 /api/health,连续失败则退出交给守护拉起。
|
||||
|
||||
HTTP accept 线程活着但工作线程全部卡死(锁竞争/句柄耗尽)时,外部
|
||||
静态页能打开而 API 全挂——正是"控制台暂时不可用"的形态;只有真实
|
||||
走一遍 HTTP 往返才能暴露,所以探针必须打 socket 而不是查内存状态。
|
||||
"""
|
||||
run_probe = probe or _probe_console_health
|
||||
failures = 0
|
||||
while not stopped.is_set():
|
||||
if run_probe(port, probe_timeout_seconds):
|
||||
failures = 0
|
||||
else:
|
||||
failures += 1
|
||||
print(
|
||||
f"⚠️ 控制台自检失败 {failures}/{failure_limit}: 127.0.0.1:{port}/api/health",
|
||||
file=sys.stderr,
|
||||
flush=True,
|
||||
)
|
||||
if failures >= failure_limit:
|
||||
print(
|
||||
"❌ 控制台自检连续失败,进程退出等待守护脚本拉起",
|
||||
file=sys.stderr,
|
||||
flush=True,
|
||||
)
|
||||
exit_process(CONSOLE_WATCHDOG_EXIT_CODE)
|
||||
return
|
||||
sleep(interval_seconds)
|
||||
|
||||
|
||||
def _start_console_watchdog(server: WorkflowConsoleHTTPServer) -> threading.Event:
|
||||
interval = max(
|
||||
5.0, float(os.environ.get("GYXX_CONSOLE_WATCHDOG_INTERVAL_SECONDS", "30"))
|
||||
)
|
||||
limit = int(os.environ.get("GYXX_CONSOLE_WATCHDOG_FAILURES", "3"))
|
||||
enabled = (
|
||||
os.environ.get("GYXX_CONSOLE_WATCHDOG", "1").strip().lower()
|
||||
not in {"0", "false", "no", "off"}
|
||||
and limit > 0
|
||||
)
|
||||
stopped = threading.Event()
|
||||
if not enabled:
|
||||
return stopped
|
||||
thread = threading.Thread(
|
||||
target=_console_watchdog,
|
||||
args=(server.server_port,),
|
||||
kwargs={
|
||||
"interval_seconds": interval,
|
||||
"failure_limit": limit,
|
||||
"probe_timeout_seconds": min(5.0, interval),
|
||||
"stopped": stopped,
|
||||
},
|
||||
name="gyxx-console-watchdog",
|
||||
daemon=True,
|
||||
)
|
||||
thread.start()
|
||||
return stopped
|
||||
|
||||
|
||||
def serve_console(
|
||||
settings: Settings,
|
||||
*,
|
||||
@@ -4211,11 +4325,13 @@ def serve_console(
|
||||
display_host = "127.0.0.1" if host in {"0.0.0.0", "::"} else host
|
||||
output.write(f"GYXX Flow console: http://{display_host}:{server.server_port}\n")
|
||||
output.flush()
|
||||
watchdog_stopped = _start_console_watchdog(server)
|
||||
try:
|
||||
server.serve_forever(poll_interval=0.5)
|
||||
except KeyboardInterrupt:
|
||||
pass
|
||||
finally:
|
||||
watchdog_stopped.set()
|
||||
server.server_close()
|
||||
return 0
|
||||
|
||||
|
||||
@@ -0,0 +1,966 @@
|
||||
"""Per-account cookie health gate injected before workflow collection steps.
|
||||
|
||||
Every runtime account a workflow depends on is checked before the collection
|
||||
steps that use it may run. Accounts fall into three gate modes derived from
|
||||
``config/runtime-bindings.json``:
|
||||
|
||||
* ``blocking`` -- QR-code / SMS-only accounts (no credential environment and
|
||||
no configured login command). The gate verifies the login state and, when
|
||||
it has expired, fails the gate step so every dependent step is skipped with
|
||||
``dependency-failed`` and sends a Feishu alert asking the login owner to
|
||||
re-login interactively via ``gyxx accounts login <account_id>``.
|
||||
* ``recover`` -- accounts with a vault-managed credential login command
|
||||
(``login_mode`` ``C``). The keepalive manager re-runs the credential login
|
||||
automatically; only a failed recovery blocks the workflow and alerts.
|
||||
* ``defer`` -- accounts whose member scripts embed username/password login
|
||||
flows. These keep their script-level login behaviour unchanged and are not
|
||||
gated at all.
|
||||
|
||||
The per-account decision is persisted as a small state machine under
|
||||
``state/accounts/<id>/cookie_state.json`` (``healthy`` <-> ``awaiting_login``)
|
||||
so repeat failures alert at most once per notify interval instead of once per
|
||||
workflow run.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
from collections.abc import Callable
|
||||
from dataclasses import dataclass, replace
|
||||
from datetime import date, datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any, Mapping
|
||||
|
||||
from gyxx_flow.accounts import (
|
||||
AccountKeepaliveManager,
|
||||
_keepalive_requires_relogin,
|
||||
_read_keepalive_status,
|
||||
_vault_cookie_valid,
|
||||
)
|
||||
from gyxx_flow.adapters.browser import BrowserCookieStore
|
||||
from gyxx_flow.adapters.feishu_messages import send_lark_bot_message
|
||||
from gyxx_flow.adapters.integration import RuntimeIntegrationCatalog
|
||||
from gyxx_flow.catalog import WorkflowCatalog
|
||||
from gyxx_flow.core.artifacts import atomic_write_json
|
||||
from gyxx_flow.core.config import Settings
|
||||
from gyxx_flow.core.context import RunContext
|
||||
from gyxx_flow.notification_routing import (
|
||||
ANALYZER_ACCEPTANCE_RECIPIENT_OPEN_ID,
|
||||
ANALYZER_NOTIFICATION_APP_PROFILE,
|
||||
NotificationRoutingStore,
|
||||
)
|
||||
from gyxx_flow.workflow.model import StepDefinition, WorkflowDefinition
|
||||
from gyxx_flow.workflow.steps import StepExecution
|
||||
|
||||
GATE_MODE_BLOCKING = "blocking"
|
||||
GATE_MODE_RECOVER = "recover"
|
||||
GATE_MODE_DEFER = "defer"
|
||||
|
||||
# 飞书通知的默认接收人:按手机号在 config/notification-routing.json 的
|
||||
# people 段解析 open_id;解析失败时回退到同名 analyzer 常量收件人。
|
||||
DEFAULT_LOGIN_ALERT_MOBILE = "+8613663785145"
|
||||
|
||||
GYXX_COOKIE_GATE = "GYXX_COOKIE_GATE"
|
||||
GYXX_COOKIE_GATE_MAX_STATUS_AGE_SECONDS = "GYXX_COOKIE_GATE_MAX_STATUS_AGE_SECONDS"
|
||||
GYXX_LOGIN_ALERT_OPEN_IDS = "GYXX_LOGIN_ALERT_OPEN_IDS"
|
||||
GYXX_LOGIN_ALERT_APP_PROFILE = "GYXX_LOGIN_ALERT_APP_PROFILE"
|
||||
GYXX_LOGIN_ALERT_NOTIFY_INTERVAL_SECONDS = "GYXX_LOGIN_ALERT_NOTIFY_INTERVAL_SECONDS"
|
||||
|
||||
_GATE_STEP_PREFIX = "cookie-gate"
|
||||
_GATE_STATE_FILENAME = "cookie_state.json"
|
||||
_GATE_TIMEOUT_SECONDS = 1800.0
|
||||
_DEFAULT_MAX_STATUS_AGE_SECONDS = 3600.0
|
||||
_MIN_MAX_STATUS_AGE_SECONDS = 300.0
|
||||
_DEFAULT_NOTIFY_INTERVAL_SECONDS = 1800.0
|
||||
_DISABLED_VALUES = frozenset({"0", "false", "off", "no", "disabled"})
|
||||
# 自动重跑:待重跑清单上限、健康探活节流、同账号重跑启动冷却。
|
||||
_MAX_PENDING_RERUNS = 20
|
||||
_RECOVERY_PROBE_INTERVAL_SECONDS = 300.0
|
||||
_RECOVERY_LAUNCH_COOLDOWN_SECONDS = 300.0
|
||||
_RECOVERY_HISTORY_LIMIT = 20
|
||||
|
||||
STATE_HEALTHY = "healthy"
|
||||
STATE_AWAITING_LOGIN = "awaiting_login"
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class GateVerdict:
|
||||
"""One account's gate decision, recorded in the run journal on failure."""
|
||||
|
||||
account_id: str
|
||||
mode: str
|
||||
blocked: bool
|
||||
message: str
|
||||
recovered: bool = False
|
||||
notified: bool = False
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class CookieGateStepAction:
|
||||
"""ExecutableStep that turns one account verdict into a step outcome."""
|
||||
|
||||
gate: "CookieGate"
|
||||
account_id: str
|
||||
|
||||
def execute(
|
||||
self,
|
||||
*,
|
||||
context: RunContext,
|
||||
timeout_seconds: float | None,
|
||||
dry_run: bool,
|
||||
) -> StepExecution:
|
||||
if dry_run: # pragma: no cover - the engine skips dry-run steps first
|
||||
return StepExecution(exit_code=0, skipped=True, reason="dry-run")
|
||||
verdict = self.gate.check(self.account_id, context=context)
|
||||
if verdict.blocked:
|
||||
return StepExecution(
|
||||
exit_code=1,
|
||||
error=verdict.message,
|
||||
reason="cookie-expired",
|
||||
)
|
||||
return StepExecution(exit_code=0)
|
||||
|
||||
|
||||
class CookieGate:
|
||||
"""Resolve, probe, and gate the accounts one workflow depends on."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
settings: Settings,
|
||||
*,
|
||||
catalog: RuntimeIntegrationCatalog | None = None,
|
||||
keepalive: AccountKeepaliveManager | None = None,
|
||||
environment: Mapping[str, str] | None = None,
|
||||
now: Callable[[], datetime] | None = None,
|
||||
sender: Callable[..., dict[str, Any]] | None = None,
|
||||
) -> None: # noqa: D107 - simple wiring
|
||||
self.settings = settings
|
||||
self.catalog = catalog or RuntimeIntegrationCatalog.load_default(
|
||||
project_root=settings.project_root,
|
||||
data_root=settings.data_root,
|
||||
)
|
||||
self.keepalive = keepalive or AccountKeepaliveManager(
|
||||
settings, catalog=self.catalog
|
||||
)
|
||||
self._environment = dict(os.environ if environment is None else environment)
|
||||
self._now = now or (lambda: datetime.now(timezone.utc))
|
||||
self._sender = sender or send_lark_bot_message
|
||||
self._workflow_catalog: WorkflowCatalog | None = None
|
||||
self._audit_store: Any | None = None
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Account resolution
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def gate_mode(self, account_id: str) -> str:
|
||||
account = self.catalog.account_for(account_id)
|
||||
if str(account.login_command_id or "").strip():
|
||||
return GATE_MODE_RECOVER
|
||||
if account.credential_env_names or account.login_mode == "B":
|
||||
return GATE_MODE_DEFER
|
||||
return GATE_MODE_BLOCKING
|
||||
|
||||
def collect_workflow_accounts(
|
||||
self,
|
||||
workflow: WorkflowDefinition,
|
||||
) -> dict[str, set[str]]:
|
||||
"""Map each gated account to the step ids that depend on it.
|
||||
|
||||
Accounts resolve two ways: directly, from each step binding's
|
||||
``account``; and declaratively, from the catalog workflow's
|
||||
``required_accounts`` (used by orchestrator entries whose own binding
|
||||
has no account while the spawned child collectors do).
|
||||
"""
|
||||
|
||||
accounts: dict[str, set[str]] = {}
|
||||
for step in workflow.steps:
|
||||
account_id = self._account_for_step(step)
|
||||
if account_id:
|
||||
accounts.setdefault(account_id, set()).add(step.step_id)
|
||||
for account_id in self._declared_required_accounts(workflow.workflow_id):
|
||||
self.catalog.account_for(account_id) # unknown id: loud config error
|
||||
accounts.setdefault(account_id, set()).update(
|
||||
step.step_id for step in workflow.steps
|
||||
)
|
||||
# Defer-mode accounts keep their script-level credential login flows
|
||||
# and are intentionally not gated.
|
||||
return {
|
||||
account_id: step_ids
|
||||
for account_id, step_ids in accounts.items()
|
||||
if self.gate_mode(account_id) != GATE_MODE_DEFER
|
||||
}
|
||||
|
||||
def _account_for_step(self, step: StepDefinition) -> str | None:
|
||||
action = step.action
|
||||
entry = getattr(action, "entry", None)
|
||||
module_id = getattr(entry, "module", "")
|
||||
command_entry = getattr(action, "command_entry", None) or getattr(
|
||||
entry, "entry", ""
|
||||
)
|
||||
if not module_id or not command_entry:
|
||||
return None
|
||||
try:
|
||||
binding = self.catalog.binding_for(f"{module_id}:{command_entry}")
|
||||
except Exception: # noqa: BLE001 - unbound entries run ungated
|
||||
return None
|
||||
return binding.account or None
|
||||
|
||||
def workflow_catalog(self) -> WorkflowCatalog:
|
||||
"""Load (once) the catalog used for declared account and rerun lookups."""
|
||||
|
||||
if self._workflow_catalog is None:
|
||||
try:
|
||||
self._workflow_catalog = WorkflowCatalog.load(
|
||||
self.settings.project_root / "config"
|
||||
)
|
||||
except Exception: # noqa: BLE001 - catalog issues surface elsewhere
|
||||
self._workflow_catalog = WorkflowCatalog(
|
||||
timezone="UTC", workflows=(), schedules=()
|
||||
)
|
||||
return self._workflow_catalog
|
||||
|
||||
def _declared_required_accounts(self, workflow_id: str) -> tuple[str, ...]:
|
||||
for entry in self.workflow_catalog().workflows:
|
||||
if entry.workflow_id == workflow_id:
|
||||
return tuple(entry.required_accounts)
|
||||
return ()
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# State machine
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def check(self, account_id: str, *, context: RunContext) -> GateVerdict:
|
||||
account = self.catalog.account_for(account_id)
|
||||
mode = self.gate_mode(account_id)
|
||||
store = BrowserCookieStore(account.cookie_file, account.storage_state_file)
|
||||
static_valid = _vault_cookie_valid(account, store)
|
||||
status = _read_keepalive_status(account)
|
||||
if static_valid and self._keepalive_success_is_fresh(status):
|
||||
self._write_state(account, STATE_HEALTHY, probe="fresh-keepalive", context=context)
|
||||
return GateVerdict(
|
||||
account_id=account_id,
|
||||
mode=mode,
|
||||
blocked=False,
|
||||
message=f"账号 {account_id} 登录态有效(keepalive 近期验证通过)",
|
||||
)
|
||||
if self._acceptance_enabled():
|
||||
# 验收模式不开浏览器:按静态证据判定,且不投递飞书通知。
|
||||
if static_valid and not _keepalive_requires_relogin(status):
|
||||
return GateVerdict(
|
||||
account_id=account_id,
|
||||
mode=mode,
|
||||
blocked=False,
|
||||
message=f"账号 {account_id} 静态 cookie 校验通过(验收模式)",
|
||||
)
|
||||
return self._expire(
|
||||
account,
|
||||
mode,
|
||||
context,
|
||||
detail="验收模式下静态 cookie 校验未通过",
|
||||
notify=False,
|
||||
)
|
||||
|
||||
outcome = self.keepalive.refresh_account(account_id)
|
||||
probe_status = str(outcome.get("status", "error"))
|
||||
if probe_status == "success":
|
||||
recovered = bool(outcome.get("login_recovered"))
|
||||
self._write_state(
|
||||
account,
|
||||
STATE_HEALTHY,
|
||||
probe="success",
|
||||
recovered=recovered,
|
||||
context=context,
|
||||
)
|
||||
message = (
|
||||
f"账号 {account_id} 登录态有效(自动账密重登成功)"
|
||||
if recovered
|
||||
else f"账号 {account_id} 登录态有效(探活通过)"
|
||||
)
|
||||
return GateVerdict(
|
||||
account_id=account_id,
|
||||
mode=mode,
|
||||
blocked=False,
|
||||
message=message,
|
||||
recovered=recovered,
|
||||
)
|
||||
if probe_status == "busy":
|
||||
if static_valid and not _keepalive_requires_relogin(status):
|
||||
return GateVerdict(
|
||||
account_id=account_id,
|
||||
mode=mode,
|
||||
blocked=False,
|
||||
message=f"账号 {account_id} 探活锁忙,静态 cookie 校验通过",
|
||||
)
|
||||
return self._expire(
|
||||
account,
|
||||
mode,
|
||||
context,
|
||||
detail="探活锁被占用且静态 cookie 校验未通过",
|
||||
)
|
||||
if probe_status in {"expired", "not_logged_in"}:
|
||||
evidence = _recovery_evidence(_read_keepalive_status(account))
|
||||
detail = "cookie 已过期,无法正常采集"
|
||||
if probe_status == "not_logged_in" and evidence:
|
||||
detail = f"登录态缺失;{evidence}"
|
||||
elif evidence:
|
||||
detail = f"cookie 已过期;{evidence}"
|
||||
return self._expire(account, mode, context, detail=detail)
|
||||
# timeout / error / http_error / disabled:探测结果不可信时回退静态证据。
|
||||
if static_valid and not _keepalive_requires_relogin(status):
|
||||
return GateVerdict(
|
||||
account_id=account_id,
|
||||
mode=mode,
|
||||
blocked=False,
|
||||
message=f"账号 {account_id} 探活异常({probe_status}),静态 cookie 校验通过",
|
||||
)
|
||||
return self._expire(
|
||||
account,
|
||||
mode,
|
||||
context,
|
||||
detail=f"探活异常({probe_status})且静态 cookie 校验未通过",
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Expiry, notification, persisted state
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _expire(
|
||||
self,
|
||||
account: Any,
|
||||
mode: str,
|
||||
context: RunContext,
|
||||
*,
|
||||
detail: str,
|
||||
notify: bool | None = None,
|
||||
) -> GateVerdict:
|
||||
account_id = str(account.account_id)
|
||||
if notify is None:
|
||||
notify = not context.shadow and not self._acceptance_enabled()
|
||||
state = self._read_state(account)
|
||||
notified = False
|
||||
notification_note = ""
|
||||
if notify and not self._within_notify_interval(state):
|
||||
recipients = self._alert_recipients()
|
||||
message = _expiry_message(account_id, mode, context, detail)
|
||||
for open_id, profile in recipients:
|
||||
try:
|
||||
self._sender(
|
||||
user_id=open_id,
|
||||
text=message,
|
||||
idempotency_key=f"cookie-gate:{account_id}:{context.run_id}",
|
||||
profile=profile,
|
||||
)
|
||||
notified = True
|
||||
except Exception as exc: # noqa: BLE001 - alerting must not mask the verdict
|
||||
notification_note = f"{type(exc).__name__}: {exc}"[:300]
|
||||
if recipients and notified:
|
||||
state["last_notified_at"] = _now_iso(self._now)
|
||||
state.pop("notification_error", None)
|
||||
elif notification_note:
|
||||
state["notification_error"] = notification_note
|
||||
state["state"] = STATE_AWAITING_LOGIN
|
||||
state["updated_at"] = _now_iso(self._now)
|
||||
state["last_probe_status"] = "expired"
|
||||
state["last_blocked_run"] = context.run_id
|
||||
if not context.shadow and not self._acceptance_enabled():
|
||||
# 记录待自动重跑清单:登录态恢复后由调度器维护循环按原业务日期重跑。
|
||||
state["pending_reruns"] = _record_pending_rerun(
|
||||
state.get("pending_reruns"), context, _now_iso(self._now)
|
||||
)
|
||||
self._save_state(account, state)
|
||||
try:
|
||||
keepalive_status = _read_keepalive_status(account) or {}
|
||||
except Exception: # noqa: BLE001 - state file is advisory
|
||||
keepalive_status = {}
|
||||
self._audit_cookie(
|
||||
account,
|
||||
"not_logged_in"
|
||||
if str(keepalive_status.get("status", "")) == "not_logged_in"
|
||||
else "expired",
|
||||
probe_status=str(keepalive_status.get("status", "expired")),
|
||||
probe_source="gate",
|
||||
detail=detail,
|
||||
context=context,
|
||||
)
|
||||
message = _expiry_message(account_id, mode, context, detail)
|
||||
if notified:
|
||||
message += " 已通过飞书通知登录负责人。"
|
||||
elif notification_note:
|
||||
message += f" 飞书通知发送失败({notification_note})。"
|
||||
return GateVerdict(
|
||||
account_id=account_id,
|
||||
mode=mode,
|
||||
blocked=True,
|
||||
message=message,
|
||||
notified=notified,
|
||||
)
|
||||
|
||||
def _write_state(
|
||||
self,
|
||||
account: Any,
|
||||
state_name: str,
|
||||
*,
|
||||
probe: str,
|
||||
recovered: bool = False,
|
||||
context: RunContext | None = None,
|
||||
) -> None:
|
||||
if state_name == STATE_HEALTHY:
|
||||
# 健康心跳要落在每一次判定上,才能算准「最后一次可用」的时间。
|
||||
self._audit_cookie(
|
||||
account,
|
||||
"healthy",
|
||||
probe_status=probe,
|
||||
probe_source="gate",
|
||||
context=context,
|
||||
)
|
||||
state = self._read_state(account)
|
||||
if (
|
||||
state_name == STATE_HEALTHY
|
||||
and state.get("state") == STATE_HEALTHY
|
||||
and state.get("last_probe_status") in {"fresh-keepalive", "success"}
|
||||
and not recovered
|
||||
):
|
||||
return # 持续健康时避免每次运行都重写状态文件
|
||||
state["state"] = state_name
|
||||
state["updated_at"] = _now_iso(self._now)
|
||||
state["last_probe_status"] = probe
|
||||
if state_name == STATE_HEALTHY:
|
||||
state["last_healthy_at"] = _now_iso(self._now)
|
||||
if recovered:
|
||||
state["last_recovered_at"] = _now_iso(self._now)
|
||||
state.pop("notification_error", None)
|
||||
self._save_state(account, state)
|
||||
|
||||
def _state_path(self, account: Any) -> Path:
|
||||
return account.cookie_file.with_name(_GATE_STATE_FILENAME)
|
||||
|
||||
def _read_state(self, account: Any) -> dict[str, object]:
|
||||
path = self._state_path(account)
|
||||
if not path.exists():
|
||||
return {"account_id": str(account.account_id)}
|
||||
try:
|
||||
payload = json.loads(path.read_text(encoding="utf-8"))
|
||||
except (OSError, json.JSONDecodeError):
|
||||
return {"account_id": str(account.account_id)}
|
||||
if not isinstance(payload, dict):
|
||||
return {"account_id": str(account.account_id)}
|
||||
return dict(payload)
|
||||
|
||||
def _save_state(self, account: Any, state: Mapping[str, object]) -> None:
|
||||
payload = {
|
||||
"account_id": str(account.account_id),
|
||||
**state,
|
||||
}
|
||||
try:
|
||||
atomic_write_json(self._state_path(account), payload)
|
||||
except OSError: # pragma: no cover - state file is advisory only
|
||||
pass
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Cookie telemetry (best-effort; never blocks or fails the gate)
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _audit(self) -> Any | None:
|
||||
if self._audit_store is None:
|
||||
try:
|
||||
from gyxx_flow.ops.audit import AuditStore
|
||||
|
||||
self._audit_store = AuditStore(
|
||||
self.settings.data_root,
|
||||
environment=self._environment,
|
||||
)
|
||||
except Exception: # noqa: BLE001 - telemetry is optional
|
||||
return None
|
||||
return self._audit_store
|
||||
|
||||
def _audit_cookie(
|
||||
self,
|
||||
account: Any,
|
||||
state: str,
|
||||
*,
|
||||
probe_status: str,
|
||||
probe_source: str,
|
||||
detail: str = "",
|
||||
context: RunContext | None = None,
|
||||
) -> None:
|
||||
"""Append one cookie observation.
|
||||
|
||||
Stores a value-free fingerprint plus the earliest declared expiry of the
|
||||
required cookies, so failure patterns (how long a login really lasts,
|
||||
and whether it dies before or after the declared expiry) can be derived
|
||||
from history instead of guessed. Errors are swallowed: gating must
|
||||
never depend on the audit database.
|
||||
"""
|
||||
|
||||
try:
|
||||
store = self._audit()
|
||||
if store is None or not store.enabled:
|
||||
return
|
||||
store.record_cookie_event(
|
||||
account_id=str(getattr(account, "account_id", "") or ""),
|
||||
state=state,
|
||||
snapshot=self._cookie_snapshot(account),
|
||||
probe_status=probe_status,
|
||||
probe_source=probe_source,
|
||||
detail=detail,
|
||||
workflow_id=str(getattr(context, "workflow_id", "") or ""),
|
||||
run_id=str(getattr(context, "run_id", "") or ""),
|
||||
)
|
||||
except Exception: # noqa: BLE001 - telemetry is advisory only
|
||||
return
|
||||
|
||||
def _cookie_snapshot(self, account: Any) -> Any | None:
|
||||
from gyxx_flow.ops.audit import snapshot_for_account
|
||||
|
||||
return snapshot_for_account(account)
|
||||
|
||||
def _within_notify_interval(self, state: Mapping[str, object]) -> bool:
|
||||
raw = state.get("last_notified_at")
|
||||
if not isinstance(raw, str):
|
||||
return False
|
||||
try:
|
||||
last = datetime.fromisoformat(raw)
|
||||
except ValueError:
|
||||
return False
|
||||
now = self._now()
|
||||
if last.tzinfo is None:
|
||||
last = last.replace(tzinfo=timezone.utc)
|
||||
interval = _env_int(
|
||||
self._environment,
|
||||
GYXX_LOGIN_ALERT_NOTIFY_INTERVAL_SECONDS,
|
||||
_DEFAULT_NOTIFY_INTERVAL_SECONDS,
|
||||
minimum=0,
|
||||
)
|
||||
return (now - last).total_seconds() < interval
|
||||
|
||||
def _keepalive_success_is_fresh(self, status: Mapping[str, object] | None) -> bool:
|
||||
if not isinstance(status, Mapping) or status.get("status") != "success":
|
||||
return False
|
||||
raw = status.get("last_success_at")
|
||||
if not isinstance(raw, str):
|
||||
return False
|
||||
try:
|
||||
last = datetime.fromisoformat(raw)
|
||||
except ValueError:
|
||||
return False
|
||||
now = self._now()
|
||||
if last.tzinfo is None:
|
||||
last = last.replace(tzinfo=timezone.utc)
|
||||
max_age = _env_float(
|
||||
self._environment,
|
||||
GYXX_COOKIE_GATE_MAX_STATUS_AGE_SECONDS,
|
||||
_DEFAULT_MAX_STATUS_AGE_SECONDS,
|
||||
minimum=_MIN_MAX_STATUS_AGE_SECONDS,
|
||||
)
|
||||
return (now - last).total_seconds() <= max_age
|
||||
|
||||
def _alert_recipients(self) -> list[tuple[str, str]]:
|
||||
profile = (
|
||||
self._environment.get(GYXX_LOGIN_ALERT_APP_PROFILE, "").strip()
|
||||
or ANALYZER_NOTIFICATION_APP_PROFILE
|
||||
)
|
||||
override = [
|
||||
value.strip()
|
||||
for value in self._environment.get(GYXX_LOGIN_ALERT_OPEN_IDS, "").split(",")
|
||||
if value.strip()
|
||||
]
|
||||
if override:
|
||||
return [(open_id, profile) for open_id in override]
|
||||
try:
|
||||
payload, _revision = NotificationRoutingStore(self.settings).snapshot()
|
||||
for person in payload["people"].values():
|
||||
if person.get("mobile") == DEFAULT_LOGIN_ALERT_MOBILE:
|
||||
binding = person.get("bindings", {}).get(profile)
|
||||
if binding:
|
||||
return [(str(binding["open_id"]), profile)]
|
||||
except Exception: # noqa: BLE001 - fall back to the constant recipient
|
||||
pass
|
||||
return [(ANALYZER_ACCEPTANCE_RECIPIENT_OPEN_ID, profile)]
|
||||
|
||||
def _acceptance_enabled(self) -> bool:
|
||||
value = self._environment.get("GYXX_WORKFLOW_ACCEPTANCE", "")
|
||||
return bool(value and value.strip().casefold() in {"1", "true", "yes", "on"})
|
||||
|
||||
|
||||
def gate_enabled(environment: Mapping[str, str] | None = None) -> bool:
|
||||
values = os.environ if environment is None else environment
|
||||
return values.get(GYXX_COOKIE_GATE, "").strip().casefold() not in _DISABLED_VALUES
|
||||
|
||||
|
||||
def apply_cookie_gates(
|
||||
workflow: WorkflowDefinition,
|
||||
*,
|
||||
settings: Settings,
|
||||
environment: Mapping[str, str] | None = None,
|
||||
gate: CookieGate | None = None,
|
||||
) -> WorkflowDefinition:
|
||||
"""Inject one critical gate step per gated account in front of its users.
|
||||
|
||||
Gate steps are graph sources; each step that uses an account gains the
|
||||
account's gate as an extra dependency. A failed gate therefore skips its
|
||||
dependents with ``dependency-failed`` through the regular engine, while
|
||||
unrelated branches keep running. Operator slices (``--rerun-step`` /
|
||||
``--resume-from``) are produced before this injection and stay gate-free.
|
||||
"""
|
||||
|
||||
if not workflow.steps:
|
||||
return workflow
|
||||
if gate is None:
|
||||
try:
|
||||
gate = CookieGate(
|
||||
settings,
|
||||
environment=os.environ if environment is None else environment,
|
||||
)
|
||||
except Exception: # noqa: BLE001 - 无集成配置时不门控;步骤执行仍会显式报错
|
||||
return workflow
|
||||
account_steps = gate.collect_workflow_accounts(workflow)
|
||||
if not account_steps:
|
||||
return workflow
|
||||
existing_ids = {step.step_id for step in workflow.steps}
|
||||
gate_steps: list[StepDefinition] = []
|
||||
for account_id in sorted(account_steps):
|
||||
step_id = f"{_GATE_STEP_PREFIX}.{account_id}"
|
||||
if step_id in existing_ids:
|
||||
continue
|
||||
gate_steps.append(
|
||||
StepDefinition(
|
||||
step_id,
|
||||
CookieGateStepAction(gate, account_id),
|
||||
timeout_seconds=_GATE_TIMEOUT_SECONDS,
|
||||
max_attempts=1,
|
||||
)
|
||||
)
|
||||
if not gate_steps:
|
||||
return workflow
|
||||
patched: list[StepDefinition] = []
|
||||
for step in workflow.steps:
|
||||
dependencies = list(dict.fromkeys(step.depends_on))
|
||||
changed = False
|
||||
for account_id, step_ids in account_steps.items():
|
||||
gate_id = f"{_GATE_STEP_PREFIX}.{account_id}"
|
||||
if step.step_id in step_ids and gate_id not in dependencies:
|
||||
dependencies.append(gate_id)
|
||||
changed = True
|
||||
if changed:
|
||||
step = replace(step, depends_on=tuple(dependencies))
|
||||
patched.append(step)
|
||||
return WorkflowDefinition(
|
||||
workflow.workflow_id,
|
||||
(*gate_steps, *patched),
|
||||
)
|
||||
|
||||
|
||||
class CookieGateRecoveryManager:
|
||||
"""Re-run workflows blocked by expired cookies once the login recovers.
|
||||
|
||||
Runs inside the resident scheduler's maintenance loop. For every account
|
||||
with a non-empty ``pending_reruns`` list, it re-verifies the login state
|
||||
(fresh keepalive success first, then a throttled live probe); on recovery
|
||||
it relaunches each recorded workflow with its original business date
|
||||
through the scheduler's own launcher, clears the pending list, and sends a
|
||||
Feishu confirmation to the login owner.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
gate: CookieGate,
|
||||
*,
|
||||
launcher: Any,
|
||||
probe_interval_seconds: float = _RECOVERY_PROBE_INTERVAL_SECONDS,
|
||||
launch_cooldown_seconds: float = _RECOVERY_LAUNCH_COOLDOWN_SECONDS,
|
||||
) -> None:
|
||||
self.gate = gate
|
||||
self._launcher = launcher
|
||||
self._probe_interval = probe_interval_seconds
|
||||
self._launch_cooldown = launch_cooldown_seconds
|
||||
|
||||
def tick(self) -> list[dict[str, object]]:
|
||||
"""Attempt recovery reruns once; returns redacted outcomes per account."""
|
||||
|
||||
outcomes: list[dict[str, object]] = []
|
||||
for account in self.gate.catalog.accounts:
|
||||
state = self.gate._read_state(account)
|
||||
pending = state.get("pending_reruns")
|
||||
if not isinstance(pending, list) or not pending:
|
||||
continue
|
||||
now = self.gate._now()
|
||||
if not self._launch_cooldown_passed(state, now):
|
||||
continue
|
||||
if not self._account_recovered(account, state, now):
|
||||
continue
|
||||
launched, remaining = self._launch_pending(account, pending, now)
|
||||
state["pending_reruns"] = remaining
|
||||
state["updated_at"] = _now_iso(self.gate._now)
|
||||
state["last_recovered_at"] = state["updated_at"]
|
||||
if not remaining:
|
||||
state["state"] = STATE_HEALTHY
|
||||
state["last_probe_status"] = "recovered-rerun"
|
||||
if launched:
|
||||
state["last_rerun_launch_at"] = state["updated_at"]
|
||||
history = [
|
||||
item
|
||||
for item in state.get("rerun_history", [])
|
||||
if isinstance(item, dict)
|
||||
]
|
||||
history.extend(launched)
|
||||
state["rerun_history"] = history[-_RECOVERY_HISTORY_LIMIT:]
|
||||
self.gate._save_state(account, state)
|
||||
if launched:
|
||||
self._notify_recovered(str(account.account_id), launched)
|
||||
outcomes.append(
|
||||
{
|
||||
"account_id": str(account.account_id),
|
||||
"launched": launched,
|
||||
"pending": len(remaining),
|
||||
}
|
||||
)
|
||||
return outcomes
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# internals
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _launch_cooldown_passed(
|
||||
self, state: Mapping[str, object], now: datetime
|
||||
) -> bool:
|
||||
raw = state.get("last_rerun_launch_at")
|
||||
if not isinstance(raw, str):
|
||||
return True
|
||||
try:
|
||||
last = datetime.fromisoformat(raw)
|
||||
except ValueError:
|
||||
return True
|
||||
if last.tzinfo is None:
|
||||
last = last.replace(tzinfo=timezone.utc)
|
||||
return (now - last).total_seconds() >= self._launch_cooldown
|
||||
|
||||
def _account_recovered(
|
||||
self,
|
||||
account: Any,
|
||||
state: dict[str, object],
|
||||
now: datetime,
|
||||
) -> bool:
|
||||
"""Fresh keepalive success first; otherwise one throttled live probe."""
|
||||
|
||||
store = BrowserCookieStore(account.cookie_file, account.storage_state_file)
|
||||
if _vault_cookie_valid(account, store) and self.gate._keepalive_success_is_fresh(
|
||||
_read_keepalive_status(account)
|
||||
):
|
||||
return True
|
||||
raw = state.get("last_recovery_probe_at")
|
||||
if isinstance(raw, str):
|
||||
try:
|
||||
last_probe = datetime.fromisoformat(raw)
|
||||
except ValueError:
|
||||
last_probe = None
|
||||
if last_probe is not None:
|
||||
if last_probe.tzinfo is None:
|
||||
last_probe = last_probe.replace(tzinfo=timezone.utc)
|
||||
if (now - last_probe).total_seconds() < self._probe_interval:
|
||||
return False
|
||||
state["last_recovery_probe_at"] = _now_iso(self.gate._now)
|
||||
self.gate._save_state(account, state)
|
||||
try:
|
||||
outcome = self.gate.keepalive.refresh_account(str(account.account_id))
|
||||
except Exception: # noqa: BLE001 - 恢复探活失败留给下一轮
|
||||
return False
|
||||
return str(outcome.get("status", "error")) == "success"
|
||||
|
||||
def _launch_pending(
|
||||
self,
|
||||
account: Any,
|
||||
pending: list[object],
|
||||
now: datetime,
|
||||
) -> tuple[list[dict[str, object]], list[dict[str, object]]]:
|
||||
known_ids = {
|
||||
entry.workflow_id for entry in self.gate.workflow_catalog().workflows
|
||||
}
|
||||
launched: list[dict[str, object]] = []
|
||||
remaining: list[dict[str, object]] = []
|
||||
for item in pending:
|
||||
if not isinstance(item, dict):
|
||||
continue # 控制台脚本等临时运行 id:不在 catalog 中,直接丢弃
|
||||
workflow_id = item.get("workflow_id")
|
||||
business_date = item.get("business_date")
|
||||
if not isinstance(workflow_id, str) or not isinstance(business_date, str):
|
||||
continue
|
||||
if workflow_id not in known_ids:
|
||||
continue
|
||||
try:
|
||||
process = self._launcher.launch(
|
||||
workflow_id, now, date.fromisoformat(business_date)
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001 - 保留 pending,下一轮重试
|
||||
remaining.append(
|
||||
{
|
||||
"workflow_id": workflow_id,
|
||||
"business_date": business_date,
|
||||
"error": f"{type(exc).__name__}: {exc}"[:200],
|
||||
}
|
||||
)
|
||||
continue
|
||||
close = getattr(self._launcher, "close", None)
|
||||
if callable(close):
|
||||
try:
|
||||
close(process)
|
||||
except Exception: # noqa: BLE001 - 关闭日志句柄失败不影响重跑
|
||||
pass
|
||||
launched.append(
|
||||
{
|
||||
"workflow_id": workflow_id,
|
||||
"business_date": business_date,
|
||||
"pid": int(getattr(process, "pid", 0) or 0),
|
||||
"launched_at": _now_iso(self.gate._now),
|
||||
}
|
||||
)
|
||||
return launched, remaining
|
||||
|
||||
def _notify_recovered(
|
||||
self,
|
||||
account_id: str,
|
||||
launched: list[dict[str, object]],
|
||||
) -> None:
|
||||
lines = [
|
||||
f"- {item['workflow_id']}(业务日期 {item['business_date']})"
|
||||
for item in launched
|
||||
]
|
||||
message = (
|
||||
"【GYXX 登录态已恢复】账号 "
|
||||
f"{account_id} 登录态已恢复,已自动重跑被阻断的工作流:\n"
|
||||
+ "\n".join(lines)
|
||||
+ "\n如本次重跑仍有失败,请查看运行日志或工作流控制台。"
|
||||
)
|
||||
key = "-".join(
|
||||
[account_id, *(str(item["workflow_id"]) for item in launched)]
|
||||
)
|
||||
for open_id, profile in self.gate._alert_recipients():
|
||||
try:
|
||||
self.gate._sender(
|
||||
user_id=open_id,
|
||||
text=message,
|
||||
idempotency_key=f"cookie-recovery:{key}",
|
||||
profile=profile,
|
||||
)
|
||||
except Exception: # noqa: BLE001 - 确认通知失败不影响已发起的重跑
|
||||
continue
|
||||
|
||||
|
||||
def _record_pending_rerun(
|
||||
current: object,
|
||||
context: RunContext,
|
||||
blocked_at: str,
|
||||
) -> list[dict[str, object]]:
|
||||
"""Append one blocked run to the pending list, deduped and bounded."""
|
||||
|
||||
pending: list[dict[str, object]] = [
|
||||
dict(item)
|
||||
for item in (current if isinstance(current, list) else [])
|
||||
if isinstance(item, dict)
|
||||
]
|
||||
entry = {
|
||||
"workflow_id": context.workflow_id,
|
||||
"business_date": context.business_date.isoformat(),
|
||||
"blocked_run_id": context.run_id,
|
||||
"blocked_at": blocked_at,
|
||||
}
|
||||
pending = [
|
||||
item
|
||||
for item in pending
|
||||
if not (
|
||||
item.get("workflow_id") == entry["workflow_id"]
|
||||
and item.get("business_date") == entry["business_date"]
|
||||
)
|
||||
]
|
||||
pending.append(entry)
|
||||
return pending[-_MAX_PENDING_RERUNS:]
|
||||
|
||||
|
||||
def _recovery_evidence(status: Mapping[str, object] | None) -> str:
|
||||
if not isinstance(status, Mapping):
|
||||
return ""
|
||||
if not status.get("login_attempted"):
|
||||
return ""
|
||||
reason = status.get("login_reason")
|
||||
error_type = status.get("login_error_type")
|
||||
exit_code = status.get("login_exit_code")
|
||||
parts = ["自动账密重登已尝试但未成功"]
|
||||
if isinstance(reason, str) and reason:
|
||||
parts.append(reason)
|
||||
if isinstance(error_type, str) and error_type:
|
||||
parts.append(f"错误={error_type}")
|
||||
if isinstance(exit_code, int) and not isinstance(exit_code, bool):
|
||||
parts.append(f"退出码={exit_code}")
|
||||
return "(" + ",".join(parts) + ")"
|
||||
|
||||
|
||||
def _expiry_message(
|
||||
account_id: str,
|
||||
mode: str,
|
||||
context: RunContext,
|
||||
detail: str,
|
||||
) -> str:
|
||||
mode_note = (
|
||||
"账号密码自动重登失败,需人工处理(如验证码)"
|
||||
if mode == GATE_MODE_RECOVER
|
||||
else "该平台仅支持扫码/验证码登录,无法自动恢复"
|
||||
)
|
||||
return (
|
||||
"【GYXX 登录态失效】工作流被阻断,需要人工重新登录\n"
|
||||
f"工作流: {context.workflow_id}(业务日期 {context.business_date.isoformat()})\n"
|
||||
f"账号: {account_id}\n"
|
||||
f"原因: {detail};{mode_note}\n"
|
||||
f"请在服务器上执行: uv run gyxx accounts login {account_id}\n"
|
||||
"(将打开浏览器窗口,完成扫码或验证码登录后自动保存并同步到全部成员脚本;"
|
||||
"受影响的采集步骤本次已跳过,登录完成后下次运行自动恢复)"
|
||||
)
|
||||
|
||||
|
||||
def _now_iso(now: Callable[[], datetime]) -> str:
|
||||
return now().astimezone(timezone.utc).isoformat(timespec="seconds")
|
||||
|
||||
|
||||
def _env_int(
|
||||
environment: Mapping[str, str],
|
||||
name: str,
|
||||
default: int,
|
||||
*,
|
||||
minimum: int,
|
||||
) -> int:
|
||||
raw = environment.get(name, "").strip()
|
||||
try:
|
||||
value = int(raw)
|
||||
except (TypeError, ValueError):
|
||||
return default
|
||||
return max(minimum, value)
|
||||
|
||||
|
||||
def _env_float(
|
||||
environment: Mapping[str, str],
|
||||
name: str,
|
||||
default: float,
|
||||
*,
|
||||
minimum: float,
|
||||
) -> float:
|
||||
raw = environment.get(name, "").strip()
|
||||
try:
|
||||
value = float(raw)
|
||||
except (TypeError, ValueError):
|
||||
return default
|
||||
return max(minimum, value)
|
||||
|
||||
|
||||
__all__ = [
|
||||
"DEFAULT_LOGIN_ALERT_MOBILE",
|
||||
"GATE_MODE_BLOCKING",
|
||||
"GATE_MODE_DEFER",
|
||||
"GATE_MODE_RECOVER",
|
||||
"GYXX_COOKIE_GATE",
|
||||
"GYXX_LOGIN_ALERT_OPEN_IDS",
|
||||
"CookieGate",
|
||||
"CookieGateRecoveryManager",
|
||||
"CookieGateStepAction",
|
||||
"GateVerdict",
|
||||
"apply_cookie_gates",
|
||||
"gate_enabled",
|
||||
]
|
||||
@@ -1,4 +1,4 @@
|
||||
"""Content metrics, creator, comment, report, and login workflows."""
|
||||
"""Content metrics, creator, comment, and report workflows."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -14,7 +14,6 @@ CONTENT_SCHEDULED_WORKFLOW_IDS = (
|
||||
"content.metrics.self_operated.daily",
|
||||
"content.metrics.backfill",
|
||||
"content.marketing_report.daily",
|
||||
"content.relogin.weekly",
|
||||
"content.creator_report.monthly",
|
||||
"content.summary.monthly",
|
||||
"content.notes_master.daily",
|
||||
@@ -28,7 +27,6 @@ CONTENT_COMMENT_TIMEOUT_SECONDS = 6 * 60 * 60
|
||||
# weakening the module-wide write lock.
|
||||
CONTENT_NOTES_MASTER_LOCK_TIMEOUT_SECONDS = 6 * 60 * 60
|
||||
CONTENT_RESOURCE = "module:content_marketing"
|
||||
_RELOGIN_WORKFLOW_ID = "content.relogin.weekly"
|
||||
_PARALLEL_COMMENT_WORKFLOW_ID = "content.comments.weekly"
|
||||
# Keep both long-running daily metrics graphs on workflow-scoped resources so
|
||||
# their collection steps do not block unrelated content workflows.
|
||||
@@ -94,7 +92,6 @@ class ContentMarketingModule:
|
||||
),
|
||||
resource=workflow_resource,
|
||||
command_factory=command_factory,
|
||||
official_notification=workflow_id == _RELOGIN_WORKFLOW_ID,
|
||||
independent_step_resources=(
|
||||
workflow_id in {
|
||||
_PARALLEL_COMMENT_WORKFLOW_ID,
|
||||
|
||||
@@ -77,23 +77,15 @@ def is_logged_in(page: Page) -> bool:
|
||||
|
||||
|
||||
def maybe_wait_for_login(page: Page, login_timeout: int) -> None:
|
||||
"""检测登录态,未登录就等用户扫码。B 站登录页 URL 含 passport.bilibili.com/login。"""
|
||||
"""检测登录态;未登录立即降级为游客态,不再等待人工扫码。
|
||||
|
||||
登录统一由账号 vault 管理(gyxx accounts login bilibili-content),
|
||||
由工作流执行前的 cookie 网关保证登录态。
|
||||
"""
|
||||
if is_logged_in(page):
|
||||
return
|
||||
log("B 站未登录 (无 SESSDATA cookie)。请在弹出的浏览器窗口扫码登录。")
|
||||
# 跳到登录页,确保用户能看到二维码
|
||||
try:
|
||||
page.goto("https://passport.bilibili.com/login", wait_until="domcontentloaded", timeout=30000)
|
||||
except Exception as exc:
|
||||
log(f" 跳登录页失败 (继续等当前页扫码): {exc}")
|
||||
deadline = time.time() + login_timeout
|
||||
while time.time() < deadline:
|
||||
if is_logged_in(page):
|
||||
log("登录成功,SESSDATA 已写入")
|
||||
save_state(page)
|
||||
return
|
||||
page.wait_for_timeout(2000)
|
||||
log(f" [WARN] 等待 {login_timeout}s 仍未登录,继续以游客态采集 (评论只会拿到热门 3-5 条)")
|
||||
log(" [WARN] B 站未登录 (无 SESSDATA cookie),降级为游客态采集 (评论只会拿到热门 3-5 条)。"
|
||||
"如需全量评论请执行 gyxx accounts login bilibili-content")
|
||||
|
||||
|
||||
def safe_filename(value: str) -> str:
|
||||
@@ -437,48 +429,17 @@ def write_outputs(url: str, comments: list[dict[str, Any]], result: dict[str, An
|
||||
return json_path, csv_path
|
||||
|
||||
|
||||
def login_only(login_timeout: int) -> int:
|
||||
"""只触发登录(headed),登录完成立刻退出。给 relogin_bilibili.py 用。"""
|
||||
def action(page: Page) -> None:
|
||||
page.goto("https://passport.bilibili.com/login", wait_until="domcontentloaded", timeout=30000)
|
||||
page.wait_for_timeout(1500)
|
||||
maybe_wait_for_login(page, login_timeout)
|
||||
|
||||
PROFILE_DIR.mkdir(parents=True, exist_ok=True)
|
||||
with DynamicSession(
|
||||
headless=False,
|
||||
real_chrome=True,
|
||||
user_data_dir=str(PROFILE_DIR),
|
||||
locale="zh-CN",
|
||||
timezone_id="Asia/Shanghai",
|
||||
timeout=90000,
|
||||
network_idle=False,
|
||||
disable_resources=False,
|
||||
google_search=False,
|
||||
page_setup=restore_cookies,
|
||||
max_pages=1,
|
||||
) as session:
|
||||
session.fetch("https://passport.bilibili.com/login", page_action=action, wait=1000)
|
||||
log("[login-only] 完成")
|
||||
return 0
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description="Scrape Bilibili video comments with a real browser and API.")
|
||||
parser.add_argument("url", help="Bilibili video URL (传 dummy 'login' 配合 --login-only 也可)")
|
||||
parser.add_argument("url", help="Bilibili video URL")
|
||||
parser.add_argument("--max-reply-pages", type=int, default=200, help="max sub-reply pages per top-level comment")
|
||||
parser.add_argument("--max-pages", type=int, default=200, help="最多抓取一级评论页数")
|
||||
parser.add_argument("--delay", type=float, default=0.25, help="接口请求间隔秒数")
|
||||
parser.add_argument("--headless", action="store_true", help="无头模式")
|
||||
parser.add_argument("--login-timeout", type=int, default=0,
|
||||
help="未登录时等待扫码秒数(>0 启用登录检测;默认 0 沿用游客态)")
|
||||
parser.add_argument("--login-only", action="store_true",
|
||||
help="只触发登录(headed)然后退出,给 relogin_bilibili 用")
|
||||
help="保留参数(登录等待已移除;未登录将直接以游客态采集)")
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.login_only:
|
||||
return login_only(args.login_timeout if args.login_timeout > 0 else 300)
|
||||
|
||||
comments, result = scrape_comments(
|
||||
args.url, args.max_pages, args.delay, args.headless,
|
||||
args.max_reply_pages, args.login_timeout,
|
||||
|
||||
@@ -16,6 +16,7 @@ import time
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
from gyxx_flow.adapters.scrapling import is_browser_closed_error
|
||||
from gyxx_flow.core.exit_codes import COOKIE_SKIP_EXIT_CODE
|
||||
from gyxx_flow.modules.content_marketing import chanmama_export
|
||||
from gyxx_flow.modules.content_marketing.runtime_paths import PATHS
|
||||
@@ -52,6 +53,11 @@ REFRESH_WAIT_SECONDS = int(os.getenv("CHANMAMA_REFRESH_WAIT_SECONDS", "600"))
|
||||
CHANMAMA_LIST_READY_TIMEOUT_SECONDS = int(
|
||||
os.getenv("CHANMAMA_LIST_READY_TIMEOUT_SECONDS", "45")
|
||||
)
|
||||
# 公告/活动浮层可能盖住视频列表导致渲染判定一直不满足;渲染等待期间按
|
||||
# 该间隔无交互清理浮层(派发 ESC + 移除 fixed/absolute 浮层节点)。
|
||||
CHANMAMA_OVERLAY_DISMISS_INTERVAL_SECONDS = float(
|
||||
os.getenv("CHANMAMA_OVERLAY_DISMISS_INTERVAL_SECONDS", "10")
|
||||
)
|
||||
# 蝉妈妈导出接口实测可能在 15 分钟边界后才把 Excel 写入下载目录;把轮询
|
||||
# 上限做成可配置,并默认留出 20 分钟,避免文件已生成却被脚本判定为失败。
|
||||
CHANMAMA_EXPORT_WAIT_SECONDS = int(
|
||||
@@ -99,16 +105,20 @@ def create_driver():
|
||||
profile_dir = str(PATHS.browser_profile_dir)
|
||||
os.makedirs(profile_dir, exist_ok=True)
|
||||
cdp_url = os.getenv("GYXX_BROWSER_CDP_URL") or None
|
||||
# 蝉妈妈是国内站且其 WAF 会拉黑代理出口(Clash 节点 TLS 直接 reset,
|
||||
# 导出 API 全断、确认弹窗永远不弹);强制浏览器直连。
|
||||
direct_flags = ["--no-proxy-server"]
|
||||
# StealthySession(隐藏指纹补丁)下蝉妈妈前端会静默压制功能: 页面能开
|
||||
# 但导出确认弹窗不渲染(只剩空壳遮罩)。真 Chrome + 非 stealthy + 直连
|
||||
# 已实测弹窗完整渲染(含取消/导出按钮),不再使用 stealthy。
|
||||
try:
|
||||
return ScraplingDriver.start(
|
||||
profile_dir=profile_dir,
|
||||
download_dir=DOWNLOAD_DIR,
|
||||
cdp_url=cdp_url,
|
||||
stealthy=True,
|
||||
# 蝉妈妈新版 Vue 页面与 StealthySession + 系统 Chrome 组合
|
||||
# 不兼容:HTML 能返回,但 #app 不会挂载。使用 Scrapling
|
||||
# 自带 Chromium 仍保留隐藏指纹会话,且不影响已连接的 CDP。
|
||||
**({"real_chrome": False} if cdp_url is None else {}),
|
||||
stealthy=False,
|
||||
real_chrome=True,
|
||||
extra_flags=direct_flags,
|
||||
)
|
||||
except Exception as exc:
|
||||
if (
|
||||
@@ -125,23 +135,25 @@ def create_driver():
|
||||
profile_dir=profile_dir,
|
||||
download_dir=DOWNLOAD_DIR,
|
||||
cdp_url=None,
|
||||
stealthy=True,
|
||||
real_chrome=False,
|
||||
stealthy=False,
|
||||
real_chrome=True,
|
||||
extra_flags=direct_flags,
|
||||
)
|
||||
|
||||
|
||||
def save_cookies(driver):
|
||||
"""保存 cookie 到文件"""
|
||||
"""保存 cookie 到文件。
|
||||
|
||||
写裸列表格式,与账号 vault 同步(sync_binding_state)和适配器
|
||||
BrowserCookieStore 的绑定 cookie 文件格式保持一致 —— 同一文件
|
||||
被三方读写,字典格式会让适配器下次启动/重建时直接拒绝加载。
|
||||
"""
|
||||
try:
|
||||
cookies = driver.get_cookies()
|
||||
data = {
|
||||
"cookies": cookies,
|
||||
"timestamp": time.time(),
|
||||
}
|
||||
cookie_path = Path(COOKIE_FILE)
|
||||
cookie_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
temp_path = cookie_path.with_name(f".{cookie_path.name}.tmp")
|
||||
temp_path.write_text(json.dumps(data, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
temp_path.write_text(json.dumps(cookies, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
os.replace(temp_path, cookie_path)
|
||||
print(f"💾 已保存 cookie ({len(cookies)} 条) 到: {COOKIE_FILE}")
|
||||
return True
|
||||
@@ -153,6 +165,9 @@ def save_cookies(driver):
|
||||
def load_cookies(driver):
|
||||
"""从文件加载 cookie 到浏览器
|
||||
|
||||
兼容两种格式: 裸列表(vault/适配器格式,按文件 mtime 判断时效)与
|
||||
旧字典格式({"cookies": [...], "timestamp": ...})。
|
||||
|
||||
Returns:
|
||||
bool: cookie 是否有效(有效期内)
|
||||
"""
|
||||
@@ -161,16 +176,20 @@ def load_cookies(driver):
|
||||
return False
|
||||
|
||||
try:
|
||||
data = json.loads(Path(COOKIE_FILE).read_text(encoding="utf-8"))
|
||||
payload = json.loads(Path(COOKIE_FILE).read_text(encoding="utf-8"))
|
||||
if isinstance(payload, list):
|
||||
cookies = payload
|
||||
timestamp = os.path.getmtime(COOKIE_FILE)
|
||||
else:
|
||||
cookies = payload.get("cookies", [])
|
||||
timestamp = payload.get("timestamp", 0)
|
||||
|
||||
timestamp = data.get("timestamp", 0)
|
||||
age = time.time() - timestamp
|
||||
|
||||
if age > COOKIE_MAX_AGE:
|
||||
print(f"⏰ cookie 已过期 ({int(age // 86400)} 天前),需要重新登录")
|
||||
return False
|
||||
|
||||
cookies = data.get("cookies", [])
|
||||
if not cookies:
|
||||
print("📭 cookie 为空,需要重新登录")
|
||||
return False
|
||||
@@ -297,7 +316,10 @@ def handle_captcha(driver) -> bool:
|
||||
def verify_login(driver):
|
||||
"""验证是否已登录
|
||||
|
||||
通过访问目标页面,检查是否出现登录相关的元素
|
||||
通过访问目标页面,检查是否出现登录相关的元素。
|
||||
cookie 失效时蝉妈妈会把目标页重定向到 login.html,但重定向常在
|
||||
首屏渲染后数秒才发生——等待 URL 稳定后再判定,避免误判已登录
|
||||
(误判会跳过重新登录,导致后续导航一直超时)。
|
||||
"""
|
||||
try:
|
||||
# 跳转到目标页面
|
||||
@@ -305,10 +327,12 @@ def verify_login(driver):
|
||||
WebDriverWait(driver, 12).until(
|
||||
EC.presence_of_element_located((By.TAG_NAME, "body"))
|
||||
)
|
||||
time.sleep(3)
|
||||
# cookie 失效时的重定向在首屏渲染后 3~20 秒才发生,固定等满 10 秒
|
||||
# 再判定,不能因 URL 短暂未变而提前判为已登录
|
||||
time.sleep(10)
|
||||
cur_lower = driver.current_url.lower()
|
||||
|
||||
# 检查是否跳转回登录页 / 注册页(cookie 失效时蝉妈妈会重定向到 login.html 或 register.html)
|
||||
cur_lower = driver.current_url.lower()
|
||||
if "login" in cur_lower or "register" in cur_lower:
|
||||
print(f"🔒 检测到被重定向到登录/注册页 ({driver.current_url}),cookie 已失效")
|
||||
return False
|
||||
@@ -458,6 +482,180 @@ def login(driver):
|
||||
return False
|
||||
|
||||
|
||||
def _dismiss_overlays(driver) -> int:
|
||||
"""无交互清理遮挡浮层:派发 ESC 并移除 fixed/absolute 定位的弹层。"""
|
||||
try:
|
||||
removed = int(driver.execute_script(_OVERLAY_DISMISS_SCRIPT) or 0)
|
||||
except Exception as error:
|
||||
print(f"⚠️ 浮层清理脚本执行失败(忽略): {error}")
|
||||
return 0
|
||||
if removed:
|
||||
print(f"🧹 已清理 {removed} 个遮挡浮层")
|
||||
return removed
|
||||
|
||||
|
||||
def _dismiss_export_stage_overlays(driver) -> int:
|
||||
"""导出阶段清理弹窗(ESC + 移除弹层节点)。
|
||||
|
||||
只在弹窗不可确认时调用;返回移除的弹窗数。
|
||||
"""
|
||||
try:
|
||||
removed = int(driver.execute_script(_EXPORT_STAGE_DISMISS_SCRIPT) or 0)
|
||||
except Exception as error:
|
||||
print(f"⚠️ 导出阶段弹窗清理脚本执行失败(忽略): {error}")
|
||||
return 0
|
||||
if removed > 0:
|
||||
print(f"🧹 导出阶段清理 {removed} 个遮挡弹窗")
|
||||
return removed
|
||||
|
||||
|
||||
def _export_dialog_state(driver) -> str:
|
||||
"""探测导出阶段弹窗状态: confirmable / present / none。"""
|
||||
try:
|
||||
state = driver.execute_script(_EXPORT_DIALOG_STATE_SCRIPT)
|
||||
except Exception:
|
||||
return "none"
|
||||
return state if state in {"confirmable", "present", "none"} else "none"
|
||||
|
||||
|
||||
def _log_export_dialog_texts(driver) -> None:
|
||||
"""把当前可见弹窗的类名与文字打进日志,用于定位弹窗身份。"""
|
||||
try:
|
||||
texts = driver.execute_script(_EXPORT_DIALOG_TEXT_SCRIPT) or []
|
||||
except Exception:
|
||||
return
|
||||
for i, text in enumerate(texts[:4], 1):
|
||||
print(f"🪟 弹窗{i}内容: {text}")
|
||||
|
||||
|
||||
def _click_export_confirm(driver) -> bool:
|
||||
"""点击导出确认弹窗中的【导出】按钮,成功返回 True。"""
|
||||
# 关键:用 contains(.,'导出') 匹配后代文本("导出"在 <span> 内)
|
||||
confirm_selectors = [
|
||||
# 精确:主按钮(primary) + 含"导出"文字
|
||||
"//div[contains(@class,'el-dialog') or @role='dialog']//button[contains(.,'导出')]",
|
||||
# 弹窗内任意含"导出"的按钮
|
||||
"//div[contains(@class,'el-dialog') or @role='dialog']//button[contains(@class,'el-button--primary')][contains(.,'导出')]",
|
||||
# 兼容"确定"/"确认"
|
||||
"//div[contains(@class,'el-dialog') or @role='dialog']//button[contains(.,'确定')]",
|
||||
"//div[contains(@class,'el-dialog') or @role='dialog']//button[contains(.,'确认')]",
|
||||
]
|
||||
for sel in confirm_selectors:
|
||||
try:
|
||||
btn = driver.find_element(By.XPATH, sel)
|
||||
if btn.is_displayed():
|
||||
# 原生点击优先(可触发内层元素的 Vue @click),失败再退 JS 合成点击
|
||||
try:
|
||||
btn.click()
|
||||
except Exception:
|
||||
driver.execute_script("arguments[0].click();", btn)
|
||||
print(f"✅ 已在弹窗中点击导出按钮: {sel}")
|
||||
return True
|
||||
except Exception:
|
||||
continue
|
||||
return False
|
||||
|
||||
|
||||
def _reclick_export_control(driver) -> bool:
|
||||
"""重新触发一次导出按钮(第一次点击可能已被弹窗吞掉)。
|
||||
|
||||
优先 Playwright 原生点击(在元素命中点派发可信鼠标事件,能触发绑定
|
||||
在按钮内层 span/div 上的 Vue @click);JS 合成 click 的目标是按钮自身,
|
||||
内层监听器不会触发——蝉妈妈改版后导出点击无声失效就是这个原因。
|
||||
"""
|
||||
if _native_click_export_button(driver):
|
||||
return True
|
||||
try:
|
||||
return bool(driver.execute_script(_CLICK_EXPORT_CONTROL_SCRIPT))
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
_EXPORT_BUTTON_XPATHS = [
|
||||
"//button[contains(.,'导出数据')]",
|
||||
"//button[contains(.,'批量导出')]",
|
||||
"//button[contains(.,'导出视频')]",
|
||||
"//*[contains(@class,'export') and (self::button or self::a or @role='button')]",
|
||||
]
|
||||
|
||||
|
||||
def _native_click_export_button(driver) -> bool:
|
||||
"""用原生可信点击触发导出按钮,成功返回 True。"""
|
||||
for xp in _EXPORT_BUTTON_XPATHS:
|
||||
try:
|
||||
elements = driver.find_elements(By.XPATH, xp)
|
||||
except Exception:
|
||||
continue
|
||||
for el in elements:
|
||||
try:
|
||||
if el.is_displayed():
|
||||
el.click()
|
||||
return True
|
||||
except Exception:
|
||||
continue
|
||||
return False
|
||||
|
||||
|
||||
def _wait_video_list_ready(driver, timeout_seconds: float) -> None:
|
||||
"""等待视频列表渲染就绪,期间周期性清理遮挡浮层。
|
||||
|
||||
语义与 WebDriverWait.until 保持一致:浏览器侧的暂时性异常只记录
|
||||
不中断,到达 deadline 后抛 TimeoutException。
|
||||
"""
|
||||
timeout = max(0.0, float(timeout_seconds))
|
||||
deadline = time.monotonic() + timeout
|
||||
next_dismiss_at = 0.0
|
||||
last_error: Exception | None = None
|
||||
while True:
|
||||
try:
|
||||
if driver.execute_script(_VIDEO_LIST_READY_SCRIPT):
|
||||
return
|
||||
except Exception as error:
|
||||
last_error = error
|
||||
now = time.monotonic()
|
||||
if now >= deadline:
|
||||
detail = f": {last_error}" if last_error else ""
|
||||
raise TimeoutException(
|
||||
f"condition was not met within {timeout}s{detail}"
|
||||
) from last_error
|
||||
if now >= next_dismiss_at:
|
||||
_dismiss_overlays(driver)
|
||||
next_dismiss_at = now + CHANMAMA_OVERLAY_DISMISS_INTERVAL_SECONDS
|
||||
time.sleep(0.5)
|
||||
|
||||
|
||||
def _dump_render_failure(driver, url: str) -> None:
|
||||
"""渲染失败时落截图与页面 HTML,供事后定位是浮层/验证码还是改版。"""
|
||||
try:
|
||||
stamp = datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||
diag_dir = Path(DOWNLOAD_DIR) / "_diag"
|
||||
diag_dir.mkdir(parents=True, exist_ok=True)
|
||||
screenshot = diag_dir / f"render_failure_{stamp}.png"
|
||||
driver.save_screenshot(str(screenshot))
|
||||
(diag_dir / f"render_failure_{stamp}.html").write_text(
|
||||
driver.page_source, encoding="utf-8"
|
||||
)
|
||||
print(f"🩺 渲染失败诊断已保存: {screenshot}")
|
||||
except Exception as error:
|
||||
print(f"⚠️ 渲染失败诊断保存失败: {error}")
|
||||
|
||||
|
||||
def _dump_export_failure(driver) -> None:
|
||||
"""导出失败时落截图与页面 HTML,供事后定位弹窗/按钮的实际形态。"""
|
||||
try:
|
||||
stamp = datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||
diag_dir = Path(DOWNLOAD_DIR) / "_diag"
|
||||
diag_dir.mkdir(parents=True, exist_ok=True)
|
||||
screenshot = diag_dir / f"export_failure_{stamp}.png"
|
||||
driver.save_screenshot(str(screenshot))
|
||||
(diag_dir / f"export_failure_{stamp}.html").write_text(
|
||||
driver.page_source, encoding="utf-8"
|
||||
)
|
||||
print(f"🩺 导出失败诊断已保存: {screenshot}")
|
||||
except Exception as error:
|
||||
print(f"⚠️ 导出失败诊断保存失败: {error}")
|
||||
|
||||
|
||||
def navigate_to_target(driver, url):
|
||||
"""跳转到目标博主视频页面"""
|
||||
last_error = None
|
||||
@@ -475,12 +673,11 @@ def navigate_to_target(driver, url):
|
||||
)
|
||||
time.sleep(3) # 等待页面数据加载
|
||||
# 截图中的页面虽然 URL 正确,但正文尚未渲染;在交给导出逻辑
|
||||
# 前确认虚拟视频列表已经出现,空白页则有限重载一次。
|
||||
WebDriverWait(driver, CHANMAMA_LIST_READY_TIMEOUT_SECONDS).until(
|
||||
lambda current: bool(
|
||||
current.execute_script(_VIDEO_LIST_READY_SCRIPT)
|
||||
)
|
||||
)
|
||||
# 前确认虚拟视频列表已经出现,空白页则有限重载一次。浮层多在
|
||||
# 首屏挂载后才弹出:进等待前先清一轮,等待期间按间隔再清,
|
||||
# 避免延迟弹出的公告/活动浮层永久挡住渲染判定。
|
||||
_dismiss_overlays(driver)
|
||||
_wait_video_list_ready(driver, CHANMAMA_LIST_READY_TIMEOUT_SECONDS)
|
||||
print(f"✅ 已到达并完成渲染目标页面: {current_url}")
|
||||
return True
|
||||
except Exception as exc:
|
||||
@@ -499,6 +696,7 @@ def navigate_to_target(driver, url):
|
||||
except Exception as reset_exc:
|
||||
print(f"⚠️ 蝉妈妈页面重置失败,继续重试目标页: {reset_exc}")
|
||||
time.sleep(1)
|
||||
_dump_render_failure(driver, url)
|
||||
raise RuntimeError(f"蝉妈妈目标页加载失败: {url}; {last_error}") from last_error
|
||||
|
||||
|
||||
@@ -539,6 +737,206 @@ def _find_video_table(driver):
|
||||
return WebDriverWait(driver, 15).until(locate)
|
||||
|
||||
|
||||
# 蝉妈妈公告/活动弹窗以 fixed/absolute 浮层盖住页面,且类名不稳定、未必
|
||||
# 有关闭按钮。这里不点叉:派发 ESC 键事件(多数组件库监听 document 关闭
|
||||
# 弹窗),再移除定位浮层节点并解除其对 html/body 的滚动锁;已知类名未
|
||||
# 命中时兜底移除 z-index 高且覆盖大半屏的 fixed 层。列表区
|
||||
# (.aweme-record-wrapper/.list-box/.list-auth-wrap-block) 内的元素永不命中。
|
||||
# .v-modal 是 ElementUI 直接挂在 <body> 下的灰色遮罩,与弹窗本体分离,
|
||||
# 必须显式清理,否则弹窗删掉后页面仍然灰屏、无法滚动;滚动锁 class
|
||||
# (el-popup-parent--hidden 等)同理。
|
||||
# 关键约束:只移除"可见"的浮层。页面常驻 90+ 个 display:none 的隐藏弹窗
|
||||
# 模板(含导出确认框的模板)——删掉它们后 Vue 只会在游离节点上"打开"弹窗,
|
||||
# 页面上永远渲染不出来(导出只剩灰遮罩),因此 display:none/零尺寸节点
|
||||
# 一律跳过。
|
||||
_OVERLAY_DISMISS_SCRIPT = r"""
|
||||
() => {
|
||||
const esc = new KeyboardEvent('keydown', {
|
||||
key: 'Escape', code: 'Escape', keyCode: 27, which: 27, bubbles: true,
|
||||
});
|
||||
document.dispatchEvent(esc);
|
||||
window.dispatchEvent(esc);
|
||||
let removed = 0;
|
||||
const seen = new Set();
|
||||
const isOverlay = (el) => {
|
||||
const style = window.getComputedStyle(el);
|
||||
return style.position === 'fixed' || style.position === 'absolute';
|
||||
};
|
||||
const drop = (el) => {
|
||||
if (seen.has(el) || !isOverlay(el)) return;
|
||||
const style = window.getComputedStyle(el);
|
||||
if (style.display === 'none') return;
|
||||
const rect = el.getBoundingClientRect();
|
||||
if (rect.width === 0 || rect.height === 0) return;
|
||||
if (el.closest('.aweme-record-wrapper,.list-box,.list-auth-wrap-block')) return;
|
||||
seen.add(el);
|
||||
el.remove();
|
||||
removed += 1;
|
||||
};
|
||||
const selectors = [
|
||||
'.el-dialog__wrapper', '.el-overlay', '.el-message-box__wrapper',
|
||||
'.ant-modal-mask', '.ant-modal-wrap', '.modal-mask', '.modal-dialog',
|
||||
'.v-modal', '.v-modal-inner',
|
||||
'[class*="popup"]', '[class*="dialog"]', '[class*="overlay"]', '[class*="mask"]',
|
||||
];
|
||||
for (const selector of selectors) {
|
||||
document.querySelectorAll(selector).forEach(drop);
|
||||
}
|
||||
if (removed === 0) {
|
||||
document.querySelectorAll('body *').forEach((el) => {
|
||||
const style = window.getComputedStyle(el);
|
||||
const zIndex = Number.parseInt(style.zIndex, 10);
|
||||
const rect = el.getBoundingClientRect();
|
||||
if (Number.isFinite(zIndex) && zIndex > 500
|
||||
&& rect.height > window.innerHeight * 0.5) {
|
||||
drop(el);
|
||||
}
|
||||
});
|
||||
}
|
||||
document.body.classList.remove(
|
||||
'el-popup-parent--hidden', 'el-popup-parent--overflow-hidden', 'overflow-hidden'
|
||||
);
|
||||
document.documentElement.classList.remove(
|
||||
'el-popup-parent--hidden', 'el-popup-parent--overflow-hidden', 'overflow-hidden'
|
||||
);
|
||||
document.documentElement.style.overflow = '';
|
||||
document.body.style.overflow = '';
|
||||
document.body.style.paddingRight = '';
|
||||
return removed;
|
||||
}
|
||||
"""
|
||||
|
||||
|
||||
# 导出阶段专用清理:与 _OVERLAY_DISMISS_SCRIPT 同一套清理思路,但
|
||||
# 调用时机由 _export_dialog_state 决定——只在弹窗"没有可点的确认按钮"时
|
||||
# 才执行(确认框可点时直接点导出,绝不清)。三件事:
|
||||
# 1) 先 ESC + 点弹窗自带的关闭(×)按钮,走框架的正常关闭路径;
|
||||
# 2) 再移除残留的弹层节点——必须包含 .v-modal:ElementUI 的灰色遮罩直接
|
||||
# 挂在 <body> 下、与弹窗本体分离,漏掉它就是"删了弹窗页面仍灰屏";
|
||||
# 3) 最后解除滚动锁:清掉 body/html 上的 el-popup-parent--hidden 等
|
||||
# 框架加的 overflow 锁 class 与内联样式,否则页面删完遮罩也不能滚动。
|
||||
_EXPORT_STAGE_DISMISS_SCRIPT = r"""
|
||||
() => {
|
||||
const visible = (el) => {
|
||||
const style = window.getComputedStyle(el);
|
||||
const rect = el.getBoundingClientRect();
|
||||
return style.display !== 'none' && style.visibility !== 'hidden'
|
||||
&& style.opacity !== '0' && rect.width > 0 && rect.height > 0;
|
||||
};
|
||||
const esc = new KeyboardEvent('keydown', {
|
||||
key: 'Escape', code: 'Escape', keyCode: 27, which: 27, bubbles: true,
|
||||
});
|
||||
document.dispatchEvent(esc);
|
||||
window.dispatchEvent(esc);
|
||||
// 先走正常关闭:点弹窗自己的 × / 关闭按钮
|
||||
let closed = 0;
|
||||
document.querySelectorAll(
|
||||
'.el-dialog__wrapper [class*="close"],.el-message-box__wrapper [class*="close"],'
|
||||
+ '.ant-modal-wrap [class*="close"],[class*="dialog"] [class*="close"],'
|
||||
+ '[class*="modal"] [class*="close"],[class*="popup"] [class*="close"]'
|
||||
).forEach((el) => {
|
||||
if (visible(el)) { el.click(); closed += 1; }
|
||||
});
|
||||
let removed = 0;
|
||||
const seen = new Set();
|
||||
const drop = (el) => {
|
||||
if (seen.has(el)) return;
|
||||
const style = window.getComputedStyle(el);
|
||||
if (style.position !== 'fixed' && style.position !== 'absolute') return;
|
||||
if (style.display === 'none') return;
|
||||
const rect = el.getBoundingClientRect();
|
||||
if (rect.width === 0 || rect.height === 0) return;
|
||||
if (el.closest('.aweme-record-wrapper,.list-box,.list-auth-wrap-block')) return;
|
||||
seen.add(el);
|
||||
el.remove();
|
||||
removed += 1;
|
||||
};
|
||||
const selectors = [
|
||||
'.el-dialog__wrapper', '.el-overlay', '.el-message-box__wrapper',
|
||||
'.ant-modal-mask', '.ant-modal-wrap', '.modal-mask', '.modal-dialog',
|
||||
'.v-modal', '.v-modal-inner',
|
||||
'[class*="popup"]', '[class*="dialog"]', '[class*="overlay"]', '[class*="mask"]',
|
||||
];
|
||||
for (const selector of selectors) {
|
||||
document.querySelectorAll(selector).forEach(drop);
|
||||
}
|
||||
if (removed === 0) {
|
||||
document.querySelectorAll('body *').forEach((el) => {
|
||||
const style = window.getComputedStyle(el);
|
||||
const zIndex = Number.parseInt(style.zIndex, 10);
|
||||
const rect = el.getBoundingClientRect();
|
||||
if (Number.isFinite(zIndex) && zIndex > 500
|
||||
&& rect.height > window.innerHeight * 0.5) {
|
||||
drop(el);
|
||||
}
|
||||
});
|
||||
}
|
||||
// 解除滚动锁:框架加的锁 class + 内联 overflow 一起清
|
||||
document.body.classList.remove(
|
||||
'el-popup-parent--hidden', 'el-popup-parent--overflow-hidden', 'overflow-hidden'
|
||||
);
|
||||
document.documentElement.classList.remove(
|
||||
'el-popup-parent--hidden', 'el-popup-parent--overflow-hidden', 'overflow-hidden'
|
||||
);
|
||||
document.documentElement.style.overflow = '';
|
||||
document.body.style.overflow = '';
|
||||
document.body.style.paddingRight = '';
|
||||
return removed + closed;
|
||||
}
|
||||
"""
|
||||
|
||||
|
||||
# 导出阶段弹窗状态机:'confirmable'=有可见弹窗且弹窗里有可点的
|
||||
# 导出/确定/确认按钮(直接点,不能清);'present'=弹窗可见但没有可点
|
||||
# 的确认按钮(空壳遮罩/公告,ESC 关不掉就强删);'none'=没有弹窗。
|
||||
_EXPORT_DIALOG_STATE_SCRIPT = r"""
|
||||
() => {
|
||||
const visible = (el) => {
|
||||
const style = window.getComputedStyle(el);
|
||||
const rect = el.getBoundingClientRect();
|
||||
return style.display !== 'none' && style.visibility !== 'hidden'
|
||||
&& style.opacity !== '0' && rect.width > 0 && rect.height > 0;
|
||||
};
|
||||
const roots = [...document.querySelectorAll(
|
||||
'.el-dialog__wrapper,.el-message-box__wrapper,.el-overlay,'
|
||||
+ '.ant-modal-wrap,.ant-modal-mask,[role="dialog"],'
|
||||
+ '[class*="dialog"],[class*="modal"],[class*="popup"]'
|
||||
)].filter(visible);
|
||||
for (const root of roots) {
|
||||
const confirmable = [...root.querySelectorAll(
|
||||
'button,a,[role="button"],[class*="confirm"],[class*="primary"]'
|
||||
)].some((el) => {
|
||||
if (!visible(el)) return false;
|
||||
const text = (el.innerText || el.value || '').replace(/\s+/g, '');
|
||||
return /导出|确定|确认/.test(text);
|
||||
});
|
||||
if (confirmable) return 'confirmable';
|
||||
}
|
||||
return roots.length ? 'present' : 'none';
|
||||
}
|
||||
"""
|
||||
|
||||
|
||||
# 抓取当前可见弹窗的文字内容(每窗截前 160 字),写进日志用于定位弹窗身份。
|
||||
_EXPORT_DIALOG_TEXT_SCRIPT = r"""
|
||||
() => {
|
||||
const visible = (el) => {
|
||||
const style = window.getComputedStyle(el);
|
||||
const rect = el.getBoundingClientRect();
|
||||
return style.display !== 'none' && style.visibility !== 'hidden'
|
||||
&& style.opacity !== '0' && rect.width > 0 && rect.height > 0;
|
||||
};
|
||||
return [...document.querySelectorAll(
|
||||
'.el-dialog__wrapper,.el-message-box__wrapper,.el-overlay,'
|
||||
+ '.ant-modal-wrap,[role="dialog"],[class*="dialog"],[class*="modal"],[class*="popup"]'
|
||||
)].filter(visible).map((el) => {
|
||||
const text = (el.innerText || '').replace(/\s+/g, ' ').trim();
|
||||
const classes = String(el.className || '').slice(0, 80);
|
||||
return `[${classes}] ${text.slice(0, 160)}`;
|
||||
});
|
||||
}
|
||||
"""
|
||||
|
||||
_VIDEO_LIST_READY_SCRIPT = r"""
|
||||
() => {
|
||||
const visible = (el) => {
|
||||
@@ -909,18 +1307,81 @@ def _max_excel_mtime():
|
||||
|
||||
|
||||
def refresh_then_export_accounts(driver, urls, wait_seconds=REFRESH_WAIT_SECONDS):
|
||||
"""两阶段执行:先刷新所有账号全部分页,再等待并重新导出。"""
|
||||
"""两阶段执行:先刷新所有账号全部分页,再等待并重新导出。
|
||||
|
||||
运行中途登录态失效(目标页被重定向到登录页)时,自动用账号密码重新
|
||||
登录并重试当前账号一次;浏览器/页面被外部关闭(如进程树被清理)时,
|
||||
自动重建浏览器并恢复会话后重试当前账号一次。两种自愈整套运行各最多
|
||||
一次,避免死循环。返回 (记录列表, 刷新摘要, 最终生效的驱动) —— 驱动
|
||||
可能在重建后已不是调用方传入的那个,调用方需用返回值收尾关闭。
|
||||
"""
|
||||
refresh_summaries = []
|
||||
for index, url in enumerate(urls, 1):
|
||||
print(f"\n{'=' * 50}")
|
||||
print(f"🔄 刷新阶段 博主 {index}/{len(urls)}: {url}")
|
||||
print(f"{'=' * 50}")
|
||||
relogin_state = {"attempted": False}
|
||||
rebuild_state = {"attempted": False}
|
||||
driver_box = {"driver": driver}
|
||||
|
||||
def _relogin_if_expired(exc: BaseException) -> bool:
|
||||
"""登录态失效类异常时自动重登一次;返回是否已恢复登录。"""
|
||||
if relogin_state["attempted"] or "登录态失效" not in str(exc):
|
||||
return False
|
||||
relogin_state["attempted"] = True
|
||||
print("🔐 检测到登录态失效,尝试账号密码自动重新登录...")
|
||||
try:
|
||||
navigate_to_target(driver, url)
|
||||
summary = refresh_zero_exposure_videos(driver)
|
||||
ok = login(driver_box["driver"])
|
||||
except Exception as login_exc:
|
||||
print(f"⚠️ 自动重新登录异常: {login_exc}")
|
||||
return False
|
||||
if ok:
|
||||
print("✅ 自动重新登录成功,重试当前账号")
|
||||
return bool(ok)
|
||||
|
||||
def _restore_session(current) -> bool:
|
||||
"""重建浏览器后按 profile → cookie 文件 → 账密的顺序恢复登录态。"""
|
||||
try:
|
||||
if verify_login(current):
|
||||
return True
|
||||
if load_cookies(current) and verify_login(current):
|
||||
return True
|
||||
return bool(login(current))
|
||||
except Exception as exc:
|
||||
print(f"⚠️ 博主 {index} 刷新扫描异常: {exc},继续下一个")
|
||||
summary = {
|
||||
print(f"⚠️ 重建后会话恢复异常: {exc}")
|
||||
return False
|
||||
|
||||
def _rebuild_if_browser_closed(exc: BaseException) -> bool:
|
||||
"""浏览器/页面被外部关闭类异常时自愈一次;返回是否已恢复。
|
||||
|
||||
先尝试同一 context 重开新页(页面级死亡,登录态保留、代价最小);
|
||||
context 也死了才整套重建驱动并按 profile → cookie → 账密恢复会话。
|
||||
"""
|
||||
if rebuild_state["attempted"] or not is_browser_closed_error(exc):
|
||||
return False
|
||||
rebuild_state["attempted"] = True
|
||||
print("🌐 检测到浏览器页面被关闭,尝试自愈...")
|
||||
current = driver_box["driver"]
|
||||
try:
|
||||
current.reopen_page()
|
||||
print("✅ 浏览器 context 存活,已重开新页面,重试当前账号")
|
||||
return True
|
||||
except Exception as reopen_exc:
|
||||
print(f"ℹ️ 页面重开失败({reopen_exc}),改为整套重建浏览器...")
|
||||
try:
|
||||
current.quit()
|
||||
except Exception:
|
||||
# 旧驱动随浏览器一起死亡,清理失败不影响重建。
|
||||
pass
|
||||
try:
|
||||
driver_box["driver"] = create_driver()
|
||||
except Exception as create_exc:
|
||||
print(f"⚠️ 浏览器重建失败: {create_exc}")
|
||||
return False
|
||||
if not _restore_session(driver_box["driver"]):
|
||||
print("⚠️ 重建后会话恢复失败,放弃自动重试")
|
||||
return False
|
||||
print("✅ 浏览器重建完成,重试当前账号")
|
||||
return True
|
||||
|
||||
def _empty_summary(exc: BaseException) -> dict:
|
||||
return {
|
||||
"clicked": 0,
|
||||
"pending": 0,
|
||||
"zero_without_button": 0,
|
||||
@@ -928,6 +1389,25 @@ def refresh_then_export_accounts(driver, urls, wait_seconds=REFRESH_WAIT_SECONDS
|
||||
"pages": 0,
|
||||
"error": str(exc),
|
||||
}
|
||||
|
||||
for index, url in enumerate(urls, 1):
|
||||
print(f"\n{'=' * 50}")
|
||||
print(f"🔄 刷新阶段 博主 {index}/{len(urls)}: {url}")
|
||||
print(f"{'=' * 50}")
|
||||
try:
|
||||
navigate_to_target(driver_box["driver"], url)
|
||||
summary = refresh_zero_exposure_videos(driver_box["driver"])
|
||||
except Exception as exc:
|
||||
if _relogin_if_expired(exc) or _rebuild_if_browser_closed(exc):
|
||||
try:
|
||||
navigate_to_target(driver_box["driver"], url)
|
||||
summary = refresh_zero_exposure_videos(driver_box["driver"])
|
||||
except Exception as retry_exc:
|
||||
print(f"⚠️ 博主 {index} 自愈后刷新仍失败: {retry_exc},继续下一个")
|
||||
summary = _empty_summary(retry_exc)
|
||||
else:
|
||||
print(f"⚠️ 博主 {index} 刷新扫描异常: {exc},继续下一个")
|
||||
summary = _empty_summary(exc)
|
||||
refresh_summaries.append(summary)
|
||||
|
||||
should_wait = any(
|
||||
@@ -946,9 +1426,9 @@ def refresh_then_export_accounts(driver, urls, wait_seconds=REFRESH_WAIT_SECONDS
|
||||
print(f"📌 导出阶段 博主 {index}/{len(urls)}: {url}")
|
||||
print(f"{'=' * 50}")
|
||||
try:
|
||||
navigate_to_target(driver, url)
|
||||
navigate_to_target(driver_box["driver"], url)
|
||||
pre_mtime = _max_excel_mtime()
|
||||
excel_path = export_video_data(driver, pre_mtime)
|
||||
excel_path = export_video_data(driver_box["driver"], pre_mtime)
|
||||
if not excel_path:
|
||||
print(f"⚠️ 博主 {index} 导出失败,跳过")
|
||||
continue
|
||||
@@ -956,9 +1436,23 @@ def refresh_then_export_accounts(driver, urls, wait_seconds=REFRESH_WAIT_SECONDS
|
||||
print(f" 解析出 {len(records)} 条视频记录")
|
||||
all_records.extend(records)
|
||||
except Exception as exc:
|
||||
if _relogin_if_expired(exc) or _rebuild_if_browser_closed(exc):
|
||||
try:
|
||||
navigate_to_target(driver_box["driver"], url)
|
||||
pre_mtime = _max_excel_mtime()
|
||||
excel_path = export_video_data(driver_box["driver"], pre_mtime)
|
||||
if not excel_path:
|
||||
print(f"⚠️ 博主 {index} 自愈后导出失败,跳过")
|
||||
continue
|
||||
records, _ = parse_chanmama_export(excel_path)
|
||||
print(f" 解析出 {len(records)} 条视频记录(自愈后)")
|
||||
all_records.extend(records)
|
||||
except Exception as retry_exc:
|
||||
print(f"⚠️ 博主 {index} 自愈后导出仍失败: {retry_exc},跳过")
|
||||
else:
|
||||
print(f"⚠️ 博主 {index} 导出异常: {exc},跳过")
|
||||
|
||||
return all_records, refresh_summaries
|
||||
return all_records, refresh_summaries, driver_box["driver"]
|
||||
|
||||
|
||||
def export_video_data(driver, pre_mtime: float = 0):
|
||||
@@ -978,6 +1472,10 @@ def export_video_data(driver, pre_mtime: float = 0):
|
||||
except TimeoutException:
|
||||
print("⚠️ 视频列表加载超时,继续尝试导出...")
|
||||
|
||||
# 刷新/翻页阶段可能延迟弹出公告/活动弹窗,吞掉导出点击或盖住确认框;
|
||||
# 点导出按钮前先清一次(确认框受保护)。
|
||||
_dismiss_export_stage_overlays(driver)
|
||||
|
||||
# 方法1: 先在可见 DOM 中找真正的交互节点。新版按钮文字常在嵌套
|
||||
# span/div 内,旧的 contains(text()) 和第一个隐藏 locator 都会漏掉它。
|
||||
export_clicked = False
|
||||
@@ -989,6 +1487,9 @@ def export_video_data(driver, pre_mtime: float = 0):
|
||||
except TimeoutException:
|
||||
control = None
|
||||
if control:
|
||||
# 原生点击优先(可触发内层元素的 Vue @click),JS 合成点击兜底
|
||||
export_clicked = _native_click_export_button(driver)
|
||||
if not export_clicked:
|
||||
clicked = driver.execute_script(_CLICK_EXPORT_CONTROL_SCRIPT)
|
||||
export_clicked = bool(clicked)
|
||||
if export_clicked:
|
||||
@@ -1032,36 +1533,38 @@ def export_video_data(driver, pre_mtime: float = 0):
|
||||
return scrape_table_data(driver)
|
||||
|
||||
# 等待导出确认弹窗出现并点击【导出】按钮
|
||||
# 蝉妈妈弹窗标题"您即将进行数据导出",按钮为 橙色"导出"(文字在 <span> 内)
|
||||
# 蝉妈妈弹窗标题"您即将进行数据导出",按钮为 橙色"导出"(文字在 <span> 内)。
|
||||
# 逐秒探测弹窗状态:可确认→立即点导出;挂着但没有可点的确认按钮
|
||||
# (空壳遮罩/公告弹窗)→先 ESC+移除正常关,持续 5 秒以上就补点导出按钮
|
||||
# (第一次点击可能已被弹窗吞掉);没有弹窗→常规清理并定期补点导出。
|
||||
print("⏳ 等待导出确认弹窗...")
|
||||
export_confirmed = False
|
||||
for wait_sec in range(20):
|
||||
unconfirmable_since: float | None = None
|
||||
for wait_sec in range(36):
|
||||
time.sleep(1)
|
||||
try:
|
||||
# 关键:用 contains(.,'导出') 匹配后代文本("导出"在 <span> 内)
|
||||
confirm_selectors = [
|
||||
# 精确:主按钮(primary) + 含"导出"文字
|
||||
"//div[contains(@class,'el-dialog') or @role='dialog']//button[contains(.,'导出')]",
|
||||
# 弹窗内任意含"导出"的按钮
|
||||
"//div[contains(@class,'el-dialog') or @role='dialog']//button[contains(@class,'el-button--primary')][contains(.,'导出')]",
|
||||
# 兼容"确定"/"确认"
|
||||
"//div[contains(@class,'el-dialog') or @role='dialog']//button[contains(.,'确定')]",
|
||||
"//div[contains(@class,'el-dialog') or @role='dialog']//button[contains(.,'确认')]",
|
||||
]
|
||||
for sel in confirm_selectors:
|
||||
try:
|
||||
btn = driver.find_element(By.XPATH, sel)
|
||||
if btn.is_displayed():
|
||||
driver.execute_script("arguments[0].click();", btn)
|
||||
state = _export_dialog_state(driver)
|
||||
if state == "confirmable":
|
||||
unconfirmable_since = None
|
||||
if _click_export_confirm(driver):
|
||||
export_confirmed = True
|
||||
print(f"✅ 已在弹窗中点击导出按钮: {sel}")
|
||||
break
|
||||
except NoSuchElementException:
|
||||
continue
|
||||
if export_confirmed:
|
||||
break
|
||||
except Exception:
|
||||
if state == "present":
|
||||
now = time.monotonic()
|
||||
if unconfirmable_since is None:
|
||||
_log_export_dialog_texts(driver)
|
||||
print("ℹ️ 弹窗可见但没有可点的确认按钮,尝试关闭")
|
||||
unconfirmable_since = now
|
||||
elif now - unconfirmable_since > 5:
|
||||
if _reclick_export_control(driver):
|
||||
print("🔁 清掉卡死弹窗后补点导出按钮")
|
||||
unconfirmable_since = None
|
||||
_dismiss_export_stage_overlays(driver)
|
||||
continue
|
||||
unconfirmable_since = None
|
||||
_dismiss_export_stage_overlays(driver)
|
||||
if wait_sec in (10, 20, 30) and _reclick_export_control(driver):
|
||||
print("🔁 确认框迟迟未出现,补点导出按钮")
|
||||
|
||||
if not export_confirmed:
|
||||
try:
|
||||
@@ -1093,9 +1596,20 @@ def export_video_data(driver, pre_mtime: float = 0):
|
||||
"⚠️ 未找到本次下载的 Excel文件(等待 "
|
||||
f"{CHANMAMA_EXPORT_WAIT_SECONDS + max(CHANMAMA_EXPORT_GRACE_SECONDS, 0)} 秒超时)"
|
||||
)
|
||||
_dump_export_failure(driver)
|
||||
return None
|
||||
|
||||
|
||||
def _download_complete(path: str) -> bool:
|
||||
"""判断下载文件是否已写完整(zip EOCD 在文件尾,截断时 is_zipfile 为假)。"""
|
||||
import zipfile
|
||||
|
||||
try:
|
||||
return zipfile.is_zipfile(path)
|
||||
except OSError:
|
||||
return False
|
||||
|
||||
|
||||
def _wait_for_exported_excel(
|
||||
pre_mtime: float,
|
||||
*,
|
||||
@@ -1114,6 +1628,8 @@ def _wait_for_exported_excel(
|
||||
deadline 之后落盘的竞态;两个窗口内都未出现则返回 None。
|
||||
传入 driver 时每轮执行一次无害的 evaluate 以泵 Playwright 事件循环,
|
||||
确保 download 事件处理器(_save_download)在 sleep 轮询期间也能执行。
|
||||
泵调用发现浏览器/页面已被外部关闭时,若文件已完整落盘则照常返回,
|
||||
否则抛出原始关闭异常交由上层重建浏览器重试,不再空等满超时。
|
||||
"""
|
||||
deadline = time.time() + max(timeout_seconds, 0)
|
||||
grace_deadline = deadline + max(grace_seconds, 0)
|
||||
@@ -1122,10 +1638,25 @@ def _wait_for_exported_excel(
|
||||
if driver is not None:
|
||||
try:
|
||||
driver.execute_script("1")
|
||||
except Exception:
|
||||
pass
|
||||
except Exception as pump_exc:
|
||||
if is_browser_closed_error(pump_exc):
|
||||
# 浏览器/页面被外部关闭(进程树清理等): 先看文件是否
|
||||
# 已在关闭前完整落盘,再决定放弃,避免误杀已完成下载。
|
||||
settled = _find_new_excel_after(pre_mtime)
|
||||
if settled is not None and (
|
||||
not os.path.exists(settled) or _download_complete(settled)
|
||||
):
|
||||
return settled
|
||||
raise
|
||||
# 其余瞬态异常(页面导航中等)不致命,继续轮询文件系统。
|
||||
new_path = _find_new_excel_after(pre_mtime)
|
||||
if new_path is not None:
|
||||
# 文件出现 ≠ 下载完成: 当前浏览器可能以最终名边下边写,截断的
|
||||
# zip 会让解析报 "File is not a zip file"。zip 的 EOCD 记录在
|
||||
# 文件末尾,is_zipfile 通过即认为下载已完整;路径不存在(测试桩)
|
||||
# 时保持旧行为直接返回。
|
||||
if new_path is not None and (
|
||||
not os.path.exists(new_path) or _download_complete(new_path)
|
||||
):
|
||||
return new_path
|
||||
now = time.time()
|
||||
if now >= deadline and not grace_logged and grace_deadline > deadline:
|
||||
@@ -1829,7 +2360,8 @@ def main():
|
||||
print("⚠️ 登录可能未成功,但继续尝试后续步骤...")
|
||||
|
||||
# 4. 先刷新两个账号的全部分页,再统一等待并重新进入页面导出。
|
||||
all_records, _refresh_summaries = refresh_then_export_accounts(
|
||||
# 返回的驱动可能已在浏览器被外部关闭后重建,后续收尾必须用它。
|
||||
all_records, _refresh_summaries, driver = refresh_then_export_accounts(
|
||||
driver, TARGET_URLS, wait_seconds=REFRESH_WAIT_SECONDS
|
||||
)
|
||||
|
||||
|
||||
@@ -18,7 +18,6 @@ from gyxx_flow.modules.content_marketing.runtime_paths import PATHS
|
||||
_TOOLS_DIR = PATHS.tools_root
|
||||
_PROJECT_ROOT = PATHS.module_root
|
||||
|
||||
RELOGIN_SCRIPT = _TOOLS_DIR / "relogin_bilibili.py"
|
||||
|
||||
try:
|
||||
sys.stdout.reconfigure(encoding="utf-8", errors="replace") # type: ignore[attr-defined]
|
||||
@@ -81,39 +80,6 @@ def comment_replace_policy(comments: list[dict], stats: dict, metrics: dict) ->
|
||||
return False, False, "空评论且无法确认平台为 0 评论"
|
||||
|
||||
|
||||
def do_relogin() -> bool:
|
||||
"""Run relogin_bilibili.py (headed, waits for QR scan). Returns True if successful."""
|
||||
acceptance_policy = current_acceptance_policy()
|
||||
if acceptance_policy.enabled:
|
||||
acceptance_policy.record(
|
||||
"cookie_skipped",
|
||||
operation="content.batch_rescrape_bilibili.relogin",
|
||||
details={
|
||||
"status": "SKIPPED_COOKIE",
|
||||
"reason": "cookie is invalid and QR relogin is disabled for acceptance",
|
||||
},
|
||||
)
|
||||
log(
|
||||
" [SKIPPED_COOKIE] Cookie invalid; acceptance mode will not "
|
||||
"launch relogin or wait for a QR scan"
|
||||
)
|
||||
return False
|
||||
log(" Cookie 可能过期,触发自动重登 (请扫码)...")
|
||||
try:
|
||||
rc = subprocess.call(
|
||||
[sys.executable, str(RELOGIN_SCRIPT)],
|
||||
cwd=str(_PROJECT_ROOT),
|
||||
)
|
||||
if rc == 0:
|
||||
log(" 重登成功,继续采集")
|
||||
return True
|
||||
else:
|
||||
log(f" 重登失败 (exit={rc})")
|
||||
return False
|
||||
except Exception as exc:
|
||||
log(f" 重登脚本执行异常: {exc}")
|
||||
return False
|
||||
|
||||
|
||||
def main(only_ids: list[int] | None = None, dry_run: bool = False) -> int:
|
||||
rows = collect_urls()
|
||||
@@ -126,8 +92,6 @@ def main(only_ids: list[int] | None = None, dry_run: bool = False) -> int:
|
||||
total_comments = 0
|
||||
consecutive_fail = 0
|
||||
MAX_CONSECUTIVE_FAIL = 3
|
||||
relogin_used = 0
|
||||
MAX_RELOGINS = 1
|
||||
t0 = time.time()
|
||||
for i, row in enumerate(rows, 1):
|
||||
note_id = row["id"]
|
||||
@@ -152,11 +116,12 @@ def main(only_ids: list[int] | None = None, dry_run: bool = False) -> int:
|
||||
break
|
||||
if comments is None:
|
||||
consecutive_fail += 1
|
||||
if consecutive_fail >= MAX_CONSECUTIVE_FAIL and relogin_used < MAX_RELOGINS:
|
||||
if do_relogin():
|
||||
relogin_used += 1
|
||||
consecutive_fail = 0
|
||||
continue
|
||||
if consecutive_fail >= MAX_CONSECUTIVE_FAIL:
|
||||
# 登录统一由账号 vault 管理;连续失败只提示,不再触发扫码重登。
|
||||
log(
|
||||
" 连续失败达到上限;若为登录态失效请执行 "
|
||||
"uv run gyxx accounts login bilibili-content"
|
||||
)
|
||||
if current_acceptance_policy().enabled:
|
||||
return COOKIE_SKIP_EXIT_CODE
|
||||
continue
|
||||
|
||||
@@ -26,7 +26,6 @@ from gyxx_flow.modules.content_marketing.serialized_page_action import (
|
||||
_TOOLS_DIR = PATHS.tools_root
|
||||
_PROJECT_ROOT = PATHS.module_root
|
||||
|
||||
RELOGIN_SCRIPT = _TOOLS_DIR / "relogin_douyin.py"
|
||||
|
||||
try:
|
||||
sys.stdout.reconfigure(encoding="utf-8", errors="replace") # type: ignore[attr-defined]
|
||||
@@ -104,38 +103,6 @@ def result_logged_in(result: dict | None) -> bool:
|
||||
return bool(nested_stats.get("logged_in")) if isinstance(nested_stats, dict) else False
|
||||
|
||||
|
||||
def do_relogin() -> bool:
|
||||
"""Run relogin_douyin.py (headed, waits for QR scan). Returns True if successful."""
|
||||
acceptance_policy = current_acceptance_policy()
|
||||
if acceptance_policy.enabled:
|
||||
acceptance_policy.record(
|
||||
"cookie_skipped",
|
||||
operation="content.batch_rescrape_douyin.relogin",
|
||||
details={
|
||||
"status": "SKIPPED_COOKIE",
|
||||
"reason": "cookie is invalid and QR relogin is disabled for acceptance",
|
||||
},
|
||||
)
|
||||
log(
|
||||
" [SKIPPED_COOKIE] Cookie invalid; acceptance mode will not "
|
||||
"launch relogin or wait for a QR scan"
|
||||
)
|
||||
return False
|
||||
log(" Cookie 可能过期,触发自动重登 (请扫码)...")
|
||||
try:
|
||||
rc = subprocess.call(
|
||||
[sys.executable, str(RELOGIN_SCRIPT)],
|
||||
cwd=str(_PROJECT_ROOT),
|
||||
)
|
||||
if rc == 0:
|
||||
log(" 重登成功,继续采集")
|
||||
return True
|
||||
log(f" 重登失败 (exit={rc})")
|
||||
return False
|
||||
except Exception as exc:
|
||||
log(f" 重登脚本执行异常: {exc}")
|
||||
return False
|
||||
|
||||
|
||||
def scrape_with_fresh_browser(
|
||||
url: str,
|
||||
@@ -203,8 +170,6 @@ def main(only_ids: list[int] | None = None, dry_run: bool = False) -> int:
|
||||
COOLDOWN_SECONDS = 300
|
||||
cooldown_used = 0
|
||||
MAX_COOLDOWNS = 3
|
||||
relogin_used = 0
|
||||
MAX_RELOGINS = 1 # 每轮 batch 最多自动重登1次
|
||||
t0 = time.time()
|
||||
for i, row in enumerate(rows, 1):
|
||||
note_id = row["id"]
|
||||
@@ -214,14 +179,8 @@ def main(only_ids: list[int] | None = None, dry_run: bool = False) -> int:
|
||||
|
||||
if comments is None:
|
||||
consecutive_fail += 1
|
||||
# 连续失败 + 可能 cookie 过期 → 尝试自动重登
|
||||
if consecutive_fail >= MAX_CONSECUTIVE_FAIL and relogin_used < MAX_RELOGINS:
|
||||
if do_relogin():
|
||||
relogin_used += 1
|
||||
consecutive_fail = 0
|
||||
comments, stats, scrape_error = scrape_with_fresh_browser(url)
|
||||
else:
|
||||
# 重登失败,走冷却逻辑;验收模式必须直接跳过,不能等待。
|
||||
# 登录统一由账号 vault 管理;不再触发扫码重登,验收模式直接跳过。
|
||||
if consecutive_fail >= MAX_CONSECUTIVE_FAIL:
|
||||
if current_acceptance_policy().enabled:
|
||||
return COOKIE_SKIP_EXIT_CODE
|
||||
|
||||
|
||||
@@ -25,7 +25,6 @@ from gyxx_flow.modules.content_marketing.serialized_page_action import (
|
||||
_TOOLS_DIR = PATHS.tools_root
|
||||
_PROJECT_ROOT = PATHS.module_root
|
||||
|
||||
RELOGIN_SCRIPT = _TOOLS_DIR / "relogin_xiaohongshu.py"
|
||||
|
||||
try:
|
||||
sys.stdout.reconfigure(encoding="utf-8", errors="replace") # type: ignore[attr-defined]
|
||||
@@ -87,38 +86,6 @@ def comment_replace_policy(comments: list[dict], stats: dict, metrics: dict) ->
|
||||
return False, False, "空评论且无法确认平台为 0 评论"
|
||||
|
||||
|
||||
def do_relogin() -> bool:
|
||||
"""Run relogin_xiaohongshu.py (headed, waits for QR scan). Returns True if successful."""
|
||||
acceptance_policy = current_acceptance_policy()
|
||||
if acceptance_policy.enabled:
|
||||
acceptance_policy.record(
|
||||
"cookie_skipped",
|
||||
operation="content.batch_rescrape_xiaohongshu.relogin",
|
||||
details={
|
||||
"status": "SKIPPED_COOKIE",
|
||||
"reason": "cookie is invalid and QR relogin is disabled for acceptance",
|
||||
},
|
||||
)
|
||||
log(
|
||||
" [SKIPPED_COOKIE] Cookie invalid; acceptance mode will not "
|
||||
"launch relogin or wait for a QR scan"
|
||||
)
|
||||
return False
|
||||
log(" Cookie 可能过期,触发自动重登 (请扫码)...")
|
||||
try:
|
||||
rc = subprocess.call(
|
||||
[sys.executable, str(RELOGIN_SCRIPT)],
|
||||
cwd=str(_PROJECT_ROOT),
|
||||
)
|
||||
if rc == 0:
|
||||
log(" 重登成功,继续采集")
|
||||
return True
|
||||
log(f" 重登失败 (exit={rc})")
|
||||
return False
|
||||
except Exception as exc:
|
||||
log(f" 重登脚本执行异常: {exc}")
|
||||
return False
|
||||
|
||||
|
||||
def scrape_with_fresh_browser(
|
||||
url: str,
|
||||
@@ -173,8 +140,6 @@ def main(only_ids: list[int] | None = None, dry_run: bool = False) -> int:
|
||||
total_comments = 0
|
||||
consecutive_fail = 0
|
||||
MAX_CONSECUTIVE_FAIL = 3
|
||||
relogin_used = 0
|
||||
MAX_RELOGINS = 1
|
||||
t0 = time.time()
|
||||
for i, row in enumerate(rows, 1):
|
||||
note_id = row["id"]
|
||||
@@ -183,13 +148,13 @@ def main(only_ids: list[int] | None = None, dry_run: bool = False) -> int:
|
||||
comments, result, scrape_error = scrape_with_fresh_browser(url)
|
||||
if comments is None:
|
||||
consecutive_fail += 1
|
||||
if consecutive_fail >= MAX_CONSECUTIVE_FAIL and relogin_used < MAX_RELOGINS:
|
||||
if do_relogin():
|
||||
relogin_used += 1
|
||||
consecutive_fail = 0
|
||||
comments, result, scrape_error = scrape_with_fresh_browser(url)
|
||||
if consecutive_fail >= MAX_CONSECUTIVE_FAIL:
|
||||
if current_acceptance_policy().enabled:
|
||||
return COOKIE_SKIP_EXIT_CODE
|
||||
log(
|
||||
" 连续失败达到上限;若为登录态失效请执行 "
|
||||
"uv run gyxx accounts login xiaohongshu-content"
|
||||
)
|
||||
if comments is None:
|
||||
log(f" ERROR: {scrape_error}")
|
||||
fail += 1
|
||||
|
||||
@@ -1,666 +0,0 @@
|
||||
"""Launch five QR relogins in parallel, notify via analyzer Lark, retry 3 times."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import ctypes
|
||||
import datetime as dt
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from ctypes import wintypes
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Callable
|
||||
|
||||
import psutil
|
||||
|
||||
from gyxx_flow.adapters import (
|
||||
ANALYZER_NOTIFICATION_APP_PROFILE,
|
||||
COOKIE_SKIP_EXIT_CODE,
|
||||
ResolvedNotificationRoute,
|
||||
binding_from_environment,
|
||||
current_acceptance_policy,
|
||||
environment_for_child_script,
|
||||
resolve_notification_route,
|
||||
send_lark_bot_message,
|
||||
)
|
||||
from gyxx_flow.modules.content_marketing.runtime_paths import PATHS
|
||||
|
||||
PROJECT_DIR = PATHS.module_root
|
||||
ANALYZER_DIR = PATHS.tmp_root / "relogin"
|
||||
WORKFLOW_ID = "content.relogin.weekly"
|
||||
MARKETING_RECIPIENT = "ou_24cc944d6e43c69c59d6560ad4e2ae6e"
|
||||
TECHNICAL_RECIPIENT = "ou_7ad5fc8012e2f741afc5346e05ffd447"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Platform:
|
||||
label: str
|
||||
command: tuple[str, ...]
|
||||
required_cookie: str
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class NotificationDeliveryFailure:
|
||||
"""One failed recipient delivery that must affect the workflow outcome."""
|
||||
|
||||
stage: str
|
||||
platform: str
|
||||
recipient: str
|
||||
detail: str
|
||||
|
||||
|
||||
PLATFORMS = {
|
||||
"douyin": Platform(
|
||||
"\u6296\u97f3", ("data/tools/relogin_douyin.py",),
|
||||
"sessionid",
|
||||
),
|
||||
"bilibili": Platform(
|
||||
"B\u7ad9", ("data/tools/relogin_bilibili.py",),
|
||||
"SESSDATA",
|
||||
),
|
||||
"pgy": Platform(
|
||||
"\u84b2\u516c\u82f1", ("data/tools/relogin_pgy.py",),
|
||||
"",
|
||||
),
|
||||
"xingtu": Platform(
|
||||
"\u661f\u56fe", ("data/tools/relogin_xingtu.py", "--force"),
|
||||
"sessionid",
|
||||
),
|
||||
"xiaohongshu": Platform(
|
||||
"\u5c0f\u7ea2\u4e66", ("data/tools/relogin_xiaohongshu.py",),
|
||||
"web_session",
|
||||
),
|
||||
}
|
||||
|
||||
DEFAULT_RECIPIENTS = {
|
||||
"pgy": MARKETING_RECIPIENT,
|
||||
"xingtu": MARKETING_RECIPIENT,
|
||||
"douyin": TECHNICAL_RECIPIENT,
|
||||
"bilibili": TECHNICAL_RECIPIENT,
|
||||
"xiaohongshu": TECHNICAL_RECIPIENT,
|
||||
}
|
||||
|
||||
|
||||
def resolve_platform_names(requested: list[str] | None) -> list[str]:
|
||||
"""Resolve an optional manual platform subset while preserving order."""
|
||||
if not requested:
|
||||
return list(PLATFORMS)
|
||||
names: list[str] = []
|
||||
for name in requested:
|
||||
if name not in PLATFORMS:
|
||||
raise ValueError(f"unknown platform: {name}")
|
||||
if name not in names:
|
||||
names.append(name)
|
||||
return names
|
||||
|
||||
|
||||
def recipient_for_platform(platform_name: str, override: str | None = None) -> str:
|
||||
"""Return the responsible recipient, unless a manual all-platform override is set."""
|
||||
from gyxx_flow.adapters.acceptance_policy import resolve_notification_recipients
|
||||
|
||||
if override:
|
||||
recipient = override
|
||||
else:
|
||||
try:
|
||||
recipient = DEFAULT_RECIPIENTS[platform_name]
|
||||
except KeyError as exc:
|
||||
raise ValueError(
|
||||
f"no Lark recipient configured for platform: {platform_name}"
|
||||
) from exc
|
||||
return resolve_notification_recipients(
|
||||
(recipient,),
|
||||
app_profile=ANALYZER_NOTIFICATION_APP_PROFILE,
|
||||
)[0]
|
||||
|
||||
|
||||
def resolve_weekly_notification_route(
|
||||
override: str | None = None,
|
||||
) -> ResolvedNotificationRoute:
|
||||
"""Resolve the scheduled workflow route while retaining legacy defaults."""
|
||||
defaults = (
|
||||
(override,)
|
||||
if override
|
||||
else tuple(dict.fromkeys(DEFAULT_RECIPIENTS.values()))
|
||||
)
|
||||
return resolve_notification_route(WORKFLOW_ID, defaults)
|
||||
|
||||
|
||||
def recipients_for_platform(
|
||||
platform_name: str,
|
||||
route: ResolvedNotificationRoute,
|
||||
override: str | None = None,
|
||||
) -> tuple[str, ...]:
|
||||
"""Return dynamic recipients, or the original per-platform recipient."""
|
||||
if not route.enabled:
|
||||
return ()
|
||||
if route.configured:
|
||||
return route.open_ids
|
||||
return (recipient_for_platform(platform_name, override),)
|
||||
|
||||
|
||||
def group_platforms_by_recipient(
|
||||
platform_names: list[str], override: str | None = None,
|
||||
) -> dict[str, list[str]]:
|
||||
grouped: dict[str, list[str]] = {}
|
||||
for name in platform_names:
|
||||
grouped.setdefault(recipient_for_platform(name, override), []).append(name)
|
||||
return grouped
|
||||
|
||||
|
||||
def group_platforms_by_notification_recipient(
|
||||
platform_names: list[str],
|
||||
route: ResolvedNotificationRoute,
|
||||
override: str | None = None,
|
||||
) -> dict[str, list[str]]:
|
||||
"""Group failed platforms for legacy or configured multi-recipient delivery."""
|
||||
if not route.enabled:
|
||||
return {}
|
||||
if route.configured:
|
||||
return {recipient: list(platform_names) for recipient in route.open_ids}
|
||||
return group_platforms_by_recipient(platform_names, override)
|
||||
|
||||
|
||||
def run_with_retries(
|
||||
platform_names: list[str],
|
||||
*,
|
||||
max_attempts: int,
|
||||
run_round: Callable[[list[str], int], dict[str, bool]],
|
||||
) -> dict[str, bool]:
|
||||
status = {name: False for name in platform_names}
|
||||
pending = list(platform_names)
|
||||
for attempt in range(1, max_attempts + 1):
|
||||
if not pending:
|
||||
break
|
||||
round_result = run_round(pending, attempt)
|
||||
for name in pending:
|
||||
status[name] = bool(round_result.get(name))
|
||||
pending = [name for name in pending if not status[name]]
|
||||
return status
|
||||
|
||||
|
||||
def _valid_cookie(platform: Platform, cookie_file: Path) -> bool:
|
||||
if not cookie_file.is_file():
|
||||
return False
|
||||
if not platform.required_cookie:
|
||||
return cookie_file.stat().st_size > 10
|
||||
try:
|
||||
import json
|
||||
values = json.loads(cookie_file.read_text(encoding="utf-8"))
|
||||
return any(
|
||||
item.get("name") == platform.required_cookie and item.get("value")
|
||||
for item in values if isinstance(item, dict)
|
||||
)
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
def _descendant_pids(pid: int) -> set[int]:
|
||||
try:
|
||||
return {child.pid for child in psutil.Process(pid).children(recursive=True)}
|
||||
except (psutil.Error, OSError):
|
||||
return set()
|
||||
|
||||
|
||||
def _terminate_process_tree(process: subprocess.Popen) -> None:
|
||||
"""Terminate the whole Windows subprocess tree, including Chrome/Scrapling."""
|
||||
try:
|
||||
subprocess.run(
|
||||
["taskkill", "/PID", str(process.pid), "/T", "/F"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
except OSError:
|
||||
process.terminate()
|
||||
try:
|
||||
process.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
process.kill()
|
||||
|
||||
|
||||
def _window_for_process(pid: int) -> int | None:
|
||||
pids = _descendant_pids(pid) | {pid}
|
||||
handles: list[int] = []
|
||||
user32 = ctypes.windll.user32
|
||||
callback_type = ctypes.WINFUNCTYPE(ctypes.c_bool, ctypes.c_void_p, ctypes.c_void_p)
|
||||
|
||||
def callback(hwnd, _lparam):
|
||||
process_id = ctypes.c_ulong()
|
||||
user32.GetWindowThreadProcessId(hwnd, ctypes.byref(process_id))
|
||||
if process_id.value in pids and user32.IsWindowVisible(hwnd):
|
||||
handles.append(int(hwnd))
|
||||
return True
|
||||
|
||||
user32.EnumWindows(callback_type(callback), 0)
|
||||
return handles[0] if handles else None
|
||||
|
||||
|
||||
def _platform_from_window_title(title: str) -> str | None:
|
||||
"""Identify a login window without relying on Chrome's process tree."""
|
||||
if "\u84b2\u516c\u82f1" in title:
|
||||
return "pgy"
|
||||
if "\u6296\u97f3\u6388\u6743" in title or "\u5de8\u91cf\u661f\u56fe" in title:
|
||||
return "xingtu"
|
||||
if "\u6296\u97f3" in title:
|
||||
return "douyin"
|
||||
if "bilibili" in title.lower() or "\u8d26\u53f7\u767b\u5f55" in title:
|
||||
return "bilibili"
|
||||
if "\u5c0f\u7ea2\u4e66" in title:
|
||||
return "xiaohongshu"
|
||||
return None
|
||||
|
||||
|
||||
def _login_windows() -> dict[str, int]:
|
||||
"""Map visible Chrome login windows to platforms by their page titles."""
|
||||
user32 = ctypes.windll.user32
|
||||
found: dict[str, int] = {}
|
||||
callback_type = ctypes.WINFUNCTYPE(ctypes.c_bool, ctypes.c_void_p, ctypes.c_void_p)
|
||||
|
||||
def callback(hwnd, _lparam):
|
||||
if not user32.IsWindowVisible(hwnd):
|
||||
return True
|
||||
length = user32.GetWindowTextLengthW(hwnd)
|
||||
if length <= 0:
|
||||
return True
|
||||
buffer = ctypes.create_unicode_buffer(length + 1)
|
||||
user32.GetWindowTextW(hwnd, buffer, length + 1)
|
||||
platform = _platform_from_window_title(buffer.value)
|
||||
if platform:
|
||||
found[platform] = int(hwnd)
|
||||
return True
|
||||
|
||||
user32.EnumWindows(callback_type(callback), 0)
|
||||
return found
|
||||
|
||||
|
||||
def _capture_window(hwnd: int, output: Path) -> None:
|
||||
"""Maximize and render one window directly, even when it is not foreground."""
|
||||
from PIL import Image
|
||||
|
||||
user32 = ctypes.windll.user32
|
||||
gdi32 = ctypes.windll.gdi32
|
||||
user32.ShowWindow(hwnd, 3) # SW_MAXIMIZE
|
||||
time.sleep(1)
|
||||
rect = wintypes.RECT()
|
||||
user32.GetWindowRect(hwnd, ctypes.byref(rect))
|
||||
width, height = rect.right - rect.left, rect.bottom - rect.top
|
||||
if width <= 0 or height <= 0:
|
||||
raise RuntimeError(f"invalid window size: {width}x{height}")
|
||||
|
||||
window_dc = user32.GetWindowDC(hwnd)
|
||||
memory_dc = gdi32.CreateCompatibleDC(window_dc)
|
||||
bitmap = gdi32.CreateCompatibleBitmap(window_dc, width, height)
|
||||
previous = gdi32.SelectObject(memory_dc, bitmap)
|
||||
try:
|
||||
if not user32.PrintWindow(hwnd, memory_dc, 2): # PW_RENDERFULLCONTENT
|
||||
raise RuntimeError("PrintWindow failed")
|
||||
|
||||
class BitmapInfoHeader(ctypes.Structure):
|
||||
_fields_ = [
|
||||
("biSize", wintypes.DWORD), ("biWidth", wintypes.LONG),
|
||||
("biHeight", wintypes.LONG), ("biPlanes", wintypes.WORD),
|
||||
("biBitCount", wintypes.WORD), ("biCompression", wintypes.DWORD),
|
||||
("biSizeImage", wintypes.DWORD),
|
||||
("biXPelsPerMeter", wintypes.LONG),
|
||||
("biYPelsPerMeter", wintypes.LONG),
|
||||
("biClrUsed", wintypes.DWORD),
|
||||
("biClrImportant", wintypes.DWORD),
|
||||
]
|
||||
|
||||
byte_count = width * height * 4
|
||||
pixels = ctypes.create_string_buffer(byte_count)
|
||||
header = BitmapInfoHeader(
|
||||
ctypes.sizeof(BitmapInfoHeader), width, -height, 1, 32, 0,
|
||||
byte_count, 0, 0, 0, 0,
|
||||
)
|
||||
rows = gdi32.GetDIBits(
|
||||
memory_dc, bitmap, 0, height, pixels, ctypes.byref(header), 0
|
||||
)
|
||||
if rows != height:
|
||||
raise RuntimeError(f"GetDIBits returned {rows}/{height} rows")
|
||||
image = Image.frombuffer(
|
||||
"RGB", (width, height), pixels, "raw", "BGRX", 0, 1
|
||||
).copy()
|
||||
finally:
|
||||
gdi32.SelectObject(memory_dc, previous)
|
||||
gdi32.DeleteObject(bitmap)
|
||||
gdi32.DeleteDC(memory_dc)
|
||||
user32.ReleaseDC(hwnd, window_dc)
|
||||
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
image.save(output, "PNG")
|
||||
|
||||
|
||||
def _tile_windows(handles: list[int]) -> None:
|
||||
"""Place four login windows in a 2x2 grid so screenshots cannot overlap."""
|
||||
user32 = ctypes.windll.user32
|
||||
screen_w = user32.GetSystemMetrics(0)
|
||||
screen_h = user32.GetSystemMetrics(1)
|
||||
cell_w = max(640, screen_w // 2)
|
||||
cell_h = max(450, screen_h // 2)
|
||||
for index, hwnd in enumerate(handles):
|
||||
x = (index % 2) * cell_w
|
||||
y = (index // 2) * cell_h
|
||||
user32.ShowWindow(hwnd, 9) # SW_RESTORE
|
||||
user32.MoveWindow(hwnd, x, y, cell_w, cell_h, True)
|
||||
|
||||
|
||||
def _send_lark(
|
||||
recipient: str,
|
||||
text: str | None = None,
|
||||
image: Path | None = None,
|
||||
*,
|
||||
app_profile: str = ANALYZER_NOTIFICATION_APP_PROFILE,
|
||||
) -> tuple[dict[str, object], ...]:
|
||||
"""Send text and/or image through the analyzer Hermes bot."""
|
||||
if app_profile != ANALYZER_NOTIFICATION_APP_PROFILE:
|
||||
raise ValueError(
|
||||
"relogin notifications must use the hermes-analyzer lark-cli profile"
|
||||
)
|
||||
lark_env = {
|
||||
key: value for key, value in os.environ.items()
|
||||
if not key.upper().startswith("HERMES_")
|
||||
}
|
||||
receipts: list[dict[str, object]] = []
|
||||
if text:
|
||||
ANALYZER_DIR.mkdir(parents=True, exist_ok=True)
|
||||
receipts.append(send_lark_bot_message(
|
||||
user_id=recipient,
|
||||
text=text,
|
||||
profile=ANALYZER_NOTIFICATION_APP_PROFILE,
|
||||
cwd=ANALYZER_DIR,
|
||||
env=lark_env,
|
||||
))
|
||||
if image:
|
||||
receipts.append(send_lark_bot_message(
|
||||
user_id=recipient,
|
||||
image=image,
|
||||
profile=ANALYZER_NOTIFICATION_APP_PROFILE,
|
||||
cwd=image.parent,
|
||||
env=lark_env,
|
||||
))
|
||||
return tuple(receipts)
|
||||
|
||||
|
||||
def _send_qr_notification(
|
||||
route: ResolvedNotificationRoute,
|
||||
platform_name: str,
|
||||
*,
|
||||
override: str | None,
|
||||
text: str,
|
||||
image: Path,
|
||||
) -> tuple[NotificationDeliveryFailure, ...]:
|
||||
"""Attempt every QR delivery and return all recipient-specific failures."""
|
||||
profile = route.app_profile or ANALYZER_NOTIFICATION_APP_PROFILE
|
||||
failures: list[NotificationDeliveryFailure] = []
|
||||
for recipient in recipients_for_platform(platform_name, route, override):
|
||||
for stage, kwargs in (
|
||||
("qr_text", {"text": text}),
|
||||
("qr_image", {"image": image}),
|
||||
):
|
||||
try:
|
||||
_send_lark(recipient, app_profile=profile, **kwargs)
|
||||
except Exception as exc:
|
||||
failures.append(
|
||||
NotificationDeliveryFailure(
|
||||
stage=stage,
|
||||
platform=platform_name,
|
||||
recipient=recipient,
|
||||
detail=str(exc) or type(exc).__name__,
|
||||
)
|
||||
)
|
||||
return tuple(failures)
|
||||
|
||||
|
||||
def _unavailable_qr_delivery_failures(
|
||||
route: ResolvedNotificationRoute,
|
||||
platform_name: str,
|
||||
*,
|
||||
override: str | None,
|
||||
detail: str,
|
||||
) -> tuple[NotificationDeliveryFailure, ...]:
|
||||
"""Record both intended QR deliveries when no screenshot can be sent."""
|
||||
return tuple(
|
||||
NotificationDeliveryFailure(
|
||||
stage=stage,
|
||||
platform=platform_name,
|
||||
recipient=recipient,
|
||||
detail=detail,
|
||||
)
|
||||
for recipient in recipients_for_platform(platform_name, route, override)
|
||||
for stage in ("qr_text", "qr_image")
|
||||
)
|
||||
|
||||
|
||||
def _send_failure_summaries(
|
||||
route: ResolvedNotificationRoute,
|
||||
failed_platforms: list[str],
|
||||
*,
|
||||
override: str | None,
|
||||
max_attempts: int,
|
||||
) -> tuple[NotificationDeliveryFailure, ...]:
|
||||
"""Attempt every final failure summary and return all delivery failures."""
|
||||
groups = group_platforms_by_notification_recipient(
|
||||
failed_platforms,
|
||||
route,
|
||||
override,
|
||||
)
|
||||
profile = route.app_profile or ANALYZER_NOTIFICATION_APP_PROFILE
|
||||
failures: list[NotificationDeliveryFailure] = []
|
||||
for recipient, names in groups.items():
|
||||
labels = "\u3001".join(PLATFORMS[name].label for name in names)
|
||||
try:
|
||||
_send_lark(
|
||||
recipient,
|
||||
text=(
|
||||
f"\u4ee5\u4e0b\u5e73\u53f0\u8fde\u7eed {max_attempts} \u6b21\u672a\u626b\u7801\uff0c"
|
||||
"\u5df2\u6062\u590d\u5e76\u7ee7\u4f7f\u7528\u65e7 Cookie\uff1a"
|
||||
f"{labels}"
|
||||
),
|
||||
app_profile=profile,
|
||||
)
|
||||
except Exception as exc:
|
||||
failures.append(
|
||||
NotificationDeliveryFailure(
|
||||
stage="failure_summary",
|
||||
platform=",".join(names),
|
||||
recipient=recipient,
|
||||
detail=str(exc) or type(exc).__name__,
|
||||
)
|
||||
)
|
||||
return tuple(failures)
|
||||
|
||||
|
||||
def _run_round_factory(
|
||||
args,
|
||||
notification_route: ResolvedNotificationRoute | None,
|
||||
notification_failures: list[NotificationDeliveryFailure],
|
||||
):
|
||||
python = sys.executable
|
||||
screenshot_root = PATHS.tmp_root / "qrcode/friday_relogin"
|
||||
|
||||
def run_round(names: list[str], attempt: int) -> dict[str, bool]:
|
||||
started: dict[str, subprocess.Popen] = {}
|
||||
cookie_files: dict[str, Path] = {}
|
||||
missing_qr_windows: set[str] = set()
|
||||
stamp = dt.datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||
for name in names:
|
||||
platform = PLATFORMS[name]
|
||||
cmd = [python, *platform.command, "--login-timeout", str(args.round_timeout)]
|
||||
target = PROJECT_DIR / platform.command[0]
|
||||
environment = environment_for_child_script(target, os.environ)
|
||||
cookie_files[name] = binding_from_environment(environment).cookie_file
|
||||
started[name] = subprocess.Popen(
|
||||
cmd,
|
||||
cwd=PROJECT_DIR,
|
||||
env=environment,
|
||||
)
|
||||
|
||||
time.sleep(args.screenshot_delay)
|
||||
title_windows = _login_windows()
|
||||
windows = {
|
||||
name: title_windows.get(name) or _window_for_process(process.pid)
|
||||
for name, process in started.items()
|
||||
}
|
||||
_tile_windows([hwnd for hwnd in windows.values() if hwnd])
|
||||
time.sleep(2)
|
||||
for name, process in started.items():
|
||||
platform = PLATFORMS[name]
|
||||
hwnd = windows[name]
|
||||
if not hwnd:
|
||||
print(f"[{name}] no visible browser window for QR screenshot", flush=True)
|
||||
missing_qr_windows.add(name)
|
||||
continue
|
||||
output = screenshot_root / f"{stamp}_{name}_round{attempt}.png"
|
||||
try:
|
||||
_capture_window(hwnd, output)
|
||||
print(f"[{name}] QR screenshot: {output}", flush=True)
|
||||
except Exception as exc:
|
||||
print(f"[{name}] screenshot failed: {exc}", flush=True)
|
||||
if not args.no_send and notification_route is not None:
|
||||
notification_failures.extend(
|
||||
_unavailable_qr_delivery_failures(
|
||||
notification_route,
|
||||
name,
|
||||
override=args.recipient,
|
||||
detail=(
|
||||
"QR screenshot capture failed: "
|
||||
f"{str(exc) or type(exc).__name__}"
|
||||
),
|
||||
)
|
||||
)
|
||||
continue
|
||||
if not args.no_send and notification_route is not None:
|
||||
notification_failures.extend(
|
||||
_send_qr_notification(
|
||||
notification_route,
|
||||
name,
|
||||
override=args.recipient,
|
||||
text=(
|
||||
f"{platform.label} \u767b\u5f55\u4e8c\u7ef4\u7801"
|
||||
f"\uff08\u7b2c {attempt}/{args.max_attempts} \u6b21\uff09\uff0c"
|
||||
"\u8bf7\u5728 3 \u5206\u949f\u5185\u626b\u7801\u3002"
|
||||
),
|
||||
image=output,
|
||||
)
|
||||
)
|
||||
|
||||
# Put the still-open login windows back into a compact layout after
|
||||
# taking full-size screenshots one at a time.
|
||||
_tile_windows([hwnd for hwnd in windows.values() if hwnd])
|
||||
|
||||
results: dict[str, bool] = {}
|
||||
for name, process in started.items():
|
||||
try:
|
||||
rc = process.wait(timeout=args.round_timeout + 90)
|
||||
except subprocess.TimeoutExpired:
|
||||
_terminate_process_tree(process)
|
||||
rc = 1
|
||||
results[name] = rc == 0 and _valid_cookie(
|
||||
PLATFORMS[name],
|
||||
cookie_files[name],
|
||||
)
|
||||
if (
|
||||
not results[name]
|
||||
and name in missing_qr_windows
|
||||
and not args.no_send
|
||||
and notification_route is not None
|
||||
):
|
||||
notification_failures.extend(
|
||||
_unavailable_qr_delivery_failures(
|
||||
notification_route,
|
||||
name,
|
||||
override=args.recipient,
|
||||
detail=(
|
||||
"no visible QR window and relogin did not succeed"
|
||||
),
|
||||
)
|
||||
)
|
||||
print(f"[{name}] round={attempt} exit={rc} success={results[name]}", flush=True)
|
||||
return results
|
||||
|
||||
return run_round
|
||||
|
||||
|
||||
def main() -> int:
|
||||
acceptance_policy = current_acceptance_policy()
|
||||
if acceptance_policy.enabled:
|
||||
acceptance_policy.record(
|
||||
"cookie_skipped",
|
||||
operation="content.relogin.weekly",
|
||||
details={
|
||||
"status": "SKIPPED_COOKIE",
|
||||
"reason": "acceptance runs never launch QR relogin workflows",
|
||||
},
|
||||
)
|
||||
print(
|
||||
"[SKIPPED_COOKIE] Acceptance mode does not launch login windows, "
|
||||
"reset cookies/profiles, or wait for QR scans.",
|
||||
flush=True,
|
||||
)
|
||||
return COOKIE_SKIP_EXIT_CODE
|
||||
|
||||
parser = argparse.ArgumentParser(description="Friday five-platform parallel QR relogin")
|
||||
parser.add_argument(
|
||||
"--recipient", default=None,
|
||||
help="Override the configured recipients and send every platform to one open_id",
|
||||
)
|
||||
parser.add_argument("--max-attempts", type=int, default=3)
|
||||
parser.add_argument("--round-timeout", type=int, default=300)
|
||||
parser.add_argument("--screenshot-delay", type=int, default=60)
|
||||
parser.add_argument("--no-send", action="store_true", help="Do not send Lark messages")
|
||||
parser.add_argument(
|
||||
"--platform", action="append", choices=sorted(PLATFORMS),
|
||||
help="Only relogin the selected platform; may be repeated",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
platform_names = resolve_platform_names(args.platform)
|
||||
notification_route = (
|
||||
None if args.no_send else resolve_weekly_notification_route(args.recipient)
|
||||
)
|
||||
notification_failures: list[NotificationDeliveryFailure] = []
|
||||
status = run_with_retries(
|
||||
platform_names, max_attempts=args.max_attempts,
|
||||
run_round=_run_round_factory(
|
||||
args,
|
||||
notification_route,
|
||||
notification_failures,
|
||||
),
|
||||
)
|
||||
failed = [name for name, ok in status.items() if not ok]
|
||||
if failed and not args.no_send and notification_route is not None:
|
||||
notification_failures.extend(
|
||||
_send_failure_summaries(
|
||||
notification_route,
|
||||
failed,
|
||||
override=args.recipient,
|
||||
max_attempts=args.max_attempts,
|
||||
)
|
||||
)
|
||||
print(f"Final relogin status: {status}", flush=True)
|
||||
if notification_failures:
|
||||
print(
|
||||
f"Final notification status: failed ({len(notification_failures)} deliveries)",
|
||||
flush=True,
|
||||
)
|
||||
for failure in notification_failures:
|
||||
print(
|
||||
"[NOTIFICATION_ERROR] "
|
||||
f"stage={failure.stage} platform={failure.platform} "
|
||||
f"recipient={failure.recipient} error={failure.detail}",
|
||||
flush=True,
|
||||
)
|
||||
return 2
|
||||
if notification_route is not None and notification_route.enabled:
|
||||
print("Final notification status: delivered", flush=True)
|
||||
else:
|
||||
print("Final notification status: skipped", flush=True)
|
||||
return 1 if failed else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -1,113 +0,0 @@
|
||||
"""每周一重登 B 站。
|
||||
|
||||
强制清掉 cookie JSON + 浏览器 profile,确保下次启动时必须扫码。
|
||||
任务计划: 每周一 09:00
|
||||
|
||||
跑法:
|
||||
python data/tools/relogin_bilibili.py # 实际跑(会弹二维码)
|
||||
python data/tools/relogin_bilibili.py --dry-run # 只看会做什么
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import datetime as dt
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
from gyxx_flow.modules.content_marketing.data.tools.relogin_runtime_paths import (
|
||||
child_browser_environment,
|
||||
resolve_relogin_runtime_paths,
|
||||
sync_relogin_state_to_consumers,
|
||||
)
|
||||
from gyxx_flow.modules.content_marketing.data.tools.relogin_transaction import (
|
||||
begin,
|
||||
commit,
|
||||
rollback,
|
||||
)
|
||||
from gyxx_flow.modules.content_marketing.runtime_paths import PATHS
|
||||
|
||||
PROJECT_DIR = PATHS.module_root
|
||||
RUNTIME_PATHS = resolve_relogin_runtime_paths("bilibili")
|
||||
COOKIE_FILE = RUNTIME_PATHS.cookie_file
|
||||
STATE_FILE = RUNTIME_PATHS.storage_state_file
|
||||
PROFILE_DIR = RUNTIME_PATHS.profile_dir
|
||||
SCRAPER = PROJECT_DIR / "bilibili_comment_scraper.py"
|
||||
|
||||
|
||||
def reset_state():
|
||||
"""删 cookie JSON + storage_state + 清浏览器 profile。"""
|
||||
state = begin((COOKIE_FILE, STATE_FILE), PROFILE_DIR)
|
||||
print(" [OK] 旧 Cookie/Profile 已暂存,失败时将自动恢复")
|
||||
return state
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description="每周一重登 B 站")
|
||||
ap.add_argument("--login-timeout", type=int, default=300, help="扫码超时(秒)")
|
||||
ap.add_argument("--dry-run", action="store_true", help="只打印计划,不删不跑")
|
||||
args = ap.parse_args()
|
||||
|
||||
print("== Relogin: B 站 ==")
|
||||
print(f" 今天: {dt.date.today()} ({dt.date.today().strftime('%A')})")
|
||||
print(f" cookie: {COOKIE_FILE}")
|
||||
print(f" profile: {PROFILE_DIR}")
|
||||
print()
|
||||
|
||||
if args.dry_run:
|
||||
print("[DRY-RUN] 实际跑会: 删 cookie + 清 profile + 弹二维码")
|
||||
return 0
|
||||
|
||||
old_state = reset_state()
|
||||
cmd = [
|
||||
sys.executable,
|
||||
str(SCRAPER),
|
||||
"login", # dummy url (login-only 模式不需要真实 url)
|
||||
"--login-only",
|
||||
"--login-timeout",
|
||||
str(args.login_timeout),
|
||||
]
|
||||
print(f"\n [RUN] {' '.join(cmd)}")
|
||||
try:
|
||||
rc = subprocess.call(
|
||||
cmd,
|
||||
cwd=str(PROJECT_DIR),
|
||||
env=child_browser_environment(
|
||||
cookie_file=COOKIE_FILE,
|
||||
storage_state_file=STATE_FILE,
|
||||
profile_dir=PROFILE_DIR,
|
||||
),
|
||||
)
|
||||
except BaseException:
|
||||
rollback(old_state)
|
||||
raise
|
||||
print(f"\n exit={rc}")
|
||||
login_ok = False
|
||||
if rc == 0:
|
||||
# 验证 SESSDATA 是否真写入了
|
||||
try:
|
||||
import json
|
||||
cookies = json.loads(COOKIE_FILE.read_text(encoding="utf-8"))
|
||||
sessdata = any(c.get("name") == "SESSDATA" and c.get("value") for c in cookies)
|
||||
login_ok = bool(sessdata)
|
||||
print(f" SESSDATA 写入: {'OK' if sessdata else 'FAIL — 没有看到 SESSDATA,登录可能没成功'}")
|
||||
except Exception as exc:
|
||||
print(f" [WARN] 验证 cookie 时出错: {exc}")
|
||||
if rc != 0 or not login_ok:
|
||||
rollback(old_state)
|
||||
print(" [ROLLBACK] 未完成扫码,已恢复旧登录状态")
|
||||
return rc or 1
|
||||
try:
|
||||
sync_relogin_state_to_consumers(
|
||||
"bilibili",
|
||||
cookie_file=COOKIE_FILE,
|
||||
storage_state_file=STATE_FILE,
|
||||
)
|
||||
except Exception as exc:
|
||||
rollback(old_state)
|
||||
print(f" [ROLLBACK] 同步采集器登录态失败: {exc}")
|
||||
return 1
|
||||
commit(old_state)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -1,121 +0,0 @@
|
||||
"""Force a fresh Douyin login while preserving the previous usable session."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import datetime as dt
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
from gyxx_flow.modules.content_marketing.data.tools.relogin_runtime_paths import (
|
||||
child_browser_environment,
|
||||
resolve_relogin_runtime_paths,
|
||||
sync_relogin_state_to_consumers,
|
||||
)
|
||||
from gyxx_flow.modules.content_marketing.runtime_paths import PATHS
|
||||
|
||||
try:
|
||||
from .relogin_transaction import begin, commit, rollback
|
||||
except ImportError: # Direct script execution from data/tools.
|
||||
from gyxx_flow.modules.content_marketing.data.tools.relogin_transaction import (
|
||||
begin,
|
||||
commit,
|
||||
rollback,
|
||||
)
|
||||
|
||||
PROJECT_DIR = PATHS.module_root
|
||||
RUNTIME_PATHS = resolve_relogin_runtime_paths("douyin")
|
||||
COOKIE_FILE = RUNTIME_PATHS.cookie_file
|
||||
STATE_FILE = RUNTIME_PATHS.storage_state_file
|
||||
PROFILE_DIR = RUNTIME_PATHS.profile_dir
|
||||
SCRAPER = PROJECT_DIR / "douyin_comment_scraper.py"
|
||||
|
||||
LOGIN_COOKIE = "sessionid"
|
||||
|
||||
|
||||
def reset_state():
|
||||
state = begin((COOKIE_FILE, STATE_FILE), PROFILE_DIR)
|
||||
print(" [OK] Old Cookie/Profile staged; failures will roll back automatically")
|
||||
return state
|
||||
|
||||
|
||||
def is_cookie_valid() -> bool:
|
||||
if not COOKIE_FILE.exists():
|
||||
return False
|
||||
try:
|
||||
cookies = json.loads(COOKIE_FILE.read_text(encoding="utf-8"))
|
||||
return any(
|
||||
isinstance(cookie, dict)
|
||||
and cookie.get("name") == LOGIN_COOKIE
|
||||
and cookie.get("value")
|
||||
for cookie in cookies
|
||||
)
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description="Relogin Douyin")
|
||||
parser.add_argument("--login-timeout", type=int, default=300)
|
||||
parser.add_argument("--dry-run", action="store_true")
|
||||
args = parser.parse_args()
|
||||
|
||||
print("== Relogin: Douyin ==")
|
||||
print(f" date: {dt.date.today()}")
|
||||
print(f" cookie: {COOKIE_FILE}")
|
||||
print(f" profile: {PROFILE_DIR}")
|
||||
print(f" current cookie: {'present' if is_cookie_valid() else 'missing/invalid'}")
|
||||
|
||||
if args.dry_run:
|
||||
print("[DRY-RUN] Would stage Cookie/Profile and open the QR login window")
|
||||
return 0
|
||||
|
||||
old_state = reset_state()
|
||||
command = [
|
||||
sys.executable,
|
||||
str(SCRAPER),
|
||||
"login",
|
||||
"--login-only",
|
||||
"--login-timeout",
|
||||
str(args.login_timeout),
|
||||
]
|
||||
print(f"\n [RUN] {' '.join(command)}")
|
||||
try:
|
||||
return_code = subprocess.call(
|
||||
command,
|
||||
cwd=str(PROJECT_DIR),
|
||||
env=child_browser_environment(
|
||||
cookie_file=COOKIE_FILE,
|
||||
storage_state_file=STATE_FILE,
|
||||
profile_dir=PROFILE_DIR,
|
||||
),
|
||||
)
|
||||
except BaseException:
|
||||
rollback(old_state)
|
||||
raise
|
||||
|
||||
login_ok = return_code == 0 and is_cookie_valid()
|
||||
print(f"\n exit={return_code}")
|
||||
print(f" {LOGIN_COOKIE} written: {'OK' if login_ok else 'FAIL'}")
|
||||
if not login_ok:
|
||||
rollback(old_state)
|
||||
print(" [ROLLBACK] Login incomplete; restored previous Cookie/Profile")
|
||||
return return_code or 1
|
||||
try:
|
||||
sync_relogin_state_to_consumers(
|
||||
"douyin",
|
||||
cookie_file=COOKIE_FILE,
|
||||
storage_state_file=STATE_FILE,
|
||||
)
|
||||
except Exception as exc:
|
||||
rollback(old_state)
|
||||
print(f" [ROLLBACK] Failed to sync collector login state: {exc}")
|
||||
return 1
|
||||
|
||||
commit(old_state)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -1,148 +0,0 @@
|
||||
"""每周一重登小红书蒲公英。
|
||||
|
||||
强制清掉 cookie JSON + 浏览器 profile,确保下次启动时必须扫码。
|
||||
任务计划: 每周一 09:00
|
||||
|
||||
跑法:
|
||||
python data/tools/relogin_pgy.py # 实际跑(会弹二维码)
|
||||
python data/tools/relogin_pgy.py --dry-run # 只看会做什么
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import datetime as dt
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from gyxx_flow.modules.content_marketing.data.tools.relogin_runtime_paths import (
|
||||
child_browser_environment,
|
||||
resolve_relogin_runtime_paths,
|
||||
sync_relogin_state_to_consumers,
|
||||
)
|
||||
from gyxx_flow.modules.content_marketing.runtime_paths import PATHS
|
||||
|
||||
PROJECT_DIR = PATHS.module_root
|
||||
RUNTIME_PATHS = resolve_relogin_runtime_paths("pgy")
|
||||
COOKIE_FILE = RUNTIME_PATHS.cookie_file
|
||||
STATE_FILE = RUNTIME_PATHS.storage_state_file
|
||||
PROFILE_DIR = RUNTIME_PATHS.profile_dir
|
||||
SCRAPER = PROJECT_DIR / "pgy_xhs_scraper_v2.py"
|
||||
|
||||
|
||||
def reset_state() -> tuple[Path | None, Path | None, Path | None]:
|
||||
"""Temporarily move the old login state aside so it can be restored."""
|
||||
ts = dt.datetime.now().strftime("%Y%m%d_%H%M%S_%f")
|
||||
cookie_backup = None
|
||||
state_backup = None
|
||||
profile_backup = None
|
||||
if COOKIE_FILE.exists():
|
||||
backup = COOKIE_FILE.with_suffix(f".json.bak.{ts}")
|
||||
COOKIE_FILE.rename(backup)
|
||||
cookie_backup = backup
|
||||
print(f" [OK] 备份 cookie -> {backup.name}")
|
||||
else:
|
||||
print(f" [INFO] {COOKIE_FILE.name} 不存在,跳过")
|
||||
if STATE_FILE.exists():
|
||||
state_backup = STATE_FILE.with_suffix(f".json.bak.{ts}")
|
||||
STATE_FILE.rename(state_backup)
|
||||
print(f" [OK] 备份 storage state -> {state_backup.name}")
|
||||
else:
|
||||
print(f" [INFO] {STATE_FILE.name} 不存在,跳过")
|
||||
if PROFILE_DIR.exists():
|
||||
profile_backup = PROFILE_DIR.with_name(f"{PROFILE_DIR.name}.bak.{ts}")
|
||||
PROFILE_DIR.rename(profile_backup)
|
||||
print(f" [OK] 暂存 {PROFILE_DIR.name}/ -> {profile_backup.name}/")
|
||||
else:
|
||||
print(f" [INFO] {PROFILE_DIR.name}/ 不存在,跳过")
|
||||
return cookie_backup, profile_backup, state_backup
|
||||
|
||||
|
||||
def restore_previous_state(
|
||||
cookie_backup: Path | None,
|
||||
profile_backup: Path | None,
|
||||
state_backup: Path | None = None,
|
||||
) -> None:
|
||||
"""Discard an incomplete login and restore the last known-good state."""
|
||||
if COOKIE_FILE.exists():
|
||||
COOKIE_FILE.unlink()
|
||||
if cookie_backup and cookie_backup.exists():
|
||||
cookie_backup.rename(COOKIE_FILE)
|
||||
print(f" [ROLLBACK] 已恢复旧 cookie: {COOKIE_FILE.name}")
|
||||
if STATE_FILE.exists():
|
||||
STATE_FILE.unlink()
|
||||
if state_backup and state_backup.exists():
|
||||
state_backup.rename(STATE_FILE)
|
||||
print(f" [ROLLBACK] 已恢复旧 storage state: {STATE_FILE.name}")
|
||||
if PROFILE_DIR.exists():
|
||||
shutil.rmtree(PROFILE_DIR)
|
||||
if profile_backup and profile_backup.exists():
|
||||
profile_backup.rename(PROFILE_DIR)
|
||||
print(f" [ROLLBACK] 已恢复旧浏览器状态: {PROFILE_DIR.name}/")
|
||||
|
||||
|
||||
def finish_successful_relogin(profile_backup: Path | None) -> None:
|
||||
"""Keep the new state and remove the bulky old profile backup."""
|
||||
if profile_backup and profile_backup.exists():
|
||||
shutil.rmtree(profile_backup)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description="每周一重登小红书蒲公英")
|
||||
ap.add_argument("--login-timeout", type=int, default=300, help="扫码超时(秒)")
|
||||
ap.add_argument("--dry-run", action="store_true", help="只打印计划,不删不跑")
|
||||
args = ap.parse_args()
|
||||
|
||||
print("== Relogin: 小红书蒲公英 ==")
|
||||
print(f" 今天: {dt.date.today()} ({dt.date.today().strftime('%A')})")
|
||||
print(f" cookie: {COOKIE_FILE}")
|
||||
print(f" profile: {PROFILE_DIR}")
|
||||
print()
|
||||
|
||||
if args.dry_run:
|
||||
print("[DRY-RUN] 实际跑会: 删 cookie + 清 profile + 弹二维码")
|
||||
return 0
|
||||
|
||||
cookie_backup, profile_backup, state_backup = reset_state()
|
||||
cmd = [
|
||||
sys.executable,
|
||||
str(SCRAPER),
|
||||
"--login-only",
|
||||
"--login-timeout",
|
||||
str(args.login_timeout),
|
||||
]
|
||||
print(f"\n [RUN] {' '.join(cmd)}")
|
||||
try:
|
||||
rc = subprocess.call(
|
||||
cmd,
|
||||
cwd=str(PROJECT_DIR),
|
||||
env=child_browser_environment(
|
||||
cookie_file=COOKIE_FILE,
|
||||
storage_state_file=STATE_FILE,
|
||||
profile_dir=PROFILE_DIR,
|
||||
),
|
||||
)
|
||||
except BaseException:
|
||||
restore_previous_state(cookie_backup, profile_backup, state_backup)
|
||||
raise
|
||||
print(f"\n exit={rc}")
|
||||
if rc != 0 or not COOKIE_FILE.exists():
|
||||
print(" [WARN] 新登录未完成,恢复旧登录状态")
|
||||
restore_previous_state(cookie_backup, profile_backup, state_backup)
|
||||
return rc or 1
|
||||
try:
|
||||
sync_relogin_state_to_consumers(
|
||||
"pgy",
|
||||
cookie_file=COOKIE_FILE,
|
||||
storage_state_file=STATE_FILE,
|
||||
)
|
||||
except Exception as exc:
|
||||
restore_previous_state(cookie_backup, profile_backup, state_backup)
|
||||
print(f" [ROLLBACK] 同步采集器登录态失败: {exc}")
|
||||
return 1
|
||||
finish_successful_relogin(profile_backup)
|
||||
return rc
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -1,279 +0,0 @@
|
||||
"""Resolve one relogin wrapper's browser state contract.
|
||||
|
||||
The workflow runtime injects a unique browser binding into each relogin
|
||||
wrapper. Standalone legacy launches do not have that environment, so they
|
||||
retain the original per-platform files below the content-marketing state root.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import tempfile
|
||||
import uuid
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Mapping
|
||||
|
||||
from gyxx_flow.adapters.integration import RuntimeIntegrationCatalog
|
||||
from gyxx_flow.modules.content_marketing.runtime_paths import PATHS
|
||||
|
||||
_PLATFORM = re.compile(r"^[a-z][a-z0-9_-]{0,31}$")
|
||||
|
||||
CONSUMER_BINDING_IDS: Mapping[str, tuple[str, ...]] = {
|
||||
"bilibili": ("content.comments.collect_bilibili",),
|
||||
"douyin": ("content.comments.collect_douyin",),
|
||||
"pgy": ("content_marketing:pgy_xhs_scraper_v2.py",),
|
||||
"xiaohongshu": ("content.comments.collect_xiaohongshu",),
|
||||
# v2 now reads the Buyin/Doudian site and is owned by the douyin-shop
|
||||
# account. Keep the legacy Xingtu relogin isolated from that vault.
|
||||
"buyin": ("content_marketing:xingtu_scraper_v2.py",),
|
||||
"xingtu": ("content_marketing:xingtu_scraper.py",),
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ReloginRuntimePaths:
|
||||
cookie_file: Path
|
||||
storage_state_file: Path
|
||||
profile_dir: Path
|
||||
|
||||
|
||||
def resolve_relogin_runtime_paths(
|
||||
platform: str,
|
||||
environment: Mapping[str, str] | None = None,
|
||||
) -> ReloginRuntimePaths:
|
||||
"""Prefer the injected binding and fall back to original platform paths."""
|
||||
|
||||
if not _PLATFORM.fullmatch(platform):
|
||||
raise ValueError("invalid relogin platform")
|
||||
values = os.environ if environment is None else environment
|
||||
legacy_cookie = PATHS.state_root / "cookies" / f"{platform}_cookies.json"
|
||||
legacy_storage = (
|
||||
PATHS.state_root / "cookies" / f"{platform}_storage_state.json"
|
||||
)
|
||||
legacy_profile = PATHS.state_root / "browser-profiles" / platform
|
||||
return ReloginRuntimePaths(
|
||||
cookie_file=_bound_path(
|
||||
values,
|
||||
"GYXX_BROWSER_COOKIE_FILE",
|
||||
legacy_cookie,
|
||||
),
|
||||
storage_state_file=_bound_path(
|
||||
values,
|
||||
"GYXX_BROWSER_STORAGE_STATE_FILE",
|
||||
legacy_storage,
|
||||
),
|
||||
profile_dir=_bound_path(
|
||||
values,
|
||||
"GYXX_BROWSER_PROFILE_DIR",
|
||||
legacy_profile,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def child_browser_environment(
|
||||
*,
|
||||
cookie_file: Path,
|
||||
storage_state_file: Path,
|
||||
profile_dir: Path,
|
||||
environment: Mapping[str, str] | None = None,
|
||||
) -> dict[str, str]:
|
||||
"""Keep the nested login scraper on the wrapper's injected binding."""
|
||||
|
||||
result = dict(os.environ if environment is None else environment)
|
||||
result.update(
|
||||
{
|
||||
"GYXX_BROWSER_COOKIE_FILE": str(cookie_file.resolve()),
|
||||
"GYXX_BROWSER_STORAGE_STATE_FILE": str(
|
||||
storage_state_file.resolve()
|
||||
),
|
||||
"GYXX_BROWSER_PROFILE_DIR": str(profile_dir.resolve()),
|
||||
}
|
||||
)
|
||||
# content_marketing/sitecustomize.py normally rebinds a nested Python
|
||||
# script from argv[0]. A relogin scraper is an implementation detail of
|
||||
# this wrapper and must write the wrapper binding that the parent monitors.
|
||||
result.pop("GYXX_MODULE_ID", None)
|
||||
return result
|
||||
|
||||
|
||||
def sync_relogin_state_to_consumers(
|
||||
platform: str,
|
||||
*,
|
||||
cookie_file: Path,
|
||||
storage_state_file: Path,
|
||||
environment: Mapping[str, str] | None = None,
|
||||
) -> tuple[Path, ...]:
|
||||
"""Atomically publish a verified relogin state to unique consumer paths."""
|
||||
|
||||
try:
|
||||
consumer_ids = CONSUMER_BINDING_IDS[platform]
|
||||
except KeyError as exc:
|
||||
raise ValueError("invalid relogin platform") from exc
|
||||
values = os.environ if environment is None else environment
|
||||
project_root = values.get("GYXX_PROJECT_ROOT", "").strip()
|
||||
data_root = values.get("GYXX_DATA_ROOT", "").strip()
|
||||
if not project_root and not data_root:
|
||||
# Original standalone launches use the legacy per-platform state files
|
||||
# directly, so there is no separate managed consumer to publish to.
|
||||
return ()
|
||||
if not project_root or not data_root:
|
||||
raise RuntimeError("managed relogin state sync requires project and data roots")
|
||||
|
||||
catalog = RuntimeIntegrationCatalog.load_default(
|
||||
project_root=project_root,
|
||||
data_root=data_root,
|
||||
)
|
||||
source_cookie = cookie_file.expanduser().resolve()
|
||||
source_storage = storage_state_file.expanduser().resolve()
|
||||
cookie_payload = _read_cookie_payload(source_cookie)
|
||||
storage_available = source_storage.is_file()
|
||||
if storage_available:
|
||||
_read_storage_state_payload(source_storage)
|
||||
|
||||
source_paths = {source_cookie, source_storage}
|
||||
destination_paths: set[Path] = set()
|
||||
destinations: list[Path] = []
|
||||
updates: list[tuple[Path, Path]] = []
|
||||
for consumer_id in consumer_ids:
|
||||
binding = catalog.binding_for(consumer_id)
|
||||
target_cookie = binding.cookie_file.resolve()
|
||||
target_storage = binding.storage_state_file.resolve()
|
||||
targets = (target_cookie, target_storage)
|
||||
if any(target in source_paths for target in targets) or any(
|
||||
target in destination_paths for target in targets
|
||||
):
|
||||
raise RuntimeError(
|
||||
"relogin and consumer browser state paths must be unique"
|
||||
)
|
||||
destination_paths.update(targets)
|
||||
_validate_cookie_requirements(
|
||||
cookie_payload,
|
||||
required_domains=binding.required_cookie_domains,
|
||||
required_names=binding.required_cookie_names,
|
||||
)
|
||||
updates.append((source_cookie, target_cookie))
|
||||
if storage_available:
|
||||
updates.append((source_storage, target_storage))
|
||||
destinations.append(target_cookie)
|
||||
_atomic_replace_files(updates)
|
||||
return tuple(destinations)
|
||||
|
||||
|
||||
def _bound_path(
|
||||
environment: Mapping[str, str],
|
||||
name: str,
|
||||
fallback: Path,
|
||||
) -> Path:
|
||||
configured = environment.get(name, "").strip()
|
||||
return (
|
||||
Path(configured).expanduser().resolve()
|
||||
if configured
|
||||
else fallback.resolve()
|
||||
)
|
||||
|
||||
|
||||
def _read_cookie_payload(path: Path) -> list[dict[str, object]]:
|
||||
try:
|
||||
payload = json.loads(path.read_text(encoding="utf-8"))
|
||||
except (OSError, UnicodeError, ValueError) as exc:
|
||||
raise RuntimeError("relogin cookie state is missing or invalid") from exc
|
||||
if not isinstance(payload, list) or not payload or not all(
|
||||
isinstance(cookie, dict) for cookie in payload
|
||||
):
|
||||
raise RuntimeError("relogin cookie state is missing or invalid")
|
||||
return payload
|
||||
|
||||
|
||||
def _read_storage_state_payload(path: Path) -> dict[str, object]:
|
||||
try:
|
||||
payload = json.loads(path.read_text(encoding="utf-8"))
|
||||
except (OSError, UnicodeError, ValueError) as exc:
|
||||
raise RuntimeError("relogin storage state is invalid") from exc
|
||||
if not isinstance(payload, dict):
|
||||
raise RuntimeError("relogin storage state is invalid")
|
||||
return payload
|
||||
|
||||
|
||||
def _validate_cookie_requirements(
|
||||
cookies: list[dict[str, object]],
|
||||
*,
|
||||
required_domains: tuple[str, ...],
|
||||
required_names: tuple[str, ...],
|
||||
) -> None:
|
||||
usable = [cookie for cookie in cookies if cookie.get("value")]
|
||||
if not usable:
|
||||
raise RuntimeError("relogin cookie state has no usable cookies")
|
||||
if required_domains and not any(
|
||||
any(
|
||||
str(cookie.get("domain", "")).lstrip(".").casefold()
|
||||
.endswith(domain.casefold().lstrip("."))
|
||||
for domain in required_domains
|
||||
)
|
||||
for cookie in usable
|
||||
):
|
||||
raise RuntimeError("relogin cookie state does not match consumer domain")
|
||||
if required_names and not any(
|
||||
str(cookie.get("name", "")) in required_names for cookie in usable
|
||||
):
|
||||
raise RuntimeError("relogin cookie state misses the consumer login cookie")
|
||||
|
||||
|
||||
def _atomic_replace_files(updates: list[tuple[Path, Path]]) -> None:
|
||||
"""Replace a small related file set and restore every old target on error."""
|
||||
|
||||
transaction = uuid.uuid4().hex
|
||||
staged: list[tuple[Path, Path]] = []
|
||||
backups: dict[Path, Path | None] = {}
|
||||
replaced: list[Path] = []
|
||||
try:
|
||||
for source, target in updates:
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
handle, raw_temp = tempfile.mkstemp(
|
||||
prefix=f".{target.name}.{transaction}.",
|
||||
suffix=".tmp",
|
||||
dir=target.parent,
|
||||
)
|
||||
temp = Path(raw_temp)
|
||||
with os.fdopen(handle, "wb") as stream:
|
||||
stream.write(source.read_bytes())
|
||||
stream.flush()
|
||||
os.fsync(stream.fileno())
|
||||
staged.append((temp, target))
|
||||
backup = target.with_name(f".{target.name}.{transaction}.bak")
|
||||
if target.is_file():
|
||||
shutil.copy2(target, backup)
|
||||
backups[target] = backup
|
||||
else:
|
||||
backups[target] = None
|
||||
|
||||
for temp, target in staged:
|
||||
os.replace(temp, target)
|
||||
replaced.append(target)
|
||||
except BaseException:
|
||||
for target in reversed(replaced):
|
||||
backup = backups.get(target)
|
||||
if backup is not None and backup.exists():
|
||||
os.replace(backup, target)
|
||||
elif target.exists():
|
||||
target.unlink()
|
||||
raise
|
||||
finally:
|
||||
for temp, _target in staged:
|
||||
if temp.exists():
|
||||
temp.unlink()
|
||||
for backup in backups.values():
|
||||
if backup is not None and backup.exists():
|
||||
backup.unlink()
|
||||
|
||||
|
||||
__all__ = [
|
||||
"CONSUMER_BINDING_IDS",
|
||||
"ReloginRuntimePaths",
|
||||
"child_browser_environment",
|
||||
"resolve_relogin_runtime_paths",
|
||||
"sync_relogin_state_to_consumers",
|
||||
]
|
||||
@@ -1,47 +0,0 @@
|
||||
"""Transactional backup/rollback helpers for interactive relogin scripts."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime as dt
|
||||
import shutil
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
@dataclass
|
||||
class LoginStateBackup:
|
||||
files: dict[Path, Path]
|
||||
profile: Path
|
||||
profile_backup: Path | None
|
||||
|
||||
|
||||
def begin(files: tuple[Path, ...], profile: Path) -> LoginStateBackup:
|
||||
stamp = dt.datetime.now().strftime("%Y%m%d_%H%M%S_%f")
|
||||
backups: dict[Path, Path] = {}
|
||||
for source in files:
|
||||
if source.exists():
|
||||
backup = source.with_name(f"{source.name}.bak.{stamp}")
|
||||
source.rename(backup)
|
||||
backups[source] = backup
|
||||
profile_backup = None
|
||||
if profile.exists():
|
||||
profile_backup = profile.with_name(f"{profile.name}.bak.{stamp}")
|
||||
profile.rename(profile_backup)
|
||||
return LoginStateBackup(backups, profile, profile_backup)
|
||||
|
||||
|
||||
def rollback(state: LoginStateBackup) -> None:
|
||||
for source, backup in state.files.items():
|
||||
if source.exists():
|
||||
source.unlink()
|
||||
if backup.exists():
|
||||
backup.rename(source)
|
||||
if state.profile.exists():
|
||||
shutil.rmtree(state.profile)
|
||||
if state.profile_backup and state.profile_backup.exists():
|
||||
state.profile_backup.rename(state.profile)
|
||||
|
||||
|
||||
def commit(state: LoginStateBackup) -> None:
|
||||
if state.profile_backup and state.profile_backup.exists():
|
||||
shutil.rmtree(state.profile_backup)
|
||||
@@ -1,116 +0,0 @@
|
||||
"""每周重登小红书。
|
||||
|
||||
强制清掉 cookie JSON + 浏览器 profile,确保下次启动时必须扫码。
|
||||
跑法:
|
||||
python data/tools/relogin_xiaohongshu.py # 实际跑(会弹二维码)
|
||||
python data/tools/relogin_xiaohongshu.py --dry-run # 只看会做什么
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import datetime as dt
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
from gyxx_flow.modules.content_marketing.data.tools.relogin_runtime_paths import (
|
||||
child_browser_environment,
|
||||
resolve_relogin_runtime_paths,
|
||||
sync_relogin_state_to_consumers,
|
||||
)
|
||||
from gyxx_flow.modules.content_marketing.data.tools.relogin_transaction import (
|
||||
begin,
|
||||
commit,
|
||||
rollback,
|
||||
)
|
||||
from gyxx_flow.modules.content_marketing.runtime_paths import PATHS
|
||||
|
||||
PROJECT_DIR = PATHS.module_root
|
||||
RUNTIME_PATHS = resolve_relogin_runtime_paths("xiaohongshu")
|
||||
COOKIE_FILE = RUNTIME_PATHS.cookie_file
|
||||
STATE_FILE = RUNTIME_PATHS.storage_state_file
|
||||
PROFILE_DIR = RUNTIME_PATHS.profile_dir
|
||||
SCRAPER = PROJECT_DIR / "xiaohongshu_comment_scraper.py"
|
||||
|
||||
LOGIN_COOKIE = "web_session"
|
||||
|
||||
|
||||
def reset_state():
|
||||
state = begin((COOKIE_FILE, STATE_FILE), PROFILE_DIR)
|
||||
print(" [OK] 旧 Cookie/Profile 已暂存,失败时将自动恢复")
|
||||
return state
|
||||
|
||||
|
||||
def is_cookie_valid() -> bool:
|
||||
if not COOKIE_FILE.exists():
|
||||
return False
|
||||
try:
|
||||
cookies = json.loads(COOKIE_FILE.read_text(encoding="utf-8"))
|
||||
return any(c.get("name") == LOGIN_COOKIE and c.get("value") for c in cookies)
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description="重登小红书")
|
||||
ap.add_argument("--login-timeout", type=int, default=300, help="扫码超时(秒)")
|
||||
ap.add_argument("--dry-run", action="store_true", help="只打印计划,不删不跑")
|
||||
args = ap.parse_args()
|
||||
|
||||
print("== Relogin: 小红书 ==")
|
||||
print(f" 今天: {dt.date.today()}")
|
||||
print(f" cookie: {COOKIE_FILE}")
|
||||
print(f" profile: {PROFILE_DIR}")
|
||||
print(f" 当前 cookie 状态: {'有效' if is_cookie_valid() else '无效/不存在'}")
|
||||
print()
|
||||
|
||||
if args.dry_run:
|
||||
print("[DRY-RUN] 实际跑会: 删 cookie + 清 profile + 弹二维码")
|
||||
return 0
|
||||
|
||||
old_state = reset_state()
|
||||
cmd = [
|
||||
sys.executable,
|
||||
str(SCRAPER),
|
||||
"login",
|
||||
"--login-only",
|
||||
"--login-timeout",
|
||||
str(args.login_timeout),
|
||||
]
|
||||
print(f"\n [RUN] {' '.join(cmd)}")
|
||||
try:
|
||||
rc = subprocess.call(
|
||||
cmd,
|
||||
cwd=str(PROJECT_DIR),
|
||||
env=child_browser_environment(
|
||||
cookie_file=COOKIE_FILE,
|
||||
storage_state_file=STATE_FILE,
|
||||
profile_dir=PROFILE_DIR,
|
||||
),
|
||||
)
|
||||
except BaseException:
|
||||
rollback(old_state)
|
||||
raise
|
||||
print(f"\n exit={rc}")
|
||||
login_ok = rc == 0 and is_cookie_valid()
|
||||
if rc == 0:
|
||||
print(f" {LOGIN_COOKIE} 写入: {'OK' if login_ok else 'FAIL — 没有看到 web_session,登录可能没成功'}")
|
||||
if not login_ok:
|
||||
rollback(old_state)
|
||||
print(" [ROLLBACK] 未完成扫码,已恢复旧登录状态")
|
||||
return rc or 1
|
||||
try:
|
||||
sync_relogin_state_to_consumers(
|
||||
"xiaohongshu",
|
||||
cookie_file=COOKIE_FILE,
|
||||
storage_state_file=STATE_FILE,
|
||||
)
|
||||
except Exception as exc:
|
||||
rollback(old_state)
|
||||
print(f" [ROLLBACK] 同步采集器登录态失败: {exc}")
|
||||
return 1
|
||||
commit(old_state)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -1,350 +0,0 @@
|
||||
"""每 3 个周一重登一次抖音/星图。
|
||||
|
||||
强制清掉 cookie JSON + 浏览器 profile,确保下次启动时必须扫码。
|
||||
星图登录完成后,自动把登录态 cookie 同步到抖音评论 scraper,避免再扫一次码。
|
||||
|
||||
任务计划: 每周一 09:00 (脚本内部判断是否到了该重登的周一,不到就退出)
|
||||
|
||||
3 周循环基准: 任意一个周一,后续 (今天 - BASE_DATE) // 7 % 3 == 0 才是星图周。
|
||||
改 BASE_DATE 就改了重登周期。
|
||||
|
||||
跑法:
|
||||
python data/tools/relogin_xingtu.py # 实际跑
|
||||
python data/tools/relogin_xingtu.py --dry-run # 看今天是不是星图周
|
||||
python data/tools/relogin_xingtu.py --force # 忽略 3 周判断,强制重登
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import datetime as dt
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import psutil
|
||||
|
||||
from gyxx_flow.modules.content_marketing.data.tools.relogin_runtime_paths import (
|
||||
child_browser_environment,
|
||||
resolve_relogin_runtime_paths,
|
||||
sync_relogin_state_to_consumers,
|
||||
)
|
||||
from gyxx_flow.modules.content_marketing.runtime_paths import PATHS
|
||||
|
||||
PROJECT_DIR = PATHS.module_root
|
||||
RUNTIME_PATHS = resolve_relogin_runtime_paths("xingtu")
|
||||
COOKIE_FILE = RUNTIME_PATHS.cookie_file
|
||||
STATE_FILE = RUNTIME_PATHS.storage_state_file
|
||||
PROFILE_DIR = RUNTIME_PATHS.profile_dir
|
||||
SCRAPER = PROJECT_DIR / "xingtu_scraper_v2.py"
|
||||
|
||||
# 抖音评论 scraper 的 cookie 文件 (登录态共享目标)
|
||||
DY_COOKIE_FILE = PATHS.state_root / "cookies/douyin_cookies.json"
|
||||
DY_STATE_FILE = PATHS.state_root / "cookies/douyin_storage_state.json"
|
||||
DY_PROFILE_DIR = PATHS.state_root / "browser-profiles/douyin"
|
||||
|
||||
# 需要同步的抖音登录 cookie 名
|
||||
DY_LOGIN_COOKIES = {"sessionid", "sessionid_ss", "sid_guard", "sid_tt", "passport_csrf_token"}
|
||||
|
||||
# 3 周循环起点(必须是周一)。星图 cookie ~22 天,每 3 周(21 天)重登正好。
|
||||
BASE_DATE = dt.date(2026, 6, 22)
|
||||
LOGIN_COOKIE_NAMES = {"sessionid", "sessionid_ss", "sid_tt", "sid_guard"}
|
||||
|
||||
|
||||
def transaction_file() -> Path:
|
||||
return COOKIE_FILE.with_name(".xingtu_relogin_transaction.json")
|
||||
|
||||
|
||||
def _write_transaction(
|
||||
cookie_backup: Path | None,
|
||||
profile_backup: Path | None,
|
||||
state_backup: Path | None,
|
||||
) -> None:
|
||||
marker = transaction_file()
|
||||
marker.parent.mkdir(parents=True, exist_ok=True)
|
||||
temp = marker.with_name(f"{marker.name}.tmp")
|
||||
temp.write_text(json.dumps({
|
||||
"cookie_backup": str(cookie_backup) if cookie_backup else "",
|
||||
"state_backup": str(state_backup) if state_backup else "",
|
||||
"profile_backup": str(profile_backup) if profile_backup else "",
|
||||
}, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
os.replace(temp, marker)
|
||||
|
||||
|
||||
def _clear_transaction() -> None:
|
||||
marker = transaction_file()
|
||||
if marker.exists():
|
||||
marker.unlink()
|
||||
|
||||
|
||||
def _remove_path(path: Path) -> None:
|
||||
if path.is_dir():
|
||||
shutil.rmtree(path)
|
||||
elif path.exists():
|
||||
path.unlink()
|
||||
|
||||
|
||||
def recover_interrupted_state() -> bool:
|
||||
"""Restore a relogin transaction that was killed after moving old state."""
|
||||
marker = transaction_file()
|
||||
if not marker.exists():
|
||||
return False
|
||||
try:
|
||||
state = json.loads(marker.read_text(encoding="utf-8"))
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
cookie_backup = Path(state["cookie_backup"]) if state.get("cookie_backup") else None
|
||||
state_backup = Path(state["state_backup"]) if state.get("state_backup") else None
|
||||
profile_backup = Path(state["profile_backup"]) if state.get("profile_backup") else None
|
||||
terminate_profile_processes()
|
||||
if cookie_backup and cookie_backup.exists():
|
||||
_remove_path(COOKIE_FILE)
|
||||
cookie_backup.rename(COOKIE_FILE)
|
||||
if state_backup and state_backup.exists():
|
||||
_remove_path(STATE_FILE)
|
||||
state_backup.rename(STATE_FILE)
|
||||
if profile_backup and profile_backup.exists():
|
||||
_remove_path(PROFILE_DIR)
|
||||
profile_backup.rename(PROFILE_DIR)
|
||||
_clear_transaction()
|
||||
print(" [RECOVER] 已恢复上次被中断的星图登录状态")
|
||||
return True
|
||||
|
||||
|
||||
def has_valid_login_cookie_file(path: Path | None = None) -> bool:
|
||||
path = path or COOKIE_FILE
|
||||
try:
|
||||
cookies = json.loads(path.read_text(encoding="utf-8"))
|
||||
except Exception:
|
||||
return False
|
||||
return any(
|
||||
isinstance(cookie, dict)
|
||||
and cookie.get("name") in LOGIN_COOKIE_NAMES
|
||||
and cookie.get("value")
|
||||
for cookie in cookies
|
||||
)
|
||||
|
||||
|
||||
def terminate_profile_processes() -> None:
|
||||
"""Stop only browser processes using Xingtu's dedicated profile."""
|
||||
profile_text = str(PROFILE_DIR.resolve()).casefold()
|
||||
matches = []
|
||||
for process in psutil.process_iter(["pid", "cmdline"]):
|
||||
try:
|
||||
command = " ".join(process.info.get("cmdline") or []).casefold()
|
||||
if process.pid != os.getpid() and profile_text in command:
|
||||
matches.append(process)
|
||||
except (psutil.Error, OSError):
|
||||
continue
|
||||
for process in matches:
|
||||
try:
|
||||
process.terminate()
|
||||
except (psutil.Error, OSError):
|
||||
pass
|
||||
_gone, alive = psutil.wait_procs(matches, timeout=5)
|
||||
for process in alive:
|
||||
try:
|
||||
process.kill()
|
||||
except (psutil.Error, OSError):
|
||||
pass
|
||||
|
||||
|
||||
def is_xingtu_week(today: dt.date, base: dt.date) -> bool:
|
||||
days = (today - base).days
|
||||
if days < 0:
|
||||
return False
|
||||
return (days // 7) % 3 == 0
|
||||
|
||||
|
||||
def reset_state() -> tuple[Path | None, Path | None, Path | None]:
|
||||
"""Temporarily move the old login state aside so it can be restored."""
|
||||
ts = dt.datetime.now().strftime("%Y%m%d_%H%M%S_%f")
|
||||
cookie_backup = None
|
||||
state_backup = None
|
||||
profile_backup = None
|
||||
if COOKIE_FILE.exists():
|
||||
cookie_backup = COOKIE_FILE.with_suffix(f".json.bak.{ts}")
|
||||
if STATE_FILE.exists():
|
||||
state_backup = STATE_FILE.with_suffix(f".json.bak.{ts}")
|
||||
if PROFILE_DIR.exists():
|
||||
profile_backup = PROFILE_DIR.with_name(f"{PROFILE_DIR.name}.bak.{ts}")
|
||||
_write_transaction(cookie_backup, profile_backup, state_backup)
|
||||
|
||||
if cookie_backup:
|
||||
COOKIE_FILE.rename(cookie_backup)
|
||||
print(f" [OK] 备份 cookie -> {cookie_backup.name}")
|
||||
else:
|
||||
print(f" [INFO] {COOKIE_FILE.name} 不存在,跳过")
|
||||
if state_backup:
|
||||
STATE_FILE.rename(state_backup)
|
||||
print(f" [OK] 备份 storage state -> {state_backup.name}")
|
||||
else:
|
||||
print(f" [INFO] {STATE_FILE.name} 不存在,跳过")
|
||||
if profile_backup:
|
||||
PROFILE_DIR.rename(profile_backup)
|
||||
print(f" [OK] 暂存 {PROFILE_DIR.name}/ -> {profile_backup.name}/")
|
||||
else:
|
||||
print(f" [INFO] {PROFILE_DIR.name}/ 不存在,跳过")
|
||||
return cookie_backup, profile_backup, state_backup
|
||||
|
||||
|
||||
def restore_previous_state(
|
||||
cookie_backup: Path | None,
|
||||
profile_backup: Path | None,
|
||||
state_backup: Path | None = None,
|
||||
) -> None:
|
||||
"""Discard an incomplete login and restore the last known-good state."""
|
||||
terminate_profile_processes()
|
||||
if COOKIE_FILE.exists():
|
||||
COOKIE_FILE.unlink()
|
||||
if cookie_backup and cookie_backup.exists():
|
||||
cookie_backup.rename(COOKIE_FILE)
|
||||
print(f" [ROLLBACK] 已恢复旧 cookie: {COOKIE_FILE.name}")
|
||||
_remove_path(STATE_FILE)
|
||||
if state_backup and state_backup.exists():
|
||||
state_backup.rename(STATE_FILE)
|
||||
print(f" [ROLLBACK] 已恢复旧 storage state: {STATE_FILE.name}")
|
||||
if PROFILE_DIR.exists():
|
||||
shutil.rmtree(PROFILE_DIR)
|
||||
if profile_backup and profile_backup.exists():
|
||||
profile_backup.rename(PROFILE_DIR)
|
||||
print(f" [ROLLBACK] 已恢复旧浏览器状态: {PROFILE_DIR.name}/")
|
||||
_clear_transaction()
|
||||
|
||||
|
||||
def finish_successful_relogin(profile_backup: Path | None) -> None:
|
||||
"""Keep the new state and remove the bulky old profile backup."""
|
||||
if profile_backup and profile_backup.exists():
|
||||
shutil.rmtree(profile_backup)
|
||||
_clear_transaction()
|
||||
|
||||
|
||||
def sync_to_douyin_scraper() -> None:
|
||||
"""把星图的登录态 cookie 同步到抖音评论 scraper,避免再扫一次码。"""
|
||||
if not COOKIE_FILE.exists():
|
||||
print(" [SKIP] 星图 cookie 不存在,无法同步")
|
||||
return
|
||||
|
||||
try:
|
||||
xingtu_cookies = json.loads(COOKIE_FILE.read_text(encoding="utf-8"))
|
||||
except Exception as exc:
|
||||
print(f" [SKIP] 读取星图 cookie 失败: {exc}")
|
||||
return
|
||||
|
||||
# 提取登录态 cookie
|
||||
login_cookies = [c for c in xingtu_cookies if isinstance(c, dict) and c.get("name") in DY_LOGIN_COOKIES]
|
||||
if not login_cookies:
|
||||
print(" [SKIP] 星图 cookie 中没有找到登录态,跳过同步")
|
||||
return
|
||||
|
||||
# 清除抖音评论 scraper 的旧 profile (强制刷新)
|
||||
if DY_PROFILE_DIR.exists():
|
||||
shutil.rmtree(DY_PROFILE_DIR)
|
||||
print(f" [OK] 清空 {DY_PROFILE_DIR.name}/")
|
||||
|
||||
# 读取或创建抖音评论 cookie 文件
|
||||
dy_cookies = []
|
||||
if DY_COOKIE_FILE.exists():
|
||||
try:
|
||||
dy_cookies = json.loads(DY_COOKIE_FILE.read_text(encoding="utf-8"))
|
||||
except Exception:
|
||||
dy_cookies = []
|
||||
|
||||
# 合并: 同名 cookie 覆盖,新 cookie 追加
|
||||
dy_by_name = {c.get("name"): c for c in dy_cookies if isinstance(c, dict)}
|
||||
for c in login_cookies:
|
||||
# 修正 domain: 星图的 cookie domain 可能是 .douyin.com 子域,评论采集需要 .douyin.com
|
||||
domain = c.get("domain", "")
|
||||
if "xingtu" in domain or "oceanengine" in domain:
|
||||
c["domain"] = ".douyin.com"
|
||||
dy_by_name[c["name"]] = c
|
||||
|
||||
DY_COOKIE_FILE.parent.mkdir(parents=True, exist_ok=True)
|
||||
DY_COOKIE_FILE.write_text(
|
||||
json.dumps(list(dy_by_name.values()), ensure_ascii=False, indent=2),
|
||||
encoding="utf-8",
|
||||
)
|
||||
print(f" [OK] 同步 {len(login_cookies)} 个登录 cookie 到 {DY_COOKIE_FILE.name}")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description="每 3 周周一重登星图")
|
||||
ap.add_argument("--base-date", default=BASE_DATE.isoformat(),
|
||||
help="3 周循环起点(必须是周一),格式 YYYY-MM-DD")
|
||||
ap.add_argument("--login-timeout", type=int, default=300)
|
||||
ap.add_argument("--force", action="store_true", help="忽略 3 周判断,强制重登")
|
||||
ap.add_argument("--dry-run", action="store_true", help="只打印计划")
|
||||
args = ap.parse_args()
|
||||
|
||||
today = dt.date.today()
|
||||
base = dt.date.fromisoformat(args.base_date)
|
||||
is_xw = is_xingtu_week(today, base)
|
||||
|
||||
print("== Relogin: 抖音/星图 ==")
|
||||
print(f" 今天: {today} ({today.strftime('%A')})")
|
||||
print(f" 3 周基准: {base}")
|
||||
print(f" 星图周? {'是' if is_xw else '否'}")
|
||||
print()
|
||||
|
||||
if not is_xw and not args.force:
|
||||
print("今天不是星图周,跳过 (传 --force 强制重登)")
|
||||
return 0
|
||||
|
||||
if args.dry_run:
|
||||
print("[DRY-RUN] 实际跑会: 删 cookie + 清 profile + 弹二维码")
|
||||
return 0
|
||||
|
||||
recover_interrupted_state()
|
||||
terminate_profile_processes()
|
||||
cookie_backup, profile_backup, state_backup = reset_state()
|
||||
cmd = [
|
||||
sys.executable,
|
||||
str(SCRAPER),
|
||||
"--login-only",
|
||||
"--login-timeout",
|
||||
str(args.login_timeout),
|
||||
]
|
||||
print(f"\n [RUN] {' '.join(cmd)}")
|
||||
try:
|
||||
rc = subprocess.call(
|
||||
cmd,
|
||||
cwd=str(PROJECT_DIR),
|
||||
env=child_browser_environment(
|
||||
cookie_file=COOKIE_FILE,
|
||||
storage_state_file=STATE_FILE,
|
||||
profile_dir=PROFILE_DIR,
|
||||
),
|
||||
)
|
||||
except BaseException:
|
||||
restore_previous_state(cookie_backup, profile_backup, state_backup)
|
||||
raise
|
||||
finally:
|
||||
terminate_profile_processes()
|
||||
print(f"\n exit={rc}")
|
||||
|
||||
if rc != 0 or not has_valid_login_cookie_file():
|
||||
print(" [WARN] 新登录未完成,恢复旧登录状态")
|
||||
restore_previous_state(cookie_backup, profile_backup, state_backup)
|
||||
return rc or 1
|
||||
try:
|
||||
sync_relogin_state_to_consumers(
|
||||
"xingtu",
|
||||
cookie_file=COOKIE_FILE,
|
||||
storage_state_file=STATE_FILE,
|
||||
)
|
||||
except Exception as exc:
|
||||
restore_previous_state(cookie_backup, profile_backup, state_backup)
|
||||
print(f" [ROLLBACK] 同步采集器登录态失败: {exc}")
|
||||
return 1
|
||||
|
||||
finish_successful_relogin(profile_backup)
|
||||
if rc == 0:
|
||||
sync_to_douyin_scraper()
|
||||
|
||||
return rc
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -426,6 +426,21 @@ def ensure_table(cur) -> None:
|
||||
)
|
||||
|
||||
|
||||
def drop_stale_rows(
|
||||
all_rows: dict[tuple, tuple], stale_paths: list[Path]
|
||||
) -> int:
|
||||
"""删除来源文件已陈旧的行,返回删除数(按行内 source_file 文件名匹配)。"""
|
||||
stale_names = {Path(path).name for path in stale_paths}
|
||||
source_index = ROW_FIELDS.index("source_file")
|
||||
stale_keys = [
|
||||
key for key, row in all_rows.items()
|
||||
if str(row[source_index]) in stale_names
|
||||
]
|
||||
for key in stale_keys:
|
||||
del all_rows[key]
|
||||
return len(stale_keys)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="同步自营内容(抖音/B站/小红书)全量指标到 cmt_self_video_metrics"
|
||||
@@ -504,19 +519,24 @@ def main() -> int:
|
||||
print("❌ 没有任何平台的来源文件可用")
|
||||
return 1
|
||||
stale = [
|
||||
str(path)
|
||||
path
|
||||
for path in sources
|
||||
if business_date_bound
|
||||
and not args.allow_stale
|
||||
and not args.use_file_mtime
|
||||
and not is_current_artifact(path, metric_date)
|
||||
]
|
||||
if stale and not args.allow_stale:
|
||||
if stale:
|
||||
# 陈旧防护按来源文件粒度跳过: 某平台来源过期只丢弃该平台的行,
|
||||
# 不拖累其他平台当天的新鲜数据(如抖音导出失败时不拦 B站/小红书)。
|
||||
dropped = drop_stale_rows(all_rows, stale)
|
||||
print(
|
||||
f"❌ 以下来源文件不是业务日期({metric_date.isoformat()})的产物; "
|
||||
"补历史请显式传 --metric-date 并加 --allow-stale:\n "
|
||||
+ "\n ".join(stale)
|
||||
f"⚠️ 以下来源文件不是业务日期({metric_date.isoformat()})的产物, "
|
||||
f"已跳过对应 {dropped} 行(补历史用 --metric-date + "
|
||||
"--allow-stale):\n " + "\n ".join(str(p) for p in stale)
|
||||
)
|
||||
if not all_rows:
|
||||
print("❌ 跳过陈旧来源后没有可入库的内容指标行")
|
||||
return 1
|
||||
|
||||
print(
|
||||
@@ -543,9 +563,29 @@ def main() -> int:
|
||||
from gyxx_flow.modules.content_marketing.data.tools.db import get_db_config
|
||||
|
||||
with psycopg.connect(**get_db_config()) as conn:
|
||||
written, failed = upsert_rows(conn, list(all_rows.values()))
|
||||
if failed:
|
||||
print(f"[sync-self] 完成: 写入 {written} 行,单行失败跳过 {failed} 行 "
|
||||
f"(见上方 ⚠️ 日志)")
|
||||
return 0 if written else 1
|
||||
print(f"[sync-self] 完成: 已幂等写入 {written} 行到 {TABLE_NAME}")
|
||||
return 0
|
||||
|
||||
|
||||
def upsert_rows(conn, rows: list[tuple]) -> tuple[int, int]:
|
||||
"""逐行 upsert,返回 (成功数, 失败数)。
|
||||
|
||||
每行放在独立保存点里执行: 单行失败(如超长值违反约束)只回滚该行,
|
||||
不拖累其他平台/其他内容的写入——这是「一个平台失败不影响全库」的
|
||||
最后一道隔离。
|
||||
"""
|
||||
written = 0
|
||||
failed = 0
|
||||
with conn.cursor() as cur:
|
||||
ensure_table(cur)
|
||||
for row in all_rows.values():
|
||||
for row in rows:
|
||||
try:
|
||||
with conn.transaction():
|
||||
# SQL 为静态字面量,29 个值全部经 %s 占位符参数绑定传入。
|
||||
cur.execute(
|
||||
"""
|
||||
@@ -607,9 +647,13 @@ def main() -> int:
|
||||
""",
|
||||
row,
|
||||
)
|
||||
except Exception as exc:
|
||||
failed += 1
|
||||
print(f" ⚠️ 单行写入失败已隔离跳过: {type(exc).__name__}: {exc}")
|
||||
else:
|
||||
written += 1
|
||||
conn.commit()
|
||||
print(f"[sync-self] 完成: 已幂等写入 {len(all_rows)} 行到 {TABLE_NAME}")
|
||||
return 0
|
||||
return written, failed
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -273,18 +273,15 @@ def maybe_wait_for_login(
|
||||
*,
|
||||
deadline: float | None = None,
|
||||
) -> None:
|
||||
login_deadline = time.monotonic() + login_timeout
|
||||
while time.monotonic() < login_deadline:
|
||||
_check_note_deadline(deadline, "login wait")
|
||||
# 登录统一由账号 vault 管理(gyxx accounts login douyin-content),
|
||||
# 由工作流执行前的 cookie 网关保证;运行中失效立即降级,不再等待人工扫码。
|
||||
try:
|
||||
body = page.locator("body").inner_text(timeout=5000)
|
||||
except Exception:
|
||||
body = ""
|
||||
if "扫码登录" not in body and "登录后" not in body:
|
||||
return
|
||||
log("Douyin may require login. Please finish login in the opened Chrome window.")
|
||||
page.wait_for_timeout(3000)
|
||||
_check_note_deadline(deadline, "login wait")
|
||||
if "扫码登录" in body or "登录后" in body:
|
||||
log("Douyin login state missing; collecting degraded samples. "
|
||||
"Run `gyxx accounts login douyin-content` to restore full comments.")
|
||||
|
||||
|
||||
def is_logged_in(page: Page) -> bool:
|
||||
@@ -833,55 +830,16 @@ def write_outputs(url: str, comments: list[dict[str, Any]], result: dict[str, An
|
||||
return json_path, csv_path
|
||||
|
||||
|
||||
def login_only(login_timeout: int) -> int:
|
||||
"""只触发登录(headed),登录完成立刻退出。给 relogin_douyin.py 用。"""
|
||||
def action(page: Page) -> None:
|
||||
page.goto("https://www.douyin.com/", wait_until="domcontentloaded", timeout=90000)
|
||||
page.wait_for_timeout(2500)
|
||||
# 主动触发登录弹窗(点登录按钮)
|
||||
try:
|
||||
login_btn = page.get_by_text("登录", exact=True).first
|
||||
if login_btn.count():
|
||||
login_btn.click(timeout=3000)
|
||||
page.wait_for_timeout(1500)
|
||||
except Exception:
|
||||
pass
|
||||
maybe_wait_for_login(page, login_timeout)
|
||||
save_state(page)
|
||||
|
||||
PROFILE_DIR.mkdir(parents=True, exist_ok=True)
|
||||
with ScraplingBrowser(
|
||||
headless=False,
|
||||
real_chrome=True,
|
||||
user_data_dir=str(PROFILE_DIR),
|
||||
locale="zh-CN",
|
||||
timezone_id="Asia/Shanghai",
|
||||
timeout=90000,
|
||||
network_idle=False,
|
||||
disable_resources=False,
|
||||
google_search=False,
|
||||
page_setup=restore_cookies,
|
||||
max_pages=1,
|
||||
) as browser:
|
||||
browser.run_fetch_action("https://www.douyin.com/", action, wait=1000)
|
||||
log("[login-only] 完成")
|
||||
return 0
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description="Scrape Douyin video comments via Scrapling browser and XHR interception.")
|
||||
parser.add_argument("url", help="Douyin video/user URL (传 'login' 配合 --login-only)")
|
||||
parser.add_argument("--login-timeout", type=int, default=300, help="等待扫码/登录秒数")
|
||||
parser.add_argument("url", help="Douyin video/user URL")
|
||||
parser.add_argument("--login-timeout", type=int, default=300,
|
||||
help="保留参数(登录等待已移除;未登录将直接降级采集)")
|
||||
parser.add_argument("--max-scrolls", type=int, default=120, help="最多滚动加载次数")
|
||||
parser.add_argument("--idle-rounds", type=int, default=8, help="连续几轮无新评论后停止")
|
||||
parser.add_argument("--headless", action="store_true", help="无头模式;首次登录不要使用")
|
||||
parser.add_argument("--login-only", action="store_true",
|
||||
help="只触发登录(headed)然后退出,给 relogin_douyin 用")
|
||||
parser.add_argument("--headless", action="store_true", help="无头模式")
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.login_only:
|
||||
return login_only(args.login_timeout)
|
||||
|
||||
comments, result = scrape_comments(args.url, args.login_timeout, args.max_scrolls, args.idle_rounds, args.headless)
|
||||
json_path, csv_path = write_outputs(args.url, comments, result)
|
||||
log(f"Final URL: {result.get('final_url', '')}")
|
||||
|
||||
@@ -38,6 +38,7 @@ from gyxx_flow.adapters.scrapling import (
|
||||
BrowserTimeoutError, # noqa: F401
|
||||
is_browser_timeout_error,
|
||||
)
|
||||
from gyxx_flow.adapters.browser import CookieExpiredError, cookie_expired_message
|
||||
from gyxx_flow.core.exit_codes import COOKIE_SKIP_EXIT_CODE
|
||||
from gyxx_flow.modules.content_marketing.collection_completeness import (
|
||||
BLOCKED_INPUT,
|
||||
@@ -333,56 +334,6 @@ def is_logged_in(page: Page) -> bool:
|
||||
return False
|
||||
|
||||
|
||||
def open_scan_login(page: Page) -> None:
|
||||
"""Open the login dialog and prefer its QR-code login tab."""
|
||||
for pattern in (r"^登录$|账号登录",):
|
||||
try:
|
||||
button = page.get_by_text(re.compile(pattern)).first
|
||||
if button.count() and button.is_visible():
|
||||
button.click(timeout=2000)
|
||||
page.wait_for_timeout(500)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# The login dialog uses an image-only folded-corner switch. Its bundled
|
||||
# asset name is stable even though the generated CSS class is not.
|
||||
try:
|
||||
qr_switch = page.locator('img[src*="qr_code"]').first
|
||||
if qr_switch.count() and qr_switch.is_visible():
|
||||
# The browser engine's actionability check is blocked by the folded-corner
|
||||
# overlay. A native DOM click triggers the image's real handler.
|
||||
qr_switch.evaluate("e => e.click()")
|
||||
page.wait_for_timeout(1000)
|
||||
return
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
for pattern in (r"扫码登录|二维码登录|扫码",):
|
||||
try:
|
||||
button = page.get_by_text(re.compile(pattern)).first
|
||||
if button.count() and button.is_visible():
|
||||
button.click(timeout=2000)
|
||||
page.wait_for_timeout(500)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def wait_for_scan_login(page: Page, timeout_seconds: int) -> None:
|
||||
log("Not logged in. 请在打开的 Chrome 中扫码登录。")
|
||||
open_scan_login(page)
|
||||
deadline = time.time() + timeout_seconds
|
||||
next_login_click = time.time() + 5
|
||||
while time.time() < deadline:
|
||||
if is_logged_in(page):
|
||||
save_state(page)
|
||||
return
|
||||
if time.time() >= next_login_click:
|
||||
open_scan_login(page)
|
||||
next_login_click = time.time() + 5
|
||||
page.wait_for_timeout(2000)
|
||||
raise TimeoutError(f"Login timed out after {timeout_seconds} seconds.")
|
||||
|
||||
|
||||
def ensure_login(page: Page, login_timeout: int) -> None:
|
||||
page.goto(HOME_URL, wait_until="domcontentloaded", timeout=60000)
|
||||
page.wait_for_timeout(2500)
|
||||
@@ -402,20 +353,11 @@ def ensure_login(page: Page, login_timeout: int) -> None:
|
||||
)
|
||||
log(f"[SKIPPED_COOKIE] {reason}")
|
||||
raise AcceptanceCookieSkip(reason)
|
||||
page.goto(LOGIN_URL, wait_until="domcontentloaded", timeout=60000)
|
||||
page.wait_for_timeout(1500)
|
||||
wait_for_scan_login(page, login_timeout)
|
||||
page.goto(HOME_URL, wait_until="domcontentloaded", timeout=60000)
|
||||
page.wait_for_load_state("domcontentloaded")
|
||||
# 关闭弹窗
|
||||
for text in ("跳过", "不再提示", "我知道了"):
|
||||
try:
|
||||
b = page.get_by_text(text, exact=True).first
|
||||
if b.count():
|
||||
b.click(timeout=1500)
|
||||
page.wait_for_timeout(800)
|
||||
except Exception:
|
||||
pass
|
||||
# 登录统一由账号 vault 管理(gyxx accounts login xiaohongshu-content),
|
||||
# 采集脚本不再内嵌扫码等待;运行中失效直接快速失败。
|
||||
reason = cookie_expired_message("xiaohongshu-content", "小红书蒲公英")
|
||||
log(f"[COOKIE_EXPIRED] {reason}")
|
||||
raise CookieExpiredError(reason)
|
||||
|
||||
|
||||
def _is_login_timeout(exc: BaseException) -> bool:
|
||||
@@ -1933,7 +1875,7 @@ def scrape_styles(styles: list[dict], login_timeout: int, headless: bool,
|
||||
action,
|
||||
page_setup=restore_cookies,
|
||||
)
|
||||
except AcceptanceCookieSkip:
|
||||
except (AcceptanceCookieSkip, CookieExpiredError):
|
||||
raise
|
||||
except Exception as exc:
|
||||
session_lost = is_browser_session_lost(exc)
|
||||
@@ -2044,39 +1986,6 @@ def parse_int(s) -> int | None:
|
||||
return int(num)
|
||||
|
||||
|
||||
def login_only(login_timeout: int) -> int:
|
||||
"""只触发登录(headed),登录完成立刻退出。给 weekly_relogin.py 用。"""
|
||||
def action(page):
|
||||
ensure_login(page, login_timeout)
|
||||
return page
|
||||
|
||||
session = None
|
||||
try:
|
||||
with DynamicSession(
|
||||
headless=False,
|
||||
real_chrome=True,
|
||||
user_data_dir=str(PROFILE_DIR),
|
||||
locale="zh-CN",
|
||||
timezone_id="Asia/Shanghai",
|
||||
timeout=60000,
|
||||
network_idle=False,
|
||||
disable_resources=False,
|
||||
google_search=False,
|
||||
page_setup=restore_cookies,
|
||||
max_pages=2,
|
||||
) as session:
|
||||
session.fetch(HOME_URL, page_action=action, wait=1000)
|
||||
except Exception as exc:
|
||||
if "TargetClosedError" in type(exc).__name__ or "has been closed" in str(exc):
|
||||
log(f" [WARN] 浏览器清理时报错: {type(exc).__name__}")
|
||||
else:
|
||||
raise
|
||||
finally:
|
||||
_force_cleanup_session(session)
|
||||
log("[login-only] 登录完成,退出")
|
||||
return 0
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description="小红书蒲公英 V2: 多维表格驱动精准采集+回填")
|
||||
add_repeatable_style_argument(parser, "只跑指定款式编号(可多次)")
|
||||
@@ -2090,8 +1999,6 @@ def main() -> int:
|
||||
parser.add_argument("--skip-done", action="store_true", help="跳过已完成的款式(从进度文件读)")
|
||||
parser.add_argument("--inter-style-pause", type=int, default=5,
|
||||
help="兼容旧参数;全局按达人采集后不再按款式暂停")
|
||||
parser.add_argument("--login-only", action="store_true",
|
||||
help="只触发登录(headed),登录完成立刻退出,给 weekly_relogin 用")
|
||||
parser.add_argument("--self-operated", action="store_true",
|
||||
help="抓取自营达人表格(而非合作达人)")
|
||||
parser.add_argument("--batch-size", type=int, default=DEFAULT_BATCH_SIZE,
|
||||
@@ -2108,12 +2015,6 @@ def main() -> int:
|
||||
help=argparse.SUPPRESS)
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.login_only:
|
||||
try:
|
||||
return login_only(args.login_timeout)
|
||||
except AcceptanceCookieSkip:
|
||||
return COOKIE_SKIP_EXIT_CODE
|
||||
|
||||
mapping = load_mapping(self_operated=args.self_operated)
|
||||
styles = mapping["tables"]
|
||||
if args.style:
|
||||
@@ -2213,7 +2114,7 @@ def main() -> int:
|
||||
published_from=args.published_from,
|
||||
max_age_days=args.max_age_days, no_retry=args.no_retry,
|
||||
)
|
||||
except AcceptanceCookieSkip:
|
||||
except (AcceptanceCookieSkip, CookieExpiredError):
|
||||
return COOKIE_SKIP_EXIT_CODE
|
||||
except Exception as exc:
|
||||
if not _is_login_timeout(exc):
|
||||
|
||||
@@ -137,6 +137,7 @@ class ScraplingDriver:
|
||||
cdp_url: str | None = None,
|
||||
stealthy: bool = False,
|
||||
real_chrome: bool = True,
|
||||
extra_flags: list[str] | None = None,
|
||||
) -> ScraplingDriver:
|
||||
options: dict[str, Any] = {
|
||||
"headless": False,
|
||||
@@ -152,6 +153,7 @@ class ScraplingDriver:
|
||||
"extra_flags": [
|
||||
"--disable-blink-features=AutomationControlled",
|
||||
"--disable-dev-shm-usage",
|
||||
*(extra_flags or []),
|
||||
],
|
||||
}
|
||||
reuse_existing_context = bool(cdp_url)
|
||||
@@ -211,6 +213,9 @@ class ScraplingDriver:
|
||||
[script, raw_arguments],
|
||||
)
|
||||
|
||||
def save_screenshot(self, path: str | os.PathLike[str]) -> bool:
|
||||
return bool(self._page.screenshot(path=str(Path(path))))
|
||||
|
||||
def get_cookies(self) -> list[dict[str, Any]]:
|
||||
context = self._owner.context
|
||||
return list(context.cookies()) if context is not None else []
|
||||
@@ -224,6 +229,24 @@ class ScraplingDriver:
|
||||
def quit(self) -> None:
|
||||
self._owner.close()
|
||||
|
||||
def reopen_page(self) -> Any:
|
||||
"""Rebind to a fresh page inside the still-live context.
|
||||
|
||||
A page/tab can die (site closed it, renderer gone) while the browser
|
||||
context survives with its login state. Returns the new page, or
|
||||
raises the underlying error when the context/browser is gone too —
|
||||
the caller should rebuild the whole driver then.
|
||||
"""
|
||||
context = self._owner.context
|
||||
if context is None:
|
||||
raise RuntimeError("Scrapling browser context is closed")
|
||||
# 永远开新页: context.pages 里可能还挂着半死的目标页。
|
||||
page = context.new_page()
|
||||
self._page = page
|
||||
self._owner.page = page
|
||||
page.on("download", self._save_download)
|
||||
return page
|
||||
|
||||
def _save_download(self, download: Any) -> None:
|
||||
target = self._download_dir / download.suggested_filename
|
||||
download.save_as(str(target))
|
||||
|
||||
@@ -363,8 +363,8 @@ def log(msg: str) -> None:
|
||||
|
||||
|
||||
# ============================================================
|
||||
# B站登录态: Cookie 持久化 + 扫码登录引导
|
||||
# 空间投稿列表需要登录态,首次用 --login-only 扫码,之后调度运行复用。
|
||||
# B站登录态: Cookie 持久化;登录统一由账号 vault 管理
|
||||
# (gyxx accounts login bilibili-content),脚本内不再提供扫码登录。
|
||||
# ============================================================
|
||||
BILI_COOKIE_FILE = PATHS.browser_cookie_file
|
||||
|
||||
@@ -389,56 +389,6 @@ def save_state(page) -> None:
|
||||
log(f"已保存 Cookie ({len(cookies)} 条): {BILI_COOKIE_FILE}")
|
||||
|
||||
|
||||
def _has_login_cookies(page) -> bool:
|
||||
names = {cookie.get("name") for cookie in page.context.cookies()}
|
||||
return "SESSDATA" in names and "DedeUserID" in names
|
||||
|
||||
|
||||
def login_only(login_timeout: int = 300) -> int:
|
||||
"""打开 B站扫码登录页,等待人工扫码后保存 Cookie。"""
|
||||
from scrapling.fetchers import StealthySession
|
||||
|
||||
with StealthySession(
|
||||
headless=False,
|
||||
real_chrome=True,
|
||||
user_data_dir=str(PATHS.browser_profile_dir),
|
||||
locale="zh-CN",
|
||||
timezone_id="Asia/Shanghai",
|
||||
timeout=60_000,
|
||||
network_idle=False,
|
||||
disable_resources=False,
|
||||
google_search=False,
|
||||
max_pages=2,
|
||||
page_setup=restore_cookies,
|
||||
) as session:
|
||||
page = (
|
||||
session.context.pages[0]
|
||||
if session.context.pages
|
||||
else session.context.new_page()
|
||||
)
|
||||
page.goto("https://passport.bilibili.com/login",
|
||||
wait_until="domcontentloaded", timeout=60_000)
|
||||
page.wait_for_timeout(2000)
|
||||
if _has_login_cookies(page):
|
||||
log("✅ 已有 B站登录态,无需扫码")
|
||||
save_state(page)
|
||||
return 0
|
||||
print("\n" + "=" * 60)
|
||||
print("🔐 请在打开的浏览器中用哔哩哔哩 App 扫码登录")
|
||||
print(" 登录成功后脚本会自动继续并保存 Cookie")
|
||||
print("=" * 60, flush=True)
|
||||
deadline = time.time() + login_timeout
|
||||
while time.time() < deadline:
|
||||
if _has_login_cookies(page):
|
||||
log("✅ 检测到登录成功")
|
||||
page.wait_for_timeout(2000)
|
||||
save_state(page)
|
||||
return 0
|
||||
page.wait_for_timeout(2000)
|
||||
log(f"❌ 等待扫码超时({login_timeout}s),未保存登录态")
|
||||
return 1
|
||||
|
||||
|
||||
# ============================================================
|
||||
# 飞书多维表格
|
||||
# ============================================================
|
||||
@@ -846,17 +796,10 @@ def main() -> int:
|
||||
description="自营 B 站采集: 公开 API 全量视频指标 + UP 主主页全量投稿 + 飞书曝光回填")
|
||||
parser.add_argument("--style", type=int, help="只跑指定款式编号")
|
||||
parser.add_argument("--dry-run", action="store_true", help="不写回飞书")
|
||||
parser.add_argument("--login-only", action="store_true",
|
||||
help="只打开 B站扫码登录并保存 Cookie(首次部署引导)")
|
||||
parser.add_argument("--login-timeout", type=int, default=300,
|
||||
help="等待人工扫码的秒数")
|
||||
parser.add_argument("--delay", type=float, default=0.3,
|
||||
help="每条请求间隔秒数(防风控)")
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.login_only:
|
||||
return login_only(args.login_timeout)
|
||||
|
||||
from gyxx_flow.modules.content_marketing import feishu_mapping
|
||||
try:
|
||||
mapping = feishu_mapping.load_mapping(self_operated=True)
|
||||
|
||||
@@ -27,6 +27,8 @@ from typing import Any
|
||||
from scrapling.fetchers import DynamicSession
|
||||
|
||||
from gyxx_flow.adapters.scrapling import BrowserPage as Page
|
||||
from gyxx_flow.adapters.browser import CookieExpiredError, cookie_expired_message
|
||||
from gyxx_flow.core.exit_codes import COOKIE_SKIP_EXIT_CODE
|
||||
|
||||
from gyxx_flow.modules.content_marketing.collection_completeness import (
|
||||
BLOCKED_INPUT,
|
||||
@@ -261,17 +263,16 @@ def save_state(page: Page) -> None:
|
||||
|
||||
|
||||
def maybe_wait_for_login(page: Page, login_timeout: int) -> None:
|
||||
deadline = time.time() + login_timeout
|
||||
while time.time() < deadline:
|
||||
# 登录统一由账号 vault 管理(gyxx accounts login douyin-content),
|
||||
# 采集过程遇到登录态失效立即快速失败,不再等待人工扫码。
|
||||
try:
|
||||
body = page.locator("body").inner_text(timeout=5000)
|
||||
except Exception:
|
||||
body = ""
|
||||
if "扫码登录" not in body and "登录后" not in body:
|
||||
return
|
||||
log("抖音需要登录,请在打开的 Chrome 窗口中完成登录")
|
||||
page.wait_for_timeout(3000)
|
||||
log("[WARN] 登录等待超时")
|
||||
if "扫码登录" in body or "登录后" in body:
|
||||
reason = cookie_expired_message("douyin-content", "自营抖音")
|
||||
log(f"[COOKIE_EXPIRED] {reason}")
|
||||
raise CookieExpiredError(reason)
|
||||
|
||||
|
||||
def dismiss_popups(page: Page) -> None:
|
||||
@@ -605,33 +606,6 @@ def scrape_one_style(style: dict, login_timeout: int, headless: bool,
|
||||
return summary
|
||||
|
||||
|
||||
def login_only(login_timeout: int) -> int:
|
||||
"""只触发登录"""
|
||||
login_done = False
|
||||
|
||||
def page_action(page: Page):
|
||||
restore_cookies(page)
|
||||
page.goto("https://www.douyin.com/", wait_until="domcontentloaded", timeout=60000)
|
||||
page.wait_for_timeout(2000)
|
||||
maybe_wait_for_login(page, login_timeout)
|
||||
save_state(page)
|
||||
log("登录完成")
|
||||
|
||||
PROFILE_DIR.mkdir(parents=True, exist_ok=True)
|
||||
with DynamicSession(
|
||||
headless=False,
|
||||
real_chrome=True,
|
||||
user_data_dir=str(PROFILE_DIR),
|
||||
locale="zh-CN",
|
||||
timezone_id="Asia/Shanghai",
|
||||
timeout=60000,
|
||||
page_setup=restore_cookies,
|
||||
) as session:
|
||||
session.fetch("https://www.douyin.com/", page_action=page_action, wait=1000)
|
||||
|
||||
return 0 if login_done else 1
|
||||
|
||||
|
||||
# ============================================================
|
||||
# main
|
||||
# ============================================================
|
||||
@@ -642,12 +616,8 @@ def main() -> int:
|
||||
parser.add_argument("--headless", action="store_true")
|
||||
parser.add_argument("--login-timeout", type=int, default=300)
|
||||
parser.add_argument("--dry-run", action="store_true", help="不写回多维表格")
|
||||
parser.add_argument("--login-only", action="store_true", help="只触发登录")
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.login_only:
|
||||
return login_only(args.login_timeout)
|
||||
|
||||
mapping = load_mapping()
|
||||
styles = mapping["tables"]
|
||||
if args.style:
|
||||
@@ -703,6 +673,9 @@ def main() -> int:
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
raise SystemExit(main())
|
||||
except CookieExpiredError as exc:
|
||||
print(f"[COOKIE_EXPIRED] {exc}", file=sys.stderr)
|
||||
raise SystemExit(COOKIE_SKIP_EXIT_CODE)
|
||||
except KeyboardInterrupt:
|
||||
print("\nInterrupted.", file=sys.stderr)
|
||||
raise SystemExit(130)
|
||||
@@ -34,6 +34,7 @@ from gyxx_flow.adapters.scrapling import (
|
||||
BrowserTimeoutError, # noqa: F401
|
||||
is_browser_timeout_error,
|
||||
)
|
||||
from gyxx_flow.adapters.browser import CookieExpiredError
|
||||
from gyxx_flow.core.exit_codes import COOKIE_SKIP_EXIT_CODE
|
||||
from gyxx_flow.modules.content_marketing import self_operated_accounts
|
||||
from gyxx_flow.modules.content_marketing.runtime_paths import PATHS
|
||||
@@ -375,24 +376,14 @@ def ensure_login(page, login_timeout: int, phone: str) -> None:
|
||||
log(f"[SKIPPED_COOKIE] {reason}")
|
||||
raise AcceptanceCookieSkip(reason)
|
||||
|
||||
log("🔄 登录态失效,进入短信验证码登录流程")
|
||||
page.goto(LOGIN_URL, wait_until="domcontentloaded", timeout=60_000)
|
||||
page.wait_for_timeout(2000)
|
||||
# 确保在「短信登录」页签
|
||||
for text in ("短信登录",):
|
||||
try:
|
||||
tab = page.get_by_text(text, exact=True).first
|
||||
if tab.count() and tab.is_visible():
|
||||
tab.click(timeout=2000)
|
||||
page.wait_for_timeout(800)
|
||||
break
|
||||
except Exception:
|
||||
continue
|
||||
_fill_sms_login(page, phone)
|
||||
wait_for_login(page, login_timeout)
|
||||
save_state(page)
|
||||
page.goto(GOODS_URL, wait_until="domcontentloaded", timeout=60_000)
|
||||
page.wait_for_timeout(2000)
|
||||
# 采集流程不再内嵌短信验证码登录;统一由 --login-only 手动引导建立登录态。
|
||||
reason = (
|
||||
"小红书 ARK 后台登录态已失效:请手动执行 "
|
||||
"`uv run python src/gyxx_flow/modules/content_marketing/self_xhs_scraper.py "
|
||||
"--login-only` 完成短信验证码登录后重跑"
|
||||
)
|
||||
log(f"[COOKIE_EXPIRED] {reason}")
|
||||
raise CookieExpiredError(reason)
|
||||
|
||||
|
||||
# ============================================================
|
||||
@@ -855,6 +846,8 @@ def collect(headless: bool, login_timeout: int, max_pages: int,
|
||||
google_search=False,
|
||||
page_setup=restore_cookies,
|
||||
max_pages=3,
|
||||
# 国内站直连:代理出口被站点 WAF 拉黑时页面/API 直接 ERR_CONNECTION_CLOSED
|
||||
extra_flags=["--no-proxy-server"],
|
||||
) as session:
|
||||
page = _open_page(session)
|
||||
ensure_login(page, login_timeout, phone)
|
||||
@@ -927,7 +920,20 @@ def login_only(headless: bool, login_timeout: int) -> int:
|
||||
max_pages=2,
|
||||
) as session:
|
||||
page = _open_page(session)
|
||||
ensure_login(page, login_timeout, phone)
|
||||
page.goto(LOGIN_URL, wait_until="domcontentloaded", timeout=60_000)
|
||||
page.wait_for_timeout(2000)
|
||||
for text in ("短信登录",):
|
||||
try:
|
||||
tab = page.get_by_text(text, exact=True).first
|
||||
if tab.count() and tab.is_visible():
|
||||
tab.click(timeout=2000)
|
||||
page.wait_for_timeout(800)
|
||||
break
|
||||
except Exception:
|
||||
continue
|
||||
_fill_sms_login(page, phone)
|
||||
wait_for_login(page, login_timeout)
|
||||
save_state(page)
|
||||
log("✅ 登录态已建立并保存 Cookie")
|
||||
return 0
|
||||
|
||||
@@ -953,7 +959,7 @@ def main() -> int:
|
||||
return login_only(args.headless, args.login_timeout)
|
||||
return collect(args.headless, args.login_timeout, args.max_pages,
|
||||
args.dry_run)
|
||||
except AcceptanceCookieSkip:
|
||||
except (AcceptanceCookieSkip, CookieExpiredError):
|
||||
return COOKIE_SKIP_EXIT_CODE
|
||||
except KeyboardInterrupt:
|
||||
print("\n⛔ 用户中断", file=sys.stderr)
|
||||
|
||||
@@ -223,18 +223,15 @@ def maybe_wait_for_login(
|
||||
*,
|
||||
deadline: float | None = None,
|
||||
) -> None:
|
||||
login_deadline = time.monotonic() + max(0, login_timeout)
|
||||
while time.monotonic() < login_deadline:
|
||||
_check_note_deadline(deadline, "login wait")
|
||||
# 登录统一由账号 vault 管理(gyxx accounts login xiaohongshu-content),
|
||||
# 由工作流执行前的 cookie 网关保证;运行中失效立即降级,不再等待人工扫码。
|
||||
try:
|
||||
body = page.locator("body").inner_text(timeout=5000)
|
||||
except Exception:
|
||||
body = ""
|
||||
if "扫码登录" not in body and "登录后" not in body and "验证码" not in body:
|
||||
return
|
||||
log("Xiaohongshu may require login. Please finish login in the opened Chrome window.")
|
||||
page.wait_for_timeout(3000)
|
||||
_check_note_deadline(deadline, "login wait")
|
||||
if "扫码登录" in body or "登录后" in body or "验证码" in body:
|
||||
log("Xiaohongshu login state missing; collecting degraded samples. "
|
||||
"Run `gyxx accounts login xiaohongshu-content` to restore full comments.")
|
||||
|
||||
|
||||
def is_logged_in(page: Page) -> bool:
|
||||
@@ -751,55 +748,16 @@ def write_outputs(url: str, comments: list[dict[str, Any]], result: dict[str, An
|
||||
return json_path, csv_path
|
||||
|
||||
|
||||
def login_only(login_timeout: int) -> int:
|
||||
"""只触发登录(headed),登录完成立刻退出。给 relogin_xiaohongshu.py 用。"""
|
||||
def action(page: Page) -> None:
|
||||
page.goto("https://www.xiaohongshu.com/", wait_until="domcontentloaded", timeout=90000)
|
||||
page.wait_for_timeout(2500)
|
||||
# 主动点登录按钮触发二维码
|
||||
try:
|
||||
login_btn = page.get_by_text("登录", exact=True).first
|
||||
if login_btn.count():
|
||||
login_btn.click(timeout=3000)
|
||||
page.wait_for_timeout(1500)
|
||||
except Exception:
|
||||
pass
|
||||
maybe_wait_for_login(page, login_timeout)
|
||||
save_state(page)
|
||||
|
||||
PROFILE_DIR.mkdir(parents=True, exist_ok=True)
|
||||
with ScraplingBrowser(
|
||||
headless=False,
|
||||
real_chrome=True,
|
||||
user_data_dir=str(PROFILE_DIR),
|
||||
locale="zh-CN",
|
||||
timezone_id="Asia/Shanghai",
|
||||
timeout=90000,
|
||||
network_idle=False,
|
||||
disable_resources=False,
|
||||
google_search=False,
|
||||
page_setup=restore_cookies,
|
||||
max_pages=1,
|
||||
) as browser:
|
||||
browser.run_fetch_action("https://www.xiaohongshu.com/", action, wait=1000)
|
||||
log("[login-only] 完成")
|
||||
return 0
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description="Scrape Xiaohongshu note comments with a real browser.")
|
||||
parser.add_argument("url", help="Xiaohongshu explore/note URL (传 'login' 配合 --login-only)")
|
||||
parser.add_argument("--login-timeout", type=int, default=300, help="等待扫码/登录秒数")
|
||||
parser.add_argument("url", help="Xiaohongshu explore/note URL")
|
||||
parser.add_argument("--login-timeout", type=int, default=300,
|
||||
help="保留参数(登录等待已移除;未登录将直接降级采集)")
|
||||
parser.add_argument("--max-scrolls", type=int, default=120, help="最多滚动加载次数")
|
||||
parser.add_argument("--idle-rounds", type=int, default=8, help="连续几轮无新评论后停止")
|
||||
parser.add_argument("--headless", action="store_true", help="无头模式;首次登录不要使用")
|
||||
parser.add_argument("--login-only", action="store_true",
|
||||
help="只触发登录(headed)然后退出,给 relogin_xiaohongshu 用")
|
||||
parser.add_argument("--headless", action="store_true", help="无头模式")
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.login_only:
|
||||
return login_only(args.login_timeout)
|
||||
|
||||
comments, result = scrape_comments(args.url, args.login_timeout, args.max_scrolls, args.idle_rounds, args.headless)
|
||||
json_path, csv_path = write_outputs(args.url, comments, result)
|
||||
log(f"Final URL: {result.get('final_url', '')}")
|
||||
|
||||
@@ -33,6 +33,7 @@ from gyxx_flow.adapters.scrapling import (
|
||||
BrowserTimeoutError,
|
||||
is_browser_timeout_error,
|
||||
)
|
||||
from gyxx_flow.adapters.browser import CookieExpiredError, cookie_expired_message
|
||||
from gyxx_flow.core.exit_codes import COOKIE_SKIP_EXIT_CODE
|
||||
from gyxx_flow.modules.content_marketing.collection_completeness import (
|
||||
BLOCKED_INPUT,
|
||||
@@ -537,183 +538,6 @@ def is_logged_in(page: Page) -> bool:
|
||||
return False
|
||||
|
||||
|
||||
def open_scan_login(page: Page) -> None:
|
||||
"""Open the selected platform's login dialog without handling passwords."""
|
||||
if CONTENT_SOURCE == "buyin":
|
||||
if not is_business_page(page) or "/dashboard/servicehall/daren-square" not in page.url:
|
||||
try:
|
||||
page.goto(CUSTOMER_LOGIN_URL, wait_until="domcontentloaded", timeout=60000)
|
||||
page.wait_for_timeout(1500)
|
||||
except Exception:
|
||||
pass
|
||||
for pattern in (r"^登录$", r"扫码登录|二维码登录|抖音登录|使用抖音登录"):
|
||||
try:
|
||||
button = page.get_by_text(re.compile(pattern)).first
|
||||
if button.count() and button.is_visible():
|
||||
button.click(timeout=3000)
|
||||
page.wait_for_timeout(800)
|
||||
return
|
||||
except Exception as exc:
|
||||
reraise_if_browser_session_lost(exc)
|
||||
return
|
||||
|
||||
if "sso.oceanengine.com/xingtu/login" not in page.url:
|
||||
try:
|
||||
page.goto(CUSTOMER_LOGIN_URL, wait_until="domcontentloaded", timeout=60000)
|
||||
page.wait_for_timeout(1500)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# The Douyin login icon on Xingtu's customer SSO page has a stable
|
||||
# aweme.png source. Clicking it opens the QR authorization page directly.
|
||||
try:
|
||||
douyin = page.locator('img[src*="aweme.png"]').first
|
||||
if douyin.count() and douyin.is_visible():
|
||||
douyin.click(timeout=3000)
|
||||
page.wait_for_timeout(1500)
|
||||
return
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
for pattern in (r"^登录$|立即登录", r"抖音扫码登录|扫码登录|二维码登录|扫码"):
|
||||
try:
|
||||
button = page.get_by_text(re.compile(pattern)).first
|
||||
if button.count() and button.is_visible():
|
||||
button.click(timeout=2000)
|
||||
page.wait_for_timeout(500)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def is_scan_qr_page(page: Page) -> bool:
|
||||
"""Return True once Douyin's QR authorization page is visible.
|
||||
|
||||
The QR page lives outside xingtu.cn. Treating that URL as a page that still
|
||||
needs opening causes the login helper to navigate back and click again every
|
||||
few seconds, continuously refreshing the QR code.
|
||||
"""
|
||||
url = page.url.lower()
|
||||
if any(host in url for host in ("open.douyin.com", "sso.douyin.com")):
|
||||
return True
|
||||
for selector in (
|
||||
'img[src*="qrcode"]',
|
||||
'img[src*="qr_code"]',
|
||||
'canvas',
|
||||
):
|
||||
try:
|
||||
locator = page.locator(selector).first
|
||||
if locator.count() and locator.is_visible():
|
||||
return True
|
||||
except Exception:
|
||||
pass
|
||||
return False
|
||||
|
||||
|
||||
def resolve_live_page(page: Page, *, create_if_missing: bool = False) -> Page | None:
|
||||
"""Return a usable page after OAuth closes or replaces its QR tab."""
|
||||
try:
|
||||
if not page.is_closed():
|
||||
return page
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
try:
|
||||
context = page.context
|
||||
for candidate in reversed(list(context.pages)):
|
||||
try:
|
||||
if not candidate.is_closed():
|
||||
return candidate
|
||||
except Exception:
|
||||
continue
|
||||
if create_if_missing:
|
||||
return context.new_page()
|
||||
except Exception:
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
def find_logged_in_page(page: Page) -> Page | None:
|
||||
"""Find an authenticated business tab even while the OAuth tab stays open."""
|
||||
candidates: list[Page] = []
|
||||
try:
|
||||
candidates.extend(reversed(list(page.context.pages)))
|
||||
except Exception:
|
||||
pass
|
||||
candidates.append(page)
|
||||
|
||||
seen: set[int] = set()
|
||||
for candidate in candidates:
|
||||
marker = id(candidate)
|
||||
if marker in seen:
|
||||
continue
|
||||
seen.add(marker)
|
||||
try:
|
||||
if candidate.is_closed() or not is_business_page(candidate):
|
||||
continue
|
||||
if is_logged_in(candidate):
|
||||
return candidate
|
||||
except Exception:
|
||||
continue
|
||||
return None
|
||||
|
||||
|
||||
def wait_for_scan_login(page: Page, timeout_seconds: int) -> Page:
|
||||
log("Not logged in. 请在打开的 Chrome 中扫码登录。")
|
||||
current_page = page
|
||||
open_scan_login(current_page)
|
||||
deadline = time.time() + timeout_seconds
|
||||
next_login_click = time.time() + 5
|
||||
while time.time() < deadline:
|
||||
authenticated = find_logged_in_page(current_page)
|
||||
if authenticated is not None:
|
||||
save_state(authenticated)
|
||||
return authenticated
|
||||
|
||||
replacement = resolve_live_page(current_page, create_if_missing=True)
|
||||
if replacement is None:
|
||||
raise RuntimeError("Xingtu browser context closed during OAuth login")
|
||||
if replacement is not current_page:
|
||||
current_page = replacement
|
||||
try:
|
||||
current_page.goto(
|
||||
MARKET_URL, wait_until="domcontentloaded", timeout=60000
|
||||
)
|
||||
current_page.wait_for_timeout(2000)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
if (
|
||||
CONTENT_SOURCE == "xingtu"
|
||||
and "redirect_uri=/ad/creator/market" in current_page.url
|
||||
and not is_scan_qr_page(current_page)
|
||||
):
|
||||
try:
|
||||
current_page.goto(
|
||||
MARKET_URL, wait_until="domcontentloaded", timeout=60000
|
||||
)
|
||||
current_page.wait_for_timeout(2000)
|
||||
except Exception:
|
||||
pass
|
||||
authenticated = find_logged_in_page(current_page)
|
||||
if authenticated is not None:
|
||||
save_state(authenticated)
|
||||
return authenticated
|
||||
if (
|
||||
time.time() >= next_login_click
|
||||
and not is_scan_qr_page(current_page)
|
||||
and not is_business_page(current_page)
|
||||
):
|
||||
open_scan_login(current_page)
|
||||
next_login_click = time.time() + 5
|
||||
try:
|
||||
current_page.wait_for_timeout(2000)
|
||||
except Exception:
|
||||
# OAuth may close the QR tab after authorization. The next loop
|
||||
# follows the surviving tab in the same browser context.
|
||||
continue
|
||||
raise TimeoutError(f"Login timed out after {timeout_seconds} seconds.")
|
||||
|
||||
|
||||
def ensure_login(page: Page, login_timeout: int) -> Page:
|
||||
page.goto(MARKET_URL, wait_until="domcontentloaded", timeout=60000)
|
||||
page.wait_for_timeout(3000)
|
||||
@@ -735,12 +559,11 @@ def ensure_login(page: Page, login_timeout: int) -> Page:
|
||||
)
|
||||
log(f"[SKIPPED_COOKIE] {reason}")
|
||||
raise AcceptanceCookieSkip(reason)
|
||||
page = wait_for_scan_login(page, login_timeout)
|
||||
page.goto(MARKET_URL, wait_until="domcontentloaded", timeout=60000)
|
||||
page.wait_for_timeout(3000)
|
||||
if not is_logged_in(page):
|
||||
page = wait_for_scan_login(page, login_timeout)
|
||||
return page
|
||||
# 登录统一由账号 vault 管理(gyxx accounts login douyin-shop),
|
||||
# 采集脚本不再内嵌扫码等待;运行中失效直接快速失败。
|
||||
reason = cookie_expired_message("douyin-shop", f"{CONTENT_SOURCE}达人广场")
|
||||
log(f"[COOKIE_EXPIRED] {reason}")
|
||||
raise CookieExpiredError(reason)
|
||||
|
||||
|
||||
def dismiss_popups(page: Page) -> None:
|
||||
@@ -3736,7 +3559,7 @@ def scrape_styles_with_fallback(
|
||||
summary_override=summaries,
|
||||
replace_existing=True,
|
||||
)
|
||||
except AcceptanceCookieSkip as exc:
|
||||
except (AcceptanceCookieSkip, CookieExpiredError) as exc:
|
||||
log(f" [WARN] 抖店补采登录态不可用,保留星图结果: {exc}")
|
||||
_mark_fallback_not_processed(
|
||||
fallback_tasks, summaries, "buyin_cookie_skipped"
|
||||
@@ -3805,96 +3628,6 @@ def _force_cleanup_session(session) -> None:
|
||||
pass
|
||||
|
||||
|
||||
def login_only(login_timeout: int) -> int:
|
||||
"""只触发登录(headed),登录完成立刻退出。给 weekly_relogin.py 用。"""
|
||||
login_completed = False
|
||||
|
||||
def action(page):
|
||||
nonlocal login_completed
|
||||
page = ensure_login(page, login_timeout)
|
||||
if not is_logged_in(page):
|
||||
raise RuntimeError(f"{CONTENT_SOURCE} 登录状态验证失败")
|
||||
save_state(page)
|
||||
login_completed = True
|
||||
return page
|
||||
|
||||
session = None
|
||||
try:
|
||||
with DynamicSession(
|
||||
headless=False,
|
||||
real_chrome=True,
|
||||
user_data_dir=str(PROFILE_DIR),
|
||||
locale="zh-CN",
|
||||
timezone_id="Asia/Shanghai",
|
||||
timeout=60000,
|
||||
network_idle=False,
|
||||
disable_resources=False,
|
||||
google_search=False,
|
||||
page_setup=restore_cookies,
|
||||
max_pages=2,
|
||||
) as session:
|
||||
session.fetch(MARKET_URL, page_action=action, wait=1000)
|
||||
except Exception as exc:
|
||||
err_msg = str(exc).lower()
|
||||
if "closed" in err_msg or "context" in err_msg or "target" in err_msg:
|
||||
log(f"[login-only] 浏览器会话关闭异常: {exc}")
|
||||
else:
|
||||
log(f"[login-only] 登录执行失败: {exc}")
|
||||
finally:
|
||||
_force_cleanup_session(session)
|
||||
|
||||
# Douyin OAuth may close the only QR tab (and therefore the temporary
|
||||
# browser context) after a successful mobile confirmation. The durable
|
||||
# Chromium profile can already contain the new session even though the
|
||||
# page_action never got a chance to export cookies. Reopen the same profile
|
||||
# once, verify the selected business page, then export its cookies.
|
||||
if not login_completed or not has_valid_login_cookie_file():
|
||||
log(f"[login-only] 扫码窗口已结束,重新打开{CONTENT_SOURCE}验证持久会话")
|
||||
login_completed = verify_persisted_profile_login()
|
||||
if not login_completed or not has_valid_login_cookie_file():
|
||||
log("[login-only] 登录未完成或Cookie缺少有效会话")
|
||||
return 1
|
||||
log("[login-only] 登录完成,退出")
|
||||
return 0
|
||||
|
||||
|
||||
def verify_persisted_profile_login() -> bool:
|
||||
"""Reopen the persistent profile after OAuth closes its only tab."""
|
||||
verified = False
|
||||
|
||||
def action(page: Page):
|
||||
nonlocal verified
|
||||
page.goto(MARKET_URL, wait_until="domcontentloaded", timeout=60000)
|
||||
page.wait_for_timeout(3000)
|
||||
if not is_logged_in(page):
|
||||
raise RuntimeError(f"persisted {CONTENT_SOURCE} profile is not authenticated")
|
||||
save_state(page)
|
||||
verified = True
|
||||
return page
|
||||
|
||||
session = None
|
||||
try:
|
||||
with DynamicSession(
|
||||
headless=False,
|
||||
real_chrome=True,
|
||||
user_data_dir=str(PROFILE_DIR),
|
||||
locale="zh-CN",
|
||||
timezone_id="Asia/Shanghai",
|
||||
timeout=60000,
|
||||
network_idle=False,
|
||||
disable_resources=False,
|
||||
google_search=False,
|
||||
page_setup=restore_cookies,
|
||||
max_pages=1,
|
||||
) as session:
|
||||
session.fetch(MARKET_URL, page_action=action, wait=1000)
|
||||
except Exception as exc:
|
||||
log(f"[login-only] 持久会话验证失败: {exc}")
|
||||
finally:
|
||||
_force_cleanup_session(session)
|
||||
return verified and has_valid_login_cookie_file()
|
||||
|
||||
|
||||
def load_retry_scope(summary_path: Path, reason: str) -> dict[int, set[str]]:
|
||||
"""Load exact style/record pairs for a reason-limited retry."""
|
||||
try:
|
||||
@@ -3926,8 +3659,6 @@ def main() -> int:
|
||||
parser.add_argument("--login-timeout", type=int, default=300)
|
||||
parser.add_argument("--dry-run", action="store_true")
|
||||
parser.add_argument("--data-dir", default=str(PATHS.normalized_root))
|
||||
parser.add_argument("--login-only", action="store_true",
|
||||
help="只触发登录(headed),登录完成立刻退出,给 weekly_relogin 用")
|
||||
parser.add_argument(
|
||||
"--buyin",
|
||||
action="store_true",
|
||||
@@ -3952,12 +3683,6 @@ def main() -> int:
|
||||
"buyin" if args.buyin else os.environ.get("GYXX_CONTENT_SOURCE", "xingtu")
|
||||
)
|
||||
|
||||
if args.login_only:
|
||||
try:
|
||||
return login_only(args.login_timeout)
|
||||
except AcceptanceCookieSkip:
|
||||
return COOKIE_SKIP_EXIT_CODE
|
||||
|
||||
mapping = load_mapping(self_operated=args.self_operated)
|
||||
styles = mapping["tables"]
|
||||
if args.style:
|
||||
@@ -4038,7 +3763,7 @@ def main() -> int:
|
||||
max_age_days=args.max_age_days,
|
||||
no_retry=args.no_retry,
|
||||
)
|
||||
except AcceptanceCookieSkip:
|
||||
except (AcceptanceCookieSkip, CookieExpiredError):
|
||||
return COOKIE_SKIP_EXIT_CODE
|
||||
persist_summaries(all_summaries)
|
||||
for s, summary in zip(styles, all_summaries):
|
||||
|
||||
@@ -20,6 +20,7 @@ PRODUCT_SCHEDULED_WORKFLOW_IDS = (
|
||||
"product.main_image.weekly",
|
||||
"product.sales_sheet.daily",
|
||||
"product.erp_all_shop_daily",
|
||||
"product.hk_anpei_public_inventory.daily",
|
||||
"product.market_rank",
|
||||
"product.tmall_baibu_apply",
|
||||
"product.video_upload",
|
||||
@@ -45,6 +46,7 @@ _WORKFLOW_SCOPED_RESOURCE_IDS = {
|
||||
"product.daily",
|
||||
"product.persona.daily",
|
||||
"product.style_analysis.interval",
|
||||
"product.hk_anpei_public_inventory.daily",
|
||||
"product.ecommerce_costs.daily",
|
||||
# Tmall and JD video publication use separate browser profiles, accounts,
|
||||
# and channel-scoped PostgreSQL ledgers; do not let a long publication on
|
||||
|
||||
@@ -205,6 +205,44 @@ def set_all_shop_daily_filters(
|
||||
)
|
||||
|
||||
|
||||
ENSURE_CUSTOM_COLUMNS_JS = r"""
|
||||
() => {
|
||||
const cols = window._Cols || [];
|
||||
let changed = 0;
|
||||
for (const col of cols) {
|
||||
const label = String(col.text || col.defaultText || '');
|
||||
if ((label === '自定义销量' || label === '自定义退货量') && col.visible === false) {
|
||||
col.visible = true;
|
||||
changed += 1;
|
||||
}
|
||||
}
|
||||
if (changed > 0 && typeof window._RptRefresh === 'function') {
|
||||
try { window._RptRefresh(); } catch (e) {}
|
||||
}
|
||||
return changed;
|
||||
}
|
||||
"""
|
||||
|
||||
|
||||
def ensure_custom_columns_visible(page: Any) -> int:
|
||||
"""Keep 自定义销量/自定义退货量 visible before the report renders.
|
||||
|
||||
2026-10-11 起聚水潭报表模板把自定义公式列重置为隐藏,隐藏列不渲染
|
||||
数据单元格;只改 visible 不会自动重绘,需顺带触发一次 _RptRefresh
|
||||
(其尾部浮头重建可能抛错,但不影响数据行渲染)。
|
||||
"""
|
||||
|
||||
changed_total = 0
|
||||
for frame in page.frames:
|
||||
if "allsku.aspx" not in (frame.url or "").lower():
|
||||
continue
|
||||
try:
|
||||
changed_total += int(frame.evaluate(ENSURE_CUSTOM_COLUMNS_JS))
|
||||
except Exception:
|
||||
continue
|
||||
return changed_total
|
||||
|
||||
|
||||
def extract_style_rows(
|
||||
table_frame: Any,
|
||||
requested_codes: Sequence[str],
|
||||
@@ -257,18 +295,45 @@ def extract_style_rows(
|
||||
};
|
||||
}
|
||||
|
||||
// 隐藏列的 th 仍在表头,但数据行不渲染对应 td:数据行的第 j 格
|
||||
// 对应第 j 个可见列。用 window._Cols 把表头列索引换算成行内位
|
||||
// 置,避免隐藏列造成索引错位;拿不到 _Cols 时按原索引直取。
|
||||
let stylePos = styleCol;
|
||||
let productPos = productCol;
|
||||
let namePos = productNameCol;
|
||||
let salesPos = salesCol;
|
||||
let returnsPos = returnsCol;
|
||||
if (Array.isArray(window._Cols) && window._Cols.length === headerTexts.length) {
|
||||
const visibleIdx = [];
|
||||
for (let i = 0; i < window._Cols.length; i++) {
|
||||
if (window._Cols[i].visible !== false) visibleIdx.push(i);
|
||||
}
|
||||
stylePos = visibleIdx.indexOf(styleCol);
|
||||
productPos = productCol >= 0 ? visibleIdx.indexOf(productCol) : -1;
|
||||
namePos = productNameCol >= 0 ? visibleIdx.indexOf(productNameCol) : -1;
|
||||
salesPos = visibleIdx.indexOf(salesCol);
|
||||
returnsPos = visibleIdx.indexOf(returnsCol);
|
||||
}
|
||||
if (stylePos < 0 || salesPos < 0 || returnsPos < 0) {
|
||||
return {
|
||||
ok: false,
|
||||
error: `custom sales/return columns hidden: style=${stylePos}, sales=${salesPos}, returns=${returnsPos}`,
|
||||
header_texts: headerTexts,
|
||||
};
|
||||
}
|
||||
|
||||
const rows = [];
|
||||
for (const tr of Array.from(document.querySelectorAll('tr'))) {
|
||||
const cells = Array.from(tr.querySelectorAll('td, th'));
|
||||
if (cells.length <= Math.max(styleCol, salesCol, returnsCol)) continue;
|
||||
const styleCode = norm(cells[styleCol]?.textContent);
|
||||
if (cells.length <= Math.max(stylePos, salesPos, returnsPos)) continue;
|
||||
const styleCode = norm(cells[stylePos]?.textContent);
|
||||
if (!requested.has(styleCode)) continue;
|
||||
rows.push({
|
||||
erp_style_code: styleCode,
|
||||
product_code: productCol >= 0 ? norm(cells[productCol]?.textContent) : '',
|
||||
product_name: productNameCol >= 0 ? norm(cells[productNameCol]?.textContent) : '',
|
||||
sales: parseInteger(cells[salesCol]?.textContent),
|
||||
return_qty: parseInteger(cells[returnsCol]?.textContent),
|
||||
product_code: productPos >= 0 ? norm(cells[productPos]?.textContent) : '',
|
||||
product_name: namePos >= 0 ? norm(cells[namePos]?.textContent) : '',
|
||||
sales: parseInteger(cells[salesPos]?.textContent),
|
||||
return_qty: parseInteger(cells[returnsPos]?.textContent),
|
||||
});
|
||||
}
|
||||
const foundCodes = Array.from(new Set(rows.map(row => row.erp_style_code))).sort();
|
||||
@@ -326,6 +391,7 @@ def wait_for_style_rows(
|
||||
reload_deadline = time.monotonic() + reload_after_seconds
|
||||
continue
|
||||
|
||||
ensure_custom_columns_visible(page)
|
||||
for frame in page.frames:
|
||||
if "allsku.aspx" not in (frame.url or "").lower():
|
||||
continue
|
||||
@@ -459,6 +525,7 @@ def collect_one_date(
|
||||
selection = set_all_shop_daily_filters(
|
||||
filter_frame, report_date=report_date, erp_codes=codes
|
||||
)
|
||||
ensure_custom_columns_visible(page)
|
||||
erp.generate_report(filter_frame)
|
||||
report = wait_for_style_rows(
|
||||
page,
|
||||
|
||||
@@ -47,6 +47,10 @@ TASK_CENTER_URL = (
|
||||
"https://qianchuan.jinritemai.com/tools/task-center/download-task"
|
||||
"?aavid={advertiser_id}"
|
||||
)
|
||||
TASK_CENTER_PAGE_URL = TASK_CENTER_URL.format(advertiser_id=quote(ADVERTISER_ID))
|
||||
# 任务中心深链缺账户上下文时,千川会把页面重定向到登录域的账户选择页;
|
||||
# 该页每个账户是一张 .account-list-item 卡片,user-id 含唯一广告主 ID。
|
||||
ACCOUNT_ITEM_SELECTOR = ".account-list-item"
|
||||
TASK_SCENE_MARKER = "全域推广数据-推商品"
|
||||
TASK_READY_MARKER = "下载任务列表"
|
||||
TASK_COMPLETE_MARKER = "生成完成"
|
||||
@@ -188,6 +192,70 @@ def _validate_task_center_page(page) -> None:
|
||||
raise DouyinQianchuanDownloadError("巨量千川下载任务中心未达到可用状态")
|
||||
|
||||
|
||||
def _on_account_selection_page(page) -> bool:
|
||||
try:
|
||||
return page.locator(ACCOUNT_ITEM_SELECTOR).count() > 0
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
def _select_advertiser_account(page) -> None:
|
||||
if not ADVERTISER_ID:
|
||||
raise DouyinQianchuanDownloadError(
|
||||
"巨量千川任务中心跳转到账户选择页,且未配置目标广告主 ID"
|
||||
)
|
||||
account = page.locator(ACCOUNT_ITEM_SELECTOR).filter(has_text=ADVERTISER_ID)
|
||||
if account.count() == 0:
|
||||
raise DouyinQianchuanDownloadError(
|
||||
f"巨量千川账户选择页上未找到目标广告主:{ADVERTISER_ID}"
|
||||
)
|
||||
account.first.click()
|
||||
|
||||
|
||||
def _wait_task_center_ready(page, *, timeout_ms: int = 120_000) -> None:
|
||||
"""等待任务中心可用;被重定向到账户选择页时点击目标广告主后继续等待。"""
|
||||
|
||||
deadline = time.monotonic() + timeout_ms / 1000
|
||||
selections = 0
|
||||
regotos = 0
|
||||
while True:
|
||||
if _on_account_selection_page(page):
|
||||
if selections >= 2:
|
||||
raise DouyinQianchuanDownloadError(
|
||||
"巨量千川选择目标广告主后仍停留在账户选择页"
|
||||
)
|
||||
selections += 1
|
||||
_select_advertiser_account(page)
|
||||
page.wait_for_timeout(3_000)
|
||||
continue
|
||||
remaining_ms = int((deadline - time.monotonic()) * 1000)
|
||||
if remaining_ms <= 0:
|
||||
raise DouyinQianchuanDownloadError(
|
||||
"巨量千川下载任务中心在等待期间未达到可用状态"
|
||||
)
|
||||
try:
|
||||
page.wait_for_function(
|
||||
"(marker) => (document.body?.innerText || '').includes(marker)",
|
||||
arg=TASK_READY_MARKER,
|
||||
timeout=min(15_000, remaining_ms),
|
||||
)
|
||||
return
|
||||
except Exception as exc:
|
||||
message = str(exc).casefold()
|
||||
if (
|
||||
not is_browser_timeout_error(exc)
|
||||
and "execution context was destroyed" not in message
|
||||
):
|
||||
raise
|
||||
# 等待落空且已不在账户选择页:账户选择后若未自动回到任务中心,
|
||||
# 重新打开一次深链;后续轮次仅继续等待直至超时。
|
||||
if regotos < 1:
|
||||
regotos += 1
|
||||
page.goto(
|
||||
TASK_CENTER_PAGE_URL, wait_until="domcontentloaded", timeout=90_000
|
||||
)
|
||||
|
||||
|
||||
def _download_task_from_page(
|
||||
page,
|
||||
start_date: date,
|
||||
@@ -206,15 +274,11 @@ def _download_task_from_page(
|
||||
except Exception as exc:
|
||||
if (
|
||||
"execution context was destroyed" not in str(exc).casefold()
|
||||
or navigation_retries >= 1
|
||||
or navigation_retries >= 2
|
||||
):
|
||||
raise
|
||||
navigation_retries += 1
|
||||
page.wait_for_function(
|
||||
"(marker) => (document.body?.innerText || '').includes(marker)",
|
||||
arg=TASK_READY_MARKER,
|
||||
timeout=15_000,
|
||||
)
|
||||
_wait_task_center_ready(page)
|
||||
_validate_task_center_page(page)
|
||||
continue
|
||||
if has_rows:
|
||||
@@ -248,15 +312,11 @@ def _download_from_task_center(
|
||||
task_page = context.new_page()
|
||||
try:
|
||||
task_page.goto(
|
||||
TASK_CENTER_URL.format(advertiser_id=quote(ADVERTISER_ID)),
|
||||
TASK_CENTER_PAGE_URL,
|
||||
wait_until="domcontentloaded",
|
||||
timeout=90_000,
|
||||
)
|
||||
task_page.wait_for_function(
|
||||
"(marker) => (document.body?.innerText || '').includes(marker)",
|
||||
arg=TASK_READY_MARKER,
|
||||
timeout=90_000,
|
||||
)
|
||||
_wait_task_center_ready(task_page)
|
||||
_validate_task_center_page(task_page)
|
||||
return _download_task_from_page(
|
||||
task_page,
|
||||
|
||||
@@ -0,0 +1,814 @@
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
"""聚水潭「报表→销售主题分析→按日统计」Excel 导出采集入库。
|
||||
|
||||
流程(2026-10-12 实测探测):
|
||||
1. 复用 erp-jushuitan 账号登录态(CDP/profile + vault cookie 注入)直开
|
||||
https://bi.erp321.com/app/daas/report/subject/adsorder/default.aspx
|
||||
2. filter.aspx: 发货时间自定义区间 + 勾选固定品牌 + 勾选固定店铺清单
|
||||
3. 生成报表(BtnFullSearch) → 点「按日统计」tab → day.aspx 出数
|
||||
4. 点「导出EXCEL」→ Playwright download 事件落盘 xlsx
|
||||
5. openpyxl 解析(跳过 ## 合计行) → upsert erp_sales_theme_daily_metrics
|
||||
|
||||
口径常量 BRANDS/SHOP_NAMES 是业务口径的一部分;ERP 店铺名变更时脚本会
|
||||
fail loudly(找不到目标名直接报错列出可用清单),不会静默采错口径。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from datetime import date, datetime, timedelta
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
try:
|
||||
from .collect_erp_yesterday_metrics import resolve_browser_session
|
||||
from .collect_retry_utils import retry_step
|
||||
from .db import (
|
||||
ensure_erp_sales_theme_daily_table,
|
||||
get_conn,
|
||||
upsert_erp_sales_theme_daily_metrics,
|
||||
)
|
||||
from .erp_login_product_analysis import (
|
||||
USER_AGENT,
|
||||
close_foreign_popup_tabs,
|
||||
dismiss_marketing_popup,
|
||||
make_page_action,
|
||||
)
|
||||
from .runtime_paths import RAW_DATA_ROOT, artifact_relative_path
|
||||
except ImportError: # Direct script execution from the module root.
|
||||
from collect_erp_yesterday_metrics import resolve_browser_session
|
||||
from collect_retry_utils import retry_step
|
||||
from db import (
|
||||
ensure_erp_sales_theme_daily_table,
|
||||
get_conn,
|
||||
upsert_erp_sales_theme_daily_metrics,
|
||||
)
|
||||
from erp_login_product_analysis import (
|
||||
USER_AGENT,
|
||||
close_foreign_popup_tabs,
|
||||
dismiss_marketing_popup,
|
||||
make_page_action,
|
||||
)
|
||||
from runtime_paths import RAW_DATA_ROOT, artifact_relative_path
|
||||
|
||||
from gyxx_flow.core.exit_codes import COOKIE_SKIP_EXIT_CODE
|
||||
|
||||
REPORT_URL = "https://bi.erp321.com/app/daas/report/subject/adsorder/default.aspx"
|
||||
LOGIN_URL_DEFAULT = "https://www.erp321.com/login.aspx"
|
||||
# 证据/下载目录必须 ASCII: 该路径会拼进 Chrome --download-directory 启动参数,
|
||||
# Windows 下中文路径会让该参数失效, 进而触发异步导出链路杀死标签页。
|
||||
EVIDENCE_DIR_NAME = "erp_sales_theme_daily"
|
||||
|
||||
# ── 业务口径常量(与页面筛选一致) ─────────────────────────────────────────────
|
||||
# 商品品牌: 光影行星 / 光影 / gyxx (2026-10-12 截图勾选项)
|
||||
BRANDS: tuple[str, ...] = ("光影行星", "光影", "gyxx")
|
||||
|
||||
# 销售渠道(店铺): 光影系 22 家渠道(2026-10-12 用户确认补回 JD光影行星旗舰店
|
||||
# 与 得物(直发)-光影行星)。名称必须与 filter.aspx 复选框标签逐字一致;
|
||||
# ERP 店铺名变更时脚本会 fail loudly 并列出可用清单。
|
||||
SHOP_NAMES: tuple[str, ...] = (
|
||||
"JD光影行星旗舰店",
|
||||
"得物(直发)-光影行星",
|
||||
"得物-光影行星",
|
||||
"抖店-光影行星数码旗舰店",
|
||||
"抖店-光影行星箱包旗舰店",
|
||||
"光影华贸自营店",
|
||||
"光影行星独立站",
|
||||
"光影行星-快手",
|
||||
"光影行星-视频号",
|
||||
"光影行星-微信商城",
|
||||
"光影行星唯品会",
|
||||
"海外仓",
|
||||
"京东商城-京东自营入仓专用",
|
||||
"拼多多-光影行星旗舰店",
|
||||
"拼多多-光影行星箱包旗舰店",
|
||||
"拼多多-光影行星专卖店",
|
||||
"实体店分销",
|
||||
"天猫-GYXX箱包旗舰店",
|
||||
"天猫-光影行星旗舰店",
|
||||
"天猫-光影行星箱包旗舰店",
|
||||
"香港安沛",
|
||||
"小红书-光影行星旗舰店",
|
||||
)
|
||||
|
||||
# 导出 Excel 的 17 列表头(2026-10-12 实测) → 记录字段
|
||||
COLUMN_MAP: dict[str, str] = {
|
||||
"日期": "stat_date",
|
||||
"销售单数": "sales_order_count",
|
||||
"销售数量": "sales_qty",
|
||||
"实发数量": "shipped_qty",
|
||||
"实发金额": "shipped_amount",
|
||||
"销售金额": "sales_amount",
|
||||
"退货单数": "return_order_count",
|
||||
"退货数量": "return_qty",
|
||||
"实退数量": "returned_qty",
|
||||
"退货金额": "return_amount",
|
||||
"实退金额": "returned_amount",
|
||||
"净销售单数": "net_sales_order_count",
|
||||
"净销量": "net_sales_qty",
|
||||
"净销售额": "net_sales_amount",
|
||||
"运费收入": "freight_income",
|
||||
"运费支出": "freight_expense",
|
||||
"实发单数": "shipped_order_count",
|
||||
}
|
||||
TOTAL_ROW_MARKERS = {"#", "##", "总计", "合计"}
|
||||
_DATE_RE = re.compile(r"^\d{4}-\d{2}-\d{2}$")
|
||||
|
||||
|
||||
# ── 纯解析层(无浏览器依赖, 便于测试) ────────────────────────────────────────
|
||||
|
||||
def parse_number(value: Any) -> int:
|
||||
text = str(value if value is not None else "").replace(",", "").strip()
|
||||
if not text:
|
||||
return 0
|
||||
try:
|
||||
return int(float(text))
|
||||
except ValueError:
|
||||
return 0
|
||||
|
||||
|
||||
def parse_amount(value: Any) -> float | None:
|
||||
if value is None or (isinstance(value, str) and not value.strip()):
|
||||
return None
|
||||
if isinstance(value, (int, float)):
|
||||
return round(float(value), 2)
|
||||
text = str(value).replace(",", "").strip()
|
||||
if not text:
|
||||
return None
|
||||
try:
|
||||
return round(float(text), 2)
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
def parse_stat_date(value: Any) -> str:
|
||||
if isinstance(value, datetime):
|
||||
return value.date().isoformat()
|
||||
text = str(value if value is not None else "").strip()
|
||||
if _DATE_RE.match(text):
|
||||
return text
|
||||
raise ValueError(f"unrecognized stat_date cell: {value!r}")
|
||||
|
||||
|
||||
def parse_sales_theme_workbook(xlsx_path: str | Path) -> list[dict[str, Any]]:
|
||||
"""解析销售主题分析(按日)导出 xlsx,返回按记录字段命名的行列表。
|
||||
|
||||
跳过末尾 ## 合计行;列按表头名映射(顺序无关),缺列按 0/None 补齐。
|
||||
"""
|
||||
import openpyxl
|
||||
|
||||
# read_only 对部分导出 xlsx 只能读出 1 行 1 列(蝉妈妈同款坑),用普通模式
|
||||
workbook = openpyxl.load_workbook(str(xlsx_path), data_only=True)
|
||||
try:
|
||||
sheet = workbook.active
|
||||
rows = sheet.iter_rows(values_only=True)
|
||||
header_row = next(rows, None)
|
||||
if not header_row:
|
||||
raise ValueError(f"{xlsx_path}: empty workbook")
|
||||
headers = [str(cell).strip() if cell is not None else "" for cell in header_row]
|
||||
field_index: dict[str, int] = {}
|
||||
for idx, header in enumerate(headers):
|
||||
field = COLUMN_MAP.get(header)
|
||||
if field and field not in field_index:
|
||||
field_index[field] = idx
|
||||
if "stat_date" not in field_index:
|
||||
raise ValueError(f"{xlsx_path}: 日期列缺失, 表头={headers}")
|
||||
|
||||
records: list[dict[str, Any]] = []
|
||||
for row in rows:
|
||||
first = str(row[0]).strip() if row and row[0] is not None else ""
|
||||
if not first or first in TOTAL_ROW_MARKERS:
|
||||
continue
|
||||
record: dict[str, Any] = {"stat_date": parse_stat_date(row[field_index["stat_date"]])}
|
||||
for field in COLUMN_MAP.values():
|
||||
if field == "stat_date":
|
||||
continue
|
||||
idx = field_index.get(field)
|
||||
raw = row[idx] if idx is not None and idx < len(row) else None
|
||||
if field.endswith(("_amount", "_income", "_expense")):
|
||||
record[field] = parse_amount(raw)
|
||||
else:
|
||||
record[field] = parse_number(raw)
|
||||
records.append(record)
|
||||
return records
|
||||
finally:
|
||||
workbook.close()
|
||||
|
||||
|
||||
# ── 页面操作层 ──────────────────────────────────────────────────────────────
|
||||
|
||||
SET_FILTERS_JS = r"""
|
||||
async ({ startDate, endDate, brands, shops }) => {
|
||||
const sleep = ms => new Promise(r => setTimeout(r, ms));
|
||||
const norm = s => String(s || '').replace(/\s+/g, '').trim();
|
||||
const waitFor = (selector, timeoutMs = 10000) => new Promise((resolve, reject) => {
|
||||
const deadline = Date.now() + timeoutMs;
|
||||
const tick = () => {
|
||||
const el = document.querySelector(selector);
|
||||
if (el) return resolve(el);
|
||||
if (Date.now() > deadline) return reject(new Error('timeout waiting for ' + selector));
|
||||
setTimeout(tick, 100);
|
||||
};
|
||||
tick();
|
||||
});
|
||||
const setChecked = async (el, wanted) => {
|
||||
if (Boolean(el.checked) !== wanted) el.click();
|
||||
for (let i = 0; i < 40 && Boolean(el.checked) !== wanted; i++) await sleep(50);
|
||||
if (Boolean(el.checked) !== wanted) throw new Error('checkbox stuck: ' + (el.id || el.value));
|
||||
};
|
||||
|
||||
// 时间口径必须是「发货时间」;否则宁报错也不采错口径
|
||||
const desc = document.querySelector('#date_desc');
|
||||
const descText = norm(desc && desc.innerText);
|
||||
if (!descText.includes('发货')) {
|
||||
throw new Error('date basis is not 发货时间: ' + (descText || '(missing #date_desc)'));
|
||||
}
|
||||
|
||||
const begin = await waitFor('#order_date_begin');
|
||||
const end = await waitFor('#order_date_end');
|
||||
for (const [el, val] of [[begin, startDate], [end, endDate]]) {
|
||||
el.value = val;
|
||||
el.dispatchEvent(new Event('input', { bubbles: true }));
|
||||
el.dispatchEvent(new Event('change', { bubbles: true }));
|
||||
el.blur();
|
||||
}
|
||||
if (begin.value !== startDate || end.value !== endDate) {
|
||||
throw new Error('date did not persist: ' + begin.value + ' ~ ' + end.value);
|
||||
}
|
||||
|
||||
// 品牌: 全清后勾目标, 回读
|
||||
const brandTargets = new Set(brands);
|
||||
const brandInputs = Array.from(document.querySelectorAll('input[name="brand_name"]'));
|
||||
for (const el of brandInputs) await setChecked(el, brandTargets.has(el.value));
|
||||
const checkedBrands = brandInputs.filter(el => el.checked).map(el => el.value);
|
||||
const missingBrands = brands.filter(b => !checkedBrands.includes(b));
|
||||
if (missingBrands.length) {
|
||||
throw new Error('brands not applied: ' + missingBrands.join(',') +
|
||||
' available=' + brandInputs.map(el => el.value).join(','));
|
||||
}
|
||||
|
||||
// 店铺: 按标签精确匹配; 目标名必须全部存在, 否则报错并列出可用清单
|
||||
const labelOf = input => norm(
|
||||
(input.closest('label') || input.parentElement || input).innerText
|
||||
);
|
||||
const shopInputs = Array.from(document.querySelectorAll('input[name="shop_id"]'));
|
||||
const byLabel = new Map();
|
||||
for (const el of shopInputs) {
|
||||
const label = labelOf(el);
|
||||
if (!label) continue;
|
||||
if (byLabel.has(label)) throw new Error('duplicate shop label: ' + label);
|
||||
byLabel.set(label, el);
|
||||
}
|
||||
const shopTargets = shops.map(norm);
|
||||
const missingShops = shopTargets.filter(name => !byLabel.has(name));
|
||||
if (missingShops.length) {
|
||||
throw new Error('shops not found in filter: ' + missingShops.join(',') +
|
||||
' | available=' + Array.from(byLabel.keys()).join(','));
|
||||
}
|
||||
for (const el of shopInputs) await setChecked(el, shopTargets.includes(labelOf(el)));
|
||||
const checkedShops = shopInputs.filter(el => el.checked).map(labelOf);
|
||||
|
||||
return {
|
||||
dateBegin: begin.value,
|
||||
dateEnd: end.value,
|
||||
dateBasis: descText,
|
||||
brands: checkedBrands,
|
||||
shops: checkedShops,
|
||||
};
|
||||
}
|
||||
"""
|
||||
|
||||
CLICK_DAY_TAB_JS = r"""
|
||||
() => {
|
||||
const ownText = el => Array.from(el.childNodes)
|
||||
.filter(n => n.nodeType === 3).map(n => n.textContent).join('').replace(/\s+/g, ' ').trim();
|
||||
const tab = Array.from(document.querySelectorAll('div,span,li,a'))
|
||||
.find(el => ownText(el) === '按日统计');
|
||||
if (!tab) throw new Error('按日统计 tab not found on report shell');
|
||||
(tab.closest('div,li,a') || tab).click();
|
||||
return true;
|
||||
}
|
||||
"""
|
||||
|
||||
DAY_BODY_SNAPSHOT_JS = r"""
|
||||
() => {
|
||||
const text = (document.body.innerText || '').replace(/\s+/g, ' ');
|
||||
const noData = text.includes('没有任何数据') || text.includes('无数据');
|
||||
const pending = text.includes('点击左侧');
|
||||
// 报表按日期倒序且分页, 首页只含最近若干天; 边界日期可能不存在
|
||||
// (如今日尚未出数), 因此抽出页面全部日期供区间判断。
|
||||
const dates = Array.from(text.matchAll(/\d{4}-\d{2}-\d{2}/g)).map(m => m[0]);
|
||||
return { text: text.slice(0, 2000), noData, pending, dates };
|
||||
}
|
||||
"""
|
||||
|
||||
CLICK_EXPORT_JS = r"""
|
||||
() => {
|
||||
const ownText = el => Array.from(el.childNodes)
|
||||
.filter(n => n.nodeType === 3).map(n => n.textContent).join('').replace(/\s+/g, ' ').trim();
|
||||
const span = Array.from(document.querySelectorAll('span'))
|
||||
.find(el => ownText(el) === '导出EXCEL');
|
||||
if (!span) throw new Error('导出EXCEL button not found in day report');
|
||||
const target = span.closest('.btn_line') || span.parentElement;
|
||||
target.dispatchEvent(new MouseEvent('mousedown', { bubbles: true }));
|
||||
target.dispatchEvent(new MouseEvent('mouseup', { bubbles: true }));
|
||||
target.click();
|
||||
return String(target.className);
|
||||
}
|
||||
"""
|
||||
|
||||
|
||||
def _is_page_closed_error(exc: BaseException) -> bool:
|
||||
"""沿着异常链判断是否为 Playwright 页面/上下文/浏览器被关闭。"""
|
||||
seen: set[int] = set()
|
||||
node: BaseException | None = exc
|
||||
while node is not None and id(node) not in seen:
|
||||
seen.add(id(node))
|
||||
if type(node).__name__ == "TargetClosedError":
|
||||
return True
|
||||
if "Target page, context or browser has been closed" in str(node):
|
||||
return True
|
||||
node = node.__cause__ or node.__context__
|
||||
return False
|
||||
|
||||
|
||||
def find_frame(page: Any, needle: str, wait_seconds: int = 20) -> Any:
|
||||
import time
|
||||
|
||||
deadline = time.monotonic() + wait_seconds
|
||||
while True:
|
||||
for frame in page.frames:
|
||||
if needle in (frame.url or ""):
|
||||
return frame
|
||||
if time.monotonic() >= deadline:
|
||||
raise RuntimeError(
|
||||
f"frame {needle} not found; frames={[f.url for f in page.frames]}"
|
||||
)
|
||||
page.wait_for_timeout(400)
|
||||
|
||||
|
||||
def wait_day_report(
|
||||
page: Any,
|
||||
*,
|
||||
start_date: str,
|
||||
end_date: str,
|
||||
timeout_seconds: int,
|
||||
) -> dict[str, Any]:
|
||||
"""等待 day.aspx 出现本区间数据(含起始日字符串)并连续两次稳定。"""
|
||||
|
||||
import time
|
||||
|
||||
deadline = time.monotonic() + timeout_seconds
|
||||
stable_key = None
|
||||
stable_count = 0
|
||||
last_error = "day report did not return data"
|
||||
while time.monotonic() < deadline:
|
||||
frame = next(
|
||||
(f for f in page.frames if "adsorder/day.aspx" in (f.url or "")), None
|
||||
)
|
||||
if frame is None:
|
||||
last_error = "day.aspx frame not present"
|
||||
page.evaluate(CLICK_DAY_TAB_JS)
|
||||
page.wait_for_timeout(2000)
|
||||
continue
|
||||
try:
|
||||
snapshot = frame.evaluate(DAY_BODY_SNAPSHOT_JS)
|
||||
except Exception as exc:
|
||||
last_error = f"{type(exc).__name__}: {exc}"
|
||||
page.wait_for_timeout(1000)
|
||||
continue
|
||||
if snapshot.get("pending"):
|
||||
last_error = "day report still waiting for 生成报表"
|
||||
regenerate_report(page)
|
||||
page.wait_for_timeout(3000)
|
||||
stable_key = None
|
||||
stable_count = 0
|
||||
continue
|
||||
fresh = any(
|
||||
start_date <= day <= end_date for day in snapshot.get("dates") or ()
|
||||
)
|
||||
if snapshot.get("no_data"):
|
||||
# 空结果也要求稳定出现两次,防止复用上一次查询的空页
|
||||
key = ("no_data", hash(snapshot["text"]))
|
||||
if key == stable_key:
|
||||
stable_count += 1
|
||||
else:
|
||||
stable_key, stable_count = key, 1
|
||||
if stable_count >= 2:
|
||||
return {"ok": True, "no_data": True, "head": snapshot["text"][:200]}
|
||||
page.wait_for_timeout(1000)
|
||||
continue
|
||||
if not fresh:
|
||||
last_error = "day report body does not cover requested range"
|
||||
stable_key = None
|
||||
stable_count = 0
|
||||
page.wait_for_timeout(1000)
|
||||
continue
|
||||
key = hash(snapshot["text"])
|
||||
if key == stable_key:
|
||||
stable_count += 1
|
||||
else:
|
||||
stable_key, stable_count = key, 1
|
||||
if stable_count >= 2:
|
||||
return {"ok": True, "no_data": False, "head": snapshot["text"][:200]}
|
||||
page.wait_for_timeout(1000)
|
||||
raise TimeoutError(f"timeout waiting for day report: {last_error}")
|
||||
|
||||
|
||||
def regenerate_report(page: Any) -> None:
|
||||
filter_frame = next(
|
||||
(f for f in page.frames if "adsorder/filter.aspx" in (f.url or "")), None
|
||||
)
|
||||
if filter_frame is None:
|
||||
return
|
||||
filter_frame.evaluate(
|
||||
"() => { if (typeof window.BtnFullSearch === 'function') window.BtnFullSearch(); }"
|
||||
)
|
||||
|
||||
|
||||
def _download_oss_export(url: str, evidence_dir: Path) -> Path:
|
||||
"""带签名 OSS 导出 URL 的直连兜底下载(页面死亡后仍可获取文件)。
|
||||
|
||||
ERP 异步导出会先上传 OSS 再返回带签名的 jstzbstatic URL;签名有有效期
|
||||
(约 20 分钟),命中后立即下载。直连不走系统代理(国内站)。
|
||||
"""
|
||||
import re as _re
|
||||
import urllib.request
|
||||
|
||||
if not _re.match(r"^https://jstzbstatic\.erp321\.com/export/[A-Za-z0-9]+\.xlsx\?", url):
|
||||
raise RuntimeError(f"refusing non-export download url: {url[:80]}")
|
||||
opener = urllib.request.build_opener(urllib.request.ProxyHandler({}))
|
||||
request = urllib.request.Request(url, headers={"User-Agent": USER_AGENT})
|
||||
with opener.open(request, timeout=120) as response:
|
||||
payload = response.read()
|
||||
safe_root = evidence_dir.resolve()
|
||||
target = (safe_root / f"销售主题分析_按日_fallback_{datetime.now():%Y%m%d%H%M%S}.xlsx").resolve()
|
||||
if target.parent != safe_root:
|
||||
raise RuntimeError("fallback download path escaped evidence dir")
|
||||
target.write_bytes(payload)
|
||||
return target
|
||||
|
||||
|
||||
def export_day_excel(
|
||||
page: Any,
|
||||
*,
|
||||
evidence_dir: Path,
|
||||
export_timeout_seconds: int,
|
||||
) -> Path:
|
||||
"""点击导出EXCEL并接住浏览器 download 事件, 落盘到证据目录。
|
||||
|
||||
双通道取文件: (1) Playwright download 事件 + save_as; (2) ERP 异步导出
|
||||
产物走 jstzbstatic OSS 签名 URL —— BI 页在异步导出链路上偶发杀死标签页
|
||||
(2026-10-12 实测, 蝉妈妈同款病), 页面死了 download 事件作废, 但签名 URL
|
||||
已经出现在网络请求里, 直连下载不受页面生命周期影响。
|
||||
"""
|
||||
import time
|
||||
|
||||
day_frame = find_frame(page, "adsorder/day.aspx", wait_seconds=15)
|
||||
received: list[Path] = []
|
||||
oss_urls: list[str] = []
|
||||
|
||||
def on_download(download: Any) -> None:
|
||||
try:
|
||||
if download.url and download.url not in oss_urls:
|
||||
oss_urls.append(download.url)
|
||||
name = download.suggested_filename or "销售主题分析_按日.xlsx"
|
||||
if not name.lower().endswith(".xlsx"):
|
||||
name += ".xlsx"
|
||||
target = evidence_dir / name
|
||||
download.save_as(str(target))
|
||||
received.append(target)
|
||||
except Exception as exc:
|
||||
print(f"[WARN] download save failed: {type(exc).__name__}: {exc}", flush=True)
|
||||
|
||||
def on_request(request: Any) -> None:
|
||||
url = request.url or ""
|
||||
if "jstzbstatic.erp321.com/export/" in url and ".xlsx" in url and url not in oss_urls:
|
||||
oss_urls.append(url)
|
||||
|
||||
page.on("download", on_download)
|
||||
page.context.on("request", on_request)
|
||||
try:
|
||||
day_frame.evaluate(CLICK_EXPORT_JS)
|
||||
deadline = time.monotonic() + export_timeout_seconds
|
||||
while time.monotonic() < deadline and not received:
|
||||
try:
|
||||
page.wait_for_timeout(1000)
|
||||
except Exception:
|
||||
# 页面已死, save_as 通道作废; 直接转入 OSS 直连兜底
|
||||
break
|
||||
finally:
|
||||
try:
|
||||
page.remove_listener("download", on_download)
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
page.context.remove_listener("request", on_request)
|
||||
except Exception:
|
||||
pass
|
||||
if received:
|
||||
return received[0]
|
||||
if oss_urls:
|
||||
path = _download_oss_export(oss_urls[-1], evidence_dir)
|
||||
print(f"[OK] 导出走 OSS 直连兜底下载: {path.name}", flush=True)
|
||||
return path
|
||||
raise TimeoutError(
|
||||
f"export download did not arrive within {export_timeout_seconds}s"
|
||||
)
|
||||
|
||||
|
||||
def collect_export(
|
||||
page: Any,
|
||||
*,
|
||||
start_date: str,
|
||||
end_date: str,
|
||||
brands: tuple[str, ...],
|
||||
shops: tuple[str, ...],
|
||||
evidence_dir: Path,
|
||||
data_wait_seconds: int,
|
||||
export_timeout_seconds: int,
|
||||
) -> dict[str, Any]:
|
||||
"""一轮完整采集: 打开报表 → 设筛选 → 生成 → 按日 → 导出。"""
|
||||
if "adsorder/default.aspx" not in (page.url or ""):
|
||||
page.goto(REPORT_URL, wait_until="domcontentloaded", timeout=60000)
|
||||
page.wait_for_timeout(3000)
|
||||
dismiss_marketing_popup(page)
|
||||
close_foreign_popup_tabs(getattr(page, "context", None), page)
|
||||
|
||||
filter_frame = find_frame(page, "adsorder/filter.aspx", wait_seconds=30)
|
||||
selection = filter_frame.evaluate(
|
||||
SET_FILTERS_JS,
|
||||
{"startDate": start_date, "endDate": end_date, "brands": list(brands), "shops": list(shops)},
|
||||
)
|
||||
regenerate_report(page)
|
||||
page.wait_for_timeout(1500)
|
||||
page.evaluate(CLICK_DAY_TAB_JS)
|
||||
report_state = wait_day_report(
|
||||
page,
|
||||
start_date=start_date,
|
||||
end_date=end_date,
|
||||
timeout_seconds=data_wait_seconds,
|
||||
)
|
||||
export_path = export_day_excel(
|
||||
page,
|
||||
evidence_dir=evidence_dir,
|
||||
export_timeout_seconds=export_timeout_seconds,
|
||||
)
|
||||
return {"selection": selection, "report": report_state, "export_path": export_path}
|
||||
|
||||
|
||||
# ── CLI ─────────────────────────────────────────────────────────────────────
|
||||
|
||||
def parse_date_arg(value: str) -> date:
|
||||
try:
|
||||
return date.fromisoformat(value)
|
||||
except ValueError as exc:
|
||||
raise argparse.ArgumentTypeError("date must be YYYY-MM-DD") from exc
|
||||
|
||||
|
||||
def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="聚水潭销售主题分析(按日)导出采集入库。"
|
||||
)
|
||||
parser.add_argument("--date", type=parse_date_arg, default=None,
|
||||
help="业务日期(区间结束日); 缺省取昨天")
|
||||
parser.add_argument("--days", type=int, default=1,
|
||||
help="发货时间区间天数(含结束日); --date 默认 1 天")
|
||||
parser.add_argument("--from", dest="from_date", type=parse_date_arg, default=None,
|
||||
help="显式区间起始日(与 --to 一起用, 覆盖 --date/--days)")
|
||||
parser.add_argument("--to", dest="to_date", type=parse_date_arg, default=None,
|
||||
help="显式区间结束日")
|
||||
parser.add_argument("--execute", action="store_true",
|
||||
help="正式执行采集并写 PostgreSQL")
|
||||
parser.add_argument("--headless", action="store_true")
|
||||
parser.add_argument("--timeout-ms", type=int, default=120_000)
|
||||
parser.add_argument("--data-wait-timeout", type=int, default=180,
|
||||
help="等待按日报表出数的秒数(默认 180)")
|
||||
parser.add_argument("--export-timeout", type=int, default=120,
|
||||
help="等待导出下载的秒数(默认 120)")
|
||||
parser.add_argument("--report-attempts", type=int, default=2,
|
||||
help="整轮采集重试次数(含首次)")
|
||||
return parser.parse_args(argv)
|
||||
|
||||
|
||||
def resolve_window(args: argparse.Namespace) -> tuple[date, date]:
|
||||
if args.from_date or args.to_date:
|
||||
if not (args.from_date and args.to_date):
|
||||
raise SystemExit("--from 与 --to 必须成对使用")
|
||||
start, end = args.from_date, args.to_date
|
||||
else:
|
||||
end = args.date or (date.today() - timedelta(days=1))
|
||||
start = end - timedelta(days=args.days - 1)
|
||||
if end < start:
|
||||
raise SystemExit(f"区间无效: {start} ~ {end}")
|
||||
if (end - start).days + 1 > 366:
|
||||
raise SystemExit("发货时间区间不能超过 366 天")
|
||||
return start, end
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
from scrapling.fetchers import DynamicFetcher
|
||||
|
||||
args = parse_args(argv)
|
||||
start, end = resolve_window(args)
|
||||
evidence_dir = RAW_DATA_ROOT / EVIDENCE_DIR_NAME / f"{start.isoformat()}_{end.isoformat()}"
|
||||
plan = {
|
||||
"range": [start.isoformat(), end.isoformat()],
|
||||
"days": (end - start).days + 1,
|
||||
"brands": list(BRANDS),
|
||||
"shops": list(SHOP_NAMES),
|
||||
"write_table": "erp_sales_theme_daily_metrics",
|
||||
"execute": bool(args.execute),
|
||||
}
|
||||
print(json.dumps(plan, ensure_ascii=False), flush=True)
|
||||
if not args.execute:
|
||||
print("[DRY-RUN] Add --execute to collect and write PostgreSQL.")
|
||||
return 0
|
||||
|
||||
import os
|
||||
|
||||
username = os.environ.get("ERP_USERNAME", "").strip()
|
||||
login_secret = (
|
||||
os.environ.get("ERP_PASSWORD", "").strip()
|
||||
or os.environ.get("GYXX_SUPPLY_ERP_PASSWORD", "").strip()
|
||||
)
|
||||
login_url = os.environ.get("GYXX_ERP_LOGIN_URL", "").strip() or LOGIN_URL_DEFAULT
|
||||
# 成员绑定的 CDP 常驻浏览器并不一定在跑(无 keepalive);只有真正在听才用,
|
||||
# 否则回退隔离 profile + vault cookie 注入(restore_erp_runtime_state)。
|
||||
cdp_url, user_data_dir = resolve_browser_session({})
|
||||
|
||||
evidence_dir.mkdir(parents=True, exist_ok=True)
|
||||
state: dict[str, Any] = {}
|
||||
|
||||
def page_action(page: Any) -> None:
|
||||
open_action = make_page_action(
|
||||
login_url=login_url,
|
||||
username=username,
|
||||
**{"pass" + "word": login_secret},
|
||||
home_url=REPORT_URL,
|
||||
timeout_ms=args.timeout_ms,
|
||||
captcha_checks=24,
|
||||
captcha_interval_seconds=5,
|
||||
hold_seconds=0,
|
||||
state=state,
|
||||
login_only=True,
|
||||
)
|
||||
open_action(page)
|
||||
if not state.get("ok"):
|
||||
return
|
||||
try:
|
||||
state["collected"] = retry_step(
|
||||
lambda: collect_export(
|
||||
page,
|
||||
start_date=start.isoformat(),
|
||||
end_date=end.isoformat(),
|
||||
brands=BRANDS,
|
||||
shops=SHOP_NAMES,
|
||||
evidence_dir=evidence_dir,
|
||||
data_wait_seconds=max(args.data_wait_timeout, min(plan["days"] * 3, 600)),
|
||||
export_timeout_seconds=args.export_timeout,
|
||||
),
|
||||
max_attempts=max(args.report_attempts, 1),
|
||||
sleep_seconds=5,
|
||||
label="ERP销售主题分析导出",
|
||||
)
|
||||
except BaseException as exc:
|
||||
# ERP BI 页偶发「点导出后 tab 崩溃」(蝉妈妈同款病)。页面已死时原地
|
||||
# 重试毫无意义:记录后正常返回,由外层重开一个全新浏览器会话自愈。
|
||||
if _is_page_closed_error(exc):
|
||||
print(
|
||||
f"[WARN] 报表页在导出阶段崩溃/关闭({type(exc).__name__}),"
|
||||
"本轮作废,由外层重开会话自愈",
|
||||
flush=True,
|
||||
)
|
||||
state["page_crashed"] = True
|
||||
else:
|
||||
state["collect_error"] = f"{type(exc).__name__}: {exc}"
|
||||
|
||||
fetch_kwargs: dict[str, Any] = {
|
||||
"headless": args.headless,
|
||||
"load_dom": True,
|
||||
"network_idle": True,
|
||||
"timeout": args.timeout_ms,
|
||||
"wait": 1000,
|
||||
# scrapling 要求 retries>=1;页面级异常已在 page_action 内转入
|
||||
# state 标志,重试语义由下方外层循环持有(每轮 = 全新浏览器会话)
|
||||
"retries": 1,
|
||||
"useragent": USER_AGENT,
|
||||
"additional_args": {
|
||||
"viewport": {"width": 1600, "height": 950},
|
||||
"screen": {"width": 1600, "height": 950},
|
||||
"accept_downloads": True,
|
||||
},
|
||||
# ERP 是国内站, 代理出口会被 WAF 拉黑(Clash 教训), 一律直连;
|
||||
# 必须显式给下载目录: 覆盖>90天数据的导出走异步导出中心
|
||||
# (CreateAsyncExport → OSS 签名 URL), headless Chrome 没有下载目录时
|
||||
# 这条链路会直接弄死标签页(2026-10-12 实测)。
|
||||
"extra_flags": [
|
||||
"--no-proxy-server",
|
||||
"--disable-dev-shm-usage",
|
||||
f"--download-directory={evidence_dir}",
|
||||
],
|
||||
"page_action": page_action,
|
||||
}
|
||||
if cdp_url:
|
||||
fetch_kwargs["cdp_url"] = cdp_url
|
||||
if user_data_dir:
|
||||
fetch_kwargs["user_data_dir"] = user_data_dir
|
||||
fetch_kwargs["real_chrome"] = True
|
||||
|
||||
passes = max(args.report_attempts, 1)
|
||||
last_error = "collect did not complete"
|
||||
for attempt in range(1, passes + 1):
|
||||
for flag in ("collected", "page_crashed", "collect_error"):
|
||||
state.pop(flag, None)
|
||||
if attempt > 1:
|
||||
print(f"[RETRY] 第 {attempt - 1}/{passes - 1} 轮重开浏览器会话重试采集", flush=True)
|
||||
try:
|
||||
DynamicFetcher.fetch(REPORT_URL, **fetch_kwargs)
|
||||
except Exception as exc:
|
||||
message = str(exc)
|
||||
if state.get("collected"):
|
||||
# 文件已到手, 采集后任何浏览器/响应转换错误都无关紧要
|
||||
print(
|
||||
f"[WARN] 报表已导出,忽略 Scrapling 收尾错误: {type(exc).__name__}",
|
||||
flush=True,
|
||||
)
|
||||
break
|
||||
last_error = f"fetch: {type(exc).__name__}: {message}"
|
||||
print(f"[FAIL] {last_error}", flush=True)
|
||||
continue
|
||||
if state.get("collected"):
|
||||
break
|
||||
if state.get("page_crashed"):
|
||||
last_error = "report page closed during export"
|
||||
continue
|
||||
if state.get("collect_error"):
|
||||
last_error = state["collect_error"]
|
||||
print(f"[FAIL] {last_error}", flush=True)
|
||||
continue
|
||||
break
|
||||
|
||||
if not state.get("ok"):
|
||||
print(
|
||||
"[SKIPPED_COOKIE] ERP 登录态失效且无法自动恢复: "
|
||||
f"{state.get('error', 'unknown error')}",
|
||||
flush=True,
|
||||
)
|
||||
return COOKIE_SKIP_EXIT_CODE
|
||||
collected = state.get("collected")
|
||||
if not collected:
|
||||
print(f"[FAIL] 采集未完成: {last_error}", flush=True)
|
||||
return 1
|
||||
|
||||
export_path: Path = collected["export_path"]
|
||||
records = parse_sales_theme_workbook(export_path)
|
||||
if not records:
|
||||
print(f"[FAIL] 导出文件未解析出数据行: {export_path}", flush=True)
|
||||
return 1
|
||||
expected_days = {(start + timedelta(days=offset)).isoformat() for offset in range(plan["days"])}
|
||||
got_days = {record["stat_date"] for record in records}
|
||||
outside = sorted(got_days - expected_days)
|
||||
if outside:
|
||||
print(f"[FAIL] 导出包含区间外日期 {outside[:5]}", flush=True)
|
||||
return 1
|
||||
missing = sorted(expected_days - got_days)
|
||||
if missing:
|
||||
print(f"[WARN] 区间内 {len(missing)} 天无数据行(ERP 未出行): {missing[:10]}", flush=True)
|
||||
|
||||
source = artifact_relative_path(export_path)
|
||||
for record in records:
|
||||
record["date_basis"] = "发货时间"
|
||||
record["brands"] = list(BRANDS)
|
||||
record["shops"] = list(collected["selection"].get("shops") or ())
|
||||
record["source_file"] = source
|
||||
manifest = {
|
||||
"range": plan["range"],
|
||||
"collected_at": datetime.now().isoformat(timespec="seconds"),
|
||||
"selection": collected["selection"],
|
||||
"report": collected["report"],
|
||||
"export_file": str(export_path),
|
||||
"row_count": len(records),
|
||||
"missing_days": missing,
|
||||
}
|
||||
manifest_path = evidence_dir / "manifest.json"
|
||||
manifest_path.write_text(json.dumps(manifest, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
|
||||
with get_conn() as conn:
|
||||
ensure_erp_sales_theme_daily_table(conn)
|
||||
written = upsert_erp_sales_theme_daily_metrics(conn, records)
|
||||
total_sales_amount = sum(float(r.get("sales_amount") or 0) for r in records)
|
||||
print(
|
||||
f"[DB] {start}~{end} upserted {written} days; "
|
||||
f"shops={len(collected['selection'].get('shops') or ())}; "
|
||||
f"sales_amount_total={total_sales_amount:.2f}; evidence={source}",
|
||||
flush=True,
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -84,8 +84,14 @@ MODE = "daily"
|
||||
# 2026-06-11: daily 走 cell 45/57 老路径 (保护 run_daily_collect.bat 定时任务, data.json 口径不变);
|
||||
# weekly 同样 cell 45/57 (2026-06-27 run_weekly_collect.bat 定时任务已删除, --mode weekly 留作手动回采);
|
||||
# backfill 单日窗口 + 表头匹配 (自定义销量 / 自定义退货量 + SPU 行汇总), 跟 daily/weekly 区分开
|
||||
# 2026-10-11: 聚水潭重置 allsku 报表模板, 38 列被置隐藏后可见列整体左移 5 列, 固定可见索引
|
||||
# 45/57 落到了「今日实退货量/今日仅退款量」(查询日清晨≈0) 上, daily 销量全 0。
|
||||
# daily 改为表头名匹配「昨日销量/昨日退货量」+ x 坐标最近格; weekly 暂留老索引
|
||||
# (语义未实测验证, 周采集复用时需先确认 7 天窗口下各列含义)。
|
||||
WEEKLY_SALES_HEADER = "__idx__45"
|
||||
WEEKLY_RETURNS_HEADER = "__idx__57"
|
||||
DAILY_SALES_HEADER = "昨日销量"
|
||||
DAILY_RETURNS_HEADER = "昨日退货量"
|
||||
BACKFILL_SALES_HEADER = "自定义销量"
|
||||
BACKFILL_RETURNS_HEADER = "自定义退货量"
|
||||
|
||||
@@ -292,8 +298,11 @@ def report_date_range() -> tuple[str, str]:
|
||||
|
||||
|
||||
def _sales_returns_idx() -> tuple[str, str]:
|
||||
if MODE in ("weekly", "daily"):
|
||||
# daily / weekly 都走 cell 45/57 老路径, 跟之前 data.json 口径一致
|
||||
if MODE == "daily":
|
||||
# 表头匹配 (昨日销量 / 昨日退货量): 模板列可见性变化会让固定索引错位
|
||||
return DAILY_SALES_HEADER, DAILY_RETURNS_HEADER
|
||||
if MODE == "weekly":
|
||||
# weekly 仍走 cell 45/57 老路径, 跟之前 data.json 口径一致
|
||||
return WEEKLY_SALES_HEADER, WEEKLY_RETURNS_HEADER
|
||||
# backfill 走表头匹配 (自定义销量 / 自定义退货量 + SPU 行求和)
|
||||
return BACKFILL_SALES_HEADER, BACKFILL_RETURNS_HEADER
|
||||
@@ -395,6 +404,44 @@ def _extract_by_idx(table_frame: Any, sales_idx: int, returns_idx: int) -> dict[
|
||||
)
|
||||
|
||||
|
||||
ENSURE_CUSTOM_COLUMNS_JS = r"""
|
||||
() => {
|
||||
const cols = window._Cols || [];
|
||||
let changed = 0;
|
||||
for (const col of cols) {
|
||||
const label = String(col.text || col.defaultText || '');
|
||||
if ((label === '自定义销量' || label === '自定义退货量') && col.visible === false) {
|
||||
col.visible = true;
|
||||
changed += 1;
|
||||
}
|
||||
}
|
||||
if (changed > 0 && typeof window._RptRefresh === 'function') {
|
||||
try { window._RptRefresh(); } catch (e) {}
|
||||
}
|
||||
return changed;
|
||||
}
|
||||
"""
|
||||
|
||||
|
||||
def ensure_custom_columns_visible(page: Any) -> int:
|
||||
"""backfill 模式: 把模板重置隐藏的自定义销量/退货量列恢复可见。
|
||||
|
||||
2026-10-11 起聚水潭报表模板把自定义公式列重置为隐藏,隐藏列不渲染
|
||||
数据单元格;只改 visible 不会自动重绘,需顺带触发一次 _RptRefresh。
|
||||
与 backfill_erp_all_shop_daily.py 同源(该脚本 10-13 实测有效)。
|
||||
"""
|
||||
|
||||
changed_total = 0
|
||||
for frame in page.frames:
|
||||
if "allsku.aspx" not in (frame.url or "").lower():
|
||||
continue
|
||||
try:
|
||||
changed_total += int(frame.evaluate(ENSURE_CUSTOM_COLUMNS_JS))
|
||||
except Exception:
|
||||
continue
|
||||
return changed_total
|
||||
|
||||
|
||||
def _extract_by_header(table_frame: Any, sales_header: str, returns_header: str) -> dict[str, Any]:
|
||||
"""backfill 模式专用: 按表头列名匹配 sales / returns 列, 直接读每 SPU 行对应 cell 的 innerText 求和.
|
||||
|
||||
@@ -487,13 +534,35 @@ def _extract_by_header(table_frame: Any, sales_header: str, returns_header: str)
|
||||
};
|
||||
}
|
||||
|
||||
// 4) 累加: 每行 cells[salesCol] / cells[returnsCol] 的 innerText
|
||||
// 4) 累加: 每行 cells[salesPos] / cells[returnsPos]
|
||||
// 2026-10-11 起模板把自定义公式列重置为隐藏: 隐藏列的 th 仍在表头,
|
||||
// 但数据行不渲染对应 td, 表头索引直接取行内格子会错位。用
|
||||
// window._Cols 把表头列索引换算成行内可见位置; 拿不到 _Cols 时按原索引直取。
|
||||
let salesPos = salesCol;
|
||||
let returnsPos = returnsCol;
|
||||
if (Array.isArray(window._Cols) && window._Cols.length === headerCells.length) {
|
||||
const visibleIdx = [];
|
||||
for (let i = 0; i < window._Cols.length; i++) {
|
||||
if (window._Cols[i].visible !== false) visibleIdx.push(i);
|
||||
}
|
||||
salesPos = visibleIdx.indexOf(salesCol);
|
||||
returnsPos = visibleIdx.indexOf(returnsCol);
|
||||
}
|
||||
if (salesPos < 0 || returnsPos < 0) {
|
||||
return {
|
||||
ok: false,
|
||||
yesterday_sales: 0,
|
||||
yesterday_returns: 0,
|
||||
error: `custom sales/return columns hidden: salesPos=${salesPos} returnsPos=${returnsPos}`,
|
||||
header_texts: headerTexts,
|
||||
};
|
||||
}
|
||||
let totalSales = 0, totalReturns = 0;
|
||||
const rowBreakdown = [];
|
||||
for (const tr of dataRows) {
|
||||
const cells = Array.from(tr.querySelectorAll('td, th'));
|
||||
const salesCell = cells[salesCol];
|
||||
const returnsCell = cells[returnsCol];
|
||||
const salesCell = cells[salesPos];
|
||||
const returnsCell = cells[returnsPos];
|
||||
const head = (tr.innerText || tr.textContent || '').slice(0, 60);
|
||||
if (!salesCell || !returnsCell) {
|
||||
rowBreakdown.push({
|
||||
@@ -662,7 +731,7 @@ def _extract_with_text(table_frame: Any) -> tuple[dict[str, Any], str]:
|
||||
ERP DOM often contains NBSP (\xa0) etc; Scrapling/Playwright on Windows gbk
|
||||
rejects those on return, so we use encodeURIComponent + tryUTF8 to make it ASCII-safe.
|
||||
|
||||
按 MODE 选 idx 模式 (daily) 或 header 模式 (weekly).
|
||||
按 MODE 选 idx 模式 (weekly) 或 header 模式 (daily / backfill)。
|
||||
"""
|
||||
sales_key, returns_key = _sales_returns_idx()
|
||||
|
||||
@@ -673,17 +742,17 @@ def _extract_with_text(table_frame: Any) -> tuple[dict[str, Any], str]:
|
||||
else:
|
||||
js_args = {"salesHeader": sales_key, "returnsHeader": returns_key, "mode": "header"}
|
||||
|
||||
# 走 SPU 行 + 自定义列名匹配 + 求和 (因为 sumRow 在 ERP 系统里不含自定义段)
|
||||
if sales_key != "__idx__45": # backfill 模式
|
||||
# backfill 走 SPU 行 + 自定义列名匹配 + 求和 (因为 sumRow 在 ERP 系统里不含自定义段);
|
||||
# daily 的 昨日销量/昨日退货量 是原生列, sumRow 里就有, 走下面 payload 的
|
||||
# mode='header' 分支 (表头名 -> x 坐标 -> sumRow 最近格), 不经过 _extract_by_header。
|
||||
if MODE == "backfill":
|
||||
header_result = _extract_by_header(table_frame, sales_key, returns_key)
|
||||
# _extract_by_header 返回 dict, 没有 body_text; wait_for_metrics 需要 erp_code in body_text 校验
|
||||
# 用 erp_code placeholder 骗校验 (它会查 erp_code in summary_text, 而 summary_text 不一定含 erp_code)
|
||||
# 但新逻辑在 row_breakdown 里已有 erp_code 关联的 SPU 编码 (10384xxx), 校验一定会过
|
||||
return header_result, header_result.get("summary_text", "") or "PLACEHOLDER"
|
||||
|
||||
# 老 idx 模式
|
||||
sales_idx = int(sales_key.removeprefix("__idx__"))
|
||||
returns_idx = int(returns_key.removeprefix("__idx__"))
|
||||
# weekly 老 idx 模式 + daily header 模式共用这段 payload (js_args 已按模式构造)
|
||||
payload = table_frame.evaluate(
|
||||
"""
|
||||
(args) => {
|
||||
@@ -712,6 +781,9 @@ def _extract_with_text(table_frame: Any) -> tuple[dict[str, Any], str]:
|
||||
}
|
||||
|
||||
let salesCell = null, returnsCell = null;
|
||||
// salesCol/returnsCol 供末尾 return 的 *_col_idx 引用, 必须在 if/else 外声明
|
||||
// (2026-10-13 daily 切到 header 模式后首测即 ReferenceError 的教训)
|
||||
let salesCol = -1, returnsCol = -1;
|
||||
if (args.mode === 'idx') {
|
||||
const cells = Array.from(sumRow.children).map((cell) => {
|
||||
const box = cell.getBoundingClientRect();
|
||||
@@ -745,8 +817,8 @@ def _extract_with_text(table_frame: Any) -> tuple[dict[str, Any], str]:
|
||||
}
|
||||
return -1;
|
||||
};
|
||||
const salesCol = findCol(args.salesHeader);
|
||||
const returnsCol = findCol(args.returnsHeader);
|
||||
salesCol = findCol(args.salesHeader);
|
||||
returnsCol = findCol(args.returnsHeader);
|
||||
if (salesCol < 0 || returnsCol < 0) {
|
||||
return { ok: false, yesterday_sales: 0, yesterday_returns: 0,
|
||||
error: `header not found: salesCol=${salesCol} returnsCol=${returnsCol}`,
|
||||
@@ -977,6 +1049,9 @@ def wait_for_metrics(page: Any, erp_code: str, timeout_seconds: int = 45,
|
||||
# previous_url 可能在 _reload_allsku_iframe 后被重置为 None
|
||||
|
||||
while True:
|
||||
# backfill 模式: 报表刷新会把自定义列的可见性重置回隐藏, 每轮恢复一次
|
||||
if MODE == "backfill":
|
||||
ensure_custom_columns_visible(page)
|
||||
# 先检测"操作频繁"弹窗 (在主页 DOM, 不在 allsku iframe)
|
||||
if filter_frame is not None and freq_dialog_handled < MAX_FREQ_DIALOG:
|
||||
if _detect_and_close_freq_dialog(page):
|
||||
@@ -1176,6 +1251,9 @@ def _run_one_code(page: Any, filter_frame: Any, platform: dict[str, Any], erp_co
|
||||
"""
|
||||
clear_code(filter_frame)
|
||||
set_platform_and_code(filter_frame, platform["shops"], erp_code)
|
||||
if MODE == "backfill":
|
||||
# 自定义销量/退货量列被模板重置隐藏时先恢复可见, 否则数据行不渲染对应格
|
||||
ensure_custom_columns_visible(page)
|
||||
prev_state = _get_allsku_frame_state(page)
|
||||
generate_report(filter_frame)
|
||||
return wait_for_metrics(
|
||||
|
||||
@@ -0,0 +1,381 @@
|
||||
{
|
||||
"target": {
|
||||
"spreadsheet_token": "LwbBswjfohdoEvtsu1Oc2rI3nac",
|
||||
"sheet_id": "2HAOmi",
|
||||
"sheet_name": "現有庫存",
|
||||
"sku_column": "C",
|
||||
"inventory_column": "D",
|
||||
"product_code_column": "H",
|
||||
"first_data_row": 2,
|
||||
"last_data_row": 52,
|
||||
"expected_sku_count": 51
|
||||
},
|
||||
"source_workbook": "光影行星香港安沛2026Q2計劃書(2026.09.28).xlsx",
|
||||
"source_sheet": "現有庫存",
|
||||
"source_as_of": "2026-09-28",
|
||||
"schedule": {
|
||||
"kind": "daily",
|
||||
"at": "09:00",
|
||||
"timezone": "Asia/Shanghai"
|
||||
},
|
||||
"availability_formula": "qty - order_lock - lock_qty + virtual_qty",
|
||||
"skus": [
|
||||
{
|
||||
"sku": "GY-001BLK",
|
||||
"product_name": "Apollo X1 Shoulder Bag",
|
||||
"ean": "4897080019369",
|
||||
"source_row": 2,
|
||||
"jushuitan_sku_id": "10388030"
|
||||
},
|
||||
{
|
||||
"sku": "GY-001WHT",
|
||||
"product_name": "Apollo X1 Shoulder Bag",
|
||||
"ean": "4897080019376",
|
||||
"source_row": 3,
|
||||
"jushuitan_sku_id": "10388031"
|
||||
},
|
||||
{
|
||||
"sku": "GY-007BLK",
|
||||
"product_name": "Liviana Shoulder Bag",
|
||||
"ean": "4897080019468",
|
||||
"source_row": 4,
|
||||
"jushuitan_sku_id": "10390039"
|
||||
},
|
||||
{
|
||||
"sku": "GY-007WHT",
|
||||
"product_name": "Liviana Shoulder Bag",
|
||||
"ean": "4897080019475",
|
||||
"source_row": 5,
|
||||
"jushuitan_sku_id": "10390040"
|
||||
},
|
||||
{
|
||||
"sku": "GY-008WHT",
|
||||
"product_name": "Starship Satchel",
|
||||
"ean": "4897080019499",
|
||||
"source_row": 6,
|
||||
"jushuitan_sku_id": "10365042"
|
||||
},
|
||||
{
|
||||
"sku": "GY-009KAI",
|
||||
"product_name": "Nebula Crossbody Bag",
|
||||
"ean": "4897080010021",
|
||||
"source_row": 7,
|
||||
"jushuitan_sku_id": "10273005"
|
||||
},
|
||||
{
|
||||
"sku": "GY-011WHT",
|
||||
"product_name": "Zeus Backpack",
|
||||
"ean": "4897080019536",
|
||||
"source_row": 8,
|
||||
"jushuitan_sku_id": "10244056"
|
||||
},
|
||||
{
|
||||
"sku": "GY-012BLK",
|
||||
"product_name": "Zeus II Travel Backpack (Expandable version)",
|
||||
"ean": "4897080019543",
|
||||
"source_row": 9,
|
||||
"jushuitan_sku_id": "10434003"
|
||||
},
|
||||
{
|
||||
"sku": "GY-013BLK",
|
||||
"product_name": "Startrail Crossbody Bag",
|
||||
"ean": "4897080019550",
|
||||
"source_row": 10,
|
||||
"jushuitan_sku_id": "10394065"
|
||||
},
|
||||
{
|
||||
"sku": "GY-013GRN",
|
||||
"product_name": "Startrail Crossbody Bag",
|
||||
"ean": "4897080012308",
|
||||
"source_row": 11,
|
||||
"jushuitan_sku_id": "10394066"
|
||||
},
|
||||
{
|
||||
"sku": "GY-013WHT",
|
||||
"product_name": "Startrail Crossbody Bag",
|
||||
"ean": "4897080019567",
|
||||
"source_row": 12,
|
||||
"jushuitan_sku_id": "10394052"
|
||||
},
|
||||
{
|
||||
"sku": "GY-021BLK",
|
||||
"product_name": "Boundary Backpack",
|
||||
"ean": "4897080012483",
|
||||
"source_row": 13,
|
||||
"jushuitan_sku_id": "10447004"
|
||||
},
|
||||
{
|
||||
"sku": "GY-022BLK",
|
||||
"product_name": "Apollo X2 Shoulder Bag",
|
||||
"ean": "4897080012490",
|
||||
"source_row": 14,
|
||||
"jushuitan_sku_id": "10442009"
|
||||
},
|
||||
{
|
||||
"sku": "GY-022WHT",
|
||||
"product_name": "Apollo X2 Shoulder Bag",
|
||||
"ean": "4897080012506",
|
||||
"source_row": 15,
|
||||
"jushuitan_sku_id": "10442010"
|
||||
},
|
||||
{
|
||||
"sku": "GY-023BLK",
|
||||
"product_name": "Apollo Air Shoulder Bag",
|
||||
"ean": "4897080012513",
|
||||
"source_row": 16,
|
||||
"jushuitan_sku_id": "10456013"
|
||||
},
|
||||
{
|
||||
"sku": "GY-023WHT",
|
||||
"product_name": "Apollo Air Shoulder Bag",
|
||||
"ean": "4897080012995",
|
||||
"source_row": 17,
|
||||
"jushuitan_sku_id": "10456021"
|
||||
},
|
||||
{
|
||||
"sku": "GY-024BLK",
|
||||
"product_name": "Skyreach Air Backpack",
|
||||
"ean": "4897080012520",
|
||||
"source_row": 18,
|
||||
"jushuitan_sku_id": "10454007"
|
||||
},
|
||||
{
|
||||
"sku": "GY-024WHT",
|
||||
"product_name": "Skyreach Air Backpack",
|
||||
"ean": "4897080012537",
|
||||
"source_row": 19,
|
||||
"jushuitan_sku_id": "10454008"
|
||||
},
|
||||
{
|
||||
"sku": "GY-0260RG",
|
||||
"product_name": "Polar Star Backpack",
|
||||
"ean": "4897080010793",
|
||||
"source_row": 20,
|
||||
"jushuitan_sku_id": "10465009"
|
||||
},
|
||||
{
|
||||
"sku": "GY-026BLK",
|
||||
"product_name": "Polar Star Backpack",
|
||||
"ean": "4897080010588",
|
||||
"source_row": 21,
|
||||
"jushuitan_sku_id": "10465003"
|
||||
},
|
||||
{
|
||||
"sku": "GY-026GRN",
|
||||
"product_name": "Polar Star Backpack",
|
||||
"ean": "4897080010786",
|
||||
"source_row": 22,
|
||||
"jushuitan_sku_id": "10465020"
|
||||
},
|
||||
{
|
||||
"sku": "GY-026WHT",
|
||||
"product_name": "Polar Star Backpack",
|
||||
"ean": "4897080010595",
|
||||
"source_row": 23,
|
||||
"jushuitan_sku_id": "10465004"
|
||||
},
|
||||
{
|
||||
"sku": "GY-027BLK",
|
||||
"product_name": "Polar Star tote bag",
|
||||
"ean": "4897080010809",
|
||||
"source_row": 24,
|
||||
"jushuitan_sku_id": "10464008"
|
||||
},
|
||||
{
|
||||
"sku": "GY-027WHT",
|
||||
"product_name": "Polar Star tote bag",
|
||||
"ean": "4897080010571",
|
||||
"source_row": 25,
|
||||
"jushuitan_sku_id": "10464004"
|
||||
},
|
||||
{
|
||||
"sku": "GY-028BLK",
|
||||
"product_name": "Poseidon tote bag (cordura)",
|
||||
"ean": "4897080010779",
|
||||
"source_row": 26,
|
||||
"jushuitan_sku_id": "10472004"
|
||||
},
|
||||
{
|
||||
"sku": "GY-029BLK",
|
||||
"product_name": "Nebula miniCrossbody Bag",
|
||||
"ean": "4897080010762",
|
||||
"source_row": 27,
|
||||
"jushuitan_sku_id": "10475015"
|
||||
},
|
||||
{
|
||||
"sku": "GY-029KAI",
|
||||
"product_name": "Nebula miniCrossbody Bag",
|
||||
"ean": "4897080010748",
|
||||
"source_row": 28,
|
||||
"jushuitan_sku_id": "10475016"
|
||||
},
|
||||
{
|
||||
"sku": "GY-029WHT",
|
||||
"product_name": "Nebula miniCrossbody Bag",
|
||||
"ean": "4897080010755",
|
||||
"source_row": 29,
|
||||
"jushuitan_sku_id": "10475014"
|
||||
},
|
||||
{
|
||||
"sku": "GY-030BLK",
|
||||
"product_name": "Morning Star miniCrossbody Bag",
|
||||
"ean": "4897080013008",
|
||||
"source_row": 30,
|
||||
"jushuitan_sku_id": "10468007"
|
||||
},
|
||||
{
|
||||
"sku": "GY-030WHT",
|
||||
"product_name": "Morning Star miniCrossbody Bag",
|
||||
"ean": "4897080010731",
|
||||
"source_row": 31,
|
||||
"jushuitan_sku_id": "10468003"
|
||||
},
|
||||
{
|
||||
"sku": "GY-031BLK",
|
||||
"product_name": "Star-by-Star EVOCrossbody Bag",
|
||||
"ean": "4897080010724",
|
||||
"source_row": 32,
|
||||
"jushuitan_sku_id": "10474004"
|
||||
},
|
||||
{
|
||||
"sku": "GY-032BLK",
|
||||
"product_name": "GlacierCrossbody Bag",
|
||||
"ean": "4897080010717",
|
||||
"source_row": 33,
|
||||
"jushuitan_sku_id": "10453005"
|
||||
},
|
||||
{
|
||||
"sku": "GY-033BLK",
|
||||
"product_name": "Morning Star 2 Crossbody Bag",
|
||||
"ean": "4897080012896",
|
||||
"source_row": 34,
|
||||
"jushuitan_sku_id": "10469003"
|
||||
},
|
||||
{
|
||||
"sku": "GY-033WHT",
|
||||
"product_name": "Morning Star 2 Crossbody Bag",
|
||||
"ean": "4897080012902",
|
||||
"source_row": 35,
|
||||
"jushuitan_sku_id": "10469004"
|
||||
},
|
||||
{
|
||||
"sku": "GY-034BLK",
|
||||
"product_name": "Startrail 2 Crossbody Bag",
|
||||
"ean": "4897080012919",
|
||||
"source_row": 36,
|
||||
"jushuitan_sku_id": "10504027"
|
||||
},
|
||||
{
|
||||
"sku": "GY-034GRN",
|
||||
"product_name": "Startrail 2 Crossbody Bag",
|
||||
"ean": "4897080012926",
|
||||
"source_row": 37,
|
||||
"jushuitan_sku_id": "10504028"
|
||||
},
|
||||
{
|
||||
"sku": "GY-034WHT",
|
||||
"product_name": "Startrail 2 Crossbody Bag",
|
||||
"ean": "4897080012933",
|
||||
"source_row": 38,
|
||||
"jushuitan_sku_id": "10504029"
|
||||
},
|
||||
{
|
||||
"sku": "GY-035BLK",
|
||||
"product_name": "Zeus 3 air Backpack",
|
||||
"ean": "4897080012940",
|
||||
"source_row": 39,
|
||||
"jushuitan_sku_id": "10482004"
|
||||
},
|
||||
{
|
||||
"sku": "GY-035WHT",
|
||||
"product_name": "Zeus 3 air Backpack",
|
||||
"ean": "4897080012957",
|
||||
"source_row": 40,
|
||||
"jushuitan_sku_id": "10482003"
|
||||
},
|
||||
{
|
||||
"sku": "GY-036BLK",
|
||||
"product_name": "Cirrus 2 Backpack",
|
||||
"ean": "4897080012964",
|
||||
"source_row": 41,
|
||||
"jushuitan_sku_id": "10481004"
|
||||
},
|
||||
{
|
||||
"sku": "GY-036GRN",
|
||||
"product_name": "Cirrus 2 Backpack",
|
||||
"ean": "4897080012988",
|
||||
"source_row": 42,
|
||||
"jushuitan_sku_id": "10481006"
|
||||
},
|
||||
{
|
||||
"sku": "GY-036WHT",
|
||||
"product_name": "Cirrus 2 Backpack",
|
||||
"ean": "4897080012971",
|
||||
"source_row": 43,
|
||||
"jushuitan_sku_id": "10481005"
|
||||
},
|
||||
{
|
||||
"sku": "GY-037BLK",
|
||||
"product_name": "Polar Star 2",
|
||||
"ean": "4897080013046",
|
||||
"source_row": 44,
|
||||
"jushuitan_sku_id": "10520009"
|
||||
},
|
||||
{
|
||||
"sku": "GY-037GRY",
|
||||
"product_name": "Polar Star 2",
|
||||
"ean": "4897080013039",
|
||||
"source_row": 45,
|
||||
"jushuitan_sku_id": "10520014"
|
||||
},
|
||||
{
|
||||
"sku": "GY-038WHT",
|
||||
"product_name": "Polar Star Shoulder Bag",
|
||||
"ean": "4897080013060",
|
||||
"source_row": 46,
|
||||
"jushuitan_sku_id": "10519010"
|
||||
},
|
||||
{
|
||||
"sku": "GY-038BLK",
|
||||
"product_name": "Polar Star Shoulder Bag",
|
||||
"ean": "4897080013053",
|
||||
"source_row": 47,
|
||||
"jushuitan_sku_id": "10519009"
|
||||
},
|
||||
{
|
||||
"sku": "GY-039BLK",
|
||||
"product_name": "Poseidon Laptop Bag",
|
||||
"ean": "4897080013084",
|
||||
"source_row": 48,
|
||||
"jushuitan_sku_id": "10515020"
|
||||
},
|
||||
{
|
||||
"sku": "GY-039GRY",
|
||||
"product_name": "Poseidon Laptop Bag",
|
||||
"ean": "4897080013077",
|
||||
"source_row": 49,
|
||||
"jushuitan_sku_id": "10515019"
|
||||
},
|
||||
{
|
||||
"sku": "GY-040BLK",
|
||||
"product_name": "Skyreach 3",
|
||||
"ean": "4897080013091",
|
||||
"source_row": 50,
|
||||
"jushuitan_sku_id": "10510019"
|
||||
},
|
||||
{
|
||||
"sku": "GY-040WHT",
|
||||
"product_name": "Skyreach 3",
|
||||
"ean": "4897080013107",
|
||||
"source_row": 51,
|
||||
"jushuitan_sku_id": "10510020"
|
||||
},
|
||||
{
|
||||
"sku": "GY-041BLK",
|
||||
"product_name": "Phantom",
|
||||
"ean": "4897080013114",
|
||||
"source_row": 52,
|
||||
"jushuitan_sku_id": "10507015"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -68,6 +68,9 @@
|
||||
fact_market_rank_notifications
|
||||
写: market_rank_hermes_notification.py (Hermes 机器人通知结果;文件名兼容保留)
|
||||
读: 同一收件人同日通知去重 / 发送审计
|
||||
erp_sales_theme_daily_metrics
|
||||
写: collect_erp_sales_theme_daily.py (聚水潭「报表→销售主题分析→按日统计」Excel 导出入库)
|
||||
读: 固定品牌+店铺口径的全渠道按日销售汇总查询
|
||||
|
||||
所有 upsert 都用 ON CONFLICT ... DO UPDATE, 重复跑覆盖不双写。
|
||||
"""
|
||||
@@ -1908,3 +1911,160 @@ def upsert_market_rank_notification(conn, record: dict) -> int:
|
||||
if not row:
|
||||
raise RuntimeError("市场排行飞书通知归档未返回记录 id")
|
||||
return int(row[0])
|
||||
|
||||
|
||||
# ── erp_sales_theme_daily_metrics ────────────────────────────────────────────
|
||||
|
||||
def ensure_erp_sales_theme_daily_table(conn) -> None:
|
||||
"""幂等创建聚水潭销售主题分析(按日)汇总表,不执行其他 schema 迁移。"""
|
||||
|
||||
sql = """
|
||||
CREATE TABLE IF NOT EXISTS erp_sales_theme_daily_metrics (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
stat_date DATE NOT NULL,
|
||||
sales_order_count INTEGER NOT NULL DEFAULT 0,
|
||||
sales_qty INTEGER NOT NULL DEFAULT 0,
|
||||
shipped_qty INTEGER NOT NULL DEFAULT 0,
|
||||
shipped_amount NUMERIC(18, 2) NOT NULL DEFAULT 0,
|
||||
sales_amount NUMERIC(18, 2) NOT NULL DEFAULT 0,
|
||||
return_order_count INTEGER NOT NULL DEFAULT 0,
|
||||
return_qty INTEGER NOT NULL DEFAULT 0,
|
||||
returned_qty INTEGER NOT NULL DEFAULT 0,
|
||||
return_amount NUMERIC(18, 2) NOT NULL DEFAULT 0,
|
||||
returned_amount NUMERIC(18, 2) NOT NULL DEFAULT 0,
|
||||
net_sales_order_count INTEGER NOT NULL DEFAULT 0,
|
||||
net_sales_qty INTEGER NOT NULL DEFAULT 0,
|
||||
net_sales_amount NUMERIC(18, 2) NOT NULL DEFAULT 0,
|
||||
freight_income NUMERIC(18, 2),
|
||||
freight_expense NUMERIC(18, 2),
|
||||
shipped_order_count INTEGER NOT NULL DEFAULT 0,
|
||||
date_basis TEXT NOT NULL DEFAULT '发货时间',
|
||||
brands TEXT[] NOT NULL DEFAULT '{}',
|
||||
shops TEXT[] NOT NULL DEFAULT '{}',
|
||||
source_file TEXT,
|
||||
imported_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
UNIQUE (stat_date)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS erp_sales_theme_daily_metrics_date_idx
|
||||
ON erp_sales_theme_daily_metrics (stat_date DESC);
|
||||
COMMENT ON TABLE erp_sales_theme_daily_metrics IS
|
||||
'聚水潭「报表→销售主题分析→按日统计」Excel 导出入库;固定品牌+店铺口径,每日一行幂等覆盖。';
|
||||
COMMENT ON COLUMN erp_sales_theme_daily_metrics.date_basis IS
|
||||
'报表时间口径,当前固定为发货时间(send_date)。';
|
||||
COMMENT ON COLUMN erp_sales_theme_daily_metrics.brands IS
|
||||
'本次查询勾选的商品品牌清单(筛选页 input[name=brand_name] 值)。';
|
||||
COMMENT ON COLUMN erp_sales_theme_daily_metrics.shops IS
|
||||
'本次查询勾选的销售渠道(店铺)名称清单。';
|
||||
"""
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(sql)
|
||||
|
||||
|
||||
_SALES_THEME_INT_FIELDS = frozenset({
|
||||
"sales_order_count",
|
||||
"sales_qty",
|
||||
"shipped_qty",
|
||||
"return_order_count",
|
||||
"return_qty",
|
||||
"returned_qty",
|
||||
"net_sales_order_count",
|
||||
"net_sales_qty",
|
||||
"shipped_order_count",
|
||||
})
|
||||
_SALES_THEME_COLUMNS = (
|
||||
"sales_order_count",
|
||||
"sales_qty",
|
||||
"shipped_qty",
|
||||
"shipped_amount",
|
||||
"sales_amount",
|
||||
"return_order_count",
|
||||
"return_qty",
|
||||
"returned_qty",
|
||||
"return_amount",
|
||||
"returned_amount",
|
||||
"net_sales_amount",
|
||||
"net_sales_order_count",
|
||||
"net_sales_qty",
|
||||
"shipped_order_count",
|
||||
"freight_income",
|
||||
"freight_expense",
|
||||
)
|
||||
# 表内 NOT NULL 的金额列;导出里无退货/无数据的日期这些单元格为空, 统一落 0
|
||||
# (运费收入/支出两列允许 NULL, 与导出口径一致)。
|
||||
_SALES_THEME_REQUIRED_AMOUNTS = frozenset({
|
||||
"shipped_amount",
|
||||
"sales_amount",
|
||||
"return_amount",
|
||||
"returned_amount",
|
||||
"net_sales_amount",
|
||||
})
|
||||
|
||||
|
||||
def upsert_erp_sales_theme_daily_metrics(conn, records: Sequence[dict]) -> int:
|
||||
"""幂等写入销售主题分析按日汇总,主键为统计日期。
|
||||
|
||||
records: [{'stat_date', 'sales_order_count', ..., 'brands', 'shops', 'source_file'}, ...]
|
||||
数量字段缺省按 0 处理;brands/shops 写入前去重排序,保证重跑结果稳定。
|
||||
"""
|
||||
|
||||
if not records:
|
||||
return 0
|
||||
dedup: dict[Any, dict] = {}
|
||||
for record in records:
|
||||
key = record["stat_date"]
|
||||
if str(key).strip() == "":
|
||||
raise ValueError("stat_date must not be empty")
|
||||
dedup[key] = dict(record)
|
||||
rows = []
|
||||
for record in dedup.values():
|
||||
row = [record["stat_date"]]
|
||||
for field in _SALES_THEME_COLUMNS:
|
||||
value = record.get(field)
|
||||
if field in _SALES_THEME_INT_FIELDS:
|
||||
row.append(int(value or 0))
|
||||
elif field in _SALES_THEME_REQUIRED_AMOUNTS:
|
||||
row.append(round(float(value), 2) if value is not None else 0)
|
||||
else:
|
||||
row.append(round(float(value), 2) if value is not None else None)
|
||||
row.append(str(record.get("date_basis") or "发货时间"))
|
||||
row.append(sorted({str(item).strip() for item in record.get("brands") or () if str(item).strip()}))
|
||||
row.append(sorted({str(item).strip() for item in record.get("shops") or () if str(item).strip()}))
|
||||
row.append(record.get("source_file"))
|
||||
rows.append(tuple(row))
|
||||
sql = """
|
||||
INSERT INTO erp_sales_theme_daily_metrics
|
||||
(stat_date, sales_order_count, sales_qty, shipped_qty,
|
||||
shipped_amount, sales_amount,
|
||||
return_order_count, return_qty, returned_qty,
|
||||
return_amount, returned_amount, net_sales_amount,
|
||||
net_sales_order_count, net_sales_qty,
|
||||
shipped_order_count, freight_income, freight_expense,
|
||||
date_basis, brands, shops, source_file)
|
||||
VALUES %s
|
||||
ON CONFLICT (stat_date) DO UPDATE SET
|
||||
sales_order_count = EXCLUDED.sales_order_count,
|
||||
sales_qty = EXCLUDED.sales_qty,
|
||||
shipped_qty = EXCLUDED.shipped_qty,
|
||||
shipped_amount = EXCLUDED.shipped_amount,
|
||||
sales_amount = EXCLUDED.sales_amount,
|
||||
return_order_count = EXCLUDED.return_order_count,
|
||||
return_qty = EXCLUDED.return_qty,
|
||||
returned_qty = EXCLUDED.returned_qty,
|
||||
return_amount = EXCLUDED.return_amount,
|
||||
returned_amount = EXCLUDED.returned_amount,
|
||||
net_sales_order_count = EXCLUDED.net_sales_order_count,
|
||||
net_sales_qty = EXCLUDED.net_sales_qty,
|
||||
net_sales_amount = EXCLUDED.net_sales_amount,
|
||||
freight_income = EXCLUDED.freight_income,
|
||||
freight_expense = EXCLUDED.freight_expense,
|
||||
shipped_order_count = EXCLUDED.shipped_order_count,
|
||||
date_basis = EXCLUDED.date_basis,
|
||||
brands = EXCLUDED.brands,
|
||||
shops = EXCLUDED.shops,
|
||||
source_file = EXCLUDED.source_file,
|
||||
updated_at = NOW()
|
||||
"""
|
||||
with conn.cursor() as cur:
|
||||
execute_values(cur, sql, rows)
|
||||
return len(rows)
|
||||
@@ -1273,3 +1273,37 @@ COMMENT ON TABLE fact_video_publications IS
|
||||
'附件级视频发布幂等账本;点击发布后无明确回执必须进入 ambiguous,禁止自动重试。';
|
||||
COMMENT ON TABLE fact_video_publication_attempts IS
|
||||
'视频发布每次尝试的标题、商品、话题、标签、提交时点与确认凭据。';
|
||||
|
||||
-- 聚水潭「报表→销售主题分析→按日统计」Excel 导出入库。
|
||||
-- 口径:发货时间区间 + 固定品牌(光影行星/光影/gyxx) + 固定勾选店铺清单,
|
||||
-- 采集脚本 collect_erp_sales_theme_daily.py;与 erp_all_shop_style_daily_metrics
|
||||
-- (全店铺按款式) 分表,本表是全渠道合并口径的按日汇总。
|
||||
CREATE TABLE IF NOT EXISTS erp_sales_theme_daily_metrics (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
stat_date DATE NOT NULL,
|
||||
sales_order_count INTEGER NOT NULL DEFAULT 0, -- 销售单数
|
||||
sales_qty INTEGER NOT NULL DEFAULT 0, -- 销售数量
|
||||
shipped_qty INTEGER NOT NULL DEFAULT 0, -- 实发数量
|
||||
shipped_amount NUMERIC(18, 2) NOT NULL DEFAULT 0, -- 实发金额
|
||||
sales_amount NUMERIC(18, 2) NOT NULL DEFAULT 0, -- 销售金额
|
||||
return_order_count INTEGER NOT NULL DEFAULT 0, -- 退货单数
|
||||
return_qty INTEGER NOT NULL DEFAULT 0, -- 退货数量
|
||||
returned_qty INTEGER NOT NULL DEFAULT 0, -- 实退数量
|
||||
return_amount NUMERIC(18, 2) NOT NULL DEFAULT 0, -- 退货金额
|
||||
returned_amount NUMERIC(18, 2) NOT NULL DEFAULT 0, -- 实退金额
|
||||
net_sales_order_count INTEGER NOT NULL DEFAULT 0, -- 净销售单数
|
||||
net_sales_qty INTEGER NOT NULL DEFAULT 0, -- 净销量
|
||||
net_sales_amount NUMERIC(18, 2) NOT NULL DEFAULT 0, -- 净销售额
|
||||
freight_income NUMERIC(18, 2), -- 运费收入(可空)
|
||||
freight_expense NUMERIC(18, 2), -- 运费支出(可空)
|
||||
shipped_order_count INTEGER NOT NULL DEFAULT 0, -- 实发单数
|
||||
date_basis TEXT NOT NULL DEFAULT '发货时间',
|
||||
brands TEXT[] NOT NULL DEFAULT '{}',
|
||||
shops TEXT[] NOT NULL DEFAULT '{}',
|
||||
source_file TEXT,
|
||||
imported_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
UNIQUE (stat_date)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS erp_sales_theme_daily_metrics_date_idx
|
||||
ON erp_sales_theme_daily_metrics (stat_date DESC);
|
||||
@@ -1437,7 +1437,8 @@ def load_jd_product_references(
|
||||
"""
|
||||
SELECT style_name, jd_spus
|
||||
FROM dim_style
|
||||
WHERE jd_spus IS NOT NULL AND cardinality(jd_spus) > 0
|
||||
WHERE jsonb_typeof(jd_spus) = 'array'
|
||||
AND jsonb_array_length(jd_spus) > 0
|
||||
"""
|
||||
)
|
||||
rows = cursor.fetchall()
|
||||
|
||||
@@ -0,0 +1,442 @@
|
||||
"""Refresh Hong Kong Anpei public inventory in the Feishu spreadsheet."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sys
|
||||
import time
|
||||
from collections.abc import Mapping
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from zoneinfo import ZoneInfo
|
||||
|
||||
from gyxx_flow.adapters.jushuitan_inventory import (
|
||||
InventoryRecord,
|
||||
JushuitanInventoryClient,
|
||||
)
|
||||
from gyxx_flow.modules.product_commerce.lark_cli_runtime import run_lark_cli
|
||||
|
||||
_WORKFLOW_ID = "product.hk_anpei_public_inventory.daily"
|
||||
_CONFIG_PATH = (
|
||||
Path(__file__).resolve().parent
|
||||
/ "config"
|
||||
/ "hk_anpei_public_inventory.json"
|
||||
)
|
||||
_REPOSITORY_ROOT = Path(__file__).resolve().parents[4]
|
||||
_TIMEZONE = ZoneInfo("Asia/Shanghai")
|
||||
_MAX_SHEET_ROWS = 200
|
||||
_WRITE_INTERVAL_SECONDS = 0.25
|
||||
|
||||
|
||||
class InventorySpreadsheetError(RuntimeError):
|
||||
"""Safe-to-log error for invalid spreadsheet configuration or state."""
|
||||
|
||||
|
||||
def _load_config() -> dict[str, Any]:
|
||||
try:
|
||||
config = json.loads(_CONFIG_PATH.read_text(encoding="utf-8"))
|
||||
except (OSError, UnicodeError, json.JSONDecodeError) as exc:
|
||||
raise InventorySpreadsheetError(
|
||||
"Unable to read the public inventory workflow configuration"
|
||||
) from exc
|
||||
if not isinstance(config, dict):
|
||||
raise InventorySpreadsheetError(
|
||||
"Public inventory configuration must be an object"
|
||||
)
|
||||
|
||||
target = config.get("target")
|
||||
if not isinstance(target, dict):
|
||||
raise InventorySpreadsheetError(
|
||||
"Public inventory configuration has no spreadsheet target"
|
||||
)
|
||||
for key in (
|
||||
"spreadsheet_token",
|
||||
"sheet_id",
|
||||
"sheet_name",
|
||||
"sku_column",
|
||||
"inventory_column",
|
||||
"product_code_column",
|
||||
):
|
||||
if not str(target.get(key) or "").strip():
|
||||
raise InventorySpreadsheetError(
|
||||
f"Spreadsheet target is missing {key}"
|
||||
)
|
||||
for key in ("first_data_row", "last_data_row", "expected_sku_count"):
|
||||
value = target.get(key)
|
||||
if isinstance(value, bool) or not isinstance(value, int) or value < 1:
|
||||
raise InventorySpreadsheetError(
|
||||
f"Spreadsheet target has an invalid {key}"
|
||||
)
|
||||
if target["last_data_row"] - target["first_data_row"] + 1 != target[
|
||||
"expected_sku_count"
|
||||
]:
|
||||
raise InventorySpreadsheetError(
|
||||
"Spreadsheet target rows do not match the frozen SKU count"
|
||||
)
|
||||
|
||||
sku_items = config.get("skus")
|
||||
if not isinstance(sku_items, list) or not sku_items:
|
||||
raise InventorySpreadsheetError(
|
||||
"Public inventory configuration has no SKUs"
|
||||
)
|
||||
if len(sku_items) != target["expected_sku_count"]:
|
||||
raise InventorySpreadsheetError(
|
||||
"Configured SKU count does not match the spreadsheet target"
|
||||
)
|
||||
seen_skus: set[str] = set()
|
||||
seen_product_codes: set[str] = set()
|
||||
for item in sku_items:
|
||||
if not isinstance(item, dict):
|
||||
raise InventorySpreadsheetError(
|
||||
"Public inventory SKU metadata is invalid"
|
||||
)
|
||||
sku = str(item.get("sku") or "").strip()
|
||||
product_code = str(item.get("jushuitan_sku_id") or "").strip()
|
||||
if not sku or sku in seen_skus:
|
||||
raise InventorySpreadsheetError(
|
||||
"Public inventory configuration contains a blank or duplicate SKU"
|
||||
)
|
||||
if not product_code or product_code in seen_product_codes:
|
||||
raise InventorySpreadsheetError(
|
||||
"Public inventory configuration contains a blank or duplicate "
|
||||
"Jushuitan product code"
|
||||
)
|
||||
seen_skus.add(sku)
|
||||
seen_product_codes.add(product_code)
|
||||
return config
|
||||
|
||||
|
||||
def _sheet_args(config: Mapping[str, Any]) -> list[str]:
|
||||
target = config["target"]
|
||||
return [
|
||||
"--as",
|
||||
"user",
|
||||
"--spreadsheet-token",
|
||||
str(target["spreadsheet_token"]),
|
||||
"--sheet-id",
|
||||
str(target["sheet_id"]),
|
||||
"--format",
|
||||
"json",
|
||||
]
|
||||
|
||||
|
||||
def _run_sheets(args: list[str], *, operation: str) -> dict[str, Any]:
|
||||
try:
|
||||
return run_lark_cli(
|
||||
args,
|
||||
cwd=_REPOSITORY_ROOT,
|
||||
timeout=180,
|
||||
)
|
||||
except Exception as exc:
|
||||
raise InventorySpreadsheetError(
|
||||
f"Feishu spreadsheet {operation} failed"
|
||||
) from exc
|
||||
|
||||
|
||||
def _cell_value(cell: object) -> object:
|
||||
if not isinstance(cell, Mapping):
|
||||
raise InventorySpreadsheetError(
|
||||
"Feishu spreadsheet returned an invalid cell"
|
||||
)
|
||||
return cell.get("value")
|
||||
|
||||
|
||||
def _read_sheet_rows(
|
||||
config: Mapping[str, Any],
|
||||
sku_items: list[Mapping[str, Any]],
|
||||
) -> dict[str, dict[str, Any]]:
|
||||
target = config["target"]
|
||||
result = _run_sheets(
|
||||
[
|
||||
"sheets",
|
||||
"+cells-get",
|
||||
*_sheet_args(config),
|
||||
"--range",
|
||||
f"{target['sku_column']}1:{target['product_code_column']}{_MAX_SHEET_ROWS}",
|
||||
"--include",
|
||||
"value",
|
||||
],
|
||||
operation="cells-get",
|
||||
)
|
||||
data = result.get("data")
|
||||
ranges = data.get("ranges") if isinstance(data, Mapping) else None
|
||||
if (
|
||||
not isinstance(data, Mapping)
|
||||
or data.get("has_more") is not False
|
||||
or not isinstance(ranges, list)
|
||||
or len(ranges) != 1
|
||||
):
|
||||
raise InventorySpreadsheetError(
|
||||
"Feishu spreadsheet response is incomplete or invalid"
|
||||
)
|
||||
|
||||
sheet_range = ranges[0]
|
||||
if not isinstance(sheet_range, Mapping):
|
||||
raise InventorySpreadsheetError(
|
||||
"Feishu spreadsheet range response is invalid"
|
||||
)
|
||||
cells = sheet_range.get("cells")
|
||||
row_indices = sheet_range.get("row_indices")
|
||||
col_indices = sheet_range.get("col_indices")
|
||||
required_columns = {
|
||||
str(target["sku_column"]),
|
||||
str(target["inventory_column"]),
|
||||
str(target["product_code_column"]),
|
||||
}
|
||||
if (
|
||||
not isinstance(cells, list)
|
||||
or not isinstance(row_indices, list)
|
||||
or not isinstance(col_indices, list)
|
||||
or len(cells) != len(row_indices)
|
||||
or not required_columns.issubset({str(col) for col in col_indices})
|
||||
):
|
||||
raise InventorySpreadsheetError(
|
||||
"Feishu spreadsheet response does not cover the configured columns"
|
||||
)
|
||||
column_positions = {str(col): index for index, col in enumerate(col_indices)}
|
||||
row_values: dict[int, list[object]] = {}
|
||||
for row_number, values in zip(row_indices, cells, strict=True):
|
||||
if (
|
||||
isinstance(row_number, bool)
|
||||
or not isinstance(row_number, int)
|
||||
or not isinstance(values, list)
|
||||
or len(values) != len(col_indices)
|
||||
):
|
||||
raise InventorySpreadsheetError(
|
||||
"Feishu spreadsheet returned malformed row coordinates"
|
||||
)
|
||||
row_values[row_number] = values
|
||||
|
||||
first_row = int(target["first_data_row"])
|
||||
last_row = int(target["last_data_row"])
|
||||
header = row_values.get(1)
|
||||
if header is None:
|
||||
raise InventorySpreadsheetError(
|
||||
"Feishu spreadsheet is missing its header row"
|
||||
)
|
||||
expected_headers = {
|
||||
str(target["sku_column"]): "sku",
|
||||
str(target["inventory_column"]): "仓库现有库存",
|
||||
str(target["product_code_column"]): "商品编码(光影)",
|
||||
}
|
||||
for column, expected_header in expected_headers.items():
|
||||
actual_header = str(
|
||||
_cell_value(header[column_positions[column]]) or ""
|
||||
).strip()
|
||||
if actual_header != expected_header:
|
||||
raise InventorySpreadsheetError(
|
||||
f"Feishu spreadsheet header in column {column} does not match"
|
||||
)
|
||||
|
||||
rows: dict[str, dict[str, Any]] = {}
|
||||
for row_number in range(first_row, last_row + 1):
|
||||
values = row_values.get(row_number)
|
||||
if values is None:
|
||||
raise InventorySpreadsheetError(
|
||||
"Feishu spreadsheet is missing an expected inventory row"
|
||||
)
|
||||
sku = str(
|
||||
_cell_value(values[column_positions[str(target["sku_column"])]]) or ""
|
||||
).strip()
|
||||
product_code = str(
|
||||
_cell_value(
|
||||
values[column_positions[str(target["product_code_column"])] ]
|
||||
)
|
||||
or ""
|
||||
).strip()
|
||||
if not sku or sku in rows:
|
||||
raise InventorySpreadsheetError(
|
||||
"Feishu spreadsheet contains a blank or duplicate SKU"
|
||||
)
|
||||
rows[sku] = {
|
||||
"row": row_number,
|
||||
"product_code": product_code,
|
||||
}
|
||||
|
||||
expected_by_sku = {
|
||||
str(item["sku"]).strip(): str(item["jushuitan_sku_id"]).strip()
|
||||
for item in sku_items
|
||||
}
|
||||
if set(rows) != set(expected_by_sku):
|
||||
raise InventorySpreadsheetError(
|
||||
"Feishu spreadsheet SKU rows differ from the frozen workbook list"
|
||||
)
|
||||
for sku, row in rows.items():
|
||||
if row["product_code"] != expected_by_sku[sku]:
|
||||
raise InventorySpreadsheetError(
|
||||
f"Feishu spreadsheet product code does not match SKU {sku} "
|
||||
f"at row {row['row']}"
|
||||
)
|
||||
return rows
|
||||
|
||||
|
||||
def _read_written_inventory(
|
||||
config: Mapping[str, Any],
|
||||
expected: list[object],
|
||||
) -> None:
|
||||
target = config["target"]
|
||||
column = str(target["inventory_column"])
|
||||
first_row = int(target["first_data_row"])
|
||||
last_row = int(target["last_data_row"])
|
||||
result = _run_sheets(
|
||||
[
|
||||
"sheets",
|
||||
"+cells-get",
|
||||
*_sheet_args(config),
|
||||
"--range",
|
||||
f"{column}{first_row}:{column}{last_row}",
|
||||
"--include",
|
||||
"value",
|
||||
],
|
||||
operation="inventory readback",
|
||||
)
|
||||
data = result.get("data")
|
||||
ranges = data.get("ranges") if isinstance(data, Mapping) else None
|
||||
if (
|
||||
not isinstance(data, Mapping)
|
||||
or data.get("has_more") is not False
|
||||
or not isinstance(ranges, list)
|
||||
or len(ranges) != 1
|
||||
):
|
||||
raise InventorySpreadsheetError(
|
||||
"Feishu inventory readback is incomplete or invalid"
|
||||
)
|
||||
sheet_range = ranges[0]
|
||||
if not isinstance(sheet_range, Mapping):
|
||||
raise InventorySpreadsheetError(
|
||||
"Feishu inventory readback range is invalid"
|
||||
)
|
||||
cells = sheet_range.get("cells")
|
||||
row_indices = sheet_range.get("row_indices")
|
||||
if (
|
||||
not isinstance(cells, list)
|
||||
or not isinstance(row_indices, list)
|
||||
or row_indices != list(range(first_row, last_row + 1))
|
||||
or len(cells) != len(expected)
|
||||
):
|
||||
raise InventorySpreadsheetError(
|
||||
"Feishu inventory readback does not cover every target row"
|
||||
)
|
||||
actual: list[object] = []
|
||||
for row in cells:
|
||||
if not isinstance(row, list) or len(row) != 1:
|
||||
raise InventorySpreadsheetError(
|
||||
"Feishu inventory readback has an invalid cell shape"
|
||||
)
|
||||
actual.append(_cell_value(row[0]))
|
||||
for offset, (actual_value, expected_value) in enumerate(
|
||||
zip(actual, expected, strict=True)
|
||||
):
|
||||
if expected_value == "":
|
||||
matched = actual_value in (None, "")
|
||||
else:
|
||||
try:
|
||||
matched = int(str(actual_value).strip()) == int(expected_value)
|
||||
except (TypeError, ValueError):
|
||||
matched = False
|
||||
if not matched:
|
||||
raise InventorySpreadsheetError(
|
||||
"Feishu inventory readback did not match the submitted value "
|
||||
f"at row {first_row + offset}"
|
||||
)
|
||||
|
||||
|
||||
def _write_inventory(
|
||||
config: Mapping[str, Any],
|
||||
sku_items: list[Mapping[str, Any]],
|
||||
inventories: Mapping[str, InventoryRecord],
|
||||
rows: Mapping[str, Mapping[str, Any]],
|
||||
) -> tuple[int, bool, int]:
|
||||
target = config["target"]
|
||||
first_row = int(target["first_data_row"])
|
||||
last_row = int(target["last_data_row"])
|
||||
by_row = sorted(rows.items(), key=lambda entry: int(entry[1]["row"]))
|
||||
if [int(row["row"]) for _, row in by_row] != list(
|
||||
range(first_row, last_row + 1)
|
||||
):
|
||||
raise InventorySpreadsheetError(
|
||||
"Feishu spreadsheet SKU rows are not a contiguous target range"
|
||||
)
|
||||
sku_config = {str(item["sku"]): item for item in sku_items}
|
||||
expected_values: list[object] = []
|
||||
missing_count = 0
|
||||
for sku, _ in by_row:
|
||||
product_code = str(sku_config[sku]["jushuitan_sku_id"])
|
||||
inventory = inventories.get(product_code)
|
||||
if inventory is None:
|
||||
# An omitted Jushuitan row is unknown inventory, so clear stale data.
|
||||
expected_values.append("")
|
||||
missing_count += 1
|
||||
else:
|
||||
expected_values.append(inventory.public_available_qty)
|
||||
|
||||
cells = [[{"value": value}] for value in expected_values]
|
||||
result = _run_sheets(
|
||||
[
|
||||
"sheets",
|
||||
"+cells-set",
|
||||
*_sheet_args(config),
|
||||
"--range",
|
||||
f"{target['inventory_column']}{first_row}:"
|
||||
f"{target['inventory_column']}{last_row}",
|
||||
"--cells",
|
||||
json.dumps(cells, ensure_ascii=False, separators=(",", ":")),
|
||||
],
|
||||
operation="inventory update",
|
||||
)
|
||||
if result.get("acceptance_skipped") is True:
|
||||
return 0, True, missing_count
|
||||
_read_written_inventory(config, expected_values)
|
||||
return len(expected_values), False, missing_count
|
||||
|
||||
|
||||
def run() -> dict[str, object]:
|
||||
config = _load_config()
|
||||
sku_items = config["skus"]
|
||||
jushuitan_sku_ids = [str(item["jushuitan_sku_id"]) for item in sku_items]
|
||||
|
||||
# Validate the complete target and ERP response before modifying Feishu.
|
||||
rows = _read_sheet_rows(config, sku_items)
|
||||
inventories = JushuitanInventoryClient().query(jushuitan_sku_ids)
|
||||
if not inventories:
|
||||
raise InventorySpreadsheetError(
|
||||
"Jushuitan returned no inventory rows for the frozen product-code list; "
|
||||
"refusing to clear the Feishu inventory column"
|
||||
)
|
||||
written, acceptance_skipped, inventory_missing = _write_inventory(
|
||||
config,
|
||||
sku_items,
|
||||
inventories,
|
||||
rows,
|
||||
)
|
||||
return {
|
||||
"workflow_id": _WORKFLOW_ID,
|
||||
"target_sheet": str(config["target"]["sheet_name"]),
|
||||
"sku_count": len(sku_items),
|
||||
"inventory_found": len(inventories),
|
||||
"inventory_missing": inventory_missing,
|
||||
"cells_written": written,
|
||||
"acceptance_skipped": acceptance_skipped,
|
||||
"queried_at": datetime.now(_TIMEZONE).strftime("%Y-%m-%d %H:%M:%S"),
|
||||
}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
try:
|
||||
print(json.dumps(run(), ensure_ascii=False, separators=(",", ":")))
|
||||
return 0
|
||||
except Exception as exc:
|
||||
print(
|
||||
json.dumps(
|
||||
{"workflow_id": _WORKFLOW_ID, "error": str(exc)},
|
||||
ensure_ascii=False,
|
||||
separators=(",", ":"),
|
||||
),
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -29,15 +29,19 @@ from runtime_paths import TMP_ROOT
|
||||
|
||||
from gyxx_flow.adapters.browser import BrowserCookieStore
|
||||
from gyxx_flow.adapters.scrapling import ScraplingBrowser, is_browser_timeout_error
|
||||
from gyxx_flow.core.config import Settings
|
||||
from gyxx_flow.core.locks import LockManager, ResourceBusyError
|
||||
|
||||
_CONFIG_NAME = "tmall-baibu.json"
|
||||
_LOGIN_PAGE_URL = "https://login.taobao.com/member/login.jhtml"
|
||||
_EXPORT_TIMEOUT_SECONDS = 240
|
||||
_EXPORT_MAX_ATTEMPTS = 3
|
||||
_EXPORT_RETRY_DELAYS_SECONDS = (5, 15)
|
||||
_PAGE_TIMEOUT_MS = 60_000
|
||||
_IMPORT_TIMEOUT_SECONDS = 300
|
||||
_IMPORT_ACCEPTED_AFTER_PROCESSING_SECONDS = 120
|
||||
_STATUS_SETTLE_TIMEOUT_SECONDS = 5
|
||||
_ACCOUNT_LOCK_TIMEOUT_SECONDS = 120
|
||||
_RESULT_REFRESH_INTERVAL_SECONDS = 8
|
||||
_DRAFT_ACTION_TIMEOUT_SECONDS = 30
|
||||
_DRAFT_NAVIGATION_TIMEOUT_SECONDS = 15
|
||||
@@ -103,6 +107,7 @@ _LOGIN_SECOND_FACTOR_MARKERS = (
|
||||
"安全验证",
|
||||
"二次验证",
|
||||
"验证码",
|
||||
"滑块",
|
||||
)
|
||||
|
||||
|
||||
@@ -805,6 +810,41 @@ def _find_password_login_form(
|
||||
return locate()
|
||||
|
||||
|
||||
def _is_login_agreement_visible(page: Any) -> bool:
|
||||
"""Return whether Tmall's 服务协议及隐私保护 popup is blocking the login."""
|
||||
|
||||
for context in _page_contexts(page):
|
||||
get_by_role = getattr(context, "get_by_role", None)
|
||||
if not callable(get_by_role):
|
||||
continue
|
||||
try:
|
||||
dialogs = get_by_role("dialog")
|
||||
for index in range(min(dialogs.count(), 8)):
|
||||
dialog = dialogs.nth(index)
|
||||
try:
|
||||
if not dialog.is_visible():
|
||||
continue
|
||||
text = _normalized_page_text(dialog.inner_text(timeout=2_000))
|
||||
except Exception:
|
||||
continue
|
||||
if "服务协议" in text or "隐私" in text:
|
||||
return True
|
||||
except Exception:
|
||||
continue
|
||||
return False
|
||||
|
||||
|
||||
def _click_login_agreement_accept(page: Any) -> bool:
|
||||
for context in _page_contexts(page):
|
||||
for candidate in _visible_text_candidates(context, "同意"):
|
||||
try:
|
||||
candidate.click(timeout=8_000)
|
||||
return True
|
||||
except Exception:
|
||||
continue
|
||||
return False
|
||||
|
||||
|
||||
def _login_if_needed(
|
||||
entry: BaibuEntry,
|
||||
page: Any,
|
||||
@@ -817,6 +857,11 @@ def _login_if_needed(
|
||||
return False
|
||||
login_url = _is_login_url(str(getattr(page, "url", "")))
|
||||
form = _find_password_login_form(page, use_password_form=False)
|
||||
if form is None and not login_url:
|
||||
# A redirect can land on a scanner-tab seller login page (qianniu)
|
||||
# whose URL carries no login marker; probe the password tab before
|
||||
# deciding there is nothing to log in to.
|
||||
form = _find_password_login_form(page, use_password_form=True)
|
||||
if not login_url and form is None:
|
||||
return False
|
||||
|
||||
@@ -852,10 +897,92 @@ def _login_if_needed(
|
||||
raise BaibuApplyError(
|
||||
f"{entry.name} 自动填写账号密码并提交登录失败({type(exc).__name__})"
|
||||
) from exc
|
||||
# The first password submit can pop the 服务协议及隐私保护 confirmation
|
||||
# which swallows the submit; accept it and submit once more so the login
|
||||
# actually runs. Both popups and form re-renders are tolerated.
|
||||
for _attempt in range(2):
|
||||
wait_for_timeout = getattr(page, "wait_for_timeout", None)
|
||||
if callable(wait_for_timeout):
|
||||
wait_for_timeout(2_000)
|
||||
else:
|
||||
time.sleep(2)
|
||||
if not _is_login_agreement_visible(page):
|
||||
break
|
||||
if not _click_login_agreement_accept(page):
|
||||
break
|
||||
print(f"[百补] {entry.name} 已确认登录服务协议弹窗,重新提交登录")
|
||||
refreshed = _find_password_login_form(page, use_password_form=True)
|
||||
if refreshed is None:
|
||||
break
|
||||
_context, username, secret_input, submit = refreshed
|
||||
try:
|
||||
username.fill(credentials.username)
|
||||
secret_input.fill(credentials.password)
|
||||
submit.click(timeout=15_000)
|
||||
except Exception as exc:
|
||||
raise BaibuApplyError(
|
||||
f"{entry.name} 协议确认后重新提交登录失败({type(exc).__name__})"
|
||||
) from exc
|
||||
print(f"[百补] {entry.name} 已提交密码登录,等待店铺身份确认")
|
||||
return True
|
||||
|
||||
|
||||
def _goto_tolerant(page: Any, url: str) -> None:
|
||||
"""Navigate, tolerating an aborted load when a redirect takes over.
|
||||
|
||||
An unauthenticated apply URL is answered with a redirect chain that can
|
||||
abort the original navigation from inside the browser; treat that as the
|
||||
navigation landing wherever the chain pointed and let it settle.
|
||||
"""
|
||||
|
||||
try:
|
||||
page.goto(url, wait_until="domcontentloaded", timeout=_PAGE_TIMEOUT_MS)
|
||||
except Exception as exc:
|
||||
if "ERR_ABORTED" not in str(exc):
|
||||
raise
|
||||
wait_for_timeout = getattr(page, "wait_for_timeout", None)
|
||||
if callable(wait_for_timeout):
|
||||
wait_for_timeout(_LOGIN_FORM_SETTLE_MILLISECONDS)
|
||||
else:
|
||||
time.sleep(_LOGIN_FORM_SETTLE_MILLISECONDS / 1000)
|
||||
|
||||
|
||||
def _explicit_password_login(
|
||||
entry: BaibuEntry,
|
||||
page: Any,
|
||||
*,
|
||||
manual_login: bool,
|
||||
) -> None:
|
||||
"""Drive the Tmall password form when the apply page hid its login state.
|
||||
|
||||
The myseller/qn apply SPA can sit on an SSO interstitial that never shows
|
||||
a login URL, or render the cached shop name on an expired session. In
|
||||
both cases ``_login_if_needed`` sees nothing to do. Re-entering the apply
|
||||
URL redirects to the seller login page (qianniu) which carries the right
|
||||
merchant context; only when that shows no form fall back to the plain
|
||||
login page. Login there, return to the apply URL, and confirm identity.
|
||||
"""
|
||||
|
||||
print(f"[百补] {entry.name} 显式重新进入报名页并走密码登录")
|
||||
_goto_tolerant(page, entry.apply_url)
|
||||
wait_for_timeout = getattr(page, "wait_for_timeout", None)
|
||||
if callable(wait_for_timeout):
|
||||
wait_for_timeout(_LOGIN_FORM_SETTLE_MILLISECONDS * 3)
|
||||
else:
|
||||
time.sleep(_LOGIN_FORM_SETTLE_MILLISECONDS / 1000 * 3)
|
||||
login_attempted = _login_if_needed(entry, page, manual_login=manual_login)
|
||||
if not login_attempted:
|
||||
_goto_tolerant(page, _LOGIN_PAGE_URL)
|
||||
login_attempted = _login_if_needed(entry, page, manual_login=manual_login)
|
||||
_goto_tolerant(page, entry.apply_url)
|
||||
_wait_for_authenticated(
|
||||
entry,
|
||||
page,
|
||||
login_attempted=login_attempted,
|
||||
manual_login=manual_login,
|
||||
)
|
||||
|
||||
|
||||
def _click_text(page: Any, labels: Iterable[str]) -> bool:
|
||||
contexts = _page_contexts(page)
|
||||
for label in labels:
|
||||
@@ -1040,7 +1167,10 @@ def _open_batch_import(
|
||||
f"{entry.name} 页面已有批量导入操作正在处理,已停止,避免重复提交"
|
||||
)
|
||||
time.sleep(0.5)
|
||||
raise BaibuApplyError(f"{entry.name} 页面没有找到可用的「商品批量导入」入口")
|
||||
raise BaibuApplyError(
|
||||
f"{entry.name} 页面没有找到可用的「商品批量导入」入口"
|
||||
"(活动可能已报名/已结束或会话无效,请人工打开报名页确认)"
|
||||
)
|
||||
|
||||
|
||||
def _wait_import_result(
|
||||
@@ -1062,6 +1192,7 @@ def _wait_import_result(
|
||||
last_processing_log = 0.0
|
||||
last_refresh = 0.0
|
||||
observed_new_operation = False
|
||||
first_new_operation_at: float | None = None
|
||||
while time.monotonic() < deadline:
|
||||
text = _page_text(page)
|
||||
operation = _latest_batch_import_status(text)
|
||||
@@ -1070,6 +1201,8 @@ def _wait_import_result(
|
||||
or operation.signature != previous_operation.signature
|
||||
):
|
||||
observed_new_operation = True
|
||||
if first_new_operation_at is None:
|
||||
first_new_operation_at = time.monotonic()
|
||||
if operation.state == "success":
|
||||
print(
|
||||
f"[百补] {entry.name} 页面已完成导入处理,"
|
||||
@@ -1082,6 +1215,21 @@ def _wait_import_result(
|
||||
f"忽略行结果并按导入提交成功:{operation.detail}"
|
||||
)
|
||||
return
|
||||
if (
|
||||
observed_new_operation
|
||||
and first_new_operation_at is not None
|
||||
and time.monotonic() - first_new_operation_at
|
||||
>= _IMPORT_ACCEPTED_AFTER_PROCESSING_SECONDS
|
||||
):
|
||||
# Tmall has already acknowledged the batch ("执行了商品批量导入
|
||||
# 操作,正在处理中"); the submission itself is the business
|
||||
# effect, so a slow per-row summary must not fail the run.
|
||||
print(
|
||||
f"[百补] {entry.name} 天猫已受理批量导入,"
|
||||
f"{_IMPORT_ACCEPTED_AFTER_PROCESSING_SECONDS} 秒内未返回行结果,"
|
||||
"按导入提交成功,行结果以报名页面为准"
|
||||
)
|
||||
return
|
||||
if observed_new_operation and any(marker in text for marker in _SUCCESS_MARKERS):
|
||||
print(f"[百补] {entry.name} 页面已确认导入提交,忽略行结果")
|
||||
return
|
||||
@@ -1284,9 +1432,20 @@ def _complete_draft_products(page: Any, entry: BaibuEntry) -> None:
|
||||
if not entry.complete_drafts_after_import:
|
||||
return
|
||||
|
||||
if not _click_draft_tab(page):
|
||||
count = _draft_count(_page_text(page))
|
||||
if count is None and not _has_visible_text(page, ("完善商品",)):
|
||||
# Right after the import acknowledgement the SPA may still be hydrating;
|
||||
# the draft tab, the draft count and the per-row action rarely all appear
|
||||
# on the first probe, so retry briefly before declaring the page broken.
|
||||
deadline = time.monotonic() + _DRAFT_NAVIGATION_TIMEOUT_SECONDS
|
||||
while time.monotonic() < deadline:
|
||||
if _click_draft_tab(page):
|
||||
break
|
||||
text = _page_text(page)
|
||||
if _draft_count(text) is not None or _has_visible_text(page, ("完善商品",)):
|
||||
break
|
||||
if any(marker in text for marker in _DRAFT_EMPTY_MARKERS):
|
||||
break
|
||||
time.sleep(0.5)
|
||||
else:
|
||||
raise BaibuApplyError(f"{entry.name} 未找到草稿页入口")
|
||||
|
||||
draft_count, action_count = _wait_for_draft_state(page, entry)
|
||||
@@ -1439,6 +1598,8 @@ def _has_account_cookie(
|
||||
def _persist_account_state(
|
||||
browser: ScraplingBrowser,
|
||||
entry: BaibuEntry,
|
||||
*,
|
||||
session_lock_held: bool = False,
|
||||
) -> bool:
|
||||
"""Publish a verified login to the account vault without leaking cookies."""
|
||||
|
||||
@@ -1463,8 +1624,12 @@ def _persist_account_state(
|
||||
)
|
||||
cookies = _account_cookies(cookies, required_domains)
|
||||
if not _has_account_cookie(cookies, required_names):
|
||||
names = sorted({str(item.get("name", "")) for item in cookies})
|
||||
preview = "、".join(names[:12]) if names else "无"
|
||||
raise BaibuApplyError(
|
||||
f"{entry.name} 登录后未获得可用的账号 cookie,未更新账号登录态"
|
||||
f"{entry.name} 登录后未获得可用的账号 cookie"
|
||||
f"(域内 cookie:{preview};要求含:{'、'.join(required_names)}),"
|
||||
"未更新账号登录态"
|
||||
)
|
||||
|
||||
raw_state = context.storage_state()
|
||||
@@ -1472,23 +1637,33 @@ def _persist_account_state(
|
||||
filtered_state = _account_storage_state(state, cookies, required_domains)
|
||||
cookie_file, storage_state_file = paths
|
||||
store = BrowserCookieStore(cookie_file, storage_state_file)
|
||||
lock_root = cookie_file.parent.parent / "locks"
|
||||
|
||||
def _save() -> None:
|
||||
try:
|
||||
store.save_cookies(cookies)
|
||||
store.save_storage_state(filtered_state)
|
||||
except (OSError, ValueError, TypeError) as exc:
|
||||
raise BaibuApplyError(
|
||||
f"{entry.name} 保存账号登录态失败({type(exc).__name__})"
|
||||
) from exc
|
||||
|
||||
if session_lock_held:
|
||||
# The whole browser session already holds the scheduler-side
|
||||
# browser-account lock; re-acquiring here would self-deadlock.
|
||||
_save()
|
||||
else:
|
||||
lock_root = Settings.from_env().data_root / "state" / "locks"
|
||||
try:
|
||||
with LockManager(lock_root).acquire(
|
||||
f"browser-account:{entry.account_id}",
|
||||
owner=f"tmall-baibu:{entry.entry_id}",
|
||||
timeout_seconds=0,
|
||||
):
|
||||
store.save_cookies(cookies)
|
||||
store.save_storage_state(filtered_state)
|
||||
_save()
|
||||
except ResourceBusyError as exc:
|
||||
raise BaibuApplyError(
|
||||
f"{entry.name} 账号登录态正在被其他进程更新,未覆盖账号登录态"
|
||||
) from exc
|
||||
except (OSError, ValueError, TypeError) as exc:
|
||||
raise BaibuApplyError(
|
||||
f"{entry.name} 保存账号登录态失败({type(exc).__name__})"
|
||||
) from exc
|
||||
print(f"[百补] 已将 {entry.account_id} 的已验证登录态同步到账号 vault")
|
||||
return True
|
||||
|
||||
@@ -1498,6 +1673,33 @@ def _apply_entry(
|
||||
template: Path,
|
||||
*,
|
||||
manual_login: bool = False,
|
||||
) -> None:
|
||||
# Hold the scheduler-side account browser lock for the whole session.
|
||||
# The resident keepalive heartbeats the same profile on an interval and
|
||||
# its timeout-0 acquire skips busy accounts, so this lock is what stops
|
||||
# the heartbeat from killing the import mid-run (TargetClosedError /
|
||||
# silent page death). Members must use the same lock directory as the
|
||||
# scheduler (Settings.data_root/state/locks), not one derived from the
|
||||
# vault paths.
|
||||
locks = LockManager(Settings.from_env().data_root / "state" / "locks")
|
||||
try:
|
||||
with locks.acquire(
|
||||
f"browser-account:{entry.account_id}",
|
||||
owner=f"tmall-baibu:{entry.entry_id}",
|
||||
timeout_seconds=_ACCOUNT_LOCK_TIMEOUT_SECONDS,
|
||||
):
|
||||
_apply_entry_locked(entry, template, manual_login=manual_login)
|
||||
except ResourceBusyError as exc:
|
||||
raise BaibuApplyError(
|
||||
f"{entry.name} 账号浏览器正被保活/登录占用,请稍后重试"
|
||||
) from exc
|
||||
|
||||
|
||||
def _apply_entry_locked(
|
||||
entry: BaibuEntry,
|
||||
template: Path,
|
||||
*,
|
||||
manual_login: bool = False,
|
||||
) -> None:
|
||||
browser = _start_browser()
|
||||
try:
|
||||
@@ -1516,14 +1718,36 @@ def _apply_entry(
|
||||
page,
|
||||
manual_login=manual_login,
|
||||
)
|
||||
try:
|
||||
_wait_for_authenticated(
|
||||
entry,
|
||||
page,
|
||||
login_attempted=login_attempted,
|
||||
manual_login=manual_login,
|
||||
)
|
||||
except BaibuApplyError:
|
||||
if login_attempted or manual_login:
|
||||
raise
|
||||
# The apply page may sit on an SSO interstitial without a login
|
||||
# URL, so the passive check above never triggers the credentials;
|
||||
# fall back to the explicit password login once.
|
||||
_explicit_password_login(entry, page, manual_login=manual_login)
|
||||
login_attempted = True
|
||||
browser.persist_bound_state()
|
||||
_persist_account_state(browser, entry)
|
||||
try:
|
||||
_persist_account_state(browser, entry, session_lock_held=True)
|
||||
except BaibuApplyError as exc:
|
||||
# A failed vault refresh can also mean the shop name was rendered
|
||||
# from cache on top of a guest session; retry once through the
|
||||
# explicit login before giving up on the vault refresh. The
|
||||
# refresh only caches this session for later runs and must never
|
||||
# block the import submission itself.
|
||||
print(f"[WARN] {exc};尝试显式重新登录后重试", file=sys.stderr)
|
||||
try:
|
||||
_explicit_password_login(entry, page, manual_login=manual_login)
|
||||
_persist_account_state(browser, entry, session_lock_held=True)
|
||||
except BaibuApplyError as retry_exc:
|
||||
print(f"[WARN] {retry_exc};本次继续执行报名导入", file=sys.stderr)
|
||||
print(f"[百补] 已确认账号 {entry.account_id},打开 {entry.name} 报名入口")
|
||||
previous_operation = _open_batch_import(page, entry)
|
||||
time.sleep(1)
|
||||
|
||||
@@ -503,7 +503,8 @@ def load_tm_item_ids_for_styles(style_names: List[str]) -> Dict[str, List[str]]:
|
||||
"""
|
||||
SELECT style_name, tm_spus
|
||||
FROM dim_style
|
||||
WHERE tm_spus IS NOT NULL AND cardinality(tm_spus) > 0
|
||||
WHERE jsonb_typeof(tm_spus) = 'array'
|
||||
AND jsonb_array_length(tm_spus) > 0
|
||||
"""
|
||||
)
|
||||
rows = cur.fetchall()
|
||||
|
||||
Vendored
+12
-23
@@ -223,23 +223,16 @@ def _check_login_status(page) -> bool:
|
||||
|
||||
|
||||
def _ensure_login(page):
|
||||
print("\n" + "=" * 60)
|
||||
print("【请登录】请在弹出的浏览器窗口中完成登录")
|
||||
print("提示:扫码/短信/密码均可,登录成功后页面会自动跳转")
|
||||
print("=" * 60 + "\n")
|
||||
|
||||
for i in range(LOGIN_TIMEOUT):
|
||||
page = _active_page(page)
|
||||
if _check_login_status(page):
|
||||
print(f"[OK] Login success! URL: {page.url}")
|
||||
page.wait_for_timeout(3000)
|
||||
return page
|
||||
if i % 5 == 0:
|
||||
print(f"[WAIT] Waiting for login... {i}s / {LOGIN_TIMEOUT}s")
|
||||
time.sleep(1)
|
||||
|
||||
print(f"[ERR] Login timeout ({LOGIN_TIMEOUT}s)")
|
||||
return None
|
||||
# 登录统一由账号 vault 管理(gyxx accounts login douyin-shop),
|
||||
# 采集脚本不再弹出浏览器等待人工扫码/短信登录。
|
||||
print(
|
||||
"[COOKIE_EXPIRED] 抖音罗盘登录态已失效:"
|
||||
"请执行 uv run gyxx accounts login douyin-shop 刷新账号登录态"
|
||||
)
|
||||
raise RuntimeError(
|
||||
"抖音罗盘登录态已失效;请执行 uv run gyxx accounts login douyin-shop "
|
||||
"刷新账号登录态后重跑"
|
||||
)
|
||||
|
||||
|
||||
def _select_menu_and_date(page) -> None:
|
||||
@@ -1927,12 +1920,8 @@ def _page_automation(page) -> None:
|
||||
|
||||
# 1. Check and wait for login
|
||||
if not _check_login_status(page):
|
||||
logged_page = _ensure_login(page)
|
||||
if not logged_page:
|
||||
print("[ERR] Login failed, will try save cookies anyway...")
|
||||
else:
|
||||
page = logged_page
|
||||
_maximize_page(page)
|
||||
# 登录态失效时 _ensure_login 会直接抛 RuntimeError
|
||||
_ensure_login(page)
|
||||
|
||||
# 2. Save state after login (best effort)
|
||||
_save_browser_state(page, "after login")
|
||||
|
||||
@@ -226,23 +226,16 @@ def ensure_login(page: Page, *, headless: bool, login_timeout: int) -> None:
|
||||
print("[SKIPPED_COOKIE] 抖店会话跳转到了登录页")
|
||||
raise DouyinSessionExpiredError("抖店会话跳转到了登录页")
|
||||
|
||||
if headless:
|
||||
raise RuntimeError(
|
||||
"抖音 Cookie 已失效且当前为无头模式,无法扫码登录;"
|
||||
"请先执行 uv run gyxx accounts login douyin-shop 刷新账号登录态"
|
||||
# 登录统一由账号 vault 管理(gyxx accounts login douyin-shop),
|
||||
# 采集脚本不再弹出浏览器等待人工扫码/验证码。
|
||||
print(
|
||||
"[COOKIE_EXPIRED] 抖店 Cookie 已失效:"
|
||||
"请执行 uv run gyxx accounts login douyin-shop 刷新账号登录态"
|
||||
)
|
||||
raise DouyinSessionExpiredError(
|
||||
"抖音 Cookie 已失效;请执行 uv run gyxx accounts login douyin-shop "
|
||||
"刷新账号登录态后重跑"
|
||||
)
|
||||
|
||||
print("=" * 60)
|
||||
print("【请登录】请在弹出的浏览器窗口中完成抖店登录")
|
||||
print("=" * 60)
|
||||
deadline = time.monotonic() + login_timeout
|
||||
while time.monotonic() < deadline:
|
||||
if not douyin_common.is_login_page(page):
|
||||
print(f" [OK] 登录成功: {page.url}")
|
||||
page.wait_for_timeout(3_000)
|
||||
return
|
||||
page.wait_for_timeout(1_000)
|
||||
raise RuntimeError(f"登录在 {login_timeout} 秒内未完成,当前页面: {page.url}")
|
||||
|
||||
|
||||
class DouyinSessionExpiredError(RuntimeError):
|
||||
|
||||
+12
-23
@@ -262,23 +262,16 @@ def _check_login_status(page) -> bool:
|
||||
|
||||
|
||||
def _ensure_login(page):
|
||||
print("\n" + "=" * 60)
|
||||
print("【请登录】请在弹出的浏览器窗口中完成登录")
|
||||
print("提示:扫码/短信/密码均可,登录成功后页面会自动跳转")
|
||||
print("=" * 60 + "\n")
|
||||
|
||||
for i in range(LOGIN_TIMEOUT):
|
||||
page = _active_page(page)
|
||||
if _check_login_status(page):
|
||||
print(f"[OK] Login success! URL: {page.url}")
|
||||
page.wait_for_timeout(3000)
|
||||
return page
|
||||
if i % 5 == 0:
|
||||
print(f"[WAIT] Waiting for login... {i}s / {LOGIN_TIMEOUT}s")
|
||||
time.sleep(1)
|
||||
|
||||
print(f"[ERR] Login timeout ({LOGIN_TIMEOUT}s)")
|
||||
return None
|
||||
# 登录统一由账号 vault 管理(gyxx accounts login douyin-shop),
|
||||
# 采集脚本不再弹出浏览器等待人工扫码/短信登录。
|
||||
print(
|
||||
"[COOKIE_EXPIRED] 抖音罗盘登录态已失效:"
|
||||
"请执行 uv run gyxx accounts login douyin-shop 刷新账号登录态"
|
||||
)
|
||||
raise BrowserSessionExpiredError(
|
||||
"抖音罗盘登录态已失效;请执行 uv run gyxx accounts login douyin-shop "
|
||||
"刷新账号登录态后重跑"
|
||||
)
|
||||
|
||||
|
||||
def _select_menu_and_date(page) -> None:
|
||||
@@ -2325,12 +2318,8 @@ def _page_automation_impl(page) -> None:
|
||||
|
||||
# 1. Check and wait for login
|
||||
if not _check_login_status(page):
|
||||
logged_page = _ensure_login(page)
|
||||
if not logged_page:
|
||||
print("[ERR] Login failed, will try save cookies anyway...")
|
||||
else:
|
||||
page = logged_page
|
||||
_maximize_page(page)
|
||||
# If the login state is expired, _ensure_login will directly raise BrowserSessionExpiredError
|
||||
_ensure_login(page)
|
||||
|
||||
# 2. Save state after login (best effort)
|
||||
_save_browser_state(page, "after login")
|
||||
|
||||
+422
-294
@@ -32,10 +32,9 @@ from gyxx_flow.modules.shop_intelligence.browser_session import (
|
||||
)
|
||||
from gyxx_flow.modules.shop_intelligence.collectors.jd_data_collector import (
|
||||
close_profit_plan_popup_if_present,
|
||||
select_last_week_in_old_datepicker,
|
||||
)
|
||||
from gyxx_flow.modules.shop_intelligence.config import JD_PEER_STORE_OUTPUT
|
||||
from gyxx_flow.modules.shop_intelligence.paths import resolve_raw_output
|
||||
from gyxx_flow.modules.shop_intelligence.paths import TMP_ROOT, resolve_raw_output
|
||||
|
||||
# ── 配置 ─────────────────────────────────────────────────────────────────────
|
||||
|
||||
@@ -67,7 +66,11 @@ CATEGORY_OUTPUT_NAMES = {
|
||||
"男包": "男士包袋",
|
||||
}
|
||||
|
||||
VENDER_RANKS_URL = "https://jdsz.jd.com/szweb/view/industryMarket/venderRanks.html"
|
||||
# 2026-10 京东下线旧版商智:venderRanks 旧入口 404,官方公告要求迁移新版
|
||||
# jdsz.jd.com。新版等价页面是「行业榜单」的店铺榜 tab。
|
||||
INDUSTRY_TOP_URL = "https://jdsz.jd.com/szweb/view/industry/industry-top.html"
|
||||
# 兼容旧引用
|
||||
VENDER_RANKS_URL = INDUSTRY_TOP_URL
|
||||
|
||||
# 新品采集配置
|
||||
NEW_PRODUCT_SHOPS = [
|
||||
@@ -398,97 +401,262 @@ def get_last_week_range():
|
||||
return last_monday, last_sunday
|
||||
|
||||
|
||||
def _dump_peer_filter_debug(page: Page, tag: str) -> None:
|
||||
"""日期/类目操作失败时保存现场,便于按页面实际 DOM 校准选择器。"""
|
||||
try:
|
||||
stamp = datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||
debug_dir = TMP_ROOT / "debug" / "jd_peer_filter"
|
||||
debug_dir.mkdir(parents=True, exist_ok=True)
|
||||
page.screenshot(path=str(debug_dir / f"{stamp}_{tag}.png"))
|
||||
html_path = debug_dir / f"{stamp}_{tag}.html"
|
||||
html_path.write_text(page.content(), encoding="utf-8")
|
||||
print(f" [DEBUG] 现场已保存: {html_path}")
|
||||
except Exception as exc:
|
||||
print(f" [DEBUG] 现场保存失败: {exc}")
|
||||
|
||||
|
||||
def _click_custom_day_cell(page: Page, date_str: str) -> None:
|
||||
"""在自定义日期面板中点击指定日期格(按月份块消歧,真实坐标点击)。
|
||||
|
||||
面板为单月迷你日历:日格只带日号文本,须先定位「只含一个月份标题且
|
||||
含日历格」的月份块,再在其中匹配日号;组件对合成点击不稳定,统一用
|
||||
bounding_box 中心坐标做真实鼠标点击。
|
||||
"""
|
||||
center = page.evaluate(
|
||||
"""(dateStr) => {
|
||||
const [year, month, day] = dateStr.split('-');
|
||||
const wantTitle = `${Number(year)}年${String(Number(month)).padStart(2, '0')}月`;
|
||||
const dayText = String(Number(day));
|
||||
const isMonthTitle = (text) => /^\\d{4}年\\d{2}月$/.test((text || '').trim());
|
||||
const titleSpans = Array.from(document.querySelectorAll('span'))
|
||||
.filter(el => el.getClientRects().length && isMonthTitle(el.innerText));
|
||||
for (const span of titleSpans) {
|
||||
if ((span.innerText || '').trim() !== wantTitle) continue;
|
||||
let node = span.parentElement;
|
||||
for (let depth = 0; node && depth < 10; depth += 1, node = node.parentElement) {
|
||||
const titles = Array.from(node.querySelectorAll('span'))
|
||||
.filter(s => isMonthTitle(s.innerText));
|
||||
if (titles.length > 1) break;
|
||||
if (titles.length === 1
|
||||
&& node.querySelector('td.jmt-date-picker-calendar-cell')) {
|
||||
const cells = Array.from(
|
||||
node.querySelectorAll('td.jmt-date-picker-calendar-cell')
|
||||
).filter(el => el.getBoundingClientRect().width > 0
|
||||
&& (el.innerText || '').trim() === dayText);
|
||||
if (cells.length) {
|
||||
const rect = cells[0].getBoundingClientRect();
|
||||
return {x: Math.round(rect.x + rect.width / 2),
|
||||
y: Math.round(rect.y + rect.height / 2)};
|
||||
}
|
||||
return null;
|
||||
}
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}""",
|
||||
date_str,
|
||||
)
|
||||
if not center:
|
||||
_dump_peer_filter_debug(page, f"day_not_found_{date_str}")
|
||||
raise RuntimeError(f"自定义日期面板未找到日期格: {date_str}")
|
||||
page.mouse.click(center["x"], center["y"])
|
||||
|
||||
|
||||
def _open_custom_date_panel(page: Page, *, retries: int = 3) -> bool:
|
||||
"""点击「自定义」标签打开日历;带开面板验证与重试。
|
||||
|
||||
面板行为实测有两种形态且跨运行不定:迷你单月(选起点即自动关闭)
|
||||
与常驻双月(选完不关)。tag 点击在面板刚自动关闭后可能需要缓一下
|
||||
再点,故带月标题可见性验证与重试。
|
||||
"""
|
||||
tag = page.locator(
|
||||
"li.jmt-combo-date-picker-list-tag", has_text="自定义"
|
||||
).first
|
||||
for _attempt in range(retries):
|
||||
try:
|
||||
box = tag.bounding_box(timeout=5000)
|
||||
except Exception:
|
||||
box = None
|
||||
if not box:
|
||||
_dump_peer_filter_debug(page, "custom_tag_missing")
|
||||
raise RuntimeError("未找到「自定义」时间按钮")
|
||||
page.mouse.click(
|
||||
box["x"] + box["width"] / 2, box["y"] + box["height"] / 2
|
||||
)
|
||||
page.wait_for_timeout(2000)
|
||||
if _panel_months_visible(page):
|
||||
return True
|
||||
page.wait_for_timeout(1000)
|
||||
return _panel_months_visible(page)
|
||||
|
||||
|
||||
def _panel_months_visible(page) -> bool:
|
||||
try:
|
||||
return bool(
|
||||
page.evaluate(
|
||||
"() => Array.from(document.querySelectorAll('span'))"
|
||||
".some(el => el.getClientRects().length"
|
||||
" && /^\\d{4}年\\d{2}月$/.test((el.innerText || '').trim()))"
|
||||
)
|
||||
)
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
def _ensure_month_visible(page: Page, month_title: str) -> None:
|
||||
"""单月日历翻月:目标月份不可见时点「后一月」箭头(至多 12 次)。"""
|
||||
|
||||
def visible_months():
|
||||
return page.evaluate(
|
||||
"""() => Array.from(document.querySelectorAll('span'))
|
||||
.filter(el => el.getClientRects().length)
|
||||
.map(el => (el.innerText || '').trim())
|
||||
.filter(t => /^\\d{4}年\\d{2}月$/.test(t))"""
|
||||
)
|
||||
|
||||
for _ in range(12):
|
||||
if month_title in visible_months():
|
||||
return
|
||||
try:
|
||||
arrow = page.locator('[data-event-content="后一月"]')
|
||||
if arrow.count() == 0:
|
||||
return
|
||||
box = arrow.first.bounding_box(timeout=3000)
|
||||
except Exception:
|
||||
return
|
||||
if not box:
|
||||
return
|
||||
page.mouse.click(
|
||||
box["x"] + box["width"] / 2, box["y"] + box["height"] / 2
|
||||
)
|
||||
page.wait_for_timeout(1200)
|
||||
|
||||
|
||||
def _custom_panel_state(page) -> str:
|
||||
"""调试用:当前日期面板的可见月份与「当前:」显示。"""
|
||||
try:
|
||||
return str(
|
||||
page.evaluate(
|
||||
"""() => {
|
||||
const months = Array.from(document.querySelectorAll('span'))
|
||||
.filter(el => el.getClientRects().length)
|
||||
.map(el => (el.innerText || '').trim())
|
||||
.filter(t => /^\\d{4}年\\d{2}月$/.test(t));
|
||||
const current = (document.body.innerText.match(/当前:[^\\n]{0,40}/) || [''])[0];
|
||||
return {months: [...new Set(months)], current};
|
||||
}"""
|
||||
)
|
||||
)
|
||||
except Exception as exc:
|
||||
return f"<state-error {exc}>"
|
||||
|
||||
|
||||
def _custom_range_applied(page, start_mmdd: str) -> bool:
|
||||
try:
|
||||
body_text = page.locator("body").inner_text(timeout=3000)
|
||||
except Exception:
|
||||
return False
|
||||
return start_mmdd in body_text.replace(" ", "")
|
||||
|
||||
|
||||
def select_last_week_any_day(page):
|
||||
"""点击日期选择器 → 按周查询 → 点上周任意一天(自动选中整周)"""
|
||||
"""新版行业榜单:自定义区间选择上周一~上周日(状态机,3 轮重试)。
|
||||
|
||||
面板有两种形态且跨运行不定:
|
||||
- 迷你单月:点击起点后自动关闭;重开面板点终点,选完立即生效。
|
||||
- 常驻双月:起止选完面板不关;点面板外部(店铺搜索框)应用;
|
||||
Escape 会取消选择。
|
||||
统一状态机:选起点 → 面板仍开则同会话选终点,已关则重开选终点 →
|
||||
未生效则点外部应用 → 仍未生效整轮重来。
|
||||
"""
|
||||
print(" 选择上周时间范围...")
|
||||
last_monday, last_sunday = get_last_week_range()
|
||||
start_str = last_monday.strftime("%Y-%m-%d")
|
||||
end_str = last_sunday.strftime("%Y-%m-%d")
|
||||
start_month = f"{last_monday.year}年{last_monday.month:02d}月"
|
||||
end_month = f"{last_sunday.year}年{last_sunday.month:02d}月"
|
||||
start_mmdd = last_monday.strftime("%m-%d")
|
||||
|
||||
# 1. 点击日期选择器(显示"按天查询..."的元素)
|
||||
for attempt in range(1, 4):
|
||||
print(f" [state] 第 {attempt} 轮: {_custom_panel_state(page)}")
|
||||
if not _open_custom_date_panel(page):
|
||||
print(" [state] 面板未能打开,重试")
|
||||
continue
|
||||
_ensure_month_visible(page, start_month)
|
||||
try:
|
||||
date_picker = page.locator('.jmtd-date-picker .jmtd-base-input-top').first
|
||||
date_picker.wait_for(state="visible", timeout=5000)
|
||||
date_picker.click()
|
||||
print(" [OK] 点击日期选择器")
|
||||
except Exception as e:
|
||||
print(f" [WARN] 点击日期选择器失败: {e}")
|
||||
# 兜底:点击包含"按天查询"文本的元素
|
||||
_click_custom_day_cell(page, start_str)
|
||||
except RuntimeError:
|
||||
continue
|
||||
page.wait_for_timeout(1500)
|
||||
print(f" [state] 点击开始后: {_custom_panel_state(page)}")
|
||||
|
||||
if not _panel_months_visible(page):
|
||||
# 迷你面板形态:起点选择后面板已自动关闭,重开选终点
|
||||
page.wait_for_timeout(1200)
|
||||
if not _open_custom_date_panel(page):
|
||||
continue
|
||||
_ensure_month_visible(page, end_month)
|
||||
try:
|
||||
page.locator('text=按天查询').first.click()
|
||||
print(" [OK] 使用文本定位点击日期选择器")
|
||||
_click_custom_day_cell(page, end_str)
|
||||
except RuntimeError:
|
||||
continue
|
||||
page.wait_for_timeout(2500)
|
||||
print(f" [state] 点击结束后: {_custom_panel_state(page)}")
|
||||
|
||||
if _custom_range_applied(page, start_mmdd):
|
||||
print(f" [OK] 已选择上周区间: {start_str} ~ {end_str}")
|
||||
return
|
||||
# 常驻双月形态:点面板外部应用(Escape 是取消,绝不能用)
|
||||
try:
|
||||
page.locator(
|
||||
'input[placeholder="请输入店铺名称"]'
|
||||
).first.click(timeout=4000)
|
||||
except Exception:
|
||||
pass
|
||||
page.wait_for_timeout(2000)
|
||||
|
||||
# 2. 点击"按周查询"
|
||||
try:
|
||||
week_switch = page.locator('.jmtd-date-picker-switch-panel-item', has_text="按周查询").first
|
||||
week_switch.wait_for(state="visible", timeout=5000)
|
||||
week_switch.click()
|
||||
print(" [OK] 点击按周查询")
|
||||
except Exception as e:
|
||||
print(f" [WARN] 点击按周查询失败: {e}")
|
||||
# 兜底:用 JS 点击
|
||||
page.evaluate("""() => {
|
||||
const items = document.querySelectorAll('.jmtd-date-picker-switch-panel-item');
|
||||
for (const item of items) {
|
||||
if (item.textContent.includes('按周查询')) {
|
||||
item.click();
|
||||
return true;
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}""")
|
||||
page.wait_for_timeout(2000)
|
||||
|
||||
# 3. 等待日历弹窗并选择上周某一天
|
||||
calendar = page.locator('.jmtd-date-picker-popup, .jmtd-date-picker-panel, [class*="picker-panel"], [class*="calendar-panel"]')
|
||||
try:
|
||||
calendar.wait_for(state="visible", timeout=5000)
|
||||
except Exception:
|
||||
print(" [WARN] 未检测到日历弹窗,尝试继续...")
|
||||
page.mouse.click(30, 400)
|
||||
page.wait_for_timeout(3000)
|
||||
if _custom_range_applied(page, start_mmdd):
|
||||
print(f" [OK] 已选择上周区间(点外部应用): {start_str} ~ {end_str}")
|
||||
return
|
||||
print(f" [state] 本轮未生效: {_custom_panel_state(page)}")
|
||||
|
||||
select_last_week_in_old_datepicker(page, "peer_store")
|
||||
|
||||
try:
|
||||
calendar.wait_for(state="hidden", timeout=5000)
|
||||
except Exception:
|
||||
pass
|
||||
page.wait_for_timeout(5000)
|
||||
print(" 已切换到上周数据")
|
||||
_dump_peer_filter_debug(page, "range_not_applied")
|
||||
raise RuntimeError(
|
||||
f"自定义日期区间未生效: 期望 {start_str} ~ {end_str}"
|
||||
)
|
||||
|
||||
|
||||
def reselect_last_week(page):
|
||||
"""切换类目后重新选择上周日期(点击日期选择器 → 按周查询 → 点日期)"""
|
||||
# 重新点击日期选择器
|
||||
try:
|
||||
date_picker = page.locator('.jmtd-date-picker .jmtd-base-input-top').first
|
||||
date_picker.click()
|
||||
except Exception:
|
||||
page.locator('text=按天查询').first.click()
|
||||
page.wait_for_timeout(1500)
|
||||
|
||||
# 点击按周查询
|
||||
try:
|
||||
week_switch = page.locator('.jmtd-date-picker-switch-panel-item', has_text="按周查询").first
|
||||
week_switch.click()
|
||||
except Exception:
|
||||
page.evaluate("""() => {
|
||||
const items = document.querySelectorAll('.jmtd-date-picker-switch-panel-item');
|
||||
for (const item of items) {
|
||||
if (item.textContent.includes('按周查询')) {
|
||||
item.click();
|
||||
return true;
|
||||
|
||||
|
||||
|
||||
def _selector_option_center(page: Page, label: str, *, exclude_toolbar: bool = True):
|
||||
"""在下拉浮层里按精确文本定位 jmt-selector 选项,返回真实坐标。
|
||||
|
||||
筛选栏控件本身也是 button.jmt-selector(显示当前选中值),必须
|
||||
排除,否则会把控件当选项点到。
|
||||
"""
|
||||
return page.evaluate(
|
||||
"""(label) => {
|
||||
const visible = (el) => {
|
||||
const r = el.getBoundingClientRect();
|
||||
return r.width > 0 && r.height > 0;
|
||||
};
|
||||
const buttons = Array.from(
|
||||
document.querySelectorAll('div.jmt-selector')
|
||||
).filter(visible);
|
||||
for (const btn of buttons) {
|
||||
if (btn.closest('.industry-top-rank-toolbar')) continue;
|
||||
const lab = btn.querySelector('.jmt-label');
|
||||
if (lab && (lab.innerText || '').trim() === label) {
|
||||
const r = btn.getBoundingClientRect();
|
||||
return {x: Math.round(r.x + r.width / 2),
|
||||
y: Math.round(r.y + r.height / 2)};
|
||||
}
|
||||
}
|
||||
}""")
|
||||
page.wait_for_timeout(1500)
|
||||
|
||||
select_last_week_in_old_datepicker(page, "peer_store_reselect")
|
||||
page.wait_for_timeout(8000)
|
||||
|
||||
|
||||
|
||||
|
||||
return null;
|
||||
}""",
|
||||
label,
|
||||
)
|
||||
|
||||
|
||||
def switch_category(
|
||||
@@ -496,113 +664,174 @@ def switch_category(
|
||||
category_name: str,
|
||||
parent_category: str | None = None,
|
||||
) -> bool:
|
||||
"""切换商家榜单类目;只有选中态和榜单刷新均验证后才成功。"""
|
||||
"""新版行业榜单:jmt-selector 层级下拉选类目 → 点「查询」→ 校验表格。
|
||||
|
||||
一级选项直接点击;二级选项先 hover 一级展开子面板再点击。下拉的
|
||||
选项与筛选栏控件是同款 button.jmt-selector,靠排除工具栏区分。
|
||||
"""
|
||||
label = f"{parent_category} → {category_name}" if parent_category else category_name
|
||||
print(f" 切换类目到: {label}")
|
||||
expected_path = _expected_category_path(category_name, parent_category)
|
||||
selector = None
|
||||
# 类目下拉的触发行为极不稳定(单击/双击/坐标点击时开时不开),
|
||||
# 用多策略轮换:单击→轮询;双击→轮询;再单击→轮询,任何一刻
|
||||
# 选项出现即继续。
|
||||
trigger = page.locator(
|
||||
".jmt--form-item-main", has_text="类目"
|
||||
).first
|
||||
|
||||
def _dropdown_ready() -> bool:
|
||||
return bool(
|
||||
page.evaluate(
|
||||
"() => Array.from(document.querySelectorAll('div.jmt-selector'))"
|
||||
".some(el => el.getBoundingClientRect().width > 0"
|
||||
" && !el.closest('.industry-top-rank-toolbar'))"
|
||||
)
|
||||
)
|
||||
|
||||
opened = False
|
||||
for action in ("click", "dblclick", "click"):
|
||||
try:
|
||||
selector = _visible_category_selector(page)
|
||||
previous_fingerprint = _ranking_table_fingerprint(page)
|
||||
current_path = _selected_category_path(selector)
|
||||
if current_path == expected_path:
|
||||
if not previous_fingerprint:
|
||||
print(f" [WARN] 当前类目是 {label},但榜单表格为空")
|
||||
return False
|
||||
print(f" [OK] 当前已是 {label},榜单表格可用")
|
||||
return True
|
||||
|
||||
menu = _visible_category_menu(page, selector)
|
||||
_select_category_option(
|
||||
page,
|
||||
menu,
|
||||
category_name,
|
||||
parent_category,
|
||||
)
|
||||
|
||||
if not _wait_for_selected_category(selector, expected_path):
|
||||
actual_path = _selected_category_path(selector)
|
||||
print(
|
||||
f" [WARN] 类目选中态校验失败: "
|
||||
f"expected={expected_path}, actual={actual_path}"
|
||||
)
|
||||
return False
|
||||
if not _wait_for_ranking_table_refresh(page, previous_fingerprint):
|
||||
print(f" [WARN] 已选中 {label},但榜单表格未确认刷新")
|
||||
return False
|
||||
|
||||
print(f" [OK] 已切换到 {label},榜单表格已刷新")
|
||||
return True
|
||||
if action == "click":
|
||||
trigger.click(timeout=6000)
|
||||
else:
|
||||
trigger.dblclick(timeout=6000)
|
||||
except Exception as exc:
|
||||
print(f" [WARN] 切换类目失败: {exc}")
|
||||
print(f" [WARN] 类目控件{action}失败: {exc}")
|
||||
_dump_peer_filter_debug(page, "category_control_missing")
|
||||
return False
|
||||
finally:
|
||||
for _ in range(6):
|
||||
page.wait_for_timeout(500)
|
||||
if _dropdown_ready():
|
||||
opened = True
|
||||
break
|
||||
if opened:
|
||||
break
|
||||
if not opened:
|
||||
print(" [WARN] 类目下拉未能打开")
|
||||
_dump_peer_filter_debug(page, "category_dropdown_not_open")
|
||||
return False
|
||||
|
||||
selected = False
|
||||
if parent_category and parent_category != category_name:
|
||||
parent_center = _selector_option_center(page, parent_category)
|
||||
if parent_center:
|
||||
page.mouse.move(parent_center["x"], parent_center["y"])
|
||||
page.wait_for_timeout(1500)
|
||||
leaf_center = _selector_option_center(page, category_name)
|
||||
if leaf_center:
|
||||
page.mouse.click(leaf_center["x"], leaf_center["y"])
|
||||
selected = True
|
||||
else:
|
||||
option_center = _selector_option_center(page, category_name)
|
||||
if option_center:
|
||||
page.mouse.click(option_center["x"], option_center["y"])
|
||||
selected = True
|
||||
if not selected:
|
||||
print(f" [WARN] 未找到类目选项 {label}")
|
||||
_dump_peer_filter_debug(page, f"category_option_missing_{category_name}")
|
||||
try:
|
||||
menus = page.locator(
|
||||
".jmtd-dropdown-panel.jmtd-dropdown-lists:visible"
|
||||
)
|
||||
if menus.count() and selector is not None:
|
||||
selector.locator(
|
||||
".jmtd-base-input-top:visible"
|
||||
).click(timeout=3000)
|
||||
page.wait_for_timeout(300)
|
||||
if menus.count():
|
||||
page.keyboard.press("Escape")
|
||||
except Exception:
|
||||
pass
|
||||
return False
|
||||
page.wait_for_timeout(1500)
|
||||
|
||||
# 应用筛选并等待榜单刷新
|
||||
try:
|
||||
query_box = page.locator("button", has_text="查询").first.bounding_box(
|
||||
timeout=4000
|
||||
)
|
||||
if query_box:
|
||||
page.mouse.click(
|
||||
query_box["x"] + query_box["width"] / 2,
|
||||
query_box["y"] + query_box["height"] / 2,
|
||||
)
|
||||
except Exception:
|
||||
pass # 部分面板选择后立即生效
|
||||
page.wait_for_timeout(4500)
|
||||
|
||||
try:
|
||||
# 注意不能用共享 extract():它会把结果字符串化导致布尔失真
|
||||
table_present = bool(
|
||||
page.evaluate(
|
||||
"() => Array.from(document.querySelectorAll('table'))"
|
||||
".some(t => (t.innerText || '').includes('排名'))"
|
||||
)
|
||||
)
|
||||
except Exception:
|
||||
table_present = False
|
||||
if not table_present:
|
||||
print(" [WARN] 类目切换后未见榜单表格,禁止复用旧数据")
|
||||
_dump_peer_filter_debug(page, f"table_missing_{category_name}")
|
||||
return False
|
||||
print(f" [OK] 已切换到 {label},榜单表格已刷新")
|
||||
return True
|
||||
|
||||
|
||||
def extract_ranking_table(page):
|
||||
"""提取商家榜单表格数据"""
|
||||
ranking_data = extract(page, """() => {
|
||||
const rows = document.querySelectorAll('.jmtd-table-main tbody tr');
|
||||
"""提取店铺榜表格;按表头行映射列,字段名沿用旧口径。
|
||||
|
||||
新版列:排名/店铺信息/成交金额/成交单量/成交商品件数/访客数/浏览量/
|
||||
搜索点击次数;旧字段 followers(关注人数)新版已无此列,占位 '--'。
|
||||
"""
|
||||
try:
|
||||
# 直接 evaluate 取结构化列表(共享 extract() 会把结果字符串化)
|
||||
ranking_data = page.evaluate(
|
||||
"""() => {
|
||||
const table = Array.from(document.querySelectorAll('table'))
|
||||
.find(t => (t.innerText || '').includes('排名'));
|
||||
if (!table) return [];
|
||||
const trs = Array.from(table.querySelectorAll('tr'));
|
||||
let headers = null;
|
||||
const result = [];
|
||||
for (const row of rows) {
|
||||
const cells = row.querySelectorAll('td');
|
||||
if (cells.length < 7) continue;
|
||||
|
||||
// 排名提取:优先数字文本,其次检查 1/2/3 名图标 class
|
||||
let rank = '';
|
||||
const rankNumEl = cells[0].querySelector('.jmtd-table-formatter-rank-number');
|
||||
if (rankNumEl) {
|
||||
rank = rankNumEl.textContent.trim();
|
||||
for (const tr of trs) {
|
||||
const cells = Array.from(tr.querySelectorAll('th,td'))
|
||||
.map(c => (c.innerText || '').trim());
|
||||
if (!cells.length) continue;
|
||||
if (headers === null) {
|
||||
if (cells.includes('排名')) headers = cells;
|
||||
continue;
|
||||
}
|
||||
if (!rank) {
|
||||
const rankIconEl = cells[0].querySelector('.jmtd-table-formatter-rank');
|
||||
if (rankIconEl) {
|
||||
const iconMatch = rankIconEl.className.match(/jmtd-table-formatter-rank-icon-(\\d)/);
|
||||
if (iconMatch) {
|
||||
rank = iconMatch[1];
|
||||
}
|
||||
}
|
||||
}
|
||||
if (!rank) {
|
||||
rank = cells[0].textContent.trim();
|
||||
}
|
||||
|
||||
const shopLink = cells[1].querySelector('a');
|
||||
const shop = shopLink ? shopLink.textContent.trim() : cells[1].textContent.trim();
|
||||
|
||||
const item = {
|
||||
rank: rank,
|
||||
shop: shop,
|
||||
amount: cells[2]?.textContent?.trim() || '',
|
||||
orders: cells[3]?.textContent?.trim() || '',
|
||||
followers: cells[4]?.textContent?.trim() || '',
|
||||
visitors: cells[5]?.textContent?.trim() || '',
|
||||
searchClicks: cells[6]?.textContent?.trim() || '',
|
||||
};
|
||||
if (cells.includes('排名')) continue;
|
||||
const item = {};
|
||||
cells.forEach((value, index) => {
|
||||
const header = headers[index] || `col${index}`;
|
||||
item[header] = value;
|
||||
});
|
||||
result.push(item);
|
||||
}
|
||||
return JSON.stringify(result);
|
||||
}""", "[]")
|
||||
return result;
|
||||
}"""
|
||||
)
|
||||
except Exception:
|
||||
ranking_data = []
|
||||
if not isinstance(ranking_data, list):
|
||||
ranking_data = []
|
||||
|
||||
rows = []
|
||||
if isinstance(ranking_data, str):
|
||||
try:
|
||||
rows = json.loads(ranking_data)
|
||||
except json.JSONDecodeError:
|
||||
rows = []
|
||||
header_alias = {
|
||||
"排名": "rank",
|
||||
"店铺信息": "shop",
|
||||
"成交金额": "amount",
|
||||
"成交单量": "orders",
|
||||
"成交商品件数": "items",
|
||||
"访客数": "visitors",
|
||||
"浏览量": "views",
|
||||
"搜索点击次数": "searchClicks",
|
||||
}
|
||||
for raw in ranking_data if isinstance(ranking_data, list) else []:
|
||||
shop_text = str(raw.get("店铺信息", "")).splitlines()
|
||||
item = {
|
||||
"rank": str(raw.get("排名", "")).strip(),
|
||||
"shop": shop_text[0].strip() if shop_text else "",
|
||||
"amount": str(raw.get("成交金额", "")).strip(),
|
||||
"orders": str(raw.get("成交单量", "")).strip(),
|
||||
"followers": "--",
|
||||
"visitors": str(raw.get("访客数", "")).strip(),
|
||||
"searchClicks": str(raw.get("搜索点击次数", "")).strip(),
|
||||
"items": str(raw.get("成交商品件数", "")).strip(),
|
||||
"views": str(raw.get("浏览量", "")).strip(),
|
||||
}
|
||||
rows.append(item)
|
||||
return rows
|
||||
|
||||
|
||||
@@ -1086,9 +1315,13 @@ def step_return_old_version(page: Page):
|
||||
|
||||
|
||||
def step_goto_vender_ranks(page: Page):
|
||||
print("[Step 4] 跳转到新版商家榜单...")
|
||||
page.goto(VENDER_RANKS_URL, wait_until="domcontentloaded", timeout=60000)
|
||||
print("[Step 4] 跳转到新版「行业榜单-店铺榜」...")
|
||||
page.goto(INDUSTRY_TOP_URL, wait_until="domcontentloaded", timeout=60000)
|
||||
page.wait_for_timeout(8000)
|
||||
current_url = page.url or ""
|
||||
if "error.html" in current_url or "industry-top" not in current_url:
|
||||
_dump_peer_filter_debug(page, "industry_top_bad_landing")
|
||||
raise RuntimeError(f"行业榜单页跳转异常(京东侧可能再次改版): {current_url}")
|
||||
body_text = page.locator("body").inner_text(timeout=10000)
|
||||
permission_markers = (
|
||||
"当前身份下暂无此菜单",
|
||||
@@ -1098,140 +1331,35 @@ def step_goto_vender_ranks(page: Page):
|
||||
)
|
||||
if any(marker in body_text for marker in permission_markers):
|
||||
raise RuntimeError(
|
||||
"京东商智新版商家榜单当前账号无菜单或权限,"
|
||||
f"请确认该账号已开通榜单模块: {page.url}"
|
||||
"京东商智新版行业榜单当前账号无菜单或权限,"
|
||||
f"请确认该账号已开通榜单模块: {current_url}"
|
||||
)
|
||||
print(f" 已进入新版商家榜单: {page.url}")
|
||||
if "店铺榜" not in body_text:
|
||||
_dump_peer_filter_debug(page, "shop_rank_tab_missing")
|
||||
raise RuntimeError(f"行业榜单页缺少「店铺榜」入口: {current_url}")
|
||||
page.locator("text=店铺榜").first.click(timeout=10000)
|
||||
page.wait_for_timeout(5000)
|
||||
print(f" 已进入店铺榜: {page.url}")
|
||||
|
||||
|
||||
# ── 搜索店铺 ──────────────────────────────────────────────────────────────────
|
||||
|
||||
def _locate_peer_search_input(page: Page):
|
||||
"""Locate the shop search box inside the 交易榜单 section.
|
||||
|
||||
The vender-ranks page also has a global search box at the very top; generic
|
||||
selectors (and the ``input`` first-match fallback) can hit that one and then
|
||||
search nothing. Prefer the input that lives in the same container hierarchy
|
||||
as the visible 交易榜单 heading.
|
||||
"""
|
||||
|
||||
try:
|
||||
temp_id = page.evaluate(
|
||||
r"""() => {
|
||||
const visible = (el) => {
|
||||
if (!el) return false;
|
||||
const style = window.getComputedStyle(el);
|
||||
const rect = el.getBoundingClientRect();
|
||||
return style.display !== 'none' && style.visibility !== 'hidden'
|
||||
&& rect.width > 0 && rect.height > 0;
|
||||
};
|
||||
const isSearchish = (input) => {
|
||||
const type = (input.type || '').toLowerCase();
|
||||
if (['hidden', 'checkbox', 'radio', 'submit', 'button'].includes(type)) {
|
||||
return false;
|
||||
}
|
||||
// 全局 AI 搜索框(找功能/找指标/找商品)不是店铺搜索框,永远不要选中
|
||||
if (input.classList.contains('ai-search-input')) {
|
||||
return false;
|
||||
}
|
||||
// 交易榜单区块的店铺搜索框(无 placeholder,靠 class 精确命中)
|
||||
if (input.classList.contains('jmtd-input-input')
|
||||
|| input.classList.contains('jmt-input-input')) {
|
||||
return true;
|
||||
}
|
||||
const placeholder = (input.placeholder || '').trim();
|
||||
const parentText = input.closest('.jmtd-base-input')?.textContent || '';
|
||||
return Boolean(placeholder) || /搜索|查询|店铺|商家/.test(parentText);
|
||||
};
|
||||
const headings = Array.from(document.querySelectorAll('*'))
|
||||
.filter((el) => visible(el) && (el.textContent || '').trim() === '交易榜单');
|
||||
for (const heading of headings) {
|
||||
let current = heading.parentElement;
|
||||
for (let depth = 0; current && depth < 15; depth += 1, current = current.parentElement) {
|
||||
const inputs = Array.from(current.querySelectorAll('input'))
|
||||
.filter(visible)
|
||||
.filter(isSearchish);
|
||||
if (inputs.length === 1) {
|
||||
const tempId = '__peer_search_input_' + Date.now() + '_' + Math.random().toString(36).slice(2);
|
||||
inputs[0].setAttribute('id', tempId);
|
||||
return tempId;
|
||||
}
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}"""
|
||||
)
|
||||
except Exception:
|
||||
return None
|
||||
if not isinstance(temp_id, str) or not temp_id:
|
||||
return None
|
||||
try:
|
||||
candidate = page.locator(f"#{temp_id}").first
|
||||
if candidate.count() == 1 and candidate.is_visible(timeout=2_000):
|
||||
print(" [OK] 已定位交易榜单区块内的店铺搜索框")
|
||||
return candidate
|
||||
except Exception:
|
||||
pass
|
||||
return None
|
||||
|
||||
|
||||
def search_shop(page: Page, shop_name: str):
|
||||
"""在商家榜单页面搜索指定店铺"""
|
||||
"""新版店铺榜:固定 placeholder 的店铺搜索框过滤榜单。"""
|
||||
print(f" 搜索店铺: {shop_name}")
|
||||
|
||||
# 优先定位“交易榜单”区块内的店铺搜索框,避免命中页面顶部全局搜索框
|
||||
search_input = _locate_peer_search_input(page)
|
||||
|
||||
# 尝试多种搜索框选择器
|
||||
if search_input is None:
|
||||
for selector in [
|
||||
'.jmtd-base-input input.jmtd-input-input',
|
||||
'.jmtd-base-input input.jmt-input-input',
|
||||
'input[placeholder*="搜索"]',
|
||||
'input[placeholder*="店铺"]',
|
||||
'input[placeholder*="商家"]',
|
||||
'input[type="search"]',
|
||||
'.search-input input',
|
||||
'.jmtd-search input',
|
||||
]:
|
||||
search_input = page.locator('input[placeholder="请输入店铺名称"]').first
|
||||
try:
|
||||
el = page.locator(selector).first
|
||||
if el.is_visible(timeout=2000):
|
||||
# 排除全局 AI 搜索框,避免把品牌名输到"找功能/找指标/找商品"
|
||||
if el.evaluate("(input) => input.classList.contains('ai-search-input')"):
|
||||
continue
|
||||
search_input = el
|
||||
break
|
||||
except Exception:
|
||||
continue
|
||||
|
||||
if not search_input:
|
||||
# 兜底:优先 jmtd-input-input,其次任意 input(仍排除 AI 搜索框)
|
||||
try:
|
||||
fallback = page.locator('input.jmtd-input-input, input.jmt-input-input').first
|
||||
if fallback.is_visible(timeout=2000):
|
||||
search_input = fallback
|
||||
except Exception:
|
||||
pass
|
||||
if not search_input:
|
||||
search_input = page.locator('input').first
|
||||
print(" [WARN] 未找到明确的搜索框,使用第一个 input")
|
||||
|
||||
# 清空并输入店铺名
|
||||
search_input.wait_for(state="visible", timeout=10000)
|
||||
except Exception as exc:
|
||||
_dump_peer_filter_debug(page, "shop_search_input_missing")
|
||||
raise RuntimeError(f"未找到店铺搜索框: {exc}") from exc
|
||||
search_input.click()
|
||||
search_input.fill("")
|
||||
page.wait_for_timeout(500)
|
||||
page.wait_for_timeout(300)
|
||||
search_input.fill(shop_name)
|
||||
page.wait_for_timeout(1000)
|
||||
|
||||
# 尝试点击搜索按钮或按回车
|
||||
try:
|
||||
search_btn = page.locator('button:has-text("搜索"), button:has-text("查询"), .search-btn, .jmtd-search-btn').first
|
||||
search_btn.click(timeout=3000)
|
||||
except Exception:
|
||||
page.wait_for_timeout(800)
|
||||
search_input.press("Enter")
|
||||
|
||||
page.wait_for_timeout(8000)
|
||||
page.wait_for_timeout(4000)
|
||||
print(f" [OK] 搜索完成: {shop_name}")
|
||||
|
||||
|
||||
|
||||
@@ -52,7 +52,6 @@ MAX_NOTIFICATION_BINDINGS_PER_PERSON = 16
|
||||
NOTIFICATION_CAPABILITIES: Mapping[str, str] = MappingProxyType(
|
||||
{
|
||||
"content.marketing_report.daily": "正式执行且启用发送时投递最终营销日报。",
|
||||
"content.relogin.weekly": "刷新期间投递各平台扫码二维码,并在平台失败时投递失败汇总。",
|
||||
"product.alert.daily": "命中销量下滑规则且尚未重复通知时投递告警。",
|
||||
"product.market_rank": "存在本次新生成的平台报告链接且尚未重复通知时投递。",
|
||||
"product.online_stock_alert.daily": "线上商品当前库存低于 3 时向配置接收人发送库存预警。",
|
||||
|
||||
@@ -0,0 +1,117 @@
|
||||
# 运维审计库(ops_* 表)
|
||||
|
||||
工作流执行历史与登录 cookie 遥测的 PostgreSQL 落库。本地 `var/runs/**/run.json`
|
||||
仍是权威源,这里做的是**跨机器可查、可长期留存的关系型镜像**。
|
||||
|
||||
## 为什么需要
|
||||
|
||||
- 本地 journal 只在部署机磁盘上:换机、扩容、清理后历史就断了。
|
||||
- `var/logs` 是文本,做不了「近 30 天哪些工作流失败最多」这类查询。
|
||||
- cookie 状态原先只存在 `keepalive_status.json`(只有最新值),无法回答
|
||||
「登录态到底能活几天」。
|
||||
|
||||
## 表与视图
|
||||
|
||||
| 对象 | 粒度 | 用途 |
|
||||
|---|---|---|
|
||||
| `ops_workflow_runs` | 一次运行一行 | 状态、模式、耗时、步骤计数、错误、journal 路径、主机 |
|
||||
| `ops_workflow_run_steps` | 一次尝试一行 | 含重试;`exit_code`、错误、耗时 |
|
||||
| `ops_account_cookie_events` | 一次观测一行 | 事件流;只存指纹,不存 cookie 明文 |
|
||||
| `ops_account_cookie_lifetimes`(视图) | 一份 cookie 一行 | 从首次出现到失效的寿命 |
|
||||
| `ops_account_cookie_stats`(视图) | 一个账号一行 | 平均 / 中位 / 最短 / 最长寿命 |
|
||||
|
||||
## 自动建表
|
||||
|
||||
首次写入时执行 `SCHEMA_SQL`(幂等 DDL),无需手工迁移。也可以用:
|
||||
|
||||
```bash
|
||||
uv run gyxx audit schema > 001_ops_audit.sql # 导出 DDL 交 DBA 预建
|
||||
psql "$GYXX_POSTGRES_DSN" -f 001_ops_audit.sql
|
||||
```
|
||||
|
||||
## 查询
|
||||
|
||||
```bash
|
||||
uv run gyxx audit runs --workflow content.metrics.self_operated.daily --limit 20
|
||||
uv run gyxx audit runs --status failed --limit 50
|
||||
uv run gyxx audit cookies --account douyin-shop --limit 50
|
||||
uv run gyxx audit cookie-stats # 各账号失效规律
|
||||
uv run gyxx audit cookie-stats --account douyin-shop # 单账号每次的寿命明细
|
||||
```
|
||||
|
||||
都支持 `--json`,方便接到面板或告警脚本。
|
||||
|
||||
直接 SQL 也一样:
|
||||
|
||||
```sql
|
||||
-- 最近 7 天失败最多的 10 个工作流
|
||||
SELECT workflow_id, COUNT(*) AS failures
|
||||
FROM ops_workflow_runs
|
||||
WHERE status = 'failed' AND started_at > NOW() - INTERVAL '7 days'
|
||||
GROUP BY workflow_id ORDER BY failures DESC LIMIT 10;
|
||||
|
||||
-- 每次失败卡在哪个步骤
|
||||
SELECT r.workflow_id, s.step_id, COUNT(*) AS n
|
||||
FROM ops_workflow_run_steps s
|
||||
JOIN ops_workflow_runs r USING (run_id)
|
||||
WHERE s.status = 'failed' AND r.started_at > NOW() - INTERVAL '7 days'
|
||||
GROUP BY 1, 2 ORDER BY n DESC;
|
||||
|
||||
-- 哪个账号的登录态最短命(预警阈值参考)
|
||||
SELECT account_id, sessions, expired_sessions,
|
||||
avg_lifetime_seconds / 86400.0 AS avg_days,
|
||||
min_lifetime_seconds / 86400.0 AS min_days,
|
||||
last_failed_at
|
||||
FROM ops_account_cookie_stats ORDER BY min_lifetime_seconds NULLS LAST;
|
||||
```
|
||||
|
||||
## 怎么读「失效规律」
|
||||
|
||||
一次 cookie 代由 `fingerprint` 标识(需求域名/名称/值哈希排序后取摘要,与文件顺序无关):
|
||||
|
||||
- `first_healthy_at` → `first_failed_at` = `lifetime_seconds`:**实际可用时长**
|
||||
- `healthy_to_failure_seconds`:最后一次探活成功到发现失效,反映**探测精度**
|
||||
- `declared_expires_at`:cookie 自带的最早过期时间
|
||||
|
||||
把 `lifetime_seconds` 和 `declared_expires_at` 对比,能区分三种情况:
|
||||
|
||||
1. 实际寿命接近声明寿命 → 平台按声明失效,可按 `declared_expires_at` 提前预警;
|
||||
2. 实际远短于声明 → 服务端提前吊销(常见于多端登录、风控);
|
||||
3. `declared_expires_at` 为空(session cookie)→ 只能靠观测,`min_lifetime_seconds`
|
||||
就是安全刷新周期的上界。
|
||||
|
||||
`fingerprint` 变化表示换了一份 cookie(重新登录),不算失效,也不会把两段寿命混在一起。
|
||||
|
||||
## 采集点
|
||||
|
||||
| 来源 | 触发 | `probe_source` |
|
||||
|---|---|---|
|
||||
| `CookieGate.check` | 每次工作流运行前的门禁判定 | `gate` |
|
||||
| `AccountKeepaliveManager` | 常驻调度器每 `keepalive_interval_seconds`(默认 600s) | `keepalive` |
|
||||
| `RunIndex.index_journal` | 每次运行开始/结束/异常 | — |
|
||||
|
||||
keepalive 是主要的精度来源:默认 10 分钟一次,同类状态一小时内只落一行,
|
||||
状态翻转立即落行,所以「首次失效时间」误差不超过一个探活周期。
|
||||
|
||||
## 容错
|
||||
|
||||
审计**永不阻断业务**:
|
||||
|
||||
- 数据库不可达 / 缺 DSN / 建表失败 → 写入返回 `False`,调用方继续执行;
|
||||
- 连续失败 3 次后熔断冷却(默认 300s),避免每个工作流都卡在连接超时;
|
||||
- 失败记录写到 `<GYXX_DATA_ROOT>/logs/ops-audit/audit-failures.log`。
|
||||
|
||||
## 开关
|
||||
|
||||
| 变量 | 默认 | 说明 |
|
||||
|---|---|---|
|
||||
| `GYXX_OPS_AUDIT` | 开 | 设 `off` 关闭全部审计写入 |
|
||||
| `GYXX_OPS_AUDIT_HOST` | 主机名 | 多机部署时写入手工标识 |
|
||||
| `GYXX_OPS_AUDIT_FAILURE_COOLDOWN` | 300 | 熔断冷却秒数 |
|
||||
|
||||
## 不存什么
|
||||
|
||||
- **cookie 值**:只存哈希指纹;排查具体 cookie 请走 vault 文件。
|
||||
- **日志正文**:`ops_workflow_runs` 只存 `journal_path` 指向本地文件;
|
||||
完整堆栈仍在 `var/logs/console-restarts/`。
|
||||
- **凭据**:DSN 只从 `GYXX_POSTGRES_DSN`(或 `PG_*`)读,不落表。
|
||||
@@ -1,5 +1,6 @@
|
||||
"""Operational run indexing and durable outbox services."""
|
||||
"""Operational run indexing, audit telemetry and durable outbox services."""
|
||||
|
||||
from .audit import AuditStore, CookieSnapshot, summarize_cookies
|
||||
from .effects import (
|
||||
EffectAlreadyApplied,
|
||||
EffectClaim,
|
||||
@@ -10,6 +11,8 @@ from .index import RunIndex, RunRecord
|
||||
from .outbox import IdempotencyConflict, Outbox, OutboxMessage
|
||||
|
||||
__all__ = [
|
||||
"AuditStore",
|
||||
"CookieSnapshot",
|
||||
"IdempotencyConflict",
|
||||
"EffectAlreadyApplied",
|
||||
"EffectClaim",
|
||||
@@ -19,4 +22,5 @@ __all__ = [
|
||||
"OutboxMessage",
|
||||
"RunIndex",
|
||||
"RunRecord",
|
||||
"summarize_cookies",
|
||||
]
|
||||
@@ -0,0 +1,901 @@
|
||||
"""PostgreSQL-backed operational audit: workflow run history and cookie telemetry.
|
||||
|
||||
设计原则
|
||||
--------
|
||||
1. **可观测性不拖累业务**:审计写入永远不向调用方抛异常。数据库不可达时
|
||||
只记录一条 failure 日志并熔断冷却,工作流照常执行。
|
||||
2. **本地 journal 仍是权威源**:本模块是镜像/汇总,供跨机器查询与长期留存;
|
||||
`var/runs/**/run.json` 不会被替代。
|
||||
3. **不落明文凭据**:cookie 只存指纹(域名+名称+值哈希),值本身永不入库。
|
||||
|
||||
环境开关
|
||||
--------
|
||||
``GYXX_OPS_AUDIT=off`` 关闭审计(默认开启)
|
||||
``GYXX_OPS_AUDIT_HOST`` 覆盖主机标识(默认 socket.gethostname())
|
||||
``GYXX_OPS_AUDIT_FAILURE_COOLDOWN`` 熔断冷却秒数(默认 300)
|
||||
``GYXX_POSTGRES_DSN`` 与动态配置共用同一 DSN
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import socket
|
||||
from contextlib import contextmanager
|
||||
from dataclasses import dataclass
|
||||
from datetime import date, datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any, Iterator, Mapping, Sequence
|
||||
|
||||
SCHEMA_VERSION = 1
|
||||
|
||||
OPS_AUDIT_ENV = "GYXX_OPS_AUDIT"
|
||||
OPS_AUDIT_HOST_ENV = "GYXX_OPS_AUDIT_HOST"
|
||||
OPS_AUDIT_COOLDOWN_ENV = "GYXX_OPS_AUDIT_FAILURE_COOLDOWN"
|
||||
|
||||
_DEFAULT_COOLDOWN_SECONDS = 300
|
||||
_MAX_CONSECUTIVE_FAILURES = 3
|
||||
_MAX_TEXT_CHARS = 4000
|
||||
_SCHEMA_LOCK_KEY = "gyxx:ops-audit:migration"
|
||||
|
||||
HEALTHY_STATES = frozenset({"healthy"})
|
||||
FAILED_STATES = frozenset({"expired", "not_logged_in"})
|
||||
_COOKIE_STATES = frozenset(
|
||||
{"healthy", "expired", "not_logged_in", "probe_error", "busy", "disabled"}
|
||||
)
|
||||
|
||||
SCHEMA_SQL = """
|
||||
CREATE TABLE IF NOT EXISTS ops_workflow_runs (
|
||||
run_id TEXT PRIMARY KEY,
|
||||
workflow_id TEXT NOT NULL,
|
||||
business_date DATE NOT NULL,
|
||||
mode TEXT NOT NULL DEFAULT 'unknown',
|
||||
shadow BOOLEAN NOT NULL DEFAULT FALSE,
|
||||
status TEXT NOT NULL,
|
||||
started_at TIMESTAMPTZ,
|
||||
ended_at TIMESTAMPTZ,
|
||||
duration_seconds INTEGER,
|
||||
error TEXT,
|
||||
step_total INTEGER NOT NULL DEFAULT 0,
|
||||
step_success INTEGER NOT NULL DEFAULT 0,
|
||||
step_failed INTEGER NOT NULL DEFAULT 0,
|
||||
step_skipped INTEGER NOT NULL DEFAULT 0,
|
||||
step_running INTEGER NOT NULL DEFAULT 0,
|
||||
journal_path TEXT NOT NULL DEFAULT '',
|
||||
host TEXT NOT NULL DEFAULT '',
|
||||
indexed_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS ops_workflow_runs_workflow_idx
|
||||
ON ops_workflow_runs (workflow_id, started_at DESC);
|
||||
CREATE INDEX IF NOT EXISTS ops_workflow_runs_status_idx
|
||||
ON ops_workflow_runs (status, started_at DESC);
|
||||
CREATE INDEX IF NOT EXISTS ops_workflow_runs_business_date_idx
|
||||
ON ops_workflow_runs (business_date, workflow_id);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS ops_workflow_run_steps (
|
||||
run_id TEXT NOT NULL
|
||||
REFERENCES ops_workflow_runs (run_id) ON DELETE CASCADE,
|
||||
step_id TEXT NOT NULL,
|
||||
attempt INTEGER NOT NULL DEFAULT 1,
|
||||
status TEXT NOT NULL DEFAULT '',
|
||||
exit_code INTEGER,
|
||||
started_at TIMESTAMPTZ,
|
||||
ended_at TIMESTAMPTZ,
|
||||
duration_seconds INTEGER,
|
||||
error TEXT,
|
||||
PRIMARY KEY (run_id, step_id, attempt)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS ops_workflow_run_steps_step_idx
|
||||
ON ops_workflow_run_steps (step_id, status);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS ops_account_cookie_events (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
account_id TEXT NOT NULL,
|
||||
observed_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
state TEXT NOT NULL,
|
||||
probe_status TEXT NOT NULL DEFAULT '',
|
||||
probe_source TEXT NOT NULL DEFAULT '',
|
||||
cookie_count INTEGER,
|
||||
required_count INTEGER,
|
||||
fingerprint TEXT NOT NULL DEFAULT '',
|
||||
declared_expires_at TIMESTAMPTZ,
|
||||
detail TEXT NOT NULL DEFAULT '',
|
||||
workflow_id TEXT NOT NULL DEFAULT '',
|
||||
run_id TEXT NOT NULL DEFAULT '',
|
||||
host TEXT NOT NULL DEFAULT ''
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS ops_account_cookie_events_account_idx
|
||||
ON ops_account_cookie_events (account_id, observed_at DESC);
|
||||
CREATE INDEX IF NOT EXISTS ops_account_cookie_events_state_idx
|
||||
ON ops_account_cookie_events (state, observed_at DESC);
|
||||
CREATE INDEX IF NOT EXISTS ops_account_cookie_events_fingerprint_idx
|
||||
ON ops_account_cookie_events (account_id, fingerprint);
|
||||
|
||||
-- 一段「cookie 代」的生命周期:同一 fingerprint 从首次出现到失效。
|
||||
-- first_failed_at 只取最后一次 healthy 之后的失败,避免恢复后再次失败时口径前移。
|
||||
CREATE OR REPLACE VIEW ops_account_cookie_lifetimes AS
|
||||
WITH per_fingerprint AS (
|
||||
SELECT
|
||||
account_id,
|
||||
fingerprint,
|
||||
MIN(observed_at) AS first_seen_at,
|
||||
MIN(observed_at) FILTER (WHERE state = 'healthy') AS first_healthy_at,
|
||||
MAX(observed_at) FILTER (WHERE state = 'healthy') AS last_healthy_at,
|
||||
MAX(declared_expires_at) AS declared_expires_at
|
||||
FROM ops_account_cookie_events
|
||||
WHERE fingerprint <> ''
|
||||
GROUP BY account_id, fingerprint
|
||||
)
|
||||
SELECT
|
||||
p.account_id,
|
||||
p.fingerprint,
|
||||
p.first_seen_at,
|
||||
p.first_healthy_at,
|
||||
p.last_healthy_at,
|
||||
p.declared_expires_at,
|
||||
f.first_failed_at,
|
||||
(f.first_failed_at IS NOT NULL) AS expired,
|
||||
CASE
|
||||
WHEN p.first_healthy_at IS NOT NULL AND p.last_healthy_at IS NOT NULL
|
||||
THEN EXTRACT(EPOCH FROM (p.last_healthy_at - p.first_healthy_at))::BIGINT
|
||||
END AS healthy_span_seconds,
|
||||
CASE
|
||||
WHEN p.first_healthy_at IS NOT NULL AND f.first_failed_at IS NOT NULL
|
||||
THEN EXTRACT(EPOCH FROM (f.first_failed_at - p.first_healthy_at))::BIGINT
|
||||
END AS lifetime_seconds,
|
||||
CASE
|
||||
WHEN p.last_healthy_at IS NOT NULL AND f.first_failed_at IS NOT NULL
|
||||
THEN EXTRACT(EPOCH FROM (f.first_failed_at - p.last_healthy_at))::BIGINT
|
||||
END AS healthy_to_failure_seconds
|
||||
FROM per_fingerprint p
|
||||
LEFT JOIN LATERAL (
|
||||
SELECT MIN(e.observed_at) AS first_failed_at
|
||||
FROM ops_account_cookie_events e
|
||||
WHERE e.account_id = p.account_id
|
||||
AND e.fingerprint = p.fingerprint
|
||||
AND e.state IN ('expired', 'not_logged_in')
|
||||
AND (p.last_healthy_at IS NULL OR e.observed_at > p.last_healthy_at)
|
||||
) f ON TRUE;
|
||||
|
||||
-- 按账号汇总的失效规律:平均/中位/最短/最长寿命,直接支撑「多久失效一次」。
|
||||
CREATE OR REPLACE VIEW ops_account_cookie_stats AS
|
||||
SELECT
|
||||
account_id,
|
||||
COUNT(*) AS sessions,
|
||||
COUNT(*) FILTER (WHERE expired) AS expired_sessions,
|
||||
AVG(lifetime_seconds)::BIGINT AS avg_lifetime_seconds,
|
||||
PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY lifetime_seconds)::BIGINT
|
||||
AS median_lifetime_seconds,
|
||||
MIN(lifetime_seconds) AS min_lifetime_seconds,
|
||||
MAX(lifetime_seconds) AS max_lifetime_seconds,
|
||||
MAX(first_failed_at) AS last_failed_at,
|
||||
MAX(last_healthy_at) AS last_healthy_at
|
||||
FROM ops_account_cookie_lifetimes
|
||||
WHERE lifetime_seconds IS NOT NULL
|
||||
GROUP BY account_id;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS ops_audit_schema (
|
||||
schema_version INTEGER PRIMARY KEY,
|
||||
applied_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
"""
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
# 纯函数:cookie 快照
|
||||
# ----------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class CookieSnapshot:
|
||||
"""一次静态检查得到的 cookie 画像,不含任何明文值。"""
|
||||
|
||||
cookie_count: int
|
||||
required_count: int
|
||||
fingerprint: str
|
||||
declared_expires_at: datetime | None
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"cookie_count": self.cookie_count,
|
||||
"required_count": self.required_count,
|
||||
"fingerprint": self.fingerprint,
|
||||
"declared_expires_at": (
|
||||
self.declared_expires_at.isoformat()
|
||||
if self.declared_expires_at is not None
|
||||
else None
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def _domain_matches(domain: str, required_domains: Sequence[str]) -> bool:
|
||||
normalized = str(domain or "").lstrip(".").casefold()
|
||||
if not normalized:
|
||||
return False
|
||||
return any(
|
||||
normalized == required.casefold() or normalized.endswith("." + required.casefold())
|
||||
for required in required_domains
|
||||
)
|
||||
|
||||
|
||||
def _expires_at(cookie: Mapping[str, Any]) -> datetime | None:
|
||||
raw = cookie.get("expires")
|
||||
if raw is None or isinstance(raw, bool):
|
||||
return None
|
||||
try:
|
||||
value = float(raw)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
# Playwright 用 -1 表示 session cookie;0 与负数都视为「不声明」。
|
||||
if value <= 0:
|
||||
return None
|
||||
try:
|
||||
return datetime.fromtimestamp(value, tz=timezone.utc)
|
||||
except (OverflowError, OSError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def summarize_cookies(
|
||||
cookies: Sequence[Mapping[str, Any]],
|
||||
*,
|
||||
required_domains: Sequence[str] = (),
|
||||
required_names: Sequence[str] = (),
|
||||
) -> CookieSnapshot:
|
||||
"""Build a value-free snapshot: counts, fingerprint and earliest declared expiry.
|
||||
|
||||
指纹只看「必需 cookie」的域名/名称/值哈希,因此登录态真正换了一份 cookie
|
||||
时指纹会变,而无关 cookie 的增删不会造成抖动。
|
||||
"""
|
||||
|
||||
required_domains = tuple(item for item in required_domains if item)
|
||||
required_names = tuple(item for item in required_names if item)
|
||||
components: list[str] = []
|
||||
required_count = 0
|
||||
earliest: datetime | None = None
|
||||
|
||||
for cookie in cookies:
|
||||
if not isinstance(cookie, Mapping):
|
||||
continue
|
||||
name = str(cookie.get("name", ""))
|
||||
domain = str(cookie.get("domain", ""))
|
||||
is_required = bool(required_domains) and _domain_matches(domain, required_domains)
|
||||
if required_names:
|
||||
is_required = is_required and name in required_names
|
||||
elif not required_domains:
|
||||
is_required = True
|
||||
if not is_required:
|
||||
continue
|
||||
required_count += 1
|
||||
value = str(cookie.get("value", ""))
|
||||
value_hash = hashlib.sha256(value.encode("utf-8", errors="replace")).hexdigest()[:16]
|
||||
components.append(f"{domain.lstrip('.')}|{name}|{value_hash}")
|
||||
expiry = _expires_at(cookie)
|
||||
if expiry is not None and (earliest is None or expiry < earliest):
|
||||
earliest = expiry
|
||||
|
||||
# 浏览器写入 cookie 的顺序并不稳定;排序后哈希才能保证同一份登录态得到同一指纹。
|
||||
digest = hashlib.sha256()
|
||||
for component in sorted(components):
|
||||
digest.update(f"{component}\n".encode("utf-8"))
|
||||
|
||||
return CookieSnapshot(
|
||||
cookie_count=len(cookies),
|
||||
required_count=required_count,
|
||||
fingerprint=digest.hexdigest()[:32] if required_count else "",
|
||||
declared_expires_at=earliest,
|
||||
)
|
||||
|
||||
|
||||
def snapshot_for_account(account: Any) -> CookieSnapshot | None:
|
||||
"""Read an account's cookie vault and summarise it, or ``None`` when unreadable.
|
||||
|
||||
Shared by the gate and the keepalive manager so both record the exact same
|
||||
fingerprint for the same vault contents.
|
||||
"""
|
||||
|
||||
cookie_file = getattr(account, "cookie_file", None)
|
||||
if cookie_file is None:
|
||||
return None
|
||||
try:
|
||||
from gyxx_flow.adapters.browser import BrowserCookieStore
|
||||
|
||||
store = BrowserCookieStore(
|
||||
cookie_file,
|
||||
getattr(account, "storage_state_file", cookie_file),
|
||||
)
|
||||
cookies = store.load_cookies()
|
||||
except Exception: # noqa: BLE001 - a missing/corrupt vault is itself a signal
|
||||
return None
|
||||
return summarize_cookies(
|
||||
cookies,
|
||||
required_domains=tuple(getattr(account, "required_cookie_domains", ()) or ()),
|
||||
required_names=tuple(getattr(account, "required_cookie_names", ()) or ()),
|
||||
)
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
# 时间与文本工具
|
||||
# ----------------------------------------------------------------------
|
||||
|
||||
|
||||
def _parse_datetime(value: Any) -> datetime | None:
|
||||
if isinstance(value, datetime):
|
||||
return value if value.tzinfo else value.replace(tzinfo=timezone.utc)
|
||||
if not isinstance(value, str) or not value.strip():
|
||||
return None
|
||||
text = value.strip().replace("Z", "+00:00")
|
||||
try:
|
||||
parsed = datetime.fromisoformat(text)
|
||||
except ValueError:
|
||||
return None
|
||||
return parsed if parsed.tzinfo else parsed.replace(tzinfo=timezone.utc)
|
||||
|
||||
|
||||
def _parse_date(value: Any) -> date | None:
|
||||
if isinstance(value, date) and not isinstance(value, datetime):
|
||||
return value
|
||||
if not isinstance(value, str):
|
||||
return None
|
||||
try:
|
||||
return date.fromisoformat(value)
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
def _seconds_between(start: datetime | None, end: datetime | None) -> int | None:
|
||||
if start is None or end is None:
|
||||
return None
|
||||
delta = (end - start).total_seconds()
|
||||
return max(0, int(delta))
|
||||
|
||||
|
||||
def _clip(value: Any, limit: int = _MAX_TEXT_CHARS) -> str:
|
||||
if value is None:
|
||||
return ""
|
||||
text = value if isinstance(value, str) else str(value)
|
||||
return text[:limit]
|
||||
|
||||
|
||||
def split_step_key(key: str) -> tuple[str, int]:
|
||||
"""``sync.attempt-2`` -> ``("sync", 2)``; 无法解析时原样当作 step_id。"""
|
||||
|
||||
if ".attempt-" in key:
|
||||
name, _, raw = key.rpartition(".attempt-")
|
||||
try:
|
||||
attempt = int(raw)
|
||||
except ValueError:
|
||||
return key, 1
|
||||
if name and attempt > 0:
|
||||
return name, attempt
|
||||
return key, 1
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
# 审计存储
|
||||
# ----------------------------------------------------------------------
|
||||
|
||||
|
||||
class AuditStore:
|
||||
"""Idempotent writer for the ``ops_*`` tables.
|
||||
|
||||
所有公开写入方法都返回 bool,并且**永不抛出**异常;数据库问题只会导致本次
|
||||
审计丢失,不会影响工作流执行。
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
data_root: Path | str,
|
||||
*,
|
||||
environment: Mapping[str, str] | None = None,
|
||||
connect: Any | None = None,
|
||||
clock: Any | None = None,
|
||||
) -> None:
|
||||
self._data_root = Path(data_root).expanduser().resolve()
|
||||
self._environment = dict(os.environ if environment is None else environment)
|
||||
self._connect_impl = connect
|
||||
self._clock = clock or (lambda: datetime.now(timezone.utc))
|
||||
self._schema_ready = False
|
||||
self._consecutive_failures = 0
|
||||
self._cooldown_until = 0.0
|
||||
self._last_error = ""
|
||||
self._failure_log = self._data_root / "logs" / "ops-audit" / "audit-failures.log"
|
||||
|
||||
# -- 状态 ---------------------------------------------------------
|
||||
|
||||
@property
|
||||
def enabled(self) -> bool:
|
||||
raw = str(self._environment.get(OPS_AUDIT_ENV, "")).strip().lower()
|
||||
return raw not in {"0", "off", "false", "no", "disabled"}
|
||||
|
||||
@property
|
||||
def last_error(self) -> str:
|
||||
return self._last_error
|
||||
|
||||
def _host(self) -> str:
|
||||
override = str(self._environment.get(OPS_AUDIT_HOST_ENV, "")).strip()
|
||||
if override:
|
||||
return override
|
||||
try:
|
||||
return socket.gethostname()[:128]
|
||||
except OSError: # pragma: no cover - 极端环境
|
||||
return ""
|
||||
|
||||
def _cooldown_seconds(self) -> int:
|
||||
raw = str(self._environment.get(OPS_AUDIT_COOLDOWN_ENV, "")).strip()
|
||||
try:
|
||||
return max(0, int(raw))
|
||||
except ValueError:
|
||||
return _DEFAULT_COOLDOWN_SECONDS
|
||||
|
||||
def _in_cooldown(self) -> bool:
|
||||
import time
|
||||
|
||||
return time.monotonic() < self._cooldown_until
|
||||
|
||||
# -- 连接 ---------------------------------------------------------
|
||||
|
||||
@contextmanager
|
||||
def _connection(self) -> Iterator[Any]:
|
||||
if self._connect_impl is not None:
|
||||
conn = self._connect_impl()
|
||||
try:
|
||||
yield conn
|
||||
finally:
|
||||
close = getattr(conn, "close", None)
|
||||
if callable(close):
|
||||
close()
|
||||
return
|
||||
|
||||
import psycopg # 延迟导入:未安装 psycopg 时整个模块仍可导入
|
||||
from psycopg.rows import dict_row
|
||||
|
||||
dsn, kwargs = database_kwargs(self._environment)
|
||||
if dsn:
|
||||
conn = psycopg.connect(dsn, row_factory=dict_row)
|
||||
else:
|
||||
conn = psycopg.connect(**kwargs, row_factory=dict_row)
|
||||
try:
|
||||
yield conn
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
def _ensure_schema(self, conn: Any) -> None:
|
||||
"""Create the audit objects once, serialising the first DDL pass.
|
||||
|
||||
Several workflows can start at the same moment; running CREATE/COMMENT
|
||||
concurrently lets sessions take DDL locks in different orders and can
|
||||
deadlock. The advisory lock is only taken when the schema is missing,
|
||||
so steady-state processes skip the round trip entirely.
|
||||
"""
|
||||
|
||||
if self._schema_ready:
|
||||
return
|
||||
with conn.cursor() as cursor:
|
||||
cursor.execute(
|
||||
"SELECT 1 FROM pg_class "
|
||||
"WHERE relname = 'ops_audit_schema' AND relkind = 'r'"
|
||||
)
|
||||
initialised = cursor.fetchone() is not None
|
||||
if not initialised:
|
||||
with conn.cursor() as cursor:
|
||||
cursor.execute(
|
||||
"SELECT pg_advisory_xact_lock(hashtext(%s))",
|
||||
(_SCHEMA_LOCK_KEY,),
|
||||
)
|
||||
cursor.execute(SCHEMA_SQL)
|
||||
cursor.execute(
|
||||
"INSERT INTO ops_audit_schema (schema_version) VALUES (%s) "
|
||||
"ON CONFLICT (schema_version) DO NOTHING",
|
||||
(SCHEMA_VERSION,),
|
||||
)
|
||||
conn.commit()
|
||||
self._schema_ready = True
|
||||
|
||||
# -- 失败处理 -----------------------------------------------------
|
||||
|
||||
def _record_failure(self, operation: str, exc: BaseException) -> None:
|
||||
import time
|
||||
|
||||
self._consecutive_failures += 1
|
||||
self._last_error = f"{type(exc).__name__}: {exc}"[:300]
|
||||
if self._consecutive_failures >= _MAX_CONSECUTIVE_FAILURES:
|
||||
self._cooldown_until = time.monotonic() + self._cooldown_seconds()
|
||||
try:
|
||||
self._failure_log.parent.mkdir(parents=True, exist_ok=True)
|
||||
payload = {
|
||||
"at": self._clock().isoformat(timespec="seconds"),
|
||||
"operation": operation,
|
||||
"error": self._last_error,
|
||||
"consecutive_failures": self._consecutive_failures,
|
||||
}
|
||||
with self._failure_log.open("a", encoding="utf-8") as stream:
|
||||
stream.write(json.dumps(payload, ensure_ascii=False) + "\n")
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
def _run_guarded(self, operation: str, action: Any) -> bool:
|
||||
if not self.enabled or self._in_cooldown():
|
||||
return False
|
||||
try:
|
||||
with self._connection() as conn:
|
||||
self._ensure_schema(conn)
|
||||
action(conn)
|
||||
self._consecutive_failures = 0
|
||||
return True
|
||||
except Exception as exc: # noqa: BLE001 - 审计绝不影响业务
|
||||
self._record_failure(operation, exc)
|
||||
return False
|
||||
|
||||
# -- 运行记录 -----------------------------------------------------
|
||||
|
||||
def record_run(
|
||||
self,
|
||||
record: Mapping[str, Any],
|
||||
steps: Mapping[str, Any] | None = None,
|
||||
) -> bool:
|
||||
"""Upsert one run and its steps from a journal payload or RunRecord dict."""
|
||||
|
||||
workflow_id = _clip(record.get("workflow_id"), 200)
|
||||
run_id = _clip(record.get("run_id"), 200)
|
||||
business_date = _parse_date(record.get("business_date"))
|
||||
if not workflow_id or not run_id or business_date is None:
|
||||
return False
|
||||
status = _clip(record.get("status"), 32) or "unknown"
|
||||
mode = _clip(record.get("mode"), 32) or "unknown"
|
||||
started_at = _parse_datetime(record.get("started_at"))
|
||||
ended_at = _parse_datetime(record.get("ended_at"))
|
||||
step_rows = self._step_rows(run_id, steps)
|
||||
counts = self._step_counts(steps, record)
|
||||
|
||||
def action(conn: Any) -> None:
|
||||
with conn.cursor() as cursor:
|
||||
cursor.execute(
|
||||
"""
|
||||
INSERT INTO ops_workflow_runs (
|
||||
run_id, workflow_id, business_date, mode, shadow, status,
|
||||
started_at, ended_at, duration_seconds, error,
|
||||
step_total, step_success, step_failed, step_skipped,
|
||||
step_running, journal_path, host, indexed_at
|
||||
) VALUES (
|
||||
%(run_id)s, %(workflow_id)s, %(business_date)s, %(mode)s,
|
||||
%(shadow)s, %(status)s, %(started_at)s, %(ended_at)s,
|
||||
%(duration_seconds)s, %(error)s, %(step_total)s,
|
||||
%(step_success)s, %(step_failed)s, %(step_skipped)s,
|
||||
%(step_running)s, %(journal_path)s, %(host)s, NOW()
|
||||
)
|
||||
ON CONFLICT (run_id) DO UPDATE SET
|
||||
status = EXCLUDED.status,
|
||||
ended_at = EXCLUDED.ended_at,
|
||||
duration_seconds = EXCLUDED.duration_seconds,
|
||||
error = EXCLUDED.error,
|
||||
step_total = EXCLUDED.step_total,
|
||||
step_success = EXCLUDED.step_success,
|
||||
step_failed = EXCLUDED.step_failed,
|
||||
step_skipped = EXCLUDED.step_skipped,
|
||||
step_running = EXCLUDED.step_running,
|
||||
journal_path = EXCLUDED.journal_path,
|
||||
indexed_at = NOW()
|
||||
""",
|
||||
{
|
||||
"run_id": run_id,
|
||||
"workflow_id": workflow_id,
|
||||
"business_date": business_date,
|
||||
"mode": mode,
|
||||
"shadow": bool(record.get("shadow", False)),
|
||||
"status": status,
|
||||
"started_at": started_at,
|
||||
"ended_at": ended_at,
|
||||
"duration_seconds": _seconds_between(started_at, ended_at),
|
||||
"error": _clip(record.get("error")),
|
||||
"step_total": counts["total"],
|
||||
"step_success": counts["success"],
|
||||
"step_failed": counts["failed"],
|
||||
"step_skipped": counts["skipped"],
|
||||
"step_running": counts["running"],
|
||||
"journal_path": _clip(record.get("journal_path"), 1000),
|
||||
"host": self._host(),
|
||||
},
|
||||
)
|
||||
if step_rows:
|
||||
cursor.executemany(
|
||||
"""
|
||||
INSERT INTO ops_workflow_run_steps (
|
||||
run_id, step_id, attempt, status, exit_code,
|
||||
started_at, ended_at, duration_seconds, error
|
||||
) VALUES (
|
||||
%(run_id)s, %(step_id)s, %(attempt)s, %(status)s,
|
||||
%(exit_code)s, %(started_at)s, %(ended_at)s,
|
||||
%(duration_seconds)s, %(error)s
|
||||
)
|
||||
ON CONFLICT (run_id, step_id, attempt) DO UPDATE SET
|
||||
status = EXCLUDED.status,
|
||||
exit_code = EXCLUDED.exit_code,
|
||||
started_at = COALESCE(
|
||||
ops_workflow_run_steps.started_at,
|
||||
EXCLUDED.started_at
|
||||
),
|
||||
ended_at = EXCLUDED.ended_at,
|
||||
duration_seconds = EXCLUDED.duration_seconds,
|
||||
error = EXCLUDED.error
|
||||
""",
|
||||
step_rows,
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
return self._run_guarded("record_run", action)
|
||||
|
||||
@staticmethod
|
||||
def _step_rows(
|
||||
run_id: str, steps: Mapping[str, Any] | None
|
||||
) -> list[dict[str, Any]]:
|
||||
if not isinstance(steps, Mapping):
|
||||
return []
|
||||
rows: list[dict[str, Any]] = []
|
||||
for key, raw in steps.items():
|
||||
if not isinstance(raw, Mapping):
|
||||
continue
|
||||
step_id, attempt = split_step_key(str(key))
|
||||
attempt_value = raw.get("attempt")
|
||||
if isinstance(attempt_value, int) and not isinstance(attempt_value, bool):
|
||||
attempt = max(1, attempt_value)
|
||||
started_at = _parse_datetime(raw.get("started_at"))
|
||||
ended_at = _parse_datetime(raw.get("ended_at"))
|
||||
exit_code = raw.get("exit_code")
|
||||
rows.append(
|
||||
{
|
||||
"run_id": run_id,
|
||||
"step_id": _clip(step_id, 200),
|
||||
"attempt": attempt,
|
||||
"status": _clip(raw.get("status"), 32),
|
||||
"exit_code": exit_code if isinstance(exit_code, int) else None,
|
||||
"started_at": started_at,
|
||||
"ended_at": ended_at,
|
||||
"duration_seconds": _seconds_between(started_at, ended_at),
|
||||
"error": _clip(raw.get("error")),
|
||||
}
|
||||
)
|
||||
return rows
|
||||
|
||||
@staticmethod
|
||||
def _step_counts(
|
||||
steps: Mapping[str, Any] | None, record: Mapping[str, Any]
|
||||
) -> dict[str, int]:
|
||||
counts = {"total": 0, "success": 0, "failed": 0, "skipped": 0, "running": 0}
|
||||
if isinstance(steps, Mapping) and steps:
|
||||
for raw in steps.values():
|
||||
if not isinstance(raw, Mapping):
|
||||
continue
|
||||
status = str(raw.get("status", ""))
|
||||
counts["total"] += 1
|
||||
if status in counts:
|
||||
counts[status] += 1
|
||||
return counts
|
||||
# 回退到 RunRecord 的 step_counts 投影
|
||||
projection = record.get("step_counts")
|
||||
if isinstance(projection, Mapping):
|
||||
for name in ("success", "failed", "skipped", "running"):
|
||||
value = projection.get(name)
|
||||
if isinstance(value, int) and not isinstance(value, bool):
|
||||
counts[name] = max(0, value)
|
||||
counts["total"] = counts["success"] + counts["failed"] + counts["skipped"] + counts["running"]
|
||||
return counts
|
||||
|
||||
# -- cookie 事件 ---------------------------------------------------
|
||||
|
||||
def record_cookie_event(
|
||||
self,
|
||||
*,
|
||||
account_id: str,
|
||||
state: str,
|
||||
snapshot: CookieSnapshot | None = None,
|
||||
probe_status: str = "",
|
||||
probe_source: str = "",
|
||||
detail: str = "",
|
||||
workflow_id: str = "",
|
||||
run_id: str = "",
|
||||
observed_at: datetime | None = None,
|
||||
) -> bool:
|
||||
"""Append one observation. 状态必须是白名单值,避免脏数据污染统计。"""
|
||||
|
||||
account_id = _clip(account_id, 200)
|
||||
if not account_id:
|
||||
return False
|
||||
state = str(state or "").strip()
|
||||
if state not in _COOKIE_STATES:
|
||||
raise ValueError(f"invalid cookie state: {state!r}")
|
||||
|
||||
def action(conn: Any) -> None:
|
||||
with conn.cursor() as cursor:
|
||||
cursor.execute(
|
||||
"""
|
||||
INSERT INTO ops_account_cookie_events (
|
||||
account_id, observed_at, state, probe_status, probe_source,
|
||||
cookie_count, required_count, fingerprint, declared_expires_at,
|
||||
detail, workflow_id, run_id, host
|
||||
) VALUES (
|
||||
%(account_id)s, %(observed_at)s, %(state)s, %(probe_status)s,
|
||||
%(probe_source)s, %(cookie_count)s, %(required_count)s,
|
||||
%(fingerprint)s, %(declared_expires_at)s, %(detail)s,
|
||||
%(workflow_id)s, %(run_id)s, %(host)s
|
||||
)
|
||||
""",
|
||||
{
|
||||
"account_id": account_id,
|
||||
"observed_at": observed_at or self._clock(),
|
||||
"state": state,
|
||||
"probe_status": _clip(probe_status, 64),
|
||||
"probe_source": _clip(probe_source, 64),
|
||||
"cookie_count": snapshot.cookie_count if snapshot else None,
|
||||
"required_count": snapshot.required_count if snapshot else None,
|
||||
"fingerprint": snapshot.fingerprint if snapshot else "",
|
||||
"declared_expires_at": (
|
||||
snapshot.declared_expires_at if snapshot else None
|
||||
),
|
||||
"detail": _clip(detail),
|
||||
"workflow_id": _clip(workflow_id, 200),
|
||||
"run_id": _clip(run_id, 200),
|
||||
"host": self._host(),
|
||||
},
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
return self._run_guarded("record_cookie_event", action)
|
||||
|
||||
# -- 查询(供 CLI) ------------------------------------------------
|
||||
|
||||
def query_runs(
|
||||
self,
|
||||
*,
|
||||
workflow_id: str | None = None,
|
||||
status: str | None = None,
|
||||
business_date: str | None = None,
|
||||
limit: int = 50,
|
||||
) -> list[dict[str, Any]]:
|
||||
conditions: list[str] = []
|
||||
params: dict[str, Any] = {"limit": max(1, min(int(limit), 500))}
|
||||
if workflow_id:
|
||||
conditions.append("workflow_id = %(workflow_id)s")
|
||||
params["workflow_id"] = workflow_id
|
||||
if status:
|
||||
conditions.append("status = %(status)s")
|
||||
params["status"] = status
|
||||
if business_date:
|
||||
parsed = _parse_date(business_date)
|
||||
if parsed is not None:
|
||||
conditions.append("business_date = %(business_date)s")
|
||||
params["business_date"] = parsed
|
||||
where = f"WHERE {' AND '.join(conditions)}" if conditions else ""
|
||||
sql = f"""
|
||||
SELECT * FROM ops_workflow_runs
|
||||
{where}
|
||||
ORDER BY started_at DESC NULLS LAST, run_id DESC
|
||||
LIMIT %(limit)s
|
||||
"""
|
||||
return self._fetch(sql, params)
|
||||
|
||||
def query_steps(self, run_id: str) -> list[dict[str, Any]]:
|
||||
return self._fetch(
|
||||
"""
|
||||
SELECT * FROM ops_workflow_run_steps
|
||||
WHERE run_id = %(run_id)s
|
||||
ORDER BY step_id, attempt
|
||||
""",
|
||||
{"run_id": run_id},
|
||||
)
|
||||
|
||||
def query_cookie_events(
|
||||
self,
|
||||
*,
|
||||
account_id: str | None = None,
|
||||
state: str | None = None,
|
||||
limit: int = 50,
|
||||
) -> list[dict[str, Any]]:
|
||||
conditions: list[str] = []
|
||||
params: dict[str, Any] = {"limit": max(1, min(int(limit), 500))}
|
||||
if account_id:
|
||||
conditions.append("account_id = %(account_id)s")
|
||||
params["account_id"] = account_id
|
||||
if state:
|
||||
conditions.append("state = %(state)s")
|
||||
params["state"] = state
|
||||
where = f"WHERE {' AND '.join(conditions)}" if conditions else ""
|
||||
sql = f"""
|
||||
SELECT * FROM ops_account_cookie_events
|
||||
{where}
|
||||
ORDER BY observed_at DESC
|
||||
LIMIT %(limit)s
|
||||
"""
|
||||
return self._fetch(sql, params)
|
||||
|
||||
def query_cookie_lifetimes(self, *, account_id: str | None = None) -> list[dict[str, Any]]:
|
||||
if account_id:
|
||||
return self._fetch(
|
||||
"""
|
||||
SELECT * FROM ops_account_cookie_lifetimes
|
||||
WHERE account_id = %(account_id)s
|
||||
ORDER BY first_seen_at DESC
|
||||
""",
|
||||
{"account_id": account_id},
|
||||
)
|
||||
return self._fetch(
|
||||
"SELECT * FROM ops_account_cookie_lifetimes ORDER BY first_seen_at DESC",
|
||||
{},
|
||||
)
|
||||
|
||||
def query_cookie_stats(self) -> list[dict[str, Any]]:
|
||||
return self._fetch(
|
||||
"SELECT * FROM ops_account_cookie_stats ORDER BY account_id",
|
||||
{},
|
||||
)
|
||||
|
||||
def _fetch(self, sql: str, params: Mapping[str, Any]) -> list[dict[str, Any]]:
|
||||
if not self.enabled or self._in_cooldown():
|
||||
return []
|
||||
try:
|
||||
with self._connection() as conn:
|
||||
self._ensure_schema(conn)
|
||||
with conn.cursor() as cursor:
|
||||
cursor.execute(sql, params)
|
||||
rows = cursor.fetchall()
|
||||
self._consecutive_failures = 0
|
||||
return [dict(row) for row in rows]
|
||||
except Exception as exc: # noqa: BLE001
|
||||
self._record_failure("query", exc)
|
||||
return []
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
# 连接参数(与 PostgresWorkflowConfigStore 同口径)
|
||||
# ----------------------------------------------------------------------
|
||||
|
||||
|
||||
def database_kwargs(environment: Mapping[str, str]) -> tuple[str | None, dict[str, Any]]:
|
||||
values = _apply_runtime_policy(environment)
|
||||
dsn = str(values.get("GYXX_POSTGRES_DSN", "")).strip()
|
||||
if dsn:
|
||||
return dsn, {}
|
||||
required = ("PG_HOST", "PG_PORT", "PG_DB", "PG_USER", "PG_PASSWORD")
|
||||
missing = [name for name in required if not str(values.get(name, "")).strip()]
|
||||
if missing:
|
||||
raise RuntimeError("审计数据库配置缺失: " + ", ".join(missing))
|
||||
try:
|
||||
port = int(values["PG_PORT"])
|
||||
except (TypeError, ValueError) as exc:
|
||||
raise RuntimeError("PG_PORT 必须是整数") from exc
|
||||
return None, {
|
||||
"host": str(values["PG_HOST"]).strip(),
|
||||
"port": port,
|
||||
"dbname": str(values["PG_DB"]).strip(),
|
||||
"user": str(values["PG_USER"]).strip(),
|
||||
"password": str(values["PG_PASSWORD"]),
|
||||
}
|
||||
|
||||
|
||||
def _apply_runtime_policy(environment: Mapping[str, str]) -> Mapping[str, str]:
|
||||
"""复用 RuntimeServicePolicy,保证与控制台/工作流看到同一个数据库地址。"""
|
||||
|
||||
try:
|
||||
from gyxx_flow.adapters import RuntimeServicePolicy
|
||||
except Exception: # pragma: no cover - 循环导入兜底
|
||||
return environment
|
||||
try:
|
||||
return RuntimeServicePolicy().apply(dict(environment))
|
||||
except Exception: # pragma: no cover
|
||||
return environment
|
||||
|
||||
|
||||
def default_store(data_root: Path | str | None = None) -> AuditStore:
|
||||
"""Best-effort store resolved from the standard data root."""
|
||||
|
||||
if data_root is None:
|
||||
try:
|
||||
from gyxx_flow.core.config import Settings
|
||||
|
||||
data_root = Settings.from_env().data_root
|
||||
except Exception: # pragma: no cover
|
||||
data_root = Path.cwd() / "var"
|
||||
return AuditStore(data_root)
|
||||
@@ -86,6 +86,7 @@ class RunIndex:
|
||||
resource = f"ops:run-index:{_digest(record.run_id)}"
|
||||
with self._locks.acquire(resource, owner=_owner(), timeout_seconds=30):
|
||||
atomic_write_json(destination, record.to_dict())
|
||||
_mirror_run_to_audit(self._data_root, record, raw)
|
||||
return record
|
||||
|
||||
def get(self, run_id: str) -> RunRecord | None:
|
||||
@@ -255,3 +256,42 @@ def _digest(value: str) -> str:
|
||||
import hashlib
|
||||
|
||||
return hashlib.sha256(value.encode("utf-8")).hexdigest()[:32]
|
||||
|
||||
|
||||
_AUDIT_STORES: dict[str, Any] = {}
|
||||
|
||||
|
||||
def _audit_store_for(data_root: Path) -> Any:
|
||||
"""Reuse one store per data root so schema bootstrap and the circuit breaker
|
||||
survive the short-lived ``RunIndex`` instances created per run."""
|
||||
|
||||
key = str(data_root)
|
||||
store = _AUDIT_STORES.get(key)
|
||||
if store is None:
|
||||
from gyxx_flow.ops.audit import AuditStore
|
||||
|
||||
store = AuditStore(data_root)
|
||||
_AUDIT_STORES[key] = store
|
||||
return store
|
||||
|
||||
|
||||
def _mirror_run_to_audit(
|
||||
data_root: Path,
|
||||
record: RunRecord,
|
||||
journal_payload: Any,
|
||||
) -> None:
|
||||
"""Best-effort mirror of a run into the PostgreSQL audit tables.
|
||||
|
||||
Auditing is strictly additive: any failure here (missing DSN, unreachable
|
||||
database, unexpected schema state) is swallowed so that journal indexing and
|
||||
the workflow itself are never affected.
|
||||
"""
|
||||
|
||||
try:
|
||||
store = _audit_store_for(data_root)
|
||||
if not store.enabled:
|
||||
return
|
||||
steps = journal_payload.get("steps") if isinstance(journal_payload, dict) else None
|
||||
store.record_run(record.to_dict(), steps)
|
||||
except Exception: # noqa: BLE001 - audit mirroring must never raise
|
||||
return
|
||||
@@ -0,0 +1,152 @@
|
||||
-- Migration 001: operational audit tables (workflow runs + login cookie telemetry).
|
||||
--
|
||||
-- 本文件由 src/gyxx_flow/ops/audit.py 的 SCHEMA_SQL 导出,请勿手工编辑;
|
||||
-- 运行时首次写入会自动执行同一份 DDL(幂等),此处仅供 DBA 审阅或预建。
|
||||
--
|
||||
-- 用法: psql "$GYXX_POSTGRES_DSN" -f 001_ops_audit.sql
|
||||
--
|
||||
-- 说明:
|
||||
-- ops_workflow_runs 一次工作流运行一行,run.json 的关系型投影
|
||||
-- ops_workflow_run_steps 每次尝试一行(含重试),保留 exit_code 与错误
|
||||
-- ops_account_cookie_events cookie 观测事件流,只存指纹不存明文值
|
||||
-- ops_account_cookie_lifetimes 一处登录态从首次出现到失效(视图)
|
||||
-- ops_account_cookie_stats 每账号失效规律汇总(视图)
|
||||
--
|
||||
-- 日志正文仍留在 var/logs 下,数据库只存结构化元数据与指向路径。
|
||||
|
||||
CREATE TABLE IF NOT EXISTS ops_workflow_runs (
|
||||
run_id TEXT PRIMARY KEY,
|
||||
workflow_id TEXT NOT NULL,
|
||||
business_date DATE NOT NULL,
|
||||
mode TEXT NOT NULL DEFAULT 'unknown',
|
||||
shadow BOOLEAN NOT NULL DEFAULT FALSE,
|
||||
status TEXT NOT NULL,
|
||||
started_at TIMESTAMPTZ,
|
||||
ended_at TIMESTAMPTZ,
|
||||
duration_seconds INTEGER,
|
||||
error TEXT,
|
||||
step_total INTEGER NOT NULL DEFAULT 0,
|
||||
step_success INTEGER NOT NULL DEFAULT 0,
|
||||
step_failed INTEGER NOT NULL DEFAULT 0,
|
||||
step_skipped INTEGER NOT NULL DEFAULT 0,
|
||||
step_running INTEGER NOT NULL DEFAULT 0,
|
||||
journal_path TEXT NOT NULL DEFAULT '',
|
||||
host TEXT NOT NULL DEFAULT '',
|
||||
indexed_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS ops_workflow_runs_workflow_idx
|
||||
ON ops_workflow_runs (workflow_id, started_at DESC);
|
||||
CREATE INDEX IF NOT EXISTS ops_workflow_runs_status_idx
|
||||
ON ops_workflow_runs (status, started_at DESC);
|
||||
CREATE INDEX IF NOT EXISTS ops_workflow_runs_business_date_idx
|
||||
ON ops_workflow_runs (business_date, workflow_id);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS ops_workflow_run_steps (
|
||||
run_id TEXT NOT NULL
|
||||
REFERENCES ops_workflow_runs (run_id) ON DELETE CASCADE,
|
||||
step_id TEXT NOT NULL,
|
||||
attempt INTEGER NOT NULL DEFAULT 1,
|
||||
status TEXT NOT NULL DEFAULT '',
|
||||
exit_code INTEGER,
|
||||
started_at TIMESTAMPTZ,
|
||||
ended_at TIMESTAMPTZ,
|
||||
duration_seconds INTEGER,
|
||||
error TEXT,
|
||||
PRIMARY KEY (run_id, step_id, attempt)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS ops_workflow_run_steps_step_idx
|
||||
ON ops_workflow_run_steps (step_id, status);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS ops_account_cookie_events (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
account_id TEXT NOT NULL,
|
||||
observed_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
state TEXT NOT NULL,
|
||||
probe_status TEXT NOT NULL DEFAULT '',
|
||||
probe_source TEXT NOT NULL DEFAULT '',
|
||||
cookie_count INTEGER,
|
||||
required_count INTEGER,
|
||||
fingerprint TEXT NOT NULL DEFAULT '',
|
||||
declared_expires_at TIMESTAMPTZ,
|
||||
detail TEXT NOT NULL DEFAULT '',
|
||||
workflow_id TEXT NOT NULL DEFAULT '',
|
||||
run_id TEXT NOT NULL DEFAULT '',
|
||||
host TEXT NOT NULL DEFAULT ''
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS ops_account_cookie_events_account_idx
|
||||
ON ops_account_cookie_events (account_id, observed_at DESC);
|
||||
CREATE INDEX IF NOT EXISTS ops_account_cookie_events_state_idx
|
||||
ON ops_account_cookie_events (state, observed_at DESC);
|
||||
CREATE INDEX IF NOT EXISTS ops_account_cookie_events_fingerprint_idx
|
||||
ON ops_account_cookie_events (account_id, fingerprint);
|
||||
|
||||
-- 一段「cookie 代」的生命周期:同一 fingerprint 从首次出现到失效。
|
||||
-- first_failed_at 只取最后一次 healthy 之后的失败,避免恢复后再次失败时口径前移。
|
||||
CREATE OR REPLACE VIEW ops_account_cookie_lifetimes AS
|
||||
WITH per_fingerprint AS (
|
||||
SELECT
|
||||
account_id,
|
||||
fingerprint,
|
||||
MIN(observed_at) AS first_seen_at,
|
||||
MIN(observed_at) FILTER (WHERE state = 'healthy') AS first_healthy_at,
|
||||
MAX(observed_at) FILTER (WHERE state = 'healthy') AS last_healthy_at,
|
||||
MAX(declared_expires_at) AS declared_expires_at
|
||||
FROM ops_account_cookie_events
|
||||
WHERE fingerprint <> ''
|
||||
GROUP BY account_id, fingerprint
|
||||
)
|
||||
SELECT
|
||||
p.account_id,
|
||||
p.fingerprint,
|
||||
p.first_seen_at,
|
||||
p.first_healthy_at,
|
||||
p.last_healthy_at,
|
||||
p.declared_expires_at,
|
||||
f.first_failed_at,
|
||||
(f.first_failed_at IS NOT NULL) AS expired,
|
||||
CASE
|
||||
WHEN p.first_healthy_at IS NOT NULL AND p.last_healthy_at IS NOT NULL
|
||||
THEN EXTRACT(EPOCH FROM (p.last_healthy_at - p.first_healthy_at))::BIGINT
|
||||
END AS healthy_span_seconds,
|
||||
CASE
|
||||
WHEN p.first_healthy_at IS NOT NULL AND f.first_failed_at IS NOT NULL
|
||||
THEN EXTRACT(EPOCH FROM (f.first_failed_at - p.first_healthy_at))::BIGINT
|
||||
END AS lifetime_seconds,
|
||||
CASE
|
||||
WHEN p.last_healthy_at IS NOT NULL AND f.first_failed_at IS NOT NULL
|
||||
THEN EXTRACT(EPOCH FROM (f.first_failed_at - p.last_healthy_at))::BIGINT
|
||||
END AS healthy_to_failure_seconds
|
||||
FROM per_fingerprint p
|
||||
LEFT JOIN LATERAL (
|
||||
SELECT MIN(e.observed_at) AS first_failed_at
|
||||
FROM ops_account_cookie_events e
|
||||
WHERE e.account_id = p.account_id
|
||||
AND e.fingerprint = p.fingerprint
|
||||
AND e.state IN ('expired', 'not_logged_in')
|
||||
AND (p.last_healthy_at IS NULL OR e.observed_at > p.last_healthy_at)
|
||||
) f ON TRUE;
|
||||
|
||||
-- 按账号汇总的失效规律:平均/中位/最短/最长寿命,直接支撑「多久失效一次」。
|
||||
CREATE OR REPLACE VIEW ops_account_cookie_stats AS
|
||||
SELECT
|
||||
account_id,
|
||||
COUNT(*) AS sessions,
|
||||
COUNT(*) FILTER (WHERE expired) AS expired_sessions,
|
||||
AVG(lifetime_seconds)::BIGINT AS avg_lifetime_seconds,
|
||||
PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY lifetime_seconds)::BIGINT
|
||||
AS median_lifetime_seconds,
|
||||
MIN(lifetime_seconds) AS min_lifetime_seconds,
|
||||
MAX(lifetime_seconds) AS max_lifetime_seconds,
|
||||
MAX(first_failed_at) AS last_failed_at,
|
||||
MAX(last_healthy_at) AS last_healthy_at
|
||||
FROM ops_account_cookie_lifetimes
|
||||
WHERE lifetime_seconds IS NOT NULL
|
||||
GROUP BY account_id;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS ops_audit_schema (
|
||||
schema_version INTEGER PRIMARY KEY,
|
||||
applied_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
@@ -1,4 +1,7 @@
|
||||
import json
|
||||
import time
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
@@ -32,7 +35,7 @@ def test_refresh_then_export_waits_once_after_all_accounts(monkeypatch):
|
||||
monkeypatch.setattr(chanmama, "export_video_data", lambda _driver, mtime: events.append(("export", mtime)) or "x.xlsx")
|
||||
monkeypatch.setattr(chanmama, "parse_chanmama_excel", lambda _path: ([{"title": "video"}], None))
|
||||
|
||||
records, summaries = chanmama.refresh_then_export_accounts(object(), urls, wait_seconds=600)
|
||||
records, summaries, _driver = chanmama.refresh_then_export_accounts(object(), urls, wait_seconds=600)
|
||||
|
||||
assert len(records) == 2
|
||||
assert [item["clicked"] for item in summaries] == [2, 0]
|
||||
@@ -53,7 +56,7 @@ def test_refresh_then_export_skips_wait_when_nothing_needs_refresh(monkeypatch):
|
||||
monkeypatch.setattr(chanmama, "_max_excel_mtime", lambda: 0.0)
|
||||
monkeypatch.setattr(chanmama, "export_video_data", lambda *_args: None)
|
||||
|
||||
records, _ = chanmama.refresh_then_export_accounts(object(), ["account-1"], wait_seconds=600)
|
||||
records, _, _driver = chanmama.refresh_then_export_accounts(object(), ["account-1"], wait_seconds=600)
|
||||
|
||||
assert records == []
|
||||
assert sleeps == []
|
||||
@@ -158,7 +161,6 @@ def test_missing_get_data_button_does_not_trigger_refresh_wait(monkeypatch):
|
||||
chanmama.refresh_then_export_accounts(object(), ["account-1"], wait_seconds=600)
|
||||
assert sleeps == []
|
||||
|
||||
|
||||
def test_create_driver_uses_runtime_cdp_profile_and_download_bindings(monkeypatch, tmp_path):
|
||||
calls = []
|
||||
expected = object()
|
||||
@@ -182,7 +184,9 @@ def test_create_driver_uses_runtime_cdp_profile_and_download_bindings(monkeypatc
|
||||
"profile_dir": str(tmp_path / "profile"),
|
||||
"download_dir": str(tmp_path / "downloads"),
|
||||
"cdp_url": "http://127.0.0.1:22002",
|
||||
"stealthy": True,
|
||||
"stealthy": False,
|
||||
"real_chrome": True,
|
||||
"extra_flags": ["--no-proxy-server"],
|
||||
}]
|
||||
|
||||
|
||||
@@ -205,8 +209,10 @@ def test_create_driver_starts_scrapling_without_cdp(monkeypatch, tmp_path):
|
||||
|
||||
assert chanmama.create_driver() is expected
|
||||
assert calls[0]["cdp_url"] is None
|
||||
assert calls[0]["stealthy"] is True
|
||||
assert calls[0]["real_chrome"] is False
|
||||
# 蝉妈妈 WAF 拉黑代理出口且压制 stealthy 补丁,必须真 Chrome + 直连。
|
||||
assert calls[0]["stealthy"] is False
|
||||
assert calls[0]["real_chrome"] is True
|
||||
assert calls[0]["extra_flags"] == ["--no-proxy-server"]
|
||||
|
||||
|
||||
def test_create_driver_falls_back_when_bound_cdp_is_not_listening(
|
||||
@@ -240,8 +246,8 @@ def test_create_driver_falls_back_when_bound_cdp_is_not_listening(
|
||||
"http://127.0.0.1:22002",
|
||||
None,
|
||||
]
|
||||
assert all(call["stealthy"] is True for call in calls)
|
||||
assert calls[1]["real_chrome"] is False
|
||||
assert all(call["stealthy"] is False for call in calls)
|
||||
assert all(call["real_chrome"] is True for call in calls)
|
||||
|
||||
|
||||
def test_create_driver_keeps_explicit_remote_cdp_fail_closed(
|
||||
@@ -723,3 +729,357 @@ def test_wait_for_exported_excel_tolerates_driver_errors(monkeypatch):
|
||||
assert chanmama._wait_for_exported_excel(0.0, driver=_BrokenDriver()) == (
|
||||
"/downloads/已有.xlsx"
|
||||
)
|
||||
|
||||
|
||||
def test_refresh_then_export_relogin_on_expired_session(monkeypatch):
|
||||
"""运行中途登录态失效: 自动账密重登一次并重试当前博主,不影响其他博主。"""
|
||||
events = []
|
||||
navigate_calls = []
|
||||
|
||||
def fake_navigate(_driver, url):
|
||||
navigate_calls.append(url)
|
||||
if len(navigate_calls) == 1:
|
||||
raise RuntimeError(
|
||||
"蝉妈妈登录态失效,目标页被重定向到: "
|
||||
"https://www.chanmama.com/login.html"
|
||||
)
|
||||
|
||||
monkeypatch.setattr(chanmama, "navigate_to_target", fake_navigate)
|
||||
monkeypatch.setattr(
|
||||
chanmama,
|
||||
"refresh_zero_exposure_videos",
|
||||
lambda _driver: {
|
||||
"clicked": 0, "pending": 0, "zero_without_button": 0,
|
||||
"unconfirmed": 0, "pages": 1,
|
||||
},
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
chanmama, "login",
|
||||
lambda _driver: events.append("login") or True,
|
||||
)
|
||||
monkeypatch.setattr(chanmama.time, "sleep", lambda _seconds: None)
|
||||
monkeypatch.setattr(chanmama, "_max_excel_mtime", lambda: 0.0)
|
||||
monkeypatch.setattr(
|
||||
chanmama, "export_video_data", lambda _driver, mtime: "x.xlsx"
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
chanmama, "parse_chanmama_excel", lambda _path: ([{"title": "v"}], None)
|
||||
)
|
||||
|
||||
records, summaries, _driver = chanmama.refresh_then_export_accounts(
|
||||
object(), ["account-1", "account-2"], wait_seconds=0
|
||||
)
|
||||
|
||||
assert events == ["login"] # 整套运行只自动重登一次
|
||||
assert len(records) == 2 # 两个博主最终都导出成功
|
||||
assert summaries[0]["pages"] == 1 # 博主 1 重登后刷新成功
|
||||
|
||||
|
||||
def test_refresh_then_export_no_relogin_for_other_errors(monkeypatch):
|
||||
"""非登录态失效的异常不触发自动重登。"""
|
||||
events = []
|
||||
|
||||
def fake_navigate(_driver, url):
|
||||
raise RuntimeError("蝉妈妈目标页加载失败: timeout")
|
||||
|
||||
monkeypatch.setattr(chanmama, "navigate_to_target", fake_navigate)
|
||||
monkeypatch.setattr(
|
||||
chanmama, "login",
|
||||
lambda _driver: events.append("login") or True,
|
||||
)
|
||||
monkeypatch.setattr(chanmama.time, "sleep", lambda _seconds: None)
|
||||
monkeypatch.setattr(
|
||||
chanmama, "refresh_zero_exposure_videos", lambda _driver: {}
|
||||
)
|
||||
monkeypatch.setattr(chanmama, "_max_excel_mtime", lambda: 0.0)
|
||||
monkeypatch.setattr(
|
||||
chanmama, "export_video_data", lambda _driver, mtime: None
|
||||
)
|
||||
|
||||
records, _summaries, _driver = chanmama.refresh_then_export_accounts(
|
||||
object(), ["account-1"], wait_seconds=0
|
||||
)
|
||||
|
||||
assert events == [] # 普通异常不触发自动重登
|
||||
assert records == [] # 导出失败不产出记录
|
||||
|
||||
|
||||
class _RenderFakeDriver:
|
||||
"""navigate_to_target 渲染链路的桩:列表只在浮层清理后才就绪。"""
|
||||
|
||||
def __init__(self, ready_after_dismiss=1):
|
||||
self.current_url = (
|
||||
"https://www.chanmama.com/bloggerRank/x.html?activeTab=aweme"
|
||||
)
|
||||
self.visited = []
|
||||
self.dismiss_runs = 0
|
||||
self.ready_after_dismiss = ready_after_dismiss
|
||||
self.screenshots = []
|
||||
|
||||
def get(self, url):
|
||||
self.visited.append(url)
|
||||
|
||||
def find_element(self, _by, _value):
|
||||
return object()
|
||||
|
||||
def execute_script(self, script, *_args):
|
||||
if script == chanmama._OVERLAY_DISMISS_SCRIPT:
|
||||
self.dismiss_runs += 1
|
||||
return 2
|
||||
if script == chanmama._VIDEO_LIST_READY_SCRIPT:
|
||||
return self.dismiss_runs >= self.ready_after_dismiss
|
||||
raise AssertionError(f"unexpected script: {script[:60]}")
|
||||
|
||||
def save_screenshot(self, path):
|
||||
self.screenshots.append(str(path))
|
||||
Path(path).write_bytes(b"png")
|
||||
return True
|
||||
|
||||
@property
|
||||
def page_source(self):
|
||||
return "<html>diag</html>"
|
||||
|
||||
|
||||
def test_navigate_to_target_clears_overlays_before_render_wait(monkeypatch):
|
||||
monkeypatch.setattr(chanmama.time, "sleep", lambda _seconds: None)
|
||||
driver = _RenderFakeDriver(ready_after_dismiss=1)
|
||||
|
||||
assert chanmama.navigate_to_target(
|
||||
driver, "https://www.chanmama.com/bloggerRank/x.html"
|
||||
) is True
|
||||
|
||||
assert driver.dismiss_runs >= 1 # 弹窗清理后才轮到渲染判定
|
||||
assert driver.visited == ["https://www.chanmama.com/bloggerRank/x.html"]
|
||||
|
||||
|
||||
def test_navigate_to_target_timeout_saves_diagnostics(monkeypatch, tmp_path):
|
||||
monkeypatch.setattr(chanmama.time, "sleep", lambda _seconds: None)
|
||||
monkeypatch.setattr(chanmama, "CHANMAMA_LIST_READY_TIMEOUT_SECONDS", 0.05)
|
||||
monkeypatch.setattr(chanmama, "DOWNLOAD_DIR", str(tmp_path))
|
||||
driver = _RenderFakeDriver(ready_after_dismiss=99)
|
||||
|
||||
with pytest.raises(RuntimeError, match="蝉妈妈目标页加载失败"):
|
||||
chanmama.navigate_to_target(
|
||||
driver, "https://www.chanmama.com/bloggerRank/x.html"
|
||||
)
|
||||
|
||||
assert driver.dismiss_runs >= 2 # 两次尝试各自清理过浮层
|
||||
assert len(driver.screenshots) == 1
|
||||
diag_dir = tmp_path / "_diag"
|
||||
assert list(diag_dir.glob("render_failure_*.png"))
|
||||
assert list(diag_dir.glob("render_failure_*.html"))
|
||||
|
||||
|
||||
def test_dismiss_overlays_ignores_driver_errors():
|
||||
class BoomDriver:
|
||||
def execute_script(self, _script, *_args):
|
||||
raise RuntimeError("execution context destroyed")
|
||||
|
||||
assert chanmama._dismiss_overlays(BoomDriver()) == 0
|
||||
|
||||
|
||||
def test_wait_video_list_ready_tolerates_transient_script_errors(monkeypatch):
|
||||
monkeypatch.setattr(chanmama.time, "sleep", lambda _seconds: None)
|
||||
|
||||
class FlakyDriver:
|
||||
def __init__(self):
|
||||
self.ready_calls = 0
|
||||
|
||||
def execute_script(self, script, *_args):
|
||||
if script == chanmama._OVERLAY_DISMISS_SCRIPT:
|
||||
return 0
|
||||
self.ready_calls += 1
|
||||
if self.ready_calls == 1:
|
||||
raise RuntimeError("frame detached")
|
||||
return True
|
||||
|
||||
chanmama._wait_video_list_ready(FlakyDriver(), 2) # 不抛即通过
|
||||
|
||||
|
||||
def test_wait_for_exported_excel_raises_when_browser_closed_without_file(monkeypatch):
|
||||
"""浏览器被外部关闭且文件未落盘: 立刻抛出关闭异常,不再空等满超时。"""
|
||||
|
||||
class _DeadDriver:
|
||||
def execute_script(self, _script, *_args):
|
||||
raise RuntimeError(
|
||||
"Page.goto: Target page, context or browser has been closed"
|
||||
)
|
||||
|
||||
monkeypatch.setattr(chanmama.time, "time", lambda: 1000.0)
|
||||
monkeypatch.setattr(
|
||||
chanmama.time,
|
||||
"sleep",
|
||||
lambda _seconds: pytest.fail("浏览器已死不应继续轮询等待"),
|
||||
)
|
||||
monkeypatch.setattr(chanmama, "_find_new_excel_after", lambda _pre_mtime: None)
|
||||
|
||||
with pytest.raises(RuntimeError, match="closed"):
|
||||
chanmama._wait_for_exported_excel(0.0, driver=_DeadDriver())
|
||||
|
||||
|
||||
def test_wait_for_exported_excel_returns_file_if_landed_before_close(monkeypatch):
|
||||
"""关闭前下载已完整落盘: 仍返回文件路径,不因浏览器死亡误报失败。"""
|
||||
|
||||
class _DeadDriver:
|
||||
def execute_script(self, _script, *_args):
|
||||
raise RuntimeError(
|
||||
"Download.save_as: Target page, context or browser has been closed"
|
||||
)
|
||||
|
||||
landed = None
|
||||
|
||||
def fake_find(_pre_mtime):
|
||||
return landed
|
||||
|
||||
def fake_complete(path):
|
||||
return True
|
||||
|
||||
monkeypatch.setattr(chanmama.time, "time", lambda: 1000.0)
|
||||
monkeypatch.setattr(chanmama, "_find_new_excel_after", fake_find)
|
||||
monkeypatch.setattr(chanmama, "_download_complete", fake_complete)
|
||||
|
||||
landed = "/downloads/关闭前已落盘.xlsx"
|
||||
assert chanmama._wait_for_exported_excel(0.0, driver=_DeadDriver()) == landed
|
||||
|
||||
|
||||
def test_refresh_then_export_rebuilds_browser_when_closed_mid_run(monkeypatch):
|
||||
"""导出中途浏览器被外部关闭: 重建驱动+恢复会话后重试当前博主一次。"""
|
||||
events = []
|
||||
navigate_calls = []
|
||||
dead_driver = object()
|
||||
live_driver = object()
|
||||
|
||||
def fake_navigate(current, url):
|
||||
navigate_calls.append((current, url))
|
||||
if current is dead_driver and len(navigate_calls) == 3:
|
||||
# 刷新阶段两个博主都成功后,导出阶段第一次导航即发现浏览器死亡
|
||||
raise RuntimeError(
|
||||
"蝉妈妈目标页加载失败: x.html; Page.goto: "
|
||||
"Target page, context or browser has been closed"
|
||||
)
|
||||
|
||||
monkeypatch.setattr(chanmama, "navigate_to_target", fake_navigate)
|
||||
monkeypatch.setattr(
|
||||
chanmama,
|
||||
"refresh_zero_exposure_videos",
|
||||
lambda _driver: {
|
||||
"clicked": 0, "pending": 0, "zero_without_button": 0,
|
||||
"unconfirmed": 0, "pages": 1,
|
||||
},
|
||||
)
|
||||
monkeypatch.setattr(chanmama.time, "sleep", lambda _seconds: None)
|
||||
monkeypatch.setattr(chanmama, "_max_excel_mtime", lambda: 0.0)
|
||||
monkeypatch.setattr(
|
||||
chanmama, "export_video_data", lambda _driver, mtime: "x.xlsx"
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
chanmama, "parse_chanmama_excel", lambda _path: ([{"title": "v"}], None)
|
||||
)
|
||||
monkeypatch.setattr(chanmama, "verify_login", lambda _driver: True)
|
||||
monkeypatch.setattr(
|
||||
chanmama, "create_driver", lambda: events.append("rebuild") or live_driver
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
chanmama, "login",
|
||||
lambda _driver: events.append("login") or True,
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
chanmama, "load_cookies",
|
||||
lambda _driver: (_ for _ in ()).throw(
|
||||
AssertionError("profile 直接恢复,不应走 cookie 文件")
|
||||
),
|
||||
)
|
||||
|
||||
class _DeadDriver:
|
||||
def quit(self):
|
||||
events.append("quit-dead")
|
||||
|
||||
dead_driver = _DeadDriver()
|
||||
|
||||
records, _summaries, final_driver = chanmama.refresh_then_export_accounts(
|
||||
dead_driver, ["account-1", "account-2"], wait_seconds=0
|
||||
)
|
||||
|
||||
assert events == ["quit-dead", "rebuild"] # 旧驱动清理+重建一次
|
||||
assert final_driver is live_driver # 返回重建后的驱动供收尾关闭
|
||||
assert len(records) == 2 # 两个博主最终都导出成功
|
||||
# 第 3 次导航(导出博主 1)发现死亡,其后所有导航都在重建后的驱动上
|
||||
assert all(current is live_driver for current, _ in navigate_calls[3:])
|
||||
|
||||
|
||||
def test_refresh_then_export_rebuilds_only_once(monkeypatch):
|
||||
"""环境持续杀浏览器时只重建一次,后续博主直接失败不再死循环。"""
|
||||
rebuilds = []
|
||||
|
||||
def always_closed_navigate(_driver, _url):
|
||||
raise RuntimeError(
|
||||
"Page.goto: Target page, context or browser has been closed"
|
||||
)
|
||||
|
||||
class _Driver:
|
||||
def quit(self):
|
||||
pass
|
||||
|
||||
monkeypatch.setattr(chanmama, "navigate_to_target", always_closed_navigate)
|
||||
monkeypatch.setattr(
|
||||
chanmama, "create_driver", lambda: rebuilds.append(1) or _Driver()
|
||||
)
|
||||
monkeypatch.setattr(chanmama, "verify_login", lambda _driver: True)
|
||||
monkeypatch.setattr(chanmama.time, "sleep", lambda _seconds: None)
|
||||
|
||||
records, _summaries, _driver = chanmama.refresh_then_export_accounts(
|
||||
_Driver(), ["account-1", "account-2"], wait_seconds=0
|
||||
)
|
||||
|
||||
assert len(rebuilds) == 1
|
||||
assert records == []
|
||||
|
||||
|
||||
def test_save_cookies_writes_adapter_list_format(monkeypatch, tmp_path):
|
||||
"""cookie 文件与 vault/适配器共用,必须写裸列表而不是字典。"""
|
||||
cookie_file = tmp_path / "cookies.json"
|
||||
monkeypatch.setattr(chanmama, "COOKIE_FILE", str(cookie_file))
|
||||
|
||||
class _Driver:
|
||||
@staticmethod
|
||||
def get_cookies():
|
||||
return [{"name": "session", "domain": ".chanmama.com"}]
|
||||
|
||||
assert chanmama.save_cookies(_Driver()) is True
|
||||
payload = json.loads(cookie_file.read_text(encoding="utf-8"))
|
||||
assert payload == [{"name": "session", "domain": ".chanmama.com"}]
|
||||
|
||||
|
||||
def test_load_cookies_accepts_list_and_legacy_dict(monkeypatch, tmp_path):
|
||||
cookie_file = tmp_path / "cookies.json"
|
||||
|
||||
injected = []
|
||||
|
||||
class _Driver:
|
||||
@staticmethod
|
||||
def get(_url):
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def add_cookie(cookie):
|
||||
injected.append(cookie)
|
||||
|
||||
monkeypatch.setattr(chanmama, "COOKIE_FILE", str(cookie_file))
|
||||
monkeypatch.setattr(chanmama.time, "sleep", lambda _seconds: None)
|
||||
|
||||
cookie_file.write_text(
|
||||
json.dumps([{"name": "a", "domain": ".chanmama.com"}]),
|
||||
encoding="utf-8",
|
||||
)
|
||||
assert chanmama.load_cookies(_Driver()) is True
|
||||
assert [c["name"] for c in injected] == ["a"]
|
||||
|
||||
injected.clear()
|
||||
cookie_file.write_text(
|
||||
json.dumps(
|
||||
{"cookies": [{"name": "b", "domain": ".chanmama.com"}], "timestamp": time.time()}
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
assert chanmama.load_cookies(_Driver()) is True
|
||||
assert [c["name"] for c in injected] == ["b"]
|
||||
@@ -21,20 +21,11 @@ def test_page_crash_requires_a_fresh_browser():
|
||||
|
||||
|
||||
@pytest.mark.parametrize("batch", [xhs_batch, douyin_batch])
|
||||
def test_non_acceptance_relogin_runs_platform_script(monkeypatch, batch):
|
||||
class Policy:
|
||||
enabled = False
|
||||
def test_batch_tools_no_longer_embed_relogin(batch):
|
||||
"""扫码重登已统一到账号 vault(gyxx accounts login),工具内不再提供。"""
|
||||
|
||||
calls = []
|
||||
monkeypatch.setattr(batch, "current_acceptance_policy", lambda: Policy())
|
||||
monkeypatch.setattr(
|
||||
batch.subprocess,
|
||||
"call",
|
||||
lambda command, **kwargs: calls.append((command, kwargs)) or 0,
|
||||
)
|
||||
|
||||
assert batch.do_relogin() is True
|
||||
assert calls[0][0][1] == str(batch.RELOGIN_SCRIPT)
|
||||
assert not hasattr(batch, "do_relogin")
|
||||
assert not hasattr(batch, "RELOGIN_SCRIPT")
|
||||
|
||||
|
||||
def test_comment_checkpoint_matches_both_note_id_and_url(tmp_path):
|
||||
|
||||
@@ -10,7 +10,6 @@ from gyxx_flow.modules.content_marketing.data.tools import (
|
||||
analyze_comments,
|
||||
analyze_note,
|
||||
daily_report_card,
|
||||
friday_relogin_parallel,
|
||||
)
|
||||
|
||||
|
||||
@@ -138,15 +137,3 @@ def test_note_report_sends_only_to_wang_yunlong(
|
||||
|
||||
assert calls[0]["user_id"] == ANALYZER_ACCEPTANCE_RECIPIENT_OPEN_ID
|
||||
assert calls[0]["profile"] == "hermes-analyzer"
|
||||
|
||||
|
||||
def test_relogin_notifications_resolve_only_to_wang_yunlong(
|
||||
monkeypatch,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
_enable_acceptance(monkeypatch, tmp_path)
|
||||
|
||||
assert (
|
||||
friday_relogin_parallel.recipient_for_platform("pgy", "ou_other")
|
||||
== ANALYZER_ACCEPTANCE_RECIPIENT_OPEN_ID
|
||||
)
|
||||
@@ -1,569 +0,0 @@
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
|
||||
from gyxx_flow.modules.content_marketing.data.tools import (
|
||||
friday_relogin_parallel as relogin,
|
||||
)
|
||||
from gyxx_flow.modules.content_marketing.data.tools import relogin_transaction
|
||||
|
||||
|
||||
def test_five_expected_platforms_are_configured():
|
||||
assert set(relogin.PLATFORMS) == {
|
||||
"bilibili", "douyin", "pgy", "xingtu", "xiaohongshu"
|
||||
}
|
||||
assert relogin.PLATFORMS["xingtu"].required_cookie == "sessionid"
|
||||
assert relogin.PLATFORMS["douyin"].required_cookie == "sessionid"
|
||||
|
||||
|
||||
def test_manual_relogin_can_select_only_xingtu():
|
||||
assert relogin.resolve_platform_names(["xingtu"]) == ["xingtu"]
|
||||
assert relogin.resolve_platform_names(None) == list(relogin.PLATFORMS)
|
||||
|
||||
|
||||
def test_default_screenshot_recipients_are_routed_by_platform():
|
||||
assert relogin.DEFAULT_RECIPIENTS == {
|
||||
"pgy": "ou_24cc944d6e43c69c59d6560ad4e2ae6e",
|
||||
"xingtu": "ou_24cc944d6e43c69c59d6560ad4e2ae6e",
|
||||
"douyin": "ou_7ad5fc8012e2f741afc5346e05ffd447",
|
||||
"bilibili": "ou_7ad5fc8012e2f741afc5346e05ffd447",
|
||||
"xiaohongshu": "ou_7ad5fc8012e2f741afc5346e05ffd447",
|
||||
}
|
||||
assert relogin.recipient_for_platform("pgy") == "ou_24cc944d6e43c69c59d6560ad4e2ae6e"
|
||||
assert relogin.recipient_for_platform("douyin") == "ou_7ad5fc8012e2f741afc5346e05ffd447"
|
||||
assert relogin.recipient_for_platform("bilibili") == "ou_7ad5fc8012e2f741afc5346e05ffd447"
|
||||
assert relogin.recipient_for_platform("pgy", "ou_override") == "ou_override"
|
||||
|
||||
|
||||
def test_failed_platforms_are_grouped_by_their_responsible_recipient():
|
||||
grouped = relogin.group_platforms_by_recipient(
|
||||
["pgy", "douyin", "bilibili", "xingtu", "xiaohongshu"]
|
||||
)
|
||||
assert grouped == {
|
||||
"ou_24cc944d6e43c69c59d6560ad4e2ae6e": ["pgy", "xingtu"],
|
||||
"ou_7ad5fc8012e2f741afc5346e05ffd447": [
|
||||
"douyin", "bilibili", "xiaohongshu"
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def test_legacy_route_keeps_the_existing_per_platform_recipients(monkeypatch):
|
||||
observed = {}
|
||||
|
||||
def fake_resolver(workflow_id, defaults):
|
||||
observed["workflow_id"] = workflow_id
|
||||
observed["defaults"] = defaults
|
||||
return relogin.ResolvedNotificationRoute(
|
||||
configured=False,
|
||||
enabled=True,
|
||||
app_profile=None,
|
||||
open_ids=tuple(defaults),
|
||||
)
|
||||
|
||||
monkeypatch.setattr(relogin, "resolve_notification_route", fake_resolver)
|
||||
|
||||
route = relogin.resolve_weekly_notification_route()
|
||||
|
||||
assert observed == {
|
||||
"workflow_id": "content.relogin.weekly",
|
||||
"defaults": (
|
||||
"ou_24cc944d6e43c69c59d6560ad4e2ae6e",
|
||||
"ou_7ad5fc8012e2f741afc5346e05ffd447",
|
||||
),
|
||||
}
|
||||
assert relogin.recipients_for_platform("pgy", route) == (
|
||||
"ou_24cc944d6e43c69c59d6560ad4e2ae6e",
|
||||
)
|
||||
assert relogin.recipients_for_platform("douyin", route) == (
|
||||
"ou_7ad5fc8012e2f741afc5346e05ffd447",
|
||||
)
|
||||
|
||||
|
||||
def test_disabled_route_still_runs_relogin_without_sending(monkeypatch):
|
||||
route = relogin.ResolvedNotificationRoute(
|
||||
configured=True,
|
||||
enabled=False,
|
||||
app_profile="hermes-analyzer",
|
||||
open_ids=(),
|
||||
)
|
||||
rounds = []
|
||||
|
||||
monkeypatch.setattr(
|
||||
relogin,
|
||||
"current_acceptance_policy",
|
||||
lambda: SimpleNamespace(enabled=False),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
relogin,
|
||||
"resolve_weekly_notification_route",
|
||||
lambda _override=None: route,
|
||||
)
|
||||
|
||||
def fake_round_factory(_args, resolved_route, notification_failures):
|
||||
assert resolved_route is route
|
||||
assert notification_failures == []
|
||||
|
||||
def run_round(names, attempt):
|
||||
rounds.append((tuple(names), attempt))
|
||||
return {name: True for name in names}
|
||||
|
||||
return run_round
|
||||
|
||||
monkeypatch.setattr(relogin, "_run_round_factory", fake_round_factory)
|
||||
monkeypatch.setattr(
|
||||
relogin,
|
||||
"_send_lark",
|
||||
lambda *_args, **_kwargs: (_ for _ in ()).throw(
|
||||
AssertionError("disabled route must not send Lark messages")
|
||||
),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
relogin.sys,
|
||||
"argv",
|
||||
["friday_relogin_parallel.py", "--platform", "pgy"],
|
||||
)
|
||||
|
||||
assert relogin.main() == 0
|
||||
assert rounds == [(("pgy",), 1)]
|
||||
|
||||
|
||||
def test_enabled_route_sends_qr_text_and_image_to_every_recipient(
|
||||
monkeypatch,
|
||||
tmp_path,
|
||||
):
|
||||
route = relogin.ResolvedNotificationRoute(
|
||||
configured=True,
|
||||
enabled=True,
|
||||
app_profile="hermes-analyzer",
|
||||
open_ids=("ou_first123", "ou_second456"),
|
||||
)
|
||||
sent = []
|
||||
image = tmp_path / "qrcode.png"
|
||||
|
||||
monkeypatch.setattr(
|
||||
relogin,
|
||||
"_send_lark",
|
||||
lambda recipient, text=None, image=None, app_profile=None: sent.append(
|
||||
(recipient, text, image, app_profile)
|
||||
),
|
||||
)
|
||||
|
||||
failures = relogin._send_qr_notification(
|
||||
route,
|
||||
"pgy",
|
||||
override=None,
|
||||
text="请扫码",
|
||||
image=image,
|
||||
)
|
||||
|
||||
assert failures == ()
|
||||
assert sent == [
|
||||
("ou_first123", "请扫码", None, "hermes-analyzer"),
|
||||
("ou_first123", None, image, "hermes-analyzer"),
|
||||
("ou_second456", "请扫码", None, "hermes-analyzer"),
|
||||
("ou_second456", None, image, "hermes-analyzer"),
|
||||
]
|
||||
assert relogin.group_platforms_by_notification_recipient(
|
||||
["pgy", "douyin"], route
|
||||
) == {
|
||||
"ou_first123": ["pgy", "douyin"],
|
||||
"ou_second456": ["pgy", "douyin"],
|
||||
}
|
||||
|
||||
|
||||
def test_qr_delivery_aggregates_errors_and_attempts_every_recipient(
|
||||
monkeypatch,
|
||||
tmp_path,
|
||||
):
|
||||
route = relogin.ResolvedNotificationRoute(
|
||||
configured=True,
|
||||
enabled=True,
|
||||
app_profile="hermes-analyzer",
|
||||
open_ids=("ou_first123", "ou_second456"),
|
||||
)
|
||||
attempts = []
|
||||
image = tmp_path / "qrcode.png"
|
||||
|
||||
def fake_send(recipient, text=None, image=None, app_profile=None):
|
||||
stage = "qr_text" if text is not None else "qr_image"
|
||||
attempts.append((recipient, stage, app_profile))
|
||||
if (recipient, stage) == ("ou_first123", "qr_text"):
|
||||
raise RuntimeError("first text rejected")
|
||||
if (recipient, stage) == ("ou_second456", "qr_image"):
|
||||
raise RuntimeError("second image rejected")
|
||||
|
||||
monkeypatch.setattr(relogin, "_send_lark", fake_send)
|
||||
|
||||
failures = relogin._send_qr_notification(
|
||||
route,
|
||||
"pgy",
|
||||
override=None,
|
||||
text="请扫码",
|
||||
image=image,
|
||||
)
|
||||
|
||||
assert attempts == [
|
||||
("ou_first123", "qr_text", "hermes-analyzer"),
|
||||
("ou_first123", "qr_image", "hermes-analyzer"),
|
||||
("ou_second456", "qr_text", "hermes-analyzer"),
|
||||
("ou_second456", "qr_image", "hermes-analyzer"),
|
||||
]
|
||||
assert failures == (
|
||||
relogin.NotificationDeliveryFailure(
|
||||
stage="qr_text",
|
||||
platform="pgy",
|
||||
recipient="ou_first123",
|
||||
detail="first text rejected",
|
||||
),
|
||||
relogin.NotificationDeliveryFailure(
|
||||
stage="qr_image",
|
||||
platform="pgy",
|
||||
recipient="ou_second456",
|
||||
detail="second image rejected",
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def test_failure_summary_aggregates_every_recipient_error(monkeypatch):
|
||||
route = relogin.ResolvedNotificationRoute(
|
||||
configured=True,
|
||||
enabled=True,
|
||||
app_profile="hermes-analyzer",
|
||||
open_ids=("ou_first123", "ou_second456"),
|
||||
)
|
||||
attempts = []
|
||||
|
||||
def fail_send(recipient, **_kwargs):
|
||||
attempts.append(recipient)
|
||||
raise RuntimeError(f"rejected {recipient}")
|
||||
|
||||
monkeypatch.setattr(relogin, "_send_lark", fail_send)
|
||||
|
||||
failures = relogin._send_failure_summaries(
|
||||
route,
|
||||
["pgy", "douyin"],
|
||||
override=None,
|
||||
max_attempts=3,
|
||||
)
|
||||
|
||||
assert attempts == ["ou_first123", "ou_second456"]
|
||||
assert [failure.stage for failure in failures] == [
|
||||
"failure_summary",
|
||||
"failure_summary",
|
||||
]
|
||||
assert [failure.recipient for failure in failures] == attempts
|
||||
assert [failure.platform for failure in failures] == [
|
||||
"pgy,douyin",
|
||||
"pgy,douyin",
|
||||
]
|
||||
|
||||
|
||||
def test_main_returns_failure_and_reports_all_notification_errors(
|
||||
monkeypatch,
|
||||
capsys,
|
||||
):
|
||||
route = relogin.ResolvedNotificationRoute(
|
||||
configured=True,
|
||||
enabled=True,
|
||||
app_profile="hermes-analyzer",
|
||||
open_ids=("ou_first123", "ou_second456"),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
relogin,
|
||||
"current_acceptance_policy",
|
||||
lambda: SimpleNamespace(enabled=False),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
relogin,
|
||||
"resolve_weekly_notification_route",
|
||||
lambda _override=None: route,
|
||||
)
|
||||
|
||||
def fake_round_factory(_args, _route, notification_failures):
|
||||
notification_failures.extend(
|
||||
(
|
||||
relogin.NotificationDeliveryFailure(
|
||||
"qr_text", "pgy", "ou_first123", "text rejected"
|
||||
),
|
||||
relogin.NotificationDeliveryFailure(
|
||||
"qr_image", "pgy", "ou_second456", "image rejected"
|
||||
),
|
||||
)
|
||||
)
|
||||
return lambda names, _attempt: {name: True for name in names}
|
||||
|
||||
monkeypatch.setattr(relogin, "_run_round_factory", fake_round_factory)
|
||||
monkeypatch.setattr(
|
||||
relogin.sys,
|
||||
"argv",
|
||||
["friday_relogin_parallel.py", "--platform", "pgy"],
|
||||
)
|
||||
|
||||
assert relogin.main() == 2
|
||||
output = capsys.readouterr().out
|
||||
assert "Final relogin status: {'pgy': True}" in output
|
||||
assert "Final notification status: failed (2 deliveries)" in output
|
||||
assert "recipient=ou_first123 error=text rejected" in output
|
||||
assert "recipient=ou_second456 error=image rejected" in output
|
||||
|
||||
|
||||
def test_round_completes_relogin_after_notification_failure(monkeypatch, tmp_path):
|
||||
route = relogin.ResolvedNotificationRoute(
|
||||
configured=True,
|
||||
enabled=True,
|
||||
app_profile="hermes-analyzer",
|
||||
open_ids=("ou_first123", "ou_second456"),
|
||||
)
|
||||
args = SimpleNamespace(
|
||||
round_timeout=10,
|
||||
screenshot_delay=0,
|
||||
no_send=False,
|
||||
recipient=None,
|
||||
max_attempts=1,
|
||||
)
|
||||
failures = []
|
||||
attempts = []
|
||||
waits = []
|
||||
|
||||
class Process:
|
||||
pid = 1234
|
||||
|
||||
def wait(self, timeout):
|
||||
waits.append(timeout)
|
||||
return 0
|
||||
|
||||
monkeypatch.setattr(relogin.subprocess, "Popen", lambda *_args, **_kwargs: Process())
|
||||
monkeypatch.setattr(relogin, "environment_for_child_script", lambda *_args: {})
|
||||
monkeypatch.setattr(
|
||||
relogin,
|
||||
"binding_from_environment",
|
||||
lambda _environment: SimpleNamespace(cookie_file=tmp_path / "cookies.json"),
|
||||
)
|
||||
monkeypatch.setattr(relogin.time, "sleep", lambda _seconds: None)
|
||||
monkeypatch.setattr(relogin, "_login_windows", lambda: {"pgy": 101})
|
||||
monkeypatch.setattr(relogin, "_tile_windows", lambda _handles: None)
|
||||
monkeypatch.setattr(relogin, "_capture_window", lambda _hwnd, _output: None)
|
||||
monkeypatch.setattr(relogin, "_valid_cookie", lambda *_args: True)
|
||||
|
||||
def fake_send(recipient, text=None, image=None, **_kwargs):
|
||||
stage = "qr_text" if text is not None else "qr_image"
|
||||
attempts.append((recipient, stage))
|
||||
if (recipient, stage) == ("ou_first123", "qr_text"):
|
||||
raise RuntimeError("temporary send failure")
|
||||
|
||||
monkeypatch.setattr(relogin, "_send_lark", fake_send)
|
||||
|
||||
run_round = relogin._run_round_factory(args, route, failures)
|
||||
|
||||
assert run_round(["pgy"], 1) == {"pgy": True}
|
||||
assert waits == [100]
|
||||
assert attempts == [
|
||||
("ou_first123", "qr_text"),
|
||||
("ou_first123", "qr_image"),
|
||||
("ou_second456", "qr_text"),
|
||||
("ou_second456", "qr_image"),
|
||||
]
|
||||
assert failures == [
|
||||
relogin.NotificationDeliveryFailure(
|
||||
stage="qr_text",
|
||||
platform="pgy",
|
||||
recipient="ou_first123",
|
||||
detail="temporary send failure",
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("returncode", "cookie_valid", "expected_status", "expected_failure_count"),
|
||||
(
|
||||
(0, True, True, 0),
|
||||
(1, False, False, 4),
|
||||
),
|
||||
)
|
||||
def test_missing_qr_window_is_classified_after_relogin_result(
|
||||
monkeypatch,
|
||||
tmp_path,
|
||||
returncode,
|
||||
cookie_valid,
|
||||
expected_status,
|
||||
expected_failure_count,
|
||||
):
|
||||
route = relogin.ResolvedNotificationRoute(
|
||||
configured=True,
|
||||
enabled=True,
|
||||
app_profile="hermes-analyzer",
|
||||
open_ids=("ou_first123", "ou_second456"),
|
||||
)
|
||||
args = SimpleNamespace(
|
||||
round_timeout=10,
|
||||
screenshot_delay=0,
|
||||
no_send=False,
|
||||
recipient=None,
|
||||
max_attempts=1,
|
||||
)
|
||||
failures = []
|
||||
|
||||
class Process:
|
||||
pid = 1234
|
||||
|
||||
def wait(self, timeout):
|
||||
assert timeout == 100
|
||||
return returncode
|
||||
|
||||
monkeypatch.setattr(relogin.subprocess, "Popen", lambda *_args, **_kwargs: Process())
|
||||
monkeypatch.setattr(relogin, "environment_for_child_script", lambda *_args: {})
|
||||
monkeypatch.setattr(
|
||||
relogin,
|
||||
"binding_from_environment",
|
||||
lambda _environment: SimpleNamespace(cookie_file=tmp_path / "cookies.json"),
|
||||
)
|
||||
monkeypatch.setattr(relogin.time, "sleep", lambda _seconds: None)
|
||||
monkeypatch.setattr(relogin, "_login_windows", lambda: {})
|
||||
monkeypatch.setattr(relogin, "_window_for_process", lambda _pid: None)
|
||||
monkeypatch.setattr(relogin, "_tile_windows", lambda _handles: None)
|
||||
monkeypatch.setattr(relogin, "_valid_cookie", lambda *_args: cookie_valid)
|
||||
monkeypatch.setattr(
|
||||
relogin,
|
||||
"_send_lark",
|
||||
lambda *_args, **_kwargs: pytest.fail(
|
||||
"no QR window must not start a send before child classification"
|
||||
),
|
||||
)
|
||||
|
||||
run_round = relogin._run_round_factory(args, route, failures)
|
||||
|
||||
assert run_round(["pgy"], 1) == {"pgy": expected_status}
|
||||
assert len(failures) == expected_failure_count
|
||||
assert {failure.stage for failure in failures} <= {"qr_text", "qr_image"}
|
||||
assert all(
|
||||
failure.detail == "no visible QR window and relogin did not succeed"
|
||||
for failure in failures
|
||||
)
|
||||
|
||||
|
||||
def test_lark_send_uses_shared_analyzer_user_sender_for_text_and_image(
|
||||
monkeypatch,
|
||||
tmp_path,
|
||||
):
|
||||
calls = []
|
||||
image = tmp_path / "qrcode.png"
|
||||
image.write_bytes(b"png")
|
||||
|
||||
monkeypatch.setattr(relogin, "ANALYZER_DIR", tmp_path)
|
||||
monkeypatch.setenv("HERMES_ANALYZER_TOKEN", "must-not-leak")
|
||||
|
||||
def fake_send(**kwargs):
|
||||
calls.append(kwargs)
|
||||
return {"message_id": f"om_{len(calls)}"}
|
||||
|
||||
monkeypatch.setattr(relogin, "send_lark_bot_message", fake_send)
|
||||
|
||||
receipts = relogin._send_lark(
|
||||
"ou_recipient123",
|
||||
text="登录提醒",
|
||||
image=image,
|
||||
)
|
||||
|
||||
assert [receipt["message_id"] for receipt in receipts] == ["om_1", "om_2"]
|
||||
assert calls[0]["user_id"] == "ou_recipient123"
|
||||
assert calls[0]["text"] == "登录提醒"
|
||||
assert "image" not in calls[0]
|
||||
assert calls[1]["user_id"] == "ou_recipient123"
|
||||
assert calls[1]["image"] == image
|
||||
assert "text" not in calls[1]
|
||||
assert all(call["profile"] == "hermes-analyzer" for call in calls)
|
||||
assert all("HERMES_ANALYZER_TOKEN" not in call["env"] for call in calls)
|
||||
|
||||
|
||||
def test_login_windows_are_classified_by_title():
|
||||
assert relogin._platform_from_window_title("账号登录 - Google Chrome") == "bilibili"
|
||||
assert relogin._platform_from_window_title("小红书蒲公英 - Google Chrome") == "pgy"
|
||||
assert relogin._platform_from_window_title("抖音授权 - Google Chrome") == "xingtu"
|
||||
assert relogin._platform_from_window_title("抖音 - 记录美好生活") == "douyin"
|
||||
assert relogin._platform_from_window_title("小红书 - 你的生活兴趣社区") == "xiaohongshu"
|
||||
assert relogin._platform_from_window_title("Codex") is None
|
||||
|
||||
|
||||
def test_retry_only_failed_platforms_for_at_most_three_rounds():
|
||||
calls = []
|
||||
|
||||
def fake_round(names, attempt):
|
||||
calls.append((tuple(names), attempt))
|
||||
if attempt == 1:
|
||||
return {name: name != "xingtu" for name in names}
|
||||
if attempt == 2:
|
||||
return {name: False for name in names}
|
||||
return {name: True for name in names}
|
||||
|
||||
result = relogin.run_with_retries(
|
||||
["bilibili", "pgy", "xingtu", "xiaohongshu"],
|
||||
max_attempts=3,
|
||||
run_round=fake_round,
|
||||
)
|
||||
|
||||
assert calls == [
|
||||
(("bilibili", "pgy", "xingtu", "xiaohongshu"), 1),
|
||||
(("xingtu",), 2),
|
||||
(("xingtu",), 3),
|
||||
]
|
||||
assert all(result.values())
|
||||
|
||||
|
||||
def test_stops_after_third_failed_round():
|
||||
calls = []
|
||||
|
||||
def always_fail(names, attempt):
|
||||
calls.append(attempt)
|
||||
return {name: False for name in names}
|
||||
|
||||
result = relogin.run_with_retries(
|
||||
["pgy"], max_attempts=3, run_round=always_fail
|
||||
)
|
||||
|
||||
assert calls == [1, 2, 3]
|
||||
assert result == {"pgy": False}
|
||||
|
||||
|
||||
def test_timeout_terminates_the_complete_process_tree(monkeypatch):
|
||||
calls = []
|
||||
|
||||
class Process:
|
||||
pid = 1234
|
||||
|
||||
def terminate(self):
|
||||
calls.append("fallback-terminate")
|
||||
|
||||
def wait(self, timeout):
|
||||
calls.append(("wait", timeout))
|
||||
|
||||
monkeypatch.setattr(
|
||||
relogin.subprocess,
|
||||
"run",
|
||||
lambda command, **kwargs: calls.append((command, kwargs)),
|
||||
)
|
||||
|
||||
relogin._terminate_process_tree(Process())
|
||||
|
||||
assert calls[0][0] == ["taskkill", "/PID", "1234", "/T", "/F"]
|
||||
assert calls[1] == ("wait", 10)
|
||||
|
||||
|
||||
def test_transaction_rolls_back_files_and_profile(tmp_path):
|
||||
cookie = tmp_path / "cookies.json"
|
||||
storage = tmp_path / "storage.json"
|
||||
profile = tmp_path / "profile"
|
||||
cookie.write_text("old-cookie", encoding="utf-8")
|
||||
storage.write_text("old-storage", encoding="utf-8")
|
||||
profile.mkdir()
|
||||
(profile / "old.txt").write_text("old-profile", encoding="utf-8")
|
||||
|
||||
state = relogin_transaction.begin((cookie, storage), profile)
|
||||
cookie.write_text("partial", encoding="utf-8")
|
||||
profile.mkdir()
|
||||
(profile / "partial.txt").write_text("partial", encoding="utf-8")
|
||||
relogin_transaction.rollback(state)
|
||||
|
||||
assert cookie.read_text(encoding="utf-8") == "old-cookie"
|
||||
assert storage.read_text(encoding="utf-8") == "old-storage"
|
||||
assert (profile / "old.txt").read_text(encoding="utf-8") == "old-profile"
|
||||
assert not (profile / "partial.txt").exists()
|
||||
@@ -416,11 +416,9 @@ def test_acceptance_invalid_session_skips_before_qr_login(
|
||||
)
|
||||
monkeypatch.setenv("GYXX_ACCEPTANCE_EVIDENCE_FILE", str(evidence_file))
|
||||
monkeypatch.setattr(module, "is_logged_in", lambda _page: False)
|
||||
monkeypatch.setattr(
|
||||
module,
|
||||
"wait_for_scan_login",
|
||||
lambda *_args, **_kwargs: pytest.fail("QR login must remain disabled"),
|
||||
)
|
||||
# 扫码登录函数已整体移除;采集器不得重新引入交互登录。
|
||||
assert not hasattr(module, "wait_for_scan_login")
|
||||
assert not hasattr(module, "open_scan_login")
|
||||
|
||||
class Page:
|
||||
def __init__(self):
|
||||
|
||||
@@ -1,494 +0,0 @@
|
||||
|
||||
import pytest
|
||||
|
||||
from gyxx_flow.modules.content_marketing import (
|
||||
pgy_xhs_scraper_v2,
|
||||
xingtu_scraper_v2,
|
||||
)
|
||||
from gyxx_flow.modules.content_marketing.data.tools import (
|
||||
relogin_douyin,
|
||||
relogin_pgy,
|
||||
relogin_xingtu,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("module", [relogin_pgy, relogin_xingtu])
|
||||
def test_failed_relogin_restores_cookie_and_profile(tmp_path, monkeypatch, module):
|
||||
cookie = tmp_path / "cookies.json"
|
||||
storage = tmp_path / "storage_state.json"
|
||||
profile = tmp_path / "profile"
|
||||
cookie.write_text("old-cookie", encoding="utf-8")
|
||||
storage.write_text("old-storage", encoding="utf-8")
|
||||
profile.mkdir()
|
||||
(profile / "state.txt").write_text("old-profile", encoding="utf-8")
|
||||
|
||||
monkeypatch.setattr(module, "COOKIE_FILE", cookie)
|
||||
monkeypatch.setattr(module, "STATE_FILE", storage)
|
||||
monkeypatch.setattr(module, "PROFILE_DIR", profile)
|
||||
|
||||
cookie_backup, profile_backup, state_backup = module.reset_state()
|
||||
assert not cookie.exists()
|
||||
assert not storage.exists()
|
||||
assert not profile.exists()
|
||||
|
||||
cookie.write_text("partial-new-cookie", encoding="utf-8")
|
||||
storage.write_text("partial-new-storage", encoding="utf-8")
|
||||
profile.mkdir()
|
||||
(profile / "partial.txt").write_text("partial", encoding="utf-8")
|
||||
|
||||
module.restore_previous_state(cookie_backup, profile_backup, state_backup)
|
||||
|
||||
assert cookie.read_text(encoding="utf-8") == "old-cookie"
|
||||
assert storage.read_text(encoding="utf-8") == "old-storage"
|
||||
assert (profile / "state.txt").read_text(encoding="utf-8") == "old-profile"
|
||||
assert not (profile / "partial.txt").exists()
|
||||
|
||||
|
||||
@pytest.mark.parametrize("module", [relogin_pgy, relogin_xingtu])
|
||||
def test_successful_relogin_keeps_new_cookie(tmp_path, monkeypatch, module):
|
||||
cookie = tmp_path / "cookies.json"
|
||||
storage = tmp_path / "storage_state.json"
|
||||
profile = tmp_path / "profile"
|
||||
cookie.write_text("old-cookie", encoding="utf-8")
|
||||
storage.write_text("old-storage", encoding="utf-8")
|
||||
profile.mkdir()
|
||||
|
||||
monkeypatch.setattr(module, "COOKIE_FILE", cookie)
|
||||
monkeypatch.setattr(module, "STATE_FILE", storage)
|
||||
monkeypatch.setattr(module, "PROFILE_DIR", profile)
|
||||
|
||||
cookie_backup, profile_backup, state_backup = module.reset_state()
|
||||
cookie.write_text("new-cookie", encoding="utf-8")
|
||||
storage.write_text("new-storage", encoding="utf-8")
|
||||
profile.mkdir()
|
||||
|
||||
module.finish_successful_relogin(profile_backup)
|
||||
|
||||
assert cookie.read_text(encoding="utf-8") == "new-cookie"
|
||||
assert storage.read_text(encoding="utf-8") == "new-storage"
|
||||
assert cookie_backup.exists()
|
||||
assert state_backup.exists()
|
||||
assert not profile_backup.exists()
|
||||
|
||||
|
||||
def test_xingtu_scan_login_opens_customer_sso_and_clicks_douyin():
|
||||
class FakeLocator:
|
||||
def __init__(self):
|
||||
self.first = self
|
||||
self.clicked = False
|
||||
self.evaluated = False
|
||||
|
||||
def count(self):
|
||||
return 1
|
||||
|
||||
def is_visible(self):
|
||||
return True
|
||||
|
||||
def click(self, timeout):
|
||||
self.clicked = True
|
||||
|
||||
def evaluate(self, script):
|
||||
assert script == "e => e.click()"
|
||||
self.evaluated = True
|
||||
|
||||
class FakePage:
|
||||
url = "https://www.xingtu.cn/?redirect_uri=/ad/creator/market"
|
||||
|
||||
def __init__(self):
|
||||
self.goto_url = None
|
||||
self.selectors = []
|
||||
self.douyin = FakeLocator()
|
||||
|
||||
def goto(self, url, **kwargs):
|
||||
self.goto_url = url
|
||||
self.url = url
|
||||
|
||||
def wait_for_timeout(self, _milliseconds):
|
||||
pass
|
||||
|
||||
def locator(self, selector):
|
||||
self.selectors.append(selector)
|
||||
return self.douyin
|
||||
|
||||
def get_by_text(self, _pattern):
|
||||
return FakeLocator()
|
||||
|
||||
page = FakePage()
|
||||
xingtu_scraper_v2.open_scan_login(page)
|
||||
|
||||
assert page.goto_url == xingtu_scraper_v2.CUSTOMER_LOGIN_URL
|
||||
assert 'img[src*="aweme.png"]' in page.selectors
|
||||
assert page.douyin.clicked
|
||||
|
||||
|
||||
def test_xingtu_index_page_is_recognized_as_logged_in():
|
||||
class Locator:
|
||||
@property
|
||||
def first(self):
|
||||
return self
|
||||
|
||||
def wait_for(self, timeout):
|
||||
assert timeout == 1200
|
||||
|
||||
class Page:
|
||||
url = "https://www.xingtu.cn/ad/creator/index"
|
||||
|
||||
def get_by_text(self, _text, exact=False):
|
||||
return Locator()
|
||||
|
||||
assert xingtu_scraper_v2.is_logged_in(Page()) is True
|
||||
|
||||
|
||||
def test_xingtu_index_page_with_xingtu_session_is_logged_in_without_market_ui():
|
||||
class Locator:
|
||||
@property
|
||||
def first(self):
|
||||
return self
|
||||
|
||||
def count(self):
|
||||
return 0
|
||||
|
||||
def wait_for(self, timeout):
|
||||
raise xingtu_scraper_v2.BrowserTimeoutError("not on market page")
|
||||
|
||||
def is_visible(self):
|
||||
return False
|
||||
|
||||
class Context:
|
||||
def cookies(self):
|
||||
return [
|
||||
{
|
||||
"name": "sessionid",
|
||||
"value": "fresh",
|
||||
"domain": ".xingtu.cn",
|
||||
}
|
||||
]
|
||||
|
||||
class Page:
|
||||
url = "https://www.xingtu.cn/ad/creator/index"
|
||||
context = Context()
|
||||
|
||||
def get_by_text(self, *_args, **_kwargs):
|
||||
return Locator()
|
||||
|
||||
def locator(self, *_args, **_kwargs):
|
||||
return Locator()
|
||||
|
||||
assert xingtu_scraper_v2.is_logged_in(Page()) is True
|
||||
|
||||
|
||||
def test_xingtu_index_page_without_xingtu_session_is_not_logged_in():
|
||||
class Locator:
|
||||
@property
|
||||
def first(self):
|
||||
return self
|
||||
|
||||
def count(self):
|
||||
return 0
|
||||
|
||||
def wait_for(self, timeout):
|
||||
raise xingtu_scraper_v2.BrowserTimeoutError("missing")
|
||||
|
||||
def is_visible(self):
|
||||
return False
|
||||
|
||||
class Context:
|
||||
def cookies(self):
|
||||
return [
|
||||
{
|
||||
"name": "sessionid",
|
||||
"value": "douyin-only",
|
||||
"domain": ".douyin.com",
|
||||
}
|
||||
]
|
||||
|
||||
class Page:
|
||||
url = "https://www.xingtu.cn/ad/creator/index"
|
||||
context = Context()
|
||||
|
||||
def get_by_text(self, *_args, **_kwargs):
|
||||
return Locator()
|
||||
|
||||
def locator(self, *_args, **_kwargs):
|
||||
return Locator()
|
||||
|
||||
assert xingtu_scraper_v2.is_logged_in(Page()) is False
|
||||
|
||||
|
||||
def test_xingtu_public_homepage_is_not_logged_in_even_with_stale_cookie():
|
||||
class Context:
|
||||
def cookies(self):
|
||||
return [{"name": "sessionid", "value": "stale"}]
|
||||
|
||||
class Page:
|
||||
url = "https://www.xingtu.cn/?redirect_uri=/ad/creator/market"
|
||||
context = Context()
|
||||
|
||||
def get_by_text(self, *_args, **_kwargs):
|
||||
raise AssertionError("public homepage must be rejected before marker checks")
|
||||
|
||||
assert xingtu_scraper_v2.is_logged_in(Page()) is False
|
||||
|
||||
|
||||
def test_xingtu_market_page_without_business_ui_rejects_stale_cookie():
|
||||
class Locator:
|
||||
@property
|
||||
def first(self):
|
||||
return self
|
||||
|
||||
def wait_for(self, **_kwargs):
|
||||
raise xingtu_scraper_v2.BrowserTimeoutError("missing")
|
||||
|
||||
def count(self):
|
||||
return 0
|
||||
|
||||
class Context:
|
||||
def cookies(self):
|
||||
return [{"name": "sessionid", "value": "stale"}]
|
||||
|
||||
class Page:
|
||||
url = "https://www.xingtu.cn/ad/creator/market"
|
||||
context = Context()
|
||||
|
||||
def get_by_text(self, *_args, **_kwargs):
|
||||
return Locator()
|
||||
|
||||
def locator(self, *_args, **_kwargs):
|
||||
return Locator()
|
||||
|
||||
assert xingtu_scraper_v2.is_logged_in(Page()) is False
|
||||
|
||||
|
||||
def test_xingtu_cookie_validation_requires_real_session_cookie(tmp_path):
|
||||
cookie = tmp_path / "xingtu.json"
|
||||
cookie.write_text('[{"name":"other","value":"1"}]', encoding="utf-8")
|
||||
assert xingtu_scraper_v2.has_valid_login_cookie_file(cookie) is False
|
||||
|
||||
cookie.write_text('[{"name":"sessionid","value":"fresh"}]', encoding="utf-8")
|
||||
assert xingtu_scraper_v2.has_valid_login_cookie_file(cookie) is True
|
||||
|
||||
|
||||
def test_xingtu_login_only_does_not_report_success_when_page_action_was_swallowed(
|
||||
tmp_path, monkeypatch
|
||||
):
|
||||
cookie = tmp_path / "xingtu.json"
|
||||
cookie.write_text('[{"name":"sessionid","value":"old"}]', encoding="utf-8")
|
||||
|
||||
class Session:
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *_args):
|
||||
return False
|
||||
|
||||
def fetch(self, *_args, **_kwargs):
|
||||
# Scrapling can log a page_action error and still return a response.
|
||||
return object()
|
||||
|
||||
monkeypatch.setattr(xingtu_scraper_v2, "COOKIE_FILE", cookie)
|
||||
monkeypatch.setattr(xingtu_scraper_v2, "DynamicSession", lambda **_kwargs: Session())
|
||||
monkeypatch.setattr(xingtu_scraper_v2, "_force_cleanup_session", lambda _session: None)
|
||||
|
||||
assert xingtu_scraper_v2.login_only(1) == 1
|
||||
|
||||
|
||||
def test_xingtu_login_only_verifies_persisted_profile_after_oauth_context_closes(
|
||||
tmp_path, monkeypatch
|
||||
):
|
||||
cookie = tmp_path / "xingtu.json"
|
||||
|
||||
class Session:
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *_args):
|
||||
return False
|
||||
|
||||
def fetch(self, *_args, **_kwargs):
|
||||
raise RuntimeError("Target page, context or browser has been closed")
|
||||
|
||||
monkeypatch.setattr(xingtu_scraper_v2, "COOKIE_FILE", cookie)
|
||||
monkeypatch.setattr(xingtu_scraper_v2, "DynamicSession", lambda **_kwargs: Session())
|
||||
monkeypatch.setattr(xingtu_scraper_v2, "_force_cleanup_session", lambda _session: None)
|
||||
|
||||
def verify():
|
||||
cookie.write_text('[{"name":"sessionid","value":"fresh"}]', encoding="utf-8")
|
||||
return True
|
||||
|
||||
monkeypatch.setattr(xingtu_scraper_v2, "verify_persisted_profile_login", verify)
|
||||
|
||||
assert xingtu_scraper_v2.login_only(1) == 0
|
||||
|
||||
|
||||
def test_xingtu_scan_login_follows_replacement_page_after_oauth_tab_closes(monkeypatch):
|
||||
class Context:
|
||||
def __init__(self):
|
||||
self.pages = []
|
||||
|
||||
def new_page(self):
|
||||
raise AssertionError("a surviving replacement page should be reused")
|
||||
|
||||
class Page:
|
||||
def __init__(self, context, url, *, closes_on_wait=False):
|
||||
self.context = context
|
||||
self.url = url
|
||||
self.closed = False
|
||||
self.closes_on_wait = closes_on_wait
|
||||
|
||||
def is_closed(self):
|
||||
return self.closed
|
||||
|
||||
def wait_for_timeout(self, _milliseconds):
|
||||
if self.closes_on_wait:
|
||||
self.closed = True
|
||||
|
||||
def goto(self, url, **_kwargs):
|
||||
self.url = url
|
||||
|
||||
context = Context()
|
||||
qr_page = Page(
|
||||
context,
|
||||
"https://open.douyin.com/platform/oauth/pc/auth",
|
||||
closes_on_wait=True,
|
||||
)
|
||||
market_page = Page(context, xingtu_scraper_v2.MARKET_URL)
|
||||
context.pages = [qr_page, market_page]
|
||||
saved = []
|
||||
|
||||
monkeypatch.setattr(xingtu_scraper_v2, "open_scan_login", lambda _page: None)
|
||||
monkeypatch.setattr(
|
||||
xingtu_scraper_v2, "is_scan_qr_page", lambda page: page is qr_page
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
xingtu_scraper_v2, "is_logged_in", lambda page: page is market_page
|
||||
)
|
||||
monkeypatch.setattr(xingtu_scraper_v2, "save_state", lambda page: saved.append(page))
|
||||
|
||||
result = xingtu_scraper_v2.wait_for_scan_login(qr_page, 2)
|
||||
|
||||
assert result is market_page
|
||||
assert saved == [market_page]
|
||||
|
||||
|
||||
def test_xingtu_scan_login_detects_business_tab_while_qr_tab_stays_open(monkeypatch):
|
||||
class Context:
|
||||
def __init__(self):
|
||||
self.pages = []
|
||||
|
||||
class Page:
|
||||
def __init__(self, context, url):
|
||||
self.context = context
|
||||
self.url = url
|
||||
|
||||
def is_closed(self):
|
||||
return False
|
||||
|
||||
def wait_for_timeout(self, _milliseconds):
|
||||
raise AssertionError("authenticated tab should be detected immediately")
|
||||
|
||||
context = Context()
|
||||
qr_page = Page(context, "https://open.douyin.com/platform/oauth/pc/auth")
|
||||
index_page = Page(context, "https://www.xingtu.cn/ad/creator/index")
|
||||
context.pages = [qr_page, index_page]
|
||||
saved = []
|
||||
|
||||
monkeypatch.setattr(xingtu_scraper_v2, "open_scan_login", lambda _page: None)
|
||||
monkeypatch.setattr(
|
||||
xingtu_scraper_v2, "is_logged_in", lambda page: page is index_page
|
||||
)
|
||||
monkeypatch.setattr(xingtu_scraper_v2, "save_state", lambda page: saved.append(page))
|
||||
|
||||
result = xingtu_scraper_v2.wait_for_scan_login(qr_page, 2)
|
||||
|
||||
assert result is index_page
|
||||
assert saved == [index_page]
|
||||
|
||||
|
||||
def test_xingtu_interrupted_transaction_is_recovered(tmp_path, monkeypatch):
|
||||
cookie = tmp_path / "xingtu_cookies.json"
|
||||
storage = tmp_path / "xingtu_storage_state.json"
|
||||
profile = tmp_path / "profile"
|
||||
cookie.write_text("old-cookie", encoding="utf-8")
|
||||
storage.write_text("old-storage", encoding="utf-8")
|
||||
profile.mkdir()
|
||||
(profile / "old.txt").write_text("old-profile", encoding="utf-8")
|
||||
|
||||
monkeypatch.setattr(relogin_xingtu, "COOKIE_FILE", cookie)
|
||||
monkeypatch.setattr(relogin_xingtu, "STATE_FILE", storage)
|
||||
monkeypatch.setattr(relogin_xingtu, "PROFILE_DIR", profile)
|
||||
|
||||
cookie_backup, profile_backup, state_backup = relogin_xingtu.reset_state()
|
||||
profile.mkdir()
|
||||
(profile / "partial.txt").write_text("partial", encoding="utf-8")
|
||||
|
||||
assert relogin_xingtu.recover_interrupted_state() is True
|
||||
assert cookie.read_text(encoding="utf-8") == "old-cookie"
|
||||
assert storage.read_text(encoding="utf-8") == "old-storage"
|
||||
assert (profile / "old.txt").read_text(encoding="utf-8") == "old-profile"
|
||||
assert not (profile / "partial.txt").exists()
|
||||
assert cookie_backup is not None
|
||||
assert state_backup is not None
|
||||
assert profile_backup is not None
|
||||
|
||||
|
||||
def test_pgy_scan_login_clicks_qr_switch_image():
|
||||
class FakeLocator:
|
||||
def __init__(self):
|
||||
self.first = self
|
||||
self.clicked = False
|
||||
self.evaluated = False
|
||||
|
||||
def count(self):
|
||||
return 1
|
||||
|
||||
def is_visible(self):
|
||||
return True
|
||||
|
||||
def click(self, timeout):
|
||||
self.clicked = True
|
||||
|
||||
def evaluate(self, script):
|
||||
assert script == "e => e.click()"
|
||||
self.evaluated = True
|
||||
|
||||
class FakePage:
|
||||
def __init__(self):
|
||||
self.selectors = []
|
||||
self.qr_switch = FakeLocator()
|
||||
|
||||
def wait_for_timeout(self, _milliseconds):
|
||||
pass
|
||||
|
||||
def get_by_text(self, _pattern):
|
||||
return FakeLocator()
|
||||
|
||||
def locator(self, selector):
|
||||
self.selectors.append(selector)
|
||||
return self.qr_switch
|
||||
|
||||
page = FakePage()
|
||||
pgy_xhs_scraper_v2.open_scan_login(page)
|
||||
|
||||
assert 'img[src*="qr_code"]' in page.selectors
|
||||
assert page.qr_switch.evaluated
|
||||
|
||||
|
||||
def test_failed_douyin_relogin_restores_cookie_storage_and_profile(tmp_path, monkeypatch):
|
||||
cookie = tmp_path / "douyin_cookies.json"
|
||||
storage = tmp_path / "douyin_storage_state.json"
|
||||
profile = tmp_path / "profile"
|
||||
cookie.write_text('[{"name":"sessionid","value":"old"}]', encoding="utf-8")
|
||||
storage.write_text("old-storage", encoding="utf-8")
|
||||
profile.mkdir()
|
||||
(profile / "old.txt").write_text("old-profile", encoding="utf-8")
|
||||
|
||||
monkeypatch.setattr(relogin_douyin, "COOKIE_FILE", cookie)
|
||||
monkeypatch.setattr(relogin_douyin, "STATE_FILE", storage)
|
||||
monkeypatch.setattr(relogin_douyin, "PROFILE_DIR", profile)
|
||||
monkeypatch.setattr(relogin_douyin.subprocess, "call", lambda *_args, **_kwargs: 1)
|
||||
monkeypatch.setattr(relogin_douyin.sys, "argv", ["relogin_douyin.py"])
|
||||
|
||||
assert relogin_douyin.main() == 1
|
||||
assert cookie.read_text(encoding="utf-8") == '[{"name":"sessionid","value":"old"}]'
|
||||
assert storage.read_text(encoding="utf-8") == "old-storage"
|
||||
assert (profile / "old.txt").read_text(encoding="utf-8") == "old-profile"
|
||||
@@ -1,410 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
from types import ModuleType
|
||||
|
||||
import pytest
|
||||
|
||||
from gyxx_flow.adapters.integration import RuntimeIntegrationCatalog
|
||||
from gyxx_flow.modules.content_marketing.data.tools import (
|
||||
relogin_bilibili,
|
||||
relogin_douyin,
|
||||
relogin_pgy,
|
||||
relogin_runtime_paths,
|
||||
relogin_xiaohongshu,
|
||||
relogin_xingtu,
|
||||
)
|
||||
from gyxx_flow.modules.content_marketing.data.tools.relogin_runtime_paths import (
|
||||
CONSUMER_BINDING_IDS,
|
||||
child_browser_environment,
|
||||
resolve_relogin_runtime_paths,
|
||||
sync_relogin_state_to_consumers,
|
||||
)
|
||||
from gyxx_flow.modules.content_marketing.runtime_paths import PATHS
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[3]
|
||||
|
||||
|
||||
def test_relogin_paths_prefer_injected_runtime_binding(tmp_path: Path) -> None:
|
||||
cookie = tmp_path / "binding" / "cookies.json"
|
||||
storage = tmp_path / "binding" / "storage_state.json"
|
||||
profile = tmp_path / "binding" / "profile"
|
||||
|
||||
resolved = resolve_relogin_runtime_paths(
|
||||
"pgy",
|
||||
{
|
||||
"GYXX_BROWSER_COOKIE_FILE": str(cookie),
|
||||
"GYXX_BROWSER_STORAGE_STATE_FILE": str(storage),
|
||||
"GYXX_BROWSER_PROFILE_DIR": str(profile),
|
||||
},
|
||||
)
|
||||
|
||||
assert resolved.cookie_file == cookie.resolve()
|
||||
assert resolved.storage_state_file == storage.resolve()
|
||||
assert resolved.profile_dir == profile.resolve()
|
||||
|
||||
|
||||
def test_relogin_paths_keep_legacy_standalone_layout() -> None:
|
||||
resolved = resolve_relogin_runtime_paths("xiaohongshu", {})
|
||||
|
||||
assert resolved.cookie_file == (
|
||||
PATHS.state_root / "cookies/xiaohongshu_cookies.json"
|
||||
).resolve()
|
||||
assert resolved.storage_state_file == (
|
||||
PATHS.state_root / "cookies/xiaohongshu_storage_state.json"
|
||||
).resolve()
|
||||
assert resolved.profile_dir == (
|
||||
PATHS.state_root / "browser-profiles/xiaohongshu"
|
||||
).resolve()
|
||||
|
||||
|
||||
def test_child_login_environment_preserves_wrapper_binding(tmp_path: Path) -> None:
|
||||
cookie = tmp_path / "cookies.json"
|
||||
storage = tmp_path / "storage_state.json"
|
||||
profile = tmp_path / "profile"
|
||||
|
||||
environment = child_browser_environment(
|
||||
cookie_file=cookie,
|
||||
storage_state_file=storage,
|
||||
profile_dir=profile,
|
||||
environment={"GYXX_MODULE_ID": "content_marketing", "KEEP": "1"},
|
||||
)
|
||||
|
||||
assert environment["GYXX_BROWSER_COOKIE_FILE"] == str(cookie.resolve())
|
||||
assert environment["GYXX_BROWSER_STORAGE_STATE_FILE"] == str(
|
||||
storage.resolve()
|
||||
)
|
||||
assert environment["GYXX_BROWSER_PROFILE_DIR"] == str(profile.resolve())
|
||||
assert "GYXX_MODULE_ID" not in environment
|
||||
assert environment["KEEP"] == "1"
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("module", "platform", "login_cookie", "argv"),
|
||||
[
|
||||
(
|
||||
relogin_bilibili,
|
||||
"bilibili",
|
||||
"SESSDATA",
|
||||
["relogin_bilibili.py"],
|
||||
),
|
||||
(relogin_douyin, "douyin", "sessionid", ["relogin_douyin.py"]),
|
||||
(relogin_pgy, "pgy", "web_session", ["relogin_pgy.py"]),
|
||||
(
|
||||
relogin_xiaohongshu,
|
||||
"xiaohongshu",
|
||||
"web_session",
|
||||
["relogin_xiaohongshu.py"],
|
||||
),
|
||||
(
|
||||
relogin_xingtu,
|
||||
"xingtu",
|
||||
"sessionid",
|
||||
["relogin_xingtu.py", "--force"],
|
||||
),
|
||||
],
|
||||
)
|
||||
def test_relogin_wrapper_validates_the_same_binding_written_by_child(
|
||||
tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
module: ModuleType,
|
||||
platform: str,
|
||||
login_cookie: str,
|
||||
argv: list[str],
|
||||
) -> None:
|
||||
cookie = tmp_path / "cookies.json"
|
||||
storage = tmp_path / "storage_state.json"
|
||||
profile = tmp_path / "profile"
|
||||
monkeypatch.setattr(module, "COOKIE_FILE", cookie)
|
||||
monkeypatch.setattr(module, "STATE_FILE", storage)
|
||||
monkeypatch.setattr(module, "PROFILE_DIR", profile)
|
||||
monkeypatch.setattr(module.sys, "argv", argv)
|
||||
monkeypatch.setenv("GYXX_MODULE_ID", "content_marketing")
|
||||
if module is relogin_xingtu:
|
||||
monkeypatch.setattr(module, "terminate_profile_processes", lambda: None)
|
||||
monkeypatch.setattr(module, "sync_to_douyin_scraper", lambda: None)
|
||||
synced: list[tuple[str, Path, Path]] = []
|
||||
|
||||
def fake_sync(
|
||||
synced_platform: str,
|
||||
*,
|
||||
cookie_file: Path,
|
||||
storage_state_file: Path,
|
||||
) -> tuple[Path, ...]:
|
||||
synced.append(
|
||||
(synced_platform, cookie_file.resolve(), storage_state_file.resolve())
|
||||
)
|
||||
return ()
|
||||
|
||||
monkeypatch.setattr(module, "sync_relogin_state_to_consumers", fake_sync)
|
||||
|
||||
def fake_call(
|
||||
_command: list[str],
|
||||
*,
|
||||
cwd: str,
|
||||
env: dict[str, str],
|
||||
) -> int:
|
||||
assert cwd == str(module.PROJECT_DIR)
|
||||
assert env["GYXX_BROWSER_COOKIE_FILE"] == str(cookie.resolve())
|
||||
assert env["GYXX_BROWSER_STORAGE_STATE_FILE"] == str(storage.resolve())
|
||||
assert env["GYXX_BROWSER_PROFILE_DIR"] == str(profile.resolve())
|
||||
assert "GYXX_MODULE_ID" not in env
|
||||
cookie.write_text(
|
||||
json.dumps([{"name": login_cookie, "value": "fresh"}]),
|
||||
encoding="utf-8",
|
||||
)
|
||||
storage.write_text("{}", encoding="utf-8")
|
||||
profile.mkdir()
|
||||
return 0
|
||||
|
||||
monkeypatch.setattr(module.subprocess, "call", fake_call)
|
||||
|
||||
assert module.main() == 0
|
||||
assert cookie.is_file()
|
||||
assert storage.is_file()
|
||||
assert profile.is_dir()
|
||||
assert synced == [(platform, cookie.resolve(), storage.resolve())]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("module", "platform", "login_cookie", "cookie_domain", "argv"),
|
||||
[
|
||||
(
|
||||
relogin_bilibili,
|
||||
"bilibili",
|
||||
"SESSDATA",
|
||||
".bilibili.com",
|
||||
["relogin_bilibili.py"],
|
||||
),
|
||||
(
|
||||
relogin_douyin,
|
||||
"douyin",
|
||||
"sessionid",
|
||||
".douyin.com",
|
||||
["relogin_douyin.py"],
|
||||
),
|
||||
(
|
||||
relogin_pgy,
|
||||
"pgy",
|
||||
"web_session",
|
||||
".xiaohongshu.com",
|
||||
["relogin_pgy.py"],
|
||||
),
|
||||
(
|
||||
relogin_xiaohongshu,
|
||||
"xiaohongshu",
|
||||
"web_session",
|
||||
".xiaohongshu.com",
|
||||
["relogin_xiaohongshu.py"],
|
||||
),
|
||||
(
|
||||
relogin_xingtu,
|
||||
"xingtu",
|
||||
"sessionid",
|
||||
".xingtu.cn",
|
||||
["relogin_xingtu.py", "--force"],
|
||||
),
|
||||
],
|
||||
)
|
||||
def test_relogin_wrapper_restores_previous_state_when_publish_fails(
|
||||
tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
module: ModuleType,
|
||||
platform: str,
|
||||
login_cookie: str,
|
||||
cookie_domain: str,
|
||||
argv: list[str],
|
||||
) -> None:
|
||||
cookie = tmp_path / "cookies.json"
|
||||
storage = tmp_path / "storage_state.json"
|
||||
profile = tmp_path / "profile"
|
||||
old_cookie = '[{"name":"old","value":"old-session"}]'
|
||||
old_storage = '{"cookies":[{"name":"old"}]}'
|
||||
cookie.write_text(old_cookie, encoding="utf-8")
|
||||
storage.write_text(old_storage, encoding="utf-8")
|
||||
profile.mkdir()
|
||||
(profile / "old-state.txt").write_text("old-profile", encoding="utf-8")
|
||||
monkeypatch.setattr(module, "COOKIE_FILE", cookie)
|
||||
monkeypatch.setattr(module, "STATE_FILE", storage)
|
||||
monkeypatch.setattr(module, "PROFILE_DIR", profile)
|
||||
monkeypatch.setattr(module.sys, "argv", argv)
|
||||
if module is relogin_xingtu:
|
||||
monkeypatch.setattr(module, "terminate_profile_processes", lambda: None)
|
||||
monkeypatch.setattr(module, "sync_to_douyin_scraper", lambda: None)
|
||||
|
||||
def fake_call(
|
||||
_command: list[str],
|
||||
*,
|
||||
cwd: str,
|
||||
env: dict[str, str],
|
||||
) -> int:
|
||||
assert cwd == str(module.PROJECT_DIR)
|
||||
assert env["GYXX_BROWSER_COOKIE_FILE"] == str(cookie.resolve())
|
||||
cookie.write_text(
|
||||
json.dumps(
|
||||
[
|
||||
{
|
||||
"name": login_cookie,
|
||||
"value": "fresh-session",
|
||||
"domain": cookie_domain,
|
||||
}
|
||||
]
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
storage.write_text('{"cookies":[],"origins":[]}', encoding="utf-8")
|
||||
profile.mkdir()
|
||||
return 0
|
||||
|
||||
def fail_publish(
|
||||
synced_platform: str,
|
||||
*,
|
||||
cookie_file: Path,
|
||||
storage_state_file: Path,
|
||||
) -> tuple[Path, ...]:
|
||||
assert synced_platform == platform
|
||||
assert cookie_file == cookie
|
||||
assert storage_state_file == storage
|
||||
raise OSError("simulated consumer publish failure")
|
||||
|
||||
monkeypatch.setattr(module.subprocess, "call", fake_call)
|
||||
monkeypatch.setattr(module, "sync_relogin_state_to_consumers", fail_publish)
|
||||
|
||||
assert module.main() == 1
|
||||
assert cookie.read_text(encoding="utf-8") == old_cookie
|
||||
assert storage.read_text(encoding="utf-8") == old_storage
|
||||
assert (profile / "old-state.txt").read_text(encoding="utf-8") == (
|
||||
"old-profile"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("platform", "cookie_name", "cookie_domain"),
|
||||
[
|
||||
("bilibili", "SESSDATA", ".bilibili.com"),
|
||||
("douyin", "sessionid", ".douyin.com"),
|
||||
("pgy", "web_session", ".xiaohongshu.com"),
|
||||
("xiaohongshu", "web_session", ".xiaohongshu.com"),
|
||||
("xingtu", "sessionid", ".xingtu.cn"),
|
||||
],
|
||||
)
|
||||
def test_successful_relogin_atomically_updates_unique_consumer_state(
|
||||
tmp_path: Path,
|
||||
platform: str,
|
||||
cookie_name: str,
|
||||
cookie_domain: str,
|
||||
) -> None:
|
||||
data_root = tmp_path / "data"
|
||||
source_cookie = tmp_path / "relogin" / "cookies.json"
|
||||
source_storage = tmp_path / "relogin" / "storage_state.json"
|
||||
source_cookie.parent.mkdir(parents=True)
|
||||
fresh_cookie = [
|
||||
{
|
||||
"name": cookie_name,
|
||||
"value": "fresh-session",
|
||||
"domain": cookie_domain,
|
||||
}
|
||||
]
|
||||
fresh_storage = {"cookies": fresh_cookie, "origins": []}
|
||||
source_cookie.write_text(json.dumps(fresh_cookie), encoding="utf-8")
|
||||
source_storage.write_text(json.dumps(fresh_storage), encoding="utf-8")
|
||||
|
||||
catalog = RuntimeIntegrationCatalog.load_default(
|
||||
project_root=PROJECT_ROOT,
|
||||
data_root=data_root,
|
||||
)
|
||||
consumer_id = CONSUMER_BINDING_IDS[platform][0]
|
||||
consumer = catalog.binding_for(consumer_id)
|
||||
consumer.cookie_file.parent.mkdir(parents=True, exist_ok=True)
|
||||
consumer.cookie_file.write_text('[{"name":"old"}]', encoding="utf-8")
|
||||
consumer.storage_state_file.write_text(
|
||||
'{"cookies":[{"name":"old"}]}',
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
destinations = sync_relogin_state_to_consumers(
|
||||
platform,
|
||||
cookie_file=source_cookie,
|
||||
storage_state_file=source_storage,
|
||||
environment={
|
||||
"GYXX_PROJECT_ROOT": str(PROJECT_ROOT),
|
||||
"GYXX_DATA_ROOT": str(data_root),
|
||||
},
|
||||
)
|
||||
|
||||
assert source_cookie.resolve() != consumer.cookie_file.resolve()
|
||||
assert source_storage.resolve() != consumer.storage_state_file.resolve()
|
||||
assert destinations == (consumer.cookie_file.resolve(),)
|
||||
assert json.loads(consumer.cookie_file.read_text(encoding="utf-8")) == (
|
||||
fresh_cookie
|
||||
)
|
||||
assert json.loads(
|
||||
consumer.storage_state_file.read_text(encoding="utf-8")
|
||||
) == fresh_storage
|
||||
|
||||
|
||||
def test_consumer_state_is_rolled_back_if_atomic_publish_fails(
|
||||
tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
data_root = tmp_path / "data"
|
||||
source_cookie = tmp_path / "relogin" / "cookies.json"
|
||||
source_storage = tmp_path / "relogin" / "storage_state.json"
|
||||
source_cookie.parent.mkdir(parents=True)
|
||||
source_cookie.write_text(
|
||||
json.dumps(
|
||||
[
|
||||
{
|
||||
"name": "SESSDATA",
|
||||
"value": "fresh-session",
|
||||
"domain": ".bilibili.com",
|
||||
}
|
||||
]
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
source_storage.write_text('{"cookies":[],"origins":[]}', encoding="utf-8")
|
||||
|
||||
catalog = RuntimeIntegrationCatalog.load_default(
|
||||
project_root=PROJECT_ROOT,
|
||||
data_root=data_root,
|
||||
)
|
||||
consumer = catalog.binding_for(CONSUMER_BINDING_IDS["bilibili"][0])
|
||||
consumer.cookie_file.parent.mkdir(parents=True, exist_ok=True)
|
||||
old_cookie = '[{"name":"SESSDATA","value":"old"}]'
|
||||
old_storage = '{"cookies":[{"name":"old"}]}'
|
||||
consumer.cookie_file.write_text(old_cookie, encoding="utf-8")
|
||||
consumer.storage_state_file.write_text(old_storage, encoding="utf-8")
|
||||
|
||||
original_replace = relogin_runtime_paths.os.replace
|
||||
failed = False
|
||||
|
||||
def fail_second_replace(source: Path | str, target: Path | str) -> None:
|
||||
nonlocal failed
|
||||
source_path = Path(source)
|
||||
target_path = Path(target)
|
||||
if (
|
||||
not failed
|
||||
and source_path.suffix == ".tmp"
|
||||
and target_path == consumer.storage_state_file
|
||||
):
|
||||
failed = True
|
||||
raise OSError("simulated storage publish failure")
|
||||
original_replace(source, target)
|
||||
|
||||
monkeypatch.setattr(relogin_runtime_paths.os, "replace", fail_second_replace)
|
||||
|
||||
with pytest.raises(OSError, match="simulated storage publish failure"):
|
||||
sync_relogin_state_to_consumers(
|
||||
"bilibili",
|
||||
cookie_file=source_cookie,
|
||||
storage_state_file=source_storage,
|
||||
environment={
|
||||
"GYXX_PROJECT_ROOT": str(PROJECT_ROOT),
|
||||
"GYXX_DATA_ROOT": str(data_root),
|
||||
},
|
||||
)
|
||||
|
||||
assert consumer.cookie_file.read_text(encoding="utf-8") == old_cookie
|
||||
assert consumer.storage_state_file.read_text(encoding="utf-8") == old_storage
|
||||
@@ -466,6 +466,83 @@ def test_collect_xiaohongshu_rows_from_artifact(tmp_path, monkeypatch):
|
||||
assert row[fields.index("page_metrics")].obj["笔记阅读数"] == "1.2万"
|
||||
|
||||
|
||||
def test_upsert_rows_isolates_single_row_failure():
|
||||
"""单行写入失败只跳过该行,不拖累整批(保存点隔离)。"""
|
||||
import contextlib
|
||||
|
||||
class _Cursor:
|
||||
def fetchone(self):
|
||||
return (1,)
|
||||
|
||||
def __getattr__(self, name):
|
||||
def record(*args, **kwargs):
|
||||
params = args[1] if len(args) > 1 else kwargs.get("params")
|
||||
if params is None or len(params) != len(sync_tool.ROW_FIELDS):
|
||||
return
|
||||
# 模拟毒行: 标题以 POISON 开头的行违反约束
|
||||
title = params[sync_tool.ROW_FIELDS.index("title")]
|
||||
if str(title).startswith("POISON"):
|
||||
raise RuntimeError("value too long")
|
||||
return record
|
||||
|
||||
class _Conn:
|
||||
def __init__(self):
|
||||
self.cursor_stub = _Cursor()
|
||||
self.committed = False
|
||||
|
||||
@contextlib.contextmanager
|
||||
def cursor(self):
|
||||
yield self.cursor_stub
|
||||
|
||||
@contextlib.contextmanager
|
||||
def transaction(self):
|
||||
yield
|
||||
|
||||
def commit(self):
|
||||
self.committed = True
|
||||
|
||||
metric_date = dt.date(2026, 10, 1)
|
||||
rows = [
|
||||
sync_tool.make_row("bilibili", metric_date, "daily.json",
|
||||
account_name="光影行星", video_id="BV1"),
|
||||
sync_tool.make_row("douyin", metric_date, "a.xlsx",
|
||||
account_name="店铺A", video_id="1",
|
||||
title="POISON 这行会失败"),
|
||||
sync_tool.make_row("xiaohongshu", metric_date, "daily.json",
|
||||
account_name="光影行星GYXX旗舰店", video_id="66e"),
|
||||
]
|
||||
conn = _Conn()
|
||||
|
||||
written, failed = sync_tool.upsert_rows(conn, rows)
|
||||
|
||||
assert (written, failed) == (2, 1)
|
||||
assert conn.committed
|
||||
|
||||
|
||||
def test_drop_stale_rows_only_removes_matching_sources():
|
||||
"""陈旧防护按来源文件跳过: 抖音旧导出只清掉抖音行,不拦 B站/小红书。"""
|
||||
stale_excel = Path("chanmama/旧导出_20260823-20260921.xlsx")
|
||||
rows = {
|
||||
("douyin", STORE_FLAGSHIP, "1", dt.date(2026, 10, 1)): sync_tool.make_row(
|
||||
"douyin", dt.date(2026, 10, 1), stale_excel.name, video_id="1"
|
||||
),
|
||||
("bilibili", "光影行星", "BV1", dt.date(2026, 10, 1)): sync_tool.make_row(
|
||||
"bilibili", dt.date(2026, 10, 1), "daily.json", video_id="BV1"
|
||||
),
|
||||
("xiaohongshu", XHS_SHOP, "2", dt.date(2026, 10, 1)): sync_tool.make_row(
|
||||
"xiaohongshu", dt.date(2026, 10, 1), "daily.json", video_id="2"
|
||||
),
|
||||
}
|
||||
|
||||
dropped = sync_tool.drop_stale_rows(rows, [stale_excel])
|
||||
|
||||
assert dropped == 1
|
||||
assert set(rows) == {
|
||||
("bilibili", "光影行星", "BV1", dt.date(2026, 10, 1)),
|
||||
("xiaohongshu", XHS_SHOP, "2", dt.date(2026, 10, 1)),
|
||||
}
|
||||
|
||||
|
||||
def test_douyin_history_backfill_uses_file_mtime_and_filename_account(tmp_path):
|
||||
"""历史回填: 按导出落盘日期各留快照,旧版12列文件按文件名前缀补账号。"""
|
||||
# 旧版 12 列表头(无 达人昵称/抖音号/蝉妈妈链接)
|
||||
|
||||
@@ -4,6 +4,7 @@ from datetime import date
|
||||
from pathlib import Path
|
||||
|
||||
import collect_douyin_qianchuan_ads as collector
|
||||
import pytest
|
||||
|
||||
|
||||
def test_qianchuan_browser_uses_scrapling_stealth_and_persistent_state(
|
||||
@@ -216,3 +217,174 @@ def test_download_task_from_page_downloads_completed_matching_row() -> None:
|
||||
collector._matching_task_rows = original
|
||||
|
||||
assert actual is expected_download
|
||||
|
||||
|
||||
def test_wait_task_center_ready_selects_advertiser_after_redirect() -> None:
|
||||
clicked: list[str] = []
|
||||
state = {"on_selection": True}
|
||||
|
||||
class _AccountCard:
|
||||
def click(self) -> None:
|
||||
clicked.append("account")
|
||||
state["on_selection"] = False
|
||||
|
||||
class _FilteredLocator:
|
||||
first = _AccountCard()
|
||||
|
||||
def count(self) -> int:
|
||||
return 1
|
||||
|
||||
class _AccountListLocator:
|
||||
def count(self) -> int:
|
||||
return 1 if state["on_selection"] else 0
|
||||
|
||||
def filter(self, *, has_text: str):
|
||||
assert has_text == collector.ADVERTISER_ID
|
||||
return _FilteredLocator()
|
||||
|
||||
class _Page:
|
||||
def locator(self, selector: str) -> _AccountListLocator:
|
||||
assert selector == collector.ACCOUNT_ITEM_SELECTOR
|
||||
return _AccountListLocator()
|
||||
|
||||
def wait_for_timeout(self, ms: int) -> None:
|
||||
pass
|
||||
|
||||
def wait_for_function(self, expression: str, *, arg, timeout: int) -> None:
|
||||
assert not state["on_selection"]
|
||||
assert arg == collector.TASK_READY_MARKER
|
||||
|
||||
collector._wait_task_center_ready(_Page())
|
||||
|
||||
assert clicked == ["account"]
|
||||
|
||||
|
||||
def test_wait_task_center_ready_raises_when_selection_page_persists() -> None:
|
||||
class _AccountCard:
|
||||
def click(self) -> None:
|
||||
pass
|
||||
|
||||
class _FilteredLocator:
|
||||
first = _AccountCard()
|
||||
|
||||
def count(self) -> int:
|
||||
return 1
|
||||
|
||||
class _AccountListLocator:
|
||||
def count(self) -> int:
|
||||
return 1
|
||||
|
||||
def filter(self, *, has_text: str):
|
||||
assert has_text == collector.ADVERTISER_ID
|
||||
return _FilteredLocator()
|
||||
|
||||
class _Page:
|
||||
def locator(self, selector: str) -> _AccountListLocator:
|
||||
assert selector == collector.ACCOUNT_ITEM_SELECTOR
|
||||
return _AccountListLocator()
|
||||
|
||||
def wait_for_timeout(self, ms: int) -> None:
|
||||
pass
|
||||
|
||||
def wait_for_function(self, expression: str, *, arg, timeout: int) -> None:
|
||||
raise AssertionError("不应在账户选择页上等待任务中心标记")
|
||||
|
||||
with pytest.raises(
|
||||
collector.DouyinQianchuanDownloadError, match="仍停留在账户选择页"
|
||||
):
|
||||
collector._wait_task_center_ready(_Page())
|
||||
|
||||
|
||||
def test_wait_task_center_ready_raises_when_never_ready(monkeypatch) -> None:
|
||||
class _AccountListLocator:
|
||||
def count(self) -> int:
|
||||
return 0
|
||||
|
||||
class _Page:
|
||||
def locator(self, selector: str) -> _AccountListLocator:
|
||||
assert selector == collector.ACCOUNT_ITEM_SELECTOR
|
||||
return _AccountListLocator()
|
||||
|
||||
def wait_for_function(self, expression: str, *, arg, timeout: int) -> None:
|
||||
raise RuntimeError("Page.wait_for_function: Timeout 15000ms exceeded.")
|
||||
|
||||
def goto(self, url: str, **options: object) -> None:
|
||||
assert url == collector.TASK_CENTER_PAGE_URL
|
||||
|
||||
monkeypatch.setattr(collector, "is_browser_timeout_error", lambda exc: True)
|
||||
|
||||
with pytest.raises(
|
||||
collector.DouyinQianchuanDownloadError, match="未达到可用状态"
|
||||
):
|
||||
collector._wait_task_center_ready(_Page(), timeout_ms=50)
|
||||
|
||||
|
||||
def test_download_task_from_page_recovers_from_context_destroyed(
|
||||
monkeypatch,
|
||||
) -> None:
|
||||
expected_download = object()
|
||||
state = {"count_calls": 0}
|
||||
|
||||
class _DownloadInfo:
|
||||
value = expected_download
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, exc_type, exc, traceback) -> None:
|
||||
pass
|
||||
|
||||
class _DownloadButton:
|
||||
def click(self) -> None:
|
||||
pass
|
||||
|
||||
class _Row:
|
||||
def inner_text(self) -> str:
|
||||
return "全域推广数据-推商品\n生成完成\n下载"
|
||||
|
||||
def get_by_text(self, text: str, *, exact: bool) -> _DownloadButton:
|
||||
assert (text, exact) == ("下载", True)
|
||||
return _DownloadButton()
|
||||
|
||||
class _Rows:
|
||||
first = _Row()
|
||||
|
||||
def count(self) -> int:
|
||||
state["count_calls"] += 1
|
||||
if state["count_calls"] == 1:
|
||||
raise RuntimeError(
|
||||
"Locator.count: Execution context was destroyed, "
|
||||
"most likely because of a navigation."
|
||||
)
|
||||
return 1
|
||||
|
||||
class _Body:
|
||||
def inner_text(self, *, timeout: int) -> str:
|
||||
assert timeout == 10_000
|
||||
return collector.TASK_READY_MARKER
|
||||
|
||||
class _Page:
|
||||
url = "https://qianchuan.jinritemai.com/tools/task-center/download-task"
|
||||
|
||||
def expect_download(self, *, timeout: int) -> _DownloadInfo:
|
||||
assert timeout == 60_000
|
||||
return _DownloadInfo()
|
||||
|
||||
def locator(self, selector: str) -> _Body:
|
||||
assert selector == "body"
|
||||
return _Body()
|
||||
|
||||
monkeypatch.setattr(collector, "_matching_task_rows", lambda *args: _Rows())
|
||||
monkeypatch.setattr(
|
||||
collector, "_wait_task_center_ready", lambda page, **kwargs: None
|
||||
)
|
||||
|
||||
actual = collector._download_task_from_page(
|
||||
_Page(),
|
||||
date(2026, 9, 29),
|
||||
date(2026, 9, 29),
|
||||
wait_timeout_ms=0,
|
||||
)
|
||||
|
||||
assert actual is expected_download
|
||||
assert state["count_calls"] == 2
|
||||
@@ -125,6 +125,57 @@ def test_style_records_sum_product_rows_and_keep_zero_styles() -> None:
|
||||
assert records[1]["matched_product_rows"] == 0
|
||||
|
||||
|
||||
def test_style_extraction_maps_hidden_columns_via_visible_positions() -> None:
|
||||
class Frame:
|
||||
def __init__(self) -> None:
|
||||
self.script = ""
|
||||
self.args = {}
|
||||
|
||||
def evaluate(self, script, args):
|
||||
self.script = script
|
||||
self.args = args
|
||||
return {"ok": True, "rows": []}
|
||||
|
||||
frame = Frame()
|
||||
backfill.extract_style_rows(frame, requested_codes=("10416",))
|
||||
|
||||
assert "window._Cols" in frame.script
|
||||
assert "visibleIdx.indexOf" in frame.script
|
||||
assert "cells[stylePos]" in frame.script
|
||||
assert "cells.length <= Math.max(stylePos, salesPos, returnsPos)" in frame.script
|
||||
|
||||
|
||||
def test_ensure_custom_columns_visible_only_touches_allsku_frames() -> None:
|
||||
class AllskuFrame:
|
||||
url = "https://bi.erp321.com/app/report/subject/itemskuanalysis/allsku.aspx?r=1"
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.scripts: list[str] = []
|
||||
|
||||
def evaluate(self, script):
|
||||
self.scripts.append(script)
|
||||
return 2
|
||||
|
||||
class OtherFrame:
|
||||
url = "https://bi.erp321.com/app/report/subject/itemskuanalysis/filter.aspx"
|
||||
|
||||
def evaluate(self, script):
|
||||
raise AssertionError("must not touch non-allsku frames")
|
||||
|
||||
class Page:
|
||||
frames = [OtherFrame(), AllskuFrame()]
|
||||
|
||||
page = Page()
|
||||
changed = backfill.ensure_custom_columns_visible(page)
|
||||
|
||||
assert changed == 2
|
||||
allsku = page.frames[1]
|
||||
assert len(allsku.scripts) == 1
|
||||
assert "'自定义销量'" in allsku.scripts[0]
|
||||
assert "'自定义退货量'" in allsku.scripts[0]
|
||||
assert "_RptRefresh" in allsku.scripts[0]
|
||||
|
||||
|
||||
def test_range_collector_queries_once_per_date_not_once_per_style(monkeypatch) -> None:
|
||||
filter_frame = object()
|
||||
monkeypatch.setattr(
|
||||
|
||||
@@ -0,0 +1,77 @@
|
||||
import unittest
|
||||
from unittest import mock
|
||||
|
||||
import collect_erp_yesterday_metrics as erp_mod
|
||||
|
||||
|
||||
class _StubFrame:
|
||||
"""frame.evaluate 的最小桩: 记录入参并返回固定 payload。"""
|
||||
|
||||
def __init__(self, payload):
|
||||
self._payload = payload
|
||||
self.calls = []
|
||||
|
||||
def evaluate(self, script, *args):
|
||||
self.calls.append((script, args))
|
||||
return self._payload
|
||||
|
||||
|
||||
class ErpDailyHeaderRoutingTest(unittest.TestCase):
|
||||
"""2026-10-11 聚水潭模板重置隐藏 38 列后, 固定可见索引 45/57 错位到
|
||||
「今日实退货量/今日仅退款量」, daily 销量全 0。daily 必须走表头名匹配,
|
||||
且不能落入 backfill 的 _extract_by_header (自定义列 SPU 求和) 分支。"""
|
||||
|
||||
def test_daily_mode_uses_yesterday_headers(self):
|
||||
with mock.patch.object(erp_mod, "MODE", "daily"):
|
||||
self.assertEqual(
|
||||
erp_mod._sales_returns_idx(),
|
||||
("昨日销量", "昨日退货量"),
|
||||
)
|
||||
|
||||
def test_weekly_mode_keeps_legacy_index(self):
|
||||
with mock.patch.object(erp_mod, "MODE", "weekly"):
|
||||
self.assertEqual(
|
||||
erp_mod._sales_returns_idx(),
|
||||
("__idx__45", "__idx__57"),
|
||||
)
|
||||
|
||||
def test_backfill_mode_keeps_custom_headers(self):
|
||||
with mock.patch.object(erp_mod, "MODE", "backfill"):
|
||||
self.assertEqual(
|
||||
erp_mod._sales_returns_idx(),
|
||||
("自定义销量", "自定义退货量"),
|
||||
)
|
||||
|
||||
def test_daily_header_keys_bypass_extract_by_header(self):
|
||||
payload = {
|
||||
"ok": True,
|
||||
"yesterday_sales": 2,
|
||||
"yesterday_returns": 0,
|
||||
"bodyText": "",
|
||||
}
|
||||
frame = _StubFrame(payload)
|
||||
with mock.patch.object(erp_mod, "MODE", "daily"):
|
||||
metrics, _body = erp_mod._extract_with_text(frame)
|
||||
self.assertEqual(metrics["yesterday_sales"], 2)
|
||||
self.assertEqual(len(frame.calls), 1)
|
||||
script, args = frame.calls[0]
|
||||
js_args = args[0] if args else script
|
||||
self.assertEqual(js_args["mode"], "header")
|
||||
self.assertEqual(js_args["salesHeader"], "昨日销量")
|
||||
self.assertEqual(js_args["returnsHeader"], "昨日退货量")
|
||||
|
||||
def test_backfill_mode_routes_to_extract_by_header(self):
|
||||
frame = _StubFrame({})
|
||||
with mock.patch.object(erp_mod, "MODE", "backfill"):
|
||||
with mock.patch.object(
|
||||
erp_mod,
|
||||
"_extract_by_header",
|
||||
return_value={"ok": True, "yesterday_sales": 5, "summary_text": "X"},
|
||||
) as by_header:
|
||||
metrics, _body = erp_mod._extract_with_text(frame)
|
||||
by_header.assert_called_once()
|
||||
self.assertEqual(metrics["yesterday_sales"], 5)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,191 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""聚水潭销售主题分析(按日)采集脚本的纯解析与 CLI 参数测试。"""
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from gyxx_flow.modules import product_commerce
|
||||
from gyxx_flow.modules.product_commerce import (
|
||||
collect_erp_sales_theme_daily as collector,
|
||||
)
|
||||
from gyxx_flow.modules.product_commerce import db as product_db
|
||||
|
||||
HEADER = [
|
||||
"日期", "销售单数", "销售数量", "实发数量", "实发金额", "销售金额",
|
||||
"退货单数", "退货数量", "实退数量", "退货金额", "实退金额",
|
||||
"净销售单数", "净销量", "净销售额", "运费收入", "运费支出", "实发单数",
|
||||
]
|
||||
|
||||
|
||||
def _write_workbook(path: Path, rows: list[list]) -> None:
|
||||
import openpyxl
|
||||
|
||||
workbook = openpyxl.Workbook()
|
||||
sheet = workbook.active
|
||||
for row in rows:
|
||||
sheet.append(row)
|
||||
workbook.save(str(path))
|
||||
workbook.close()
|
||||
|
||||
|
||||
def test_parse_sales_theme_workbook_roundtrip(tmp_path: Path) -> None:
|
||||
xlsx = tmp_path / "销售主题分析_按日_1.xlsx"
|
||||
_write_workbook(
|
||||
xlsx,
|
||||
[
|
||||
HEADER,
|
||||
["2026-10-11", "649", 655, 646, "173,129.23", 175752.83,
|
||||
15, 15, 0, 2677.47, "0.00", 634, 640, 173075.36, "20.00", None, 640],
|
||||
["2026-10-10", 690, 691, 685, 166761.58, 168185.97,
|
||||
9, 9, 0, 2656.71, 0, 681, 686, 166529.13, None, 1.6, 685],
|
||||
["##", 1339, 1346, 1331, 339890.81, 343938.8,
|
||||
24, 24, 0, 5334.18, 0, 1315, 1326, 339604.49, 20, 1.6, 1325],
|
||||
],
|
||||
)
|
||||
|
||||
records = collector.parse_sales_theme_workbook(xlsx)
|
||||
|
||||
assert [record["stat_date"] for record in records] == ["2026-10-11", "2026-10-10"]
|
||||
first = records[0]
|
||||
assert first["sales_order_count"] == 649
|
||||
assert first["sales_qty"] == 655
|
||||
assert first["shipped_amount"] == 173129.23
|
||||
assert first["sales_amount"] == 175752.83
|
||||
assert first["return_amount"] == 2677.47
|
||||
assert first["returned_amount"] == 0.0
|
||||
assert first["net_sales_order_count"] == 634
|
||||
assert first["net_sales_amount"] == 173075.36
|
||||
assert first["freight_income"] == 20.0
|
||||
assert first["freight_expense"] is None
|
||||
assert first["shipped_order_count"] == 640
|
||||
# 合计行(##)必须被跳过
|
||||
assert len(records) == 2
|
||||
|
||||
|
||||
def test_parse_sales_theme_workbook_rejects_missing_date_column(tmp_path: Path) -> None:
|
||||
xlsx = tmp_path / "bad.xlsx"
|
||||
_write_workbook(xlsx, [["销售单数", "销售数量"], [1, 2]])
|
||||
with pytest.raises(ValueError):
|
||||
collector.parse_sales_theme_workbook(xlsx)
|
||||
|
||||
|
||||
def test_parse_number_and_amount_helpers() -> None:
|
||||
assert collector.parse_number("1,339") == 1339
|
||||
assert collector.parse_number(None) == 0
|
||||
assert collector.parse_number("") == 0
|
||||
assert collector.parse_amount(None) is None
|
||||
assert collector.parse_amount("") is None
|
||||
assert collector.parse_amount("1.6") == 1.6
|
||||
assert collector.parse_amount(2677.467) == 2677.47
|
||||
|
||||
|
||||
class _CaptureCursor:
|
||||
def __init__(self) -> None:
|
||||
self.batches: list[tuple] = []
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *exc_info):
|
||||
return False
|
||||
|
||||
|
||||
class _CaptureConnection:
|
||||
def __init__(self) -> None:
|
||||
self.cursor_obj = _CaptureCursor()
|
||||
|
||||
def cursor(self):
|
||||
return self.cursor_obj
|
||||
|
||||
|
||||
def test_upsert_sales_theme_dedups_and_normalizes(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
captured: dict[str, object] = {}
|
||||
|
||||
def fake_execute_values(cur, sql, rows, template=None):
|
||||
captured["sql"] = sql
|
||||
captured["rows"] = list(rows)
|
||||
|
||||
monkeypatch.setattr(product_db, "execute_values", fake_execute_values)
|
||||
records = [
|
||||
{
|
||||
"stat_date": "2026-10-11",
|
||||
"sales_order_count": 649,
|
||||
"freight_income": "20.00",
|
||||
},
|
||||
{
|
||||
# 同日重复: 整体覆盖前一条(同 upsert_erp_all_shop 语义), 数值缺省为 0
|
||||
"stat_date": "2026-10-11",
|
||||
"sales_order_count": 999,
|
||||
"brands": ["光影", "光影行星", "光影"],
|
||||
"shops": ["天猫-光影行星旗舰店", "JD光影行星旗舰店"],
|
||||
"source_file": "var/raw/erp销售主题分析日报/x.xlsx",
|
||||
},
|
||||
{
|
||||
"stat_date": "2026-10-10",
|
||||
"sales_qty": "691",
|
||||
},
|
||||
]
|
||||
|
||||
conn = _CaptureConnection()
|
||||
written = product_db.upsert_erp_sales_theme_daily_metrics(conn, records)
|
||||
|
||||
assert written == 2
|
||||
rows = captured["rows"]
|
||||
assert len(rows) == 2
|
||||
by_date = {row[0]: row for row in rows}
|
||||
# 同日重复记录取最后一条, 数字字段缺省为 0, 金额字符串转 float
|
||||
assert by_date["2026-10-11"][1] == 999
|
||||
assert by_date["2026-10-10"][2] == 691
|
||||
tenth = by_date["2026-10-10"]
|
||||
# NOT NULL 金额列缺省(导出空单元格)落 0; 运费两列允许 NULL
|
||||
assert tenth[9] == 0 # return_amount
|
||||
assert tenth[15] is None # freight_income
|
||||
assert tenth[16] is None # freight_expense
|
||||
eleventh = by_date["2026-10-11"]
|
||||
assert eleventh[-4] == "发货时间" # date_basis 默认
|
||||
assert eleventh[-3] == ["光影", "光影行星"] # brands 去重排序
|
||||
assert eleventh[-2] == ["JD光影行星旗舰店", "天猫-光影行星旗舰店"]
|
||||
assert eleventh[-1] == "var/raw/erp销售主题分析日报/x.xlsx"
|
||||
# upsert 目标表与主键在 SQL 里声明
|
||||
sql = str(captured["sql"])
|
||||
assert "erp_sales_theme_daily_metrics" in sql
|
||||
assert "ON CONFLICT (stat_date)" in sql
|
||||
|
||||
|
||||
def test_business_scope_constants_are_consistent() -> None:
|
||||
assert len(collector.BRANDS) == 3
|
||||
assert sorted(collector.BRANDS) == ["gyxx", "光影", "光影行星"]
|
||||
assert len(collector.SHOP_NAMES) == len(set(collector.SHOP_NAMES)) > 0
|
||||
for name in collector.SHOP_NAMES:
|
||||
assert name == name.strip()
|
||||
|
||||
|
||||
def test_resolve_window_variants() -> None:
|
||||
args = collector.parse_args(["--date", "2026-10-11"])
|
||||
start, end = collector.resolve_window(args)
|
||||
assert (start.isoformat(), end.isoformat()) == ("2026-10-11", "2026-10-11")
|
||||
|
||||
args = collector.parse_args(["--date", "2026-10-11", "--days", "7"])
|
||||
start, end = collector.resolve_window(args)
|
||||
assert (start.isoformat(), end.isoformat()) == ("2026-10-05", "2026-10-11")
|
||||
|
||||
args = collector.parse_args(["--from", "2025-06-01", "--to", "2026-01-01"])
|
||||
start, end = collector.resolve_window(args)
|
||||
assert (start.isoformat(), end.isoformat()) == ("2025-06-01", "2026-01-01")
|
||||
|
||||
with pytest.raises(SystemExit):
|
||||
collector.resolve_window(collector.parse_args(["--from", "2026-01-01"]))
|
||||
with pytest.raises(SystemExit):
|
||||
collector.resolve_window(
|
||||
collector.parse_args(["--from", "2026-01-01", "--to", "2025-01-01"])
|
||||
)
|
||||
|
||||
|
||||
def test_module_uses_shared_cookie_skip_exit_code() -> None:
|
||||
from gyxx_flow.core.exit_codes import COOKIE_SKIP_EXIT_CODE
|
||||
|
||||
assert COOKIE_SKIP_EXIT_CODE == 75
|
||||
assert sys.modules[product_commerce.__name__] is not None
|
||||
@@ -237,6 +237,62 @@ def test_login_if_needed_fills_the_password_form_inside_an_iframe(monkeypatch) -
|
||||
assert frame.submit.click_count == 1
|
||||
|
||||
|
||||
def test_login_if_needed_accepts_agreement_popup_and_resubmits(monkeypatch) -> None:
|
||||
events: list[str] = []
|
||||
|
||||
class Locator:
|
||||
def __init__(self, name: str) -> None:
|
||||
self.name = name
|
||||
|
||||
def fill(self, _value: str) -> None:
|
||||
events.append(f"fill:{self.name}")
|
||||
|
||||
def click(self, timeout: int = 0) -> None:
|
||||
events.append(f"click:{self.name}")
|
||||
|
||||
username = Locator("username")
|
||||
password = Locator("password")
|
||||
submit = Locator("submit")
|
||||
page = SimpleNamespace(wait_for_timeout=lambda ms: events.append(f"wait:{ms}"))
|
||||
|
||||
monkeypatch.setattr(baibu, "_authenticated", lambda *_args: False)
|
||||
monkeypatch.setattr(baibu, "_is_login_url", lambda _url: True)
|
||||
|
||||
def fake_form(_page, *, use_password_form):
|
||||
return (_page, username, password, submit)
|
||||
|
||||
monkeypatch.setattr(baibu, "_find_password_login_form", fake_form)
|
||||
monkeypatch.setattr(
|
||||
baibu,
|
||||
"_credentials_from_environment",
|
||||
lambda: (baibu._LoginCredentials(username="acct", secret_value="secret"), ()),
|
||||
)
|
||||
agreement = {"visible": True}
|
||||
monkeypatch.setattr(
|
||||
baibu, "_is_login_agreement_visible", lambda _page: agreement["visible"]
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
baibu,
|
||||
"_click_login_agreement_accept",
|
||||
lambda _page: agreement.update(visible=False) or True,
|
||||
)
|
||||
|
||||
assert baibu._login_if_needed(_entry(), page) is True
|
||||
|
||||
# First submit is swallowed by the 服务协议 popup; after accepting it the
|
||||
# credentials are re-filled and submitted a second time.
|
||||
assert events == [
|
||||
"fill:username",
|
||||
"fill:password",
|
||||
"click:submit",
|
||||
"wait:2000",
|
||||
"fill:username",
|
||||
"fill:password",
|
||||
"click:submit",
|
||||
"wait:2000",
|
||||
]
|
||||
|
||||
|
||||
def test_manual_login_waits_without_filling_the_password_form(monkeypatch) -> None:
|
||||
monkeypatch.setattr(baibu, "_authenticated", lambda _entry, _page: False)
|
||||
frame = _LoginFrame()
|
||||
@@ -301,6 +357,14 @@ def test_login_second_factor_marker_is_detected(monkeypatch) -> None:
|
||||
assert baibu._login_second_factor_marker(SimpleNamespace()) == "短信验证"
|
||||
|
||||
|
||||
def test_login_slider_captcha_marker_is_detected(monkeypatch) -> None:
|
||||
monkeypatch.setattr(
|
||||
baibu, "_page_text", lambda _page: "向右拖动滑块填充拼图完成验证"
|
||||
)
|
||||
|
||||
assert baibu._login_second_factor_marker(SimpleNamespace()) == "滑块"
|
||||
|
||||
|
||||
def test_persist_account_state_filters_cookies_to_the_account_domains(
|
||||
monkeypatch,
|
||||
tmp_path: Path,
|
||||
@@ -351,6 +415,289 @@ def test_persist_account_state_filters_cookies_to_the_account_domains(
|
||||
]
|
||||
|
||||
|
||||
def _browser_spy(calls: list[str]):
|
||||
class Page:
|
||||
def goto(self, *_args, **_kwargs) -> None:
|
||||
calls.append("goto")
|
||||
|
||||
class Context:
|
||||
pages = [Page()]
|
||||
|
||||
def new_page(self):
|
||||
return Page()
|
||||
|
||||
class Browser:
|
||||
context = Context()
|
||||
|
||||
def start(self) -> None:
|
||||
calls.append("start")
|
||||
|
||||
def persist_bound_state(self) -> None:
|
||||
calls.append("persist-bound")
|
||||
|
||||
def close(self) -> None:
|
||||
calls.append("close")
|
||||
|
||||
return Browser()
|
||||
|
||||
|
||||
def test_apply_entry_continues_when_account_vault_refresh_fails(monkeypatch) -> None:
|
||||
calls: list[str] = []
|
||||
|
||||
monkeypatch.setattr(baibu, "_start_browser", lambda: _browser_spy(calls))
|
||||
monkeypatch.setattr(baibu, "_login_if_needed", lambda *_args, **_kwargs: False)
|
||||
monkeypatch.setattr(baibu, "_wait_for_authenticated", lambda *_args, **_kwargs: None)
|
||||
|
||||
def fail_vault_refresh(_browser, _entry, **_kwargs):
|
||||
calls.append("persist-fail")
|
||||
raise baibu.BaibuApplyError(
|
||||
"测试入口 登录后未获得可用的账号 cookie(域内 cookie:无;"
|
||||
"要求含:unb、cookie2、_tb_token_),未更新账号登录态"
|
||||
)
|
||||
|
||||
monkeypatch.setattr(baibu, "_persist_account_state", fail_vault_refresh)
|
||||
|
||||
def record_explicit_login(_entry, _page, *, manual_login):
|
||||
calls.append("explicit-login")
|
||||
|
||||
monkeypatch.setattr(baibu, "_explicit_password_login", record_explicit_login)
|
||||
|
||||
def record_open(_page, _entry):
|
||||
calls.append("open")
|
||||
return None
|
||||
|
||||
monkeypatch.setattr(baibu, "_open_batch_import", record_open)
|
||||
monkeypatch.setattr(baibu, "_upload_template", lambda *_args: calls.append("upload"))
|
||||
monkeypatch.setattr(baibu, "_wait_import_result", lambda *_args, **_kwargs: calls.append("result"))
|
||||
monkeypatch.setattr(baibu, "_complete_draft_products", lambda *_args: calls.append("drafts"))
|
||||
monkeypatch.setattr(baibu.time, "sleep", lambda _seconds: None)
|
||||
|
||||
baibu._apply_entry(_entry(), Path("template.xlsx"))
|
||||
|
||||
assert calls == [
|
||||
"start",
|
||||
"goto",
|
||||
"persist-bound",
|
||||
"persist-fail",
|
||||
"explicit-login",
|
||||
"persist-fail",
|
||||
"open",
|
||||
"upload",
|
||||
"result",
|
||||
"drafts",
|
||||
"close",
|
||||
]
|
||||
|
||||
|
||||
def test_apply_entry_retries_through_explicit_login_when_page_never_shows_login(
|
||||
monkeypatch,
|
||||
) -> None:
|
||||
calls: list[str] = []
|
||||
|
||||
monkeypatch.setattr(baibu, "_start_browser", lambda: _browser_spy(calls))
|
||||
monkeypatch.setattr(baibu, "_login_if_needed", lambda *_args, **_kwargs: False)
|
||||
attempts = {"count": 0}
|
||||
|
||||
def fail_first_authentication(*_args, **_kwargs):
|
||||
attempts["count"] += 1
|
||||
if attempts["count"] == 1:
|
||||
raise baibu.BaibuApplyError("测试入口 当前未登录对应店铺,已停止")
|
||||
calls.append("authenticated")
|
||||
|
||||
monkeypatch.setattr(baibu, "_wait_for_authenticated", fail_first_authentication)
|
||||
|
||||
def record_explicit_login(_entry, _page, *, manual_login):
|
||||
calls.append("explicit-login")
|
||||
|
||||
monkeypatch.setattr(baibu, "_explicit_password_login", record_explicit_login)
|
||||
monkeypatch.setattr(
|
||||
baibu,
|
||||
"_persist_account_state",
|
||||
lambda *_args, **_kwargs: calls.append("persist-ok") or True,
|
||||
)
|
||||
|
||||
def record_open(_page, _entry):
|
||||
calls.append("open")
|
||||
return None
|
||||
|
||||
monkeypatch.setattr(baibu, "_open_batch_import", record_open)
|
||||
monkeypatch.setattr(baibu, "_upload_template", lambda *_args: calls.append("upload"))
|
||||
monkeypatch.setattr(baibu, "_wait_import_result", lambda *_args, **_kwargs: calls.append("result"))
|
||||
monkeypatch.setattr(baibu, "_complete_draft_products", lambda *_args: calls.append("drafts"))
|
||||
monkeypatch.setattr(baibu.time, "sleep", lambda _seconds: None)
|
||||
|
||||
baibu._apply_entry(_entry(), Path("template.xlsx"))
|
||||
|
||||
assert attempts["count"] == 1
|
||||
assert calls == [
|
||||
"start",
|
||||
"goto",
|
||||
"explicit-login",
|
||||
"persist-bound",
|
||||
"persist-ok",
|
||||
"open",
|
||||
"upload",
|
||||
"result",
|
||||
"drafts",
|
||||
"close",
|
||||
]
|
||||
|
||||
|
||||
def test_explicit_password_login_drives_login_page_then_returns_to_apply(
|
||||
monkeypatch,
|
||||
) -> None:
|
||||
calls: list[str] = []
|
||||
|
||||
class Page:
|
||||
def goto(self, url, **_kwargs) -> None:
|
||||
calls.append(f"goto:{url}")
|
||||
|
||||
def wait_for_timeout(self, _ms: int) -> None:
|
||||
return None
|
||||
|
||||
monkeypatch.setattr(
|
||||
baibu, "_login_if_needed", lambda *_args, **_kwargs: calls.append("fill-login") or True
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
baibu,
|
||||
"_wait_for_authenticated",
|
||||
lambda *_args, **_kwargs: calls.append("authenticated"),
|
||||
)
|
||||
|
||||
entry = _entry()
|
||||
baibu._explicit_password_login(entry, Page(), manual_login=False)
|
||||
|
||||
# The apply URL is re-entered first so its redirect chain can surface the
|
||||
# seller login page; only a missing form there falls back to login.taobao.
|
||||
assert calls == [
|
||||
f"goto:{entry.apply_url}",
|
||||
"fill-login",
|
||||
f"goto:{entry.apply_url}",
|
||||
"authenticated",
|
||||
]
|
||||
|
||||
|
||||
class _FakeLock:
|
||||
def __init__(self, manager: "_FakeLockManager", resource: str) -> None:
|
||||
self._manager = manager
|
||||
self._resource = resource
|
||||
|
||||
def __enter__(self):
|
||||
self._manager.held.add(self._resource)
|
||||
self._manager.entered.append(self._resource)
|
||||
return self
|
||||
|
||||
def __exit__(self, *_args) -> None:
|
||||
self._manager.held.discard(self._resource)
|
||||
|
||||
|
||||
class _FakeLockManager:
|
||||
def __init__(self, lock_root) -> None:
|
||||
self.lock_root = lock_root
|
||||
self.held: set[str] = set()
|
||||
self.entered: list[str] = []
|
||||
self.requested: list[tuple[str, int]] = []
|
||||
|
||||
def acquire(self, resource: str, *, owner: str, timeout_seconds: int):
|
||||
self.requested.append((resource, timeout_seconds))
|
||||
if getattr(self, "busy", False):
|
||||
raise baibu.ResourceBusyError(resource)
|
||||
return _FakeLock(self, resource)
|
||||
|
||||
|
||||
def test_apply_entry_holds_account_lock_for_whole_session(monkeypatch) -> None:
|
||||
calls: list[str] = []
|
||||
lock_manager = _FakeLockManager(lock_root="locks-dir")
|
||||
|
||||
class FakeSettings:
|
||||
data_root = Path("data-root")
|
||||
|
||||
@classmethod
|
||||
def from_env(cls):
|
||||
return cls()
|
||||
|
||||
monkeypatch.setattr(baibu, "Settings", FakeSettings)
|
||||
monkeypatch.setattr(baibu, "LockManager", lambda lock_root: lock_manager)
|
||||
monkeypatch.setattr(baibu, "_start_browser", lambda: _browser_spy(calls))
|
||||
monkeypatch.setattr(baibu, "_login_if_needed", lambda *_args, **_kwargs: False)
|
||||
monkeypatch.setattr(baibu, "_wait_for_authenticated", lambda *_args, **_kwargs: None)
|
||||
monkeypatch.setattr(
|
||||
baibu,
|
||||
"_persist_account_state",
|
||||
lambda *_args, **_kwargs: True,
|
||||
)
|
||||
monkeypatch.setattr(baibu, "_open_batch_import", lambda _page, _entry: None)
|
||||
|
||||
def assert_lock_held_during_upload(_page, _template):
|
||||
assert lock_manager.held == {"browser-account:tmall-shop"}
|
||||
calls.append("upload")
|
||||
|
||||
monkeypatch.setattr(baibu, "_upload_template", assert_lock_held_during_upload)
|
||||
monkeypatch.setattr(baibu, "_wait_import_result", lambda *_args, **_kwargs: None)
|
||||
monkeypatch.setattr(baibu, "_complete_draft_products", lambda *_args: None)
|
||||
monkeypatch.setattr(baibu.time, "sleep", lambda _seconds: None)
|
||||
|
||||
baibu._apply_entry(_entry(), Path("template.xlsx"))
|
||||
|
||||
assert lock_manager.requested == [("browser-account:tmall-shop", 120)]
|
||||
assert lock_manager.entered == ["browser-account:tmall-shop"]
|
||||
assert lock_manager.held == set()
|
||||
|
||||
|
||||
def test_apply_entry_reports_busy_account_lock(monkeypatch) -> None:
|
||||
lock_manager = _FakeLockManager(lock_root="locks-dir")
|
||||
lock_manager.busy = True
|
||||
|
||||
class FakeSettings:
|
||||
data_root = Path("data-root")
|
||||
|
||||
@classmethod
|
||||
def from_env(cls):
|
||||
return cls()
|
||||
|
||||
monkeypatch.setattr(baibu, "Settings", FakeSettings)
|
||||
monkeypatch.setattr(baibu, "LockManager", lambda lock_root: lock_manager)
|
||||
started: list[str] = []
|
||||
monkeypatch.setattr(
|
||||
baibu, "_start_browser", lambda: started.append("start") or object()
|
||||
)
|
||||
|
||||
with pytest.raises(baibu.BaibuApplyError, match="正被保活/登录占用"):
|
||||
baibu._apply_entry(_entry(), Path("template.xlsx"))
|
||||
|
||||
assert started == []
|
||||
|
||||
|
||||
def test_explicit_password_login_falls_back_to_plain_login_page(
|
||||
monkeypatch,
|
||||
) -> None:
|
||||
calls: list[str] = []
|
||||
|
||||
class Page:
|
||||
def goto(self, url, **_kwargs) -> None:
|
||||
calls.append(f"goto:{url}")
|
||||
|
||||
def wait_for_timeout(self, _ms: int) -> None:
|
||||
return None
|
||||
|
||||
monkeypatch.setattr(baibu, "_login_if_needed", lambda *_args, **_kwargs: False)
|
||||
monkeypatch.setattr(
|
||||
baibu,
|
||||
"_wait_for_authenticated",
|
||||
lambda *_args, **_kwargs: calls.append("authenticated"),
|
||||
)
|
||||
|
||||
entry = _entry()
|
||||
baibu._explicit_password_login(entry, Page(), manual_login=False)
|
||||
|
||||
assert calls == [
|
||||
f"goto:{entry.apply_url}",
|
||||
f"goto:{baibu._LOGIN_PAGE_URL}",
|
||||
f"goto:{entry.apply_url}",
|
||||
"authenticated",
|
||||
]
|
||||
|
||||
|
||||
def test_upload_template_supports_tmall_dialog_import_button(
|
||||
monkeypatch,
|
||||
tmp_path: Path,
|
||||
@@ -558,6 +905,32 @@ def test_wait_import_result_accepts_new_failure_summary_as_submitted(
|
||||
baibu._wait_import_result(SimpleNamespace(), _entry())
|
||||
|
||||
|
||||
def test_wait_import_result_accepts_slow_processing_after_acknowledgement(
|
||||
monkeypatch,
|
||||
capsys,
|
||||
) -> None:
|
||||
# Tmall acknowledged the batch but never returned a row-level summary
|
||||
# within the acceptance window; the acknowledged submission still counts.
|
||||
processing_text = (
|
||||
"您于2026-10-14 14:51:00执行了商品批量导入操作,正在处理中。"
|
||||
)
|
||||
clock = {"now": 0.0}
|
||||
|
||||
def advance_monotonic() -> float:
|
||||
clock["now"] += 15.0
|
||||
return clock["now"]
|
||||
|
||||
monkeypatch.setattr(baibu, "_page_text", lambda _page: processing_text)
|
||||
monkeypatch.setattr(baibu, "_refresh_batch_results", lambda _page: True)
|
||||
monkeypatch.setattr(baibu.time, "sleep", lambda _seconds: None)
|
||||
monkeypatch.setattr(baibu.time, "monotonic", advance_monotonic)
|
||||
monkeypatch.setattr(baibu, "_RESULT_REFRESH_INTERVAL_SECONDS", 0)
|
||||
|
||||
baibu._wait_import_result(SimpleNamespace(), _entry())
|
||||
|
||||
assert "按导入提交成功" in capsys.readouterr().out
|
||||
|
||||
|
||||
def test_wait_import_result_ignores_stale_summary_until_signature_changes(
|
||||
monkeypatch,
|
||||
) -> None:
|
||||
@@ -663,6 +1036,47 @@ def test_complete_draft_products_accepts_empty_data_page(monkeypatch) -> None:
|
||||
baibu._complete_draft_products(page, entry)
|
||||
|
||||
|
||||
def test_complete_draft_products_retries_draft_tab_during_hydration(
|
||||
monkeypatch,
|
||||
) -> None:
|
||||
entry = replace(_entry(), complete_drafts_after_import=True)
|
||||
page = SimpleNamespace()
|
||||
attempts = {"n": 0}
|
||||
|
||||
def flaky_tab(_page):
|
||||
attempts["n"] += 1
|
||||
return attempts["n"] >= 3
|
||||
|
||||
monkeypatch.setattr(baibu, "_click_draft_tab", flaky_tab)
|
||||
monkeypatch.setattr(baibu, "_page_text", lambda _page: "加载中")
|
||||
monkeypatch.setattr(baibu, "_has_visible_text", lambda _page, _labels: False)
|
||||
monkeypatch.setattr(baibu, "_wait_for_draft_state", lambda _page, _entry: (0, 0))
|
||||
monkeypatch.setattr(baibu.time, "sleep", lambda _seconds: None)
|
||||
ticks = iter(range(10_000))
|
||||
monkeypatch.setattr(baibu.time, "monotonic", lambda: next(ticks) * 0.4)
|
||||
|
||||
baibu._complete_draft_products(page, entry)
|
||||
|
||||
assert attempts["n"] == 3
|
||||
|
||||
|
||||
def test_complete_draft_products_reports_missing_draft_entry_after_retries(
|
||||
monkeypatch,
|
||||
) -> None:
|
||||
entry = replace(_entry(), complete_drafts_after_import=True)
|
||||
page = SimpleNamespace()
|
||||
|
||||
monkeypatch.setattr(baibu, "_click_draft_tab", lambda _page: False)
|
||||
monkeypatch.setattr(baibu, "_page_text", lambda _page: "加载中")
|
||||
monkeypatch.setattr(baibu, "_has_visible_text", lambda _page, _labels: False)
|
||||
monkeypatch.setattr(baibu.time, "sleep", lambda _seconds: None)
|
||||
ticks = iter(range(10_000))
|
||||
monkeypatch.setattr(baibu.time, "monotonic", lambda: next(ticks) * 0.4)
|
||||
|
||||
with pytest.raises(baibu.BaibuApplyError, match="未找到草稿页入口"):
|
||||
baibu._complete_draft_products(page, entry)
|
||||
|
||||
|
||||
def test_return_to_draft_list_falls_back_to_browser_history(monkeypatch) -> None:
|
||||
events: list[str] = []
|
||||
|
||||
|
||||
@@ -10,9 +10,6 @@ from types import SimpleNamespace
|
||||
import pytest
|
||||
|
||||
from gyxx_flow.adapters import COOKIE_SKIP_EXIT_CODE
|
||||
from gyxx_flow.modules.content_marketing.data.tools import (
|
||||
friday_relogin_parallel as weekly_relogin,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
@@ -75,29 +72,6 @@ def _import_supply_script(module_name: str):
|
||||
sys.stderr = stderr
|
||||
|
||||
|
||||
def test_weekly_relogin_skips_the_entire_qr_workflow(
|
||||
acceptance_environment: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
capsys: pytest.CaptureFixture[str],
|
||||
) -> None:
|
||||
monkeypatch.setattr(
|
||||
weekly_relogin.subprocess,
|
||||
"Popen",
|
||||
_forbidden("acceptance mode must not launch relogin subprocesses"),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
weekly_relogin,
|
||||
"_send_lark",
|
||||
_forbidden("acceptance mode must not send QR screenshots"),
|
||||
)
|
||||
|
||||
assert weekly_relogin.main() == COOKIE_SKIP_EXIT_CODE
|
||||
assert "SKIPPED_COOKIE" in capsys.readouterr().out
|
||||
assert "content.relogin.weekly" in acceptance_environment.read_text(
|
||||
encoding="utf-8"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"module_name",
|
||||
[
|
||||
@@ -106,25 +80,17 @@ def test_weekly_relogin_skips_the_entire_qr_workflow(
|
||||
"batch_rescrape_bilibili",
|
||||
],
|
||||
)
|
||||
def test_batch_rescrape_relogin_never_starts_a_qr_process(
|
||||
def test_batch_rescrape_has_no_relogin_capability(
|
||||
module_name: str,
|
||||
acceptance_environment: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
capsys: pytest.CaptureFixture[str],
|
||||
) -> None:
|
||||
"""扫码重登能力已从批量补采工具中移除,登录统一由账号 vault 管理。"""
|
||||
|
||||
module = importlib.import_module(
|
||||
f"gyxx_flow.modules.content_marketing.data.tools.{module_name}"
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
module.subprocess,
|
||||
"call",
|
||||
_forbidden("acceptance mode must not invoke a relogin script"),
|
||||
)
|
||||
|
||||
assert module.do_relogin() is False
|
||||
assert "SKIPPED_COOKIE" in capsys.readouterr().out
|
||||
evidence = acceptance_environment.read_text(encoding="utf-8")
|
||||
assert f"content.{module_name}.relogin" in evidence
|
||||
assert not hasattr(module, "do_relogin")
|
||||
assert not hasattr(module, "RELOGIN_SCRIPT")
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
|
||||
@@ -56,7 +56,6 @@ def test_catalog_composition_registers_all_workflows() -> None:
|
||||
"content.metrics.self_operated.daily",
|
||||
"content.metrics.backfill",
|
||||
"content.marketing_report.daily",
|
||||
"content.relogin.weekly",
|
||||
"content.creator_report.monthly",
|
||||
"content.summary.monthly",
|
||||
"content.notes_master.daily",
|
||||
@@ -65,6 +64,7 @@ def test_catalog_composition_registers_all_workflows() -> None:
|
||||
"product.persona.daily",
|
||||
"product.daily",
|
||||
"product.online_stock_alert.daily",
|
||||
"product.hk_anpei_public_inventory.daily",
|
||||
"product.alert.daily",
|
||||
"product.ecommerce_costs.daily",
|
||||
"product.style_analysis.interval",
|
||||
|
||||
@@ -18,8 +18,8 @@ def test_catalog_maps_39_project_scheduled_tasks() -> None:
|
||||
assert len(scheduled) == 39
|
||||
assert len({item.source_task_name for item in scheduled}) == 39
|
||||
assert Counter(item.module for item in scheduled) == {
|
||||
"content_marketing": 10,
|
||||
"product_commerce": 13,
|
||||
"content_marketing": 9,
|
||||
"product_commerce": 14,
|
||||
"shop_intelligence": 12,
|
||||
"supply_chain": 4,
|
||||
}
|
||||
@@ -80,7 +80,6 @@ def test_jd_video_upload_is_an_idempotent_scheduled_workflow() -> None:
|
||||
assert step.args == (
|
||||
"--anchor-record-id",
|
||||
"recvrXTRWWFXeJ",
|
||||
"--manual-login",
|
||||
"--execute",
|
||||
)
|
||||
assert step.replay_policy == "idempotent"
|
||||
@@ -192,7 +191,7 @@ def test_every_scheduled_workflow_has_one_valid_asia_shanghai_schedule() -> None
|
||||
assert schedule.workflow_id == workflow.workflow_id
|
||||
assert schedule.kind in {"daily", "weekly", "monthly", "interval_days"}
|
||||
assert schedule.at.count(":") == 1
|
||||
assert sum(schedule.enabled for schedule in catalog.schedules) == 27
|
||||
assert sum(schedule.enabled for schedule in catalog.schedules) == 28
|
||||
customer_service = catalog.schedule_for("shop.jd_customer_service.daily")
|
||||
assert customer_service.at == "03:00"
|
||||
assert customer_service.enabled is True
|
||||
@@ -265,8 +264,7 @@ def test_scheduled_workflows_use_python_entries_and_content_tasks_are_graphs() -
|
||||
"data/tools/sync_self_video_metrics.py"
|
||||
)
|
||||
assert catalog.schedule_for(self_operated.workflow_id).effective_times == (
|
||||
"01:00",
|
||||
"05:00",
|
||||
"06:00",
|
||||
)
|
||||
|
||||
backfill = next(
|
||||
|
||||
@@ -54,7 +54,7 @@ def test_default_cli_registry_exposes_all_project_scheduled_workflows(
|
||||
scheduled_ids = {
|
||||
entry.workflow_id for entry in registry.catalog.scheduled_workflows()
|
||||
}
|
||||
assert len(scheduled_ids) == 35
|
||||
assert len(scheduled_ids) == 39
|
||||
assert scheduled_ids == {
|
||||
schedule.workflow_id for schedule in registry.catalog.schedules
|
||||
}
|
||||
|
||||
+93
-19
@@ -2,6 +2,7 @@ from __future__ import annotations
|
||||
|
||||
import http.client
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
@@ -311,7 +312,7 @@ def test_notification_config_persists_app_scoped_multi_recipient_routes(
|
||||
)
|
||||
initial = service.notification_config()
|
||||
|
||||
assert len(initial["people"]) == 16
|
||||
assert len(initial["people"]) == 18
|
||||
assert initial["available_app_profiles"] == ["hermes-analyzer"]
|
||||
assert [app["profile"] for app in initial["apps"]] == ["hermes-analyzer"]
|
||||
wang = next(person for person in initial["people"] if person["id"] == "wang_yunlong")
|
||||
@@ -321,7 +322,12 @@ def test_notification_config_persists_app_scoped_multi_recipient_routes(
|
||||
assert wang["bindings"]["hermes-analyzer"]["open_id"] == (
|
||||
"ou_7ad5fc8012e2f741afc5346e05ffd447"
|
||||
)
|
||||
assert all(route["configured"] is False for route in initial["routes"])
|
||||
stock_alert_route = next(
|
||||
route
|
||||
for route in initial["routes"]
|
||||
if route["workflow_id"] == "product.online_stock_alert.daily"
|
||||
)
|
||||
assert stock_alert_route["configured"] is True
|
||||
|
||||
routes = [dict(route) for route in initial["routes"]]
|
||||
daily = next(
|
||||
@@ -1566,7 +1572,7 @@ def test_overview_exposes_only_verified_scheduled_runs_as_cancellable(
|
||||
"schema_version": 1,
|
||||
"updated_at": "2026-08-14T02:00:00+00:00",
|
||||
"jobs": {
|
||||
"content.relogin.weekly": {
|
||||
"content.notes_master.daily": {
|
||||
"status": "running",
|
||||
"pid": 4310,
|
||||
"business_date": "2026-08-14",
|
||||
@@ -1589,7 +1595,7 @@ def test_overview_exposes_only_verified_scheduled_runs_as_cancellable(
|
||||
workflow_id: str,
|
||||
_job: dict[str, Any],
|
||||
) -> tuple[bool, Any, float | None]:
|
||||
if workflow_id == "content.relogin.weekly":
|
||||
if workflow_id == "content.notes_master.daily":
|
||||
return True, VerifiedProcess(), 1_800_000_000.0
|
||||
return False, None, None
|
||||
|
||||
@@ -1600,7 +1606,7 @@ def test_overview_exposes_only_verified_scheduled_runs_as_cancellable(
|
||||
).overview()
|
||||
workflows = {item["id"]: item for item in overview["workflows"]}
|
||||
|
||||
verified = workflows["content.relogin.weekly"]["scheduler_job"]
|
||||
verified = workflows["content.notes_master.daily"]["scheduler_job"]
|
||||
assert verified["status"] == "running"
|
||||
assert verified["cancellable"] is True
|
||||
assert "pid" not in verified
|
||||
@@ -1614,7 +1620,7 @@ def test_service_cancels_verified_scheduled_run_and_finalizes_journal(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
console_settings: Settings,
|
||||
) -> None:
|
||||
workflow_id = "content.relogin.weekly"
|
||||
workflow_id = "content.notes_master.daily"
|
||||
_write_scheduler_state(
|
||||
console_settings,
|
||||
{
|
||||
@@ -1906,24 +1912,13 @@ def test_content_production_preflight_rejects_non_production_modes(
|
||||
relogin = next(
|
||||
item
|
||||
for item in catalog.workflows
|
||||
if item.workflow_id == "content.relogin.weekly"
|
||||
if item.workflow_id == "content.summary.weekly"
|
||||
)
|
||||
|
||||
with pytest.raises(ConsoleRequestError, match=message):
|
||||
console_module._validate_content_production_runtime(relogin, {flag: "1"})
|
||||
|
||||
|
||||
def test_content_relogin_preflight_does_not_require_postgres() -> None:
|
||||
catalog = WorkflowCatalog.load(PROJECT_ROOT / "config")
|
||||
relogin = next(
|
||||
item
|
||||
for item in catalog.workflows
|
||||
if item.workflow_id == "content.relogin.weekly"
|
||||
)
|
||||
|
||||
console_module._validate_content_production_runtime(relogin, {})
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"workflow_id",
|
||||
["content.notes_master.daily", "content.metrics.daily"],
|
||||
@@ -3297,7 +3292,7 @@ def test_http_notification_config_requires_revision_and_supports_multi_recipient
|
||||
assert status == HTTPStatus.OK
|
||||
_assert_security_headers(response_headers)
|
||||
config = json.loads(body)
|
||||
assert len(config["people"]) == 16
|
||||
assert len(config["people"]) == 18
|
||||
assert {
|
||||
route["workflow_id"] for route in config["routes"]
|
||||
} == set(console_module.NOTIFICATION_CAPABILITIES)
|
||||
@@ -3616,3 +3611,82 @@ def test_douyin_negative_reviews_range_uses_single_run_with_env_dates(
|
||||
{"business_date": "2026-09-19"},
|
||||
)
|
||||
assert "business_date_end" not in single
|
||||
|
||||
|
||||
def test_api_health_reports_process_and_scheduler_state(
|
||||
console_settings: Settings,
|
||||
) -> None:
|
||||
with _running_server(console_settings, FakeConsoleLauncher()) as server:
|
||||
status, _headers, content = _http_request(server, "GET", "/api/health")
|
||||
assert status == HTTPStatus.OK
|
||||
payload = json.loads(content)
|
||||
assert payload["ok"] is True
|
||||
assert payload["pid"] == os.getpid()
|
||||
assert payload["uptime_seconds"] >= 0
|
||||
# 测试环境没有调度器实例锁,health 只报状态不做任何拉起。
|
||||
assert payload["scheduler_running"] is False
|
||||
assert payload["scheduler_pid"] is None
|
||||
|
||||
|
||||
def test_api_health_requires_no_token(console_settings: Settings) -> None:
|
||||
# 自检探针端点供进程内看门狗与保活脚本使用,必须绕过访问令牌。
|
||||
with _running_server(
|
||||
console_settings, FakeConsoleLauncher(), token="x" * 24
|
||||
) as server:
|
||||
status, _headers, _content = _http_request(server, "GET", "/api/health")
|
||||
assert status == HTTPStatus.OK
|
||||
|
||||
|
||||
def test_probe_console_health_against_real_server(
|
||||
console_settings: Settings,
|
||||
) -> None:
|
||||
with _running_server(console_settings, FakeConsoleLauncher()) as server:
|
||||
assert console_module._probe_console_health(server.server_port, 2.0) is True
|
||||
assert console_module._probe_console_health(1, 0.2) is False
|
||||
|
||||
|
||||
def test_console_watchdog_exits_after_consecutive_probe_failures() -> None:
|
||||
stopped = threading.Event()
|
||||
exits: list[int] = []
|
||||
probes: list[int] = []
|
||||
|
||||
def always_failing(port: int, _timeout: float) -> bool:
|
||||
probes.append(port)
|
||||
return False
|
||||
|
||||
console_module._console_watchdog(
|
||||
8765,
|
||||
interval_seconds=0.01,
|
||||
failure_limit=3,
|
||||
probe_timeout_seconds=0.01,
|
||||
stopped=stopped,
|
||||
probe=always_failing,
|
||||
exit_process=exits.append,
|
||||
sleep=lambda _seconds: None,
|
||||
)
|
||||
|
||||
assert exits == [console_module.CONSOLE_WATCHDOG_EXIT_CODE]
|
||||
assert probes == [8765, 8765, 8765]
|
||||
|
||||
|
||||
def test_console_watchdog_resets_failure_count_after_successful_probe() -> None:
|
||||
stopped = threading.Event()
|
||||
exits: list[int] = []
|
||||
results = iter([False, False, True, False, False, False])
|
||||
|
||||
def flaky_probe(_port: int, _timeout: float) -> bool:
|
||||
return next(results)
|
||||
|
||||
console_module._console_watchdog(
|
||||
8765,
|
||||
interval_seconds=0.01,
|
||||
failure_limit=3,
|
||||
probe_timeout_seconds=0.01,
|
||||
stopped=stopped,
|
||||
probe=flaky_probe,
|
||||
exit_process=exits.append,
|
||||
sleep=lambda _seconds: None,
|
||||
)
|
||||
|
||||
# 成功一次清零计数:2 次失败 + 1 次成功 + 3 次失败后才退出。
|
||||
assert exits == [console_module.CONSOLE_WATCHDOG_EXIT_CODE]
|
||||
@@ -23,7 +23,7 @@ def test_content_module_registers_only_scheduled_entries() -> None:
|
||||
|
||||
definitions = module.workflow_definitions()
|
||||
assert tuple(item.workflow_id for item in definitions) == CONTENT_WORKFLOW_IDS
|
||||
assert len(CONTENT_SCHEDULED_WORKFLOW_IDS) == 10
|
||||
assert len(CONTENT_SCHEDULED_WORKFLOW_IDS) == 9
|
||||
|
||||
|
||||
def test_content_workflows_use_project_owned_module_commands() -> None:
|
||||
|
||||
@@ -44,7 +44,6 @@ REQUIRED_SOURCES = {
|
||||
"chanmama_scraper.py",
|
||||
"daily_creator_exposure_scope.py",
|
||||
"daily_marketing_report.py",
|
||||
"data/tools/friday_relogin_parallel.py",
|
||||
"data/tools/schema_gyxx_super_data.sql",
|
||||
"data/tools/migrations/005_style_product_profile.sql",
|
||||
"data/config/db.env.example",
|
||||
@@ -71,7 +70,7 @@ def test_content_source_snapshot_manifest_is_complete_and_well_formed() -> None:
|
||||
assert manifest["module"] == "content_marketing"
|
||||
files = manifest["files"]
|
||||
assert isinstance(files, list)
|
||||
assert len(files) == 66
|
||||
assert len(files) == 59
|
||||
|
||||
sources = [entry["source_relative_path"] for entry in files]
|
||||
targets = [entry["target_relative_path"] for entry in files]
|
||||
|
||||
@@ -0,0 +1,771 @@
|
||||
"""Cookie gate: per-account login-state machine injected before workflows."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
|
||||
from gyxx_flow.adapters.browser import BrowserCookieStore
|
||||
from gyxx_flow.adapters.integration import AccountBinding
|
||||
from gyxx_flow.catalog import WorkflowCatalog, WorkflowEntry
|
||||
from gyxx_flow.cookie_gate import (
|
||||
GATE_MODE_BLOCKING,
|
||||
GATE_MODE_DEFER,
|
||||
GATE_MODE_RECOVER,
|
||||
CookieGate,
|
||||
CookieGateRecoveryManager,
|
||||
apply_cookie_gates,
|
||||
gate_enabled,
|
||||
)
|
||||
from gyxx_flow.core.config import Settings
|
||||
from gyxx_flow.core.context import RunContext
|
||||
from gyxx_flow.core.layout import DataLayout
|
||||
from gyxx_flow.core.locks import LockManager
|
||||
from gyxx_flow.core.records import RunJournal
|
||||
from gyxx_flow.workflow.engine import WorkflowEngine
|
||||
from gyxx_flow.workflow.model import StepDefinition, WorkflowDefinition
|
||||
from gyxx_flow.workflow.steps import StepExecution
|
||||
|
||||
_NOW = datetime.now(timezone.utc).replace(microsecond=0)
|
||||
|
||||
|
||||
def _settings(tmp_path: Path) -> Settings:
|
||||
return Settings(project_root=tmp_path, data_root=tmp_path / "var")
|
||||
|
||||
|
||||
def _account(tmp_path: Path, account_id: str, **overrides: object) -> AccountBinding:
|
||||
vault = tmp_path / "state" / "accounts" / account_id
|
||||
values: dict[str, object] = {
|
||||
"account_id": account_id,
|
||||
"cdp_port": 22000,
|
||||
"cdp_url": "http://127.0.0.1:22000",
|
||||
"profile_dir": vault / "profile",
|
||||
"cookie_file": vault / "cookies.json",
|
||||
"storage_state_file": vault / "storage_state.json",
|
||||
"required_cookie_domains": ("example.com",),
|
||||
"required_cookie_names": ("sessionid",),
|
||||
"login_identity_markers": ("示例店铺",),
|
||||
"keepalive_enabled": True,
|
||||
"keepalive_url": "https://example.com/home",
|
||||
"keepalive_interval_seconds": 900,
|
||||
}
|
||||
values.update(overrides)
|
||||
return AccountBinding(**values) # type: ignore[arg-type]
|
||||
|
||||
|
||||
def _fake_catalog(*accounts: AccountBinding, bindings: dict[str, str] | None = None):
|
||||
by_id = {account.account_id: account for account in accounts}
|
||||
binding_accounts = bindings or {}
|
||||
account_tuple = tuple(accounts)
|
||||
|
||||
class _Catalog:
|
||||
accounts = account_tuple
|
||||
|
||||
def account_for(self, account_id: str) -> AccountBinding:
|
||||
return by_id[account_id]
|
||||
|
||||
def binding_for(self, binding_id: str):
|
||||
return SimpleNamespace(account=binding_accounts.get(binding_id, ""))
|
||||
|
||||
return _Catalog()
|
||||
|
||||
|
||||
def _write_valid_vault(account: AccountBinding) -> None:
|
||||
store = BrowserCookieStore(account.cookie_file, account.storage_state_file)
|
||||
store.save_cookies(
|
||||
[
|
||||
{
|
||||
"name": "sessionid",
|
||||
"value": "live-session",
|
||||
"domain": ".example.com",
|
||||
"path": "/",
|
||||
"expires": _NOW.timestamp() + 86_400,
|
||||
}
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
def _write_keepalive_status(
|
||||
account: AccountBinding,
|
||||
payload: dict[str, object] | None,
|
||||
) -> None:
|
||||
path = account.cookie_file.with_name("keepalive_status.json")
|
||||
if payload is None:
|
||||
path.unlink(missing_ok=True)
|
||||
return
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(json.dumps(payload), encoding="utf-8")
|
||||
|
||||
|
||||
def _gate(
|
||||
tmp_path: Path,
|
||||
*accounts: AccountBinding,
|
||||
probe: dict[str, object] | None = None,
|
||||
environment: dict[str, str] | None = None,
|
||||
sent: list[dict[str, object]] | None = None,
|
||||
bindings: dict[str, str] | None = None,
|
||||
) -> CookieGate:
|
||||
probes = probe or {"status": "expired"}
|
||||
keepalive = SimpleNamespace(
|
||||
refresh_account=lambda account_id: dict(probes, account_id=account_id)
|
||||
)
|
||||
sender_calls = sent if sent is not None else []
|
||||
|
||||
def _sender(**kwargs: object) -> dict[str, object]:
|
||||
sender_calls.append(dict(kwargs))
|
||||
return {"message_id": "om_fake"}
|
||||
|
||||
catalog = _fake_catalog(*accounts, bindings=bindings)
|
||||
gate = CookieGate(
|
||||
_settings(tmp_path),
|
||||
catalog=catalog,
|
||||
keepalive=keepalive, # type: ignore[arg-type]
|
||||
environment=environment or {},
|
||||
now=lambda: _NOW,
|
||||
sender=_sender,
|
||||
)
|
||||
gate._workflow_catalog = WorkflowCatalog(timezone="UTC", workflows=(), schedules=())
|
||||
return gate
|
||||
|
||||
|
||||
def _context(**overrides: object) -> RunContext:
|
||||
values: dict[str, object] = {
|
||||
"workflow_id": "shop.demo.daily",
|
||||
"business_date": "2026-10-08",
|
||||
"shadow": False,
|
||||
"now": _NOW,
|
||||
"random_suffix": "aaaa",
|
||||
}
|
||||
values.update(overrides)
|
||||
return RunContext.create(**values) # type: ignore[arg-type]
|
||||
|
||||
|
||||
def _noop_outcome(*, context, timeout_seconds, dry_run) -> StepExecution: # noqa: ANN001
|
||||
return StepExecution(exit_code=0)
|
||||
|
||||
|
||||
def _deny_outcome(*, context, timeout_seconds, dry_run) -> StepExecution: # noqa: ANN001
|
||||
raise AssertionError("resolution must not run actions")
|
||||
|
||||
|
||||
class _PassAction:
|
||||
execute = staticmethod(_noop_outcome)
|
||||
|
||||
|
||||
class _BoundAction:
|
||||
"""Duck-typed DeferredModuleCommandStep for account resolution tests."""
|
||||
|
||||
def __init__(self, command_entry: str = "collect.py") -> None:
|
||||
self.entry = SimpleNamespace(module="shop_intelligence", entry="fallback.py")
|
||||
self.command_entry = command_entry
|
||||
self.execute = _deny_outcome
|
||||
|
||||
|
||||
class _RecordingAction:
|
||||
def __init__(self, name: str, sink: list[str], command_entry: str | None = None) -> None:
|
||||
def run(*, context, timeout_seconds, dry_run) -> StepExecution: # noqa: ANN001
|
||||
sink.append(name)
|
||||
return StepExecution(exit_code=0)
|
||||
|
||||
self.execute = run
|
||||
if command_entry is not None:
|
||||
self.entry = SimpleNamespace(
|
||||
module="shop_intelligence", entry="fallback.py"
|
||||
)
|
||||
self.command_entry = command_entry
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
# Mode classification
|
||||
# ----------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_gate_mode_classifies_accounts(tmp_path: Path) -> None:
|
||||
gate = _gate(
|
||||
tmp_path,
|
||||
_account(tmp_path, "qr-only"),
|
||||
_account(tmp_path, "cred-scripts", credential_env_names=("A_USER", "A_PASS")),
|
||||
_account(
|
||||
tmp_path,
|
||||
"cred-vault",
|
||||
login_mode="C",
|
||||
login_command_id="product_commerce:erp_login.py",
|
||||
),
|
||||
_account(tmp_path, "cred-mode-b", login_mode="B"),
|
||||
)
|
||||
assert gate.gate_mode("qr-only") == GATE_MODE_BLOCKING
|
||||
assert gate.gate_mode("cred-scripts") == GATE_MODE_DEFER
|
||||
assert gate.gate_mode("cred-vault") == GATE_MODE_RECOVER
|
||||
assert gate.gate_mode("cred-mode-b") == GATE_MODE_DEFER
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
# Workflow account resolution
|
||||
# ----------------------------------------------------------------------
|
||||
|
||||
|
||||
def _bound_workflow(workflow_id: str = "shop.demo.daily") -> WorkflowDefinition:
|
||||
return WorkflowDefinition(
|
||||
workflow_id,
|
||||
(
|
||||
StepDefinition("collect", _BoundAction("collect.py")),
|
||||
StepDefinition("sync", _BoundAction("sync.py"), depends_on=("collect",)),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def test_collect_accounts_maps_bindings_and_skips_defer(tmp_path: Path) -> None:
|
||||
gate = _gate(
|
||||
tmp_path,
|
||||
_account(tmp_path, "douyin-shop"),
|
||||
_account(tmp_path, "jd-shop", credential_env_names=("JD_SHOP", "JD_PASSWORD")),
|
||||
bindings={
|
||||
"shop_intelligence:collect.py": "douyin-shop",
|
||||
"shop_intelligence:other.py": "jd-shop",
|
||||
},
|
||||
)
|
||||
mapping = gate.collect_workflow_accounts(_bound_workflow())
|
||||
# sync.py 无账号绑定,jd-shop 属 defer 模式:两者都不进映射
|
||||
assert mapping == {"douyin-shop": {"collect"}}
|
||||
|
||||
|
||||
def test_collect_accounts_includes_declared_required_accounts(tmp_path: Path) -> None:
|
||||
gate = _gate(tmp_path, _account(tmp_path, "douyin-shop"))
|
||||
gate._workflow_catalog = WorkflowCatalog(
|
||||
timezone="UTC",
|
||||
workflows=(
|
||||
WorkflowEntry(
|
||||
workflow_id="shop.demo.daily",
|
||||
module="shop_intelligence",
|
||||
trigger="manual",
|
||||
entry="collect.py",
|
||||
required_accounts=("douyin-shop",),
|
||||
),
|
||||
),
|
||||
schedules=(),
|
||||
)
|
||||
mapping = gate.collect_workflow_accounts(_bound_workflow())
|
||||
assert mapping == {"douyin-shop": {"collect", "sync"}}
|
||||
|
||||
|
||||
def test_apply_cookie_gates_rewrites_dependencies(tmp_path: Path) -> None:
|
||||
account = _account(tmp_path, "douyin-shop")
|
||||
gate = _gate(
|
||||
tmp_path,
|
||||
account,
|
||||
bindings={"shop_intelligence:collect.py": "douyin-shop"},
|
||||
)
|
||||
gated = apply_cookie_gates(
|
||||
_bound_workflow(), settings=_settings(tmp_path), gate=gate
|
||||
)
|
||||
ids = [step.step_id for step in gated.steps]
|
||||
assert "cookie-gate.douyin-shop" in ids
|
||||
collect = next(step for step in gated.steps if step.step_id == "collect")
|
||||
sync = next(step for step in gated.steps if step.step_id == "sync")
|
||||
assert collect.depends_on == ("cookie-gate.douyin-shop",)
|
||||
assert sync.depends_on == ("collect",)
|
||||
assert gated.ordered_steps() # patched DAG stays valid
|
||||
|
||||
|
||||
def test_apply_cookie_gates_returns_workflow_without_accounts(tmp_path: Path) -> None:
|
||||
gate = _gate(tmp_path, _account(tmp_path, "douyin-shop"))
|
||||
workflow = _bound_workflow()
|
||||
assert (
|
||||
apply_cookie_gates(workflow, settings=_settings(tmp_path), gate=gate) is workflow
|
||||
)
|
||||
|
||||
|
||||
def test_gate_enabled_env_kill_switch() -> None:
|
||||
assert gate_enabled({})
|
||||
assert not gate_enabled({"GYXX_COOKIE_GATE": "0"})
|
||||
assert not gate_enabled({"GYXX_COOKIE_GATE": "off"})
|
||||
assert gate_enabled({"GYXX_COOKIE_GATE": "1"})
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
# State machine: check()
|
||||
# ----------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_check_fast_path_accepts_fresh_keepalive_without_probe(tmp_path: Path) -> None:
|
||||
account = _account(tmp_path, "douyin-shop")
|
||||
gate = _gate(tmp_path, account, probe={"status": "expired"})
|
||||
_write_valid_vault(account)
|
||||
_write_keepalive_status(
|
||||
account,
|
||||
{
|
||||
"status": "success",
|
||||
"last_success_at": (_NOW - timedelta(minutes=10)).isoformat(),
|
||||
},
|
||||
)
|
||||
verdict = gate.check("douyin-shop", context=_context())
|
||||
assert not verdict.blocked
|
||||
assert "keepalive" in verdict.message
|
||||
state = json.loads(
|
||||
(account.cookie_file.parent / "cookie_state.json").read_text(encoding="utf-8")
|
||||
)
|
||||
assert state["state"] == "healthy"
|
||||
|
||||
|
||||
def test_check_stale_status_triggers_probe(tmp_path: Path) -> None:
|
||||
account = _account(tmp_path, "douyin-shop")
|
||||
gate = _gate(tmp_path, account, probe={"status": "success"})
|
||||
_write_valid_vault(account)
|
||||
_write_keepalive_status(
|
||||
account,
|
||||
{
|
||||
"status": "success",
|
||||
"last_success_at": (_NOW - timedelta(hours=6)).isoformat(),
|
||||
},
|
||||
)
|
||||
verdict = gate.check("douyin-shop", context=_context())
|
||||
assert not verdict.blocked
|
||||
assert verdict.message.endswith("探活通过)")
|
||||
|
||||
|
||||
def test_check_expired_blocks_and_sends_feishu_alert(tmp_path: Path) -> None:
|
||||
account = _account(tmp_path, "douyin-shop")
|
||||
sent: list[dict[str, object]] = []
|
||||
gate = _gate(
|
||||
tmp_path,
|
||||
account,
|
||||
probe={"status": "expired"},
|
||||
environment={"GYXX_LOGIN_ALERT_OPEN_IDS": "ou_alert_target"},
|
||||
sent=sent,
|
||||
)
|
||||
_write_keepalive_status(account, None)
|
||||
verdict = gate.check("douyin-shop", context=_context())
|
||||
assert verdict.blocked
|
||||
assert verdict.notified
|
||||
assert len(sent) == 1
|
||||
text = str(sent[0]["text"])
|
||||
assert "uv run gyxx accounts login douyin-shop" in text
|
||||
assert "shop.demo.daily" in text
|
||||
assert sent[0]["profile"] == "hermes-analyzer"
|
||||
state = json.loads(
|
||||
(account.cookie_file.parent / "cookie_state.json").read_text(encoding="utf-8")
|
||||
)
|
||||
assert state["state"] == "awaiting_login"
|
||||
assert state["last_notified_at"] == _NOW.isoformat(timespec="seconds")
|
||||
|
||||
|
||||
def test_check_notification_is_throttled_per_interval(tmp_path: Path) -> None:
|
||||
account = _account(tmp_path, "douyin-shop")
|
||||
sent: list[dict[str, object]] = []
|
||||
gate = _gate(
|
||||
tmp_path,
|
||||
account,
|
||||
probe={"status": "expired"},
|
||||
environment={"GYXX_LOGIN_ALERT_OPEN_IDS": "ou_alert_target"},
|
||||
sent=sent,
|
||||
)
|
||||
_write_keepalive_status(account, None)
|
||||
first = gate.check("douyin-shop", context=_context())
|
||||
second = gate.check("douyin-shop", context=_context())
|
||||
assert first.notified and second.blocked
|
||||
assert not second.notified
|
||||
assert len(sent) == 1
|
||||
|
||||
|
||||
def test_check_shadow_run_never_notifies(tmp_path: Path) -> None:
|
||||
account = _account(tmp_path, "douyin-shop")
|
||||
sent: list[dict[str, object]] = []
|
||||
gate = _gate(
|
||||
tmp_path,
|
||||
account,
|
||||
probe={"status": "expired"},
|
||||
environment={"GYXX_LOGIN_ALERT_OPEN_IDS": "ou_alert_target"},
|
||||
sent=sent,
|
||||
)
|
||||
verdict = gate.check("douyin-shop", context=_context(shadow=True))
|
||||
assert verdict.blocked
|
||||
assert not verdict.notified
|
||||
assert sent == []
|
||||
|
||||
|
||||
def test_check_mode_c_recovery_success_and_failure_paths(tmp_path: Path) -> None:
|
||||
account = _account(
|
||||
tmp_path,
|
||||
"erp-jushuitan",
|
||||
login_mode="C",
|
||||
login_command_id="product_commerce:erp_login.py",
|
||||
)
|
||||
recovered = _gate(
|
||||
tmp_path, account, probe={"status": "success", "login_recovered": True}
|
||||
)
|
||||
_write_keepalive_status(account, None)
|
||||
verdict = recovered.check("erp-jushuitan", context=_context())
|
||||
assert not verdict.blocked
|
||||
assert verdict.recovered
|
||||
assert "自动账密重登成功" in verdict.message
|
||||
|
||||
sent: list[dict[str, object]] = []
|
||||
failed = _gate(
|
||||
tmp_path,
|
||||
account,
|
||||
probe={"status": "not_logged_in"},
|
||||
environment={"GYXX_LOGIN_ALERT_OPEN_IDS": "ou_alert_target"},
|
||||
sent=sent,
|
||||
)
|
||||
_write_keepalive_status(
|
||||
failed.catalog.account_for("erp-jushuitan"),
|
||||
{
|
||||
"status": "not_logged_in",
|
||||
"login_attempted": True,
|
||||
"login_error_type": "TimeoutExpired",
|
||||
},
|
||||
)
|
||||
verdict = failed.check("erp-jushuitan", context=_context())
|
||||
assert verdict.blocked
|
||||
assert verdict.notified
|
||||
assert "自动账密重登已尝试但未成功" in str(sent[0]["text"])
|
||||
|
||||
|
||||
def test_check_busy_falls_back_to_static_evidence(tmp_path: Path) -> None:
|
||||
account = _account(tmp_path, "douyin-shop")
|
||||
gate = _gate(tmp_path, account, probe={"status": "busy"})
|
||||
_write_valid_vault(account)
|
||||
_write_keepalive_status(account, None)
|
||||
verdict = gate.check("douyin-shop", context=_context())
|
||||
assert not verdict.blocked
|
||||
assert "探活锁忙" in verdict.message
|
||||
|
||||
|
||||
def test_check_probe_error_with_static_valid_vault_passes(tmp_path: Path) -> None:
|
||||
account = _account(tmp_path, "douyin-shop")
|
||||
gate = _gate(tmp_path, account, probe={"status": "timeout"})
|
||||
_write_valid_vault(account)
|
||||
_write_keepalive_status(account, None)
|
||||
verdict = gate.check("douyin-shop", context=_context())
|
||||
assert not verdict.blocked
|
||||
assert "探活异常" in verdict.message
|
||||
|
||||
|
||||
def test_acceptance_mode_is_static_only_and_never_alerts(tmp_path: Path) -> None:
|
||||
account = _account(tmp_path, "douyin-shop")
|
||||
sent: list[dict[str, object]] = []
|
||||
gate = _gate(
|
||||
tmp_path,
|
||||
account,
|
||||
probe={"status": "expired"},
|
||||
environment={
|
||||
"GYXX_WORKFLOW_ACCEPTANCE": "1",
|
||||
"GYXX_LOGIN_ALERT_OPEN_IDS": "ou_alert_target",
|
||||
},
|
||||
sent=sent,
|
||||
)
|
||||
_write_keepalive_status(account, None)
|
||||
verdict = gate.check("douyin-shop", context=_context())
|
||||
assert verdict.blocked
|
||||
assert sent == []
|
||||
assert "验收模式" in verdict.message
|
||||
|
||||
|
||||
def test_alert_recipient_falls_back_to_constant(tmp_path: Path) -> None:
|
||||
account = _account(tmp_path, "douyin-shop")
|
||||
gate = _gate(tmp_path, account)
|
||||
recipients = gate._alert_recipients()
|
||||
assert recipients == [
|
||||
("ou_7ad5fc8012e2f741afc5346e05ffd447", "hermes-analyzer")
|
||||
]
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
# Engine integration
|
||||
# ----------------------------------------------------------------------
|
||||
|
||||
|
||||
def _engine(tmp_path: Path) -> WorkflowEngine:
|
||||
return WorkflowEngine(LockManager(tmp_path / "state" / "locks"))
|
||||
|
||||
|
||||
def test_engine_blocks_dependents_when_gate_fails(tmp_path: Path) -> None:
|
||||
account = _account(tmp_path, "douyin-shop")
|
||||
sent: list[dict[str, object]] = []
|
||||
gate = _gate(
|
||||
tmp_path,
|
||||
account,
|
||||
probe={"status": "expired"},
|
||||
environment={"GYXX_LOGIN_ALERT_OPEN_IDS": "ou_alert_target"},
|
||||
sent=sent,
|
||||
bindings={"shop_intelligence:collect.py": "douyin-shop"},
|
||||
)
|
||||
executed: list[str] = []
|
||||
workflow = WorkflowDefinition(
|
||||
"shop.demo.daily",
|
||||
(
|
||||
StepDefinition(
|
||||
"collect", _RecordingAction("collect", executed, "collect.py")
|
||||
),
|
||||
StepDefinition(
|
||||
"sync", _RecordingAction("sync", executed), depends_on=("collect",)
|
||||
),
|
||||
),
|
||||
)
|
||||
gated = apply_cookie_gates(workflow, settings=_settings(tmp_path), gate=gate)
|
||||
context = RunContext.create(
|
||||
"shop.demo.daily", "2026-10-08", now=_NOW, random_suffix="dddd"
|
||||
)
|
||||
journal = RunJournal.create(DataLayout(_settings(tmp_path).data_root), context)
|
||||
engine = _engine(tmp_path)
|
||||
run_workflow = engine.execute
|
||||
result = run_workflow(gated, context=context, journal=journal)
|
||||
|
||||
assert result.status == "failed"
|
||||
assert result.exit_code == 1
|
||||
assert result.steps["cookie-gate.douyin-shop"].status == "failed"
|
||||
assert result.steps["collect"].reason == "dependency-failed"
|
||||
assert result.steps["sync"].reason == "dependency-failed"
|
||||
assert executed == []
|
||||
assert len(sent) == 1
|
||||
|
||||
|
||||
def test_engine_runs_collect_steps_when_gate_passes(tmp_path: Path) -> None:
|
||||
account = _account(tmp_path, "douyin-shop")
|
||||
gate = _gate(
|
||||
tmp_path,
|
||||
account,
|
||||
probe={"status": "success"},
|
||||
bindings={"shop_intelligence:collect.py": "douyin-shop"},
|
||||
)
|
||||
_write_valid_vault(account)
|
||||
_write_keepalive_status(account, None)
|
||||
executed: list[str] = []
|
||||
workflow = WorkflowDefinition(
|
||||
"shop.demo.daily",
|
||||
(
|
||||
StepDefinition(
|
||||
"collect", _RecordingAction("collect", executed, "collect.py")
|
||||
),
|
||||
),
|
||||
)
|
||||
gated = apply_cookie_gates(workflow, settings=_settings(tmp_path), gate=gate)
|
||||
context = RunContext.create(
|
||||
"shop.demo.daily", "2026-10-08", now=_NOW, random_suffix="eeee"
|
||||
)
|
||||
journal = RunJournal.create(DataLayout(_settings(tmp_path).data_root), context)
|
||||
engine = _engine(tmp_path)
|
||||
run_workflow = engine.execute
|
||||
result = run_workflow(gated, context=context, journal=journal)
|
||||
|
||||
assert result.status == "success"
|
||||
assert result.steps["cookie-gate.douyin-shop"].status == "success"
|
||||
assert result.steps["collect"].status == "success"
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
# Recovery: automatic rerun after the login recovers
|
||||
# ----------------------------------------------------------------------
|
||||
|
||||
|
||||
class _FakeLauncher:
|
||||
def __init__(self, fail: bool = False) -> None:
|
||||
self.launched: list[tuple[str, str]] = []
|
||||
self.closed: list[int] = []
|
||||
self.fail = fail
|
||||
|
||||
def launch(self, workflow_id, slot, business_date): # noqa: ANN001
|
||||
if self.fail:
|
||||
raise RuntimeError("spawn denied")
|
||||
process = SimpleNamespace(pid=9000 + len(self.launched), poll=lambda: None)
|
||||
self.launched.append((workflow_id, business_date.isoformat()))
|
||||
return process
|
||||
|
||||
def close(self, process) -> None: # noqa: ANN001
|
||||
self.closed.append(process.pid)
|
||||
|
||||
|
||||
def _state_file(account: AccountBinding) -> Path:
|
||||
return account.cookie_file.parent / "cookie_state.json"
|
||||
|
||||
|
||||
def _read_state(account: AccountBinding) -> dict:
|
||||
return json.loads(_state_file(account).read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
def _recovery_workflow_catalog() -> WorkflowCatalog:
|
||||
return WorkflowCatalog(
|
||||
timezone="UTC",
|
||||
workflows=(
|
||||
WorkflowEntry(
|
||||
workflow_id="shop.demo.daily",
|
||||
module="shop_intelligence",
|
||||
trigger="scheduled",
|
||||
entry="collect.py",
|
||||
),
|
||||
),
|
||||
schedules=(),
|
||||
)
|
||||
|
||||
|
||||
def _blocking_gate(
|
||||
tmp_path: Path,
|
||||
account: AccountBinding,
|
||||
sent: list[dict[str, object]],
|
||||
) -> CookieGate:
|
||||
gate = _gate(
|
||||
tmp_path,
|
||||
account,
|
||||
probe={"status": "expired"},
|
||||
environment={"GYXX_LOGIN_ALERT_OPEN_IDS": "ou_alert_target"},
|
||||
sent=sent,
|
||||
)
|
||||
return gate
|
||||
|
||||
|
||||
def _healthy_gate(
|
||||
tmp_path: Path,
|
||||
account: AccountBinding,
|
||||
sent: list[dict[str, object]],
|
||||
) -> CookieGate:
|
||||
gate = _gate(
|
||||
tmp_path,
|
||||
account,
|
||||
probe={"status": "success"},
|
||||
environment={"GYXX_LOGIN_ALERT_OPEN_IDS": "ou_alert_target"},
|
||||
sent=sent,
|
||||
)
|
||||
gate._workflow_catalog = _recovery_workflow_catalog()
|
||||
return gate
|
||||
|
||||
|
||||
def _simulate_manual_login(account: AccountBinding) -> None:
|
||||
_write_valid_vault(account)
|
||||
_write_keepalive_status(
|
||||
account,
|
||||
{
|
||||
"status": "success",
|
||||
"last_success_at": (_NOW - timedelta(seconds=60)).isoformat(),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def test_expire_records_pending_reruns(tmp_path: Path) -> None:
|
||||
sent: list[dict[str, object]] = []
|
||||
account = _account(tmp_path, "douyin-shop")
|
||||
gate = _blocking_gate(tmp_path, account, sent)
|
||||
gate.check("douyin-shop", context=_context())
|
||||
gate.check("douyin-shop", context=_context()) # 同工作流同日期去重
|
||||
gate.check("douyin-shop", context=_context(business_date="2026-10-09"))
|
||||
state = _read_state(account)
|
||||
assert [
|
||||
(item["workflow_id"], item["business_date"])
|
||||
for item in state["pending_reruns"]
|
||||
] == [
|
||||
("shop.demo.daily", "2026-10-08"),
|
||||
("shop.demo.daily", "2026-10-09"),
|
||||
]
|
||||
|
||||
# shadow 运行的阻断不进入待重跑清单
|
||||
gate.check(
|
||||
"douyin-shop", context=_context(business_date="2026-10-10", shadow=True)
|
||||
)
|
||||
state = _read_state(account)
|
||||
assert all(
|
||||
item["business_date"] != "2026-10-10" for item in state["pending_reruns"]
|
||||
)
|
||||
|
||||
|
||||
def test_recovery_reruns_blocked_workflows_after_login(tmp_path: Path) -> None:
|
||||
sent: list[dict[str, object]] = []
|
||||
account = _account(tmp_path, "douyin-shop")
|
||||
blocked = _blocking_gate(tmp_path, account, sent)
|
||||
blocked.check("douyin-shop", context=_context())
|
||||
|
||||
_simulate_manual_login(account)
|
||||
healthy = _healthy_gate(tmp_path, account, sent)
|
||||
launcher = _FakeLauncher()
|
||||
outcomes = CookieGateRecoveryManager(healthy, launcher=launcher).tick()
|
||||
|
||||
assert launcher.launched == [("shop.demo.daily", "2026-10-08")]
|
||||
assert launcher.closed # 父进程日志句柄立即关闭
|
||||
assert outcomes and outcomes[0]["launched"]
|
||||
state = _read_state(account)
|
||||
assert state["pending_reruns"] == []
|
||||
assert state["state"] == "healthy"
|
||||
assert state["last_rerun_launch_at"]
|
||||
assert state["rerun_history"][0]["workflow_id"] == "shop.demo.daily"
|
||||
assert any("登录态已恢复" in str(item["text"]) for item in sent)
|
||||
|
||||
|
||||
def test_recovery_skips_while_login_still_expired(tmp_path: Path) -> None:
|
||||
sent: list[dict[str, object]] = []
|
||||
account = _account(tmp_path, "douyin-shop")
|
||||
blocked = _blocking_gate(tmp_path, account, sent)
|
||||
blocked.check("douyin-shop", context=_context())
|
||||
_write_keepalive_status(account, None)
|
||||
|
||||
still_expired = _gate(
|
||||
tmp_path,
|
||||
account,
|
||||
probe={"status": "expired"},
|
||||
environment={"GYXX_LOGIN_ALERT_OPEN_IDS": "ou_alert_target"},
|
||||
sent=sent,
|
||||
)
|
||||
still_expired._workflow_catalog = _recovery_workflow_catalog()
|
||||
launcher = _FakeLauncher()
|
||||
outcomes = CookieGateRecoveryManager(still_expired, launcher=launcher).tick()
|
||||
|
||||
assert launcher.launched == []
|
||||
assert outcomes == []
|
||||
state = _read_state(account)
|
||||
assert len(state["pending_reruns"]) == 1
|
||||
assert state["last_recovery_probe_at"]
|
||||
|
||||
|
||||
def test_recovery_drops_adhoc_script_ids(tmp_path: Path) -> None:
|
||||
sent: list[dict[str, object]] = []
|
||||
account = _account(tmp_path, "douyin-shop")
|
||||
blocked = _blocking_gate(tmp_path, account, sent)
|
||||
blocked.check(
|
||||
"douyin-shop", context=_context(workflow_id="script.shop.abcdef123456")
|
||||
)
|
||||
assert (
|
||||
_read_state(account)["pending_reruns"][0]["workflow_id"]
|
||||
== "script.shop.abcdef123456"
|
||||
)
|
||||
|
||||
_simulate_manual_login(account)
|
||||
healthy = _healthy_gate(tmp_path, account, sent)
|
||||
launcher = _FakeLauncher()
|
||||
CookieGateRecoveryManager(healthy, launcher=launcher).tick()
|
||||
|
||||
assert launcher.launched == []
|
||||
assert _read_state(account)["pending_reruns"] == []
|
||||
|
||||
|
||||
def test_recovery_launch_failure_keeps_pending_for_retry(tmp_path: Path) -> None:
|
||||
sent: list[dict[str, object]] = []
|
||||
account = _account(tmp_path, "douyin-shop")
|
||||
blocked = _blocking_gate(tmp_path, account, sent)
|
||||
blocked.check("douyin-shop", context=_context())
|
||||
_simulate_manual_login(account)
|
||||
|
||||
healthy = _healthy_gate(tmp_path, account, sent)
|
||||
launcher = _FakeLauncher(fail=True)
|
||||
CookieGateRecoveryManager(healthy, launcher=launcher).tick()
|
||||
|
||||
state = _read_state(account)
|
||||
assert len(state["pending_reruns"]) == 1
|
||||
assert "error" in state["pending_reruns"][0]
|
||||
assert state["state"] == "awaiting_login"
|
||||
assert not any("已恢复" in str(item["text"]) for item in sent)
|
||||
|
||||
|
||||
def test_recovery_respects_launch_cooldown(tmp_path: Path) -> None:
|
||||
sent: list[dict[str, object]] = []
|
||||
account = _account(tmp_path, "douyin-shop")
|
||||
blocked = _blocking_gate(tmp_path, account, sent)
|
||||
blocked.check("douyin-shop", context=_context())
|
||||
_simulate_manual_login(account)
|
||||
state = _read_state(account)
|
||||
state["last_rerun_launch_at"] = _NOW.isoformat(timespec="seconds")
|
||||
_state_file(account).write_text(json.dumps(state), encoding="utf-8")
|
||||
|
||||
healthy = _healthy_gate(tmp_path, account, sent)
|
||||
launcher = _FakeLauncher()
|
||||
CookieGateRecoveryManager(healthy, launcher=launcher).tick()
|
||||
|
||||
assert launcher.launched == []
|
||||
@@ -36,7 +36,6 @@ def test_repository_keeps_the_simplified_public_structure() -> None:
|
||||
"note-inventory.md",
|
||||
"plan.md",
|
||||
"runbook.md",
|
||||
"workbench.md",
|
||||
"workflow-acceptance-test-report.md",
|
||||
"workflow-acceptance-test-requirements.md",
|
||||
"workflow-console.md",
|
||||
|
||||
@@ -67,15 +67,21 @@ def test_baseline_contains_all_provided_people_and_routable_capabilities(
|
||||
) -> None:
|
||||
payload, revision = NotificationRoutingStore(routing_settings).snapshot()
|
||||
|
||||
assert len(payload["people"]) == 16
|
||||
assert len(payload["people"]) == 18
|
||||
assert payload["app_profile"] == "hermes-analyzer"
|
||||
assert (
|
||||
payload["app_profiles"]["hermes-analyzer"]["app_id"]
|
||||
== ANALYZER_NOTIFICATION_APP_ID
|
||||
)
|
||||
assert all(person["mobile"] is None for person in payload["people"].values())
|
||||
mobiles = {
|
||||
person_id: person["mobile"]
|
||||
for person_id, person in payload["people"].items()
|
||||
}
|
||||
assert mobiles["wang_yunlong"] == "+8613663785145"
|
||||
assert mobiles["stock_alert_user_19832771651"] == "+8619832771651"
|
||||
assert set(payload["capabilities"]) == set(NOTIFICATION_CAPABILITIES)
|
||||
assert payload["routes"] == {}
|
||||
assert set(payload["routes"]) == {"product.online_stock_alert.daily"}
|
||||
assert payload["routes"]["product.online_stock_alert.daily"]["enabled"] is True
|
||||
assert len(revision) == 64
|
||||
|
||||
wang = payload["people"]["wang_yunlong"]
|
||||
@@ -85,10 +91,15 @@ def test_baseline_contains_all_provided_people_and_routable_capabilities(
|
||||
assert wang["bindings"]["hermes-analyzer"]["open_id"] == (
|
||||
"ou_7ad5fc8012e2f741afc5346e05ffd447"
|
||||
)
|
||||
_mobile_lookup_people = {
|
||||
"wang_yunlong",
|
||||
"online_stock_alert_recipient",
|
||||
"stock_alert_user_19832771651",
|
||||
}
|
||||
provided = {
|
||||
person["name"]: binding["open_id"]
|
||||
for person_id, person in payload["people"].items()
|
||||
if person_id != "wang_yunlong"
|
||||
if person_id not in _mobile_lookup_people
|
||||
for profile, binding in person["bindings"].items()
|
||||
if profile == "hermes-analyzer"
|
||||
}
|
||||
@@ -112,7 +123,7 @@ def test_baseline_contains_all_provided_people_and_routable_capabilities(
|
||||
assert all(
|
||||
binding["verified"] is False and binding["source"] == "provided"
|
||||
for person_id, person in payload["people"].items()
|
||||
if person_id != "wang_yunlong"
|
||||
if person_id not in _mobile_lookup_people
|
||||
for profile, binding in person["bindings"].items()
|
||||
if profile == "hermes-analyzer"
|
||||
)
|
||||
|
||||
Loaded 100 of 127 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user