wangyunlong/gyxx-flow
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: complete production workflow migration

Browse Source
This commit is contained in:
wangyunlong
2026-08-06 14:29:57 +08:00
parent 7f215e79c4
commit 8df5266abb
448 changed files with 56937 additions and 14619 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitlab-ci.yml
+38
View File
@@ -0,0 +1,38 @@
image: ghcr.io/astral-sh/uv:python3.12-bookworm-slim
stages:
- quality
- test
- build
variables:
UV_CACHE_DIR: "$CI_PROJECT_DIR/.uv-cache"
cache:
paths:
- .uv-cache/
before_script:
- uv sync --frozen --group dev
lint:
stage: quality
script:
- uv run ruff check src tests
- uv run ruff check src/gyxx_flow/modules/content_marketing/daily_creator_exposure_scope.py src/gyxx_flow/modules/product_commerce/market_rank_limits.py src/gyxx_flow/modules/product_commerce/market_rank_product_import.py
test:
stage: test
script:
- uv run pytest
- uv run pytest tests/modules/content_marketing/test_daily_creator_exposure_scope.py tests/modules/content_marketing/test_chanmama_refresh_before_export.py tests/modules/content_marketing/test_daily_exposure_fields.py
- uv run pytest tests/modules/product_commerce/test_market_rank_limits.py tests/modules/product_commerce/test_market_rank_product_import.py
build:
stage: build
script:
- uv build
artifacts:
paths:
- dist/
expire_in: 7 days
AGENTS.md
+39
View File
@@ -0,0 +1,39 @@
# Repository Guidelines
## Project Structure & Module Organization
GYXX Flow is a Python 3.12 `src`-layout package. Framework code lives in `src/gyxx_flow/`; business implementations belong directly in `src/gyxx_flow/modules/<module>/`. The former `runtime/` directories are compatibility namespaces only and must not receive new production code.
All project and module tests live in `tests/`. Configuration lives in `config/`, runbooks in `docs/`, and service scripts in `deploy/`. Treat `var/` as generated state, not source; production deployments must set an external `GYXX_DATA_ROOT`.
## Build, Test, and Development Commands
Use `uv` from the repository root:
```powershell
uv sync --python 3.12 --group dev # create/update the development environment
uv run pytest # run the project test suite
uv run pytest tests/test_cli.py # run a focused test file
uv run ruff check src tests # lint imports and Python errors
uv build # create distribution artifacts
uv run gyxx doctor --json # validate local configuration
uv run gyxx list # inspect registered workflows
```
Workflow and command execution defaults to dry-run behavior. Add `--execute` only when real external effects are intended. Public manual commands must be declared in `config/commands.json`; never reintroduce recursive executable-script discovery.
Production schedules run only through the Python scheduler service; do not add Windows Task Scheduler registration. Manual commands remain dry-run by default, while `gyxx schedule run` executes enabled jobs unless `--dry-run` is supplied. PostgreSQL uses the cloud service selected through runtime-only credentials, both Hermes roles remain loopback-only, Feishu keeps the existing integration behavior, credentials must not have non-empty source defaults, and every browser entry must retain its unique CDP/Profile/Cookie/storage-state binding.
Each executable `workflow_id` compiles to a LangGraph `StateGraph`. Preserve the public workflow model, journal, lock, shadow, and effect-ledger contracts when changing graph execution. Check upstream drift with `uv run gyxx sources status`; automated source application is limited to untransformed, conflict-free entries and still requires `--execute`.
## Coding Style & Naming Conventions
Follow existing Python conventions: four-space indentation, type annotations, concise docstrings, and imports ordered as standard library, third party, then local modules. Ruff enforces `E4`, `E7`, `E9`, `F`, and `I` rules. Use `snake_case` for modules, functions, variables, and test files; `PascalCase` for classes; and uppercase names for constants. Keep cross-module contracts in shared framework packages rather than importing another business module’s internals.
## Testing Guidelines
Tests use pytest and follow `test_*.py` / `test_*` naming. Add focused unit tests for new logic and regression tests for fixed defects. Changes to workflows, schedules, manifests, runtime paths, or CLI behavior should include acceptance or boundary coverage. Avoid tests that require live credentials or mutate production services.
## Commit & Pull Request Guidelines
Use short Conventional Commit subjects such as `feat: consolidate legacy workflows`. Keep commits scoped. Pull requests should explain behavior, list verification, link issues, and call out configuration or migration effects. Never commit secrets, cookies, tokens, `.env` files, or generated `var/` contents.
README.md
+122 -65
View File
@@ -1,99 +1,156 @@
# GYXX Flow
GYXX Flow 是面向业务自动化的模块化工作流平台，统一管理数据采集、经营分析、供应链处理、定时调度和外部系统集成。
GYXX Flow 是一个 Python 3.12 业务自动化平台，用 LangGraph 编排内容营销、商品经营、
店铺分析和供应链工作流，并由项目内 Python 调度器统一定时运行。
- `content_marketing`：内容与营销采集
- `product_commerce`：商品、平台和经营分析
- `shop_intelligence`：店铺与竞店采集
- `supply_chain`：供应链采集与通知
项目采用模块化单体：公共调度、工作流、数据和外部系统契约集中维护，业务实现留在各自
模块内。运行时不依赖其他源码目录，也不使用 Windows Task Scheduler。
项目采用模块化单体结构。框架能力位于 `src/gyxx_flow`，各业务模块运行时代码位于
`src/gyxx_flow/modules/<module>/runtime`。工作流、脚本、数据目录和外部系统配置均由
统一入口管理，同时保持模块间高内聚、低耦合。
## 能力概览
## 核心能力
- 23 条调度工作流（内容 8、商品 8、店铺 4、供应链 3），当前全部启用。
- 工作流目录只保存定时任务；补采、重试、映射刷新和受保护写操作统一由手动命令承载。
- Python 常驻调度支持日、周、月、间隔日、错过触发补偿、防重复和优雅停止。
- 31 个显式命令覆盖工作流节点和手动补偿入口；136 条内部浏览器绑定继续使用唯一 CDP、Profile、Cookie 和 storage state。
- PostgreSQL 使用运行时注入的云端 DSN；地址、数据库、用户和密码均不在源码中提供默认值。
- Hermes 保持本机 `data-collector`、`data-analyzer` 两个角色；飞书保持既有身份和接口。
- JSON、Markdown、CSV、Excel、下载文件和截图统一写入可迁移的数据根。
- `run`、`backfill`、`scripts run` 默认 dry-run，只有 `--execute` 允许真实副作用。
- 声明式工作流目录、依赖编排、超时、重试和失败恢复
- 计划任务生成、漂移检查和逐任务部署
- 手工执行、定时执行、日期回填、dry-run 和 shadow 模式
- 每次运行使用稳定的 `run_id`，记录步骤状态、日志、产物和外部写入
- JSON、Markdown、CSV、Excel、下载文件和截图统一分层存储
- 每个浏览器脚本独立 CDP 端口、Profile、Cookie 和 storage state
- 飞书、云端 PostgreSQL、本机 Hermes 和浏览器能力统一接入
## 目录
## 安装与验证
```text
config/ 工作流、时间表、脚本绑定和服务策略
deploy/ PostgreSQL 与调度服务部署文件
docs/ 部署、运维、源码同步和回滚手册
src/gyxx_flow/
adapters/ PostgreSQL、Hermes、飞书、浏览器适配边界
core/ 配置、运行上下文、日志、锁、产物
workflow/ LangGraph 模型、工厂与执行引擎
modules/<module>/ 四个业务域的工作流入口与内部实现
scheduler_service.py Python 常驻调度器
tests/ 项目级契约与回归测试
var/ 默认运行数据；不属于源码
```
要求 Python 3.12，推荐使用 `uv`：
业务模块：
- `content_marketing`：内容指标、达人、评论和营销报告。
- `product_commerce`：商品数据、人群画像、市场排行、主图和经营分析。
- `shop_intelligence`：店铺、竞店和京东自营业绩。
- `supply_chain`：采购确认、补货、库存预警和采购单更新。
## 开发环境
```powershell
cd D:\gyxx-flow
uv sync --python 3.12 --extra test
uv sync --python 3.12 --group dev
uv run ruff check src tests
uv run pytest
uv build
uv run gyxx doctor --json
uv run gyxx acceptance status --json
```
## 运行工作流
## 云端 PostgreSQL
由密钥系统向当前进程注入完整云端 DSN：
```powershell
# 示例只展示变量名；真实 DSN 由部署环境提供
$env:GYXX_POSTGRES_DSN = '<secret-manager-provided-postgresql-dsn>'
```
应用优先从 `GYXX_POSTGRES_DSN` 读取云端连接，并映射到各业务模块使用的 `PG_*`、`DB_*` 和
`AUTOFLOW_PG_*` 变量。源码、示例文件和运行报告均不保存真实地址或凭据；云端模式会拒绝回环数据库地址。
## 本机 Hermes
运行时使用两个本机业务 API base：
- analyzer：`http://127.0.0.1:8642/v1`
- collector：`http://127.0.0.1:8643/v1`
`28790/28791` 不作为工作流业务端点。
密钥通过 `GYXX_HERMES_API_KEY` 注入。非回环 Hermes 地址会在业务脚本启动前被拒绝。
不使用 Hermes 的纯采集工作流可在无 AI 环境运行；依赖分析或通知的工作流需要对应本机
角色可用。
## 工作流与脚本
```powershell
uv run gyxx list
uv run gyxx run product.daily --date 2026-07-27
uv run gyxx run product.daily --date 2026-07-27 --execute
uv run gyxx run product.daily --date 2026-08-01
uv run gyxx run product.daily --date 2026-08-01 --execute
uv run gyxx scripts list --module shop_intelligence
uv run gyxx scripts run shop.jd_self_operated.collect_product --date 2026-08-01
```
`run`、`backfill` 和 `scripts run` 默认都是无副作用 dry-run；只有显式加
`--execute` 才会启动项目内的业务脚本。
`gyxx list` 只展示调度工作流。手动补采和维护操作使用 `gyxx scripts run`；已配置的业务日期参数会从 `--date` 自动渲染，仍然只有显式添加 `--execute` 才会真实执行。
## 运行任意脚本
主图定时采集只注册为 `product.main_image.weekly`：每周日 08:30 同时启动京东和天猫两个独立分支。每个平台都在各自的采集完成后调用对应插入脚本，分别写入云端 PostgreSQL `main_image_creatives` 和飞书主图表；任一分支失败都不会取消、跳过或回滚另一分支的采集与写入，两个分支结束后工作流再汇总状态，并在失败摘要中标明具体平台和错误。真实写入仍受 `--execute`、凭据、登录态和副作用门禁约束。
| 场景 | 当前入口 |
|---|---|
| 内容映射重建 | `gyxx scripts run content.mapping.rebuild --date <日期>` |
| 内容失败任务重试 | `gyxx scripts run content.failed.retry --date <日期>` |
| 内容日报按日期重跑 | `gyxx backfill content.metrics.daily --from <日期> --to <日期>` |
| 京东自营品牌单日回采 | `gyxx scripts run shop.jd_self_operated.collect_brand --date <日期>` |
| 商品历史补采 | `gyxx scripts run product.backfill.run --date <日期>` |
| 商品评价补采 | `gyxx scripts run product.review.orchestrate --date <日期>` |
| 采购单更新 | `gyxx scripts run supply.workflow.run --date <日期>` |
每次运行生成稳定 `run_id`，并记录图节点状态、日志、产物、资源锁和外部副作用账本。
业务脚本通过共享适配器取得项目根、数据根、业务日期和服务配置，不应导入其他业务
模块的内部代码。旧 `module:entry` 脚本 ID 暂时保留为兼容别名。
## Python 定时调度
```powershell
uv run gyxx scripts list
uv run gyxx scripts list --module content_marketing
uv run gyxx scripts run content_marketing:run_all.py --date 2026-07-27
uv run gyxx scripts run content_marketing:run_all.py --date 2026-07-27 --execute
uv run gyxx schedule run --dry-run --once
uv run gyxx schedule status
uv run gyxx schedule run
```
脚本 ID 格式为 `<module>:<runtime 内相对路径>`。Python、BAT/CMD 和
PowerShell 入口均受统一项目根、数据根、业务日期、run_id 和 shadow 环境约束。
## 浏览器与外部系统绑定
`config/runtime-bindings.json` 为当前 131 个脚本各自分配固定且唯一的 CDP 端口。
无论从 workflow、`scripts run` 还是嵌套脚本启动，目标脚本都会重新取得自己的端口和
`state/browser/<module>/<script>/` 下的 Profile、Cookie、storage state；同一脚本下次
运行会复用登录态，不同脚本不会共享浏览器状态。
飞书继续走原来的 lark-cli/OpenAPI 身份；数据库继续走现有云端 PostgreSQL；Hermes
继续走本机服务。运行时会拒绝 localhost 数据库和非本机 Hermes 地址。不要在源码或
`runtime-bindings.json` 中写入 Cookie、数据库密码、飞书密钥或 Hermes token。
时间规则集中在 `config/schedules.json`：23 条声明全部启用，统一使用
`Asia/Shanghai`；单个日计划可以声明多个执行时间。服务器只托管一个 `gyxx schedule run` 进程，不要把业务规则复制为
Windows 任务或多条 cron。状态位于 `state/scheduler`，子任务日志位于 `logs/scheduler`。
## 数据目录
默认数据根为 `D:\gyxx-flow\var`，可用 `GYXX_DATA_ROOT` 配置到其他磁盘或服务器目录。
业务数据按模块写入 `data/raw`、`data/normalized`、`data/curated`、`data/exports`
和 `data/evidence`；运行状态、日志和临时文件分别进入 `state`、`logs` 和 `tmp`。
原始 JSON/CSV/XLS/XLSX/下载文件进 raw，清洗结果进 normalized，聚合数据进 curated，
最终 Markdown/Excel 报告进 exports。浏览器 Profile、Cookie 和 storage state 位于
`state/browser/<module>/<script>/` 并按脚本复用。源码与运行数据相互隔离，部署或更换
数据磁盘时只需调整环境变量。
开发环境默认数据根是项目下的 `var`；生产环境必须通过 `GYXX_DATA_ROOT` 外置，例如
Linux 使用 `/var/lib/gyxx-flow`。现有数据可以整体迁移，不应删除或写回源码目录。
## 定时任务
```powershell
uv run gyxx schedule plan `
--output D:\gyxx-flow\var\schedule-plan\candidate `
--start-date 2026-07-27 `
--python-executable D:\gyxx-flow\.venv\Scripts\python.exe
```text
<GYXX_DATA_ROOT>/
data/raw/<module>/ 原始 JSON/CSV/XLS/XLSX、下载文件、截图
data/normalized/<module>/ 清洗与标准化结果
data/curated/<module>/ 聚合和业务事实
data/exports/<module>/ Markdown、Excel 等最终报告
data/evidence/<module>/ 验收及对账证据
state/browser/<module>/ 每脚本 Profile、Cookie、storage state
state/ 调度、工作流和幂等状态
logs/ 运行日志
tmp/ 可清理临时文件
```
该命令只生成 21 个任务定义和审核脚本，不会直接注册、禁用或修改系统任务。
生产启用时应先执行环境检查和 dry-run，再按工作流逐项应用调度配置。
## 业务源码同步
## 关键文件
源码同步采用清单驱动的三方哈希比较，状态检查默认只读。源目录只用于发现更新，不是项目
运行依赖：
- `design.md`：系统架构和模块边界
- `plan.md`：逐项验收清单
- `config/workflows.json`：工作流定义与本地执行入口
- `config/schedules.json`：定时调度配置
- `config/runtime-bindings.json`：脚本 CDP 端口与外部服务策略
- `docs/`：部署、运行和回滚手册
```powershell
uv run gyxx sources status `
--source-root content_marketing=<内容源码目录> `
--source-root product_commerce=<商品源码目录> `
--source-root shop_intelligence=<店铺源码目录> `
--source-root supply_chain=<供应链源码目录>
```
只有未转换、无冲突的源端单边变化可通过 `sources apply --execute` 自动复制；经过统一适配
的文件必须人工复核并更新来源/目标哈希。Cookie、Profile、凭据、日志和采集结果不会进入
源码同步清单。
详细操作见 [架构说明](docs/architecture.md)、[部署手册](docs/deployment.md)、
[运维手册](docs/runbook.md) 和 [验收报告](docs/acceptance-report.md)。
config/commands.json
+41
View File
@@ -0,0 +1,41 @@
{
"schema_version": 1,
"commands": [
{"id":"content.metrics.collect_collaborators","module":"content_marketing","entry":"run_all.py","kind":"python"},
{"id":"content.mapping.refresh_self","module":"content_marketing","entry":"data/tools/refresh_self_mapping.py","kind":"python"},
{"id":"content.metrics.collect_self_bilibili","module":"content_marketing","entry":"self_bilibili_scraper.py","kind":"python"},
{"id":"content.metrics.collect_self_douyin","module":"content_marketing","entry":"chanmama_scraper.py","kind":"python"},
{"id":"content.metrics.sync","module":"content_marketing","entry":"data/tools/sync_metrics_to_cmt_notes.py","kind":"python"},
{"id":"content.marketing_report.generate","module":"content_marketing","entry":"daily_marketing_report.py","kind":"python"},
{"id":"content.login.refresh","module":"content_marketing","entry":"data/tools/friday_relogin_parallel.py","kind":"python"},
{"id":"content.creator_report.generate","module":"content_marketing","entry":"data/tools/generate_creator_report.py","kind":"python"},
{"id":"content.summary.monthly","module":"content_marketing","entry":"monthly_summary_all.py","kind":"python"},
{"id":"content.cooperations.sync","module":"content_marketing","entry":"data/tools/sync_cooperations.py","kind":"python"},
{"id":"content.comments.collect_bilibili","module":"content_marketing","entry":"data/tools/batch_rescrape_bilibili.py","kind":"python"},
{"id":"content.comments.collect_xiaohongshu","module":"content_marketing","entry":"data/tools/batch_rescrape_xiaohongshu.py","kind":"python"},
{"id":"content.comments.collect_douyin","module":"content_marketing","entry":"data/tools/batch_rescrape_douyin.py","kind":"python"},
{"id":"content.summary.weekly","module":"content_marketing","entry":"weekly_summary_all.py","kind":"python"},
{"id":"content.mapping.rebuild","module":"content_marketing","entry":"data/tools/rebuild_mapping.py","kind":"python"},
{"id":"content.failed.retry","module":"content_marketing","entry":"data/tools/retry_failed.py","kind":"python"},
{"id":"shop.metrics.collect","module":"shop_intelligence","entry":"runners/run_shop.py","kind":"python"},
{"id":"shop.competitor.collect","module":"shop_intelligence","entry":"runners/run_peer_store.py","kind":"python"},
{"id":"shop.jd_self_operated.collect_brand","module":"shop_intelligence","entry":"collectors/jd_self_operated_brand_daily.py","kind":"python","default_args":["--start-date","{business_date}","--end-date","{business_date}","--headless"]},
{"id":"shop.jd_self_operated.collect_product","module":"shop_intelligence","entry":"collectors/jd_self_operated_product_daily.py","kind":"python","default_args":["--start-date","{business_date}","--end-date","{business_date}","--headless"]},
{"id":"shop.douyin_price_appeal","module":"shop_intelligence","entry":"collectors/dy_store_competitor_store_scraping.py","kind":"python","default_args":["--price-appeal","--execute"]},
{"id":"product.persona.collect","module":"product_commerce","entry":"run_daily_persona.py","kind":"python"},
{"id":"product.daily.orchestrate","module":"product_commerce","entry":"orchestrate_daily_collection.py","kind":"python"},
{"id":"product.alert.run","module":"product_commerce","entry":"commands/run_alerts.py","kind":"python"},
{"id":"product.import.daily","module":"product_commerce","entry":"commands/import_daily.py","kind":"python"},
{"id":"product.style_analysis.run","module":"product_commerce","entry":"analyze_style_with_hermes.py","kind":"python"},
{"id":"product.main_image.collect_jd","module":"product_commerce","entry":"run_weekly_jd_main_image.py","kind":"python"},
{"id":"product.main_image.collect_tmall","module":"product_commerce","entry":"run_weekly_main_image.py","kind":"python"},
{"id":"product.sales_sheet.sync","module":"product_commerce","entry":"sync_monthly_sales_sheet.py","kind":"python","default_args":["--month","{business_date}","--execute"]},
{"id":"product.backfill.run","module":"product_commerce","entry":"backfill_collect.py","kind":"python","default_args":["--from","{business_date}","--to","{business_date}"]},
{"id":"product.market_rank.orchestrate","module":"product_commerce","entry":"orchestrate_market_rank_collection.py","kind":"python"},
{"id":"product.review.orchestrate","module":"product_commerce","entry":"orchestrate_review_collection.py","kind":"python","default_args":["--target-date","{business_date}"]},
{"id":"supply.workflow.run","module":"supply_chain","entry":"run.py","kind":"python","default_args":["mcp-run","purchase-order-update"]}
]
}
config/product-monthly-sales.json
+49
View File
@@ -0,0 +1,49 @@
{
"spreadsheet_url": "https://bu0zgpibak.feishu.cn/sheets/O7nGsVXyvhI0W6tnOaFcSABVnqb",
"base_url": "https://bu0zgpibak.feishu.cn/base/TH1JbxhfUaxetis6F4pcoqvWncd?table=tbl8q0hbIPMNn0k8&view=vew7j57rli",
"base_token": "TH1JbxhfUaxetis6F4pcoqvWncd",
"table_id": "tbl8q0hbIPMNn0k8",
"view_id": "vew7j57rli",
"product_field": "品名",
"target_field": "目标销量",
"month_field": "月份",
"lark_profile": "cli_aa8c4fb4c4f81cd3",
"style_base_url": "https://bu0zgpibak.feishu.cn/base/TtoCb1NuQaDy3NsZWTpc0GIvnph?table=tblKCjplVAFrRwMC&view=vewvy33xEk",
"style_base_token": "TtoCb1NuQaDy3NsZWTpc0GIvnph",
"style_table_id": "tblKCjplVAFrRwMC",
"style_view_id": "vewvy33xEk",
"erp_shop_labels": [
"JD光影行星旗舰店",
"ozko抖音店",
"OZKO京东店",
"OZKO拼多多店",
"ozko旗舰店-天猫",
"ozko-视频号",
"ozko小红书店",
"得物-ozko",
"得物-光影行星",
"抖店-光影行星数码旗舰店",
"抖店-光影行星箱包旗舰店",
"光影行星-快手",
"光影行星旗舰店-小红书",
"光影行星-视频号",
"光影行星-微信商城",
"拼多多-光影行星旗舰店",
"拼多多-光影行星箱包旗舰店",
"拼多多-光影行星专卖店",
"天猫-光影行星旗舰店",
"天猫-光影行星箱包旗舰店"
],
"product_aliases": {
"盖亚微单Pro": "盖亚微单",
"星迹&星迹2": "星迹2",
"宙斯双肩包": "宙斯",
"瑞白双肩包": "瑞白",
"拾影双肩电脑包": "拾影双肩",
"拾影相机包": "拾影斜挎相机包",
"极星单肩包": "极星单肩",
"极星托特": "极星托特",
"云栖相机双肩包": "云栖",
"轻风双肩包": "轻风双肩包"
}
}
config/runtime-bindings.json
+142 -133
View File
@@ -1,143 +1,152 @@
{
"schema_version": 1,
"schema_version": 2,
"cdp_host": "127.0.0.1",
"scripts": {
"content_marketing:bilibili_comment_scraper.py": 22000,
"content_marketing:bilibili_scraper.py": 22001,
"content_marketing:chanmama_scraper.py": 22002,
"content_marketing:daily_marketing_report.py": 22003,
"content_marketing:data/tools/analyze_comments.py": 22004,
"content_marketing:data/tools/analyze_note.py": 22005,
"content_marketing:data/tools/batch_rescrape_bilibili.py": 22006,
"content_marketing:data/tools/batch_rescrape_douyin.py": 22007,
"content_marketing:data/tools/batch_rescrape_xiaohongshu.py": 22008,
"content_marketing:data/tools/check_status.py": 22009,
"content_marketing:data/tools/collect_note_metrics.py": 22010,
"content_marketing:data/tools/daily_marketing_report.bat": 22011,
"content_marketing:data/tools/daily_run.bat": 22012,
"content_marketing:data/tools/daily_run_with_backfill.bat": 22013,
"content_marketing:data/tools/db.py": 22014,
"content_marketing:data/tools/feishu_comment_batch.py": 22015,
"content_marketing:data/tools/feishu_doc_writer.py": 22016,
"content_marketing:data/tools/friday_relogin.bat": 22017,
"content_marketing:data/tools/friday_relogin_parallel.py": 22018,
"content_marketing:data/tools/generate_creator_report.py": 22019,
"content_marketing:data/tools/init_db.py": 22020,
"content_marketing:data/tools/kill_project_chrome.ps1": 22021,
"content_marketing:data/tools/list_project_chrome.ps1": 22022,
"content_marketing:data/tools/monday_self_run.bat": 22023,
"content_marketing:data/tools/monthly_creator_report.bat": 22024,
"content_marketing:data/tools/monthly_summary.bat": 22025,
"content_marketing:data/tools/pull_all_tables.py": 22026,
"content_marketing:data/tools/rebuild_mapping.py": 22027,
"content_marketing:data/tools/relogin_bilibili.py": 22028,
"content_marketing:data/tools/relogin_douyin.py": 22029,
"content_marketing:data/tools/relogin_pgy.py": 22030,
"content_marketing:data/tools/relogin_xiaohongshu.py": 22031,
"content_marketing:data/tools/relogin_xingtu.py": 22032,
"content_marketing:data/tools/retry_failed.py": 22033,
"content_marketing:data/tools/sync_cooperations.bat": 22034,
"content_marketing:data/tools/sync_cooperations.py": 22035,
"content_marketing:data/tools/sync_metrics_to_cmt_notes.py": 22036,
"content_marketing:data/tools/sync_style_categories.py": 22037,
"content_marketing:data/tools/validate_mapping.py": 22038,
"content_marketing:data/tools/weekly_comment_scrape.bat": 22039,
"content_marketing:data/tools/weekly_summary.bat": 22040,
"content_marketing:data/tools/write_notes_to_doc.py": 22041,
"content_marketing:douyin_comment_scraper.py": 22042,
"content_marketing:feishu_mapping.py": 22043,
"content_marketing:login_helper.py": 22044,
"content_marketing:monthly_summary_all.py": 22045,
"content_marketing:pgy_xhs_scraper.py": 22046,
"content_marketing:pgy_xhs_scraper_v2.py": 22047,
"content_marketing:run_all.py": 22048,
"content_marketing:self_bilibili_scraper.py": 22049,
"content_marketing:self_douyin_scraper.py": 22050,
"content_marketing:weekly_summary_all.py": 22051,
"content_marketing:weekly_summary_xingyun2.py": 22052,
"content_marketing:xiaohongshu_comment_scraper.py": 22053,
"content_marketing:xingtu_scraper.py": 22054,
"content_marketing:xingtu_scraper_v2.py": 22055,
"product_commerce:aggregate_daily_final.py": 22056,
"product_commerce:analyze_style_with_hermes.py": 22057,
"product_commerce:backfill_poseidon_sales.py": 22130,
"product_commerce:backfill_collect.py": 22058,
"product_commerce:backfill_one_day.py": 22059,
"product_commerce:check_nine_day_decline.py": 22060,
"product_commerce:collect_dy_market_rank.py": 22061,
"product_commerce:collect_dy_persona_to_bitable.py": 22062,
"product_commerce:collect_erp_yesterday_metrics.py": 22063,
"product_commerce:collect_jd_market_rank.py": 22064,
"product_commerce:collect_jd_persona_to_bitable.py": 22065,
"product_commerce:collect_persona_to_bitable.py": 22066,
"product_commerce:collect_sycm_market_rank.py": 22067,
"product_commerce:commands/import_daily.py": 22068,
"product_commerce:commands/run_alerts.py": 22069,
"product_commerce:db/sync_dim_style.py": 22070,
"product_commerce:db/sync_sku_master.py": 22071,
"product_commerce:dy_audience_profile_collect.py": 22072,
"product_commerce:dy_product_scraping.py": 22073,
"product_commerce:erp_login_product_analysis.py": 22074,
"product_commerce:export_bitable_records.py": 22075,
"product_commerce:import_product_daily.py": 22076,
"product_commerce:import_product_reviews.py": 22077,
"product_commerce:insert_bitable_records.py": 22078,
"product_commerce:jd_main_image_collector.py": 22079,
"product_commerce:jd_product_data_collector.py": 22080,
"product_commerce:jd_self_inventory_sales_collector.py": 22081,
"product_commerce:orchestrate_daily_collection.py": 22082,
"product_commerce:orchestrate_market_rank_collection.py": 22083,
"product_commerce:orchestrate_review_collection.py": 22084,
"product_commerce:reapply_erp_override.py": 22085,
"product_commerce:rebuild_market_rank_documents.py": 22086,
"product_commerce:run_alerts_with_retry.py": 22087,
"product_commerce:run_daily_persona.py": 22088,
"product_commerce:run_weekly_jd_main_image.py": 22089,
"product_commerce:run_weekly_main_image.py": 22090,
"product_commerce:scripts/insert_jd_main_image_records.py": 22091,
"product_commerce:scripts/insert_main_image_records.py": 22092,
"product_commerce:taobao_dmp_item_crowd_insight_screenshots.py": 22093,
"product_commerce:taobao_sycm_collect.py": 22094,
"product_commerce:taobao_sycm_collect_backfill.py": 22095,
"product_commerce:taobao_sycm_products.py": 22096,
"product_commerce:taobao_wanxiang_ai_creative_report.py": 22097,
"product_commerce:upload_video_to_guanghe.py": 22098,
"product_commerce:vendors/dy-data-flow/dy_store_competitor_store_scraping.py": 22099,
"product_commerce:vendors/jd-data-flow/jd_data_collector.py": 22100,
"product_commerce:vendors/jd-data-flow/jd_peer_product_data_collector.py": 22101,
"product_commerce:vendors/jd-data-flow/jd_product_data_collector.py": 22102,
"product_commerce:weekly_aggregate.py": 22103,
"shop_intelligence:collectors/dy_store_competitor_store_scraping.py": 22104,
"shop_intelligence:collectors/jd_data_collector.py": 22105,
"shop_intelligence:collectors/jd_peer_store_data_collector.py": 22106,
"shop_intelligence:collectors/taobao_sycm.py": 22107,
"shop_intelligence:runners/run_peer_store.py": 22108,
"shop_intelligence:runners/run_shop.py": 22109,
"shop_intelligence:scripts/remove_scheduler.ps1": 22110,
"shop_intelligence:scripts/setup_scheduler.ps1": 22111,
"supply_chain:orchestrator/mcp_workflow.py": 22112,
"supply_chain:orchestrator/monitor.py": 22113,
"supply_chain:orchestrator/runner.py": 22114,
"supply_chain:orchestrator/scripts/ProductReplenishment.py": 22115,
"supply_chain:orchestrator/scripts/PurchaseConfirmation.py": 22116,
"supply_chain:orchestrator/scripts/PurchaseOrderUpdate.py": 22117,
"supply_chain:orchestrator/scripts/batch_process.py": 22118,
"supply_chain:orchestrator/scripts/collect_confirmation.ps1": 22119,
"supply_chain:orchestrator/scripts/collect_purchase_order_update.ps1": 22120,
"supply_chain:orchestrator/scripts/collect_replenishment.ps1": 22121,
"supply_chain:orchestrator/scripts/insert_replenishment_bitable.py": 22122,
"supply_chain:orchestrator/scripts/send_card_notification.py": 22123,
"supply_chain:orchestrator/scripts/trigger_purchase_order_update.py": 22124,
"supply_chain:orchestrator/sql/backfill_history.py": 22125,
"supply_chain:run.py": 22126,
"supply_chain:scripts/purchase-confirmation.bat": 22127,
"supply_chain:scripts/replenishment-alert.bat": 22128,
"supply_chain:scripts/replenishment.bat": 22129
"content_marketing:bilibili_comment_scraper.py": {"script_id":"content_marketing:bilibili_comment_scraper.py","state_key":"content_marketing:bilibili_comment_scraper.py","aliases":["content_marketing:bilibili_comment_scraper.py"],"cdp_port":22000},
"content_marketing:bilibili_scraper.py": {"script_id":"content_marketing:bilibili_scraper.py","state_key":"content_marketing:bilibili_scraper.py","aliases":["content_marketing:bilibili_scraper.py"],"cdp_port":22001,"login_mode":"A","required_cookie_domains":["bilibili.com"],"required_cookie_names":["SESSDATA"]},
"content.metrics.collect_self_douyin": {"script_id":"content_marketing:chanmama_scraper.py","state_key":"content_marketing:chanmama_scraper.py","aliases":["content_marketing:chanmama_scraper.py"],"cdp_port":22002,"login_mode":"C","required_cookie_domains":["chanmama.com"],"credential_env_names":["CHANMAMA_ACCOUNT","CHANMAMA_PASSWORD"]},
"content.marketing_report.generate": {"script_id":"content_marketing:daily_marketing_report.py","state_key":"content_marketing:daily_marketing_report.py","aliases":["content_marketing:daily_marketing_report.py"],"cdp_port":22003},
"content_marketing:data/tools/analyze_comments.py": {"script_id":"content_marketing:data/tools/analyze_comments.py","state_key":"content_marketing:data/tools/analyze_comments.py","aliases":["content_marketing:data/tools/analyze_comments.py"],"cdp_port":22004},
"content_marketing:data/tools/analyze_note.py": {"script_id":"content_marketing:data/tools/analyze_note.py","state_key":"content_marketing:data/tools/analyze_note.py","aliases":["content_marketing:data/tools/analyze_note.py"],"cdp_port":22005},
"content.comments.collect_bilibili": {"script_id":"content_marketing:data/tools/batch_rescrape_bilibili.py","state_key":"content_marketing:data/tools/batch_rescrape_bilibili.py","aliases":["content_marketing:data/tools/batch_rescrape_bilibili.py"],"cdp_port":22006,"login_mode":"A","required_cookie_domains":["bilibili.com"],"required_cookie_names":["SESSDATA"]},
"content.comments.collect_douyin": {"script_id":"content_marketing:data/tools/batch_rescrape_douyin.py","state_key":"content_marketing:data/tools/batch_rescrape_douyin.py","aliases":["content_marketing:data/tools/batch_rescrape_douyin.py"],"cdp_port":22007,"login_mode":"A","required_cookie_domains":["douyin.com"],"required_cookie_names":["sessionid"]},
"content.comments.collect_xiaohongshu": {"script_id":"content_marketing:data/tools/batch_rescrape_xiaohongshu.py","state_key":"content_marketing:data/tools/batch_rescrape_xiaohongshu.py","aliases":["content_marketing:data/tools/batch_rescrape_xiaohongshu.py"],"cdp_port":22008,"login_mode":"A","required_cookie_domains":["xiaohongshu.com"],"required_cookie_names":["web_session"]},
"content_marketing:data/tools/check_status.py": {"script_id":"content_marketing:data/tools/check_status.py","state_key":"content_marketing:data/tools/check_status.py","aliases":["content_marketing:data/tools/check_status.py"],"cdp_port":22009},
"content_marketing:data/tools/collect_note_metrics.py": {"script_id":"content_marketing:data/tools/collect_note_metrics.py","state_key":"content_marketing:data/tools/collect_note_metrics.py","aliases":["content_marketing:data/tools/collect_note_metrics.py"],"cdp_port":22010},
"content_marketing:data/tools/daily_marketing_report.bat": {"script_id":"content_marketing:data/tools/daily_marketing_report.bat","state_key":"content_marketing:data/tools/daily_marketing_report.bat","aliases":["content_marketing:data/tools/daily_marketing_report.bat"],"cdp_port":22011},
"content_marketing:data/tools/daily_run.bat": {"script_id":"content_marketing:data/tools/daily_run.bat","state_key":"content_marketing:data/tools/daily_run.bat","aliases":["content_marketing:data/tools/daily_run.bat"],"cdp_port":22012},
"content_marketing:data/tools/daily_run_with_backfill.bat": {"script_id":"content_marketing:data/tools/daily_run_with_backfill.bat","state_key":"content_marketing:data/tools/daily_run_with_backfill.bat","aliases":["content_marketing:data/tools/daily_run_with_backfill.bat"],"cdp_port":22013},
"content_marketing:data/tools/db.py": {"script_id":"content_marketing:data/tools/db.py","state_key":"content_marketing:data/tools/db.py","aliases":["content_marketing:data/tools/db.py"],"cdp_port":22014},
"content_marketing:data/tools/feishu_comment_batch.py": {"script_id":"content_marketing:data/tools/feishu_comment_batch.py","state_key":"content_marketing:data/tools/feishu_comment_batch.py","aliases":["content_marketing:data/tools/feishu_comment_batch.py"],"cdp_port":22015},
"content_marketing:data/tools/feishu_doc_writer.py": {"script_id":"content_marketing:data/tools/feishu_doc_writer.py","state_key":"content_marketing:data/tools/feishu_doc_writer.py","aliases":["content_marketing:data/tools/feishu_doc_writer.py"],"cdp_port":22016},
"content_marketing:data/tools/friday_relogin.bat": {"script_id":"content_marketing:data/tools/friday_relogin.bat","state_key":"content_marketing:data/tools/friday_relogin.bat","aliases":["content_marketing:data/tools/friday_relogin.bat"],"cdp_port":22017},
"content.login.refresh": {"script_id":"content_marketing:data/tools/friday_relogin_parallel.py","state_key":"content_marketing:data/tools/friday_relogin_parallel.py","aliases":["content_marketing:data/tools/friday_relogin_parallel.py"],"cdp_port":22018,"login_mode":"D"},
"content.creator_report.generate": {"script_id":"content_marketing:data/tools/generate_creator_report.py","state_key":"content_marketing:data/tools/generate_creator_report.py","aliases":["content_marketing:data/tools/generate_creator_report.py"],"cdp_port":22019},
"content_marketing:data/tools/init_db.py": {"script_id":"content_marketing:data/tools/init_db.py","state_key":"content_marketing:data/tools/init_db.py","aliases":["content_marketing:data/tools/init_db.py"],"cdp_port":22020},
"content_marketing:data/tools/kill_project_chrome.ps1": {"script_id":"content_marketing:data/tools/kill_project_chrome.ps1","state_key":"content_marketing:data/tools/kill_project_chrome.ps1","aliases":["content_marketing:data/tools/kill_project_chrome.ps1"],"cdp_port":22021},
"content_marketing:data/tools/list_project_chrome.ps1": {"script_id":"content_marketing:data/tools/list_project_chrome.ps1","state_key":"content_marketing:data/tools/list_project_chrome.ps1","aliases":["content_marketing:data/tools/list_project_chrome.ps1"],"cdp_port":22022},
"content_marketing:data/tools/monday_self_run.bat": {"script_id":"content_marketing:data/tools/monday_self_run.bat","state_key":"content_marketing:data/tools/monday_self_run.bat","aliases":["content_marketing:data/tools/monday_self_run.bat"],"cdp_port":22023},
"content_marketing:data/tools/monthly_creator_report.bat": {"script_id":"content_marketing:data/tools/monthly_creator_report.bat","state_key":"content_marketing:data/tools/monthly_creator_report.bat","aliases":["content_marketing:data/tools/monthly_creator_report.bat"],"cdp_port":22024},
"content_marketing:data/tools/monthly_summary.bat": {"script_id":"content_marketing:data/tools/monthly_summary.bat","state_key":"content_marketing:data/tools/monthly_summary.bat","aliases":["content_marketing:data/tools/monthly_summary.bat"],"cdp_port":22025},
"content_marketing:data/tools/pull_all_tables.py": {"script_id":"content_marketing:data/tools/pull_all_tables.py","state_key":"content_marketing:data/tools/pull_all_tables.py","aliases":["content_marketing:data/tools/pull_all_tables.py"],"cdp_port":22026},
"content.mapping.rebuild": {"script_id":"content_marketing:data/tools/rebuild_mapping.py","state_key":"content_marketing:data/tools/rebuild_mapping.py","aliases":["content_marketing:data/tools/rebuild_mapping.py"],"cdp_port":22027},
"content.mapping.refresh_self": {"script_id":"content_marketing:data/tools/refresh_self_mapping.py","state_key":"content_marketing:data/tools/refresh_self_mapping.py","aliases":["content_marketing:data/tools/refresh_self_mapping.py"],"cdp_port":22135},
"content_marketing:data/tools/relogin_bilibili.py": {"script_id":"content_marketing:data/tools/relogin_bilibili.py","state_key":"content_marketing:data/tools/relogin_bilibili.py","aliases":["content_marketing:data/tools/relogin_bilibili.py"],"cdp_port":22028,"login_mode":"D","required_cookie_domains":["bilibili.com"],"required_cookie_names":["SESSDATA"]},
"content_marketing:data/tools/relogin_douyin.py": {"script_id":"content_marketing:data/tools/relogin_douyin.py","state_key":"content_marketing:data/tools/relogin_douyin.py","aliases":["content_marketing:data/tools/relogin_douyin.py"],"cdp_port":22029,"login_mode":"D","required_cookie_domains":["douyin.com"],"required_cookie_names":["sessionid"]},
"content_marketing:data/tools/relogin_pgy.py": {"script_id":"content_marketing:data/tools/relogin_pgy.py","state_key":"content_marketing:data/tools/relogin_pgy.py","aliases":["content_marketing:data/tools/relogin_pgy.py"],"cdp_port":22030,"login_mode":"D","required_cookie_domains":["xiaohongshu.com"]},
"content_marketing:data/tools/relogin_xiaohongshu.py": {"script_id":"content_marketing:data/tools/relogin_xiaohongshu.py","state_key":"content_marketing:data/tools/relogin_xiaohongshu.py","aliases":["content_marketing:data/tools/relogin_xiaohongshu.py"],"cdp_port":22031,"login_mode":"D","required_cookie_domains":["xiaohongshu.com"],"required_cookie_names":["web_session"]},
"content_marketing:data/tools/relogin_xingtu.py": {"script_id":"content_marketing:data/tools/relogin_xingtu.py","state_key":"content_marketing:data/tools/relogin_xingtu.py","aliases":["content_marketing:data/tools/relogin_xingtu.py"],"cdp_port":22032,"login_mode":"D","required_cookie_domains":["douyin.com","oceanengine.com","xingtu.cn"],"required_cookie_names":["sessionid","sessionid_ss","sid_tt","sid_guard"]},
"content.failed.retry": {"script_id":"content_marketing:data/tools/retry_failed.py","state_key":"content_marketing:data/tools/retry_failed.py","aliases":["content_marketing:data/tools/retry_failed.py"],"cdp_port":22033},
"content_marketing:data/tools/sync_cooperations.bat": {"script_id":"content_marketing:data/tools/sync_cooperations.bat","state_key":"content_marketing:data/tools/sync_cooperations.bat","aliases":["content_marketing:data/tools/sync_cooperations.bat"],"cdp_port":22034},
"content.cooperations.sync": {"script_id":"content_marketing:data/tools/sync_cooperations.py","state_key":"content_marketing:data/tools/sync_cooperations.py","aliases":["content_marketing:data/tools/sync_cooperations.py"],"cdp_port":22035},
"content.metrics.sync": {"script_id":"content_marketing:data/tools/sync_metrics_to_cmt_notes.py","state_key":"content_marketing:data/tools/sync_metrics_to_cmt_notes.py","aliases":["content_marketing:data/tools/sync_metrics_to_cmt_notes.py"],"cdp_port":22036},
"content_marketing:data/tools/sync_style_categories.py": {"script_id":"content_marketing:data/tools/sync_style_categories.py","state_key":"content_marketing:data/tools/sync_style_categories.py","aliases":["content_marketing:data/tools/sync_style_categories.py"],"cdp_port":22037},
"content_marketing:data/tools/validate_mapping.py": {"script_id":"content_marketing:data/tools/validate_mapping.py","state_key":"content_marketing:data/tools/validate_mapping.py","aliases":["content_marketing:data/tools/validate_mapping.py"],"cdp_port":22038},
"content_marketing:data/tools/weekly_comment_scrape.bat": {"script_id":"content_marketing:data/tools/weekly_comment_scrape.bat","state_key":"content_marketing:data/tools/weekly_comment_scrape.bat","aliases":["content_marketing:data/tools/weekly_comment_scrape.bat"],"cdp_port":22039},
"content_marketing:data/tools/weekly_summary.bat": {"script_id":"content_marketing:data/tools/weekly_summary.bat","state_key":"content_marketing:data/tools/weekly_summary.bat","aliases":["content_marketing:data/tools/weekly_summary.bat"],"cdp_port":22040},
"content_marketing:data/tools/write_notes_to_doc.py": {"script_id":"content_marketing:data/tools/write_notes_to_doc.py","state_key":"content_marketing:data/tools/write_notes_to_doc.py","aliases":["content_marketing:data/tools/write_notes_to_doc.py"],"cdp_port":22041},
"content_marketing:douyin_comment_scraper.py": {"script_id":"content_marketing:douyin_comment_scraper.py","state_key":"content_marketing:douyin_comment_scraper.py","aliases":["content_marketing:douyin_comment_scraper.py"],"cdp_port":22042},
"content_marketing:feishu_mapping.py": {"script_id":"content_marketing:feishu_mapping.py","state_key":"content_marketing:feishu_mapping.py","aliases":["content_marketing:feishu_mapping.py"],"cdp_port":22043},
"content_marketing:login_helper.py": {"script_id":"content_marketing:login_helper.py","state_key":"content_marketing:login_helper.py","aliases":["content_marketing:login_helper.py"],"cdp_port":22044},
"content.summary.monthly": {"script_id":"content_marketing:monthly_summary_all.py","state_key":"content_marketing:monthly_summary_all.py","aliases":["content_marketing:monthly_summary_all.py"],"cdp_port":22045},
"content_marketing:pgy_xhs_scraper.py": {"script_id":"content_marketing:pgy_xhs_scraper.py","state_key":"content_marketing:pgy_xhs_scraper.py","aliases":["content_marketing:pgy_xhs_scraper.py"],"cdp_port":22046},
"content_marketing:pgy_xhs_scraper_v2.py": {"script_id":"content_marketing:pgy_xhs_scraper_v2.py","state_key":"content_marketing:pgy_xhs_scraper_v2.py","aliases":["content_marketing:pgy_xhs_scraper_v2.py"],"cdp_port":22047,"login_mode":"A","required_cookie_domains":["xiaohongshu.com"]},
"content.metrics.collect_collaborators": {"script_id":"content_marketing:run_all.py","state_key":"content_marketing:run_all.py","aliases":["content_marketing:run_all.py"],"cdp_port":22048},
"content.metrics.collect_self_bilibili": {"script_id":"content_marketing:self_bilibili_scraper.py","state_key":"content_marketing:self_bilibili_scraper.py","aliases":["content_marketing:self_bilibili_scraper.py"],"cdp_port":22049,"login_mode":"A","required_cookie_domains":["bilibili.com"],"required_cookie_names":["SESSDATA"]},
"content_marketing:self_douyin_scraper.py": {"script_id":"content_marketing:self_douyin_scraper.py","state_key":"content_marketing:self_douyin_scraper.py","aliases":["content_marketing:self_douyin_scraper.py"],"cdp_port":22050,"login_mode":"A","required_cookie_domains":["douyin.com"],"required_cookie_names":["sessionid","sessionid_ss","sid_tt","sid_guard"]},
"content.summary.weekly": {"script_id":"content_marketing:weekly_summary_all.py","state_key":"content_marketing:weekly_summary_all.py","aliases":["content_marketing:weekly_summary_all.py"],"cdp_port":22051},
"content_marketing:weekly_summary_xingyun2.py": {"script_id":"content_marketing:weekly_summary_xingyun2.py","state_key":"content_marketing:weekly_summary_xingyun2.py","aliases":["content_marketing:weekly_summary_xingyun2.py"],"cdp_port":22052},
"content_marketing:xiaohongshu_comment_scraper.py": {"script_id":"content_marketing:xiaohongshu_comment_scraper.py","state_key":"content_marketing:xiaohongshu_comment_scraper.py","aliases":["content_marketing:xiaohongshu_comment_scraper.py"],"cdp_port":22053},
"content_marketing:xingtu_scraper.py": {"script_id":"content_marketing:xingtu_scraper.py","state_key":"content_marketing:xingtu_scraper.py","aliases":["content_marketing:xingtu_scraper.py"],"cdp_port":22054},
"content_marketing:xingtu_scraper_v2.py": {"script_id":"content_marketing:xingtu_scraper_v2.py","state_key":"content_marketing:xingtu_scraper_v2.py","aliases":["content_marketing:xingtu_scraper_v2.py"],"cdp_port":22055,"login_mode":"A","required_cookie_domains":["douyin.com","oceanengine.com","xingtu.cn"],"required_cookie_names":["sessionid","sessionid_ss","sid_tt","sid_guard"]},
"product_commerce:aggregate_daily_final.py": {"script_id":"product_commerce:aggregate_daily_final.py","state_key":"product_commerce:aggregate_daily_final.py","aliases":["product_commerce:aggregate_daily_final.py"],"cdp_port":22056},
"product.style_analysis.run": {"script_id":"product_commerce:analyze_style_with_hermes.py","state_key":"product_commerce:analyze_style_with_hermes.py","aliases":["product_commerce:analyze_style_with_hermes.py"],"cdp_port":22057},
"product_commerce:backfill_poseidon_sales.py": {"script_id":"product_commerce:backfill_poseidon_sales.py","state_key":"product_commerce:backfill_poseidon_sales.py","aliases":["product_commerce:backfill_poseidon_sales.py"],"cdp_port":22130},
"product.backfill.run": {"script_id":"product_commerce:backfill_collect.py","state_key":"product_commerce:backfill_collect.py","aliases":["product_commerce:backfill_collect.py"],"cdp_port":22058},
"product_commerce:backfill_one_day.py": {"script_id":"product_commerce:backfill_one_day.py","state_key":"product_commerce:backfill_one_day.py","aliases":["product_commerce:backfill_one_day.py"],"cdp_port":22059},
"product_commerce:check_nine_day_decline.py": {"script_id":"product_commerce:check_nine_day_decline.py","state_key":"product_commerce:check_nine_day_decline.py","aliases":["product_commerce:check_nine_day_decline.py"],"cdp_port":22060},
"product_commerce:collect_dy_market_rank.py": {"script_id":"product_commerce:collect_dy_market_rank.py","state_key":"product_commerce:collect_dy_market_rank.py","aliases":["product_commerce:collect_dy_market_rank.py"],"cdp_port":22061,"login_mode":"A","required_cookie_domains":["jinritemai.com"]},
"product_commerce:collect_dy_persona_to_bitable.py": {"script_id":"product_commerce:collect_dy_persona_to_bitable.py","state_key":"product_commerce:collect_dy_persona_to_bitable.py","aliases":["product_commerce:collect_dy_persona_to_bitable.py"],"cdp_port":22062},
"product_commerce:collect_erp_yesterday_metrics.py": {"script_id":"product_commerce:collect_erp_yesterday_metrics.py","state_key":"product_commerce:collect_erp_yesterday_metrics.py","aliases":["product_commerce:collect_erp_yesterday_metrics.py"],"cdp_port":22063,"login_mode":"C","required_cookie_domains":["erp321.com"]},
"product_commerce:collect_jd_market_rank.py": {"script_id":"product_commerce:collect_jd_market_rank.py","state_key":"product_commerce:collect_jd_market_rank.py","aliases":["product_commerce:collect_jd_market_rank.py"],"cdp_port":22064,"login_mode":"C","required_cookie_domains":["jd.com"],"credential_env_names":["JD_PASSWORD"]},
"product_commerce:collect_jd_persona_to_bitable.py": {"script_id":"product_commerce:collect_jd_persona_to_bitable.py","state_key":"product_commerce:collect_jd_persona_to_bitable.py","aliases":["product_commerce:collect_jd_persona_to_bitable.py"],"cdp_port":22065,"login_mode":"C","required_cookie_domains":["jd.com"],"credential_env_names":["JD_PASSWORD"]},
"product_commerce:collect_persona_to_bitable.py": {"script_id":"product_commerce:collect_persona_to_bitable.py","state_key":"product_commerce:collect_persona_to_bitable.py","aliases":["product_commerce:collect_persona_to_bitable.py"],"cdp_port":22066},
"product_commerce:collect_sycm_market_rank.py": {"script_id":"product_commerce:collect_sycm_market_rank.py","state_key":"product_commerce:collect_sycm_market_rank.py","aliases":["product_commerce:collect_sycm_market_rank.py"],"cdp_port":22067,"login_mode":"C","required_cookie_domains":["taobao.com"],"credential_env_names":["SYCM_ACCOUNT","SYCM_PASSWORD"]},
"product.import.daily": {"script_id":"product_commerce:commands/import_daily.py","state_key":"product_commerce:commands/import_daily.py","aliases":["product_commerce:commands/import_daily.py"],"cdp_port":22068},
"product.alert.run": {"script_id":"product_commerce:commands/run_alerts.py","state_key":"product_commerce:commands/run_alerts.py","aliases":["product_commerce:commands/run_alerts.py"],"cdp_port":22069},
"product_commerce:db/sync_dim_style.py": {"script_id":"product_commerce:db/sync_dim_style.py","state_key":"product_commerce:db/sync_dim_style.py","aliases":["product_commerce:db/sync_dim_style.py"],"cdp_port":22070},
"product_commerce:db/sync_sku_master.py": {"script_id":"product_commerce:db/sync_sku_master.py","state_key":"product_commerce:db/sync_sku_master.py","aliases":["product_commerce:db/sync_sku_master.py"],"cdp_port":22071},
"product_commerce:dy_audience_profile_collect.py": {"script_id":"product_commerce:dy_audience_profile_collect.py","state_key":"product_commerce:dy_audience_profile_collect.py","aliases":["product_commerce:dy_audience_profile_collect.py"],"cdp_port":22072,"login_mode":"A","required_cookie_domains":["jinritemai.com"]},
"product_commerce:dy_product_scraping.py": {"script_id":"product_commerce:dy_product_scraping.py","state_key":"product_commerce:dy_product_scraping.py","aliases":["product_commerce:dy_product_scraping.py"],"cdp_port":22073,"login_mode":"A","required_cookie_domains":["jinritemai.com"]},
"product_commerce:erp_login_product_analysis.py": {"script_id":"product_commerce:erp_login_product_analysis.py","state_key":"product_commerce:erp_login_product_analysis.py","aliases":["product_commerce:erp_login_product_analysis.py"],"cdp_port":22074},
"product_commerce:export_bitable_records.py": {"script_id":"product_commerce:export_bitable_records.py","state_key":"product_commerce:export_bitable_records.py","aliases":["product_commerce:export_bitable_records.py"],"cdp_port":22075},
"product_commerce:import_product_daily.py": {"script_id":"product_commerce:import_product_daily.py","state_key":"product_commerce:import_product_daily.py","aliases":["product_commerce:import_product_daily.py"],"cdp_port":22076},
"product_commerce:import_product_reviews.py": {"script_id":"product_commerce:import_product_reviews.py","state_key":"product_commerce:import_product_reviews.py","aliases":["product_commerce:import_product_reviews.py"],"cdp_port":22077},
"product_commerce:insert_bitable_records.py": {"script_id":"product_commerce:insert_bitable_records.py","state_key":"product_commerce:insert_bitable_records.py","aliases":["product_commerce:insert_bitable_records.py"],"cdp_port":22078},
"product_commerce:jd_main_image_collector.py": {"script_id":"product_commerce:jd_main_image_collector.py","state_key":"product_commerce:jd_main_image_collector.py","aliases":["product_commerce:jd_main_image_collector.py"],"cdp_port":22079,"login_mode":"C","required_cookie_domains":["jd.com"],"credential_env_names":["JD_PASSWORD"]},
"product_commerce:jd_product_data_collector.py": {"script_id":"product_commerce:jd_product_data_collector.py","state_key":"product_commerce:jd_product_data_collector.py","aliases":["product_commerce:jd_product_data_collector.py"],"cdp_port":22080,"login_mode":"C","required_cookie_domains":["jd.com"],"credential_env_names":["JD_PASSWORD"]},
"product_commerce:jd_self_inventory_sales_collector.py": {"script_id":"product_commerce:jd_self_inventory_sales_collector.py","state_key":"product_commerce:jd_self_inventory_sales_collector.py","aliases":["product_commerce:jd_self_inventory_sales_collector.py"],"cdp_port":22081},
"product_commerce:market_rank_product_import.py": {"script_id":"product_commerce:market_rank_product_import.py","state_key":"product_commerce:market_rank_product_import.py","aliases":["product_commerce:market_rank_product_import.py"],"cdp_port":22131},
"product.daily.orchestrate": {"script_id":"product_commerce:orchestrate_daily_collection.py","state_key":"product_commerce:orchestrate_daily_collection.py","aliases":["product_commerce:orchestrate_daily_collection.py"],"cdp_port":22082},
"product.market_rank.orchestrate": {"script_id":"product_commerce:orchestrate_market_rank_collection.py","state_key":"product_commerce:orchestrate_market_rank_collection.py","aliases":["product_commerce:orchestrate_market_rank_collection.py"],"cdp_port":22083},
"product.review.orchestrate": {"script_id":"product_commerce:orchestrate_review_collection.py","state_key":"product_commerce:orchestrate_review_collection.py","aliases":["product_commerce:orchestrate_review_collection.py"],"cdp_port":22084},
"product_commerce:reapply_erp_override.py": {"script_id":"product_commerce:reapply_erp_override.py","state_key":"product_commerce:reapply_erp_override.py","aliases":["product_commerce:reapply_erp_override.py"],"cdp_port":22085},
"product_commerce:rebuild_market_rank_documents.py": {"script_id":"product_commerce:rebuild_market_rank_documents.py","state_key":"product_commerce:rebuild_market_rank_documents.py","aliases":["product_commerce:rebuild_market_rank_documents.py"],"cdp_port":22086},
"product_commerce:run_alerts_with_retry.py": {"script_id":"product_commerce:run_alerts_with_retry.py","state_key":"product_commerce:run_alerts_with_retry.py","aliases":["product_commerce:run_alerts_with_retry.py"],"cdp_port":22087},
"product.persona.collect": {"script_id":"product_commerce:run_daily_persona.py","state_key":"product_commerce:run_daily_persona.py","aliases":["product_commerce:run_daily_persona.py"],"cdp_port":22088},
"product.main_image.collect_jd": {"script_id":"product_commerce:run_weekly_jd_main_image.py","state_key":"product_commerce:run_weekly_jd_main_image.py","aliases":["product_commerce:run_weekly_jd_main_image.py"],"cdp_port":22089},
"product.main_image.collect_tmall": {"script_id":"product_commerce:run_weekly_main_image.py","state_key":"product_commerce:run_weekly_main_image.py","aliases":["product_commerce:run_weekly_main_image.py"],"cdp_port":22090},
"product.sales_sheet.sync": {"script_id":"product_commerce:sync_monthly_sales_sheet.py","state_key":"product_commerce:sync_monthly_sales_sheet.py","aliases":["product_commerce:sync_monthly_sales_sheet.py"],"cdp_port":22136,"login_mode":"C","required_cookie_domains":["erp321.com"]},
"product_commerce:scripts/insert_jd_main_image_records.py": {"script_id":"product_commerce:scripts/insert_jd_main_image_records.py","state_key":"product_commerce:scripts/insert_jd_main_image_records.py","aliases":["product_commerce:scripts/insert_jd_main_image_records.py"],"cdp_port":22091},
"product_commerce:scripts/insert_main_image_records.py": {"script_id":"product_commerce:scripts/insert_main_image_records.py","state_key":"product_commerce:scripts/insert_main_image_records.py","aliases":["product_commerce:scripts/insert_main_image_records.py"],"cdp_port":22092},
"product_commerce:taobao_dmp_item_crowd_insight_screenshots.py": {"script_id":"product_commerce:taobao_dmp_item_crowd_insight_screenshots.py","state_key":"product_commerce:taobao_dmp_item_crowd_insight_screenshots.py","aliases":["product_commerce:taobao_dmp_item_crowd_insight_screenshots.py"],"cdp_port":22093,"login_mode":"C","required_cookie_domains":["taobao.com"],"credential_env_names":["SYCM_ACCOUNT","SYCM_PASSWORD"]},
"product_commerce:taobao_sycm_collect.py": {"script_id":"product_commerce:taobao_sycm_collect.py","state_key":"product_commerce:taobao_sycm_collect.py","aliases":["product_commerce:taobao_sycm_collect.py"],"cdp_port":22094,"login_mode":"C","required_cookie_domains":["taobao.com"],"credential_env_names":["SYCM_ACCOUNT","SYCM_PASSWORD"]},
"product_commerce:taobao_sycm_collect_backfill.py": {"script_id":"product_commerce:taobao_sycm_collect_backfill.py","state_key":"product_commerce:taobao_sycm_collect_backfill.py","aliases":["product_commerce:taobao_sycm_collect_backfill.py"],"cdp_port":22095,"login_mode":"C","required_cookie_domains":["taobao.com"],"credential_env_names":["SYCM_ACCOUNT","SYCM_PASSWORD"]},
"product_commerce:taobao_sycm_products.py": {"script_id":"product_commerce:taobao_sycm_products.py","state_key":"product_commerce:taobao_sycm_products.py","aliases":["product_commerce:taobao_sycm_products.py"],"cdp_port":22096,"login_mode":"C","required_cookie_domains":["taobao.com"],"credential_env_names":["SYCM_ACCOUNT","SYCM_PASSWORD"]},
"product_commerce:taobao_wanxiang_ai_creative_report.py": {"script_id":"product_commerce:taobao_wanxiang_ai_creative_report.py","state_key":"product_commerce:taobao_wanxiang_ai_creative_report.py","aliases":["product_commerce:taobao_wanxiang_ai_creative_report.py"],"cdp_port":22097,"login_mode":"C","required_cookie_domains":["taobao.com","alimama.com"],"credential_env_names":["SYCM_ACCOUNT","SYCM_PASSWORD"]},
"product_commerce:upload_video_to_guanghe.py": {"script_id":"product_commerce:upload_video_to_guanghe.py","state_key":"product_commerce:upload_video_to_guanghe.py","aliases":["product_commerce:upload_video_to_guanghe.py"],"cdp_port":22098},
"product_commerce:vendors/dy-data-flow/dy_store_competitor_store_scraping.py": {"script_id":"product_commerce:vendors/dy-data-flow/dy_store_competitor_store_scraping.py","state_key":"product_commerce:vendors/dy-data-flow/dy_store_competitor_store_scraping.py","aliases":["product_commerce:vendors/dy-data-flow/dy_store_competitor_store_scraping.py"],"cdp_port":22099,"login_mode":"A","required_cookie_domains":["jinritemai.com"]},
"product_commerce:vendors/jd-data-flow/jd_data_collector.py": {"script_id":"product_commerce:vendors/jd-data-flow/jd_data_collector.py","state_key":"product_commerce:vendors/jd-data-flow/jd_data_collector.py","aliases":["product_commerce:vendors/jd-data-flow/jd_data_collector.py"],"cdp_port":22100,"login_mode":"B","required_cookie_domains":["jd.com"],"credential_env_names":["JD_PASSWORD"]},
"product_commerce:vendors/jd-data-flow/jd_peer_product_data_collector.py": {"script_id":"product_commerce:vendors/jd-data-flow/jd_peer_product_data_collector.py","state_key":"product_commerce:vendors/jd-data-flow/jd_peer_product_data_collector.py","aliases":["product_commerce:vendors/jd-data-flow/jd_peer_product_data_collector.py"],"cdp_port":22101},
"product_commerce:vendors/jd-data-flow/jd_product_data_collector.py": {"script_id":"product_commerce:vendors/jd-data-flow/jd_product_data_collector.py","state_key":"product_commerce:vendors/jd-data-flow/jd_product_data_collector.py","aliases":["product_commerce:vendors/jd-data-flow/jd_product_data_collector.py"],"cdp_port":22102,"login_mode":"C","required_cookie_domains":["jd.com"],"credential_env_names":["JD_PASSWORD"]},
"product_commerce:weekly_aggregate.py": {"script_id":"product_commerce:weekly_aggregate.py","state_key":"product_commerce:weekly_aggregate.py","aliases":["product_commerce:weekly_aggregate.py"],"cdp_port":22103},
"shop.douyin_price_appeal": {"script_id":"shop_intelligence:collectors/dy_store_competitor_store_scraping.py","state_key":"shop_intelligence:collectors/dy_store_competitor_store_scraping.py","aliases":["shop_intelligence:collectors/dy_store_competitor_store_scraping.py"],"cdp_port":22104,"login_mode":"A","required_cookie_domains":["jinritemai.com"]},
"shop_intelligence:collectors/jd_data_collector.py": {"script_id":"shop_intelligence:collectors/jd_data_collector.py","state_key":"shop_intelligence:collectors/jd_data_collector.py","aliases":["shop_intelligence:collectors/jd_data_collector.py"],"cdp_port":22105,"login_mode":"B","required_cookie_domains":["jd.com"],"credential_env_names":["JD_PASSWORD"]},
"shop_intelligence:collectors/jd_peer_store_data_collector.py": {"script_id":"shop_intelligence:collectors/jd_peer_store_data_collector.py","state_key":"shop_intelligence:collectors/jd_peer_store_data_collector.py","aliases":["shop_intelligence:collectors/jd_peer_store_data_collector.py"],"cdp_port":22106,"login_mode":"B","required_cookie_domains":["jd.com"],"credential_env_names":["JD_PASSWORD"]},
"shop.jd_self_operated.collect_brand": {"script_id":"shop_intelligence:collectors/jd_self_operated_brand_daily.py","state_key":"shop_intelligence:collectors/jd_self_operated_brand_daily.py","aliases":["shop_intelligence:collectors/jd_self_operated_brand_daily.py"],"cdp_port":22132,"login_mode":"C","required_cookie_domains":["jd.com"],"credential_env_names":["JD_SELF_OPERATED_ACCOUNT","JD_SELF_OPERATED_PASSWORD"]},
"shop.jd_self_operated.collect_product": {"script_id":"shop_intelligence:collectors/jd_self_operated_product_daily.py","state_key":"shop_intelligence:collectors/jd_self_operated_product_daily.py","aliases":["shop_intelligence:collectors/jd_self_operated_product_daily.py"],"cdp_port":22133,"login_mode":"C","required_cookie_domains":["jd.com"],"credential_env_names":["JD_SELF_OPERATED_ACCOUNT","JD_SELF_OPERATED_PASSWORD"]},
"shop_intelligence:collectors/taobao_sycm.py": {"script_id":"shop_intelligence:collectors/taobao_sycm.py","state_key":"shop_intelligence:collectors/taobao_sycm.py","aliases":["shop_intelligence:collectors/taobao_sycm.py"],"cdp_port":22107,"login_mode":"C","required_cookie_domains":["taobao.com"],"credential_env_names":["SYCM_ACCOUNT","SYCM_PASSWORD"]},
"shop.competitor.collect": {"script_id":"shop_intelligence:runners/run_peer_store.py","state_key":"shop_intelligence:runners/run_peer_store.py","aliases":["shop_intelligence:runners/run_peer_store.py"],"cdp_port":22108},
"shop_intelligence:runners/run_jd_self_operated_daily.py": {"script_id":"shop_intelligence:runners/run_jd_self_operated_daily.py","state_key":"shop_intelligence:runners/run_jd_self_operated_daily.py","aliases":["shop_intelligence:runners/run_jd_self_operated_daily.py"],"cdp_port":22134},
"shop.metrics.collect": {"script_id":"shop_intelligence:runners/run_shop.py","state_key":"shop_intelligence:runners/run_shop.py","aliases":["shop_intelligence:runners/run_shop.py"],"cdp_port":22109},
"shop_intelligence:scripts/remove_scheduler.ps1": {"script_id":"shop_intelligence:scripts/remove_scheduler.ps1","state_key":"shop_intelligence:scripts/remove_scheduler.ps1","aliases":["shop_intelligence:scripts/remove_scheduler.ps1"],"cdp_port":22110},
"shop_intelligence:scripts/setup_scheduler.ps1": {"script_id":"shop_intelligence:scripts/setup_scheduler.ps1","state_key":"shop_intelligence:scripts/setup_scheduler.ps1","aliases":["shop_intelligence:scripts/setup_scheduler.ps1"],"cdp_port":22111},
"supply_chain:orchestrator/mcp_workflow.py": {"script_id":"supply_chain:orchestrator/mcp_workflow.py","state_key":"supply_chain:orchestrator/mcp_workflow.py","aliases":["supply_chain:orchestrator/mcp_workflow.py"],"cdp_port":22112},
"supply_chain:orchestrator/monitor.py": {"script_id":"supply_chain:orchestrator/monitor.py","state_key":"supply_chain:orchestrator/monitor.py","aliases":["supply_chain:orchestrator/monitor.py"],"cdp_port":22113},
"supply_chain:orchestrator/runner.py": {"script_id":"supply_chain:orchestrator/runner.py","state_key":"supply_chain:orchestrator/runner.py","aliases":["supply_chain:orchestrator/runner.py"],"cdp_port":22114},
"supply_chain:orchestrator/scripts/ProductReplenishment.py": {"script_id":"supply_chain:orchestrator/scripts/ProductReplenishment.py","state_key":"supply_chain:orchestrator/scripts/ProductReplenishment.py","aliases":["supply_chain:orchestrator/scripts/ProductReplenishment.py"],"cdp_port":22115,"login_mode":"C","required_cookie_domains":["erp321.com"],"required_cookie_names":["u_id","u_co_id","ASP.NET_SessionId",".ASPXAUTH","token"],"credential_env_names":["GYXX_SUPPLY_ERP_PASSWORD"]},
"supply_chain:orchestrator/scripts/PurchaseConfirmation.py": {"script_id":"supply_chain:orchestrator/scripts/PurchaseConfirmation.py","state_key":"supply_chain:orchestrator/scripts/PurchaseConfirmation.py","aliases":["supply_chain:orchestrator/scripts/PurchaseConfirmation.py"],"cdp_port":22116,"login_mode":"C","required_cookie_domains":["erp321.com"],"required_cookie_names":["ASP.NET_SessionId",".ASPXAUTH","token"],"credential_env_names":["GYXX_SUPPLY_ERP_PASSWORD"]},
"supply_chain:orchestrator/scripts/PurchaseOrderUpdate.py": {"script_id":"supply_chain:orchestrator/scripts/PurchaseOrderUpdate.py","state_key":"supply_chain:orchestrator/scripts/PurchaseOrderUpdate.py","aliases":["supply_chain:orchestrator/scripts/PurchaseOrderUpdate.py"],"cdp_port":22117,"login_mode":"C","required_cookie_domains":["erp321.com"],"required_cookie_names":["ASP.NET_SessionId",".ASPXAUTH","token"],"credential_env_names":["GYXX_SUPPLY_ERP_PASSWORD"]},
"supply_chain:orchestrator/scripts/batch_process.py": {"script_id":"supply_chain:orchestrator/scripts/batch_process.py","state_key":"supply_chain:orchestrator/scripts/batch_process.py","aliases":["supply_chain:orchestrator/scripts/batch_process.py"],"cdp_port":22118},
"supply_chain:orchestrator/scripts/collect_confirmation.ps1": {"script_id":"supply_chain:orchestrator/scripts/collect_confirmation.ps1","state_key":"supply_chain:orchestrator/scripts/collect_confirmation.ps1","aliases":["supply_chain:orchestrator/scripts/collect_confirmation.ps1"],"cdp_port":22119},
"supply_chain:orchestrator/scripts/collect_purchase_order_update.ps1": {"script_id":"supply_chain:orchestrator/scripts/collect_purchase_order_update.ps1","state_key":"supply_chain:orchestrator/scripts/collect_purchase_order_update.ps1","aliases":["supply_chain:orchestrator/scripts/collect_purchase_order_update.ps1"],"cdp_port":22120},
"supply_chain:orchestrator/scripts/collect_replenishment.ps1": {"script_id":"supply_chain:orchestrator/scripts/collect_replenishment.ps1","state_key":"supply_chain:orchestrator/scripts/collect_replenishment.ps1","aliases":["supply_chain:orchestrator/scripts/collect_replenishment.ps1"],"cdp_port":22121},
"supply_chain:orchestrator/scripts/insert_replenishment_bitable.py": {"script_id":"supply_chain:orchestrator/scripts/insert_replenishment_bitable.py","state_key":"supply_chain:orchestrator/scripts/insert_replenishment_bitable.py","aliases":["supply_chain:orchestrator/scripts/insert_replenishment_bitable.py"],"cdp_port":22122},
"supply_chain:orchestrator/scripts/send_card_notification.py": {"script_id":"supply_chain:orchestrator/scripts/send_card_notification.py","state_key":"supply_chain:orchestrator/scripts/send_card_notification.py","aliases":["supply_chain:orchestrator/scripts/send_card_notification.py"],"cdp_port":22123},
"supply_chain:orchestrator/scripts/trigger_purchase_order_update.py": {"script_id":"supply_chain:orchestrator/scripts/trigger_purchase_order_update.py","state_key":"supply_chain:orchestrator/scripts/trigger_purchase_order_update.py","aliases":["supply_chain:orchestrator/scripts/trigger_purchase_order_update.py"],"cdp_port":22124},
"supply_chain:orchestrator/sql/backfill_history.py": {"script_id":"supply_chain:orchestrator/sql/backfill_history.py","state_key":"supply_chain:orchestrator/sql/backfill_history.py","aliases":["supply_chain:orchestrator/sql/backfill_history.py"],"cdp_port":22125},
"supply.workflow.run": {"script_id":"supply_chain:run.py","state_key":"supply_chain:run.py","aliases":["supply_chain:run.py"],"cdp_port":22126},
"supply_chain:scripts/purchase-confirmation.bat": {"script_id":"supply_chain:scripts/purchase-confirmation.bat","state_key":"supply_chain:scripts/purchase-confirmation.bat","aliases":["supply_chain:scripts/purchase-confirmation.bat"],"cdp_port":22127},
"supply_chain:scripts/replenishment-alert.bat": {"script_id":"supply_chain:scripts/replenishment-alert.bat","state_key":"supply_chain:scripts/replenishment-alert.bat","aliases":["supply_chain:scripts/replenishment-alert.bat"],"cdp_port":22128},
"supply_chain:scripts/replenishment.bat": {"script_id":"supply_chain:scripts/replenishment.bat","state_key":"supply_chain:scripts/replenishment.bat","aliases":["supply_chain:scripts/replenishment.bat"],"cdp_port":22129}
},
"services": {
"feishu": "legacy",
"postgres": "cloud",
"hermes": "local",
"hermes_url": "http://127.0.0.1:8642/v1/chat/completions"
"hermes_url": "http://127.0.0.1:8642/v1/chat/completions",
"hermes_collector_url": "http://127.0.0.1:8643/v1/chat/completions",
"hermes_analyzer_gateway_url": "http://127.0.0.1:8642/v1",
"hermes_collector_gateway_url": "http://127.0.0.1:8643/v1"
}
}
config/schedules.json
+154 -27
View File
@@ -1,30 +1,157 @@
{
"schema_version": 1,
"timezone": "Asia/Shanghai",
"schedules": [
{"workflow_id":"content.metrics.daily","kind":"daily","at":"06:00"},
{"workflow_id":"content.cooperations.daily","kind":"daily","at":"09:00"},
{"workflow_id":"content.marketing_report.daily","kind":"daily","at":"10:00"},
{"workflow_id":"content.relogin.weekly","kind":"weekly","days":["Friday"],"at":"10:00"},
{"workflow_id":"content.self_operated.weekly","kind":"weekly","days":["Monday"],"at":"13:00"},
{"workflow_id":"content.comments.weekly","kind":"weekly","days":["Sunday"],"at":"12:00"},
{"workflow_id":"content.summary.weekly","kind":"weekly","days":["Tuesday"],"at":"10:00"},
{"workflow_id":"content.summary.monthly","kind":"monthly","day_of_month":1,"at":"08:00"},
{"workflow_id":"content.creator_report.monthly","kind":"monthly","day_of_month":1,"at":"08:30"},
{"workflow_id":"shop.metrics.weekly","kind":"weekly","days":["Monday"],"at":"12:00"},
{"workflow_id":"shop.competitor.weekly","kind":"weekly","days":["Monday"],"at":"12:30"},
{"workflow_id":"product.daily","kind":"daily","at":"08:40"},
{"workflow_id":"product.persona.daily","kind":"daily","at":"10:00"},
{"workflow_id":"product.import.daily","kind":"daily","at":"19:00"},
{"workflow_id":"product.alert.daily","kind":"daily","at":"23:00"},
{"workflow_id":"product.style_analysis.interval","kind":"interval_days","every_days":3,"anchor_date":"2026-07-25","at":"11:00"},
{"workflow_id":"product.main_image.jd.weekly","kind":"weekly","days":["Sunday"],"at":"08:30"},
{"workflow_id":"product.main_image.weekly","kind":"weekly","days":["Sunday"],"at":"09:30"},
{"workflow_id":"supply.replenishment_alert.daily","kind":"daily","at":"07:00"},
{"workflow_id":"supply.purchase_confirmation.daily","kind":"daily","at":"08:00"},
{"workflow_id":"supply.replenishment.weekly","kind":"weekly","days":["Monday"],"at":"08:00"}
]
{
"at": "22:00",
"business_date_offset_days": 0,
"enabled": true,
"kind": "daily",
"workflow_id": "content.metrics.daily"
},
{
"at": "09:00",
"kind": "daily",
"workflow_id": "content.cooperations.daily"
},
{
"at": "10:00",
"kind": "daily",
"workflow_id": "content.marketing_report.daily"
},
{
"at": "10:00",
"days": [
"Friday"
],
"kind": "weekly",
"workflow_id": "content.relogin.weekly"
},
{
"at": "12:00",
"days": [
"Sunday"
],
"kind": "weekly",
"workflow_id": "content.comments.weekly"
},
{
"at": "10:00",
"days": [
"Tuesday"
],
"kind": "weekly",
"workflow_id": "content.summary.weekly"
},
{
"at": "08:00",
"day_of_month": 1,
"kind": "monthly",
"workflow_id": "content.summary.monthly"
},
{
"at": "08:30",
"day_of_month": 1,
"kind": "monthly",
"workflow_id": "content.creator_report.monthly"
},
{
"at": "12:00",
"days": [
"Monday"
],
"kind": "weekly",
"workflow_id": "shop.metrics.weekly"
},
{
"at": "12:30",
"days": [
"Monday"
],
"kind": "weekly",
"workflow_id": "shop.competitor.weekly"
},
{
"at": [
"08:00",
"16:00",
"22:00"
],
"kind": "daily",
"workflow_id": "shop.douyin_price_appeal"
},
{
"at": "16:00",
"business_date_offset_days": -1,
"kind": "daily",
"workflow_id": "shop.jd_self_operated.daily"
},
{
"at": "08:40",
"business_date_offset_days": -1,
"kind": "daily",
"workflow_id": "product.daily"
},
{
"at": "10:00",
"kind": "daily",
"workflow_id": "product.persona.daily"
},
{
"at": "19:00",
"kind": "daily",
"workflow_id": "product.import.daily"
},
{
"at": "23:00",
"kind": "daily",
"workflow_id": "product.alert.daily"
},
{
"anchor_date": "2026-07-25",
"at": "11:00",
"every_days": 3,
"kind": "interval_days",
"workflow_id": "product.style_analysis.interval"
},
{
"at": "08:30",
"days": [
"Sunday"
],
"kind": "weekly",
"workflow_id": "product.main_image.weekly"
},
{
"at": "18:00",
"kind": "daily",
"workflow_id": "product.sales_sheet.daily"
},
{
"at": "10:00",
"days": [
"Monday"
],
"kind": "weekly",
"workflow_id": "product.market_rank"
},
{
"at": "07:00",
"kind": "daily",
"workflow_id": "supply.replenishment_alert.daily"
},
{
"at": "08:00",
"kind": "daily",
"workflow_id": "supply.purchase_confirmation.daily"
},
{
"at": "08:00",
"days": [
"Monday"
],
"kind": "weekly",
"workflow_id": "supply.replenishment.weekly"
}
],
"schema_version": 1,
"timezone": "Asia/Shanghai"
}
config/secrets.example.env
+3
View File
@@ -3,3 +3,6 @@ GYXX_POSTGRES_PASSWORD=
GYXX_FEISHU_APP_ID=
GYXX_FEISHU_APP_SECRET=
GYXX_HERMES_API_KEY=
GYXX_NOTIFICATION_RECIPIENT_OPEN_ID=
JD_SELF_OPERATED_ACCOUNT=
JD_SELF_OPERATED_PASSWORD=
config/source-manifests/content_marketing.json
+704 -884
View File
File diff suppressed because it is too large Load Diff
config/source-manifests/product_commerce.json
+1078 -970
View File
File diff suppressed because it is too large Load Diff
config/source-manifests/shop_intelligence.json
+374 -303
View File
@@ -1,305 +1,376 @@
{
"schema_version": 1,
"module": "shop_intelligence",
"source_project": "shop-data-flow",
"files": [
{
"source_relative_path": ".env.example",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/config.example.env",
"category": "config",
"source_sha256": "d0a005429c4363020681076afe4ec389712bd86b041271b8e9d52b977d79bfc1",
"target_sha256": "a1e2bad32d3a2eb6cec1c60e13bef1298f872365301c048a8e8df826c783c122",
"transformed": true
},
{
"source_relative_path": "adaptive_selectors.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/adaptive_selectors.py",
"category": "source",
"source_sha256": "7f60d6eedf0b9bf4aad0087305a86814dc9e2691128cce4010f9b3deff403bae",
"target_sha256": "9ee3b786a5eecb6877c31d583de87bf75e22326c9126d9b679d2d4460db70454",
"transformed": true
},
{
"source_relative_path": "collectors/__init__.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/collectors/__init__.py",
"category": "source",
"source_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"target_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"transformed": false
},
{
"source_relative_path": "collectors/dy_store_competitor_store_scraping.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/collectors/dy_store_competitor_store_scraping.py",
"category": "source",
"source_sha256": "009879a7dd341ca6ecbc8eb66f5700114c73bcaba3a53f159014a76232d0aa6b",
"target_sha256": "8d319196d56a52e084c375f872faf729d33fca400f4cabae901accb6c0bdde47",
"transformed": true
},
{
"source_relative_path": "collectors/jd_data_collector.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/collectors/jd_data_collector.py",
"category": "source",
"source_sha256": "6f00ae3b2b74e08d3c34c3b911ea819dfa4c6ce55bceef282afb3b2f565aa3ec",
"target_sha256": "1825f123b3c1604340bd95a79cad31cbaba23834d50dae69ce5b68068a5b055e",
"transformed": true
},
{
"source_relative_path": "collectors/jd_peer_store_data_collector.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/collectors/jd_peer_store_data_collector.py",
"category": "source",
"source_sha256": "4ebdb52df7b384d2bfdd2fc840d55d14e18864002c20a36685b63cee1966cf4c",
"target_sha256": "23f445aae6bcf41193db5286ce1a519d50b93e20a55f27f6df58315aafb30a08",
"transformed": true
},
{
"source_relative_path": "collectors/taobao_sycm.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/collectors/taobao_sycm.py",
"category": "source",
"source_sha256": "6978333bd60c1946e3dd8eeced1d0589461c4a26377511da7875f60bd0e31739",
"target_sha256": "6042dc251c0dbff92ca9eb8ed0a4db4322db5810e43fadf1cae79219067496fd",
"transformed": true
},
{
"source_relative_path": "config.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/config.py",
"category": "source",
"source_sha256": "2cfa101ed07643ad822f176e4b21bb72bf7ff32e3945837082811ea792a9e21b",
"target_sha256": "d9a8e32513071dacc8fd303901a3f665336d49e5dd8afe2ba3a0bc8f9dc19985",
"transformed": true
},
{
"source_relative_path": "db/__init__.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/db/__init__.py",
"category": "source",
"source_sha256": "e75ec31954ddf3c6a84e5915a6ff662027d097fdf7fba91aa5c84d048b8e960c",
"target_sha256": "e75ec31954ddf3c6a84e5915a6ff662027d097fdf7fba91aa5c84d048b8e960c",
"transformed": false
},
{
"source_relative_path": "db/db.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/db/db.py",
"category": "source",
"source_sha256": "bf600de53e60be0492426b6ffade60f75474d885263963b12b1bc4a42c03e216",
"target_sha256": "567df11eac1462111594668d66e1210fe1d1808f0f77250e557ad1cf5b66af44",
"transformed": true
},
{
"source_relative_path": "db/schema.sql",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/db/schema.sql",
"category": "sql",
"source_sha256": "5c3b90a70af4aa9e69039be8f3ec79a136ab5892e5d1ff31fc8a20d735a5dfe5",
"target_sha256": "5c3b90a70af4aa9e69039be8f3ec79a136ab5892e5d1ff31fc8a20d735a5dfe5",
"transformed": false
},
{
"source_relative_path": "lark_cli.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/lark_cli.py",
"category": "source",
"source_sha256": "434394c2864b44e612a34b16c242df2a5f89dbfb298505b22f2133af925d0233",
"target_sha256": "dd2c06c0e491f310e563b6203b078f41fd1cf1427b946332a2735df522cf13d1",
"transformed": true
},
{
"source_relative_path": "pyproject.toml",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/dependencies.toml",
"category": "dependency",
"source_sha256": "002035ffb503798d48241f962216c5baa460de4ee8a1d7d696a3d6ba6afb2975",
"target_sha256": "002035ffb503798d48241f962216c5baa460de4ee8a1d7d696a3d6ba6afb2975",
"transformed": false
},
{
"source_relative_path": "runners/__init__.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/runners/__init__.py",
"category": "source",
"source_sha256": "79b41ece107032dcaa3a7b07df8454673fae8d09f763052ec12c81e3be5126d0",
"target_sha256": "79b41ece107032dcaa3a7b07df8454673fae8d09f763052ec12c81e3be5126d0",
"transformed": false
},
{
"source_relative_path": "runners/run_peer_store.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/runners/run_peer_store.py",
"category": "launcher",
"source_sha256": "363a04b516dd6b011f07bf34feb480ccd0f1da46cb7d9512653ea7d5f4f7a3a4",
"target_sha256": "5db5365ba1a69ec325cc9e5b90aa77609045df84d5b3b8956e1718faaf30be55",
"transformed": true
},
{
"source_relative_path": "runners/run_shop.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/runners/run_shop.py",
"category": "launcher",
"source_sha256": "5c5a37fb230f531e55563e75b7cf361a3cba190707f01f516e244ac643c73027",
"target_sha256": "f12d29bd72533780270393eae855412b87f716f2540b05fed82da097821d20cb",
"transformed": true
},
{
"source_relative_path": "runners/utils.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/runners/utils.py",
"category": "source",
"source_sha256": "3bdf504593634d467524d06322d7f0ca20ab7bdf8c115573c154b8f5559c1cad",
"target_sha256": "43340178902f3531d3d68ae3dc51b2dc8117003b43c6172f32a53f04b3f6f2cb",
"transformed": true
},
{
"source_relative_path": "scripts/remove_scheduler.ps1",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/scripts/remove_scheduler.ps1",
"category": "launcher",
"source_sha256": "0532c87a6de5f7f7f65796b8c2077126091cda7d4f065c6015624599b6df1ee6",
"target_sha256": "c376ac34da5aa777634ccea5a5d2e5718c69309b17a0219cac5c19a3c6d46ca4",
"transformed": true
},
{
"source_relative_path": "scripts/setup_scheduler.ps1",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/scripts/setup_scheduler.ps1",
"category": "launcher",
"source_sha256": "8ff5418b494886ceb76acb9c8dcbccf85183f1e85fb18426c0053802ae5eb26b",
"target_sha256": "88915b6d75c62c61d13bc39d7fb959ea4b308068b9edab08db43c5c18f295f69",
"transformed": true
},
{
"source_relative_path": "writers/__init__.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/writers/__init__.py",
"category": "source",
"source_sha256": "d830cb8dd53601d06abb9481c907472bcefcadb8b65ceb0356f5baf037c3af10",
"target_sha256": "d830cb8dd53601d06abb9481c907472bcefcadb8b65ceb0356f5baf037c3af10",
"transformed": false
},
{
"source_relative_path": "writers/peer_store_writer.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/writers/peer_store_writer.py",
"category": "source",
"source_sha256": "71bf69c423bf3dd1f3d68d4e33f7024fbe0e446567d2cc3b25ec0cb71f59ab7e",
"target_sha256": "075b2862d486f43a406a3be58439813b60f3354ebba244f4d0be25a82ff618d7",
"transformed": true
},
{
"source_relative_path": "writers/shop_base_writer.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/writers/shop_base_writer.py",
"category": "source",
"source_sha256": "01aa18955a0ec92f4fb745df916e5b7f1c69646c8a194e29bc0042120f59c722",
"target_sha256": "3ec3d71af95e7d50fa77d7ec684f1e86634eea0eed6be6b5728fe19a3f04ce0d",
"transformed": true
}
],
"entrypoints": [
{
"source_relative_path": "runners/run_shop.py",
"target_relative_path": "runners/run_shop.py",
"classification": "scheduled",
"workflows": [
"shop.metrics.weekly"
]
},
{
"source_relative_path": "runners/run_peer_store.py",
"target_relative_path": "runners/run_peer_store.py",
"classification": "scheduled",
"workflows": [
"shop.competitor.weekly"
]
},
{
"source_relative_path": "collectors/jd_data_collector.py",
"target_relative_path": "collectors/jd_data_collector.py",
"classification": "internal",
"workflows": [
"shop.metrics.weekly"
]
},
{
"source_relative_path": "collectors/jd_peer_store_data_collector.py",
"target_relative_path": "collectors/jd_peer_store_data_collector.py",
"classification": "internal",
"workflows": [
"shop.competitor.weekly"
]
},
{
"source_relative_path": "collectors/dy_store_competitor_store_scraping.py",
"target_relative_path": "collectors/dy_store_competitor_store_scraping.py",
"classification": "internal",
"workflows": [
"shop.metrics.weekly",
"shop.competitor.weekly"
]
},
{
"source_relative_path": "collectors/taobao_sycm.py",
"target_relative_path": "collectors/taobao_sycm.py",
"classification": "internal",
"workflows": [
"shop.metrics.weekly"
]
},
{
"source_relative_path": "scripts/setup_scheduler.ps1",
"target_relative_path": "scripts/setup_scheduler.ps1",
"classification": "maintenance",
"workflows": [
]
},
{
"source_relative_path": "scripts/remove_scheduler.ps1",
"target_relative_path": "scripts/remove_scheduler.ps1",
"classification": "maintenance",
"workflows": [
]
}
],
"intentionally_excluded": [
{
"pattern": ".git/**",
"reason": "runtime, generated, sensitive, or non-business material is not copied into module source"
},
{
"pattern": ".venv/**",
"reason": "runtime, generated, sensitive, or non-business material is not copied into module source"
},
{
"pattern": "**/__pycache__/**",
"reason": "runtime, generated, sensitive, or non-business material is not copied into module source"
},
{
"pattern": ".env",
"reason": "runtime, generated, sensitive, or non-business material is not copied into module source"
},
{
"pattern": "**/*cookies*",
"reason": "runtime, generated, sensitive, or non-business material is not copied into module source"
},
{
"pattern": "**/*profile*/**",
"reason": "runtime, generated, sensitive, or non-business material is not copied into module source"
},
{
"pattern": "**/data/**",
"reason": "runtime, generated, sensitive, or non-business material is not copied into module source"
},
{
"pattern": "**/logs/**",
"reason": "runtime, generated, sensitive, or non-business material is not copied into module source"
},
{
"pattern": "debug/**",
"reason": "runtime, generated, sensitive, or non-business material is not copied into module source"
},
{
"pattern": "uv.lock",
"reason": "runtime, generated, sensitive, or non-business material is not copied into module source"
},
{
"pattern": "README.md",
"reason": "runtime, generated, sensitive, or non-business material is not copied into module source"
},
{
"pattern": "CLAUDE.md",
"reason": "runtime, generated, sensitive, or non-business material is not copied into module source"
},
{
"pattern": "AGENTS.md",
"reason": "runtime, generated, sensitive, or non-business material is not copied into module source"
}
],
"snapshot": "current-filesystem"
"schema_version": 1,
"module": "shop_intelligence",
"source_project": "shop-data-flow",
"files": [
{
"source_relative_path": ".env.example",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/config.example.env",
"category": "config",
"source_sha256": "d0a005429c4363020681076afe4ec389712bd86b041271b8e9d52b977d79bfc1",
"target_sha256": "2d5c9bd6f7749bee87c3716c0d934883f0dcc3548ccf0df75441911b882a98ad",
"transformed": true
},
{
"source_relative_path": "adaptive_selectors.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/adaptive_selectors.py",
"category": "source",
"source_sha256": "7f60d6eedf0b9bf4aad0087305a86814dc9e2691128cce4010f9b3deff403bae",
"target_sha256": "3417782d86c96939ca7fc682c8d9e85a8f4b156f25df802f16a168a3fbe421e6",
"transformed": true
},
{
"source_relative_path": "collectors/__init__.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/collectors/__init__.py",
"category": "source",
"source_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"target_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"transformed": false
},
{
"source_relative_path": "collectors/dy_store_competitor_store_scraping.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/collectors/dy_store_competitor_store_scraping.py",
"category": "source",
"source_sha256": "009879a7dd341ca6ecbc8eb66f5700114c73bcaba3a53f159014a76232d0aa6b",
"target_sha256": "e0b9664ad71c5c44132a730ccb274e30e6749229e912c5df816105aa7905405b",
"transformed": true
},
{
"source_relative_path": "collectors/jd_data_collector.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/collectors/jd_data_collector.py",
"category": "source",
"source_sha256": "6f00ae3b2b74e08d3c34c3b911ea819dfa4c6ce55bceef282afb3b2f565aa3ec",
"target_sha256": "caaef9f448c1fd803ee7aac5149d5ddf49890fc793c1d4d6e1ef58f8404e0be9",
"transformed": true
},
{
"source_relative_path": "collectors/jd_peer_store_data_collector.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/collectors/jd_peer_store_data_collector.py",
"category": "source",
"source_sha256": "4ebdb52df7b384d2bfdd2fc840d55d14e18864002c20a36685b63cee1966cf4c",
"target_sha256": "d0a75b08ca50b746dd28b609d15d769b1137a8f6aba575a253a0fc3988bbf34b",
"transformed": true
},
{
"source_relative_path": "collectors/jd_self_operated_brand_daily.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/collectors/jd_self_operated_brand_daily.py",
"category": "source",
"source_sha256": "f03e6e6a65faa1cbafae57b641fe2efec9aae8ba685afd1c0cf63680adf7588e",
"target_sha256": "9190b09f739461fa7d231391754ba4373093ec8dede9df77ae06d866306da1a6",
"transformed": true
},
{
"source_relative_path": "collectors/jd_self_operated_product_daily.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/collectors/jd_self_operated_product_daily.py",
"category": "source",
"source_sha256": "92f5c8619eede17c25bad8482b8821ff7a7dac3ff649e4a02d69c21f8dc89de1",
"target_sha256": "8c28a8684bf8812f87e8ca7764fed3d6b026cf107d69dbac6507e0fb4630f54d",
"transformed": true
},
{
"source_relative_path": "collectors/taobao_sycm.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/collectors/taobao_sycm.py",
"category": "source",
"source_sha256": "6978333bd60c1946e3dd8eeced1d0589461c4a26377511da7875f60bd0e31739",
"target_sha256": "b832f7421f0782401850c3bf6d6c7c1c827af10230c7c4b0b12db303edac3d61",
"transformed": true
},
{
"source_relative_path": "config.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/config.py",
"category": "source",
"source_sha256": "2cfa101ed07643ad822f176e4b21bb72bf7ff32e3945837082811ea792a9e21b",
"target_sha256": "0e8819fc9ec12fabbe91b25f51db895ca95f4a725b69bde0d6b807da5b8d4416",
"transformed": true
},
{
"source_relative_path": "db/__init__.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/db/__init__.py",
"category": "source",
"source_sha256": "ffc89b78ff0081a129e04eb6de4985ab0a2a4613c9e8a1d8b2750d8c17e8bc05",
"target_sha256": "ffc89b78ff0081a129e04eb6de4985ab0a2a4613c9e8a1d8b2750d8c17e8bc05",
"transformed": false
},
{
"source_relative_path": "db/db.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/db/db.py",
"category": "source",
"source_sha256": "168b6f87c1827b267f763ddce7eb15d0c8d652c8c0c7af18dcd0c6c7da5cddd1",
"target_sha256": "b8f15975605e2d9b75d25ac09a67768d4583401c1365288de078a97d0d3f3abf",
"transformed": true
},
{
"source_relative_path": "db/schema.sql",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/db/schema.sql",
"category": "sql",
"source_sha256": "73230590bde7ae6359c02c82c2db096c0d3faa55b374e30348c47d7b40ea498b",
"target_sha256": "a0aa2a22c8fdbb1f748dc7b206fe06a43f10899ca7036e433cdd91d7a1f5cba9",
"transformed": true
},
{
"source_relative_path": "lark_cli.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/lark_cli.py",
"category": "source",
"source_sha256": "434394c2864b44e612a34b16c242df2a5f89dbfb298505b22f2133af925d0233",
"target_sha256": "1b227902a8ebe05ada4f0bda7c2b69bdfd639a4f5b339c949df0b656c11e6144",
"transformed": true
},
{
"source_relative_path": "pyproject.toml",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/dependencies.toml",
"category": "dependency",
"source_sha256": "002035ffb503798d48241f962216c5baa460de4ee8a1d7d696a3d6ba6afb2975",
"target_sha256": "002035ffb503798d48241f962216c5baa460de4ee8a1d7d696a3d6ba6afb2975",
"transformed": false
},
{
"source_relative_path": "tests/test_jd_self_operated_brand_daily.py",
"target_relative_path": "tests/modules/shop_intelligence/test_jd_self_operated_brand_daily.py",
"category": "test",
"source_sha256": "71186dd318bd582648b43b72641b91c540c1e6908b28f2b9591f70c3590f22e8",
"target_sha256": "8dddfb16c48556b00f59b00477407e7d289e5f893ecff60255b04829b18a4fb5",
"transformed": true
},
{
"source_relative_path": "tests/test_jd_self_operated_product_daily.py",
"target_relative_path": "tests/modules/shop_intelligence/test_jd_self_operated_product_daily.py",
"category": "test",
"source_sha256": "2b8d404951bea91438dc7fa85ad525bf0235cd758de14c35cc17f0a3f6c4b06b",
"target_sha256": "21667a07f324da3755d70c67cba4b2d50f922614bee41464a52ae154aff56fc2",
"transformed": true
},
{
"source_relative_path": "tests/test_run_jd_self_operated_daily.py",
"target_relative_path": "tests/modules/shop_intelligence/test_run_jd_self_operated_daily.py",
"category": "test",
"source_sha256": "8883b9f6cf6313dd49422a1329307288a240dee6018ed40c71683969b565885e",
"target_sha256": "1fafc65993c4c0b853f60ae0bad2ca53d2b396a7c7a1b602be7830ec2a5ba5f5",
"transformed": true
},
{
"source_relative_path": "runners/__init__.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runners/__init__.py",
"category": "source",
"source_sha256": "79b41ece107032dcaa3a7b07df8454673fae8d09f763052ec12c81e3be5126d0",
"target_sha256": "79b41ece107032dcaa3a7b07df8454673fae8d09f763052ec12c81e3be5126d0",
"transformed": false
},
{
"source_relative_path": "runners/run_peer_store.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runners/run_peer_store.py",
"category": "launcher",
"source_sha256": "363a04b516dd6b011f07bf34feb480ccd0f1da46cb7d9512653ea7d5f4f7a3a4",
"target_sha256": "a603b281f5db5c43bc022c47f7dd3c8a381a33980f6974c5c30aac5c06431917",
"transformed": true
},
{
"source_relative_path": "runners/run_jd_self_operated_daily.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runners/run_jd_self_operated_daily.py",
"category": "launcher",
"source_sha256": "b0bad4bcb45f9566d004358796c9588d3e1f0714f5a9b523fae1d44a94b1745e",
"target_sha256": "a26baf610b91d86684366db5809ca031b737663b765fd8672b0addbf4a58a67d",
"transformed": true
},
{
"source_relative_path": "runners/run_shop.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runners/run_shop.py",
"category": "launcher",
"source_sha256": "7c5f50e44613fa4271d6b3d8dbc28d43c884a06a6a2cd04299f7155a9dc572e3",
"target_sha256": "94e0024de5f92a7e72d864f52f30474ca193feff93a370dcbb6d8a346590386e",
"transformed": true
},
{
"source_relative_path": "runners/utils.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runners/utils.py",
"category": "source",
"source_sha256": "3bdf504593634d467524d06322d7f0ca20ab7bdf8c115573c154b8f5559c1cad",
"target_sha256": "d3b8d86770401b12fbfc8855763e765c5599b97eddb6cd65ea49fe7b34ce2ce4",
"transformed": true
},
{
"source_relative_path": "writers/__init__.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/writers/__init__.py",
"category": "source",
"source_sha256": "d830cb8dd53601d06abb9481c907472bcefcadb8b65ceb0356f5baf037c3af10",
"target_sha256": "d830cb8dd53601d06abb9481c907472bcefcadb8b65ceb0356f5baf037c3af10",
"transformed": false
},
{
"source_relative_path": "writers/peer_store_writer.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/writers/peer_store_writer.py",
"category": "source",
"source_sha256": "71bf69c423bf3dd1f3d68d4e33f7024fbe0e446567d2cc3b25ec0cb71f59ab7e",
"target_sha256": "56b2d41af0b5ac19e189b5445b765776efdacea8c963bf0013ad377c399cc569",
"transformed": true
},
{
"source_relative_path": "writers/shop_base_writer.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/writers/shop_base_writer.py",
"category": "source",
"source_sha256": "f16f6dcb1ca028ca4ec1fe70d77127459656b4f9f6fc982c30bc426ad09ca3f5",
"target_sha256": "3bcd1dc039b25370d27cdbefa697a47349707ab94ff5e999c5ab25fa74168421",
"transformed": true
}
],
"entrypoints": [
{
"source_relative_path": "runners/run_shop.py",
"target_relative_path": "runners/run_shop.py",
"classification": "scheduled",
"workflows": [
"shop.metrics.weekly"
]
},
{
"source_relative_path": "runners/run_peer_store.py",
"target_relative_path": "runners/run_peer_store.py",
"classification": "scheduled",
"workflows": [
"shop.competitor.weekly"
]
},
{
"source_relative_path": "collectors/jd_data_collector.py",
"target_relative_path": "collectors/jd_data_collector.py",
"classification": "internal",
"workflows": [
"shop.metrics.weekly"
]
},
{
"source_relative_path": "collectors/jd_peer_store_data_collector.py",
"target_relative_path": "collectors/jd_peer_store_data_collector.py",
"classification": "internal",
"workflows": [
"shop.competitor.weekly"
]
},
{
"source_relative_path": "collectors/dy_store_competitor_store_scraping.py",
"target_relative_path": "collectors/dy_store_competitor_store_scraping.py",
"classification": "internal",
"workflows": [
"shop.metrics.weekly",
"shop.competitor.weekly",
"shop.douyin_price_appeal"
],
"commands": [
"shop.douyin_price_appeal"
]
},
{
"source_relative_path": "collectors/taobao_sycm.py",
"target_relative_path": "collectors/taobao_sycm.py",
"classification": "internal",
"workflows": [
"shop.metrics.weekly"
]
},
{
"source_relative_path": "collectors/jd_self_operated_brand_daily.py",
"target_relative_path": "collectors/jd_self_operated_brand_daily.py",
"classification": "manual",
"workflows": [
"shop.jd_self_operated.daily"
],
"commands": [
"shop.jd_self_operated.collect_brand"
]
},
{
"source_relative_path": "runners/run_jd_self_operated_daily.py",
"target_relative_path": "runners/run_jd_self_operated_daily.py",
"classification": "scheduled",
"workflows": [
"shop.jd_self_operated.daily"
]
},
{
"source_relative_path": "collectors/jd_self_operated_product_daily.py",
"target_relative_path": "collectors/jd_self_operated_product_daily.py",
"classification": "internal",
"workflows": [
"shop.jd_self_operated.daily"
]
}
],
"intentionally_excluded": [
{
"pattern": "scripts/setup_scheduler.ps1",
"reason": "historical Windows scheduler setup is archived under docs/history and is not production source"
},
{
"pattern": "scripts/remove_scheduler.ps1",
"reason": "historical Windows scheduler removal is archived under docs/history and is not production source"
},
{
"pattern": ".gitignore",
"reason": "source-project ignore rules are not module runtime source"
},
{
"pattern": ".learnings/**",
"reason": "local learning state is not runtime source"
},
{
"pattern": "docs/**",
"reason": "historical design and review documents are not runtime dependencies"
},
{
"pattern": "**/*storage_state*",
"reason": "browser storage state is sensitive mutable runtime data"
},
{
"pattern": ".git/**",
"reason": "runtime, generated, sensitive, or non-business material is not copied into module source"
},
{
"pattern": ".venv/**",
"reason": "runtime, generated, sensitive, or non-business material is not copied into module source"
},
{
"pattern": "**/__pycache__/**",
"reason": "runtime, generated, sensitive, or non-business material is not copied into module source"
},
{
"pattern": ".env",
"reason": "runtime, generated, sensitive, or non-business material is not copied into module source"
},
{
"pattern": "**/*cookies*",
"reason": "runtime, generated, sensitive, or non-business material is not copied into module source"
},
{
"pattern": "**/*profile*/**",
"reason": "runtime, generated, sensitive, or non-business material is not copied into module source"
},
{
"pattern": "**/data/**",
"reason": "runtime, generated, sensitive, or non-business material is not copied into module source"
},
{
"pattern": "**/logs/**",
"reason": "runtime, generated, sensitive, or non-business material is not copied into module source"
},
{
"pattern": "debug/**",
"reason": "runtime, generated, sensitive, or non-business material is not copied into module source"
},
{
"pattern": "uv.lock",
"reason": "runtime, generated, sensitive, or non-business material is not copied into module source"
},
{
"pattern": "README.md",
"reason": "runtime, generated, sensitive, or non-business material is not copied into module source"
},
{
"pattern": "CLAUDE.md",
"reason": "runtime, generated, sensitive, or non-business material is not copied into module source"
},
{
"pattern": "AGENTS.md",
"reason": "runtime, generated, sensitive, or non-business material is not copied into module source"
}
],
"snapshot": "current-filesystem"
}
config/source-manifests/supply_chain.json
+564 -533
View File
File diff suppressed because it is too large Load Diff
config/source-projects.json
+33
View File
@@ -0,0 +1,33 @@
{
"schema_version": 1,
"projects": [
{
"module": "content_marketing",
"source_project": "yingxiaoyunying",
"manifest": "config/source-manifests/content_marketing.json",
"root_env": "GYXX_SOURCE_CONTENT_MARKETING_ROOT",
"target_root": "src/gyxx_flow/modules/content_marketing"
},
{
"module": "product_commerce",
"source_project": "product-collector-analyze-flow",
"manifest": "config/source-manifests/product_commerce.json",
"root_env": "GYXX_SOURCE_PRODUCT_COMMERCE_ROOT",
"target_root": "src/gyxx_flow/modules/product_commerce"
},
{
"module": "shop_intelligence",
"source_project": "shop-data-flow",
"manifest": "config/source-manifests/shop_intelligence.json",
"root_env": "GYXX_SOURCE_SHOP_INTELLIGENCE_ROOT",
"target_root": "src/gyxx_flow/modules/shop_intelligence"
},
{
"module": "supply_chain",
"source_project": "auto-flow",
"manifest": "config/source-manifests/supply_chain.json",
"root_env": "GYXX_SOURCE_SUPPLY_CHAIN_ROOT",
"target_root": "src/gyxx_flow/modules/supply_chain"
}
]
}
config/workflows.json
+24 -31
View File
@@ -1,38 +1,31 @@
{
"schema_version": 2,
"schema_version": 3,
"workflows": [
{"id":"content.metrics.daily","module":"content_marketing","trigger":"scheduled","execution":{"entry":"data/tools/daily_run.bat"},"provenance":{"source_project":"content","task_name":"YingxiaoYunying_DailyRun"}},
{"id":"content.marketing_report.daily","module":"content_marketing","trigger":"scheduled","execution":{"entry":"data/tools/daily_marketing_report.bat"},"provenance":{"source_project":"content","task_name":"YingxiaoYunying_DailyMarketingReport"}},
{"id":"content.relogin.weekly","module":"content_marketing","trigger":"scheduled","execution":{"entry":"data/tools/friday_relogin.bat"},"provenance":{"source_project":"content","task_name":"YingxiaoYunying_FridayRelogin"}},
{"id":"content.self_operated.weekly","module":"content_marketing","trigger":"scheduled","execution":{"entry":"data/tools/monday_self_run.bat"},"provenance":{"source_project":"content","task_name":"YingxiaoYunying_MondaySelf"}},
{"id":"content.creator_report.monthly","module":"content_marketing","trigger":"scheduled","execution":{"entry":"data/tools/monthly_creator_report.bat"},"provenance":{"source_project":"content","task_name":"YingxiaoYunying_MonthlyCreatorReport"}},
{"id":"content.summary.monthly","module":"content_marketing","trigger":"scheduled","execution":{"entry":"data/tools/monthly_summary.bat"},"provenance":{"source_project":"content","task_name":"YingxiaoYunying_MonthlySummary"}},
{"id":"content.cooperations.daily","module":"content_marketing","trigger":"scheduled","execution":{"entry":"data/tools/sync_cooperations.bat"},"provenance":{"source_project":"content","task_name":"YingxiaoYunying_SyncCooperations"}},
{"id":"content.comments.weekly","module":"content_marketing","trigger":"scheduled","execution":{"entry":"data/tools/weekly_comment_scrape.bat"},"provenance":{"source_project":"content","task_name":"YingxiaoYunying_WeeklyCommentScrape"}},
{"id":"content.summary.weekly","module":"content_marketing","trigger":"scheduled","execution":{"entry":"data/tools/weekly_summary.bat"},"provenance":{"source_project":"content","task_name":"YingxiaoYunying_WeeklySummary"}},
{"id":"content.metrics.daily","module":"content_marketing","trigger":"scheduled","execution":{"steps":[{"id":"collect_collaborators","name":"采集合作达人指标","description":"按每日范围并行抓取合作达人在 B站、小红书蒲公英和抖音星图的曝光等指标，生成各平台采集结果。","data_flow":{"sources":[{"system":"飞书","label":"合作达人及内容清单"},{"system":"内容平台","label":"B站、小红书蒲公英与抖音星图指标"}],"processing":["按业务日期拆分平台采集任务","并行抓取合作内容曝光与互动指标"],"destinations":[{"system":"本地采集产物","label":"合作达人多平台指标结果"}]},"entry":"run_all.py","args":["--daily-scope"]},{"id":"refresh_self_mapping","name":"刷新自营款式映射","description":"从飞书索引刷新自营账号、款式与多维表映射，供后续自营采集准确定位目标记录。","data_flow":{"sources":[{"system":"飞书","label":"内容索引、账号与款式配置"}],"processing":["刷新自营账号、款式和多维表对应关系"],"destinations":[{"system":"本地映射","label":"自营账号、款式与目标表索引"}]},"entry":"data/tools/refresh_self_mapping.py","depends_on":["collect_collaborators"],"run_after_failure":true},{"id":"collect_self_bilibili","name":"采集自营 B站指标","description":"读取自营映射中的 B站笔记链接，通过公开 API 获取播放量和发布时间并回填对应飞书表。","data_flow":{"sources":[{"system":"本地映射","label":"自营 B站笔记链接"},{"system":"B站公开 API","label":"播放量与发布时间"}],"processing":["按映射定位自营笔记并读取公开指标"],"destinations":[{"system":"飞书","label":"自营 B站内容指标表"},{"system":"本地采集产物","label":"B站自营指标结果"}]},"entry":"self_bilibili_scraper.py","depends_on":["refresh_self_mapping"],"run_after_failure":true},{"id":"collect_self_douyin","name":"采集自营抖音指标","description":"使用蝉妈妈登录态导出自营抖音视频数据，解析曝光指标并回填飞书自营表。","data_flow":{"sources":[{"system":"本地映射","label":"自营抖音账号与款式"},{"system":"蝉妈妈","label":"已登录的抖音视频导出数据"}],"processing":["导出自营视频数据并解析曝光指标"],"destinations":[{"system":"飞书","label":"自营抖音内容指标表"},{"system":"本地采集产物","label":"抖音自营指标结果"}]},"entry":"chanmama_scraper.py","depends_on":["collect_self_bilibili"],"run_after_failure":true},{"id":"sync","name":"同步曝光指标入库","description":"汇总本轮合作与自营曝光结果，按记录映射更新或补建 PostgreSQL 内容笔记数据。","data_flow":{"sources":[{"system":"本轮采集产物","label":"合作达人及自营内容指标"},{"system":"本地映射","label":"内容记录对应关系"}],"processing":["合并各平台指标并匹配内容记录","更新已有记录或补建缺失记录"],"destinations":[{"system":"PostgreSQL","label":"内容笔记与曝光指标"}]},"entry":"data/tools/sync_metrics_to_cmt_notes.py","depends_on":["collect_self_douyin"],"run_after_failure":true}]},"provenance":{"source_project":"content","task_name":"YingxiaoYunying_DailyRun"},"note":"每日先采集合作达人，再通过专有自营映射、B站和蝉妈妈脚本采集自营笔记；各阶段保留批处理继续执行语义，最终任一步失败则工作流失败。"},
{"id":"content.marketing_report.daily","module":"content_marketing","trigger":"scheduled","execution":{"steps":[{"id":"generate_and_send","name":"生成并发送营销日报","description":"聚合各款式内容与评论分析，生成一页全域种草日报并发送给飞书负责人。","data_flow":{"sources":[{"system":"PostgreSQL","label":"内容、经营、评论与达人数据","detail":"款式指标、笔记、评论、合作和画像等"},{"system":"飞书","label":"款式、负责人及素材配置"}],"processing":["聚合业务事实与款式表现","调用 Hermes 生成营销分析","生成图表和 Markdown 日报"],"destinations":[{"system":"本地报告","label":"Markdown 日报与 PNG 数据看板"},{"system":"飞书","label":"负责人日报卡片","condition":"正式执行且启用 --send"}]},"entry":"daily_marketing_report.py","args":["--send"]}]},"provenance":{"source_project":"content","task_name":"YingxiaoYunying_DailyMarketingReport"}},
{"id":"content.relogin.weekly","module":"content_marketing","trigger":"scheduled","execution":{"steps":[{"id":"relogin","name":"刷新内容平台登录态","description":"并行启动五个平台的二维码重新登录流程，按配置轮次重试，持久化登录状态并通知结果。","data_flow":{"sources":[{"system":"内容平台","label":"五个平台登录页与当前会话状态"}],"processing":["并行发起二维码登录并按轮次重试","校验并持久化新的登录会话"],"destinations":[{"system":"浏览器状态","label":"各平台 Cookie 与 storage state"},{"system":"飞书","label":"登录刷新结果通知"}]},"entry":"data/tools/friday_relogin_parallel.py","args":["--max-attempts","3","--round-timeout","300","--screenshot-delay","60"]}]},"provenance":{"source_project":"content","task_name":"YingxiaoYunying_FridayRelogin"}},
{"id":"content.creator_report.monthly","module":"content_marketing","trigger":"scheduled","execution":{"steps":[{"id":"generate","name":"生成达人合作月报","description":"从 PostgreSQL 读取达人、款式和合作数据，生成完整的报价与合作分析报告。","data_flow":{"sources":[{"system":"PostgreSQL","label":"达人、款式、合作与内容表现数据"}],"processing":["统计达人报价、合作和内容效果","生成月度筛选与报价分析"],"destinations":[{"system":"本地报告","label":"达人合作月报 Markdown"},{"system":"飞书文档","label":"达人合作分析报告"},{"system":"PostgreSQL","label":"月报记录与报告链接"}]},"entry":"data/tools/generate_creator_report.py"}]},"provenance":{"source_project":"content","task_name":"YingxiaoYunying_MonthlyCreatorReport"}},
{"id":"content.summary.monthly","module":"content_marketing","trigger":"scheduled","execution":{"steps":[{"id":"summarize","name":"生成内容月度汇总","description":"分析上月各款全部笔记并做跨平台横向对比，生成飞书文档、回写生命进程表并入库。","data_flow":{"sources":[{"system":"PostgreSQL","label":"上月各款内容、互动与评论数据"},{"system":"飞书","label":"款式与生命进程配置"}],"processing":["按款式分析全部笔记","完成跨平台对比与月度总结"],"destinations":[{"system":"飞书文档","label":"内容月度报告"},{"system":"飞书","label":"款式生命进程表"},{"system":"PostgreSQL","label":"月度汇总结果"}]},"entry":"monthly_summary_all.py","args":["--max-workers","4"]}]},"provenance":{"source_project":"content","task_name":"YingxiaoYunying_MonthlySummary"}},
{"id":"content.cooperations.daily","module":"content_marketing","trigger":"scheduled","execution":{"steps":[{"id":"sync","name":"同步达人合作记录","description":"遍历飞书款式合作表，幂等写入或更新 PostgreSQL 中的达人属性和合作记录。","data_flow":{"sources":[{"system":"飞书","label":"各款式达人合作表"}],"processing":["刷新款式映射并规范化达人合作字段","按业务主键幂等新增或更新"],"destinations":[{"system":"PostgreSQL","label":"达人档案与合作记录"}]},"entry":"data/tools/sync_cooperations.py","args":["--refresh-mapping"]}]},"provenance":{"source_project":"content","task_name":"YingxiaoYunying_SyncCooperations"}},
{"id":"content.comments.weekly","module":"content_marketing","trigger":"scheduled","execution":{"steps":[{"id":"bilibili","name":"重采 B站评论","description":"从数据库读取 B站视频，重新采集互动指标和评论，替换评论表数据并保存本地产物。","data_flow":{"sources":[{"system":"PostgreSQL","label":"B站视频清单"},{"system":"B站","label":"视频互动指标与评论"}],"processing":["逐条重新采集视频互动和评论"],"destinations":[{"system":"PostgreSQL","label":"B站评论与互动指标"},{"system":"本地采集产物","label":"B站评论结果"}]},"entry":"data/tools/batch_rescrape_bilibili.py"},{"id":"xiaohongshu","name":"重采小红书评论","description":"从数据库读取小红书笔记，重新采集互动指标和评论，替换评论表数据并保存本地产物。","data_flow":{"sources":[{"system":"PostgreSQL","label":"小红书笔记清单"},{"system":"小红书","label":"笔记互动指标与评论"}],"processing":["逐条重新采集笔记互动和评论"],"destinations":[{"system":"PostgreSQL","label":"小红书评论与互动指标"},{"system":"本地采集产物","label":"小红书评论结果"}]},"entry":"data/tools/batch_rescrape_xiaohongshu.py"},{"id":"douyin","name":"重采抖音评论","description":"从数据库读取抖音视频，重新采集互动指标和评论，替换评论表数据并保存本地产物。","data_flow":{"sources":[{"system":"PostgreSQL","label":"抖音视频清单"},{"system":"抖音","label":"视频互动指标与评论"}],"processing":["逐条重新采集视频互动和评论"],"destinations":[{"system":"PostgreSQL","label":"抖音评论与互动指标"},{"system":"本地采集产物","label":"抖音评论结果"}]},"entry":"data/tools/batch_rescrape_douyin.py"}]},"provenance":{"source_project":"content","task_name":"YingxiaoYunying_WeeklyCommentScrape"},"note":"保留来源批处理语义：三平台并行启动、全部等待，任一平台失败则工作流最终失败。"},
{"id":"content.summary.weekly","module":"content_marketing","trigger":"scheduled","execution":{"steps":[{"id":"summarize","name":"生成内容周度汇总","description":"分析上周各款新增笔记，调用模型生成综合评级，创建飞书周报并回写生命进程表。","data_flow":{"sources":[{"system":"PostgreSQL","label":"上周各款新增笔记与互动数据"},{"system":"飞书","label":"款式与生命进程配置"}],"processing":["按款式汇总新增内容","调用模型生成评级与周度分析"],"destinations":[{"system":"飞书文档","label":"内容周报"},{"system":"飞书","label":"款式生命进程表"},{"system":"PostgreSQL","label":"周度汇总结果"}]},"entry":"weekly_summary_all.py","args":["--max-workers","4"]}]},"provenance":{"source_project":"content","task_name":"YingxiaoYunying_WeeklySummary"}},
{"id":"shop.metrics.weekly","module":"shop_intelligence","trigger":"scheduled","execution":{"entry":"runners/run_shop.py"},"provenance":{"source_project":"shop","task_name":"shop-data-collection"}},
{"id":"shop.competitor.weekly","module":"shop_intelligence","trigger":"scheduled","execution":{"entry":"runners/run_peer_store.py"},"provenance":{"source_project":"shop","task_name":"peer-store-data-collection"}},
{"id":"shop.metrics.weekly","module":"shop_intelligence","trigger":"scheduled","execution":{"steps":[{"id":"jd","name":"采集京东店铺经营指标","description":"独立采集京东店铺经营数据，生成本次产物并写入 PostgreSQL；生产模式写入飞书 Base。","data_flow":{"sources":[{"system":"京东商智","label":"店铺经营指标"}],"processing":["按业务日期采集并规范化京东店铺数据"],"destinations":[{"system":"本地采集产物","label":"京东店铺指标"},{"system":"PostgreSQL","label":"店铺经营指标"},{"system":"飞书 Base","label":"京东店铺周数据","condition":"正式执行"}]},"entry":"runners/run_shop.py","args":["--platform","jd"]},{"id":"dy","name":"采集抖音店铺经营指标","description":"独立采集抖音店铺经营数据，生成本次产物并写入 PostgreSQL；生产模式写入飞书 Base。","data_flow":{"sources":[{"system":"抖音电商罗盘","label":"店铺经营指标"}],"processing":["按业务日期采集并规范化抖音店铺数据"],"destinations":[{"system":"本地采集产物","label":"抖音店铺指标"},{"system":"PostgreSQL","label":"店铺经营指标"},{"system":"飞书 Base","label":"抖音店铺周数据","condition":"正式执行"}]},"entry":"runners/run_shop.py","args":["--platform","dy"]},{"id":"tm","name":"采集天猫店铺经营指标","description":"独立采集天猫店铺经营数据，生成本次产物并写入 PostgreSQL；按原逻辑不写飞书店铺表。","data_flow":{"sources":[{"system":"天猫生意参谋","label":"店铺经营指标"}],"processing":["按业务日期采集并规范化天猫店铺数据"],"destinations":[{"system":"本地采集产物","label":"天猫店铺指标"},{"system":"PostgreSQL","label":"店铺经营指标"}]},"entry":"runners/run_shop.py","args":["--platform","tm"]}]},"provenance":{"source_project":"shop","task_name":"shop-data-collection"},"note":"三个平台作为独立 LangGraph 节点并行执行，分别入库和收敛错误，单个平台失败不阻断其他平台。"},
{"id":"shop.competitor.weekly","module":"shop_intelligence","trigger":"scheduled","execution":{"steps":[{"id":"jd","name":"采集京东竞店经营指标","description":"独立采集京东竞店排行，生成本次产物并写入 PostgreSQL；生产模式写入飞书 Base。","data_flow":{"sources":[{"system":"京东商智","label":"竞店排行与经营指标"}],"processing":["采集目标竞店并规范化排行数据"],"destinations":[{"system":"本地采集产物","label":"京东竞店指标"},{"system":"PostgreSQL","label":"竞店经营指标"},{"system":"飞书 Base","label":"京东竞店周数据","condition":"正式执行"}]},"entry":"runners/run_peer_store.py","args":["--platform","jd"]},{"id":"dy","name":"采集抖音竞店经营指标","description":"独立采集抖音竞店排行，生成本次产物并写入 PostgreSQL；生产模式写入飞书 Base。","data_flow":{"sources":[{"system":"抖音电商罗盘","label":"竞店排行与经营指标"}],"processing":["采集目标竞店并规范化排行数据"],"destinations":[{"system":"本地采集产物","label":"抖音竞店指标"},{"system":"PostgreSQL","label":"竞店经营指标"},{"system":"飞书 Base","label":"抖音竞店周数据","condition":"正式执行"}]},"entry":"runners/run_peer_store.py","args":["--platform","dy"]}]},"provenance":{"source_project":"shop","task_name":"peer-store-data-collection"},"note":"京东和抖音作为独立 LangGraph 节点并行执行，分别入库和收敛错误，单个平台失败不阻断另一个平台；当前无天猫竞店采集任务。"},
{"id":"shop.jd_self_operated.daily","module":"shop_intelligence","trigger":"scheduled","execution":{"steps":[{"id":"brand","name":"采集京东自营品牌业绩","description":"按业务日期采集京东自营品牌维度业绩，保存原始产物并写入 PostgreSQL。","data_flow":{"sources":[{"system":"京东自营后台","label":"品牌维度经营数据"}],"processing":["按业务日期采集并整理品牌业绩"],"destinations":[{"system":"本地原始产物","label":"品牌业绩数据"},{"system":"PostgreSQL","label":"京东自营品牌日指标"}]},"entry":"collectors/jd_self_operated_brand_daily.py","args":["--start-date","{business_date}","--end-date","{business_date}","--headless"],"timeout_seconds":600},{"id":"product","name":"采集京东自营商品业绩","description":"按业务日期逐页采集京东自营商品维度业绩，保存原始产物并写入 PostgreSQL。","data_flow":{"sources":[{"system":"京东自营后台","label":"商品维度经营数据"}],"processing":["逐页采集商品明细并整理日业绩"],"destinations":[{"system":"本地原始产物","label":"商品业绩数据"},{"system":"PostgreSQL","label":"京东自营商品日指标"}]},"entry":"collectors/jd_self_operated_product_daily.py","args":["--start-date","{business_date}","--end-date","{business_date}","--headless"],"depends_on":["brand"],"run_after_failure":true,"timeout_seconds":1800}]},"provenance":{"source_project":"shop","task_name":"京东自营昨日业绩每日采集"},"note":"每天采集计划日期前一天；品牌失败仍继续商品，任一步失败则工作流失败。"},
{"id":"shop.douyin_price_appeal","module":"shop_intelligence","trigger":"scheduled","execution":{"steps":[{"id":"dy","name":"提交抖音待改价申诉","description":"复用店铺周采集的抖音商家账号 Cookie 与 Profile，打开活动广场；待改价不为 0 时逐条进入立即改价，兼容“申诉”和“发起申诉”入口，全选 SKU，并以固定原因提交申诉。","data_flow":{"sources":[{"system":"抖音电商","label":"活动广场待改价商品"},{"system":"浏览器状态","label":"店铺周采集账号 Cookie 与 Profile"}],"processing":["检查待改价数量并打开待处理列表","逐条进入立即改价并兼容两种申诉入口","全选 SKU，选择“对议价价格存疑 / 认为识别有误，该规格没有这么低的价格”并提交"],"destinations":[{"system":"抖音电商","label":"待改价议价申诉","condition":"正式执行（控制台或定时调度）"}]},"entry":"collectors/dy_store_competitor_store_scraping.py","args":["--price-appeal","--execute"],"timeout_seconds":1800,"replay_policy":"idempotent"}]},"provenance":{"source_project":"shop","task_name":"douyin-price-appeal-daily"},"note":"每天 08:00、16:00、22:00 自动执行；仍可在控制台手动执行。每个时点都会重新扫描，已申诉商品不会再次出现可申诉入口；安全预演不访问抖音也不提交申诉。"},
{"id":"product.persona.daily","module":"product_commerce","trigger":"scheduled","execution":{"steps":[{"id":"collect","name":"采集三平台人群画像","description":"并行运行天猫、抖音和京东人群画像采集器，等待全部完成并写入对应飞书多维表。","data_flow":{"sources":[{"system":"天猫、抖音与京东","label":"商品人群画像后台数据"},{"system":"本地配置","label":"款式与目标表映射"}],"processing":["并行采集三平台画像","统一画像维度并匹配款式"],"destinations":[{"system":"飞书多维表","label":"各平台商品人群画像"}]},"entry":"run_daily_persona.py","args":["--date","{business_date}"]}]},"provenance":{"source_project":"product","task_name":"PersonaDailyCollect"}},
{"id":"product.daily","module":"product_commerce","trigger":"scheduled","execution":{"steps":[{"id":"collect_analyze_publish","name":"采集、汇总并发布商品日报","description":"按业务日期采集 ERP 与三平台数据，完成汇总分析、导出并写入飞书日报记录。","data_flow":{"sources":[{"system":"ERP","label":"商品库存与销售数据"},{"system":"天猫、京东与抖音","label":"商品经营指标"}],"processing":["按业务日期采集各数据源","汇总并分析商品经营表现","导出日报记录"],"destinations":[{"system":"本地日报产物","label":"商品经营汇总与导出文件"},{"system":"飞书","label":"商品日报记录"}]},"entry":"orchestrate_daily_collection.py","args":["--target-date","{business_date}","--stages","collect,analyze,export,insert"]}]},"provenance":{"source_project":"product","task_name":"ProductCollectorDailyCollect"}},
{"id":"product.alert.daily","module":"product_commerce","trigger":"scheduled","execution":{"steps":[{"id":"detect_and_notify","name":"检测并通知销量下滑","description":"检测连续销量下滑，幂等记录告警事件，并通过 Hermes 与飞书通知配置对象。","data_flow":{"sources":[{"system":"PostgreSQL","label":"商品日销量与历史趋势"},{"system":"告警配置","label":"检测规则与通知对象"}],"processing":["检测连续销量下滑","调用 Hermes 生成告警说明","幂等判定是否需要通知"],"destinations":[{"system":"PostgreSQL","label":"销量告警事件"},{"system":"飞书","label":"配置对象告警通知","condition":"命中规则且未重复发送"}]},"entry":"commands/run_alerts.py"}]},"provenance":{"source_project":"product","task_name":"ProductCollectorSalesAlert"}},
{"id":"product.import.daily","module":"product_commerce","trigger":"scheduled","execution":{"steps":[{"id":"import","name":"导入商品日报原始数据","description":"扫描业务日期前一天的三平台原始报表，统一字段后幂等写入 PostgreSQL 商品日指标表。","replay_policy":"idempotent","data_flow":{"sources":[{"system":"本地原始报表","label":"天猫、京东与抖音商品日报"}],"processing":["按业务日期定位报表","统一三平台字段并校验记录","按业务主键幂等导入"],"destinations":[{"system":"PostgreSQL","label":"商品日经营指标"}]},"entry":"commands/import_daily.py"}]},"provenance":{"source_project":"product","task_name":"ProductDailyImport"}},
{"id":"product.style_analysis.interval","module":"product_commerce","trigger":"scheduled","execution":{"steps":[{"id":"analyze","name":"生成款式周期分析","description":"聚合每个款式最近 3 天经营指标，调用 Hermes 分析，创建飞书文档并保存报告链接。","data_flow":{"sources":[{"system":"PostgreSQL","label":"最近 3 天款式经营指标"},{"system":"款式配置","label":"待分析款式与最小分析间隔"}],"processing":["聚合款式周期指标","调用 Hermes 生成经营分析","跳过分析间隔未满足的款式"],"destinations":[{"system":"飞书文档","label":"款式周期分析报告"},{"system":"PostgreSQL","label":"报告链接与分析状态"}]},"entry":"analyze_style_with_hermes.py","args":["--all-styles","--days","3","--min-interval-days","3","--skip-existing","--end-date","{business_date}"]}]},"provenance":{"source_project":"product","task_name":"StyleAnalysisEvery3Days"}},
{"id":"product.main_image.weekly","module":"product_commerce","trigger":"scheduled","execution":{"steps":[{"id":"jd","name":"采集京东主图表现","description":"采集京东款式主图及曝光、点击指标，保存 JSON，并写入每款飞书主图表和 PostgreSQL。","data_flow":{"sources":[{"system":"京东商智","label":"款式主图、曝光与点击指标"},{"system":"本地配置","label":"款式与飞书目标表映射"}],"processing":["采集并按款式整理京东主图表现"],"destinations":[{"system":"本地 JSON","label":"京东主图采集结果"},{"system":"飞书","label":"各款式京东主图表"},{"system":"PostgreSQL","label":"京东主图表现指标"}]},"entry":"run_weekly_jd_main_image.py","args":["--date","{business_date}","--headless"]},{"id":"tmall","name":"采集天猫主图表现","description":"采集天猫万相台款式主图聚合指标，保存 JSON，并写入每款飞书主图表和 PostgreSQL。","data_flow":{"sources":[{"system":"天猫万相台","label":"款式主图聚合表现"},{"system":"本地配置","label":"款式与飞书目标表映射"}],"processing":["采集并按款式整理天猫主图表现"],"destinations":[{"system":"本地 JSON","label":"天猫主图采集结果"},{"system":"飞书","label":"各款式天猫主图表"},{"system":"PostgreSQL","label":"天猫主图表现指标"}]},"entry":"run_weekly_main_image.py","args":["--date","{business_date}","--headless"]}]},"provenance":{"source_project":"product","task_name":"WeeklyJdAndTmallMainImageCollect"},"note":"每周日 08:30 并行执行京东与天猫万相主图采集；两个平台分别完成 PG 入库和飞书插入，资源、状态及失败互不影响。任一平台失败时另一个仍继续，工作流最终报告具体的平台错误。"},
{"id":"product.sales_sheet.daily","module":"product_commerce","trigger":"scheduled","execution":{"steps":[{"id":"sync","name":"刷新商品月度销量表","description":"全量读取预算产品与款式主表，不按预算月份过滤款式；所选月份只控制聚水潭查询区间和飞书目标月份页。","replay_policy":"idempotent","data_flow":{"sources":[{"system":"飞书多维表格","label":"全部预算产品、目标销量、款式名与 ERP 款式编码"},{"system":"聚水潭商品主体分析","label":"所选自然月全渠道自定义销量与自定义退货量"}],"processing":["合并预算 Base 与款式主表的全量款式并匹配 ERP 编码","按所选月月初到月末逐 ERP 编码生成一次全渠道报表","累加全部商品行和同款全部 ERP 编码并计算退货率、实销和目标达成率"],"destinations":[{"system":"飞书电子表格","label":"当月产品销量页","condition":"正式执行"}]},"entry":"sync_monthly_sales_sheet.py","args":["--month","{business_date}","--execute","--allow-missing-erp-as-zero"]}]},"provenance":{"source_project":"product","task_name":"ProductMonthlySalesSheetDailySync"},"note":"同名月份页幂等覆盖；月份只控制聚水潭自然月区间与目标页，产品全集不按预算月份筛选；抓取和校验全部 ERP 报表成功后才清空写入；无 ERP 编码产品按 0 写入；缺页时复制最近历史月份页继承格式。"},
{"id":"product.persona.daily","module":"product_commerce","trigger":"scheduled","execution":{"entry":"run_daily_persona.py"},"provenance":{"source_project":"product","task_name":"PersonaDailyCollect"}},
{"id":"product.daily","module":"product_commerce","trigger":"scheduled","execution":{"entry":"orchestrate_daily_collection.py","args":["--stages","collect,analyze,export,insert"]},"provenance":{"source_project":"product","task_name":"ProductCollectorDailyCollect"}},
{"id":"product.alert.daily","module":"product_commerce","trigger":"scheduled","execution":{"entry":"commands/run_alerts.py"},"provenance":{"source_project":"product","task_name":"ProductCollectorSalesAlert"}},
{"id":"product.import.daily","module":"product_commerce","trigger":"scheduled","execution":{"entry":"commands/import_daily.py"},"provenance":{"source_project":"product","task_name":"ProductDailyImport"}},
{"id":"product.style_analysis.interval","module":"product_commerce","trigger":"scheduled","execution":{"entry":"analyze_style_with_hermes.py","args":["--all-styles","--days","3","--min-interval-days","3","--skip-existing"]},"provenance":{"source_project":"product","task_name":"StyleAnalysisEvery3Days"}},
{"id":"product.main_image.jd.weekly","module":"product_commerce","trigger":"scheduled","execution":{"entry":"run_weekly_jd_main_image.py","args":["--headless"]},"provenance":{"source_project":"product","task_name":"WeeklyJdMainImageCollect"}},
{"id":"product.main_image.weekly","module":"product_commerce","trigger":"scheduled","execution":{"entry":"run_weekly_main_image.py","args":["--headless"]},"provenance":{"source_project":"product","task_name":"WeeklyMainImageCollect"}},
{"id":"supply.purchase_confirmation.daily","module":"supply_chain","trigger":"scheduled","execution":{"steps":[{"id":"collect_analyze_notify","name":"生成采购确认通知","description":"从 ERP 识别库存缺口商品并检查协议到货时间，写入 PostgreSQL 后发送采购确认通知。","replay_policy":"idempotent","data_flow":{"sources":[{"system":"ERP","label":"商品库存、在途与采购数据"},{"system":"采购配置","label":"协议到货时间与确认规则"}],"processing":["识别库存缺口商品","核对协议到货时间并生成确认项"],"destinations":[{"system":"PostgreSQL","label":"采购确认处理记录"},{"system":"飞书","label":"采购确认通知"}]},"entry":"run.py","args":["mcp-run","purchase-confirmation"]}]},"provenance":{"source_project":"supply","task_name":"auto-flow-purchase-confirmation"}},
{"id":"supply.replenishment.weekly","module":"supply_chain","trigger":"scheduled","execution":{"steps":[{"id":"collect_analyze_notify","name":"生成并发送补货建议","description":"导出 ERP 库存并计算补货量，写入 PostgreSQL 与飞书多维表，最后发送补货结果通知。","replay_policy":"idempotent","data_flow":{"sources":[{"system":"ERP","label":"库存、销售与商品数据"},{"system":"补货配置","label":"SKU 范围与补货规则"}],"processing":["导出并规范化库存数据","按规则计算建议补货量","生成补货批次结果"],"destinations":[{"system":"PostgreSQL","label":"补货批次与建议明细"},{"system":"飞书多维表","label":"补货建议记录"},{"system":"飞书","label":"补货结果通知"}]},"entry":"run.py","args":["mcp-run","replenishment"]}]},"provenance":{"source_project":"supply","task_name":"auto-flow-replenishment"}},
{"id":"supply.replenishment_alert.daily","module":"supply_chain","trigger":"scheduled","execution":{"steps":[{"id":"collect_analyze_notify","name":"生成库存阈值预警","description":"依据配置 SKU 和库存阈值生成预警并写入 PostgreSQL，仅在存在预警时发送业务通知。","replay_policy":"idempotent","data_flow":{"sources":[{"system":"ERP","label":"SKU 当前库存"},{"system":"预警配置","label":"监控 SKU 与库存阈值"}],"processing":["比较当前库存与配置阈值","生成需要处理的库存预警"],"destinations":[{"system":"PostgreSQL","label":"库存阈值预警记录"},{"system":"飞书","label":"库存预警通知","condition":"存在命中阈值的 SKU"}]},"entry":"run.py","args":["mcp-run","replenishment-alert"]}]},"provenance":{"source_project":"supply","task_name":"auto-flow-replenishment-alert"}},
{"id":"supply.purchase_confirmation.daily","module":"supply_chain","trigger":"scheduled","execution":{"entry":"run.py","args":["mcp-run","purchase-confirmation"]},"provenance":{"source_project":"supply","task_name":"auto-flow-purchase-confirmation"}},
{"id":"supply.replenishment.weekly","module":"supply_chain","trigger":"scheduled","execution":{"entry":"run.py","args":["mcp-run","replenishment"]},"provenance":{"source_project":"supply","task_name":"auto-flow-replenishment"}},
{"id":"supply.replenishment_alert.daily","module":"supply_chain","trigger":"scheduled","execution":{"entry":"run.py","args":["mcp-run","replenishment-alert"]},"provenance":{"source_project":"supply","task_name":"auto-flow-replenishment-alert"}},
{"id":"content.mapping.refresh","module":"content_marketing","trigger":"manual","execution":{"entry":"data/tools/rebuild_mapping.py"},"provenance":{"source_project":"content"}},
{"id":"content.retry_failed","module":"content_marketing","trigger":"manual","execution":{"entry":"data/tools/retry_failed.py"},"provenance":{"source_project":"content"}},
{"id":"content.metrics.backfill","module":"content_marketing","trigger":"manual","execution":{"entry":"data/tools/daily_run_with_backfill.bat"},"provenance":{"source_project":"content"}},
{"id":"product.backfill","module":"product_commerce","trigger":"manual","execution":{"entry":"backfill_collect.py"},"provenance":{"source_project":"product"}},
{"id":"product.market_rank","module":"product_commerce","trigger":"manual","execution":{"entry":"orchestrate_market_rank_collection.py"},"provenance":{"source_project":"product"},"note":"A launcher exists but no matching system task is registered."},
{"id":"product.review_collection","module":"product_commerce","trigger":"manual","execution":{"entry":"orchestrate_review_collection.py"},"provenance":{"source_project":"product"}},
{"id":"supply.purchase_order_update","module":"supply_chain","trigger":"manual","execution":{"entry":"run.py","args":["mcp-run","purchase-order-update"]},"provenance":{"source_project":"supply"}},
{"id":"product.weekly_aggregate.documented_missing","module":"product_commerce","trigger":"unavailable","execution":{"entry":"run_weekly_collect.py"},"provenance":{"source_project":"product"},"note":"Documented in README but the entry file and scheduled task are absent."}
{"id":"product.market_rank","module":"product_commerce","trigger":"scheduled","execution":{"steps":[{"id":"collect_analyze_publish","name":"采集并发布三平台市场排行","description":"并行采集天猫、京东和抖音市场排行，汇总平台结果并发送飞书报告链接。","data_flow":{"sources":[{"system":"天猫、京东与抖音","label":"市场商品排行与表现数据"},{"system":"本地配置","label":"类目、关键词与采集范围"}],"processing":["并行采集三平台排行","汇总平台结果并生成市场分析","归档报告并准备通知"],"destinations":[{"system":"本地报告产物","label":"三平台市场排行结果"},{"system":"飞书","label":"市场排行报告链接与通知"}]},"entry":"orchestrate_market_rank_collection.py","args":["--report-date","{business_date}"]}]},"provenance":{"source_project":"product","task_name":"三平台市场排行周采集"},"note":"每周一 10:00 由项目内 Python 调度器触发。"}
]
}
deploy/gyxx-flow.service
+23
View File
@@ -0,0 +1,23 @@
[Unit]
Description=GYXX Flow Python Scheduler
Wants=network-online.target
After=network-online.target
[Service]
Type=simple
User=gyxx-flow
Group=gyxx-flow
WorkingDirectory=/opt/gyxx-flow
Environment=PYTHONUTF8=1
Environment=GYXX_PROJECT_ROOT=/opt/gyxx-flow
Environment=GYXX_DATA_ROOT=/var/lib/gyxx-flow
EnvironmentFile=-/etc/gyxx-flow/gyxx-flow.env
ExecStart=/opt/gyxx-flow/.venv/bin/python -m gyxx_flow schedule run
Restart=on-failure
RestartSec=10s
TimeoutStopSec=90s
KillSignal=SIGTERM
NoNewPrivileges=true
[Install]
WantedBy=multi-user.target
deploy/postgres.compose.yml
+26
View File
@@ -0,0 +1,26 @@
services:
postgres:
image: postgres:16-alpine
container_name: gyxx-flow-postgres
restart: unless-stopped
environment:
POSTGRES_DB: gyxx_super_data
POSTGRES_USER: gyxx_flow
POSTGRES_PASSWORD: ${GYXX_POSTGRES_PASSWORD:?set GYXX_POSTGRES_PASSWORD}
ports:
- "127.0.0.1:5432:5432"
volumes:
- gyxx_flow_postgres:/var/lib/postgresql/data
- ../src/gyxx_flow/modules/content_marketing/data/tools/schema_gyxx_super_data.sql:/docker-entrypoint-initdb.d/10-content-marketing.sql:ro
- ../src/gyxx_flow/modules/product_commerce/db/schema.sql:/docker-entrypoint-initdb.d/20-product-commerce.sql:ro
- ../src/gyxx_flow/modules/shop_intelligence/db/schema.sql:/docker-entrypoint-initdb.d/30-shop-intelligence.sql:ro
- ../src/gyxx_flow/modules/supply_chain/orchestrator/sql/001_init.sql:/docker-entrypoint-initdb.d/40-supply-chain.sql:ro
- ../src/gyxx_flow/modules/supply_chain/orchestrator/sql/002_workflow_v2.sql:/docker-entrypoint-initdb.d/41-supply-chain-v2.sql:ro
healthcheck:
test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
interval: 5s
timeout: 5s
retries: 20
volumes:
gyxx_flow_postgres:
deploy/windows-service/install.ps1
+50
View File
@@ -0,0 +1,50 @@
# Legacy Windows compatibility only.
# NSSM supervises the single Python scheduler process; this script never creates
# Windows Task Scheduler entries and does not own business schedule rules.
param(
[string]$ProjectRoot = (Resolve-Path (Join-Path $PSScriptRoot "..\..")).Path,
[string]$DataRoot = "",
[string]$ServiceName = "gyxx-flow-scheduler",
[string]$NssmPath = "nssm.exe"
)
$ErrorActionPreference = "Stop"
Set-StrictMode -Version Latest
$project = (Resolve-Path -LiteralPath $ProjectRoot).Path
if ([string]::IsNullOrWhiteSpace($DataRoot)) {
$DataRoot = Join-Path $project "var"
}
$data = [IO.Path]::GetFullPath($DataRoot)
$python = Join-Path $project ".venv\Scripts\python.exe"
if (-not (Test-Path -LiteralPath $python -PathType Leaf)) {
throw "Python environment not found: $python"
}
$nssm = (Get-Command $NssmPath -ErrorAction Stop).Source
$serviceLogRoot = Join-Path $data "logs\scheduler-service"
New-Item -ItemType Directory -Path $serviceLogRoot -Force | Out-Null
function Invoke-Nssm([string[]]$Arguments) {
& $nssm @Arguments
if ($LASTEXITCODE -ne 0) {
throw "NSSM failed with exit code $LASTEXITCODE"
}
}
Invoke-Nssm -Arguments @("install", $ServiceName, $python)
Invoke-Nssm -Arguments @("set", $ServiceName, "AppDirectory", $project)
Invoke-Nssm -Arguments @("set", $ServiceName, "AppParameters", "-m gyxx_flow schedule run")
Invoke-Nssm -Arguments @(
"set", $ServiceName, "AppEnvironmentExtra",
"GYXX_PROJECT_ROOT=$project", "GYXX_DATA_ROOT=$data", "PYTHONUTF8=1"
)
Invoke-Nssm -Arguments @("set", $ServiceName, "AppStdout", (Join-Path $serviceLogRoot "service.log"))
Invoke-Nssm -Arguments @("set", $ServiceName, "AppStderr", (Join-Path $serviceLogRoot "service-error.log"))
Invoke-Nssm -Arguments @("set", $ServiceName, "AppRotateFiles", "1")
Invoke-Nssm -Arguments @("set", $ServiceName, "AppRotateBytes", "10485760")
Invoke-Nssm -Arguments @("set", $ServiceName, "AppExit", "Default", "Restart")
Invoke-Nssm -Arguments @("set", $ServiceName, "Start", "SERVICE_AUTO_START")
Write-Host "Service installed: $ServiceName"
Write-Host "Before starting it, set the Log On account and production environment as documented."
Write-Host "Start with: nssm start $ServiceName"
deploy/windows-service/uninstall.ps1
+20
View File
@@ -0,0 +1,20 @@
# Legacy Windows compatibility only. This removes the NSSM process supervisor;
# project data, browser state, and Python scheduler state are preserved.
param(
[string]$ServiceName = "gyxx-flow-scheduler",
[string]$NssmPath = "nssm.exe"
)
$ErrorActionPreference = "Stop"
Set-StrictMode -Version Latest
$nssm = (Get-Command $NssmPath -ErrorAction Stop).Source
& $nssm stop $ServiceName
if ($LASTEXITCODE -notin @(0, 3)) {
throw "Unable to stop service $ServiceName (exit code $LASTEXITCODE)"
}
& $nssm remove $ServiceName confirm
if ($LASTEXITCODE -ne 0) {
throw "Unable to remove service $ServiceName (exit code $LASTEXITCODE)"
}
Write-Host "Service removed. Project data and scheduler state were preserved."
design.md
-157
View File
@@ -1,157 +0,0 @@
# GYXX Flow 统一编排平台设计
## 1. 目标
把 `D:\yingxiaoyunying`、`D:\shop-data-flow`、
`D:\product-collector-analyze-flow` 和 `E:\auto-flow` 的业务源码、启动脚本和工作流
完整迁移到 `D:\gyxx-flow`。新系统采用模块化单体、统一 CLI、声明式调度和分层
数据目录，同时允许旧任务在迁移期继续运行并能逐任务回滚。
“完整迁移”的硬性定义是：删除、改名或断开四个旧项目目录后，新项目仍能完成
入口发现、导入、dry-run、手工执行和调度执行。旧项目只能作为迁移输入和生产对照，
不得成为新项目运行时依赖。仅登记旧入口或通过环境变量调用旧脚本不算迁移完成。
## 2. 边界
- 新代码和迁移工具只写入 `D:\gyxx-flow`。
- 在生产切换门禁前，不修改四个旧项目或现有计划任务。
- 允许从旧项目做一次性只读复制；复制后的代码归属新项目并在新项目内改造、测试。
- 运行时代码不得读取 `GYXX_LEGACY_*_ROOT`，不得导入或启动四个旧目录中的文件。
- `D:\comment-data-collector` 不在迁移范围内；它与内容模块共享数据库表的字段
所有权以外部契约表示。
- 真实 Cookie、密码、应用密钥、访问令牌和浏览器 Profile 不进入源码仓库。
## 3. 方案
选择模块化单体而不是原样拼目录或直接拆微服务。四个业务模块为：
1. `content_marketing`：内容指标、评论、达人、周报/月报和登录维护。
2. `product_commerce`：ERP、商品、画像、主图、市场排名、分析和告警。
3. `shop_intelligence`：店铺及竞店周数据。
4. `supply_chain`：采购确认、库存预警、补货和采购单更新。
共享能力通过端口和适配器提供：文件产物、PostgreSQL、飞书、Hermes、浏览器和
平台采集。业务模块不得导入其他业务模块的内部实现。
每个业务模块包含自己的 `jobs/`、`collectors/`、`services/`、`models/` 和
`resources/`。一次性复制阶段可以在模块内保留原相对结构以降低行为变化风险，但
所有入口必须改为由模块清单定位，所有数据/日志/Profile/临时文件必须改由
`RunContext` 与 `DataLayout` 提供。模块间共享只允许依赖 `core`、`workflow`、
`adapters` 和显式契约。
## 4. 运行模型
调度器只调用稳定命令 `gyxx run <workflow-id>`。工作流引擎负责：
- 创建 `run_id` 和 `RunContext`；
- 获取工作流及资源锁；
- 按依赖顺序执行步骤；
- 记录步骤状态、退出码、耗时和错误；
- 进行有上限的步骤级重试；
- 生成不可变产物清单和 SHA-256；
- 在 shadow 模式关闭生产写入和正式通知。
Windows Task Scheduler 只是一个调度适配器。工作流和时间表的唯一事实来源是仓库
内的声明式配置，因此可以生成 Windows、Cron 或其他平台的调度配置。
## 5. 数据模型
运行根目录由 `GYXX_DATA_ROOT` 指定，默认是仓库下 `var`：
```text
var/
data/raw/<domain>/<source>/<dataset>/business_date=<date>/run_id=<id>/
data/normalized/<domain>/<dataset>/schema_vN/
data/curated/<domain>/<dataset>/
data/exports/<consumer>/<workflow>/<date>/<run_id>/
data/evidence/<workflow>/<run_id>/
data/legacy/<source_project>/
runs/<workflow>/<yyyy>/<mm>/<dd>/<run_id>/
state/{browser_profiles,cookies,checkpoints,locks}/
logs/<workflow>/<yyyy>/<mm>/<dd>/
quarantine/
tmp/
```
原始数据只追加。每个产物都有 `artifact_id`、数据集、业务日期、schema 版本、行数、
字节数、SHA-256、源路径和上游产物引用。历史文件先原样复制到 `legacy`，校验通过后
再生成标准化数据。
## 6. 安全和幂等
- 配置只保存环境变量名或凭据引用，不保存密钥值。
- 生产写入使用 `workflow + business_date + entity_id + schema_version` 幂等键。
- 外部通知使用 outbox/回执，失败重试不会重复发送。
- 飞书、数据库和通知都是可替换 Sink；shadow 模式使用测试 Sink。
- 浏览器 Profile 使用独立资源锁，禁止不同工作流并发写同一 Profile。
## 7. 迁移策略
采用源码接管迁移：基线清单 -> 新基础设施 -> 源码只读复制 -> 新项目内路径与配置
改造 -> 无旧目录测试 -> 影子运行 -> 单任务切换 -> 观察 -> 退役。旧命令适配器只
用于盘点阶段，不能出现在最终工作流定义中。切换时只禁用一个旧任务，不删除；失败
时关闭新任务并重新启用旧任务。旧项目至少保留只读 30 天。
迁移顺序为 `shop_intelligence`、`supply_chain`、`content_marketing`、
`product_commerce`。最后才合并真正重复的适配器，避免因名称相同而过早抽象不同语义
的采集器。
## 8. 验收原则
工程验收依赖自动化测试、静态扫描、源码哈希清单、入口覆盖、旧根目录引用扫描、
隔离旧目录的导入/dry-run、调度生成和历史回放。四模块所有纳入迁移范围的源码必须
在新项目中有目标文件和来源哈希；所有真实入口必须标明 scheduled、manual、library
或 intentionally-excluded，不能静默遗漏。生产验收必须以真实任务运行记录为证据：
日任务连续 7 天，周任务连续 2 个周期，月任务通过历史月份回放。任何缺少证据的
项目保持未勾选状态。
## 9. 统一运行时适配器
每个可执行脚本以完整 `script_id=<module>:<entry>` 作为运行时隔离主键。
`config/runtime-bindings.json` 显式保存 130 个脚本的固定 CDP 端口，范围为
`22000..22999`；配置加载时必须同时满足入口 100% 覆盖、端口唯一和回环地址约束。
新增脚本只能领取未使用端口，已有脚本端口不得随发现顺序漂移。
每个脚本的浏览器状态固定落到：
```text
<GYXX_DATA_ROOT>/state/browser/<module>/<script-name>-<id-digest>/
profile/
cookies.json
storage_state.json
```
同一脚本跨运行复用上述目录，不同脚本不共享端口、Profile、Cookie 或 storage state。
顶层 workflow 和 `gyxx scripts run` 均由 `ModuleCommandAdapter` 注入绑定；嵌套 Python
进程在业务模块导入前由 adapter-owned `sitecustomize` 按真实子脚本路径重新绑定，且会
纠正父进程遗留的 `--cdp-port`、`--cdp-url` 和 `--user-data-dir` 参数。PowerShell/BAT
需要浏览器时通过 `python -m gyxx_flow.runtime_exec env <script-id>` 读取目标叶子脚本绑定。
外部系统策略是运行时硬边界，而不是更换原业务后端：
- 飞书继续使用原来的 lark-cli profile、身份和原 OpenAPI 应用；适配器只透传并禁止误覆盖。
- PostgreSQL 继续使用现有云端配置，统一映射 `PG_*`、`DB_*`、`AUTOFLOW_PG_*`，拒绝回环数据库。
- Hermes 继续使用本机 HTTP/CLI，所有 HTTP 端点只允许 `localhost`、`127.0.0.1` 或 `::1`。
- Cookie 和 storage state 通过原子替换保存；日志和绑定输出只包含路径、端口和计数，不包含值。
dry-run 只解析和验证绑定，不创建 Profile/Cookie 目录，不探测或启动浏览器，也不触发任何
飞书、数据库或 Hermes 调用。
## 10. 采集数据统一分层
`GYXX_DATA_ROOT` 表示整套运行数据根，不直接等同于 raw 目录。四个业务模块通过共享的 `ModuleDataPaths` 解析固定目录，模块脚本继续使用原有公开常量，避免业务逻辑与物理目录结构耦合：
```text
<GYXX_DATA_ROOT>/
data/
raw/<module>/ # 原始 JSON、CSV、Excel、下载文件、原始截图
normalized/<module>/ # 清洗并统一字段后的数据
curated/<module>/ # 聚合、分析和可供下游复用的数据
exports/<module>/ # Markdown、Excel 等面向人员或外部消费的报告
evidence/<module>/ # 产物清单、哈希和验收证据
state/<module>/ # checkpoint；浏览器状态仍按 script_id 隔离
logs/<module>/
tmp/<module>/
```
raw 数据保持追加语义；浏览器直接下载的 XLS/XLSX/CSV、接口原始 JSON 和采集截图归 raw，处理过程中的临时下载归 tmp，清洗结果归 normalized，跨来源汇总归 curated，最终 MD/XLSX 报告归 exports。现有 `data/raw/<module>/legacy` 只读保留，不在本次收口中移动或重写。路径对象只负责解析，不在导入或 dry-run 时创建目录，因此项目根和数据根可分别迁移。
docs/acceptance-report.md
+139
View File
@@ -0,0 +1,139 @@
# GYXX Flow 工作流合并验收报告
验收日期：2026-08-01
## 验收结论
项目工作流目录已收敛为 21 个真实且启用的调度任务。每个调度任务对应一个 LangGraph 工作流；补采、重试、映射刷新和受保护写操作不再重复注册为工作流，统一通过显式命令执行。
重复且原本禁用的 `content.self_operated.weekly` 已删除，其自营映射、B 站、蝉妈妈和指标同步脚本继续由每日 `content.metrics.daily` 调用。京东与天猫主图周采集收敛为唯一工作流 `product.main_image.weekly`，其余时间规则和失败隔离边界保持不变。
## 数量总览
| 项目 | 数量 | 说明 |
|---|---:|---|
| 业务模块 | 4 | 内容营销、商品经营、店铺洞察、供应链 |
| 调度工作流 | 21 | 内容 8、商品 7、店铺 3、供应链 3 |
| 启用状态 | 21 / 21 | 所有当前调度均启用 |
| LangGraph 节点 | 29 | 所有调度工作流均声明显式节点 |
| 公开命令 | 31 | 覆盖工作流节点及手动补偿入口 |
| 浏览器状态绑定 | 136 | 唯一 CDP、Profile、Cookie 与 storage state |
| 源同步清单 | 233 | 运行时不依赖外部源码目录 |
## 工作流清单
所有时间均使用 `Asia/Shanghai`。
### 内容营销（8）
| 工作流 | 时间 | 用途 |
|---|---|---|
| `content.cooperations.daily` | 每日 09:00 | 刷新业务映射并同步合作记录。 |
| `content.marketing_report.daily` | 每日 10:00 | 生成营销日报并按现有飞书策略发送。 |
| `content.metrics.daily` | 每日 22:00 | 依次完成合作达人、自营映射、自营 B 站、蝉妈妈和指标同步。 |
| `content.relogin.weekly` | 周五 10:00 | 维护各内容平台可复用登录状态。 |
| `content.comments.weekly` | 周日 12:00 | 并行补采 B 站、小红书和抖音评论。 |
| `content.summary.weekly` | 周二 10:00 | 生成跨平台内容周汇总。 |
| `content.summary.monthly` | 每月 1 日 08:00 | 生成跨平台内容月汇总。 |
| `content.creator_report.monthly` | 每月 1 日 08:30 | 生成达人月度报告。 |
### 店铺洞察（4）
| 工作流 | 时间 | 用途 |
|---|---|---|
| `shop.metrics.weekly` | 周一 12:00 | 并行采集京东、抖音和天猫店铺经营指标。 |
| `shop.competitor.weekly` | 周一 12:30 | 采集京东、抖音竞店数据并形成对比结果。 |
| `shop.douyin_price_appeal` | 每日 08:00、16:00、22:00 | 复用店铺周采集登录态，扫描全部待改价列表并对可申诉商品提交固定原因申诉。 |
| `shop.jd_self_operated.daily` | 每日 16:00，业务日 -1 | 京东自营数据从下午开始推送；预留刷新窗口后先采集品牌，再采集商品，品牌失败后商品仍继续。 |
### 商品经营（7）
| 工作流 | 时间 | 用途 |
|---|---|---|
| `product.daily` | 每日 08:40，业务日 -1 | 编排 ERP 与三平台采集、分析、导出和入库。 |
| `product.persona.daily` | 每日 10:00 | 并行采集天猫、抖音和京东商品画像。 |
| `product.import.daily` | 每日 19:00 | 幂等导入三平台商品日报。 |
| `product.alert.daily` | 每日 23:00 | 独立检测商品异常并按条件通知。 |
| `product.style_analysis.interval` | 每 3 天 11:00 | 使用本地 Hermes 分析到期款式。 |
| `product.main_image.weekly` | 周日 08:30 | 并行启动 JD、TM 两个独立主图分支，各自写入本地 PG 和飞书；一个平台失败不阻断另一平台，全部结束后汇总整体状态和平台级具体错误。 |
| `product.market_rank` | 周一 10:00 | 并行采集天猫、京东、抖音市场排行并汇总归档。 |
### 供应链（3）
| 工作流 | 时间 | 用途 |
|---|---|---|
| `supply.replenishment_alert.daily` | 每日 07:00 | 检测补货风险并执行告警链路。 |
| `supply.purchase_confirmation.daily` | 每日 08:00 | 采集采购确认数据，经本地 Hermes 分析后通知。 |
| `supply.replenishment.weekly` | 周一 08:00 | 采集并分析周度补货数据。 |
## 手动能力合并结果
| 原目录入口 | 当前入口 | 合并结果 |
|---|---|---|
| `content.mapping.refresh` | `content.mapping.rebuild` | 改为内容映射重建命令。 |
| `content.retry_failed` | `content.failed.retry` | 改为失败任务重试命令。 |
| `content.metrics.backfill` | `gyxx backfill content.metrics.daily` | 合并到标准内容日报图，按日期范围重跑。 |
| `shop.jd_self_operated.history` | `shop.jd_self_operated.collect_brand` | 保留品牌单日回采；`--date` 自动渲染起止日期。 |
| `product.backfill` | `product.backfill.run` | 保留商品补采语义；`--date` 自动渲染 `--from/--to`。 |
| `product.review_collection` | `product.review.orchestrate` | 保留评价补采；`--date` 自动渲染目标日期。 |
| `supply.purchase_order_update` | `supply.workflow.run` | 保留受保护 ERP 更新；自动选择 `mcp-run purchase-order-update`。 |
`content.self_operated.weekly` 因与每日指标工作流重复且原调度已禁用而删除；底层自营脚本继续保留。`product.weekly_aggregate.documented_missing` 没有对应脚本和调度任务，也只在历史证据中保留说明。
## 整体架构
```mermaid
flowchart TD
S["Python Scheduler"] --> C["21 Scheduled Workflows"]
U["CLI Manual Commands"] --> M["31 Explicit Commands"]
C --> G["LangGraph StateGraph"]
G --> B1["content_marketing"]
G --> B2["product_commerce"]
G --> B3["shop_intelligence"]
G --> B4["supply_chain"]
M --> B1
M --> B2
M --> B3
M --> B4
B1 --> A["Shared Adapters"]
B2 --> A
B2 --> MI["product.main_image.weekly"]
MI -->|"parallel independent branch"| JD["JD collect + PG + Feishu"]
MI -->|"parallel independent branch"| TM["TM collect + PG + Feishu"]
B3 --> A
B4 --> A
A --> P["Local PostgreSQL"]
A --> H["Local Hermes Collector / Analyzer"]
A --> F["Existing Feishu Integration"]
A --> R["Per-script Browser State"]
G --> J["Journal / Locks / Effect Ledger"]
G --> D["Layered Data Layout"]
```
业务模块只依赖共享 `core`、`workflow`、`adapters` 和 `ops` 契约，不跨模块导入内部实现。时间规则集中在 `config/schedules.json`，生产环境只守护一个 `gyxx schedule run` 进程。
## 数据与外部系统边界
- PostgreSQL：`127.0.0.1:5432/gyxx_super_data`，非回环地址会被拒绝。
- Hermes：本机 analyzer 与 collector 两个角色，分别使用独立 API 和 gateway。
- 飞书：保持现有身份、应用和业务调用方式，凭据只从运行环境注入。
- 浏览器：每个脚本独立 CDP 端口、Profile、Cookie 和 storage state。
- 数据：JSON、CSV、Markdown、Excel、下载文件和截图按模块及处理阶段写入 `GYXX_DATA_ROOT`。
## 工程验收
全量回归结果为 `940 passed, 1 skipped`。
本次主图合并、失败语义与双 sink 静态定向回归为 `21 passed`；自动化测试没有连接生产 PostgreSQL，也没有执行真实飞书写入。另已在本地 `gyxx_super_data` 将 `main_image_creatives` 主键迁移为四列，3681 条存量数据数量不变；同键 JD/TM 双行事务探针成功并已回滚测试数据。
- [x] 工作流目录仅包含 21 个有效调度任务，且与 21 条时间规则一一对应。
- [x] 21 个当前调度全部启用，重复禁用项已删除。
- [x] 主图调度仅保留 `product.main_image.weekly`，JD、TM 以独立资源并行编排；分支失败隔离、成功写入保留、最终状态汇总和平台级错误明细已有配置和自动化测试覆盖。
- [x] 京东、天猫入口默认均调用飞书插入与本地 PG upsert 链路；相关 21 项定向测试通过，不代表已执行真实外部写入。
- [x] 本地 PG 存量主键已完成平台隔离迁移；同日、同款、同图片名的京东与天猫记录可同时存在。
- [x] 7 个手动能力已转为命令或合并进标准工作流，默认仍为 dry-run。
- [x] 单日回采、商品补采、评价补采和采购单更新具备安全默认参数。
- [x] 不可执行历史项不再参与目录、注册和数量统计。
- [x] 最终全量测试、Ruff、构建、doctor、调度 dry-run 与敏感信息扫描通过。
真实浏览器登录、飞书写入、Hermes 分析和 ERP 修改不由本次结构合并自动触发。各业务链路最近一次实跑结果与未通过原因见 [工作流验收测试报告](workflow-acceptance-test-report.md)。
docs/architecture.md
+106
View File
@@ -0,0 +1,106 @@
# GYXX Flow 架构
## 目标
GYXX Flow 是一个 Python 3.12 模块化单体。项目在同一部署单元中提供工作流目录、LangGraph 编排、Python 定时调度、外部系统适配和运行审计，同时保持业务模块之间互不依赖。
核心约束：
- 每个定时任务都是一个独立工作流，并编译为 LangGraph `StateGraph`。
- 业务模块只能依赖共享契约，不能导入其他业务模块的内部实现。
- PostgreSQL 使用运行时注入的云端连接；Hermes 使用本机回环地址；飞书保持现有身份和调用方式。
- 手动执行默认 dry-run，真实外部副作用必须显式使用 `--execute`。
- 代码、配置和运行数据分离；生产数据根目录必须位于项目目录之外。
## 项目边界
```text
gyxx-flow/
├─ src/gyxx_flow/
│ ├─ core/ 配置、上下文、产物、记录和锁
│ ├─ workflow/ LangGraph 构建、步骤协议和执行引擎
│ ├─ adapters/ 进程、浏览器和外部服务适配
│ ├─ modules/ 四个业务模块，生产代码直接位于模块目录
│ ├─ source_sync/ 可选的业务源码漂移检查
│ └─ scheduler_service.py 跨平台 Python 常驻调度器
├─ config/ 工作流、时间规则和运行绑定
├─ deploy/ PostgreSQL 与进程守护定义
├─ docs/ 架构、部署和运维主文档
└─ tests/ 开发和验收测试，不进入生产运行包
```
`var/` 只是开发环境的默认运行目录，不是源码。生产环境通过 `GYXX_DATA_ROOT` 使用独立持久化目录。
## 运行链路
```mermaid
flowchart TD
A["CLI / Python Scheduler"] --> B["WorkflowCatalog"]
B --> C["Workflow Registry + Factory"]
C --> D["LangGraph StateGraph"]
D --> E1["content_marketing"]
D --> E2["product_commerce"]
D --> E3["shop_intelligence"]
D --> E4["supply_chain"]
E1 --> F["Shared Adapters"]
E2 --> F
E3 --> F
E4 --> F
F --> G1["Cloud PostgreSQL"]
F --> G2["Local Hermes collector / analyzer"]
F --> G3["Existing Feishu identity"]
F --> G4["Per-script CDP and browser state"]
D --> H["Journal / Locks / EffectLedger"]
D --> I["Layered DataLayout"]
```
## 工作流与调度
`config/workflows.json` 只保存由 Python 调度器托管的工作流，记录稳定 ID、所属模块、入口、显式步骤、依赖和失败策略。每个目录项必须在 `config/schedules.json` 中有且只有一条时间规则。模块工厂负责注入超时、重试、资源锁和副作用策略。
`config/commands.json` 保存可手动执行的脚本白名单，也承载补采、重试、映射刷新和受保护写操作。命令不是新的定时工作流；`--date` 可渲染命令声明的 `{business_date}` 默认参数，`--arg` 用于追加脚本参数。
`config/schedules.json` 只保存时间规则。`scheduler_service.py` 负责 `Asia/Shanghai` 时区计算、业务日期偏移、时间槽防重、同工作流不重叠、有限错过触发补偿和优雅停止。
服务器只守护一个 `gyxx schedule run` 进程。systemd 或兼容的 NSSM 只负责进程生命周期，不保存业务时间规则；不得再向 Windows Task Scheduler 或多条 cron 复制工作流时间。
## 外部系统
- PostgreSQL：云端模式不提供地址、数据库、用户或密码的源码默认值；由 `GYXX_POSTGRES_DSN` 在运行时注入，并拒绝回环数据库地址。
- Hermes：保留本机 `data-analyzer` 和 `data-collector` 两个角色，以及各自 API 和 gateway。
- 飞书：复用现有 lark-cli profile、应用身份、表格和消息调用方式。
- 浏览器：每个脚本从 `config/runtime-bindings.json` 获得唯一 CDP 端口及独立 Profile、Cookie、storage state 路径，不共享可写 Profile。
## 数据布局
所有运行路径由 `GYXX_DATA_ROOT` 推导：
```text
data/raw 原始响应、JSON、CSV、Excel、下载文件和截图
data/normalized 清洗和标准化数据
data/curated 聚合事实和业务结果
data/exports Markdown、Excel 等交付文件
data/evidence 对账、验收和追溯证据
state 调度、图状态、锁、账本和浏览器状态
logs 调度器与工作流日志
tmp 可清理临时文件
```
原始数据采用追加式保存。迁移部署时停止调度器，完整复制数据根目录并重新注入 `GYXX_DATA_ROOT`，不修改代码中的路径。
## 扩展方式
新增定时任务时：
1. 在所属 `modules/<module>/` 中实现可独立测试的业务入口。
2. 在工作流目录声明 LangGraph 步骤和依赖，并同步增加唯一时间规则。
3. 为工作流使用的脚本登记公开命令和独立浏览器绑定。
4. 通过共享适配器访问数据库、Hermes、飞书和数据目录。
5. 增加 dry-run、失败边界、业务日期和模块回归测试。
新增补采或维护能力时，只在命令目录注册，不增加工作流或调度规则；只有形成新的独立定时任务后才升级为工作流。
新增模块不应要求修改其他业务模块；新增步骤不应要求修改调度器或执行引擎。
四个模块下的 `runtime/__init__.py` 仅用于兼容历史 Python 导入路径，不承载生产
代码。项目内部代码和新增实现必须使用模块正式路径。
docs/deployment.md
+132 -27
View File
@@ -1,39 +1,144 @@
# 部署手册
# GYXX Flow 部署
## 推荐部署方式
生产环境推荐使用 Linux 主机：
- PostgreSQL 使用云端服务，完整 DSN 由服务器密钥环境注入。
- Python 调度器、本机 Hermes 和需要登录状态的浏览器运行在宿主机。
- systemd 只守护一个 Python 调度进程，所有业务时间规则仍来自 `config/schedules.json`。
- 运行数据使用 `/var/lib/gyxx-flow`，不得把生产 `GYXX_DATA_ROOT` 指向源码目录中的 `var/`。
这种方式能够直接访问两个本机 Hermes 角色和浏览器 CDP，也不会把业务定时规则复制到 systemd timer、cron 或 Windows Task Scheduler。
## 环境要求
- Windows 10/11 或 Windows Server，系统时区 `China Standard Time`
- Python 3.12、`uv`、Windows Task Scheduler
- 凭据由环境变量或外部密钥系统提供，不写入源码或清单
- Python 3.12 和 `uv`
- 可访问的 PostgreSQL 13+ 云端实例及运行时注入的 `GYXX_POSTGRES_DSN`
- 本机 Hermes `data-analyzer` 与 `data-collector`
- Chrome/Playwright，以及个别业务入口仍需要的 PowerShell 运行条件
- 可访问现有飞书身份的专用系统用户
## 安装
创建生产目录和服务账户：
```powershell
cd D:\gyxx-flow
uv sync --python 3.12 --extra test
$env:GYXX_DATA_ROOT = 'D:\gyxx-flow\var'
.\.venv\Scripts\python.exe -m gyxx_flow doctor --json
.\.venv\Scripts\python.exe -m pytest
```bash
sudo useradd --system --create-home --shell /usr/sbin/nologin gyxx-flow
sudo install -d -o gyxx-flow -g gyxx-flow /opt/gyxx-flow
sudo install -d -o gyxx-flow -g gyxx-flow /var/lib/gyxx-flow
sudo install -d -o root -g gyxx-flow -m 0750 /etc/gyxx-flow
```
不再配置任何 `GYXX_LEGACY_*_ROOT`。运行代码和资源随 `gyxx_flow` 包部署，
四个旧项目可以不挂载。商品模块需要独立配置时，可设置 `GYXX_PRODUCT_CONFIG`；
该变量只指向新部署的配置文件。
将代码发布到 `/opt/gyxx-flow` 后安装锁定依赖：
## 生成候选调度计划
```powershell
.\.venv\Scripts\python.exe -m gyxx_flow schedule plan `
--output D:\gyxx-flow\var\schedule-plan\candidate `
--start-date 2026-07-27 `
--python-executable D:\gyxx-flow\.venv\Scripts\python.exe
```bash
cd /opt/gyxx-flow
sudo -u gyxx-flow uv sync --python 3.12 --no-group dev --frozen
```
输出包含 21 个 XML、`install.ps1`、`plan.json` 和 `drift.json`。生成计划不会
注册任务。只有生产门禁通过并获得明确授权后，才能人工审阅并逐个使用
`install.ps1 -Apply -WorkflowId <id>`；安装器强制一次只处理一个任务。
凭据放入 `/etc/gyxx-flow/gyxx-flow.env`，权限设为 `0640`。该文件不提交到 Git，至少按实际环境注入数据库密码、飞书身份和可选 Hermes 密钥。
## 可迁移部署
控制台和调度器都支持可重复的 `--env-file`，只加载显式列出的文件。systemd 的
`EnvironmentFile=` 或当前进程环境优先于文件中的同名值，避免本地文件意外覆盖密钥系统
注入值。
复制项目或安装 wheel 后，只需重新设置 `GYXX_DATA_ROOT` 和凭据。禁止在生产任务
命令中出现旧项目盘符、用户目录解释器或旧项目工作目录。
## PostgreSQL
从受限环境文件加载云端数据库连接：
```bash
cd /opt/gyxx-flow
set -a
source /etc/gyxx-flow/gyxx-flow.env
set +a
uv run gyxx doctor --json
```
生产环境必须设置 `GYXX_POSTGRES_DSN`，且 DSN 主机必须是非回环地址。项目不会把云端地址、用户名或密码写入源码；`deploy/postgres.compose.yml` 仅保留为开发和恢复场景的可选本地工具，不是当前生产数据库入口。
## 本机 Hermes
启动并验证两个独立角色：
```text
data-analyzer: API base http://127.0.0.1:8642/v1
data-collector: API base http://127.0.0.1:8643/v1
```
`28790/28791` 不作为工作流业务端点。
运行时配置必须保持回环地址。Hermes 不可用时，纯采集、文件处理和数据库同步仍可运行；依赖 Hermes 分析或通知的工作流应保持停用或手工执行，不得静默改用远程 AI。
## 上线前验证
使用生产服务账户运行：
```bash
cd /opt/gyxx-flow
sudo -u gyxx-flow env GYXX_DATA_ROOT=/var/lib/gyxx-flow uv run gyxx doctor --json
sudo -u gyxx-flow env GYXX_DATA_ROOT=/var/lib/gyxx-flow uv run gyxx schedule run --dry-run --once
sudo -u gyxx-flow env GYXX_DATA_ROOT=/var/lib/gyxx-flow uv run gyxx list --json
```
开发或发布流水线另外执行：
```bash
uv run ruff check src tests
uv run pytest
uv build
uv run gyxx acceptance status --json
```
## systemd 调度服务
项目提供 `deploy/gyxx-flow.service`。安装并启动：
```bash
sudo cp /opt/gyxx-flow/deploy/gyxx-flow.service /etc/systemd/system/gyxx-flow.service
sudo systemctl daemon-reload
sudo systemctl enable --now gyxx-flow.service
sudo systemctl status gyxx-flow.service
```
unit 的唯一业务入口是：
```text
/opt/gyxx-flow/.venv/bin/python -m gyxx_flow schedule run
```
若 unit 不使用 systemd `EnvironmentFile=`，入口必须显式追加：
```text
--env-file /etc/gyxx-flow/gyxx-flow.env
```
调度状态写入 `/var/lib/gyxx-flow/state/scheduler`，日志和子进程产物写入同一外置数据根。修改 `config/schedules.json` 后先运行一次 dry-run，再重启服务：
```bash
sudo -u gyxx-flow env GYXX_DATA_ROOT=/var/lib/gyxx-flow \
/opt/gyxx-flow/.venv/bin/python -m gyxx_flow schedule run --dry-run --once
sudo systemctl restart gyxx-flow.service
```
不要为单个工作流创建 systemd timer 或 cron 条目，也不得同时运行两个调度器实例。
## 发布更新
```bash
sudo systemctl stop gyxx-flow.service
cd /opt/gyxx-flow
# 切换到已验收版本后：
sudo -u gyxx-flow uv sync --python 3.12 --no-group dev --frozen
sudo -u gyxx-flow env GYXX_DATA_ROOT=/var/lib/gyxx-flow uv run gyxx doctor --json
sudo -u gyxx-flow env GYXX_DATA_ROOT=/var/lib/gyxx-flow uv run gyxx schedule run --dry-run --once
sudo systemctl start gyxx-flow.service
```
代码发布和回滚都不得覆盖 `/var/lib/gyxx-flow`。
## Docker 边界
当前 Compose 只负责 PostgreSQL。完整应用若进入容器，容器内 `127.0.0.1` 不再指向宿主机的两个 Hermes 和浏览器 CDP；同时部分工作流仍可能依赖可见桌面登录或 PowerShell。因此在完成网络、安全、浏览器 Profile 持久化和目标工作流验收前，不将完整生产应用声明为纯容器部署。
## Windows 兼容入口
`deploy/windows-service/` 暂时保留为 legacy NSSM 兼容入口。NSSM 只守护同一个 `gyxx schedule run` 进程，不注册 Windows Task Scheduler，也不保存业务时间规则。新服务器部署以 Linux systemd 为准。
docs/history/content-windows-launchers/README.md
+6
View File
@@ -0,0 +1,6 @@
# Archived content-marketing Windows launchers
These BAT and PowerShell files are retained only as migration history. They are
not production commands, are not registered by GYXX Flow, and are not used by
the Python scheduler. The active workflow steps are declared in
`config/workflows.json` and scheduled through the Python scheduler service.
src/gyxx_flow/modules/content_marketing/runtime/data/tools/daily_marketing_report.bat → docs/history/content-windows-launchers/daily_marketing_report.bat
+3 -16
View File
@@ -1,7 +1,7 @@
@echo off
if not defined GYXX_PYTHON set "GYXX_PYTHON=python"
REM Daily SKU marketing operations report: Hermes analysis + Feishu dashboard and full report
REM Register: schtasks /Create /SC DAILY /TN YingxiaoYunying_DailyMarketingReport /TR %PROJECT_DIR%\data\tools\daily_marketing_report.bat /ST 10:00 /F
REM Historical launcher retained for provenance; production scheduling is owned by gyxx schedule run.
setlocal
@@ -31,21 +31,8 @@ set LOG_FILE=%LOG_DIR%\daily_marketing_report_%TS%.log
echo === Daily marketing report started at %date% %time% === > "%LOG_FILE%"
echo PROJECT_DIR=%PROJECT_DIR% >> "%LOG_FILE%"
REM Wait for today's daily_run to finish (poll every 60s, max 90 min), so that
REM cmt_notes is synced before the report reads yesterday's new notes.
set TODAY=%date:~0,4%%date:~5,2%%date:~8,2%
set /a WAITED=0
:wait_daily_run
set DAILY_DONE=
for %%F in ("%LOG_DIR%\daily_run_%TODAY%_*.log") do findstr /C:"Daily run finished" /C:"Daily run skipped" "%%F" >nul 2>&1 && set DAILY_DONE=1
if defined DAILY_DONE goto daily_run_ready
if %WAITED% GEQ 5400 goto daily_run_timeout
timeout /t 60 /nobreak >nul
set /a WAITED+=60
goto wait_daily_run
:daily_run_timeout
echo WARN: daily_run not finished after 90 min wait, generating report anyway >> "%LOG_FILE%"
:daily_run_ready
REM Exposure collection finishes at 22:00. The 10:00 report reads the latest
REM completed database snapshot and must not wait for a same-day collection.
cd /d "%PROJECT_DIR%"
call %PYTHON% -u -X utf8 daily_marketing_report.py --send >> "%LOG_FILE%" 2>&1
docs/history/content-windows-launchers/daily_run.bat
+67
View File
@@ -0,0 +1,67 @@
@echo off
if not defined GYXX_PYTHON set "GYXX_PYTHON=python"
REM Daily 22:00 task: collect collaborator + self-operated exposure, write back to Feishu, then sync cmt_notes.
REM Historical launcher retained for provenance; production scheduling is owned by gyxx schedule run.
setlocal
for %%I in ("%~dp0..\..") do set "PROJECT_DIR=%%~fI"
if not defined GYXX_DATA_ROOT (
if not defined GYXX_PROJECT_ROOT (
echo ERROR: GYXX_DATA_ROOT or GYXX_PROJECT_ROOT must be defined. 1>&2
exit /b 3
)
set "GYXX_DATA_ROOT=%GYXX_PROJECT_ROOT%\var"
)
set LOG_DIR=%GYXX_DATA_ROOT%\logs\content_marketing\scheduled
REM The Python scheduler service may inject a project-specific interpreter.
set PYTHON=%GYXX_PYTHON%
where %PYTHON% >nul 2>&1 || set PYTHON=python
if not exist "%LOG_DIR%" mkdir "%LOG_DIR%"
set TS=%date:~0,4%%date:~5,2%%date:~8,2%_%time:~0,2%%time:~3,2%%time:~6,2%
set TS=%TS: =0%
set LOG_FILE=%LOG_DIR%\daily_run_%TS%.log
echo === Daily run started at %date% %time% === > "%LOG_FILE%"
echo PROJECT_DIR=%PROJECT_DIR% >> "%LOG_FILE%"
cd /d "%PROJECT_DIR%"
echo --- [step 1] run_all.py (collaborator only; includes collaborator mapping refresh) --- >> "%LOG_FILE%"
call %PYTHON% run_all.py --daily-scope >> "%LOG_FILE%" 2>&1
set RC_RUN=%ERRORLEVEL%
echo run_all exit=%RC_RUN% >> "%LOG_FILE%"
echo --- [step 2] refresh_self_mapping.py --- >> "%LOG_FILE%"
call %PYTHON% data\tools\refresh_self_mapping.py >> "%LOG_FILE%" 2>&1
set RC_SELF_MAP=%ERRORLEVEL%
echo refresh_self_mapping exit=%RC_SELF_MAP% >> "%LOG_FILE%"
echo --- [step 3] self_bilibili_scraper.py --- >> "%LOG_FILE%"
call %PYTHON% self_bilibili_scraper.py >> "%LOG_FILE%" 2>&1
set RC_SELF_BILI=%ERRORLEVEL%
echo self_bilibili exit=%RC_SELF_BILI% >> "%LOG_FILE%"
echo --- [step 4] chanmama_scraper.py (self-operated Douyin) --- >> "%LOG_FILE%"
call %PYTHON% chanmama_scraper.py >> "%LOG_FILE%" 2>&1
set RC_SELF_DOUYIN=%ERRORLEVEL%
echo chanmama exit=%RC_SELF_DOUYIN% >> "%LOG_FILE%"
echo --- [step 5] sync_metrics_to_cmt_notes.py --- >> "%LOG_FILE%"
call %PYTHON% data\tools\sync_metrics_to_cmt_notes.py >> "%LOG_FILE%" 2>&1
set RC_SYNC=%ERRORLEVEL%
echo sync exit=%RC_SYNC% >> "%LOG_FILE%"
set FINAL_RC=0
if not "%RC_RUN%"=="0" set FINAL_RC=1
if not "%RC_SELF_MAP%"=="0" set FINAL_RC=1
if not "%RC_SELF_BILI%"=="0" set FINAL_RC=1
if not "%RC_SELF_DOUYIN%"=="0" set FINAL_RC=1
if not "%RC_SYNC%"=="0" set FINAL_RC=1
echo === Daily run finished at %date% %time% (run=%RC_RUN%, self_map=%RC_SELF_MAP%, self_bili=%RC_SELF_BILI%, self_douyin=%RC_SELF_DOUYIN%, sync=%RC_SYNC%, final=%FINAL_RC%) === >> "%LOG_FILE%"
endlocal & exit /b %FINAL_RC%
src/gyxx_flow/modules/content_marketing/runtime/data/tools/daily_run_with_backfill.bat → docs/history/content-windows-launchers/daily_run_with_backfill.bat
+6 -18
View File
@@ -31,31 +31,19 @@ echo PROJECT_DIR=%PROJECT_DIR% >> "%LOG_FILE%"
cd /d "%PROJECT_DIR%"
echo --- [step 1] run_all.py (includes feishu_mapping refresh) --- >> "%LOG_FILE%"
call %PYTHON% run_all.py >> "%LOG_FILE%" 2>&1
call %PYTHON% run_all.py --daily-scope >> "%LOG_FILE%" 2>&1
set RC_RUN=%ERRORLEVEL%
echo run_all exit=%RC_RUN% >> "%LOG_FILE%"
echo --- [step 2] sync_metrics_to_cmt_notes.py (主采集后首次落库) --- >> "%LOG_FILE%"
echo --- [step 2] sync_metrics_to_cmt_notes.py --- >> "%LOG_FILE%"
call %PYTHON% data\tools\sync_metrics_to_cmt_notes.py >> "%LOG_FILE%" 2>&1
set RC_SYNC_PRE=%ERRORLEVEL%
echo sync_pre exit=%RC_SYNC_PRE% >> "%LOG_FILE%"
echo --- [step 3] retry_failed.py --- >> "%LOG_FILE%"
call %PYTHON% data\tools\retry_failed.py --max-attempts 1 --stale-hours 4 >> "%LOG_FILE%" 2>&1
set RC_RETRY=%ERRORLEVEL%
echo retry_failed exit=%RC_RETRY% >> "%LOG_FILE%"
echo --- [step 4] sync_metrics_to_cmt_notes.py (补跑后再次落库) --- >> "%LOG_FILE%"
call %PYTHON% data\tools\sync_metrics_to_cmt_notes.py >> "%LOG_FILE%" 2>&1
set RC_SYNC_POST=%ERRORLEVEL%
echo sync_post exit=%RC_SYNC_POST% >> "%LOG_FILE%"
set RC_SYNC=%ERRORLEVEL%
echo sync exit=%RC_SYNC% >> "%LOG_FILE%"
set FINAL_RC=0
if not "%RC_RUN%"=="0" set FINAL_RC=1
if not "%RC_SYNC_PRE%"=="0" set FINAL_RC=1
if not "%RC_RETRY%"=="0" set FINAL_RC=1
if not "%RC_SYNC_POST%"=="0" set FINAL_RC=1
if not "%RC_SYNC%"=="0" set FINAL_RC=1
echo === Monday run finished at %date% %time% (run=%RC_RUN%, sync_pre=%RC_SYNC_PRE%, retry=%RC_RETRY%, sync_post=%RC_SYNC_POST%, final=%FINAL_RC%) === >> "%LOG_FILE%"
echo === Monday run finished at %date% %time% (run=%RC_RUN%, sync=%RC_SYNC%, final=%FINAL_RC%) === >> "%LOG_FILE%"
endlocal & exit /b %FINAL_RC%
src/gyxx_flow/modules/content_marketing/runtime/data/tools/friday_relogin.bat → docs/history/content-windows-launchers/friday_relogin.bat
View File
src/gyxx_flow/modules/content_marketing/runtime/data/tools/kill_project_chrome.ps1 → docs/history/content-windows-launchers/kill_project_chrome.ps1
View File
src/gyxx_flow/modules/content_marketing/runtime/data/tools/list_project_chrome.ps1 → docs/history/content-windows-launchers/list_project_chrome.ps1
View File
src/gyxx_flow/modules/content_marketing/runtime/data/tools/monday_self_run.bat → docs/history/content-windows-launchers/monday_self_run.bat
View File
src/gyxx_flow/modules/content_marketing/runtime/data/tools/monthly_creator_report.bat → docs/history/content-windows-launchers/monthly_creator_report.bat
View File
src/gyxx_flow/modules/content_marketing/runtime/data/tools/monthly_summary.bat → docs/history/content-windows-launchers/monthly_summary.bat
View File
src/gyxx_flow/modules/content_marketing/runtime/data/tools/sync_cooperations.bat → docs/history/content-windows-launchers/sync_cooperations.bat
View File
src/gyxx_flow/modules/content_marketing/runtime/data/tools/weekly_comment_scrape.bat → docs/history/content-windows-launchers/weekly_comment_scrape.bat
View File
src/gyxx_flow/modules/content_marketing/runtime/data/tools/weekly_summary.bat → docs/history/content-windows-launchers/weekly_summary.bat
View File
docs/history/product-commerce/MIGRATION_STATUS.md
+15
View File
@@ -0,0 +1,15 @@
# Product-commerce migration history
Product-commerce production code now lives directly in
`src/gyxx_flow/modules/product_commerce/`. The module is fully owned by
`gyxx-flow`; it does not import executable code from the legacy checkout.
`launchers_reference/` is provenance only. Those files document the old task
arguments and must never be registered as final scheduled-task actions.
The migrated regression tests live in `tests/modules/product_commerce/` and are
part of the repository's normal pytest collection.
The authoritative source/target hashes, transformations, and six intentionally
excluded diagnostic entries are recorded in
`config/source-manifests/product_commerce.json`.
src/gyxx_flow/modules/product_commerce/runtime/launchers_reference/README.md → docs/history/product-commerce/launchers_reference/README.md
View File
src/gyxx_flow/modules/product_commerce/runtime/launchers_reference/run_alerts.bat → docs/history/product-commerce/launchers_reference/run_alerts.bat
View File
src/gyxx_flow/modules/product_commerce/runtime/launchers_reference/run_daily_collect.bat → docs/history/product-commerce/launchers_reference/run_daily_collect.bat
View File
src/gyxx_flow/modules/product_commerce/runtime/launchers_reference/run_daily_import.bat → docs/history/product-commerce/launchers_reference/run_daily_import.bat
View File
src/gyxx_flow/modules/product_commerce/runtime/launchers_reference/run_daily_import.ps1 → docs/history/product-commerce/launchers_reference/run_daily_import.ps1
View File
src/gyxx_flow/modules/product_commerce/runtime/launchers_reference/run_daily_persona.bat → docs/history/product-commerce/launchers_reference/run_daily_persona.bat
View File
src/gyxx_flow/modules/product_commerce/runtime/launchers_reference/run_style_analysis_3d.bat → docs/history/product-commerce/launchers_reference/run_style_analysis_3d.bat
View File
src/gyxx_flow/modules/product_commerce/runtime/launchers_reference/run_weekly_jd_main_image.bat → docs/history/product-commerce/launchers_reference/run_weekly_jd_main_image.bat
View File
src/gyxx_flow/modules/product_commerce/runtime/launchers_reference/run_weekly_main_image.bat → docs/history/product-commerce/launchers_reference/run_weekly_main_image.bat
View File
src/gyxx_flow/modules/product_commerce/runtime/launchers_reference/run_weekly_market_rank.bat → docs/history/product-commerce/launchers_reference/run_weekly_market_rank.bat
View File
docs/history/shop-windows-task-scheduler/remove_scheduler.ps1
+21
View File
@@ -0,0 +1,21 @@
#Requires -RunAsAdministrator
param(
[string]$ProjectRoot = "",
[string]$NssmPath = "nssm.exe"
)
$ErrorActionPreference = "Stop"
if ([string]::IsNullOrWhiteSpace($ProjectRoot)) {
$ProjectRoot = if ($env:GYXX_PROJECT_ROOT) {
$env:GYXX_PROJECT_ROOT
} else {
(Resolve-Path (Join-Path $PSScriptRoot "..\..\..\..\..\..")).Path
}
}
$uninstaller = Join-Path $ProjectRoot "deploy\windows-service\uninstall.ps1"
if (-not (Test-Path -LiteralPath $uninstaller -PathType Leaf)) {
throw "Server scheduler uninstaller is missing: $uninstaller"
}
& $uninstaller -NssmPath $NssmPath
exit $LASTEXITCODE
docs/history/shop-windows-task-scheduler/setup_scheduler.ps1
+22
View File
@@ -0,0 +1,22 @@
#Requires -RunAsAdministrator
param(
[string]$ProjectRoot = "",
[string]$DataRoot = "",
[string]$NssmPath = "nssm.exe"
)
$ErrorActionPreference = "Stop"
if ([string]::IsNullOrWhiteSpace($ProjectRoot)) {
$ProjectRoot = if ($env:GYXX_PROJECT_ROOT) {
$env:GYXX_PROJECT_ROOT
} else {
(Resolve-Path (Join-Path $PSScriptRoot "..\..\..\..\..\..")).Path
}
}
$installer = Join-Path $ProjectRoot "deploy\windows-service\install.ps1"
if (-not (Test-Path -LiteralPath $installer -PathType Leaf)) {
throw "Server scheduler installer is missing: $installer"
}
& $installer -ProjectRoot $ProjectRoot -DataRoot $DataRoot -NssmPath $NssmPath
exit $LASTEXITCODE
src/gyxx_flow/modules/supply_chain/runtime/skills/analyzer/SKILL.md → docs/history/supply-chain/skills/analyzer/SKILL.md
View File
src/gyxx_flow/modules/supply_chain/runtime/skills/analyzer/workflow/purchase-order-update/SKILL.md → docs/history/supply-chain/skills/analyzer/workflow/purchase-order-update/SKILL.md
View File
src/gyxx_flow/modules/supply_chain/runtime/skills/collector/SKILL.md → docs/history/supply-chain/skills/collector/SKILL.md
View File
src/gyxx_flow/modules/supply_chain/runtime/skills/collector/workflow/purchase-confirmation-workflow/SKILL.md → docs/history/supply-chain/skills/collector/workflow/purchase-confirmation-workflow/SKILL.md
View File
src/gyxx_flow/modules/supply_chain/runtime/skills/collector/workflow/purchase-order-update-workflow/SKILL.md → docs/history/supply-chain/skills/collector/workflow/purchase-order-update-workflow/SKILL.md
View File
src/gyxx_flow/modules/supply_chain/runtime/skills/collector/workflow/replenishment-alert-workflow/SKILL.md → docs/history/supply-chain/skills/collector/workflow/replenishment-alert-workflow/SKILL.md
View File
src/gyxx_flow/modules/supply_chain/runtime/skills/collector/workflow/replenishment-workflow/SKILL.md → docs/history/supply-chain/skills/collector/workflow/replenishment-workflow/SKILL.md
View File
src/gyxx_flow/modules/supply_chain/runtime/skills/collector/workflow/workflow-trigger-only/SKILL.md → docs/history/supply-chain/skills/collector/workflow/workflow-trigger-only/SKILL.md
View File
src/gyxx_flow/modules/supply_chain/runtime/scripts/purchase-confirmation.bat → docs/history/supply-chain/windows-launchers/purchase-confirmation.bat
View File
src/gyxx_flow/modules/supply_chain/runtime/scripts/replenishment-alert.bat → docs/history/supply-chain/windows-launchers/replenishment-alert.bat
View File
src/gyxx_flow/modules/supply_chain/runtime/scripts/replenishment.bat → docs/history/supply-chain/windows-launchers/replenishment.bat
View File
docs/migration-runbook.md
-29
View File
@@ -1,29 +0,0 @@
# 迁移手册
## 当前边界
- 四个旧项目的源码、数据和现有计划任务保持原状。
- 工作流及其业务脚本已复制到 `src/gyxx_flow/modules/*/runtime`，运行时不引用旧项目。
- 21 个定时工作流可生成候选系统任务；默认不应用。
- 所有发现到的本地可执行脚本可通过 `gyxx scripts` 手工 dry-run 或执行。
- 历史数据副本按模块落入新数据根，并保留数量、字节和 SHA-256 对账证据。
## 工程验收
1. 四份 `config/source-manifests/*.json` 的目标文件和 SHA-256 全部通过。
2. 扫描源码、配置和启动器，确认无四个旧项目根目录或 `GYXX_LEGACY_*_ROOT`。
3. 在不设置旧项目环境变量的进程中导入目录、列出脚本并 dry-run 21 个工作流。
4. 执行 `uv run pytest`、`uv run gyxx doctor --json` 和 wheel 构建检查。
5. 只生成调度计划，不注册、不禁用任何系统任务。
## 逐任务生产切换
1. 轮换历史明文凭据，并将新凭据放入外部密钥系统或环境变量。
2. 对目标 workflow 做 dry-run 和 shadow 对账。
3. 审核 run journal、artifact manifest、effect ledger 和 outbox。
4. 取得负责人明确授权，只安装一个新任务并禁用对应旧任务；旧任务不删除。
5. 日任务连续观察 7 天，周任务观察 2 个周期，月任务完成指定历史月回放。
6. 数据库、飞书、文件产物和通知对账通过后，才迁移下一任务。
推荐顺序：`shop_intelligence` → `supply_chain` → `content_marketing` →
`product_commerce`。
docs/operations.md
-29
View File
@@ -1,29 +0,0 @@
# 日常运维手册
## 常用命令
```powershell
gyxx list --json
gyxx scripts list --json
gyxx doctor --json
gyxx acceptance status --json
gyxx run shop.metrics.weekly --date 2026-07-27
gyxx backfill shop.metrics.weekly --from 2026-07-21 --to 2026-07-27
gyxx scripts run shop_intelligence:runners/run_shop.py --date 2026-07-27
```
`run`、`backfill`、`scripts run` 默认 dry-run。`--execute` 才启动本地迁入脚本；
Task Scheduler 使用 `--scheduled`，按 Asia/Shanghai 当日执行。
## 数据与证据
- `var/runs/.../<run_id>/run.json`：步骤、尝试、退出码和 trace
- `var/state/ops/run-index/`：按 run_id/workflow/date/status 查询
- `var/state/ops/effects/`：生产 Sink 幂等回执
- `var/state/ops/outbox/messages/`：外部写入消息
- `var/logs/`、`var/data/evidence/`：日志和截图证据
- `var/state/profiles/`：浏览器 Profile
- `var/state/locks/`：工作流、Profile 和共享资源锁
effect 为 `ambiguous` 或长时间 `in_progress` 时，先人工核对外部系统；不要删除回执
或强制重跑。手工脚本执行同样会产生统一 run journal 和 effect 记录。
docs/plan.md
+77
View File
@@ -0,0 +1,77 @@
# GYXX Flow 重构执行计划
验收规则：只有在具备代码、配置或自动化测试证据后才能勾选。本次验收统一使用 dry-run；手动业务入口默认 dry-run，调度预演显式使用 `--dry-run --once`。本轮不启动真实采集，不写飞书、PostgreSQL、Hermes 或业务浏览器会话。
## P0 业务代码同步
- [x] P0.1 只读比较 4 个业务源码目录与项目清单，识别最新更新、目标适配和冲突。
- [x] P0.2 将最新业务源码完整复制到对应模块，不通过导入、挂载或子进程引用外部项目代码。
- [x] P0.3 对需要统一路径、数据库、Hermes 和浏览器状态的文件执行目标项目适配。
- [x] P0.4 更新 233 个来源/目标清单 SHA-256；同步状态为 233 个 `in_sync`。
## P1 LangGraph 工作流
- [x] P1.1 建立 4 个高内聚业务模块和统一模块契约，禁止业务模块互相导入内部实现。
- [x] P1.2 将每个定时任务声明为独立工作流，并为每个定时工作流显式声明图节点。
- [x] P1.3 21 个定时工作流全部编译为 LangGraph `StateGraph` 并启用。
- [x] P1.4 工作流目录收敛为 21 个有效调度项；7 个手动能力转为命令，重复或失效历史项不再注册。
- [x] P1.5 京东自营每日工作流按品牌、商品两节点执行；品牌失败后商品仍运行，最终状态保留失败。
- [x] P1.6 统一重试、超时、依赖、fan-in、日志、run_id、资源锁和外部副作用幂等契约。
## P2 Python 调度
- [x] P2.1 使用单一 Python 常驻调度器读取 `config/schedules.json`。
- [x] P2.2 支持日、周、月、间隔日、Asia/Shanghai、错过触发补偿和优雅停止。
- [x] P2.3 持久化时间槽并提供单实例锁、同工作流不重叠和重启防重复。
- [x] P2.4 京东自营每日工作流 10:00 运行，业务日期固定为调度日期前一天。
- [x] P2.5 退出 Windows Task Scheduler 执行链；服务器只托管一个 Python 调度进程。
## P3 外部系统与数据目录
- [x] P3.1 PostgreSQL 统一使用运行时注入的云端 DSN，源码不设置地址、数据库、用户或密码默认值，回环地址立即拒绝。
- [x] P3.2 Hermes 保持本机采集端和分析端两个角色，统一配置 2 个 API 和 2 个 gateway。
- [x] P3.3 飞书保持既有身份、应用和调用方式，密钥只从运行时环境注入。
- [x] P3.4 31 个公开命令显式注册；136 个浏览器状态键各自使用唯一 CDP 端口、Profile、Cookie 和 storage state。
- [x] P3.5 JSON、Markdown、CSV、Excel、下载文件和截图按模块及数据阶段分目录保存。
- [x] P3.6 运行状态、日志、临时文件与业务数据分离，整体可随 `GYXX_DATA_ROOT` 迁移。
## P4 可部署性与低耦合
- [x] P4.1 项目运行不依赖外部源码目录、固定盘符或 Windows 定时任务。
- [x] P4.2 生产代码直接位于 `modules/<module>/`，业务模块只依赖共享 workflow、adapters、core 契约。
- [x] P4.3 提供本地 PostgreSQL Compose，初始化 4 个模块 schema 且只监听回环地址。
- [x] P4.4 手动命令保留 dry-run 默认值；调度器提供显式 dry-run 预演。
## P5 自动化验收
- [x] P5.1 工作流目录、模块注册、LangGraph、fan-in、调度日期和失败语义测试通过。
- [x] P5.2 全量项目回归 940 passed、1 skipped。
- [x] P5.3 Ruff、compileall、构建和差异格式检查通过。
- [x] P5.4 `gyxx doctor`、`gyxx list`、调度 dry-run 和验收状态通过。
- [x] P5.5 敏感信息扫描、本地服务策略、31 个公开命令、136 个浏览器绑定和可迁移性测试通过。
- [x] P5.6 最终验收报告记录完整工作流清单、用途、架构和验证证据。
## P6 目录精简
- [x] P6.1 四个模块生产代码上移到模块根，`runtime/` 只保留单文件历史导入兼容层。
- [x] P6.2 47 个模块内测试统一迁入 `tests/modules/`，生产包不再夹带测试目录。
- [x] P6.3 Windows 启动器和历史技能说明归档到 `docs/history/`，不参与命令发现或服务器调度。
- [x] P6.4 脚本发现改为 `config/commands.json` 显式白名单，不再递归暴露内部工具和历史脚本。
- [x] P6.5 项目文档收口为架构、部署、运维、计划和验收报告，删除重复跳转文档。
- [x] P6.6 四个模块的新路径、命令别名、浏览器状态键和源同步哈希均完成回归校验。
## P7 工作流合并
- [x] P7.1 核对实际调度来源，确认 21 条有效调度定义并全部启用。
- [x] P7.2 从工作流目录移出 7 个手动入口，并删除 1 个不可执行历史目录项。
- [x] P7.3 通过命令注册保留补采、重试、映射刷新和采购单更新能力。
- [x] P7.4 为单日回采、商品补采、评价补采和采购单更新增加业务日期或固定子命令默认参数。
- [x] P7.5 更新项目文档并完成定向、全量、Lint、构建、doctor 和调度 dry-run 验收。
- [x] P7.6 删除重复且禁用的 `content.self_operated.weekly`，保留每日指标工作流仍依赖的底层自营脚本并完成回归。
- [x] P7.7 将京东、天猫主图周任务合并为 `product.main_image.weekly` 单图，周日 08:30 并行启动两个无依赖、资源隔离的分支；任一平台失败都不阻断另一平台采集、PG upsert 和飞书插入，全部结束后汇总失败状态及平台级具体错误。
- [x] P7.8 核对两个主图入口默认均调用飞书插入和本地 PG upsert，并以目录、图依赖、失败语义和入口测试完成双 sink 静态验收；相关 21 项定向测试通过，未将自动化测试表述为真实外部写入。
- [x] P7.9 将本地 `main_image_creatives` 存量主键迁移为日期、款式、平台、图片四列；迁移前后均为 3681 行，并用同键 JD/TM 双行事务探针验证后回滚测试数据。
## 生产启用条件
以下是部署后的运维门禁，不属于本次无副作用代码验收：配置真实凭据、初始化本地数据库、启动两个本机 Hermes 角色、准备浏览器首次登录态，以及逐工作流执行真实数据与外部写入对账。
docs/rollback.md
-17
View File
@@ -1,17 +0,0 @@
# 回滚手册
## 单任务回滚
1. 记录失败的新任务名、workflow ID、run_id 和业务日期。
2. 禁用对应 `\GYXX\<workflow-id>`，保留任务定义和运行证据。
3. 重新启用原计划任务，核对触发时间、账号和旧工作目录仍与基线一致。
4. 检查 `state/ops/effects`；`in_progress` 或 `ambiguous` 必须先做外部对账。
5. 检查 outbox；已 `sent` 的消息不得重发，`failed` 只能以同一幂等键 replay。
6. 记录回滚结果和恢复时间，保留新旧日志。
## 原则
- 一次只切换或回滚一个 workflow。
- 旧任务只禁用/启用，不删除；旧项目在全部观察期完成前保持只读可回滚。
- 不删除新系统数据、run journal、effect receipt 或 outbox 消息。
- 未完成对账时，不允许用直接运行脚本绕过 effect ledger。
docs/runbook.md
+149
View File
@@ -0,0 +1,149 @@
# GYXX Flow 运维手册
## 日常检查
```bash
uv run gyxx doctor --json
uv run gyxx schedule status
uv run gyxx list
```
Linux 服务同时检查：
```bash
systemctl status gyxx-flow.service
journalctl -u gyxx-flow.service --since today
```
工作流失败时，使用 `workflow_id`、业务日期和 `run_id` 对照 `<GYXX_DATA_ROOT>/logs`、运行记录、节点产物和副作用账本。
## 手工执行
```bash
uv run gyxx run <workflow_id> --date 2026-08-01
uv run gyxx run <workflow_id> --date 2026-08-01 --execute
uv run gyxx scripts list --json
uv run gyxx scripts run <command_id> --date 2026-08-01
uv run gyxx scripts run <command_id> --date 2026-08-01 --execute
```
`gyxx run` 只接受 22 个调度工作流；补采和维护操作使用 `gyxx scripts run`。两类入口都默认 dry-run，只有业务日期、凭据、浏览器登录态、PostgreSQL 和对应 Hermes 角色全部确认后才使用 `--execute`。
| 手动场景 | 命令 ID | 日期行为 |
|---|---|---|
| 内容映射重建 | `content.mapping.rebuild` | 业务日期只用于运行审计 |
| 内容失败任务重试 | `content.failed.retry` | 业务日期只用于运行审计 |
| 内容日报重跑 | `gyxx backfill content.metrics.daily` | 使用 `--from/--to` 指定日期范围 |
| 京东自营品牌单日回采 | `shop.jd_self_operated.collect_brand` | `--date` 自动成为起止日期 |
| 商品历史补采 | `product.backfill.run` | `--date` 自动成为 `--from/--to` |
| 商品评价补采 | `product.review.orchestrate` | `--date` 自动成为目标日期 |
| 采购单更新 | `supply.workflow.run` | 自动选择 `mcp-run purchase-order-update` |
命令的额外脚本参数使用可重复的 `--arg=<值>` 传入。`supply.workflow.run --execute` 会进入真实 ERP 修改链路，必须在任务文件、目标范围和回滚条件全部复核后执行。
### 主图周工作流
主图调度的唯一工作流 ID 是 `product.main_image.weekly`，每周日 08:30 启动。图同时运行 JD、TM 两个无依赖分支；一个平台失败不会取消、跳过或回滚另一个平台的采集、PG upsert 和飞书插入。两个分支全部结束后再汇总工作流状态：任一平台失败时整体状态为失败，并在错误摘要中标明具体平台和错误，另一平台已经成功的结果继续保留。
两个平台默认都在采集后执行飞书主图表插入和本地 PostgreSQL `main_image_creatives` upsert。执行前必须同时确认两套浏览器登录态、飞书身份和本地数据库；排障时分别检查 `jd`、`tmall` 节点以及对应插入脚本日志，不能只以采集文件存在判断双 sink 已完成。
## 调度操作
```bash
uv run gyxx schedule run --dry-run --once
uv run gyxx schedule status
sudo systemctl restart gyxx-flow.service
```
调度服务也只加载显式指定的受控环境文件：
```bash
uv run gyxx schedule run --env-file /etc/gyxx-flow/gyxx-flow.env
```
业务时间只在 `config/schedules.json` 中维护。systemd 负责进程守护，Python scheduler 负责全部业务定时；不要重复注册 cron、systemd timer 或 Windows 任务计划。
需要停用单个工作流时，优先修改该工作流的调度启用状态，先 dry-run，再重启服务。其他模块继续运行。
## 数据和浏览器状态
生产 `GYXX_DATA_ROOT` 必须位于源码目录之外，Linux 推荐 `/var/lib/gyxx-flow`：
- `data/raw/<module>`：原始采集结果，不按临时文件清理。
- `data/normalized`、`data/curated`：清洗、标准化和聚合数据。
- `data/exports`：Markdown、Excel 等交付文件。
- `state/browser/<module>/<script>`：独立 Cookie、Profile 和 storage state。
- `state/scheduler`、`state/locks`、运行账本：调度和幂等状态。
- `tmp`：仅存放可重建临时文件。
项目不会自动搬动或删除已有 `var/`。迁移数据根时应停止调度器，完整备份和复制原目录，修改环境变量后依次运行 doctor、dry-run 和一次指定日期的对账执行。
## 清理和保留
可以在确认没有任务运行后清理缓存、`tmp`、dry-run 目录和可重建构建产物。以下内容不得作为普通缓存删除：
- `data/raw` 原始数据；
- `state/scheduler` 和资源锁；
- EffectLedger、运行 journal 和浏览器状态；
- 尚未核对或尚未导入数据库的 Excel、JSON、Markdown 导出。
旧 Windows 任务计划 XML 只属于历史部署证据，确认 Python scheduler 已接管且完成备份后再归档。
## 故障处理
1. 按 `workflow_id`、业务日期和 `run_id` 定位失败节点。
2. 检查本机 PostgreSQL、对应 Hermes 角色、飞书身份和脚本专属 CDP 端口。
3. 检查 Cookie/storage state 是否存在且未失效。
4. 修复后按同一业务日期重跑，依靠唯一键和 EffectLedger 防止重复写入。
5. 仍失败时停用该工作流，不停止其他模块。
### 副作用账本对账
正式节点失败后，EffectLedger 会保留 `ambiguous` 回执，避免不确定状态下重复写库或重复
发送飞书。不要删除或手工修改回执文件。只有先从 PostgreSQL、飞书消息回执和目标文档
确认该节点是否已产生外部效果，才可执行审计式对账：
```bash
uv run gyxx effects reconcile <workflow_id> \
--date 2026-08-04 \
--step <step_id> \
--expected-run-id <原失败run_id> \
--action retry \
--operator <操作人> \
--reason <重试原因> \
--evidence <已核对的证据> \
--execute
```
`retry` 仅在证明确实没有完成外部写入时允许同日重试；若证据表明外部写入已经完成，使用
`--action applied`。命令只接受当前仍为 `ambiguous` 且 `run_id` 精确匹配的回执，并保留
操作人、原因、证据和时间，不能用于绕过正在执行的节点。
## 回滚
代码回滚时先停止 `gyxx-flow.service`，切换到上一个已验收版本，重新安装锁定依赖并执行 doctor 和调度 dry-run；确认后再启动服务。不得用代码回滚覆盖 `GYXX_DATA_ROOT`。
数据撤销必须针对明确表、业务键和 `run_id`，先备份 PostgreSQL，再执行经过复核的 SQL。不得删除整个数据库、数据卷或数据根。
浏览器状态异常时，只备份并处理目标脚本的 `state/browser/<module>/<script>`，不要清理其他脚本状态。调度状态异常时先停止唯一调度进程并备份 `state/scheduler`，不要同时启动第二实例。
## 业务源码同步
外部业务源码目录只用于发现更新，GYXX Flow 运行时不导入或挂载这些目录。只读检查：
```powershell
uv run gyxx sources status `
--source-root content_marketing=<内容源码目录> `
--source-root product_commerce=<商品源码目录> `
--source-root shop_intelligence=<店铺源码目录> `
--source-root supply_chain=<供应链源码目录> `
--json
```
只有未转换、无冲突的 `source_changed` 可以显式执行：
```powershell
uv run gyxx sources apply ... --execute --json
```
经过统一路径、数据库、Hermes、飞书或浏览器适配的文件必须人工合并并运行回归测试。新增可执行脚本还需登记工作流、调度和独立 CDP 绑定。同步完成后执行测试、doctor、acceptance 和 source status；真实采集不在源码同步过程中自动触发。
docs/workflow-acceptance-test-report.md
+197
View File
@@ -0,0 +1,197 @@
# GYXX Flow 全工作流验收测试报告
测试日期：2026-08-03
测试时区：Asia/Shanghai
测试顺序：供应链 → 店铺 → 商品 → 内容
## 1. 最终结论
当前架构共有 21 个定时工作流并全部启用，模块分布为内容 8、商品 7、店铺 3、供应链 3。13 项达到 `PASS` 或 `PASS_WITH_SKIPS`，合并后的 `product.main_image.weekly` 已完成工程结构验证但尚无新两节点图的业务实跑，因此业务通过和规则完成均为 13/21，未通过或待复测为 8/21。7 个手动入口、2 个已删除/废弃项以及 2 条合并前主图子链实跑证据均单列，不进入当前工作流分母。
| 结果 | 数量 | 打勾 | 解释 |
|---|---:|---|---|
| `PASS` | 4 | 是 | 允许执行的采集、处理和本地写入成功；包含合法空结果 |
| `PASS_WITH_SKIPS` | 9 | 是 | 主链路或受控入口验证成功，禁止/无需执行的外部副作用已明确跳过 |
| `FAIL_AUTH_OR_UI` | 1 | 否 | 登录、配置或目标页面控件条件不满足 |
| `PARTIAL_FAILED` | 2 | 否 | 有部分有效结果，但必要子链明确失败 |
| `PARTIAL_TIMEOUT` | 2 | 否 | 有部分有效结果，但整体在有界等待内未收敛 |
| `BLOCKED_EXTERNAL_PERMISSION` | 2 | 否 | 登录成功后被外部系统权限阻断 |
| `PENDING_RETEST_AFTER_MERGE` | 1 | 否 | 合并后的结构已验证，但新拓扑尚无受控业务实跑证据 |
| 合计 | 21 | 13/21 | 13 项业务通过且规则完成，8 项未通过或待复测 |
手动命令历史场景另计：`PASS` 3、`PARTIAL_TIMEOUT` 3、`EXCLUDED_BY_REQUEST` 1，共 7 项；`content.self_operated.weekly` 和 `product.weekly_aggregate.documented_missing` 仅作为已删除/废弃历史证据保存。合并前京东 `PASS` 和天猫 `SKIPPED_COOKIE` 的 dated run 只作子链历史追溯，不能计入上表。
## 2. 云端数据库迁移与本地数据基线
云端 `8.148.185.119/data_hub` 已在 2026-08-01 16:31（Asia/Shanghai）完成一致性快照恢复到本地 `127.0.0.1:5432/gyxx_super_data`，本地所有者和运行角色均为 `gyxx_flow`。迁移凭据只在进程环境中临时注入，没有写入仓库、dump 元数据或报告。
- 云端 PostgreSQL 17.6，本地 PostgreSQL 17.10；源端和目标端均为 359 张业务表。
- 恢复后校验到 5,815 个可见列、606 个索引、433 个约束、337 个序列、7 个触发器、1 个业务函数，语义结构一致。
- 359 张表全部完成行数和双指纹校验：357 张与云端实时状态完全匹配；`qrtz_scheduler_state` 的调度心跳、`system_oauth2_access_token` 及其序列的新增属于一致性 dump 完成后的正常在线漂移，不是恢复丢失。
- 本机没有 PostGIS，迁移只排除了扩展自有表；359 张业务表均无 PostGIS 列，因此不影响业务数据完整性。
- 云端一致性 dump 保存在 `var/migrations/cloud_data_hub_to_gyxx_super_data_20260801T162327/cloud_data_hub.dump`；恢复前本地库 dump 保存在同目录的 `local_gyxx_super_data_before.dump`。
- 恢复前数据库仍保留为 `gyxx_super_data_before_20260801_162327`，已禁止新连接；旧库和两个 custom dump 共同构成可回滚点。
- 恢复使用单事务；临时 `pg_hba.conf` 规则已恢复。连接、schema usage/create 及“建表 → 插入 → 读取 → 更新 → 回滚”探针全部通过，探针未持久化。
- 会产生本地验收写入的实际工作流均在上述迁移和一致性校验完成后才执行；因此 357+2 的迁移结论没有被后续验收数据污染。
权威迁移证据为 `var/migrations/cloud_data_hub_to_gyxx_super_data_20260801T162327/migration-report.json`。
## 3. 隔离、安全与运行环境结论
### 飞书与通知
- 店铺复测中曾因一次命令漏设验收总开关误建 1 条京东周数据记录；随后通过飞书 Base 精确读取核对后只删除该 record id，并回读确认 `record_not_found`。除此之外飞书 Base/Bitable/Sheets 写动作均由验收门禁记录为 `feishu_write_skipped`，最终业务表未留下本轮测试记录。
- 通知策略只允许王云龙作为收件人，其他收件人和业务群在验收态被拒绝。
- 商品告警因无符合阈值的数据没有通知；营销日报未启用发送参数。采购确认 journal 仅记录通用 `step:collect_analyze_notify:production_sink`，没有独立消息正文或投递回执，因此不声称该通用副作用的具体内容或投递结果。
### Hermes
- 分析端 `127.0.0.1:8642`、采集端 `127.0.0.1:8643` 均保持本机回环绑定，项目没有回退到远程 AI。店铺指标采集的分析端请求因当前运行进程未注入 `GYXX_HERMES_API_KEY` 返回 401，AI 分析结果为空，但采集、文件与 PostgreSQL 主链路均成功。
- 营销日报已真实调用本地分析 Hermes 并生成 Markdown/PNG。
- 项目客户端全部使用回环地址；两个 Hermes 进程当前仍监听 `0.0.0.0`，服务器部署前应通过服务配置或防火墙收紧。这是部署风险，不改变本轮本地调用结论。
### 浏览器、Cookie 与来源只读
- 136 个运行绑定保留各自唯一的 CDP/Profile/Cookie/storage-state/state-key 组合。
- 15/15 个已识别的来源状态映射已复制到本项目独立目录，共 5,037,233,753 B、31,213 个文件；运行配置不引用来源路径。
- 实跑先复用本项目/已复制状态，再使用脚本明确支持的账号密码回退；仍不可用才 exit 75。验收过程中未扫码、未人工过验证码、未覆盖来源 Cookie。
- 来源项目和来源浏览器状态保持只读，未被本轮命令写入；来源清单检查为 233/233 `in_sync`。
- JSON、Markdown、CSV、Excel、图片、日志和 journal 均落入本项目 `GYXX_DATA_ROOT` 分层目录；没有把来源目录当运行时依赖。
### 凭据清理
- 历史遗留的实际 `db.env`、`analyze.env` 已删除；项目只保留无凭据示例和统一运行时环境变量入口。
- 全仓 secret scan 结果为 0；报告、验收事件和迁移清单均未记录账号、密码、Cookie、token 或密钥。
## 4. 框架安全修复（6 项）
以下为框架层修复，与下一节业务逻辑对齐分开统计：
1. `CommandStep` 超时时完整终止进程树：Windows 使用 Job Object，POSIX 使用 process group，避免父进程退出后浏览器或子脚本继续产生副作用。
2. secret scanner 收紧动态引用、`getenv` fallback 和 lambda 边界；最终全仓扫描命中 0。
3. 商品配置路径优先级固定为 `GYXX_PRODUCT_CONFIG > AUTO_FLOW_CONFIG > AUTOFLOW_CONFIG_PATH > canonical`，避免旧兼容变量覆盖显式配置。
4. `NamedResourceLock` 增加原子发布、PID create-time/lease 校验和 OS reclaim guard，对陈旧锁采取保守恢复，避免误夺仍存活进程的锁。
5. acceptance evidence 增加跨线程/跨进程文件锁：Windows 使用 `msvcrt`、POSIX 使用 `flock`；`register_at_fork` 后重置状态，锁等待超时按 fail-closed 处理。
6. `_safe_details` 对 `Mapping`、`list`、`tuple` 递归脱敏并检测循环引用，避免嵌套验收细节泄露敏感值或因自引用崩溃。
## 5. 业务逻辑与编排对齐
- 商品日报显式下传 `--target-date {business_date}`，08:40 定时任务使用前一业务日；评价采集同样显式下传目标日期。
- 内容评论恢复 B 站、小红书、抖音三平台并发、全部等待和最终聚合失败语义。
- 内容指标回补恢复“采集失败仍执行同步、最后返回聚合失败”的原逻辑。
- 商品主图收敛为 `product.main_image.weekly`：周日 08:30 并行启动 JD、Tmall 两个独立资源分支，两个子链均执行本地 PG 入库与飞书插入逻辑；任一受控节点失败都不阻断另一平台完成采集和写入，末尾聚合工作流失败状态和平台级具体错误。
- 蝉妈妈保留分页兜底、逐页异常处理及“获取数据 → 最长等待 10 分钟 → 重新进入导出”。
- 商品告警使用统一通知配置，验收态只允许王云龙。
- 供应链补货输出先写本轮目录再归档，停产 SKU 状态改为项目路径；批处理只消费本轮 Excel，空补货结果按合法业务结果处理。
- 供应链外层增加 Cookie/CDP 和 ERP 修改门禁，修复 PowerShell 在 Python 防护前启动浏览器的窗口。
- 21 个定时 `workflow_id` 均编译为 LangGraph `StateGraph`，模块分布为内容 8、商品 7、店铺 3、供应链 3；手动命令继续通过 `config/commands.json` 显式注册，不再混入定时工作流目录，也没有恢复递归脚本发现或 Windows 定时任务。
## 6. 逐定时工作流最终结果
### 供应链
| 完成 | 工作流 | 调度 | 用途 | 最终状态、run_id 与关键证据 |
|---|---|---|---|---|
| ✅ | `supply.purchase_confirmation.daily` | 每天 08:00 | 采购确认采集、分析和通知 | `PASS`；`supply.purchase_confirmation.daily__20260729__20260801T092309Z__5daace`；约 443 秒，ERP 原始 Excel、汇总 CSV、本地 PG 状态均成功。通用 `production_sink` 不作为消息内容/回执证明 |
| ⬜ | `supply.replenishment.weekly` | 周一 08:00 | 计算并输出补货建议 | `BLOCKED_EXTERNAL_PERMISSION`；`supply.replenishment.weekly__20260728__20260801T094413Z__743f4a`；登录成功后 ERP 拒绝“普通商品资料”，exit 1、外部写入为空 |
| ⬜ | `supply.replenishment_alert.daily` | 每天 07:00 | 指定 SKU 库存阈值预警 | `BLOCKED_EXTERNAL_PERMISSION`；`supply.replenishment_alert.daily__20260727__20260801T094546Z__500e91`，复现 `supply.replenishment_alert.daily__20260728__20260801T094100Z__4b2da2`；同一 ERP 权限阻断 |
### 店铺
| 完成 | 工作流 | 调度 | 用途 | 最终状态、run_id 与关键证据 |
|---|---|---|---|---|
| ✅ | `shop.metrics.weekly` | 周一 12:00 | 并行采集京东、抖音、天猫店铺指标 | `PASS`；`shop.metrics.weekly__20260804__20260804T065204Z__bb3b69`；三节点 3/3 成功，产物与云端 PG 写入完成；京东、抖音真实更新飞书 Base，天猫按原逻辑不写店铺表；抖音采用确认后的净销售额 `294,892.98` |
| ✅ | `shop.competitor.weekly` | 周一 12:30 | 并行采集京东、抖音竞店指标 | `PASS`；`shop.competitor.weekly__20260804__20260804T075530Z__4fd7c1`；京东/抖音 2/2 成功并分别 upsert 49/24 行，飞书 Base 分别更新 7/9 条品牌记录 |
| ✅ | `shop.jd_self_operated.daily` | 每天 16:00，业务日 -1 | 京东自营品牌后接商品采集 | `PASS`；`shop.jd_self_operated.daily__20260801__20260804T082226Z__255cfc`；品牌/商品 2/2 成功并分别采集 1/50 行，收入合计均为 `19,974.27`；同日真实重跑 `shop.jd_self_operated.daily__20260801__20260804T082722Z__0ced7e` 再次 2/2 成功且仍为 1/50 行，CDP 22132/22133 均关闭 |
### 商品
| 完成 | 工作流 | 调度 | 用途 | 最终状态、run_id 与关键证据 |
|---|---|---|---|---|
| ⬜ | `product.persona.daily` | 每天 10:00 | 三平台商品人群画像采集 | `PARTIAL_FAILED`；`product.persona.daily__20260815__20260801T103908Z__2d6028`；天猫缺 DMP、抖音缺搜索控件、京东完成 19/30 后浏览器崩溃 |
| ⬜ | `product.daily` | 每天 08:40，业务日 -1 | ERP/三平台采集、分析、导出、入库 | `PARTIAL_TIMEOUT`；`product.daily__20260731__20260801T113513Z__b59304`；ERP/JD/TM 完成，DY 超时，raw 257；独立下游完成分析、导出 32、本地 upsert 与 32 个飞书写跳过 |
| ✅ | `product.alert.daily` | 每天 23:00 | 商品异常检测和条件通知 | `PASS`；`script.product_commerce.955b1dfb0cdf__20260801__20260801T085614Z__4fefb3`；真实读取本地库，无符合阈值异常，未产生通知 |
| ✅ | `product.import.daily` | 每天 19:00 | 三平台日报幂等导入本地 PG | `PASS`；`script.product_commerce.9137239938d5__20260727__20260801T085541Z__59c16c`；抖音/京东/天猫 163/90/173 行，无重复 |
| ✅ | `product.style_analysis.interval` | 每 3 天 11:00 | 本地 Hermes 分析到期款式 | `PASS_WITH_SKIPS`；`product.style_analysis.interval__20260801__20260801T084137Z__952204`；数据库/Hermes 检查和实现入口 dry-run 成功，外部写关闭 |
| ⬜ | `product.main_image.weekly` | 周日 08:30 | JD、Tmall 主图并行采集，两边独立 PG 入库与飞书插入 | `PENDING_RETEST_AFTER_MERGE`；尚无合并后新图的业务 run_id。工程结构要求为两个无依赖且资源隔离的分支；任一分支受控失败不阻断另一分支，全部结束后整体失败并报告平台级具体错误。合并前 run 仅见下方历史证据表 |
| ⬜ | `product.market_rank` | 周一 10:00 | 三平台周市场排行采集、归档 | `FAIL_AUTH_OR_UI`；`product.market_rank__20260814__20260801T103628Z__5a9024`；天猫认证、京东类目配置、抖音控件分别失败；目标与来源脚本哈希一致 |
### 内容
| 完成 | 工作流 | 调度 | 用途 | 最终状态、run_id 与关键证据 |
|---|---|---|---|---|
| ⬜ | `content.metrics.daily` | 每天 22:00 | 达人/自营/B 站/蝉妈妈采集和指标同步 | `PARTIAL_FAILED`；`content.metrics.daily__20260813__20260801T101312Z__28d505`；71 分 05 秒后完整度检查失败，已完成证据和飞书写跳过保留 |
| ✅ | `content.marketing_report.daily` | 每天 10:00 | 从本地数据生成营销日报 | `PASS_WITH_SKIPS`；`script.content_marketing.62b3b647254f__20260804__20260801T091245Z__f20319`；真实 PG + 本地 Hermes，生成 Markdown/PNG，未启用发送 |
| ✅ | `content.relogin.weekly` | 周五 10:00 | 并行刷新内容平台登录态 | `PASS_WITH_SKIPS`；`content.relogin.weekly__20260811__20260801T100706Z__45a8db`；验收态按设计禁止 QR/交互登录并留跳过证据，未修改 Cookie |
| ✅ | `content.creator_report.monthly` | 每月 1 日 08:30 | 生成达人月报 | `PASS_WITH_SKIPS`；`content.creator_report.monthly__20260801__20260801T084140Z__bd9fdd`；数据库检查和入口 dry-run 成功，飞书写关闭 |
| ✅ | `content.summary.monthly` | 每月 1 日 08:00 | 内容月度汇总 | `PASS_WITH_SKIPS`；`content.summary.monthly__20260801__20260801T084142Z__9d73c1`；数据库检查和入口 dry-run 成功，外部写关闭 |
| ✅ | `content.cooperations.daily` | 每天 09:00 | 刷新映射并同步合作记录 | `PASS_WITH_SKIPS`；`content.cooperations.daily__20260801__20260801T084143Z__02b89a`；真实飞书读取与本地 PG 读取，内部 dry-run 禁止写入 |
| ⬜ | `content.comments.weekly` | 周日 12:00 | 三平台评论并行补采和汇总 | `PARTIAL_TIMEOUT`；`content.comments.weekly__20260812__20260801T100904Z__530989`；60 分 56 秒，B 站完成，XHS/DY 有界停止，229 CSV + 229 JSON、约 5.45 MB |
| ✅ | `content.summary.weekly` | 周二 10:00 | 内容周度汇总 | `PASS_WITH_SKIPS`；`content.summary.weekly__20260801__20260801T084145Z__7e6081`；数据库检查和入口 dry-run 成功，外部写关闭 |
### 手动命令/历史验收场景（不计当前工作流分母）
| 旧验收标识 | 合并后的入口与用途 | 历史状态、run_id 与关键证据 |
|---|---|---|
| `supply.purchase_order_update` | `supply.workflow.run`；默认参数自动选择 `mcp-run purchase-order-update` | `EXCLUDED_BY_REQUEST`；无 run_id，按要求未执行、未访问 ERP |
| `shop.jd_self_operated.history` | `shop.jd_self_operated.collect_brand`；`--date` 经 `{business_date}` 自动渲染起止日期 | `PASS`；`shop.jd_self_operated.history__20260728__20260801T103803Z__22f456`；约 43 秒，品牌 JSON 533 B，本地库 +1 |
| `product.backfill` | `product.backfill.run`；`--date` 经 `{business_date}` 自动渲染 `--from/--to` | `PARTIAL_TIMEOUT`；`product.backfill__20260730__20260801T112223Z__af7f00`；抖音第二轮未收敛 |
| `product.review_collection` | `product.review.orchestrate`；`--date` 经 `{business_date}` 自动渲染目标日期 | `PASS`（空结果）；`product.review_collection__20260731__20260801T111838Z__e99652`；约 92 秒，JSON/Markdown 成功，当日新增 0 条 |
| `content.mapping.refresh` | `content.mapping.rebuild`；飞书只读并生成本地内容/款式映射 | `PASS`；`content.mapping.refresh__20260818__20260801T112822Z__917ed7`；约 25 秒，35 张表、117,505 B 本地 JSON |
| `content.retry_failed` | `content.failed.retry`；根据失败清单重试采集 | `PARTIAL_TIMEOUT`；`content.retry_failed__20260820__20260801T113412Z__e0c8af`；20 分 56 秒，B 站 4/4、蒲公英 14/22、星图 0 |
| `content.metrics.backfill` | `gyxx backfill content.metrics.daily`；合并到标准日报图并按日期范围重跑 | `PARTIAL_TIMEOUT`；`content.metrics.backfill__20260821__20260801T115644Z__1cb626`；15 分 30 秒，采集有界停止、同步成功，133/133 验收证据完整 |
历史场景分类为 `PASS` 3、`PARTIAL_TIMEOUT` 3、`EXCLUDED_BY_REQUEST` 1，共 7 项；这些状态不改变当前 21 个定时工作流的统计。
### 已废弃历史证据（不计任何当前分母）
- `content.self_operated.weekly`：原禁用兼容工作流与 `content.metrics.daily` 重复，已删除工作流和调度定义；底层脚本仍由日报调用。历史状态 `PASS_WITH_SKIPS`，run_id `content.self_operated.weekly__20260819__20260801T113119Z__b71197`。
- `product.weekly_aggregate.documented_missing`：`NOT_EXECUTABLE`；无 run_id；已确认没有脚本、注册项或调度入口，仅保留原审计结论。
### 合并前主图子链实跑证据（不计任何当前分母）
| 合并前工作流/版本 | 历史状态、run_id 与关键证据 |
|---|---|
| `product.main_image.jd.weekly` | `PASS`；`product.main_image.jd.weekly__20260812__20260801T102149Z__b2d3b7`；京东旧独立工作流生成 120 个文件、约 15.5 MB，本地 PG 288，82 个飞书写动作由验收门禁跳过 |
| `product.main_image.weekly`（合并前单平台版本） | `SKIPPED_COOKIE`；`product.main_image.weekly__20260813__20260801T103413Z__b48e34`；天猫旧独立工作流无法建立有效非交互登录态，exit 75，未扫码、未覆盖来源状态 |
以上 dated `run_id` 只证明合并前子链的历史表现，不证明当前 JD、Tmall 并行两节点图已经实跑，且不增加当前目录数量。
### 合并后的旧→新映射
- 当前 21 条有效调度定义对应 21 个顶层 LangGraph 工作流并全部启用；原两个主图调度已合并为一个周日 08:30 的 `product.main_image.weekly`，旧 run 仅作非加总历史证据。
- `shop.jd_self_operated.history`、`product.backfill`、`product.review_collection` 由命令声明的 `default_args` 和 `{business_date}` 自动渲染日期，`supply.purchase_order_update` 自动注入固定子命令；调用者无需重复传日期范围或 `mcp-run purchase-order-update`。
- `content.metrics.backfill` 收敛为 `gyxx backfill content.metrics.daily`；`content.mapping.refresh`、`content.retry_failed` 分别映射到 `content.mapping.rebuild`、`content.failed.retry`。
- `product.weekly_aggregate.documented_missing` 不再注册、不再调度，只保留上述历史证据。
## 7. 证据与自动化校验
- 现行 21 个定时工作流均已通过隔离 `GYXX_DATA_ROOT` 下的 LangGraph 注册/编排验证；合并主图的 JD、Tmall 并行启动、独立资源、失败隔离、成功写入保留和最终错误聚合已通过工程测试，但尚未完成受控业务实跑。7 个手动命令保留原实跑或排除证据；真实验收状态以第 6 节为准，工程验证或 dry-run 不替代业务实跑。
- 数据库阻塞已解除：8 个原数据库阻塞工作流均完成表/列/权限检查和事务回滚探针；商品导入、商品告警、营销日报随后执行了真实安全分支。
- 本次合并后的最终全量回归数量由交付汇总统一更新；Ruff、构建、`gyxx doctor --json` 和来源状态检查仍是必验项，不在此预写新的工程测试数量。
- 全仓 secret scan 命中 0；飞书误建的 1 条记录已精确删除并确认不存在，其余业务表写入均跳过；来源项目实际写入 0。
主要证据位置：
- 数据库迁移：`var/migrations/cloud_data_hub_to_gyxx_super_data_20260801T162327/`
- 浏览器状态复制：`var/migrations/browser-state/20260801T085006Z-5c217b14/`
- 验收事件：`var/reports/workflow-acceptance/evidence.jsonl`
- 店铺实跑：`var/reports/workflow-acceptance/shop-live-20260801.jsonl`
- 店铺指标三平台复测：`var/reports/workflow-acceptance/shop-metrics-full-20260909/`
- 店铺竞店两平台复测：`var/reports/workflow-acceptance/shop-competitor-full-20260913/`
- 京东自营日采集复测：`var/reports/workflow-acceptance/shop-jd-self-operated-full-20260729/`
- 商品京东主图（合并前子链证据）：`var/reports/workflow-acceptance/product-main-image-jd-20260812.jsonl`
- 商品日报下游复核：`var/reports/workflow-acceptance/product-daily-downstream-20260731-retry.jsonl`
- 内容长任务：`var/reports/workflow-acceptance/runs/`
- 工作流 journal：`var/runs/<workflow_id>/<business-date>/<run_id>/run.json`
- 逐项执行清单：`docs/workflow-acceptance-test-requirements.md`
## 8. 未通过项的下一步
1. 为供应链测试账号授予 ERP“普通商品资料”权限后，只复测补货周任务和补货告警；采购单更新仍保持排除，除非另行授权并提供可回滚测试数据。
2. 服务器运行店铺指标工作流前注入有效 `GYXX_HERMES_API_KEY`，复核可选 AI 分析输出；该项不阻塞三个店铺工作流的采集、文件和云端 PostgreSQL 验收结论。
3. 补齐商品天猫 DMP、京东类目配置，更新抖音页面控件定位；处理京东画像浏览器崩溃与抖音采集未收敛。
4. 为天猫万相提供有效且可复用的非交互登录态后，对合并后的主图两节点工作流执行受控业务复测：验证周日 08:30 并行启动、两个分支资源隔离、两边 PG/飞书边界，以及任一受控失败时另一边完成写入、整体失败并给出平台级具体错误；仍不得弹出扫码流程，飞书保持零写门禁。
5. 对内容完整度、评论、失败重试和指标回补设置可分段续跑的检查点，复测时继续保持飞书表零写入和唯一通知收件人策略。
最终结论是：工作流框架、数据库迁移、本地服务绑定和安全隔离已达到验收要求；当前 21 个定时工作流中，13 项业务通过且规则完成，7 项未通过，1 项合并后待受控业务复测。另有 7 个手动命令历史场景、2 个已删除/废弃历史证据和 2 条合并前主图子链实跑证据单列追溯，不能表述为“全部生产链路通过”。
docs/workflow-acceptance-test-requirements.md
+130
View File
@@ -0,0 +1,130 @@
# GYXX Flow 全工作流验收测试需求
测试日期：2026-08-03
测试时区：Asia/Shanghai
测试顺序：供应链 → 店铺 → 商品 → 内容
## 1. 验收目标与范围
当前架构的验收分母是 21 个定时工作流并全部启用，按模块分为内容 8、商品 7、店铺 3、供应链 3。原先独立调度的 `product.main_image.jd.weekly` 已与原单平台 `product.main_image.weekly` 合并为一个两节点工作流；合并前两个 dated `run_id` 只作为子链历史证据，不得冒充新图实跑。原目录中的另外 7 个可执行项现归类为手动命令/历史验收场景，不进入当前分母；`content.self_operated.weekly` 和 `product.weekly_aggregate.documented_missing` 继续仅保留历史证据。
验收覆盖工作流注册、LangGraph 编排、业务日期、浏览器状态、数据采集、云端 PostgreSQL、文件落盘、Hermes、飞书隔离和副作用账本。真实外部链路只在安全门禁允许时执行；失败、超时、权限阻塞和安全跳过必须如实保留，不能用 dry-run 冒充业务通过。
## 2. 强制隔离与数据标准
1. PostgreSQL 只允许使用运行时注入的云端 DSN；回环数据库立即失败，源码、报告和示例文件不得保存真实凭据。
2. 开始供应链工作流前，先验证云端目标库的连接、schema、读写权限和事务回滚能力；测试数据必须使用唯一 run_id，避免覆盖既有业务记录。
3. 飞书 Base、Bitable、Sheets 的新增、更新、删除全部物理跳过并留证；飞书读取可以执行。
4. 通知收件人只允许王云龙；不得向原业务群、原收件人或其他用户发送。通用 `production_sink` 只能证明进入副作用边界，不能据此推断消息正文或投递回执。
5. 每个浏览器入口保留独立 CDP、Profile、Cookie 和 storage state。登录策略固定为“本项目状态 → 已复制的来源状态 → 受支持的账号密码回退 → 安全跳过”；不弹出二维码、不人工通过验证码。
6. 本机 Hermes 分析端和采集端只通过回环地址访问，不得回退到远程 AI。
7. JSON、Markdown、CSV、Excel、图片、日志和 journal 必须按模块、工作流、业务日期落入 `GYXX_DATA_ROOT`；来源目录只读且不得成为运行时依赖。
8. 凭据只允许运行时注入。历史遗留 `db.env`、`analyze.env` 必须删除，仓库和验收报告不得记录账号、密码、Cookie、token 或密钥。
## 3. 状态与打勾规则
| 状态 | 含义 | 打勾 |
|---|---|---|
| `PASS` | 原逻辑、采集、处理和本轮允许的本地写入成功；允许出现业务空结果 | 是 |
| `PASS_WITH_SKIPS` | 已验证的主链路成功，本轮禁止或无需执行的外部副作用被明确跳过 | 是 |
| `SKIPPED_COOKIE` | 状态复用和受支持的非交互登录均不能建立有效登录态，且按规则安全退出 | 是，但不代表生产采集通过 |
| `NOT_EXECUTABLE` | 确认没有脚本、注册项或调度入口；仅用于已废弃历史证据 | 是，但不计当前工作流分母 |
| `PENDING_RETEST_AFTER_MERGE` | 合并后的工作流结构、调度和失败传播已完成工程验证，但新拓扑尚无受控业务实跑证据 | 否 |
| `FAIL_AUTH_OR_UI` | 登录、授权、配置或目标页面控件不满足，业务链路未完成 | 否 |
| `PARTIAL_FAILED` | 部分子链路已有有效结果，但至少一个必要子链路明确失败 | 否 |
| `PARTIAL_TIMEOUT` | 部分子链路已有有效结果，但整体在有界等待内未收敛 | 否 |
| `BLOCKED_EXTERNAL_PERMISSION` | 登录成功后，外部系统明确拒绝目标菜单或资源权限 | 否 |
| `EXCLUDED_BY_REQUEST` | 用户明确排除，本轮没有执行；仅用于手动命令历史场景 | 否，不计当前工作流分母 |
打勾表示“本轮验收计划已得到规则允许的确定结论”，不等同于生产链路全部无条件成功。当前 21 个定时工作流中，业务通过仅包括 `PASS` 和 `PASS_WITH_SKIPS`；`PENDING_RETEST_AFTER_MERGE` 不计业务通过或规则完成。`SKIPPED_COOKIE` 和 `NOT_EXECUTABLE` 在本文仍可用于历史证据，但当前结果表没有这两类状态。
## 4. 执行计划与验收方法
1. 冻结来源与验收范围：核对工作流清单、调度、入口、业务日期和源文件基线，来源目录保持只读。
2. 建立数据基线：确认云端目标库、schema、表数量和读权限，并使用可回滚事务验证写权限。
3. 建立安全门禁：统一云端 PG、本地 Hermes、唯一浏览器绑定、Cookie 复用、飞书写隔离、收件人白名单和 effect ledger。
4. 按“供应链 → 店铺 → 商品 → 内容”逐项执行；为每项保存 `run_id`、节点结果、退出码、产物、数据库变化和副作用证据。
5. 对超时任务执行有界观察；已经成功的独立下游只作为局部证据，不把部分成功升级为整体通过。
6. 汇总分类时分别核对“21 个定时工作流”“7 个手动命令场景”“2 个已删除/废弃历史证据”和“2 条合并前主图子链实跑证据”；最后一类只作非加总追溯，不进入任何当前状态分母。
## 5. 供应链工作流
- [x] `supply.purchase_confirmation.daily`：采购确认采集、分析和通知。`PASS`；run_id `supply.purchase_confirmation.daily__20260729__20260801T092309Z__5daace`，ERP 采集、原始 Excel、汇总 CSV 和本地 PG `workflow_runs=completed` 均成功。journal 仅记录通用 `production_sink`，不扩展解释其消息内容或回执。
- [ ] `supply.replenishment.weekly`：按销量、库存和在途生成补货建议。`BLOCKED_EXTERNAL_PERMISSION`；run_id `supply.replenishment.weekly__20260728__20260801T094413Z__743f4a`，账号登录成功后 ERP 明确拒绝“普通商品资料”权限，exit 1，阻断前外部写入为空。
- [ ] `supply.replenishment_alert.daily`：指定 SKU 库存阈值预警。`BLOCKED_EXTERNAL_PERMISSION`；主要 run_id `supply.replenishment_alert.daily__20260727__20260801T094546Z__500e91`，复现 run_id `supply.replenishment_alert.daily__20260728__20260801T094100Z__4b2da2`；均在登录后被同一菜单权限阻断，外部写入为空。
## 6. 店铺工作流
- [x] `shop.metrics.weekly`：并行采集京东、抖音、天猫店铺经营指标。`PASS`；run_id `shop.metrics.weekly__20260804__20260804T065204Z__bb3b69`，京东、抖音、天猫 3/3 节点成功，产物落盘并写入云端 PostgreSQL；京东和抖音按原逻辑真实更新飞书 Base，天猫按原逻辑不写店铺表。抖音销售额采用确认后的净销售额 `294,892.98`，毛销售额保留在原始产物和 PostgreSQL。
- [x] `shop.competitor.weekly`：并行采集京东、抖音竞店指标；当前没有天猫竞店采集任务。`PASS`；run_id `shop.competitor.weekly__20260804__20260804T075530Z__4fd7c1`，京东和抖音 2/2 节点成功，分别 upsert 49/24 行；飞书 Base 分别更新 7/9 条品牌记录，无法排名的类目只在页面明确显示“暂无数据”时记为未上榜。
- [x] `shop.jd_self_operated.daily`：按前一业务日依次采集京东自营品牌和商品。`PASS`；run_id `shop.jd_self_operated.daily__20260801__20260804T082226Z__255cfc`，品牌和商品 2/2 节点成功，分别采集 1/50 行，收入合计均为 `19,974.27`。Cookie 复用时从京东 `pin` 识别账号 `gyxx2022`，云端 PostgreSQL 按 `account + data_date + brand/product_code` UPSERT；同日真实重跑 `shop.jd_self_operated.daily__20260801__20260804T082722Z__0ced7e` 再次 2/2 成功且产物仍为 1/50 行，CDP 22132/22133 均已关闭。
## 7. 商品工作流
- [ ] `product.persona.daily`：并行采集天猫、抖音、京东商品人群画像。`PARTIAL_FAILED`；run_id `product.persona.daily__20260815__20260801T103908Z__2d6028`。天猫缺少 DMP 入口，抖音缺少搜索控件，京东完成 19/30 后浏览器崩溃，部分结果不能代表三平台完成。
- [ ] `product.daily`：ERP 与三平台采集、分析、导出和入库。`PARTIAL_TIMEOUT`；run_id `product.daily__20260731__20260801T113513Z__b59304`。ERP、京东、天猫完成，抖音超时；保留 257 个 raw 文件。独立下游验证完成分析、32 条导出、本地 PG upsert 和 32 次飞书写跳过，但不把下游成功升级为整体通过。
- [x] `product.alert.daily`：读取商品数据并检测销量异常、按条件通知。`PASS`；实际入口 run_id `script.product_commerce.955b1dfb0cdf__20260801__20260801T085614Z__4fefb3`。本地库读取成功，当次无符合阈值的异常，因此没有告警新增或通知。
- [x] `product.import.daily`：把三平台商品日报幂等导入本地 PG。`PASS`；实际入口 run_id `script.product_commerce.9137239938d5__20260727__20260801T085541Z__59c16c`。抖音、京东、天猫分别处理 163、90、173 行，无重复。
- [x] `product.style_analysis.interval`：使用本地 Hermes 分析到期款式。`PASS_WITH_SKIPS`；workflow run_id `product.style_analysis.interval__20260801__20260801T084137Z__952204`。数据库表/权限与 Hermes 健康检查通过，单款实现入口内部 dry-run 成功，未写外部副作用目标。
- [ ] `product.main_image.weekly`：周日 08:30 并行启动 JD、Tmall 两个无依赖且资源隔离的主图分支，两个子链均独立执行本地 PG 入库与飞书插入逻辑。`PENDING_RETEST_AFTER_MERGE`；当前没有可归属于合并后新图的业务 run_id。工程结构要求为：任一分支受控失败时，另一分支仍完成自己的采集和双 sink 写入；全部结束后整体聚合为失败，并在错误摘要中标明具体平台和错误。下一轮需在现有飞书零写门禁下完成受控业务复测。
- [ ] `product.market_rank`：并行采集天猫、京东、抖音三平台周市场排行。`FAIL_AUTH_OR_UI`；run_id `product.market_rank__20260814__20260801T103628Z__5a9024`。天猫认证失败、京东缺配置类目、抖音目标控件缺失；目标脚本与来源哈希一致，当前证据不支持判定为迁移改坏。
## 8. 内容工作流
- [ ] `content.metrics.daily`：采集合作达人、自营 B 站、蝉妈妈数据并同步指标。`PARTIAL_FAILED`；run_id `content.metrics.daily__20260813__20260801T101312Z__28d505`，运行 71 分 05 秒后完整度检查失败；已完成子链和飞书写跳过证据保留，但必要采集链未全部完成。
- [x] `content.marketing_report.daily`：从本地数据生成营销日报并按条件通知。`PASS_WITH_SKIPS`；实际入口 run_id `script.content_marketing.62b3b647254f__20260804__20260801T091245Z__f20319`。真实读取本地 PG、调用本地 Hermes，生成 Markdown/PNG；未启用发送参数。
- [x] `content.relogin.weekly`：并行刷新内容平台登录态。`PASS_WITH_SKIPS`；run_id `content.relogin.weekly__20260811__20260801T100706Z__45a8db`。LangGraph 和验收门禁按设计阻止二维码/交互登录并安全记录跳过；没有扫码、没有覆盖 Cookie 或来源状态。
- [x] `content.creator_report.monthly`：生成达人月报。`PASS_WITH_SKIPS`；workflow run_id `content.creator_report.monthly__20260801__20260801T084140Z__bd9fdd`。本地数据库检查和实现入口 dry-run 成功，飞书表写入关闭。
- [x] `content.summary.monthly`：生成内容月度汇总。`PASS_WITH_SKIPS`；workflow run_id `content.summary.monthly__20260801__20260801T084142Z__9d73c1`。本地数据库检查和实现入口 dry-run 成功，外部写入关闭。
- [x] `content.cooperations.daily`：刷新映射并同步合作记录。`PASS_WITH_SKIPS`；workflow run_id `content.cooperations.daily__20260801__20260801T084143Z__02b89a`。真实完成飞书读取和本地数据库读取，内部 dry-run 阻止数据库/飞书写入。
- [ ] `content.comments.weekly`：并行补采 B 站、小红书、抖音评论并汇总。`PARTIAL_TIMEOUT`；run_id `content.comments.weekly__20260812__20260801T100904Z__530989`，运行 60 分 56 秒。B 站完成，小红书/抖音有界停止；保留 229 个 CSV、229 个 JSON，合计约 5.45 MB。
- [x] `content.summary.weekly`：生成内容周度汇总。`PASS_WITH_SKIPS`；workflow run_id `content.summary.weekly__20260801__20260801T084145Z__7e6081`。本地数据库检查和实现入口 dry-run 成功，外部写入关闭。
## 9. 手动命令与历史验收场景
下列 7 项保留手动执行能力和既有验收证据，但不再注册为定时工作流，也不进入 21 个工作流分母。
| 旧验收标识 | 合并后的入口与用途 | 历史状态、run_id 与关键证据 |
|---|---|---|
| `supply.purchase_order_update` | `supply.workflow.run`；受保护的 ERP 更新命令，默认参数自动选择 `mcp-run purchase-order-update` | `EXCLUDED_BY_REQUEST`；无 run_id，按要求未测试、未访问 ERP |
| `shop.jd_self_operated.history` | `shop.jd_self_operated.collect_brand`；`--date` 经 `{business_date}` 自动渲染起止日期 | `PASS`；`shop.jd_self_operated.history__20260728__20260801T103803Z__22f456`；约 43 秒，生成 533 B 品牌 JSON，本地库新增 1 条 |
| `product.backfill` | `product.backfill.run`；`--date` 经 `{business_date}` 自动渲染 `--from/--to` | `PARTIAL_TIMEOUT`；`product.backfill__20260730__20260801T112223Z__af7f00`；抖音第二轮等待未收敛，保留已完成平台证据 |
| `product.review_collection` | `product.review.orchestrate`；`--date` 经 `{business_date}` 自动渲染目标日期 | `PASS`（空结果）；`product.review_collection__20260731__20260801T111838Z__e99652`；约 92 秒，JSON/Markdown 产物成功，当日数据库新增 0 条 |
| `content.mapping.refresh` | `content.mapping.rebuild`；读取内容/款式映射并生成本地 JSON | `PASS`；`content.mapping.refresh__20260818__20260801T112822Z__917ed7`；约 25 秒，35 张表、117,505 B 本地映射文件，飞书只读 |
| `content.retry_failed` | `content.failed.retry`；按失败清单重试采集并汇总 | `PARTIAL_TIMEOUT`；`content.retry_failed__20260820__20260801T113412Z__e0c8af`；20 分 56 秒，B 站 4/4、蒲公英 14/22、星图 0 |
| `content.metrics.backfill` | `gyxx backfill content.metrics.daily`；合并到标准日报图并按日期范围重跑 | `PARTIAL_TIMEOUT`；`content.metrics.backfill__20260821__20260801T115644Z__1cb626`；15 分 30 秒，采集有界停止、同步成功，133/133 条验收证据完整 |
手动场景历史分类为：`PASS` 3、`PARTIAL_TIMEOUT` 3、`EXCLUDED_BY_REQUEST` 1，合计 7。
## 10. 当前分母外的历史证据
### 已删除/废弃项
- `content.self_operated.weekly`：原禁用兼容工作流与 `content.metrics.daily` 的自营链重复，现已删除工作流和调度定义；底层自营脚本仍由日报工作流调用。历史结论为 `PASS_WITH_SKIPS`，run_id `content.self_operated.weekly__20260819__20260801T113119Z__b71197`，仅作追溯。
- `product.weekly_aggregate.documented_missing`：历史文档中的商品周聚合说明项。`NOT_EXECUTABLE`；无 run_id；已确认来源和目标均不存在脚本、注册项或调度入口。该项不进入当前工作流或手动命令分母。
### 合并前主图子链实跑证据
| 合并前工作流/版本 | 历史状态、run_id 与关键证据 |
|---|---|
| `product.main_image.jd.weekly` | `PASS`；`product.main_image.jd.weekly__20260812__20260801T102149Z__b2d3b7`；京东旧独立工作流生成 120 个文件、约 15.5 MB，本地 PG 记录 288，82 个飞书写动作由验收门禁跳过 |
| `product.main_image.weekly`（合并前单平台版本） | `SKIPPED_COOKIE`；`product.main_image.weekly__20260813__20260801T103413Z__b48e34`；天猫旧独立工作流无法建立有效非交互会话，exit 75 安全退出，未扫码、未覆盖来源状态 |
这两个 dated `run_id` 只能说明合并前京东、天猫子链各自的历史表现，不是合并后 JD、Tmall 并行两节点图的实跑结果，也不参与当前 21 项状态统计。
## 11. 合并口径与旧→新映射
- 当前 21 条有效调度定义对应 21 个顶层 LangGraph 工作流并全部启用，模块分布为内容 8、商品 7、店铺 3、供应链 3；重复且禁用的 `content.self_operated.weekly` 已删除，其主要采集链继续由 `content.metrics.daily` 执行。
- 原 `product.main_image.jd.weekly` 与原单平台 `product.main_image.weekly` 已收敛为当前 `product.main_image.weekly`：周日 08:30 并行启动 JD、Tmall 两个独立分支，两边均进入各自 PG 与飞书写入边界；受控失败不阻断另一边，末尾聚合失败状态和平台级具体错误。当前仅工程结构验证完成，业务状态保持 `PENDING_RETEST_AFTER_MERGE`。
- `shop.jd_self_operated.history`、`product.backfill`、`product.review_collection` 使用命令层 `default_args` 与 `{business_date}` 渲染日期；`supply.purchase_order_update` 使用固定默认子命令，调用者无需重复传日期范围或固定子命令。`content.metrics.backfill` 合并为标准日报工作流的 `gyxx backfill` 入口，其余内容项为显式维护/补偿命令。
- `content.self_operated.weekly` 和 `product.weekly_aggregate.documented_missing` 仅作为历史审计证据保存，不再出现在工作流注册、调度或当前数量统计中。
## 12. 最终验收标准与完成度
- 21/21 个定时工作流均有用途和状态；20 项保留既有 dated 业务证据，合并后的主图工作流尚无新图 run_id，明确列为待受控业务复测。当前调度全部启用。
- 当前工作流结果分类：`PASS` 4、`PASS_WITH_SKIPS` 9、`FAIL_AUTH_OR_UI` 1、`PARTIAL_FAILED` 2、`PARTIAL_TIMEOUT` 2、`BLOCKED_EXTERNAL_PERMISSION` 2、`PENDING_RETEST_AFTER_MERGE` 1，合计 21。
- 当前工作流业务通过和规则完成均为 13/21；未通过或待复测为 8/21。
- 7 个手动命令历史场景、2 个已删除/废弃历史证据和 2 条合并前主图子链实跑证据均单列保存，不进入当前工作流分母；主图历史证据是非加总 run 记录，不能机械增加目录项数量。
- 店铺复测中曾因一次命令漏设验收总开关误建 1 条京东周数据记录；已按精确 record id 删除并回读确认不存在，除此之外飞书业务表写入均由门禁跳过。通知策略唯一允许王云龙，但供应链通用 `production_sink` 没有独立消息正文或投递回执，不作过度结论。
- 云端快照迁移、旧库/dump 回滚、本地读写、Hermes、浏览器状态复制、来源只读、凭据清理和 secret scan 证据必须在最终报告中单列。
- 当前工作流未通过或待复测项保留真实边界：2 项外部 ERP 权限、1 项认证/UI、2 项部分失败、2 项部分超时、1 项合并后受控业务复测；手动命令另有 3 项部分超时、1 项用户排除。主图项是由拓扑合并产生的明确复测要求，不沿用旧的模糊“待浏览器复测”表述。
docs/workflow-console.md
+69
View File
@@ -0,0 +1,69 @@
# 工作流控制台
GYXX Flow 自带一个只依赖 Python 运行时的工作流控制台。页面从当前工作流目录、定时配置、
调度状态和运行索引动态取数，不维护第二套工作流清单。
## 启动
```powershell
cd D:\gyxx-flow
uv run gyxx console
```
生产凭据不会从源码或任意 `.env` 自动发现。需要真实运行时，必须显式指定受控环境文件；
已经由服务环境注入的同名变量优先，不会被文件覆盖：
```powershell
uv run gyxx console --env-file D:\secure\gyxx-flow.env
```
商品经营正式执行会在启动子进程前校验云端 PostgreSQL、`hermes-analyzer` 飞书身份、
所需 Hermes 密钥和负责人 `open_id`。缺少任一必需项时，前端会返回明确的配置错误，
不会创建一个注定失败的“正式运行”。
默认地址为 `http://127.0.0.1:8765`。控制台进程与调度进程职责分离；服务器仍只运行一个
项目内调度服务：
```powershell
uv run gyxx schedule run
```
页面可以：
- 按内容营销、商品经营、店铺洞察和供应链分别展示全部工作流；
- 查看工作流定义、执行步骤、定时规则和下一次启动时间；
- 修改定时类型、一个或多个时间、日期规则、启停状态和业务日期偏移；
- 以安全预演或正式执行方式手动触发已注册工作流；
- 查看最近运行状态、步骤统计和经过脱敏的错误详情。
定时配置保存到 `config/schedules.json`。常驻 Python 调度器会在下一次轮询时重新验证并
加载配置，不需要创建或修改 Windows Task Scheduler 任务。若新规则在错过触发补偿窗口
内已经到期，下一次轮询可能立即启动该工作流。
## 执行安全
手动运行默认选择“安全预演”，不会产生真实外部副作用。只有在页面中选择“正式执行”并
确认影响后，控制台才会通过固定的 `python -m gyxx_flow run ... --execute` 入口启动独立
子进程。浏览器不能提交脚本路径、命令参数、环境变量或凭据。
控制台沿用 `GYXX_DATA_ROOT`，因此运行日志、工作流锁、运行索引和副作用账本与 CLI、调度
服务保持同一边界。生产部署必须为所有进程设置同一个外部数据根。
页面中的“预演完成”和“正式成功”是两种不同状态。预演全部跳过外部写入时不会再显示为
最近正式成功；运行详情和历史记录会保留 `dry_run` 或 `execute` 模式。
## 远程访问
默认只监听回环地址。若需要绑定非回环地址，必须先通过安全环境注入不少于 24 个字符的
访问令牌：
```powershell
$env:GYXX_CONSOLE_TOKEN = '<由密钥系统注入的随机令牌>'
uv run gyxx console --host 0.0.0.0 --port 8765
```
令牌不会写入源码、配置、URL 或日志。页面会在当前浏览器会话中临时保存令牌。生产环境
应再通过 HTTPS 反向代理或 SSH 隧道访问，不应直接把明文 HTTP 控制端口暴露到公网。
控制台不提供工作流入口、参数、运行时绑定或凭据的在线编辑能力；这些仍由受版本控制的
项目配置和代码维护。
final_report.md
-72
View File
@@ -1,72 +0,0 @@
# GYXX Flow 源码迁移阶段报告
## 当前结论
核心采集脚本已经修改，但修改的是复制到 `D:\gyxx-flow` 的新项目副本；四个旧项目的
原始代码和现有定时任务均未修改。新工作流不再通过环境变量、动态导入或子进程引用旧
项目代码。
当前可用能力：
- 统一列出并 dry-run/执行 21 个定时工作流和 7 个正式手工工作流；
- 统一发现并 dry-run/执行 131 个迁入后的本地脚本；
- 生成 21 个 Windows Task Scheduler XML 和单任务安装器；
- 将新数据按模块写入 `data/{raw,normalized,curated,exports,evidence}/<module>`，运行状态、
日志和临时文件写入 `state/logs/tmp`；
- 更换项目根和数据根后继续完成入口发现与 dry-run。
## 已完成
- [x] 四模块源码实体迁移及逐文件来源/目标哈希清单
- [x] 21 个计划任务到新项目本地入口的一一映射
- [x] 131 个本地可执行脚本目录与统一手工入口
- [x] 旧项目根目录、旧环境变量和旧配置回退清零
- [x] 数据、日志、Profile、状态和临时目录迁入可配置数据根
- [x] 敏感默认值清理、来源清单校验、重定位测试和全量自动化测试
- [x] 只生成不应用的候选调度包；安装器强制逐任务应用
- [x] 部署、迁移、运维和回滚手册更新
- [x] 131 个脚本唯一 CDP 端口、独立可迁移 Profile/Cookie/storage state
- [x] 原飞书透传、现有云 PostgreSQL 和本机 Hermes 的统一运行时边界
## 未完成
- [ ] 为每个外部写入证明 run_id 追踪和重跑幂等
- [ ] 用户授权后的凭据轮换、逐任务生产切换和回滚演练
- [ ] 日任务 7 天、周任务 2 个周期、月任务历史回放及最终对账
`gyxx acceptance status` 在上述生产门禁完成前应继续返回 incomplete。这是有意保留的
安全状态，不影响当前已具备的本地手工运行和候选任务注册能力。
## 统一适配器补充验收（2026-07-27）
- `config/runtime-bindings.json` 覆盖 131/131 脚本，CDP 端口 131 个且无重复。
- 顶层 workflow/manual 与嵌套 Python/PowerShell 均按目标脚本 ID 重新绑定。
- Cookie、storage state 原子保存并跨运行复用，全部位于可迁移数据根。
- 生产源码中固定 `9222/18801/18802/18803`、源码目录 Profile 和临时即删 Profile 已清除。
- 飞书身份/调用后端保持原样；三套数据库变量映射到现有云端，回环数据库被拒绝；
Hermes 非回环 URL 被拒绝。
- Python 3.12 全量测试 `294 passed`；框架 Ruff、compileall、敏感信息和迁移哈希通过。
## 采集数据目录收口补充验收（2026-07-27）
- [x] 四模块统一使用 `data/{raw,normalized,curated,exports,evidence}/<module>`。
- [x] JSON、CSV、XLS/XLSX、浏览器下载和原始截图进入 raw；清洗结果进入 normalized；
聚合结果进入 curated；最终 Markdown/Excel 报告进入 exports。
- [x] checkpoint、缓存、锁、浏览器状态、日志和临时处理文件不再混入 raw 或源码目录。
- [x] CLI/vendor 自定义输出受数据根边界校验，不能写到 cwd、源码树或数据根之外。
- [x] 历史 `legacy` 前后均为 48,961 个文件、1,091,967,729 字节，无移动、删除或改写。
- [x] raw 按 `run_id` 追加保留，不清理历史原始产物；浏览器 Profile/Cookie 在任务结束后保留复用。
- [x] 空/纯空白数据根安全回退项目 `var`，不会把 cwd 当成数据根。
- [x] 全量 `294 passed`，Ruff、compileall、PowerShell AST 和来源清单哈希通过。
本次只修改 `D:\gyxx-flow` 内迁入副本和工程文件，没有修改四个旧项目，也没有注册、
禁用或改动任何现有定时任务；未启动真实浏览器采集，未写飞书、云端数据库或 Hermes。
## 四源项目增量重新对齐（2026-07-28）
- [x] 内容营销 4 个源端修复文件及两份动态映射资源已重新对齐。
- [x] 商品商业 ERP 天猫权威来源修复已重新对齐。
- [x] 源端新增波塞冬历史销量回填脚本及测试已实体迁入并改接统一数据分层。
- [x] 新脚本可通过统一脚本目录手工启动，使用独立 CDP 端口和可复用 Cookie 状态。
- [x] 四份来源清单现覆盖 250 个文件；目标哈希、全量 294 项测试、18 项定向回归、compileall 和 doctor 通过。
- [ ] Ruff 未安装在当前 `.venv`，未作为本轮通过项；P6/P7 的真实外部写入和生产切换门禁继续保持未完成。
plan.md
-126
View File
@@ -1,126 +0,0 @@
# GYXX Flow 执行计划与验收清单
规则：只有完成对应验证并保存证据后才把 `[ ]` 改为 `[x]`。每次更新复核
`python -m pytest`、`gyxx acceptance status` 和本文件。
## P0 设计与迁移基线
- [x] P0.1 固化模块化单体、数据分层、影子运行和逐任务回滚设计。
- 文件：`design.md`
- 验证：设计包含目标、边界、模块、数据、安全、迁移和验收。
- [x] P0.2 建立项目骨架、依赖文件和忽略规则。
- 文件：`pyproject.toml`、`.gitignore`、`README.md`、`src/`、`tests/`
- 验证：干净环境可安装，CLI 能显示帮助。
- [x] P0.3 生成四个旧项目的代码、任务、数据基线清单。
- 文件：`var/baseline/<timestamp>/`
- 验证：包含 21 个实际任务、代码哈希、数据汇总；不包含密钥值。
- [x] P0.4 建立敏感信息扫描并记录轮换门禁。
- 文件：`src/gyxx_flow/security/`、`config/secrets.example.env`
- 验证：仓库扫描无明文凭据；生产切换要求凭据轮换确认。
## P1 核心运行与数据基础设施
- [x] P1.1 TDD 实现可移植路径配置，不允许业务代码硬编码旧盘符。
- [x] P1.2 TDD 实现 `RunContext`、稳定 `run_id` 和业务日期校验。
- [x] P1.3 TDD 实现原子 JSON 写入、SHA-256 和 artifact manifest。
- [x] P1.4 TDD 实现 raw/normalized/curated/export/evidence/state/log/tmp 目录解析。
- [x] P1.5 TDD 实现结构化运行及步骤状态记录。
- [x] P1.6 TDD 实现文件锁、工作流锁和命名资源锁。
## P2 工作流引擎与 CLI
- [x] P2.1 TDD 实现工作流、步骤、依赖、超时和重试模型。
- [x] P2.2 TDD 实现命令步骤及无副作用 dry-run。
- [x] P2.3 TDD 实现 shadow 模式，禁止生产 Sink 和正式通知。
- [x] P2.4 TDD 实现 `gyxx list`、`gyxx run`、`gyxx backfill`。
- [x] P2.5 TDD 实现失败恢复、步骤级重跑和最终退出码。
- [x] P2.6 建立 `ops` 运行记录和 outbox 接口。
## P3 工作流目录与调度
- [x] P3.1 建立四个模块及模块依赖约束测试。
- [x] P3.2 映射 21 个实际计划任务到稳定 workflow ID。
- [x] P3.3 登记未调度/手工工作流，保持原触发语义。
- [x] P3.4 建立声明式 schedule 配置及 schema 校验。
- [x] P3.5 生成 Windows Task Scheduler XML/PowerShell 安装计划，但默认不应用。
- [x] P3.6 实现 `gyxx schedule plan`，检测系统任务漂移。
## P4 迁移和验收工具
- [x] P4.1 实现旧命令适配器，所有旧根目录均通过配置注入。
- [x] P4.2 实现历史数据只复制、不删除的迁移计划及校验清单。
- [x] P4.3 实现文件数量、字节数、SHA-256 对账。
- [x] P4.4 实现 shadow 结果主键、行数、指标和错误对比。
- [x] P4.5 实现 `gyxx doctor` 环境、解释器、CLI、路径和权限预检。
- [x] P4.6 实现 `gyxx acceptance status` 和机器可读验收报告。
- [x] P4.7 生成四个旧项目的源码迁移清单；每个文件记录来源哈希、目标路径和处置类型。
## P5 分模块源码迁移
- [x] P5.1 将 `shop_intelligence` 全部业务源码、资源和入口复制并改造到新模块；两个周任务可独立运行。
- [x] P5.2 将 `supply_chain` 全部业务源码、资源和入口复制并改造到新模块；三个定时和全部手工入口可独立运行。
- [x] P5.3 将 `content_marketing` 全部业务源码、资源和入口复制并改造到新模块；全部定时和手工入口可独立运行。
- [x] P5.4 将 `product_commerce` 全部业务源码、资源、内置 vendors 和入口复制并改造到新模块；全部定时和手工入口可独立运行。
- [x] P5.5 消除营销二维码写入商品项目、商品配置回退到 Auto Flow，以及任何跨业务模块内部导入。
- [x] P5.6 将重复的飞书、PostgreSQL、Hermes、浏览器能力收敛为适配器，业务行为保持可回放对账。
- 证据：`src/gyxx_flow/adapters/integration.py`、`browser.py`、`bootstrap.py`、`runtime_exec.py`；原飞书后端不变，数据库只允许云端，Hermes 只允许本机。
- [x] P5.7 删除最终工作流中的 `DeferredLegacyCommandStep`，运行时不再要求 `GYXX_LEGACY_*_ROOT`。
- [x] P5.8 全量入口分类完成：scheduled、manual、library、intentionally-excluded 均有证据，不遗漏脚本。
## P6 自动化工程验收
- [x] P6.1 21 个计划任务逐项有且只有一个新项目本地入口映射，没有意外新增。
- [x] P6.2 源码、配置、启动命令和动态导入中均不存在四个旧项目的运行时依赖。
- [x] P6.3 临时隐藏四个旧项目路径并更换项目根、数据根后，入口导入和 dry-run 全部通过。
- [ ] P6.4 每个迁移后的业务运行可从 run_id 追溯输入、输出、日志和外部写入。
- [ ] P6.5 迁移后的业务代码重跑不会重复写数据库、飞书或正式通知。
- [x] P6.6 旧数据复制前后数量、大小和 SHA-256 一致。
- [x] P6.7 迁移后的浏览器 Profile 和共享资源全部进入新数据根，且并发锁验证通过。
- [x] P6.8 源码仓库敏感信息扫描通过。
- [x] P6.9 全量测试、静态检查、源码覆盖清单和代码评审无阻断项。
## P7 生产切换门禁
- [ ] P7.1 用户确认迁移基线、密钥轮换和第一个切换任务。
- [ ] P7.2 单次只切一个任务；旧任务只禁用、不删除。
- [ ] P7.3 回滚演练证明能关闭新任务并重新启用旧任务。
- [ ] P7.4 日任务真实连续成功 7 天。
- [ ] P7.5 周任务真实连续成功 2 个周期。
- [ ] P7.6 月任务指定历史月份回放通过。
- [ ] P7.7 生产通知无重复，数据库/飞书对账通过。
- [ ] P7.8 旧项目只读保留期满并取得退役确认。
## P8 完成
- [ ] P8.1 所有上述检查项均有证据且已勾选。
- [x] P8.2 按源码完整迁移的新口径重新生成 `review.md` 和 `final_report.md`。
- [x] P8.3 按独立运行的新口径更新部署、迁移、回滚和日常运维手册。
## P9 每脚本运行时隔离与外部系统统一边界
- [x] P9.1 为 131 个可执行脚本建立显式、稳定、唯一的 `22000..22999` CDP 端口分配。
- [x] P9.2 每脚本独立 Profile、Cookie、storage state，全部位于 `GYXX_DATA_ROOT/state/browser`。
- [x] P9.3 顶层 workflow、手工脚本和嵌套 Python/PowerShell 子脚本均按真实 script ID 重新绑定。
- [x] P9.4 Cookie/storage state 支持首次为空、原子保存和后续运行复用，内容不进入日志或源码。
- [x] P9.5 飞书保持原 lark-cli profile、身份和 OpenAPI 应用，不切换后端或凭据来源。
- [x] P9.6 `PG_*`、`DB_*`、`AUTOFLOW_PG_*` 统一映射现有云端 PostgreSQL，拒绝本地回环数据库。
- [x] P9.7 Hermes 保持本机 HTTP/CLI，非回环 Hermes URL 在业务调用前失败。
- [x] P9.8 移除生产代码固定 9222/18801/18802/18803 和源码目录/临时 Profile，刷新迁移哈希。
- [x] P9.9 Python 3.12 全量 `294 passed`，compileall、敏感信息、端口和旧根依赖扫描通过；历史 Ruff 证据保留。
## P10 采集数据统一分层与目录收口
- [x] P10.1 四个模块的新产物统一使用 `<GYXX_DATA_ROOT>/data/{raw,normalized,curated,exports,evidence}/<module>`，运行状态、日志和临时文件分别使用 `state/`、`logs/`、`tmp/`。
- [x] P10.2 JSON、Markdown、CSV、XLS/XLSX、浏览器下载和截图均由模块路径边界定位，不写入源码目录、模块目录或当前工作目录。
- [x] P10.3 `var/data/raw/<module>/legacy` 历史数据不移动、不删除、不改写，迁移前后文件数和总字节数一致。
- [x] P10.4 四模块保留原业务脚本公开路径常量，统一改接共享 `ModuleDataPaths`，新增模块只需声明模块名即可扩展。
- [x] P10.5 更换 `GYXX_DATA_ROOT` 后所有产物路径整体迁移，路径解析不创建目录、不访问四个旧项目。
- [x] P10.6 路径红测、静态旁路扫描、全量测试、Ruff、compileall、敏感信息和旧根依赖扫描全部通过，清单与验收报告已刷新。
## P11 四源项目增量 Bug 修复重新对齐（2026-07-28）
- [x] P11.1 以四份来源清单 SHA-256 为基线，识别源端变更、新增、删除和目标端二次改造。
- [x] P11.2 合并内容营销模块发布链接非空触发、星图普通失败不熔断和精确续跑修复。
- [x] P11.3 合并商品模块“ERP 编码只读取天猫分组”的权威来源修复，并迁入波塞冬历史销量回填工具。
- [x] P11.4 刷新来源/目标哈希、两份动态业务映射、迁移清单和新增回归测试，不纳入运行数据、临时文件或敏感状态。
- [x] P11.5 定向测试、全量测试、compileall、敏感信息、旧根依赖和 doctor 通过；acceptance 正确保留 P6/P7 生产门禁为 incomplete，Ruff 因当前虚拟环境未安装而未执行。
pyproject.toml
+19 -4
View File
@@ -40,6 +40,13 @@ dependencies = [
[project.optional-dependencies]
test = ["pytest>=8.0"]
[dependency-groups]
dev = [
"build>=1.2",
"pytest>=8.0",
"ruff>=0.12",
]
[project.scripts]
gyxx = "gyxx_flow.cli:main"
@@ -47,10 +54,11 @@ gyxx = "gyxx_flow.cli:main"
where = ["src"]
[tool.setuptools.package-data]
"gyxx_flow.modules.content_marketing" = ["runtime/**/*"]
"gyxx_flow.modules.product_commerce" = ["runtime/**/*"]
"gyxx_flow.modules.shop_intelligence" = ["runtime/**/*"]
"gyxx_flow.modules.supply_chain" = ["runtime/**/*"]
"gyxx_flow.web" = ["*.html", "*.css", "*.js"]
"gyxx_flow.modules.content_marketing" = ["**/*"]
"gyxx_flow.modules.product_commerce" = ["**/*"]
"gyxx_flow.modules.shop_intelligence" = ["**/*"]
"gyxx_flow.modules.supply_chain" = ["**/*"]
[tool.setuptools.exclude-package-data]
"*" = ["**/__pycache__/**", "**/*.pyc", "**/*.pyo"]
@@ -59,5 +67,12 @@ where = ["src"]
pythonpath = ["src"]
testpaths = ["tests"]
[tool.ruff]
# The migrated business scripts keep their original lint baseline while the
# framework and focused critical scripts are checked in CI. New module code
# should be added to the focused CI list until this compatibility exclusion is
# retired module by module.
extend-exclude = ["src/gyxx_flow/modules/*/**"]
[tool.ruff.lint]
select = ["E4", "E7", "E9", "F", "I"]
review.md
-90
View File
@@ -1,90 +0,0 @@
# 工程复核报告
## 结论
四个旧项目的工作流、定时入口及纳入范围的业务源码已实体复制到新项目，并完成路径、
配置、数据目录和启动方式改造。运行时入口只解析
`src/gyxx_flow/modules/<module>/runtime`，不再导入或启动四个旧目录中的代码。
新项目已经具备统一手工运行和生成 21 个 Windows 定时任务定义的工程能力；本轮没有
注册新任务，也没有禁用、删除或修改旧任务。生产安全切换仍受 P6.4、P6.5 和
P7 门禁约束，不能把“可生成/可 dry-run”表述为“已经生产投运”。
## 源码接管证据
| 模块 | 来源清单文件 | 迁移文件 | 可执行脚本 |
|---|---:|---:|---:|
| content_marketing | 90 | 90 + 1 个新路径模块 | 56 |
| product_commerce | 101 | 101 + 1 个新路径模块 | 49 |
| shop_intelligence | 22 | 22 | 8 |
| supply_chain | 37 | 37 | 18 |
| 合计 | 250 | 256（含 2 个路径模块和 4 个运行时 bootstrap） | 131 |
- 四份 `config/source-manifests/*.json` 逐文件记录来源相对路径、来源 SHA-256、目标
相对路径、目标 SHA-256、分类和是否改造。
- 对四个旧目录做只读复核并同步两份动态业务映射后，250/250 个纳入文件来源哈希与
当前迁移清单一致；旧项目没有被本次迁移写入。
- 源码、配置、启动器及动态入口扫描中，四个旧根目录、`GYXX_LEGACY_*_ROOT`、
`DeferredLegacyCommandStep` 和 `AUTO_FLOW_CONFIG` 的运行时命中均为 0。
- 项目和数据根整体迁移到临时目录后，目录发现、131 个脚本发现及四模块代表工作流
dry-run 均通过。
## 入口与调度证据
- 29 个目录项：21 个 scheduled、7 个 manual、1 个有证据的 unavailable。
- `gyxx scripts list` 发现 131 个新项目本地 Python/BAT/PowerShell 入口。
- `gyxx run`、`gyxx backfill` 和 `gyxx scripts run` 默认 dry-run；`--execute` 才启动
迁入后的本地副本。
- 最终候选调度包包含 21 个 XML，动作统一为
`D:\gyxx-flow\.venv\Scripts\python.exe -m gyxx_flow run <id> --scheduled`。
- 安装器默认只展示计划；即使显式 `-Apply` 也必须提供一个 `-WorkflowId`，不允许一次
批量切换。
## 自动化验证
- 全量 pytest：221 passed；仅有复制脚本内嵌 JavaScript 正则产生的非阻断
`SyntaxWarning`。
- 新框架与测试 Ruff：通过。
- 所有可执行 runtime 的 fatal Ruff 规则 `E9/F63/F7/F82`：通过；一个不独立执行的
上游源码摘录已在 manifest 中标记 `source_resource`。
- Python compileall、PowerShell AST、来源清单哈希、旧路径扫描和敏感信息扫描：通过。
- 历史数据：48,964 个文件、1,091,997,415 字节，复制前后聚合 SHA-256 一致。
## 适配器状态与尚未通过的门禁
- P5.6 已通过：131 个脚本统一绑定唯一 CDP、独立 Profile/Cookie/storage state；飞书
保持原身份/后端，PostgreSQL 只允许现有云端，Hermes 只允许本机。
- P6.4/P6.5：编排层已有 run journal、effect ledger 和 outbox，但尚未证明每个旧业务
脚本的所有内部外部写入都可追踪且全量幂等。
- P7：需要用户授权、凭据轮换、逐任务切换、真实 7 天/2 周观察和数据/通知对账。
因此工程“源码归属和独立入口”已通过，生产“全量安全执行和切换”尚未通过。
## 四源项目增量对齐复核（2026-07-28）
- 内容营销已合并“发布链接非空即进入采集”、星图普通达人失败不触发全局熔断、按款式和记录精确续跑。
- 商品商业已合并“ERP 编码只认天猫分组”，并把源端新增的波塞冬历史销量回填脚本完整迁入；脚本不引用旧项目。
- 波塞冬回填按 `raw → normalized → curated → exports` 分层，数据库继续使用云端配置，飞书仍使用原身份和项目内写入器。
- 新脚本登记独立 CDP 端口 `22130`，Profile/Cookie/storage state 继续由统一绑定派生并可迁移复用。
- 本轮全量 `294 passed`，修复定向测试 18 项通过，compileall、doctor、敏感信息与来源/目标哈希验证通过；当前虚拟环境未安装 Ruff，生产门禁仍保持 incomplete。
## 采集数据统一目录复核（2026-07-27）
- 四模块路径层统一改接共享 `DataLayout.for_module()` / `ModuleDataPaths`；新采集产物固定进入
`data/{raw,normalized,curated,exports,evidence}/<module>`，状态、日志和临时文件分别进入
`state/<module>`、`logs/<module>` 和 `tmp/<module>`。
- 内容模块的动态 mapping 和 V2 结果进入 normalized，跨平台 run_all 报告进入 curated，
checkpoint 进入 state；商品模块的 checkpoint、失败日志、调试文件和 vendor 下载已收口；
店铺调试输出进入 tmp；供应链缓存/锁/status 进入 state，原始下载、临时处理和最终报表分别
进入 raw、tmp 和 exports。
- 所有审计到的任意 `--output`、`--data-root` 和 vendor 环境输出均限制在
`GYXX_DATA_ROOT` 内；绝对外部路径或 `..` 逃逸会在写入前失败。
- `var/data/raw/<module>/legacy` 前后均为 48,961 个文件、1,091,967,729 字节；四模块分项
文件数和字节数完全一致，且不存在旁路 `var/raw` 或 `var/exports`。证据见
`var/evidence/unified-data-layout-20260727.json`。
- 供应链 raw 产物按 `run_id=<id>` 追加保存，编排器只读取本次运行目录且不会清理 raw；
采购更新触发任务放在 state，脚本结束只关闭浏览器进程并保留 Profile/Cookie 供复用。
- 空字符串或纯空白 `GYXX_DATA_ROOT` 会安全回退项目 `var`，不会把 cwd 当数据根。
- 全量回归为 294 passed；新代码 Ruff、compileall、PowerShell AST、四份来源清单目标哈希、
敏感信息和旧根依赖扫描全部通过。5 条 warning 是迁入脚本内嵌 JavaScript 正则的既有
`SyntaxWarning`，不影响本次目录验收。
src/gyxx_flow/acceptance.py
+111 -19
View File
@@ -10,7 +10,7 @@ from pathlib import Path
from gyxx_flow.catalog import CatalogError, WorkflowCatalog
from gyxx_flow.core.config import Settings
from gyxx_flow.script_catalog import ScriptCatalog
from gyxx_flow.script_catalog import ScriptCatalog, ScriptCatalogError
from gyxx_flow.security import scan_repository
_CHECKLIST = re.compile(r"^\s*-\s*\[([ xX])\]\s+(P\d+\.\d+)\b", re.MULTILINE)
@@ -61,38 +61,101 @@ def parse_plan_checklist(path: Path) -> tuple[PlanItem, ...]:
def build_acceptance_report(settings: Settings) -> AcceptanceReport:
project_root = settings.project_root
items = parse_plan_checklist(project_root / "plan.md")
items = parse_plan_checklist(project_root / "docs" / "plan.md")
checks = {
"catalog_21_tasks": _catalog_has_21_tasks(project_root),
"baseline_21_tasks": _baseline_has_21_tasks(settings.data_root),
"catalog_22_tasks": _catalog_has_22_tasks(project_root),
"scheduled_graphs_explicit": _scheduled_graphs_are_explicit(project_root),
"python_scheduler_only": _python_scheduler_is_the_only_scheduler(project_root),
"runtime_service_policy": _runtime_service_policy_is_configured(project_root),
"native_entrypoints_local": _native_entrypoints_are_local(project_root),
"runtime_sources_decoupled": _runtime_sources_are_decoupled(project_root),
"runnable_script_catalog": _runnable_script_catalog_is_complete(),
"public_command_registry_complete": _public_command_registry_is_complete(
project_root
),
"source_manifests_verified": _source_manifests_are_verified(project_root),
"secret_scan_clean": not scan_repository(project_root),
}
return AcceptanceReport(items, checks)
def _catalog_has_21_tasks(project_root: Path) -> bool:
def _catalog_has_22_tasks(project_root: Path) -> bool:
try:
catalog = WorkflowCatalog.load(project_root / "config")
except Exception:
return False
scheduled = catalog.scheduled_workflows()
return len(scheduled) == 21 and len(catalog.schedules) == 21
return (
len(scheduled) == 23
and len(catalog.schedules) == 23
and sum(schedule.enabled for schedule in catalog.schedules) == 23
)
def _baseline_has_21_tasks(data_root: Path) -> bool:
candidates = sorted((Path(data_root) / "baseline").glob("*/manifest.json"))
if not candidates:
return False
def _scheduled_graphs_are_explicit(project_root: Path) -> bool:
try:
payload = json.loads(candidates[-1].read_text(encoding="utf-8"))
tasks = payload["scheduled_tasks"]
return tasks["actual_count"] == 21 and len(tasks["tasks"]) == 21
catalog = WorkflowCatalog.load(project_root / "config")
except Exception:
return False
return all(workflow.steps for workflow in catalog.scheduled_workflows())
def _python_scheduler_is_the_only_scheduler(project_root: Path) -> bool:
scheduler_service = project_root / "src" / "gyxx_flow" / "scheduler_service.py"
retired_windows_scheduler = project_root / "src" / "gyxx_flow" / "scheduler.py"
systemd_unit = project_root / "deploy" / "gyxx-flow.service"
legacy_installer = project_root / "deploy" / "windows-service" / "install.ps1"
try:
unit = systemd_unit.read_text(encoding="utf-8").casefold()
except OSError:
return False
legacy_is_safe = True
if legacy_installer.exists():
try:
installer = legacy_installer.read_text(encoding="utf-8").casefold()
except OSError:
return False
legacy_is_safe = (
"schedule run" in installer
and "schtasks" not in installer
and "new-scheduledtask" not in installer
)
return (
scheduler_service.is_file()
and not retired_windows_scheduler.exists()
and "execstart=/opt/gyxx-flow/.venv/bin/python -m gyxx_flow schedule run"
in unit
and "environment=gyxx_data_root=/var/lib/gyxx-flow" in unit
and "oncalendar=" not in unit
and legacy_is_safe
)
def _runtime_service_policy_is_configured(project_root: Path) -> bool:
try:
payload = json.loads(
(project_root / "config" / "runtime-bindings.json").read_text(
encoding="utf-8"
)
)
services = payload["services"]
except (OSError, json.JSONDecodeError, KeyError, TypeError):
return False
local_urls = (
services.get("hermes_url", ""),
services.get("hermes_collector_url", ""),
services.get("hermes_analyzer_gateway_url", ""),
services.get("hermes_collector_gateway_url", ""),
)
return (
services.get("postgres") == "cloud"
and not services.get("postgres_host")
and not services.get("postgres_database")
and not services.get("postgres_user")
and services.get("hermes") == "local"
and all(url.startswith("http://127.0.0.1:") for url in local_urls)
)
def _native_entrypoints_are_local(project_root: Path) -> bool:
@@ -107,7 +170,6 @@ def _native_entrypoints_are_local(project_root: Path) -> bool:
/ "gyxx_flow"
/ "modules"
/ workflow.module
/ "runtime"
/ workflow.entry
).resolve(strict=True)
if not target.is_file() or not target.is_relative_to(project_root):
@@ -120,12 +182,42 @@ def _native_entrypoints_are_local(project_root: Path) -> bool:
return False
def _runnable_script_catalog_is_complete() -> bool:
def _public_command_registry_is_complete(project_root: Path) -> bool:
try:
scripts = ScriptCatalog.discover_default().scripts
except (OSError, ValueError):
workflows = WorkflowCatalog.load(project_root / "config").workflows
commands = ScriptCatalog.discover_default()
except (OSError, ValueError, ScriptCatalogError):
return False
return len(scripts) >= 100 and len({item.module for item in scripts}) == 4
runnable = tuple(item for item in workflows if item.trigger != "unavailable")
expected_modules = {item.module for item in runnable}
scripts = commands.scripts
if (
not scripts
or len(commands.command_ids) != len(set(commands.command_ids))
or {item.module for item in scripts} != expected_modules
):
return False
resolved_root = project_root.resolve()
for script in scripts:
if not script.path.is_file() or not script.path.is_relative_to(resolved_root):
return False
try:
for workflow in runnable:
entries = (
tuple(step.entry for step in workflow.steps)
if workflow.steps
else (workflow.entry,)
)
for entry in entries:
command = commands.get(f"{workflow.module}:{entry}")
if command.module != workflow.module or command.entry != entry:
return False
except ScriptCatalogError:
return False
return True
def _runtime_sources_are_decoupled(project_root: Path) -> bool:
src/gyxx_flow/adapters/__init__.py
+20
View File
@@ -1,5 +1,15 @@
"""Replaceable infrastructure adapters exposed to business modules."""
from .acceptance_policy import (
COOKIE_SKIP_EXIT_CODE,
WORKFLOW_ACCEPTANCE_RECIPIENT_OPEN_ID,
CookiePreflightResult,
WorkflowAcceptancePolicy,
WorkflowAcceptancePolicyError,
current_acceptance_policy,
resolve_notification_recipients,
skip_feishu_table_write,
)
from .browser import BrowserCookieStore, BrowserProfileLease, BrowserProfileManager
from .external import (
FeishuOutboxAdapter,
@@ -14,6 +24,7 @@ from .integration import (
RuntimeServicePolicy,
binding_from_environment,
environment_for_child_script,
resolve_hermes_profile_api_key,
)
from .native import (
DeferredModuleCommandStep,
@@ -27,6 +38,8 @@ __all__ = [
"BrowserCookieStore",
"BrowserProfileLease",
"BrowserProfileManager",
"COOKIE_SKIP_EXIT_CODE",
"CookiePreflightResult",
"DeferredModuleCommandStep",
"FeishuOutboxAdapter",
"HermesCommandAdapter",
@@ -40,6 +53,13 @@ __all__ = [
"RuntimeIntegrationCatalog",
"RuntimeIntegrationError",
"RuntimeServicePolicy",
"WORKFLOW_ACCEPTANCE_RECIPIENT_OPEN_ID",
"WorkflowAcceptancePolicy",
"WorkflowAcceptancePolicyError",
"binding_from_environment",
"current_acceptance_policy",
"environment_for_child_script",
"resolve_notification_recipients",
"resolve_hermes_profile_api_key",
"skip_feishu_table_write",
]
src/gyxx_flow/adapters/acceptance_policy.py
+690
View File
@@ -0,0 +1,690 @@
"""Fail-closed runtime policy for isolated workflow acceptance runs."""
from __future__ import annotations
import json
import os
import socket
import sqlite3
import threading
import time
from contextlib import contextmanager
from dataclasses import asdict, dataclass
from datetime import UTC, datetime
from pathlib import Path
from typing import Any, Mapping, Sequence
from urllib.parse import urlparse
from gyxx_flow.core.exit_codes import COOKIE_SKIP_EXIT_CODE
from .integration import RuntimeIntegrationBinding
WORKFLOW_ACCEPTANCE_RECIPIENT_OPEN_ID = "ou_8ee224968aa26a74c7d30ba27fed5eeb"
_TRUE_VALUES = frozenset({"1", "true", "yes", "on"})
_FALSE_VALUES = frozenset({"0", "false", "no", "off"})
_EVIDENCE_THREAD_LOCK = threading.Lock()
_EVIDENCE_LOCK_TIMEOUT_SECONDS = 30.0
def _reset_evidence_thread_lock_after_fork() -> None:
global _EVIDENCE_THREAD_LOCK
_EVIDENCE_THREAD_LOCK = threading.Lock()
if hasattr(os, "register_at_fork"):
os.register_at_fork(after_in_child=_reset_evidence_thread_lock_after_fork)
class WorkflowAcceptancePolicyError(ValueError):
"""Raised when an acceptance run would violate its isolation contract."""
@dataclass(frozen=True, slots=True)
class CookiePreflightResult:
"""A non-interactive decision about one browser binding's login state."""
status: str
reason: str
cookie_file: str
storage_state_file: str
profile_dir: str
cdp_url: str
@property
def should_skip(self) -> bool:
return self.status == "SKIPPED_COOKIE"
@dataclass(frozen=True, slots=True)
class WorkflowAcceptancePolicy:
"""Acceptance-only controls propagated to every migrated child process."""
enabled: bool
skip_feishu_table_writes: bool
notification_recipient_open_id: str | None
skip_invalid_cookie: bool
evidence_file: Path | None
@classmethod
def from_environment(
cls,
environment: Mapping[str, str] | None = None,
) -> "WorkflowAcceptancePolicy":
values = os.environ if environment is None else environment
enabled = _read_bool(values, "GYXX_WORKFLOW_ACCEPTANCE", default=False)
if not enabled:
return cls(
enabled=False,
skip_feishu_table_writes=False,
notification_recipient_open_id=None,
skip_invalid_cookie=False,
evidence_file=None,
)
skip_writes = _read_bool(
values,
"GYXX_FEISHU_TABLE_WRITE_DISABLED",
default=True,
)
skip_cookie = _read_bool(
values,
"GYXX_COOKIE_INVALID_SKIP",
default=True,
)
recipient = values.get(
"GYXX_NOTIFICATION_RECIPIENT_OPEN_ID",
WORKFLOW_ACCEPTANCE_RECIPIENT_OPEN_ID,
).strip()
if not skip_writes:
raise WorkflowAcceptancePolicyError(
"acceptance mode requires Feishu table writes to be disabled"
)
if not skip_cookie:
raise WorkflowAcceptancePolicyError(
"acceptance mode requires invalid cookies to be skipped"
)
if recipient != WORKFLOW_ACCEPTANCE_RECIPIENT_OPEN_ID:
raise WorkflowAcceptancePolicyError(
"acceptance notification recipient must be Wang Yunlong"
)
evidence_value = values.get("GYXX_ACCEPTANCE_EVIDENCE_FILE", "").strip()
if evidence_value:
evidence_file = Path(evidence_value).expanduser().resolve()
else:
data_root = values.get("GYXX_DATA_ROOT", "var").strip() or "var"
evidence_file = (
Path(data_root).expanduser().resolve()
/ "reports"
/ "workflow-acceptance"
/ "evidence.jsonl"
)
return cls(
enabled=True,
skip_feishu_table_writes=True,
notification_recipient_open_id=recipient,
skip_invalid_cookie=True,
evidence_file=evidence_file,
)
def environment(self) -> dict[str, str]:
"""Return canonical child-process variables for this policy."""
if not self.enabled:
return {}
assert self.notification_recipient_open_id is not None
assert self.evidence_file is not None
return {
"GYXX_WORKFLOW_ACCEPTANCE": "1",
"GYXX_FEISHU_TABLE_WRITE_DISABLED": "1",
"GYXX_COOKIE_INVALID_SKIP": "1",
"GYXX_NOTIFICATION_RECIPIENT_OPEN_ID": (
self.notification_recipient_open_id
),
"GYXX_ACCEPTANCE_EVIDENCE_FILE": str(self.evidence_file),
}
def notification_recipients(self, defaults: Sequence[str]) -> tuple[str, ...]:
"""Return the only recipients permitted for the current run."""
if self.enabled:
assert self.notification_recipient_open_id is not None
return (self.notification_recipient_open_id,)
return tuple(dict.fromkeys(item.strip() for item in defaults if item.strip()))
def skip_feishu_write(
self,
operation: str,
*,
details: Mapping[str, Any] | None = None,
) -> bool:
"""Record and approve a required Feishu table-write skip."""
if not self.enabled or not self.skip_feishu_table_writes:
return False
self.record(
"feishu_write_skipped",
operation=operation,
details=details or {},
)
return True
def preflight_cookie(
self,
binding: RuntimeIntegrationBinding,
*,
now_epoch: float | None = None,
environment: Mapping[str, str] | None = None,
) -> CookiePreflightResult:
"""Skip missing, invalid or expired state without opening a login page."""
if self.enabled and binding.login_mode == "D":
result = _cookie_result(
binding,
"SKIPPED_COOKIE",
"interactive login is disabled during acceptance runs",
)
else:
result = _inspect_cookie_state(binding, now_epoch=now_epoch)
values = os.environ if environment is None else environment
if (
result.should_skip
and binding.login_mode in {"B", "C"}
and _credentials_are_available(binding, values)
):
result = _cookie_result(
binding,
"READY",
"credential login fallback is available",
)
if self.enabled and self.skip_invalid_cookie and result.should_skip:
self.record("cookie_skipped", details=asdict(result))
return result
def record(
self,
event: str,
*,
operation: str | None = None,
details: Mapping[str, Any] | None = None,
) -> None:
"""Append non-secret, run-scoped acceptance evidence as JSONL."""
if not self.enabled or self.evidence_file is None:
return
payload = {
"timestamp": datetime.now(UTC).isoformat(),
"event": event,
"workflow_id": os.environ.get("GYXX_WORKFLOW_ID", ""),
"run_id": os.environ.get("GYXX_RUN_ID", ""),
"operation": operation or "",
"details": _safe_details(details or {}),
}
self.evidence_file.parent.mkdir(parents=True, exist_ok=True)
line = json.dumps(payload, ensure_ascii=False, sort_keys=True) + "\n"
lock_file = self.evidence_file.with_suffix(self.evidence_file.suffix + ".lock")
with _exclusive_file_lock(lock_file):
with self.evidence_file.open("a", encoding="utf-8", newline="") as handle:
handle.write(line)
@contextmanager
def _exclusive_file_lock(lock_file: Path):
"""Serialize evidence appends across threads and child processes."""
lock_file.parent.mkdir(parents=True, exist_ok=True)
if not _EVIDENCE_THREAD_LOCK.acquire(timeout=_EVIDENCE_LOCK_TIMEOUT_SECONDS):
raise WorkflowAcceptancePolicyError(
"timed out while locking acceptance evidence in this process"
)
try:
handle = lock_file.open("a+b")
except BaseException:
_EVIDENCE_THREAD_LOCK.release()
raise
try:
deadline = time.monotonic() + _EVIDENCE_LOCK_TIMEOUT_SECONDS
while not _try_lock_file(handle):
if time.monotonic() >= deadline:
raise WorkflowAcceptancePolicyError(
"timed out while locking the acceptance evidence file"
)
time.sleep(0.01)
try:
yield
finally:
_unlock_file(handle)
finally:
handle.close()
_EVIDENCE_THREAD_LOCK.release()
def _try_lock_file(handle) -> bool:
if os.name == "nt":
import msvcrt
handle.seek(0, os.SEEK_END)
if handle.tell() == 0:
handle.write(b"\0")
handle.flush()
handle.seek(0)
try:
msvcrt.locking(handle.fileno(), msvcrt.LK_NBLCK, 1)
except OSError:
return False
return True
import fcntl
try:
fcntl.flock(handle.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)
except OSError:
return False
return True
def _unlock_file(handle) -> None:
if os.name == "nt":
import msvcrt
handle.seek(0)
msvcrt.locking(handle.fileno(), msvcrt.LK_UNLCK, 1)
return
import fcntl
fcntl.flock(handle.fileno(), fcntl.LOCK_UN)
def current_acceptance_policy(
environment: Mapping[str, str] | None = None,
) -> WorkflowAcceptancePolicy:
return WorkflowAcceptancePolicy.from_environment(environment)
def resolve_notification_recipients(
defaults: Sequence[str],
environment: Mapping[str, str] | None = None,
) -> tuple[str, ...]:
return current_acceptance_policy(environment).notification_recipients(defaults)
def skip_feishu_table_write(
operation: str,
*,
details: Mapping[str, Any] | None = None,
environment: Mapping[str, str] | None = None,
) -> bool:
return current_acceptance_policy(environment).skip_feishu_write(
operation,
details=details,
)
def _inspect_cookie_state(
binding: RuntimeIntegrationBinding,
*,
now_epoch: float | None,
) -> CookiePreflightResult:
now = time.time() if now_epoch is None else now_epoch
cookies: list[Mapping[str, Any]] = []
storage_origins: list[str] = []
invalid_reason = ""
if binding.cookie_file.is_file():
try:
payload = json.loads(binding.cookie_file.read_text(encoding="utf-8"))
loaded, origins, invalid_reason = _cookie_payload(
payload,
label="cookie file",
)
cookies.extend(loaded)
storage_origins.extend(origins)
except (OSError, json.JSONDecodeError):
invalid_reason = "cookie file is unreadable"
if not invalid_reason and binding.storage_state_file.is_file():
try:
payload = json.loads(
binding.storage_state_file.read_text(encoding="utf-8")
)
loaded, origins, invalid_reason = _cookie_payload(
payload,
label="storage state",
)
cookies.extend(loaded)
storage_origins.extend(origins)
except (OSError, json.JSONDecodeError):
invalid_reason = "storage state is unreadable"
if invalid_reason:
return _cookie_result(binding, "SKIPPED_COOKIE", invalid_reason)
state_failure_reason = ""
if cookies:
usable: list[Mapping[str, Any]] = []
for cookie in cookies:
if not cookie.get("value"):
continue
expires = cookie.get("expires", cookie.get("expirationDate"))
if expires in (None, "", 0, -1):
usable.append(cookie)
continue
try:
if float(expires) > now:
usable.append(cookie)
except (TypeError, ValueError):
return _cookie_result(
binding,
"SKIPPED_COOKIE",
"cookie expiry is invalid",
)
if not usable:
return _cookie_result(binding, "SKIPPED_COOKIE", "all cookies are expired")
domain_matched = _cookies_match_required_domains(
usable,
binding.required_cookie_domains,
)
if not domain_matched:
state_failure_reason = (
"cookie state does not match the required platform domain"
)
elif binding.required_cookie_names and not any(
str(cookie.get("name", "")) in binding.required_cookie_names
for cookie in domain_matched
):
state_failure_reason = "required platform login cookie is missing"
else:
return _cookie_result(binding, "READY", "cookie state is available")
if storage_origins and _origins_match_required_domains(
storage_origins,
binding.required_cookie_domains,
):
return _cookie_result(binding, "READY", "browser storage state is available")
if _profile_has_usable_cookies(
binding.profile_dir,
now_epoch=now,
required_domains=binding.required_cookie_domains,
required_names=binding.required_cookie_names,
):
return _cookie_result(binding, "READY", "browser profile state is available")
if (
not binding.required_cookie_domains
and not binding.required_cookie_names
and _cdp_is_listening(binding)
):
return _cookie_result(binding, "READY", "bound CDP endpoint is listening")
return _cookie_result(
binding,
"SKIPPED_COOKIE",
state_failure_reason or "browser state is missing",
)
def _cookie_payload(
payload: object,
*,
label: str,
) -> tuple[list[Mapping[str, Any]], list[str], str]:
if isinstance(payload, list):
cookies = payload
origins: object = []
elif isinstance(payload, dict):
cookies = payload.get("cookies", [])
origins = payload.get("origins", [])
else:
return [], [], f"{label} is not a cookie list or storage-state object"
if not isinstance(cookies, list) or not all(
isinstance(item, dict) for item in cookies
):
return [], [], f"{label} cookies are invalid"
if not isinstance(origins, list) or not all(
isinstance(item, dict) and isinstance(item.get("origin"), str)
for item in origins
):
return [], [], f"{label} origins are invalid"
return cookies, [item["origin"] for item in origins], ""
def _cookie_domain_matches(cookie_domain: str, required_domain: str) -> bool:
cookie_host = cookie_domain.strip().casefold().lstrip(".")
required_host = required_domain.strip().casefold().lstrip(".")
if not cookie_host or not required_host:
return False
return (
cookie_host == required_host
or cookie_host.endswith(f".{required_host}")
or required_host.endswith(f".{cookie_host}")
)
def _cookies_match_required_domains(
cookies: Sequence[Mapping[str, Any]],
required_domains: Sequence[str],
) -> list[Mapping[str, Any]]:
if not required_domains:
return list(cookies)
return [
cookie
for cookie in cookies
if any(
_cookie_domain_matches(str(cookie.get("domain", "")), required)
for required in required_domains
)
]
def _origins_match_required_domains(
origins: Sequence[str],
required_domains: Sequence[str],
) -> bool:
if not required_domains:
return bool(origins)
for origin in origins:
try:
host = urlparse(origin).hostname
except (TypeError, ValueError):
host = None
if host and any(
_cookie_domain_matches(host, required) for required in required_domains
):
return True
return False
def _credentials_are_available(
binding: RuntimeIntegrationBinding,
environment: Mapping[str, str],
) -> bool:
return bool(binding.credential_env_names) and all(
environment.get(name, "").strip()
for name in binding.credential_env_names
)
def _cookie_result(
binding: RuntimeIntegrationBinding,
status: str,
reason: str,
) -> CookiePreflightResult:
return CookiePreflightResult(
status=status,
reason=reason,
cookie_file=str(binding.cookie_file),
storage_state_file=str(binding.storage_state_file),
profile_dir=str(binding.profile_dir),
cdp_url=binding.cdp_url,
)
def _profile_has_usable_cookies(
path: Path,
*,
now_epoch: float,
required_domains: Sequence[str] = (),
required_names: Sequence[str] = (),
) -> bool:
"""Treat a Chrome profile as reusable only when it has live cookies.
A newly-created profile already contains many files, so directory
non-emptiness is not evidence of an authenticated session. Chromium stores
expiry values as microseconds since 1601-01-01.
"""
if not path.is_dir():
return False
chrome_now = int((now_epoch + 11_644_473_600) * 1_000_000)
candidates = [path / "Network" / "Cookies", path / "Default" / "Network" / "Cookies"]
try:
candidates.extend(
profile / "Network" / "Cookies"
for profile in path.glob("Profile *")
if profile.is_dir()
)
except OSError:
return False
for cookie_db in candidates:
if not cookie_db.is_file():
continue
try:
uri = f"{cookie_db.resolve().as_uri()}?mode=ro"
with sqlite3.connect(uri, uri=True, timeout=0.2) as connection:
columns = {
str(row[1])
for row in connection.execute("PRAGMA table_info(cookies)")
}
if "expires_utc" not in columns:
continue
if required_domains and "host_key" not in columns:
continue
if required_names and "name" not in columns:
continue
selected = [
column
for column in ("host_key", "name")
if column in columns
]
rows = connection.execute(
f"SELECT {', '.join(selected) or 'expires_utc'} FROM cookies "
"WHERE expires_utc = 0 OR expires_utc > ?",
(chrome_now,),
).fetchall()
host_index = selected.index("host_key") if "host_key" in selected else None
name_index = selected.index("name") if "name" in selected else None
if any(
(
not required_domains
or (
host_index is not None
and any(
_cookie_domain_matches(str(row[host_index]), required)
for required in required_domains
)
)
)
and (
not required_names
or (
name_index is not None
and str(row[name_index]) in required_names
)
)
for row in rows
):
return True
except (OSError, sqlite3.Error):
continue
return False
def _cdp_is_listening(binding: RuntimeIntegrationBinding) -> bool:
try:
with socket.create_connection(("127.0.0.1", binding.cdp_port), timeout=0.2):
return True
except OSError:
return False
def _read_bool(
values: Mapping[str, str],
name: str,
*,
default: bool,
) -> bool:
raw = values.get(name, "").strip().casefold()
if not raw:
return default
if raw in _TRUE_VALUES:
return True
if raw in _FALSE_VALUES:
return False
raise WorkflowAcceptancePolicyError(f"{name} must be a boolean value")
def _safe_details(details: Mapping[str, Any]) -> dict[str, Any]:
return _safe_detail_mapping(details, seen=set())
def _safe_detail_mapping(
details: Mapping[object, Any],
*,
seen: set[int],
) -> dict[str, Any]:
identity = id(details)
if identity in seen:
return {"recursive": "<redacted>"}
seen.add(identity)
safe: dict[str, Any] = {}
for key, value in details.items():
safe_key = str(key)
if _is_sensitive_detail_key(safe_key):
safe[safe_key] = "<redacted>"
else:
safe[safe_key] = _safe_detail_value(value, seen=seen)
seen.remove(identity)
return safe
def _safe_detail_value(value: Any, *, seen: set[int]) -> Any:
if value is None or isinstance(value, (bool, int, float, str)):
return value
if isinstance(value, Mapping):
return _safe_detail_mapping(value, seen=seen)
if isinstance(value, list):
identity = id(value)
if identity in seen:
return "<redacted>"
seen.add(identity)
safe = [_safe_detail_value(item, seen=seen) for item in value]
seen.remove(identity)
return safe
if isinstance(value, tuple):
identity = id(value)
if identity in seen:
return "<redacted>"
seen.add(identity)
safe = tuple(_safe_detail_value(item, seen=seen) for item in value)
seen.remove(identity)
return safe
return str(value)
def _is_sensitive_detail_key(key: str) -> bool:
return any(
marker in key.casefold()
for marker in ("password", "secret", "token", "key")
)
__all__ = [
"COOKIE_SKIP_EXIT_CODE",
"CookiePreflightResult",
"WORKFLOW_ACCEPTANCE_RECIPIENT_OPEN_ID",
"WorkflowAcceptancePolicy",
"WorkflowAcceptancePolicyError",
"current_acceptance_policy",
"resolve_notification_recipients",
"skip_feishu_table_write",
]
src/gyxx_flow/adapters/integration.py
+569 -52
View File
@@ -19,6 +19,49 @@ class RuntimeIntegrationError(ValueError):
"""Raised when an integration binding is incomplete or unsafe."""
def resolve_hermes_profile_api_key(
profile: str,
environment: Mapping[str, str] | None = None,
*,
preferred_environment_names: tuple[str, ...] = (),
) -> str:
"""Resolve a local Hermes gateway key without copying it into project files."""
if (
not isinstance(profile, str)
or not profile.strip()
or profile.strip() in {".", ".."}
or any(separator in profile for separator in ("/", "\\"))
):
raise RuntimeIntegrationError("Hermes profile must be a safe directory name")
values = os.environ if environment is None else environment
for name in (
*preferred_environment_names,
"GYXX_HERMES_API_KEY",
"HERMES_API_KEY",
):
configured = values.get(name, "").strip()
if configured:
return configured
hermes_home = values.get("HERMES_HOME", "").strip()
if not hermes_home:
return ""
env_path = Path(hermes_home).expanduser() / "profiles" / profile.strip() / ".env"
try:
lines = env_path.read_text(encoding="utf-8").splitlines()
except OSError:
return ""
for raw_line in lines:
line = raw_line.strip()
if not line or line.startswith("#") or "=" not in line:
continue
key, value = line.split("=", 1)
if key.strip() == "API_SERVER_KEY":
return value.strip().strip('"').strip("'")
return ""
@dataclass(frozen=True, slots=True)
class RuntimeIntegrationBinding:
script_id: str
@@ -29,40 +72,138 @@ class RuntimeIntegrationBinding:
profile_dir: Path
cookie_file: Path
storage_state_file: Path
command_id: str = ""
state_key: str = ""
aliases: tuple[str, ...] = ()
login_mode: str = "A"
required_cookie_domains: tuple[str, ...] = ()
required_cookie_names: tuple[str, ...] = ()
credential_env_names: tuple[str, ...] = ()
def __post_init__(self) -> None:
# Defaults keep direct construction and schema-v1 callers compatible.
if not self.command_id:
object.__setattr__(self, "command_id", self.script_id)
if not self.state_key:
object.__setattr__(self, "state_key", self.script_id)
object.__setattr__(self, "aliases", tuple(self.aliases))
login_mode = self.login_mode.strip().upper()
if login_mode not in {"A", "B", "C", "D"}:
raise RuntimeIntegrationError("browser login mode must be A, B, C or D")
object.__setattr__(self, "login_mode", login_mode)
object.__setattr__(
self,
"required_cookie_domains",
_normalize_cookie_domains(self.required_cookie_domains),
)
object.__setattr__(
self,
"required_cookie_names",
_normalize_binding_strings(
self.required_cookie_names,
field="required cookie names",
),
)
object.__setattr__(
self,
"credential_env_names",
_normalize_binding_strings(
self.credential_env_names,
field="credential environment names",
require_env_name=True,
),
)
@dataclass(frozen=True, slots=True)
class RuntimeServicePolicy:
"""Keep legacy Feishu, cloud PostgreSQL, and loopback-only Hermes."""
"""Keep legacy Feishu, configurable PostgreSQL, and local Hermes."""
postgres_mode: str = "cloud"
postgres_host: str = ""
postgres_port: int = 5432
postgres_database: str = ""
postgres_user: str = ""
hermes_url: str = "http://127.0.0.1:8642/v1/chat/completions"
hermes_collector_url: str = "http://127.0.0.1:8643/v1/chat/completions"
hermes_analyzer_gateway_url: str = "http://127.0.0.1:8642/v1"
hermes_collector_gateway_url: str = "http://127.0.0.1:8643/v1"
def __post_init__(self) -> None:
postgres_mode = self.postgres_mode.strip().casefold()
if postgres_mode not in {"cloud", "local"}:
raise RuntimeIntegrationError("PostgreSQL mode must be cloud or local")
object.__setattr__(self, "postgres_mode", postgres_mode)
if not 1 <= self.postgres_port <= 65535:
raise RuntimeIntegrationError("PostgreSQL port must be valid")
if postgres_mode == "local":
if not _is_loopback_host(self.postgres_host):
raise RuntimeIntegrationError("local PostgreSQL requires loopback host")
if not self.postgres_database or not self.postgres_user:
raise RuntimeIntegrationError(
"local PostgreSQL database and user are required"
)
elif self.postgres_host and _is_loopback_host(self.postgres_host):
raise RuntimeIntegrationError("cloud PostgreSQL requires a remote host")
_require_loopback_url(self.hermes_url, field="Hermes local URL")
_require_loopback_url(
self.hermes_collector_url,
field="Hermes collector URL",
)
_require_loopback_url(
self.hermes_analyzer_gateway_url,
field="Hermes analyzer gateway URL",
)
_require_loopback_url(
self.hermes_collector_gateway_url,
field="Hermes collector gateway URL",
)
def apply(self, environment: Mapping[str, str]) -> dict[str, str]:
result = dict(environment)
_apply_canonical_secrets(result)
_fill_database_aliases(result)
_validate_cloud_database(result)
if self.postgres_mode == "local":
_fill_local_database_defaults(
result,
host=self.postgres_host,
port=self.postgres_port,
database=self.postgres_database,
user=self.postgres_user,
)
_validate_local_database(result)
else:
_validate_cloud_database(result)
configured_hermes = _configured_hermes_urls(result)
for field, value in configured_hermes:
_require_loopback_url(value, field=field)
result.update(
{
"GYXX_FEISHU_MODE": "legacy",
"GYXX_POSTGRES_MODE": "cloud",
"GYXX_POSTGRES_MODE": self.postgres_mode,
"GYXX_HERMES_MODE": "local",
}
)
# Preserve every original Feishu/DB/Hermes setting. Defaults are only
# supplied for the two historical Hermes HTTP variable names.
result.setdefault("HERMES_ANALYZER_URL", self.hermes_url)
result.setdefault("ANALYZER_API_SERVER_URL", self.hermes_url)
_set_if_blank(result, "HERMES_ANALYZER_URL", self.hermes_url)
_set_if_blank(result, "ANALYZER_API_SERVER_URL", self.hermes_url)
_set_if_blank(result, "COLLECTOR_API_SERVER_URL", self.hermes_collector_url)
_set_if_blank(
result,
"ANALYZER_HERMES_GATEWAY_URL",
self.hermes_analyzer_gateway_url,
)
_set_if_blank(
result,
"COLLECTOR_HERMES_GATEWAY_URL",
self.hermes_collector_gateway_url,
)
return result
class RuntimeIntegrationCatalog:
"""Immutable exact allocation of one portable browser binding per script."""
"""Immutable browser bindings addressed by stable IDs and legacy aliases."""
def __init__(
self,
@@ -70,7 +211,24 @@ class RuntimeIntegrationCatalog:
*,
service_policy: RuntimeServicePolicy,
) -> None:
self._bindings = MappingProxyType(dict(sorted(bindings.items())))
primary = dict(sorted(bindings.items()))
aliases: dict[str, str] = {}
for command_id, binding in primary.items():
if command_id != binding.command_id:
raise RuntimeIntegrationError(
f"runtime binding key does not match command_id: {command_id}"
)
for alias in (command_id, binding.script_id, *binding.aliases):
if not isinstance(alias, str) or not alias.strip():
raise RuntimeIntegrationError("runtime binding aliases must be non-empty")
existing = aliases.get(alias)
if existing is not None and existing != command_id:
raise RuntimeIntegrationError(
f"runtime binding alias is ambiguous: {alias}"
)
aliases[alias] = command_id
self._bindings = MappingProxyType(primary)
self._aliases = MappingProxyType(dict(sorted(aliases.items())))
self.service_policy = service_policy
@classmethod
@@ -88,60 +246,87 @@ class RuntimeIntegrationCatalog:
raise RuntimeIntegrationError(
f"cannot load runtime integration catalog: {path.name}"
) from exc
if not isinstance(payload, dict) or payload.get("schema_version") != 1:
if not isinstance(payload, dict) or payload.get("schema_version") not in {1, 2}:
raise RuntimeIntegrationError("unsupported runtime binding schema")
host = payload.get("cdp_host")
if host not in {"127.0.0.1", "localhost", "::1"}:
raise RuntimeIntegrationError("browser CDP host must be loopback")
allocations = payload.get("scripts")
if not isinstance(allocations, dict):
configured_bindings = payload.get("scripts")
if not isinstance(configured_bindings, dict):
raise RuntimeIntegrationError("runtime binding scripts must be an object")
configured = set(allocations)
discovered = set(scripts.script_ids)
if configured != discovered:
missing = len(discovered - configured)
extra = len(configured - discovered)
raise RuntimeIntegrationError(
f"runtime binding coverage mismatch; missing={missing}, extra={extra}"
)
ports = list(allocations.values())
if (
not all(isinstance(port, int) and 22000 <= port <= 22999 for port in ports)
or len(set(ports)) != len(ports)
):
raise RuntimeIntegrationError(
"runtime binding ports must be unique integers in 22000..22999"
)
root = Path(data_root).expanduser().resolve()
bindings: dict[str, RuntimeIntegrationBinding] = {}
for script in scripts.scripts:
port = allocations[script.script_id]
state_root = _browser_state_root(root, script.module, script.script_id)
bindings[script.script_id] = RuntimeIntegrationBinding(
script_id=script.script_id,
module=script.module,
entry=script.entry,
cdp_port=port,
cdp_url=f"http://{host}:{port}",
profile_dir=state_root / "profile",
cookie_file=state_root / "cookies.json",
storage_state_file=state_root / "storage_state.json",
if payload["schema_version"] == 1:
bindings = _load_legacy_bindings(
configured_bindings,
scripts=scripts,
data_root=root,
cdp_host=host,
)
else:
bindings = _load_stable_bindings(
configured_bindings,
scripts=scripts,
data_root=root,
cdp_host=host,
)
services = payload.get("services")
if not isinstance(services, dict):
raise RuntimeIntegrationError("runtime binding services must be an object")
postgres_mode = services.get("postgres")
if (
services.get("feishu") != "legacy"
or services.get("postgres") != "cloud"
or postgres_mode not in {"cloud", "local"}
or services.get("hermes") != "local"
):
raise RuntimeIntegrationError("unsupported runtime service policy")
hermes_url = services.get("hermes_url")
if not isinstance(hermes_url, str):
raise RuntimeIntegrationError("runtime service policy requires Hermes URL")
hermes_collector_url = services.get("hermes_collector_url")
hermes_analyzer_gateway_url = services.get("hermes_analyzer_gateway_url")
hermes_collector_gateway_url = services.get("hermes_collector_gateway_url")
if not all(
isinstance(value, str)
for value in (
hermes_collector_url,
hermes_analyzer_gateway_url,
hermes_collector_gateway_url,
)
):
raise RuntimeIntegrationError(
"runtime service policy requires both local Hermes roles"
)
local_postgres = postgres_mode == "local"
postgres_host = services.get(
"postgres_host", "127.0.0.1" if local_postgres else ""
)
postgres_port = services.get("postgres_port", 5432)
postgres_database = services.get(
"postgres_database", "gyxx_super_data" if local_postgres else ""
)
postgres_user = services.get(
"postgres_user", "gyxx_flow" if local_postgres else ""
)
if (
not isinstance(postgres_host, str)
or not isinstance(postgres_port, int)
or not isinstance(postgres_database, str)
or not isinstance(postgres_user, str)
):
raise RuntimeIntegrationError("invalid PostgreSQL service settings")
return cls(
bindings,
service_policy=RuntimeServicePolicy(hermes_url=hermes_url),
service_policy=RuntimeServicePolicy(
postgres_mode=postgres_mode,
postgres_host=postgres_host,
postgres_port=postgres_port,
postgres_database=postgres_database,
postgres_user=postgres_user,
hermes_url=hermes_url,
hermes_collector_url=hermes_collector_url,
hermes_analyzer_gateway_url=hermes_analyzer_gateway_url,
hermes_collector_gateway_url=hermes_collector_gateway_url,
),
)
@classmethod
@@ -161,31 +346,57 @@ class RuntimeIntegrationCatalog:
@property
def script_ids(self) -> tuple[str, ...]:
"""Return current physical script IDs for bootstrap compatibility."""
return tuple(sorted(binding.script_id for binding in self._bindings.values()))
@property
def command_ids(self) -> tuple[str, ...]:
"""Return stable logical command IDs."""
return tuple(self._bindings)
def binding_for(self, script_id: str) -> RuntimeIntegrationBinding:
def binding_for(self, binding_id: str) -> RuntimeIntegrationBinding:
try:
return self._bindings[script_id]
command_id = self._aliases[binding_id]
return self._bindings[command_id]
except KeyError as exc:
raise RuntimeIntegrationError(f"unknown runtime script binding: {script_id}") from exc
raise RuntimeIntegrationError(
f"unknown runtime script binding: {binding_id}"
) from exc
def environment_for(
self,
script_id: str,
binding_id: str,
base_environment: Mapping[str, str] | None = None,
) -> dict[str, str]:
binding = self.binding_for(script_id)
binding = self.binding_for(binding_id)
result = self.service_policy.apply(
os.environ if base_environment is None else base_environment
)
result.update(
{
"GYXX_SCRIPT_ID": binding.script_id,
"GYXX_COMMAND_ID": binding.command_id,
"GYXX_BROWSER_STATE_KEY": binding.state_key,
"GYXX_BROWSER_CDP_PORT": str(binding.cdp_port),
"GYXX_BROWSER_CDP_URL": binding.cdp_url,
"GYXX_BROWSER_PROFILE_DIR": str(binding.profile_dir),
"GYXX_BROWSER_COOKIE_FILE": str(binding.cookie_file),
"GYXX_BROWSER_STORAGE_STATE_FILE": str(binding.storage_state_file),
"GYXX_BROWSER_LOGIN_MODE": binding.login_mode,
"GYXX_BROWSER_REQUIRED_COOKIE_DOMAINS": json.dumps(
binding.required_cookie_domains,
ensure_ascii=True,
),
"GYXX_BROWSER_REQUIRED_COOKIE_NAMES": json.dumps(
binding.required_cookie_names,
ensure_ascii=True,
),
"GYXX_BROWSER_CREDENTIAL_ENV_NAMES": json.dumps(
binding.credential_env_names,
ensure_ascii=True,
),
# Compatibility aliases consumed by the migrated browser engines.
# They are deliberately overwritten so every engine uses the
# catalog allocation instead of a legacy shared profile/port.
@@ -199,6 +410,7 @@ class RuntimeIntegrationCatalog:
"WANXIANG_USER_DATA_DIR": str(binding.profile_dir),
"GUANGHE_USER_DATA_DIR": str(binding.profile_dir),
"GUANGHE_LUGGAGE_USER_DATA_DIR": str(binding.profile_dir),
"DY_COOKIES_FILE": str(binding.cookie_file),
"DY_STORAGE_STATE_FILE": str(binding.storage_state_file),
}
)
@@ -242,13 +454,30 @@ def binding_from_environment(
try:
script_id = values["GYXX_SCRIPT_ID"]
module, entry = script_id.split(":", 1)
command_id = values.get("GYXX_COMMAND_ID", script_id).strip()
state_key = values.get("GYXX_BROWSER_STATE_KEY", script_id).strip()
port = int(values["GYXX_BROWSER_CDP_PORT"])
cdp_url = values["GYXX_BROWSER_CDP_URL"]
profile = Path(values["GYXX_BROWSER_PROFILE_DIR"]).expanduser().resolve()
cookie = Path(values["GYXX_BROWSER_COOKIE_FILE"]).expanduser().resolve()
storage = Path(values["GYXX_BROWSER_STORAGE_STATE_FILE"]).expanduser().resolve()
login_mode = values.get("GYXX_BROWSER_LOGIN_MODE", "A")
required_cookie_domains = _binding_tuple_from_environment(
values,
"GYXX_BROWSER_REQUIRED_COOKIE_DOMAINS",
)
required_cookie_names = _binding_tuple_from_environment(
values,
"GYXX_BROWSER_REQUIRED_COOKIE_NAMES",
)
credential_env_names = _binding_tuple_from_environment(
values,
"GYXX_BROWSER_CREDENTIAL_ENV_NAMES",
)
except (KeyError, ValueError) as exc:
raise RuntimeIntegrationError("current process has no valid browser binding") from exc
if not command_id or not state_key:
raise RuntimeIntegrationError("current process has no stable browser identity")
_require_loopback_url(cdp_url, field="browser CDP URL")
if not 22000 <= port <= 22999:
raise RuntimeIntegrationError("browser CDP port is outside the managed range")
@@ -261,15 +490,235 @@ def binding_from_environment(
profile_dir=profile,
cookie_file=cookie,
storage_state_file=storage,
command_id=command_id,
state_key=state_key,
aliases=(script_id,),
login_mode=login_mode,
required_cookie_domains=required_cookie_domains,
required_cookie_names=required_cookie_names,
credential_env_names=credential_env_names,
)
def _browser_state_root(data_root: Path, module: str, script_id: str) -> Path:
entry = script_id.split(":", 1)[1]
stem = Path(entry).stem
def _load_legacy_bindings(
allocations: Mapping[str, object],
*,
scripts: ScriptCatalog,
data_root: Path,
cdp_host: str,
) -> dict[str, RuntimeIntegrationBinding]:
configured = set(allocations)
discovered = set(scripts.script_ids)
if configured != discovered:
missing = len(discovered - configured)
extra = len(configured - discovered)
raise RuntimeIntegrationError(
f"runtime binding coverage mismatch; missing={missing}, extra={extra}"
)
_validate_unique_ports(allocations.values())
bindings: dict[str, RuntimeIntegrationBinding] = {}
for script in scripts.scripts:
script_id = script.script_id
command_id = getattr(script, "command_id", script_id)
port = allocations[script_id]
if not isinstance(port, int):
raise RuntimeIntegrationError(
"runtime binding ports must be unique integers in 22000..22999"
)
state_root = _browser_state_root(data_root, script.module, script_id)
bindings[command_id] = RuntimeIntegrationBinding(
script_id=script_id,
module=script.module,
entry=script.entry,
cdp_port=port,
cdp_url=f"http://{cdp_host}:{port}",
profile_dir=state_root / "profile",
cookie_file=state_root / "cookies.json",
storage_state_file=state_root / "storage_state.json",
command_id=command_id,
state_key=script_id,
aliases=(script_id,),
)
return bindings
def _load_stable_bindings(
allocations: Mapping[str, object],
*,
scripts: ScriptCatalog,
data_root: Path,
cdp_host: str,
) -> dict[str, RuntimeIntegrationBinding]:
bindings: dict[str, RuntimeIntegrationBinding] = {}
ports: list[object] = []
state_keys: list[str] = []
for command_id, raw_binding in allocations.items():
if not isinstance(command_id, str) or not command_id.strip():
raise RuntimeIntegrationError("runtime command IDs must be non-empty")
if not isinstance(raw_binding, dict):
raise RuntimeIntegrationError(
f"stable runtime binding must be an object: {command_id}"
)
script_id = raw_binding.get("script_id")
state_key = raw_binding.get("state_key")
port = raw_binding.get("cdp_port")
configured_aliases = raw_binding.get("aliases", [])
login_mode = raw_binding.get("login_mode", "A")
required_cookie_domains = raw_binding.get("required_cookie_domains", [])
required_cookie_names = raw_binding.get("required_cookie_names", [])
credential_env_names = raw_binding.get("credential_env_names", [])
if (
not isinstance(script_id, str)
or ":" not in script_id
or not isinstance(state_key, str)
or not state_key.strip()
or not isinstance(configured_aliases, list)
or not all(
isinstance(alias, str) and alias.strip()
for alias in configured_aliases
)
or not isinstance(login_mode, str)
or not isinstance(required_cookie_domains, list)
or not isinstance(required_cookie_names, list)
or not isinstance(credential_env_names, list)
or not all(
isinstance(value, str)
for values in (
required_cookie_domains,
required_cookie_names,
credential_env_names,
)
for value in values
)
):
raise RuntimeIntegrationError(
f"invalid stable runtime binding: {command_id}"
)
module, entry = script_id.split(":", 1)
if not module or not entry:
raise RuntimeIntegrationError(
f"invalid stable runtime script ID: {command_id}"
)
ports.append(port)
state_keys.append(state_key)
state_root = _browser_state_root(data_root, module, state_key)
bindings[command_id] = RuntimeIntegrationBinding(
script_id=script_id,
module=module,
entry=entry,
cdp_port=port,
cdp_url=f"http://{cdp_host}:{port}",
profile_dir=state_root / "profile",
cookie_file=state_root / "cookies.json",
storage_state_file=state_root / "storage_state.json",
command_id=command_id,
state_key=state_key,
aliases=tuple(configured_aliases),
login_mode=login_mode,
required_cookie_domains=tuple(required_cookie_domains),
required_cookie_names=tuple(required_cookie_names),
credential_env_names=tuple(credential_env_names),
)
_validate_unique_ports(ports)
if len(set(state_keys)) != len(state_keys):
raise RuntimeIntegrationError("runtime binding state keys must be unique")
aliases = {
alias: command_id
for command_id, binding in bindings.items()
for alias in (binding.script_id, *binding.aliases)
}
missing: list[str] = []
mismatched: list[str] = []
for script in scripts.scripts:
legacy_id = getattr(script, "legacy_script_id", script.script_id)
command_id = getattr(script, "command_id", None)
if command_id is None:
resolved_command_id = aliases.get(legacy_id)
else:
resolved_command_id = command_id if command_id in bindings else None
if resolved_command_id is None:
missing.append(command_id or legacy_id)
continue
binding = bindings[resolved_command_id]
if binding.module != script.module or binding.entry != script.entry:
mismatched.append(command_id or legacy_id)
if missing or mismatched:
raise RuntimeIntegrationError(
"runtime binding coverage mismatch; "
f"missing={len(missing)}, mismatched={len(mismatched)}"
)
return bindings
def _validate_unique_ports(ports: object) -> None:
values = list(ports) # type: ignore[arg-type]
if (
not all(isinstance(port, int) and 22000 <= port <= 22999 for port in values)
or len(set(values)) != len(values)
):
raise RuntimeIntegrationError(
"runtime binding ports must be unique integers in 22000..22999"
)
def _normalize_cookie_domains(values: tuple[str, ...]) -> tuple[str, ...]:
normalized: list[str] = []
for value in values:
if not isinstance(value, str):
raise RuntimeIntegrationError("required cookie domains must be strings")
domain = value.strip().casefold().lstrip(".")
if not domain or any(character.isspace() for character in domain):
raise RuntimeIntegrationError("required cookie domains must be valid hosts")
normalized.append(domain)
return tuple(dict.fromkeys(normalized))
def _normalize_binding_strings(
values: tuple[str, ...],
*,
field: str,
require_env_name: bool = False,
) -> tuple[str, ...]:
normalized: list[str] = []
for value in values:
if not isinstance(value, str) or not value.strip():
raise RuntimeIntegrationError(f"{field} must be non-empty strings")
item = value.strip()
if require_env_name and not (
item.replace("_", "a").isalnum() and not item[0].isdigit()
):
raise RuntimeIntegrationError(
"credential environment names must be valid identifiers"
)
normalized.append(item)
return tuple(dict.fromkeys(normalized))
def _binding_tuple_from_environment(
values: Mapping[str, str],
name: str,
) -> tuple[str, ...]:
raw = values.get(name, "").strip()
if not raw:
return ()
try:
payload = json.loads(raw)
except json.JSONDecodeError as exc:
raise RuntimeIntegrationError(f"{name} must be a JSON string array") from exc
if not isinstance(payload, list) or not all(
isinstance(item, str) for item in payload
):
raise RuntimeIntegrationError(f"{name} must be a JSON string array")
return tuple(payload)
def _browser_state_root(data_root: Path, module: str, state_key: str) -> Path:
label = state_key.split(":", 1)[1] if ":" in state_key else state_key
stem = Path(label).stem
safe = "".join(character if character.isalnum() else "-" for character in stem)
safe = safe.strip("-")[:48] or "script"
digest = hashlib.sha256(script_id.encode("utf-8")).hexdigest()[:12]
digest = hashlib.sha256(state_key.encode("utf-8")).hexdigest()[:12]
return data_root / "state" / "browser" / module / f"{safe}-{digest}"
@@ -284,17 +733,85 @@ def _configured_hermes_urls(environment: Mapping[str, str]) -> tuple[tuple[str,
return tuple((name, environment[name]) for name in names if environment.get(name, "").strip())
def _validate_local_database(environment: Mapping[str, str]) -> None:
for name in ("PG_HOST", "DB_HOST", "AUTOFLOW_PG_HOST"):
value = environment.get(name, "").strip()
if value and not _is_loopback_host(value):
raise RuntimeIntegrationError(f"local PostgreSQL requires loopback host in {name}")
for name in ("DATABASE_URL", "DB_URL"):
value = environment.get(name, "").strip()
if value:
host = urlparse(value).hostname
if host is None or not _is_loopback_host(host):
raise RuntimeIntegrationError(f"local PostgreSQL requires loopback host in {name}")
def _validate_cloud_database(environment: Mapping[str, str]) -> None:
for name in ("PG_HOST", "DB_HOST", "AUTOFLOW_PG_HOST"):
value = environment.get(name, "").strip()
if value and _is_loopback_host(value):
raise RuntimeIntegrationError(f"cloud PostgreSQL cannot use loopback host in {name}")
raise RuntimeIntegrationError(
f"cloud PostgreSQL requires a remote host in {name}"
)
for name in ("DATABASE_URL", "DB_URL"):
value = environment.get(name, "").strip()
if value:
host = urlparse(value).hostname
if host is None or _is_loopback_host(host):
raise RuntimeIntegrationError(f"cloud PostgreSQL requires a remote host in {name}")
raise RuntimeIntegrationError(
f"cloud PostgreSQL requires a remote host in {name}"
)
def _fill_local_database_defaults(
environment: dict[str, str],
*,
host: str,
port: int,
database: str,
user: str,
) -> None:
defaults = (
(("PG_HOST", "DB_HOST", "AUTOFLOW_PG_HOST"), host),
(("PG_PORT", "DB_PORT", "AUTOFLOW_PG_PORT"), str(port)),
(("PG_DB", "DB_NAME", "AUTOFLOW_PG_DB"), database),
(("PG_USER", "DB_USER", "AUTOFLOW_PG_USER"), user),
)
for aliases, value in defaults:
for name in aliases:
_set_if_blank(environment, name, value)
def _apply_canonical_secrets(environment: dict[str, str]) -> None:
dsn = environment.get("GYXX_POSTGRES_DSN", "").strip()
if dsn:
parsed = urlparse(dsn)
if parsed.scheme not in {"postgres", "postgresql"} or not parsed.hostname:
raise RuntimeIntegrationError("GYXX_POSTGRES_DSN must be a PostgreSQL URL")
_set_if_blank(environment, "DATABASE_URL", dsn)
_set_if_blank(environment, "PG_HOST", parsed.hostname)
_set_if_blank(environment, "PG_PORT", str(parsed.port or 5432))
_set_if_blank(environment, "PG_DB", parsed.path.lstrip("/"))
if parsed.username:
_set_if_blank(environment, "PG_USER", parsed.username)
if parsed.password:
_set_if_blank(environment, "PG_PASSWORD", parsed.password)
postgres_credential = environment.get("GYXX_POSTGRES_PASSWORD", "").strip()
if postgres_credential:
for name in ("PG_PASSWORD", "DB_PASSWORD", "AUTOFLOW_PG_PASSWORD"):
_set_if_blank(environment, name, postgres_credential)
hermes_key = environment.get("GYXX_HERMES_API_KEY", "").strip()
if hermes_key:
_set_if_blank(environment, "HERMES_API_KEY", hermes_key)
_set_if_blank(environment, "HERMES_ANALYZER_TOKEN", hermes_key)
_set_if_blank(environment, "GYXX_SUPPLY_HERMES_TOKEN", hermes_key)
def _set_if_blank(environment: dict[str, str], name: str, value: str) -> None:
if not environment.get(name, "").strip():
environment[name] = value
def _fill_database_aliases(environment: dict[str, str]) -> None:
src/gyxx_flow/adapters/native.py
+57 -7
View File
@@ -5,11 +5,12 @@ from __future__ import annotations
import os
import subprocess
import sys
from dataclasses import dataclass
from dataclasses import dataclass, replace
from pathlib import Path, PurePosixPath
from types import MappingProxyType
from typing import Callable, Mapping
from gyxx_flow.adapters.acceptance_policy import current_acceptance_policy
from gyxx_flow.adapters.integration import RuntimeIntegrationCatalog
from gyxx_flow.catalog import WorkflowEntry
from gyxx_flow.core.config import Settings
@@ -116,6 +117,17 @@ class ModuleCommandAdapter:
"GYXX_SHADOW": "true" if context.shadow else "false",
"PYTHONUNBUFFERED": "1",
}
if entry.module == "product_commerce":
product_paths = DataLayout(self._data_root).for_module("product_commerce")
configured_product_config = env.get("GYXX_PRODUCT_CONFIG", "").strip()
original_product_config = env.get("AUTO_FLOW_CONFIG", "").strip()
portable_product_config = env.get("AUTOFLOW_CONFIG_PATH", "").strip()
env["GYXX_PRODUCT_CONFIG"] = (
configured_product_config
or original_product_config
or portable_product_config
or str(product_paths.state_root / "auto-flow-config.json")
)
if entry.module == "supply_chain":
supply_paths = DataLayout(self._data_root).for_module("supply_chain")
env.update(
@@ -127,9 +139,11 @@ class ModuleCommandAdapter:
}
)
if self._integration_catalog is not None:
env = self._integration_catalog.environment_for(
f"{entry.module}:{entry.entry}", env
)
legacy_script_id = f"{entry.module}:{entry.entry}"
binding = self._integration_catalog.binding_for(legacy_script_id)
env = self._integration_catalog.environment_for(binding.command_id, env)
acceptance_policy = current_acceptance_policy(env)
env.update(acceptance_policy.environment())
existing_pythonpath = env.get("PYTHONPATH", "").strip()
env["PYTHONPATH"] = (
f"{root}{os.pathsep}{existing_pythonpath}" if existing_pythonpath else str(root)
@@ -142,10 +156,10 @@ ModuleCommandFactory = Callable[[WorkflowEntry, RunContext], ExecutableStep]
def _command_from_project(entry: WorkflowEntry, context: RunContext) -> ExecutableStep:
package_root = Path(__file__).resolve().parents[1]
runtime_root = package_root / "modules" / entry.module / "runtime"
module_root = package_root / "modules" / entry.module
settings = Settings.from_env()
return ModuleCommandAdapter(
ModuleSourceRoots({entry.module: runtime_root}),
ModuleSourceRoots({entry.module: module_root}),
project_root=settings.project_root,
data_root=settings.data_root,
).build(entry, context=context)
@@ -157,10 +171,16 @@ class DeferredModuleCommandStep:
entry: WorkflowEntry
command_factory: ModuleCommandFactory = _command_from_project
command_entry: str | None = None
command_args: tuple[str, ...] | None = None
def __post_init__(self) -> None:
if not callable(self.command_factory):
raise TypeError("command_factory must be callable")
if self.command_entry is not None and not self.command_entry:
raise ValueError("command_entry must be non-empty when provided")
if self.command_args is not None:
object.__setattr__(self, "command_args", tuple(self.command_args))
def execute(
self,
@@ -171,7 +191,19 @@ class DeferredModuleCommandStep:
) -> StepExecution:
if dry_run:
return StepExecution(exit_code=0, skipped=True, reason="dry-run")
command = self.command_factory(self.entry, context)
resolved_entry = self.entry
if self.command_entry is not None or self.command_args is not None:
configured_args = (
self.entry.args
if self.command_args is None
else self.command_args
)
resolved_entry = replace(
self.entry,
entry=self.command_entry or self.entry.entry,
args=_render_command_args(configured_args, context),
)
command = self.command_factory(resolved_entry, context)
if not callable(getattr(command, "execute", None)):
raise TypeError("command_factory must return an executable step")
return command.execute(
@@ -181,6 +213,24 @@ class DeferredModuleCommandStep:
)
def _render_command_args(
arguments: tuple[str, ...],
context: RunContext,
) -> tuple[str, ...]:
replacements = {
"{business_date}": context.business_date.isoformat(),
"{run_id}": context.run_id,
"{workflow_id}": context.workflow_id,
}
rendered = []
for argument in arguments:
value = argument
for marker, replacement in replacements.items():
value = value.replace(marker, replacement)
rendered.append(value)
return tuple(rendered)
def _resolve_entry(root: Path, configured_entry: str) -> Path:
if not configured_entry or "\\" in configured_entry or ":" in configured_entry:
raise ModuleSourceError("module entry must use a safe relative path")
src/gyxx_flow/catalog.py
+310 -18
View File
@@ -3,13 +3,15 @@
from __future__ import annotations
import json
import math
import re
from dataclasses import dataclass
from datetime import time
from datetime import date
from pathlib import Path, PurePosixPath
from typing import Any, Literal
_WORKFLOW_ID = re.compile(r"^[a-z][a-z0-9]*(?:[._-][a-z0-9]+)*$")
_SCHEDULE_TIME = re.compile(r"^(?:[01]\d|2[0-3]):[0-5]\d$")
_MODULES = {
"content_marketing",
"product_commerce",
@@ -25,6 +27,35 @@ class CatalogError(ValueError):
"""Raised for an invalid workflow catalog."""
@dataclass(frozen=True, slots=True)
class WorkflowDataEndpoint:
label: str
system: str | None = None
detail: str | None = None
condition: str | None = None
@dataclass(frozen=True, slots=True)
class WorkflowDataFlow:
sources: tuple[WorkflowDataEndpoint, ...]
processing: tuple[str, ...]
destinations: tuple[WorkflowDataEndpoint, ...]
@dataclass(frozen=True, slots=True)
class WorkflowStepEntry:
step_id: str
entry: str
name: str | None = None
description: str | None = None
args: tuple[str, ...] = ()
depends_on: tuple[str, ...] = ()
run_after_failure: bool = False
timeout_seconds: float | None = None
replay_policy: Literal["guarded", "idempotent"] | None = None
data_flow: WorkflowDataFlow | None = None
@dataclass(frozen=True, slots=True)
class WorkflowEntry:
workflow_id: str
@@ -35,6 +66,7 @@ class WorkflowEntry:
source_project: str | None = None
source_task_name: str | None = None
note: str | None = None
steps: tuple[WorkflowStepEntry, ...] = ()
@dataclass(frozen=True, slots=True)
@@ -46,6 +78,14 @@ class ScheduleEntry:
day_of_month: int | None = None
every_days: int | None = None
anchor_date: str | None = None
enabled: bool = True
business_date_offset_days: int = 0
at_times: tuple[str, ...] = ()
@property
def effective_times(self) -> tuple[str, ...]:
"""Return every configured wall-clock time, including legacy ``at``."""
return self.at_times or (self.at,)
@dataclass(frozen=True, slots=True)
@@ -59,7 +99,7 @@ class WorkflowCatalog:
root = Path(config_dir)
workflow_payload = _load_json(root / "workflows.json")
schedule_payload = _load_json(root / "schedules.json")
_require_schema(workflow_payload, "workflows.json", expected=2)
_require_schema(workflow_payload, "workflows.json", expected=3)
_require_schema(schedule_payload, "schedules.json", expected=1)
workflows = tuple(_parse_workflow(item) for item in workflow_payload.get("workflows", []))
schedules = tuple(_parse_schedule(item) for item in schedule_payload.get("schedules", []))
@@ -144,18 +184,27 @@ def _parse_workflow(item: Any) -> WorkflowEntry:
raise CatalogError(f"missing execution definition for {workflow_id}")
if not isinstance(provenance, dict):
raise CatalogError(f"invalid provenance definition for {workflow_id}")
entry = execution.get("entry")
if not _is_relative_entry(entry):
raise CatalogError(f"execution entry must be relative for {workflow_id}")
raw_steps = execution.get("steps")
if raw_steps is None:
entry = execution.get("entry")
if not _is_relative_entry(entry):
raise CatalogError(f"execution entry must be relative for {workflow_id}")
args = _parse_args(execution.get("args", []), workflow_id)
steps: tuple[WorkflowStepEntry, ...] = ()
else:
if "entry" in execution or "args" in execution:
raise CatalogError(
f"execution cannot mix entry and steps for {workflow_id}"
)
steps = _parse_workflow_steps(raw_steps, workflow_id)
entry = steps[0].entry
args = steps[0].args
project = provenance.get("source_project")
if project is not None and (not isinstance(project, str) or not project):
raise CatalogError(f"invalid source project for {workflow_id}")
task_name = provenance.get("task_name")
if trigger == "scheduled" and (not isinstance(task_name, str) or not task_name):
raise CatalogError(f"scheduled workflow requires source task name: {workflow_id}")
args = execution.get("args", [])
if not isinstance(args, list) or not all(isinstance(value, str) for value in args):
raise CatalogError(f"execution args must be strings for {workflow_id}")
return WorkflowEntry(
workflow_id=workflow_id,
module=module,
@@ -165,9 +214,195 @@ def _parse_workflow(item: Any) -> WorkflowEntry:
source_project=project,
source_task_name=task_name,
note=item.get("note"),
steps=steps,
)
def _parse_workflow_steps(
raw_steps: Any,
workflow_id: str,
) -> tuple[WorkflowStepEntry, ...]:
if not isinstance(raw_steps, list) or not raw_steps:
raise CatalogError(f"execution steps must be a non-empty list for {workflow_id}")
steps: list[WorkflowStepEntry] = []
for raw_step in raw_steps:
if not isinstance(raw_step, dict):
raise CatalogError(f"workflow step must be an object for {workflow_id}")
step_id = raw_step.get("id")
entry = raw_step.get("entry")
if not isinstance(step_id, str) or not _WORKFLOW_ID.fullmatch(step_id):
raise CatalogError(f"invalid workflow step id for {workflow_id}: {step_id!r}")
if not _is_relative_entry(entry):
raise CatalogError(
f"workflow step entry must be relative for {workflow_id}.{step_id}"
)
depends_on = raw_step.get("depends_on", [])
if not isinstance(depends_on, list) or not all(
isinstance(value, str) and _WORKFLOW_ID.fullmatch(value)
for value in depends_on
):
raise CatalogError(
f"workflow step dependencies are invalid for {workflow_id}.{step_id}"
)
run_after_failure = raw_step.get("run_after_failure", False)
if not isinstance(run_after_failure, bool):
raise CatalogError(
f"workflow step run_after_failure is invalid for {workflow_id}.{step_id}"
)
timeout_seconds = raw_step.get("timeout_seconds")
if timeout_seconds is not None and (
isinstance(timeout_seconds, bool)
or not isinstance(timeout_seconds, (int, float))
or not math.isfinite(timeout_seconds)
or timeout_seconds <= 0
):
raise CatalogError(
f"workflow step timeout_seconds is invalid for {workflow_id}.{step_id}"
)
replay_policy = raw_step.get("replay_policy")
if replay_policy is not None and replay_policy not in {
"guarded",
"idempotent",
}:
raise CatalogError(
f"workflow step replay_policy is invalid for {workflow_id}.{step_id}"
)
name = raw_step.get("name")
if name is not None and (not isinstance(name, str) or not name.strip()):
raise CatalogError(
f"workflow step name is invalid for {workflow_id}.{step_id}"
)
description = raw_step.get("description")
if description is not None and (
not isinstance(description, str) or not description.strip()
):
raise CatalogError(
f"workflow step description is invalid for {workflow_id}.{step_id}"
)
steps.append(
WorkflowStepEntry(
step_id=step_id,
entry=entry,
name=name.strip() if name is not None else None,
description=description.strip() if description is not None else None,
args=_parse_args(raw_step.get("args", []), workflow_id),
depends_on=tuple(depends_on),
run_after_failure=run_after_failure,
timeout_seconds=(
float(timeout_seconds) if timeout_seconds is not None else None
),
replay_policy=replay_policy,
data_flow=_parse_step_data_flow(
raw_step.get("data_flow"),
workflow_id,
step_id,
),
)
)
step_ids = [step.step_id for step in steps]
duplicates = _duplicates(step_ids)
if duplicates:
raise CatalogError(
f"duplicate workflow step id for {workflow_id}: {sorted(duplicates)[0]}"
)
known = set(step_ids)
for step in steps:
unknown = set(step.depends_on) - known
if unknown:
raise CatalogError(
f"unknown workflow step dependency for {workflow_id}.{step.step_id}: "
f"{sorted(unknown)[0]}"
)
_validate_step_graph(steps, workflow_id)
return tuple(steps)
def _parse_step_data_flow(
value: Any,
workflow_id: str,
step_id: str,
) -> WorkflowDataFlow | None:
if value is None:
return None
label = f"{workflow_id}.{step_id}"
if not isinstance(value, dict):
raise CatalogError(f"workflow step data_flow must be an object for {label}")
allowed = {"sources", "processing", "destinations"}
if set(value) != allowed:
raise CatalogError(f"workflow step data_flow fields are invalid for {label}")
processing = value.get("processing")
if not isinstance(processing, list) or not processing or not all(
isinstance(item, str) and item.strip() for item in processing
):
raise CatalogError(f"workflow step processing is invalid for {label}")
return WorkflowDataFlow(
sources=_parse_data_endpoints(value.get("sources"), label, "sources"),
processing=tuple(item.strip() for item in processing),
destinations=_parse_data_endpoints(
value.get("destinations"),
label,
"destinations",
),
)
def _parse_data_endpoints(
value: Any,
step_label: str,
field: str,
) -> tuple[WorkflowDataEndpoint, ...]:
if not isinstance(value, list) or not value:
raise CatalogError(f"workflow step {field} is invalid for {step_label}")
endpoints: list[WorkflowDataEndpoint] = []
allowed = {"label", "system", "detail", "condition"}
for item in value:
if not isinstance(item, dict) or set(item) - allowed:
raise CatalogError(f"workflow step {field} is invalid for {step_label}")
endpoint_label = item.get("label")
if not isinstance(endpoint_label, str) or not endpoint_label.strip():
raise CatalogError(f"workflow step {field} is invalid for {step_label}")
optional: dict[str, str | None] = {}
for key in ("system", "detail", "condition"):
raw = item.get(key)
if raw is not None and (not isinstance(raw, str) or not raw.strip()):
raise CatalogError(f"workflow step {field} is invalid for {step_label}")
optional[key] = raw.strip() if raw is not None else None
endpoints.append(
WorkflowDataEndpoint(
label=endpoint_label.strip(),
system=optional["system"],
detail=optional["detail"],
condition=optional["condition"],
)
)
return tuple(endpoints)
def _parse_args(value: Any, workflow_id: str) -> tuple[str, ...]:
if not isinstance(value, list) or not all(isinstance(item, str) for item in value):
raise CatalogError(f"execution args must be strings for {workflow_id}")
return tuple(value)
def _validate_step_graph(
steps: list[WorkflowStepEntry],
workflow_id: str,
) -> None:
unresolved = {step.step_id: set(step.depends_on) for step in steps}
completed: set[str] = set()
while unresolved:
ready = [
step_id
for step_id, dependencies in unresolved.items()
if dependencies <= completed
]
if not ready:
raise CatalogError(f"workflow step dependency cycle for {workflow_id}")
completed.update(ready)
for step_id in ready:
del unresolved[step_id]
def _is_relative_entry(entry: Any) -> bool:
if not isinstance(entry, str) or not entry or "\\" in entry or ":" in entry:
return False
@@ -180,24 +415,78 @@ def _parse_schedule(item: Any) -> ScheduleEntry:
raise CatalogError("schedule entry must be an object")
workflow_id = item.get("workflow_id")
kind = item.get("kind")
at = item.get("at")
raw_at = item.get("at")
if not isinstance(workflow_id, str) or not _WORKFLOW_ID.fullmatch(workflow_id):
raise CatalogError(f"invalid schedule workflow id: {workflow_id!r}")
if kind not in _KINDS:
raise CatalogError(f"invalid schedule kind for {workflow_id}: {kind!r}")
try:
time.fromisoformat(at)
except (TypeError, ValueError) as exc:
raise CatalogError(f"invalid schedule time for {workflow_id}: {at!r}") from exc
days = tuple(item.get("days", []))
if kind == "weekly" and (not days or not set(days) <= _WEEKDAYS):
if isinstance(raw_at, str):
if not _SCHEDULE_TIME.fullmatch(raw_at):
raise CatalogError(f"invalid schedule time for {workflow_id}: {raw_at!r}")
at = raw_at
at_times: tuple[str, ...] = ()
elif isinstance(raw_at, list):
if (
not raw_at
or not all(
isinstance(value, str) and _SCHEDULE_TIME.fullmatch(value)
for value in raw_at
)
or len(raw_at) != len(set(raw_at))
):
raise CatalogError(f"invalid schedule times for {workflow_id}: {raw_at!r}")
at_times = tuple(raw_at)
at = at_times[0]
else:
raise CatalogError(f"invalid schedule time for {workflow_id}: {raw_at!r}")
raw_days = item.get("days", [])
if not isinstance(raw_days, list) or not all(
isinstance(day, str) for day in raw_days
):
raise CatalogError(f"invalid weekly days for {workflow_id}")
days = tuple(raw_days)
if kind == "weekly" and (
not days
or len(days) != len(set(days))
or not set(days) <= _WEEKDAYS
):
raise CatalogError(f"invalid weekly days for {workflow_id}")
day_of_month = item.get("day_of_month")
if kind == "monthly" and (not isinstance(day_of_month, int) or not 1 <= day_of_month <= 31):
if kind == "monthly" and (
isinstance(day_of_month, bool)
or not isinstance(day_of_month, int)
or not 1 <= day_of_month <= 31
):
raise CatalogError(f"invalid day_of_month for {workflow_id}")
every_days = item.get("every_days")
if kind == "interval_days" and (not isinstance(every_days, int) or every_days < 1):
if kind == "interval_days" and (
isinstance(every_days, bool)
or not isinstance(every_days, int)
or every_days < 1
):
raise CatalogError(f"invalid every_days for {workflow_id}")
anchor_date = item.get("anchor_date")
if kind == "interval_days":
try:
parsed_anchor = date.fromisoformat(anchor_date)
except (TypeError, ValueError) as exc:
raise CatalogError(
f"invalid anchor_date for {workflow_id}: {anchor_date!r}"
) from exc
if parsed_anchor.isoformat() != anchor_date:
raise CatalogError(
f"invalid anchor_date for {workflow_id}: {anchor_date!r}"
)
enabled = item.get("enabled", True)
if not isinstance(enabled, bool):
raise CatalogError(f"invalid enabled flag for {workflow_id}")
business_date_offset_days = item.get("business_date_offset_days", 0)
if (
isinstance(business_date_offset_days, bool)
or not isinstance(business_date_offset_days, int)
or not -31 <= business_date_offset_days <= 31
):
raise CatalogError(f"invalid business date offset for {workflow_id}")
return ScheduleEntry(
workflow_id=workflow_id,
kind=kind,
@@ -205,7 +494,10 @@ def _parse_schedule(item: Any) -> ScheduleEntry:
days=days,
day_of_month=day_of_month,
every_days=every_days,
anchor_date=item.get("anchor_date"),
anchor_date=anchor_date,
enabled=enabled,
business_date_offset_days=business_date_offset_days,
at_times=at_times,
)
src/gyxx_flow/cli.py
+282 -90
View File
@@ -5,13 +5,17 @@ from __future__ import annotations
import argparse
import hashlib
import json
import os
import signal
import sys
import threading
from collections.abc import Sequence
from contextlib import suppress
from datetime import date, datetime, timedelta, timezone
from datetime import date, timedelta
from pathlib import Path
from typing import TextIO
from zoneinfo import ZoneInfo
from dotenv import dotenv_values
from gyxx_flow.acceptance import build_acceptance_report
from gyxx_flow.adapters.native import DeferredModuleCommandStep
@@ -24,15 +28,14 @@ from gyxx_flow.core.locks import LockManager
from gyxx_flow.core.records import RunJournal
from gyxx_flow.diagnostics import run_doctor
from gyxx_flow.modules import create_default_registry as create_module_registry
from gyxx_flow.ops import EffectLedger, RunIndex
from gyxx_flow.scheduler import (
PowerShellCurrentTaskProvider,
SchedulerConfig,
build_schedule_plan,
detect_schedule_drift,
write_schedule_plan_bundle,
from gyxx_flow.ops import EffectLedger, EffectStateAmbiguous, RunIndex
from gyxx_flow.scheduler_service import (
PythonScheduler,
SchedulerInstanceLock,
SubprocessWorkflowLauncher,
)
from gyxx_flow.script_catalog import ScriptCatalog, ScriptCatalogError
from gyxx_flow.source_sync.cli import add_sources_parser, run_sources_command
from gyxx_flow.workflow.engine import WorkflowEngine, WorkflowRunResult
from gyxx_flow.workflow.model import StepDefinition, WorkflowDefinition
from gyxx_flow.workflow.registry import WorkflowRegistry, WorkflowRegistryError
@@ -64,11 +67,13 @@ def build_parser() -> argparse.ArgumentParser:
)
commands = parser.add_subparsers(dest="command", required=True)
list_parser = commands.add_parser("list", help="list catalog workflows")
add_sources_parser(commands)
list_parser = commands.add_parser("list", help="list scheduled workflows")
list_parser.add_argument("--json", action="store_true", help="emit machine-readable JSON")
scripts_parser = commands.add_parser(
"scripts", help="discover and manually run project-owned module scripts"
"scripts", help="list and manually run explicitly registered commands"
)
script_commands = scripts_parser.add_subparsers(dest="scripts_command", required=True)
scripts_list_parser = script_commands.add_parser("list", help="list runnable scripts")
@@ -76,9 +81,9 @@ def build_parser() -> argparse.ArgumentParser:
scripts_list_parser.add_argument("--json", action="store_true")
scripts_run_parser = script_commands.add_parser("run", help="run one local script")
scripts_run_parser.add_argument("script_id")
scripts_run_date = scripts_run_parser.add_mutually_exclusive_group(required=True)
scripts_run_date.add_argument("--date", help="business date (YYYY-MM-DD)")
scripts_run_date.add_argument("--scheduled", action="store_true")
scripts_run_parser.add_argument(
"--date", required=True, help="business date (YYYY-MM-DD)"
)
scripts_run_parser.add_argument("--execute", action="store_true")
scripts_run_parser.add_argument("--shadow", action="store_true")
scripts_run_parser.add_argument(
@@ -89,14 +94,35 @@ def build_parser() -> argparse.ArgumentParser:
help="one argument passed to the script; repeat for multiple arguments",
)
effects_parser = commands.add_parser(
"effects", help="inspect or reconcile guarded production effects"
)
effect_commands = effects_parser.add_subparsers(
dest="effects_command", required=True
)
reconcile_parser = effect_commands.add_parser(
"reconcile", help="resolve one verified ambiguous effect receipt"
)
reconcile_parser.add_argument("workflow_id")
reconcile_parser.add_argument("--date", required=True, dest="business_date")
reconcile_parser.add_argument("--step", required=True, dest="step_id")
reconcile_parser.add_argument("--expected-run-id", required=True)
reconcile_parser.add_argument(
"--action", required=True, choices=("retry", "applied")
)
reconcile_parser.add_argument("--operator", required=True)
reconcile_parser.add_argument("--reason", required=True)
reconcile_parser.add_argument("--evidence", required=True)
reconcile_parser.add_argument(
"--execute",
action="store_true",
help="confirm the audited receipt mutation",
)
run_parser = commands.add_parser("run", help="run one workflow business date")
_add_execution_arguments(run_parser)
run_date = run_parser.add_mutually_exclusive_group(required=True)
run_date.add_argument("--date", help="business date (YYYY-MM-DD)")
run_date.add_argument(
"--scheduled",
action="store_true",
help="execute for today's Asia/Shanghai business date",
run_parser.add_argument(
"--date", required=True, help="business date (YYYY-MM-DD)"
)
backfill_parser = commands.add_parser(
@@ -110,16 +136,42 @@ def build_parser() -> argparse.ArgumentParser:
"--to", dest="to_date", required=True, help="last business date"
)
schedule_parser = commands.add_parser("schedule", help="plan managed schedules")
schedule_parser = commands.add_parser(
"schedule", help="run the project-owned resident scheduler"
)
schedule_commands = schedule_parser.add_subparsers(
dest="schedule_command", required=True
)
plan_parser = schedule_commands.add_parser(
"plan", help="write an inert Task Scheduler bundle and drift report"
run_schedule_parser = schedule_commands.add_parser(
"run", help="run the cross-platform resident Python scheduler"
)
run_schedule_parser.add_argument("--python-executable", type=Path)
run_schedule_parser.add_argument("--poll-seconds", type=float, default=15.0)
run_schedule_parser.add_argument("--misfire-grace-seconds", type=int, default=21600)
run_schedule_parser.add_argument("--shutdown-timeout-seconds", type=float, default=300.0)
run_schedule_parser.add_argument("--once", action="store_true")
run_schedule_parser.add_argument("--dry-run", action="store_true")
run_schedule_parser.add_argument(
"--env-file",
action="append",
default=[],
type=Path,
help="explicit runtime environment file; repeatable, existing process values win",
)
schedule_commands.add_parser("status", help="show persisted Python scheduler state")
console_parser = commands.add_parser(
"console", help="serve the workflow operator web console"
)
console_parser.add_argument("--host", default="127.0.0.1")
console_parser.add_argument("--port", type=int, default=8765)
console_parser.add_argument(
"--env-file",
action="append",
default=[],
type=Path,
help="explicit runtime environment file; repeatable, existing process values win",
)
plan_parser.add_argument("--output", type=Path)
plan_parser.add_argument("--start-date", help="trigger boundary date (YYYY-MM-DD)")
plan_parser.add_argument("--python-executable", type=Path, required=True)
doctor_parser = commands.add_parser("doctor", help="run environment preflight checks")
doctor_parser.add_argument("--json", action="store_true")
@@ -167,7 +219,6 @@ def main(
settings: Settings | None = None,
stdout: TextIO | None = None,
stderr: TextIO | None = None,
current_task_provider: PowerShellCurrentTaskProvider | None = None,
) -> int:
"""Execute a CLI command and return a stable process exit code."""
@@ -179,10 +230,19 @@ def main(
except SystemExit as exc:
return int(exc.code)
resolved_settings = settings or Settings.from_env()
try:
_load_runtime_environment_files(arguments)
resolved_settings = settings or Settings.from_env()
if arguments.command == "doctor":
return _doctor_command(arguments, resolved_settings, output)
if arguments.command == "console":
return _console_command(arguments, resolved_settings, output)
if arguments.command == "sources":
return run_sources_command(
arguments,
project_root=resolved_settings.project_root,
output=output,
)
if (
arguments.command == "acceptance"
and arguments.acceptance_command == "status"
@@ -192,6 +252,13 @@ def main(
return _list_scripts(arguments, output)
if arguments.command == "scripts" and arguments.scripts_command == "run":
return _run_script(arguments, resolved_settings, output)
if (
arguments.command == "effects"
and arguments.effects_command == "reconcile"
):
return _effects_reconcile_command(
arguments, resolved_settings, output
)
resolved_registry = registry or build_default_registry(resolved_settings)
if arguments.command == "list":
return _list_workflows(arguments, resolved_registry, output)
@@ -199,14 +266,10 @@ def main(
return _run_command(arguments, resolved_registry, resolved_settings, output)
if arguments.command == "backfill":
return _backfill_command(arguments, resolved_registry, resolved_settings, output)
if arguments.command == "schedule" and arguments.schedule_command == "plan":
return _schedule_plan_command(
arguments,
resolved_registry.catalog,
resolved_settings,
output,
current_task_provider=current_task_provider,
)
if arguments.command == "schedule" and arguments.schedule_command == "run":
return _schedule_run_command(arguments, resolved_registry.catalog, resolved_settings, output)
if arguments.command == "schedule" and arguments.schedule_command == "status":
return _schedule_status_command(resolved_registry.catalog, resolved_settings, output)
raise CliConfigurationError(f"unsupported command: {arguments.command}")
except (
CatalogError,
@@ -222,6 +285,26 @@ def main(
return EXIT_RUNTIME
def _load_runtime_environment_files(arguments: argparse.Namespace) -> None:
"""Load only explicitly named service environment files without leaking values."""
raw_paths = getattr(arguments, "env_file", ()) or ()
for raw_path in raw_paths:
path = Path(raw_path).expanduser().resolve()
if not path.is_file():
raise CliConfigurationError(f"runtime environment file not found: {path}")
try:
values = dotenv_values(path, encoding="utf-8")
except (OSError, UnicodeError) as exc:
raise CliConfigurationError(
f"cannot read runtime environment file: {path}"
) from exc
for key, value in values.items():
if not isinstance(key, str) or not key or value is None:
continue
os.environ.setdefault(key, value)
def _doctor_command(
arguments: argparse.Namespace, settings: Settings, output: TextIO
) -> int:
@@ -235,6 +318,48 @@ def _doctor_command(
return EXIT_SUCCESS if report.is_healthy else EXIT_RUNTIME
def _effects_reconcile_command(
arguments: argparse.Namespace,
settings: Settings,
output: TextIO,
) -> int:
if not arguments.execute:
raise CliConfigurationError("effect reconciliation requires --execute")
try:
result = EffectLedger(settings.data_root).reconcile(
workflow_id=arguments.workflow_id,
business_date=arguments.business_date,
step_id=arguments.step_id,
expected_run_id=arguments.expected_run_id,
action=arguments.action,
operator=arguments.operator,
reason=arguments.reason,
evidence=arguments.evidence,
)
except EffectStateAmbiguous as exc:
raise CliConfigurationError(str(exc)) from exc
_write_json(output, result)
return EXIT_SUCCESS
def _console_command(
arguments: argparse.Namespace,
settings: Settings,
output: TextIO,
) -> int:
from gyxx_flow.console import serve_console
try:
return serve_console(
settings,
host=arguments.host,
port=arguments.port,
output=output,
)
except OSError as exc:
raise CliRuntimeError("cannot start workflow console") from exc
def _acceptance_status_command(
arguments: argparse.Namespace, settings: Settings, output: TextIO
) -> int:
@@ -264,14 +389,19 @@ def _list_workflows(
"module": entry.module,
"registered": registry.is_registered(entry.workflow_id),
"trigger": entry.trigger,
"enabled": registry.catalog.schedule_for(entry.workflow_id).enabled,
}
for entry in registry.catalog.workflows
for entry in registry.catalog.scheduled_workflows()
]
if arguments.json:
_write_json(output, rows)
else:
for row in rows:
state = "ready" if row["registered"] else "not-registered"
state = (
"not-registered"
if not row["registered"]
else "ready" if row["enabled"] else "disabled"
)
output.write(
f"{row['id']}\t{row['module']}\t{row['trigger']}\t{state}\n"
)
@@ -279,13 +409,16 @@ def _list_workflows(
def _list_scripts(arguments: argparse.Namespace, output: TextIO) -> int:
catalog = ScriptCatalog.discover_default()
catalog = ScriptCatalog.load_default()
rows = [
{
"id": script.script_id,
"id": script.command_id,
"command_id": script.command_id,
"legacy_script_id": script.legacy_script_id,
"module": script.module,
"entry": script.entry,
"kind": script.kind,
"default_args": list(script.default_args),
}
for script in catalog.scripts
if arguments.module is None or script.module == arguments.module
@@ -305,10 +438,19 @@ def _run_script(
settings: Settings,
output: TextIO,
) -> int:
script = ScriptCatalog.discover_default().get(arguments.script_id)
suffix = hashlib.sha256(script.script_id.encode("utf-8")).hexdigest()[:12]
script = ScriptCatalog.load_default().get(arguments.script_id)
business_date = _parse_date(arguments.date)
script_args = (
*_render_script_default_args(script.default_args, business_date),
*arguments.script_args,
)
invocation_identity = json.dumps(
[script.command_id, *script_args],
ensure_ascii=False,
separators=(",", ":"),
)
suffix = hashlib.sha256(invocation_identity.encode("utf-8")).hexdigest()[:12]
workflow_id = f"script.{script.module}.{suffix}"
script_args = tuple(arguments.script_args)
entry = WorkflowEntry(
workflow_id=workflow_id,
module=script.module,
@@ -329,19 +471,28 @@ def _run_script(
),
),
)
business_date = _today_shanghai() if arguments.scheduled else _parse_date(arguments.date)
exit_code, payload = _execute_once(
workflow,
business_date=business_date,
shadow=arguments.shadow,
dry_run=not (arguments.execute or arguments.scheduled),
dry_run=not arguments.execute,
settings=settings,
)
payload["script_id"] = script.script_id
payload["command_id"] = script.command_id
payload["script_id"] = script.legacy_script_id
_write_json(output, payload)
return exit_code
def _render_script_default_args(
arguments: tuple[str, ...], business_date: date
) -> tuple[str, ...]:
return tuple(
argument.replace("{business_date}", business_date.isoformat())
for argument in arguments
)
def _run_command(
arguments: argparse.Namespace,
registry: WorkflowRegistry,
@@ -350,8 +501,8 @@ def _run_command(
) -> int:
registered = registry.resolve(arguments.workflow_id)
workflow = _select_workflow(registered.definition, arguments)
business_date = _today_shanghai() if arguments.scheduled else _parse_date(arguments.date)
dry_run = not (arguments.execute or arguments.scheduled)
business_date = _parse_date(arguments.date)
dry_run = not arguments.execute
exit_code, payload = _execute_once(
workflow,
business_date=business_date,
@@ -363,51 +514,91 @@ def _run_command(
return exit_code
def _schedule_plan_command(
def _python_scheduler(
catalog: WorkflowCatalog,
settings: Settings,
*,
python_executable: Path | None = None,
misfire_grace_seconds: int = 21600,
dry_run: bool = False,
) -> PythonScheduler:
launcher = SubprocessWorkflowLauncher(
settings.project_root, settings.data_root, python_executable=python_executable
)
return PythonScheduler(
catalog,
settings.data_root,
launcher=launcher,
misfire_grace_seconds=misfire_grace_seconds,
dry_run=dry_run,
catalog_loader=lambda: WorkflowCatalog.load(
settings.project_root / "config"
),
)
def _schedule_run_command(
arguments: argparse.Namespace,
catalog: WorkflowCatalog,
settings: Settings,
output: TextIO,
*,
current_task_provider: PowerShellCurrentTaskProvider | None,
) -> int:
start_date = (
_parse_date(arguments.start_date)
if arguments.start_date
else _today_shanghai()
)
destination = arguments.output
if destination is None:
timestamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
destination = settings.data_root / "evidence" / "schedule-plans" / timestamp
config = SchedulerConfig(
if arguments.poll_seconds <= 0:
raise CliConfigurationError("poll seconds must be positive")
if arguments.shutdown_timeout_seconds < 0:
raise CliConfigurationError("shutdown timeout must be non-negative")
scheduler = _python_scheduler(
catalog,
settings,
python_executable=arguments.python_executable,
project_root=settings.project_root,
misfire_grace_seconds=arguments.misfire_grace_seconds,
dry_run=arguments.dry_run,
)
plan = build_schedule_plan(catalog, config, start_date=start_date)
lock_path = settings.data_root / "state" / "scheduler" / "service.lock"
try:
bundle = write_schedule_plan_bundle(destination, plan)
except FileExistsError as exc:
raise CliConfigurationError(
f"schedule plan destination already exists: {destination}"
) from exc
provider = current_task_provider or PowerShellCurrentTaskProvider()
try:
current = provider.current_tasks(config.task_path)
except Exception as exc:
raise CliRuntimeError("cannot read current managed scheduled tasks") from exc
drift = detect_schedule_drift(plan, current)
atomic_write_json(bundle / "drift.json", drift.as_dict())
_write_json(
output,
{
"path": str(bundle),
"desired_count": len(plan.tasks),
"drift_count": len(drift.items),
"is_clean": drift.is_clean,
"applied": False,
},
)
with SchedulerInstanceLock(lock_path):
first = scheduler.tick()
if arguments.once or arguments.dry_run:
if not arguments.dry_run:
scheduler.wait_for_active(arguments.shutdown_timeout_seconds)
_write_json(output, {
"mode": "dry-run" if arguments.dry_run else "once",
"due_or_started": first,
"state": scheduler.status(),
})
return EXIT_SUCCESS
stop = threading.Event()
previous_handlers: dict[int, object] = {}
def request_stop(_signum, _frame) -> None: # type: ignore[no-untyped-def]
stop.set()
for signum in (signal.SIGINT, signal.SIGTERM):
previous_handlers[signum] = signal.getsignal(signum)
signal.signal(signum, request_stop)
try:
while not stop.wait(arguments.poll_seconds):
scheduler.tick()
finally:
for signum, handler in previous_handlers.items():
signal.signal(signum, handler)
scheduler.wait_for_active(arguments.shutdown_timeout_seconds)
_write_json(output, {"mode": "service", "state": scheduler.status()})
return EXIT_SUCCESS
except RuntimeError as exc:
raise CliRuntimeError(str(exc)) from exc
def _schedule_status_command(
catalog: WorkflowCatalog, settings: Settings, output: TextIO
) -> int:
scheduler = _python_scheduler(catalog, settings)
_write_json(output, {
"timezone": catalog.timezone,
"scheduled_count": len(catalog.scheduled_workflows()),
"state": scheduler.status(),
})
return EXIT_SUCCESS
@@ -481,7 +672,12 @@ def _execute_once(
layout = DataLayout(settings.data_root)
journal: RunJournal | None = None
try:
journal = RunJournal.create(layout, context)
journal = RunJournal.create(
layout,
context,
mode="dry_run" if dry_run else "execute",
)
RunIndex(settings.data_root).index_journal(journal)
result = WorkflowEngine(
LockManager(settings.data_root / "state" / "locks"),
effect_ledger=EffectLedger(settings.data_root),
@@ -536,10 +732,6 @@ def _parse_date(value: str) -> date:
return parsed
def _today_shanghai() -> date:
return datetime.now(ZoneInfo("Asia/Shanghai")).date()
def _write_json(output: TextIO, payload: object) -> None:
json.dump(payload, output, ensure_ascii=False, sort_keys=True)
output.write("\n")
src/gyxx_flow/console.py
+2086
View File
File diff suppressed because it is too large Load Diff
src/gyxx_flow/core/exit_codes.py
+5
View File
@@ -0,0 +1,5 @@
"""Reserved process exit codes shared by framework and runtime adapters."""
COOKIE_SKIP_EXIT_CODE = 75
__all__ = ["COOKIE_SKIP_EXIT_CODE"]
src/gyxx_flow/core/locks.py
+304 -17
View File
@@ -2,23 +2,193 @@
from __future__ import annotations
import ctypes
import hashlib
import json
import os
import re
import time
import uuid
from contextlib import contextmanager
from dataclasses import dataclass
from datetime import datetime, timezone
from pathlib import Path
import psutil
_RESOURCE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9:._-]*$")
_OWNER = re.compile(r"^[A-Za-z0-9][A-Za-z0-9:._-]*$")
_PARTIAL_PID = re.compile(r'["\']pid["\']\s*:\s*(?P<pid>\d+)')
_PARTIAL_PROCESS_STARTED_AT = re.compile(
r'["\']process_started_at["\']\s*:\s*(?P<started>\d+(?:\.\d+)?)'
)
_PROCESS_QUERY_LIMITED_INFORMATION = 0x1000
_STILL_ACTIVE = 259
_ERROR_INVALID_PARAMETER = 87
_MALFORMED_LOCK_GRACE_SECONDS = 60.0
_PROCESS_START_TOLERANCE_SECONDS = 1.0
_RELEASE_GUARD_TIMEOUT_SECONDS = 30.0
class ResourceBusyError(RuntimeError):
"""Raised when a named resource cannot be acquired before its deadline."""
def _process_is_running(pid: int) -> bool | None:
"""Return process liveness, or ``None`` when it cannot be proven safely."""
if pid <= 0:
return None
if os.name == "nt":
from ctypes import wintypes
kernel32 = ctypes.WinDLL("kernel32", use_last_error=True)
kernel32.OpenProcess.argtypes = (wintypes.DWORD, wintypes.BOOL, wintypes.DWORD)
kernel32.OpenProcess.restype = wintypes.HANDLE
kernel32.GetExitCodeProcess.argtypes = (
wintypes.HANDLE,
ctypes.POINTER(wintypes.DWORD),
)
kernel32.GetExitCodeProcess.restype = wintypes.BOOL
kernel32.CloseHandle.argtypes = (wintypes.HANDLE,)
kernel32.CloseHandle.restype = wintypes.BOOL
handle = kernel32.OpenProcess(
_PROCESS_QUERY_LIMITED_INFORMATION,
False,
pid,
)
if not handle:
return (
False
if ctypes.get_last_error() == _ERROR_INVALID_PARAMETER
else None
)
try:
exit_code = wintypes.DWORD()
if not kernel32.GetExitCodeProcess(handle, ctypes.byref(exit_code)):
return None
return exit_code.value == _STILL_ACTIVE
finally:
kernel32.CloseHandle(handle)
try:
os.kill(pid, 0)
except ProcessLookupError:
return False
except (PermissionError, OSError):
return None
return True
def _process_started_at(pid: int) -> float | None:
try:
return psutil.Process(pid).create_time()
except (psutil.Error, OSError):
return None
def _owner_is_provably_stale(
pid: int,
process_started_at: float | None,
) -> bool:
running = _process_is_running(pid)
if running is False:
return True
if running is not True or process_started_at is None:
return False
actual_started_at = _process_started_at(pid)
if actual_started_at is None:
return False
return (
abs(actual_started_at - process_started_at)
> _PROCESS_START_TOLERANCE_SECONDS
)
def _coerce_process_started_at(value: object) -> float | None:
if isinstance(value, bool) or not isinstance(value, (int, float)):
return None
started_at = float(value)
return started_at if started_at > 0 else None
def _partial_pid(text: str) -> int | None:
match = _PARTIAL_PID.search(text)
if match is None:
return None
pid = int(match.group("pid"))
return pid if pid > 0 else None
def _partial_process_started_at(text: str) -> float | None:
match = _PARTIAL_PROCESS_STARTED_AT.search(text)
if match is None:
return None
return _coerce_process_started_at(float(match.group("started")))
def _file_identity(stat: os.stat_result) -> tuple[int, int, int, int]:
return (stat.st_dev, stat.st_ino, stat.st_size, stat.st_mtime_ns)
def _try_lock_file(handle) -> bool: # type: ignore[no-untyped-def]
if os.name == "nt":
import msvcrt
handle.seek(0, os.SEEK_END)
if handle.tell() == 0:
handle.write(b"\0")
handle.flush()
handle.seek(0)
try:
msvcrt.locking(handle.fileno(), msvcrt.LK_NBLCK, 1)
except OSError:
return False
return True
import fcntl
try:
fcntl.flock(handle.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)
except OSError:
return False
return True
def _unlock_file(handle) -> None: # type: ignore[no-untyped-def]
if os.name == "nt":
import msvcrt
handle.seek(0)
msvcrt.locking(handle.fileno(), msvcrt.LK_UNLCK, 1)
return
import fcntl
fcntl.flock(handle.fileno(), fcntl.LOCK_UN)
@contextmanager
def _exclusive_reclaim_guard(
path: Path,
*,
resource: str,
deadline: float,
poll_seconds: float,
):
"""Use an OS lock so a crashed reclaimer cannot strand the guard."""
path.parent.mkdir(parents=True, exist_ok=True)
with path.open("a+b") as handle:
while not _try_lock_file(handle):
if time.monotonic() >= deadline:
raise ResourceBusyError(f"resource is busy: {resource}")
time.sleep(poll_seconds)
try:
yield
finally:
_unlock_file(handle)
@dataclass(slots=True)
class NamedResourceLock:
path: Path
@@ -27,36 +197,153 @@ class NamedResourceLock:
timeout_seconds: float
poll_seconds: float
_acquired: bool = False
_lease_id: str | None = None
_acquired_pid: int | None = None
def __enter__(self) -> "NamedResourceLock":
deadline = time.monotonic() + max(0.0, self.timeout_seconds)
self.path.parent.mkdir(parents=True, exist_ok=True)
reclaim_path = self.path.with_suffix(self.path.suffix + ".reclaim")
while True:
try:
descriptor = os.open(self.path, os.O_CREAT | os.O_EXCL | os.O_WRONLY)
except FileExistsError:
if time.monotonic() >= deadline:
raise ResourceBusyError(f"resource is busy: {self.resource}")
time.sleep(self.poll_seconds)
with _exclusive_reclaim_guard(
reclaim_path,
resource=self.resource,
deadline=deadline,
poll_seconds=self.poll_seconds,
):
if self.path.exists():
recovered = self._recover_stale_lock()
else:
recovered = False
if self._publish_lock():
self._acquired = True
return self
if recovered:
continue
metadata = {
"resource": self.resource,
"owner": self.owner,
"acquired_at": datetime.now(timezone.utc).isoformat(timespec="seconds"),
"pid": os.getpid(),
}
with os.fdopen(descriptor, "w", encoding="utf-8", newline="\n") as handle:
if time.monotonic() >= deadline:
raise ResourceBusyError(f"resource is busy: {self.resource}")
time.sleep(self.poll_seconds)
def _publish_lock(self) -> bool:
lease_id = uuid.uuid4().hex
pid = os.getpid()
metadata: dict[str, object] = {
"resource": self.resource,
"owner": self.owner,
"acquired_at": datetime.now(timezone.utc).isoformat(timespec="seconds"),
"pid": pid,
"lease_id": lease_id,
}
process_started_at = _process_started_at(pid)
if process_started_at is not None:
metadata["process_started_at"] = process_started_at
candidate = self.path.with_name(
f".{self.path.name}.{pid}.{lease_id}.tmp"
)
try:
with candidate.open("x", encoding="utf-8", newline="\n") as handle:
json.dump(metadata, handle, ensure_ascii=False, indent=2, sort_keys=True)
handle.write("\n")
handle.flush()
os.fsync(handle.fileno())
self._acquired = True
return self
try:
os.link(candidate, self.path)
except FileExistsError:
return False
finally:
candidate.unlink(missing_ok=True)
self._lease_id = lease_id
self._acquired_pid = pid
return True
def _recover_stale_lock(self) -> bool:
"""Quarantine a lock only when its recorded owner PID is provably dead."""
try:
snapshot = self.path.stat()
text = self.path.read_text(encoding="utf-8")
except FileNotFoundError:
return True
except (OSError, UnicodeError):
return False
try:
metadata = json.loads(text)
except json.JSONDecodeError:
metadata = None
if isinstance(metadata, dict):
pid = metadata.get("pid")
if (
metadata.get("resource") == self.resource
and isinstance(pid, int)
and not isinstance(pid, bool)
):
started_at = _coerce_process_started_at(
metadata.get("process_started_at")
)
if not _owner_is_provably_stale(pid, started_at):
return False
return self._quarantine_if_unchanged(snapshot)
age_seconds = max(0.0, time.time() - snapshot.st_mtime)
if age_seconds < _MALFORMED_LOCK_GRACE_SECONDS:
return False
partial_pid = _partial_pid(text)
if partial_pid is not None:
partial_started_at = _partial_process_started_at(text)
if not _owner_is_provably_stale(partial_pid, partial_started_at):
return False
return self._quarantine_if_unchanged(snapshot)
def _quarantine_if_unchanged(self, snapshot: os.stat_result) -> bool:
try:
current = self.path.stat()
except FileNotFoundError:
return True
except OSError:
return False
if _file_identity(current) != _file_identity(snapshot):
return False
stale_root = self.path.parent / "stale"
stale_root.mkdir(parents=True, exist_ok=True)
stale_path = stale_root / (
f"{self.path.stem}.{time.time_ns()}.{uuid.uuid4().hex}.lock"
)
try:
os.replace(self.path, stale_path)
except FileNotFoundError:
return True
return True
def __exit__(self, exc_type, exc, traceback) -> None: # type: ignore[no-untyped-def]
if self._acquired:
self.path.unlink(missing_ok=True)
self._acquired = False
reclaim_path = self.path.with_suffix(self.path.suffix + ".reclaim")
deadline = time.monotonic() + _RELEASE_GUARD_TIMEOUT_SECONDS
try:
with _exclusive_reclaim_guard(
reclaim_path,
resource=self.resource,
deadline=deadline,
poll_seconds=self.poll_seconds,
):
try:
metadata = json.loads(self.path.read_text(encoding="utf-8"))
except (FileNotFoundError, OSError, UnicodeError, json.JSONDecodeError):
metadata = None
if (
isinstance(metadata, dict)
and metadata.get("lease_id") == self._lease_id
and metadata.get("pid") == self._acquired_pid
and self._acquired_pid == os.getpid()
):
self.path.unlink(missing_ok=True)
finally:
self._acquired = False
self._lease_id = None
self._acquired_pid = None
@dataclass(frozen=True, slots=True)
src/gyxx_flow/core/records.py
+88 -36
View File
@@ -3,9 +3,11 @@
from __future__ import annotations
import json
from dataclasses import dataclass
from collections.abc import Sequence
from dataclasses import dataclass, field
from datetime import datetime, timezone
from pathlib import Path
from threading import RLock
from typing import Any, Literal
from .artifacts import atomic_write_json
@@ -14,6 +16,7 @@ from .layout import DataLayout
StepStatus = Literal["success", "failed", "skipped"]
RunStatus = Literal["success", "failed", "cancelled"]
RunMode = Literal["execute", "dry_run", "unknown"]
def _now() -> str:
@@ -23,9 +26,18 @@ def _now() -> str:
@dataclass(frozen=True, slots=True)
class RunJournal:
path: Path
_lock: Any = field(default_factory=RLock, compare=False, repr=False)
@classmethod
def create(cls, layout: DataLayout, context: RunContext) -> "RunJournal":
def create(
cls,
layout: DataLayout,
context: RunContext,
*,
mode: RunMode = "unknown",
) -> "RunJournal":
if not isinstance(mode, str) or mode not in {"execute", "dry_run", "unknown"}:
raise ValueError(f"invalid run mode: {mode!r}")
run_dir = layout.run_dir(
context.workflow_id,
context.business_date.isoformat(),
@@ -42,6 +54,7 @@ class RunJournal:
"workflow_id": context.workflow_id,
"run_id": context.run_id,
"business_date": context.business_date.isoformat(),
"mode": mode,
"shadow": context.shadow,
"started_at": context.started_at.isoformat(timespec="seconds"),
"ended_at": None,
@@ -66,19 +79,20 @@ class RunJournal:
return json.loads(self.path.read_text(encoding="utf-8"))
def start_step(self, step_id: str, *, attempt: int) -> None:
payload = self._read()
previous = payload["steps"].get(step_id)
if previous and previous["status"] == "running":
raise ValueError(f"step {step_id!r} is already running")
payload["steps"][step_id] = {
"status": "running",
"attempt": attempt,
"started_at": _now(),
"ended_at": None,
"exit_code": None,
"error": None,
}
atomic_write_json(self.path, payload)
with self._lock:
payload = self._read()
previous = payload["steps"].get(step_id)
if previous and previous["status"] == "running":
raise ValueError(f"step {step_id!r} is already running")
payload["steps"][step_id] = {
"status": "running",
"attempt": attempt,
"started_at": _now(),
"ended_at": None,
"exit_code": None,
"error": None,
}
atomic_write_json(self.path, payload)
def finish_step(
self,
@@ -90,26 +104,28 @@ class RunJournal:
) -> None:
if status not in {"success", "failed", "skipped"}:
raise ValueError(f"invalid step status: {status!r}")
payload = self._read()
step = payload["steps"].get(step_id)
if not step or step["status"] != "running":
raise ValueError(f"step {step_id!r} is not running")
step.update(
{
"status": status,
"ended_at": _now(),
"exit_code": exit_code,
"error": error,
}
)
atomic_write_json(self.path, payload)
with self._lock:
payload = self._read()
step = payload["steps"].get(step_id)
if not step or step["status"] != "running":
raise ValueError(f"step {step_id!r} is not running")
step.update(
{
"status": status,
"ended_at": _now(),
"exit_code": exit_code,
"error": error,
}
)
atomic_write_json(self.path, payload)
def finalize(self, status: RunStatus, *, error: str | None = None) -> None:
if status not in {"success", "failed", "cancelled"}:
raise ValueError(f"invalid run status: {status!r}")
payload = self._read()
payload.update({"status": status, "ended_at": _now(), "error": error})
atomic_write_json(self.path, payload)
with self._lock:
payload = self._read()
payload.update({"status": status, "ended_at": _now(), "error": error})
atomic_write_json(self.path, payload)
def record_input(self, reference: str) -> None:
self._record_trace("inputs", reference)
@@ -120,6 +136,41 @@ class RunJournal:
def record_external_write(self, reference: str) -> None:
self._record_trace("external_writes", reference)
def normalize_step_order(self, step_ids: Sequence[str]) -> None:
"""Make concurrent graph journal collections deterministic by DAG order."""
ordered = tuple(step_ids)
if len(set(ordered)) != len(ordered) or any(not value for value in ordered):
raise ValueError("step_ids must be unique non-empty strings")
ranks = {step_id: index for index, step_id in enumerate(ordered)}
def rank(reference: str, original_index: int) -> tuple[int, int]:
if reference.startswith("step:"):
step_id = reference.split(":", 2)[1]
else:
step_id = reference.split(".attempt-", 1)[0]
return ranks.get(step_id, len(ranks)), original_index
with self._lock:
payload = self._read()
step_items = list(payload["steps"].items())
payload["steps"] = dict(
sorted(
step_items,
key=lambda item: rank(item[0], step_items.index(item)),
)
)
for category in ("inputs", "outputs", "external_writes"):
references = payload["trace"][category]
payload["trace"][category] = [
reference
for _, reference in sorted(
enumerate(references),
key=lambda item: rank(item[1], item[0]),
)
]
atomic_write_json(self.path, payload)
def _record_trace(self, category: str, reference: str) -> None:
if (
not isinstance(reference, str)
@@ -128,8 +179,9 @@ class RunJournal:
or any(ord(character) < 32 for character in reference)
):
raise ValueError("invalid trace reference")
payload = self._read()
references = payload["trace"][category]
if reference not in references:
references.append(reference)
atomic_write_json(self.path, payload)
with self._lock:
payload = self._read()
references = payload["trace"][category]
if reference not in references:
references.append(reference)
atomic_write_json(self.path, payload)
src/gyxx_flow/core/text.py
+25
View File
@@ -0,0 +1,25 @@
"""Small text helpers shared by runtime and presentation boundaries."""
from __future__ import annotations
_OMISSION_MARKER = "\n...[middle output omitted]...\n"
def bounded_head_tail(value: str, limit: int) -> str:
"""Bound text while retaining both its context and final diagnostic lines."""
if limit < 1:
raise ValueError("limit must be positive")
if len(value) <= limit:
return value
if limit <= len(_OMISSION_MARKER):
return value[-limit:]
available = limit - len(_OMISSION_MARKER)
head_length = max(1, available // 4)
tail_length = available - head_length
return (
value[:head_length]
+ _OMISSION_MARKER
+ value[-tail_length:]
)
src/gyxx_flow/diagnostics.py
+1 -2
View File
@@ -54,7 +54,7 @@ def run_doctor(settings: Settings) -> DoctorReport:
]
try:
catalog = WorkflowCatalog.load(settings.project_root / "config")
catalog_ok = len(catalog.scheduled_workflows()) == 21
catalog_ok = len(catalog.scheduled_workflows()) == 23
catalog_message = f"catalog has {len(catalog.scheduled_workflows())} scheduled workflows"
except Exception:
catalog_ok = False
@@ -91,4 +91,3 @@ def _probe_write_permission(path: Path) -> tuple[bool, str]:
except OSError:
return False, "data-root parent is not writable"
return True, "data-root parent write probe passed"
src/gyxx_flow/modules/content_marketing/__init__.py
+15 -29
View File
@@ -5,29 +5,25 @@ from __future__ import annotations
from collections.abc import Iterable
from typing import Any
from gyxx_flow.adapters.native import DeferredModuleCommandStep, ModuleCommandFactory
from gyxx_flow.workflow import StepDefinition, WorkflowDefinition
from gyxx_flow.adapters.native import ModuleCommandFactory
from gyxx_flow.workflow import WorkflowDefinition
from gyxx_flow.workflow.factory import build_catalog_workflow
CONTENT_SCHEDULED_WORKFLOW_IDS = (
"content.metrics.daily",
"content.marketing_report.daily",
"content.relogin.weekly",
"content.self_operated.weekly",
"content.creator_report.monthly",
"content.summary.monthly",
"content.cooperations.daily",
"content.comments.weekly",
"content.summary.weekly",
)
CONTENT_MANUAL_WORKFLOW_IDS = (
"content.mapping.refresh",
"content.retry_failed",
"content.metrics.backfill",
)
CONTENT_WORKFLOW_IDS = (*CONTENT_SCHEDULED_WORKFLOW_IDS, *CONTENT_MANUAL_WORKFLOW_IDS)
CONTENT_WORKFLOW_IDS = CONTENT_SCHEDULED_WORKFLOW_IDS
CONTENT_TIMEOUT_SECONDS = 4 * 60 * 60
CONTENT_RESOURCE = "module:content_marketing"
_RELOGIN_WORKFLOW_ID = "content.relogin.weekly"
_PARALLEL_COMMENT_WORKFLOW_ID = "content.comments.weekly"
class ContentMarketingModule:
@@ -54,7 +50,7 @@ class ContentMarketingModule:
entries = {
entry.workflow_id: entry
for entry in catalog.workflows
if entry.module == cls.module_id and entry.trigger in {"scheduled", "manual"}
if entry.module == cls.module_id and entry.trigger == "scheduled"
}
expected = set(CONTENT_WORKFLOW_IDS)
if set(entries) != expected:
@@ -65,25 +61,16 @@ class ContentMarketingModule:
definitions: list[WorkflowDefinition] = []
for workflow_id in CONTENT_WORKFLOW_IDS:
entry = entries[workflow_id]
action = (
DeferredModuleCommandStep(entry)
if command_factory is None
else DeferredModuleCommandStep(entry, command_factory=command_factory)
)
definitions.append(
WorkflowDefinition(
workflow_id,
(
StepDefinition(
"module_run",
action,
timeout_seconds=CONTENT_TIMEOUT_SECONDS,
max_attempts=1,
resources=(CONTENT_RESOURCE,),
production_sink=True,
official_notification=workflow_id
== _RELOGIN_WORKFLOW_ID,
),
build_catalog_workflow(
entry,
default_step_id="module_run",
timeout_seconds=CONTENT_TIMEOUT_SECONDS,
resource=CONTENT_RESOURCE,
command_factory=command_factory,
official_notification=workflow_id == _RELOGIN_WORKFLOW_ID,
independent_step_resources=(
workflow_id == _PARALLEL_COMMENT_WORKFLOW_ID
),
)
)
@@ -95,7 +82,6 @@ class ContentMarketingModule:
__all__ = [
"CONTENT_WORKFLOW_IDS",
"CONTENT_MANUAL_WORKFLOW_IDS",
"CONTENT_SCHEDULED_WORKFLOW_IDS",
"CONTENT_RESOURCE",
"CONTENT_TIMEOUT_SECONDS",
src/gyxx_flow/modules/content_marketing/runtime/bilibili_comment_scraper.py → src/gyxx_flow/modules/content_marketing/bilibili_comment_scraper.py
+1 -1
View File
@@ -7,7 +7,7 @@ import time
from datetime import datetime
from pathlib import Path
from gyxx_flow.modules.content_marketing.runtime.runtime_paths import PATHS
from gyxx_flow.modules.content_marketing.runtime_paths import PATHS
from typing import Any
from urllib.parse import parse_qs, urlencode, urlsplit, urlunsplit
src/gyxx_flow/modules/content_marketing/runtime/bilibili_scraper.py → src/gyxx_flow/modules/content_marketing/bilibili_scraper.py
+98 -13
View File
@@ -48,19 +48,22 @@ import sys
import time
from datetime import datetime, date
from pathlib import Path
from gyxx_flow.modules.content_marketing.runtime.runtime_paths import (
from gyxx_flow.modules.content_marketing.runtime_paths import (
PATHS,
resolve_layer_output,
)
import requests
from gyxx_flow.modules.content_marketing.runtime.collection_completeness import (
from gyxx_flow.modules.content_marketing.collection_completeness import (
add_repeatable_style_argument,
atomic_write_json,
merge_global_summaries,
select_requested_styles,
)
from gyxx_flow.modules.content_marketing.daily_creator_exposure_scope import (
classify_publish_scope,
)
BASE_DIR = PATHS.module_root
DEFAULT_DATA_DIR = PATHS.normalized_root
@@ -217,8 +220,12 @@ def call_lark_json(args: list[str]) -> dict:
def load_mapping(data_dir: Path, self_operated: bool = False) -> dict:
# 动态:从「合作达人」/「自营达人」多维表格地址表读取款式→各表地址,实时拉字段重建对照表
from gyxx_flow.modules.content_marketing.runtime import feishu_mapping
return feishu_mapping.load_mapping(None, self_operated=self_operated)
from gyxx_flow.modules.content_marketing import feishu_mapping
mapping = feishu_mapping.load_mapping(None, self_operated=self_operated)
if self_operated:
for style in mapping.get("tables", []):
style.get("field_map", {}).pop("daily_exposure", None)
return mapping
def list_all_records(base_token: str, table_id: str) -> list[dict]:
@@ -266,6 +273,14 @@ def write_record(base_token: str, table_id: str, record_id: str,
if dry_run:
log(f" [DRY-RUN] 写入: {json.dumps(fields, ensure_ascii=False)[:200]}")
return True
from gyxx_flow.adapters.acceptance_policy import skip_feishu_table_write
if skip_feishu_table_write(
"content_marketing.bilibili.record-upsert",
details={"table_id": table_id, "record_id": record_id},
):
log(f" [ACCEPTANCE-SKIP] record_id={record_id}")
return True
pf = _write_payload_to_cwd(fields)
cmd = [LARK_CLI, "base", "+record-upsert",
"--base-token", base_token,
@@ -305,6 +320,14 @@ def create_record(base_token: str, table_id: str,
if dry_run:
log(f" [DRY-RUN] 新建: {json.dumps(fields, ensure_ascii=False)[:200]}")
return "rec_dryrun"
from gyxx_flow.adapters.acceptance_policy import skip_feishu_table_write
if skip_feishu_table_write(
"content_marketing.bilibili.record-create",
details={"table_id": table_id},
):
log(" [ACCEPTANCE-SKIP] create record")
return "acceptance-skipped"
pf = _write_payload_to_cwd(fields)
cmd = [LARK_CLI, "base", "+record-upsert",
"--base-token", base_token,
@@ -549,7 +572,9 @@ def save_state(data_dir: Path, state: dict) -> None:
def process_style(style: dict, only_record_ids: set[str] | None,
dry_run: bool, delay: float, session: requests.Session,
state: dict, force_today: date | None,
first_run: bool = False) -> dict:
first_run: bool = False,
published_from: date | None = None,
max_age_days: int | None = None) -> dict:
base_token = style["base_token"]
table_id = style["table_id"]
fmap = style.get("field_map", {})
@@ -562,6 +587,10 @@ def process_style(style: dict, only_record_ids: set[str] | None,
fid_month = fmap.get(KEY_MONTH, {}).get("field_id", "")
fid_parent = fmap.get(KEY_PARENT, {}).get("field_id", "")
fid_week = fmap.get(KEY_WEEK, {}).get("field_id", "")
daily_info = fmap.get("daily_exposure") or {}
daily_fid = daily_info.get("field_id", "")
daily_field_name = daily_info.get("field_name", "")
use_daily_exposure = bool(daily_fid)
# 5 档槽位的 field_id 列表
slot_fids = [get_slot_fid(fmap, ln) or "" for ln, _ in SLOTS]
@@ -573,7 +602,12 @@ def process_style(style: dict, only_record_ids: set[str] | None,
missing_mapping.append("platform")
if not fid_url:
missing_mapping.append("note_url")
if first_run:
if (published_from is not None or max_age_days is not None) and not fid_pubtime:
missing_mapping.append("publish_time")
if use_daily_exposure:
if not daily_fid:
missing_mapping.append("daily_exposure")
elif first_run:
if not slot_fids[0]:
missing_mapping.append("read_count_7d")
else:
@@ -617,12 +651,35 @@ def process_style(style: dict, only_record_ids: set[str] | None,
log(f" 发布链接过滤: 待采 {len(b_records)} 条,未触发 {skipped_no_url} 条")
today = force_today or date.today()
scope_skip_reasons: dict[str, int] = {}
if published_from is not None or max_age_days is not None:
scoped_records = []
for record in b_records:
publish_date = parse_pub_date(record.get(fid_pubtime))
reason = classify_publish_scope(
publish_date,
today,
published_from=published_from,
max_age_days=max_age_days,
)
if reason:
scope_skip_reasons[reason] = scope_skip_reasons.get(reason, 0) + 1
else:
scoped_records.append(record)
b_records = scoped_records
if scope_skip_reasons:
log(
f" 每日范围过滤: 跳过 {sum(scope_skip_reasons.values())} 条 "
f"{scope_skip_reasons}"
)
style_state = state.setdefault(style["name"], {})
summary = {
"style": style["name"], "index": style["index"],
"run_started_at": time.time(),
"source_b_records": source_b_records,
"skipped_no_url": skipped_no_url,
"skipped_publish_scope": sum(scope_skip_reasons.values()),
"publish_scope_skip_reasons": scope_skip_reasons,
"total_b_records": len(b_records), "updated": 0, "skipped": 0,
"details": [],
}
@@ -637,7 +694,7 @@ def process_style(style: dict, only_record_ids: set[str] | None,
# 解析发布时间
pub_date = parse_pub_date(pubtime_raw)
if pub_date is None and not first_run:
if pub_date is None and not first_run and not use_daily_exposure:
log(f" [SKIP] {rid[:10]}.. {creator} 发布时间为空,无法判断槽位")
summary["skipped"] += 1
summary["details"].append({
@@ -648,13 +705,20 @@ def process_style(style: dict, only_record_ids: set[str] | None,
continue
# 算槽位: (今天 - 发布日) 天数差
if first_run:
if use_daily_exposure:
if pub_date is not None and pub_date > today:
target_slot_fid, target_slot_key = None, None
else:
target_slot_fid, target_slot_key = daily_fid, daily_field_name
elif first_run:
target_slot_fid, target_slot_key = slot_fids[0] or None, SLOTS[0][1] if slot_fids[0] else None
else:
days_elapsed = (today - pub_date).days
target_slot_fid, target_slot_key = pick_slot_by_days(days_elapsed, slot_fids)
if target_slot_fid is None:
if first_run:
if use_daily_exposure:
log(f" [SKIP] {rid[:10]}.. {creator} 发布时间 {pub_date} 在 {today} 之后")
elif first_run:
log(f" [SKIP] {rid[:10]}.. {creator} 7天曝光量字段缺失")
else:
log(f" [SKIP] {rid[:10]}.. {creator} 对应槽位字段缺失或发布时间 {pub_date} 在 {today} 之后")
@@ -681,9 +745,10 @@ def process_style(style: dict, only_record_ids: set[str] | None,
continue
# 覆盖写入 (同槽覆盖语义,数字更新)
action = "first_run_fill" if first_run else "update_slot"
action = "daily_snapshot" if use_daily_exposure else ("first_run_fill" if first_run else "update_slot")
target_label = str(target_slot_key) if use_daily_exposure else f"{target_slot_key}天曝光量"
log(f" {rid[:10]}.. {creator} 发布={pub_date} 距今={(today - pub_date).days if pub_date else '-'}天"
f" → 写 {target_slot_key}天曝光量={play}")
f" → 写 {target_label}={play}")
ok = write_record(base_token, table_id, rid,
{target_slot_fid: play}, dry_run)
if ok:
@@ -693,7 +758,8 @@ def process_style(style: dict, only_record_ids: set[str] | None,
"record_id": rid, "creator": creator,
"publish_date": str(pub_date) if pub_date else None,
"days_elapsed": (today - pub_date).days if pub_date else None,
"slot": target_slot_key, "play_count": play,
"slot": "daily_exposure" if use_daily_exposure else target_slot_key,
"field_name": target_label, "play_count": play,
"action": action, "ok": ok,
"matched": True,
"write_ok": ok if not dry_run else None,
@@ -771,6 +837,12 @@ def main() -> int:
help="首次回填: 所有 B 站记录全部填 7天曝光量,跳过周次逻辑")
parser.add_argument("--self-operated", action="store_true",
help="抓取自营达人表格(而非合作达人)")
parser.add_argument("--published-from", type=date.fromisoformat,
help="仅采集此日期及以后发布的笔记,格式 YYYY-MM-DD")
parser.add_argument("--max-age-days", type=int,
help="发布满此天数后停止采集")
parser.add_argument("--no-retry", action="store_true", help=argparse.SUPPRESS)
parser.add_argument("--skip-field-prepare", action="store_true", help=argparse.SUPPRESS)
args = parser.parse_args()
data_dir = resolve_layer_output(
@@ -789,6 +861,17 @@ def main() -> int:
if args.today:
force_today = date.fromisoformat(args.today)
if not args.self_operated and not args.skip_field_prepare:
from gyxx_flow.modules.content_marketing import feishu_mapping
prepared = feishu_mapping.ensure_daily_exposure_fields(
mapping, target_date=force_today, dry_run=args.dry_run,
write=not args.dry_run,
selected_indices={style["index"] for style in styles},
)
if not prepared["ok"]:
log(f"[ERROR] 当天曝光字段准备失败: {prepared['errors']}")
return 1
state_path = BILIBILI_CHECKPOINT_DIR / STATE_FILENAME
if args.reset_state and state_path.exists():
state_path.unlink()
@@ -806,7 +889,9 @@ def main() -> int:
final = out_dir / f"_all_summaries{self_sfx}_bilibili.json"
for s in styles:
summary = process_style(s, only_rids, args.dry_run, args.delay,
session, state, force_today, args.first_run)
session, state, force_today, args.first_run,
published_from=args.published_from,
max_age_days=args.max_age_days)
# 传了 --record 但本款式没匹配到 → 跳过 (不落空盘,不计入总汇总)
if only_rids and summary["total_b_records"] == 0 and summary["updated"] == 0 \
and summary["skipped"] == 0:
src/gyxx_flow/modules/content_marketing/runtime/chanmama_scraper.py → src/gyxx_flow/modules/content_marketing/chanmama_scraper.py
+414 -55
View File
@@ -5,30 +5,34 @@
依赖：pip install selenium webdriver-manager
"""
import sys
import io
import time
import os
import csv
import re
import json
import argparse
import csv
import json
import os
import re
import subprocess
import sys
import time
from datetime import datetime
from pathlib import Path
from gyxx_flow.modules.content_marketing.runtime.runtime_paths import PATHS
# 修复 Windows 控制台输出编码
sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding="utf-8", errors="replace")
sys.stderr = io.TextIOWrapper(sys.stderr.buffer, encoding="utf-8", errors="replace")
from gyxx_flow.core.exit_codes import COOKIE_SKIP_EXIT_CODE
from gyxx_flow.modules.content_marketing.runtime_paths import PATHS
# 避免替换并关闭 pytest、服务管理器等宿主提供的捕获流。
for _stream in (sys.stdout, sys.stderr):
try:
_stream.reconfigure(encoding="utf-8", errors="replace")
except (AttributeError, OSError, ValueError):
pass
try:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import NoSuchElementException, TimeoutException
from selenium.webdriver.chrome.options import Options
from selenium.common.exceptions import TimeoutException, NoSuchElementException
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
HAS_SELENIUM = True
except ImportError:
HAS_SELENIUM = False
@@ -52,6 +56,8 @@ TARGET_URLS = [
DOWNLOAD_DIR = str(PATHS.raw_root / "chanmama")
COOKIE_FILE = str(PATHS.browser_cookie_file)
COOKIE_MAX_AGE = 7 * 24 * 3600 # cookie 有效期 7 天（秒）
REFRESH_WAIT_SECONDS = int(os.getenv("CHANMAMA_REFRESH_WAIT_SECONDS", "600"))
REFRESH_MAX_PAGES = 50
# ================================================
@@ -193,7 +199,7 @@ def load_cookies(driver):
return False
def handle_captcha(driver):
def handle_captcha(driver) -> bool:
"""处理登录时的验证码（滑块/图形/短信等）
点击登录后轮询等待：检测到验证码时提示用户在浏览器手动完成，
@@ -241,12 +247,29 @@ def handle_captcha(driver):
if not _has_captcha():
if _logged_in_ok():
print("✅ 登录成功，无需验证码")
return
return True
# 可能验证码延迟出现，再观察一下
time.sleep(3)
if not _has_captcha() and _logged_in_ok():
print("✅ 登录成功，无需验证码")
return
return True
from gyxx_flow.adapters.acceptance_policy import current_acceptance_policy
acceptance_policy = current_acceptance_policy()
if acceptance_policy.enabled:
reason = (
"captcha or interactive verification is required"
if _has_captcha()
else "credential login did not complete without interaction"
)
acceptance_policy.record(
"cookie_skipped",
operation="content.chanmama.login",
details={"status": "SKIPPED_COOKIE", "reason": reason},
)
print(f"[SKIPPED_COOKIE] {reason}", flush=True)
return False
if _has_captcha():
print("\n" + "=" * 50)
@@ -262,7 +285,7 @@ def handle_captcha(driver):
while waited < max_wait:
if _logged_in_ok():
print("✅ 检测到登录成功，继续后续流程...")
return
return True
# 还在登录页且仍有验证码 -> 继续等用户操作
waited += interval
remaining = max_wait - waited
@@ -271,6 +294,7 @@ def handle_captcha(driver):
time.sleep(interval)
print("⚠️ 等待验证码超时（180s），继续尝试后续步骤...")
return False
def verify_login(driver):
@@ -387,7 +411,6 @@ def login(driver):
# ② 我已阅读并同意 → 是一个自定义 div.cursor-pointer（不是 checkbox！）
# 必须点击这个 div 本体才生效
agree_clicked = False
try:
agree_div = driver.find_element(
By.XPATH,
@@ -396,7 +419,6 @@ def login(driver):
driver.execute_script("arguments[0].click();", agree_div)
time.sleep(0.3)
# 验证：点击后该 div 或其内部是否出现勾选标记（class 变化或新增元素）
agree_clicked = True
print("✅ 已点击【我已阅读并同意】协议区域")
except Exception as e:
print(f"⚠️ 协议 div 定位失败: {e}，尝试备用选择器")
@@ -406,7 +428,6 @@ def login(driver):
"//div[contains(.,'我已阅读并同意') and contains(@class,'cursor-pointer')]"
)
driver.execute_script("arguments[0].click();", agree_div)
agree_clicked = True
print("✅ 已点击【我已阅读并同意】协议区域(备用)")
except Exception as e2:
print(f"⚠️ 协议勾选全部失败: {e2}")
@@ -422,7 +443,8 @@ def login(driver):
print("✅ 已点击【登录】按钮 (备用)")
# 步骤6：处理验证码（如有）：等待用户在浏览器中手动完成
handle_captcha(driver)
if not handle_captcha(driver):
return False
# 等待登录跳转完成
print("⏳ 等待登录完成...")
@@ -451,6 +473,334 @@ def navigate_to_target(driver, url):
return True
def _classify_exposure_text(text):
"""正数曝光视为就绪；零值或任意非空状态文字需要刷新。"""
normalized = re.sub(r"\s+", "", "" if text is None else str(text))
if not normalized:
return "ready"
numeric = normalized.lower().replace(",", "")
match = re.fullmatch(r"(\d+(?:\.\d+)?)(?:万|w)?\+?", numeric)
if match:
return "refresh" if float(match.group(1)) == 0 else "ready"
return "refresh"
def _pagination_element_disabled(element):
try:
if not element.is_enabled():
return True
except Exception:
pass
css = str(element.get_attribute("class") or "").lower()
aria = str(element.get_attribute("aria-disabled") or "").lower()
disabled = str(element.get_attribute("disabled") or "").lower()
return "disabled" in css or aria == "true" or disabled in {"true", "disabled"}
def _find_video_table(driver):
def locate(current_driver):
for table in current_driver.find_elements(By.CSS_SELECTOR, "table"):
try:
headers = table.find_elements(By.CSS_SELECTOR, "thead th, thead td")
if table.is_displayed() and "预估曝光" in "|".join(h.text or "" for h in headers):
return table
except Exception:
continue
return False
return WebDriverWait(driver, 15).until(locate)
def _exposure_column_index(table):
for index, header in enumerate(table.find_elements(By.CSS_SELECTOR, "thead th, thead td")):
if "预估曝光" in (header.text or "").replace("\n", ""):
return index
return 2
def _page_signature(driver):
parts = []
for selector in (".el-pagination .el-pager li.active", ".el-pagination .el-pager li.is-active", ".el-pager li.active"):
try:
active = driver.find_elements(By.CSS_SELECTOR, selector)
if active:
parts.append(active[0].text.strip())
break
except Exception:
continue
try:
parts.extend((row.text or "").strip() for row in _find_video_table(driver).find_elements(By.CSS_SELECTOR, "tbody tr")[:3])
except Exception:
pass
return "|".join(parts)
def _find_next_page_button(driver):
selectors = [
(By.CSS_SELECTOR, ".el-pagination .btn-next"),
(By.CSS_SELECTOR, "button.btn-next"),
(By.CSS_SELECTOR, "li.next"),
(By.CSS_SELECTOR, "button[aria-label*='下一页']"),
(By.CSS_SELECTOR, "[title='下一页']"),
(By.XPATH, "//button[contains(.,'下一页')]"),
(By.XPATH, "//span[normalize-space()='下一页']/ancestor::button[1]"),
]
for by, selector in selectors:
try:
for element in driver.find_elements(by, selector):
if element.is_displayed():
return element
except Exception:
continue
return None
def _find_get_data_targets(driver):
"""查找当前页真正绑定点击事件的“获取数据”节点。"""
targets = []
seen = set()
try:
candidates = driver.find_elements(By.CSS_SELECTOR, "td .cursor-pointer")
except Exception:
candidates = []
for candidate in candidates:
try:
if not candidate.is_displayed():
continue
css_classes = str(candidate.get_attribute("class") or "").split()
if "cursor-pointer" not in css_classes:
continue
if re.sub(r"\s+", "", candidate.text or "") != "获取数据":
continue
key = getattr(candidate, "id", None) or id(candidate)
if key in seen:
continue
seen.add(key)
targets.append(candidate)
except Exception:
continue
return targets
def _get_data_target_gone(target):
"""点击后节点消失、隐藏或文字变化，才视为页面已接受操作。"""
try:
return (
not target.is_displayed()
or re.sub(r"\s+", "", target.text or "") != "获取数据"
)
except Exception:
return True
def _click_and_confirm_get_data(driver, target, timeout=8):
"""点击真实 cursor-pointer，并验证“获取数据”状态已经变化。"""
before_count = max(len(_find_get_data_targets(driver)), 1)
def click_confirmed():
if not _get_data_target_gone(target):
return False
return len(_find_get_data_targets(driver)) < before_count
try:
driver.execute_script(
"arguments[0].scrollIntoView({block:'center'});", target
)
except Exception:
pass
try:
target.click()
except Exception:
try:
driver.execute_script("arguments[0].click();", target)
except Exception:
return False
deadline = time.monotonic() + max(timeout, 0)
js_retried = False
while time.monotonic() <= deadline:
if click_confirmed():
return True
if not js_retried and time.monotonic() >= deadline - max(timeout - 2, 0):
try:
driver.execute_script("arguments[0].click();", target)
except Exception:
pass
js_retried = True
time.sleep(0.25)
return click_confirmed()
def _count_pending_exposures(driver):
"""统计当前页已经进入“更新中”的曝光单元格。"""
try:
cells = driver.find_elements(
By.XPATH,
"//td[contains(normalize-space(.),'更新中')]",
)
except Exception:
return 0
count = 0
for cell in cells:
try:
if cell.is_displayed() and "更新中" in re.sub(r"\s+", "", cell.text or ""):
count += 1
except Exception:
continue
return count
def _refresh_current_page(driver):
"""只点击当前页“预估曝光”列中的“获取数据”，返回本页统计。"""
result = {
"clicked": 0,
"pending": 0,
"zero_without_button": 0,
"unconfirmed": 0,
}
attempted = set()
while True:
targets = []
for target in _find_get_data_targets(driver):
key = getattr(target, "id", None) or id(target)
if key not in attempted:
targets.append((key, target))
if not targets:
break
key, target = targets[0]
if _click_and_confirm_get_data(driver, target):
result["clicked"] += 1
else:
attempted.add(key)
result["unconfirmed"] += 1
print(" ⚠️ 已点击“获取数据”，但页面状态未变化，不计为成功")
result["pending"] = _count_pending_exposures(driver)
return result
def refresh_zero_exposure_videos(driver, max_pages=REFRESH_MAX_PAGES):
"""遍历当前博主的所有分页，触发零曝光/待更新记录的数据更新。"""
total = {
"clicked": 0,
"pending": 0,
"zero_without_button": 0,
"unconfirmed": 0,
"pages": 0,
}
seen_signatures = set()
for page_number in range(1, max_pages + 1):
try:
before = _page_signature(driver)
if before and before in seen_signatures:
print("📄 检测到重复页面，停止翻页")
break
if before:
seen_signatures.add(before)
page_result = _refresh_current_page(driver)
total["pages"] += 1
for key in (
"clicked",
"pending",
"zero_without_button",
"unconfirmed",
):
total[key] += page_result.get(key, 0)
print(
f"📄 第 {page_number} 页: 点击“获取数据” {page_result['clicked']} 条, "
f"更新中 {page_result['pending']} 条, "
f"点击未确认 {page_result.get('unconfirmed', 0)} 条"
)
next_button = _find_next_page_button(driver)
if next_button is None or _pagination_element_disabled(next_button):
print("📄 已到最后一页")
break
driver.execute_script("arguments[0].click();", next_button)
changed = False
for _ in range(20):
time.sleep(0.5)
after = _page_signature(driver)
if after and after != before:
changed = True
break
if not changed:
print("⚠️ 点击下一页后内容未变化，停止翻页以避免死循环")
break
except Exception as exc:
print(f"⚠️ 第 {page_number} 页刷新扫描失败: {exc}")
break
print(
f"🔄 刷新扫描完成: {total['pages']} 页, 点击“获取数据” {total['clicked']} 条, "
f"更新中 {total['pending']} 条, "
f"点击未确认 {total['unconfirmed']} 条"
)
return total
def _max_excel_mtime():
root = Path(DOWNLOAD_DIR)
if not root.exists():
return 0.0
return max((p.stat().st_mtime for p in root.glob("*.xlsx") if not p.name.startswith("~$")), default=0.0)
def refresh_then_export_accounts(driver, urls, wait_seconds=REFRESH_WAIT_SECONDS):
"""两阶段执行：先刷新所有账号全部分页，再等待并重新导出。"""
refresh_summaries = []
for index, url in enumerate(urls, 1):
print(f"\n{'=' * 50}")
print(f"🔄 刷新阶段 博主 {index}/{len(urls)}: {url}")
print(f"{'=' * 50}")
try:
navigate_to_target(driver, url)
summary = refresh_zero_exposure_videos(driver)
except Exception as exc:
print(f"⚠️ 博主 {index} 刷新扫描异常: {exc},继续下一个")
summary = {
"clicked": 0,
"pending": 0,
"zero_without_button": 0,
"unconfirmed": 0,
"pages": 0,
"error": str(exc),
}
refresh_summaries.append(summary)
should_wait = any(
item.get("clicked", 0) > 0 or item.get("pending", 0) > 0
for item in refresh_summaries
)
if should_wait and wait_seconds > 0:
print(f"\n⏳ 已触发曝光更新，等待 {wait_seconds} 秒后刷新导出...")
time.sleep(wait_seconds)
else:
print("\n✅ 没有待更新或更新中的曝光，直接进入导出阶段")
all_records = []
for index, url in enumerate(urls, 1):
print(f"\n{'=' * 50}")
print(f"📌 导出阶段 博主 {index}/{len(urls)}: {url}")
print(f"{'=' * 50}")
try:
navigate_to_target(driver, url)
pre_mtime = _max_excel_mtime()
excel_path = export_video_data(driver, pre_mtime)
if not excel_path:
print(f"⚠️ 博主 {index} 导出失败,跳过")
continue
records, _ = parse_chanmama_excel(excel_path)
print(f" 解析出 {len(records)} 条视频记录")
all_records.extend(records)
except Exception as exc:
print(f"⚠️ 博主 {index} 导出异常: {exc},跳过")
return all_records, refresh_summaries
def export_video_data(driver, pre_mtime: float = 0):
"""导出视频记录数据,返回下载的 Excel 绝对路径。
pre_mtime: 进入导出前 DOWNLOAD_DIR 中已有文件的最大 mtime,用于识别本次新下载的文件。
@@ -554,8 +904,6 @@ def scrape_table_data(driver):
"""备用方案：直接从页面抓取表格数据保存为CSV"""
print("\n📋 正在抓取页面表格数据...")
rows_data = []
try:
# 等待表格加载
table = WebDriverWait(driver, 10).until(
@@ -633,7 +981,7 @@ def scrape_table_data(driver):
writer.writerow(headers + ["链接"])
writer.writerows(all_rows)
print(f"\n✅ 数据抓取完成！")
print("\n✅ 数据抓取完成！")
print(f" 📁 文件路径: {csv_path}")
print(f" 📊 共计 {len(all_rows)} 条记录")
return csv_path
@@ -853,6 +1201,14 @@ def _list_records_by_table(base_token, table_id):
def _write_back(base_token, table_id, record_id, field_id, value):
from gyxx_flow.adapters.acceptance_policy import skip_feishu_table_write
if skip_feishu_table_write(
"content_marketing.chanmama.record-upsert",
details={"table_id": table_id, "record_id": record_id, "field_id": field_id},
):
print(f" [ACCEPTANCE-SKIP] record_id={record_id}")
return True
payload = {field_id: value}
resp = _call_lark_json([
"base", "+record-upsert",
@@ -952,7 +1308,7 @@ def _title_match(haystack, needle):
def backfill_self_tables(excel_records, dry_run=False, only_style=None):
"""遍历自营 mapping,用 excel_records 回填飞书自营表(抖音平台)"""
from gyxx_flow.modules.content_marketing.runtime import feishu_mapping
from gyxx_flow.modules.content_marketing import feishu_mapping
try:
mapping = feishu_mapping.load_mapping(self_operated=True)
except Exception as exc:
@@ -1221,9 +1577,12 @@ def main():
print("\n🌐 启动浏览器...")
driver = create_driver()
# 2. 优先用 cookie 登录(避免每次都触发密码登录的验证码)
logged_in = False
if load_cookies(driver):
# 2. 优先复用独占 Chrome Profile，再尝试显式 cookie 文件。
logged_in = verify_login(driver)
if logged_in:
save_cookies(driver)
print("✨ 使用已复制的浏览器 Profile 登录成功,跳过手动登录!")
elif load_cookies(driver):
print("🔍 正在验证 cookie 有效性...")
if verify_login(driver):
logged_in = True
@@ -1231,37 +1590,31 @@ def main():
# 3. cookie 无效或过期时,回退到账号密码登录
if not logged_in:
from gyxx_flow.adapters.acceptance_policy import current_acceptance_policy
acceptance_policy = current_acceptance_policy()
if acceptance_policy.enabled and (not ACCOUNT or not PASSWORD):
reason = "profile/cookie session is invalid and credential fallback is unavailable"
acceptance_policy.record(
"cookie_skipped",
operation="content.chanmama.login",
details={"status": "SKIPPED_COOKIE", "reason": reason},
)
print(f"[SKIPPED_COOKIE] {reason}", flush=True)
return COOKIE_SKIP_EXIT_CODE
print("\n🔐 cookie 不可用,执行账号密码登录...")
success = login(driver)
if success:
logged_in = True
elif acceptance_policy.enabled:
return COOKIE_SKIP_EXIT_CODE
else:
print("⚠️ 登录可能未成功,但继续尝试后续步骤...")
# 4. 循环每个博主 URL 导出 Excel
for i, url in enumerate(TARGET_URLS, 1):
print(f"\n{'='*50}")
print(f"📌 博主 {i}/{len(TARGET_URLS)}: {url}")
print(f"{'='*50}")
# 记录导出前已有文件的最大 mtime,用于识别本次新下载的文件
pre_mtime = 0
if os.path.exists(DOWNLOAD_DIR):
for f in os.listdir(DOWNLOAD_DIR):
if f.lower().endswith('.xlsx') and not f.startswith('~$'):
pre_mtime = max(pre_mtime,
os.path.getmtime(os.path.join(DOWNLOAD_DIR, f)))
try:
navigate_to_target(driver, url)
excel_path = export_video_data(driver, pre_mtime)
if not excel_path:
print(f"⚠️ 博主 {i} 导出失败,跳过")
continue
records, _ = parse_chanmama_excel(excel_path)
print(f" 解析出 {len(records)} 条视频记录")
all_records.extend(records)
except Exception as exc:
print(f"⚠️ 博主 {i} 异常: {exc},跳过")
continue
# 4. 先刷新两个账号的全部分页，再统一等待并重新进入页面导出。
all_records, _refresh_summaries = refresh_then_export_accounts(
driver, TARGET_URLS, wait_seconds=REFRESH_WAIT_SECONDS
)
print("\n" + "=" * 50)
print(f"🎉 导出完成!共 {len(all_records)} 条 records")
@@ -1289,5 +1642,11 @@ def main():
return 0
def cli() -> None:
"""Run the script and preserve the business exit code for the workflow engine."""
raise SystemExit(main())
if __name__ == "__main__":
main()
cli()
src/gyxx_flow/modules/content_marketing/runtime/collection_completeness.py → src/gyxx_flow/modules/content_marketing/collection_completeness.py
View File
src/gyxx_flow/modules/content_marketing/runtime/creator_task_grouping.py → src/gyxx_flow/modules/content_marketing/creator_task_grouping.py
View File
src/gyxx_flow/modules/content_marketing/daily_creator_exposure_scope.py
+55
View File
@@ -0,0 +1,55 @@
"""Business scope for the scheduled collaborator exposure collection."""
from __future__ import annotations
from collections.abc import Iterable
from datetime import date
DEFAULT_PUBLISHED_FROM = date(2026, 7, 1)
DEFAULT_MAX_AGE_DAYS = 30
EXCLUDED_STYLE_NAMES = frozenset({"盖亚微单", "逐星GT", "晨星2", "觅光"})
def select_daily_styles(
styles: Iterable[dict],
requested_indices: Iterable[int] | None = None,
) -> list[dict]:
"""Return requested styles after applying the daily exclusion list."""
requested = (
{int(index) for index in requested_indices}
if requested_indices is not None
else None
)
return [
style
for style in styles
if (requested is None or int(style["index"]) in requested)
and str(style.get("name") or "").strip() not in EXCLUDED_STYLE_NAMES
]
def classify_publish_scope(
published_on: date | None,
collected_on: date,
*,
published_from: date | None = None,
max_age_days: int | None = None,
) -> str | None:
"""Return a skip reason, or ``None`` when a note should be collected.
A 30-day window means ages 0 through 29 are collected. Once the note is
30 days old it has completed 30 daily snapshots and leaves the queue.
"""
scoped = published_from is not None or max_age_days is not None
if not scoped:
return None
if published_on is None:
return "publish_time_missing"
if published_from is not None and published_on < published_from:
return "before_publish_cutoff"
age_days = (collected_on - published_on).days
if age_days < 0:
return "publish_time_in_future"
if max_age_days is not None and age_days >= max_age_days:
return "collection_window_complete"
return None
src/gyxx_flow/modules/content_marketing/runtime/daily_marketing_report.py → src/gyxx_flow/modules/content_marketing/daily_marketing_report.py
+6 -6
View File
@@ -10,7 +10,7 @@ from collections import Counter
from datetime import date, datetime, timedelta
from pathlib import Path
from typing import Any
from gyxx_flow.modules.content_marketing.runtime.runtime_paths import PATHS
from gyxx_flow.modules.content_marketing.runtime_paths import PATHS
if hasattr(sys.stdout, "reconfigure"):
sys.stdout.reconfigure(encoding="utf-8", errors="replace")
@@ -20,13 +20,13 @@ if hasattr(sys.stderr, "reconfigure"):
ROOT = PATHS.module_root
TOOLS_DIR = PATHS.tools_root
from gyxx_flow.modules.content_marketing.runtime.data.tools.analyze_comments import ( # noqa: E402
from gyxx_flow.modules.content_marketing.data.tools.analyze_comments import ( # noqa: E402
call_hermes_analyzer,
)
from gyxx_flow.modules.content_marketing.runtime.data.tools.db import get_conn # noqa: E402
from gyxx_flow.modules.content_marketing.runtime.data.tools.daily_report_charts import generate_dashboard # noqa: E402
from gyxx_flow.modules.content_marketing.runtime.data.tools.daily_report_card import send_daily_report_cards # noqa: E402
from gyxx_flow.modules.content_marketing.runtime.data.tools.daily_dashboard_analytics import ( # noqa: E402
from gyxx_flow.modules.content_marketing.data.tools.db import get_conn # noqa: E402
from gyxx_flow.modules.content_marketing.data.tools.daily_report_charts import generate_dashboard # noqa: E402
from gyxx_flow.modules.content_marketing.data.tools.daily_report_card import send_daily_report_cards # noqa: E402
from gyxx_flow.modules.content_marketing.data.tools.daily_dashboard_analytics import ( # noqa: E402
build_dashboard_facts,
load_creator_connections,
load_style_categories,
src/gyxx_flow/modules/content_marketing/runtime/data/config/analyze.env.example → src/gyxx_flow/modules/content_marketing/data/config/analyze.env.example
View File
src/gyxx_flow/modules/content_marketing/runtime/data/config/db.env.example → src/gyxx_flow/modules/content_marketing/data/config/db.env.example
+4 -4
View File
@@ -1,8 +1,8 @@
# PG 连接信息模板
# 复制为 db.env 并填真实值 (db.env 已被 .gitignore 排除)
PG_HOST=8.148.185.119
PG_HOST=127.0.0.1
PG_PORT=5432
PG_DB=data_hub
PG_USER=data_hub
PG_PASSWORD=${GYXX_PG_PASSWORD}
PG_DB=gyxx_super_data
PG_USER=gyxx_flow
PG_PASSWORD=
src/gyxx_flow/modules/content_marketing/runtime/data/config/款式_多维表格_对照.json → src/gyxx_flow/modules/content_marketing/data/config/款式_多维表格_对照.json
+2548 -4648
View File
File diff suppressed because it is too large Load Diff
src/gyxx_flow/modules/content_marketing/runtime/data/config/款式_多维表格_对照_自营.json → src/gyxx_flow/modules/content_marketing/data/config/款式_多维表格_对照_自营.json
+1545 -1207
View File
File diff suppressed because it is too large Load Diff
src/gyxx_flow/modules/content_marketing/runtime/data/tools/analyze_comments.py → src/gyxx_flow/modules/content_marketing/data/tools/analyze_comments.py
+16 -3
View File
@@ -20,7 +20,6 @@ import concurrent.futures
import csv
import json
import os
from gyxx_flow.adapters import RuntimeServicePolicy
import random
import re
import subprocess
@@ -30,7 +29,9 @@ import urllib.error
import urllib.request
from pathlib import Path
from typing import Any
from gyxx_flow.modules.content_marketing.runtime.runtime_paths import (
from gyxx_flow.adapters import RuntimeServicePolicy
from gyxx_flow.modules.content_marketing.runtime_paths import (
PATHS,
resolve_layer_output,
)
@@ -377,8 +378,17 @@ def build_batch_report(style_results: list[tuple[str, str, int, int]]) -> str:
# ---------------------------------------------------------------------------
# Hermes 分析端调用
# ---------------------------------------------------------------------------
def _chat_completions_url(configured: str) -> str:
"""Accept either an OpenAI API base URL or the full completions URL."""
normalized = configured.rstrip("/")
if normalized.endswith("/chat/completions"):
return normalized
return normalized + "/chat/completions"
def call_hermes_analyzer(system_prompt: str, user_content: str) -> str:
url = HERMES_ANALYZER_URL.rstrip("/") + "/chat/completions"
url = _chat_completions_url(HERMES_ANALYZER_URL)
payload = {
"model": HERMES_ANALYZER_MODEL,
"messages": [
@@ -733,6 +743,9 @@ def write_summary(summary: str, path: Path) -> Path:
def send_feishu_summary(summary: str, open_id: str) -> dict[str, Any]:
"""通过 lark-cli --profile hermes-analyzer 以分析端应用身份发送汇总。"""
from gyxx_flow.adapters.acceptance_policy import resolve_notification_recipients
open_id = resolve_notification_recipients((open_id,))[0]
text = summary
if len(text) > FEISHU_TEXT_LIMIT:
text = text[:FEISHU_TEXT_LIMIT] + "\n\n...（内容过长，已截断，完整内容见项目 data/summary 汇总文件）"
src/gyxx_flow/modules/content_marketing/runtime/data/tools/analyze_note.py → src/gyxx_flow/modules/content_marketing/data/tools/analyze_note.py
+6 -3
View File
@@ -17,7 +17,7 @@ import sys
import time
from pathlib import Path
from typing import Any
from gyxx_flow.modules.content_marketing.runtime.runtime_paths import (
from gyxx_flow.modules.content_marketing.runtime_paths import (
PATHS,
resolve_layer_output,
)
@@ -26,7 +26,7 @@ from gyxx_flow.modules.content_marketing.runtime.runtime_paths import (
_TOOLS_DIR = PATHS.tools_root
_PROJECT_ROOT = PATHS.module_root
from gyxx_flow.modules.content_marketing.runtime.data.tools.analyze_comments import (
from gyxx_flow.modules.content_marketing.data.tools.analyze_comments import (
HERMES_ANALYZER_MODEL,
HERMES_ANALYZER_TOKEN,
HERMES_ANALYZER_URL,
@@ -40,7 +40,7 @@ from gyxx_flow.modules.content_marketing.runtime.data.tools.analyze_comments imp
call_hermes_analyzer,
)
from gyxx_flow.modules.content_marketing.runtime.data.tools import db
from gyxx_flow.modules.content_marketing.data.tools import db
PROJECT_ROOT = _PROJECT_ROOT
DATA_DIR = PATHS.raw_root
@@ -677,6 +677,9 @@ def send_feishu_report(
Windows 下直接走 lark-cli.cmd 会撞到命令行长度上限且对 &|<>^% 转义敏感，
改为调 node + run.js，args 用 list 传，由 node 自己处理。
"""
from gyxx_flow.adapters.acceptance_policy import resolve_notification_recipients
open_id = resolve_notification_recipients((open_id,))[0]
card = build_card_payload(
info, metrics, distribution,
raw_comments, filtered_comments, analysis,
src/gyxx_flow/modules/content_marketing/runtime/data/tools/batch_rescrape_bilibili.py → src/gyxx_flow/modules/content_marketing/data/tools/batch_rescrape_bilibili.py
+22 -5
View File
@@ -9,15 +9,15 @@ Updates:
import subprocess
import sys
import time
from pathlib import Path
from gyxx_flow.modules.content_marketing.runtime.runtime_paths import PATHS
from gyxx_flow.adapters import COOKIE_SKIP_EXIT_CODE, current_acceptance_policy
from gyxx_flow.modules.content_marketing import bilibili_comment_scraper as scraper
from gyxx_flow.modules.content_marketing.data.tools import db
from gyxx_flow.modules.content_marketing.runtime_paths import PATHS
_TOOLS_DIR = PATHS.tools_root
_PROJECT_ROOT = PATHS.module_root
from gyxx_flow.modules.content_marketing.runtime.data.tools import db
from gyxx_flow.modules.content_marketing.runtime import bilibili_comment_scraper as scraper
RELOGIN_SCRIPT = _TOOLS_DIR / "relogin_bilibili.py"
try:
@@ -83,6 +83,21 @@ def comment_replace_policy(comments: list[dict], stats: dict, metrics: dict) ->
def do_relogin() -> bool:
"""Run relogin_bilibili.py (headed, waits for QR scan). Returns True if successful."""
acceptance_policy = current_acceptance_policy()
if acceptance_policy.enabled:
acceptance_policy.record(
"cookie_skipped",
operation="content.batch_rescrape_bilibili.relogin",
details={
"status": "SKIPPED_COOKIE",
"reason": "cookie is invalid and QR relogin is disabled for acceptance",
},
)
log(
" [SKIPPED_COOKIE] Cookie invalid; acceptance mode will not "
"launch relogin or wait for a QR scan"
)
return False
log(" Cookie 可能过期,触发自动重登 (请扫码)...")
try:
rc = subprocess.call(
@@ -142,6 +157,8 @@ def main(only_ids: list[int] | None = None, dry_run: bool = False) -> int:
relogin_used += 1
consecutive_fail = 0
continue
if current_acceptance_policy().enabled:
return COOKIE_SKIP_EXIT_CODE
continue
consecutive_fail = 0
src/gyxx_flow/modules/content_marketing/runtime/data/tools/batch_rescrape_douyin.py → src/gyxx_flow/modules/content_marketing/data/tools/batch_rescrape_douyin.py
+23 -7
View File
@@ -9,16 +9,16 @@ Updates:
import subprocess
import sys
import time
from pathlib import Path
from gyxx_flow.modules.content_marketing.runtime.runtime_paths import PATHS
from gyxx_flow.adapters import COOKIE_SKIP_EXIT_CODE, current_acceptance_policy
from gyxx_flow.modules.content_marketing import douyin_comment_scraper as scraper
from gyxx_flow.modules.content_marketing.data.tools import db
from gyxx_flow.modules.content_marketing.runtime_paths import PATHS
# Path setup: db.py is in data/tools/, scrapers are in project root
_TOOLS_DIR = PATHS.tools_root
_PROJECT_ROOT = PATHS.module_root
from gyxx_flow.modules.content_marketing.runtime.data.tools import db
from gyxx_flow.modules.content_marketing.runtime import douyin_comment_scraper as scraper
RELOGIN_SCRIPT = _TOOLS_DIR / "relogin_douyin.py"
try:
@@ -99,6 +99,21 @@ def result_logged_in(result: dict | None) -> bool:
def do_relogin() -> bool:
"""Run relogin_douyin.py (headed, waits for QR scan). Returns True if successful."""
acceptance_policy = current_acceptance_policy()
if acceptance_policy.enabled:
acceptance_policy.record(
"cookie_skipped",
operation="content.batch_rescrape_douyin.relogin",
details={
"status": "SKIPPED_COOKIE",
"reason": "cookie is invalid and QR relogin is disabled for acceptance",
},
)
log(
" [SKIPPED_COOKIE] Cookie invalid; acceptance mode will not "
"launch relogin or wait for a QR scan"
)
return False
log(" Cookie 可能过期,触发自动重登 (请扫码)...")
try:
rc = subprocess.call(
@@ -176,8 +191,9 @@ def main(only_ids: list[int] | None = None, dry_run: bool = False) -> int:
consecutive_fail = 0
continue # retry the current URL in next iteration
else:
# 重登失败,走冷却逻辑
pass
# 重登失败,走冷却逻辑；验收模式必须直接跳过，不能等待。
if current_acceptance_policy().enabled:
return COOKIE_SKIP_EXIT_CODE
if consecutive_fail >= MAX_CONSECUTIVE_FAIL:
if cooldown_used < MAX_COOLDOWNS:

Some files were not shown because too many files have changed in this diff Show More