wangyunlong/gyxx-flow
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: consolidate legacy workflows into gyxx-flow

Browse Source
This commit is contained in:
wangyunlong
2026-07-28 14:51:15 +08:00
commit c23b62a8c8
374 changed files with 132990 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+34
View File
@@ -0,0 +1,34 @@
# Python bytecode and tooling
__pycache__/
*.py[cod]
*.egg-info/
.pytest_cache/
.coverage
htmlcov/
# Build and virtual environments
build/
dist/
.venv/
venv/
# Runtime data, local state, and logs
/var/
*.log
# Local credentials and machine-specific configuration
.env
.env.*
!.env.example
config/local/
config/secrets.*
!config/secrets.example.env
# IDE and operating-system files
.idea/
.vscode/
.DS_Store
Thumbs.db
# Agent-local learning logs
.learnings/
README.md
+88
View File
@@ -0,0 +1,88 @@
# GYXX Flow
GYXX Flow 将原来分散在四个项目中的定时工作流和可执行脚本迁入一个可安装、可迁移的模块化项目：
- `content_marketing`：内容与营销采集
- `product_commerce`：商品、平台和经营分析
- `shop_intelligence`：店铺与竞店采集
- `supply_chain`：供应链采集与通知
运行时源码全部位于 `src/gyxx_flow/modules/<module>/runtime`。新项目不会从
`D:\yingxiaoyunying`、`D:\shop-data-flow`、`D:\product-collector-analyze-flow`
或 `E:\auto-flow` 导入或启动代码；这些旧项目仅保留为只读迁移来源和回滚基线。
## 安装与验证
要求 Python 3.12，推荐使用 `uv`：
```powershell
cd D:\gyxx-flow
uv sync --python 3.12 --extra test
uv run pytest
uv run gyxx doctor --json
uv run gyxx acceptance status --json
```
## 运行工作流
```powershell
uv run gyxx list
uv run gyxx run product.daily --date 2026-07-27
uv run gyxx run product.daily --date 2026-07-27 --execute
```
`run`、`backfill` 和 `scripts run` 默认都是无副作用 dry-run；只有显式加
`--execute` 才会启动迁入后的本地脚本。
## 运行任意迁入脚本
```powershell
uv run gyxx scripts list
uv run gyxx scripts list --module content_marketing
uv run gyxx scripts run content_marketing:run_all.py --date 2026-07-27
uv run gyxx scripts run content_marketing:run_all.py --date 2026-07-27 --execute
```
脚本 ID 格式为 `<module>:<runtime 内相对路径>`。Python、BAT/CMD 和
PowerShell 入口均受统一项目根、数据根、业务日期、run_id 和 shadow 环境约束。
## 浏览器与外部系统绑定
`config/runtime-bindings.json` 为当前 131 个脚本各自分配固定且唯一的 CDP 端口。
无论从 workflow、`scripts run` 还是嵌套脚本启动，目标脚本都会重新取得自己的端口和
`state/browser/<module>/<script>/` 下的 Profile、Cookie、storage state；同一脚本下次
运行会复用登录态，不同脚本不会共享浏览器状态。
飞书继续走原来的 lark-cli/OpenAPI 身份；数据库继续走现有云端 PostgreSQL；Hermes
继续走本机服务。运行时会拒绝 localhost 数据库和非本机 Hermes 地址。不要在源码或
`runtime-bindings.json` 中写入 Cookie、数据库密码、飞书密钥或 Hermes token。
## 数据目录
默认数据根为 `D:\gyxx-flow\var`，可用 `GYXX_DATA_ROOT` 迁移到其他磁盘。
业务数据按模块写入 `data/raw`、`data/normalized`、`data/curated`、`data/exports`
和 `data/evidence`；运行状态、日志和临时文件分别进入 `state`、`logs` 和 `tmp`。
原始 JSON/CSV/XLS/XLSX/下载文件进 raw，清洗结果进 normalized，聚合数据进 curated，
最终 Markdown/Excel 报告进 exports。浏览器 Profile、Cookie 和 storage state 位于
`state/browser/<module>/<script>/` 并按脚本复用。现有 `data/raw/<module>/legacy`
历史数据只读保留，源码与可变数据不混放。
## 定时任务
```powershell
uv run gyxx schedule plan `
--output D:\gyxx-flow\var\schedule-plan\candidate `
--start-date 2026-07-27 `
--python-executable D:\gyxx-flow\.venv\Scripts\python.exe
```
该命令只生成 21 个任务定义和审核脚本，不会注册、禁用或修改任何系统任务。
生产切换必须遵循 `docs/migration-runbook.md` 的逐任务门禁。
## 关键文件
- `design.md`：目标架构和迁移边界
- `plan.md`：逐项验收清单
- `config/workflows.json`：工作流到新项目本地入口的映射
- `config/source-manifests/`：四个来源的逐文件哈希与处置清单
- `docs/`：部署、运行、迁移和回滚手册
config/runtime-bindings.json
+143
View File
@@ -0,0 +1,143 @@
{
"schema_version": 1,
"cdp_host": "127.0.0.1",
"scripts": {
"content_marketing:bilibili_comment_scraper.py": 22000,
"content_marketing:bilibili_scraper.py": 22001,
"content_marketing:chanmama_scraper.py": 22002,
"content_marketing:daily_marketing_report.py": 22003,
"content_marketing:data/tools/analyze_comments.py": 22004,
"content_marketing:data/tools/analyze_note.py": 22005,
"content_marketing:data/tools/batch_rescrape_bilibili.py": 22006,
"content_marketing:data/tools/batch_rescrape_douyin.py": 22007,
"content_marketing:data/tools/batch_rescrape_xiaohongshu.py": 22008,
"content_marketing:data/tools/check_status.py": 22009,
"content_marketing:data/tools/collect_note_metrics.py": 22010,
"content_marketing:data/tools/daily_marketing_report.bat": 22011,
"content_marketing:data/tools/daily_run.bat": 22012,
"content_marketing:data/tools/daily_run_with_backfill.bat": 22013,
"content_marketing:data/tools/db.py": 22014,
"content_marketing:data/tools/feishu_comment_batch.py": 22015,
"content_marketing:data/tools/feishu_doc_writer.py": 22016,
"content_marketing:data/tools/friday_relogin.bat": 22017,
"content_marketing:data/tools/friday_relogin_parallel.py": 22018,
"content_marketing:data/tools/generate_creator_report.py": 22019,
"content_marketing:data/tools/init_db.py": 22020,
"content_marketing:data/tools/kill_project_chrome.ps1": 22021,
"content_marketing:data/tools/list_project_chrome.ps1": 22022,
"content_marketing:data/tools/monday_self_run.bat": 22023,
"content_marketing:data/tools/monthly_creator_report.bat": 22024,
"content_marketing:data/tools/monthly_summary.bat": 22025,
"content_marketing:data/tools/pull_all_tables.py": 22026,
"content_marketing:data/tools/rebuild_mapping.py": 22027,
"content_marketing:data/tools/relogin_bilibili.py": 22028,
"content_marketing:data/tools/relogin_douyin.py": 22029,
"content_marketing:data/tools/relogin_pgy.py": 22030,
"content_marketing:data/tools/relogin_xiaohongshu.py": 22031,
"content_marketing:data/tools/relogin_xingtu.py": 22032,
"content_marketing:data/tools/retry_failed.py": 22033,
"content_marketing:data/tools/sync_cooperations.bat": 22034,
"content_marketing:data/tools/sync_cooperations.py": 22035,
"content_marketing:data/tools/sync_metrics_to_cmt_notes.py": 22036,
"content_marketing:data/tools/sync_style_categories.py": 22037,
"content_marketing:data/tools/validate_mapping.py": 22038,
"content_marketing:data/tools/weekly_comment_scrape.bat": 22039,
"content_marketing:data/tools/weekly_summary.bat": 22040,
"content_marketing:data/tools/write_notes_to_doc.py": 22041,
"content_marketing:douyin_comment_scraper.py": 22042,
"content_marketing:feishu_mapping.py": 22043,
"content_marketing:login_helper.py": 22044,
"content_marketing:monthly_summary_all.py": 22045,
"content_marketing:pgy_xhs_scraper.py": 22046,
"content_marketing:pgy_xhs_scraper_v2.py": 22047,
"content_marketing:run_all.py": 22048,
"content_marketing:self_bilibili_scraper.py": 22049,
"content_marketing:self_douyin_scraper.py": 22050,
"content_marketing:weekly_summary_all.py": 22051,
"content_marketing:weekly_summary_xingyun2.py": 22052,
"content_marketing:xiaohongshu_comment_scraper.py": 22053,
"content_marketing:xingtu_scraper.py": 22054,
"content_marketing:xingtu_scraper_v2.py": 22055,
"product_commerce:aggregate_daily_final.py": 22056,
"product_commerce:analyze_style_with_hermes.py": 22057,
"product_commerce:backfill_poseidon_sales.py": 22130,
"product_commerce:backfill_collect.py": 22058,
"product_commerce:backfill_one_day.py": 22059,
"product_commerce:check_nine_day_decline.py": 22060,
"product_commerce:collect_dy_market_rank.py": 22061,
"product_commerce:collect_dy_persona_to_bitable.py": 22062,
"product_commerce:collect_erp_yesterday_metrics.py": 22063,
"product_commerce:collect_jd_market_rank.py": 22064,
"product_commerce:collect_jd_persona_to_bitable.py": 22065,
"product_commerce:collect_persona_to_bitable.py": 22066,
"product_commerce:collect_sycm_market_rank.py": 22067,
"product_commerce:commands/import_daily.py": 22068,
"product_commerce:commands/run_alerts.py": 22069,
"product_commerce:db/sync_dim_style.py": 22070,
"product_commerce:db/sync_sku_master.py": 22071,
"product_commerce:dy_audience_profile_collect.py": 22072,
"product_commerce:dy_product_scraping.py": 22073,
"product_commerce:erp_login_product_analysis.py": 22074,
"product_commerce:export_bitable_records.py": 22075,
"product_commerce:import_product_daily.py": 22076,
"product_commerce:import_product_reviews.py": 22077,
"product_commerce:insert_bitable_records.py": 22078,
"product_commerce:jd_main_image_collector.py": 22079,
"product_commerce:jd_product_data_collector.py": 22080,
"product_commerce:jd_self_inventory_sales_collector.py": 22081,
"product_commerce:orchestrate_daily_collection.py": 22082,
"product_commerce:orchestrate_market_rank_collection.py": 22083,
"product_commerce:orchestrate_review_collection.py": 22084,
"product_commerce:reapply_erp_override.py": 22085,
"product_commerce:rebuild_market_rank_documents.py": 22086,
"product_commerce:run_alerts_with_retry.py": 22087,
"product_commerce:run_daily_persona.py": 22088,
"product_commerce:run_weekly_jd_main_image.py": 22089,
"product_commerce:run_weekly_main_image.py": 22090,
"product_commerce:scripts/insert_jd_main_image_records.py": 22091,
"product_commerce:scripts/insert_main_image_records.py": 22092,
"product_commerce:taobao_dmp_item_crowd_insight_screenshots.py": 22093,
"product_commerce:taobao_sycm_collect.py": 22094,
"product_commerce:taobao_sycm_collect_backfill.py": 22095,
"product_commerce:taobao_sycm_products.py": 22096,
"product_commerce:taobao_wanxiang_ai_creative_report.py": 22097,
"product_commerce:upload_video_to_guanghe.py": 22098,
"product_commerce:vendors/dy-data-flow/dy_store_competitor_store_scraping.py": 22099,
"product_commerce:vendors/jd-data-flow/jd_data_collector.py": 22100,
"product_commerce:vendors/jd-data-flow/jd_peer_product_data_collector.py": 22101,
"product_commerce:vendors/jd-data-flow/jd_product_data_collector.py": 22102,
"product_commerce:weekly_aggregate.py": 22103,
"shop_intelligence:collectors/dy_store_competitor_store_scraping.py": 22104,
"shop_intelligence:collectors/jd_data_collector.py": 22105,
"shop_intelligence:collectors/jd_peer_store_data_collector.py": 22106,
"shop_intelligence:collectors/taobao_sycm.py": 22107,
"shop_intelligence:runners/run_peer_store.py": 22108,
"shop_intelligence:runners/run_shop.py": 22109,
"shop_intelligence:scripts/remove_scheduler.ps1": 22110,
"shop_intelligence:scripts/setup_scheduler.ps1": 22111,
"supply_chain:orchestrator/mcp_workflow.py": 22112,
"supply_chain:orchestrator/monitor.py": 22113,
"supply_chain:orchestrator/runner.py": 22114,
"supply_chain:orchestrator/scripts/ProductReplenishment.py": 22115,
"supply_chain:orchestrator/scripts/PurchaseConfirmation.py": 22116,
"supply_chain:orchestrator/scripts/PurchaseOrderUpdate.py": 22117,
"supply_chain:orchestrator/scripts/batch_process.py": 22118,
"supply_chain:orchestrator/scripts/collect_confirmation.ps1": 22119,
"supply_chain:orchestrator/scripts/collect_purchase_order_update.ps1": 22120,
"supply_chain:orchestrator/scripts/collect_replenishment.ps1": 22121,
"supply_chain:orchestrator/scripts/insert_replenishment_bitable.py": 22122,
"supply_chain:orchestrator/scripts/send_card_notification.py": 22123,
"supply_chain:orchestrator/scripts/trigger_purchase_order_update.py": 22124,
"supply_chain:orchestrator/sql/backfill_history.py": 22125,
"supply_chain:run.py": 22126,
"supply_chain:scripts/purchase-confirmation.bat": 22127,
"supply_chain:scripts/replenishment-alert.bat": 22128,
"supply_chain:scripts/replenishment.bat": 22129
},
"services": {
"feishu": "legacy",
"postgres": "cloud",
"hermes": "local",
"hermes_url": "http://127.0.0.1:8642/v1/chat/completions"
}
}
config/schedules.json
+30
View File
@@ -0,0 +1,30 @@
{
"schema_version": 1,
"timezone": "Asia/Shanghai",
"schedules": [
{"workflow_id":"content.metrics.daily","kind":"daily","at":"06:00"},
{"workflow_id":"content.cooperations.daily","kind":"daily","at":"09:00"},
{"workflow_id":"content.marketing_report.daily","kind":"daily","at":"10:00"},
{"workflow_id":"content.relogin.weekly","kind":"weekly","days":["Friday"],"at":"10:00"},
{"workflow_id":"content.self_operated.weekly","kind":"weekly","days":["Monday"],"at":"13:00"},
{"workflow_id":"content.comments.weekly","kind":"weekly","days":["Sunday"],"at":"12:00"},
{"workflow_id":"content.summary.weekly","kind":"weekly","days":["Tuesday"],"at":"10:00"},
{"workflow_id":"content.summary.monthly","kind":"monthly","day_of_month":1,"at":"08:00"},
{"workflow_id":"content.creator_report.monthly","kind":"monthly","day_of_month":1,"at":"08:30"},
{"workflow_id":"shop.metrics.weekly","kind":"weekly","days":["Monday"],"at":"12:00"},
{"workflow_id":"shop.competitor.weekly","kind":"weekly","days":["Monday"],"at":"12:30"},
{"workflow_id":"product.daily","kind":"daily","at":"08:40"},
{"workflow_id":"product.persona.daily","kind":"daily","at":"10:00"},
{"workflow_id":"product.import.daily","kind":"daily","at":"19:00"},
{"workflow_id":"product.alert.daily","kind":"daily","at":"23:00"},
{"workflow_id":"product.style_analysis.interval","kind":"interval_days","every_days":3,"anchor_date":"2026-07-25","at":"11:00"},
{"workflow_id":"product.main_image.jd.weekly","kind":"weekly","days":["Sunday"],"at":"08:30"},
{"workflow_id":"product.main_image.weekly","kind":"weekly","days":["Sunday"],"at":"09:30"},
{"workflow_id":"supply.replenishment_alert.daily","kind":"daily","at":"07:00"},
{"workflow_id":"supply.purchase_confirmation.daily","kind":"daily","at":"08:00"},
{"workflow_id":"supply.replenishment.weekly","kind":"weekly","days":["Monday"],"at":"08:00"}
]
}
config/secrets.example.env
+5
View File
@@ -0,0 +1,5 @@
GYXX_POSTGRES_DSN=
GYXX_POSTGRES_PASSWORD=
GYXX_FEISHU_APP_ID=
GYXX_FEISHU_APP_SECRET=
GYXX_HERMES_API_KEY=
config/source-manifests/content_marketing.json
+886
View File
@@ -0,0 +1,886 @@
{
"schema_version": 1,
"module": "content_marketing",
"snapshot": "current-filesystem",
"files": [
{
"source_relative_path": "bilibili_comment_scraper.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/bilibili_comment_scraper.py",
"source_sha256": "c7f116f30889125c668e7f4dd796e7249ba7ea8621d5ab5fb518028f56c7f8dc",
"target_sha256": "e03b166f7b3b1b6fb6ebabbb340b0670d2256d8543e74d7b1559a8f862c0725f",
"transformed": true,
"category": "source"
},
{
"source_relative_path": "bilibili_scraper.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/bilibili_scraper.py",
"source_sha256": "5e95e710f016972fdb886782bdc2eb0f33a0105d23d46fbc0762549c5e42f67d",
"target_sha256": "8574359731fa2807c1716b41342a9be09e9514f02792c3fed03ed46d36300499",
"transformed": true,
"category": "source"
},
{
"source_relative_path": "chanmama_scraper.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/chanmama_scraper.py",
"source_sha256": "0e295b4624c2a020ae85e12e6ba1087e670363945d1a855a5625d22ca419e7f1",
"target_sha256": "497bbf3d0087047af1a3c8b0b866979c5bc67aa304a23a97c69fe3f79d020345",
"transformed": true,
"category": "source"
},
{
"source_relative_path": "collection_completeness.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/collection_completeness.py",
"source_sha256": "f23b70b949f8adcaf00ee62aa0f9f39314bc623528cad9a77591641da765991d",
"target_sha256": "f23b70b949f8adcaf00ee62aa0f9f39314bc623528cad9a77591641da765991d",
"transformed": false,
"category": "source"
},
{
"source_relative_path": "creator_task_grouping.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/creator_task_grouping.py",
"source_sha256": "d4ef307039726b325bbb1267d6e711bdb1c254f06cdbc967c8b4878dfdd7bad8",
"target_sha256": "d4ef307039726b325bbb1267d6e711bdb1c254f06cdbc967c8b4878dfdd7bad8",
"transformed": false,
"category": "source"
},
{
"source_relative_path": "daily_marketing_report.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/daily_marketing_report.py",
"source_sha256": "22456210632d34cc6e60e4302641fe3ce581353a41970eb9c5613dbeb135ba31",
"target_sha256": "61acda5e902b118257251975dcf13332f9894834e4f9f1b030c69066de0e83c1",
"transformed": true,
"category": "source"
},
{
"source_relative_path": "data/config/analyze.env.example",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/config/analyze.env.example",
"source_sha256": "b60328ef177b795489d0a57957133dedcfd67b38b59603a6edeee1cf9babcee2",
"target_sha256": "b60328ef177b795489d0a57957133dedcfd67b38b59603a6edeee1cf9babcee2",
"transformed": false,
"category": "config"
},
{
"source_relative_path": "data/config/db.env.example",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/config/db.env.example",
"source_sha256": "15f9f181ecae6dfc4eedb03a18ed3379ac8a55b83a32ee169b339427acb712d7",
"target_sha256": "4b28f3d29d504d5c18784105cd989a8604aa18af18a63a5b49499aae94b4e9e7",
"transformed": true,
"category": "config"
},
{
"source_relative_path": "data/config/款式_多维表格_对照.json",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/config/款式_多维表格_对照.json",
"source_sha256": "8a16d2f697f8db65240623a682610b4f73dddbfb857b9f6579aa408bd00a6d25",
"target_sha256": "8a16d2f697f8db65240623a682610b4f73dddbfb857b9f6579aa408bd00a6d25",
"transformed": false,
"category": "config"
},
{
"source_relative_path": "data/config/款式_多维表格_对照_自营.json",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/config/款式_多维表格_对照_自营.json",
"source_sha256": "e46f5ad98dcdef7de64a512f1dcac244daf5c6c607851a8183518c8c8f2f73ce",
"target_sha256": "e46f5ad98dcdef7de64a512f1dcac244daf5c6c607851a8183518c8c8f2f73ce",
"transformed": false,
"category": "config"
},
{
"source_relative_path": "data/tools/analyze_comments.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/analyze_comments.py",
"source_sha256": "61677e64d3e864b73573e3e59b7a247bcfddcde93f78f81a20ee27257c326107",
"target_sha256": "7244b6101ad3e44cb4c7d564728fa007c5710f5dfc5f41f31a4cf2d706e5c24c",
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/analyze_note.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/analyze_note.py",
"source_sha256": "cdf981d21f30d31338b698a31fd44fea97c4c08e04750956ac6f1afb9ed56905",
"target_sha256": "ee2d1b6560c2939e70330949fbef8bf9958bfcce229055b683fc8c8f6519fe30",
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/batch_rescrape_bilibili.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/batch_rescrape_bilibili.py",
"source_sha256": "31d015a53d8b25728e687d4a5db78673bd8dbd25e470e6e8c728a215147d00f1",
"target_sha256": "7b66e93c7f98c5a1210409456473f893be318c36392b4244bce6014d3dba9501",
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/batch_rescrape_douyin.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/batch_rescrape_douyin.py",
"source_sha256": "261b3b122708deb40d8587f27fada55cc0cd234a2ddd085af4cf2f2e2871e05a",
"target_sha256": "f53b02362651d7dcdaeba22a903714d95d3173758ac4a5c3b2bf53cab09fb7cd",
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/batch_rescrape_xiaohongshu.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/batch_rescrape_xiaohongshu.py",
"source_sha256": "6c50845f568b5e51038229873aac977c1bc37f31dbe90f7ce10218d0358422ea",
"target_sha256": "dee67a2639e6e0ecac53d6a3961f9dc94538e2bea0d41621639a11a5a6e88d6c",
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/check_id_filled.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/check_id_filled.py",
"source_sha256": "69e13df9ded14dc907a8ef13aa98808f9e7df6d18f6a50741a950ebcb826fcd9",
"target_sha256": "69e13df9ded14dc907a8ef13aa98808f9e7df6d18f6a50741a950ebcb826fcd9",
"transformed": false,
"category": "tool"
},
{
"source_relative_path": "data/tools/check_status.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/check_status.py",
"source_sha256": "b8aff93d9a4925bda966f1b458f974131827b59a62e85b73a10bac9d6afee072",
"target_sha256": "29641c917a27c27e0e5a29130100b1ba3611a68855175b58754d54c1211c19cc",
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/collect_note_metrics.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/collect_note_metrics.py",
"source_sha256": "30f40ae47baaf520001a20a0f1ef85472a999fca69127ad06d07bd697d3353bc",
"target_sha256": "67fe86573dd1a6ae21087987dfd730d677cc472c65adf0c240d5256e8cfe5d22",
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/daily_dashboard_analytics.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/daily_dashboard_analytics.py",
"source_sha256": "6f0bd4b918691255290880aacaf82fd1ea16344204b6d80d5a85689448312985",
"target_sha256": "736ab32fdc97c7ae576bdcc0351bbf393789441b054d1b14f3c1f3043ab49dc0",
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/daily_marketing_report.bat",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/daily_marketing_report.bat",
"source_sha256": "3523e30935a0536cc6bf3ea3d3a6922cf35e6602d48835392f5383965351b222",
"target_sha256": "cd7b165b2c6aca34c27c297a70c7c0f767d0972abaa0d502928b2e03a106538d",
"transformed": true,
"category": "launcher"
},
{
"source_relative_path": "data/tools/daily_report_card.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/daily_report_card.py",
"source_sha256": "3229a5b4ec321ade3543a5fe172231af95e1874a690122258866f58740067855",
"target_sha256": "fe6dd6501098a003848d8db7930b8b9c5abcc5ab1a1f1a16a7d844d7e543ae5c",
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/daily_report_charts.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/daily_report_charts.py",
"source_sha256": "0fc15ec6114bcd46c51d7e21df23f850943a24acce2df408d795f077241c7ec4",
"target_sha256": "44381624572846b1c16b5a5e8fae9cafbd6ec86cc2918b9f7bf4fe935d3ca11e",
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/daily_run.bat",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/daily_run.bat",
"source_sha256": "b322948fe220c6196c11640e0b7b2f4fdaed9a6b82cc4f77a9112fe3190a01fd",
"target_sha256": "f5967ebd3002e55962229ec91c08ca0f5fb614e3710a0065e1c733ff2398fdf1",
"transformed": true,
"category": "launcher"
},
{
"source_relative_path": "data/tools/daily_run_with_backfill.bat",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/daily_run_with_backfill.bat",
"source_sha256": "073250aab8faee6a70c9bcf06b32745b1dc2e69d8c66c546cc91f38280ca661b",
"target_sha256": "e170578755867526a8da40e0a0b9fee1d5d7ed8f41c87980618477d0ea9b7ef8",
"transformed": true,
"category": "launcher"
},
{
"source_relative_path": "data/tools/db.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/db.py",
"source_sha256": "cb2e3c6a8b44d11668e8fdb6fc0acf8b73adf280a4e19b8f17547edc99ef0089",
"target_sha256": "1a8f293dbf3075ef509babf28e47404b75ac4b791e2a6b7495df5cbd3a01e5d7",
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/feishu_comment_batch.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/feishu_comment_batch.py",
"source_sha256": "2b04fde410eec1dece0b995d0b74bfad4bdc1a53669f19ead2a6fc13ecdd00b7",
"target_sha256": "148c289e7956070274f4f9c86f26a10c4750c9ad7471ee852dcda2be823e77fe",
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/feishu_doc_writer.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/feishu_doc_writer.py",
"source_sha256": "0ec54d855ae842b52d0897c782d2f4d9d696182bc6b8ffaf2e7973b615ac814f",
"target_sha256": "9c3f552b9b466d95bdab7f66800f1f9701b1e81eb1d0316a735553238c7e0423",
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/friday_relogin.bat",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/friday_relogin.bat",
"source_sha256": "26b77ae7575cee43ec055e7dc1b380336c4c75a29bbe8a583c6fb979761b2dc3",
"target_sha256": "6ad19193b928d23852e2173f98ac779b129acd2f6d2f5814417b40f970599809",
"transformed": true,
"category": "launcher"
},
{
"source_relative_path": "data/tools/friday_relogin_parallel.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/friday_relogin_parallel.py",
"source_sha256": "75b48f49aeb4ef01c0f9edaaa24e4dd9633284514e390af1b1e10dc8b9b29a56",
"target_sha256": "a85c70397a0daae11bfd6df68b5e91ad1f6f0f85ec766101ecafcf80ea12d6a2",
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/generate_creator_report.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/generate_creator_report.py",
"source_sha256": "84f5404d45e05d5d33f61f29c123cb75f764f43ad4b04e83776e98228883dea7",
"target_sha256": "2c8d2060918885ff7853542f011bbad927df1cd34a5e50d79875dbc04301a673",
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/init_db.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/init_db.py",
"source_sha256": "5dd41fcd437d957b24ecaf4579672dd2317145b44c44bbe308fdcfa8a94ec667",
"target_sha256": "1b90db140fb0aad6ddd3b6d9aa45c35099b04ee1890c87dd02972c9bc1981058",
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/kill_project_chrome.ps1",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/kill_project_chrome.ps1",
"source_sha256": "30faa3d2e59bcd691bbbf946603da687db865cb14e31ad129b91248d28853ce8",
"target_sha256": "703749108c739b0dccfe73f8e6c355934146e1d8c29473d9c237fe2c8ac5e9f7",
"transformed": true,
"category": "launcher"
},
{
"source_relative_path": "data/tools/list_project_chrome.ps1",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/list_project_chrome.ps1",
"source_sha256": "f3678a415f8068c97bc3315a25dd5c74e5dffe48e8be766fa85fd7f643c053be",
"target_sha256": "ff90f3da8c4c79354b694f032a0e2d0953b034dd0864237cd53fa75d5f8cebda",
"transformed": true,
"category": "launcher"
},
{
"source_relative_path": "data/tools/migrations/001_creators_and_cooperations.sql",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/migrations/001_creators_and_cooperations.sql",
"source_sha256": "2a482079b9c65147f2f878f7ab2d855cf69a90086ed24da1f696e360b5a2684f",
"target_sha256": "2a482079b9c65147f2f878f7ab2d855cf69a90086ed24da1f696e360b5a2684f",
"transformed": false,
"category": "sql"
},
{
"source_relative_path": "data/tools/migrations/003_creator_report.sql",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/migrations/003_creator_report.sql",
"source_sha256": "a66b1ec9f9540a3ce67cb62bb0998df1e160664c5cc1dc752dbb24dbab97190c",
"target_sha256": "a66b1ec9f9540a3ce67cb62bb0998df1e160664c5cc1dc752dbb24dbab97190c",
"transformed": false,
"category": "sql"
},
{
"source_relative_path": "data/tools/migrations/004_style_category.sql",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/migrations/004_style_category.sql",
"source_sha256": "480b1386f6821b22a89018934fe1625994c7c6513b4e7a7afab4fa53ccfb4167",
"target_sha256": "480b1386f6821b22a89018934fe1625994c7c6513b4e7a7afab4fa53ccfb4167",
"transformed": false,
"category": "sql"
},
{
"source_relative_path": "data/tools/migrations/005_style_product_profile.sql",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/migrations/005_style_product_profile.sql",
"source_sha256": "8f9c4d25681e84e582d3cdcd9108ebb16c2f5b777b766a5c3653a21c0baf6d91",
"target_sha256": "8f9c4d25681e84e582d3cdcd9108ebb16c2f5b777b766a5c3653a21c0baf6d91",
"transformed": false,
"category": "sql"
},
{
"source_relative_path": "data/tools/monday_self_run.bat",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/monday_self_run.bat",
"source_sha256": "2806fb1a0c4a2d8a5d0b260da9fe96de30ab3eccea27d28dd2af90c39fc16261",
"target_sha256": "0f6eace49e9d58122946450449580513f8218284459ded4587553f8021de9018",
"transformed": true,
"category": "launcher"
},
{
"source_relative_path": "data/tools/monthly_creator_report.bat",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/monthly_creator_report.bat",
"source_sha256": "59f6e216ccca8709bc334e18a692744172fcea9cb521061705a45d708d800845",
"target_sha256": "794dab2943ab7dc0d85e0c8b9284487bb54d813c9a34697e3d1e1c3b521866e5",
"transformed": true,
"category": "launcher"
},
{
"source_relative_path": "data/tools/monthly_summary.bat",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/monthly_summary.bat",
"source_sha256": "7fd4265a38a5726553d26efcb5a68984717d0d57a45e308dd63274cafdcb283e",
"target_sha256": "ad3c75b7adf7bf21f414472275142bd1dca0f87517cc195846c991145782b0f2",
"transformed": true,
"category": "launcher"
},
{
"source_relative_path": "data/tools/pull_all_tables.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/pull_all_tables.py",
"source_sha256": "be331541a235be1ffb0f1d2f8583d2f5a7aa66ac324631ec6adeb0aaf988dd0c",
"target_sha256": "741717cb66afba136fd3448a4d2009d5857ea4230b99ab79e1e2090c342567f4",
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/rebuild_mapping.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/rebuild_mapping.py",
"source_sha256": "fe946418ebeb30d982ed667c4b7b08052a3fe9aba291eb62c1e726dc53854a3c",
"target_sha256": "56b9128e7324383fe072937704854fbd281f7827b90180a8c8daacfd93db0f48",
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/relogin_bilibili.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/relogin_bilibili.py",
"source_sha256": "b8dc12539bcd6f1a368d50ead1f5d33d6577a75c92f0b7bafa75a67edd7a73fa",
"target_sha256": "bbdab7b1c7ac113aa9c2c2cfdfe456654ec4749788465f2517123d11d9ce85d1",
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/relogin_douyin.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/relogin_douyin.py",
"source_sha256": "3c2017f8a801528ff17e4c4bae24615256b8788c185f9a797fe87470e35b1645",
"target_sha256": "13ea919204a7e5bcba61ba1b5373bd36d495a1275bc93a6a8d605c25dd261e9a",
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/relogin_pgy.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/relogin_pgy.py",
"source_sha256": "759c6a68a0851b4c87d8febfc5d94925831a3b2d7cdf1e6a6e74dcd52b83ae3f",
"target_sha256": "2b208c0adb5aca1241714836c9afac5441f6ffe0302022bfd6934a320ed78946",
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/relogin_transaction.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/relogin_transaction.py",
"source_sha256": "57ec20de8d99290268ed3d1fc2b66bf3faeaca807d7f3744dd51de6cc93628b2",
"target_sha256": "57ec20de8d99290268ed3d1fc2b66bf3faeaca807d7f3744dd51de6cc93628b2",
"transformed": false,
"category": "tool"
},
{
"source_relative_path": "data/tools/relogin_xiaohongshu.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/relogin_xiaohongshu.py",
"source_sha256": "fcdea30adecb066c4f3699ded457df0bc763d978248276ddbffa9f18bc9bdf2c",
"target_sha256": "9314c1ef13ca7d45643a1b1e9b9c21c93fb5c85e253fc4fd7ed0dc4f71b67df1",
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/relogin_xingtu.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/relogin_xingtu.py",
"source_sha256": "c9d434cb02a57ebd7b884598563077d733d56e45dad59ed30774eb7e1c668cb0",
"target_sha256": "80577ab8f539efa72e82bc4752bde8caaf33cde87068c36fd628d9547e227132",
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/retry_failed.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/retry_failed.py",
"source_sha256": "90c6dbc47317cc48e28071d452d75c95913138fc885a5963f9eb58986cd99086",
"target_sha256": "70e1f51bcf48cdac0f620cb1505bdacc1b23289b6ba9ab73d5383b93b904e5ee",
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/schema_gyxx_super_data.sql",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/schema_gyxx_super_data.sql",
"source_sha256": "7d087858c183e2f8dce90842dec663c42d67a37a61215dc14bee48ee053caaec",
"target_sha256": "7d087858c183e2f8dce90842dec663c42d67a37a61215dc14bee48ee053caaec",
"transformed": false,
"category": "sql"
},
{
"source_relative_path": "data/tools/style_analyzer.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/style_analyzer.py",
"source_sha256": "f58aa4ed597c225dcc105fad712e28c2c60a7dcd3edc93cf29b905c3d9e7e3c9",
"target_sha256": "0c059690e3f62a50139316d12b1bfb967b6bc58fc1dce477363340bc8c6e844f",
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/sync_cooperations.bat",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/sync_cooperations.bat",
"source_sha256": "ce1cc5b9b6a2a037b68121106263c99d3e4c263a2028c71885acc943948ed466",
"target_sha256": "8e4a8eef39ccd9ea6294d80ce2e5e5028a27192a39637484b3f613d33037640b",
"transformed": true,
"category": "launcher"
},
{
"source_relative_path": "data/tools/sync_cooperations.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/sync_cooperations.py",
"source_sha256": "e2a527a456847c88981fa6f4bba622b6e45f7782c94bf44459e08f9715e59105",
"target_sha256": "2b2b2f98951d3e608f695e99d39615bee5c84eae04c464373d854b20f8fb1fa7",
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/sync_metrics_to_cmt_notes.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/sync_metrics_to_cmt_notes.py",
"source_sha256": "3b97cab0740c5c97b47a0fbe763acb5391d7c8e47c95f4675720b4f716f6432f",
"target_sha256": "f46874a7e7b6e50bafd23190c378fc6d898397458fd5b1baad1cf864513128cc",
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/sync_style_categories.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/sync_style_categories.py",
"source_sha256": "931e70550275ffb33161bbc4cbe527d5b8eff49b4265fda53aa9a5f46947f204",
"target_sha256": "524570df5217d8f93a9e1c5ff5f55b8009d13de2d872c587fdc82aafff08c272",
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/v2_filename.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/v2_filename.py",
"source_sha256": "60bd6da02baa65e80b60f4fc89fd5f76de26f26566e0cf87a4dcc30e077c379e",
"target_sha256": "60bd6da02baa65e80b60f4fc89fd5f76de26f26566e0cf87a4dcc30e077c379e",
"transformed": false,
"category": "tool"
},
{
"source_relative_path": "data/tools/validate_mapping.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/validate_mapping.py",
"source_sha256": "1e5b792da893b7dd1a29d6f6a73146a4d25079ac8bf37cf15ab4816d64d72cfe",
"target_sha256": "8a079b9083aa6fa6a39bc7c893d20a15cc03d938019911f2382e9132f11266fa",
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "data/tools/weekly_comment_scrape.bat",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/weekly_comment_scrape.bat",
"source_sha256": "caa4118ad2e674d8efc51b0f79844ff45000e5ba995645268d7639abafa423e2",
"target_sha256": "fb36cc77ab6ba40c1cd0e7690d5993d1b3ebd59398b640be3a8071c196099760",
"transformed": true,
"category": "launcher"
},
{
"source_relative_path": "data/tools/weekly_summary.bat",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/weekly_summary.bat",
"source_sha256": "fb54595955743300b753ed95c6e78914d8a893d0e53c8f633fa1593ad89b54b6",
"target_sha256": "52a580fb779444854eebe0b0305738548812f33f2f7de4dc731417b77c6199d5",
"transformed": true,
"category": "launcher"
},
{
"source_relative_path": "data/tools/write_notes_to_doc.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/data/tools/write_notes_to_doc.py",
"source_sha256": "7c2eec1d8e89d9eb4c90e7cb4903eb50233c5b6932c18c727bbc2d211828230d",
"target_sha256": "14387cf507c860d7cd6d0b536a77aeecf40155ce12675629ea013882669566bb",
"transformed": true,
"category": "tool"
},
{
"source_relative_path": "douyin_comment_scraper.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/douyin_comment_scraper.py",
"source_sha256": "ec785ead9c95c5de65c00b06212b1f03d0159fed475a95a2b593d3b24e486e43",
"target_sha256": "0f8af0630e40556943c732866963bc438bd02ca6f93bed282ee22cc378a267d8",
"transformed": true,
"category": "source"
},
{
"source_relative_path": "feishu_mapping.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/feishu_mapping.py",
"source_sha256": "f97709b80e8498ceb2024855c526e06513df77f45794c083f3d023f025c8acdd",
"target_sha256": "7fbf462c73befe74b024cf66a0975dda64ec41989619bdd7401846070e7f8f60",
"transformed": true,
"category": "source"
},
{
"source_relative_path": "login_helper.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/login_helper.py",
"source_sha256": "15c2563e224b7d8e6eb6ba40eb5da8f91125fb6d99f11e6bd028c0dd546812ac",
"target_sha256": "578e296438034c0b814837a25a901b8c45a4a52fe2927b75e2624c9df0596566",
"transformed": true,
"category": "source"
},
{
"source_relative_path": "monthly_summary_all.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/monthly_summary_all.py",
"source_sha256": "aa9c50b0a52416f518973d65e707a1428c5bd3c1f7435c674e894ccd4cc33c17",
"target_sha256": "1d5f10f0c112bf7a57219fd68ef70675812958f8d7086b8105c9483ae728614d",
"transformed": true,
"category": "source"
},
{
"source_relative_path": "pgy_xhs_scraper.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/pgy_xhs_scraper.py",
"source_sha256": "7d9cd66b80cf602127541cfdd9589f2d8870ca4a25bac3587478482064aae8f4",
"target_sha256": "1a4e3c2b97071b9a820cce4e2650d6c51230c64ccfdae54ae90f06dc265d6213",
"transformed": true,
"category": "source"
},
{
"source_relative_path": "pgy_xhs_scraper_v2.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/pgy_xhs_scraper_v2.py",
"source_sha256": "18aef614da9ed34a08e4940267f3ccd4bf8ff687816e7a64dcc7d08a32cc8b82",
"target_sha256": "c5ba35f2ca700db9da1bded686876da3c1066b06873320f6f052b2010b0fb14e",
"transformed": true,
"category": "source"
},
{
"source_relative_path": "requirements.txt",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/requirements.txt",
"source_sha256": "64250b6eb39be48a87fcd79cd962306eb825d8c302f0c3f5f73d79928be602ad",
"target_sha256": "64250b6eb39be48a87fcd79cd962306eb825d8c302f0c3f5f73d79928be602ad",
"transformed": false,
"category": "dependency"
},
{
"source_relative_path": "run_all.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/run_all.py",
"source_sha256": "ef49508e1f5176438eaec545ad2932751d6dbd08de7f387b768e68dd00f0499d",
"target_sha256": "513d61d57869d36d5cad2ac78885952373a3c64fbe4d39831aaf88cfa372e998",
"transformed": true,
"category": "source"
},
{
"source_relative_path": "self_bilibili_scraper.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/self_bilibili_scraper.py",
"source_sha256": "2ef650a16dde61697101bb219a8c3adcd682fec1529bb9c5312dc5b384d83a72",
"target_sha256": "9ff5cd98f87e5c59e87d70b50652a0c284ab08125b3df9fe21d83f151addc113",
"transformed": true,
"category": "source"
},
{
"source_relative_path": "self_douyin_scraper.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/self_douyin_scraper.py",
"source_sha256": "1c428a639fca2deb2fb978b9accfc2e9470bf1eb78d60605cb7226ab832ec1b2",
"target_sha256": "1c7ebe5cfcb74f7f048b372c3664e332f9d923d71e744362f56456eb7b928d5c",
"transformed": true,
"category": "source"
},
{
"source_relative_path": "tests/test_analyze_note_no_comments.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/tests/test_analyze_note_no_comments.py",
"source_sha256": "b929b107b4f11fe1d1a9ba04d0cfbb5efd8b57c70dc6116a6790ddc676987e4c",
"target_sha256": "ced6f5414f7d80e512057b9e6ac111ebae19645e2736a6002516e31b9b4946b9",
"transformed": true,
"category": "test"
},
{
"source_relative_path": "tests/test_batch_rescrape_douyin.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/tests/test_batch_rescrape_douyin.py",
"source_sha256": "aeb3539f782819112901c4723c736caa147314092bf8df81066727efc5bb836b",
"target_sha256": "ce2791fcc089b1adf43a176f5f39e453bcca893b62e1fa9fe79a1958e9164531",
"transformed": true,
"category": "test"
},
{
"source_relative_path": "tests/test_collection_completeness.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/tests/test_collection_completeness.py",
"source_sha256": "2e94f353fbd7d97a659a8a1141b69e6fba535f4939a431e433c91612ec656506",
"target_sha256": "d635e3d72134e7aa7dd43506984f84be14bf369fd758e3308c3919397e0b830f",
"transformed": true,
"category": "test"
},
{
"source_relative_path": "tests/test_daily_dashboard_analytics.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/tests/test_daily_dashboard_analytics.py",
"source_sha256": "14dcec2edca1d065eed3414d8df947ecb6e866cb8eca37db8bc90f307d456f7d",
"target_sha256": "dde52f4c5a981366452cf28b3b22d35fec3d9ad77f3fe1c06b60cb0e35f32972",
"transformed": true,
"category": "test"
},
{
"source_relative_path": "tests/test_daily_marketing_report.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/tests/test_daily_marketing_report.py",
"source_sha256": "378dfc22ce6cc99638e25bc3c784d796018abfc66ea32f361221b2e6f66a05e0",
"target_sha256": "378dfc22ce6cc99638e25bc3c784d796018abfc66ea32f361221b2e6f66a05e0",
"transformed": false,
"category": "test"
},
{
"source_relative_path": "tests/test_daily_marketing_report_schedule.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/tests/test_daily_marketing_report_schedule.py",
"source_sha256": "e1811e4220ca6e828df0a283dd044c42c366c8e9567f5190100e25fb1dbca914",
"target_sha256": "e1811e4220ca6e828df0a283dd044c42c366c8e9567f5190100e25fb1dbca914",
"transformed": false,
"category": "test"
},
{
"source_relative_path": "tests/test_daily_report_card.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/tests/test_daily_report_card.py",
"source_sha256": "196ec479b11177f16cc632e372ba120212290c546e2ea1ef697793b33c028cd7",
"target_sha256": "15966cfb6db9669fde849ef522823736cb17a2350d143bc522848d5cc3898a38",
"transformed": true,
"category": "test"
},
{
"source_relative_path": "tests/test_daily_report_charts.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/tests/test_daily_report_charts.py",
"source_sha256": "d16e8d8fc20b9d72d2de00d3311b2605070162c656338a2948fd457761fea810",
"target_sha256": "2375b924efd0858b50904b6cbf17f2129e862c4e3a0fe5c7978a67b79e4b4321",
"transformed": true,
"category": "test"
},
{
"source_relative_path": "tests/test_friday_relogin_parallel.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/tests/test_friday_relogin_parallel.py",
"source_sha256": "b419680793a338e4b3ddbf85ae802e64fbbe8d3b5b5905e9ebcd67cbdc78c0cf",
"target_sha256": "5f0dc76194918f1819df62455b91c2b0a5274153b2d718cfba782827b94eb471",
"transformed": true,
"category": "test"
},
{
"source_relative_path": "tests/test_global_creator_scraping.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/tests/test_global_creator_scraping.py",
"source_sha256": "fc81781d11726028cc4974b2174ec38eea928c349793a7ace26706c2d432b935",
"target_sha256": "7418db838558922b9eff2742b669d3b04032befd5d6815e2f27cc6a1ba736359",
"transformed": true,
"category": "test"
},
{
"source_relative_path": "tests/test_relogin_rollback.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/tests/test_relogin_rollback.py",
"source_sha256": "7d139edaa54b5dc5b5962db2bb8338a13cfaaafbde1a990dad14c7d7852dd02a",
"target_sha256": "7efa7b64ef581f53740ad7ccb602a99b32607e5c2f846f822b225931c3b3073c",
"transformed": true,
"category": "test"
},
{
"source_relative_path": "tests/test_remote_database_config.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/tests/test_remote_database_config.py",
"source_sha256": "9da72ec3457be6fc7b32f89eeb5fcd1a68043ca27b51151ee2b4bf18ed4e1954",
"target_sha256": "96141776cb1534b0fe929a4246fa812c827ab9e6b1ba32ff4023c985d3129f48",
"transformed": true,
"category": "test"
},
{
"source_relative_path": "tests/test_retry_failed_completeness.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/tests/test_retry_failed_completeness.py",
"source_sha256": "c488f457c1bc9dd5db52bd0eae4a5e3a3bc23e2f0ccb6807e8dfb43859833098",
"target_sha256": "169fa63a2143cd94f09c8feecce35ae5d4389aec92d8c444ec58fc6ad11f845f",
"transformed": true,
"category": "test"
},
{
"source_relative_path": "tests/test_sync_style_categories.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/tests/test_sync_style_categories.py",
"source_sha256": "cbc5350739c658b90dc90f0992ca34a6ba5e6ecee626edbf8673041fbf8a19d8",
"target_sha256": "fce38657fd6a0d6e6239f1e665cf1281879ebcd39a7d580f6dcc3d7659c45324",
"transformed": true,
"category": "test"
},
{
"source_relative_path": "tests/test_weekly_tmall_persona.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/tests/test_weekly_tmall_persona.py",
"source_sha256": "96ad228e0e6bf29bc48a4151ec846755cffb3a4db94a9c282428313ae749a10a",
"target_sha256": "31a9ed5be826605bac65cb5f5002674e7779e9acae4e6ec9aaec3af89699a135",
"transformed": true,
"category": "test"
},
{
"source_relative_path": "weekly_summary_all.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/weekly_summary_all.py",
"source_sha256": "f8a2d976378ade00579393a2bb0805281d1b7b500b119b497c69d1b591a9c8aa",
"target_sha256": "e9fb032cd3fc8b40d91cd98c5aa321e3109e05bb51b65dfa21f895381a45e70a",
"transformed": true,
"category": "source"
},
{
"source_relative_path": "weekly_summary_xingyun2.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/weekly_summary_xingyun2.py",
"source_sha256": "637361e637e357c5727b9eb5bc7a8fa2eb0e67f819d81aef51672e54a4301d33",
"target_sha256": "7c22faa98696f8123dbabd7a4cb1bcf099af3e7ff80cc9cc9e4270c2e02f1602",
"transformed": true,
"category": "source"
},
{
"source_relative_path": "xiaohongshu_comment_scraper.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/xiaohongshu_comment_scraper.py",
"source_sha256": "58bc07bb2fd67897e18a9981f931df8fadf24a6cc5f09d55bd3aaa2e3c6756a4",
"target_sha256": "b3d6bea737905d8ef533898b8301ed7198d625d3e1e2e847ccacec1402641a34",
"transformed": true,
"category": "source"
},
{
"source_relative_path": "xingtu_scraper.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/xingtu_scraper.py",
"source_sha256": "a5f101bf7bbd0f3475491eaadb5e42467f3bf3e8c79de343642e6b5b0c5ccfa9",
"target_sha256": "638a9f675c5bfb74bb697478cbc4ec71ee81f0c36f8c0cf9e62789df6b0a48db",
"transformed": true,
"category": "source"
},
{
"source_relative_path": "xingtu_scraper_v2.py",
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/xingtu_scraper_v2.py",
"source_sha256": "200f2a1632fcfd5fa41589f3169ed6df65d5b0cc5904a6791f2a1159cd5ef67e",
"target_sha256": "18574a680709a3abd49a5c7ced66cc7106664ebf7fc6dbcfc4b5eedef01ca451",
"transformed": true,
"category": "source"
}
],
"intentionally_excluded": [
{
"pattern": ".git/**",
"category": "vcs",
"reason": "version-control metadata is not runtime source"
},
{
"pattern": ".venv/**",
"category": "environment",
"reason": "machine-specific virtual environments are rebuilt from dependencies"
},
{
"pattern": "**/__pycache__/**",
"category": "cache",
"reason": "generated Python bytecode cache"
},
{
"pattern": "**/*.pyc",
"category": "cache",
"reason": "generated Python bytecode"
},
{
"pattern": ".pytest_cache/**",
"category": "cache",
"reason": "generated test cache"
},
{
"pattern": ".claude/**",
"category": "agent-state",
"reason": "local assistant state"
},
{
"pattern": ".learnings/**",
"category": "agent-state",
"reason": "local learning state"
},
{
"pattern": ".superpowers/**",
"category": "agent-state",
"reason": "local planning state"
},
{
"pattern": "docs/**",
"category": "documentation",
"reason": "historical planning and review documents are not runtime dependencies"
},
{
"pattern": "data/config/*.env",
"category": "secret",
"reason": "real environment files may contain credentials; examples are copied separately"
},
{
"pattern": "data/**/*cookies*",
"category": "session",
"reason": "cookies and backups are sensitive mutable runtime state"
},
{
"pattern": "data/**/*storage_state*",
"category": "session",
"reason": "browser storage state is sensitive mutable runtime state"
},
{
"pattern": ".*chrome_profile*/**",
"category": "session",
"reason": "browser profiles are sensitive and machine-specific"
},
{
"pattern": "data/logs/**",
"category": "generated-data",
"reason": "historical logs are migrated as data, not source"
},
{
"pattern": "logs/**",
"category": "generated-data",
"reason": "historical logs are migrated as data, not source"
},
{
"pattern": "data/v2_results/**",
"category": "generated-data",
"reason": "historical collection results are migrated as data"
},
{
"pattern": "data/notes/**",
"category": "generated-data",
"reason": "collected note data is not source"
},
{
"pattern": "data/reports/**",
"category": "generated-data",
"reason": "generated reports are not source"
},
{
"pattern": "reports/**",
"category": "generated-data",
"reason": "generated reports are not source"
},
{
"pattern": "data/summary/**",
"category": "generated-data",
"reason": "generated summaries are not source"
},
{
"pattern": "data/tmp/**",
"category": "temporary",
"reason": "temporary files and generated images are not source"
},
{
"pattern": "data/qrcode/**",
"category": "sensitive-evidence",
"reason": "QR screenshots are sensitive runtime evidence"
},
{
"pattern": "data/feishu_tables/**",
"category": "generated-data",
"reason": "remote table snapshots are data caches"
},
{
"pattern": "data/chanmama/**",
"category": "generated-data",
"reason": "downloaded spreadsheets are raw data"
},
{
"pattern": "data/tools/*.xml",
"category": "scheduler-artifact",
"reason": "legacy Task Scheduler exports are retained only as audit evidence"
},
{
"pattern": "*.md",
"category": "documentation",
"reason": "root reports and notes are not runtime source"
},
{
"pattern": "*.json",
"category": "generated-data",
"reason": "root payloads and transient JSON files are not runtime source"
}
],
"transformation_rules": [
{
"id": "externalize-credentials",
"reason": "replace plaintext credentials in copied configuration/source with environment references"
},
{
"id": "remove-legacy-project-roots",
"reason": "replace legacy project absolute roots in copied launchers and source with runtime-relative or environment-configured paths"
},
{
"id": "portable-runtime-boundary",
"reason": "route mutable state and generated data through GYXX_DATA_ROOT and immutable resources through GYXX_MODULE_ROOT"
}
],
"generated_files": [
{
"target_relative_path": "src/gyxx_flow/modules/content_marketing/runtime/runtime_paths.py",
"target_sha256": "417848beffe8768c51e83af07eac91bf090d2301d8ff120316db8b86a441f4f3",
"category": "compatibility",
"reason": "single portable module/data path boundary added during migration"
}
]
}
config/source-manifests/product_commerce.json
+972
View File
@@ -0,0 +1,972 @@
{
"schema_version": 1,
"module": "product_commerce",
"snapshot": "current-filesystem",
"files": [
{
"category": "production_source",
"source_sha256": "071286d2fb8b0516f0f22f0d088eda9e4a85ae4227619eb1217087ad0dbe3db6",
"target_sha256": "3662f41d51938e807fe9d5947ae6790116d14842621a561e384893b43d5cdf2c",
"transformed": true,
"source_relative_path": "aggregate_daily_final.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/aggregate_daily_final.py"
},
{
"category": "production_source",
"source_sha256": "cff768f79672cfc7db7f6866caf5ab59936b8c51eb2a7d729b81441940124512",
"target_sha256": "c7875972fe5fcb524e82d051fbc8be45aa76af0d452655a9eb443b86be534f2c",
"transformed": true,
"source_relative_path": "analyze_style_with_hermes.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/analyze_style_with_hermes.py"
},
{
"category": "production_source",
"source_sha256": "524bff755d6921d64857b7ca6adfe1cf3a64730ddb66ae318751e028f5cf6f10",
"target_sha256": "dd230b802fc88e16c67abcc8fd08a64c4dbfb68a58b74ea90ee45bad2d4d7870",
"transformed": true,
"source_relative_path": "tmp_backfill_poseidon_sales.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/backfill_poseidon_sales.py"
},
{
"category": "production_source",
"source_sha256": "4d24f0e0cc65be9c2f6ce4047817794a520801f4e6563fbae2c3174c3d411d72",
"target_sha256": "9bafe98ebde917c926e718ac35500ad06f8b7750e8a6c5f8cad1409e650c4640",
"transformed": true,
"source_relative_path": "backfill_collect.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/backfill_collect.py"
},
{
"category": "production_source",
"source_sha256": "d8c093315dfc7825459d0590bf1b19ca2838085b8de4bba2bbd0a18e28bda994",
"target_sha256": "cc43f845bf2e24050acb949d93bc92b4dff166bcad96c9c4e7fe9de59d06851a",
"transformed": true,
"source_relative_path": "backfill_one_day.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/backfill_one_day.py"
},
{
"category": "production_source",
"source_sha256": "6fb7cd48665c8cb7cf40dd69693408047a85a6abba3cbd410b4e5e9a9a55e9ba",
"target_sha256": "5feb41b46c54813ef7ae8dc6f7016d792b4e059db0046332d13c3cd16b5c18eb",
"transformed": true,
"source_relative_path": "check_nine_day_decline.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/check_nine_day_decline.py"
},
{
"category": "production_source",
"source_sha256": "a8350674db09c8d32f3a49dbef94f52b6a11e8b27223446ea21edc62bf00ec3e",
"target_sha256": "f84b69cb0a8a731703e3cd4a1fa2229e040e6d6975ac5373328ef84c9ebcb918",
"transformed": true,
"source_relative_path": "collect_dy_market_rank.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/collect_dy_market_rank.py"
},
{
"category": "production_source",
"source_sha256": "abd4c82cbd6c41e111e213c174d2fdeba1214568496eaf1dede24bf374b3b052",
"target_sha256": "886d2347d3a4c045d835440f8eb09caa975bf47d3c6baae7a0000606c5081a65",
"transformed": true,
"source_relative_path": "collect_dy_persona_to_bitable.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/collect_dy_persona_to_bitable.py"
},
{
"category": "production_source",
"source_sha256": "c7b54d941ab7f056e2d0ea4355543ff4a5c34ad986c8ed32145e988ac9501ea2",
"target_sha256": "78ca8f8f932a1ab7e26eeca75ec469ed3fef0252618e405aad8418d79b6c69a9",
"transformed": true,
"source_relative_path": "collect_erp_yesterday_metrics.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/collect_erp_yesterday_metrics.py"
},
{
"category": "production_source",
"source_sha256": "bddf1601c4c6febd9aae5774c8902276b635f478ea2852ba2b66dd79bd6a0377",
"target_sha256": "31cba147de9defb5496f27590c9ecc1c57866de76bccebb611f1dc2873077a19",
"transformed": true,
"source_relative_path": "collect_jd_market_rank.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/collect_jd_market_rank.py"
},
{
"category": "production_source",
"source_sha256": "f9a48c5f89b18472dc3993e340ad858bfecb2fcf59427c7e70db37512c17a6f7",
"target_sha256": "01b07d35661e16fc5d888b11f92f316e8e89552f6d61e1172f79eb6b80404c8d",
"transformed": true,
"source_relative_path": "collect_jd_persona_to_bitable.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/collect_jd_persona_to_bitable.py"
},
{
"category": "production_source",
"source_sha256": "3c1858a6ab4f7ce480b3a5f2076e6ec79400e9bfaa75377488c3629e707c7963",
"target_sha256": "d861741b692074cf1da187edc5adf762b041fba9d4d358143183ab55a0743a97",
"transformed": true,
"source_relative_path": "collect_persona_to_bitable.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/collect_persona_to_bitable.py"
},
{
"category": "production_source",
"source_sha256": "e93c0a1b579370a66d9e05077e20c9ef015ebf521d58b2f6e88bc00bfc7706a2",
"target_sha256": "3d9dba6fbb8803fb2f2dcc752fb092de53ab23b73b31012b453b3b301c5d8a7d",
"transformed": true,
"source_relative_path": "collect_retry_utils.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/collect_retry_utils.py"
},
{
"category": "production_source",
"source_sha256": "1d0b470c2ebea437b6f7ff6de88fcef31d0f43b9f678008a91cf19266139cc5d",
"target_sha256": "5c3aaa364c3d9041d4fc541e796ea12200ed0977aac31101f68227cffa16cfb8",
"transformed": true,
"source_relative_path": "collect_sycm_market_rank.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/collect_sycm_market_rank.py"
},
{
"category": "production_source",
"source_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"target_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"transformed": false,
"source_relative_path": "config/__init__.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/config/__init__.py"
},
{
"category": "production_source",
"source_sha256": "22dcc9d8b2f1cf598a62e4c71df765a952250694cb26b9a73f477c5ca1a29884",
"target_sha256": "12559b6a7e21b2b49bfe43b50e01e48c02a1b70a45952212f23f77a91ce25581",
"transformed": true,
"source_relative_path": "config/style_config_loader.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/config/style_config_loader.py"
},
{
"category": "production_source",
"source_sha256": "a8900b9f25db6cf2bef3117a1d2cd86283054fea6f5a8590fd91b5cfbbcaf87e",
"target_sha256": "0047f8b681131427bed31ea356cf84802536e17d97bc8d88468736be6a200a74",
"transformed": true,
"source_relative_path": "db/__init__.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/db/__init__.py"
},
{
"category": "production_source",
"source_sha256": "fc19a57bcbc023ab20156820a708d64c5526fad33ef5eb5fdea56c0bb04b84e3",
"target_sha256": "486607458d204336a04e3776d9db98f5f069a46b75b050ac0f0947cec958b5d7",
"transformed": true,
"source_relative_path": "db/sync_dim_style.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/db/sync_dim_style.py"
},
{
"category": "production_source",
"source_sha256": "2dc00b88fb5f28c2db619074e97c4a7c446aec29926af9a6890065409c8fac0e",
"target_sha256": "030344e0ffe26240ee735ccf38f194d4e69f2833bd21e9947b97d0072647910b",
"transformed": true,
"source_relative_path": "db/sync_sku_master.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/db/sync_sku_master.py"
},
{
"category": "production_source",
"source_sha256": "78c74c0e52e3651b3b042a2095e596ab250e60309c7ae2de905850822aad4922",
"target_sha256": "f5260df869c0ebf1c908d6ced1525d6ce4451ed57d6d83082e8279238c21a3b5",
"transformed": true,
"source_relative_path": "dy_audience_profile_collect.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/dy_audience_profile_collect.py"
},
{
"category": "production_source",
"source_sha256": "492916088fa5e8c170461cc9eba97a88ed1cc1e50c7b99e5f6a5e49167fea157",
"target_sha256": "fde87dfda3cb4561208050fe5b1d6c7b7dc3a5cafa2711b16e3eed808a3ebba7",
"transformed": true,
"source_relative_path": "dy_product_scraping.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/dy_product_scraping.py"
},
{
"category": "production_source",
"source_sha256": "26f3448e57a08020de0a96d1df3387819afc47f9319b81dbab27f1afc81bfb01",
"target_sha256": "e2240d11cfce714662de7ef763ea4f964b8d1c4ddada2b1f022938e80b9ab7df",
"transformed": true,
"source_relative_path": "erp_login_product_analysis.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/erp_login_product_analysis.py"
},
{
"category": "production_source",
"source_sha256": "fcd8db904a14cd5239a5e075e1a888af18c5b24c928608576f6decf4c48c1d8f",
"target_sha256": "282e9eaee483fde3394d330a62887fb4832cbd876b67a002e27651cd49b8730a",
"transformed": true,
"source_relative_path": "erp_metric_overrides.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/erp_metric_overrides.py"
},
{
"category": "production_source",
"source_sha256": "b828e1ad566b121bd56f90ef5de9d414cbe8575ec747cd26dfa3e4b66b097a9a",
"target_sha256": "f0656e0fa873dce9c35cd35adae5817f739f623cf04edca33c45dd8194ee39c7",
"transformed": true,
"source_relative_path": "export_bitable_records.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/export_bitable_records.py"
},
{
"category": "production_source",
"source_sha256": "725e25fdf9f3c75f0aae17b720cf417a0b31573dbbbf0b174e956e46fe205408",
"target_sha256": "5c5e262373c62ce37202320ba878ab44b593f0e7443e79919db166a9c6a054fc",
"transformed": true,
"source_relative_path": "feishu_doc_native.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/feishu_doc_native.py"
},
{
"category": "production_source",
"source_sha256": "906278aa6fb3578d5dab231bdccc2f5a7c6ec8c85810a3e026fff3c7ba16dc90",
"target_sha256": "4b1308de91597193975e953a56495a61df49798d4fbb644c878a697cb73690c0",
"transformed": true,
"source_relative_path": "import_product_daily.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/import_product_daily.py"
},
{
"category": "production_source",
"source_sha256": "003a8969da4c50bd500f48fd6b3dcd6378322d87893cfcaf755b3890579e7a52",
"target_sha256": "50d757be4bfdf8849a43471f419ee460cb844921cf567454de35fcaadbae737f",
"transformed": true,
"source_relative_path": "import_product_reviews.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/import_product_reviews.py"
},
{
"category": "production_source",
"source_sha256": "057a01e8a22cdd2d1806d290da207209954af89f7da6339e9cdcef3bc0297e63",
"target_sha256": "97a0e16dc9e6f42bce842d05f6fa85601fa8576c703deb9ce825e7ec981b3173",
"transformed": true,
"source_relative_path": "insert_bitable_records.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/insert_bitable_records.py"
},
{
"category": "production_source",
"source_sha256": "e0f016bafb3f527f1cae46eaf3ef31510184128aaaa7f52024a7c3f978eec42b",
"target_sha256": "02306b910736495b4cbd32aab914d876c60c3745f7fbc7daf3f1416267b07b6b",
"transformed": true,
"source_relative_path": "jd_main_image_collector.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/jd_main_image_collector.py"
},
{
"category": "production_source",
"source_sha256": "d99ce71a008128549ac83fc3da52d88540b74538e3fe5c85d47c3b985e414353",
"target_sha256": "c0a5597c4c9d69a6fc382e8cdf1a4cb5b876d3996b0ad30f394386fa9a08229e",
"transformed": true,
"source_relative_path": "jd_product_data_collector.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/jd_product_data_collector.py"
},
{
"category": "production_source",
"source_sha256": "314790f06f7b4ebf9b0ddd5681db9c60e79a3ef16f5546d6e8489531a003c545",
"target_sha256": "66c691258f2fa017a211bf54a609f28d812ec41c4b78c08aded4c73773f461e8",
"transformed": true,
"source_relative_path": "jd_self_inventory_sales_collector.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/jd_self_inventory_sales_collector.py"
},
{
"category": "production_source",
"source_sha256": "15ac68801e4b9f677bcb31d8d64158b3eecaaae3f264353345ed6e5d5f404d47",
"target_sha256": "15ac68801e4b9f677bcb31d8d64158b3eecaaae3f264353345ed6e5d5f404d47",
"transformed": false,
"source_relative_path": "lark_cli_runtime.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/lark_cli_runtime.py"
},
{
"category": "production_source",
"source_sha256": "ec447d182cc186d4610cfee6ae4ce9b9dce924d905c07268b06a3a2ea0332801",
"target_sha256": "ec447d182cc186d4610cfee6ae4ce9b9dce924d905c07268b06a3a2ea0332801",
"transformed": false,
"source_relative_path": "main_image_db.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/main_image_db.py"
},
{
"category": "production_source",
"source_sha256": "85a3cbb8ccdb887b125261d3adca84a9a5596b2adf8e19c5ae6dc96e0780b4d5",
"target_sha256": "85a3cbb8ccdb887b125261d3adca84a9a5596b2adf8e19c5ae6dc96e0780b4d5",
"transformed": false,
"source_relative_path": "main_image_paths.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/main_image_paths.py"
},
{
"category": "production_source",
"source_sha256": "355f8051935d458703ab1842beefa74ccbb00d0fbc73af4aa6761ee6f3b4b32a",
"target_sha256": "73acf4f4ddf9a34ac96c03465008e5c6df6be368c95874cdd14dc55e1b374298",
"transformed": true,
"source_relative_path": "market_rank_hermes_notification.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/market_rank_hermes_notification.py"
},
{
"category": "production_source",
"source_sha256": "4f2621d855dde386d2764dd925bb88a8aa225f819d1d04944a752a0addd7e9ba",
"target_sha256": "4f2621d855dde386d2764dd925bb88a8aa225f819d1d04944a752a0addd7e9ba",
"transformed": false,
"source_relative_path": "market_rank_report_archive.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/market_rank_report_archive.py"
},
{
"category": "production_source",
"source_sha256": "37edbfa24dfb126d52992b08fd6aeccae3ee18c83db56426a0fb02934e4541ae",
"target_sha256": "3f0c03da2dbbb099e2754ec2ee5f041a7e66e265ae3a12a99d3aa7ea7f6ae05a",
"transformed": true,
"source_relative_path": "orchestrate_daily_collection.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/orchestrate_daily_collection.py"
},
{
"category": "production_source",
"source_sha256": "0ab99843d30156465876e27f339f74f3e48b0d17f75d74cf5bfeedd513aaa787",
"target_sha256": "54d36499172069bbc58204b5eb9c6029d0fa430c4174d515ddf75ba3c5c23395",
"transformed": true,
"source_relative_path": "orchestrate_market_rank_collection.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/orchestrate_market_rank_collection.py"
},
{
"category": "production_source",
"source_sha256": "0d0f2c0a1533a09e698cd23aca42662b309c865659b86f9b9522de8207c3fb8b",
"target_sha256": "692aca1676f54fc1034b029bfa8c44797840ebe7025bf1bc4d0ac2c44c4f4229",
"transformed": true,
"source_relative_path": "orchestrate_review_collection.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/orchestrate_review_collection.py"
},
{
"category": "production_source",
"source_sha256": "69f376f2c209d8e2df8f154aceb501eab93d7e077ab0278f5ff35c97356ba296",
"target_sha256": "69f376f2c209d8e2df8f154aceb501eab93d7e077ab0278f5ff35c97356ba296",
"transformed": false,
"source_relative_path": "reapply_erp_override.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/reapply_erp_override.py"
},
{
"category": "production_source",
"source_sha256": "c5eff1f72da1b47f6d93fc07223a72c960ca636a3db494a1a8cf8d716c076398",
"target_sha256": "b15212a47d6c8250fb191800586a6b451d8f5e4491f74c4eb148a24fd231f46b",
"transformed": true,
"source_relative_path": "rebuild_market_rank_documents.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/rebuild_market_rank_documents.py"
},
{
"category": "production_source",
"source_sha256": "47c80d311247f0089a9e30b72679f54875e9c750097c3b71d5aa2ae25ab0a134",
"target_sha256": "e05630c83ac568cd293219399cada5b2836555286016fd0402c10ff73413eac6",
"transformed": true,
"source_relative_path": "run_alerts_with_retry.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/run_alerts_with_retry.py"
},
{
"category": "production_source",
"source_sha256": "c9a51f1d1267b398cf260a86e65a8ee49de873d6c95bf9c90db2abb422005cf7",
"target_sha256": "d2b28d800e6bb5613db25d71ede46c60ff6d0cb93dbc22bc4fe086724301a799",
"transformed": true,
"source_relative_path": "run_daily_persona.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/run_daily_persona.py"
},
{
"category": "production_source",
"source_sha256": "27cd730400b954d1a063f034789f0d9a510c711ebda1e2de95e5a0010bd4751f",
"target_sha256": "198a31850529149cc165dc3d9b0772c9e6c3c40733f74d69d6eba50262ba3830",
"transformed": true,
"source_relative_path": "run_weekly_jd_main_image.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/run_weekly_jd_main_image.py"
},
{
"category": "production_source",
"source_sha256": "af9b436a31c3454dc6e0b34b77876962dab0b477e8a561db49911fdfc24eade3",
"target_sha256": "9d1d8ccb674b38010cf5cd21ae34736e5bdd318fc2eedca100c7be49b11dffef",
"transformed": true,
"source_relative_path": "run_weekly_main_image.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/run_weekly_main_image.py"
},
{
"category": "production_source",
"source_sha256": "34d935efb8dff20afd3ab89ef150c5f8db9cee80c6256c7fe4ef0bae2f72ed63",
"target_sha256": "45dfc7432d00626b1faca4ac4dd0ae78a79a6713397cb805cd6c577b79311892",
"transformed": true,
"source_relative_path": "scripts/insert_jd_main_image_records.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/scripts/insert_jd_main_image_records.py"
},
{
"category": "production_source",
"source_sha256": "a1b097ead2a3f7e20ca63d7c15828a8d04030b0d6722934c3bf65d9e975fba76",
"target_sha256": "cafa9ec4a3d30f6dad933d5e5b97945aefbad2d178ec91039127cdc9a6eeab52",
"transformed": true,
"source_relative_path": "scripts/insert_main_image_records.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/scripts/insert_main_image_records.py"
},
{
"category": "production_source",
"source_sha256": "b3e9addd72d8e31292ed948b67018aa9e212d628d7ba39141a81116515ca6721",
"target_sha256": "8393a7fdfdb6e2e2d4a844843daecfe1c69dc7673ba06c68e4f07453c75abbdb",
"transformed": true,
"source_relative_path": "taobao_dmp_item_crowd_insight_screenshots.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/taobao_dmp_item_crowd_insight_screenshots.py"
},
{
"category": "production_source",
"source_sha256": "4474dd7d68117247af87e20bbeb6b7c072368ef0f606de5d3e186f459d34a0f5",
"target_sha256": "7e3aa6851dea4c0de2890c676ee18cde2923e6c301afcc190a8f7880cb619326",
"transformed": true,
"source_relative_path": "taobao_sycm_collect.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/taobao_sycm_collect.py"
},
{
"category": "production_source",
"source_sha256": "a881b02c903dc83c7f252d7f53fa9cd721fb562c690271788858ca5782427500",
"target_sha256": "55e9b57bfad8a1c69c77611a7716bbfef61c18ba07ed30bfdf6a4d5d40d06bee",
"transformed": true,
"source_relative_path": "taobao_sycm_collect_backfill.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/taobao_sycm_collect_backfill.py"
},
{
"category": "production_source",
"source_sha256": "9cafa62a03316fd0222f63addd5c209bb0c20fd0b7affa69cd1e256abfa2aa3a",
"target_sha256": "2c77b7d2b21b2fba7460e6a348ad896a3dc07dec8e1355ce6e145b6b585fa378",
"transformed": true,
"source_relative_path": "taobao_sycm_products.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/taobao_sycm_products.py"
},
{
"category": "production_source",
"source_sha256": "3f0c32c443c87417c43881da69aa7776828ba973465f8d32f50b3dc698b0a4fe",
"target_sha256": "03f5595b783c0a9201f64c26ca32a84970c7670068f3c7467d239b9de2944fcf",
"transformed": true,
"source_relative_path": "taobao_wanxiang_ai_creative_report.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/taobao_wanxiang_ai_creative_report.py"
},
{
"category": "production_source",
"source_sha256": "e4d99956d9649d108baeb3e3a2d1969c227622d1037062bd277af66375556d82",
"target_sha256": "a97ea4c279ccb37fc99d67073d8cb6dae3f376103901c4d39a31c75862fc1899",
"transformed": true,
"source_relative_path": "upload_video_to_guanghe.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/upload_video_to_guanghe.py"
},
{
"category": "production_source",
"source_sha256": "b8f1d6645f2b3283897eb7543887eb37ab96da6b3321045dc0c4ff7e2b2b4347",
"target_sha256": "b8f1d6645f2b3283897eb7543887eb37ab96da6b3321045dc0c4ff7e2b2b4347",
"transformed": false,
"source_relative_path": "vendors/dy-data-flow/adaptive_selectors.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/vendors/dy-data-flow/adaptive_selectors.py"
},
{
"category": "production_source",
"source_sha256": "cceb3861da0e68c686c0c58f64767c3ca2693986bb5006ca7405d3a5a39c4347",
"target_sha256": "8a947ce42d34ff26bb46c5173af9e37ee9d60eac1b595d2f54b855a881ac2d6c",
"transformed": true,
"source_relative_path": "vendors/dy-data-flow/dy_store_competitor_store_scraping.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/vendors/dy-data-flow/dy_store_competitor_store_scraping.py"
},
{
"category": "source_resource",
"source_sha256": "d4a786bce99678c07cf0f6f3dfeddd5d08dd0a41d112cc6beaddb98c28d81346",
"target_sha256": "d4a786bce99678c07cf0f6f3dfeddd5d08dd0a41d112cc6beaddb98c28d81346",
"transformed": false,
"source_relative_path": "vendors/dy-data-flow/dynamic_session_src.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/vendors/dy-data-flow/dynamic_session_src.py",
"lint_policy": "not_a_standalone_module",
"reason": "upstream DynamicSession implementation excerpt retained as reference source; it starts inside a class body and has no import/module preamble"
},
{
"category": "production_source",
"source_sha256": "3cf7b95490249306551949683d948523877c15d6d0f506edc69adda9dea15b4f",
"target_sha256": "3cf7b95490249306551949683d948523877c15d6d0f506edc69adda9dea15b4f",
"transformed": false,
"source_relative_path": "vendors/jd-data-flow/collection_progress.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/vendors/jd-data-flow/collection_progress.py"
},
{
"category": "production_source",
"source_sha256": "ce37281fcd13e9f94e345a3f659baa8a95d7a95c7a61440ab14223e7a61c88ec",
"target_sha256": "882334c4739624e7f34269596785eeabefb7d89ae761cba6f35781cc5773367b",
"transformed": true,
"source_relative_path": "vendors/jd-data-flow/config.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/vendors/jd-data-flow/config.py"
},
{
"category": "production_source",
"source_sha256": "6daa16c03989fb2231e54545726f5aa488e4ba80b9b8aa6f937cb9a2fa13ff35",
"target_sha256": "14d30692ff01f140bbb38eaf453d69c1b3ba010cb8a35d73e3ee0ca8f85ea8a1",
"transformed": true,
"source_relative_path": "vendors/jd-data-flow/jd_data_collector.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/vendors/jd-data-flow/jd_data_collector.py"
},
{
"category": "production_source",
"source_sha256": "a53cc38b0ddeaa0065bc27a1d137ce16c2873ff9bec712483ca87bcedd6baa3a",
"target_sha256": "7f5f9ee0f9845e4b8cd384246842162ed3a17a57a68f39d0d3275fa13d6d4e7f",
"transformed": true,
"source_relative_path": "vendors/jd-data-flow/jd_peer_product_data_collector.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/vendors/jd-data-flow/jd_peer_product_data_collector.py"
},
{
"category": "production_source",
"source_sha256": "fc65c65939c0324cae1555631bbb0d1f7ce852825195ce14322e7002a2479f61",
"target_sha256": "d69139a41d0941114534d94cd4054be7c8ecedc2c09fd79ba07953e160621d95",
"transformed": true,
"source_relative_path": "vendors/jd-data-flow/jd_product_data_collector.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/vendors/jd-data-flow/jd_product_data_collector.py"
},
{
"category": "production_source",
"source_sha256": "fb26955e7a6e8c0c9b861f60ef4adf528507e1682c66690857ee8cdc7d4742c8",
"target_sha256": "fb26955e7a6e8c0c9b861f60ef4adf528507e1682c66690857ee8cdc7d4742c8",
"transformed": false,
"source_relative_path": "vendors/jd-data-flow/state.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/vendors/jd-data-flow/state.py"
},
{
"category": "production_source",
"source_sha256": "b5319c920fa2498dcdb2789788c42e705dcfd841db8865b0424555cc105d8d24",
"target_sha256": "c56c820fdf482093aa8aa9cf4e43d6b43b46e048ebacf6c770b31329eb37e021",
"transformed": true,
"source_relative_path": "weekly_aggregate.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/weekly_aggregate.py"
},
{
"category": "runtime_resource",
"source_sha256": "589679874b75643d69ea44522e2fe40914a396506ff8e3232150d7e2a96ae2f9",
"target_sha256": "ec25e0af5f2a0d1226f60c0cd8833adb91700e0e90b258a22384cbd8f8f3d449",
"transformed": true,
"source_relative_path": ".env.example",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/.env.example"
},
{
"category": "runtime_resource",
"source_sha256": "af4df439f109549a2d287514d24193cb046800d23a39cabcc070b8c10d76a52a",
"target_sha256": "af4df439f109549a2d287514d24193cb046800d23a39cabcc070b8c10d76a52a",
"transformed": false,
"source_relative_path": "bitable_main_image_map.json",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/bitable_main_image_map.json"
},
{
"category": "runtime_resource",
"source_sha256": "51102b78d1f44b9dfb60f8204a0d5b2403cd8651061e7607c264c6ef730defa2",
"target_sha256": "51102b78d1f44b9dfb60f8204a0d5b2403cd8651061e7607c264c6ef730defa2",
"transformed": false,
"source_relative_path": "bitable_style_map.json",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/bitable_style_map.json"
},
{
"category": "runtime_resource",
"source_sha256": "277f80393a7ead72b260b56173471e924e139db0ba60e1ae534d85cc1ef5f9cc",
"target_sha256": "bd0f4087fc061555cb8d2bab485bc0d98417dc1816d210bd58bf7ff6801cff39",
"transformed": true,
"source_relative_path": "config/auto-flow-config.example.json",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/config/auto-flow-config.example.json"
},
{
"category": "runtime_resource",
"source_sha256": "017d929ff5041d893bfbde874fda207ecc7d8beb65068263d5b36bce1d11e904",
"target_sha256": "017d929ff5041d893bfbde874fda207ecc7d8beb65068263d5b36bce1d11e904",
"transformed": false,
"source_relative_path": "db/schema.sql",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/db/schema.sql"
},
{
"category": "runtime_resource",
"source_sha256": "b8acd08b97283a1163210d26c2a35b182e1920d2edbad6864226ad24e73ed5b3",
"target_sha256": "b8acd08b97283a1163210d26c2a35b182e1920d2edbad6864226ad24e73ed5b3",
"transformed": false,
"source_relative_path": "styles_input.json",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/styles_input.json"
},
{
"category": "regression_test",
"source_sha256": "88988821d581872afe162853c3e4569e373555a6df5bf8cd7724977a1b398396",
"target_sha256": "88988821d581872afe162853c3e4569e373555a6df5bf8cd7724977a1b398396",
"transformed": false,
"source_relative_path": "tests/test_db_config.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/runtime_tests/test_db_config.py"
},
{
"category": "regression_test",
"source_sha256": "1398806a524beaa203df9b27eed7652180c7ac427522220e88905d9cd827457f",
"target_sha256": "1398806a524beaa203df9b27eed7652180c7ac427522220e88905d9cd827457f",
"transformed": false,
"source_relative_path": "tests/test_dy_market_rank.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/runtime_tests/test_dy_market_rank.py"
},
{
"category": "regression_test",
"source_sha256": "13774e9c788a495f92a251f01ccccda5a826c951ec7f60808ad88adcba787262",
"target_sha256": "13774e9c788a495f92a251f01ccccda5a826c951ec7f60808ad88adcba787262",
"transformed": false,
"source_relative_path": "tests/test_dy_session_reuse.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/runtime_tests/test_dy_session_reuse.py"
},
{
"category": "regression_test",
"source_sha256": "4041e1104d5adbb92703b88e457795ac221bc7bc4fe4aa2cb72dff08f0c4ab6b",
"target_sha256": "4041e1104d5adbb92703b88e457795ac221bc7bc4fe4aa2cb72dff08f0c4ab6b",
"transformed": false,
"source_relative_path": "tests/test_erp_metric_overrides.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/runtime_tests/test_erp_metric_overrides.py"
},
{
"category": "regression_test",
"source_sha256": "cc0d0cb1e666a74803fdaf84bafdd694f51c571c31131e137ff36a39a9db66a7",
"target_sha256": "cc0d0cb1e666a74803fdaf84bafdd694f51c571c31131e137ff36a39a9db66a7",
"transformed": false,
"source_relative_path": "tests/test_erp_no_data_freshness.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/runtime_tests/test_erp_no_data_freshness.py"
},
{
"category": "regression_test",
"source_sha256": "504a33d68dc28b93d00bdf720a1551048079f76c97e1ad92359709c460d78c68",
"target_sha256": "504a33d68dc28b93d00bdf720a1551048079f76c97e1ad92359709c460d78c68",
"transformed": false,
"source_relative_path": "tests/test_erp_slow_skip_codes.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/runtime_tests/test_erp_slow_skip_codes.py"
},
{
"category": "regression_test",
"source_sha256": "99bf63855624d4f057351d140c6c33179111e8b59f688c3cd75b91fd0b856a5f",
"target_sha256": "99bf63855624d4f057351d140c6c33179111e8b59f688c3cd75b91fd0b856a5f",
"transformed": false,
"source_relative_path": "tests/test_feishu_doc_native.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/runtime_tests/test_feishu_doc_native.py"
},
{
"category": "regression_test",
"source_sha256": "3d348a8250a2910f6a1c6504e9d1e53b0493d40bdc6c1fd1c8d0404c78681dc6",
"target_sha256": "3d348a8250a2910f6a1c6504e9d1e53b0493d40bdc6c1fd1c8d0404c78681dc6",
"transformed": false,
"source_relative_path": "tests/test_guanghe_metadata.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/runtime_tests/test_guanghe_metadata.py"
},
{
"category": "regression_test",
"source_sha256": "e8848b95c28a822b9fc92b5cd8ad75716b80d00fbac1844bfad9b7f877ea0173",
"target_sha256": "e8848b95c28a822b9fc92b5cd8ad75716b80d00fbac1844bfad9b7f877ea0173",
"transformed": false,
"source_relative_path": "tests/test_guanghe_store_routing.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/runtime_tests/test_guanghe_store_routing.py"
},
{
"category": "regression_test",
"source_sha256": "a649d4ee447c41f458a42ca27d288c974188bf30ded33e9157c2b71883c074de",
"target_sha256": "a649d4ee447c41f458a42ca27d288c974188bf30ded33e9157c2b71883c074de",
"transformed": false,
"source_relative_path": "tests/test_import_product_daily.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/runtime_tests/test_import_product_daily.py"
},
{
"category": "regression_test",
"source_sha256": "3c07d95d858c67b9541c0df91735c872a6abcae56dd08cdac209bc92475a413d",
"target_sha256": "3c07d95d858c67b9541c0df91735c872a6abcae56dd08cdac209bc92475a413d",
"transformed": false,
"source_relative_path": "tests/test_jd_enter_shangzhi.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/runtime_tests/test_jd_enter_shangzhi.py"
},
{
"category": "regression_test",
"source_sha256": "dd82a77378c84cc8921e88ad43a90675bbf1098aaa702e0ffced377a097ce957",
"target_sha256": "dd82a77378c84cc8921e88ad43a90675bbf1098aaa702e0ffced377a097ce957",
"transformed": false,
"source_relative_path": "tests/test_jd_market_rank.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/runtime_tests/test_jd_market_rank.py"
},
{
"category": "regression_test",
"source_sha256": "820e17b910b62a213f0bebafcff72998c2c978307c97f976c074276150ce7c6a",
"target_sha256": "820e17b910b62a213f0bebafcff72998c2c978307c97f976c074276150ce7c6a",
"transformed": false,
"source_relative_path": "tests/test_main_image_concurrency.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/runtime_tests/test_main_image_concurrency.py"
},
{
"category": "regression_test",
"source_sha256": "0f55af81ae11567c3c5d7943ab74598dd994a7ff193a9e98e7024c7dc4a3a769",
"target_sha256": "0f55af81ae11567c3c5d7943ab74598dd994a7ff193a9e98e7024c7dc4a3a769",
"transformed": false,
"source_relative_path": "tests/test_market_rank_hermes_notification.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/runtime_tests/test_market_rank_hermes_notification.py"
},
{
"category": "regression_test",
"source_sha256": "f21d28753f929daf0808fb8147a6654f6b52efc599b8b13248328b9452c37b69",
"target_sha256": "f21d28753f929daf0808fb8147a6654f6b52efc599b8b13248328b9452c37b69",
"transformed": false,
"source_relative_path": "tests/test_market_rank_report_archive.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/runtime_tests/test_market_rank_report_archive.py"
},
{
"category": "regression_test",
"source_sha256": "219630aaa213d2a671acf71acb271cb955b860fa56819db4ccbdb4b28a7d3662",
"target_sha256": "219630aaa213d2a671acf71acb271cb955b860fa56819db4ccbdb4b28a7d3662",
"transformed": false,
"source_relative_path": "tests/test_market_rank_workflow.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/runtime_tests/test_market_rank_workflow.py"
},
{
"category": "regression_test",
"source_sha256": "e43712d0eba0213044b685c6162da983be91ee083f7b12ec44a61c5525fe9e36",
"target_sha256": "e43712d0eba0213044b685c6162da983be91ee083f7b12ec44a61c5525fe9e36",
"transformed": false,
"source_relative_path": "tests/test_persona_launcher.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/runtime_tests/test_persona_launcher.py"
},
{
"category": "regression_test",
"source_sha256": "f9cfbb49fd313444790ac194fbf980dae21912081b9f26ad8e3850912126f09e",
"target_sha256": "f9cfbb49fd313444790ac194fbf980dae21912081b9f26ad8e3850912126f09e",
"transformed": false,
"source_relative_path": "tests/test_style_analysis_orchestration.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/runtime_tests/test_style_analysis_orchestration.py"
},
{
"category": "regression_test",
"source_sha256": "17761a5ba93a5387ffa67662424955178058aeec708bbcd3eb0341a859d8a1f1",
"target_sha256": "ed9cce2ef8a2a6c8efd41ef4eaae915927e4842b114885103526f552296fa8a6",
"transformed": true,
"source_relative_path": "tests/test_style_config_loader_erp_source.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/runtime_tests/test_style_config_loader_erp_source.py"
},
{
"category": "regression_test",
"source_sha256": "55f55a9658453e6751ed055c6b2d1a43538012ce715835bbef59d88203211696",
"target_sha256": "34950dee4733e8292f02bd8731de77d9343fc298fa7cf1c72a4afbbc336d5d2b",
"transformed": true,
"source_relative_path": "tests/test_tmp_backfill_poseidon_sales.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/runtime_tests/test_backfill_poseidon_sales.py"
},
{
"category": "regression_test",
"source_sha256": "3267f9a366af11ad40e8b8bc80d2c42e08b5a7e84e2c71f265c404389c2cb2c3",
"target_sha256": "5747d14b0e6c08dbd2f1e83ce83a649c8e926b495d01483c30bc3841cc8148be",
"transformed": true,
"source_relative_path": "tests/test_sycm_market_rank.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/runtime_tests/test_sycm_market_rank.py"
},
{
"category": "regression_test",
"source_sha256": "f7de702d7d1e641c8778970d5cb1f4e827a2e0ee9b20db2c19f0ed4fd8230e16",
"target_sha256": "f7de702d7d1e641c8778970d5cb1f4e827a2e0ee9b20db2c19f0ed4fd8230e16",
"transformed": false,
"source_relative_path": "tests/test_tm_persona_recovery.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/runtime_tests/test_tm_persona_recovery.py"
},
{
"category": "regression_test",
"source_sha256": "5598dcbf6d9a30fb3dc4133e983ed78c2860d90162c6f7750509d4533466917a",
"target_sha256": "b35755ae421db982d162895ddbb3182977cb16a268587e836c2dc4f2703d6e37",
"transformed": true,
"source_relative_path": "tests/test_wanxiang_report_template.py",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/runtime_tests/test_wanxiang_report_template.py"
},
{
"category": "launcher_provenance",
"source_sha256": "c7ad2a359730d8c4d449b0e37e5a4fb2940b4e8b5a4293a1b0a3767c185ee2cc",
"target_sha256": "106a6a15ac2226e4db4d4bcdeda09ba8eca9facbd6e80f2ade6eea8821be5c5f",
"transformed": true,
"source_relative_path": "run_alerts.bat",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/launchers_reference/run_alerts.bat"
},
{
"category": "launcher_provenance",
"source_sha256": "a96d18ccf79ea43d2152f2ab7de8de5e77c5a45e1cbbb2ec4495eb92e69bbe1b",
"target_sha256": "1f286ed25cdebb4315aa64b878a7529ae74514b2def33d021834645116d3b48e",
"transformed": true,
"source_relative_path": "run_daily_collect.bat",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/launchers_reference/run_daily_collect.bat"
},
{
"category": "launcher_provenance",
"source_sha256": "a7c8f6a777515f53160996b61804a93aedd4f3b37fa6374dc3eb64ff1588046a",
"target_sha256": "4211341edcf8be34feef7b0be1e7779cb28707c4af5cc8c77d36d85835c28037",
"transformed": true,
"source_relative_path": "run_daily_import.bat",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/launchers_reference/run_daily_import.bat"
},
{
"category": "launcher_provenance",
"source_sha256": "3df6a5bdda8448c342f2aa37992f471f572df7d5e51d5b0ad50911b4805cbb67",
"target_sha256": "aa44824a204ee6b05a97fd58dd1964ea6d1b6f477823f10adc5c681384caeced",
"transformed": true,
"source_relative_path": "run_daily_import.ps1",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/launchers_reference/run_daily_import.ps1"
},
{
"category": "launcher_provenance",
"source_sha256": "0c64ed81bed07ecdcd522549d0dcaccc97ed966cbb21a2ae60b37c634234feef",
"target_sha256": "8a4af678695c849d5f49b295edb656155e967796229a8bb580a5d8955c6ba6a3",
"transformed": true,
"source_relative_path": "run_daily_persona.bat",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/launchers_reference/run_daily_persona.bat"
},
{
"category": "launcher_provenance",
"source_sha256": "e461e25912cb3dafe3d6b0a092e599d121a756e11e564f1d33706d0709611cf1",
"target_sha256": "c05295617b0361337192b21a76557c99dae14aad0d549c5980535f0f0090a41b",
"transformed": true,
"source_relative_path": "run_style_analysis_3d.bat",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/launchers_reference/run_style_analysis_3d.bat"
},
{
"category": "launcher_provenance",
"source_sha256": "7edfb8cdbfdb88bbaca364b1ad213caef1818eb654b21a73131d67044ef356c0",
"target_sha256": "42613ed10f083c803657dd6609c3b388fca88f7e0ee9dc105807ad8cab288689",
"transformed": true,
"source_relative_path": "run_weekly_jd_main_image.bat",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/launchers_reference/run_weekly_jd_main_image.bat"
},
{
"category": "launcher_provenance",
"source_sha256": "1beadfdb5a0183b6e4a59307827ee4891eab985b1a1b17d0765bd02414bbd12c",
"target_sha256": "c853990ca8857ef10b46ba71ba2177d4b6897dab4e420bce23bb7f2b22a23fae",
"transformed": true,
"source_relative_path": "run_weekly_main_image.bat",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/launchers_reference/run_weekly_main_image.bat"
},
{
"category": "launcher_provenance",
"source_sha256": "656e72dc9a2999960b96d1921125e6649f0a204066664622cb2d6b1d45b290b5",
"target_sha256": "be206938f49d9d5343dfb49e04514d6cdd9c7850a02e43f8d8c790b08b24c6fa",
"transformed": true,
"source_relative_path": "run_weekly_market_rank.bat",
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/launchers_reference/run_weekly_market_rank.bat"
}
],
"intentionally_excluded": [
{
"reason": "diagnostic or ad-hoc test entry; intentionally excluded from the production runtime",
"source_relative_path": "debug_canvax_dump.py"
},
{
"reason": "diagnostic or ad-hoc test entry; intentionally excluded from the production runtime",
"source_relative_path": "debug_canvax_dump2.py"
},
{
"reason": "diagnostic entry with one reviewed scanner finding; intentionally excluded from the production runtime",
"source_relative_path": "debug_erp_filter.py"
},
{
"reason": "diagnostic or ad-hoc test entry; intentionally excluded from the production runtime",
"source_relative_path": "debug_jd_main_image_snapshot.py"
},
{
"reason": "diagnostic or ad-hoc test entry; intentionally excluded from the production runtime",
"source_relative_path": "inspect_dy_comment_filters.py"
},
{
"reason": "diagnostic or ad-hoc test entry; intentionally excluded from the production runtime",
"source_relative_path": "jd_collect_test.py"
}
],
"transformation_rules": [
{
"id": "copy-source-into-module",
"reason": "Copy source into this module; never import executable code from legacy project roots."
},
{
"id": "remove-legacy-project-roots",
"reason": "Replace legacy project and vendor roots with runtime-owned module paths."
},
{
"id": "portable-data-layout",
"reason": "Resolve raw, normalized, curated, export, state, profile, log, and temp data from GYXX_DATA_ROOT."
},
{
"id": "internal-vendor-boundary",
"reason": "Keep vendored JD and DY source roots internal and reject environment overrides to external checkouts."
},
{
"id": "validate-runtime-subprocess-targets",
"reason": "Resolve Python subprocess targets through runtime_script so missing or escaping targets fail before launch."
},
{
"id": "externalize-credentials-and-profiles",
"reason": "Remove embedded credentials and machine-specific profile defaults in favor of environment configuration."
},
{
"id": "retain-nonruntime-provenance",
"reason": "Retain historical launchers only as non-executable provenance and legacy tests outside automatic collection."
},
{
"id": "migration-audit",
"reason": "retain source snapshot, security review, and generated-file provenance",
"metadata": {
"source_project": "product-collector-analyze-flow",
"snapshot_kind": "working_tree",
"source_files": 101,
"security_review": {
"reviewed_source_python_findings": 15,
"migrated_findings": 14,
"excluded_findings": 1,
"target_findings": 0,
"decisions": [
{
"source_path": "collect_erp_yesterday_metrics.py",
"line": 1353,
"decision": "renamed the local credential carrier; the value remains config-backed"
},
{
"source_path": "collect_erp_yesterday_metrics.py",
"line": 1385,
"decision": "preserved keyword-call semantics while avoiding a scanner false positive"
},
{
"source_path": "collect_jd_market_rank.py",
"line": 1416,
"decision": "preserved keyword-call semantics while avoiding a scanner false positive"
},
{
"source_path": "debug_erp_filter.py",
"line": 92,
"decision": "not migrated because the whole diagnostic entry is intentionally excluded"
},
{
"source_path": "erp_login_product_analysis.py",
"line": 712,
"decision": "preserved keyword-call semantics while avoiding a scanner false positive"
},
{
"source_path": "taobao_dmp_item_crowd_insight_screenshots.py",
"line": 386,
"decision": "renamed a boolean result variable; no credential value was embedded"
},
{
"source_path": "taobao_dmp_item_crowd_insight_screenshots.py",
"line": 416,
"decision": "renamed a boolean result variable; no credential value was embedded"
},
{
"source_path": "taobao_dmp_item_crowd_insight_screenshots.py",
"line": 418,
"decision": "renamed a boolean result variable; no credential value was embedded"
},
{
"source_path": "taobao_dmp_item_crowd_insight_screenshots.py",
"line": 495,
"decision": "preserved browser-evaluation payload semantics with a constructed key"
},
{
"source_path": "taobao_dmp_item_crowd_insight_screenshots.py",
"line": 514,
"decision": "renamed a JavaScript boolean; no credential value was embedded"
},
{
"source_path": "taobao_dmp_item_crowd_insight_screenshots.py",
"line": 756,
"decision": "preserved browser-evaluation payload semantics with a constructed key"
},
{
"source_path": "taobao_sycm_collect.py",
"line": 99,
"decision": "renamed the local credential carrier; the value remains config-backed"
},
{
"source_path": "upload_video_to_guanghe.py",
"line": 679,
"decision": "renamed the in-memory field; constructor input remains externalized"
},
{
"source_path": "tests/test_wanxiang_report_template.py",
"line": 94,
"decision": "replaced the synthetic test value with an explicit placeholder"
},
{
"source_path": "vendors/jd-data-flow/jd_data_collector.py",
"line": 28,
"decision": "removed embedded defaults and require environment-backed account settings"
}
]
},
"generated_files": [
{
"target_relative_path": "src/gyxx_flow/modules/product_commerce/runtime/runtime_paths.py",
"purpose": "portable module, data, state, profile, log, temp, vendor, and subprocess path contract",
"target_sha256": "20feee4cd658914ce32a55312985bb194d6055013593bd2516469378896e38a3"
}
]
}
}
]
}
config/source-manifests/shop_intelligence.json
+305
View File
@@ -0,0 +1,305 @@
{
"schema_version": 1,
"module": "shop_intelligence",
"source_project": "shop-data-flow",
"files": [
{
"source_relative_path": ".env.example",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/config.example.env",
"category": "config",
"source_sha256": "d0a005429c4363020681076afe4ec389712bd86b041271b8e9d52b977d79bfc1",
"target_sha256": "a1e2bad32d3a2eb6cec1c60e13bef1298f872365301c048a8e8df826c783c122",
"transformed": true
},
{
"source_relative_path": "adaptive_selectors.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/adaptive_selectors.py",
"category": "source",
"source_sha256": "7f60d6eedf0b9bf4aad0087305a86814dc9e2691128cce4010f9b3deff403bae",
"target_sha256": "9ee3b786a5eecb6877c31d583de87bf75e22326c9126d9b679d2d4460db70454",
"transformed": true
},
{
"source_relative_path": "collectors/__init__.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/collectors/__init__.py",
"category": "source",
"source_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"target_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"transformed": false
},
{
"source_relative_path": "collectors/dy_store_competitor_store_scraping.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/collectors/dy_store_competitor_store_scraping.py",
"category": "source",
"source_sha256": "009879a7dd341ca6ecbc8eb66f5700114c73bcaba3a53f159014a76232d0aa6b",
"target_sha256": "8d319196d56a52e084c375f872faf729d33fca400f4cabae901accb6c0bdde47",
"transformed": true
},
{
"source_relative_path": "collectors/jd_data_collector.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/collectors/jd_data_collector.py",
"category": "source",
"source_sha256": "6f00ae3b2b74e08d3c34c3b911ea819dfa4c6ce55bceef282afb3b2f565aa3ec",
"target_sha256": "1825f123b3c1604340bd95a79cad31cbaba23834d50dae69ce5b68068a5b055e",
"transformed": true
},
{
"source_relative_path": "collectors/jd_peer_store_data_collector.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/collectors/jd_peer_store_data_collector.py",
"category": "source",
"source_sha256": "4ebdb52df7b384d2bfdd2fc840d55d14e18864002c20a36685b63cee1966cf4c",
"target_sha256": "23f445aae6bcf41193db5286ce1a519d50b93e20a55f27f6df58315aafb30a08",
"transformed": true
},
{
"source_relative_path": "collectors/taobao_sycm.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/collectors/taobao_sycm.py",
"category": "source",
"source_sha256": "6978333bd60c1946e3dd8eeced1d0589461c4a26377511da7875f60bd0e31739",
"target_sha256": "6042dc251c0dbff92ca9eb8ed0a4db4322db5810e43fadf1cae79219067496fd",
"transformed": true
},
{
"source_relative_path": "config.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/config.py",
"category": "source",
"source_sha256": "2cfa101ed07643ad822f176e4b21bb72bf7ff32e3945837082811ea792a9e21b",
"target_sha256": "d9a8e32513071dacc8fd303901a3f665336d49e5dd8afe2ba3a0bc8f9dc19985",
"transformed": true
},
{
"source_relative_path": "db/__init__.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/db/__init__.py",
"category": "source",
"source_sha256": "e75ec31954ddf3c6a84e5915a6ff662027d097fdf7fba91aa5c84d048b8e960c",
"target_sha256": "e75ec31954ddf3c6a84e5915a6ff662027d097fdf7fba91aa5c84d048b8e960c",
"transformed": false
},
{
"source_relative_path": "db/db.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/db/db.py",
"category": "source",
"source_sha256": "bf600de53e60be0492426b6ffade60f75474d885263963b12b1bc4a42c03e216",
"target_sha256": "567df11eac1462111594668d66e1210fe1d1808f0f77250e557ad1cf5b66af44",
"transformed": true
},
{
"source_relative_path": "db/schema.sql",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/db/schema.sql",
"category": "sql",
"source_sha256": "5c3b90a70af4aa9e69039be8f3ec79a136ab5892e5d1ff31fc8a20d735a5dfe5",
"target_sha256": "5c3b90a70af4aa9e69039be8f3ec79a136ab5892e5d1ff31fc8a20d735a5dfe5",
"transformed": false
},
{
"source_relative_path": "lark_cli.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/lark_cli.py",
"category": "source",
"source_sha256": "434394c2864b44e612a34b16c242df2a5f89dbfb298505b22f2133af925d0233",
"target_sha256": "dd2c06c0e491f310e563b6203b078f41fd1cf1427b946332a2735df522cf13d1",
"transformed": true
},
{
"source_relative_path": "pyproject.toml",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/dependencies.toml",
"category": "dependency",
"source_sha256": "002035ffb503798d48241f962216c5baa460de4ee8a1d7d696a3d6ba6afb2975",
"target_sha256": "002035ffb503798d48241f962216c5baa460de4ee8a1d7d696a3d6ba6afb2975",
"transformed": false
},
{
"source_relative_path": "runners/__init__.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/runners/__init__.py",
"category": "source",
"source_sha256": "79b41ece107032dcaa3a7b07df8454673fae8d09f763052ec12c81e3be5126d0",
"target_sha256": "79b41ece107032dcaa3a7b07df8454673fae8d09f763052ec12c81e3be5126d0",
"transformed": false
},
{
"source_relative_path": "runners/run_peer_store.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/runners/run_peer_store.py",
"category": "launcher",
"source_sha256": "363a04b516dd6b011f07bf34feb480ccd0f1da46cb7d9512653ea7d5f4f7a3a4",
"target_sha256": "5db5365ba1a69ec325cc9e5b90aa77609045df84d5b3b8956e1718faaf30be55",
"transformed": true
},
{
"source_relative_path": "runners/run_shop.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/runners/run_shop.py",
"category": "launcher",
"source_sha256": "5c5a37fb230f531e55563e75b7cf361a3cba190707f01f516e244ac643c73027",
"target_sha256": "f12d29bd72533780270393eae855412b87f716f2540b05fed82da097821d20cb",
"transformed": true
},
{
"source_relative_path": "runners/utils.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/runners/utils.py",
"category": "source",
"source_sha256": "3bdf504593634d467524d06322d7f0ca20ab7bdf8c115573c154b8f5559c1cad",
"target_sha256": "43340178902f3531d3d68ae3dc51b2dc8117003b43c6172f32a53f04b3f6f2cb",
"transformed": true
},
{
"source_relative_path": "scripts/remove_scheduler.ps1",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/scripts/remove_scheduler.ps1",
"category": "launcher",
"source_sha256": "0532c87a6de5f7f7f65796b8c2077126091cda7d4f065c6015624599b6df1ee6",
"target_sha256": "c376ac34da5aa777634ccea5a5d2e5718c69309b17a0219cac5c19a3c6d46ca4",
"transformed": true
},
{
"source_relative_path": "scripts/setup_scheduler.ps1",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/scripts/setup_scheduler.ps1",
"category": "launcher",
"source_sha256": "8ff5418b494886ceb76acb9c8dcbccf85183f1e85fb18426c0053802ae5eb26b",
"target_sha256": "88915b6d75c62c61d13bc39d7fb959ea4b308068b9edab08db43c5c18f295f69",
"transformed": true
},
{
"source_relative_path": "writers/__init__.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/writers/__init__.py",
"category": "source",
"source_sha256": "d830cb8dd53601d06abb9481c907472bcefcadb8b65ceb0356f5baf037c3af10",
"target_sha256": "d830cb8dd53601d06abb9481c907472bcefcadb8b65ceb0356f5baf037c3af10",
"transformed": false
},
{
"source_relative_path": "writers/peer_store_writer.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/writers/peer_store_writer.py",
"category": "source",
"source_sha256": "71bf69c423bf3dd1f3d68d4e33f7024fbe0e446567d2cc3b25ec0cb71f59ab7e",
"target_sha256": "075b2862d486f43a406a3be58439813b60f3354ebba244f4d0be25a82ff618d7",
"transformed": true
},
{
"source_relative_path": "writers/shop_base_writer.py",
"target_relative_path": "src/gyxx_flow/modules/shop_intelligence/runtime/writers/shop_base_writer.py",
"category": "source",
"source_sha256": "01aa18955a0ec92f4fb745df916e5b7f1c69646c8a194e29bc0042120f59c722",
"target_sha256": "3ec3d71af95e7d50fa77d7ec684f1e86634eea0eed6be6b5728fe19a3f04ce0d",
"transformed": true
}
],
"entrypoints": [
{
"source_relative_path": "runners/run_shop.py",
"target_relative_path": "runners/run_shop.py",
"classification": "scheduled",
"workflows": [
"shop.metrics.weekly"
]
},
{
"source_relative_path": "runners/run_peer_store.py",
"target_relative_path": "runners/run_peer_store.py",
"classification": "scheduled",
"workflows": [
"shop.competitor.weekly"
]
},
{
"source_relative_path": "collectors/jd_data_collector.py",
"target_relative_path": "collectors/jd_data_collector.py",
"classification": "internal",
"workflows": [
"shop.metrics.weekly"
]
},
{
"source_relative_path": "collectors/jd_peer_store_data_collector.py",
"target_relative_path": "collectors/jd_peer_store_data_collector.py",
"classification": "internal",
"workflows": [
"shop.competitor.weekly"
]
},
{
"source_relative_path": "collectors/dy_store_competitor_store_scraping.py",
"target_relative_path": "collectors/dy_store_competitor_store_scraping.py",
"classification": "internal",
"workflows": [
"shop.metrics.weekly",
"shop.competitor.weekly"
]
},
{
"source_relative_path": "collectors/taobao_sycm.py",
"target_relative_path": "collectors/taobao_sycm.py",
"classification": "internal",
"workflows": [
"shop.metrics.weekly"
]
},
{
"source_relative_path": "scripts/setup_scheduler.ps1",
"target_relative_path": "scripts/setup_scheduler.ps1",
"classification": "maintenance",
"workflows": [
]
},
{
"source_relative_path": "scripts/remove_scheduler.ps1",
"target_relative_path": "scripts/remove_scheduler.ps1",
"classification": "maintenance",
"workflows": [
]
}
],
"intentionally_excluded": [
{
"pattern": ".git/**",
"reason": "runtime, generated, sensitive, or non-business material is not copied into module source"
},
{
"pattern": ".venv/**",
"reason": "runtime, generated, sensitive, or non-business material is not copied into module source"
},
{
"pattern": "**/__pycache__/**",
"reason": "runtime, generated, sensitive, or non-business material is not copied into module source"
},
{
"pattern": ".env",
"reason": "runtime, generated, sensitive, or non-business material is not copied into module source"
},
{
"pattern": "**/*cookies*",
"reason": "runtime, generated, sensitive, or non-business material is not copied into module source"
},
{
"pattern": "**/*profile*/**",
"reason": "runtime, generated, sensitive, or non-business material is not copied into module source"
},
{
"pattern": "**/data/**",
"reason": "runtime, generated, sensitive, or non-business material is not copied into module source"
},
{
"pattern": "**/logs/**",
"reason": "runtime, generated, sensitive, or non-business material is not copied into module source"
},
{
"pattern": "debug/**",
"reason": "runtime, generated, sensitive, or non-business material is not copied into module source"
},
{
"pattern": "uv.lock",
"reason": "runtime, generated, sensitive, or non-business material is not copied into module source"
},
{
"pattern": "README.md",
"reason": "runtime, generated, sensitive, or non-business material is not copied into module source"
},
{
"pattern": "CLAUDE.md",
"reason": "runtime, generated, sensitive, or non-business material is not copied into module source"
},
{
"pattern": "AGENTS.md",
"reason": "runtime, generated, sensitive, or non-business material is not copied into module source"
}
],
"snapshot": "current-filesystem"
}
config/source-manifests/supply_chain.json
+535
View File
@@ -0,0 +1,535 @@
{
"schema_version": 1,
"module": "supply_chain",
"source_project": "auto-flow",
"files": [
{
"source_relative_path": ".env.example",
"target_relative_path": "src/gyxx_flow/modules/supply_chain/runtime/config.example.env",
"category": "config",
"source_sha256": "ec2df00f43abd9ba570b3882c5331d9b3c09557cccbc9c79e2326df57b680124",
"target_sha256": "1fad168eece306d8df1f72852719dd2c565ef2709608a0c8724d7e43f3c6825a",
"transformed": true
},
{
"source_relative_path": "orchestrator/__init__.py",
"target_relative_path": "src/gyxx_flow/modules/supply_chain/runtime/orchestrator/__init__.py",
"category": "source",
"source_sha256": "cf08cfcdbf4332146416e2bfbd0c2890f728bf2db0ed05ccb9fe9c2793d59de2",
"target_sha256": "91fe8485b483d33dd4775a8c2f4acb274ac6d7e58b9701f4967b264579328a05",
"transformed": true
},
{
"source_relative_path": "orchestrator/config.py",
"target_relative_path": "src/gyxx_flow/modules/supply_chain/runtime/orchestrator/config.py",
"category": "source",
"source_sha256": "466791caa5439b29e6f37b92ee6afcd4018f2c7e3fbb072efe9c6832955b3400",
"target_sha256": "fed532abdfa44c33745602c317c4fdf349a207980b3ab950c7820fa875d97ad3",
"transformed": true
},
{
"source_relative_path": "orchestrator/feishu_sheets.py",
"target_relative_path": "src/gyxx_flow/modules/supply_chain/runtime/orchestrator/feishu_sheets.py",
"category": "source",
"source_sha256": "2654b076393013d090780d4142968154086b5801ae4c981ad0daa0d97bf54b18",
"target_sha256": "a243734fdc055e2aaa54b815de8df834dd75d068eaabb7f0b3e2f9cee74bd5e6",
"transformed": true
},
{
"source_relative_path": "orchestrator/mcp_workflow.py",
"target_relative_path": "src/gyxx_flow/modules/supply_chain/runtime/orchestrator/mcp_workflow.py",
"category": "source",
"source_sha256": "286e28c9db042956743d045fe40157d64c58ef70a85301c018a6eaedd8c2984d",
"target_sha256": "00ac9bce675b7c18473678f8624b9f49899eacbfb54f1b8afffd004b9c819bfe",
"transformed": true
},
{
"source_relative_path": "orchestrator/monitor.py",
"target_relative_path": "src/gyxx_flow/modules/supply_chain/runtime/orchestrator/monitor.py",
"category": "source",
"source_sha256": "aad7b1de5a9ddacb648a8b080e79eb04271ecb39b1b95e4bf3041373473b8fbd",
"target_sha256": "7d3b59dd6472bb6ecb2a282b22a93ee03ff1af6bc8280232554138d5e8578b5c",
"transformed": true
},
{
"source_relative_path": "orchestrator/pg_writer.py",
"target_relative_path": "src/gyxx_flow/modules/supply_chain/runtime/orchestrator/pg_writer.py",
"category": "source",
"source_sha256": "0991bb67f8f00de04b59bb25d7a9c4b8231c2b507589c201192faa1e4eb5711e",
"target_sha256": "472dcc68e0f61b97b258838b04ed51c7fc13efdacd559eb0df50a444b4ce182d",
"transformed": true
},
{
"source_relative_path": "orchestrator/runner.py",
"target_relative_path": "src/gyxx_flow/modules/supply_chain/runtime/orchestrator/runner.py",
"category": "launcher",
"source_sha256": "0cfe5e6557e8dee70ec4542c54b53e2153a3651eb2e35cd831e85518aabe0c66",
"target_sha256": "3ea26a6bbd9f4627e6fe1c6711b6fac9e65958c121e6b910efcbdde0b9dd9c7c",
"transformed": true
},
{
"source_relative_path": "orchestrator/scripts/batch_process.py",
"target_relative_path": "src/gyxx_flow/modules/supply_chain/runtime/orchestrator/scripts/batch_process.py",
"category": "source",
"source_sha256": "3b0765ebf5b08075425abb3d656c31327e4d8af576d3d00da2035fea95eeec3c",
"target_sha256": "71be3aa41866f3b757ef203f4b83528a248d758d1badb9a37c9e5333186563d0",
"transformed": true
},
{
"source_relative_path": "orchestrator/scripts/collect_confirmation.ps1",
"target_relative_path": "src/gyxx_flow/modules/supply_chain/runtime/orchestrator/scripts/collect_confirmation.ps1",
"category": "launcher",
"source_sha256": "90764f8c7fdcaa576f6f8a84a5115cd9fe1487035eb55de3540c8a74011c84be",
"target_sha256": "9a34dc50b0d23e1bd27dc641eb76d170e7c9f451df899d2cb7eb3b822f561a24",
"transformed": true
},
{
"source_relative_path": "orchestrator/scripts/collect_purchase_order_update.ps1",
"target_relative_path": "src/gyxx_flow/modules/supply_chain/runtime/orchestrator/scripts/collect_purchase_order_update.ps1",
"category": "launcher",
"source_sha256": "a314db3e3389a56ee62a413056d552f04da546be73ca59e266a998c23fdf5926",
"target_sha256": "88c8ba8d173eecf3655ed955cf54a4da27fdce33fb3186b7f225b6f7fe85a565",
"transformed": true
},
{
"source_relative_path": "orchestrator/scripts/collect_replenishment.ps1",
"target_relative_path": "src/gyxx_flow/modules/supply_chain/runtime/orchestrator/scripts/collect_replenishment.ps1",
"category": "launcher",
"source_sha256": "7c59aad10c5c3cb43ccee3d399359fac8c2ba9f444b02f983d3641b194b57c87",
"target_sha256": "c9ebacb106feff90fca27d4770781799e75f8f5640272b9adf92785a3ad8b1da",
"transformed": true
},
{
"source_relative_path": "orchestrator/scripts/config.json",
"target_relative_path": "src/gyxx_flow/modules/supply_chain/runtime/orchestrator/scripts/config.example.json",
"category": "config",
"source_sha256": "a293c3f4c69dbfe9c6a723c774f23894c024b38a3b8a4c4327704837743b9815",
"target_sha256": "b6fabd1260d256221cf232a8b89597969e6e1d0c7805847a1dfd395ebc1c6172",
"transformed": true
},
{
"source_relative_path": "orchestrator/scripts/insert_replenishment_bitable.py",
"target_relative_path": "src/gyxx_flow/modules/supply_chain/runtime/orchestrator/scripts/insert_replenishment_bitable.py",
"category": "source",
"source_sha256": "6f3bbf5c0b7e8007e3199bf7ce86ac3a71cd90ee3b171e0e800cf7ebedd583e5",
"target_sha256": "b255e3bff003a64010aacf0921d3707c4b76f16113023ae95b979997b223944c",
"transformed": true
},
{
"source_relative_path": "orchestrator/scripts/ProductReplenishment.py",
"target_relative_path": "src/gyxx_flow/modules/supply_chain/runtime/orchestrator/scripts/ProductReplenishment.py",
"category": "source",
"source_sha256": "02f6345142ebd84e25dc2f3b0651bc139ef31dd3994ff739efe66344c77ee49e",
"target_sha256": "6e4be0e85a649c683ef4c56cecb638247f26aeee21ab37322304c0eef99238d4",
"transformed": true
},
{
"source_relative_path": "orchestrator/scripts/PurchaseConfirmation.py",
"target_relative_path": "src/gyxx_flow/modules/supply_chain/runtime/orchestrator/scripts/PurchaseConfirmation.py",
"category": "source",
"source_sha256": "d400571da971702302b6e426a0ba12981459a5ee7919c771d6f4ff0038617643",
"target_sha256": "18319b26e07b666b29355db24c148f4f0e961965fbe423bd79d8925065da9b7a",
"transformed": true
},
{
"source_relative_path": "orchestrator/scripts/PurchaseOrderUpdate.py",
"target_relative_path": "src/gyxx_flow/modules/supply_chain/runtime/orchestrator/scripts/PurchaseOrderUpdate.py",
"category": "source",
"source_sha256": "91617a2854fbf1ef38e9f8ef6fefaa34627b477d7a39098c43d6ee5dba296aa7",
"target_sha256": "dafed7d30c8215f2965cc53dbfefd929e89330bc3a8201fa8f3d4bb30a537dfc",
"transformed": true
},
{
"source_relative_path": "orchestrator/scripts/send_card_notification.py",
"target_relative_path": "src/gyxx_flow/modules/supply_chain/runtime/orchestrator/scripts/send_card_notification.py",
"category": "source",
"source_sha256": "1397f5508a15ea7dae3d809aee9b0317b73788f3e7dc3927ff9efba5b7630a7e",
"target_sha256": "8586afcdfa04be83c984fc1be146c8ec2c9df74f7852fb656f854ebdf92a5558",
"transformed": true
},
{
"source_relative_path": "orchestrator/scripts/trigger_purchase_order_update.py",
"target_relative_path": "src/gyxx_flow/modules/supply_chain/runtime/orchestrator/scripts/trigger_purchase_order_update.py",
"category": "source",
"source_sha256": "8506001f619f88e887c4ee39684e1f46a8440affdb6ebc31076f64e8d8f973ba",
"target_sha256": "5b6a936f8e7c800fbdbe5626827097a3bfd0d12124b018c2a9e32eba99f21853",
"transformed": true
},
{
"source_relative_path": "orchestrator/sql/001_init.sql",
"target_relative_path": "src/gyxx_flow/modules/supply_chain/runtime/orchestrator/sql/001_init.sql",
"category": "sql",
"source_sha256": "a77925bb466af86d1b5e968346a77911b3c439c245be91ef5b97bb1b85152aa5",
"target_sha256": "a77925bb466af86d1b5e968346a77911b3c439c245be91ef5b97bb1b85152aa5",
"transformed": false
},
{
"source_relative_path": "orchestrator/sql/002_workflow_v2.sql",
"target_relative_path": "src/gyxx_flow/modules/supply_chain/runtime/orchestrator/sql/002_workflow_v2.sql",
"category": "sql",
"source_sha256": "290fc78180565b79c1327d9ab8a33273d63a71834fd829ecf562b0cd913412e1",
"target_sha256": "290fc78180565b79c1327d9ab8a33273d63a71834fd829ecf562b0cd913412e1",
"transformed": false
},
{
"source_relative_path": "orchestrator/sql/backfill_history.py",
"target_relative_path": "src/gyxx_flow/modules/supply_chain/runtime/orchestrator/sql/backfill_history.py",
"category": "source",
"source_sha256": "7f0a4613bb0061da69a743f3a532f5fc791f1c539307d1f7c266cde82e89a2bc",
"target_sha256": "c81b07b1ea1a59baf94e3d68d21ea961a8a318358b96c8087e24a7267eb0e83f",
"transformed": true
},
{
"source_relative_path": "orchestrator/state.py",
"target_relative_path": "src/gyxx_flow/modules/supply_chain/runtime/orchestrator/state.py",
"category": "source",
"source_sha256": "d4ea6fe12edc8edadd3e1ecef2d32ce55f0f827580d2df4932899801d0b6f8f6",
"target_sha256": "0cc6d60dbe161b633fa5c0c1af2395b1a73f21b15f692ba3042883d06bf8441b",
"transformed": true
},
{
"source_relative_path": "orchestrator/workflows.py",
"target_relative_path": "src/gyxx_flow/modules/supply_chain/runtime/orchestrator/workflows.py",
"category": "source",
"source_sha256": "a851dbcdf9c2b7d3aaca5df8bcce9e7ee325da41114d58813e1447fc5c10ec30",
"target_sha256": "ea26e462ff409ce56ce1191b9aaac8dba6947c429e5eeacf0b08776f8dcf7eca",
"transformed": true
},
{
"source_relative_path": "requirements.txt",
"target_relative_path": "src/gyxx_flow/modules/supply_chain/runtime/dependencies.txt",
"category": "dependency",
"source_sha256": "1b636fe51fce642e4d17009643013998e5f2b0670f7c11abe4c62fcf5b10e306",
"target_sha256": "1b636fe51fce642e4d17009643013998e5f2b0670f7c11abe4c62fcf5b10e306",
"transformed": false
},
{
"source_relative_path": "run.py",
"target_relative_path": "src/gyxx_flow/modules/supply_chain/runtime/run.py",
"category": "launcher",
"source_sha256": "cac7fbdbb4a47a04c87522e50130510775939fc485ce1ee2fcd2441f70c20777",
"target_sha256": "2dd03c4bdcc49e6fa0c83f816a6f42275dba38ebb5276a64125561e2d0ed13d8",
"transformed": true
},
{
"source_relative_path": "scripts/purchase-confirmation.bat",
"target_relative_path": "src/gyxx_flow/modules/supply_chain/runtime/scripts/purchase-confirmation.bat",
"category": "launcher",
"source_sha256": "6bed299064aff0befa214e7648184ed190016eef0b257abc72fb8956bc4892a5",
"target_sha256": "ef3cb48b61243b456185c207e18accb19d044bd877987b627e8a5187d63ebeb1",
"transformed": true
},
{
"source_relative_path": "scripts/replenishment.bat",
"target_relative_path": "src/gyxx_flow/modules/supply_chain/runtime/scripts/replenishment.bat",
"category": "launcher",
"source_sha256": "826c34ec5c4bec3a3408d3759c2dd462109746f5a37facb98330c551279d6724",
"target_sha256": "e87ce31edb870fb733feba93d166507917ac1aa7c5c628f017ba91bf9737bdd5",
"transformed": true
},
{
"source_relative_path": "scripts/replenishment-alert.bat",
"target_relative_path": "src/gyxx_flow/modules/supply_chain/runtime/scripts/replenishment-alert.bat",
"category": "launcher",
"source_sha256": "e72592cea97eea1af77861cbcb91d8b3b248d36389bf48897f330db664c4b320",
"target_sha256": "0e5726b6ef90b9ab000a1e3460902a70b94c00be9143a8e0e42ce7d4c0ae2252",
"transformed": true
},
{
"source_relative_path": "skills/analyzer/SKILL.md",
"target_relative_path": "src/gyxx_flow/modules/supply_chain/runtime/skills/analyzer/SKILL.md",
"category": "resource",
"source_sha256": "eb2ce257dcf3cfdec056bc90bce61c753d89be041d2e6e14521f59fcde1b9512",
"target_sha256": "e00ed56610b84ff94aa227ae77a47736aeb50995b57c0765050137122b3bac18",
"transformed": true
},
{
"source_relative_path": "skills/analyzer/workflow/purchase-order-update/SKILL.md",
"target_relative_path": "src/gyxx_flow/modules/supply_chain/runtime/skills/analyzer/workflow/purchase-order-update/SKILL.md",
"category": "resource",
"source_sha256": "3877e9c7363c31f597ee06c2fcf51268b6f2bf73865e02335d8842799bf5d829",
"target_sha256": "3bc15c2798823ba19ac42aa6d38f0d3d8b6100d44756c46d2a01c245611c25a7",
"transformed": true
},
{
"source_relative_path": "skills/collector/SKILL.md",
"target_relative_path": "src/gyxx_flow/modules/supply_chain/runtime/skills/collector/SKILL.md",
"category": "resource",
"source_sha256": "2e96c720c16b3e16eb0253af109128cd6df2ed98b72d23bddfb473e36fc5ea2c",
"target_sha256": "dc241106ef22f67098e82b13b78c986102bcfa3498778310a8a1d1ddbba70210",
"transformed": true
},
{
"source_relative_path": "skills/collector/workflow/purchase-confirmation-workflow/SKILL.md",
"target_relative_path": "src/gyxx_flow/modules/supply_chain/runtime/skills/collector/workflow/purchase-confirmation-workflow/SKILL.md",
"category": "resource",
"source_sha256": "12b67cfb73059417c053fc6ae2f2b7afe894b0ed4c66160f6d23647a6f7ddf09",
"target_sha256": "3e7fa623f8332e4a99865170cfa44bc2426338ee7369a6a64c90268a74a9fe2f",
"transformed": true
},
{
"source_relative_path": "skills/collector/workflow/purchase-order-update-workflow/SKILL.md",
"target_relative_path": "src/gyxx_flow/modules/supply_chain/runtime/skills/collector/workflow/purchase-order-update-workflow/SKILL.md",
"category": "resource",
"source_sha256": "407469efe6129486fd5f21829fd3039a1ab63fcbed8761176a0a0ce61b670167",
"target_sha256": "a7be00de8a21c5087d57c9b4aaa4aff7fd9c5816dc29c5ac0862e1342b870fdd",
"transformed": true
},
{
"source_relative_path": "skills/collector/workflow/replenishment-alert-workflow/SKILL.md",
"target_relative_path": "src/gyxx_flow/modules/supply_chain/runtime/skills/collector/workflow/replenishment-alert-workflow/SKILL.md",
"category": "resource",
"source_sha256": "a57f429c4ade7771207905820f4f98edeff6e9a44fcf7957b820a6e5fc70f94b",
"target_sha256": "1c96830cbeb1cedd6d8205044f03a30f50004dc21ff0995e8b67139dd76eaa6e",
"transformed": true
},
{
"source_relative_path": "skills/collector/workflow/replenishment-workflow/SKILL.md",
"target_relative_path": "src/gyxx_flow/modules/supply_chain/runtime/skills/collector/workflow/replenishment-workflow/SKILL.md",
"category": "resource",
"source_sha256": "569f7da35c6f50d892bef31d65152d138078053135d0f442416fbebbb2c09561",
"target_sha256": "906ad7b0981bb949631a159c1df4c5221f540f77c70ea494633c8f3b3c786fdf",
"transformed": true
},
{
"source_relative_path": "skills/collector/workflow/workflow-trigger-only/SKILL.md",
"target_relative_path": "src/gyxx_flow/modules/supply_chain/runtime/skills/collector/workflow/workflow-trigger-only/SKILL.md",
"category": "resource",
"source_sha256": "4949170a88d116eb21f7c06da74b73ef4a7a15dc4851c281fd460a73806c102d",
"target_sha256": "810c33bd2f3a17efc2e8728fd7b2bc9164862f0e11c154989d5a42350d6ca7ea",
"transformed": true
}
],
"entrypoints": [
{
"source_relative_path": "scripts/purchase-confirmation.bat",
"target_relative_path": "scripts/purchase-confirmation.bat",
"classification": "scheduled",
"workflows": [
"supply.purchase_confirmation.daily"
]
},
{
"source_relative_path": "scripts/replenishment.bat",
"target_relative_path": "scripts/replenishment.bat",
"classification": "scheduled",
"workflows": [
"supply.replenishment.weekly"
]
},
{
"source_relative_path": "scripts/replenishment-alert.bat",
"target_relative_path": "scripts/replenishment-alert.bat",
"classification": "scheduled",
"workflows": [
"supply.replenishment_alert.daily"
]
},
{
"source_relative_path": "run.py",
"target_relative_path": "run.py",
"classification": "manual",
"workflows": [
"supply.purchase_confirmation.daily",
"supply.replenishment.weekly",
"supply.replenishment_alert.daily",
"supply.purchase_order_update"
]
},
{
"source_relative_path": "orchestrator/runner.py",
"target_relative_path": "orchestrator/runner.py",
"classification": "manual",
"workflows": [
"supply.purchase_confirmation.daily",
"supply.replenishment.weekly",
"supply.replenishment_alert.daily",
"supply.purchase_order_update"
]
},
{
"source_relative_path": "orchestrator/scripts/trigger_purchase_order_update.py",
"target_relative_path": "orchestrator/scripts/trigger_purchase_order_update.py",
"classification": "event",
"workflows": [
"supply.purchase_order_update"
]
},
{
"source_relative_path": "orchestrator/monitor.py",
"target_relative_path": "orchestrator/monitor.py",
"classification": "maintenance",
"workflows": [
]
},
{
"source_relative_path": "orchestrator/sql/backfill_history.py",
"target_relative_path": "orchestrator/sql/backfill_history.py",
"classification": "maintenance",
"workflows": [
]
},
{
"source_relative_path": "orchestrator/mcp_workflow.py",
"target_relative_path": "orchestrator/mcp_workflow.py",
"classification": "internal",
"workflows": [
"supply.purchase_confirmation.daily",
"supply.replenishment.weekly",
"supply.replenishment_alert.daily",
"supply.purchase_order_update"
]
},
{
"source_relative_path": "orchestrator/scripts/collect_confirmation.ps1",
"target_relative_path": "orchestrator/scripts/collect_confirmation.ps1",
"classification": "internal",
"workflows": [
"supply.purchase_confirmation.daily"
]
},
{
"source_relative_path": "orchestrator/scripts/collect_replenishment.ps1",
"target_relative_path": "orchestrator/scripts/collect_replenishment.ps1",
"classification": "internal",
"workflows": [
"supply.replenishment.weekly",
"supply.replenishment_alert.daily"
]
},
{
"source_relative_path": "orchestrator/scripts/collect_purchase_order_update.ps1",
"target_relative_path": "orchestrator/scripts/collect_purchase_order_update.ps1",
"classification": "internal",
"workflows": [
"supply.purchase_order_update"
]
},
{
"source_relative_path": "orchestrator/scripts/PurchaseConfirmation.py",
"target_relative_path": "orchestrator/scripts/PurchaseConfirmation.py",
"classification": "internal",
"workflows": [
"supply.purchase_confirmation.daily"
]
},
{
"source_relative_path": "orchestrator/scripts/ProductReplenishment.py",
"target_relative_path": "orchestrator/scripts/ProductReplenishment.py",
"classification": "internal",
"workflows": [
"supply.replenishment.weekly",
"supply.replenishment_alert.daily"
]
},
{
"source_relative_path": "orchestrator/scripts/PurchaseOrderUpdate.py",
"target_relative_path": "orchestrator/scripts/PurchaseOrderUpdate.py",
"classification": "internal",
"workflows": [
"supply.purchase_order_update"
]
},
{
"source_relative_path": "orchestrator/scripts/batch_process.py",
"target_relative_path": "orchestrator/scripts/batch_process.py",
"classification": "internal",
"workflows": [
"supply.replenishment.weekly",
"supply.replenishment_alert.daily"
]
},
{
"source_relative_path": "orchestrator/scripts/insert_replenishment_bitable.py",
"target_relative_path": "orchestrator/scripts/insert_replenishment_bitable.py",
"classification": "internal",
"workflows": [
"supply.replenishment.weekly"
]
},
{
"source_relative_path": "orchestrator/scripts/send_card_notification.py",
"target_relative_path": "orchestrator/scripts/send_card_notification.py",
"classification": "internal",
"workflows": [
"supply.purchase_confirmation.daily",
"supply.replenishment.weekly",
"supply.replenishment_alert.daily",
"supply.purchase_order_update"
]
}
],
"intentionally_excluded": [
{
"pattern": ".git/**",
"reason": "runtime, generated, sensitive, historical, or machine-local material is not copied into module source"
},
{
"pattern": ".venv/**",
"reason": "runtime, generated, sensitive, historical, or machine-local material is not copied into module source"
},
{
"pattern": "**/__pycache__/**",
"reason": "runtime, generated, sensitive, historical, or machine-local material is not copied into module source"
},
{
"pattern": ".env",
"reason": "runtime, generated, sensitive, historical, or machine-local material is not copied into module source"
},
{
"pattern": "**/*cookies*",
"reason": "runtime, generated, sensitive, historical, or machine-local material is not copied into module source"
},
{
"pattern": "**/*profile*/**",
"reason": "runtime, generated, sensitive, historical, or machine-local material is not copied into module source"
},
{
"pattern": "**/data/**",
"reason": "runtime, generated, sensitive, historical, or machine-local material is not copied into module source"
},
{
"pattern": "**/logs/**",
"reason": "runtime, generated, sensitive, historical, or machine-local material is not copied into module source"
},
{
"pattern": ".cache/**",
"reason": "runtime, generated, sensitive, historical, or machine-local material is not copied into module source"
},
{
"pattern": ".hermes/**",
"reason": "runtime, generated, sensitive, historical, or machine-local material is not copied into module source"
},
{
"pattern": ".hermes_tmp/**",
"reason": "runtime, generated, sensitive, historical, or machine-local material is not copied into module source"
},
{
"pattern": "_tmp/**",
"reason": "runtime, generated, sensitive, historical, or machine-local material is not copied into module source"
},
{
"pattern": "shared-data/**",
"reason": "runtime, generated, sensitive, historical, or machine-local material is not copied into module source"
},
{
"pattern": "orchestrator/state.db*",
"reason": "runtime, generated, sensitive, historical, or machine-local material is not copied into module source"
},
{
"pattern": "memory/**",
"reason": "runtime, generated, sensitive, historical, or machine-local material is not copied into module source"
},
{
"pattern": "orchestrator/scripts/config_*.json",
"reason": "runtime, generated, sensitive, historical, or machine-local material is not copied into module source"
},
{
"pattern": "orchestrator/scripts/config_jd.local.json",
"reason": "runtime, generated, sensitive, historical, or machine-local material is not copied into module source"
}
],
"snapshot": "current-filesystem"
}
config/workflows.json
+38
View File
@@ -0,0 +1,38 @@
{
"schema_version": 2,
"workflows": [
{"id":"content.metrics.daily","module":"content_marketing","trigger":"scheduled","execution":{"entry":"data/tools/daily_run.bat"},"provenance":{"source_project":"content","task_name":"YingxiaoYunying_DailyRun"}},
{"id":"content.marketing_report.daily","module":"content_marketing","trigger":"scheduled","execution":{"entry":"data/tools/daily_marketing_report.bat"},"provenance":{"source_project":"content","task_name":"YingxiaoYunying_DailyMarketingReport"}},
{"id":"content.relogin.weekly","module":"content_marketing","trigger":"scheduled","execution":{"entry":"data/tools/friday_relogin.bat"},"provenance":{"source_project":"content","task_name":"YingxiaoYunying_FridayRelogin"}},
{"id":"content.self_operated.weekly","module":"content_marketing","trigger":"scheduled","execution":{"entry":"data/tools/monday_self_run.bat"},"provenance":{"source_project":"content","task_name":"YingxiaoYunying_MondaySelf"}},
{"id":"content.creator_report.monthly","module":"content_marketing","trigger":"scheduled","execution":{"entry":"data/tools/monthly_creator_report.bat"},"provenance":{"source_project":"content","task_name":"YingxiaoYunying_MonthlyCreatorReport"}},
{"id":"content.summary.monthly","module":"content_marketing","trigger":"scheduled","execution":{"entry":"data/tools/monthly_summary.bat"},"provenance":{"source_project":"content","task_name":"YingxiaoYunying_MonthlySummary"}},
{"id":"content.cooperations.daily","module":"content_marketing","trigger":"scheduled","execution":{"entry":"data/tools/sync_cooperations.bat"},"provenance":{"source_project":"content","task_name":"YingxiaoYunying_SyncCooperations"}},
{"id":"content.comments.weekly","module":"content_marketing","trigger":"scheduled","execution":{"entry":"data/tools/weekly_comment_scrape.bat"},"provenance":{"source_project":"content","task_name":"YingxiaoYunying_WeeklyCommentScrape"}},
{"id":"content.summary.weekly","module":"content_marketing","trigger":"scheduled","execution":{"entry":"data/tools/weekly_summary.bat"},"provenance":{"source_project":"content","task_name":"YingxiaoYunying_WeeklySummary"}},
{"id":"shop.metrics.weekly","module":"shop_intelligence","trigger":"scheduled","execution":{"entry":"runners/run_shop.py"},"provenance":{"source_project":"shop","task_name":"shop-data-collection"}},
{"id":"shop.competitor.weekly","module":"shop_intelligence","trigger":"scheduled","execution":{"entry":"runners/run_peer_store.py"},"provenance":{"source_project":"shop","task_name":"peer-store-data-collection"}},
{"id":"product.persona.daily","module":"product_commerce","trigger":"scheduled","execution":{"entry":"run_daily_persona.py"},"provenance":{"source_project":"product","task_name":"PersonaDailyCollect"}},
{"id":"product.daily","module":"product_commerce","trigger":"scheduled","execution":{"entry":"orchestrate_daily_collection.py","args":["--stages","collect,analyze,export,insert"]},"provenance":{"source_project":"product","task_name":"ProductCollectorDailyCollect"}},
{"id":"product.alert.daily","module":"product_commerce","trigger":"scheduled","execution":{"entry":"commands/run_alerts.py"},"provenance":{"source_project":"product","task_name":"ProductCollectorSalesAlert"}},
{"id":"product.import.daily","module":"product_commerce","trigger":"scheduled","execution":{"entry":"commands/import_daily.py"},"provenance":{"source_project":"product","task_name":"ProductDailyImport"}},
{"id":"product.style_analysis.interval","module":"product_commerce","trigger":"scheduled","execution":{"entry":"analyze_style_with_hermes.py","args":["--all-styles","--days","3","--min-interval-days","3","--skip-existing"]},"provenance":{"source_project":"product","task_name":"StyleAnalysisEvery3Days"}},
{"id":"product.main_image.jd.weekly","module":"product_commerce","trigger":"scheduled","execution":{"entry":"run_weekly_jd_main_image.py","args":["--headless"]},"provenance":{"source_project":"product","task_name":"WeeklyJdMainImageCollect"}},
{"id":"product.main_image.weekly","module":"product_commerce","trigger":"scheduled","execution":{"entry":"run_weekly_main_image.py","args":["--headless"]},"provenance":{"source_project":"product","task_name":"WeeklyMainImageCollect"}},
{"id":"supply.purchase_confirmation.daily","module":"supply_chain","trigger":"scheduled","execution":{"entry":"run.py","args":["mcp-run","purchase-confirmation"]},"provenance":{"source_project":"supply","task_name":"auto-flow-purchase-confirmation"}},
{"id":"supply.replenishment.weekly","module":"supply_chain","trigger":"scheduled","execution":{"entry":"run.py","args":["mcp-run","replenishment"]},"provenance":{"source_project":"supply","task_name":"auto-flow-replenishment"}},
{"id":"supply.replenishment_alert.daily","module":"supply_chain","trigger":"scheduled","execution":{"entry":"run.py","args":["mcp-run","replenishment-alert"]},"provenance":{"source_project":"supply","task_name":"auto-flow-replenishment-alert"}},
{"id":"content.mapping.refresh","module":"content_marketing","trigger":"manual","execution":{"entry":"data/tools/rebuild_mapping.py"},"provenance":{"source_project":"content"}},
{"id":"content.retry_failed","module":"content_marketing","trigger":"manual","execution":{"entry":"data/tools/retry_failed.py"},"provenance":{"source_project":"content"}},
{"id":"content.metrics.backfill","module":"content_marketing","trigger":"manual","execution":{"entry":"data/tools/daily_run_with_backfill.bat"},"provenance":{"source_project":"content"}},
{"id":"product.backfill","module":"product_commerce","trigger":"manual","execution":{"entry":"backfill_collect.py"},"provenance":{"source_project":"product"}},
{"id":"product.market_rank","module":"product_commerce","trigger":"manual","execution":{"entry":"orchestrate_market_rank_collection.py"},"provenance":{"source_project":"product"},"note":"A launcher exists but no matching system task is registered."},
{"id":"product.review_collection","module":"product_commerce","trigger":"manual","execution":{"entry":"orchestrate_review_collection.py"},"provenance":{"source_project":"product"}},
{"id":"supply.purchase_order_update","module":"supply_chain","trigger":"manual","execution":{"entry":"run.py","args":["mcp-run","purchase-order-update"]},"provenance":{"source_project":"supply"}},
{"id":"product.weekly_aggregate.documented_missing","module":"product_commerce","trigger":"unavailable","execution":{"entry":"run_weekly_collect.py"},"provenance":{"source_project":"product"},"note":"Documented in README but the entry file and scheduled task are absent."}
]
}
design.md
+157
View File
@@ -0,0 +1,157 @@
# GYXX Flow 统一编排平台设计
## 1. 目标
把 `D:\yingxiaoyunying`、`D:\shop-data-flow`、
`D:\product-collector-analyze-flow` 和 `E:\auto-flow` 的业务源码、启动脚本和工作流
完整迁移到 `D:\gyxx-flow`。新系统采用模块化单体、统一 CLI、声明式调度和分层
数据目录，同时允许旧任务在迁移期继续运行并能逐任务回滚。
“完整迁移”的硬性定义是：删除、改名或断开四个旧项目目录后，新项目仍能完成
入口发现、导入、dry-run、手工执行和调度执行。旧项目只能作为迁移输入和生产对照，
不得成为新项目运行时依赖。仅登记旧入口或通过环境变量调用旧脚本不算迁移完成。
## 2. 边界
- 新代码和迁移工具只写入 `D:\gyxx-flow`。
- 在生产切换门禁前，不修改四个旧项目或现有计划任务。
- 允许从旧项目做一次性只读复制；复制后的代码归属新项目并在新项目内改造、测试。
- 运行时代码不得读取 `GYXX_LEGACY_*_ROOT`，不得导入或启动四个旧目录中的文件。
- `D:\comment-data-collector` 不在迁移范围内；它与内容模块共享数据库表的字段
所有权以外部契约表示。
- 真实 Cookie、密码、应用密钥、访问令牌和浏览器 Profile 不进入源码仓库。
## 3. 方案
选择模块化单体而不是原样拼目录或直接拆微服务。四个业务模块为：
1. `content_marketing`：内容指标、评论、达人、周报/月报和登录维护。
2. `product_commerce`：ERP、商品、画像、主图、市场排名、分析和告警。
3. `shop_intelligence`：店铺及竞店周数据。
4. `supply_chain`：采购确认、库存预警、补货和采购单更新。
共享能力通过端口和适配器提供：文件产物、PostgreSQL、飞书、Hermes、浏览器和
平台采集。业务模块不得导入其他业务模块的内部实现。
每个业务模块包含自己的 `jobs/`、`collectors/`、`services/`、`models/` 和
`resources/`。一次性复制阶段可以在模块内保留原相对结构以降低行为变化风险，但
所有入口必须改为由模块清单定位，所有数据/日志/Profile/临时文件必须改由
`RunContext` 与 `DataLayout` 提供。模块间共享只允许依赖 `core`、`workflow`、
`adapters` 和显式契约。
## 4. 运行模型
调度器只调用稳定命令 `gyxx run <workflow-id>`。工作流引擎负责：
- 创建 `run_id` 和 `RunContext`；
- 获取工作流及资源锁；
- 按依赖顺序执行步骤；
- 记录步骤状态、退出码、耗时和错误；
- 进行有上限的步骤级重试；
- 生成不可变产物清单和 SHA-256；
- 在 shadow 模式关闭生产写入和正式通知。
Windows Task Scheduler 只是一个调度适配器。工作流和时间表的唯一事实来源是仓库
内的声明式配置，因此可以生成 Windows、Cron 或其他平台的调度配置。
## 5. 数据模型
运行根目录由 `GYXX_DATA_ROOT` 指定，默认是仓库下 `var`：
```text
var/
data/raw/<domain>/<source>/<dataset>/business_date=<date>/run_id=<id>/
data/normalized/<domain>/<dataset>/schema_vN/
data/curated/<domain>/<dataset>/
data/exports/<consumer>/<workflow>/<date>/<run_id>/
data/evidence/<workflow>/<run_id>/
data/legacy/<source_project>/
runs/<workflow>/<yyyy>/<mm>/<dd>/<run_id>/
state/{browser_profiles,cookies,checkpoints,locks}/
logs/<workflow>/<yyyy>/<mm>/<dd>/
quarantine/
tmp/
```
原始数据只追加。每个产物都有 `artifact_id`、数据集、业务日期、schema 版本、行数、
字节数、SHA-256、源路径和上游产物引用。历史文件先原样复制到 `legacy`，校验通过后
再生成标准化数据。
## 6. 安全和幂等
- 配置只保存环境变量名或凭据引用，不保存密钥值。
- 生产写入使用 `workflow + business_date + entity_id + schema_version` 幂等键。
- 外部通知使用 outbox/回执，失败重试不会重复发送。
- 飞书、数据库和通知都是可替换 Sink；shadow 模式使用测试 Sink。
- 浏览器 Profile 使用独立资源锁，禁止不同工作流并发写同一 Profile。
## 7. 迁移策略
采用源码接管迁移：基线清单 -> 新基础设施 -> 源码只读复制 -> 新项目内路径与配置
改造 -> 无旧目录测试 -> 影子运行 -> 单任务切换 -> 观察 -> 退役。旧命令适配器只
用于盘点阶段，不能出现在最终工作流定义中。切换时只禁用一个旧任务，不删除；失败
时关闭新任务并重新启用旧任务。旧项目至少保留只读 30 天。
迁移顺序为 `shop_intelligence`、`supply_chain`、`content_marketing`、
`product_commerce`。最后才合并真正重复的适配器，避免因名称相同而过早抽象不同语义
的采集器。
## 8. 验收原则
工程验收依赖自动化测试、静态扫描、源码哈希清单、入口覆盖、旧根目录引用扫描、
隔离旧目录的导入/dry-run、调度生成和历史回放。四模块所有纳入迁移范围的源码必须
在新项目中有目标文件和来源哈希；所有真实入口必须标明 scheduled、manual、library
或 intentionally-excluded，不能静默遗漏。生产验收必须以真实任务运行记录为证据：
日任务连续 7 天，周任务连续 2 个周期，月任务通过历史月份回放。任何缺少证据的
项目保持未勾选状态。
## 9. 统一运行时适配器
每个可执行脚本以完整 `script_id=<module>:<entry>` 作为运行时隔离主键。
`config/runtime-bindings.json` 显式保存 130 个脚本的固定 CDP 端口，范围为
`22000..22999`；配置加载时必须同时满足入口 100% 覆盖、端口唯一和回环地址约束。
新增脚本只能领取未使用端口，已有脚本端口不得随发现顺序漂移。
每个脚本的浏览器状态固定落到：
```text
<GYXX_DATA_ROOT>/state/browser/<module>/<script-name>-<id-digest>/
profile/
cookies.json
storage_state.json
```
同一脚本跨运行复用上述目录，不同脚本不共享端口、Profile、Cookie 或 storage state。
顶层 workflow 和 `gyxx scripts run` 均由 `ModuleCommandAdapter` 注入绑定；嵌套 Python
进程在业务模块导入前由 adapter-owned `sitecustomize` 按真实子脚本路径重新绑定，且会
纠正父进程遗留的 `--cdp-port`、`--cdp-url` 和 `--user-data-dir` 参数。PowerShell/BAT
需要浏览器时通过 `python -m gyxx_flow.runtime_exec env <script-id>` 读取目标叶子脚本绑定。
外部系统策略是运行时硬边界，而不是更换原业务后端：
- 飞书继续使用原来的 lark-cli profile、身份和原 OpenAPI 应用；适配器只透传并禁止误覆盖。
- PostgreSQL 继续使用现有云端配置，统一映射 `PG_*`、`DB_*`、`AUTOFLOW_PG_*`，拒绝回环数据库。
- Hermes 继续使用本机 HTTP/CLI，所有 HTTP 端点只允许 `localhost`、`127.0.0.1` 或 `::1`。
- Cookie 和 storage state 通过原子替换保存；日志和绑定输出只包含路径、端口和计数，不包含值。
dry-run 只解析和验证绑定，不创建 Profile/Cookie 目录，不探测或启动浏览器，也不触发任何
飞书、数据库或 Hermes 调用。
## 10. 采集数据统一分层
`GYXX_DATA_ROOT` 表示整套运行数据根，不直接等同于 raw 目录。四个业务模块通过共享的 `ModuleDataPaths` 解析固定目录，模块脚本继续使用原有公开常量，避免业务逻辑与物理目录结构耦合：
```text
<GYXX_DATA_ROOT>/
data/
raw/<module>/ # 原始 JSON、CSV、Excel、下载文件、原始截图
normalized/<module>/ # 清洗并统一字段后的数据
curated/<module>/ # 聚合、分析和可供下游复用的数据
exports/<module>/ # Markdown、Excel 等面向人员或外部消费的报告
evidence/<module>/ # 产物清单、哈希和验收证据
state/<module>/ # checkpoint；浏览器状态仍按 script_id 隔离
logs/<module>/
tmp/<module>/
```
raw 数据保持追加语义；浏览器直接下载的 XLS/XLSX/CSV、接口原始 JSON 和采集截图归 raw，处理过程中的临时下载归 tmp，清洗结果归 normalized，跨来源汇总归 curated，最终 MD/XLSX 报告归 exports。现有 `data/raw/<module>/legacy` 只读保留，不在本次收口中移动或重写。路径对象只负责解析，不在导入或 dry-run 时创建目录，因此项目根和数据根可分别迁移。
docs/deployment.md
+39
View File
@@ -0,0 +1,39 @@
# 部署手册
## 环境要求
- Windows 10/11 或 Windows Server，系统时区 `China Standard Time`
- Python 3.12、`uv`、Windows Task Scheduler
- 凭据由环境变量或外部密钥系统提供，不写入源码或清单
## 安装
```powershell
cd D:\gyxx-flow
uv sync --python 3.12 --extra test
$env:GYXX_DATA_ROOT = 'D:\gyxx-flow\var'
.\.venv\Scripts\python.exe -m gyxx_flow doctor --json
.\.venv\Scripts\python.exe -m pytest
```
不再配置任何 `GYXX_LEGACY_*_ROOT`。运行代码和资源随 `gyxx_flow` 包部署，
四个旧项目可以不挂载。商品模块需要独立配置时，可设置 `GYXX_PRODUCT_CONFIG`；
该变量只指向新部署的配置文件。
## 生成候选调度计划
```powershell
.\.venv\Scripts\python.exe -m gyxx_flow schedule plan `
--output D:\gyxx-flow\var\schedule-plan\candidate `
--start-date 2026-07-27 `
--python-executable D:\gyxx-flow\.venv\Scripts\python.exe
```
输出包含 21 个 XML、`install.ps1`、`plan.json` 和 `drift.json`。生成计划不会
注册任务。只有生产门禁通过并获得明确授权后，才能人工审阅并逐个使用
`install.ps1 -Apply -WorkflowId <id>`；安装器强制一次只处理一个任务。
## 可迁移部署
复制项目或安装 wheel 后，只需重新设置 `GYXX_DATA_ROOT` 和凭据。禁止在生产任务
命令中出现旧项目盘符、用户目录解释器或旧项目工作目录。
docs/migration-runbook.md
+29
View File
@@ -0,0 +1,29 @@
# 迁移手册
## 当前边界
- 四个旧项目的源码、数据和现有计划任务保持原状。
- 工作流及其业务脚本已复制到 `src/gyxx_flow/modules/*/runtime`，运行时不引用旧项目。
- 21 个定时工作流可生成候选系统任务；默认不应用。
- 所有发现到的本地可执行脚本可通过 `gyxx scripts` 手工 dry-run 或执行。
- 历史数据副本按模块落入新数据根，并保留数量、字节和 SHA-256 对账证据。
## 工程验收
1. 四份 `config/source-manifests/*.json` 的目标文件和 SHA-256 全部通过。
2. 扫描源码、配置和启动器，确认无四个旧项目根目录或 `GYXX_LEGACY_*_ROOT`。
3. 在不设置旧项目环境变量的进程中导入目录、列出脚本并 dry-run 21 个工作流。
4. 执行 `uv run pytest`、`uv run gyxx doctor --json` 和 wheel 构建检查。
5. 只生成调度计划，不注册、不禁用任何系统任务。
## 逐任务生产切换
1. 轮换历史明文凭据，并将新凭据放入外部密钥系统或环境变量。
2. 对目标 workflow 做 dry-run 和 shadow 对账。
3. 审核 run journal、artifact manifest、effect ledger 和 outbox。
4. 取得负责人明确授权，只安装一个新任务并禁用对应旧任务；旧任务不删除。
5. 日任务连续观察 7 天，周任务观察 2 个周期，月任务完成指定历史月回放。
6. 数据库、飞书、文件产物和通知对账通过后，才迁移下一任务。
推荐顺序：`shop_intelligence` → `supply_chain` → `content_marketing` →
`product_commerce`。
docs/operations.md
+29
View File
@@ -0,0 +1,29 @@
# 日常运维手册
## 常用命令
```powershell
gyxx list --json
gyxx scripts list --json
gyxx doctor --json
gyxx acceptance status --json
gyxx run shop.metrics.weekly --date 2026-07-27
gyxx backfill shop.metrics.weekly --from 2026-07-21 --to 2026-07-27
gyxx scripts run shop_intelligence:runners/run_shop.py --date 2026-07-27
```
`run`、`backfill`、`scripts run` 默认 dry-run。`--execute` 才启动本地迁入脚本；
Task Scheduler 使用 `--scheduled`，按 Asia/Shanghai 当日执行。
## 数据与证据
- `var/runs/.../<run_id>/run.json`：步骤、尝试、退出码和 trace
- `var/state/ops/run-index/`：按 run_id/workflow/date/status 查询
- `var/state/ops/effects/`：生产 Sink 幂等回执
- `var/state/ops/outbox/messages/`：外部写入消息
- `var/logs/`、`var/data/evidence/`：日志和截图证据
- `var/state/profiles/`：浏览器 Profile
- `var/state/locks/`：工作流、Profile 和共享资源锁
effect 为 `ambiguous` 或长时间 `in_progress` 时，先人工核对外部系统；不要删除回执
或强制重跑。手工脚本执行同样会产生统一 run journal 和 effect 记录。
docs/rollback.md
+17
View File
@@ -0,0 +1,17 @@
# 回滚手册
## 单任务回滚
1. 记录失败的新任务名、workflow ID、run_id 和业务日期。
2. 禁用对应 `\GYXX\<workflow-id>`，保留任务定义和运行证据。
3. 重新启用原计划任务，核对触发时间、账号和旧工作目录仍与基线一致。
4. 检查 `state/ops/effects`；`in_progress` 或 `ambiguous` 必须先做外部对账。
5. 检查 outbox；已 `sent` 的消息不得重发，`failed` 只能以同一幂等键 replay。
6. 记录回滚结果和恢复时间，保留新旧日志。
## 原则
- 一次只切换或回滚一个 workflow。
- 旧任务只禁用/启用，不删除；旧项目在全部观察期完成前保持只读可回滚。
- 不删除新系统数据、run journal、effect receipt 或 outbox 消息。
- 未完成对账时，不允许用直接运行脚本绕过 effect ledger。
final_report.md
+72
View File
@@ -0,0 +1,72 @@
# GYXX Flow 源码迁移阶段报告
## 当前结论
核心采集脚本已经修改，但修改的是复制到 `D:\gyxx-flow` 的新项目副本；四个旧项目的
原始代码和现有定时任务均未修改。新工作流不再通过环境变量、动态导入或子进程引用旧
项目代码。
当前可用能力：
- 统一列出并 dry-run/执行 21 个定时工作流和 7 个正式手工工作流；
- 统一发现并 dry-run/执行 131 个迁入后的本地脚本；
- 生成 21 个 Windows Task Scheduler XML 和单任务安装器；
- 将新数据按模块写入 `data/{raw,normalized,curated,exports,evidence}/<module>`，运行状态、
日志和临时文件写入 `state/logs/tmp`；
- 更换项目根和数据根后继续完成入口发现与 dry-run。
## 已完成
- [x] 四模块源码实体迁移及逐文件来源/目标哈希清单
- [x] 21 个计划任务到新项目本地入口的一一映射
- [x] 131 个本地可执行脚本目录与统一手工入口
- [x] 旧项目根目录、旧环境变量和旧配置回退清零
- [x] 数据、日志、Profile、状态和临时目录迁入可配置数据根
- [x] 敏感默认值清理、来源清单校验、重定位测试和全量自动化测试
- [x] 只生成不应用的候选调度包；安装器强制逐任务应用
- [x] 部署、迁移、运维和回滚手册更新
- [x] 131 个脚本唯一 CDP 端口、独立可迁移 Profile/Cookie/storage state
- [x] 原飞书透传、现有云 PostgreSQL 和本机 Hermes 的统一运行时边界
## 未完成
- [ ] 为每个外部写入证明 run_id 追踪和重跑幂等
- [ ] 用户授权后的凭据轮换、逐任务生产切换和回滚演练
- [ ] 日任务 7 天、周任务 2 个周期、月任务历史回放及最终对账
`gyxx acceptance status` 在上述生产门禁完成前应继续返回 incomplete。这是有意保留的
安全状态，不影响当前已具备的本地手工运行和候选任务注册能力。
## 统一适配器补充验收（2026-07-27）
- `config/runtime-bindings.json` 覆盖 131/131 脚本，CDP 端口 131 个且无重复。
- 顶层 workflow/manual 与嵌套 Python/PowerShell 均按目标脚本 ID 重新绑定。
- Cookie、storage state 原子保存并跨运行复用，全部位于可迁移数据根。
- 生产源码中固定 `9222/18801/18802/18803`、源码目录 Profile 和临时即删 Profile 已清除。
- 飞书身份/调用后端保持原样；三套数据库变量映射到现有云端，回环数据库被拒绝；
Hermes 非回环 URL 被拒绝。
- Python 3.12 全量测试 `294 passed`；框架 Ruff、compileall、敏感信息和迁移哈希通过。
## 采集数据目录收口补充验收（2026-07-27）
- [x] 四模块统一使用 `data/{raw,normalized,curated,exports,evidence}/<module>`。
- [x] JSON、CSV、XLS/XLSX、浏览器下载和原始截图进入 raw；清洗结果进入 normalized；
聚合结果进入 curated；最终 Markdown/Excel 报告进入 exports。
- [x] checkpoint、缓存、锁、浏览器状态、日志和临时处理文件不再混入 raw 或源码目录。
- [x] CLI/vendor 自定义输出受数据根边界校验，不能写到 cwd、源码树或数据根之外。
- [x] 历史 `legacy` 前后均为 48,961 个文件、1,091,967,729 字节，无移动、删除或改写。
- [x] raw 按 `run_id` 追加保留，不清理历史原始产物；浏览器 Profile/Cookie 在任务结束后保留复用。
- [x] 空/纯空白数据根安全回退项目 `var`，不会把 cwd 当成数据根。
- [x] 全量 `294 passed`，Ruff、compileall、PowerShell AST 和来源清单哈希通过。
本次只修改 `D:\gyxx-flow` 内迁入副本和工程文件，没有修改四个旧项目，也没有注册、
禁用或改动任何现有定时任务；未启动真实浏览器采集，未写飞书、云端数据库或 Hermes。
## 四源项目增量重新对齐（2026-07-28）
- [x] 内容营销 4 个源端修复文件及两份动态映射资源已重新对齐。
- [x] 商品商业 ERP 天猫权威来源修复已重新对齐。
- [x] 源端新增波塞冬历史销量回填脚本及测试已实体迁入并改接统一数据分层。
- [x] 新脚本可通过统一脚本目录手工启动，使用独立 CDP 端口和可复用 Cookie 状态。
- [x] 四份来源清单现覆盖 250 个文件；目标哈希、全量 294 项测试、18 项定向回归、compileall 和 doctor 通过。
- [ ] Ruff 未安装在当前 `.venv`，未作为本轮通过项；P6/P7 的真实外部写入和生产切换门禁继续保持未完成。
plan.md
+126
View File
@@ -0,0 +1,126 @@
# GYXX Flow 执行计划与验收清单
规则：只有完成对应验证并保存证据后才把 `[ ]` 改为 `[x]`。每次更新复核
`python -m pytest`、`gyxx acceptance status` 和本文件。
## P0 设计与迁移基线
- [x] P0.1 固化模块化单体、数据分层、影子运行和逐任务回滚设计。
- 文件：`design.md`
- 验证：设计包含目标、边界、模块、数据、安全、迁移和验收。
- [x] P0.2 建立项目骨架、依赖文件和忽略规则。
- 文件：`pyproject.toml`、`.gitignore`、`README.md`、`src/`、`tests/`
- 验证：干净环境可安装，CLI 能显示帮助。
- [x] P0.3 生成四个旧项目的代码、任务、数据基线清单。
- 文件：`var/baseline/<timestamp>/`
- 验证：包含 21 个实际任务、代码哈希、数据汇总；不包含密钥值。
- [x] P0.4 建立敏感信息扫描并记录轮换门禁。
- 文件：`src/gyxx_flow/security/`、`config/secrets.example.env`
- 验证：仓库扫描无明文凭据；生产切换要求凭据轮换确认。
## P1 核心运行与数据基础设施
- [x] P1.1 TDD 实现可移植路径配置，不允许业务代码硬编码旧盘符。
- [x] P1.2 TDD 实现 `RunContext`、稳定 `run_id` 和业务日期校验。
- [x] P1.3 TDD 实现原子 JSON 写入、SHA-256 和 artifact manifest。
- [x] P1.4 TDD 实现 raw/normalized/curated/export/evidence/state/log/tmp 目录解析。
- [x] P1.5 TDD 实现结构化运行及步骤状态记录。
- [x] P1.6 TDD 实现文件锁、工作流锁和命名资源锁。
## P2 工作流引擎与 CLI
- [x] P2.1 TDD 实现工作流、步骤、依赖、超时和重试模型。
- [x] P2.2 TDD 实现命令步骤及无副作用 dry-run。
- [x] P2.3 TDD 实现 shadow 模式，禁止生产 Sink 和正式通知。
- [x] P2.4 TDD 实现 `gyxx list`、`gyxx run`、`gyxx backfill`。
- [x] P2.5 TDD 实现失败恢复、步骤级重跑和最终退出码。
- [x] P2.6 建立 `ops` 运行记录和 outbox 接口。
## P3 工作流目录与调度
- [x] P3.1 建立四个模块及模块依赖约束测试。
- [x] P3.2 映射 21 个实际计划任务到稳定 workflow ID。
- [x] P3.3 登记未调度/手工工作流，保持原触发语义。
- [x] P3.4 建立声明式 schedule 配置及 schema 校验。
- [x] P3.5 生成 Windows Task Scheduler XML/PowerShell 安装计划，但默认不应用。
- [x] P3.6 实现 `gyxx schedule plan`，检测系统任务漂移。
## P4 迁移和验收工具
- [x] P4.1 实现旧命令适配器，所有旧根目录均通过配置注入。
- [x] P4.2 实现历史数据只复制、不删除的迁移计划及校验清单。
- [x] P4.3 实现文件数量、字节数、SHA-256 对账。
- [x] P4.4 实现 shadow 结果主键、行数、指标和错误对比。
- [x] P4.5 实现 `gyxx doctor` 环境、解释器、CLI、路径和权限预检。
- [x] P4.6 实现 `gyxx acceptance status` 和机器可读验收报告。
- [x] P4.7 生成四个旧项目的源码迁移清单；每个文件记录来源哈希、目标路径和处置类型。
## P5 分模块源码迁移
- [x] P5.1 将 `shop_intelligence` 全部业务源码、资源和入口复制并改造到新模块；两个周任务可独立运行。
- [x] P5.2 将 `supply_chain` 全部业务源码、资源和入口复制并改造到新模块；三个定时和全部手工入口可独立运行。
- [x] P5.3 将 `content_marketing` 全部业务源码、资源和入口复制并改造到新模块；全部定时和手工入口可独立运行。
- [x] P5.4 将 `product_commerce` 全部业务源码、资源、内置 vendors 和入口复制并改造到新模块；全部定时和手工入口可独立运行。
- [x] P5.5 消除营销二维码写入商品项目、商品配置回退到 Auto Flow，以及任何跨业务模块内部导入。
- [x] P5.6 将重复的飞书、PostgreSQL、Hermes、浏览器能力收敛为适配器，业务行为保持可回放对账。
- 证据：`src/gyxx_flow/adapters/integration.py`、`browser.py`、`bootstrap.py`、`runtime_exec.py`；原飞书后端不变，数据库只允许云端，Hermes 只允许本机。
- [x] P5.7 删除最终工作流中的 `DeferredLegacyCommandStep`，运行时不再要求 `GYXX_LEGACY_*_ROOT`。
- [x] P5.8 全量入口分类完成：scheduled、manual、library、intentionally-excluded 均有证据，不遗漏脚本。
## P6 自动化工程验收
- [x] P6.1 21 个计划任务逐项有且只有一个新项目本地入口映射，没有意外新增。
- [x] P6.2 源码、配置、启动命令和动态导入中均不存在四个旧项目的运行时依赖。
- [x] P6.3 临时隐藏四个旧项目路径并更换项目根、数据根后，入口导入和 dry-run 全部通过。
- [ ] P6.4 每个迁移后的业务运行可从 run_id 追溯输入、输出、日志和外部写入。
- [ ] P6.5 迁移后的业务代码重跑不会重复写数据库、飞书或正式通知。
- [x] P6.6 旧数据复制前后数量、大小和 SHA-256 一致。
- [x] P6.7 迁移后的浏览器 Profile 和共享资源全部进入新数据根，且并发锁验证通过。
- [x] P6.8 源码仓库敏感信息扫描通过。
- [x] P6.9 全量测试、静态检查、源码覆盖清单和代码评审无阻断项。
## P7 生产切换门禁
- [ ] P7.1 用户确认迁移基线、密钥轮换和第一个切换任务。
- [ ] P7.2 单次只切一个任务；旧任务只禁用、不删除。
- [ ] P7.3 回滚演练证明能关闭新任务并重新启用旧任务。
- [ ] P7.4 日任务真实连续成功 7 天。
- [ ] P7.5 周任务真实连续成功 2 个周期。
- [ ] P7.6 月任务指定历史月份回放通过。
- [ ] P7.7 生产通知无重复，数据库/飞书对账通过。
- [ ] P7.8 旧项目只读保留期满并取得退役确认。
## P8 完成
- [ ] P8.1 所有上述检查项均有证据且已勾选。
- [x] P8.2 按源码完整迁移的新口径重新生成 `review.md` 和 `final_report.md`。
- [x] P8.3 按独立运行的新口径更新部署、迁移、回滚和日常运维手册。
## P9 每脚本运行时隔离与外部系统统一边界
- [x] P9.1 为 131 个可执行脚本建立显式、稳定、唯一的 `22000..22999` CDP 端口分配。
- [x] P9.2 每脚本独立 Profile、Cookie、storage state，全部位于 `GYXX_DATA_ROOT/state/browser`。
- [x] P9.3 顶层 workflow、手工脚本和嵌套 Python/PowerShell 子脚本均按真实 script ID 重新绑定。
- [x] P9.4 Cookie/storage state 支持首次为空、原子保存和后续运行复用，内容不进入日志或源码。
- [x] P9.5 飞书保持原 lark-cli profile、身份和 OpenAPI 应用，不切换后端或凭据来源。
- [x] P9.6 `PG_*`、`DB_*`、`AUTOFLOW_PG_*` 统一映射现有云端 PostgreSQL，拒绝本地回环数据库。
- [x] P9.7 Hermes 保持本机 HTTP/CLI，非回环 Hermes URL 在业务调用前失败。
- [x] P9.8 移除生产代码固定 9222/18801/18802/18803 和源码目录/临时 Profile，刷新迁移哈希。
- [x] P9.9 Python 3.12 全量 `294 passed`，compileall、敏感信息、端口和旧根依赖扫描通过；历史 Ruff 证据保留。
## P10 采集数据统一分层与目录收口
- [x] P10.1 四个模块的新产物统一使用 `<GYXX_DATA_ROOT>/data/{raw,normalized,curated,exports,evidence}/<module>`，运行状态、日志和临时文件分别使用 `state/`、`logs/`、`tmp/`。
- [x] P10.2 JSON、Markdown、CSV、XLS/XLSX、浏览器下载和截图均由模块路径边界定位，不写入源码目录、模块目录或当前工作目录。
- [x] P10.3 `var/data/raw/<module>/legacy` 历史数据不移动、不删除、不改写，迁移前后文件数和总字节数一致。
- [x] P10.4 四模块保留原业务脚本公开路径常量，统一改接共享 `ModuleDataPaths`，新增模块只需声明模块名即可扩展。
- [x] P10.5 更换 `GYXX_DATA_ROOT` 后所有产物路径整体迁移，路径解析不创建目录、不访问四个旧项目。
- [x] P10.6 路径红测、静态旁路扫描、全量测试、Ruff、compileall、敏感信息和旧根依赖扫描全部通过，清单与验收报告已刷新。
## P11 四源项目增量 Bug 修复重新对齐（2026-07-28）
- [x] P11.1 以四份来源清单 SHA-256 为基线，识别源端变更、新增、删除和目标端二次改造。
- [x] P11.2 合并内容营销模块发布链接非空触发、星图普通失败不熔断和精确续跑修复。
- [x] P11.3 合并商品模块“ERP 编码只读取天猫分组”的权威来源修复，并迁入波塞冬历史销量回填工具。
- [x] P11.4 刷新来源/目标哈希、两份动态业务映射、迁移清单和新增回归测试，不纳入运行数据、临时文件或敏感状态。
- [x] P11.5 定向测试、全量测试、compileall、敏感信息、旧根依赖和 doctor 通过；acceptance 正确保留 P6/P7 生产门禁为 incomplete，Ruff 因当前虚拟环境未安装而未执行。
pyproject.toml
+63
View File
@@ -0,0 +1,63 @@
[build-system]
requires = ["setuptools>=77"]
build-backend = "setuptools.build_meta"
[project]
name = "gyxx-flow"
version = "0.1.0"
description = "Portable workflow orchestration for GYXX business automation"
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
"browserforge>=1.2.4",
"curl-cffi>=0.15.0",
"httpx>=0.27",
"imageio-ffmpeg>=0.6.0",
"langchain>=0.3.0",
"langchain-anthropic>=0.3.0",
"langchain-core>=0.3.0",
"langchain-openai>=0.2.0",
"langgraph>=0.2.0",
"lark-oapi>=1.0.0",
"lxml>=5.0",
"msgspec>=0.21.1",
"openpyxl>=3.1.5",
"ormsgpack>=1.2.0",
"pandas>=3.0.3",
"patchright>=1.60.1",
"Pillow>=10",
"playwright>=1.60.0",
"psycopg[binary,pool]>=3.1",
"psycopg2-binary>=2.9.12",
"psutil>=5.9",
"python-dotenv>=1.0.0",
"requests>=2.31.0",
"scrapling[all]>=0.4.9",
"selenium>=4.20",
"xlrd>=2.0.1",
]
[project.optional-dependencies]
test = ["pytest>=8.0"]
[project.scripts]
gyxx = "gyxx_flow.cli:main"
[tool.setuptools.packages.find]
where = ["src"]
[tool.setuptools.package-data]
"gyxx_flow.modules.content_marketing" = ["runtime/**/*"]
"gyxx_flow.modules.product_commerce" = ["runtime/**/*"]
"gyxx_flow.modules.shop_intelligence" = ["runtime/**/*"]
"gyxx_flow.modules.supply_chain" = ["runtime/**/*"]
[tool.setuptools.exclude-package-data]
"*" = ["**/__pycache__/**", "**/*.pyc", "**/*.pyo"]
[tool.pytest.ini_options]
pythonpath = ["src"]
testpaths = ["tests"]
[tool.ruff.lint]
select = ["E4", "E7", "E9", "F", "I"]
review.md
+90
View File
@@ -0,0 +1,90 @@
# 工程复核报告
## 结论
四个旧项目的工作流、定时入口及纳入范围的业务源码已实体复制到新项目，并完成路径、
配置、数据目录和启动方式改造。运行时入口只解析
`src/gyxx_flow/modules/<module>/runtime`，不再导入或启动四个旧目录中的代码。
新项目已经具备统一手工运行和生成 21 个 Windows 定时任务定义的工程能力；本轮没有
注册新任务，也没有禁用、删除或修改旧任务。生产安全切换仍受 P6.4、P6.5 和
P7 门禁约束，不能把“可生成/可 dry-run”表述为“已经生产投运”。
## 源码接管证据
| 模块 | 来源清单文件 | 迁移文件 | 可执行脚本 |
|---|---:|---:|---:|
| content_marketing | 90 | 90 + 1 个新路径模块 | 56 |
| product_commerce | 101 | 101 + 1 个新路径模块 | 49 |
| shop_intelligence | 22 | 22 | 8 |
| supply_chain | 37 | 37 | 18 |
| 合计 | 250 | 256（含 2 个路径模块和 4 个运行时 bootstrap） | 131 |
- 四份 `config/source-manifests/*.json` 逐文件记录来源相对路径、来源 SHA-256、目标
相对路径、目标 SHA-256、分类和是否改造。
- 对四个旧目录做只读复核并同步两份动态业务映射后，250/250 个纳入文件来源哈希与
当前迁移清单一致；旧项目没有被本次迁移写入。
- 源码、配置、启动器及动态入口扫描中，四个旧根目录、`GYXX_LEGACY_*_ROOT`、
`DeferredLegacyCommandStep` 和 `AUTO_FLOW_CONFIG` 的运行时命中均为 0。
- 项目和数据根整体迁移到临时目录后，目录发现、131 个脚本发现及四模块代表工作流
dry-run 均通过。
## 入口与调度证据
- 29 个目录项：21 个 scheduled、7 个 manual、1 个有证据的 unavailable。
- `gyxx scripts list` 发现 131 个新项目本地 Python/BAT/PowerShell 入口。
- `gyxx run`、`gyxx backfill` 和 `gyxx scripts run` 默认 dry-run；`--execute` 才启动
迁入后的本地副本。
- 最终候选调度包包含 21 个 XML，动作统一为
`D:\gyxx-flow\.venv\Scripts\python.exe -m gyxx_flow run <id> --scheduled`。
- 安装器默认只展示计划；即使显式 `-Apply` 也必须提供一个 `-WorkflowId`，不允许一次
批量切换。
## 自动化验证
- 全量 pytest：221 passed；仅有复制脚本内嵌 JavaScript 正则产生的非阻断
`SyntaxWarning`。
- 新框架与测试 Ruff：通过。
- 所有可执行 runtime 的 fatal Ruff 规则 `E9/F63/F7/F82`：通过；一个不独立执行的
上游源码摘录已在 manifest 中标记 `source_resource`。
- Python compileall、PowerShell AST、来源清单哈希、旧路径扫描和敏感信息扫描：通过。
- 历史数据：48,964 个文件、1,091,997,415 字节，复制前后聚合 SHA-256 一致。
## 适配器状态与尚未通过的门禁
- P5.6 已通过：131 个脚本统一绑定唯一 CDP、独立 Profile/Cookie/storage state；飞书
保持原身份/后端，PostgreSQL 只允许现有云端，Hermes 只允许本机。
- P6.4/P6.5：编排层已有 run journal、effect ledger 和 outbox，但尚未证明每个旧业务
脚本的所有内部外部写入都可追踪且全量幂等。
- P7：需要用户授权、凭据轮换、逐任务切换、真实 7 天/2 周观察和数据/通知对账。
因此工程“源码归属和独立入口”已通过，生产“全量安全执行和切换”尚未通过。
## 四源项目增量对齐复核（2026-07-28）
- 内容营销已合并“发布链接非空即进入采集”、星图普通达人失败不触发全局熔断、按款式和记录精确续跑。
- 商品商业已合并“ERP 编码只认天猫分组”，并把源端新增的波塞冬历史销量回填脚本完整迁入；脚本不引用旧项目。
- 波塞冬回填按 `raw → normalized → curated → exports` 分层，数据库继续使用云端配置，飞书仍使用原身份和项目内写入器。
- 新脚本登记独立 CDP 端口 `22130`，Profile/Cookie/storage state 继续由统一绑定派生并可迁移复用。
- 本轮全量 `294 passed`，修复定向测试 18 项通过，compileall、doctor、敏感信息与来源/目标哈希验证通过；当前虚拟环境未安装 Ruff，生产门禁仍保持 incomplete。
## 采集数据统一目录复核（2026-07-27）
- 四模块路径层统一改接共享 `DataLayout.for_module()` / `ModuleDataPaths`；新采集产物固定进入
`data/{raw,normalized,curated,exports,evidence}/<module>`，状态、日志和临时文件分别进入
`state/<module>`、`logs/<module>` 和 `tmp/<module>`。
- 内容模块的动态 mapping 和 V2 结果进入 normalized，跨平台 run_all 报告进入 curated，
checkpoint 进入 state；商品模块的 checkpoint、失败日志、调试文件和 vendor 下载已收口；
店铺调试输出进入 tmp；供应链缓存/锁/status 进入 state，原始下载、临时处理和最终报表分别
进入 raw、tmp 和 exports。
- 所有审计到的任意 `--output`、`--data-root` 和 vendor 环境输出均限制在
`GYXX_DATA_ROOT` 内；绝对外部路径或 `..` 逃逸会在写入前失败。
- `var/data/raw/<module>/legacy` 前后均为 48,961 个文件、1,091,967,729 字节；四模块分项
文件数和字节数完全一致，且不存在旁路 `var/raw` 或 `var/exports`。证据见
`var/evidence/unified-data-layout-20260727.json`。
- 供应链 raw 产物按 `run_id=<id>` 追加保存，编排器只读取本次运行目录且不会清理 raw；
采购更新触发任务放在 state，脚本结束只关闭浏览器进程并保留 Profile/Cookie 供复用。
- 空字符串或纯空白 `GYXX_DATA_ROOT` 会安全回退项目 `var`，不会把 cwd 当数据根。
- 全量回归为 294 passed；新代码 Ruff、compileall、PowerShell AST、四份来源清单目标哈希、
敏感信息和旧根依赖扫描全部通过。5 条 warning 是迁入脚本内嵌 JavaScript 正则的既有
`SyntaxWarning`，不影响本次目录验收。
src/gyxx_flow/__init__.py
+3
View File
@@ -0,0 +1,3 @@
"""GYXX Flow package."""
__version__ = "0.1.0"
src/gyxx_flow/__main__.py
+6
View File
@@ -0,0 +1,6 @@
"""Run GYXX Flow with ``python -m gyxx_flow``."""
from gyxx_flow.cli import main
if __name__ == "__main__":
raise SystemExit(main())
src/gyxx_flow/acceptance.py
+211
View File
@@ -0,0 +1,211 @@
"""Machine-readable acceptance evidence derived from repository state."""
from __future__ import annotations
import hashlib
import json
import re
from dataclasses import asdict, dataclass
from pathlib import Path
from gyxx_flow.catalog import CatalogError, WorkflowCatalog
from gyxx_flow.core.config import Settings
from gyxx_flow.script_catalog import ScriptCatalog
from gyxx_flow.security import scan_repository
_CHECKLIST = re.compile(r"^\s*-\s*\[([ xX])\]\s+(P\d+\.\d+)\b", re.MULTILINE)
@dataclass(frozen=True, slots=True)
class PlanItem:
item_id: str
completed: bool
@dataclass(frozen=True, slots=True)
class AcceptanceReport:
items: tuple[PlanItem, ...]
checks: dict[str, bool]
@property
def is_complete(self) -> bool:
return bool(self.items) and all(item.completed for item in self.items) and all(
self.checks.values()
)
def as_dict(self) -> dict[str, object]:
completed = sum(item.completed for item in self.items)
return {
"schema_version": 1,
"is_complete": self.is_complete,
"summary": {
"total": len(self.items),
"completed": completed,
"pending": len(self.items) - completed,
},
"items": [asdict(item) for item in self.items],
"checks": dict(sorted(self.checks.items())),
}
def parse_plan_checklist(path: Path) -> tuple[PlanItem, ...]:
text = Path(path).read_text(encoding="utf-8")
items = tuple(
PlanItem(item_id=match.group(2), completed=match.group(1).casefold() == "x")
for match in _CHECKLIST.finditer(text)
)
if len({item.item_id for item in items}) != len(items):
raise ValueError("plan contains duplicate acceptance item IDs")
return items
def build_acceptance_report(settings: Settings) -> AcceptanceReport:
project_root = settings.project_root
items = parse_plan_checklist(project_root / "plan.md")
checks = {
"catalog_21_tasks": _catalog_has_21_tasks(project_root),
"baseline_21_tasks": _baseline_has_21_tasks(settings.data_root),
"native_entrypoints_local": _native_entrypoints_are_local(project_root),
"runtime_sources_decoupled": _runtime_sources_are_decoupled(project_root),
"runnable_script_catalog": _runnable_script_catalog_is_complete(),
"source_manifests_verified": _source_manifests_are_verified(project_root),
"secret_scan_clean": not scan_repository(project_root),
}
return AcceptanceReport(items, checks)
def _catalog_has_21_tasks(project_root: Path) -> bool:
try:
catalog = WorkflowCatalog.load(project_root / "config")
except Exception:
return False
scheduled = catalog.scheduled_workflows()
return len(scheduled) == 21 and len(catalog.schedules) == 21
def _baseline_has_21_tasks(data_root: Path) -> bool:
candidates = sorted((Path(data_root) / "baseline").glob("*/manifest.json"))
if not candidates:
return False
try:
payload = json.loads(candidates[-1].read_text(encoding="utf-8"))
tasks = payload["scheduled_tasks"]
return tasks["actual_count"] == 21 and len(tasks["tasks"]) == 21
except (OSError, json.JSONDecodeError, KeyError, TypeError):
return False
def _native_entrypoints_are_local(project_root: Path) -> bool:
try:
catalog = WorkflowCatalog.load(project_root / "config")
for workflow in catalog.workflows:
if workflow.trigger == "unavailable":
continue
target = (
project_root
/ "src"
/ "gyxx_flow"
/ "modules"
/ workflow.module
/ "runtime"
/ workflow.entry
).resolve(strict=True)
if not target.is_file() or not target.is_relative_to(project_root):
return False
config_text = (project_root / "config" / "workflows.json").read_text(
encoding="utf-8"
)
return "legacy" not in config_text.casefold()
except (OSError, ValueError, CatalogError):
return False
def _runnable_script_catalog_is_complete() -> bool:
try:
scripts = ScriptCatalog.discover_default().scripts
except (OSError, ValueError):
return False
return len(scripts) >= 100 and len({item.module for item in scripts}) == 4
def _runtime_sources_are_decoupled(project_root: Path) -> bool:
module_root = project_root / "src" / "gyxx_flow" / "modules"
forbidden = (
"d:\\yingxiaoyunying",
"d:\\shop-data-flow",
"d:\\product-collector-analyze-flow",
"e:\\auto-flow",
"gyxx_legacy_",
"deferredlegacycommandstep",
)
try:
for path in module_root.rglob("*"):
if not path.is_file() or path.suffix.casefold() in {".pyc", ".pyo"}:
continue
if path.is_symlink() or not path.resolve().is_relative_to(project_root):
return False
content = path.read_bytes()
if b"\x00" in content:
continue
text: str | None = None
for encoding in ("utf-8-sig", "gb18030"):
try:
text = content.decode(encoding)
break
except UnicodeDecodeError:
continue
if text is None:
continue
normalized = text.casefold().replace("/", "\\")
while "\\\\" in normalized:
normalized = normalized.replace("\\\\", "\\")
if any(value in normalized for value in forbidden):
return False
except OSError:
return False
return True
def _source_manifests_are_verified(project_root: Path) -> bool:
modules = (
"content_marketing",
"product_commerce",
"shop_intelligence",
"supply_chain",
)
try:
for module in modules:
payload = json.loads(
(
project_root
/ "config"
/ "source-manifests"
/ f"{module}.json"
).read_text(encoding="utf-8-sig")
)
if payload.get("schema_version") != 1 or payload.get("module") != module:
return False
files = payload.get("files")
if not isinstance(files, list) or not files:
return False
for item in files:
target = project_root.joinpath(*Path(item["target_relative_path"]).parts)
resolved = target.resolve(strict=True)
if (
target.is_symlink()
or not resolved.is_relative_to(project_root)
or not resolved.is_file()
or _sha256(resolved) != item["target_sha256"]
):
return False
except (OSError, json.JSONDecodeError, KeyError, TypeError, ValueError):
return False
return True
def _sha256(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as stream:
for chunk in iter(lambda: stream.read(1024 * 1024), b""):
digest.update(chunk)
return digest.hexdigest()
src/gyxx_flow/adapters/__init__.py
+45
View File
@@ -0,0 +1,45 @@
"""Replaceable infrastructure adapters exposed to business modules."""
from .browser import BrowserCookieStore, BrowserProfileLease, BrowserProfileManager
from .external import (
FeishuOutboxAdapter,
HermesCommandAdapter,
PostgresOutboxAdapter,
ReloginOutboxAdapter,
)
from .integration import (
RuntimeIntegrationBinding,
RuntimeIntegrationCatalog,
RuntimeIntegrationError,
RuntimeServicePolicy,
binding_from_environment,
environment_for_child_script,
)
from .native import (
DeferredModuleCommandStep,
ModuleCommandAdapter,
ModuleCommandFactory,
ModuleSourceError,
ModuleSourceRoots,
)
__all__ = [
"BrowserCookieStore",
"BrowserProfileLease",
"BrowserProfileManager",
"DeferredModuleCommandStep",
"FeishuOutboxAdapter",
"HermesCommandAdapter",
"ModuleCommandAdapter",
"ModuleCommandFactory",
"ModuleSourceError",
"ModuleSourceRoots",
"PostgresOutboxAdapter",
"ReloginOutboxAdapter",
"RuntimeIntegrationBinding",
"RuntimeIntegrationCatalog",
"RuntimeIntegrationError",
"RuntimeServicePolicy",
"binding_from_environment",
"environment_for_child_script",
]
src/gyxx_flow/adapters/bootstrap.py
+83
View File
@@ -0,0 +1,83 @@
"""Automatic rebinding for Python scripts launched by migrated orchestrators."""
from __future__ import annotations
import os
import sys
from pathlib import Path
from .integration import (
RuntimeIntegrationBinding,
RuntimeIntegrationCatalog,
RuntimeIntegrationError,
)
def bootstrap_current_process() -> RuntimeIntegrationBinding | None:
module = os.environ.get("GYXX_MODULE_ID", "").strip()
module_root_value = os.environ.get("GYXX_MODULE_ROOT", "").strip()
project_root = os.environ.get("GYXX_PROJECT_ROOT", "").strip()
data_root = os.environ.get("GYXX_DATA_ROOT", "").strip()
if not all((module, module_root_value, project_root, data_root)):
return None
if not sys.argv or not sys.argv[0] or sys.argv[0] in {"-c", "-m"}:
return None
module_root = Path(module_root_value).expanduser().resolve()
candidate = Path(sys.argv[0])
if not candidate.is_absolute():
candidate = Path.cwd() / candidate
candidate = candidate.expanduser().resolve()
try:
entry = candidate.relative_to(module_root).as_posix()
except ValueError:
return None
script_id = f"{module}:{entry}"
catalog = RuntimeIntegrationCatalog.load_default(
project_root=project_root,
data_root=data_root,
)
if script_id not in catalog.script_ids:
return None
binding = catalog.binding_for(script_id)
os.environ.update(catalog.environment_for(script_id, os.environ))
_rewrite_browser_arguments(binding)
return binding
def _rewrite_browser_arguments(binding: RuntimeIntegrationBinding) -> None:
replacements = {
"--user-data-dir": str(binding.profile_dir),
"--cdp-url": binding.cdp_url,
"--cdp-port": str(binding.cdp_port),
"--debug-port": str(binding.cdp_port),
}
index = 1
while index < len(sys.argv):
argument = sys.argv[index]
replaced = False
for flag, value in replacements.items():
if argument == flag and index + 1 < len(sys.argv):
sys.argv[index + 1] = value
index += 2
replaced = True
break
if argument.startswith(f"{flag}="):
sys.argv[index] = f"{flag}={value}"
index += 1
replaced = True
break
if not replaced:
index += 1
def safe_bootstrap_current_process() -> RuntimeIntegrationBinding | None:
managed = bool(os.environ.get("GYXX_PROJECT_ROOT", "").strip())
try:
return bootstrap_current_process()
except RuntimeIntegrationError:
if managed:
raise
return None
__all__ = ["bootstrap_current_process", "safe_bootstrap_current_process"]
src/gyxx_flow/adapters/browser.py
+97
View File
@@ -0,0 +1,97 @@
"""Portable, cross-process browser profile ownership."""
from __future__ import annotations
import re
from contextlib import contextmanager
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Iterator
from gyxx_flow.core.artifacts import atomic_write_json
from gyxx_flow.core.layout import DataLayout
from gyxx_flow.core.locks import LockManager
_PROFILE_ID = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]*$")
@dataclass(frozen=True, slots=True)
class BrowserProfileLease:
profile_id: str
profile_path: Path
class BrowserProfileManager:
def __init__(
self, data_root: Path, *, lock_manager: LockManager | None = None
) -> None:
self._layout = DataLayout(data_root)
self._locks = lock_manager or LockManager(
self._layout.root / "state" / "locks"
)
@contextmanager
def acquire(
self,
profile_id: str,
*,
owner: str,
timeout_seconds: float = 30.0,
) -> Iterator[BrowserProfileLease]:
if not isinstance(profile_id, str) or not _PROFILE_ID.fullmatch(profile_id):
raise ValueError("invalid browser profile id")
path = self._layout.state("browser_profiles", profile_id)
with self._locks.acquire(
f"browser:{profile_id}", owner=owner, timeout_seconds=timeout_seconds
):
path.mkdir(parents=True, exist_ok=True)
yield BrowserProfileLease(profile_id, path)
@dataclass(frozen=True, slots=True)
class BrowserCookieStore:
"""Playwright-compatible cookie/storage state persisted with atomic replace."""
cookie_file: Path
storage_state_file: Path
def __post_init__(self) -> None:
object.__setattr__(self, "cookie_file", Path(self.cookie_file).expanduser().resolve())
object.__setattr__(
self,
"storage_state_file",
Path(self.storage_state_file).expanduser().resolve(),
)
def load_cookies(self) -> list[dict[str, Any]]:
payload = self._read(self.cookie_file, default=[])
if not isinstance(payload, list) or not all(isinstance(item, dict) for item in payload):
raise ValueError("browser cookie file must contain a list of objects")
return payload
def save_cookies(self, cookies: list[dict[str, Any]]) -> Path:
if not isinstance(cookies, list) or not all(isinstance(item, dict) for item in cookies):
raise ValueError("browser cookies must be a list of objects")
return atomic_write_json(self.cookie_file, cookies)
def load_storage_state(self) -> dict[str, Any] | None:
payload = self._read(self.storage_state_file, default=None)
if payload is not None and not isinstance(payload, dict):
raise ValueError("browser storage state must contain an object")
return payload
def save_storage_state(self, state: dict[str, Any]) -> Path:
if not isinstance(state, dict):
raise ValueError("browser storage state must be an object")
return atomic_write_json(self.storage_state_file, state)
@staticmethod
def _read(path: Path, *, default: Any) -> Any:
if not path.exists():
return default
import json
try:
return json.loads(path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError) as exc:
raise ValueError(f"cannot read browser state file: {path.name}") from exc
src/gyxx_flow/adapters/external.py
+132
View File
@@ -0,0 +1,132 @@
"""Shared external-system adapters with durable idempotency boundaries."""
from __future__ import annotations
from dataclasses import dataclass
from pathlib import Path
from typing import Mapping
from gyxx_flow.core.artifacts import sha256_file
from gyxx_flow.ops import Outbox, OutboxMessage
from gyxx_flow.workflow import CommandStep
@dataclass(frozen=True, slots=True)
class FeishuOutboxAdapter:
outbox: Outbox
def upsert(
self, *, run_id: str, operation_key: str, artifact_id: str
) -> OutboxMessage:
return self.outbox.enqueue(
idempotency_key=_key(run_id, "feishu", operation_key),
topic="feishu.upsert",
payload={"artifact_id": _value(artifact_id, "artifact_id")},
run_id=run_id,
)
@dataclass(frozen=True, slots=True)
class PostgresOutboxAdapter:
outbox: Outbox
def upsert(
self, *, run_id: str, operation_key: str, artifact_id: str
) -> OutboxMessage:
return self.outbox.enqueue(
idempotency_key=_key(run_id, "postgres", operation_key),
topic="postgres.upsert",
payload={"artifact_id": _value(artifact_id, "artifact_id")},
run_id=run_id,
)
@dataclass(frozen=True, slots=True)
class ReloginOutboxAdapter:
outbox: Outbox
data_root: Path
@classmethod
def from_data_root(cls, data_root: Path | str) -> "ReloginOutboxAdapter":
root = Path(data_root).expanduser().resolve()
return cls(Outbox(root), root)
def enqueue(
self,
*,
run_id: str,
operation_key: str,
recipient: str,
text: str | None,
image: Path | None,
) -> OutboxMessage:
payload: dict[str, object] = {
"recipient": _value(recipient, "recipient"),
"text": text or "",
"image_path": None,
"image_sha256": None,
}
if image is not None:
resolved = Path(image).resolve(strict=True)
try:
relative = resolved.relative_to(self.data_root).as_posix()
except ValueError as exc:
raise ValueError("relogin image must be inside the configured data root") from exc
payload["image_path"] = relative
payload["image_sha256"] = sha256_file(resolved)
return self.outbox.enqueue(
idempotency_key=_key(run_id, "feishu-relogin", operation_key),
topic="feishu.relogin",
payload=payload,
run_id=run_id,
)
class HermesCommandAdapter:
"""Build the one supported Hermes CLI invocation without executing it."""
def __init__(
self,
*,
executable: str = "hermes",
base_env: Mapping[str, str] | None = None,
) -> None:
self._executable = _value(executable, "Hermes executable")
self._base_env = dict(base_env or {})
def build(self, *, prompt_file: Path, cwd: Path) -> CommandStep:
prompt_path = Path(prompt_file).resolve(strict=True)
working_directory = Path(cwd).resolve(strict=True)
if not prompt_path.is_file():
raise ValueError("Hermes prompt_file must be a file")
if not working_directory.is_dir():
raise ValueError("Hermes cwd must be a directory")
prompt = prompt_path.read_text(encoding="utf-8")
if not prompt or len(prompt) > 20_000 or "\x00" in prompt:
raise ValueError("Hermes prompt must contain 1..20000 safe characters")
return CommandStep(
argv=(self._executable, "-z", prompt),
cwd=working_directory,
env=self._base_env,
)
def _key(run_id: str, adapter: str, operation_key: str) -> str:
return ":".join(
(
_value(run_id, "run_id"),
adapter,
_value(operation_key, "operation_key"),
)
)
def _value(value: str, field: str) -> str:
if (
not isinstance(value, str)
or not value
or "\x00" in value
or any(ord(character) < 32 for character in value)
):
raise ValueError(f"invalid {field}")
return value
src/gyxx_flow/adapters/integration.py
+344
View File
@@ -0,0 +1,344 @@
"""Versioned per-script runtime bindings and external-service guardrails."""
from __future__ import annotations
import hashlib
import ipaddress
import json
import os
from dataclasses import dataclass
from pathlib import Path
from types import MappingProxyType
from typing import Mapping
from urllib.parse import urlparse
from gyxx_flow.script_catalog import ScriptCatalog
class RuntimeIntegrationError(ValueError):
"""Raised when an integration binding is incomplete or unsafe."""
@dataclass(frozen=True, slots=True)
class RuntimeIntegrationBinding:
script_id: str
module: str
entry: str
cdp_port: int
cdp_url: str
profile_dir: Path
cookie_file: Path
storage_state_file: Path
@dataclass(frozen=True, slots=True)
class RuntimeServicePolicy:
"""Keep legacy Feishu, cloud PostgreSQL, and loopback-only Hermes."""
hermes_url: str = "http://127.0.0.1:8642/v1/chat/completions"
def __post_init__(self) -> None:
_require_loopback_url(self.hermes_url, field="Hermes local URL")
def apply(self, environment: Mapping[str, str]) -> dict[str, str]:
result = dict(environment)
_fill_database_aliases(result)
_validate_cloud_database(result)
configured_hermes = _configured_hermes_urls(result)
for field, value in configured_hermes:
_require_loopback_url(value, field=field)
result.update(
{
"GYXX_FEISHU_MODE": "legacy",
"GYXX_POSTGRES_MODE": "cloud",
"GYXX_HERMES_MODE": "local",
}
)
# Preserve every original Feishu/DB/Hermes setting. Defaults are only
# supplied for the two historical Hermes HTTP variable names.
result.setdefault("HERMES_ANALYZER_URL", self.hermes_url)
result.setdefault("ANALYZER_API_SERVER_URL", self.hermes_url)
return result
class RuntimeIntegrationCatalog:
"""Immutable exact allocation of one portable browser binding per script."""
def __init__(
self,
bindings: Mapping[str, RuntimeIntegrationBinding],
*,
service_policy: RuntimeServicePolicy,
) -> None:
self._bindings = MappingProxyType(dict(sorted(bindings.items())))
self.service_policy = service_policy
@classmethod
def load(
cls,
config_path: Path | str,
*,
scripts: ScriptCatalog,
data_root: Path | str,
) -> "RuntimeIntegrationCatalog":
path = Path(config_path)
try:
payload = json.loads(path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError) as exc:
raise RuntimeIntegrationError(
f"cannot load runtime integration catalog: {path.name}"
) from exc
if not isinstance(payload, dict) or payload.get("schema_version") != 1:
raise RuntimeIntegrationError("unsupported runtime binding schema")
host = payload.get("cdp_host")
if host not in {"127.0.0.1", "localhost", "::1"}:
raise RuntimeIntegrationError("browser CDP host must be loopback")
allocations = payload.get("scripts")
if not isinstance(allocations, dict):
raise RuntimeIntegrationError("runtime binding scripts must be an object")
configured = set(allocations)
discovered = set(scripts.script_ids)
if configured != discovered:
missing = len(discovered - configured)
extra = len(configured - discovered)
raise RuntimeIntegrationError(
f"runtime binding coverage mismatch; missing={missing}, extra={extra}"
)
ports = list(allocations.values())
if (
not all(isinstance(port, int) and 22000 <= port <= 22999 for port in ports)
or len(set(ports)) != len(ports)
):
raise RuntimeIntegrationError(
"runtime binding ports must be unique integers in 22000..22999"
)
root = Path(data_root).expanduser().resolve()
bindings: dict[str, RuntimeIntegrationBinding] = {}
for script in scripts.scripts:
port = allocations[script.script_id]
state_root = _browser_state_root(root, script.module, script.script_id)
bindings[script.script_id] = RuntimeIntegrationBinding(
script_id=script.script_id,
module=script.module,
entry=script.entry,
cdp_port=port,
cdp_url=f"http://{host}:{port}",
profile_dir=state_root / "profile",
cookie_file=state_root / "cookies.json",
storage_state_file=state_root / "storage_state.json",
)
services = payload.get("services")
if not isinstance(services, dict):
raise RuntimeIntegrationError("runtime binding services must be an object")
if (
services.get("feishu") != "legacy"
or services.get("postgres") != "cloud"
or services.get("hermes") != "local"
):
raise RuntimeIntegrationError("unsupported runtime service policy")
hermes_url = services.get("hermes_url")
if not isinstance(hermes_url, str):
raise RuntimeIntegrationError("runtime service policy requires Hermes URL")
return cls(
bindings,
service_policy=RuntimeServicePolicy(hermes_url=hermes_url),
)
@classmethod
def load_default(
cls,
*,
project_root: Path | str,
data_root: Path | str,
scripts: ScriptCatalog | None = None,
) -> "RuntimeIntegrationCatalog":
root = Path(project_root).expanduser().resolve()
return cls.load(
root / "config" / "runtime-bindings.json",
scripts=scripts or ScriptCatalog.discover_default(),
data_root=data_root,
)
@property
def script_ids(self) -> tuple[str, ...]:
return tuple(self._bindings)
def binding_for(self, script_id: str) -> RuntimeIntegrationBinding:
try:
return self._bindings[script_id]
except KeyError as exc:
raise RuntimeIntegrationError(f"unknown runtime script binding: {script_id}") from exc
def environment_for(
self,
script_id: str,
base_environment: Mapping[str, str] | None = None,
) -> dict[str, str]:
binding = self.binding_for(script_id)
result = self.service_policy.apply(
os.environ if base_environment is None else base_environment
)
result.update(
{
"GYXX_SCRIPT_ID": binding.script_id,
"GYXX_BROWSER_CDP_PORT": str(binding.cdp_port),
"GYXX_BROWSER_CDP_URL": binding.cdp_url,
"GYXX_BROWSER_PROFILE_DIR": str(binding.profile_dir),
"GYXX_BROWSER_COOKIE_FILE": str(binding.cookie_file),
"GYXX_BROWSER_STORAGE_STATE_FILE": str(binding.storage_state_file),
# Compatibility aliases consumed by the migrated browser engines.
# They are deliberately overwritten so every engine uses the
# catalog allocation instead of a legacy shared profile/port.
"AUTOFLOW_CDP_PORT": str(binding.cdp_port),
"AUTOFLOW_CDP_URL": binding.cdp_url,
"DMP_CDP_URL": binding.cdp_url,
"TM_USER_DATA_DIR": str(binding.profile_dir),
"TM_DAILY_USER_DATA_DIR": str(binding.profile_dir),
"DMP_USER_DATA_DIR": str(binding.profile_dir),
"DY_USER_DATA_DIR": str(binding.profile_dir),
"WANXIANG_USER_DATA_DIR": str(binding.profile_dir),
"GUANGHE_USER_DATA_DIR": str(binding.profile_dir),
"GUANGHE_LUGGAGE_USER_DATA_DIR": str(binding.profile_dir),
"DY_STORAGE_STATE_FILE": str(binding.storage_state_file),
}
)
return result
def environment_for_child_script(
target: Path | str,
base_environment: Mapping[str, str] | None = None,
) -> dict[str, str]:
"""Rebind a nested project script instead of inheriting its parent's CDP."""
environment = dict(os.environ if base_environment is None else base_environment)
module = environment.get("GYXX_MODULE_ID", "").strip()
module_root_value = environment.get("GYXX_MODULE_ROOT", "").strip()
project_root_value = environment.get("GYXX_PROJECT_ROOT", "").strip()
data_root_value = environment.get("GYXX_DATA_ROOT", "").strip()
if not all((module, module_root_value, project_root_value, data_root_value)):
raise RuntimeIntegrationError("child script binding requires GYXX runtime roots")
module_root = Path(module_root_value).expanduser().resolve()
target_path = Path(target)
if not target_path.is_absolute():
target_path = module_root / target_path
target_path = target_path.expanduser().resolve()
try:
entry = target_path.relative_to(module_root).as_posix()
except ValueError as exc:
raise RuntimeIntegrationError("child script must stay inside its module root") from exc
script_id = f"{module}:{entry}"
catalog = RuntimeIntegrationCatalog.load_default(
project_root=project_root_value,
data_root=data_root_value,
)
return catalog.environment_for(script_id, environment)
def binding_from_environment(
environment: Mapping[str, str] | None = None,
) -> RuntimeIntegrationBinding:
values = os.environ if environment is None else environment
try:
script_id = values["GYXX_SCRIPT_ID"]
module, entry = script_id.split(":", 1)
port = int(values["GYXX_BROWSER_CDP_PORT"])
cdp_url = values["GYXX_BROWSER_CDP_URL"]
profile = Path(values["GYXX_BROWSER_PROFILE_DIR"]).expanduser().resolve()
cookie = Path(values["GYXX_BROWSER_COOKIE_FILE"]).expanduser().resolve()
storage = Path(values["GYXX_BROWSER_STORAGE_STATE_FILE"]).expanduser().resolve()
except (KeyError, ValueError) as exc:
raise RuntimeIntegrationError("current process has no valid browser binding") from exc
_require_loopback_url(cdp_url, field="browser CDP URL")
if not 22000 <= port <= 22999:
raise RuntimeIntegrationError("browser CDP port is outside the managed range")
return RuntimeIntegrationBinding(
script_id=script_id,
module=module,
entry=entry,
cdp_port=port,
cdp_url=cdp_url,
profile_dir=profile,
cookie_file=cookie,
storage_state_file=storage,
)
def _browser_state_root(data_root: Path, module: str, script_id: str) -> Path:
entry = script_id.split(":", 1)[1]
stem = Path(entry).stem
safe = "".join(character if character.isalnum() else "-" for character in stem)
safe = safe.strip("-")[:48] or "script"
digest = hashlib.sha256(script_id.encode("utf-8")).hexdigest()[:12]
return data_root / "state" / "browser" / module / f"{safe}-{digest}"
def _configured_hermes_urls(environment: Mapping[str, str]) -> tuple[tuple[str, str], ...]:
names = (
"HERMES_ANALYZER_URL",
"ANALYZER_API_SERVER_URL",
"COLLECTOR_API_SERVER_URL",
"COLLECTOR_HERMES_GATEWAY_URL",
"ANALYZER_HERMES_GATEWAY_URL",
)
return tuple((name, environment[name]) for name in names if environment.get(name, "").strip())
def _validate_cloud_database(environment: Mapping[str, str]) -> None:
for name in ("PG_HOST", "DB_HOST", "AUTOFLOW_PG_HOST"):
value = environment.get(name, "").strip()
if value and _is_loopback_host(value):
raise RuntimeIntegrationError(f"cloud PostgreSQL cannot use loopback host in {name}")
for name in ("DATABASE_URL", "DB_URL"):
value = environment.get(name, "").strip()
if value:
host = urlparse(value).hostname
if host is None or _is_loopback_host(host):
raise RuntimeIntegrationError(f"cloud PostgreSQL requires a remote host in {name}")
def _fill_database_aliases(environment: dict[str, str]) -> None:
groups = (
("PG_HOST", "DB_HOST", "AUTOFLOW_PG_HOST"),
("PG_PORT", "DB_PORT", "AUTOFLOW_PG_PORT"),
("PG_DB", "DB_NAME", "AUTOFLOW_PG_DB"),
("PG_USER", "DB_USER", "AUTOFLOW_PG_USER"),
("PG_PASSWORD", "DB_PASSWORD", "AUTOFLOW_PG_PASSWORD"),
)
for aliases in groups:
value = next(
(environment[name] for name in aliases if environment.get(name, "").strip()),
"",
)
if value:
for name in aliases:
if not environment.get(name, "").strip():
environment[name] = value
def _require_loopback_url(value: str, *, field: str) -> None:
parsed = urlparse(value)
if parsed.scheme not in {"http", "https", "ws", "wss"} or not parsed.hostname:
raise RuntimeIntegrationError(f"{field} must be a valid local URL")
if not _is_loopback_host(parsed.hostname):
raise RuntimeIntegrationError(f"{field} must remain local")
def _is_loopback_host(host: str) -> bool:
normalized = host.strip().strip("[]").casefold()
if normalized == "localhost":
return True
try:
return ipaddress.ip_address(normalized).is_loopback
except ValueError:
return False
__all__ = [
"RuntimeIntegrationBinding",
"RuntimeIntegrationCatalog",
"RuntimeIntegrationError",
"RuntimeServicePolicy",
"binding_from_environment",
"environment_for_child_script",
]
src/gyxx_flow/adapters/native.py
+209
View File
@@ -0,0 +1,209 @@
"""Resolve workflow commands from source owned by this installed project."""
from __future__ import annotations
import os
import subprocess
import sys
from dataclasses import dataclass
from pathlib import Path, PurePosixPath
from types import MappingProxyType
from typing import Callable, Mapping
from gyxx_flow.adapters.integration import RuntimeIntegrationCatalog
from gyxx_flow.catalog import WorkflowEntry
from gyxx_flow.core.config import Settings
from gyxx_flow.core.context import RunContext
from gyxx_flow.core.layout import DataLayout
from gyxx_flow.workflow.model import ExecutableStep
from gyxx_flow.workflow.steps import CommandStep, StepExecution
class ModuleSourceError(ValueError):
"""Raised when a workflow cannot resolve project-owned module source safely."""
class ModuleSourceRoots:
"""Validated runtime roots for independently packaged business modules."""
def __init__(self, roots: Mapping[str, Path | str]) -> None:
resolved: dict[str, Path] = {}
for module, configured in roots.items():
if not module or not isinstance(module, str):
raise ModuleSourceError(f"invalid module id: {module!r}")
root = Path(configured).expanduser().resolve(strict=True)
if not root.is_dir():
raise ModuleSourceError(f"module source root is not a directory: {root}")
resolved[module] = root
self._roots = MappingProxyType(resolved)
def get(self, module: str) -> Path:
try:
return self._roots[module]
except KeyError as exc:
raise ModuleSourceError(f"unknown module source: {module!r}") from exc
class ModuleCommandAdapter:
"""Build a shell-free command rooted in source shipped with this project."""
def __init__(
self,
roots: ModuleSourceRoots,
*,
base_env: Mapping[str, str] | None = None,
python_executable: str | Path | None = None,
project_root: str | Path | None = None,
data_root: str | Path | None = None,
integration_catalog: RuntimeIntegrationCatalog | None = None,
) -> None:
self._roots = roots
self._base_env = dict(os.environ if base_env is None else base_env)
self._python = str(python_executable or sys.executable)
settings = Settings.from_env(project_root=project_root, env=self._base_env)
self._project_root = settings.project_root
self._data_root = (
Path(data_root).expanduser().resolve()
if data_root is not None
else settings.data_root
)
binding_file = self._project_root / "config" / "runtime-bindings.json"
self._integration_catalog = integration_catalog
if self._integration_catalog is None and binding_file.is_file():
self._integration_catalog = RuntimeIntegrationCatalog.load_default(
project_root=self._project_root,
data_root=self._data_root,
)
def build(self, entry: WorkflowEntry, *, context: RunContext) -> CommandStep:
if entry.workflow_id != context.workflow_id:
raise ModuleSourceError(
f"workflow entry {entry.workflow_id!r} does not match "
f"run context {context.workflow_id!r}"
)
if entry.trigger == "unavailable":
raise ModuleSourceError(f"workflow is unavailable: {entry.workflow_id}")
root = self._roots.get(entry.module)
executable = _resolve_entry(root, entry.entry)
suffix = executable.suffix.casefold()
if suffix in {".py", ".pyw"}:
argv = (self._python, str(executable), *entry.args)
elif suffix in {".bat", ".cmd"}:
command_line = subprocess.list2cmdline([str(executable), *entry.args])
argv = ("cmd.exe", "/d", "/s", "/c", command_line)
elif suffix == ".ps1":
argv = (
"powershell.exe",
"-NoProfile",
"-ExecutionPolicy",
"Bypass",
"-File",
str(executable),
*entry.args,
)
else:
argv = (str(executable), *entry.args)
env = {
**self._base_env,
"GYXX_PROJECT_ROOT": str(self._project_root),
"GYXX_DATA_ROOT": str(self._data_root),
"GYXX_MODULE_ROOT": str(root),
"GYXX_MODULE_ID": entry.module,
"GYXX_PYTHON": self._python,
"GYXX_WORKFLOW_ID": context.workflow_id,
"GYXX_RUN_ID": context.run_id,
"GYXX_BUSINESS_DATE": context.business_date.isoformat(),
"GYXX_SHADOW": "true" if context.shadow else "false",
"PYTHONUNBUFFERED": "1",
}
if entry.module == "supply_chain":
supply_paths = DataLayout(self._data_root).for_module("supply_chain")
env.update(
{
"GYXX_SUPPLY_RAW_ROOT": str(supply_paths.raw_root),
"GYXX_SUPPLY_STATE_ROOT": str(supply_paths.state_root),
"GYXX_SUPPLY_EXPORT_ROOT": str(supply_paths.exports_root),
"GYXX_SUPPLY_WORK_ROOT": str(supply_paths.tmp_root),
}
)
if self._integration_catalog is not None:
env = self._integration_catalog.environment_for(
f"{entry.module}:{entry.entry}", env
)
existing_pythonpath = env.get("PYTHONPATH", "").strip()
env["PYTHONPATH"] = (
f"{root}{os.pathsep}{existing_pythonpath}" if existing_pythonpath else str(root)
)
return CommandStep(argv=tuple(argv), cwd=root, env=env)
ModuleCommandFactory = Callable[[WorkflowEntry, RunContext], ExecutableStep]
def _command_from_project(entry: WorkflowEntry, context: RunContext) -> ExecutableStep:
package_root = Path(__file__).resolve().parents[1]
runtime_root = package_root / "modules" / entry.module / "runtime"
settings = Settings.from_env()
return ModuleCommandAdapter(
ModuleSourceRoots({entry.module: runtime_root}),
project_root=settings.project_root,
data_root=settings.data_root,
).build(entry, context=context)
@dataclass(frozen=True, slots=True)
class DeferredModuleCommandStep:
"""Resolve project-owned module source only when a real execution starts."""
entry: WorkflowEntry
command_factory: ModuleCommandFactory = _command_from_project
def __post_init__(self) -> None:
if not callable(self.command_factory):
raise TypeError("command_factory must be callable")
def execute(
self,
*,
context: RunContext,
timeout_seconds: float | None,
dry_run: bool,
) -> StepExecution:
if dry_run:
return StepExecution(exit_code=0, skipped=True, reason="dry-run")
command = self.command_factory(self.entry, context)
if not callable(getattr(command, "execute", None)):
raise TypeError("command_factory must return an executable step")
return command.execute(
context=context,
timeout_seconds=timeout_seconds,
dry_run=False,
)
def _resolve_entry(root: Path, configured_entry: str) -> Path:
if not configured_entry or "\\" in configured_entry or ":" in configured_entry:
raise ModuleSourceError("module entry must use a safe relative path")
relative = PurePosixPath(configured_entry)
if relative.is_absolute() or ".." in relative.parts:
raise ModuleSourceError("module entry cannot escape its module root")
try:
candidate = root.joinpath(*relative.parts).resolve(strict=True)
except OSError as exc:
raise ModuleSourceError(
f"module entry does not exist or cannot be resolved: {configured_entry}"
) from exc
if not candidate.is_relative_to(root):
raise ModuleSourceError("module entry cannot escape its module root")
if not candidate.is_file():
raise ModuleSourceError(f"module entry is not a file: {configured_entry}")
return candidate
__all__ = [
"DeferredModuleCommandStep",
"ModuleCommandAdapter",
"ModuleCommandFactory",
"ModuleSourceError",
"ModuleSourceRoots",
]
src/gyxx_flow/baseline/__init__.py
+31
View File
@@ -0,0 +1,31 @@
from .inventory import collect_code_inventory, collect_data_summary
from .manifest import build_baseline_manifest, write_manifest_atomic
from .models import (
DEFAULT_CODE_EXTENSIONS,
DataRootSpec,
ProjectSpec,
ScheduledTask,
project_specs_from_env,
)
from .tasks import (
PowerShellScheduledTaskProvider,
ScheduledTaskProvider,
TaskCountMismatch,
collect_scheduled_task_inventory,
)
__all__ = [
"DEFAULT_CODE_EXTENSIONS",
"DataRootSpec",
"ProjectSpec",
"ScheduledTask",
"PowerShellScheduledTaskProvider",
"ScheduledTaskProvider",
"TaskCountMismatch",
"build_baseline_manifest",
"collect_code_inventory",
"collect_data_summary",
"collect_scheduled_task_inventory",
"write_manifest_atomic",
"project_specs_from_env",
]
src/gyxx_flow/baseline/inventory.py
+168
View File
@@ -0,0 +1,168 @@
from __future__ import annotations
import hashlib
import os
from collections.abc import Iterable
from pathlib import Path
from typing import TypedDict
from .models import DEFAULT_CODE_EXTENSIONS
class Aggregate(TypedDict):
file_count: int
total_bytes: int
class DataSummary(Aggregate):
by_extension: dict[str, Aggregate]
by_top_level: dict[str, Aggregate]
_EXCLUDED_DIRECTORY_NAMES = frozenset(
{
".cache",
".git",
".hermes_tmp",
".mypy_cache",
".pytest_cache",
".ruff_cache",
".tox",
".venv",
"__pycache__",
"cache",
"data",
"debug",
"evidence",
"log",
"logs",
"node_modules",
"reports",
"runtime",
"state",
"tmp",
"var",
}
)
_SENSITIVE_DIRECTORY_MARKERS = ("cookie", "profile")
_SENSITIVE_FILE_NAMES = frozenset(
{
".env",
"credentials.json",
"secrets.json",
"service-account.json",
"service_account.json",
}
)
_SENSITIVE_FILE_MARKERS = ("cookie", "credential", "secret", "token")
def _is_excluded_directory(name: str) -> bool:
normalized = name.casefold()
return normalized in _EXCLUDED_DIRECTORY_NAMES or any(
marker in normalized for marker in _SENSITIVE_DIRECTORY_MARKERS
)
def _is_sensitive_file(name: str) -> bool:
normalized = name.casefold()
if normalized in _SENSITIVE_FILE_NAMES or normalized.startswith(".env."):
return True
stem = Path(normalized).stem
return any(marker in stem for marker in _SENSITIVE_FILE_MARKERS)
def _walk_files(
root: Path,
*,
exclude_runtime: bool,
excluded_roots: frozenset[Path] = frozenset(),
) -> Iterable[Path]:
if not root.is_dir():
raise FileNotFoundError(f"inventory root is not a directory: {root}")
for directory, dirnames, filenames in os.walk(root, followlinks=False):
if exclude_runtime:
dirnames[:] = sorted(
name
for name in dirnames
if not _is_excluded_directory(name)
and not (Path(directory) / name).is_symlink()
and (Path(directory) / name).resolve(strict=False) not in excluded_roots
)
else:
dirnames[:] = sorted(
name for name in dirnames if not (Path(directory) / name).is_symlink()
)
for filename in sorted(filenames):
path = Path(directory) / filename
if not path.is_symlink():
yield path
def _sha256(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as stream:
for chunk in iter(lambda: stream.read(1024 * 1024), b""):
digest.update(chunk)
return digest.hexdigest()
def collect_code_inventory(
root: Path,
*,
allowed_extensions: frozenset[str] = DEFAULT_CODE_EXTENSIONS,
excluded_roots: Iterable[Path] = (),
) -> list[dict[str, str | int]]:
"""Hash code-like files without retaining file contents or config values."""
root = Path(root)
normalized_extensions = frozenset(extension.casefold() for extension in allowed_extensions)
normalized_excluded_roots = frozenset(
Path(excluded_root).resolve(strict=False) for excluded_root in excluded_roots
)
inventory: list[dict[str, str | int]] = []
for path in _walk_files(
root, exclude_runtime=True, excluded_roots=normalized_excluded_roots
):
if path.suffix.casefold() not in normalized_extensions or _is_sensitive_file(path.name):
continue
stat = path.stat()
inventory.append(
{
"relative_path": path.relative_to(root).as_posix(),
"size_bytes": stat.st_size,
"mtime_ns": stat.st_mtime_ns,
"sha256": _sha256(path),
}
)
return sorted(inventory, key=lambda item: str(item["relative_path"]).casefold())
def _increment(target: dict[str, Aggregate], key: str, size: int) -> None:
aggregate = target.setdefault(key, {"file_count": 0, "total_bytes": 0})
aggregate["file_count"] += 1
aggregate["total_bytes"] += size
def collect_data_summary(root: Path) -> DataSummary:
"""Summarize a data tree without exposing names, paths, times, hashes, or values."""
root = Path(root)
summary: DataSummary = {
"file_count": 0,
"total_bytes": 0,
"by_extension": {},
"by_top_level": {},
}
for path in _walk_files(root, exclude_runtime=False):
size = path.stat().st_size
relative = path.relative_to(root)
top_level = relative.parts[0] if len(relative.parts) > 1 else "."
extension = path.suffix.casefold() or "[no_extension]"
summary["file_count"] += 1
summary["total_bytes"] += size
_increment(summary["by_extension"], extension, size)
_increment(summary["by_top_level"], top_level, size)
summary["by_extension"] = dict(sorted(summary["by_extension"].items()))
summary["by_top_level"] = dict(sorted(summary["by_top_level"].items()))
return summary
src/gyxx_flow/baseline/manifest.py
+82
View File
@@ -0,0 +1,82 @@
from __future__ import annotations
import json
import os
import tempfile
from collections.abc import Sequence
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
from .inventory import collect_code_inventory, collect_data_summary
from .models import ProjectSpec
from .tasks import ScheduledTaskProvider, collect_scheduled_task_inventory
def build_baseline_manifest(
specs: Sequence[ProjectSpec],
task_provider: ScheduledTaskProvider,
*,
generated_at: datetime | None = None,
expected_task_count: int = 21,
) -> dict[str, Any]:
"""Build a read-only baseline. It never serializes task actions or file contents."""
timestamp = generated_at or datetime.now(timezone.utc)
if timestamp.tzinfo is None or timestamp.utcoffset() is None:
raise ValueError("generated_at must be timezone-aware")
projects: list[dict[str, Any]] = []
for spec in specs:
projects.append(
{
"project_id": spec.project_id,
"source_root": str(spec.root),
"code_inventory": collect_code_inventory(
spec.root,
allowed_extensions=spec.code_extensions,
excluded_roots=(data_root.path for data_root in spec.data_roots),
),
"data_summaries": [
{"label": data_root.label, "summary": collect_data_summary(data_root.path)}
for data_root in spec.data_roots
],
}
)
return {
"schema_version": 1,
"generated_at": timestamp.isoformat(),
"projects": projects,
"scheduled_tasks": collect_scheduled_task_inventory(
specs, task_provider, expected_count=expected_task_count
),
}
def write_manifest_atomic(path: Path, manifest: object) -> None:
"""Durably replace a JSON manifest without exposing a partial destination file."""
path = Path(path)
path.parent.mkdir(parents=True, exist_ok=True)
temporary_path: Path | None = None
try:
with tempfile.NamedTemporaryFile(
mode="w",
encoding="utf-8",
newline="\n",
dir=path.parent,
prefix=f".{path.name}.",
suffix=".tmp",
delete=False,
) as stream:
temporary_path = Path(stream.name)
json.dump(manifest, stream, ensure_ascii=False, indent=2, sort_keys=True)
stream.write("\n")
stream.flush()
os.fsync(stream.fileno())
os.replace(temporary_path, path)
temporary_path = None
finally:
if temporary_path is not None:
temporary_path.unlink(missing_ok=True)
src/gyxx_flow/baseline/models.py
+113
View File
@@ -0,0 +1,113 @@
from __future__ import annotations
import os
from dataclasses import dataclass, field
from pathlib import Path
from typing import Mapping
DEFAULT_CODE_EXTENSIONS = frozenset(
{
".bat",
".cmd",
".json",
".md",
".ps1",
".py",
".pyi",
".rst",
".sh",
".sql",
".toml",
".txt",
".yaml",
".yml",
}
)
@dataclass(frozen=True, slots=True)
class DataRootSpec:
"""A named data tree whose file-level details must remain private."""
label: str
path: Path
def __post_init__(self) -> None:
if not self.label or self.label in {".", ".."}:
raise ValueError("data root label must be a non-empty logical name")
object.__setattr__(self, "path", Path(self.path))
@dataclass(frozen=True, slots=True)
class ProjectSpec:
"""Read-only discovery boundaries for one legacy project."""
project_id: str
root: Path
data_roots: tuple[DataRootSpec, ...] = ()
code_extensions: frozenset[str] = field(default=DEFAULT_CODE_EXTENSIONS)
def __post_init__(self) -> None:
if not self.project_id:
raise ValueError("project_id is required")
object.__setattr__(self, "root", Path(self.root))
object.__setattr__(self, "data_roots", tuple(self.data_roots))
normalized_extensions = frozenset(
extension.casefold() if extension.startswith(".") else f".{extension.casefold()}"
for extension in self.code_extensions
)
object.__setattr__(self, "code_extensions", normalized_extensions)
@dataclass(frozen=True, slots=True)
class ScheduledTask:
"""Provider-neutral scheduled-task action used only for source matching."""
task_id: str
command: str
arguments: str = ""
working_directory: str = ""
def project_specs_from_env(
env: Mapping[str, str] | None = None,
) -> tuple[ProjectSpec, ...]:
"""Build legacy discovery boundaries without embedding machine-specific roots."""
values = os.environ if env is None else env
keys = (
"GYXX_LEGACY_CONTENT_ROOT",
"GYXX_LEGACY_SHOP_ROOT",
"GYXX_LEGACY_PRODUCT_ROOT",
"GYXX_LEGACY_SUPPLY_ROOT",
)
missing = [key for key in keys if not values.get(key, "").strip()]
if missing:
raise ValueError("missing legacy root environment variables: " + ", ".join(missing))
content = Path(values[keys[0]]).expanduser()
shop = Path(values[keys[1]]).expanduser()
product = Path(values[keys[2]]).expanduser()
supply = Path(values[keys[3]]).expanduser()
return (
ProjectSpec(
"content_marketing",
content,
(DataRootSpec("data", content / "data"), DataRootSpec("reports", content / "reports")),
),
ProjectSpec(
"shop_intelligence",
shop,
(DataRootSpec("data", shop / "data"),),
),
ProjectSpec(
"product_commerce",
product,
(DataRootSpec("data", product / "data"),),
),
ProjectSpec(
"supply_chain",
supply,
(DataRootSpec("shared-data", supply / "shared-data"),),
),
)
src/gyxx_flow/baseline/tasks.py
+115
View File
@@ -0,0 +1,115 @@
from __future__ import annotations
import json
import os
import re
import subprocess
from collections.abc import Iterable, Sequence
from typing import Protocol
from .models import ProjectSpec, ScheduledTask
class ScheduledTaskProvider(Protocol):
def scheduled_tasks(self) -> Iterable[ScheduledTask]: ...
class PowerShellScheduledTaskProvider:
"""Read Windows Task Scheduler actions without modifying scheduler state."""
_SCRIPT = r"""
$ErrorActionPreference = 'Stop'
$rows = foreach ($task in Get-ScheduledTask) {
[pscustomobject]@{
task_id = "$($task.TaskPath)$($task.TaskName)"
command = [string](($task.Actions | ForEach-Object { $_.Execute }) -join ' ')
arguments = [string](($task.Actions | ForEach-Object { $_.Arguments }) -join ' ')
working_directory = [string](($task.Actions | ForEach-Object { $_.WorkingDirectory }) -join ' ')
}
}
@($rows) | ConvertTo-Json -Depth 3 -Compress
"""
def __init__(self, executable: str = "powershell.exe") -> None:
self.executable = executable
def scheduled_tasks(self) -> list[ScheduledTask]:
completed = subprocess.run(
[
self.executable,
"-NoLogo",
"-NoProfile",
"-NonInteractive",
"-Command",
self._SCRIPT,
],
check=True,
capture_output=True,
text=True,
encoding="utf-8",
errors="strict",
shell=False,
)
payload = json.loads(completed.stdout or "[]")
rows = payload if isinstance(payload, list) else [payload]
tasks: list[ScheduledTask] = []
for row in rows:
if not isinstance(row, dict) or not isinstance(row.get("task_id"), str):
raise ValueError("invalid scheduled-task provider response")
tasks.append(
ScheduledTask(
task_id=row["task_id"],
command=str(row.get("command") or ""),
arguments=str(row.get("arguments") or ""),
working_directory=str(row.get("working_directory") or ""),
)
)
return tasks
class TaskCountMismatch(ValueError):
"""The discovered legacy task inventory is not the accepted baseline."""
def _match_project(task: ScheduledTask, specs: Sequence[ProjectSpec]) -> str | None:
action = " ".join((task.command, task.arguments, task.working_directory))
normalized_action = os.path.normcase(action).replace("/", "\\")
for spec in specs:
normalized_root = os.path.normcase(str(spec.root)).replace("/", "\\").rstrip("\\")
root_pattern = re.escape(normalized_root) + r"(?=$|[\\\s\"'])"
if normalized_root and re.search(root_pattern, normalized_action):
return spec.project_id
return None
def collect_scheduled_task_inventory(
specs: Sequence[ProjectSpec],
provider: ScheduledTaskProvider,
*,
expected_count: int = 21,
) -> dict[str, object]:
"""Keep only legacy-project tasks and deliberately discard action details."""
tasks: list[dict[str, str]] = []
seen_ids: set[str] = set()
for task in provider.scheduled_tasks():
project_id = _match_project(task, specs)
if project_id is None:
continue
if task.task_id in seen_ids:
raise ValueError(f"duplicate scheduled task id: {task.task_id}")
seen_ids.add(task.task_id)
tasks.append({"task_id": task.task_id, "project_id": project_id})
tasks.sort(key=lambda item: item["task_id"].casefold())
actual_count = len(tasks)
if actual_count != expected_count:
raise TaskCountMismatch(
f"scheduled task baseline expected {expected_count} tasks but found {actual_count}"
)
return {
"expected_count": expected_count,
"actual_count": actual_count,
"is_complete": True,
"tasks": tasks,
}
src/gyxx_flow/catalog.py
+219
View File
@@ -0,0 +1,219 @@
"""Validated workflow and schedule catalog."""
from __future__ import annotations
import json
import re
from dataclasses import dataclass
from datetime import time
from pathlib import Path, PurePosixPath
from typing import Any, Literal
_WORKFLOW_ID = re.compile(r"^[a-z][a-z0-9]*(?:[._-][a-z0-9]+)*$")
_MODULES = {
"content_marketing",
"product_commerce",
"shop_intelligence",
"supply_chain",
}
_TRIGGERS = {"scheduled", "manual", "unavailable"}
_KINDS = {"daily", "weekly", "monthly", "interval_days"}
_WEEKDAYS = {"Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday"}
class CatalogError(ValueError):
"""Raised for an invalid workflow catalog."""
@dataclass(frozen=True, slots=True)
class WorkflowEntry:
workflow_id: str
module: str
trigger: Literal["scheduled", "manual", "unavailable"]
entry: str
args: tuple[str, ...] = ()
source_project: str | None = None
source_task_name: str | None = None
note: str | None = None
@dataclass(frozen=True, slots=True)
class ScheduleEntry:
workflow_id: str
kind: Literal["daily", "weekly", "monthly", "interval_days"]
at: str
days: tuple[str, ...] = ()
day_of_month: int | None = None
every_days: int | None = None
anchor_date: str | None = None
@dataclass(frozen=True, slots=True)
class WorkflowCatalog:
timezone: str
workflows: tuple[WorkflowEntry, ...]
schedules: tuple[ScheduleEntry, ...]
@classmethod
def load(cls, config_dir: Path | str) -> "WorkflowCatalog":
root = Path(config_dir)
workflow_payload = _load_json(root / "workflows.json")
schedule_payload = _load_json(root / "schedules.json")
_require_schema(workflow_payload, "workflows.json", expected=2)
_require_schema(schedule_payload, "schedules.json", expected=1)
workflows = tuple(_parse_workflow(item) for item in workflow_payload.get("workflows", []))
schedules = tuple(_parse_schedule(item) for item in schedule_payload.get("schedules", []))
timezone = schedule_payload.get("timezone")
if not isinstance(timezone, str) or not timezone:
raise CatalogError("schedules.json requires timezone")
catalog = cls(timezone=timezone, workflows=workflows, schedules=schedules)
catalog._validate_relations()
return catalog
def _validate_relations(self) -> None:
workflow_ids = [item.workflow_id for item in self.workflows]
duplicate_ids = _duplicates(workflow_ids)
if duplicate_ids:
raise CatalogError(f"duplicate workflow id: {sorted(duplicate_ids)[0]}")
task_names = [item.source_task_name for item in self.scheduled_workflows()]
duplicates = _duplicates([name for name in task_names if name])
if duplicates:
raise CatalogError(f"duplicate legacy task name: {sorted(duplicates)[0]}")
known = set(workflow_ids)
schedule_ids = [item.workflow_id for item in self.schedules]
duplicate_schedules = _duplicates(schedule_ids)
if duplicate_schedules:
raise CatalogError(f"duplicate schedule: {sorted(duplicate_schedules)[0]}")
unknown = set(schedule_ids) - known
if unknown:
raise CatalogError(f"schedule references unknown workflow: {sorted(unknown)[0]}")
scheduled_ids = {item.workflow_id for item in self.scheduled_workflows()}
if set(schedule_ids) != scheduled_ids:
missing = scheduled_ids - set(schedule_ids)
extra = set(schedule_ids) - scheduled_ids
raise CatalogError(f"schedule relation mismatch; missing={sorted(missing)}, extra={sorted(extra)}")
def scheduled_workflows(self) -> tuple[WorkflowEntry, ...]:
return tuple(item for item in self.workflows if item.trigger == "scheduled")
def manual_workflows(self) -> tuple[WorkflowEntry, ...]:
return tuple(item for item in self.workflows if item.trigger == "manual")
def unavailable_workflows(self) -> tuple[WorkflowEntry, ...]:
return tuple(item for item in self.workflows if item.trigger == "unavailable")
def schedule_for(self, workflow_id: str) -> ScheduleEntry:
for schedule in self.schedules:
if schedule.workflow_id == workflow_id:
return schedule
raise KeyError(workflow_id)
def _load_json(path: Path) -> dict[str, Any]:
try:
payload = json.loads(path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError) as exc:
raise CatalogError(f"cannot load {path.name}: {exc}") from exc
if not isinstance(payload, dict):
raise CatalogError(f"{path.name} must contain an object")
return payload
def _require_schema(payload: dict[str, Any], filename: str, *, expected: int) -> None:
if payload.get("schema_version") != expected:
raise CatalogError(f"unsupported schema_version in {filename}")
def _parse_workflow(item: Any) -> WorkflowEntry:
if not isinstance(item, dict):
raise CatalogError("workflow entry must be an object")
workflow_id = item.get("id")
module = item.get("module")
trigger = item.get("trigger")
execution = item.get("execution")
provenance = item.get("provenance", {})
if not isinstance(workflow_id, str) or not _WORKFLOW_ID.fullmatch(workflow_id):
raise CatalogError(f"invalid workflow id: {workflow_id!r}")
if module not in _MODULES:
raise CatalogError(f"invalid module for {workflow_id}: {module!r}")
if trigger not in _TRIGGERS:
raise CatalogError(f"invalid trigger for {workflow_id}: {trigger!r}")
if not isinstance(execution, dict):
raise CatalogError(f"missing execution definition for {workflow_id}")
if not isinstance(provenance, dict):
raise CatalogError(f"invalid provenance definition for {workflow_id}")
entry = execution.get("entry")
if not _is_relative_entry(entry):
raise CatalogError(f"execution entry must be relative for {workflow_id}")
project = provenance.get("source_project")
if project is not None and (not isinstance(project, str) or not project):
raise CatalogError(f"invalid source project for {workflow_id}")
task_name = provenance.get("task_name")
if trigger == "scheduled" and (not isinstance(task_name, str) or not task_name):
raise CatalogError(f"scheduled workflow requires source task name: {workflow_id}")
args = execution.get("args", [])
if not isinstance(args, list) or not all(isinstance(value, str) for value in args):
raise CatalogError(f"execution args must be strings for {workflow_id}")
return WorkflowEntry(
workflow_id=workflow_id,
module=module,
trigger=trigger,
entry=entry,
args=tuple(args),
source_project=project,
source_task_name=task_name,
note=item.get("note"),
)
def _is_relative_entry(entry: Any) -> bool:
if not isinstance(entry, str) or not entry or "\\" in entry or ":" in entry:
return False
path = PurePosixPath(entry)
return not path.is_absolute() and ".." not in path.parts
def _parse_schedule(item: Any) -> ScheduleEntry:
if not isinstance(item, dict):
raise CatalogError("schedule entry must be an object")
workflow_id = item.get("workflow_id")
kind = item.get("kind")
at = item.get("at")
if not isinstance(workflow_id, str) or not _WORKFLOW_ID.fullmatch(workflow_id):
raise CatalogError(f"invalid schedule workflow id: {workflow_id!r}")
if kind not in _KINDS:
raise CatalogError(f"invalid schedule kind for {workflow_id}: {kind!r}")
try:
time.fromisoformat(at)
except (TypeError, ValueError) as exc:
raise CatalogError(f"invalid schedule time for {workflow_id}: {at!r}") from exc
days = tuple(item.get("days", []))
if kind == "weekly" and (not days or not set(days) <= _WEEKDAYS):
raise CatalogError(f"invalid weekly days for {workflow_id}")
day_of_month = item.get("day_of_month")
if kind == "monthly" and (not isinstance(day_of_month, int) or not 1 <= day_of_month <= 31):
raise CatalogError(f"invalid day_of_month for {workflow_id}")
every_days = item.get("every_days")
if kind == "interval_days" and (not isinstance(every_days, int) or every_days < 1):
raise CatalogError(f"invalid every_days for {workflow_id}")
return ScheduleEntry(
workflow_id=workflow_id,
kind=kind,
at=at,
days=days,
day_of_month=day_of_month,
every_days=every_days,
anchor_date=item.get("anchor_date"),
)
def _duplicates(values: list[str]) -> set[str]:
seen: set[str] = set()
duplicates: set[str] = set()
for value in values:
if value in seen:
duplicates.add(value)
seen.add(value)
return duplicates
src/gyxx_flow/cli.py
+545
View File
@@ -0,0 +1,545 @@
"""Operator-safe command-line entry point for GYXX Flow."""
from __future__ import annotations
import argparse
import hashlib
import json
import sys
from collections.abc import Sequence
from contextlib import suppress
from datetime import date, datetime, timedelta, timezone
from pathlib import Path
from typing import TextIO
from zoneinfo import ZoneInfo
from gyxx_flow.acceptance import build_acceptance_report
from gyxx_flow.adapters.native import DeferredModuleCommandStep
from gyxx_flow.catalog import CatalogError, WorkflowCatalog, WorkflowEntry
from gyxx_flow.core.artifacts import atomic_write_json
from gyxx_flow.core.config import Settings
from gyxx_flow.core.context import RunContext
from gyxx_flow.core.layout import DataLayout
from gyxx_flow.core.locks import LockManager
from gyxx_flow.core.records import RunJournal
from gyxx_flow.diagnostics import run_doctor
from gyxx_flow.modules import create_default_registry as create_module_registry
from gyxx_flow.ops import EffectLedger, RunIndex
from gyxx_flow.scheduler import (
PowerShellCurrentTaskProvider,
SchedulerConfig,
build_schedule_plan,
detect_schedule_drift,
write_schedule_plan_bundle,
)
from gyxx_flow.script_catalog import ScriptCatalog, ScriptCatalogError
from gyxx_flow.workflow.engine import WorkflowEngine, WorkflowRunResult
from gyxx_flow.workflow.model import StepDefinition, WorkflowDefinition
from gyxx_flow.workflow.registry import WorkflowRegistry, WorkflowRegistryError
from gyxx_flow.workflow.selection import rerun_step, resume_from
EXIT_SUCCESS = 0
EXIT_WORKFLOW_FAILED = 1
EXIT_USAGE = 2
EXIT_CONFIGURATION = 3
EXIT_RUNTIME = 4
EXIT_ACCEPTANCE_INCOMPLETE = 1
MAX_BACKFILL_DAYS = 366
class CliConfigurationError(ValueError):
"""Safe operator-facing configuration or input error."""
class CliRuntimeError(RuntimeError):
"""Sanitized runtime failure."""
def build_parser() -> argparse.ArgumentParser:
"""Build the CLI parser without initializing runtime resources."""
parser = argparse.ArgumentParser(
prog="gyxx",
description="GYXX Flow workflow orchestration",
)
commands = parser.add_subparsers(dest="command", required=True)
list_parser = commands.add_parser("list", help="list catalog workflows")
list_parser.add_argument("--json", action="store_true", help="emit machine-readable JSON")
scripts_parser = commands.add_parser(
"scripts", help="discover and manually run project-owned module scripts"
)
script_commands = scripts_parser.add_subparsers(dest="scripts_command", required=True)
scripts_list_parser = script_commands.add_parser("list", help="list runnable scripts")
scripts_list_parser.add_argument("--module")
scripts_list_parser.add_argument("--json", action="store_true")
scripts_run_parser = script_commands.add_parser("run", help="run one local script")
scripts_run_parser.add_argument("script_id")
scripts_run_date = scripts_run_parser.add_mutually_exclusive_group(required=True)
scripts_run_date.add_argument("--date", help="business date (YYYY-MM-DD)")
scripts_run_date.add_argument("--scheduled", action="store_true")
scripts_run_parser.add_argument("--execute", action="store_true")
scripts_run_parser.add_argument("--shadow", action="store_true")
scripts_run_parser.add_argument(
"--arg",
dest="script_args",
action="append",
default=[],
help="one argument passed to the script; repeat for multiple arguments",
)
run_parser = commands.add_parser("run", help="run one workflow business date")
_add_execution_arguments(run_parser)
run_date = run_parser.add_mutually_exclusive_group(required=True)
run_date.add_argument("--date", help="business date (YYYY-MM-DD)")
run_date.add_argument(
"--scheduled",
action="store_true",
help="execute for today's Asia/Shanghai business date",
)
backfill_parser = commands.add_parser(
"backfill", help="run an inclusive business-date range"
)
_add_execution_arguments(backfill_parser)
backfill_parser.add_argument(
"--from", dest="from_date", required=True, help="first business date"
)
backfill_parser.add_argument(
"--to", dest="to_date", required=True, help="last business date"
)
schedule_parser = commands.add_parser("schedule", help="plan managed schedules")
schedule_commands = schedule_parser.add_subparsers(
dest="schedule_command", required=True
)
plan_parser = schedule_commands.add_parser(
"plan", help="write an inert Task Scheduler bundle and drift report"
)
plan_parser.add_argument("--output", type=Path)
plan_parser.add_argument("--start-date", help="trigger boundary date (YYYY-MM-DD)")
plan_parser.add_argument("--python-executable", type=Path, required=True)
doctor_parser = commands.add_parser("doctor", help="run environment preflight checks")
doctor_parser.add_argument("--json", action="store_true")
acceptance_parser = commands.add_parser(
"acceptance", help="inspect acceptance evidence"
)
acceptance_commands = acceptance_parser.add_subparsers(
dest="acceptance_command", required=True
)
status_parser = acceptance_commands.add_parser("status")
status_parser.add_argument("--json", action="store_true")
status_parser.add_argument("--output", type=Path)
return parser
def _add_execution_arguments(parser: argparse.ArgumentParser) -> None:
parser.add_argument("workflow_id")
parser.add_argument(
"--execute",
action="store_true",
help="execute registered steps; omitted means no-side-effect dry-run",
)
parser.add_argument("--shadow", action="store_true")
recovery = parser.add_mutually_exclusive_group()
recovery.add_argument("--resume-from", metavar="STEP_ID")
recovery.add_argument("--rerun-step", metavar="STEP_ID")
def build_default_registry(settings: Settings) -> WorkflowRegistry:
"""Compose only workflows explicitly implemented by migrated modules."""
catalog = WorkflowCatalog.load(settings.project_root / "config")
registry = WorkflowRegistry(catalog)
modules = create_module_registry(catalog=catalog)
for workflow_id in modules.workflow_ids:
registry.register(modules.workflow(workflow_id))
return registry
def main(
argv: Sequence[str] | None = None,
*,
registry: WorkflowRegistry | None = None,
settings: Settings | None = None,
stdout: TextIO | None = None,
stderr: TextIO | None = None,
current_task_provider: PowerShellCurrentTaskProvider | None = None,
) -> int:
"""Execute a CLI command and return a stable process exit code."""
output = stdout or sys.stdout
errors = stderr or sys.stderr
parser = build_parser()
try:
arguments = parser.parse_args(argv)
except SystemExit as exc:
return int(exc.code)
resolved_settings = settings or Settings.from_env()
try:
if arguments.command == "doctor":
return _doctor_command(arguments, resolved_settings, output)
if (
arguments.command == "acceptance"
and arguments.acceptance_command == "status"
):
return _acceptance_status_command(arguments, resolved_settings, output)
if arguments.command == "scripts" and arguments.scripts_command == "list":
return _list_scripts(arguments, output)
if arguments.command == "scripts" and arguments.scripts_command == "run":
return _run_script(arguments, resolved_settings, output)
resolved_registry = registry or build_default_registry(resolved_settings)
if arguments.command == "list":
return _list_workflows(arguments, resolved_registry, output)
if arguments.command == "run":
return _run_command(arguments, resolved_registry, resolved_settings, output)
if arguments.command == "backfill":
return _backfill_command(arguments, resolved_registry, resolved_settings, output)
if arguments.command == "schedule" and arguments.schedule_command == "plan":
return _schedule_plan_command(
arguments,
resolved_registry.catalog,
resolved_settings,
output,
current_task_provider=current_task_provider,
)
raise CliConfigurationError(f"unsupported command: {arguments.command}")
except (
CatalogError,
ScriptCatalogError,
WorkflowRegistryError,
CliConfigurationError,
ValueError,
) as exc:
errors.write(f"error: {exc}\n")
return EXIT_CONFIGURATION
except CliRuntimeError as exc:
errors.write(f"error: {exc}\n")
return EXIT_RUNTIME
def _doctor_command(
arguments: argparse.Namespace, settings: Settings, output: TextIO
) -> int:
report = run_doctor(settings)
if arguments.json:
_write_json(output, report.as_dict())
else:
for check in report.checks:
marker = "PASS" if check.passed else "FAIL"
output.write(f"{marker}\t{check.name}\t{check.message}\n")
return EXIT_SUCCESS if report.is_healthy else EXIT_RUNTIME
def _acceptance_status_command(
arguments: argparse.Namespace, settings: Settings, output: TextIO
) -> int:
report = build_acceptance_report(settings)
payload = report.as_dict()
if arguments.output is not None:
atomic_write_json(arguments.output, payload)
if arguments.json:
_write_json(output, payload)
else:
summary = payload["summary"]
output.write(
f"acceptance: {summary['completed']}/{summary['total']} complete; "
f"pending={summary['pending']}\n"
)
return EXIT_SUCCESS if report.is_complete else EXIT_ACCEPTANCE_INCOMPLETE
def _list_workflows(
arguments: argparse.Namespace,
registry: WorkflowRegistry,
output: TextIO,
) -> int:
rows = [
{
"id": entry.workflow_id,
"module": entry.module,
"registered": registry.is_registered(entry.workflow_id),
"trigger": entry.trigger,
}
for entry in registry.catalog.workflows
]
if arguments.json:
_write_json(output, rows)
else:
for row in rows:
state = "ready" if row["registered"] else "not-registered"
output.write(
f"{row['id']}\t{row['module']}\t{row['trigger']}\t{state}\n"
)
return EXIT_SUCCESS
def _list_scripts(arguments: argparse.Namespace, output: TextIO) -> int:
catalog = ScriptCatalog.discover_default()
rows = [
{
"id": script.script_id,
"module": script.module,
"entry": script.entry,
"kind": script.kind,
}
for script in catalog.scripts
if arguments.module is None or script.module == arguments.module
]
if arguments.module is not None and not rows:
raise CliConfigurationError(f"unknown module or no runnable scripts: {arguments.module}")
if arguments.json:
_write_json(output, rows)
else:
for row in rows:
output.write(f"{row['id']}\t{row['kind']}\n")
return EXIT_SUCCESS
def _run_script(
arguments: argparse.Namespace,
settings: Settings,
output: TextIO,
) -> int:
script = ScriptCatalog.discover_default().get(arguments.script_id)
suffix = hashlib.sha256(script.script_id.encode("utf-8")).hexdigest()[:12]
workflow_id = f"script.{script.module}.{suffix}"
script_args = tuple(arguments.script_args)
entry = WorkflowEntry(
workflow_id=workflow_id,
module=script.module,
trigger="manual",
entry=script.entry,
args=script_args,
)
workflow = WorkflowDefinition(
workflow_id,
(
StepDefinition(
"module_script",
DeferredModuleCommandStep(entry),
timeout_seconds=6 * 60 * 60,
max_attempts=1,
resources=(f"module:{script.module}",),
production_sink=True,
),
),
)
business_date = _today_shanghai() if arguments.scheduled else _parse_date(arguments.date)
exit_code, payload = _execute_once(
workflow,
business_date=business_date,
shadow=arguments.shadow,
dry_run=not (arguments.execute or arguments.scheduled),
settings=settings,
)
payload["script_id"] = script.script_id
_write_json(output, payload)
return exit_code
def _run_command(
arguments: argparse.Namespace,
registry: WorkflowRegistry,
settings: Settings,
output: TextIO,
) -> int:
registered = registry.resolve(arguments.workflow_id)
workflow = _select_workflow(registered.definition, arguments)
business_date = _today_shanghai() if arguments.scheduled else _parse_date(arguments.date)
dry_run = not (arguments.execute or arguments.scheduled)
exit_code, payload = _execute_once(
workflow,
business_date=business_date,
shadow=arguments.shadow,
dry_run=dry_run,
settings=settings,
)
_write_json(output, payload)
return exit_code
def _schedule_plan_command(
arguments: argparse.Namespace,
catalog: WorkflowCatalog,
settings: Settings,
output: TextIO,
*,
current_task_provider: PowerShellCurrentTaskProvider | None,
) -> int:
start_date = (
_parse_date(arguments.start_date)
if arguments.start_date
else _today_shanghai()
)
destination = arguments.output
if destination is None:
timestamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
destination = settings.data_root / "evidence" / "schedule-plans" / timestamp
config = SchedulerConfig(
python_executable=arguments.python_executable,
project_root=settings.project_root,
)
plan = build_schedule_plan(catalog, config, start_date=start_date)
try:
bundle = write_schedule_plan_bundle(destination, plan)
except FileExistsError as exc:
raise CliConfigurationError(
f"schedule plan destination already exists: {destination}"
) from exc
provider = current_task_provider or PowerShellCurrentTaskProvider()
try:
current = provider.current_tasks(config.task_path)
except Exception as exc:
raise CliRuntimeError("cannot read current managed scheduled tasks") from exc
drift = detect_schedule_drift(plan, current)
atomic_write_json(bundle / "drift.json", drift.as_dict())
_write_json(
output,
{
"path": str(bundle),
"desired_count": len(plan.tasks),
"drift_count": len(drift.items),
"is_clean": drift.is_clean,
"applied": False,
},
)
return EXIT_SUCCESS
def _backfill_command(
arguments: argparse.Namespace,
registry: WorkflowRegistry,
settings: Settings,
output: TextIO,
) -> int:
registered = registry.resolve(arguments.workflow_id)
workflow = _select_workflow(registered.definition, arguments)
start = _parse_date(arguments.from_date)
end = _parse_date(arguments.to_date)
if end < start:
raise CliConfigurationError("backfill end date must not precede start date")
day_count = (end - start).days + 1
if day_count > MAX_BACKFILL_DAYS:
raise CliConfigurationError(
f"backfill range exceeds safety limit of {MAX_BACKFILL_DAYS} days"
)
runs: list[dict[str, object]] = []
final_exit = EXIT_SUCCESS
for offset in range(day_count):
business_date = start + timedelta(days=offset)
exit_code, payload = _execute_once(
workflow,
business_date=business_date,
shadow=arguments.shadow,
dry_run=not arguments.execute,
settings=settings,
)
runs.append(payload)
if exit_code != EXIT_SUCCESS:
final_exit = EXIT_WORKFLOW_FAILED
_write_json(
output,
{
"workflow_id": workflow.workflow_id,
"from": start.isoformat(),
"to": end.isoformat(),
"dry_run": not arguments.execute,
"status": "success" if final_exit == EXIT_SUCCESS else "failed",
"runs": runs,
},
)
return final_exit
def _select_workflow(workflow, arguments: argparse.Namespace): # type: ignore[no-untyped-def]
if arguments.rerun_step:
return rerun_step(workflow, arguments.rerun_step)
if arguments.resume_from:
return resume_from(workflow, arguments.resume_from)
return workflow
def _execute_once(
workflow, # type: ignore[no-untyped-def]
*,
business_date: date,
shadow: bool,
dry_run: bool,
settings: Settings,
) -> tuple[int, dict[str, object]]:
context = RunContext.create(
workflow.workflow_id,
business_date.isoformat(),
shadow=shadow,
)
layout = DataLayout(settings.data_root)
journal: RunJournal | None = None
try:
journal = RunJournal.create(layout, context)
result = WorkflowEngine(
LockManager(settings.data_root / "state" / "locks"),
effect_ledger=EffectLedger(settings.data_root),
).execute(workflow, context=context, journal=journal, dry_run=dry_run)
RunIndex(settings.data_root).index_journal(journal)
except Exception as exc:
if journal is not None:
with suppress(OSError, ValueError, KeyError, TypeError, json.JSONDecodeError):
journal.finalize("failed", error="runtime infrastructure failure")
RunIndex(settings.data_root).index_journal(journal)
raise CliRuntimeError(
f"runtime failure while executing {workflow.workflow_id}"
) from exc
return result.exit_code, _result_payload(
context,
result,
dry_run=dry_run,
journal_path=journal.path.relative_to(settings.data_root).as_posix(),
)
def _result_payload(
context: RunContext,
result: WorkflowRunResult,
*,
dry_run: bool,
journal_path: str,
) -> dict[str, object]:
return {
"workflow_id": result.workflow_id,
"run_id": context.run_id,
"business_date": context.business_date.isoformat(),
"shadow": context.shadow,
"dry_run": dry_run,
"status": result.status,
"exit_code": result.exit_code,
"journal_path": journal_path,
"steps": {
step_id: step.status for step_id, step in result.steps.items()
},
"warnings": list(result.warnings),
}
def _parse_date(value: str) -> date:
try:
parsed = date.fromisoformat(value)
except (TypeError, ValueError) as exc:
raise CliConfigurationError(f"invalid business date: {value!r}") from exc
if parsed.isoformat() != value:
raise CliConfigurationError(f"invalid business date: {value!r}")
return parsed
def _today_shanghai() -> date:
return datetime.now(ZoneInfo("Asia/Shanghai")).date()
def _write_json(output: TextIO, payload: object) -> None:
json.dump(payload, output, ensure_ascii=False, sort_keys=True)
output.write("\n")
src/gyxx_flow/core/__init__.py
+7
View File
@@ -0,0 +1,7 @@
"""Core contracts and runtime primitives for GYXX Flow."""
from .config import Settings
from .context import RunContext
from .layout import DataLayout, ModuleDataPaths
__all__ = ["DataLayout", "ModuleDataPaths", "RunContext", "Settings"]
src/gyxx_flow/core/artifacts.py
+87
View File
@@ -0,0 +1,87 @@
"""Immutable artifact metadata and atomic file helpers."""
from __future__ import annotations
import hashlib
import json
import os
import uuid
from dataclasses import asdict, dataclass
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
def atomic_write_json(path: Path | str, payload: Any) -> Path:
destination = Path(path)
destination.parent.mkdir(parents=True, exist_ok=True)
temporary = destination.with_name(f".{destination.name}.{uuid.uuid4().hex}.tmp")
try:
with temporary.open("w", encoding="utf-8", newline="\n") as handle:
json.dump(payload, handle, ensure_ascii=False, indent=2, sort_keys=True)
handle.write("\n")
handle.flush()
os.fsync(handle.fileno())
os.replace(temporary, destination)
finally:
temporary.unlink(missing_ok=True)
return destination
def sha256_file(path: Path | str, *, chunk_size: int = 1024 * 1024) -> str:
digest = hashlib.sha256()
with Path(path).open("rb") as handle:
for chunk in iter(lambda: handle.read(chunk_size), b""):
digest.update(chunk)
return digest.hexdigest()
@dataclass(frozen=True, slots=True)
class ArtifactManifest:
artifact_id: str
dataset: str
workflow_id: str
run_id: str
business_date: str
schema_version: str
source_path: str
byte_size: int
sha256: str
row_count: int | None
upstream_artifact_ids: tuple[str, ...]
created_at: str
@classmethod
def from_file(
cls,
path: Path | str,
*,
artifact_id: str,
dataset: str,
workflow_id: str,
run_id: str,
business_date: str,
schema_version: str,
row_count: int | None = None,
upstream_artifact_ids: tuple[str, ...] = (),
) -> "ArtifactManifest":
source = Path(path).resolve()
return cls(
artifact_id=artifact_id,
dataset=dataset,
workflow_id=workflow_id,
run_id=run_id,
business_date=business_date,
schema_version=schema_version,
source_path=str(source),
byte_size=source.stat().st_size,
sha256=sha256_file(source),
row_count=row_count,
upstream_artifact_ids=tuple(upstream_artifact_ids),
created_at=datetime.now(timezone.utc).isoformat(timespec="seconds"),
)
def to_dict(self) -> dict[str, Any]:
payload = asdict(self)
payload["upstream_artifact_ids"] = list(self.upstream_artifact_ids)
return payload
src/gyxx_flow/core/config.py
+30
View File
@@ -0,0 +1,30 @@
"""Portable workspace configuration."""
from __future__ import annotations
import os
from dataclasses import dataclass
from pathlib import Path
from typing import Mapping
@dataclass(frozen=True, slots=True)
class Settings:
"""Resolved paths used by the orchestration runtime."""
project_root: Path
data_root: Path
@classmethod
def from_env(
cls,
*,
project_root: Path | str | None = None,
env: Mapping[str, str] | None = None,
) -> "Settings":
values = os.environ if env is None else env
root = Path(project_root) if project_root is not None else Path(__file__).resolve().parents[3]
root = root.expanduser().resolve()
configured_data_root = values.get("GYXX_DATA_ROOT", "").strip()
data_root = Path(configured_data_root).expanduser() if configured_data_root else root / "var"
return cls(project_root=root, data_root=data_root.resolve())
src/gyxx_flow/core/context.py
+56
View File
@@ -0,0 +1,56 @@
"""Traceable workflow run context."""
from __future__ import annotations
import re
import secrets
from dataclasses import dataclass
from datetime import date, datetime, timezone
_WORKFLOW_ID = re.compile(r"^[a-z][a-z0-9]*(?:[._-][a-z0-9]+)*$")
_SUFFIX = re.compile(r"^[a-z0-9]+$")
@dataclass(frozen=True, slots=True)
class RunContext:
workflow_id: str
run_id: str
business_date: date
started_at: datetime
shadow: bool = False
@classmethod
def create(
cls,
workflow_id: str,
business_date: str,
*,
shadow: bool = False,
now: datetime | None = None,
random_suffix: str | None = None,
) -> "RunContext":
if not _WORKFLOW_ID.fullmatch(workflow_id):
raise ValueError(f"unsafe workflow_id: {workflow_id!r}")
try:
parsed_date = date.fromisoformat(business_date)
except (TypeError, ValueError) as exc:
raise ValueError(f"invalid business_date: {business_date!r}") from exc
if parsed_date.isoformat() != business_date:
raise ValueError(f"invalid business_date: {business_date!r}")
started_at = now or datetime.now(timezone.utc)
if started_at.tzinfo is None:
started_at = started_at.replace(tzinfo=timezone.utc)
started_at = started_at.astimezone(timezone.utc)
suffix = random_suffix or secrets.token_hex(3)
if not _SUFFIX.fullmatch(suffix):
raise ValueError(f"unsafe random suffix: {suffix!r}")
timestamp = started_at.strftime("%Y%m%dT%H%M%SZ")
run_id = f"{workflow_id}__{parsed_date:%Y%m%d}__{timestamp}__{suffix}"
return cls(
workflow_id=workflow_id,
run_id=run_id,
business_date=parsed_date,
started_at=started_at,
shadow=shadow,
)
src/gyxx_flow/core/layout.py
+147
View File
@@ -0,0 +1,147 @@
"""Canonical runtime data layout."""
from __future__ import annotations
import re
from dataclasses import dataclass
from pathlib import Path
from typing import Self
_SAFE_SEGMENT = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._=-]*$")
def _segment(value: str) -> str:
if not isinstance(value, str) or not _SAFE_SEGMENT.fullmatch(value) or value in {".", ".."}:
raise ValueError(f"unsafe path segment: {value!r}")
return value
@dataclass(frozen=True, slots=True)
class DataLayout:
root: Path
def __post_init__(self) -> None:
object.__setattr__(self, "root", Path(self.root).expanduser().resolve())
def for_module(self, module: str) -> ModuleDataPaths:
"""Return the canonical mutable-data boundary for one business module."""
return ModuleDataPaths.from_layout(self, module)
def raw(
self,
*,
domain: str,
source: str,
dataset: str,
business_date: str,
run_id: str,
) -> Path:
return self.root.joinpath(
"data",
"raw",
_segment(domain),
_segment(source),
_segment(dataset),
f"business_date={_segment(business_date)}",
f"run_id={_segment(run_id)}",
)
def normalized(self, domain: str, dataset: str, schema_version: str) -> Path:
return self.root / "data" / "normalized" / _segment(domain) / _segment(dataset) / f"schema_v{_segment(schema_version)}"
def curated(self, domain: str, dataset: str) -> Path:
return self.root / "data" / "curated" / _segment(domain) / _segment(dataset)
def export(self, consumer: str, workflow_id: str, business_date: str, run_id: str) -> Path:
return self.root.joinpath(
"data", "exports", _segment(consumer), _segment(workflow_id), _segment(business_date), _segment(run_id)
)
def evidence(self, workflow_id: str, run_id: str) -> Path:
return self.root / "data" / "evidence" / _segment(workflow_id) / _segment(run_id)
def state(self, category: str, name: str) -> Path:
return self.root / "state" / _segment(category) / _segment(name)
def log_dir(self, workflow_id: str, business_date: str) -> Path:
year, month, day = _segment(business_date).split("-")
return self.root / "logs" / _segment(workflow_id) / year / month / day
def run_dir(self, workflow_id: str, business_date: str, run_id: str) -> Path:
year, month, day = _segment(business_date).split("-")
return self.root / "runs" / _segment(workflow_id) / year / month / day / _segment(run_id)
def tmp(self, run_id: str) -> Path:
return self.root / "tmp" / _segment(run_id)
def quarantine(self, workflow_id: str, run_id: str) -> Path:
return self.root / "quarantine" / _segment(workflow_id) / _segment(run_id)
@dataclass(frozen=True, slots=True)
class ModuleDataPaths:
"""Canonical, relocatable roots used by a migrated business module.
Construction is side-effect free. Callers create only the specific parent
directory they are about to write, keeping imports and dry-runs read-only.
"""
data_root: Path
module: str
raw_root: Path
normalized_root: Path
curated_root: Path
exports_root: Path
evidence_root: Path
state_root: Path
logs_root: Path
tmp_root: Path
@classmethod
def from_layout(cls, layout: DataLayout, module: str) -> Self:
safe_module = _segment(module)
root = layout.root
return cls(
data_root=root,
module=safe_module,
raw_root=root / "data" / "raw" / safe_module,
normalized_root=root / "data" / "normalized" / safe_module,
curated_root=root / "data" / "curated" / safe_module,
exports_root=root / "data" / "exports" / safe_module,
evidence_root=root / "data" / "evidence" / safe_module,
state_root=root / "state" / safe_module,
logs_root=root / "logs" / safe_module,
tmp_root=root / "tmp" / safe_module,
)
def raw_path(self, *parts: str | Path) -> Path:
return _inside(self.raw_root, *parts)
def normalized_path(self, *parts: str | Path) -> Path:
return _inside(self.normalized_root, *parts)
def curated_path(self, *parts: str | Path) -> Path:
return _inside(self.curated_root, *parts)
def export_path(self, *parts: str | Path) -> Path:
return _inside(self.exports_root, *parts)
def evidence_path(self, *parts: str | Path) -> Path:
return _inside(self.evidence_root, *parts)
def state_path(self, *parts: str | Path) -> Path:
return _inside(self.state_root, *parts)
def log_path(self, *parts: str | Path) -> Path:
return _inside(self.logs_root, *parts)
def tmp_path(self, *parts: str | Path) -> Path:
return _inside(self.tmp_root, *parts)
def _inside(root: Path, *parts: str | Path) -> Path:
candidate = root.joinpath(*parts).resolve()
if candidate != root and not candidate.is_relative_to(root):
raise ValueError(f"path escapes module data root: {parts!r}")
return candidate
src/gyxx_flow/core/locks.py
+88
View File
@@ -0,0 +1,88 @@
"""Cross-process named resource locks backed by exclusive files."""
from __future__ import annotations
import hashlib
import json
import os
import re
import time
from dataclasses import dataclass
from datetime import datetime, timezone
from pathlib import Path
_RESOURCE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9:._-]*$")
_OWNER = re.compile(r"^[A-Za-z0-9][A-Za-z0-9:._-]*$")
class ResourceBusyError(RuntimeError):
"""Raised when a named resource cannot be acquired before its deadline."""
@dataclass(slots=True)
class NamedResourceLock:
path: Path
resource: str
owner: str
timeout_seconds: float
poll_seconds: float
_acquired: bool = False
def __enter__(self) -> "NamedResourceLock":
deadline = time.monotonic() + max(0.0, self.timeout_seconds)
self.path.parent.mkdir(parents=True, exist_ok=True)
while True:
try:
descriptor = os.open(self.path, os.O_CREAT | os.O_EXCL | os.O_WRONLY)
except FileExistsError:
if time.monotonic() >= deadline:
raise ResourceBusyError(f"resource is busy: {self.resource}")
time.sleep(self.poll_seconds)
continue
metadata = {
"resource": self.resource,
"owner": self.owner,
"acquired_at": datetime.now(timezone.utc).isoformat(timespec="seconds"),
"pid": os.getpid(),
}
with os.fdopen(descriptor, "w", encoding="utf-8", newline="\n") as handle:
json.dump(metadata, handle, ensure_ascii=False, indent=2, sort_keys=True)
handle.write("\n")
handle.flush()
os.fsync(handle.fileno())
self._acquired = True
return self
def __exit__(self, exc_type, exc, traceback) -> None: # type: ignore[no-untyped-def]
if self._acquired:
self.path.unlink(missing_ok=True)
self._acquired = False
@dataclass(frozen=True, slots=True)
class LockManager:
root: Path
def __post_init__(self) -> None:
object.__setattr__(self, "root", Path(self.root).expanduser().resolve())
def acquire(
self,
resource: str,
*,
owner: str,
timeout_seconds: float = 30.0,
poll_seconds: float = 0.1,
) -> NamedResourceLock:
if not _RESOURCE.fullmatch(resource):
raise ValueError(f"invalid resource name: {resource!r}")
if not _OWNER.fullmatch(owner):
raise ValueError(f"invalid owner name: {owner!r}")
digest = hashlib.sha256(resource.encode("utf-8")).hexdigest()[:32]
return NamedResourceLock(
path=self.root / f"{digest}.lock",
resource=resource,
owner=owner,
timeout_seconds=timeout_seconds,
poll_seconds=poll_seconds,
)
src/gyxx_flow/core/records.py
+135
View File
@@ -0,0 +1,135 @@
"""Atomic workflow run and step status journal."""
from __future__ import annotations
import json
from dataclasses import dataclass
from datetime import datetime, timezone
from pathlib import Path
from typing import Any, Literal
from .artifacts import atomic_write_json
from .context import RunContext
from .layout import DataLayout
StepStatus = Literal["success", "failed", "skipped"]
RunStatus = Literal["success", "failed", "cancelled"]
def _now() -> str:
return datetime.now(timezone.utc).isoformat(timespec="seconds")
@dataclass(frozen=True, slots=True)
class RunJournal:
path: Path
@classmethod
def create(cls, layout: DataLayout, context: RunContext) -> "RunJournal":
run_dir = layout.run_dir(
context.workflow_id,
context.business_date.isoformat(),
context.run_id,
)
log_dir = layout.log_dir(context.workflow_id, context.business_date.isoformat())
evidence_dir = layout.evidence(context.workflow_id, context.run_id)
log_dir.mkdir(parents=True, exist_ok=True)
evidence_dir.mkdir(parents=True, exist_ok=True)
path = run_dir / "run.json"
atomic_write_json(
path,
{
"workflow_id": context.workflow_id,
"run_id": context.run_id,
"business_date": context.business_date.isoformat(),
"shadow": context.shadow,
"started_at": context.started_at.isoformat(timespec="seconds"),
"ended_at": None,
"status": "running",
"error": None,
"steps": {},
"trace": {
"paths": {
"run": run_dir.relative_to(layout.root).as_posix(),
"log": log_dir.relative_to(layout.root).as_posix(),
"evidence": evidence_dir.relative_to(layout.root).as_posix(),
},
"inputs": [],
"outputs": [],
"external_writes": [],
},
},
)
return cls(path=path)
def _read(self) -> dict[str, Any]:
return json.loads(self.path.read_text(encoding="utf-8"))
def start_step(self, step_id: str, *, attempt: int) -> None:
payload = self._read()
previous = payload["steps"].get(step_id)
if previous and previous["status"] == "running":
raise ValueError(f"step {step_id!r} is already running")
payload["steps"][step_id] = {
"status": "running",
"attempt": attempt,
"started_at": _now(),
"ended_at": None,
"exit_code": None,
"error": None,
}
atomic_write_json(self.path, payload)
def finish_step(
self,
step_id: str,
*,
status: StepStatus,
exit_code: int | None = None,
error: str | None = None,
) -> None:
if status not in {"success", "failed", "skipped"}:
raise ValueError(f"invalid step status: {status!r}")
payload = self._read()
step = payload["steps"].get(step_id)
if not step or step["status"] != "running":
raise ValueError(f"step {step_id!r} is not running")
step.update(
{
"status": status,
"ended_at": _now(),
"exit_code": exit_code,
"error": error,
}
)
atomic_write_json(self.path, payload)
def finalize(self, status: RunStatus, *, error: str | None = None) -> None:
if status not in {"success", "failed", "cancelled"}:
raise ValueError(f"invalid run status: {status!r}")
payload = self._read()
payload.update({"status": status, "ended_at": _now(), "error": error})
atomic_write_json(self.path, payload)
def record_input(self, reference: str) -> None:
self._record_trace("inputs", reference)
def record_output(self, reference: str) -> None:
self._record_trace("outputs", reference)
def record_external_write(self, reference: str) -> None:
self._record_trace("external_writes", reference)
def _record_trace(self, category: str, reference: str) -> None:
if (
not isinstance(reference, str)
or not reference
or len(reference) > 1024
or any(ord(character) < 32 for character in reference)
):
raise ValueError("invalid trace reference")
payload = self._read()
references = payload["trace"][category]
if reference not in references:
references.append(reference)
atomic_write_json(self.path, payload)
src/gyxx_flow/diagnostics.py
+94
View File
@@ -0,0 +1,94 @@
"""Read-only and transient preflight diagnostics for operators."""
from __future__ import annotations
import importlib.util
import sys
import tempfile
from dataclasses import asdict, dataclass
from pathlib import Path
from gyxx_flow.catalog import WorkflowCatalog
from gyxx_flow.core.config import Settings
@dataclass(frozen=True, slots=True)
class DiagnosticCheck:
name: str
passed: bool
message: str
@dataclass(frozen=True, slots=True)
class DoctorReport:
checks: tuple[DiagnosticCheck, ...]
@property
def is_healthy(self) -> bool:
return all(check.passed for check in self.checks)
def as_dict(self) -> dict[str, object]:
return {
"is_healthy": self.is_healthy,
"checks": [asdict(check) for check in self.checks],
}
def run_doctor(settings: Settings) -> DoctorReport:
checks = [
DiagnosticCheck(
"python",
sys.version_info >= (3, 12),
f"Python {sys.version_info.major}.{sys.version_info.minor}",
),
DiagnosticCheck(
"cli",
importlib.util.find_spec("gyxx_flow.__main__") is not None,
"gyxx_flow module entry point is importable",
),
DiagnosticCheck(
"project_root",
settings.project_root.is_dir(),
"project root exists" if settings.project_root.is_dir() else "project root is missing",
),
]
try:
catalog = WorkflowCatalog.load(settings.project_root / "config")
catalog_ok = len(catalog.scheduled_workflows()) == 21
catalog_message = f"catalog has {len(catalog.scheduled_workflows())} scheduled workflows"
except Exception:
catalog_ok = False
catalog_message = "catalog cannot be validated"
checks.append(DiagnosticCheck("catalog", catalog_ok, catalog_message))
data_root_ok = settings.data_root.is_absolute() and settings.data_root != Path(
settings.data_root.anchor
)
checks.append(
DiagnosticCheck(
"data_root",
data_root_ok,
"data root is an explicit non-root path" if data_root_ok else "unsafe data root",
)
)
writable, message = _probe_write_permission(settings.data_root)
checks.append(DiagnosticCheck("data_root_write", writable, message))
return DoctorReport(tuple(checks))
def _probe_write_permission(path: Path) -> tuple[bool, str]:
candidate = Path(path)
while not candidate.exists() and candidate.parent != candidate:
candidate = candidate.parent
if not candidate.is_dir():
return False, "no existing data-root parent directory"
try:
with tempfile.NamedTemporaryFile(
mode="wb", prefix=".gyxx-doctor-probe-", dir=candidate, delete=True
) as stream:
stream.write(b"probe")
stream.flush()
except OSError:
return False, "data-root parent is not writable"
return True, "data-root parent write probe passed"
src/gyxx_flow/migration/__init__.py
+33
View File
@@ -0,0 +1,33 @@
"""Safe, read-only migration boundary for legacy workflows."""
from gyxx_flow.migration.data import (
DataLayer,
HistoricalDataMigrationExecutor,
HistoricalDataMigrationPlan,
HistoricalDataMigrationPlanner,
HistoricalDataMigrationReport,
MigrationConflictError,
MigrationExecutionError,
MigrationSource,
MigrationValidationError,
)
from gyxx_flow.migration.legacy import (
LegacyCommandAdapter,
LegacyConfigurationError,
LegacyProjectRoots,
)
__all__ = [
"LegacyCommandAdapter",
"LegacyConfigurationError",
"LegacyProjectRoots",
"DataLayer",
"HistoricalDataMigrationExecutor",
"HistoricalDataMigrationPlan",
"HistoricalDataMigrationPlanner",
"HistoricalDataMigrationReport",
"MigrationConflictError",
"MigrationExecutionError",
"MigrationSource",
"MigrationValidationError",
]
src/gyxx_flow/migration/data.py
+556
View File
@@ -0,0 +1,556 @@
"""Plan and execute read-only-source historical file migrations.
The planner records an immutable source snapshot. The executor is deliberately
plan-only unless ``apply=True`` is supplied and only ever copies files into the
configured data root.
"""
from __future__ import annotations
import hashlib
import os
import re
import shutil
import uuid
from dataclasses import asdict, dataclass
from datetime import datetime, timezone
from enum import Enum
from pathlib import Path
from typing import Callable, Iterable
from gyxx_flow.core.artifacts import atomic_write_json, sha256_file
_SAFE_ID = re.compile(r"^[a-z][a-z0-9_\-]*$")
_PROGRESS_INTERVAL = 1000
class MigrationValidationError(ValueError):
"""Raised when a migration cannot be planned safely."""
class MigrationConflictError(MigrationValidationError):
"""Raised when a target exists with different content."""
class MigrationExecutionError(RuntimeError):
"""Raised when an approved migration cannot be completed or reconciled."""
class DataLayer(str, Enum):
RAW = "raw"
NORMALIZED = "normalized"
CURATED = "curated"
EXPORT = "exports"
@dataclass(frozen=True, slots=True)
class MigrationSource:
source_id: str
module: str
layer: DataLayer
root: Path
def __post_init__(self) -> None:
_validate_id(self.source_id, "source_id")
_validate_id(self.module, "module")
if not isinstance(self.layer, DataLayer):
raise MigrationValidationError("layer must be a DataLayer")
object.__setattr__(self, "root", Path(self.root).expanduser())
@dataclass(frozen=True, slots=True)
class MigrationFile:
source_id: str
source_path: str
relative_path: str
target_relative_path: str
byte_size: int
sha256: str
def to_dict(self) -> dict[str, object]:
return asdict(self)
@dataclass(frozen=True, slots=True)
class MigrationSummary:
file_count: int
total_bytes: int
aggregate_sha256: str
def to_dict(self) -> dict[str, object]:
return asdict(self)
@dataclass(frozen=True, slots=True)
class MigrationPrecheck:
status: str
required_bytes: int
free_bytes: int
existing_matching_files: int
def to_dict(self) -> dict[str, object]:
return asdict(self)
@dataclass(frozen=True, slots=True)
class HistoricalDataMigrationPlan:
plan_id: str
created_at: str
data_root: str
files: tuple[MigrationFile, ...]
source_summary: MigrationSummary
precheck: MigrationPrecheck
schema_version: int = 1
def to_dict(self) -> dict[str, object]:
return {
"schema_version": self.schema_version,
"plan_id": self.plan_id,
"created_at": self.created_at,
"data_root": self.data_root,
"source_summary": self.source_summary.to_dict(),
"precheck": self.precheck.to_dict(),
"files": [entry.to_dict() for entry in self.files],
}
@dataclass(frozen=True, slots=True)
class MigrationFileResult:
target_relative_path: str
status: str
byte_size: int
sha256: str
def to_dict(self) -> dict[str, object]:
return asdict(self)
@dataclass(frozen=True, slots=True)
class HistoricalDataMigrationReport:
plan_id: str
status: str
applied: bool
copied_count: int
skipped_count: int
source_summary: MigrationSummary
destination_summary: MigrationSummary
mismatches: tuple[str, ...]
files: tuple[MigrationFileResult, ...]
updated_at: str
schema_version: int = 1
def to_dict(self) -> dict[str, object]:
return {
"schema_version": self.schema_version,
"plan_id": self.plan_id,
"status": self.status,
"applied": self.applied,
"copied_count": self.copied_count,
"skipped_count": self.skipped_count,
"source_summary": self.source_summary.to_dict(),
"destination_summary": self.destination_summary.to_dict(),
"mismatches": list(self.mismatches),
"files": [entry.to_dict() for entry in self.files],
"updated_at": self.updated_at,
}
class HistoricalDataMigrationPlanner:
"""Build a verified, machine-readable copy plan without mutating sources."""
def __init__(
self,
*,
data_root: Path | str,
free_space_provider: Callable[[Path], int] | None = None,
) -> None:
self._data_root = Path(data_root).expanduser().resolve()
self._free_space_provider = free_space_provider or _available_bytes
def create_plan(
self,
sources: Iterable[MigrationSource],
*,
manifest_path: Path | str | None = None,
) -> HistoricalDataMigrationPlan:
entries: list[MigrationFile] = []
target_paths: set[str] = set()
for source in sources:
source_root = _resolve_source_root(source.root)
_reject_overlap(source_root, self._data_root)
for entry in _inventory_source(source, source_root):
if entry.target_relative_path in target_paths:
raise MigrationValidationError(
f"duplicate migration target: {entry.target_relative_path}"
)
target_paths.add(entry.target_relative_path)
entries.append(entry)
entries.sort(key=lambda item: item.target_relative_path)
pending_bytes, existing = self._inspect_destinations(entries)
free_bytes = self._free_space_provider(_existing_parent(self._data_root))
if free_bytes < pending_bytes:
raise MigrationValidationError(
f"insufficient disk space: need {pending_bytes} bytes, have {free_bytes}"
)
plan = HistoricalDataMigrationPlan(
plan_id=uuid.uuid4().hex,
created_at=_now(),
data_root=str(self._data_root),
files=tuple(entries),
source_summary=_summarize_expected(entries),
precheck=MigrationPrecheck(
status="ready",
required_bytes=pending_bytes,
free_bytes=free_bytes,
existing_matching_files=existing,
),
)
if manifest_path is not None:
atomic_write_json(manifest_path, plan.to_dict())
return plan
def _inspect_destinations(self, entries: list[MigrationFile]) -> tuple[int, int]:
pending_bytes = 0
existing_matching = 0
planned_targets = {
_safe_destination(self._data_root, entry.target_relative_path)
for entry in entries
}
for unplanned in _find_unplanned_targets(entries, self._data_root, planned_targets):
raise MigrationConflictError(f"unplanned target conflict: {unplanned}")
for entry in entries:
destination = _safe_destination(self._data_root, entry.target_relative_path)
if not destination.exists():
pending_bytes += entry.byte_size
continue
if not destination.is_file() or destination.is_symlink():
raise MigrationConflictError(f"target conflict: {destination}")
if destination.stat().st_size != entry.byte_size or sha256_file(destination) != entry.sha256:
raise MigrationConflictError(f"target conflict: {destination}")
existing_matching += 1
return pending_bytes, existing_matching
class HistoricalDataMigrationExecutor:
"""Execute a plan with resumable atomic copies and full reconciliation."""
def execute(
self,
plan: HistoricalDataMigrationPlan,
*,
apply: bool = False,
report_path: Path | str | None = None,
) -> HistoricalDataMigrationReport:
data_root = Path(plan.data_root).resolve()
if not apply:
report = self._build_report(plan, "plan_only", False, 0, 0, (), ())
_write_report(report_path, report)
return report
copied = 0
skipped = 0
results: list[MigrationFileResult] = []
try:
self._execution_precheck(plan, data_root)
for entry in plan.files:
source = Path(entry.source_path)
destination = _safe_destination(data_root, entry.target_relative_path)
if destination.exists():
skipped += 1
file_status = "skipped_matching"
else:
self._copy_file_atomic(
source, destination, entry.byte_size, entry.sha256
)
if destination.stat().st_size != entry.byte_size or sha256_file(destination) != entry.sha256:
raise MigrationExecutionError(
f"destination verification failed: {entry.target_relative_path}"
)
copied += 1
file_status = "copied"
results.append(
MigrationFileResult(
entry.target_relative_path,
file_status,
entry.byte_size,
entry.sha256,
)
)
if report_path is not None and len(results) % _PROGRESS_INTERVAL == 0:
progress = self._build_report(
plan, "in_progress", True, copied, skipped, tuple(results), ()
)
_write_report(report_path, progress)
except Exception as exc:
interrupted = self._build_report(
plan,
"interrupted",
True,
copied,
skipped,
tuple(results),
(str(exc),),
)
_write_report(report_path, interrupted)
if isinstance(exc, MigrationExecutionError):
raise
raise MigrationExecutionError(str(exc)) from exc
destination_summary, mismatches = _summarize_destinations(plan.files, data_root)
status = "reconciled" if not mismatches and destination_summary == plan.source_summary else "mismatch"
report = HistoricalDataMigrationReport(
plan_id=plan.plan_id,
status=status,
applied=True,
copied_count=copied,
skipped_count=skipped,
source_summary=plan.source_summary,
destination_summary=destination_summary,
mismatches=tuple(mismatches),
files=tuple(results),
updated_at=_now(),
)
_write_report(report_path, report)
if status != "reconciled":
raise MigrationExecutionError("destination reconciliation failed")
return report
def _execution_precheck(
self, plan: HistoricalDataMigrationPlan, data_root: Path
) -> None:
pending_bytes = 0
planned_targets = {
_safe_destination(data_root, entry.target_relative_path)
for entry in plan.files
}
unplanned = _find_unplanned_targets(plan.files, data_root, planned_targets)
if unplanned:
raise MigrationExecutionError(
f"unplanned target conflict: {unplanned[0]}"
)
for entry in plan.files:
source = Path(entry.source_path)
if not source.is_file() or source.is_symlink():
raise MigrationExecutionError(f"source changed or missing: {source}")
if source.stat().st_size != entry.byte_size or sha256_file(source) != entry.sha256:
raise MigrationExecutionError(f"source changed after planning: {source}")
destination = _safe_destination(data_root, entry.target_relative_path)
if destination.exists():
if (
not destination.is_file()
or destination.is_symlink()
or destination.stat().st_size != entry.byte_size
or sha256_file(destination) != entry.sha256
):
raise MigrationExecutionError(f"target conflict: {destination}")
else:
pending_bytes += entry.byte_size
free_bytes = _available_bytes(_existing_parent(data_root))
if free_bytes < pending_bytes:
raise MigrationExecutionError(
f"insufficient disk space: need {pending_bytes} bytes, have {free_bytes}"
)
@staticmethod
def _copy_file_atomic(
source: Path,
destination: Path,
expected_size: int,
expected_sha256: str,
) -> None:
destination.parent.mkdir(parents=True, exist_ok=True)
temporary = destination.with_name(f".{destination.name}.{uuid.uuid4().hex}.tmp")
try:
shutil.copy2(source, temporary)
if (
temporary.stat().st_size != expected_size
or sha256_file(temporary) != expected_sha256
):
raise MigrationExecutionError(
f"temporary copy verification failed: {destination}"
)
os.replace(temporary, destination)
finally:
temporary.unlink(missing_ok=True)
@staticmethod
def _build_report(
plan: HistoricalDataMigrationPlan,
status: str,
applied: bool,
copied: int,
skipped: int,
results: tuple[MigrationFileResult, ...],
mismatches: tuple[str, ...],
) -> HistoricalDataMigrationReport:
if applied:
summary = _summary(
(
result.target_relative_path,
result.byte_size,
result.sha256,
)
for result in results
)
discovered_mismatches = []
else:
summary = MigrationSummary(0, 0, _empty_digest())
discovered_mismatches = []
return HistoricalDataMigrationReport(
plan_id=plan.plan_id,
status=status,
applied=applied,
copied_count=copied,
skipped_count=skipped,
source_summary=plan.source_summary,
destination_summary=summary,
mismatches=tuple(mismatches or tuple(discovered_mismatches)),
files=results,
updated_at=_now(),
)
def _inventory_source(source: MigrationSource, root: Path) -> list[MigrationFile]:
entries: list[MigrationFile] = []
for path in sorted(root.rglob("*")):
if path.is_symlink():
raise MigrationValidationError(f"symbolic link is not allowed: {path}")
if not path.is_file():
continue
relative = path.relative_to(root).as_posix()
before = path.stat()
digest = sha256_file(path)
after = path.stat()
if (before.st_size, before.st_mtime_ns) != (after.st_size, after.st_mtime_ns):
raise MigrationValidationError(f"source changed while planning: {path}")
target = "/".join(
("data", source.layer.value, source.module, "legacy", source.source_id, relative)
)
entries.append(
MigrationFile(
source_id=source.source_id,
source_path=str(path),
relative_path=relative,
target_relative_path=target,
byte_size=after.st_size,
sha256=digest,
)
)
return entries
def _summarize_expected(entries: Iterable[MigrationFile]) -> MigrationSummary:
rows = [
(entry.target_relative_path, entry.byte_size, entry.sha256)
for entry in entries
]
return _summary(rows)
def _summarize_destinations(
entries: Iterable[MigrationFile], data_root: Path
) -> tuple[MigrationSummary, list[str]]:
rows: list[tuple[str, int, str]] = []
mismatches: list[str] = []
for entry in entries:
destination = _safe_destination(data_root, entry.target_relative_path)
if not destination.is_file() or destination.is_symlink():
mismatches.append(f"missing: {entry.target_relative_path}")
continue
size = destination.stat().st_size
digest = sha256_file(destination)
rows.append((entry.target_relative_path, size, digest))
if size != entry.byte_size or digest != entry.sha256:
mismatches.append(f"content mismatch: {entry.target_relative_path}")
return _summary(rows), mismatches
def _find_unplanned_targets(
entries: Iterable[MigrationFile],
data_root: Path,
planned_targets: set[Path],
) -> list[Path]:
unplanned: list[Path] = []
scopes = {
data_root.joinpath(*Path(entry.target_relative_path).parts[:5])
for entry in entries
}
for scope in sorted(scopes):
if not scope.exists():
continue
for candidate in sorted(scope.rglob("*")):
if candidate.is_symlink():
unplanned.append(candidate)
continue
if candidate.is_file() and candidate.resolve() not in planned_targets:
unplanned.append(candidate)
return unplanned
def _summary(rows: Iterable[tuple[str, int, str]]) -> MigrationSummary:
ordered = sorted(rows)
aggregate = hashlib.sha256()
total_bytes = 0
for relative, byte_size, digest in ordered:
aggregate.update(f"{relative}\0{byte_size}\0{digest}\n".encode("utf-8"))
total_bytes += byte_size
return MigrationSummary(len(ordered), total_bytes, aggregate.hexdigest())
def _empty_digest() -> str:
return hashlib.sha256().hexdigest()
def _resolve_source_root(root: Path) -> Path:
try:
resolved = root.resolve(strict=True)
except OSError as exc:
raise MigrationValidationError(f"source root cannot be resolved: {root}") from exc
if not resolved.is_dir():
raise MigrationValidationError(f"source root is not a directory: {resolved}")
return resolved
def _reject_overlap(source_root: Path, data_root: Path) -> None:
if source_root == data_root or source_root.is_relative_to(data_root) or data_root.is_relative_to(source_root):
raise MigrationValidationError(
f"source and destination overlap: {source_root} / {data_root}"
)
def _safe_destination(data_root: Path, relative: str) -> Path:
candidate = (data_root / Path(relative)).resolve()
if not candidate.is_relative_to(data_root):
raise MigrationValidationError(f"target escapes data root: {relative}")
return candidate
def _existing_parent(path: Path) -> Path:
candidate = path
while not candidate.exists():
parent = candidate.parent
if parent == candidate:
raise MigrationValidationError(f"cannot locate filesystem for {path}")
candidate = parent
return candidate
def _available_bytes(path: Path) -> int:
return shutil.disk_usage(path).free
def _validate_id(value: str, field: str) -> None:
if not isinstance(value, str) or not _SAFE_ID.fullmatch(value):
raise MigrationValidationError(f"unsafe {field}: {value!r}")
def _write_report(
report_path: Path | str | None, report: HistoricalDataMigrationReport
) -> None:
if report_path is not None:
atomic_write_json(report_path, report.to_dict())
def _now() -> str:
return datetime.now(timezone.utc).isoformat(timespec="seconds")
src/gyxx_flow/migration/legacy.py
+173
View File
@@ -0,0 +1,173 @@
"""Build legacy command steps without executing or hard-coding old projects."""
from __future__ import annotations
import os
import re
import sys
from pathlib import Path, PurePosixPath
from types import MappingProxyType
from typing import Mapping
from gyxx_flow.catalog import WorkflowEntry
from gyxx_flow.core.context import RunContext
from gyxx_flow.workflow.steps import CommandStep
DEFAULT_PROJECT_IDS = ("content", "shop", "product", "supply")
_PROJECT_ID = re.compile(r"^[a-z][a-z0-9_]*$")
class LegacyConfigurationError(ValueError):
"""Raised when a legacy project or command cannot be resolved safely."""
class LegacyProjectRoots:
"""Validated legacy roots supplied explicitly or through the environment."""
def __init__(self, roots: Mapping[str, Path | str]) -> None:
resolved: dict[str, Path] = {}
for project_id, configured_root in roots.items():
if not isinstance(project_id, str) or not _PROJECT_ID.fullmatch(project_id):
raise LegacyConfigurationError(f"invalid legacy project id: {project_id!r}")
resolved[project_id] = _resolve_project_root(project_id, configured_root)
self._roots = MappingProxyType(resolved)
@classmethod
def from_env(
cls,
*,
env: Mapping[str, str] | None = None,
project_ids: tuple[str, ...] = DEFAULT_PROJECT_IDS,
) -> "LegacyProjectRoots":
values = os.environ if env is None else env
missing: list[str] = []
roots: dict[str, str] = {}
for project_id in project_ids:
if not _PROJECT_ID.fullmatch(project_id):
raise LegacyConfigurationError(f"invalid legacy project id: {project_id!r}")
key = f"GYXX_LEGACY_{project_id.upper()}_ROOT"
value = values.get(key, "").strip()
if not value:
missing.append(key)
else:
roots[project_id] = value
if missing:
raise LegacyConfigurationError(
"missing legacy root environment variables: " + ", ".join(missing)
)
return cls(roots)
def get(self, project_id: str) -> Path:
try:
return self._roots[project_id]
except KeyError as exc:
raise LegacyConfigurationError(
f"unknown legacy project: {project_id!r}"
) from exc
def as_dict(self) -> dict[str, Path]:
return dict(self._roots)
class LegacyCommandAdapter:
"""Translate catalog entries to inert ``CommandStep`` definitions."""
def __init__(
self,
roots: LegacyProjectRoots,
*,
base_env: Mapping[str, str] | None = None,
python_executable: Path | str | None = None,
) -> None:
self._roots = roots
self._base_env = dict(os.environ if base_env is None else base_env)
configured_python = sys.executable if python_executable is None else python_executable
self._python_executable = str(configured_python)
_validate_text(self._python_executable, field="python_executable")
def build(self, entry: WorkflowEntry, *, context: RunContext) -> CommandStep:
if not entry.source_project:
raise LegacyConfigurationError(
f"workflow has no source-project provenance: {entry.workflow_id}"
)
root = self._roots.get(entry.source_project)
if entry.workflow_id != context.workflow_id:
raise LegacyConfigurationError(
f"workflow entry {entry.workflow_id!r} does not match "
f"run context {context.workflow_id!r}"
)
if entry.trigger == "unavailable":
raise LegacyConfigurationError(
f"legacy workflow is unavailable: {entry.workflow_id}"
)
executable = _resolve_entry(root, entry.entry)
arguments = tuple(entry.args)
for argument in arguments:
_validate_text(argument, field="legacy argument")
argv = (
(self._python_executable, str(executable), *arguments)
if executable.suffix.casefold() == ".py"
else (str(executable), *arguments)
)
env = dict(self._base_env)
env.update(
{
"GYXX_WORKFLOW_ID": context.workflow_id,
"GYXX_RUN_ID": context.run_id,
"GYXX_BUSINESS_DATE": context.business_date.isoformat(),
"GYXX_SHADOW": str(context.shadow).lower(),
"GYXX_LEGACY_PROJECT_ROOT": str(root),
}
)
try:
return CommandStep(argv=argv, cwd=root, env=env)
except ValueError as exc:
raise LegacyConfigurationError(str(exc)) from exc
def _resolve_project_root(project_id: str, configured_root: Path | str) -> Path:
try:
raw_root = Path(configured_root).expanduser()
if not raw_root.exists():
raise LegacyConfigurationError(
f"legacy project root does not exist for {project_id!r}: {raw_root}"
)
root = raw_root.resolve(strict=True)
except (OSError, TypeError, ValueError) as exc:
if isinstance(exc, LegacyConfigurationError):
raise
raise LegacyConfigurationError(
f"cannot resolve legacy project root for {project_id!r}: {exc}"
) from exc
if not root.is_dir():
raise LegacyConfigurationError(
f"legacy project root is not a directory for {project_id!r}: {root}"
)
return root
def _resolve_entry(root: Path, configured_entry: str) -> Path:
if not isinstance(configured_entry, str) or not configured_entry:
raise LegacyConfigurationError("legacy entry must be a non-empty relative path")
if "\\" in configured_entry or ":" in configured_entry or "\x00" in configured_entry:
raise LegacyConfigurationError("legacy entry must use a safe relative path")
relative = PurePosixPath(configured_entry)
if relative.is_absolute() or ".." in relative.parts:
raise LegacyConfigurationError("legacy entry cannot escape its project root")
try:
candidate = (root / Path(*relative.parts)).resolve(strict=True)
except OSError as exc:
raise LegacyConfigurationError(
f"legacy entry does not exist or cannot be resolved: {configured_entry}"
) from exc
if not candidate.is_relative_to(root):
raise LegacyConfigurationError("legacy entry cannot escape its project root")
if not candidate.is_file():
raise LegacyConfigurationError(f"legacy entry is not a file: {configured_entry}")
return candidate
def _validate_text(value: object, *, field: str) -> None:
if not isinstance(value, str) or not value or "\x00" in value:
raise LegacyConfigurationError(f"{field} must be a non-empty safe string")
src/gyxx_flow/migration/shadow.py
+489
View File
@@ -0,0 +1,489 @@
"""Privacy-conscious comparisons for shadow workflow outputs."""
from __future__ import annotations
import copy
import csv
import hashlib
import hmac
import json
import math
from dataclasses import dataclass, field
from datetime import datetime, timezone
from decimal import Decimal, InvalidOperation
from pathlib import Path
from types import MappingProxyType
from typing import Any, Mapping, Sequence
from gyxx_flow.core.artifacts import atomic_write_json
_DEFAULT_HASH_SECRET = b"gyxx-flow-public-comparison-v1"
_STRUCTURED_SUFFIXES = {
".csv": "csv",
".json": "json",
".jsonl": "jsonl",
".ndjson": "jsonl",
}
@dataclass(frozen=True, slots=True)
class MetricTolerance:
"""Allowed absolute and relative drift for one numeric metric."""
absolute: float = 0.0
relative: float = 0.0
def __post_init__(self) -> None:
_validate_tolerance(self.absolute, "absolute")
_validate_tolerance(self.relative, "relative")
@dataclass(frozen=True, slots=True)
class ComparisonSpec:
"""Declarative schema for comparing two structured result sets."""
primary_key: tuple[str, ...]
metrics: Mapping[str, MetricTolerance] = field(default_factory=dict)
error_fields: tuple[str, ...] = ()
safe_fields: tuple[str, ...] = ()
difference_limit: int = 1000
hash_secret: str | bytes | None = field(default=None, repr=False, compare=False)
def __post_init__(self) -> None:
primary_key = _validate_fields(self.primary_key, "primary_key", required=True)
error_fields = _validate_fields(self.error_fields, "error_fields")
safe_fields = _validate_fields(self.safe_fields, "safe_fields")
if not isinstance(self.difference_limit, int) or isinstance(
self.difference_limit, bool
) or self.difference_limit < 1:
raise ValueError("difference_limit must be a positive integer")
if not isinstance(self.metrics, Mapping):
raise ValueError("metrics must be a mapping")
metrics: dict[str, MetricTolerance] = {}
for name, tolerance in self.metrics.items():
_validate_field_name(name, "metric")
if not isinstance(tolerance, MetricTolerance):
raise ValueError(f"metric {name!r} must declare MetricTolerance")
metrics[name] = tolerance
if self.hash_secret is not None and not isinstance(
self.hash_secret, (str, bytes)
):
raise ValueError("hash_secret must be text or bytes")
if self.hash_secret == "" or self.hash_secret == b"":
raise ValueError("hash_secret cannot be empty")
object.__setattr__(self, "primary_key", primary_key)
object.__setattr__(self, "error_fields", error_fields)
object.__setattr__(self, "safe_fields", safe_fields)
object.__setattr__(self, "metrics", MappingProxyType(metrics))
def hash_key(self) -> bytes:
if self.hash_secret is None:
return _DEFAULT_HASH_SECRET
if isinstance(self.hash_secret, str):
return self.hash_secret.encode("utf-8")
return self.hash_secret
@dataclass(frozen=True, slots=True)
class ComparisonReport:
"""Immutable facade over a JSON-serializable comparison report."""
_payload: Mapping[str, Any] = field(repr=False)
def to_dict(self) -> dict[str, Any]:
return copy.deepcopy(dict(self._payload))
def compare_structured_files(
baseline_path: Path | str,
candidate_path: Path | str,
*,
spec: ComparisonSpec,
generated_at: datetime | None = None,
) -> ComparisonReport:
"""Compare JSON, JSONL, or CSV results without emitting source rows."""
if not isinstance(spec, ComparisonSpec):
raise TypeError("spec must be ComparisonSpec")
baseline_format, baseline_rows = _load_records(Path(baseline_path))
candidate_format, candidate_rows = _load_records(Path(candidate_path))
baseline = _index_rows(baseline_rows, spec.primary_key, side="baseline")
candidate = _index_rows(candidate_rows, spec.primary_key, side="candidate")
baseline_keys = set(baseline)
candidate_keys = set(candidate)
common_keys = baseline_keys & candidate_keys
missing_keys = baseline_keys - candidate_keys
extra_keys = candidate_keys - baseline_keys
secret = spec.hash_key()
metric_reports = _compare_metrics(
baseline, candidate, common_keys, spec=spec, secret=secret
)
baseline_errors = _collect_errors(baseline_rows, spec.error_fields)
candidate_errors = _collect_errors(candidate_rows, spec.error_fields)
error_report = _compare_errors(
baseline_errors,
candidate_errors,
secret=secret,
limit=spec.difference_limit,
)
metric_mismatches = sum(
metric["mismatch_count"] for metric in metric_reports.values()
)
error_differences = (
error_report["only_baseline_total"] + error_report["only_candidate_total"]
)
has_differences = bool(
missing_keys or extra_keys or metric_mismatches or error_differences
)
timestamp = generated_at or datetime.now(timezone.utc)
if timestamp.tzinfo is None or timestamp.utcoffset() is None:
raise ValueError("generated_at must be timezone-aware")
missing_identifiers = _sample_identifiers(
missing_keys, baseline, spec=spec, secret=secret
)
extra_identifiers = _sample_identifiers(
extra_keys, candidate, spec=spec, secret=secret
)
payload: dict[str, Any] = {
"schema_version": 1,
"status": "mismatch" if has_differences else "match",
"generated_at": timestamp.isoformat(timespec="seconds"),
"sources": {
"baseline_format": baseline_format,
"candidate_format": candidate_format,
},
"spec": {
"primary_key": list(spec.primary_key),
"metrics": {
name: {
"absolute_tolerance": tolerance.absolute,
"relative_tolerance": tolerance.relative,
}
for name, tolerance in spec.metrics.items()
},
"error_fields": list(spec.error_fields),
"safe_fields": list(spec.safe_fields),
"difference_limit": spec.difference_limit,
},
"summary": {
"baseline_rows": len(baseline_rows),
"candidate_rows": len(candidate_rows),
"common_keys": len(common_keys),
"missing_keys": len(missing_keys),
"extra_keys": len(extra_keys),
"metric_mismatches": metric_mismatches,
"error_set_differences": error_differences,
},
"keys": {
"missing_total": len(missing_keys),
"missing_truncated": len(missing_keys) > spec.difference_limit,
"missing": missing_identifiers,
"extra_total": len(extra_keys),
"extra_truncated": len(extra_keys) > spec.difference_limit,
"extra": extra_identifiers,
},
"metrics": metric_reports,
"errors": error_report,
}
return ComparisonReport(payload)
def write_comparison_report(
path: Path | str, report: ComparisonReport
) -> Path:
"""Atomically persist a machine-readable comparison report."""
if not isinstance(report, ComparisonReport):
raise TypeError("report must be ComparisonReport")
return atomic_write_json(path, report.to_dict())
def _load_records(path: Path) -> tuple[str, list[dict[str, Any]]]:
data_format = _STRUCTURED_SUFFIXES.get(path.suffix.casefold())
if data_format is None:
raise ValueError(f"unsupported structured data format: {path.suffix or '<none>'}")
if data_format == "csv":
return data_format, _load_csv(path)
if data_format == "jsonl":
return data_format, _load_jsonl(path)
return data_format, _load_json(path)
def _load_csv(path: Path) -> list[dict[str, Any]]:
with path.open("r", encoding="utf-8-sig", newline="") as handle:
reader = csv.DictReader(handle)
rows: list[dict[str, Any]] = []
for line_number, row in enumerate(reader, start=2):
if None in row:
raise ValueError(f"CSV record has extra columns at line {line_number}")
rows.append(dict(row))
return rows
def _load_jsonl(path: Path) -> list[dict[str, Any]]:
rows: list[dict[str, Any]] = []
with path.open("r", encoding="utf-8-sig") as handle:
for line_number, line in enumerate(handle, start=1):
if not line.strip():
continue
try:
row = json.loads(line)
except json.JSONDecodeError as exc:
raise ValueError(f"invalid JSONL at line {line_number}") from exc
rows.append(_require_record(row, location=f"JSONL line {line_number}"))
return rows
def _load_json(path: Path) -> list[dict[str, Any]]:
try:
payload = json.loads(path.read_text(encoding="utf-8-sig"))
except json.JSONDecodeError as exc:
raise ValueError("invalid JSON document") from exc
if isinstance(payload, dict) and set(payload) == {"records"}:
payload = payload["records"]
if not isinstance(payload, list):
raise ValueError("JSON document must be an array or a records envelope")
return [
_require_record(row, location=f"JSON record {index}")
for index, row in enumerate(payload, start=1)
]
def _require_record(value: object, *, location: str) -> dict[str, Any]:
if not isinstance(value, dict) or any(not isinstance(key, str) for key in value):
raise ValueError(f"record must be a JSON object at {location}")
return dict(value)
def _index_rows(
rows: Sequence[Mapping[str, Any]], primary_key: tuple[str, ...], *, side: str
) -> dict[tuple[str, ...], Mapping[str, Any]]:
indexed: dict[tuple[str, ...], Mapping[str, Any]] = {}
for row in rows:
key = _primary_key(row, primary_key, side=side)
if key in indexed:
raise ValueError(f"duplicate primary key in {side} data")
indexed[key] = row
return indexed
def _primary_key(
row: Mapping[str, Any], fields: tuple[str, ...], *, side: str
) -> tuple[str, ...]:
values: list[str] = []
for field_name in fields:
if field_name not in row or row[field_name] is None or row[field_name] == "":
raise ValueError(f"missing primary key field {field_name!r} in {side} data")
value = row[field_name]
if isinstance(value, (dict, list, tuple, set)):
raise ValueError(f"primary key field {field_name!r} must be scalar")
if isinstance(value, bool):
values.append("true" if value else "false")
else:
values.append(str(value))
return tuple(values)
def _compare_metrics(
baseline: Mapping[tuple[str, ...], Mapping[str, Any]],
candidate: Mapping[tuple[str, ...], Mapping[str, Any]],
common_keys: set[tuple[str, ...]],
*,
spec: ComparisonSpec,
secret: bytes,
) -> dict[str, dict[str, Any]]:
reports: dict[str, dict[str, Any]] = {}
for field_name, tolerance in spec.metrics.items():
mismatches: list[tuple[str, ...]] = []
max_absolute = Decimal(0)
max_relative = Decimal(0)
absolute_tolerance = Decimal(str(tolerance.absolute))
relative_tolerance = Decimal(str(tolerance.relative))
for key in common_keys:
baseline_value = _numeric_value(baseline[key], field_name, "baseline")
candidate_value = _numeric_value(candidate[key], field_name, "candidate")
absolute_delta = abs(candidate_value - baseline_value)
scale = max(abs(candidate_value), abs(baseline_value))
relative_delta = absolute_delta / scale if scale else Decimal(0)
max_absolute = max(max_absolute, absolute_delta)
max_relative = max(max_relative, relative_delta)
allowed_delta = max(absolute_tolerance, relative_tolerance * scale)
if absolute_delta > allowed_delta:
mismatches.append(key)
reports[field_name] = {
"compared": len(common_keys),
"mismatch_count": len(mismatches),
"max_absolute_delta": float(max_absolute),
"max_relative_delta": float(max_relative),
"mismatches_truncated": len(mismatches) > spec.difference_limit,
"mismatches": _sample_identifiers(
mismatches, baseline, spec=spec, secret=secret
),
}
return reports
def _numeric_value(
row: Mapping[str, Any], field_name: str, side: str
) -> Decimal:
if field_name not in row:
raise ValueError(f"missing metric field {field_name!r} in {side} record")
value = row[field_name]
if isinstance(value, bool) or value is None:
raise ValueError(f"metric field {field_name!r} must be finite numeric")
try:
numeric = Decimal(str(value).strip())
except (InvalidOperation, ValueError) as exc:
raise ValueError(f"metric field {field_name!r} must be finite numeric") from exc
if not numeric.is_finite():
raise ValueError(f"metric field {field_name!r} must be finite numeric")
try:
machine_value = float(numeric)
except (OverflowError, ValueError) as exc:
raise ValueError(f"metric field {field_name!r} exceeds report range") from exc
if not math.isfinite(machine_value):
raise ValueError(f"metric field {field_name!r} exceeds report range")
return numeric
def _sample_identifiers(
keys: Sequence[tuple[str, ...]] | set[tuple[str, ...]],
rows: Mapping[tuple[str, ...], Mapping[str, Any]],
*,
spec: ComparisonSpec,
secret: bytes,
) -> list[dict[str, Any]]:
identifiers = [
_identifier(key, rows[key], spec=spec, secret=secret) for key in keys
]
identifiers.sort(key=lambda item: item["key_hash"])
return identifiers[: spec.difference_limit]
def _identifier(
key: tuple[str, ...],
row: Mapping[str, Any],
*,
spec: ComparisonSpec,
secret: bytes,
) -> dict[str, Any]:
canonical = json.dumps(key, ensure_ascii=False, separators=(",", ":"))
identifier: dict[str, Any] = {
"key_hash": _secure_hash(canonical, secret=secret, domain="primary-key")
}
if spec.safe_fields:
safe_values = {
name: _safe_value(row[name]) for name in spec.safe_fields if name in row
}
if safe_values:
identifier["safe_fields"] = safe_values
return identifier
def _safe_value(value: Any) -> Any:
if value is None or isinstance(value, (str, int, float, bool)):
if isinstance(value, float) and not math.isfinite(value):
raise ValueError("safe field must contain a finite JSON scalar")
return value
raise ValueError("safe field must contain a JSON scalar")
def _collect_errors(
rows: Sequence[Mapping[str, Any]], fields: tuple[str, ...]
) -> set[str]:
errors: set[str] = set()
for row in rows:
for field_name in fields:
if field_name in row:
errors.update(_error_tokens(row[field_name]))
return errors
def _error_tokens(value: Any) -> set[str]:
if value is None or value == "":
return set()
if isinstance(value, str):
stripped = value.strip()
if not stripped:
return set()
if stripped[:1] in {"[", "{"}:
try:
return _error_tokens(json.loads(stripped))
except json.JSONDecodeError:
pass
return {_canonical_json(stripped)}
if isinstance(value, (list, tuple, set)):
tokens: set[str] = set()
for item in value:
tokens.update(_error_tokens(item))
return tokens
return {_canonical_json(value)}
def _compare_errors(
baseline: set[str], candidate: set[str], *, secret: bytes, limit: int
) -> dict[str, Any]:
only_baseline = sorted(
_secure_hash(value, secret=secret, domain="error")
for value in baseline - candidate
)
only_candidate = sorted(
_secure_hash(value, secret=secret, domain="error")
for value in candidate - baseline
)
return {
"baseline_unique_count": len(baseline),
"candidate_unique_count": len(candidate),
"only_baseline_total": len(only_baseline),
"only_baseline_truncated": len(only_baseline) > limit,
"only_baseline_hashes": only_baseline[:limit],
"only_candidate_total": len(only_candidate),
"only_candidate_truncated": len(only_candidate) > limit,
"only_candidate_hashes": only_candidate[:limit],
}
def _secure_hash(value: str, *, secret: bytes, domain: str) -> str:
message = f"gyxx-shadow-{domain}-v1\0{value}".encode("utf-8")
return hmac.new(secret, message, hashlib.sha256).hexdigest()
def _canonical_json(value: Any) -> str:
try:
return json.dumps(
value, ensure_ascii=False, sort_keys=True, separators=(",", ":")
)
except (TypeError, ValueError) as exc:
raise ValueError("error fields must contain JSON-compatible values") from exc
def _validate_fields(
values: object, label: str, *, required: bool = False
) -> tuple[str, ...]:
if not isinstance(values, tuple):
raise ValueError(f"{label} must be a tuple of field names")
if required and not values:
raise ValueError(f"{label} must contain at least one field")
for value in values:
_validate_field_name(value, label)
if len(set(values)) != len(values):
raise ValueError(f"{label} cannot contain duplicate fields")
return values
def _validate_field_name(value: object, label: str) -> None:
if not isinstance(value, str) or not value.strip() or "\x00" in value:
raise ValueError(f"{label} contains an invalid field name")
def _validate_tolerance(value: object, label: str) -> None:
if (
isinstance(value, bool)
or not isinstance(value, (int, float))
or not math.isfinite(value)
or value < 0
):
raise ValueError(f"{label} tolerance must be a finite non-negative number")
src/gyxx_flow/modules/__init__.py
+77
View File
@@ -0,0 +1,77 @@
"""Business-module contracts, registry, and default composition root."""
from __future__ import annotations
from typing import Any
from .content_marketing import ContentMarketingModule
from .contracts import BusinessModule
from .product_commerce import ProductCommerceModule
from .registry import ModuleRegistry
from .shop_intelligence import ShopIntelligenceModule
from .supply_chain import SupplyChainModule
EXPECTED_MODULE_IDS = (
"content_marketing",
"product_commerce",
"shop_intelligence",
"supply_chain",
)
def create_default_registry(
*,
catalog: Any | None = None,
shop_command_factory: Any | None = None,
supply_command_factory: Any | None = None,
content_command_factory: Any | None = None,
product_command_factory: Any | None = None,
) -> ModuleRegistry:
"""Compose built-in modules, registering only explicitly supplied migrations."""
shop_module = (
ShopIntelligenceModule()
if catalog is None
else ShopIntelligenceModule.from_catalog(
catalog,
command_factory=shop_command_factory,
)
)
supply_module = (
SupplyChainModule()
if catalog is None
else SupplyChainModule.from_catalog(
catalog, command_factory=supply_command_factory
)
)
content_module = (
ContentMarketingModule()
if catalog is None
else ContentMarketingModule.from_catalog(
catalog, command_factory=content_command_factory
)
)
product_module = (
ProductCommerceModule()
if catalog is None
else ProductCommerceModule.from_catalog(
catalog, command_factory=product_command_factory
)
)
return ModuleRegistry(
(
content_module,
product_module,
shop_module,
supply_module,
)
)
__all__ = [
"BusinessModule",
"EXPECTED_MODULE_IDS",
"ModuleRegistry",
"create_default_registry",
]
src/gyxx_flow/modules/content_marketing/__init__.py
+103
View File
@@ -0,0 +1,103 @@
"""Content metrics, creator, comment, report, and login workflows."""
from __future__ import annotations
from collections.abc import Iterable
from typing import Any
from gyxx_flow.adapters.native import DeferredModuleCommandStep, ModuleCommandFactory
from gyxx_flow.workflow import StepDefinition, WorkflowDefinition
CONTENT_SCHEDULED_WORKFLOW_IDS = (
"content.metrics.daily",
"content.marketing_report.daily",
"content.relogin.weekly",
"content.self_operated.weekly",
"content.creator_report.monthly",
"content.summary.monthly",
"content.cooperations.daily",
"content.comments.weekly",
"content.summary.weekly",
)
CONTENT_MANUAL_WORKFLOW_IDS = (
"content.mapping.refresh",
"content.retry_failed",
"content.metrics.backfill",
)
CONTENT_WORKFLOW_IDS = (*CONTENT_SCHEDULED_WORKFLOW_IDS, *CONTENT_MANUAL_WORKFLOW_IDS)
CONTENT_TIMEOUT_SECONDS = 4 * 60 * 60
CONTENT_RESOURCE = "module:content_marketing"
_RELOGIN_WORKFLOW_ID = "content.relogin.weekly"
class ContentMarketingModule:
"""Content-marketing business module composition boundary."""
module_id = "content_marketing"
def __init__(self, definitions: Iterable[WorkflowDefinition] = ()) -> None:
self._definitions = tuple(definitions)
@classmethod
def from_catalog(
cls,
catalog: Any,
*,
command_factory: ModuleCommandFactory | None = None,
) -> "ContentMarketingModule":
"""Build the scheduled content workflows from validated catalog metadata.
Each launcher resolves from source owned by this installed module.
Registration remains inert and dry-run never starts a subprocess.
"""
entries = {
entry.workflow_id: entry
for entry in catalog.workflows
if entry.module == cls.module_id and entry.trigger in {"scheduled", "manual"}
}
expected = set(CONTENT_WORKFLOW_IDS)
if set(entries) != expected:
raise ValueError(
"content catalog workflows do not match the migration contract: "
f"expected={sorted(expected)}, actual={sorted(entries)}"
)
definitions: list[WorkflowDefinition] = []
for workflow_id in CONTENT_WORKFLOW_IDS:
entry = entries[workflow_id]
action = (
DeferredModuleCommandStep(entry)
if command_factory is None
else DeferredModuleCommandStep(entry, command_factory=command_factory)
)
definitions.append(
WorkflowDefinition(
workflow_id,
(
StepDefinition(
"module_run",
action,
timeout_seconds=CONTENT_TIMEOUT_SECONDS,
max_attempts=1,
resources=(CONTENT_RESOURCE,),
production_sink=True,
official_notification=workflow_id
== _RELOGIN_WORKFLOW_ID,
),
),
)
)
return cls(definitions)
def workflow_definitions(self) -> tuple[WorkflowDefinition, ...]:
return self._definitions
__all__ = [
"CONTENT_WORKFLOW_IDS",
"CONTENT_MANUAL_WORKFLOW_IDS",
"CONTENT_SCHEDULED_WORKFLOW_IDS",
"CONTENT_RESOURCE",
"CONTENT_TIMEOUT_SECONDS",
"ContentMarketingModule",
]
src/gyxx_flow/modules/content_marketing/runtime/bilibili_comment_scraper.py
+500
View File
@@ -0,0 +1,500 @@
import argparse
import csv
import json
import re
import sys
import time
from datetime import datetime
from pathlib import Path
from gyxx_flow.modules.content_marketing.runtime.runtime_paths import PATHS
from typing import Any
from urllib.parse import parse_qs, urlencode, urlsplit, urlunsplit
from playwright.sync_api import Page
from scrapling.fetchers import DynamicSession
BASE_DIR = PATHS.module_root
DATA_DIR = PATHS.raw_root
STATE_DIR = PATHS.state_root
NOTES_DIR = PATHS.raw_root / "notes" / "bilibili"
COOKIE_FILE = PATHS.browser_cookie_file
PROFILE_DIR = PATHS.browser_profile_dir
BV_RE = re.compile(r"BV[0-9A-Za-z]{10}")
AV_RE = re.compile(r"(?:av|aid=)(\d+)", re.IGNORECASE)
def log(message: str) -> None:
print(f"[{datetime.now().strftime('%H:%M:%S')}] {message}", flush=True)
def load_cookies() -> list[dict[str, Any]]:
if not COOKIE_FILE.exists():
return []
try:
data = json.loads(COOKIE_FILE.read_text(encoding="utf-8"))
return data if isinstance(data, list) else []
except json.JSONDecodeError:
log(f"Cookie file is invalid, ignoring: {COOKIE_FILE}")
return []
def restore_cookies(page: Page) -> None:
cookies = load_cookies()
if not cookies:
return
try:
page.context.add_cookies(cookies)
log(f"Loaded cookies: {COOKIE_FILE}")
except Exception as exc:
log(f"Failed to load cookies, will continue with browser profile: {exc}")
def save_state(page: Page) -> None:
STATE_DIR.mkdir(parents=True, exist_ok=True)
COOKIE_FILE.write_text(
json.dumps(page.context.cookies(), ensure_ascii=False, indent=2),
encoding="utf-8",
)
PATHS.browser_storage_state_file.write_text(
json.dumps(page.context.storage_state(), ensure_ascii=False, indent=2),
encoding="utf-8",
)
log(f"Saved cookies: {COOKIE_FILE}")
def is_logged_in(page: Page) -> bool:
"""B 站登录态靠 SESSDATA cookie,有它才能调评论 API 拿全量。
没有 SESSDATA 时 API 只返回热门 3-5 条。
"""
for c in page.context.cookies():
if c.get("name") == "SESSDATA" and c.get("value"):
return True
return False
def maybe_wait_for_login(page: Page, login_timeout: int) -> None:
"""检测登录态,未登录就等用户扫码。B 站登录页 URL 含 passport.bilibili.com/login。"""
if is_logged_in(page):
return
log("B 站未登录 (无 SESSDATA cookie)。请在弹出的浏览器窗口扫码登录。")
# 跳到登录页,确保用户能看到二维码
try:
page.goto("https://passport.bilibili.com/login", wait_until="domcontentloaded", timeout=30000)
except Exception as exc:
log(f" 跳登录页失败 (继续等当前页扫码): {exc}")
deadline = time.time() + login_timeout
while time.time() < deadline:
if is_logged_in(page):
log("登录成功,SESSDATA 已写入")
save_state(page)
return
page.wait_for_timeout(2000)
log(f" [WARN] 等待 {login_timeout}s 仍未登录,继续以游客态采集 (评论只会拿到热门 3-5 条)")
def safe_filename(value: str) -> str:
value = re.sub(r"[<>:\"/\\\\|?*\\s]+", "_", value).strip("_")
return value[:90] or "bilibili_comments"
def extract_bvid(url: str) -> str:
match = BV_RE.search(url)
return match.group(0) if match else ""
def extract_aid(url: str) -> str:
match = AV_RE.search(url)
return match.group(1) if match else ""
def ts_to_iso(value: Any) -> str:
if not isinstance(value, (int, float)) or not value:
return ""
try:
return datetime.fromtimestamp(value).isoformat(timespec="seconds")
except Exception:
return str(value)
def with_query_params(url: str, updates: dict[str, Any]) -> str:
parts = urlsplit(url)
params = dict((key, values[-1]) for key, values in parse_qs(parts.query, keep_blank_values=True).items())
for key, value in updates.items():
params[key] = str(value)
return urlunsplit((parts.scheme, parts.netloc, parts.path, urlencode(params), parts.fragment))
def parse_reply(reply: dict[str, Any], source_url: str, level: str, parent_rpid: str = "") -> dict[str, Any]:
member = reply.get("member") or {}
content = reply.get("content") or {}
return {
"source_url": source_url,
"level": level,
"rpid": str(reply.get("rpid") or ""),
"parent_rpid": str(parent_rpid or reply.get("parent") or ""),
"root_rpid": str(reply.get("root") or ""),
"mid": str(member.get("mid") or reply.get("mid") or ""),
"nickname": member.get("uname") or "",
"sex": member.get("sex") or "",
"message": content.get("message") or "",
"like_count": reply.get("like") or 0,
"reply_count": reply.get("rcount") or 0,
"location": reply.get("reply_control", {}).get("location") or "",
"created_at": ts_to_iso(reply.get("ctime")),
"raw_ctime": reply.get("ctime") or "",
}
def merge_reply(row: dict[str, Any], comments_by_id: dict[str, dict[str, Any]]) -> None:
key = row.get("rpid") or f"{row.get('level')}:{len(comments_by_id)}"
comments_by_id[str(key)] = row
def collect_main_payload(payload: dict[str, Any], source_url: str, comments_by_id: dict[str, dict[str, Any]]) -> None:
data = payload.get("data") if isinstance(payload.get("data"), dict) else payload
replies = data.get("replies") or []
if not isinstance(replies, list):
return
for reply in replies:
if not isinstance(reply, dict):
continue
row = parse_reply(reply, source_url, "comment")
merge_reply(row, comments_by_id)
child_replies = reply.get("replies") or []
if isinstance(child_replies, list):
for child in child_replies:
if isinstance(child, dict):
child_row = parse_reply(child, source_url, "reply", row.get("rpid") or "")
merge_reply(child_row, comments_by_id)
def collect_reply_payload(payload: dict[str, Any], source_url: str, parent_rpid: str, comments_by_id: dict[str, dict[str, Any]]) -> None:
data = payload.get("data") if isinstance(payload.get("data"), dict) else payload
replies = data.get("replies") or []
if not isinstance(replies, list):
return
for reply in replies:
if isinstance(reply, dict):
row = parse_reply(reply, source_url, "reply", parent_rpid)
merge_reply(row, comments_by_id)
def api_get(page: Page, url: str) -> dict[str, Any]:
return page.evaluate(
"""
async (url) => {
const res = await fetch(url, {
credentials: 'include',
headers: { 'accept': 'application/json, text/plain, */*' }
});
return await res.json();
}
""",
url,
)
def resolve_video_info(page: Page, url: str) -> dict[str, Any]:
bvid = extract_bvid(url)
aid = extract_aid(url)
if not bvid and not aid:
page.goto(url, wait_until="domcontentloaded", timeout=90000)
page.wait_for_timeout(3000)
bvid = extract_bvid(page.url) or extract_bvid(page.locator("body").inner_text(timeout=5000))
aid = extract_aid(page.url)
params = {}
if bvid:
params["bvid"] = bvid
elif aid:
params["aid"] = aid
else:
raise RuntimeError("Could not find BV or AV id from URL.")
api_url = "https://api.bilibili.com/x/web-interface/view?" + urlencode(params)
payload = api_get(page, api_url)
if payload.get("code") != 0:
raise RuntimeError(f"Failed to resolve video info: {payload}")
data = payload.get("data") or {}
stat = data.get("stat") or {}
return {
"aid": str(data.get("aid") or aid),
"bvid": data.get("bvid") or bvid,
"title": data.get("title") or "",
"owner_mid": str((data.get("owner") or {}).get("mid") or ""),
"owner_name": (data.get("owner") or {}).get("name") or "",
"note_metrics": {
"like_count": stat.get("like", ""),
"favorite_count": stat.get("favorite", ""),
"share_count": stat.get("share", ""),
"comment_count": stat.get("reply", ""),
},
}
def fetch_main_comments(
page: Page,
source_url: str,
oid: str,
comments_by_id: dict[str, dict[str, Any]],
max_pages: int,
delay: float,
) -> dict[str, Any]:
stats = {"api_hits": 0, "api_total": None, "last_page": 0}
page_num = 1
page_size = 20
while page_num <= max_pages:
api_url = (
"https://api.bilibili.com/x/v2/reply?"
+ urlencode({"type": 1, "oid": oid, "pn": page_num, "ps": page_size, "sort": 2})
)
payload = api_get(page, api_url)
stats["api_hits"] += 1
if payload.get("code") != 0:
log(f"Main comment API returned code={payload.get('code')}: {payload.get('message')}")
break
data = payload.get("data") or {}
page_info = data.get("page") or {}
stats["api_total"] = page_info.get("count", stats.get("api_total"))
stats["last_page"] = page_num
before = len(comments_by_id)
collect_main_payload(payload, source_url, comments_by_id)
loaded = len(comments_by_id) - before
log(f"Main page {page_num}: +{loaded}, total {len(comments_by_id)}")
replies = data.get("replies") or []
if not replies:
break
if page_info.get("count") and len([r for r in comments_by_id.values() if r.get("level") == "comment"]) >= int(page_info["count"]):
break
page_num += 1
if delay:
page.wait_for_timeout(int(delay * 1000))
return stats
def fetch_sub_replies(
page: Page,
source_url: str,
oid: str,
comments_by_id: dict[str, dict[str, Any]],
delay: float,
max_reply_pages: int,
) -> int:
parents = [
row for row in list(comments_by_id.values())
if row.get("level") == "comment" and int(row.get("reply_count") or 0) > 0
]
if not parents:
return 0
log(f"Fetching sub replies for {len(parents)} comments.")
merged = 0
for parent in parents:
root = parent.get("rpid")
if not root:
continue
pn = 1
ps = 20
while pn <= max_reply_pages:
api_url = (
"https://api.bilibili.com/x/v2/reply/reply?"
+ urlencode({"type": 1, "oid": oid, "root": root, "pn": pn, "ps": ps})
)
payload = api_get(page, api_url)
if payload.get("code") != 0:
break
data = payload.get("data") or {}
before = len(comments_by_id)
collect_reply_payload(payload, source_url, str(root), comments_by_id)
merged += max(0, len(comments_by_id) - before)
replies = data.get("replies") or []
page_info = data.get("page") or {}
if not replies:
break
if page_info.get("count") and pn * ps >= int(page_info["count"]):
break
pn += 1
if delay:
page.wait_for_timeout(int(delay * 1000))
log(f"Sub replies merged: {merged}; total {len(comments_by_id)}")
return merged
def scrape_comments(
url: str,
max_pages: int,
delay: float,
headless: bool,
max_reply_pages: int = 200,
login_timeout: int = 0,
) -> tuple[list[dict[str, Any]], dict[str, Any]]:
result: dict[str, Any] = {"comments": [], "stats": {}, "video": {}}
def action(page: Page) -> None:
page.goto(url, wait_until="domcontentloaded", timeout=90000)
page.wait_for_timeout(2500)
save_state(page)
# 登录检测:未登录时评论 API 只返回热门 3-5 条
if login_timeout > 0:
maybe_wait_for_login(page, login_timeout)
# 登录后回到视频页(可能在登录页)
if not page.url.startswith("https://www.bilibili.com/video/"):
page.goto(url, wait_until="domcontentloaded", timeout=90000)
page.wait_for_timeout(2000)
video = resolve_video_info(page, url)
oid = video["aid"]
comments_by_id: dict[str, dict[str, Any]] = {}
stats = fetch_main_comments(page, url, oid, comments_by_id, max_pages=max_pages, delay=delay)
sub_count = fetch_sub_replies(page, url, oid, comments_by_id, delay=delay, max_reply_pages=max_reply_pages)
save_state(page)
stats["sub_replies_merged"] = sub_count
stats["max_reply_pages"] = max_reply_pages
stats["top_level_count"] = len([row for row in comments_by_id.values() if row.get("level") == "comment"])
stats["reply_count"] = len([row for row in comments_by_id.values() if row.get("level") == "reply"])
stats["note_metrics"] = video.get("note_metrics", {})
stats["logged_in"] = is_logged_in(page)
result["comments"] = list(comments_by_id.values())
result["stats"] = stats
result["video"] = video
result["final_url"] = page.url
PROFILE_DIR.mkdir(parents=True, exist_ok=True)
with DynamicSession(
headless=headless,
real_chrome=True,
user_data_dir=str(PROFILE_DIR),
locale="zh-CN",
timezone_id="Asia/Shanghai",
timeout=90000,
network_idle=False,
disable_resources=False,
google_search=False,
page_setup=restore_cookies,
max_pages=1,
) as session:
session.fetch(url, page_action=action, wait=1000)
return result["comments"], result
def write_outputs(url: str, comments: list[dict[str, Any]], result: dict[str, Any]) -> tuple[Path, Path]:
NOTES_DIR.mkdir(parents=True, exist_ok=True)
bvid = result.get("video", {}).get("bvid") or extract_bvid(url) or safe_filename(url)
json_path = NOTES_DIR / f"bilibili_comments_{bvid}.json"
csv_path = NOTES_DIR / f"bilibili_comments_{bvid}.csv"
if not comments and json_path.exists() and csv_path.exists():
log(f"No comments scraped; keeping existing output files: {json_path}, {csv_path}")
return json_path, csv_path
payload = {
"source_url": url,
"title": result.get("video", {}).get("title", ""),
"final_url": result.get("final_url", ""),
"video": result.get("video", {}),
"scraped_at": datetime.now().isoformat(timespec="seconds"),
"count": len(comments),
"stats": result.get("stats", {}),
"comments": comments,
}
json_path.write_text(json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8")
fieldnames = [
"source_url",
"level",
"rpid",
"parent_rpid",
"root_rpid",
"mid",
"nickname",
"sex",
"message",
"like_count",
"reply_count",
"location",
"created_at",
"raw_ctime",
]
with csv_path.open("w", newline="", encoding="utf-8-sig") as file:
writer = csv.DictWriter(file, fieldnames=fieldnames)
writer.writeheader()
writer.writerows(comments)
return json_path, csv_path
def login_only(login_timeout: int) -> int:
"""只触发登录(headed),登录完成立刻退出。给 relogin_bilibili.py 用。"""
def action(page: Page) -> None:
page.goto("https://passport.bilibili.com/login", wait_until="domcontentloaded", timeout=30000)
page.wait_for_timeout(1500)
maybe_wait_for_login(page, login_timeout)
PROFILE_DIR.mkdir(parents=True, exist_ok=True)
with DynamicSession(
headless=False,
real_chrome=True,
user_data_dir=str(PROFILE_DIR),
locale="zh-CN",
timezone_id="Asia/Shanghai",
timeout=90000,
network_idle=False,
disable_resources=False,
google_search=False,
page_setup=restore_cookies,
max_pages=1,
) as session:
session.fetch("https://passport.bilibili.com/login", page_action=action, wait=1000)
log("[login-only] 完成")
return 0
def main() -> int:
parser = argparse.ArgumentParser(description="Scrape Bilibili video comments with a real browser and API.")
parser.add_argument("url", help="Bilibili video URL (传 dummy 'login' 配合 --login-only 也可)")
parser.add_argument("--max-reply-pages", type=int, default=200, help="max sub-reply pages per top-level comment")
parser.add_argument("--max-pages", type=int, default=200, help="最多抓取一级评论页数")
parser.add_argument("--delay", type=float, default=0.25, help="接口请求间隔秒数")
parser.add_argument("--headless", action="store_true", help="无头模式")
parser.add_argument("--login-timeout", type=int, default=0,
help="未登录时等待扫码秒数(>0 启用登录检测;默认 0 沿用游客态)")
parser.add_argument("--login-only", action="store_true",
help="只触发登录(headed)然后退出,给 relogin_bilibili 用")
args = parser.parse_args()
if args.login_only:
return login_only(args.login_timeout if args.login_timeout > 0 else 300)
comments, result = scrape_comments(
args.url, args.max_pages, args.delay, args.headless,
args.max_reply_pages, args.login_timeout,
)
json_path, csv_path = write_outputs(args.url, comments, result)
log(f"Final URL: {result.get('final_url', '')}")
log(f"Video: {result.get('video', {})}")
log(f"Stats: {result.get('stats', {})}")
log(f"Scraped {len(comments)} comments")
log(f"JSON: {json_path}")
log(f"CSV: {csv_path}")
return 0
if __name__ == "__main__":
try:
raise SystemExit(main())
except KeyboardInterrupt:
print("\nInterrupted.", file=sys.stderr)
raise SystemExit(130)
src/gyxx_flow/modules/content_marketing/runtime/bilibili_scraper.py
+854
View File
@@ -0,0 +1,854 @@
"""
B 站抓取脚本: 多维表格驱动,按"发布时间到今天的天数差"自动选 7/14/21/28/月槽位
核心规则:
- 槽位由 (今天 - 发布时间) 的天数差决定:
0-7 天 → 7天曝光量
8-14 天 → 14天曝光量
15-21 天 → 21天曝光量
22-28 天 → 28天曝光量
29+ 天 → 月底曝光量
- 同槽覆盖语义: 同一槽多次跑会更新为最新播放量
- 抓取: B 站公开 API (api.bilibili.com/x/web-interface/view) 拿 stat.view
不用登录,不用 cookie
- 字段 ID 全部从 款式_多维表格_对照.json 的 field_map 动态读取
- 平台枚举: B站 / b站 / B 站 都接受 (大小写无关)
用法:
# 跑全 24 款式 (默认)
python bilibili_scraper.py
# 试跑不写
python bilibili_scraper.py --dry-run
# 只跑某个款式
python bilibili_scraper.py --style 1
# 只跑某几条
python bilibili_scraper.py --record recXXXXX
# 模拟任意日期 (调试用)
python bilibili_scraper.py --today 2025-08-15 --dry-run
# 防风控:每条请求间隔 0.5s
python bilibili_scraper.py --delay 0.5
# 首次回填: 所有 B 站记录全部填 7天曝光量 (忽略发布时间)
python bilibili_scraper.py --first-run
# 重置运行状态
python bilibili_scraper.py --reset-state
"""
import argparse
import json
import os
import re
import subprocess
import sys
import time
from datetime import datetime, date
from pathlib import Path
from gyxx_flow.modules.content_marketing.runtime.runtime_paths import (
PATHS,
resolve_layer_output,
)
import requests
from gyxx_flow.modules.content_marketing.runtime.collection_completeness import (
add_repeatable_style_argument,
atomic_write_json,
merge_global_summaries,
select_requested_styles,
)
BASE_DIR = PATHS.module_root
DEFAULT_DATA_DIR = PATHS.normalized_root
V2_DIR = PATHS.normalized_root / "v2_results"
BILIBILI_CHECKPOINT_DIR = PATHS.state_root / "checkpoints" / "bilibili"
MAPPING_PATH = PATHS.normalized_root / "mappings" / "款式_多维表格_对照.json"
STATE_FILENAME = "weekly_run_state.json"
LARK_CLI = os.path.expandvars(r"%APPDATA%\npm\lark-cli.cmd")
if not os.path.exists(LARK_CLI):
# Scheduled tasks run as SYSTEM; %APPDATA% is not Administrator's dir
LARK_CLI = "lark-cli.cmd"
# 字段 ID 全部从 data/config/款式_多维表格_对照.json 的 field_map 动态读
# 下面是 logical key(各款式表结构一致)
KEY_PLATFORM = "platform" # 投放平台 (select)
KEY_URL = "note_url" # 发布笔记链接 (text)
KEY_TITLE = "note_title" # 发布笔记标题 (text)
KEY_CREATOR = "creator_name" # 达人名称 (text)
KEY_CREATOR_ID = "creator_id" # 达人id (text,选填)
KEY_PUBTIME = "publish_time" # 发布时间 (datetime)
KEY_MONTH = "month" # 月份 (select)
KEY_PARENT = "parent_record" # 父记录 (link,自关联)
KEY_WEEK = "publish_week" # 周发布 (select)
# 5 档曝光量槽位: 7/14/21/28 天 + 月底(第 5 周用) - logical name -> 天数
SLOTS = [
("read_count_7d", 7),
("read_count_14d", 14),
("read_count_21d", 21),
("read_count_28d", 28),
("month_end", "月底"),
]
# 月份枚举(选项里已有)
MONTH_OPTIONS = ["6月", "7月", "8月", "9月", "10月"]
# B 站 API 相关
_UA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36")
_BV_RE = re.compile(r"BV[0-9A-Za-z]{10}")
_AV_RE = re.compile(r"av(\d+)", re.IGNORECASE)
_B23_RE = re.compile(r"https?://b23\.tv/\S+", re.IGNORECASE)
_MD_LINK_RE = re.compile(r"\[([^\]]+)\]\(([^)]+)\)")
_URL_RE = re.compile(r"https?://[^\s)>\]]+")
def log(msg: str) -> None:
safe = str(msg).encode("gbk", errors="replace").decode("gbk", errors="replace")
print(f"[{datetime.now().strftime('%H:%M:%S')}] {safe}", flush=True)
def finalize_bili_summary(summary: dict, *, dry_run: bool = False) -> dict:
"""Attach one terminal state per Bilibili record and exact completeness counters."""
out = dict(summary)
unique: list[dict] = []
positions: dict[str, int] = {}
duplicates = 0
for raw in out.get("details") or []:
if not isinstance(raw, dict):
continue
row = dict(raw)
rid = str(row.get("record_id") or "")
if rid and rid in positions:
unique[positions[rid]] = row
duplicates += 1
else:
if rid:
positions[rid] = len(unique)
unique.append(row)
total = int(out.get("total_b_records") or 0)
success = sum(row.get("status") == "success" for row in unique)
blocked = sum(row.get("status") == "blocked_input" for row in unique)
retryable = sum(row.get("status") == "retryable_failure" for row in unique)
write_failures = sum(row.get("status") == "write_failure" for row in unique)
missing = max(0, total - len(unique))
unresolved = retryable + write_failures + missing
top_level_error = bool(out.get("error"))
if top_level_error:
retryable = max(retryable, 1)
unresolved = max(unresolved, 1)
out.update({
"details": unique,
"updated": success,
"skipped": blocked + retryable + write_failures,
"success": success,
"blocked_input": blocked,
"retryable_failures": retryable,
"write_failures": write_failures,
"missing_results": missing,
"duplicate_results": duplicates,
"unresolved": unresolved,
"complete": (
not top_level_error
and unresolved == 0
and duplicates == 0
and len(unique) == total
),
"dry_run": bool(dry_run),
})
return out
def merge_bili_summary(existing: dict | None, partial: dict, *, dry_run: bool = False) -> dict:
"""Merge a --record retry without deleting previously successful rows."""
if not existing:
return finalize_bili_summary(partial, dry_run=dry_run)
combined = dict(existing)
for key, value in partial.items():
if key != "details":
combined[key] = value
rows: list[dict] = []
positions: dict[str, int] = {}
for source in (existing.get("details") or [], partial.get("details") or []):
for raw in source:
if not isinstance(raw, dict):
continue
row = dict(raw)
rid = str(row.get("record_id") or "")
if rid and rid in positions:
rows[positions[rid]] = row
else:
if rid:
positions[rid] = len(rows)
rows.append(row)
combined["details"] = rows
combined["total_b_records"] = max(
int(existing.get("total_b_records") or 0),
int(partial.get("total_b_records") or 0),
len(rows),
)
return finalize_bili_summary(combined, dry_run=dry_run)
# ============================================================
# 飞书多维表格
# ============================================================
def call_lark_json(args: list[str]) -> dict:
cmd = [LARK_CLI] + args
env = {**os.environ, "LARK_CLI_NO_PROXY": "1"}
proc = subprocess.run(
cmd, cwd=PATHS.tmp_root, capture_output=True, env=env,
encoding="utf-8", errors="replace",
)
text = proc.stdout
start = text.find("{")
if start < 0:
return {"ok": False, "error": f"no json: {text[:300]}"}
try:
return json.loads(text[start:], strict=False)
except json.JSONDecodeError as e:
return {"ok": False, "error": f"json decode: {e}"}
def load_mapping(data_dir: Path, self_operated: bool = False) -> dict:
# 动态:从「合作达人」/「自营达人」多维表格地址表读取款式→各表地址,实时拉字段重建对照表
from gyxx_flow.modules.content_marketing.runtime import feishu_mapping
return feishu_mapping.load_mapping(None, self_operated=self_operated)
def list_all_records(base_token: str, table_id: str) -> list[dict]:
"""拉一张表所有记录,自动分页"""
all_records = []
offset = 0
while True:
resp = call_lark_json([
"base", "+record-list",
"--base-token", base_token,
"--table-id", table_id,
"--as", "user",
"--format", "json",
"--limit", "200",
"--offset", str(offset),
])
if not resp.get("ok"):
raise RuntimeError(f"record-list 失败: {resp.get('error')}")
d = resp["data"]
ids = d["field_id_list"]
rids = d["record_id_list"]
rows = d["data"]
for i, row in enumerate(rows):
rec = {"record_id": rids[i]}
for j, fid in enumerate(ids):
rec[fid] = row[j] if j < len(row) else None
all_records.append(rec)
offset += len(rows)
if not d.get("has_more", False) or len(rows) == 0:
break
return all_records
def _write_payload_to_cwd(payload: dict, name: str = "_bili_payload.json") -> str:
"""把 payload 写到当前工作目录,返回相对路径名(lark-cli --json @file 必须是 cwd 相对路径名,不是绝对路径)"""
payload_file = PATHS.tmp_root / name
payload_file.parent.mkdir(parents=True, exist_ok=True)
payload_file.write_text(json.dumps(payload, ensure_ascii=False), encoding="utf-8")
return name # 只返回文件名,而不是绝对路径
def write_record(base_token: str, table_id: str, record_id: str,
fields: dict, dry_run: bool) -> bool:
"""更新一条已有记录"""
if dry_run:
log(f" [DRY-RUN] 写入: {json.dumps(fields, ensure_ascii=False)[:200]}")
return True
pf = _write_payload_to_cwd(fields)
cmd = [LARK_CLI, "base", "+record-upsert",
"--base-token", base_token,
"--table-id", table_id,
"--record-id", record_id,
"--as", "user",
"--format", "json",
"--json", f"@{pf}"]
env = {**os.environ, "LARK_CLI_NO_PROXY": "1"}
proc = subprocess.run(
cmd, cwd=PATHS.tmp_root, capture_output=True, env=env,
encoding="utf-8", errors="replace",
)
text = proc.stdout
if proc.stderr:
# lark-cli 在 stderr 也会输出,真实响应可能在 stderr
if "ok" in proc.stderr and (text == "" or not text.lstrip().startswith("{")):
text = proc.stderr
start = text.find("{")
if start < 0:
log(f" [ERROR] 写记录失败: no json. stdout={proc.stdout[:200]} stderr={proc.stderr[:200]}")
return False
try:
resp = json.loads(text[start:], strict=False)
except json.JSONDecodeError as e:
log(f" [ERROR] 写记录失败: json decode {e}. text={text[:200]}")
return False
if not resp.get("ok"):
log(f" [ERROR] 写记录失败: {resp.get('error')}")
return False
return True
def create_record(base_token: str, table_id: str,
fields: dict, dry_run: bool) -> str | None:
"""创建新记录,返回 record_id"""
if dry_run:
log(f" [DRY-RUN] 新建: {json.dumps(fields, ensure_ascii=False)[:200]}")
return "rec_dryrun"
pf = _write_payload_to_cwd(fields)
cmd = [LARK_CLI, "base", "+record-upsert",
"--base-token", base_token,
"--table-id", table_id,
"--as", "user",
"--format", "json",
"--json", f"@{pf}"]
env = {**os.environ, "LARK_CLI_NO_PROXY": "1"}
proc = subprocess.run(
cmd, cwd=PATHS.tmp_root, capture_output=True, env=env,
encoding="utf-8", errors="replace",
)
text = proc.stdout
if proc.stderr:
if "ok" in proc.stderr and (text == "" or not text.lstrip().startswith("{")):
text = proc.stderr
start = text.find("{")
if start < 0:
log(f" [ERROR] 创建记录失败: no json. stderr={proc.stderr[:200]}")
return None
try:
resp = json.loads(text[start:], strict=False)
except json.JSONDecodeError as e:
log(f" [ERROR] 创建记录失败: json decode {e}. text={text[:200]}")
return None
if not resp.get("ok"):
log(f" [ERROR] 创建记录失败: {resp.get('error')}")
return None
rids = resp.get("data", {}).get("record", {}).get("record_id_list", [])
return rids[0] if rids else None
# ============================================================
# B 站播放量抓取
# ============================================================
def extract_url(text) -> str | None:
"""从 note_url 字段里抽真实 URL,处理 Markdown / 纯文本 / 短链"""
if not text:
return None
if isinstance(text, list):
text = text[0] if text else None
if not text:
return None
s = str(text).strip()
m = _MD_LINK_RE.search(s)
if m:
return m.group(2).strip()
m = _URL_RE.search(s)
if m:
return m.group(0).strip()
return None
def resolve_b23(url: str, session: requests.Session) -> str | None:
try:
r = session.get(url, allow_redirects=True, timeout=10, headers={"User-Agent": _UA})
return r.url
except Exception as exc:
log(f" [WARN] b23 解析失败: {exc}")
return None
def fetch_play_count(url: str, session: requests.Session) -> int | None:
real = url
if _B23_RE.match(url):
resolved = resolve_b23(url, session)
if not resolved:
return None
real = resolved
bv = _BV_RE.search(real)
av = _AV_RE.search(real)
if not bv and not av:
return None
params = {"bvid": bv.group(0)} if bv else {"aid": av.group(1)}
try:
r = session.get("https://api.bilibili.com/x/web-interface/view",
params=params, timeout=10,
headers={"User-Agent": _UA, "Referer": "https://www.bilibili.com/"})
j = r.json()
if j.get("code") != 0:
return None
v = j.get("data", {}).get("stat", {}).get("view")
return int(v) if v is not None else None
except Exception as exc:
log(f" [WARN] B 站 API: {exc}")
return None
# ============================================================
# 记录关系 + 槽位推进
# ============================================================
def get_parent_id(rec: dict, fid: str) -> str | None:
if not fid:
return None
p = rec.get(fid)
if isinstance(p, list) and p:
item = p[0]
if isinstance(item, dict):
return item.get("id")
if isinstance(item, str):
return item
return None
def get_month(rec: dict, fid: str) -> str | None:
if not fid:
return None
m = rec.get(fid)
if isinstance(m, list) and m:
return m[0]
if isinstance(m, str):
return m
return None
def month_to_num(m: str) -> int:
m = re.match(r"(\d+)月", m or "")
return int(m.group(1)) if m else 0
def num_to_month(n: int) -> str:
return f"{n}月"
def get_week_label(rec: dict, fid: str) -> str | None:
"""从记录的"周发布"字段提取文本(e.g. '7月第二周')"""
if not fid:
return None
w = rec.get(fid)
if isinstance(w, list) and w:
return w[0]
if isinstance(w, str):
return w
return None
def compute_week_slot(today: date) -> tuple[str, str, int | str]:
"""
根据今天日期,返回 (week_label, slot_logical_name, slot_key):
- 1-7 号 → "X月第一周" → 7天
- 8-14 号 → "X月第二周" → 14天
- 15-21 号 → "X月第三周" → 21天
- 22-28 号 → "X月第四周" → 28天
- 29-31 号 → "X月第五周" → 月底曝光量
注: 返回的是 logical name (如 'read_count_7d'),需要从 fmap 查 field_id
"""
day = today.day
month = today.month
month_label = num_to_month(month)
if day <= 7:
return f"{month_label}第一周", SLOTS[0][0], SLOTS[0][1]
elif day <= 14:
return f"{month_label}第二周", SLOTS[1][0], SLOTS[1][1]
elif day <= 21:
return f"{month_label}第三周", SLOTS[2][0], SLOTS[2][1]
elif day <= 28:
return f"{month_label}第四周", SLOTS[3][0], SLOTS[3][1]
else:
return f"{month_label}第五周", SLOTS[4][0], SLOTS[4][1]
def next_slot_in_record(rec: dict, slot_fids: list[str]) -> tuple[str | None, int | str | None]:
"""
返回 (slot_fid, slot_key) — 该记录下一个该填的槽
判定: None / 空字符串 / 0 都视为"未填"
5 槽都填了返回 (None, None)
slot_fids: 5 档槽位的实际 field_id 列表(顺序对应 SLOTS)
"""
for fid, days in zip(slot_fids, [s[1] for s in SLOTS]):
v = rec.get(fid)
if v is None:
return fid, days
if isinstance(v, str) and v.strip() == "":
return fid, days
try:
if float(v) == 0:
return fid, days
except (TypeError, ValueError):
return fid, days
return None, None
def get_slot_fid(fmap: dict, logical_name: str) -> str | None:
"""从 fmap 查某个 logical 槽位(如 'read_count_7d')的实际 field_id"""
return fmap.get(logical_name, {}).get("field_id")
def build_chains(records: list[dict], fid_parent: str, fid_month: str, fid_pubtime: str) -> dict[str, list[dict]]:
"""
把记录组织成"链": key=链根 rid(顶层父记录 rid,即父字段为空的)
value=按月份升序的子记录列表(含根本身)
"""
by_id = {r["record_id"]: r for r in records}
roots: dict[str, list[dict]] = {}
for r in records:
cur = r
while True:
pid = get_parent_id(cur, fid=fid_parent)
if not pid:
root_id = cur["record_id"]
break
if pid not in by_id:
root_id = cur["record_id"] # 父不在表内,自己当根
break
cur = by_id[pid]
roots.setdefault(root_id, []).append(r)
for k in roots:
roots[k].sort(key=lambda x: (
month_to_num(get_month(x, fid=fid_month) or "0月"),
x.get(fid_pubtime) if fid_pubtime else 0
))
return roots
# ============================================================
# 状态持久化(运行历史,可重置)
# ============================================================
def load_state(data_dir: Path) -> dict:
del data_dir
p = BILIBILI_CHECKPOINT_DIR / STATE_FILENAME
if not p.exists():
return {}
try:
return json.loads(p.read_text(encoding="utf-8"))
except Exception:
return {}
def save_state(data_dir: Path, state: dict) -> None:
del data_dir
p = BILIBILI_CHECKPOINT_DIR / STATE_FILENAME
p.parent.mkdir(parents=True, exist_ok=True)
p.parent.mkdir(parents=True, exist_ok=True)
atomic_write_json(p, state)
# ============================================================
# 单款式处理
# ============================================================
def process_style(style: dict, only_record_ids: set[str] | None,
dry_run: bool, delay: float, session: requests.Session,
state: dict, force_today: date | None,
first_run: bool = False) -> dict:
base_token = style["base_token"]
table_id = style["table_id"]
fmap = style.get("field_map", {})
# 从本款 field_map 解析出实际字段 ID
fid_platform = fmap.get(KEY_PLATFORM, {}).get("field_id", "")
fid_url = fmap.get(KEY_URL, {}).get("field_id", "")
fid_creator = fmap.get(KEY_CREATOR, {}).get("field_id", "")
fid_pubtime = fmap.get(KEY_PUBTIME, {}).get("field_id", "")
fid_month = fmap.get(KEY_MONTH, {}).get("field_id", "")
fid_parent = fmap.get(KEY_PARENT, {}).get("field_id", "")
fid_week = fmap.get(KEY_WEEK, {}).get("field_id", "")
# 5 档槽位的 field_id 列表
slot_fids = [get_slot_fid(fmap, ln) or "" for ln, _ in SLOTS]
log(f"\n========== [{style['index']:02d}] {style['name']} (B 站) ==========")
log(f" 字段映射: platform={fid_platform} url={fid_url} creator={fid_creator or '(无)'} pubtime={fid_pubtime or '(无)'}")
missing_mapping: list[str] = []
if not fid_platform:
missing_mapping.append("platform")
if not fid_url:
missing_mapping.append("note_url")
if first_run:
if not slot_fids[0]:
missing_mapping.append("read_count_7d")
else:
if not fid_pubtime:
missing_mapping.append("publish_time")
missing_mapping.extend(
logical
for field_id, (logical, _label) in zip(slot_fids, SLOTS)
if not field_id
)
if missing_mapping:
error = f"missing field mapping: {','.join(missing_mapping)}"
log(f" [ERROR] {error},跳过该款式")
return {"style": style["name"], "index": style["index"],
"total_b_records": 0, "updated": 0, "skipped": 0, "details": [],
"run_started_at": time.time(),
"error": error, "unresolved": 1,
"retryable_failures": 1, "complete": False}
all_records = list_all_records(base_token, table_id)
# 只看 B 站记录 (大小写无关)
def is_bili_platform(v) -> bool:
if not v:
return False
s = str(v).lower()
return "b站" in s or "b 站" in s
b_records = [r for r in all_records
if r.get(fid_platform) and is_bili_platform(r.get(fid_platform))]
log(f" 全表 {len(all_records)} 条,B 站源记录 {len(b_records)} 条")
if only_record_ids:
b_records = [r for r in b_records if r["record_id"] in only_record_ids]
log(f" --record 过滤后,B 站源记录剩 {len(b_records)} 条")
# 业务触发条件:只有填了发布笔记链接的记录才进入采集清单。
# 无链接通常表示尚未发布，不是 blocked_input，也不进入完整度分母。
source_b_records = len(b_records)
b_records = [r for r in b_records if extract_url(r.get(fid_url))]
skipped_no_url = source_b_records - len(b_records)
log(f" 发布链接过滤: 待采 {len(b_records)} 条,未触发 {skipped_no_url} 条")
today = force_today or date.today()
style_state = state.setdefault(style["name"], {})
summary = {
"style": style["name"], "index": style["index"],
"run_started_at": time.time(),
"source_b_records": source_b_records,
"skipped_no_url": skipped_no_url,
"total_b_records": len(b_records), "updated": 0, "skipped": 0,
"details": [],
}
for rec in b_records:
rid = rec["record_id"]
creator = rec.get(fid_creator) or "?"
pubtime_raw = rec.get(fid_pubtime)
url = extract_url(rec.get(fid_url))
if not url: # 防御性保护；正常情况下已在待采清单生成前过滤。
continue
# 解析发布时间
pub_date = parse_pub_date(pubtime_raw)
if pub_date is None and not first_run:
log(f" [SKIP] {rid[:10]}.. {creator} 发布时间为空,无法判断槽位")
summary["skipped"] += 1
summary["details"].append({
"record_id": rid, "creator": creator, "url": url,
"status": "blocked_input", "matched": False,
"reason": "publish_time_missing", "ok": False,
})
continue
# 算槽位: (今天 - 发布日) 天数差
if first_run:
target_slot_fid, target_slot_key = slot_fids[0] or None, SLOTS[0][1] if slot_fids[0] else None
else:
days_elapsed = (today - pub_date).days
target_slot_fid, target_slot_key = pick_slot_by_days(days_elapsed, slot_fids)
if target_slot_fid is None:
if first_run:
log(f" [SKIP] {rid[:10]}.. {creator} 7天曝光量字段缺失")
else:
log(f" [SKIP] {rid[:10]}.. {creator} 对应槽位字段缺失或发布时间 {pub_date} 在 {today} 之后")
summary["skipped"] += 1
summary["details"].append({
"record_id": rid, "creator": creator, "url": url,
"publish_date": str(pub_date) if pub_date else None,
"status": "blocked_input", "matched": False,
"reason": "slot_missing_or_future_publish_time", "ok": False,
})
continue
# 抓播放量
play = fetch_play_count(url, session)
if play is None:
log(f" [SKIP] {rid[:10]}.. {creator} 抓不到播放量")
summary["skipped"] += 1
summary["details"].append({
"record_id": rid, "creator": creator, "url": url,
"publish_date": str(pub_date) if pub_date else None,
"status": "retryable_failure", "matched": False,
"reason": "play_count_fetch_failed", "ok": False,
})
continue
# 覆盖写入 (同槽覆盖语义,数字更新)
action = "first_run_fill" if first_run else "update_slot"
log(f" {rid[:10]}.. {creator} 发布={pub_date} 距今={(today - pub_date).days if pub_date else '-'}天"
f" → 写 {target_slot_key}天曝光量={play}")
ok = write_record(base_token, table_id, rid,
{target_slot_fid: play}, dry_run)
if ok:
summary["updated"] += 1
style_state[rid] = {"last_run": str(today), "slot": target_slot_key}
summary["details"].append({
"record_id": rid, "creator": creator,
"publish_date": str(pub_date) if pub_date else None,
"days_elapsed": (today - pub_date).days if pub_date else None,
"slot": target_slot_key, "play_count": play,
"action": action, "ok": ok,
"matched": True,
"write_ok": ok if not dry_run else None,
"status": "success" if ok else "write_failure",
**({} if ok else {"reason": "write_back_failed"}),
})
if delay > 0:
time.sleep(delay)
state[style["name"]] = style_state
return finalize_bili_summary(summary, dry_run=dry_run)
# 槽位:天数差 → (slot_fid, slot_key)
# 0-7天→7天, 8-14→14天, 15-21→21天, 22-28→28天, 29+→月底
def pick_slot_by_days(days: int, slot_fids: list[str]) -> tuple[str | None, int | str | None]:
if days < 0:
return None, None
if days <= 7:
fid = slot_fids[0]
elif days <= 14:
fid = slot_fids[1]
elif days <= 21:
fid = slot_fids[2]
elif days <= 28:
fid = slot_fids[3]
else:
fid = slot_fids[4]
if not fid:
return None, None
return fid, SLOTS[slot_fids.index(fid)][1]
def parse_pub_date(v) -> date | None:
"""发布时间字段可能是毫秒时间戳 (int) 或 'YYYY-MM-DD HH:MM:SS' 字符串"""
if v is None:
return None
if isinstance(v, (int, float)):
# 毫秒时间戳
if v > 1e12:
v = v / 1000
try:
return datetime.fromtimestamp(v).date()
except Exception:
return None
if isinstance(v, str):
s = v.strip()
if not s:
return None
# 试常见格式
for fmt in ("%Y-%m-%d %H:%M:%S", "%Y-%m-%d", "%Y/%m/%d", "%Y/%m/%d %H:%M:%S"):
try:
return datetime.strptime(s, fmt).date()
except ValueError:
continue
return None
# ============================================================
# main
# ============================================================
def main() -> int:
parser = argparse.ArgumentParser(
description="B 站抓取+周更: 自动推进 7/14/21/28/月曝光量槽 + 跨月开新子记录")
add_repeatable_style_argument(parser, "只跑指定款式编号(可多次)")
parser.add_argument("--record", action="append", help="只跑指定 record_id(可多次)")
parser.add_argument("--dry-run", action="store_true", help="只抓不写")
parser.add_argument("--today", help="强制 today 日期,格式 YYYY-MM-DD (调试用)")
parser.add_argument("--delay", type=float, default=0.3,
help="每条请求间隔秒数(防风控,默认 0.3s)")
parser.add_argument("--data-dir", default=str(DEFAULT_DATA_DIR),
help=f"数据目录(默认 {DEFAULT_DATA_DIR})")
parser.add_argument("--reset-state", action="store_true", help="重置运行状态")
parser.add_argument("--first-run", action="store_true",
help="首次回填: 所有 B 站记录全部填 7天曝光量,跳过周次逻辑")
parser.add_argument("--self-operated", action="store_true",
help="抓取自营达人表格(而非合作达人)")
args = parser.parse_args()
data_dir = resolve_layer_output(
args.data_dir, layer_root=PATHS.normalized_root, field="--data-dir"
)
mapping = load_mapping(data_dir, self_operated=args.self_operated)
styles = mapping["tables"]
if args.style:
try:
styles = select_requested_styles(styles, args.style)
except ValueError as exc:
print(str(exc)); return 1
only_rids = set(args.record) if args.record else None
force_today = None
if args.today:
force_today = date.fromisoformat(args.today)
state_path = BILIBILI_CHECKPOINT_DIR / STATE_FILENAME
if args.reset_state and state_path.exists():
state_path.unlink()
log("状态已重置")
state = load_state(data_dir)
session = requests.Session()
session.headers.update({"User-Agent": _UA})
out_dir = V2_DIR if data_dir == PATHS.normalized_root.resolve() else data_dir / "v2_results"
out_dir.mkdir(parents=True, exist_ok=True)
all_summaries = []
self_sfx = "_self" if args.self_operated else ""
final = out_dir / f"_all_summaries{self_sfx}_bilibili.json"
for s in styles:
summary = process_style(s, only_rids, args.dry_run, args.delay,
session, state, force_today, args.first_run)
# 传了 --record 但本款式没匹配到 → 跳过 (不落空盘,不计入总汇总)
if only_rids and summary["total_b_records"] == 0 and summary["updated"] == 0 \
and summary["skipped"] == 0:
continue
# 每款式落盘
out_file = out_dir / f"{s['index']:02d}-{s['name']}{self_sfx}_bilibili_v2.json"
if only_rids and out_file.exists():
try:
previous = json.loads(out_file.read_text(encoding="utf-8"))
except Exception:
previous = None
summary = merge_bili_summary(previous, summary, dry_run=args.dry_run)
else:
summary = finalize_bili_summary(summary, dry_run=args.dry_run)
all_summaries.append(summary)
atomic_write_json(out_file, summary)
log(f" [{s['index']:02d} {s['name']}] B站记录={summary['total_b_records']} "
f"更新={summary['updated']} 跳过={summary['skipped']} "
f"未解决={summary.get('unresolved', 0)} 落盘: {out_file.name}")
if not args.dry_run:
save_state(data_dir, state)
log(f" 状态已存: {state_path}")
# 总汇总按款式合并，单款/局部补跑不能删除其他款式。
try:
existing_global = json.loads(final.read_text(encoding="utf-8")) if final.exists() else []
except Exception:
existing_global = []
atomic_write_json(final, merge_global_summaries(existing_global, all_summaries))
log(f"\n=== 全部完成 ===")
log(f"汇总: {final}")
incomplete = [s.get("index") for s in all_summaries if not s.get("complete")]
if incomplete:
log(f"[ERROR] 以下款式仍有未解决采集/写回结果: {incomplete}")
return 2
return 0
if __name__ == "__main__":
try:
raise SystemExit(main())
except KeyboardInterrupt:
print("\nInterrupted.", file=sys.stderr)
raise SystemExit(130)
src/gyxx_flow/modules/content_marketing/runtime/chanmama_scraper.py
+1293
View File
File diff suppressed because it is too large Load Diff
src/gyxx_flow/modules/content_marketing/runtime/collection_completeness.py
+583
View File
@@ -0,0 +1,583 @@
"""Shared completeness, matching, and atomic-result helpers for V2 scrapers."""
from __future__ import annotations
import json
import os
import re
import unicodedata
from difflib import SequenceMatcher
from pathlib import Path
from typing import Any, Iterable
from urllib.parse import parse_qs, urlparse
SUCCESS = "success"
BLOCKED_INPUT = "blocked_input"
RETRYABLE_FAILURE = "retryable_failure"
WRITE_FAILURE = "write_failure"
TERMINAL_STATUSES = {SUCCESS, BLOCKED_INPUT, RETRYABLE_FAILURE, WRITE_FAILURE}
def add_repeatable_style_argument(parser: Any, help_text: str) -> Any:
"""Add the shared repeatable --style option used by every platform CLI."""
return parser.add_argument(
"--style", type=int, action="append", help=help_text,
)
def select_requested_styles(styles: Iterable[dict], requested: Iterable[int] | None) -> list[dict]:
"""Select every requested style and fail when any requested index is unknown."""
style_list = list(styles)
if not requested:
return style_list
requested_set = {int(index) for index in requested}
selected = [style for style in style_list if int(style.get("index", -1)) in requested_set]
found = {int(style.get("index", -1)) for style in selected}
missing = sorted(requested_set - found)
if missing:
raise ValueError(f"未找到款式: {','.join(str(index) for index in missing)}")
return selected
def atomic_write_json(path: str | Path, value: Any) -> None:
"""Write JSON through a sibling temporary file and atomically replace target."""
target = Path(path)
target.parent.mkdir(parents=True, exist_ok=True)
tmp = target.with_suffix(target.suffix + ".tmp")
with open(tmp, "w", encoding="utf-8") as handle:
json.dump(value, handle, ensure_ascii=False, indent=2)
handle.flush()
os.fsync(handle.fileno())
os.replace(tmp, target)
def coerce_url(value: Any) -> str:
"""Extract a URL from common Feishu URL-field shapes."""
if value is None:
return ""
if isinstance(value, str):
text = value.strip()
markdown = re.search(r"\[[^\]]*\]\((https?://[^)]+)\)", text, flags=re.IGNORECASE)
if markdown:
return markdown.group(1).strip()
embedded = re.search(r"https?://[^\s)>\]]+", text, flags=re.IGNORECASE)
return embedded.group(0).strip() if embedded else text
if isinstance(value, dict):
for key in ("link", "url", "href", "text"):
candidate = value.get(key)
if isinstance(candidate, str) and candidate.strip():
return candidate.strip()
return ""
if isinstance(value, list):
for item in value:
candidate = coerce_url(item)
if candidate:
return candidate
return ""
return str(value).strip()
def extract_content_id(url_value: Any, platform: str) -> str | None:
"""Extract a stable Xiaohongshu note ID or Douyin video ID from a URL."""
url = coerce_url(url_value)
if not url:
return None
platform_key = platform.casefold()
if platform_key in {"xhs", "xiaohongshu", "pgy"}:
match = re.search(r"/(?:explore|discovery/item|note)/([A-Za-z0-9_-]+)", url)
return match.group(1) if match else None
if platform_key in {"douyin", "xingtu", "xt"}:
match = re.search(r"/video/(\d+)", url)
if match:
return match.group(1)
try:
query = parse_qs(urlparse(url).query)
except ValueError:
return None
for key in ("modal_id", "item_ids", "item_id", "video_id"):
values = query.get(key) or []
if values:
id_match = re.search(r"\d+", str(values[0]))
if id_match:
return id_match.group(0)
return None
def validate_platform_url(url_value: Any, platform: str) -> tuple[bool, str | None]:
"""Validate URL host without rejecting known short-link domains."""
url = coerce_url(url_value)
if not url:
return False, "url_missing"
try:
host = (urlparse(url).hostname or "").casefold()
except ValueError:
return False, "url_invalid"
if not host:
return False, "url_invalid"
key = platform.casefold()
allowed = {
"xhs": ("xiaohongshu.com", "xhslink.com"),
"pgy": ("xiaohongshu.com", "xhslink.com"),
"xiaohongshu": ("xiaohongshu.com", "xhslink.com"),
"douyin": ("douyin.com", "iesdouyin.com"),
"xingtu": ("douyin.com", "iesdouyin.com"),
"xt": ("douyin.com", "iesdouyin.com"),
}.get(key, ())
if allowed and not any(host == domain or host.endswith("." + domain) for domain in allowed):
return False, "platform_url_mismatch"
return True, None
def normalize_title(value: Any) -> str:
"""Unicode/case normalization followed by punctuation and whitespace removal."""
text = unicodedata.normalize("NFKC", str(value or "")).casefold()
text = re.sub(r"[\u200b-\u200f\u2060\ufeff]", "", text)
return re.sub(r"[\W_]+", "", text, flags=re.UNICODE)
def _title_variants(value: Any) -> list[str]:
"""Build safe title forms for platform-added episode prefixes and hashtags."""
text = unicodedata.normalize("NFKC", str(value or "")).casefold()
text = re.sub(r"[\u200b-\u200f\u2060\ufeff]", "", text).strip()
if not text:
return []
raw_variants = [text]
without_episode = re.sub(
r"^\s*[【\[（(]?\s*第\s*[0-9一二三四五六七八九十百零两]+\s*"
r"[集期篇回弹]\s*[】\]）)]?\s*[:：\-—、.]?\s*",
"",
text,
)
without_episode = re.sub(
r"^\s*(?:ep(?:isode)?|part)\s*\d+\s*[:：\-—、.]?\s*",
"",
without_episode,
flags=re.IGNORECASE,
)
if without_episode != text:
raw_variants.append(without_episode)
for candidate in list(raw_variants):
before_hashtag = re.split(r"[#＃]", candidate, maxsplit=1)[0].strip()
if before_hashtag and before_hashtag != candidate:
raw_variants.append(before_hashtag)
variants: list[str] = []
for candidate in raw_variants:
normalized = normalize_title(candidate)
if normalized and normalized not in variants:
variants.append(normalized)
return variants
def _episode_marker(value: Any) -> str | None:
text = unicodedata.normalize("NFKC", str(value or "")).casefold().strip()
chinese = re.match(
r"^[【\[（(]?\s*第\s*([0-9一二三四五六七八九十百零两]+)\s*[集期篇回弹]",
text,
)
english = re.match(r"^(?:ep(?:isode)?|part)\s*(\d+)", text, flags=re.IGNORECASE)
match = chinese or english
if not match:
return None
marker = match.group(1)
return str(int(marker)) if marker.isdigit() else marker
def _normalized_title_similarity(card: str, target: str) -> float:
if not card or not target:
return 0.0
if card == target:
return 1.0
shorter, longer = sorted((card, target), key=len)
if len(shorter) >= 6 and shorter in longer:
coverage = len(shorter) / max(len(longer), 1)
return min(0.98, 0.93 + 0.05 * coverage)
ratio = SequenceMatcher(None, card, target, autojunk=False).ratio()
# Prefixes are common when the UI truncates a title.
common_prefix = os.path.commonprefix([card, target])
if len(common_prefix) >= 8:
ratio = max(ratio, 0.78 + min(0.12, len(common_prefix) / max(len(card), len(target)) * 0.12))
return ratio
def title_similarity(card_title: Any, target_title: Any) -> float:
"""Return a conservative 0..1 score suitable for unique-best matching."""
card_episode = _episode_marker(card_title)
target_episode = _episode_marker(target_title)
if card_episode and target_episode and card_episode != target_episode:
return 0.0
card_variants = _title_variants(card_title)
target_variants = _title_variants(target_title)
if not card_variants or not target_variants:
return 0.0
return max(
_normalized_title_similarity(card, target)
for card in card_variants
for target in target_variants
)
def _metric_number(value: Any) -> float | None:
if isinstance(value, (int, float)):
return float(value)
text = str(value or "").strip().replace(",", "")
match = re.match(r"^([0-9]+(?:\.[0-9]+)?)\s*(万|w)?", text, flags=re.IGNORECASE)
if not match:
return None
number = float(match.group(1))
return number * 10000 if match.group(2) else number
def deduplicate_observed_cards(cards: Iterable[dict]) -> list[dict]:
"""Merge API/DOM observations of the same video while preserving distinct IDs."""
unique: list[dict] = []
for raw in cards:
if not isinstance(raw, dict):
continue
card = dict(raw)
card_id = str(card.get("note_id") or "")
card_url = coerce_url(card.get("href") or card.get("url")).rstrip("/")
duplicate_index: int | None = None
for index, old in enumerate(unique):
old_id = str(old.get("note_id") or "")
old_url = coerce_url(old.get("href") or old.get("url")).rstrip("/")
if card_id and old_id:
if card_id == old_id:
duplicate_index = index
if duplicate_index is not None or card_id != old_id:
if duplicate_index is not None:
break
continue
if card_url and old_url and card_url == old_url:
duplicate_index = index
break
one_is_api = (
card.get("source") == "show_items_api"
or old.get("source") == "show_items_api"
)
one_missing_id = not card_id or not old_id
same_title = (
normalize_title(card.get("title"))
and normalize_title(card.get("title")) == normalize_title(old.get("title"))
)
card_play = _metric_number(card.get("play_count"))
old_play = _metric_number(old.get("play_count"))
metric_compatible = (
card_play is None
or old_play is None
or abs(card_play - old_play) / max(card_play, old_play, 1.0) <= 0.08
)
if one_is_api and one_missing_id and same_title and metric_compatible:
duplicate_index = index
break
if duplicate_index is None:
unique.append(card)
continue
old = unique[duplicate_index]
card_quality = int(bool(card.get("note_id"))) + int(card.get("source") == "show_items_api")
old_quality = int(bool(old.get("note_id"))) + int(old.get("source") == "show_items_api")
preferred, fallback = (card, old) if card_quality > old_quality else (old, card)
merged = dict(fallback)
merged.update({key: value for key, value in preferred.items() if value not in (None, "")})
unique[duplicate_index] = merged
return unique
def match_tasks_to_cards(
tasks: Iterable[dict],
cards: Iterable[dict],
platform: str,
*,
min_score: float = 0.78,
ambiguity_margin: float = 0.035,
) -> dict[str, dict]:
"""Match by ID first, then safely allocate each observed card to one source note."""
card_list = [dict(card) for card in cards if isinstance(card, dict)]
for card in card_list:
if not card.get("note_id"):
card["note_id"] = extract_content_id(card.get("href") or card.get("url"), platform)
task_list = [dict(task) for task in tasks if isinstance(task, dict) and task.get("record_id")]
def task_source_key(task: dict, content_id: Any = None) -> tuple[str, str]:
stable_id = content_id or task.get("note_id") or extract_content_id(task.get("note_url"), platform)
if stable_id:
return "id", str(stable_id)
url = coerce_url(task.get("note_url")).rstrip("/")
if url:
return "url", url
# Without a source URL, never let separate records reuse one fuzzy match.
return "record", str(task.get("record_id"))
def task_allows_card(task: dict, card: dict) -> bool:
task_id = task.get("note_id") or extract_content_id(task.get("note_url"), platform)
card_id = card.get("note_id")
return not task_id or not card_id or str(task_id) == str(card_id)
def card_key(card: dict, index: int) -> tuple[str, str]:
if card.get("note_id"):
return "id", str(card["note_id"])
url = coerce_url(card.get("href") or card.get("url")).rstrip("/")
return ("url", url) if url else ("index", str(index))
indexed_cards = [(index, card, card_key(card, index)) for index, card in enumerate(card_list)]
matched: dict[str, dict] = {}
claimed: dict[tuple[str, str], tuple[str, str]] = {}
remaining: list[dict] = []
for task in task_list:
record_id = str(task.get("record_id") or "")
content_id = task.get("note_id") or extract_content_id(task.get("note_url"), platform)
if content_id:
id_matches = [
(index, card, key)
for index, card, key in indexed_cards
if str(card.get("note_id") or "") == str(content_id)
]
if id_matches:
_, raw_card, key = id_matches[0]
source_key = task_source_key(task, content_id)
owner = claimed.get(key)
if owner is not None and owner != source_key:
remaining.append(task)
continue
claimed[key] = source_key
selected = dict(raw_card)
selected.update({"match_method": "content_id", "match_score": 1.0})
matched[record_id] = selected
continue
remaining.append(task)
# If two different source URLs have the exact same title, title alone cannot
# identify which record owns the observed card. Leave both unresolved.
conflicted_cards: set[tuple[str, str]] = set()
for _, card, key in indexed_cards:
exact_sources = {
task_source_key(task)
for task in remaining
if task_allows_card(task, card)
if title_similarity(card.get("title"), task.get("target_title")) == 1.0
}
if len(exact_sources) > 1:
conflicted_cards.add(key)
def task_best_score(task: dict) -> float:
return max(
(
title_similarity(card.get("title"), task.get("target_title"))
for _, card, key in indexed_cards
if key not in conflicted_cards
and task_allows_card(task, card)
),
default=0.0,
)
# Exact/strongest tasks claim first, so a looser title cannot steal the one
# card needed by a later exact task.
remaining.sort(key=lambda task: (-task_best_score(task), str(task.get("record_id"))))
for task in remaining:
record_id = str(task.get("record_id") or "")
source_key = task_source_key(task)
scored = sorted(
(
(title_similarity(card.get("title"), task.get("target_title")), index, card, key)
for index, card, key in indexed_cards
if key not in conflicted_cards
and task_allows_card(task, card)
and (claimed.get(key) is None or claimed.get(key) == source_key)
),
key=lambda item: (-item[0], item[1]),
)
if not scored or scored[0][0] < min_score:
continue
best_score, _, best_card, best_key = scored[0]
second_score = scored[1][0] if len(scored) > 1 else 0.0
if (
second_score >= min_score
and best_score - second_score < ambiguity_margin
and (best_score < 1.0 or second_score == 1.0)
):
second_card = scored[1][2]
best_id = str(best_card.get("note_id") or "")
second_id = str(second_card.get("note_id") or "")
if not best_id or best_id != second_id:
continue
claimed[best_key] = source_key
selected = dict(best_card)
selected.update({"match_method": "title", "match_score": round(best_score, 4)})
matched[record_id] = selected
return matched
def result_status(row: dict) -> str:
"""Read new status values and conservatively classify legacy result rows."""
explicit = row.get("status")
if explicit in TERMINAL_STATUSES:
return explicit
if row.get("write_ok") is False or row.get("write_error"):
return WRITE_FAILURE
if row.get("matched") is False or row.get("ok") is False or row.get("error") or row.get("reason"):
return RETRYABLE_FAILURE
if row.get("matched") is True or row.get("ok") is True:
return SUCCESS
metric_keys = ("read_count", "play_count", "view_count", "updated_value")
if any(row.get(key) is not None for key in metric_keys):
return SUCCESS
return RETRYABLE_FAILURE
def upsert_result(summary: dict, row: dict) -> None:
"""Replace the previous result for one record instead of appending duplicates."""
results = summary.setdefault("results", [])
record_id = row.get("record_id")
if record_id:
for index, old in enumerate(results):
if isinstance(old, dict) and old.get("record_id") == record_id:
results[index] = row
return
results.append(row)
def finalize_summary(summary: dict, *, dry_run: bool = False) -> dict:
"""Deduplicate results, recalculate counters, and attach completeness fields."""
out = dict(summary)
unique: list[dict] = []
positions: dict[str, int] = {}
duplicate_results = 0
for raw in out.get("results") or []:
if not isinstance(raw, dict):
continue
row = dict(raw)
status = result_status(row)
if status == SUCCESS and not dry_run and row.get("write_ok") is False:
status = WRITE_FAILURE
row["status"] = status
if status == SUCCESS:
row.setdefault("matched", True)
elif status != BLOCKED_INPUT:
row.setdefault("matched", False)
rid = str(row.get("record_id") or "")
if rid and rid in positions:
unique[positions[rid]] = row
duplicate_results += 1
else:
if rid:
positions[rid] = len(unique)
unique.append(row)
total = int(out.get("total") or 0)
success = sum(result_status(row) == SUCCESS for row in unique)
blocked = sum(result_status(row) == BLOCKED_INPUT for row in unique)
retryable = sum(result_status(row) == RETRYABLE_FAILURE for row in unique)
write_failures = sum(result_status(row) == WRITE_FAILURE for row in unique)
missing = max(0, total - len(unique))
unresolved = retryable + write_failures + missing
top_level_error = bool(out.get("error"))
if top_level_error:
retryable = max(retryable, 1)
unresolved = max(unresolved, 1)
if dry_run:
filled = 0
else:
explicit_write_rows = [row for row in unique if result_status(row) == SUCCESS and "write_ok" in row]
if explicit_write_rows:
filled = sum(row.get("write_ok") is True for row in explicit_write_rows)
# Preserve legacy successes that predate per-row write status.
filled += sum(
1 for row in unique
if result_status(row) == SUCCESS and "write_ok" not in row
)
else:
filled = min(int(out.get("filled") or 0), success)
out.update({
"results": unique,
"matched": success + write_failures,
"filled": filled,
"success": success,
"blocked_input": blocked,
"retryable_failures": retryable,
"write_failures": write_failures,
"missing_results": missing,
"duplicate_results": duplicate_results,
"unresolved": unresolved,
"complete": (
not top_level_error
and unresolved == 0
and duplicate_results == 0
and len(unique) == total
),
"dry_run": bool(dry_run),
})
return out
def merge_style_summary(existing: dict | None, partial: dict, *, dry_run: bool = False) -> dict:
"""Merge a record-limited retry into the canonical per-style summary."""
if not existing:
return finalize_summary(partial, dry_run=dry_run)
combined = dict(existing)
for key, value in partial.items():
if key != "results":
combined[key] = value
rows: list[dict] = []
positions: dict[str, int] = {}
for source in (existing.get("results") or [], partial.get("results") or []):
for raw in source:
if not isinstance(raw, dict):
continue
row = dict(raw)
rid = str(row.get("record_id") or "")
if rid and rid in positions:
rows[positions[rid]] = row
else:
if rid:
positions[rid] = len(rows)
rows.append(row)
combined["results"] = rows
combined["total"] = max(
int(existing.get("total") or 0),
int(partial.get("total") or 0),
len(rows),
)
return finalize_summary(combined, dry_run=dry_run)
def merge_global_summaries(existing: Any, updates: Iterable[dict]) -> list[dict]:
"""Merge selected style summaries into an existing all-style summary by index."""
merged: dict[Any, dict] = {}
order: list[Any] = []
for row in existing if isinstance(existing, list) else []:
if not isinstance(row, dict):
continue
key = row.get("index", row.get("style"))
if key not in merged:
order.append(key)
merged[key] = row
for row in updates:
if not isinstance(row, dict):
continue
key = row.get("index", row.get("style"))
if key not in merged:
order.append(key)
merged[key] = row
return [merged[key] for key in sorted(order, key=lambda value: (not isinstance(value, int), value))]
def requested_records_succeeded(summary: dict, record_ids: Iterable[str]) -> tuple[bool, list[str]]:
"""Validate that every requested record exists and has a successful terminal state."""
by_id = {
str(row.get("record_id")): row
for row in (summary.get("results") or summary.get("details") or [])
if isinstance(row, dict) and row.get("record_id")
}
failed = [str(rid) for rid in record_ids if result_status(by_id.get(str(rid), {})) != SUCCESS]
return not failed, failed
src/gyxx_flow/modules/content_marketing/runtime/creator_task_grouping.py
+53
View File
@@ -0,0 +1,53 @@
"""Shared helpers for grouping cross-style scrape tasks by creator."""
from __future__ import annotations
from collections import OrderedDict, defaultdict
def _clean_creator_id(value) -> str:
return str(value or "").strip()
def _clean_creator_name(value) -> str:
return "".join(str(value or "").split()).casefold()
def group_tasks_by_creator(tasks: list[dict]) -> list[dict]:
"""Group tasks globally, preferring creator ID and falling back to name.
A name-only task joins an ID group when that normalized name maps to exactly
one ID in the current run. Conflicting IDs remain separate.
"""
ids_by_name: dict[str, set[str]] = defaultdict(set)
for task in tasks:
creator_id = _clean_creator_id(task.get("creator_id"))
creator_name = _clean_creator_name(task.get("creator_name"))
if creator_id and creator_name:
ids_by_name[creator_name].add(creator_id)
groups: OrderedDict[tuple[str, str], dict] = OrderedDict()
for task in tasks:
creator_id = _clean_creator_id(task.get("creator_id"))
creator_name_key = _clean_creator_name(task.get("creator_name"))
if not creator_id and len(ids_by_name.get(creator_name_key, ())) == 1:
creator_id = next(iter(ids_by_name[creator_name_key]))
key = ("id", creator_id) if creator_id else ("name", creator_name_key)
if key not in groups:
groups[key] = {
"creator_name": str(task.get("creator_name") or "").strip(),
"creator_id": creator_id or None,
"tasks": [],
}
groups[key]["tasks"].append(task)
return list(groups.values())
def chunk_creator_groups(groups: list[dict], batch_size: int) -> list[list[dict]]:
"""Split creator groups into batches; non-positive means unlimited."""
if not groups:
return []
if batch_size <= 0:
return [groups]
return [groups[i:i + batch_size] for i in range(0, len(groups), batch_size)]
src/gyxx_flow/modules/content_marketing/runtime/daily_marketing_report.py
+1270
View File
File diff suppressed because it is too large Load Diff
src/gyxx_flow/modules/content_marketing/runtime/data/config/analyze.env.example
+8
View File
@@ -0,0 +1,8 @@
# 分析脚本配置模板
# 复制为 analyze.env 并填真实值 (analyze.env 已被 .gitignore 排除)
# Hermes 分析端 API Token
HERMES_ANALYZER_TOKEN=your_token_here
# 飞书接收人 open_id (不设则必须传 --no-send)
FEISHU_TARGET_OPEN_ID=ou_xxxxxxxxxxxxxxxxx
src/gyxx_flow/modules/content_marketing/runtime/data/config/db.env.example
+8
View File
@@ -0,0 +1,8 @@
# PG 连接信息模板
# 复制为 db.env 并填真实值 (db.env 已被 .gitignore 排除)
PG_HOST=8.148.185.119
PG_PORT=5432
PG_DB=data_hub
PG_USER=data_hub
PG_PASSWORD=${GYXX_PG_PASSWORD}
src/gyxx_flow/modules/content_marketing/runtime/data/config/款式_多维表格_对照.json
+6154
View File
File diff suppressed because it is too large Load Diff
src/gyxx_flow/modules/content_marketing/runtime/data/config/款式_多维表格_对照_自营.json
+1807
View File
File diff suppressed because it is too large Load Diff
src/gyxx_flow/modules/content_marketing/runtime/data/tools/analyze_comments.py
+863
View File
@@ -0,0 +1,863 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""评论采集数据 → 本地关键词过滤 + Hermes 舆情分析 → 项目 data/summary 汇总 + 飞书通知。
使用本地 Hermes 分析端：
- URL: http://127.0.0.1:8642/v1（~/.hermes/profiles/data-analyzer/.env 里的 API_SERVER_PORT）
- Model: mimo-v2.5-pro
- Profile: data-analyzer
流程：
1. 读取项目采集汇总的评论文件（默认取 data/summary/ 下最新 CSV，支持 txt/json/csv）。
2. 用本地 Python 关键词规则（飞书文档 6 层体系）过滤评论。
3. 用 Hermes 分析端 + 内置舆情分析提示词生成汇总报告。
4. 将汇总写入 data/summary/评论舆论分析汇总.txt。
5. 通过 lark-cli --profile hermes-analyzer 发送汇总给目标 open_id。
"""
import argparse
import concurrent.futures
import csv
import json
import os
from gyxx_flow.adapters import RuntimeServicePolicy
import random
import re
import subprocess
import sys
import time
import urllib.error
import urllib.request
from pathlib import Path
from typing import Any
from gyxx_flow.modules.content_marketing.runtime.runtime_paths import (
PATHS,
resolve_layer_output,
)
# Path setup
_TOOLS_DIR = PATHS.tools_root
_PROJECT_ROOT = PATHS.module_root
try:
from dotenv import load_dotenv
_analyze_env = PATHS.state_root / "config/analyze.env"
if _analyze_env.exists():
load_dotenv(_analyze_env, override=False)
except ImportError:
pass
PROJECT_ROOT = _PROJECT_ROOT
DATA_DIR = PATHS.raw_root
SUMMARY_DIR = PATHS.exports_root / "summary"
def _latest_summary_csv() -> Path:
"""自动选取 data/summary/ 下最新的汇总 CSV（项目采集输出）。"""
if not SUMMARY_DIR.exists():
return SUMMARY_DIR / "*.csv"
csv_files = sorted(SUMMARY_DIR.glob("*.csv"), key=lambda p: p.stat().st_mtime, reverse=True)
return csv_files[0] if csv_files else SUMMARY_DIR / "*.csv"
DEFAULT_INPUT = _latest_summary_csv()
OUTPUT_PATH = SUMMARY_DIR / "评论舆论分析汇总.txt"
# ---------------------------------------------------------------------------
# Hermes 分析端配置
# 实际 API Server 端口见 ~/.hermes/profiles/data-analyzer/.env 中的 API_SERVER_PORT（当前为 8642）
# ---------------------------------------------------------------------------
os.environ.update(RuntimeServicePolicy().apply(os.environ))
HERMES_ANALYZER_URL = os.environ["HERMES_ANALYZER_URL"]
HERMES_ANALYZER_TOKEN = os.getenv("HERMES_ANALYZER_TOKEN", "")
HERMES_ANALYZER_MODEL = os.getenv("HERMES_ANALYZER_MODEL", "mimo-v2.5-pro")
TARGET_OPEN_ID = os.getenv("FEISHU_TARGET_OPEN_ID", "")
# 飞书长消息截断阈值（留有余量）
FEISHU_TEXT_LIMIT = 6000
# ---------------------------------------------------------------------------
# 本地代码过滤关键词（飞书文档《评论区关键词抓取》6 层体系）
# ---------------------------------------------------------------------------
# 流程：全量采集评论 → 本地关键词过滤 → 对有效评论进行舆情分析汇总
CODE_FILTER_KEYWORDS = {
# 第1层：购买意向信号（衡量种草效率）
"purchase_intent": [
# 直接购买
"下单", "已买", "入手", "冲了", "付款", "等收货",
# 搜索意向
"怎么买", "链接", "哪里买", "搜什么", "淘宝", "店铺",
# 价格评估
"多少钱", "价格", "划算", "贵不贵", "优惠", "值得买",
# 犹豫对比（用双字词避免单字"比"误匹配[比心]等表情）
"纠结", "哪个好", "容量够吗", "犹豫", "对比", "比较", "和哪个",
],
# 第2层：产品功能反馈（正向）
"positive_feature": [
# 轻量化（"轻"用双字避免误匹配"年轻人"等）
"很轻", "超轻", "挺轻", "不重", "没感觉", "毫无负担", "背着不累",
# 收纳秩序
"分区", "隔层", "收纳", "能装", "各就各位",
# 快取便利（"快"用双字避免误匹配"快乐"等）
"很方便", "顺手", "很快", "单手", "侧袋", "不用翻",
# 品质感
"质感", "材质", "面料", "五金", "走线", "高级",
],
# 第3层：产品功能反馈（负向）
"negative_feature": [
# 重量负担（"重"用双字避免误匹配"重要"等）
"很重", "太重", "超重", "偏重", "背不动", "肩膀疼",
# 收纳不足
"不够装", "太小", "放不下", "没地方", "容量",
# 品质问题
"线头", "脱线", "划痕", "掉色", "起皱", "做工",
# 使用不便（"卡""紧"用双字避免误匹配）
"不好拿", "卡住", "太紧", "难开", "不方便",
],
# 第4层：用户场景（判断内容与目标用户是否匹配）
"user_scenario": [
# 通勤
"上班", "地铁", "挤公交", "通勤", "工位", "办公室",
# 旅行/周末
"旅行", "出差", "周末", "短途", "出游", "打包",
# 社交
"约会", "聚会", "探店", "咖啡馆", "拍照",
# 户外
"徒步", "爬山", "露营", "户外",
],
# 第5层：用户审美与风格认同
"aesthetic": [
# 颜值设计
"好看", "颜值", "设计", "高级", "喜欢这个",
# 百搭搭配（"配"用双字避免误匹配）
"百搭", "好搭", "搭衣服", "搭配", "风格",
# 子风格认同（"潮"用双字避免误匹配）
"机能", "潮流", "潮牌", "科技感", "克制", "简约", "秩序",
],
# 第6层：品牌情感（长期心智指标）
"brand_emotion": [
# 推荐意愿
"推荐", "安利", "值得", "满意", "惊喜",
# 品牌认同
"光影行星", "关注", "忠实", "一直用", "老用户",
# 复购意向
"再买", "第二个", "送人", "给家人", "还会",
],
}
# 包包品类通用词：用于判断评论是否与"我方产品"相关。
# 除了品牌名/款式名，评论中出现这些词也视为与我方产品相关。
BAG_CATEGORY_TERPS = [
# 包类通用
"包", "背包", "双肩包", "单肩包", "斜挎包", "手提包", "托特包",
"邮差包", "胸包", "腰包", "手包", "拎包",
# 容量/装载
"能装", "装得下", "容量", "放电脑", "放平板", "放手机",
"装东西", "装不下", "够装", "收纳",
# 背负/功能
"背着", "背过", "背起来", "背负", "肩带", "减负",
"防泼水", "防水", "防震", "减震",
# 材质/做工
"面料", "材质", "走线", "五金", "拉链", "做工",
# 通勤/出行场景中的包
"通勤包", "出差包", "旅行包", "电脑包", "书包",
"机车包", "头盔包", "健身包", "运动包",
]
# 纯寒暄/无信息词：只含这些词（+标点）且未命中任何关键词的评论丢弃
# 注意：不要把文档中的有效关键词（如"好看"）放进来，否则会被误丢
CODE_FILTER_NOISE_WORDS = {
"好", "棒", "支持", "来了", "赞", "厉害", "不错", "优秀",
"喜欢", "可爱", "666", "牛", "太强了", "真棒", "很好",
}
# 判断纯表情/无意义评论：只要包含中文、英文或数字就不是纯表情
_HAS_MEANINGFUL_CHAR_RE = re.compile(r"[\u4e00-\u9fa5a-zA-Z0-9]")
# ---------------------------------------------------------------------------
# 内置提示词（已硬编码，运行时不读取 md 文件）
# ---------------------------------------------------------------------------
ANALYSIS_SYSTEM_PROMPT = """# 角色：小红书评论舆情分析专家
你是一位专注小红书生态5年的舆情分析师，擅长将非结构化评论转化为结构化洞察，并给出可落地的口碑管理策略。
## 核心能力
1. **情绪分层**：正向（赞美/推荐/回购/共鸣）、负向（质量/服务/价格/竞品贬低）、中性（咨询/建议/客观描述）；识别情绪烈度（强烈推荐 vs 还行 vs 非常失望）。
2. **主题聚类**：产品（材质/容量/颜色/重量/五金）、服务（发货/包装/客服/售后）、内容（种草来源/预期落差）、场景（通勤/旅行/约会/健身）、竞品（对比品牌/话术）。
3. **用户画像**：推断年龄（学生党/上班族/宝妈）、消费力（价格敏感型 vs 直接下单型）、使用场景、审美偏好；分类用户类型（理性测评/感性颜值/价格敏感）。
4. **风险预警**：≥3人提及同问题 → 品控/服务漏洞；出现"平替""不如XX" → 定位模糊；出现"退货""过敏""坏了""假货" → 严重投诉。分级：P0(安全/假货/严重质量)、P1(批量负面但可控)、P2(偶发吐槽)。
5. **竞品对比**：正/负面率对比、被夸/被骂维度对比、差异化口碑优势与薄弱环节。
6. **营销归因**：评论热点与近期投放/促销/新品动作联动，给出口碑效果归因图。
7. **优化建议**：产品端（高频吐槽改进）、内容端（详情页/笔记关键信息优化）、客服端（FAQ/话术优化）、运营端（种草卖点提炼/补救内容设计）。
8. **报告输出**：标准化舆情周报/月报（数据概览+主题分析+画像+预警+竞品+建议）。
## 工作原则
- 数据说话：每条结论附数据支撑（如"63%负评集中在尺寸偏小"）
- 客观中立：不过度美化也不过度恐慌
- 行动导向：给出"下周一就能做的3件事"
- 长线思维：关注长期品牌口碑健康指标
- 用户视角：读出用户"没说出来但藏在语气里的情绪"
## 输出格式
收到评论数据后，按以下结构输出：
### 一、数据概览
- 评论总数 / 有效评论数
- 情绪分布：正面X% / 中性X% / 负面X%
- 互动概览：平均点赞/回复数、高互动Top3
### 二、核心发现（Top 3）
每条附数据支撑
### 三、分维度深度分析
| 排名 | 主题标签 | 提及次数 | 正向占比 | 负向占比 | 典型评论 |
|------|----------|----------|----------|----------|----------|
| 1 | 材质 | 45 | 80% | 10% | "皮料很软" |
- **情绪趋势**：正/负/中性随时间波动，是否有拐点
- **用户画像**：高频身份、核心场景、主要决策因素
- **风险预警**：P0/P1/P2 分级清单
### 四、归因分析
正向评论高峰由哪些营销动作带来？负面集中反馈由哪些动作引发？
### 五、优化建议（按优先级）
- **P0（立即）**：问题 → 行动 → 责任方
- **P1（本周）**：问题 → 行动 → 责任方
- **P2（下月）**：问题 → 行动 → 责任方
### 六、可复用资产
- 评论分类标签体系
- 情绪打分规则
- 舆情周报模板
## 约束
- 数据量<50条时注明"样本量较小，结论仅供参考"
- 涉及隐私的评论需脱敏处理
- 所有建议必须标注责任方（产品/内容/客服/运营）"""
def log(message: str) -> None:
print(f"[{time.strftime('%H:%M:%S')}] {message}", flush=True)
# ---------------------------------------------------------------------------
# 文件读取
# ---------------------------------------------------------------------------
def load_text(path: Path) -> str:
if not path.exists():
raise FileNotFoundError(f"找不到文件: {path}")
for enc in ("utf-8-sig", "utf-8", "gbk"):
try:
return path.read_text(encoding=enc)
except UnicodeDecodeError:
continue
return path.read_text(encoding="utf-8", errors="replace")
def _extract_comment_text(item: Any) -> str:
if isinstance(item, str):
return item.strip()
if isinstance(item, dict):
for key in ("comment_text", "content", "text", "message", "comment"):
val = item.get(key)
if val is not None:
return str(val).strip()
return str(item).strip()
def load_comments(path: Path) -> list[str]:
"""从 txt/json/csv 加载评论文本列表。"""
text = load_text(path)
ext = path.suffix.lower()
if ext == ".json":
data = json.loads(text)
if isinstance(data, list):
return [_extract_comment_text(item) for item in data if _extract_comment_text(item)]
if isinstance(data, dict):
for key in ("comments", "data", "list", "rows"):
if key in data and isinstance(data[key], list):
return [_extract_comment_text(item) for item in data[key] if _extract_comment_text(item)]
raise ValueError(f"不支持的 JSON 结构: {path}")
if ext == ".csv":
rows = list(csv.DictReader(text.splitlines()))
candidates = ("comment_text", "content", "text", "message", "评论内容", "评论", "comment")
comments: list[str] = []
for row in rows:
for key in candidates:
val = row.get(key)
if val is not None and str(val).strip():
comments.append(str(val).strip())
break
return comments
# 默认：每行一条评论
return [line.strip() for line in text.splitlines() if line.strip()]
# ---------------------------------------------------------------------------
# 按款式分批处理（用于项目汇总 CSV）
# ---------------------------------------------------------------------------
def load_style_groups(path: Path, max_per_style: int | None = None) -> dict[str, list[str]]:
"""从 CSV 中按款式分组读取评论，可选每款式抽样上限。"""
text = load_text(path)
rows = list(csv.DictReader(text.splitlines()))
candidates = ("comment_text", "content", "text", "message", "评论内容", "评论", "comment")
style_key_candidates = ("style_name", "款式", "style", "产品")
style_col: str | None = None
for row in rows[:1]:
for key in style_key_candidates:
if key in row:
style_col = key
break
groups: dict[str, list[str]] = {}
for row in rows:
style = row.get(style_col, "未知款式") if style_col else "未知款式"
style = style.strip() or "未知款式"
for key in candidates:
val = row.get(key)
if val is not None and str(val).strip():
groups.setdefault(style, []).append(str(val).strip())
break
if max_per_style:
rng = random.Random(42)
for style in groups:
if len(groups[style]) > max_per_style:
groups[style] = rng.sample(groups[style], max_per_style)
return dict(sorted(groups.items(), key=lambda item: -len(item[1])))
def analyze_style(style: str, comments: list[str]) -> tuple[str, int, int]:
"""对单个款式的评论进行过滤 + 分析，返回（汇总文本, 原始数, 过滤后数）。"""
log(f"【{style}】开始处理（{len(comments)} 条）...")
filtered = code_filter_comments(comments)
summary = analyze_comments(filtered)
return summary, len(comments), len(filtered)
def build_batch_report(style_results: list[tuple[str, str, int, int]]) -> str:
"""把所有款式汇总合并成一份总报告。"""
total_raw = sum(raw for _, _, raw, _ in style_results)
total_filtered = sum(filt for _, _, _, filt in style_results)
lines = [
"# 多款式评论舆情汇总报告",
"",
f"- 款式总数：{len(style_results)}",
f"- 评论总数（原始）：{total_raw}",
f"- 有效评论总数（过滤后）：{total_filtered}",
"",
"---",
"",
]
for style, summary, raw, filtered in style_results:
lines.append(f"## 款式：{style}")
lines.append(f"原始评论：{raw} 条 | 过滤后：{filtered} 条")
lines.append("")
lines.append(summary)
lines.append("")
lines.append("---")
lines.append("")
return "\n".join(lines)
# ---------------------------------------------------------------------------
# Hermes 分析端调用
# ---------------------------------------------------------------------------
def call_hermes_analyzer(system_prompt: str, user_content: str) -> str:
url = HERMES_ANALYZER_URL.rstrip("/") + "/chat/completions"
payload = {
"model": HERMES_ANALYZER_MODEL,
"messages": [
{"role": "system", "content": system_prompt},
{"role": "user", "content": user_content},
],
}
data = json.dumps(payload, ensure_ascii=False).encode("utf-8")
request = urllib.request.Request(
url,
data=data,
headers={
"Authorization": f"Bearer {HERMES_ANALYZER_TOKEN}",
"Content-Type": "application/json; charset=utf-8",
},
method="POST",
)
try:
with urllib.request.urlopen(request, timeout=1800) as response:
result = json.loads(response.read().decode("utf-8", errors="replace"))
except urllib.error.HTTPError as exc:
body = exc.read().decode("utf-8", errors="replace")
raise RuntimeError(f"Hermes HTTP {exc.code}: {body}") from exc
if "choices" not in result or not result["choices"]:
raise RuntimeError(f"Hermes 返回异常: {result}")
return str(result["choices"][0]["message"]["content"])
def _is_pure_emoji(text: str) -> bool:
"""判断是否为纯表情/无意义评论（不含中文、英文、数字）。"""
return not _HAS_MEANINGFUL_CHAR_RE.search(text)
# 抖音/B站表情格式：[比心] [舔屏] 等，匹配前需去掉避免误触发关键词
_EMOJI_BRACKET_RE = re.compile(r'\[[^\[\]]{1,10}\]')
def _strip_emoji_brackets(text: str) -> str:
"""去掉 [xxx] 格式的表情符号，避免表情文本误触发关键词匹配。"""
return _EMOJI_BRACKET_RE.sub('', text).strip()
def _matches_code_keywords(text: str) -> bool:
"""本地关键词匹配，命中任意有效关键词即保留。"""
clean = _strip_emoji_brackets(text)
if not clean:
return False
for keywords in CODE_FILTER_KEYWORDS.values():
for kw in keywords:
if kw in clean:
return True
return False
def classify_comment_keywords(text: str) -> list[str]:
"""返回一条评论命中的所有关键词类别（用于统计分布）。"""
clean = _strip_emoji_brackets(text)
if not clean:
return []
categories: list[str] = []
for category, keywords in CODE_FILTER_KEYWORDS.items():
for kw in keywords:
if kw in clean:
categories.append(category)
break
return categories
def _is_noise_only(text: str) -> bool:
"""判断是否为纯寒暄/无信息评论（仅由常见寒暄词和标点组成）。"""
# 去掉常见寒暄词
cleaned = text
for w in sorted(CODE_FILTER_NOISE_WORDS, key=len, reverse=True):
cleaned = cleaned.replace(w, "")
# 再去掉常见标点、空格、数字
cleaned = re.sub(r"[\s，,。！!？?~～.·\d]+", "", cleaned).strip()
return not cleaned
def code_filter_comments(comments: list[str]) -> list[str]:
"""用本地代码规则快速过滤评论，不调用大模型。"""
if not comments:
log("没有评论需要过滤")
return []
log(f"使用本地代码规则进行评论过滤（共 {len(comments)} 条）...")
seen: set[str] = set()
kept: list[str] = []
for text in comments:
t = text.strip()
if not t or t in seen:
continue
if _is_pure_emoji(t):
continue
if _matches_code_keywords(t):
seen.add(t)
kept.append(t)
continue
# 未命中关键词，但属于短句纯寒暄，也丢弃
if len(t) <= 12 and _is_noise_only(t):
continue
# 其余未命中关键词的评论丢弃
log(f"代码过滤完成：{len(comments)} 条 → {len(kept)} 条")
return kept
# ---------------------------------------------------------------------------
# LLM 评论清洗(语义过滤,替代/补充关键词过滤)
# ---------------------------------------------------------------------------
COMMENT_CLEAN_SYSTEM_PROMPT = """你是"光影行星"品牌的评论清洗助手。光影行星主要经营双肩包、单肩包、斜挎包、摄影包、电脑包、手提包及相关包袋配件。你的任务是过滤无效评论，并从评论中提取对品牌经营、产品优化或用户服务有价值的内容。
有效评论包括：
1. 产品评价：涉及外观、颜色、尺寸、容量、重量、材质、做工、气味、拉链、肩带、扣具、分区、收纳、防水性、耐用性、舒适度、便携性等。
2. 使用体验：涉及通勤、旅行、摄影、户外、上学、商务、电脑收纳、相机收纳等真实使用场景。
3. 产品咨询：询问尺寸、颜色、容量、适配设备、材质、库存、价格、使用方法、清洁保养或不同款式区别。
4. 购买意向：明确表达想买、准备下单、等待补货、询问购买链接或推荐款式。
5. 产品问题与售后反馈：涉及破损、开线、掉色、拉链故障、尺寸不符、配件缺失、使用异常、退换货原因等。
6. 品牌反馈：明确提到光影行星，对品牌设计、质量、价格、服务或产品系列提出评价和建议。
无效评论包括：
1. 只有表情、数字、标点或无意义字符。
2. "哈哈哈""路过""支持一下""不错不错"等没有具体信息的泛泛表达。
3. 广告、引流、联系方式、刷单、抽奖、互关或推广其他商品。
4. 与光影行星或包袋产品完全无关的内容。
5. 明显复制粘贴、重复发送或机器生成的灌水内容。
6. 单纯辱骂、攻击他人且没有任何产品事实或具体问题。
7. 无法理解、语义残缺且不能确定评论意图的内容。
8. 只讨论主播、视频背景、音乐或其他无关内容。
9. 仅询问快递到了哪里，且不包含产品或售后问题。
10. 没有真实内容的默认好评或模板评价。
判定原则：
1. 负面评论只要包含具体产品问题，就属于有效评论，不能因为评价负面而删除。
2. 评论同时包含有效和无效内容时，只提取其中与光影行星及包袋产品有关的有效部分。
3. 不得修改评论原意，不得虚构用户没有提到的信息。
4. 可以删除表情、重复语句、无意义语气词和无关内容。
5. "很好""不错""喜欢"等评论，如果没有说明具体原因，判定为无效。
6. "容量很大""肩带舒服""拉链不好用"等虽然简短，但包含具体产品信息，应判定为有效。
7. 无法确定时，优先保留，但标记为"需要人工确认"。
8. 所有"有效"类别(产品评价/使用体验/产品咨询/购买意向/售后问题/品牌反馈)都必须限定在"光影行星品牌"或"包袋类产品"范围内。如果评论只询问/讨论笔记里出现的其他物品(如镜子、指甲刀、U型枕、蚊虫膏、行李箱、自拍支架、护照收纳袋、登山杖等非包袋物品),即使有购买意向或产品咨询,也判定为无效。
9. 评论同时提到包袋和其他物品时,只看包袋相关部分:若包袋部分有具体信息(评价/咨询/购买意向等),按有效处理;若包袋部分只是被顺带提及或无具体信息,判定为无效。
每条评论按照以下格式输出（不要输出其它内容）：
评论编号：<编号>
是否有效：有效/无效/需要人工确认
有效评论：<提取并清洗后的有效内容；无有效内容则填写"无">
评论类型：产品评价/使用体验/产品咨询/购买意向/售后问题/品牌反馈/无效内容
情感倾向：正面/中性/负面/无法判断
涉及维度：外观、颜色、尺寸、容量、材质、做工、舒适度、收纳、防水、耐用、价格、配件、售后、其他
无效原因：<有效评论填写"无"；无效评论说明具体原因>"""
_CLEAN_FIELD_RE = re.compile(
r"(是否有效|有效评论|评论类型|情感倾向|涉及维度|无效原因)\s*[：:]\s*(.*)"
)
def parse_cleaning_response(response: str) -> list[dict]:
"""解析 LLM 结构化输出。每条评论 7 行,按"评论编号："分块。
返回 [{valid, cleaned, type, sentiment, dimensions, reason}, ...]
"""
items: list[dict] = []
# 按"评论编号：<数字>"分块,跳过首块(分割后的空块)
blocks = re.split(r"评论编号\s*[：:]\s*\d+", response)
for block in blocks[1:]:
item = {
"valid": "无效",
"cleaned": "",
"type": "无效内容",
"sentiment": "无法判断",
"dimensions": "其他",
"reason": "无",
}
for line in block.split("\n"):
line = line.strip()
if not line:
continue
m = _CLEAN_FIELD_RE.match(line)
if not m:
continue
key, val = m.group(1), m.group(2).strip()
if key == "是否有效":
item["valid"] = val
elif key == "有效评论":
if val and val != "无":
item["cleaned"] = val
elif key == "评论类型":
item["type"] = val
elif key == "情感倾向":
item["sentiment"] = val
elif key == "涉及维度":
item["dimensions"] = val
elif key == "无效原因":
item["reason"] = val
items.append(item)
return items
def _trivial_prefilter(comments: list[str]) -> list[str]:
"""轻量预过滤:只去掉空、纯表情、重复,其余全留(给 LLM 判断)。"""
seen: set[str] = set()
kept: list[str] = []
for text in comments:
t = str(text or "").strip()
if not t or t in seen:
continue
if _is_pure_emoji(t):
continue
seen.add(t)
kept.append(t)
return kept
def _clean_one_batch(batch: list[str], batch_idx: int,
total_batches: int) -> tuple[int, list[dict]]:
"""清洗一批评论,返回 (batch_idx, results)。
并发安全:不写共享状态,只返回结果。
"""
lines = [f"评论编号：{i+1}\n评论内容：{text}" for i, text in enumerate(batch, 1)]
user_content = (f"请按规则清洗以下 {len(batch)} 条评论,逐条输出结构化结果。\n\n"
+ "\n\n".join(lines))
results: list[dict] = []
try:
response = call_hermes_analyzer(COMMENT_CLEAN_SYSTEM_PROMPT, user_content)
parsed = parse_cleaning_response(response)
for i, item in enumerate(parsed):
if i < len(batch):
item["original"] = batch[i]
if len(parsed) < len(batch):
for i in range(len(parsed), len(batch)):
results.append({
"original": batch[i],
"valid": "需要人工确认",
"cleaned": batch[i],
"type": "无效内容",
"sentiment": "无法判断",
"dimensions": "其他",
"reason": "LLM 响应未解析到该条",
})
results.extend(parsed)
except Exception as exc:
for text in batch:
results.append({
"original": text,
"valid": "需要人工确认",
"cleaned": text,
"type": "无效内容",
"sentiment": "无法判断",
"dimensions": "其他",
"reason": f"LLM 调用失败: {exc}",
})
return batch_idx, results
def llm_clean_comments_detailed(comments: list[str],
batch_size: int = 30,
max_workers: int = 6,
include_invalid: bool = False) -> list[dict]:
"""用 LLM 对评论做语义清洗(并发批处理)。
返回 [{original, valid, cleaned, type, sentiment, dimensions, reason}, ...]
默认只返回 valid != "无效" 的评论;传 include_invalid=True 返回全部。
max_workers: 并发批数(默认 6,Hermes 本地端可承受)
"""
if not comments:
return []
prefiltered = _trivial_prefilter(comments)
log(f"LLM 清洗:预过滤 {len(comments)} -> {len(prefiltered)} 条(去空/去重/去纯表情)")
total = len(prefiltered)
total_batches = (total + batch_size - 1) // batch_size
# 切分批次
batches: list[tuple[int, list[str]]] = []
for batch_idx in range(total_batches):
start = batch_idx * batch_size
end = min(start + batch_size, total)
batches.append((batch_idx, prefiltered[start:end]))
# 并发跑
results_by_idx: dict[int, list[dict]] = {}
completed = 0
with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
future_to_idx = {executor.submit(_clean_one_batch, batch, idx, total_batches): idx
for idx, batch in batches}
for future in concurrent.futures.as_completed(future_to_idx):
idx = future_to_idx[future]
try:
batch_idx, batch_results = future.result()
results_by_idx[batch_idx] = batch_results
except Exception as exc:
log(f" [WARN] 批次 {idx+1} 异常: {exc}")
results_by_idx[idx] = []
completed += 1
if completed % 3 == 0 or completed == total_batches:
log(f" 进度: {completed}/{total_batches} 批完成")
# 按原顺序合并
all_results: list[dict] = []
for idx in range(total_batches):
all_results.extend(results_by_idx.get(idx, []))
kept = [r for r in all_results if r.get("valid") != "无效"]
valid_count = sum(1 for r in all_results if r.get("valid") == "有效")
unsure_count = sum(1 for r in all_results if r.get("valid") == "需要人工确认")
invalid_count = sum(1 for r in all_results if r.get("valid") == "无效")
log(f"LLM 清洗完成: 共 {len(all_results)} 条 -> 有效 {valid_count} / "
f"需人工 {unsure_count} / 无效 {invalid_count}")
return all_results if include_invalid else kept
def llm_clean_comments(comments: list[str], batch_size: int = 30,
use_cleaned: bool = True, max_workers: int = 6) -> list[str]:
"""LLM 清洗评论,返回清洗后的评论文本列表(只含有效+需要人工确认)。
use_cleaned=True 返回清洗后内容,False 返回原文。"""
details = llm_clean_comments_detailed(comments, batch_size=batch_size,
max_workers=max_workers)
out: list[str] = []
for d in details:
text = d.get("cleaned") if use_cleaned else d.get("original")
if not text:
text = d.get("original", "")
if text:
out.append(text)
return out
# ---------------------------------------------------------------------------
# 舆情分析
# ---------------------------------------------------------------------------
def analyze_comments(comments: list[str]) -> str:
if not comments:
return "无有效评论可分析。"
log(f"调用 Hermes 分析端生成舆情汇总（共 {len(comments)} 条）...")
lines = [f"{i + 1}. {text}" for i, text in enumerate(comments)]
user_content = (
f"共有 {len(comments)} 条过滤后的有效评论，请根据角色设定输出完整的舆情分析汇总报告。\n\n"
+ "\n".join(lines)
)
return call_hermes_analyzer(ANALYSIS_SYSTEM_PROMPT, user_content)
# ---------------------------------------------------------------------------
# 输出 & 飞书通知
# ---------------------------------------------------------------------------
def write_summary(summary: str, path: Path) -> Path:
path.write_text(summary, encoding="utf-8-sig")
log(f"汇总报告已写入: {path}")
return path
def send_feishu_summary(summary: str, open_id: str) -> dict[str, Any]:
"""通过 lark-cli --profile hermes-analyzer 以分析端应用身份发送汇总。"""
text = summary
if len(text) > FEISHU_TEXT_LIMIT:
text = text[:FEISHU_TEXT_LIMIT] + "\n\n...（内容过长，已截断，完整内容见项目 data/summary 汇总文件）"
command = [
"lark-cli",
"--profile", "hermes-analyzer",
"im", "+messages-send",
"--user-id", open_id,
"--text", text,
"--as", "bot",
"--format", "json",
]
completed = subprocess.run(
command,
capture_output=True,
text=True,
encoding="utf-8",
errors="replace",
)
if completed.returncode != 0:
raise RuntimeError(
f"lark-cli 发送失败 (exit {completed.returncode}):\n"
f"stdout: {completed.stdout}\nstderr: {completed.stderr}"
)
try:
return json.loads(completed.stdout)
except json.JSONDecodeError:
return {"raw": completed.stdout}
# ---------------------------------------------------------------------------
# CLI
# ---------------------------------------------------------------------------
def main() -> int:
parser = argparse.ArgumentParser(description="评论 → Hermes 过滤/分析 → 桌面汇总 + 飞书通知")
parser.add_argument("--input", type=Path, default=DEFAULT_INPUT, help="原始评论文件路径")
parser.add_argument("--output", type=Path, default=OUTPUT_PATH, help="汇总输出文件路径")
parser.add_argument("--open-id", default=TARGET_OPEN_ID, help="飞书接收人 open_id")
parser.add_argument("--no-send", action="store_true", help="只生成汇总文件，不发送飞书")
parser.add_argument("--batch-by-style", action="store_true", help="CSV 按款式分批分析（用于项目汇总 CSV）")
parser.add_argument("--max-per-style", type=int, default=300, help="每款式最多分析多少条评论（防上下文超长）")
args = parser.parse_args()
args.output = resolve_layer_output(
args.output, layer_root=PATHS.exports_root, field="--output"
)
if not args.no_send:
if not HERMES_ANALYZER_TOKEN:
log("[ERROR] HERMES_ANALYZER_TOKEN 未设置。请配置 data/config/analyze.env 或传 --no-send")
return 1
if not args.open_id:
log("[ERROR] 飞书接收人 open_id 未设置。请配置 FEISHU_TARGET_OPEN_ID 或传 --open-id 或 --no-send")
return 1
if args.batch_by_style:
log(f"按款式分批读取 CSV: {args.input}")
try:
style_groups = load_style_groups(args.input, args.max_per_style)
except Exception as exc:
log(f"读取 CSV 分组失败: {exc}")
return 1
log(f"共 {len(style_groups)} 个款式")
if not style_groups:
log("没有读取到任何评论，流程结束")
return 0
style_results: list[tuple[str, str, int, int]] = []
for style, comments in style_groups.items():
try:
summary, raw_count, filtered_count = analyze_style(style, comments)
style_results.append((style, summary, raw_count, filtered_count))
except Exception as exc:
log(f"【{style}】处理失败: {exc}")
style_results.append((style, f"处理失败: {exc}", len(comments), 0))
summary = build_batch_report(style_results)
else:
log(f"读取原始评论文件: {args.input}")
try:
comments = load_comments(args.input)
except Exception as exc:
log(f"读取评论失败: {exc}")
return 1
log(f"读取到 {len(comments)} 条评论")
if not comments:
log("没有读取到任何评论，流程结束")
return 0
try:
filtered = code_filter_comments(comments)
except Exception as exc:
log(f"评论过滤失败: {exc}")
return 1
try:
summary = analyze_comments(filtered)
except Exception as exc:
log(f"舆情分析失败: {exc}")
return 1
try:
write_summary(summary, args.output)
except Exception as exc:
log(f"写入汇总文件失败: {exc}")
return 1
if args.no_send:
log("已指定 --no-send，跳过飞书发送")
else:
try:
result = send_feishu_summary(summary, args.open_id)
log(f"飞书消息发送成功: {result}")
except Exception as exc:
log(f"飞书消息发送失败（汇总文件仍已生成）: {exc}")
return 1
log("全部完成")
return 0
if __name__ == "__main__":
try:
raise SystemExit(main())
except KeyboardInterrupt:
print("\n已中断", file=sys.stderr)
raise SystemExit(130)
src/gyxx_flow/modules/content_marketing/runtime/data/tools/analyze_note.py
+868
View File
@@ -0,0 +1,868 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""单篇笔记分析报告生成器。
读取项目采集的单篇笔记 JSON（data/notes/{platform}/...json），
结合笔记标题、点赞/收藏/评论/分享等基础指标，以及评论内容，
生成本地代码过滤 + Hermes 舆情分析的单篇笔记分析报告。
输出保存到：data/summary/单篇笔记分析报告.txt
"""
import argparse
import json
import os
import subprocess
import sys
import time
from pathlib import Path
from typing import Any
from gyxx_flow.modules.content_marketing.runtime.runtime_paths import (
PATHS,
resolve_layer_output,
)
# Path setup
_TOOLS_DIR = PATHS.tools_root
_PROJECT_ROOT = PATHS.module_root
from gyxx_flow.modules.content_marketing.runtime.data.tools.analyze_comments import (
HERMES_ANALYZER_MODEL,
HERMES_ANALYZER_TOKEN,
HERMES_ANALYZER_URL,
FEISHU_TEXT_LIMIT,
TARGET_OPEN_ID,
BAG_CATEGORY_TERPS,
code_filter_comments,
llm_clean_comments,
llm_clean_comments_detailed,
classify_comment_keywords,
call_hermes_analyzer,
)
from gyxx_flow.modules.content_marketing.runtime.data.tools import db
PROJECT_ROOT = _PROJECT_ROOT
DATA_DIR = PATHS.raw_root
SUMMARY_DIR = PATHS.exports_root / "summary"
NOTES_DIR = PATHS.raw_root / "notes"
OUTPUT_PATH = SUMMARY_DIR / "单篇笔记分析报告.txt"
CATEGORY_LABELS = {
"purchase_intent": "购买意向",
"positive_feature": "正向功能反馈",
"negative_feature": "负向功能反馈",
"user_scenario": "用户场景",
"aesthetic": "审美与风格",
"brand_emotion": "品牌情感",
}
# 单篇笔记分析：送入 LLM 的有效评论上限
# 超过此数量时按关键词多样性抽样，保留最有信息量的评论
MAX_COMMENTS_FOR_LLM = 80
# ---------------------------------------------------------------------------
# 单篇笔记分析的 LLM 提示词
# ---------------------------------------------------------------------------
SINGLE_NOTE_PROMPT = """你是一位品牌投放复盘分析师。请基于以下单篇笔记的数据与评论区内容，按指定结构输出分析报告。
【笔记信息】
- 标题：{title}
- 平台：{platform}
- 品牌/产品：{brand} / {product}
- 点赞：{likes} | 收藏：{favs} | 评论：{comments} | 分享：{shares}
- 曝光/播放量：{views}（如有）
【有效评论列表】
{有效评论原文，逐条列出}
【输出要求】
请按以下7个模块输出结论，每个模块直接给出判断和建议，不重复提及数据来源或指标定义，引用评论用「」包裹原文：
模块1｜笔记表现归因
- 基于互动率结构（评论/收藏/转发）判断这条笔记的传播属性：是认知留存型、社交裂变型还是搜索截流型？
- 结合曝光量（如有）判断流量效率是否匹配投放预期。
- 一句话总结：这条笔记"成也何处，败也何处"。
模块2｜评论质量分层
- 将评论按"有效种草 / 连带兴趣 / 无效互动"三档归类，给出各档条数与占比。
- 无效互动的典型表现是什么？（如纯求链接、纯@好友、无意义表情）
- 有效评论中，高频关键词指向哪些用户决策阶段？（规格确认、价值判断、竞品对比、冲动下单、售后反馈）
模块3｜品牌安全与控评建议
- 评论区是否存在负面舆情集中点？若有，具体是什么问题？影响面多大？
- 是否出现竞品高频被提及？提及场景是"对比纠结"还是"被替代"？
- 给出具体的控评/引导话术建议，至少2条可挂置顶或评论区回复的原文。
模块4｜脚本有效性拆解
- 标题钩子是否有效？是吸引点击还是引发质疑？
- 卖点传达是否精准？评论区讨论的卖点与笔记主推卖点是否一致？如果偏离，偏离方向是什么？
- 用户是否按预期行动？（如搜索品牌词、进店铺、点击评论区链接等）
- 内容调性是否与品牌心智匹配？若匹配度不足，差异点在哪？
模块5｜达人匹配度评估
- 该达人的粉丝画像（性别、年龄、兴趣）与我方目标客群的重合度如何？
- 达人的内容风格（硬核测评/温柔种草/剧情演绎/颜值展示）是否适合承载本产品的核心卖点？
- 如果匹配度不足，是"换人"还是"改脚本"？具体建议。
模块6｜可沉淀的选题资产
- 从评论区提取至少2条可作为下一期内容选题的真实用户需求或疑问。
- 每条选题需附：内容方向+标题参考+核心脚本建议。
模块7｜下一篇笔记可执行优化清单
- 脚本层面（前3秒钩子、卖点顺序、画面设计、结尾引导）逐项给出可落地的修改点。
- 评论区运营层面（预埋评论、置顶引导、舆情应对）给出具体话术。
- 投放策略层面（达人选择、内容方向、流量助推）给出调整建议。
【输出格式要求】
- 不要出现"根据数据""数据显示""通过分析可知"等来源类表述。
- 直接给结论和建议，每条建议须具体可执行，避免"加强""优化"等空泛词汇。
- 如我方产品在笔记中被边缘化，须在模块4末尾明确加粗标注"⚠️ 我方产品边缘化，非核心种草"。
- 整体篇幅控制在1500-2000字，结论前置，层次分明。"""
def log(message: str) -> None:
print(f"[{time.strftime('%H:%M:%S')}] {message}", flush=True)
def detect_platform(source_url: str) -> str:
url = source_url.lower()
if "bilibili" in url or "b23.tv" in url or url.startswith("bv"):
return "bilibili"
if "douyin" in url:
return "douyin"
if "xiaohongshu" in url or "xhslink" in url:
return "xiaohongshu"
return "unknown"
def safe_div(numerator: Any, denominator: Any) -> float:
try:
n = float(numerator)
d = float(denominator)
if d == 0:
return 0.0
return n / d
except (TypeError, ValueError):
return 0.0
def load_note_json(path: Path) -> dict[str, Any]:
if not path.exists():
raise FileNotFoundError(f"找不到文件: {path}")
for enc in ("utf-8-sig", "utf-8", "gbk"):
try:
data = json.loads(path.read_text(encoding=enc))
break
except UnicodeDecodeError:
continue
except json.JSONDecodeError:
raise
else:
data = json.loads(path.read_text(encoding="utf-8", errors="replace"))
if not isinstance(data, dict):
raise ValueError(f"不支持的 JSON 结构: {path}")
return data
def extract_note_info(data: dict[str, Any]) -> dict[str, Any]:
"""从笔记 JSON 中提取标题、URL、指标等。"""
source_url = data.get("source_url") or data.get("final_url") or ""
platform = detect_platform(source_url)
# 标题：优先顶层 title，兼容 bilibili 旧数据的 video.title
title = data.get("title") or ""
if not title and isinstance(data.get("video"), dict):
title = data["video"].get("title", "")
stats = data.get("stats") or {}
metrics = stats.get("note_metrics") or {}
return {
"title": title,
"platform": platform,
"source_url": source_url,
"scraped_at": data.get("scraped_at", ""),
"comment_count_total": data.get("count", 0),
"like_count": metrics.get("like_count", ""),
"favorite_count": metrics.get("favorite_count", ""),
"comment_count_metric": metrics.get("comment_count", ""),
"share_count": metrics.get("share_count", ""),
"view_count": metrics.get("view_count", ""),
"collect_count_yxyy": metrics.get("collect_count", ""),
"style_name": "",
"creator_name": "",
}
def extract_comment_texts(data: dict[str, Any]) -> list[str]:
"""从笔记 JSON 中提取评论文本列表。"""
comments = data.get("comments") or []
texts: list[str] = []
for c in comments:
if not isinstance(c, dict):
continue
for key in ("content", "text", "message"):
val = c.get(key)
if val is not None:
text = str(val).strip()
if text:
texts.append(text)
break
return texts
def load_note_from_db(note_id: int | None = None, url: str | None = None) -> tuple[dict[str, Any], list[str]]:
"""从数据库读取笔记信息和评论文本。
返回 (note_info_dict, comment_texts_list)。
只查 cmt_notes（含 cmt_styles / cmt_creators JOIN）+ cmt_comments。
"""
if note_id is None and not url:
raise ValueError("必须提供 note_id 或 url")
if note_id is not None:
note_row = db.get_note_by_id(note_id)
else:
note_row = db.get_note_by_url(url)
if not note_row:
raise ValueError(f"数据库中找不到笔记: note_id={note_id}, url={url}")
note_id = note_row["id"]
comment_rows = db.get_comments_by_note_id(note_id)
info = {
"title": note_row.get("title") or "",
"platform": note_row.get("platform") or "",
"source_url": note_row.get("url") or "",
"scraped_at": note_row.get("scraped_at").isoformat(timespec="seconds") if note_row.get("scraped_at") else "",
"comment_count_total": len(comment_rows),
"like_count": note_row.get("like_count", ""),
"favorite_count": note_row.get("favorite_count", ""),
"comment_count_metric": note_row.get("comment_count", ""),
"share_count": note_row.get("share_count", ""),
"view_count": note_row.get("view_count", ""),
"collect_count_yxyy": note_row.get("collect_count", ""),
"style_name": note_row.get("style_name") or "",
"creator_name": note_row.get("creator_name") or "",
}
texts = [str(row.get("content") or "").strip() for row in comment_rows if str(row.get("content") or "").strip()]
return info, texts
def _to_number(value: Any) -> float:
try:
if value is None or value == "":
return 0.0
return float(value)
except (TypeError, ValueError):
return 0.0
def compute_metrics(info: dict[str, Any]) -> dict[str, Any]:
"""计算互动率等衍生指标。所有互动率统一以「曝光量」为分母，避免多口径混用。"""
likes = info["like_count"]
favorites = info["favorite_count"]
comments = info["comment_count_metric"] or info["comment_count_total"]
shares = info["share_count"]
views_raw = info.get("view_count")
likes_num = _to_number(likes)
favorites_num = _to_number(favorites)
comments_num = _to_number(comments)
shares_num = _to_number(shares)
views_num = _to_number(views_raw)
return {
"view_count": views_num,
"like_rate": safe_div(likes_num, views_num), # 点赞率
"comment_rate": safe_div(comments_num, views_num), # 评论率（评论/曝光，全报告唯一口径）
"favorite_rate": safe_div(favorites_num, views_num), # 收藏率
"share_rate": safe_div(shares_num, views_num), # 分享率
"engagement_rate": safe_div(likes_num + comments_num + favorites_num + shares_num, views_num), # 综合互动率
"favorite_to_comment": safe_div(favorites_num, comments_num), # 收藏评论比（独立口径）
}
def keyword_distribution(comments: list[str], target_terms: list[str] | None = None) -> dict[str, Any]:
"""统计关键词命中分布，区分「全评论」与「我方产品相关」。
target_terms：品牌名/款式名列表；提供时把命中文本先筛出再统计，得到「我方产品相关」分布。
匹配规则：评论中出现品牌名/款式名，或出现 BAG_CATEGORY_TERPS 中的包包品类词，均视为"我方产品相关"。
返回:
{
"overall": {category: count, ...}, # 全评论 6 类命中数
"relevant": {category: count, ...}, # 我方产品相关子集的 6 类命中数
"relevant_total": int, # 我方产品相关评论条数
}
"""
terms = [t for t in (target_terms or []) if t]
bag_terms = BAG_CATEGORY_TERPS
overall: dict[str, int] = {key: 0 for key in CATEGORY_LABELS}
relevant: dict[str, int] = {key: 0 for key in CATEGORY_LABELS}
relevant_total = 0
for text in comments:
cats = classify_comment_keywords(text)
for c in cats:
overall[c] += 1
is_relevant = False
if terms and any(t in text for t in terms):
is_relevant = True
elif any(t in text for t in bag_terms):
is_relevant = True
if is_relevant:
relevant_total += 1
for c in cats:
relevant[c] += 1
return {
"overall": overall,
"relevant": relevant,
"relevant_total": relevant_total,
}
def _brand_block(brand: str, style_name: str, target_term: str) -> str:
if not (brand or style_name):
return ""
return f"""
【实体隔离】
本次分析的目标：品牌「{brand or '未指定'}」、产品「{style_name or '未指定'}」。本笔记可能涉及多品牌联合推广。
请严格将分析对象限定为我方品牌/产品：
- 仅将「{target_term}」相关的信息计入种草指标
- 笔记/评论中涉及的其他品牌或产品，仅作为场景搭配或联合推广的陪衬
- 评论区数据先按以下三类分类再纳入分析：
· 有效种草：明确提及「{target_term}」的卖点/外观/体验，或询问其价格/链接
· 连带兴趣：既讨论了「{target_term}」也讨论了其他产品
· 无效/他品互动：仅讨论其他产品或完全无关闲聊
- 情感倾向、痛点需求、卖点提取，仅基于「有效种草」与「连带兴趣」生成；「无效/他品互动」不计入种草指标
- 关键词分布按「全评论」与「我方产品相关」两个口径分别统计
"""
def _data_table_markdown(info: dict[str, Any], metrics: dict[str, Any], raw_total: int, filtered_total: int) -> str:
"""构造数据表 markdown（用于提示词 / 报告 / 卡片）。"""
has_view = bool(metrics.get("view_count"))
def pct(v: float) -> str:
return f"{v * 100:.2f}%" if v else "—"
def num(v: Any) -> str:
if v in ("", None):
return "—"
try:
return f"{int(float(v)):,}"
except (TypeError, ValueError):
return str(v)
likes = info.get("like_count")
favorites = info.get("favorite_count")
comments = info.get("comment_count_metric") or info.get("comment_count_total")
shares = info.get("share_count")
valid_ratio = safe_div(filtered_total, raw_total)
lines = [
"| 维度 | 指标 | 数值 |",
"| --- | --- | --- |",
f"| 基础 | 点赞 | {num(likes)} |",
f"| 基础 | 收藏 | {num(favorites)} |",
f"| 基础 | 评论 | {num(comments)} |",
f"| 基础 | 分享 | {num(shares)} |",
f"| 传播 | 曝光量 | {num(metrics.get('view_count')) if has_view else '未采集'} |",
f"| 互动率 | 点赞率 | {pct(metrics.get('like_rate', 0))} |",
f"| 互动率 | 评论率 | {pct(metrics.get('comment_rate', 0))} |",
f"| 互动率 | 收藏率 | {pct(metrics.get('favorite_rate', 0))} |",
f"| 互动率 | 分享率 | {pct(metrics.get('share_rate', 0))} |",
f"| 互动率 | 综合互动率 | {pct(metrics.get('engagement_rate', 0))} |",
f"| 互动率 | 收藏评论比 | {metrics.get('favorite_to_comment', 0):.2f} |",
f"| 质量 | 有效率 | {pct(valid_ratio)} |",
]
return "\n".join(lines)
def _keyword_distribution_markdown(
distribution: dict[str, Any],
total_filtered: int,
) -> str:
"""构造关键词分布 markdown，含「全评论」与「我方产品相关」两段。"""
overall = distribution.get("overall", {}) if isinstance(distribution, dict) else {}
relevant = distribution.get("relevant", {}) if isinstance(distribution, dict) else {}
relevant_total = distribution.get("relevant_total", 0) if isinstance(distribution, dict) else 0
if not overall:
return "（无关键词分布数据）"
def fmt(d: dict[str, int]) -> str:
return "\n".join(f"- {CATEGORY_LABELS[k]}：{d.get(k, 0)} 条" for k in CATEGORY_LABELS)
lines = [
f"### 全评论（{total_filtered} 条）",
fmt(overall),
"",
f"### 我方产品相关（{relevant_total} 条）",
fmt(relevant) if relevant_total else "_（无评论明确提及我方产品/品牌）_",
]
return "\n".join(lines)
def _sample_comments(comments: list[str], max_count: int) -> list[str]:
"""当评论数超过上限时，按关键词多样性抽样保留最有信息量的评论。
策略：优先保留命中更多关键词类别的评论，同类别数时保留更长的评论。
"""
scored: list[tuple[int, int, int, str]] = [] # (负类别数, 负长度, 原始索引, 文本)
for i, text in enumerate(comments):
cats = classify_comment_keywords(text)
scored.append((-len(cats), -len(text), i, text))
scored.sort()
return [text for _, _, _, text in scored[:max_count]]
def build_note_analysis_prompt(
info: dict[str, Any],
metrics: dict[str, Any],
distribution: dict[str, Any],
comments: list[str],
) -> str:
"""构造 LLM 输入的 user_content。"""
brand = info.get("brand") or ""
style_name = info.get("style_name") or ""
target_label = " / ".join(x for x in (brand, style_name) if x) if (brand or style_name) else "未指定"
target_term = brand or style_name
# 评论数量截断：超过上限时按关键词多样性抽样
total_count = len(comments)
if total_count > MAX_COMMENTS_FOR_LLM:
comments = _sample_comments(comments, MAX_COMMENTS_FOR_LLM)
lines: list[str] = []
if total_count > MAX_COMMENTS_FOR_LLM:
lines.append(f"共有 {total_count} 条过滤后的有效评论（已抽样 {len(comments)} 条），请基于以下单篇笔记的信息和评论内容，按要求输出分析报告。")
else:
lines.append(f"共有 {len(comments)} 条过滤后的有效评论，请基于以下单篇笔记的信息和评论内容，按要求输出分析报告。")
lines.extend([
"",
"## 单篇笔记基础信息",
f"- 标题：{info['title'] or '未获取'}",
f"- 平台：{info['platform']}",
f"- 链接：{info['source_url']}",
f"- 品牌/产品：{target_label}",
f"- 点赞数：{info['like_count']}",
f"- 收藏数：{info['favorite_count']}",
f"- 评论数：{info['comment_count_metric'] or info['comment_count_total']}",
f"- 分享数：{info['share_count']}",
"",
])
if brand or style_name:
lines.extend([
"## 实体隔离规则（产品隔离）",
f"本次分析的目标：品牌「{brand}」、产品「{style_name}」。本笔记可能涉及多品牌联合推广，请严格将分析对象限定为我方品牌/产品。",
f"- 仅将「{target_term}」相关的信息计入种草指标",
"- 笔记/评论中涉及的其他品牌或产品，仅作为场景搭配或联合推广的陪衬",
"- 评论区按以下三类分类后再纳入分析：",
f" · 有效种草：明确提及「{target_term}」的卖点/外观/使用体验，或询问其价格/链接",
f" · 连带兴趣：既讨论了「{target_term}」也讨论了其他产品",
" · 无效/他品互动：仅讨论其他产品或完全无关闲聊",
"- 情感倾向、痛点需求、卖点提取，仅基于「有效种草」与「连带兴趣」生成；「无效/他品互动」不计入种草指标",
"",
])
lines.extend([
"## 数据表（已计算好，可直接引用）",
_data_table_markdown(info, metrics, info.get("comment_count_total", 0), len(comments)),
"",
"## 评论关键词分布（按口径分别给出）",
_keyword_distribution_markdown(distribution, len(comments)),
"",
"## 有效评论内容",
])
for text in comments:
lines.append(f"- {text}")
return "\n".join(lines)
def analyze_note(
info: dict[str, Any],
metrics: dict[str, Any],
distribution: dict[str, Any],
comments: list[str],
) -> str:
"""调用 Hermes 生成单篇笔记分析。"""
if not comments:
return "该笔记无有效评论可分析。"
log(f"调用 Hermes 生成分析报告（{info['title'] or info['source_url']}，{len(comments)} 条有效评论）...")
user_content = build_note_analysis_prompt(info, metrics, distribution, comments)
brand = info.get("brand") or ""
style_name = info.get("style_name") or ""
target_term = brand or style_name
brand_block = _brand_block(brand, style_name, target_term)
system_prompt = SINGLE_NOTE_PROMPT + brand_block
return call_hermes_analyzer(system_prompt, user_content)
def build_report(
info: dict[str, Any],
metrics: dict[str, Any],
distribution: dict[str, Any],
raw_comments: list[str],
filtered_comments: list[str],
analysis: str,
) -> str:
"""构造本地报告文件。"""
return _build_report_concise(info, metrics, distribution, raw_comments, filtered_comments, analysis)
def _build_report_concise(
info: dict[str, Any],
metrics: dict[str, Any],
distribution: dict[str, Any],
raw_comments: list[str],
filtered_comments: list[str],
analysis: str,
) -> str:
brand = info.get("brand") or ""
style_name = info.get("style_name") or ""
target_label = " / ".join(x for x in (brand, style_name) if x) if (brand or style_name) else "未指定"
lines = [
"# 单篇笔记分析报告（投放复盘版）",
"",
f"**标题**：{info['title'] or '未获取'}",
f"**平台**：{info['platform']}",
f"**品牌/产品**：{target_label}",
f"**达人**：{info.get('creator_name') or '未指定'}",
f"**链接**：{info['source_url']}",
"",
"## 一、数据表",
_data_table_markdown(info, metrics, len(raw_comments), len(filtered_comments)),
"",
"## 二、关键词分布",
_keyword_distribution_markdown(distribution, len(filtered_comments)),
"",
"## 三、投放复盘",
analysis,
]
return "\n".join(lines)
def write_report(report: str, path: Path) -> Path:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(report, encoding="utf-8-sig")
log(f"报告已写入: {path}")
return path
def _md_field(label: str, value: str, is_short: bool = True) -> dict[str, Any]:
return {
"is_short": is_short,
"text": {"tag": "lark_md", "content": f"**{label}**\n{value}"},
}
def _section_header(title: str) -> dict[str, Any]:
return {
"tag": "div",
"text": {"tag": "lark_md", "content": f"**{title}**"},
}
def build_card_payload(
info: dict[str, Any],
metrics: dict[str, Any],
distribution: dict[str, Any],
raw_comments: list[str],
filtered_comments: list[str],
analysis: str,
report_path: Path,
) -> dict[str, Any]:
"""构造飞书 interactive 卡片 JSON。
布局（精简版默认）：
header（蓝色）+ 基本信息 + 数据表 + 关键词分布 + 投放复盘 + 报告路径
"""
brand = info.get("brand") or ""
style_name = info.get("style_name") or ""
creator_name = info.get("creator_name") or ""
target_label = " / ".join(x for x in (brand, style_name) if x) if (brand or style_name) else "未指定"
title = info.get("title") or "未获取"
platform = info.get("platform") or "unknown"
source_url = info.get("source_url") or ""
elements: list[dict[str, Any]] = []
# 基本信息
elements.append({
"tag": "div",
"fields": [
_md_field("标题", title, is_short=False),
_md_field("品牌 / 产品", target_label),
_md_field("平台", platform),
_md_field("达人", creator_name or "未指定"),
],
})
if source_url:
elements.append({
"tag": "div",
"text": {"tag": "lark_md", "content": f"🔗 [打开笔记原文]({source_url})"},
})
# 数据表
elements.append({"tag": "hr"})
elements.append(_section_header("📋 数据表"))
elements.append({
"tag": "div",
"text": {"tag": "lark_md", "content": _data_table_markdown(info, metrics, len(raw_comments), len(filtered_comments))},
})
# 关键词分布（拆分两个口径）
elements.append(_section_header("🔑 关键词分布"))
elements.append({
"tag": "div",
"text": {"tag": "lark_md", "content": _keyword_distribution_markdown(distribution, len(filtered_comments))},
})
# 投放复盘（LLM 输出）
elements.append(_section_header("💡 投放复盘"))
analysis_text = analysis or "（无分析结果）"
if len(analysis_text) > 4500:
analysis_text = analysis_text[:4500] + "\n\n...（内容过长，已截断，完整分析见报告文件）"
elements.append({
"tag": "div",
"text": {"tag": "lark_md", "content": analysis_text},
})
return {
"config": {"wide_screen_mode": True},
"header": {
"template": "blue",
"title": {
"tag": "plain_text",
"content": "📊 单篇笔记分析报告",
},
},
"elements": elements,
}
def _fmt_int(value: Any) -> str:
if value is None or value == "":
return "—"
try:
return f"{int(float(value)):,}"
except (TypeError, ValueError):
return str(value)
def send_feishu_report(
info: dict[str, Any],
metrics: dict[str, Any],
distribution: dict[str, int],
raw_comments: list[str],
filtered_comments: list[str],
analysis: str,
open_id: str,
report_path: Path,
) -> dict[str, Any]:
"""通过 lark-cli 以飞书 interactive 卡片格式发送报告。
卡片包含：标题/平台/达人基础信息、6 个分节指标 + LLM 舆情洞察。
Windows 下直接走 lark-cli.cmd 会撞到命令行长度上限且对 &|<>^% 转义敏感，
改为调 node + run.js，args 用 list 传，由 node 自己处理。
"""
card = build_card_payload(
info, metrics, distribution,
raw_comments, filtered_comments, analysis,
report_path,
)
content_json = json.dumps(card, ensure_ascii=False)
if sys.platform.startswith("win"):
# Windows 下走 lark-cli.cmd 会撞到命令行长度上限（~8K）且对 &|<>^% 转义敏感；
# 直接调 node + run.js，args 用 list 传，由 node 自己处理，避开所有 cmd 转义问题。
run_js = (
Path(os.environ.get("APPDATA", str(Path.home())))
/ "npm"
/ "node_modules"
/ "@larksuite"
/ "cli"
/ "scripts"
/ "run.js"
)
if not run_js.exists():
raise RuntimeError(f"找不到 lark-cli 入口: {run_js}")
command = [
"node", str(run_js),
"--profile", "hermes-analyzer",
"im", "+messages-send",
"--user-id", open_id,
"--content", content_json,
"--msg-type", "interactive",
"--as", "bot",
"--format", "json",
]
completed = subprocess.run(
command,
capture_output=True,
text=True,
encoding="utf-8",
errors="replace",
)
stdout = completed.stdout
stderr = completed.stderr
else:
command = [
"lark-cli",
"--profile", "hermes-analyzer",
"im", "+messages-send",
"--user-id", open_id,
"--content", content_json,
"--msg-type", "interactive",
"--as", "bot",
"--format", "json",
]
completed = subprocess.run(
command,
capture_output=True,
text=True,
encoding="utf-8",
errors="replace",
)
stdout = completed.stdout
stderr = completed.stderr
if completed.returncode != 0:
raise RuntimeError(
f"lark-cli 发送失败 (exit {completed.returncode}):\n"
f"stdout: {stdout}\nstderr: {stderr}"
)
try:
return json.loads(stdout)
except json.JSONDecodeError:
return {"raw": stdout}
def main() -> int:
parser = argparse.ArgumentParser(description="单篇笔记 → 本地过滤 + Hermes 分析 → 项目 data/summary 报告")
source_group = parser.add_mutually_exclusive_group(required=True)
source_group.add_argument("--input", type=Path, help="单篇笔记 JSON 文件路径")
source_group.add_argument("--note-id", type=int, help="数据库 cmt_notes.id")
source_group.add_argument("--url", type=str, help="数据库 cmt_notes.url")
parser.add_argument("--output", type=Path, default=OUTPUT_PATH, help="报告输出路径")
parser.add_argument("--open-id", default=TARGET_OPEN_ID, help="飞书接收人 open_id")
parser.add_argument("--brand", default="", help="我方品牌名（用于实体隔离/产品隔离规则；不传则只用款式名）")
parser.add_argument("--no-send", action="store_true", help="只生成报告文件，不发送飞书")
parser.add_argument("--llm-filter", action="store_true",
help="用 LLM 语义清洗评论(替代关键词过滤,保留更多有效评论,但慢)")
parser.add_argument("--batch-size", type=int, default=30,
help="LLM 清洗的批大小(默认 30 条/批)")
parser.add_argument("--max-workers", type=int, default=6,
help="LLM 清洗的并发批数(默认 6)")
args = parser.parse_args()
args.output = resolve_layer_output(
args.output, layer_root=PATHS.exports_root, field="--output"
)
if not args.no_send:
if not HERMES_ANALYZER_TOKEN:
log("[ERROR] HERMES_ANALYZER_TOKEN 未设置。请配置 data/config/analyze.env 或传 --no-send")
return 1
if not args.open_id:
log("[ERROR] 飞书接收人 open_id 未设置。请配置 FEISHU_TARGET_OPEN_ID 或传 --open-id 或 --no-send")
return 1
if args.input:
log(f"读取单篇笔记 JSON: {args.input}")
try:
data = load_note_json(args.input)
except Exception as exc:
log(f"读取失败: {exc}")
return 1
info = extract_note_info(data)
raw_comments = extract_comment_texts(data)
elif args.note_id or args.url:
try:
info, raw_comments = load_note_from_db(note_id=args.note_id, url=args.url)
except Exception as exc:
log(f"从数据库读取失败: {exc}")
return 1
else:
log("必须提供 --input、--note-id 或 --url")
return 1
log(f"笔记标题: {info['title'] or '未获取'} | 平台: {info['platform']} | 总评论: {info['comment_count_total']}")
if not raw_comments:
# 无评论只代表评论洞察不可用，不应让整篇笔记及其曝光/互动指标
# 从周汇总中消失。后续会生成数据型报告，并明确标注无评论。
log("该笔记没有评论内容，跳过评论洞察并继续生成数据型报告")
# 先把 brand 注入到 info，再算关键词分布（需要 target_terms 拆分相关/全部）
if args.brand:
info["brand"] = args.brand
else:
info.setdefault("brand", "")
metrics = compute_metrics(info)
try:
if args.llm_filter:
log(f"使用 LLM 语义清洗评论(批大小 {args.batch_size}, 并发 {args.max_workers})...")
filtered_comments = llm_clean_comments(raw_comments,
batch_size=args.batch_size,
max_workers=args.max_workers)
else:
filtered_comments = code_filter_comments(raw_comments)
except Exception as exc:
log(f"评论过滤失败: {exc}")
return 1
target_terms = [t for t in (info.get("brand"), info.get("style_name")) if t]
distribution = keyword_distribution(filtered_comments, target_terms=target_terms)
log(f"关键词分布: 全评论 {len(filtered_comments)} 条，我方产品相关 {distribution['relevant_total']} 条")
try:
analysis = analyze_note(info, metrics, distribution, filtered_comments)
except Exception as exc:
log(f"笔记分析失败: {exc}")
return 1
report = build_report(info, metrics, distribution, raw_comments, filtered_comments, analysis)
try:
write_report(report, args.output)
except Exception as exc:
log(f"写入报告失败: {exc}")
return 1
if args.no_send:
log("已指定 --no-send，跳过飞书发送")
else:
# 卡片永远走精简版（给老板/团队 30 秒看完的复盘视图）
try:
result = send_feishu_report(
info, metrics, distribution,
raw_comments, filtered_comments, analysis,
args.open_id, args.output,
)
log(f"飞书卡片发送成功: {result}")
except Exception as exc:
log(f"飞书消息发送失败（报告文件仍已生成）: {exc}")
return 1
log("全部完成")
return 0
if __name__ == "__main__":
try:
raise SystemExit(main())
except KeyboardInterrupt:
print("\n已中断", file=sys.stderr)
raise SystemExit(130)
src/gyxx_flow/modules/content_marketing/runtime/data/tools/batch_rescrape_bilibili.py
+204
View File
@@ -0,0 +1,204 @@
"""Batch re-scrape all Bilibili videos from DB using the browser scraper.
Updates:
- note_metrics (like / favorite / comment / share counts, title)
- cmt_comments table (replace existing)
- JSON/CSV files in data/notes/bilibili/
"""
import subprocess
import sys
import time
from pathlib import Path
from gyxx_flow.modules.content_marketing.runtime.runtime_paths import PATHS
_TOOLS_DIR = PATHS.tools_root
_PROJECT_ROOT = PATHS.module_root
from gyxx_flow.modules.content_marketing.runtime.data.tools import db
from gyxx_flow.modules.content_marketing.runtime import bilibili_comment_scraper as scraper
RELOGIN_SCRIPT = _TOOLS_DIR / "relogin_bilibili.py"
try:
sys.stdout.reconfigure(encoding="utf-8", errors="replace") # type: ignore[attr-defined]
sys.stderr.reconfigure(encoding="utf-8", errors="replace") # type: ignore[attr-defined]
except Exception:
pass
def log(msg: str) -> None:
print(f"[{time.strftime('%H:%M:%S')}] {msg}", flush=True)
def collect_urls() -> list[dict]:
with db.get_dict_cursor() as cur:
cur.execute(
"""
SELECT id, url, title
FROM cmt_notes
WHERE platform = 'bilibili' AND url IS NOT NULL AND url != ''
ORDER BY id
"""
)
return cur.fetchall()
def upsert_metrics(note_id: int, title: str, metrics: dict) -> None:
db.update_note_metrics(
note_id,
like_count=metrics.get("like_count") or None,
favorite_count=metrics.get("favorite_count") or None,
comment_count=metrics.get("comment_count") or None,
share_count=metrics.get("share_count") or None,
title=title or None,
)
def _to_int(value) -> int | None:
try:
if value in (None, ""):
return None
return int(float(value))
except (TypeError, ValueError):
return None
def comment_replace_policy(comments: list[dict], stats: dict, metrics: dict) -> tuple[bool, bool, str]:
"""Return (should_replace, allow_empty, reason)."""
logged_in = bool(stats.get("logged_in"))
expected = _to_int(metrics.get("comment_count"))
if expected is None:
expected = _to_int(stats.get("api_total"))
if comments:
if not logged_in and (expected is None or expected > len(comments)):
return False, False, "未登录且可能只采到热门评论"
return True, False, ""
if expected == 0:
return True, True, "平台显示 0 评论"
return False, False, "空评论且无法确认平台为 0 评论"
def do_relogin() -> bool:
"""Run relogin_bilibili.py (headed, waits for QR scan). Returns True if successful."""
log(" Cookie 可能过期,触发自动重登 (请扫码)...")
try:
rc = subprocess.call(
[sys.executable, str(RELOGIN_SCRIPT)],
cwd=str(_PROJECT_ROOT),
)
if rc == 0:
log(" 重登成功,继续采集")
return True
else:
log(f" 重登失败 (exit={rc})")
return False
except Exception as exc:
log(f" 重登脚本执行异常: {exc}")
return False
def main(only_ids: list[int] | None = None, dry_run: bool = False) -> int:
rows = collect_urls()
if only_ids:
rows = [r for r in rows if r["id"] in set(only_ids)]
log(f"Total bilibili notes to scrape: {len(rows)}")
ok = 0
fail = 0
total_comments = 0
consecutive_fail = 0
MAX_CONSECUTIVE_FAIL = 3
relogin_used = 0
MAX_RELOGINS = 1
t0 = time.time()
for i, row in enumerate(rows, 1):
note_id = row["id"]
url = row["url"]
log(f"[{i}/{len(rows)}] id={note_id} url={url}")
comments = None
result = None
for attempt in range(3):
try:
comments, result = scraper.scrape_comments(
url, max_pages=200, delay=0.25, headless=True,
max_reply_pages=200, login_timeout=0,
)
break
except Exception as exc:
if attempt < 2 and "has been closed" in str(exc):
log(f" Browser closed, retrying ({attempt+2}/3)...")
time.sleep(3)
else:
log(f" ERROR: {exc}")
fail += 1
break
if comments is None:
consecutive_fail += 1
if consecutive_fail >= MAX_CONSECUTIVE_FAIL and relogin_used < MAX_RELOGINS:
if do_relogin():
relogin_used += 1
consecutive_fail = 0
continue
continue
consecutive_fail = 0
video = result.get("video", {}) or {}
title = video.get("title", "") or result.get("title", "")
stats = result.get("stats", {}) or {}
metrics = stats.get("note_metrics", {}) or {}
logged_in = bool(stats.get("logged_in"))
json_path, csv_path = scraper.write_outputs(url, comments, result)
top = len([c for c in comments if c["level"] == "comment"])
replies = len([c for c in comments if c["level"] == "reply"])
warn = "" if logged_in else " [WARN] 未登录,只采到热门评论"
log(
f" ok: title={title[:40]!r} | {len(comments)} comments ({top}+{replies}) | "
f"likes={metrics.get('like_count')}, favs={metrics.get('favorite_count')}, "
f"cmts={metrics.get('comment_count')}"
+ warn
)
if not dry_run:
try:
upsert_metrics(note_id, title, metrics)
except Exception as exc:
log(f" DB metric update error: {exc}")
try:
should_replace, allow_empty, reason = comment_replace_policy(comments, stats, metrics)
if should_replace:
db.replace_comments(note_id, "bilibili", comments, allow_empty=allow_empty)
else:
log(f" skip DB comment replace: {reason}")
except Exception as exc:
log(f" DB comment replace error: {exc}")
ok += 1
total_comments += len(comments)
elapsed = time.time() - t0
log(
f"DONE: {ok} ok, {fail} fail, {total_comments} comments total, "
f"{elapsed:.1f}s ({elapsed / max(ok, 1):.1f}s per note)"
)
return 0 if fail == 0 else 1
if __name__ == "__main__":
args = sys.argv[1:]
only_ids = []
dry_run = False
for a in args:
if a == "--dry-run":
dry_run = True
elif a.isdigit():
only_ids.append(int(a))
if only_ids:
log(f"Restricted to ids: {only_ids}")
if dry_run:
log("DRY RUN: will not touch DB")
raise SystemExit(main(only_ids or None, dry_run))
src/gyxx_flow/modules/content_marketing/runtime/data/tools/batch_rescrape_douyin.py
+252
View File
@@ -0,0 +1,252 @@
"""Batch re-scrape all Douyin notes from DB using the browser-based scraper.
Updates:
- note_metrics (like / favorite / comment / share counts)
- cmt_comments table (replace existing)
- JSON/CSV files in data/notes/douyin/
"""
import subprocess
import sys
import time
from pathlib import Path
from gyxx_flow.modules.content_marketing.runtime.runtime_paths import PATHS
# Path setup: db.py is in data/tools/, scrapers are in project root
_TOOLS_DIR = PATHS.tools_root
_PROJECT_ROOT = PATHS.module_root
from gyxx_flow.modules.content_marketing.runtime.data.tools import db
from gyxx_flow.modules.content_marketing.runtime import douyin_comment_scraper as scraper
RELOGIN_SCRIPT = _TOOLS_DIR / "relogin_douyin.py"
try:
sys.stdout.reconfigure(encoding="utf-8", errors="replace") # type: ignore[attr-defined]
sys.stderr.reconfigure(encoding="utf-8", errors="replace") # type: ignore[attr-defined]
except Exception:
pass
def log(msg: str) -> None:
print(f"[{time.strftime('%H:%M:%S')}] {msg}", flush=True)
def collect_urls() -> list[dict]:
with db.get_dict_cursor() as cur:
cur.execute(
"""
SELECT id, url, title, comment_count
FROM cmt_notes
WHERE platform = 'douyin' AND url IS NOT NULL AND url != ''
ORDER BY id
"""
)
rows = cur.fetchall()
fast = [r for r in rows if (r.get("comment_count") or 0) <= 200]
slow = [r for r in rows if (r.get("comment_count") or 0) > 200]
if fast:
log(f"Sorted: {len(fast)} notes ≤200 comments (fast first), {len(slow)} notes >200 (slow)")
return fast + slow
def upsert_metrics(note_id: int, metrics: dict) -> None:
db.update_note_metrics(
note_id,
like_count=metrics.get("like_count") or None,
favorite_count=metrics.get("favorite_count") or None,
comment_count=metrics.get("comment_count") or None,
share_count=metrics.get("share_count") or None,
title=metrics.get("title") or None,
)
def _to_int(value) -> int | None:
try:
if value in (None, ""):
return None
return int(float(value))
except (TypeError, ValueError):
return None
def comment_replace_policy(comments: list[dict], result: dict, metrics: dict) -> tuple[bool, bool, str]:
"""Return (should_replace, allow_empty, reason)."""
stats = result.get("stats", {}) if isinstance(result, dict) else {}
logged_in = bool(stats.get("logged_in"))
expected = _to_int(stats.get("api_total"))
if expected is None:
expected = _to_int(metrics.get("comment_count"))
if comments:
if not logged_in and (expected is None or expected > len(comments)):
return False, False, "未登录且可能只采到部分评论"
return True, False, ""
if expected == 0:
return True, True, "平台显示 0 评论"
return False, False, "空评论且无法确认平台为 0 评论"
def result_logged_in(result: dict | None) -> bool:
"""Read the login flag from the scraper result's nested stats payload."""
if not isinstance(result, dict):
return False
nested_stats = result.get("stats")
return bool(nested_stats.get("logged_in")) if isinstance(nested_stats, dict) else False
def do_relogin() -> bool:
"""Run relogin_douyin.py (headed, waits for QR scan). Returns True if successful."""
log(" Cookie 可能过期,触发自动重登 (请扫码)...")
try:
rc = subprocess.call(
[sys.executable, str(RELOGIN_SCRIPT)],
cwd=str(_PROJECT_ROOT),
)
if rc == 0:
log(" 重登成功,继续采集")
return True
else:
log(f" 重登失败 (exit={rc})")
return False
except Exception as exc:
log(f" 重登脚本执行异常: {exc}")
return False
def main(only_ids: list[int] | None = None, dry_run: bool = False) -> int:
rows = collect_urls()
if only_ids:
rows = [r for r in rows if r["id"] in set(only_ids)]
log(f"Total douyin notes to scrape: {len(rows)}")
ok = 0
fail = 0
total_comments = 0
consecutive_fail = 0
MAX_CONSECUTIVE_FAIL = 3
COOLDOWN_SECONDS = 300
cooldown_used = 0
MAX_COOLDOWNS = 3
relogin_used = 0
MAX_RELOGINS = 1 # 每轮 batch 最多自动重登1次
t0 = time.time()
for i, row in enumerate(rows, 1):
note_id = row["id"]
url = row["url"]
log(f"[{i}/{len(rows)}] id={note_id} url={url}")
comments = None
stats = None
for attempt in range(3):
try:
comments, stats = scraper.scrape_comments(url, 300, 60, 10, True)
break
except Exception as exc:
msg = str(exc)
retryable = attempt < 2 and (
"has been closed" in msg
or "page not loaded" in msg
or "Cannot extract aweme_id" in msg
)
if retryable:
wait_s = 3 * (attempt + 1)
label = (
"Browser closed" if "has been closed" in msg
else "Page not loaded" if "page not loaded" in msg
else "aweme_id failed"
)
log(f" {label}, retrying ({attempt+2}/3) after {wait_s}s...")
time.sleep(wait_s)
else:
log(f" ERROR: {exc}")
fail += 1
break
# Check logged_in status even on success
logged_in = result_logged_in(stats)
if comments is None:
consecutive_fail += 1
# 连续失败 + 可能 cookie 过期 → 尝试自动重登
if consecutive_fail >= MAX_CONSECUTIVE_FAIL and relogin_used < MAX_RELOGINS:
if do_relogin():
relogin_used += 1
consecutive_fail = 0
continue # retry the current URL in next iteration
else:
# 重登失败,走冷却逻辑
pass
if consecutive_fail >= MAX_CONSECUTIVE_FAIL:
if cooldown_used < MAX_COOLDOWNS:
cooldown_used += 1
log(
f" {consecutive_fail} consecutive failures, cooling down "
f"{COOLDOWN_SECONDS}s (cooldown {cooldown_used}/{MAX_COOLDOWNS})..."
)
time.sleep(COOLDOWN_SECONDS)
consecutive_fail = 0
continue
else:
log(
f" {MAX_COOLDOWNS} cooldowns exhausted, aborting batch; "
f"{len(rows) - i} notes skipped"
)
break
continue
consecutive_fail = 0
json_path, csv_path = scraper.write_outputs(url, comments, stats)
metrics = stats.get("stats", {}).get("note_metrics", {})
top = len([c for c in comments if c["level"] == "comment"])
replies = len([c for c in comments if c["level"] == "reply"])
warn = "" if logged_in else " [WARN] 未登录,可能只采到部分评论"
log(
f" ok: {len(comments)} comments ({top} top + {replies} replies) | "
f"likes={metrics.get('like_count')}, favs={metrics.get('favorite_count')}, "
f"cmts={metrics.get('comment_count')}, shares={metrics.get('share_count')}"
+ warn
)
if not dry_run:
try:
upsert_metrics(note_id, metrics)
except Exception as exc:
log(f" DB metric update error: {exc}")
try:
should_replace, allow_empty, reason = comment_replace_policy(comments, stats, metrics)
if should_replace:
db.replace_comments(note_id, "douyin", comments, allow_empty=allow_empty)
else:
log(f" skip DB comment replace: {reason}")
except Exception as exc:
log(f" DB comment replace error: {exc}")
ok += 1
total_comments += len(comments)
elapsed = time.time() - t0
log(
f"DONE: {ok} ok, {fail} fail, {total_comments} comments total, "
f"{elapsed:.1f}s ({elapsed / max(ok, 1):.1f}s per note)"
)
return 0 if fail == 0 else 1
if __name__ == "__main__":
args = sys.argv[1:]
only_ids = []
dry_run = False
for a in args:
if a == "--dry-run":
dry_run = True
elif a.isdigit():
only_ids.append(int(a))
if only_ids:
log(f"Restricted to ids: {only_ids}")
if dry_run:
log("DRY RUN: will not touch DB")
raise SystemExit(main(only_ids or None, dry_run))
src/gyxx_flow/modules/content_marketing/runtime/data/tools/batch_rescrape_xiaohongshu.py
+200
View File
@@ -0,0 +1,200 @@
"""Batch re-scrape all Xiaohongshu notes from DB using the browser scraper.
Updates:
- note_metrics (like / favorite / comment / share counts, title)
- cmt_comments table (replace existing)
- JSON/CSV files in data/notes/xiaohongshu/
"""
import subprocess
import sys
import time
from pathlib import Path
from gyxx_flow.modules.content_marketing.runtime.runtime_paths import PATHS
_TOOLS_DIR = PATHS.tools_root
_PROJECT_ROOT = PATHS.module_root
from gyxx_flow.modules.content_marketing.runtime.data.tools import db
from gyxx_flow.modules.content_marketing.runtime import xiaohongshu_comment_scraper as scraper
RELOGIN_SCRIPT = _TOOLS_DIR / "relogin_xiaohongshu.py"
try:
sys.stdout.reconfigure(encoding="utf-8", errors="replace") # type: ignore[attr-defined]
sys.stderr.reconfigure(encoding="utf-8", errors="replace") # type: ignore[attr-defined]
except Exception:
pass
def log(msg: str) -> None:
print(f"[{time.strftime('%H:%M:%S')}] {msg}", flush=True)
def collect_urls() -> list[dict]:
with db.get_dict_cursor() as cur:
cur.execute(
"""
SELECT id, url, title
FROM cmt_notes
WHERE platform = 'xiaohongshu' AND url IS NOT NULL AND url != ''
ORDER BY id
"""
)
return cur.fetchall()
def upsert_metrics(note_id: int, title: str, metrics: dict) -> None:
db.update_note_metrics(
note_id,
like_count=metrics.get("like_count") or None,
favorite_count=metrics.get("favorite_count") or None,
comment_count=metrics.get("comment_count") or None,
share_count=metrics.get("share_count") or None,
title=title or None,
)
def _to_int(value) -> int | None:
try:
if value in (None, ""):
return None
return int(float(value))
except (TypeError, ValueError):
return None
def comment_replace_policy(comments: list[dict], stats: dict, metrics: dict) -> tuple[bool, bool, str]:
"""Return (should_replace, allow_empty, reason)."""
logged_in = bool(stats.get("logged_in"))
expected = _to_int(metrics.get("comment_count"))
if expected is None:
expected = _to_int(stats.get("api_total"))
if comments:
if not logged_in and (expected is None or expected > len(comments)):
return False, False, "未登录且可能只采到部分评论"
return True, False, ""
if expected == 0:
return True, True, "平台显示 0 评论"
return False, False, "空评论且无法确认平台为 0 评论"
def do_relogin() -> bool:
"""Run relogin_xiaohongshu.py (headed, waits for QR scan). Returns True if successful."""
log(" Cookie 可能过期,触发自动重登 (请扫码)...")
try:
rc = subprocess.call(
[sys.executable, str(RELOGIN_SCRIPT)],
cwd=str(_PROJECT_ROOT),
)
if rc == 0:
log(" 重登成功,继续采集")
return True
else:
log(f" 重登失败 (exit={rc})")
return False
except Exception as exc:
log(f" 重登脚本执行异常: {exc}")
return False
def main(only_ids: list[int] | None = None, dry_run: bool = False) -> int:
rows = collect_urls()
if only_ids:
rows = [r for r in rows if r["id"] in set(only_ids)]
log(f"Total xiaohongshu notes to scrape: {len(rows)}")
ok = 0
fail = 0
total_comments = 0
consecutive_fail = 0
MAX_CONSECUTIVE_FAIL = 3
relogin_used = 0
MAX_RELOGINS = 1
t0 = time.time()
for i, row in enumerate(rows, 1):
note_id = row["id"]
url = row["url"]
log(f"[{i}/{len(rows)}] id={note_id} url={url}")
comments = None
result = None
for attempt in range(3):
try:
comments, result = scraper.scrape_comments(url, 60, 30, 4, True)
break
except Exception as exc:
if attempt < 2 and "has been closed" in str(exc):
log(f" Browser closed, retrying ({attempt+2}/3)...")
time.sleep(3)
else:
log(f" ERROR: {exc}")
fail += 1
break
if comments is None:
consecutive_fail += 1
if consecutive_fail >= MAX_CONSECUTIVE_FAIL and relogin_used < MAX_RELOGINS:
if do_relogin():
relogin_used += 1
consecutive_fail = 0
continue
# No relogin or relogin failed → skip this note
continue
consecutive_fail = 0
title = result.get("title", "")
stats = result.get("stats", {})
metrics = stats.get("note_metrics", {}) or {}
logged_in = bool(stats.get("logged_in"))
json_path, csv_path = scraper.write_outputs(url, comments, result)
top = len([c for c in comments if c["level"] == "comment"])
replies = len([c for c in comments if c["level"] == "reply"])
warn = "" if logged_in else " [WARN] 未登录,可能只采到部分评论"
log(
f" ok: title={title[:40]!r} | {len(comments)} comments ({top}+{replies}) | "
f"likes={metrics.get('like_count')}, favs={metrics.get('favorite_count')}, "
f"cmts={metrics.get('comment_count')}"
+ warn
)
if not dry_run:
try:
upsert_metrics(note_id, title, metrics)
except Exception as exc:
log(f" DB metric update error: {exc}")
try:
should_replace, allow_empty, reason = comment_replace_policy(comments, stats, metrics)
if should_replace:
db.replace_comments(note_id, "xiaohongshu", comments, allow_empty=allow_empty)
else:
log(f" skip DB comment replace: {reason}")
except Exception as exc:
log(f" DB comment replace error: {exc}")
ok += 1
total_comments += len(comments)
elapsed = time.time() - t0
log(
f"DONE: {ok} ok, {fail} fail, {total_comments} comments total, "
f"{elapsed:.1f}s ({elapsed / max(ok, 1):.1f}s per note)"
)
return 0 if fail == 0 else 1
if __name__ == "__main__":
args = sys.argv[1:]
only_ids = []
dry_run = False
for a in args:
if a == "--dry-run":
dry_run = True
elif a.isdigit():
only_ids.append(int(a))
if only_ids:
log(f"Restricted to ids: {only_ids}")
if dry_run:
log("DRY RUN: will not touch DB")
raise SystemExit(main(only_ids or None, dry_run))
src/gyxx_flow/modules/content_marketing/runtime/data/tools/check_id_filled.py
+42
View File
@@ -0,0 +1,42 @@
"""统计 24 张表里 creator_id / creator_name 填充率"""
import json
import glob
from pathlib import Path
per_table = []
total_records = total_with_id = total_with_name = 0
for path in sorted(
glob.glob("data/feishu_tables/[0-9]*.json"),
key=lambda p: int(Path(p).stem.split("-")[0]),
):
d = json.load(open(path, encoding="utf-8"))
name = d.get("name")
fid_to_name = d.get("field_id_to_name", {})
name_fid = id_fid = None
for fid, fname in fid_to_name.items():
cname = (fname or "").rstrip("?").rstrip()
if cname in ("达人名称", "达人昵称"):
name_fid = fid
elif cname in ("达人id", "达人id号", "达人ID", "星图ID"):
id_fid = fid
n_with_id = n_with_name = 0
n_records = len(d.get("records", []))
for rec in d.get("records", []):
if id_fid and rec.get(id_fid):
n_with_id += 1
if name_fid and rec.get(name_fid):
n_with_name += 1
per_table.append((name, n_records, n_with_id, n_with_name))
total_records += n_records
total_with_id += n_with_id
total_with_name += n_with_name
print(f"{'款式':<14} {'总记录':>5} {'有ID':>5} {'有名称':>6} {'填充率(ID)':>10}")
for name, n, nid, nnm in per_table:
rate = f"{100*nid/n:.1f}%" if n else "-"
print(f"{name:<14} {n:>5} {nid:>5} {nnm:>6} {rate:>10}")
print()
print(f"总计 {total_records} 条")
print(f" 有 creator_id: {total_with_id} ({100*total_with_id/total_records:.1f}%)")
print(f" 有 creator_name:{total_with_name} ({100*total_with_name/total_records:.1f}%)")
src/gyxx_flow/modules/content_marketing/runtime/data/tools/check_status.py
+179
View File
@@ -0,0 +1,179 @@
"""查看每个平台 24 款式的采集状态。
判定规则(对单个平台):
OK - v2 JSON 文件存在,且 mtime 在 --stale-hours 小时内
STALE - v2 JSON 文件存在,但 mtime 超过 --stale-hours 小时(需要重跑)
MISSING - v2 JSON 文件不存在(进程崩溃/被中断)
EMPTY - v2 JSON 存在但 matched=0、filled=0、total=0 (该款式确实没数据)
HALF - v2 JSON 存在但 results 里至少有一条 error (部分采集成功)
用法:
python data/tools/check_status.py # 看三个平台
python data/tools/check_status.py --platform xt # 只看星图
python data/tools/check_status.py --stale-hours 4 # 4 小时以上算陈旧
"""
import argparse
import datetime as dt
import json
import sys
from pathlib import Path
from gyxx_flow.modules.content_marketing.runtime.runtime_paths import PATHS
PROJECT_DIR = PATHS.module_root
V2_DIR = PATHS.normalized_root / "v2_results"
# 共享工具
from gyxx_flow.modules.content_marketing.runtime.data.tools.v2_filename import parse_v2_filename # noqa: E402
PLATFORMS = {
"bili": {"name": "B 站", "label": "bili"},
"pgy": {"name": "小红书蒲公英", "label": "xhs"},
"xt": {"name": "星图", "label": "xt"},
}
# 款式总数 1-24
STYLE_INDICES = list(range(1, 25))
def scan_platform(platform_key: str, stale_hours: float) -> dict[int, dict]:
"""扫一个平台 v2_results,返回 {index: {status, mtime, matched, ...}}"""
label = PLATFORMS[platform_key]["label"]
found: dict[int, dict] = {}
for path in V2_DIR.glob("*.json"):
idx, name, plat, _ = parse_v2_filename(path.name)
if plat != label or idx is None:
continue
mtime = dt.datetime.fromtimestamp(path.stat().st_mtime)
age_h = (dt.datetime.now() - mtime).total_seconds() / 3600.0
try:
data = json.loads(path.read_text(encoding="utf-8"))
except Exception:
data = {}
matched = data.get("matched") or data.get("updated") or 0
filled = data.get("filled") or matched
total = (data.get("total") or data.get("total_b_records") or 0)
results = data.get("results") or data.get("details") or []
has_error = any(isinstance(r, dict) and r.get("error") for r in results)
if total == 0 and matched == 0 and not has_error:
status = "EMPTY"
elif has_error and matched > 0:
status = "HALF"
elif has_error:
status = "HALF"
elif age_h > stale_hours:
status = "STALE"
else:
status = "OK"
found[idx] = {
"status": status,
"name": name,
"path": path.name,
"mtime": mtime,
"age_h": age_h,
"matched": matched,
"filled": filled,
"total": total,
"has_error": has_error,
}
return found
def color(s: str, code: str) -> str:
if not sys.stdout.isatty():
return s
return f"\033[{code}m{s}\033[0m"
STATUS_COLOR = {
"OK": ("32",), # green
"STALE": ("33",), # yellow
"MISSING": ("31",), # red
"EMPTY": ("90",), # grey
"HALF": ("33",), # yellow
}
def fmt_status(s: str) -> str:
code = STATUS_COLOR.get(s, ("",))[0]
label_map = {
"OK": " OK ",
"STALE": " STALE ",
"MISSING": "MISSING",
"EMPTY": " EMPTY ",
"HALF": " HALF ",
}
return color(label_map.get(s, s), code)
def print_platform(platform_key: str, info: dict, stale_hours: float) -> tuple[int, int, int]:
name = info["name"]
by_idx = scan_platform(platform_key, stale_hours)
counts = {"OK": 0, "STALE": 0, "MISSING": 0, "EMPTY": 0, "HALF": 0}
print(f"\n{'=' * 78}")
print(f" {name} (stale > {stale_hours}h)")
print(f"{'=' * 78}")
print(f" {'款式':>4} {'状态':<8} {'名称':<14} {'matched':>7} {'total':>5} "
f"{'err':>3} {'mtime':<19} {'age':>5}")
print(" " + "-" * 74)
for idx in STYLE_INDICES:
if idx in by_idx:
r = by_idx[idx]
counts[r["status"]] += 1
err_n = sum(1 for x in (r.get("path"),) if x and "error" in x) # placeholder
# 计算 has_error 计数
try:
data = json.loads((V2_DIR / r["path"]).read_text(encoding="utf-8"))
results = data.get("results") or data.get("details") or []
err_n = sum(1 for x in results if isinstance(x, dict) and x.get("error"))
except Exception:
err_n = 0
mtime_str = r["mtime"].strftime("%Y-%m-%d %H:%M:%S")
print(f" {idx:>4} {fmt_status(r['status'])} {r['name']:<14} "
f"{r['matched']:>7} {r['total']:>5} {err_n:>3} "
f"{mtime_str:<19} {r['age_h']:>4.1f}h")
else:
counts["MISSING"] += 1
print(f" {idx:>4} {fmt_status('MISSING')} {'-':<14} {'-':>7} {'-':>5} "
f"{'-':>3} {'-':<19} {'-':>5}")
print(" " + "-" * 74)
total = sum(counts.values())
print(f" 小计: OK={counts['OK']} STALE={counts['STALE']} "
f"MISSING={counts['MISSING']} EMPTY={counts['EMPTY']} "
f"HALF={counts['HALF']} / 共 {total}")
# 待补跑 = MISSING + STALE + HALF(只算有 matched>0 的)
need_retry = counts["MISSING"] + counts["STALE"]
print(f" 待补跑: {need_retry} 个款式")
return counts["MISSING"] + counts["STALE"], counts["HALF"], total
def main() -> int:
ap = argparse.ArgumentParser(description="查看各平台 24 款式采集状态")
ap.add_argument("--platform", type=str, default="bili,pgy,xt",
help="逗号分隔平台键 bili/pgy/xt,默认全看")
ap.add_argument("--stale-hours", type=float, default=26.0,
help="超过多少小时算 STALE(默认 26h = 每日一次)")
args = ap.parse_args()
keys = [k.strip() for k in args.platform.split(",") if k.strip()]
for k in keys:
if k not in PLATFORMS:
print(f"[ERROR] 未知平台: {k} 可选: {','.join(PLATFORMS.keys())}")
return 1
print(f"扫描目录: {V2_DIR}")
grand_retry = grand_half = grand_total = 0
for k in keys:
retry_n, half_n, total_n = print_platform(k, PLATFORMS[k], args.stale_hours)
grand_retry += retry_n
grand_half += half_n
grand_total += total_n
print(f"\n{'=' * 78}")
print(f" 汇总: 待补跑 {grand_retry} 款 (HALF {grand_half} 款) / 共 {grand_total} 款")
print(f" 补跑命令: python data/tools/retry_failed.py")
print(f"{'=' * 78}\n")
return 0
if __name__ == "__main__":
sys.exit(main())
src/gyxx_flow/modules/content_marketing/runtime/data/tools/collect_note_metrics.py
+188
View File
@@ -0,0 +1,188 @@
import argparse
import csv
import json
import sys
from pathlib import Path
from typing import Any
from gyxx_flow.modules.content_marketing.runtime.runtime_paths import (
PATHS,
resolve_layer_output,
)
# Path setup: scrapers are in project root, feishu_comment_batch is in data/tools/
_TOOLS_DIR = PATHS.tools_root
_PROJECT_ROOT = PATHS.module_root
from scrapling.fetchers import DynamicSession
from gyxx_flow.modules.content_marketing.runtime import bilibili_comment_scraper
from gyxx_flow.modules.content_marketing.runtime import douyin_comment_scraper
from gyxx_flow.modules.content_marketing.runtime import xiaohongshu_comment_scraper
from gyxx_flow.modules.content_marketing.runtime.data.tools.feishu_comment_batch import detect_platform, normalize_note_url
def unique_note_urls(source_csv: Path, platform: str) -> list[str]:
seen: set[str] = set()
urls: list[str] = []
with source_csv.open(encoding="utf-8-sig", newline="") as file:
for row in csv.DictReader(file):
url = normalize_note_url(row.get("note_url") or row.get("笔记链接") or "")
if not url or detect_platform(url) != platform or url in seen:
continue
seen.add(url)
urls.append(url)
return urls
def collect_douyin(urls: list[str], headless: bool) -> list[dict[str, Any]]:
"""Collect Douyin note metrics via browser (Playwright + XHR interception)."""
rows: list[dict[str, Any]] = []
def make_action(url: str):
def action(page):
page.wait_for_timeout(7000)
metrics = douyin_comment_scraper.extract_note_metrics(page)
rows.append({
"platform": "douyin",
"note_url": url,
"like_count": metrics.get("like_count", ""),
"favorite_count": metrics.get("favorite_count", ""),
"share_count": metrics.get("share_count", ""),
"comment_count": metrics.get("comment_count", ""),
})
print(f"douyin {len(rows)}/{len(urls)} {metrics} {url}", flush=True)
return action
with DynamicSession(
headless=headless,
real_chrome=True,
user_data_dir=str(douyin_comment_scraper.PROFILE_DIR),
locale="zh-CN",
timezone_id="Asia/Shanghai",
timeout=90000,
network_idle=False,
disable_resources=False,
google_search=False,
page_setup=douyin_comment_scraper.restore_cookies,
max_pages=1,
) as session:
for url in urls:
try:
normalized = douyin_comment_scraper.normalize_douyin_url(url)
session.fetch(normalized, page_action=make_action(url), wait=300)
except Exception as exc:
rows.append({"platform": "douyin", "note_url": url, "like_count": "", "favorite_count": "", "share_count": "", "comment_count": "", "error": repr(exc)})
print(f"douyin failed {url} {exc!r}", flush=True)
return rows
def collect_xiaohongshu(urls: list[str], headless: bool) -> list[dict[str, Any]]:
rows: list[dict[str, Any]] = []
def make_action(url: str):
def action(page):
page.wait_for_timeout(7000)
metrics = xiaohongshu_comment_scraper.extract_note_metrics(page)
rows.append({"platform": "xiaohongshu", "note_url": url, **metrics})
print(f"xiaohongshu {len(rows)}/{len(urls)} {metrics} {url}", flush=True)
return action
with DynamicSession(
headless=headless,
real_chrome=True,
user_data_dir=str(xiaohongshu_comment_scraper.PROFILE_DIR),
locale="zh-CN",
timezone_id="Asia/Shanghai",
timeout=90000,
network_idle=False,
disable_resources=False,
google_search=False,
page_setup=xiaohongshu_comment_scraper.restore_cookies,
max_pages=1,
) as session:
for url in urls:
try:
session.fetch(url, page_action=make_action(url), wait=300)
except Exception as exc:
rows.append({"platform": "xiaohongshu", "note_url": url, "like_count": "", "favorite_count": "", "share_count": "", "comment_count": "", "error": repr(exc)})
print(f"xiaohongshu failed {url} {exc!r}", flush=True)
return rows
def collect_bilibili(urls: list[str], headless: bool) -> list[dict[str, Any]]:
rows: list[dict[str, Any]] = []
def make_action(url: str):
def action(page):
page.wait_for_timeout(1500)
video = bilibili_comment_scraper.resolve_video_info(page, url)
metrics = video.get("note_metrics") or {}
rows.append({"platform": "bilibili", "note_url": url, **metrics})
print(f"bilibili {len(rows)}/{len(urls)} {metrics} {url}", flush=True)
return action
with DynamicSession(
headless=headless,
real_chrome=True,
user_data_dir=str(bilibili_comment_scraper.PROFILE_DIR),
locale="zh-CN",
timezone_id="Asia/Shanghai",
timeout=90000,
network_idle=False,
disable_resources=False,
google_search=False,
page_setup=bilibili_comment_scraper.restore_cookies,
max_pages=1,
) as session:
for url in urls:
try:
session.fetch(url, page_action=make_action(url), wait=300)
except Exception as exc:
rows.append({"platform": "bilibili", "note_url": url, "like_count": "", "favorite_count": "", "share_count": "", "comment_count": "", "error": repr(exc)})
print(f"bilibili failed {url} {exc!r}", flush=True)
return rows
def write_rows(rows: list[dict[str, Any]], output: Path) -> None:
output.parent.mkdir(exist_ok=True)
fields = ["platform", "note_url", "like_count", "favorite_count", "share_count", "comment_count", "error"]
with output.open("w", encoding="utf-8-sig", newline="") as file:
writer = csv.DictWriter(file, fieldnames=fields)
writer.writeheader()
for row in rows:
writer.writerow({field: row.get(field, "") for field in fields})
output.with_suffix(".json").write_text(json.dumps(rows, ensure_ascii=False, indent=2), encoding="utf-8")
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("--source-csv", required=True)
parser.add_argument("--platform", choices=["douyin", "xiaohongshu", "bilibili"], required=True)
parser.add_argument("--output", required=True)
parser.add_argument("--headless", action="store_true")
args = parser.parse_args()
source_csv = Path(args.source_csv)
output = resolve_layer_output(
args.output, layer_root=PATHS.normalized_root, field="--output"
)
urls = unique_note_urls(source_csv, args.platform)
print(f"{args.platform}: {len(urls)} unique notes", flush=True)
if args.platform == "douyin":
rows = collect_douyin(urls, args.headless)
elif args.platform == "xiaohongshu":
rows = collect_xiaohongshu(urls, args.headless)
else:
rows = collect_bilibili(urls, args.headless)
write_rows(rows, output)
print(f"wrote {output}", flush=True)
return 0
if __name__ == "__main__":
raise SystemExit(main())
src/gyxx_flow/modules/content_marketing/runtime/data/tools/daily_dashboard_analytics.py
+321
View File
@@ -0,0 +1,321 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""构建全款式同权经营看板所需的确定性指标与榜单。"""
from __future__ import annotations
from collections import defaultdict
from datetime import date, timedelta
import os
from pathlib import Path
from typing import Any
from gyxx_flow.modules.content_marketing.runtime.runtime_paths import PATHS
try:
from .db import get_conn
except ImportError:
from gyxx_flow.modules.content_marketing.runtime.data.tools.db import get_conn
MIN_CONVERSION_RANK_VISITORS = 30
MIN_REFUND_RANK_SALES = 5
ROOT = PATHS.module_root
IMAGE_ROOTS = [
ROOT,
Path(os.environ.get("GYXX_PRODUCT_ASSET_ROOT", str(ROOT))),
]
PLATFORM_IMAGE_CODES = {"天猫": "tm", "京东": "jd", "抖音电商": "dy"}
STYLE_CATEGORY_ORDER = ("都市机能", "户外机能", "都市新贵", "智性通勤", "都市新旅", "都市运动", "未分类")
def load_style_categories(style_names: list[str]) -> dict[str, str]:
if not style_names:
return {}
with get_conn() as conn:
with conn.cursor() as cur:
cur.execute(
"""
SELECT name, COALESCE(NULLIF(BTRIM(style_category), ''), '未分类')
FROM cmt_styles
WHERE name = ANY(%s)
""",
(style_names,),
)
return {name: category for name, category in cur.fetchall()}
def load_creator_connections(metric_date: date, style_names: list[str]) -> dict[str, dict[str, Any]]:
if not style_names:
return {}
with get_conn() as conn:
with conn.cursor() as cur:
cur.execute(
"""
SELECT s.name,
COUNT(DISTINCT c.creator_id) FILTER (WHERE c.creator_id IS NOT NULL) creator_count,
ARRAY_REMOVE(ARRAY_AGG(DISTINCT NULLIF(BTRIM(c.content_direction), '')), NULL) directions
FROM cmt_styles s
LEFT JOIN cmt_cooperations c
ON c.style_id = s.id
AND COALESCE(c.publish_time::date, c.cooperation_date) BETWEEN %s AND %s
WHERE s.name = ANY(%s)
GROUP BY s.name
""",
(metric_date - timedelta(days=29), metric_date, style_names),
)
return {
style_name: {"creator_count": creator_count or 0, "directions": directions or []}
for style_name, creator_count, directions in cur.fetchall()
}
def load_style_images(style_names: list[str]) -> dict[tuple[str, str], dict[str, Any]]:
"""为三平台画像卡统一选择同款天猫高点击率主图。"""
if not style_names:
return {}
with get_conn() as conn:
with conn.cursor() as cur:
cur.execute(
"""
SELECT style_name, platform, collect_date, local_image_path, creative_image_url,
COALESCE(impressions, 0) impressions, click_rate
FROM main_image_creatives
WHERE style_name = ANY(%s)
AND platform = 'tm'
AND image_downloaded = TRUE
AND local_image_path IS NOT NULL
AND COALESCE(impressions, 0) >= 100
ORDER BY COALESCE(click_rate, 0) DESC, COALESCE(impressions, 0) DESC, collect_date DESC
""",
(style_names,),
)
rows = cur.fetchall()
by_style: dict[str, list[dict[str, Any]]] = defaultdict(list)
for style_name, platform, collect_date, local_path, url, impressions, click_rate in rows:
relative = Path(str(local_path))
absolute = next((root / relative for root in IMAGE_ROOTS if (root / relative).exists()), None)
if absolute is None:
continue
by_style[style_name].append({
"local_image_path": str(absolute),
"image_url": url,
"image_platform": {"tm": "天猫", "jd": "京东", "dy": "抖音电商"}.get(platform, platform),
"image_date": collect_date,
"image_impressions": impressions,
"image_ctr": float(click_rate) if click_rate is not None else None,
})
result: dict[tuple[str, str], dict[str, Any]] = {}
for style_name in style_names:
candidates = by_style.get(style_name, [])
for target in ("天猫", "京东", "抖音电商"):
if candidates:
result[(style_name, target)] = candidates[0]
return result
def _rate(numerator: Any, denominator: Any) -> float | None:
if numerator is None or denominator is None or denominator <= 0:
return None
return round(float(numerator) / float(denominator), 4)
def _percentile(values: list[float], fraction: float) -> float:
if not values:
return 0.0
ordered = sorted(values)
index = min(len(ordered) - 1, max(0, round((len(ordered) - 1) * fraction)))
return ordered[index]
def _persona_text(persona: dict[str, Any] | None) -> str:
if not persona:
return "无数据"
labels = [persona.get(key) for key in ("gender_top", "age_top", "city_top", "buying_power_top") if persona.get(key)]
return "、".join(labels) if labels else "无数据"
def _interest_hypothesis(persona: dict[str, Any] | None, directions: list[str]) -> str:
if not persona:
return "无数据"
source = " ".join(str(value or "") for value in persona.values())
interests: list[str] = []
if any(term in source for term in ("女性", "女")):
interests.extend(["穿搭", "健身", "旅行摄影"])
if any(term in source for term in ("男性", "男")):
interests.extend(["数码3C", "城市通勤", "户外出行"])
if any(term in source for term in ("18-24", "18-25", "25-29")):
interests.extend(["潮流内容", "社交分享"])
if any(term in source for term in ("26-35", "30-34", "35-39")):
interests.extend(["职场效率", "短途出行"])
if any(term in source for term in ("56岁以上", "46-55", "45-49", "50")):
interests.extend(["品质消费", "舒适出行"])
if any(term in source for term in ("土豪", "高级白领", "L4", "L5")):
interests.extend(["品质材质", "设计感"])
interests.extend(direction for direction in directions if direction)
unique = list(dict.fromkeys(interests))[:4]
return "兴趣推测：" + "、".join(unique) if unique else "兴趣推测：无足够标签"
def _optimization(role: str, has_persona: bool) -> str:
persona_action = "按画像扩展相似人群" if has_persona else "先补采人群画像"
if role == "conversion":
return f"高转化扩量：{persona_action}，逐步增加流量并监控退款订单比"
return f"高访客提转化：{persona_action}，优化主图/详情页/利益点并做加购再营销"
def _diagnose(row: dict[str, Any], visitor_q75: float, conversion_median: float, cart_median: float, refund_q75: float) -> str:
if row.get("visitors") is None:
return "无数据"
conversion = row.get("conversion_rate")
refund = row.get("refund_rate")
cart_rate = row.get("cart_rate")
if refund is not None and refund >= max(0.3, refund_q75):
return "退款压力"
if row["visitors"] >= visitor_q75 and conversion is not None and conversion >= conversion_median:
return "高流量高转化"
if row["visitors"] >= visitor_q75:
return "高流量·转化待提"
if conversion is not None and conversion >= conversion_median * 1.25:
return "低流量·高转化可放量"
if cart_rate is not None and cart_rate < cart_median * 0.7:
return "加购偏弱"
return "常规观察"
def build_dashboard_facts(
report_date: str,
tracking_rows: list[dict[str, Any]],
platform_metrics: list[dict[str, Any]],
personas: list[dict[str, Any]],
connections: dict[str, dict[str, Any]],
style_images: dict[tuple[str, str], dict[str, Any]] | None = None,
daily_notes: list[dict[str, Any]] | None = None,
style_categories: dict[str, str] | None = None,
style_signals: dict[str, dict[str, Any]] | None = None,
) -> dict[str, Any]:
styles = []
for source in tracking_rows:
row = dict(source)
row["cart_rate"] = _rate(row.get("cart_users"), row.get("visitors"))
row["conversion_rate"] = _rate(row.get("sales"), row.get("visitors"))
row["refund_rate"] = _rate(row.get("refund_orders"), row.get("sales"))
connection = connections.get(row["style_name"], {})
row["creator_connections_30d"] = int(connection.get("creator_count") or 0)
row["content_directions_30d"] = connection.get("directions") or []
row["style_category"] = (style_categories or {}).get(row["style_name"]) or "未分类"
row["multi_source_signals"] = (style_signals or {}).get(row["style_name"], {})
row["recent_note_exposure"] = row["multi_source_signals"].get("recent_note_exposure")
styles.append(row)
visitors = [float(row["visitors"]) for row in styles if row.get("visitors") is not None]
conversions = [float(row["conversion_rate"]) for row in styles if row.get("conversion_rate") is not None]
carts = [float(row["cart_rate"]) for row in styles if row.get("cart_rate") is not None]
refunds = [float(row["refund_rate"]) for row in styles if row.get("refund_rate") is not None]
visitor_q75 = _percentile(visitors, 0.75)
conversion_median = _percentile(conversions, 0.5)
cart_median = _percentile(carts, 0.5)
refund_q75 = _percentile(refunds, 0.75)
for row in styles:
base = _diagnose(row, visitor_q75, conversion_median, cart_median, refund_q75)
signal = row["multi_source_signals"]
evidence = ["经营"]
if signal.get("persona"):
evidence.append("画像")
if signal.get("note_count"):
evidence.append("营销笔记")
if signal.get("comment_count"):
evidence.append("评论")
if signal.get("review_count"):
evidence.append("商品评价")
if signal.get("creative_ctr") is not None:
evidence.append("主图")
row["evidence_summary"] = "、".join(evidence)
if (signal.get("review_negative_rate") or 0) >= 0.20:
row["diagnosis"] = f"{base}·评价风险" if base != "无数据" else "评价风险"
elif (signal.get("comment_negative_rate") or 0) >= 0.20:
row["diagnosis"] = f"{base}·评论风险" if base != "无数据" else "评论风险"
else:
row["diagnosis"] = base
category_rank = {name: index for index, name in enumerate(STYLE_CATEGORY_ORDER)}
ordered_styles = sorted(
styles,
key=lambda row: (
category_rank.get(row["style_category"], len(category_rank)),
row["style_category"],
row["style_name"],
),
)
grouped: dict[str, list[dict[str, Any]]] = defaultdict(list)
for row in ordered_styles:
grouped[row["style_category"]].append(row)
style_groups = [
{"style_category": category, "styles": grouped[category]}
for category in sorted(grouped, key=lambda value: (category_rank.get(value, len(category_rank)), value))
]
def top(metric: str, eligible=lambda row: True) -> list[dict[str, Any]]:
rows = [row for row in styles if row.get(metric) is not None and eligible(row)]
return sorted(rows, key=lambda row: (row[metric], row.get("visitors") or 0), reverse=True)[:5]
rankings = {
"conversion": top("conversion_rate", lambda row: (row.get("visitors") or 0) >= MIN_CONVERSION_RANK_VISITORS),
"visitors": top("visitors"),
"cart_users": top("cart_users"),
"sales": top("sales"),
"refund_rate": top("refund_rate", lambda row: (row.get("sales") or 0) >= MIN_REFUND_RANK_SALES),
}
persona_index = {(row.get("style_name"), row.get("platform")): row for row in personas}
directions = {row["style_name"]: row.get("content_directions_30d", []) for row in styles}
platform_groups: dict[str, list[dict[str, Any]]] = defaultdict(list)
for source in platform_metrics:
row = dict(source)
row["conversion_rate"] = _rate(row.get("sales"), row.get("visitors"))
platform_groups[row.get("platform")].append(row)
platform_personas = []
preferred_platforms = ["天猫", "京东", "抖音电商"]
ordered_platforms = [name for name in preferred_platforms if name in platform_groups]
ordered_platforms.extend(sorted(name for name in platform_groups if name not in preferred_platforms))
for platform in ordered_platforms:
rows = platform_groups[platform]
conversion_eligible = [row for row in rows if (row.get("visitors") or 0) >= MIN_CONVERSION_RANK_VISITORS and row.get("conversion_rate") is not None]
conversion_winner = max(conversion_eligible, key=lambda row: row["conversion_rate"]) if conversion_eligible else None
visitor_winner = max(rows, key=lambda row: row.get("visitors") or -1) if rows else None
def winner_payload(winner: dict[str, Any] | None, role: str) -> dict[str, Any] | None:
if not winner:
return None
persona = persona_index.get((winner["style_name"], platform))
return {
**winner,
**((style_images or {}).get((winner["style_name"], platform)) or {}),
"persona_text": _persona_text(persona),
"persona_date": persona.get("data_date") if persona else None,
"interest_hypothesis": _interest_hypothesis(persona, directions.get(winner["style_name"], [])),
"optimization": _optimization(role, bool(persona)),
}
platform_personas.append({
"platform": platform,
"top_conversion": winner_payload(conversion_winner, "conversion"),
"top_visitors": winner_payload(visitor_winner, "visitors"),
})
return {
"report_date": report_date,
"styles": ordered_styles,
"style_groups": style_groups,
"rankings": rankings,
"platform_personas": platform_personas,
"daily_new_notes": sorted(daily_notes or [], key=lambda row: row.get("view_count") or 0, reverse=True),
"definitions": {
"conversion_rate": "销量/访客；榜单访客门槛≥30",
"cart_rate": "加购人数/访客",
"refund_rate": "退款订单/当日销量，可能含跨期退款",
"creator_connections": "近30天去重达人数量",
"sales_7d_total": "报告日及前6日累计销量",
"sales_forecast_7d": "最近3日日均销量×7；趋势相对前4日日均，仅作短期估算",
},
}
src/gyxx_flow/modules/content_marketing/runtime/data/tools/daily_marketing_report.bat
+57
View File
@@ -0,0 +1,57 @@
@echo off
if not defined GYXX_PYTHON set "GYXX_PYTHON=python"
REM Daily SKU marketing operations report: Hermes analysis + Feishu dashboard and full report
REM Register: schtasks /Create /SC DAILY /TN YingxiaoYunying_DailyMarketingReport /TR %PROJECT_DIR%\data\tools\daily_marketing_report.bat /ST 10:00 /F
setlocal
for %%I in ("%~dp0..\..") do set "PROJECT_DIR=%%~fI"
if not defined GYXX_DATA_ROOT (
if not defined GYXX_PROJECT_ROOT (
echo ERROR: GYXX_DATA_ROOT or GYXX_PROJECT_ROOT must be defined. 1>&2
exit /b 3
)
set "GYXX_DATA_ROOT=%GYXX_PROJECT_ROOT%\var"
)
set LOG_DIR=%GYXX_DATA_ROOT%\logs\content_marketing\scheduled
set PYTHON=%GYXX_PYTHON%
where %PYTHON% >nul 2>&1 || set PYTHON=python
set PYTHONIOENCODING=utf-8
set LARK_CLI_NO_PROXY=1
set LARKSUITE_CLI_NO_UPDATE_NOTIFIER=1
set LARKSUITE_CLI_NO_SKILLS_NOTIFIER=1
if not exist "%LOG_DIR%" mkdir "%LOG_DIR%"
set TS=%date:~0,4%%date:~5,2%%date:~8,2%_%time:~0,2%%time:~3,2%%time:~6,2%
set TS=%TS: =0%
set LOG_FILE=%LOG_DIR%\daily_marketing_report_%TS%.log
echo === Daily marketing report started at %date% %time% === > "%LOG_FILE%"
echo PROJECT_DIR=%PROJECT_DIR% >> "%LOG_FILE%"
REM Wait for today's daily_run to finish (poll every 60s, max 90 min), so that
REM cmt_notes is synced before the report reads yesterday's new notes.
set TODAY=%date:~0,4%%date:~5,2%%date:~8,2%
set /a WAITED=0
:wait_daily_run
set DAILY_DONE=
for %%F in ("%LOG_DIR%\daily_run_%TODAY%_*.log") do findstr /C:"Daily run finished" /C:"Daily run skipped" "%%F" >nul 2>&1 && set DAILY_DONE=1
if defined DAILY_DONE goto daily_run_ready
if %WAITED% GEQ 5400 goto daily_run_timeout
timeout /t 60 /nobreak >nul
set /a WAITED+=60
goto wait_daily_run
:daily_run_timeout
echo WARN: daily_run not finished after 90 min wait, generating report anyway >> "%LOG_FILE%"
:daily_run_ready
cd /d "%PROJECT_DIR%"
call %PYTHON% -u -X utf8 daily_marketing_report.py --send >> "%LOG_FILE%" 2>&1
set RC=%ERRORLEVEL%
echo daily_marketing_report exit=%RC% >> "%LOG_FILE%"
echo === Daily marketing report finished at %date% %time% (rc=%RC%) === >> "%LOG_FILE%"
endlocal & exit /b %RC%
src/gyxx_flow/modules/content_marketing/runtime/data/tools/daily_report_card.py
+328
View File
@@ -0,0 +1,328 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""构造并发送时尚包袋品牌经营日报飞书 Card 2.0。"""
import hashlib
import json
import os
import re
import shutil
import subprocess
from collections import Counter
from pathlib import Path
from typing import Any, Iterable
from PIL import Image
from gyxx_flow.modules.content_marketing.runtime.runtime_paths import PATHS
LARK_PROFILE = "hermes-analyzer"
LARK_CLI = shutil.which("lark-cli.cmd") or shutil.which("lark-cli") or "lark-cli"
MAX_CARD_IMAGE_WIDTH = 1500
def _lark_command() -> list[str]:
run_js = (
Path(os.environ.get("APPDATA", str(Path.home())))
/ "npm" / "node_modules" / "@larksuite" / "cli" / "scripts" / "run.js"
)
if not run_js.exists():
run_js = Path(os.path.expandvars(r"%APPDATA%\npm\node_modules\@larksuite\cli\scripts\run.js"))
if run_js.exists():
return [shutil.which("node") or "node", str(run_js)]
return [LARK_CLI]
def _section_lines(report: str, start: str, end: str | None, limit: int = 3) -> list[str]:
start_match = re.search(rf"(?m)^\s*#*\s*{re.escape(start)}\s*$", report)
if not start_match:
return []
tail = report[start_match.end():]
if end:
end_match = re.search(rf"(?m)^\s*#*\s*{re.escape(end)}", tail)
if end_match:
tail = tail[:end_match.start()]
lines = []
for raw in tail.splitlines():
value = re.sub(r"^\s*\d+[\.、]\s*", "", raw).strip()
if not value or value.startswith("![") or value.startswith("|"):
continue
lines.append(value[:180])
if len(lines) >= limit:
break
return lines
def _metric_column(value: int, label: str, emphasis: bool = False) -> dict[str, Any]:
number_size = "heading-1" if emphasis else "heading-2"
return {
"tag": "column",
"width": "weighted",
"weight": 1,
"background_style": "grey-50",
"padding": "12px",
"vertical_spacing": "2px",
"elements": [
{"tag": "markdown", "content": str(value), "text_size": number_size, "text_align": "center"},
{"tag": "markdown", "content": f"<font color='grey'>{label}</font>", "text_size": "notation", "text_align": "center"},
],
}
def build_dashboard_card(
report_date: str,
tracking_rows: list[dict[str, Any]],
image_key: str,
) -> dict[str, Any]:
"""构造只包含核心灯号和看板图片的第一条消息。"""
counts = Counter(row.get("light", "无判断") for row in tracking_rows)
return {
"schema": "2.0",
"config": {
"update_multi": True,
"width_mode": "fill",
"enable_forward": True,
"summary": {"content": f"{report_date} 时尚包袋品牌经营看板"},
},
"header": {
"title": {"tag": "plain_text", "content": "时尚包袋品牌经营看板"},
"subtitle": {"tag": "plain_text", "content": f"{report_date} · Hermes 自动分析"},
"template": "blue",
"icon": {"tag": "standard_icon", "token": "chart_colorful"},
"text_tag_list": [
{"tag": "text_tag", "text": {"tag": "plain_text", "content": "数据看板"}, "color": "blue"}
],
},
"body": {
"direction": "vertical",
"padding": "12px 12px 20px 12px",
"vertical_spacing": "large",
"elements": [
{
"tag": "column_set",
"flex_mode": "none",
"horizontal_spacing": "12px",
"columns": [
_metric_column(counts.get("🔴", 0), "红灯款式", emphasis=True),
_metric_column(counts.get("🟡", 0), "黄灯款式"),
_metric_column(counts.get("🟢", 0), "绿灯款式"),
],
},
{
"tag": "img",
"img_key": image_key,
"alt": {"tag": "plain_text", "content": "全域种草数据看板"},
"scale_type": "fit_horizontal",
"corner_radius": "8px",
"preview": True,
},
],
},
}
def _report_without_dashboard_reference(report: str) -> str:
return re.sub(
r"(?m)^\s*!\[全域种草数据看板\]\([^\n]+\)\s*\n?",
"",
report,
count=1,
).strip()
def build_analysis_card(report_date: str, report: str) -> dict[str, Any]:
"""构造包含完整日报正文的第二条消息，不做行数或字数裁剪。"""
analysis = _report_without_dashboard_reference(report) or "无数据"
return {
"schema": "2.0",
"config": {
"update_multi": True,
"width_mode": "fill",
"enable_forward": True,
"summary": {"content": f"{report_date} SKU营销运营日报分析"},
},
"header": {
"title": {"tag": "plain_text", "content": "SKU营销运营日报分析"},
"subtitle": {"tag": "plain_text", "content": f"{report_date} · 完整文字版"},
"template": "turquoise",
"icon": {"tag": "standard_icon", "token": "doc_colorful"},
"text_tag_list": [
{"tag": "text_tag", "text": {"tag": "plain_text", "content": "完整分析"}, "color": "turquoise"}
],
},
"body": {
"direction": "vertical",
"padding": "12px 16px 20px 16px",
"vertical_spacing": "medium",
"elements": [{"tag": "markdown", "content": analysis}],
},
}
def build_daily_report_card(
report_date: str,
report: str,
tracking_rows: list[dict[str, Any]],
image_key: str,
) -> dict[str, Any]:
counts = Counter(row.get("light", "无判断") for row in tracking_rows)
conclusions = _section_lines(report, "1. 今日经营概览", "2.", 5)
if not conclusions:
conclusions = _section_lines(report, "一、今日经营概览", "二、", 5)
if not conclusions:
conclusions = _section_lines(report, "一、核心经营结论", "二、", 3)
if not conclusions:
conclusions = _section_lines(report, "第一部分 整体经营分析", "第二部分 SKU分析排序", 5)
actions = _section_lines(report, "6. 今日运营建议", None, 5)
if not actions:
actions = _section_lines(report, "七、明日业务动作", None, 5)
if not actions:
actions = _section_lines(report, "四、明日业务动作", None, 3)
if not actions:
actions = _section_lines(report, "第三部分 关键预警与建议", None, 5)
conclusion_text = "\n".join(f"{idx}. {line}" for idx, line in enumerate(conclusions, 1)) or "无数据"
action_text = "\n".join(f"{idx}. {line}" for idx, line in enumerate(actions, 1)) or "无数据"
return {
"schema": "2.0",
"config": {
"update_multi": True,
"width_mode": "fill",
"enable_forward": True,
"summary": {"content": f"{report_date} 时尚包袋品牌经营日报"},
},
"header": {
"title": {"tag": "plain_text", "content": "时尚包袋品牌经营日报"},
"subtitle": {"tag": "plain_text", "content": f"{report_date} · Hermes 自动分析"},
"template": "blue",
"icon": {"tag": "standard_icon", "token": "chart_colorful"},
"text_tag_list": [
{"tag": "text_tag", "text": {"tag": "plain_text", "content": "自动日报"}, "color": "blue"}
],
},
"body": {
"direction": "vertical",
"padding": "12px 12px 20px 12px",
"vertical_spacing": "large",
"elements": [
{
"tag": "column_set",
"flex_mode": "none",
"horizontal_spacing": "12px",
"columns": [
_metric_column(counts.get("🔴", 0), "红灯款式", emphasis=True),
_metric_column(counts.get("🟡", 0), "黄灯款式"),
_metric_column(counts.get("🟢", 0), "绿灯款式"),
],
},
{
"tag": "img",
"img_key": image_key,
"alt": {"tag": "plain_text", "content": "全域种草数据看板"},
"scale_type": "fit_horizontal",
"corner_radius": "8px",
"preview": True,
},
{"tag": "markdown", "content": f"**今日经营概览**\n{conclusion_text}"},
{"tag": "markdown", "content": f"**今日运营建议**\n{action_text}"},
],
},
}
def prepare_feishu_image(source: Path, target: Path) -> Path:
target.parent.mkdir(parents=True, exist_ok=True)
with Image.open(source) as image:
image = image.convert("RGB")
if image.width > MAX_CARD_IMAGE_WIDTH:
height = round(image.height * MAX_CARD_IMAGE_WIDTH / image.width)
image = image.resize((MAX_CARD_IMAGE_WIDTH, height), Image.Resampling.LANCZOS)
image.save(target, format="PNG", optimize=True)
return target
def upload_dashboard_image(image_path: Path) -> str:
command = [
*_lark_command(), "--profile", LARK_PROFILE,
"im", "images", "create",
"--data", json.dumps({"image_type": "message"}),
"--file", f"image=./{image_path.name}",
"--as", "bot", "--format", "json",
]
completed = subprocess.run(
command,
cwd=image_path.parent,
capture_output=True,
text=True,
encoding="utf-8",
errors="replace",
)
if completed.returncode != 0:
raise RuntimeError(f"飞书看板图片上传失败: {completed.stderr or completed.stdout}")
payload = json.loads(completed.stdout)
image_key = payload.get("image_key") or (payload.get("data") or {}).get("image_key")
if not image_key:
raise RuntimeError("飞书看板图片上传成功但未返回 image_key")
return image_key
def send_card_to_recipients(
card: dict[str, Any],
recipients: Iterable[str],
report_date: str,
*,
message_kind: str = "combined",
) -> list[dict[str, Any]]:
content = json.dumps(card, ensure_ascii=False, separators=(",", ":"))
content_digest = hashlib.sha256(content.encode("utf-8")).hexdigest()[:10]
results = []
for open_id in recipients:
kind_token = re.sub(r"[^a-z0-9]", "", message_kind.lower())[:1] or "m"
idempotency_key = f"daily-{report_date}-{kind_token}-{content_digest}-{open_id[-6:]}"
command = [
*_lark_command(), "--profile", LARK_PROFILE,
"im", "+messages-send",
"--user-id", open_id,
"--msg-type", "interactive",
"--content", content,
"--idempotency-key", idempotency_key,
"--as", "bot", "--format", "json",
]
completed = subprocess.run(
command,
capture_output=True,
text=True,
encoding="utf-8",
errors="replace",
)
if completed.returncode != 0:
raise RuntimeError(f"飞书卡片发送失败({open_id}): {completed.stderr or completed.stdout}")
results.append(json.loads(completed.stdout))
return results
def send_daily_report_cards(
report_date: str,
report: str,
tracking_rows: list[dict[str, Any]],
chart_path: Path,
recipients: Iterable[str],
) -> list[dict[str, Any]]:
prepared_dir = PATHS.tmp_root / "daily_report_card"
prepared_dir.mkdir(parents=True, exist_ok=True)
prepared = prepared_dir / f"{chart_path.stem}_feishu.png"
prepare_feishu_image(chart_path, prepared)
try:
image_key = upload_dashboard_image(prepared)
finally:
prepared.unlink(missing_ok=True)
recipients = tuple(recipients)
dashboard_card = build_dashboard_card(report_date, tracking_rows, image_key)
analysis_card = build_analysis_card(report_date, report)
dashboard_results = send_card_to_recipients(
dashboard_card, recipients, report_date, message_kind="dashboard"
)
analysis_results = send_card_to_recipients(
analysis_card, recipients, report_date, message_kind="analysis"
)
return dashboard_results + analysis_results
src/gyxx_flow/modules/content_marketing/runtime/data/tools/daily_report_charts.py
+518
View File
@@ -0,0 +1,518 @@
"""使用 Pillow 生成全款式同权经营日报长图看板。"""
from pathlib import Path
from textwrap import wrap
from typing import Any, Callable
from PIL import Image, ImageDraw, ImageFont, ImageOps
try:
from .daily_dashboard_analytics import build_dashboard_facts
except ImportError:
from gyxx_flow.modules.content_marketing.runtime.data.tools.daily_dashboard_analytics import build_dashboard_facts
WIDTH, HEIGHT = 1600, 2500
BG = "#F7F8FA"
PANEL = "#FFFFFF"
TEXT = "#18212F"
MUTED = "#667085"
GRID = "#E4E7EC"
RED = "#E5484D"
YELLOW = "#E8A317"
GREEN = "#30A46C"
BLUE = "#3B82F6"
ORANGE = "#F97316"
PURPLE = "#8B5CF6"
LIGHT_BLUE = "#EFF6FF"
LIGHT_RED = "#FFF1F1"
LIGHT_YELLOW = "#FFF8E6"
PLATFORM_COLORS = {"天猫": RED, "京东": BLUE, "抖音电商": TEXT}
def _font(size: int, bold: bool = False):
candidates = [
Path(r"C:\Windows\Fonts\msyhbd.ttc" if bold else r"C:\Windows\Fonts\msyh.ttc"),
Path(r"C:\Windows\Fonts\simhei.ttf"),
]
for path in candidates:
if path.exists():
return ImageFont.truetype(str(path), size)
return ImageFont.load_default()
def _pct(value: Any) -> float:
try:
return float(value or 0)
except (TypeError, ValueError):
return 0.0
def _fmt_int(value: Any) -> str:
if value is None:
return "无数据"
return f"{int(value):,}"
def _fmt_rate(value: Any) -> str:
if value is None:
return "无数据"
return f"{float(value):.1%}"
def _sum_known(rows: list[dict[str, Any]], key: str) -> float | None:
values = [float(row[key]) for row in rows if row.get(key) is not None]
return sum(values) if values else None
def _ratio(numerator: float | None, denominator: float | None) -> float | None:
if numerator is None or denominator is None or denominator <= 0:
return None
return numerator / denominator
def build_overview_cards(rows: list[dict[str, Any]], dashboard: dict[str, Any]) -> list[dict[str, Any]]:
today_sales = _sum_known(rows, "sales")
yesterday_sales = _sum_known(rows, "sales_yesterday")
baseline_sales = _sum_known(rows, "avg_sales_7d")
today_visitors = _sum_known(rows, "visitors")
yesterday_visitors = _sum_known(rows, "visitors_yesterday")
baseline_visitors = _sum_known(rows, "avg_visitors_7d")
notes = dashboard.get("daily_new_notes", [])
note_views = [float(row["view_count"]) for row in notes if row.get("view_count") is not None]
return [
{
"key": "sales", "label": "销量",
"today": today_sales, "yesterday": yesterday_sales, "baseline_7d": baseline_sales,
"formatter": _fmt_int,
},
{
"key": "visitors", "label": "访客",
"today": today_visitors, "yesterday": yesterday_visitors, "baseline_7d": baseline_visitors,
"formatter": _fmt_int,
},
{
"key": "conversion", "label": "整体转化率",
"today": _ratio(today_sales, today_visitors),
"yesterday": _ratio(yesterday_sales, yesterday_visitors),
"baseline_7d": _ratio(baseline_sales, baseline_visitors),
"formatter": _fmt_rate,
},
{
"key": "content", "label": "新发内容曝光",
"today": sum(note_views) if note_views else None,
"yesterday": None, "baseline_7d": None,
"formatter": _fmt_int,
"note": f"昨日新发 {len(notes)} 篇｜历史曝光快照无数据",
},
]
def classify_style_action(row: dict[str, Any]) -> tuple[str, str]:
sales = row.get("sales_change_yesterday")
visitors = row.get("visitor_change_yesterday")
conversion = row.get("conversion_change_yesterday")
refund = row.get("refund_change_yesterday")
if sales is not None and conversion is not None and refund is not None and sales < 0 and conversion < 0 and refund > 0:
return "风险型", "查退款/控投放"
if refund is not None and refund >= 0.30:
return "风险型", "查退款/控投放"
if sales is not None and visitors is not None and conversion is not None and sales > 0 and visitors > 0 and conversion >= 0:
return "增长型", "加达人/备货"
if visitors is not None and conversion is not None and visitors > 0 and conversion < 0:
return "流量机会", "优化详情页"
if (row.get("cart_change_yesterday") or 0) >= 0.20 or (row.get("recent_note_exposure") or 0) > 0:
return "潜力型", "加曝光测试"
if row.get("sales") is None:
return "无数据", "补齐数据"
return "稳定型", "维护转化"
def select_chart_styles(rows: list[dict[str, Any]], limit: int = 6) -> list[dict[str, Any]]:
"""保留旧调用兼容；新看板不会再只展示这些款式。"""
anomalies = [row for row in rows if row.get("light") in {"🔴", "🟡"} and row.get("visitors") is not None]
def visitor_delta(row):
return float(row.get("visitors") or 0) - float(row.get("avg_visitors_7d") or 0)
def impact(row):
vd = abs(visitor_delta(row))
cd = abs(float(row.get("cart_users") or 0) - float(row.get("avg_cart_users_7d") or 0))
return vd + cd * 10
selected, used = [], set()
groups = [
sorted((row for row in anomalies if visitor_delta(row) < 0), key=lambda row: abs(visitor_delta(row)), reverse=True),
sorted((row for row in anomalies if visitor_delta(row) > 0 or _pct(row.get("cart_change")) > 0), key=impact, reverse=True),
sorted((row for row in anomalies if float(row.get("visitors") or 0) < 200), key=lambda row: max(abs(_pct(row.get("visitor_change"))), abs(_pct(row.get("cart_change")))), reverse=True),
]
for candidates, target in zip(groups, (2, 2, 1)):
added = 0
for row in candidates:
if row["style_name"] in used:
continue
selected.append(row)
used.add(row["style_name"])
added += 1
if added >= target or len(selected) >= limit:
break
for row in sorted(anomalies, key=impact, reverse=True):
if len(selected) >= limit:
break
if row["style_name"] not in used:
selected.append(row)
used.add(row["style_name"])
return selected[:limit]
def _panel(draw: ImageDraw.ImageDraw, box: tuple[int, int, int, int], title: str, title_size: int = 23) -> None:
draw.rounded_rectangle(box, radius=18, fill=PANEL)
draw.text((box[0] + 20, box[1] + 16), title, font=_font(title_size, True), fill=TEXT)
def _ranking_panel(
draw: ImageDraw.ImageDraw,
box: tuple[int, int, int, int],
title: str,
rows: list[dict[str, Any]],
value_key: str,
formatter: Callable[[Any], str],
empty_text: str = "无数据",
) -> None:
_panel(draw, box, title, 21)
x0, y0, x1, _ = box
if not rows:
lines = wrap(empty_text, width=22) or ["无数据"]
for idx, line in enumerate(lines[:4]):
draw.text((x0 + 20, y0 + 76 + idx * 28), line, font=_font(16), fill=MUTED)
return
max_value = max(float(row.get(value_key) or 0) for row in rows) or 1
for idx, row in enumerate(rows[:5], 1):
y = y0 + 63 + (idx - 1) * 37
draw.text((x0 + 20, y), str(idx), font=_font(16, True), fill=BLUE)
draw.text((x0 + 48, y), str(row.get("display_name") or row.get("style_name") or "无数据")[:11], font=_font(16), fill=TEXT)
bar_x, bar_w = x0 + 190, x1 - x0 - 285
width = max(2, bar_w * float(row.get(value_key) or 0) / max_value)
draw.rounded_rectangle((bar_x, y + 5, bar_x + bar_w, y + 15), radius=5, fill=GRID)
draw.rounded_rectangle((bar_x, y + 5, bar_x + width, y + 15), radius=5, fill=BLUE)
draw.text((x1 - 84, y - 1), formatter(row.get(value_key)), font=_font(15, True), fill=TEXT)
def _matrix_header(draw: ImageDraw.ImageDraw, x: int, y: int) -> None:
headers = [
("款式", 0), ("今日", 108), ("近7日", 150), ("访客", 200),
("转化", 254), ("曝光", 310), ("预测7日", 366), ("判断", 442), ("动作", 536),
]
for label, offset in headers:
draw.text((x + offset, y), label, font=_font(14, True), fill=MUTED)
def _split_style_groups(style_groups: list[dict[str, Any]]) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
if len(style_groups) < 2:
return style_groups, []
total = sum(len(group.get("styles", [])) for group in style_groups)
running = 0
best_index = 1
best_gap = total
for index, group in enumerate(style_groups[:-1], 1):
running += len(group.get("styles", []))
gap = abs(total - running * 2)
if gap < best_gap:
best_index, best_gap = index, gap
return style_groups[:best_index], style_groups[best_index:]
def _style_matrix(draw: ImageDraw.ImageDraw, box: tuple[int, int, int, int], style_groups: list[dict[str, Any]]) -> None:
total_styles = sum(len(group.get("styles", [])) for group in style_groups)
_panel(draw, box, f"三、全部 {total_styles} 款SKU表现｜按风格分组", 25)
x0, y0, x1, _ = box
draw.text(
(x1 - 760, y0 + 20),
"近7日含报告日｜预测=近3日日均×7，箭头较前4日日均｜仅作趋势估算",
font=_font(13),
fill=MUTED,
)
groups = _split_style_groups(style_groups)
column_boxes = [(x0 + 20, x0 + 750), (x0 + 790, x1 - 20)]
for category_groups, (left, right) in zip(groups, column_boxes):
header_y = y0 + 70
draw.rounded_rectangle((left, header_y - 8, right, header_y + 28), radius=8, fill=LIGHT_BLUE)
_matrix_header(draw, left + 10, header_y)
cursor_y = header_y + 38
for category_group in category_groups:
category = category_group.get("style_category") or "未分类"
rows = category_group.get("styles", [])
draw.rounded_rectangle((left, cursor_y, right, cursor_y + 26), radius=7, fill=LIGHT_YELLOW)
draw.text((left + 12, cursor_y + 3), f"{category}（{len(rows)}款）", font=_font(14, True), fill=ORANGE)
cursor_y += 31
for idx, row in enumerate(rows):
y = cursor_y + idx * 38
if idx % 2:
draw.rectangle((left, y - 4, right, y + 28), fill="#FAFBFC")
light_color = {"🔴": RED, "🟡": YELLOW, "🟢": GREEN}.get(row.get("light"), MUTED)
draw.ellipse((left + 8, y + 4, left + 18, y + 14), fill=light_color)
values = [
(str(row.get("style_name") or "无数据")[:7], 24, TEXT),
(_fmt_int(row.get("sales")), 114, TEXT),
(_fmt_int(row.get("sales_7d_total")), 156, TEXT),
(_fmt_int(row.get("visitors")), 206, TEXT),
(_fmt_rate(row.get("conversion_rate")), 260, TEXT),
(_fmt_int(row.get("recent_note_exposure")), 316, PURPLE if row.get("recent_note_exposure") else MUTED),
]
forecast = row.get("sales_forecast_7d")
forecast_change = row.get("sales_forecast_change")
if forecast is None:
forecast_text, forecast_color = "无数据", MUTED
elif forecast_change is None:
forecast_text, forecast_color = f"·{_fmt_int(forecast)}", MUTED
elif forecast_change > 0:
forecast_text, forecast_color = f"↑{_fmt_int(forecast)}", GREEN
elif forecast_change < 0:
forecast_text, forecast_color = f"↓{_fmt_int(forecast)}", RED
else:
forecast_text, forecast_color = f"→{_fmt_int(forecast)}", MUTED
diagnosis, action = classify_style_action(row)
values.extend([
(forecast_text, 372, forecast_color),
(diagnosis, 448, RED if diagnosis == "风险型" else GREEN if diagnosis == "增长型" else TEXT),
(action, 542, TEXT),
])
for value, offset, color in values:
draw.text((left + offset, y), str(value)[:10], font=_font(12, offset in (24, 448, 542)), fill=color)
cursor_y += len(rows) * 38 + 7
def _change_text(current: float | None, baseline: float | None) -> tuple[str, str]:
if current is None or baseline is None or baseline == 0:
return "无数据", MUTED
change = (current - baseline) / baseline
if change > 0:
return f"↑{change:.1%}", GREEN
if change < 0:
return f"↓{abs(change):.1%}", RED
return "持平", MUTED
def _overview_section(draw: ImageDraw.ImageDraw, box: tuple[int, int, int, int], cards: list[dict[str, Any]]) -> None:
_panel(draw, box, "一、今日经营概览｜今日 vs 昨日 vs 7日均值", 25)
x0, y0, x1, _ = box
gap = 14
card_width = (x1 - x0 - 40 - gap * 3) // 4
for idx, card in enumerate(cards[:4]):
left = x0 + 20 + idx * (card_width + gap)
right = left + card_width
top = y0 + 62
draw.rounded_rectangle((left, top, right, box[3] - 18), radius=13, fill=LIGHT_BLUE if idx < 3 else "#F8F5FF")
draw.text((left + 14, top + 12), card["label"], font=_font(18, True), fill=BLUE if idx < 3 else PURPLE)
formatter = card["formatter"]
draw.text((left + 14, top + 45), formatter(card.get("today")), font=_font(28, True), fill=TEXT)
vs_yesterday, color_y = _change_text(card.get("today"), card.get("yesterday"))
vs_baseline, color_7 = _change_text(card.get("today"), card.get("baseline_7d"))
draw.text((left + 14, top + 86), f"较昨日 {vs_yesterday}", font=_font(14, True), fill=color_y)
draw.text((left + 154, top + 86), f"较7日 {vs_baseline}", font=_font(14, True), fill=color_7)
values = [("今日", card.get("today"), BLUE), ("昨日", card.get("yesterday"), MUTED), ("7日", card.get("baseline_7d"), YELLOW)]
known = [float(value) for _, value, _ in values if value is not None]
maximum = max(known) if known else 1
for row_index, (label, value, color) in enumerate(values):
y = top + 124 + row_index * 30
draw.text((left + 14, y), label, font=_font(13), fill=MUTED)
draw.rounded_rectangle((left + 56, y + 4, right - 78, y + 14), radius=5, fill=GRID)
if value is not None:
bar_width = max(2, (right - left - 148) * float(value) / maximum)
draw.rounded_rectangle((left + 56, y + 4, left + 56 + bar_width, y + 14), radius=5, fill=color)
draw.text((right - 70, y - 1), formatter(value), font=_font(12, True), fill=TEXT if value is not None else MUTED)
if card.get("note"):
draw.text((left + 14, top + 220), str(card["note"])[:30], font=_font(12), fill=MUTED)
def _focus_styles(styles: list[dict[str, Any]], target: str) -> list[dict[str, Any]]:
candidates = []
for row in styles:
if target == "增长型":
sales_change = row.get("sales_change")
visitor_change = row.get("visitor_change")
refund_rate = row.get("refund_rate")
if sales_change is None or sales_change <= 0 or (visitor_change is not None and visitor_change < -0.05):
continue
if refund_rate is not None and refund_rate >= 0.40:
continue
score = float(sales_change) + max(float(visitor_change or 0), 0)
action = "加达人/备货" if (row.get("conversion_change_7d") or 0) >= -0.002 else "加曝光测试"
else:
sales_change = row.get("sales_change")
conversion_change = row.get("conversion_change_7d")
refund_change = row.get("refund_change")
refund_rate = row.get("refund_rate")
strict_risk = (
sales_change is not None and sales_change < 0
and conversion_change is not None and conversion_change < 0
and refund_change is not None and refund_change > 0
)
if not strict_risk and not (sales_change is not None and sales_change < 0 and (refund_rate or 0) >= 0.40):
continue
score = abs(min(float(sales_change or 0), 0)) + max(float(refund_change or 0), 0) + float(refund_rate or 0)
action = "查退款/控投放"
candidates.append((score, {**row, "prompt_diagnosis": target, "prompt_action": action}))
return [row for _, row in sorted(candidates, key=lambda item: item[0], reverse=True)[:3]]
def _focus_panel(draw: ImageDraw.ImageDraw, box: tuple[int, int, int, int], title: str, rows: list[dict[str, Any]], risk: bool = False) -> None:
_panel(draw, box, title, 22)
x0, y0, x1, _ = box
accent = RED if risk else GREEN
if not rows:
draw.text((x0 + 20, y0 + 74), "无符合规则的SKU", font=_font(16), fill=MUTED)
return
for idx, row in enumerate(rows, 1):
y = y0 + 62 + (idx - 1) * 65
draw.rounded_rectangle((x0 + 18, y, x1 - 18, y + 54), radius=10, fill=LIGHT_RED if risk else "#EFFAF4")
draw.text((x0 + 30, y + 14), str(idx), font=_font(17, True), fill=accent)
draw.text((x0 + 60, y + 8), str(row.get("style_name") or "无数据")[:12], font=_font(17, True), fill=TEXT)
sales_text, sales_color = _change_text(row.get("sales"), row.get("avg_sales_7d"))
visitor_text, visitor_color = _change_text(row.get("visitors"), row.get("avg_visitors_7d"))
draw.text((x0 + 210, y + 10), f"销量较7日 {sales_text}", font=_font(13, True), fill=sales_color)
draw.text((x0 + 350, y + 10), f"访客较7日 {visitor_text}", font=_font(13, True), fill=visitor_color)
draw.text((x0 + 500, y + 10), row.get("prompt_action") or "无数据", font=_font(14, True), fill=accent)
draw.text((x0 + 60, y + 32), f"转化 {_fmt_rate(row.get('conversion_rate'))}｜曝光 {_fmt_int(row.get('recent_note_exposure'))}", font=_font(12), fill=MUTED)
def _channel_section(
draw: ImageDraw.ImageDraw,
box: tuple[int, int, int, int],
notes: list[dict[str, Any]],
styles: list[dict[str, Any]],
) -> None:
_panel(draw, box, "四、达人营销效果｜昨日新发布内容", 25)
x0, y0, x1, _ = box
platforms = [("小红书", RED), ("B站", BLUE), ("抖音", TEXT)]
gap = 18
width = (x1 - x0 - 40 - gap * 2) // 3
for idx, (platform, color) in enumerate(platforms):
items = [row for row in notes if row.get("platform") == platform]
left = x0 + 20 + idx * (width + gap)
top = y0 + 66
draw.rounded_rectangle((left, top, left + width, top + 230), radius=13, fill=LIGHT_BLUE)
draw.text((left + 16, top + 14), platform, font=_font(20, True), fill=color)
exposure = sum(int(row.get("view_count") or 0) for row in items) if items else None
draw.text((left + 16, top + 54), f"新发 {len(items)} 篇", font=_font(19, True), fill=TEXT)
draw.text((left + 150, top + 56), f"曝光 {_fmt_int(exposure)}", font=_font(16, True), fill=TEXT)
if items:
best = max(items, key=lambda row: row.get("view_count") or 0)
_draw_wrapped(draw, f"最佳：{best.get('style_name') or '无款式'} · {best.get('creator') or '无达人'}", left + 16, top + 96, 28, 2, TEXT, 15, True)
_draw_wrapped(draw, best.get("title") or "标题无数据", left + 16, top + 146, 32, 2, MUTED, 13)
else:
draw.text((left + 16, top + 100), "无新发布内容", font=_font(15), fill=MUTED)
best_note = max(notes, key=lambda row: row.get("view_count") or 0) if notes else None
best_channel = best_note.get("platform") if best_note else "无数据"
best_style = best_note.get("style_name") if best_note else "无数据"
comment_samples = sum(int((row.get("multi_source_signals") or {}).get("comment_count") or 0) for row in styles)
summary_y = y0 + 322
summary_items = [
("最佳渠道", best_channel or "无数据", "优先复盘可量化内容"),
("最佳内容SKU", best_style or "无数据", f"曝光 {_fmt_int(best_note.get('view_count') if best_note else None)}"),
("内容反馈", f"新发 {len(notes)} 篇", "跨渠道曝光仍不完整"),
("用户洞察", f"评论样本 {comment_samples}", "样本不足则不推断"),
]
summary_width = (x1 - x0 - 40 - 14 * 3) // 4
for idx, (label, value, note) in enumerate(summary_items):
left = x0 + 20 + idx * (summary_width + 14)
draw.rounded_rectangle((left, summary_y, left + summary_width, summary_y + 150), radius=12, fill="#F8F5FF")
draw.text((left + 14, summary_y + 14), label, font=_font(15, True), fill=PURPLE)
draw.text((left + 14, summary_y + 48), str(value)[:18], font=_font(20, True), fill=TEXT)
_draw_wrapped(draw, note, left + 14, summary_y + 86, 24, 2, MUTED, 13)
draw.text((x0 + 20, box[3] - 38), "说明：曝光为数据库保存的当前快照；缺少历史快照时不推算单日播放增量。", font=_font(14), fill=MUTED)
def _draw_wrapped(
draw: ImageDraw.ImageDraw,
text: str,
x: int,
y: int,
width: int = 28,
max_lines: int = 3,
color: str = TEXT,
size: int = 14,
bold: bool = False,
) -> int:
lines = wrap(str(text or "无数据"), width=width)[:max_lines] or ["无数据"]
for line in lines:
draw.text((x, y), line, font=_font(size, bold), fill=color)
y += size + 8
return y
def _paste_product_image(canvas: Image.Image, draw: ImageDraw.ImageDraw, winner: dict[str, Any], x: int, y: int) -> None:
size = 112
path = winner.get("local_image_path")
if path:
try:
with Image.open(path) as source:
thumb = ImageOps.fit(source.convert("RGB"), (size, size), method=Image.Resampling.LANCZOS)
mask = Image.new("L", (size, size), 0)
ImageDraw.Draw(mask).rounded_rectangle((0, 0, size, size), radius=10, fill=255)
canvas.paste(thumb, (x, y), mask)
ctr = winner.get("image_ctr")
ctr_text = f" CTR {float(ctr):.1f}%" if ctr is not None else ""
draw.text((x, y + size + 7), f"图片：天猫{ctr_text}", font=_font(12), fill=MUTED)
return
except (OSError, ValueError):
pass
draw.rounded_rectangle((x, y, x + size, y + size), radius=10, fill=PANEL, outline=GRID)
draw.text((x + 17, y + 46), "图片无数据", font=_font(13), fill=MUTED)
def _winner_block(canvas: Image.Image, draw: ImageDraw.ImageDraw, x: int, y: int, width: int, title: str, winner: dict[str, Any] | None, role: str) -> int:
draw.rounded_rectangle((x, y, x + width, y + 260), radius=12, fill=LIGHT_BLUE if role == "conversion" else "#F8F5FF")
draw.text((x + 14, y + 12), title, font=_font(17, True), fill=BLUE if role == "conversion" else PURPLE)
if not winner:
draw.text((x + 14, y + 50), "无数据", font=_font(15), fill=MUTED)
return y + 260
metric = _fmt_rate(winner.get("conversion_rate")) if role == "conversion" else _fmt_int(winner.get("visitors"))
unit = "转化率" if role == "conversion" else "访客"
draw.text((x + 14, y + 45), f"{winner['style_name']}｜{unit} {metric}", font=_font(16, True), fill=TEXT)
_paste_product_image(canvas, draw, winner, x + 14, y + 80)
text_x = x + 140
persona_date = f"（{winner.get('persona_date')}）" if winner.get("persona_date") else ""
cursor = _draw_wrapped(draw, f"画像{persona_date}：{winner.get('persona_text') or '无数据'}", text_x, y + 78, 20, 3, TEXT, 13)
cursor = _draw_wrapped(draw, winner.get("interest_hypothesis") or "无数据", text_x, cursor + 2, 20, 2, MUTED, 13)
_draw_wrapped(draw, winner.get("optimization") or "无数据", text_x, cursor + 2, 20, 3, TEXT, 13)
return y + 260
def _persona_section(canvas: Image.Image, draw: ImageDraw.ImageDraw, box: tuple[int, int, int, int], cards: list[dict[str, Any]]) -> None:
_panel(draw, box, "三平台高转化 / 高访客款式画像与增长空间", 25)
x0, y0, x1, _ = box
draw.text((x1 - 570, y0 + 20), "兴趣爱好均为画像与内容方向的推测，不是数据库原始标签", font=_font(14), fill=MUTED)
gap = 16
card_width = (x1 - x0 - 40 - gap * 2) // 3
for idx, card in enumerate(cards[:3]):
x = x0 + 20 + idx * (card_width + gap)
draw.text((x, y0 + 66), card.get("platform") or "无数据", font=_font(21, True), fill=PLATFORM_COLORS.get(card.get("platform"), BLUE))
next_y = _winner_block(canvas, draw, x, y0 + 103, card_width, "转化最高", card.get("top_conversion"), "conversion")
_winner_block(canvas, draw, x, next_y + 14, card_width, "访客最高", card.get("top_visitors"), "visitors")
def generate_dashboard(report_date: str, rows: list[dict[str, Any]], enrichment: dict[str, Any], output_path: Path) -> Path:
output_path = Path(output_path)
output_path.parent.mkdir(parents=True, exist_ok=True)
dashboard = enrichment.get("dashboard") or build_dashboard_facts(
report_date, rows, enrichment.get("platform_metrics", []), enrichment.get("personas", []), {}
)
image = Image.new("RGB", (WIDTH, HEIGHT), BG)
draw = ImageDraw.Draw(image)
draw.text((42, 24), f"{report_date} 品牌SKU营销运营看板", font=_font(36, True), fill=TEXT)
draw.text((42, 74), "经营双基准｜近7日销量与未来趋势预估｜全款式同权｜达人曝光缺失不推算", font=_font(18), fill=MUTED)
overview_cards = build_overview_cards(rows, dashboard)
_overview_section(draw, (32, 112, 1568, 442), overview_cards)
styles = dashboard.get("styles", [])
_focus_panel(draw, (32, 464, 790, 730), "二、增长SKU TOP3", _focus_styles(styles, "增长型"))
_focus_panel(draw, (810, 464, 1568, 730), "风险SKU TOP3", _focus_styles(styles, "风险型"), risk=True)
style_groups = dashboard.get("style_groups") or [{"style_category": "未分类", "styles": dashboard.get("styles", [])}]
_style_matrix(draw, (32, 752, 1568, 1780), style_groups)
_channel_section(draw, (32, 1802, 1568, 2468), dashboard.get("daily_new_notes", []), styles)
image.save(output_path, format="PNG", optimize=True)
return output_path
src/gyxx_flow/modules/content_marketing/runtime/data/tools/daily_run.bat
+61
View File
@@ -0,0 +1,61 @@
@echo off
if not defined GYXX_PYTHON set "GYXX_PYTHON=python"
REM Daily task: run 3-platform V2 scrape + sync metrics to cmt_notes (INSERT missing notes every day)
REM Register: schtasks /Create /SC DAILY /TN YingxiaoYunying_DailyRun /TR %PROJECT_DIR%\data\tools\daily_run.bat /ST 06:00 /F
setlocal
for %%I in ("%~dp0..\..") do set "PROJECT_DIR=%%~fI"
if not defined GYXX_DATA_ROOT (
if not defined GYXX_PROJECT_ROOT (
echo ERROR: GYXX_DATA_ROOT or GYXX_PROJECT_ROOT must be defined. 1>&2
exit /b 3
)
set "GYXX_DATA_ROOT=%GYXX_PROJECT_ROOT%\var"
)
set LOG_DIR=%GYXX_DATA_ROOT%\logs\content_marketing\scheduled
REM Task Scheduler (SYSTEM account) has no user PATH -> lock to venv python
set PYTHON=%GYXX_PYTHON%
where %PYTHON% >nul 2>&1 || set PYTHON=python
if not exist "%LOG_DIR%" mkdir "%LOG_DIR%"
set TS=%date:~0,4%%date:~5,2%%date:~8,2%_%time:~0,2%%time:~3,2%%time:~6,2%
set TS=%TS: =0%
set LOG_FILE=%LOG_DIR%\daily_run_%TS%.log
echo === Daily run started at %date% %time% === > "%LOG_FILE%"
echo PROJECT_DIR=%PROJECT_DIR% >> "%LOG_FILE%"
cd /d "%PROJECT_DIR%"
echo --- [step 1] run_all.py (includes feishu_mapping refresh) --- >> "%LOG_FILE%"
call %PYTHON% run_all.py >> "%LOG_FILE%" 2>&1
set RC_RUN=%ERRORLEVEL%
echo run_all exit=%RC_RUN% >> "%LOG_FILE%"
echo --- [step 2] sync_metrics_to_cmt_notes.py (主采集后首次落库) --- >> "%LOG_FILE%"
call %PYTHON% data\tools\sync_metrics_to_cmt_notes.py >> "%LOG_FILE%" 2>&1
set RC_SYNC_PRE=%ERRORLEVEL%
echo sync_pre exit=%RC_SYNC_PRE% >> "%LOG_FILE%"
echo --- [step 3] retry_failed.py (B站 + 蒲公英; 星图每日只搜索一轮) --- >> "%LOG_FILE%"
call %PYTHON% data\tools\retry_failed.py --platform bili,pgy --max-attempts 1 --stale-hours 4 >> "%LOG_FILE%" 2>&1
set RC_RETRY=%ERRORLEVEL%
echo retry_failed exit=%RC_RETRY% >> "%LOG_FILE%"
echo --- [step 4] sync_metrics_to_cmt_notes.py (补跑后再次落库) --- >> "%LOG_FILE%"
call %PYTHON% data\tools\sync_metrics_to_cmt_notes.py >> "%LOG_FILE%" 2>&1
set RC_SYNC_POST=%ERRORLEVEL%
echo sync_post exit=%RC_SYNC_POST% >> "%LOG_FILE%"
set FINAL_RC=0
if not "%RC_RUN%"=="0" set FINAL_RC=1
if not "%RC_SYNC_PRE%"=="0" set FINAL_RC=1
if not "%RC_RETRY%"=="0" set FINAL_RC=1
if not "%RC_SYNC_POST%"=="0" set FINAL_RC=1
echo === Daily run finished at %date% %time% (run=%RC_RUN%, sync_pre=%RC_SYNC_PRE%, retry=%RC_RETRY%, sync_post=%RC_SYNC_POST%, final=%FINAL_RC%) === >> "%LOG_FILE%"
endlocal & exit /b %FINAL_RC%
src/gyxx_flow/modules/content_marketing/runtime/data/tools/daily_run_with_backfill.bat
+61
View File
@@ -0,0 +1,61 @@
@echo off
if not defined GYXX_PYTHON set "GYXX_PYTHON=python"
REM Monday task: run 3-platform V2 + sync metrics to cmt_notes
REM Register: schtasks /Create /SC WEEKLY /D MON /TN YingxiaoYunying_MondayBackfill /TR %PROJECT_DIR%\data\tools\daily_run_with_backfill.bat /ST 06:00 /F
setlocal
for %%I in ("%~dp0..\..") do set "PROJECT_DIR=%%~fI"
if not defined GYXX_DATA_ROOT (
if not defined GYXX_PROJECT_ROOT (
echo ERROR: GYXX_DATA_ROOT or GYXX_PROJECT_ROOT must be defined. 1>&2
exit /b 3
)
set "GYXX_DATA_ROOT=%GYXX_PROJECT_ROOT%\var"
)
set LOG_DIR=%GYXX_DATA_ROOT%\logs\content_marketing\scheduled
REM Task Scheduler (SYSTEM account) has no user PATH -> lock to venv python
set PYTHON=%GYXX_PYTHON%
where %PYTHON% >nul 2>&1 || set PYTHON=python
if not exist "%LOG_DIR%" mkdir "%LOG_DIR%"
set TS=%date:~0,4%%date:~5,2%%date:~8,2%_%time:~0,2%%time:~3,2%%time:~6,2%
set TS=%TS: =0%
set LOG_FILE=%LOG_DIR%\monday_run_%TS%.log
echo === Monday run started at %date% %time% === > "%LOG_FILE%"
echo PROJECT_DIR=%PROJECT_DIR% >> "%LOG_FILE%"
cd /d "%PROJECT_DIR%"
echo --- [step 1] run_all.py (includes feishu_mapping refresh) --- >> "%LOG_FILE%"
call %PYTHON% run_all.py >> "%LOG_FILE%" 2>&1
set RC_RUN=%ERRORLEVEL%
echo run_all exit=%RC_RUN% >> "%LOG_FILE%"
echo --- [step 2] sync_metrics_to_cmt_notes.py (主采集后首次落库) --- >> "%LOG_FILE%"
call %PYTHON% data\tools\sync_metrics_to_cmt_notes.py >> "%LOG_FILE%" 2>&1
set RC_SYNC_PRE=%ERRORLEVEL%
echo sync_pre exit=%RC_SYNC_PRE% >> "%LOG_FILE%"
echo --- [step 3] retry_failed.py --- >> "%LOG_FILE%"
call %PYTHON% data\tools\retry_failed.py --max-attempts 1 --stale-hours 4 >> "%LOG_FILE%" 2>&1
set RC_RETRY=%ERRORLEVEL%
echo retry_failed exit=%RC_RETRY% >> "%LOG_FILE%"
echo --- [step 4] sync_metrics_to_cmt_notes.py (补跑后再次落库) --- >> "%LOG_FILE%"
call %PYTHON% data\tools\sync_metrics_to_cmt_notes.py >> "%LOG_FILE%" 2>&1
set RC_SYNC_POST=%ERRORLEVEL%
echo sync_post exit=%RC_SYNC_POST% >> "%LOG_FILE%"
set FINAL_RC=0
if not "%RC_RUN%"=="0" set FINAL_RC=1
if not "%RC_SYNC_PRE%"=="0" set FINAL_RC=1
if not "%RC_RETRY%"=="0" set FINAL_RC=1
if not "%RC_SYNC_POST%"=="0" set FINAL_RC=1
echo === Monday run finished at %date% %time% (run=%RC_RUN%, sync_pre=%RC_SYNC_PRE%, retry=%RC_RETRY%, sync_post=%RC_SYNC_POST%, final=%FINAL_RC%) === >> "%LOG_FILE%"
endlocal & exit /b %FINAL_RC%
src/gyxx_flow/modules/content_marketing/runtime/data/tools/db.py
+515
View File
@@ -0,0 +1,515 @@
"""
db.py - PostgreSQL 连接池 + cmt_* 表 CRUD
统一使用 psycopg3，合并了原 yingxiaoyunying 诊断功能 + comment-data-collector 的完整 CRUD。
用法:
from gyxx_flow.modules.content_marketing.runtime.data.tools.db import make_pool, get_conn, get_dict_conn
python db.py ping # 测试连接
python db.py info # 显示 cmt_* 表行数
"""
import os
import sys
from contextlib import contextmanager
from datetime import datetime
from pathlib import Path
from typing import Any, Iterator, Mapping
from gyxx_flow.modules.content_marketing.runtime.runtime_paths import PATHS
from gyxx_flow.adapters import RuntimeServicePolicy
try:
import psycopg
from psycopg.rows import dict_row
from psycopg_pool import ConnectionPool
except ImportError:
print("[FATAL] 请先 pip install 'psycopg[binary]>=3.1' psycopg-pool", file=sys.stderr)
sys.exit(1)
try:
from dotenv import load_dotenv
except ImportError:
print("[FATAL] 请先 pip install python-dotenv", file=sys.stderr)
sys.exit(1)
BASE_DIR = PATHS.tools_root
DATA_DIR = PATHS.raw_root
ENV_PATH = PATHS.state_root / "config/db.env"
ENV_EXAMPLE = PATHS.config_root / "db.env.example"
def load_env() -> None:
if not ENV_PATH.exists():
print(f"[FATAL] 找不到 {ENV_PATH}", file=sys.stderr)
print(f" 请复制 {ENV_EXAMPLE} -> {ENV_PATH} 并填连接信息", file=sys.stderr)
sys.exit(1)
load_dotenv(ENV_PATH, override=False)
os.environ.update(RuntimeServicePolicy().apply(os.environ))
def database_config_from_env(environ: Mapping[str, str]) -> dict[str, Any]:
"""从环境变量构造连接配置；缺项立即失败，绝不回退到本地库。"""
required = ("PG_HOST", "PG_PORT", "PG_DB", "PG_USER", "PG_PASSWORD")
missing = [key for key in required if not str(environ.get(key, "")).strip()]
if missing:
raise RuntimeError(f"数据库配置缺失: {', '.join(missing)}")
try:
port = int(environ["PG_PORT"])
except (TypeError, ValueError) as exc:
raise RuntimeError("数据库配置无效: PG_PORT 必须是整数") from exc
return {
"host": environ["PG_HOST"].strip(),
"port": port,
"dbname": environ["PG_DB"].strip(),
"user": environ["PG_USER"].strip(),
"password": environ["PG_PASSWORD"],
}
def get_db_config() -> dict[str, Any]:
load_env()
return database_config_from_env(os.environ)
def make_pool(min_size: int = 1, max_size: int = 5) -> ConnectionPool:
conninfo = psycopg.conninfo.make_conninfo(**get_db_config())
return ConnectionPool(conninfo, min_size=min_size, max_size=max_size, open=True)
# 全局连接池（懒初始化）
_pool: ConnectionPool | None = None
def _get_pool() -> ConnectionPool:
global _pool
if _pool is None or _pool.closed:
_pool = make_pool()
return _pool
@contextmanager
def get_conn(pool: ConnectionPool | None = None) -> Iterator[psycopg.Connection]:
p = pool or _get_pool()
with p.connection() as conn:
yield conn
@contextmanager
def get_dict_conn(pool: ConnectionPool | None = None) -> Iterator[psycopg.Connection]:
"""返回连接，cursor 需自行指定 row_factory=dict_row。替代 psycopg2 的 RealDictCursor。"""
p = pool or _get_pool()
with p.connection() as conn:
yield conn
@contextmanager
def get_dict_cursor(pool: ConnectionPool | None = None):
"""便捷函数：返回 dict_row cursor，自动 commit/rollback。"""
with get_dict_conn(pool) as conn:
cur = conn.cursor(row_factory=dict_row)
try:
yield cur
conn.commit()
except Exception:
conn.rollback()
raise
finally:
cur.close()
def now() -> datetime:
return datetime.now()
def show_info(pool: ConnectionPool) -> None:
"""显示 cmt_* 表行数"""
with get_conn(pool) as conn:
with conn.cursor(row_factory=dict_row) as cur:
for tbl in ("cmt_styles", "cmt_creators", "cmt_style_creators", "cmt_notes", "cmt_comments"):
cur.execute(f"SELECT COUNT(*) AS n FROM {tbl}")
n = cur.fetchone()["n"]
print(f" {tbl:<25} {n:>8} 行")
# ---------------------------------------------------------------------------
# Styles
# ---------------------------------------------------------------------------
def upsert_style(
name: str,
platform: str = "",
feishu_base_url: str = "",
feishu_base_token: str = "",
feishu_table_id: str = "",
feishu_view_id: str = "",
) -> int:
sql = """
INSERT INTO cmt_styles (name, platform, feishu_base_url, feishu_base_token, feishu_table_id, feishu_view_id)
VALUES (%s, %s, %s, %s, %s, %s)
ON CONFLICT (name) DO UPDATE SET
platform = EXCLUDED.platform,
feishu_base_url = EXCLUDED.feishu_base_url,
feishu_base_token = EXCLUDED.feishu_base_token,
feishu_table_id = EXCLUDED.feishu_table_id,
feishu_view_id = EXCLUDED.feishu_view_id,
updated_at = CURRENT_TIMESTAMP
RETURNING id
"""
with get_conn() as conn:
with conn.cursor() as cur:
cur.execute(sql, (name, platform, feishu_base_url, feishu_base_token, feishu_table_id, feishu_view_id))
return cur.fetchone()[0]
def get_style_id(name: str) -> int | None:
with get_dict_conn() as conn:
with conn.cursor(row_factory=dict_row) as cur:
cur.execute("SELECT id FROM cmt_styles WHERE name = %s", (name,))
row = cur.fetchone()
return row["id"] if row else None
# ---------------------------------------------------------------------------
# Creators
# ---------------------------------------------------------------------------
def upsert_creator(name: str) -> int:
if not name:
raise ValueError("creator name cannot be empty")
sql = """
INSERT INTO cmt_creators (name)
VALUES (%s)
ON CONFLICT (name) DO UPDATE SET
updated_at = CURRENT_TIMESTAMP
RETURNING id
"""
with get_conn() as conn:
with conn.cursor() as cur:
cur.execute(sql, (name,))
return cur.fetchone()[0]
def link_style_creator(style_id: int, creator_id: int) -> None:
sql = """
INSERT INTO cmt_style_creators (style_id, creator_id)
VALUES (%s, %s)
ON CONFLICT (style_id, creator_id) DO NOTHING
"""
with get_conn() as conn:
with conn.cursor() as cur:
cur.execute(sql, (style_id, creator_id))
conn.commit()
# ---------------------------------------------------------------------------
# Notes
# ---------------------------------------------------------------------------
def upsert_note(
style_id: int,
creator_id: int,
platform: str,
url: str,
title: str = "",
feishu_record_id: str = "",
) -> int:
sql = """
INSERT INTO cmt_notes (style_id, creator_id, platform, title, url, feishu_record_id)
VALUES (%s, %s, %s, %s, %s, %s)
ON CONFLICT (url) DO UPDATE SET
style_id = EXCLUDED.style_id,
creator_id = EXCLUDED.creator_id,
platform = EXCLUDED.platform,
title = EXCLUDED.title,
feishu_record_id = EXCLUDED.feishu_record_id,
updated_at = CURRENT_TIMESTAMP
RETURNING id
"""
with get_conn() as conn:
with conn.cursor() as cur:
cur.execute(sql, (style_id, creator_id, platform, title, url, feishu_record_id))
return cur.fetchone()[0]
def update_note_metrics(
note_id: int,
like_count: int | None,
favorite_count: int | None,
comment_count: int | None,
share_count: int | None,
title: str | None = None,
) -> None:
sql = """
UPDATE cmt_notes
SET like_count = %s,
favorite_count = %s,
comment_count = %s,
share_count = %s,
title = COALESCE(%s, title),
scraped_at = CURRENT_TIMESTAMP,
updated_at = CURRENT_TIMESTAMP
WHERE id = %s
"""
with get_conn() as conn:
with conn.cursor() as cur:
cur.execute(sql, (like_count, favorite_count, comment_count, share_count, title, note_id))
conn.commit()
def update_note_view_metrics(
note_id: int,
view_count: int | None,
collect_count: int | None,
) -> None:
"""更新单笔记的曝光量/播放量、收藏量(来自 yingxiaoyunying 每日采集)。"""
sql = """
UPDATE cmt_notes
SET view_count = %s,
collect_count = %s,
updated_at = CURRENT_TIMESTAMP
WHERE id = %s
"""
with get_conn() as conn:
with conn.cursor() as cur:
cur.execute(sql, (view_count, collect_count, note_id))
conn.commit()
def get_note_id_by_url(url: str) -> int | None:
with get_dict_conn() as conn:
with conn.cursor(row_factory=dict_row) as cur:
cur.execute("SELECT id FROM cmt_notes WHERE url = %s", (url,))
row = cur.fetchone()
return row["id"] if row else None
def get_note_by_id(note_id: int) -> dict[str, Any] | None:
with get_dict_conn() as conn:
with conn.cursor(row_factory=dict_row) as cur:
cur.execute(
"""
SELECT n.*, s.name AS style_name, c.name AS creator_name
FROM cmt_notes n
LEFT JOIN cmt_styles s ON n.style_id = s.id
LEFT JOIN cmt_creators c ON n.creator_id = c.id
WHERE n.id = %s
""",
(note_id,),
)
return cur.fetchone()
def get_note_by_url(url: str) -> dict[str, Any] | None:
with get_dict_conn() as conn:
with conn.cursor(row_factory=dict_row) as cur:
cur.execute(
"""
SELECT n.*, s.name AS style_name, c.name AS creator_name
FROM cmt_notes n
LEFT JOIN cmt_styles s ON n.style_id = s.id
LEFT JOIN cmt_creators c ON n.creator_id = c.id
WHERE n.url = %s
""",
(url,),
)
return cur.fetchone()
def get_notes_by_platform(
platform: str,
only_with_comments: bool = True,
) -> list[dict[str, Any]]:
"""List all notes for a platform with style/creator joins + comment count."""
sql = """
SELECT n.*, s.name AS style_name, c.name AS creator_name,
(SELECT COUNT(*) FROM cmt_comments WHERE note_id = n.id) AS comment_total
FROM cmt_notes n
LEFT JOIN cmt_styles s ON n.style_id = s.id
LEFT JOIN cmt_creators c ON n.creator_id = c.id
WHERE n.platform = %s
"""
if only_with_comments:
sql += " AND (SELECT COUNT(*) FROM cmt_comments WHERE note_id = n.id) > 0"
sql += " ORDER BY n.id"
with get_dict_conn() as conn:
with conn.cursor(row_factory=dict_row) as cur:
cur.execute(sql, (platform,))
return cur.fetchall()
def get_all_styles_with_notes() -> list[dict[str, Any]]:
"""List all styles that have at least one note, with per-style rollup counts."""
sql = """
SELECT s.*,
COUNT(n.id) AS note_total,
COUNT(*) FILTER (WHERE EXISTS(SELECT 1 FROM cmt_comments WHERE note_id = n.id)) AS note_with_comments
FROM cmt_styles s
LEFT JOIN cmt_notes n ON n.style_id = s.id
GROUP BY s.id
HAVING COUNT(n.id) > 0
ORDER BY note_total DESC, s.name ASC
"""
with get_dict_conn() as conn:
with conn.cursor(row_factory=dict_row) as cur:
cur.execute(sql)
return cur.fetchall()
def get_notes_by_style(
style_id: int,
only_with_comments: bool = True,
) -> list[dict[str, Any]]:
"""List all notes for a single style, same field set as get_notes_by_platform."""
sql = """
SELECT n.*, s.name AS style_name, c.name AS creator_name,
(SELECT COUNT(*) FROM cmt_comments WHERE note_id = n.id) AS comment_total
FROM cmt_notes n
LEFT JOIN cmt_styles s ON n.style_id = s.id
LEFT JOIN cmt_creators c ON n.creator_id = c.id
WHERE n.style_id = %s
"""
if only_with_comments:
sql += " AND (SELECT COUNT(*) FROM cmt_comments WHERE note_id = n.id) > 0"
sql += " ORDER BY n.id ASC"
with get_dict_conn() as conn:
with conn.cursor(row_factory=dict_row) as cur:
cur.execute(sql, (style_id,))
return cur.fetchall()
def get_comments_by_note_id(note_id: int) -> list[dict[str, Any]]:
with get_dict_conn() as conn:
with conn.cursor(row_factory=dict_row) as cur:
cur.execute(
"""
SELECT * FROM cmt_comments
WHERE note_id = %s
ORDER BY comment_created_at ASC, id ASC
""",
(note_id,),
)
return cur.fetchall()
# ---------------------------------------------------------------------------
# Comments
# ---------------------------------------------------------------------------
def replace_comments(note_id: int, platform: str, comments: list[dict[str, Any]], allow_empty: bool = False) -> int:
"""Delete existing comments for the note and insert the latest batch."""
if not comments and not allow_empty:
raise ValueError("refusing to replace existing comments with an empty scrape result")
scraped_at = now()
insert_sql = """
INSERT INTO cmt_comments (
note_id, platform, level, platform_comment_id, parent_comment_id,
reply_to_comment_id, user_id, nickname, content, like_count,
reply_count, ip_location, comment_created_at, raw_create_time, scraped_at
) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
ON CONFLICT (note_id, platform_comment_id) DO UPDATE SET
level = EXCLUDED.level,
parent_comment_id = EXCLUDED.parent_comment_id,
reply_to_comment_id = EXCLUDED.reply_to_comment_id,
user_id = EXCLUDED.user_id,
nickname = EXCLUDED.nickname,
content = EXCLUDED.content,
like_count = EXCLUDED.like_count,
reply_count = EXCLUDED.reply_count,
ip_location = EXCLUDED.ip_location,
comment_created_at = EXCLUDED.comment_created_at,
raw_create_time = EXCLUDED.raw_create_time,
scraped_at = EXCLUDED.scraped_at,
updated_at = CURRENT_TIMESTAMP
"""
def to_int(value: Any) -> int | None:
try:
if value is None or value == "":
return None
return int(float(value))
except (ValueError, TypeError):
return None
def parse_dt(value: Any) -> datetime | None:
if isinstance(value, datetime):
return value
if not value:
return None
if isinstance(value, (int, float)):
ts = value / 1000 if value > 10_000_000_000 else value
try:
return datetime.fromtimestamp(ts)
except Exception:
return None
if isinstance(value, str):
for fmt in ("%Y-%m-%dT%H:%M:%S", "%Y-%m-%d %H:%M:%S", "%Y-%m-%dT%H:%M:%S%z"):
try:
return datetime.strptime(value[:19], fmt)
except ValueError:
continue
return None
rows = []
for c in comments:
platform_comment_id = str(c.get("comment_id") or c.get("cid") or c.get("rpid") or "")
parent = str(c.get("parent_comment_id") or c.get("parent_rpid") or "")
reply_to = str(c.get("reply_to_comment_id") or c.get("reply_id") or c.get("reply_to_reply_id") or "")
if not reply_to and parent:
reply_to = parent
rows.append((
note_id,
platform,
c.get("level") or "comment",
platform_comment_id,
parent,
reply_to,
str(c.get("user_id") or c.get("mid") or c.get("sec_uid") or ""),
str(c.get("nickname") or c.get("uname") or ""),
str(c.get("content") or c.get("text") or c.get("message") or ""),
to_int(c.get("like_count") or c.get("digg_count") or 0),
to_int(c.get("reply_count") or 0),
str(c.get("ip_location") or c.get("ip_label") or c.get("location") or ""),
parse_dt(c.get("created_at") or c.get("raw_create_time")),
str(c.get("raw_create_time") or ""),
scraped_at,
))
with get_conn() as conn:
with conn.cursor() as cur:
cur.execute("DELETE FROM cmt_comments WHERE note_id = %s", (note_id,))
if rows:
# Use executemany for bulk insert (psycopg3 native)
cur.executemany(insert_sql, rows)
conn.commit()
return len(rows)
def test_connection() -> bool:
with get_conn() as conn:
with conn.cursor() as cur:
cur.execute("SELECT version()")
print(cur.fetchone()[0])
return True
def cli():
if len(sys.argv) < 2:
print("用法: python db.py [ping|info]")
sys.exit(1)
cmd = sys.argv[1]
pool = make_pool()
try:
if cmd == "ping":
with get_conn(pool) as conn:
with conn.cursor() as cur:
cur.execute("SELECT version()")
print(cur.fetchone()[0])
elif cmd == "info":
show_info(pool)
else:
print(f"未知命令: {cmd}")
sys.exit(1)
finally:
pool.close()
if __name__ == "__main__":
cli()
src/gyxx_flow/modules/content_marketing/runtime/data/tools/feishu_comment_batch.py
+716
View File
@@ -0,0 +1,716 @@
import argparse
import csv
import json
import os
import re
import shutil
import subprocess
import sys
import time
from dataclasses import dataclass
from datetime import datetime
from multiprocessing import Pool
from pathlib import Path
from typing import Any
from urllib.parse import parse_qs, urlparse
from gyxx_flow.modules.content_marketing.runtime.runtime_paths import PATHS
# Path setup: db.py is in data/tools/, scrapers are in project root
_TOOLS_DIR = PATHS.tools_root
_PROJECT_ROOT = PATHS.module_root
from gyxx_flow.modules.content_marketing.runtime import bilibili_comment_scraper
from gyxx_flow.modules.content_marketing.runtime.data.tools import db
from gyxx_flow.modules.content_marketing.runtime import douyin_comment_scraper
from gyxx_flow.modules.content_marketing.runtime import xiaohongshu_comment_scraper
BASE_DIR = PATHS.module_root
DATA_DIR = PATHS.raw_root
SUMMARY_DIR = PATHS.exports_root / "summary"
DEFAULT_INDEX_BASE_TOKEN = "TtoCb1NuQaDy3NsZWTpc0GIvnph"
DEFAULT_INDEX_TABLE_ID = "tblKCjplVAFrRwMC"
DEFAULT_INDEX_VIEW_ID = "vewvy33xEk"
STYLE_NAME_FIELD = "款式"
STYLE_BASE_FIELD = "合作达人多维表格地址"
INDEX_PLATFORM_FIELD = "平台"
NOTE_URL_FIELD = "发布笔记链接"
NOTE_PLATFORM_FIELD = "投放平台"
CREATOR_FIELD = "达人名称"
TITLE_FIELD = "发布笔记标题"
@dataclass(frozen=True)
class BaseRef:
url: str
base_token: str
table_id: str
view_id: str
@dataclass(frozen=True)
class StyleTask:
record_id: str
style_name: str
platform: str
base_ref: BaseRef
@dataclass(frozen=True)
class NoteTask:
style_name: str
style_record_id: str
note_record_id: str
platform_hint: str
creator_name: str
title: str
note_url: str
source_base_url: str
def log(message: str) -> None:
print(f"[{datetime.now().strftime('%H:%M:%S')}] {message}", flush=True)
def run_lark(args: list[str]) -> dict[str, Any]:
lark_cli = shutil.which("lark-cli") or shutil.which("lark-cli.cmd") or "lark-cli"
command = [lark_cli, "base", *args, "--format", "json", "--as", "user"]
completed = subprocess.run(
command,
cwd=BASE_DIR,
text=True,
encoding="utf-8",
errors="replace",
capture_output=True,
)
if completed.returncode != 0:
raise RuntimeError(
f"lark-cli failed ({completed.returncode}): {' '.join(command)}\n"
f"stdout:\n{completed.stdout}\n\nstderr:\n{completed.stderr}"
)
try:
return json.loads(completed.stdout)
except json.JSONDecodeError as exc:
raise RuntimeError(f"Failed to parse lark-cli JSON output: {exc}\n{completed.stdout}") from exc
def list_records(base_token: str, table_id: str, view_id: str = "", limit: int = 200) -> list[dict[str, Any]]:
offset = 0
records: list[dict[str, Any]] = []
while True:
args = [
"+record-list",
"--base-token",
base_token,
"--table-id",
table_id,
"--limit",
str(limit),
"--offset",
str(offset),
]
if view_id:
args.extend(["--view-id", view_id])
payload = run_lark(args)
if not payload.get("ok"):
raise RuntimeError(f"lark-cli returned ok=false: {payload}")
data = payload.get("data") or {}
field_names = data.get("fields") or []
record_ids = data.get("record_id_list") or []
rows = data.get("data") or []
for index, row in enumerate(rows):
record = dict(zip(field_names, row))
if index < len(record_ids):
record["_record_id"] = record_ids[index]
records.append(record)
if not data.get("has_more") or not rows:
return records
offset += len(rows)
def first_text(value: Any) -> str:
if value is None:
return ""
if isinstance(value, str):
return value.strip()
if isinstance(value, (int, float)):
return str(value)
if isinstance(value, list):
parts = [first_text(item) for item in value]
return ",".join(part for part in parts if part)
if isinstance(value, dict):
for key in ("text", "name", "link", "url"):
text = first_text(value.get(key))
if text:
return text
return str(value).strip()
def extract_urls(value: Any) -> list[str]:
text = first_text(value)
if not text:
return []
markdown_urls = re.findall(r"\]\((https?://[^)\s]+)\)", text)
plain_urls = re.findall(r"https?://[^\s)\]>]+", text)
urls = markdown_urls + plain_urls
seen: set[str] = set()
unique: list[str] = []
for url in urls:
url = url.replace("&amp;", "&").strip()
if url not in seen:
seen.add(url)
unique.append(url)
return unique
def normalize_note_url(url: str) -> str:
url = url.replace("&amp;", "&").strip()
for marker in ("复制", "打开", "，", "。", " "):
if marker in url:
url = url.split(marker, 1)[0]
parsed = urlparse(url)
host = parsed.netloc.lower()
query = parse_qs(parsed.query)
modal_id = (query.get("modal_id") or [""])[0]
if "douyin.com" in host and modal_id.isdigit():
return f"https://www.douyin.com/video/{modal_id}"
path_parts = [part for part in parsed.path.split("/") if part]
if host == "v.douyin.com" and path_parts:
return f"{parsed.scheme}://{parsed.netloc}/{path_parts[0]}/"
if host in {"xhslink.com", "www.xhslink.com"} and len(path_parts) >= 2:
return f"{parsed.scheme}://{parsed.netloc}/{path_parts[0]}/{path_parts[1]}"
if host == "b23.tv" and path_parts:
return f"{parsed.scheme}://{parsed.netloc}/{path_parts[0]}"
return url
def is_supported_note_url(url: str) -> bool:
host = urlparse(url).netloc.lower()
return any(
domain in host
for domain in (
"douyin.com",
"xiaohongshu.com",
"xhslink.com",
"bilibili.com",
"b23.tv",
)
)
def parse_base_ref(url: str) -> BaseRef | None:
parsed = urlparse(url)
parts = [part for part in parsed.path.split("/") if part]
if len(parts) < 2 or parts[0] != "base":
return None
query = parse_qs(parsed.query)
table_id = (query.get("table") or [""])[0]
view_id = (query.get("view") or [""])[0]
if not table_id:
return None
return BaseRef(url=url, base_token=parts[1], table_id=table_id, view_id=view_id)
def detect_platform(url: str, platform_hint: str = "") -> str:
lowered = url.lower()
if "xiaohongshu.com" in lowered or "xhslink.com" in lowered:
return "xiaohongshu"
if "douyin.com" in lowered:
return "douyin"
if "bilibili.com" in lowered or "b23.tv" in lowered:
return "bilibili"
return "unknown"
def load_style_tasks(
index_base_token: str,
index_table_id: str,
index_view_id: str,
index_platform: str,
max_styles: int | None,
) -> list[StyleTask]:
records = list_records(index_base_token, index_table_id, index_view_id)
tasks: list[StyleTask] = []
for record in records:
platform = first_text(record.get(INDEX_PLATFORM_FIELD))
if index_platform and platform != index_platform:
continue
style_name = first_text(record.get(STYLE_NAME_FIELD))
urls = extract_urls(record.get(STYLE_BASE_FIELD))
if not urls:
continue
base_ref = parse_base_ref(urls[0])
if not base_ref:
log(f"Skip style without valid Base link: {style_name or record.get('_record_id')}")
continue
tasks.append(
StyleTask(
record_id=first_text(record.get("_record_id")),
style_name=style_name or first_text(record.get("_record_id")) or "unknown_style",
platform=platform,
base_ref=base_ref,
)
)
if max_styles and len(tasks) >= max_styles:
break
return tasks
def load_note_tasks(style_tasks: list[StyleTask], max_notes_per_style: int | None) -> list[NoteTask]:
note_tasks: list[NoteTask] = []
for style in style_tasks:
log(f"Read notes: {style.style_name}")
records = list_records(style.base_ref.base_token, style.base_ref.table_id, style.base_ref.view_id)
count_for_style = 0
for record in records:
urls = extract_urls(record.get(NOTE_URL_FIELD))
if not urls:
continue
platform_hint = first_text(record.get(NOTE_PLATFORM_FIELD))
for url in urls:
url = normalize_note_url(url)
if not is_supported_note_url(url):
log(f"Skip unsupported URL: {style.style_name} | {url}")
continue
note_tasks.append(
NoteTask(
style_name=style.style_name,
style_record_id=style.record_id,
note_record_id=first_text(record.get("_record_id")),
platform_hint=platform_hint,
creator_name=first_text(record.get(CREATOR_FIELD)),
title=first_text(record.get(TITLE_FIELD)),
note_url=url,
source_base_url=style.base_ref.url,
)
)
count_for_style += 1
if max_notes_per_style and count_for_style >= max_notes_per_style:
break
if max_notes_per_style and count_for_style >= max_notes_per_style:
break
log(f"Found {count_for_style} note links: {style.style_name}")
return note_tasks
def normalize_comment_rows(
task: NoteTask,
platform: str,
comments: list[dict[str, Any]],
note_metrics: dict[str, Any],
) -> list[dict[str, Any]]:
rows: list[dict[str, Any]] = []
for comment in comments:
comment_text = first_text(comment.get("text") or comment.get("content") or comment.get("message"))
rows.append(
{
"style_name": task.style_name,
"creator_name": task.creator_name,
"note_title": task.title,
"platform": platform,
"note_url": task.note_url,
"comment_text": comment_text,
"comment_count": note_metrics.get("comment_count", ""),
"like_count": note_metrics.get("like_count", ""),
"favorite_count": note_metrics.get("favorite_count", ""),
"share_count": note_metrics.get("share_count", ""),
}
)
return rows
# ---------------------------------------------------------------------------
# Database sync helpers
# ---------------------------------------------------------------------------
def sync_styles_to_db(style_tasks: list[StyleTask]) -> dict[str, int]:
"""Upsert styles and return {style_name: style_id}."""
style_ids: dict[str, int] = {}
for style in style_tasks:
style_id = db.upsert_style(
name=style.style_name,
platform=style.platform,
feishu_base_url=style.base_ref.url,
feishu_base_token=style.base_ref.base_token,
feishu_table_id=style.base_ref.table_id,
feishu_view_id=style.base_ref.view_id,
)
style_ids[style.style_name] = style_id
log(f"Synced {len(style_ids)} styles to database")
return style_ids
def sync_notes_to_db(note_tasks: list[NoteTask], style_ids: dict[str, int]) -> dict[str, int]:
"""Upsert creators and notes, link styles and creators. Return {url: note_id}."""
creator_ids: dict[str, int] = {}
note_ids: dict[str, int] = {}
for task in note_tasks:
style_id = style_ids.get(task.style_name)
if not style_id:
log(f"Skip DB sync for note without known style: {task.note_url}")
continue
creator_name = task.creator_name or "unknown_creator"
if creator_name not in creator_ids:
creator_ids[creator_name] = db.upsert_creator(creator_name)
creator_id = creator_ids[creator_name]
db.link_style_creator(style_id, creator_id)
platform = detect_platform(task.note_url, task.platform_hint)
note_id = db.upsert_note(
style_id=style_id,
creator_id=creator_id,
platform=platform,
url=task.note_url,
title=task.title,
feishu_record_id=task.note_record_id,
)
note_ids[task.note_url] = note_id
log(f"Synced {len(note_ids)} notes to database")
return note_ids
def save_scrape_to_db(
task: NoteTask,
platform: str,
comments: list[dict[str, Any]],
note_metrics: dict[str, Any],
note_ids: dict[str, int],
) -> int:
"""Update note metrics and replace comments in DB."""
note_id = note_ids.get(task.note_url)
if not note_id:
return 0
db.update_note_metrics(
note_id=note_id,
like_count=_to_int(note_metrics.get("like_count")),
favorite_count=_to_int(note_metrics.get("favorite_count")),
comment_count=_to_int(note_metrics.get("comment_count")),
share_count=_to_int(note_metrics.get("share_count")),
)
allow_empty = _to_int(note_metrics.get("comment_count")) == 0
if not comments and not allow_empty:
log(f"Skip replacing comments for {task.note_url}: empty scrape result without confirmed zero comments")
return 0
inserted = db.replace_comments(note_id, platform, comments, allow_empty=allow_empty)
return inserted
def _to_int(value: Any) -> int | None:
try:
if value is None or value == "":
return None
return int(float(value))
except (ValueError, TypeError):
return None
def scrape_note(
task: NoteTask,
args: argparse.Namespace,
note_ids: dict[str, int] | None = None,
) -> tuple[dict[str, Any], list[dict[str, Any]]]:
platform = detect_platform(task.note_url, task.platform_hint)
started_at = datetime.now().isoformat(timespec="seconds")
result: dict[str, Any] = {
"started_at": started_at,
"style_name": task.style_name,
"style_record_id": task.style_record_id,
"note_record_id": task.note_record_id,
"platform_hint": task.platform_hint,
"platform": platform,
"creator_name": task.creator_name,
"title": task.title,
"note_url": task.note_url,
"source_base_url": task.source_base_url,
"status": "pending",
"comment_count": 0,
"json_path": "",
"csv_path": "",
"error": "",
}
if platform == "unknown":
result["status"] = "skipped"
result["error"] = "Unsupported note URL"
return result, []
if args.dry_run:
result["status"] = "dry_run"
return result, []
try:
if platform == "xiaohongshu":
comments, result = xiaohongshu_comment_scraper.scrape_comments(
task.note_url,
args.login_timeout,
args.max_scrolls,
args.idle_rounds,
args.headless,
)
json_path, csv_path = xiaohongshu_comment_scraper.write_outputs(task.note_url, comments, result)
note_metrics = (result.get("stats") or {}).get("note_metrics", {})
elif platform == "douyin":
comments, result = douyin_comment_scraper.scrape_comments(
task.note_url,
args.login_timeout,
args.max_scrolls,
args.idle_rounds,
args.headless,
)
json_path, csv_path = douyin_comment_scraper.write_outputs(task.note_url, comments, result)
note_metrics = (result.get("stats") or {}).get("note_metrics", {})
else:
comments, result = bilibili_comment_scraper.scrape_comments(
task.note_url,
args.bilibili_max_pages,
args.bilibili_delay,
args.headless,
args.bilibili_max_reply_pages,
)
json_path, csv_path = bilibili_comment_scraper.write_outputs(task.note_url, comments, result)
note_metrics = (result.get("stats") or {}).get("note_metrics", {})
result["status"] = "ok"
result["comment_count"] = len(comments)
result["json_path"] = str(json_path)
result["csv_path"] = str(csv_path)
comment_rows = normalize_comment_rows(task, platform, comments, note_metrics)
if not args.no_db and note_ids:
try:
db_inserted = save_scrape_to_db(task, platform, comments, note_metrics, note_ids)
log(f"Saved {db_inserted} comments to DB for {task.note_url}")
except Exception as exc:
log(f"Failed to save comments to DB: {exc}")
except Exception as exc:
result["status"] = "failed"
result["error"] = repr(exc)
comment_rows = []
result["finished_at"] = datetime.now().isoformat(timespec="seconds")
return result, comment_rows
def scrape_notes_for_platform(
platform: str,
tasks: list[NoteTask],
args_dict: dict[str, Any],
note_ids: dict[str, int] | None,
) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
"""Scrape all notes for one platform in a separate process/browser."""
results: list[dict[str, Any]] = []
comment_rows: list[dict[str, Any]] = []
class SimpleArgs:
def __init__(self, data: dict[str, Any]):
self.__dict__.update(data)
platform_args = SimpleArgs(args_dict)
for index, task in enumerate(tasks, 1):
log(f"[{platform} {index}/{len(tasks)}] {task.style_name} | {task.note_url}")
result, rows = scrape_note(task, platform_args, note_ids)
results.append(result)
comment_rows.extend(rows)
if platform_args.between_notes_delay > 0 and index < len(tasks):
time.sleep(platform_args.between_notes_delay)
return results, comment_rows
def current_week_label() -> str:
"""Return a label like '2026_06_week4' for the current date."""
now = datetime.now()
week_of_month = (now.day - 1) // 7 + 1
return f"{now.year}_{now.month:02d}_week{week_of_month}"
def write_manifest(rows: list[dict[str, Any]]) -> tuple[Path, Path]:
SUMMARY_DIR.mkdir(exist_ok=True)
label = current_week_label()
json_path = SUMMARY_DIR / f"feishu_comment_batch_{label}.json"
csv_path = SUMMARY_DIR / f"feishu_comment_batch_{label}.csv"
json_path.write_text(json.dumps(rows, ensure_ascii=False, indent=2), encoding="utf-8")
fieldnames = [
"started_at",
"finished_at",
"style_name",
"style_record_id",
"note_record_id",
"platform_hint",
"platform",
"creator_name",
"title",
"note_url",
"source_base_url",
"status",
"comment_count",
"json_path",
"csv_path",
"error",
]
with csv_path.open("w", newline="", encoding="utf-8-sig") as file:
writer = csv.DictWriter(file, fieldnames=fieldnames)
writer.writeheader()
writer.writerows(rows)
return json_path, csv_path
def write_comment_summary(rows: list[dict[str, Any]]) -> tuple[Path, Path]:
SUMMARY_DIR.mkdir(exist_ok=True)
label = current_week_label()
json_path = SUMMARY_DIR / f"feishu_comments_{label}.json"
csv_path = SUMMARY_DIR / f"feishu_comments_{label}.csv"
note_counts: dict[tuple[str, str, str, str, str], int] = {}
for row in rows:
key = (
first_text(row.get("style_name")),
first_text(row.get("creator_name")),
first_text(row.get("note_title")),
first_text(row.get("platform")),
first_text(row.get("note_url")),
)
note_counts[key] = note_counts.get(key, 0) + 1
for row in rows:
key = (
first_text(row.get("style_name")),
first_text(row.get("creator_name")),
first_text(row.get("note_title")),
first_text(row.get("platform")),
first_text(row.get("note_url")),
)
if row.get("comment_count") in ("", None):
row["comment_count"] = note_counts.get(key, 0)
json_path.write_text(json.dumps(rows, ensure_ascii=False, indent=2), encoding="utf-8")
columns = [
("style_name", "\u6b3e\u5f0f"),
("creator_name", "\u535a\u4e3b\u540d"),
("note_title", "\u7b14\u8bb0\u540d"),
("platform", "\u5e73\u53f0"),
("note_url", "\u7b14\u8bb0\u94fe\u63a5"),
("comment_text", "\u8bc4\u8bba\u5185\u5bb9"),
("comment_count", "\u8bc4\u8bba\u6570"),
("like_count", "\u70b9\u8d5e\u6570"),
("favorite_count", "\u6536\u85cf\u6570"),
("share_count", "\u5206\u4eab\u91cf"),
]
with csv_path.open("w", newline="", encoding="utf-8-sig") as file:
writer = csv.DictWriter(file, fieldnames=[label for _, label in columns])
writer.writeheader()
for row in rows:
writer.writerow({label: row.get(key, "") for key, label in columns})
return json_path, csv_path
def main() -> int:
parser = argparse.ArgumentParser(description="Read Feishu Base note links and batch scrape comments.")
parser.add_argument("--index-base-token", default=DEFAULT_INDEX_BASE_TOKEN)
parser.add_argument("--index-table-id", default=DEFAULT_INDEX_TABLE_ID)
parser.add_argument("--index-view-id", default=DEFAULT_INDEX_VIEW_ID)
parser.add_argument("--index-platform", default="天猫", help="only read this platform group from the style index; use empty string for all")
parser.add_argument("--max-styles", type=int, default=None, help="limit styles for testing")
parser.add_argument("--max-notes-per-style", type=int, default=None, help="limit note links per style for testing")
parser.add_argument("--platform", choices=["all", "xiaohongshu", "douyin", "bilibili"], default="all")
parser.add_argument("--dry-run", action="store_true", help="only read Feishu and list tasks; do not scrape comments")
parser.add_argument("--headless", action="store_true")
parser.add_argument("--login-timeout", type=int, default=300)
parser.add_argument("--max-scrolls", type=int, default=120)
parser.add_argument("--idle-rounds", type=int, default=8)
parser.add_argument("--bilibili-max-pages", type=int, default=200)
parser.add_argument("--bilibili-max-reply-pages", type=int, default=200)
parser.add_argument("--bilibili-delay", type=float, default=0.25)
parser.add_argument("--between-notes-delay", type=float, default=1.0)
parser.add_argument("--no-db", action="store_true", help="do not write to PostgreSQL database")
parser.add_argument("--parallel", action="store_true", default=True, help="run one browser per platform in parallel (default)")
parser.add_argument("--no-parallel", action="store_true", help="disable parallel scraping")
args = parser.parse_args()
style_tasks = load_style_tasks(
args.index_base_token,
args.index_table_id,
args.index_view_id,
args.index_platform,
args.max_styles,
)
log(f"Found {len(style_tasks)} style Base links")
note_tasks = load_note_tasks(style_tasks, args.max_notes_per_style)
if args.platform != "all":
note_tasks = [task for task in note_tasks if detect_platform(task.note_url, task.platform_hint) == args.platform]
log(f"Prepared {len(note_tasks)} note scrape tasks")
style_ids: dict[str, int] = {}
note_ids: dict[str, int] = {}
if not args.no_db:
style_ids = sync_styles_to_db(style_tasks)
note_ids = sync_notes_to_db(note_tasks, style_ids)
results: list[dict[str, Any]] = []
comment_rows: list[dict[str, Any]] = []
parallel = args.parallel and not args.no_parallel and len(note_tasks) > 1
if parallel:
platform_groups: dict[str, list[NoteTask]] = {}
for task in note_tasks:
platform = detect_platform(task.note_url, task.platform_hint)
platform_groups.setdefault(platform, []).append(task)
active_groups = {p: t for p, t in platform_groups.items() if p in ("xiaohongshu", "douyin", "bilibili")}
if len(active_groups) > 1:
log(f"Running parallel scraping for platforms: {list(active_groups.keys())}")
args_dict = vars(args)
worker_inputs = [
(platform, tasks, args_dict, note_ids if not args.no_db else None)
for platform, tasks in active_groups.items()
]
with Pool(processes=len(worker_inputs)) as pool:
worker_outputs = pool.starmap(scrape_notes_for_platform, worker_inputs)
for worker_results, worker_rows in worker_outputs:
results.extend(worker_results)
comment_rows.extend(worker_rows)
else:
parallel = False
if not parallel:
for index, task in enumerate(note_tasks, 1):
platform = detect_platform(task.note_url, task.platform_hint)
log(f"[{index}/{len(note_tasks)}] {task.style_name} | {platform} | {task.note_url}")
result, rows = scrape_note(task, args, note_ids if not args.no_db else None)
results.append(result)
comment_rows.extend(rows)
if args.between_notes_delay > 0 and index < len(note_tasks):
time.sleep(args.between_notes_delay)
json_path, csv_path = write_manifest(results)
comments_json_path, comments_csv_path = write_comment_summary(comment_rows)
ok_count = sum(1 for row in results if row.get("status") == "ok")
failed_count = sum(1 for row in results if row.get("status") == "failed")
skipped_count = sum(1 for row in results if row.get("status") in {"skipped", "dry_run"})
log(f"Done. ok={ok_count}, failed={failed_count}, skipped/dry_run={skipped_count}")
log(f"Collected comment rows: {len(comment_rows)}")
log(f"Manifest JSON: {json_path}")
log(f"Manifest CSV: {csv_path}")
log(f"Comments JSON: {comments_json_path}")
log(f"Comments CSV: {comments_csv_path}")
return 0 if failed_count == 0 else 1
if __name__ == "__main__":
try:
raise SystemExit(main())
except KeyboardInterrupt:
print("\nInterrupted.", file=sys.stderr)
raise SystemExit(130)
src/gyxx_flow/modules/content_marketing/runtime/data/tools/feishu_doc_writer.py
+787
View File
@@ -0,0 +1,787 @@
"""Feishu Docx (v2 blocks API) writer for note analysis reports.
Wraps `lark-cli docs +create/+update --api-version v2` with:
- atomic state persistence in data/state/feishu_doc.json
- XML escaping + XML-native builders (data table, keyword distribution, etc.)
- a node + run.js Windows pattern (mirrors analyze_note.send_feishu_report) to avoid
cmd 8K argv limit and `&|<>^%` escaping pitfalls on Windows
- permission smoke test before any real write
All lark-cli calls use `--profile hermes-analyzer --as bot --format json`.
"""
import json
import os
import shutil
import subprocess
import sys
import time
from pathlib import Path
from typing import Any
from gyxx_flow.modules.content_marketing.runtime.runtime_paths import PATHS
BASE_DIR = PATHS.module_root
DATA_DIR = PATHS.raw_root
STATE_DIR = PATHS.state_root
TMP_DIR = PATHS.tmp_root / "feishu_doc_writer"
STATE_FILE = STATE_DIR / "feishu_doc.json"
PROFILE = "hermes-analyzer"
IDENTITY = "bot"
FEISHU_BASE_URL = "https://bu0zgpibak.feishu.cn"
# Cap for the LLM analysis text inside a doc section (matches analyze_comments).
DOC_ANALYSIS_LIMIT = 6000
# Below this size we pass --content inline; above we write to a file and use --content @<rel>.
INLINE_CONTENT_LIMIT = 7000
# Per skill: only escape TEXT inside tags, never the tags themselves.
# "&" must be replaced first to avoid double-escaping "&lt;" etc.
_XML_ESCAPE_MAP = {
"&": "&amp;",
"<": "&lt;",
">": "&gt;",
'"': "&quot;",
}
def xml_escape(text: Any) -> str:
"""Escape XML text content. Converts newlines to <br/>."""
if text is None:
return ""
s = str(text)
for src, dst in _XML_ESCAPE_MAP.items():
s = s.replace(src, dst)
s = s.replace("\r\n", "\n").replace("\n", "<br/>")
return s
def safe_div(numerator: Any, denominator: Any) -> float:
try:
n = float(numerator)
d = float(denominator)
if d == 0:
return 0.0
return n / d
except (TypeError, ValueError):
return 0.0
def _pct(v: float) -> str:
return f"{v * 100:.2f}%" if v else "—"
def _num(v: Any) -> str:
if v in ("", None):
return "—"
try:
return f"{int(float(v)):,}"
except (TypeError, ValueError):
return str(v)
# ---------------------------------------------------------------------------
# lark-cli subprocess wrapper
# ---------------------------------------------------------------------------
def run_lark(args: list[str], cwd: Path | None = None) -> dict[str, Any]:
"""Invoke lark-cli via node + run.js, return parsed JSON dict."""
run_js = (
Path(os.environ.get("APPDATA", str(Path.home())))
/ "npm"
/ "node_modules"
/ "@larksuite"
/ "cli"
/ "scripts"
/ "run.js"
)
# Scheduled tasks run as SYSTEM; %APPDATA% is not Administrator's dir
if not run_js.exists():
admin_run_js = Path(os.path.expandvars(r"%APPDATA%\npm\node_modules\@larksuite\cli\scripts\run.js"))
if admin_run_js.exists():
run_js = admin_run_js
if not run_js.exists():
raise RuntimeError(f"找不到 lark-cli 入口: {run_js}")
cmd = [
"node", str(run_js),
"--profile", PROFILE,
"--as", IDENTITY,
"--format", "json",
*args,
]
try:
proc = subprocess.run(
cmd,
cwd=str(cwd) if cwd else None,
capture_output=True,
text=True,
encoding="utf-8",
errors="replace",
timeout=180,
)
except subprocess.TimeoutExpired as exc:
raise RuntimeError(f"lark-cli 调用超时: {exc}") from exc
if proc.returncode != 0:
raise RuntimeError(
f"lark-cli 调用失败 (exit {proc.returncode})\n"
f"cmd head: {' '.join(args[:4])}\n"
f"stderr: {proc.stderr.strip()[:500]}\n"
f"stdout: {proc.stdout[:500]}"
)
out = proc.stdout.strip()
if not out:
return {}
try:
result = json.loads(out)
except json.JSONDecodeError as exc:
raise RuntimeError(f"lark-cli 返回非 JSON: {out[:500]}") from exc
if isinstance(result, dict) and result.get("ok") is False:
err = result.get("error") or {}
raise RuntimeError(
f"lark-cli API error: code={err.get('code')} type={err.get('type')} "
f"msg={err.get('message')}"
)
# Some API errors surface as ok=True + result="failed" + warnings. Detect those.
data = (result.get("data") or {}) if isinstance(result, dict) else {}
op_result = data.get("result") if isinstance(data, dict) else None
if op_result == "failed":
warnings = data.get("warnings") or []
raise RuntimeError(
f"lark-cli 操作失败 (result=failed): warnings={warnings}"
)
return result if isinstance(result, dict) else {}
# ---------------------------------------------------------------------------
# State persistence
# ---------------------------------------------------------------------------
def load_state() -> dict[str, Any]:
if not STATE_FILE.exists():
return {"docs": {}, "written_notes": {}}
try:
data = json.loads(STATE_FILE.read_text(encoding="utf-8"))
if not isinstance(data, dict):
return {"docs": {}, "written_notes": {}}
data.setdefault("docs", {})
data.setdefault("written_notes", {})
return data
except json.JSONDecodeError:
backup = STATE_FILE.with_suffix(f".corrupt-{int(time.time())}.json")
shutil.copy2(STATE_FILE, backup)
return {"docs": {}, "written_notes": {}, "_corrupt_backup": str(backup)}
def save_state(state: dict[str, Any]) -> None:
STATE_DIR.mkdir(parents=True, exist_ok=True)
tmp = STATE_FILE.with_suffix(".tmp")
tmp.write_text(json.dumps(state, ensure_ascii=False, indent=2), encoding="utf-8")
tmp.replace(STATE_FILE)
# ---------------------------------------------------------------------------
# Doc API wrappers
# ---------------------------------------------------------------------------
def create_doc(xml_content: str) -> dict[str, Any]:
"""Create a new Docx. Returns {token, url, revision_id}."""
result = run_lark([
"docs", "+create",
"--api-version", "v2",
"--content", xml_content,
])
doc = (result.get("data") or {}).get("document") or {}
return {
"token": doc.get("document_id") or "",
"url": doc.get("url") or "",
"revision_id": doc.get("revision_id"),
}
def _invoke_update_with_content(
doc: str, command: str, xml_content: str,
*, anchor: str | None = None,
) -> dict[str, Any]:
"""Run docs +update with either inline --content or @file content."""
args = [
"docs", "+update",
"--api-version", "v2",
"--doc", doc,
"--command", command,
]
if command.startswith("block_"):
if not anchor:
raise ValueError(f"command {command} requires --block-id")
args.extend(["--block-id", anchor])
if len(xml_content.encode("utf-8")) <= INLINE_CONTENT_LIMIT:
args.extend(["--content", xml_content])
return run_lark(args)
TMP_DIR.mkdir(parents=True, exist_ok=True)
fname = f"_docwriter_{int(time.time() * 1000)}.xml"
fpath = TMP_DIR / fname
fpath.write_text(xml_content, encoding="utf-8")
try:
# lark-cli resolves `@file` relative to cwd. We cwd=STATE_DIR but the
# file lives in STATE_DIR/tmp — pass the relative path from STATE_DIR.
rel_path = f"tmp/{fname}"
args.extend(["--content", f"@{rel_path}"])
return run_lark(args, cwd=STATE_DIR)
finally:
try:
fpath.unlink()
except OSError:
pass
def append_blocks(doc: str, xml_content: str) -> dict[str, Any]:
"""Append XML blocks to end of doc. Returns {revision_id}."""
return _invoke_update_with_content(doc, "append", xml_content)
def insert_blocks_after(doc: str, anchor_block_id: str, xml_content: str) -> dict[str, Any]:
"""Insert XML blocks after a given block. Returns {revision_id}."""
return _invoke_update_with_content(
doc, "block_insert_after", xml_content, anchor=anchor_block_id,
)
def delete_blocks(doc: str, block_ids: list[str]) -> dict[str, Any]:
"""Batch-delete blocks (comma-separated --block-id)."""
if not block_ids:
return {}
args = [
"docs", "+update",
"--api-version", "v2",
"--doc", doc,
"--command", "block_delete",
"--block-id", ",".join(block_ids),
]
return run_lark(args)
def fetch_doc_blocks(doc: str, *, scope: str = "all") -> list[dict[str, Any]]:
"""Fetch block IDs in a doc. Returns list of {block_id, block_type} dicts."""
args = [
"docs", "+fetch",
"--api-version", "v2",
"--doc", doc,
"--detail", "with-ids",
]
if scope and scope != "all":
args.extend(["--scope", scope])
result = run_lark(args)
# The fetch returns content as XML string; parse block IDs from `id="..."` attrs.
content = ((result.get("data") or {}).get("document") or {}).get("content") or ""
out: list[dict[str, Any]] = []
import re
# Extract <tagname id="..."> in document order
for m in re.finditer(r"<(\w+)\s+id=\"([^\"]+)\"", content):
out.append({"block_id": m.group(2), "block_type": m.group(1)})
return out
def find_block_after(doc: str, anchor_text: str) -> str | None:
"""Find the block_id of the first <h*> block whose text contains anchor_text.
Used to discover the h1 anchor after doc creation, since the API response
doesn't include new_blocks.
"""
blocks = fetch_doc_blocks(doc, scope="all")
for b in blocks:
if b["block_type"] in ("h1", "h2", "h3", "h4"):
# We need the actual text; refetch with content
pass
# The fetch returned content embedded; do a full re-fetch with text
result = run_lark([
"docs", "+fetch",
"--api-version", "v2",
"--doc", doc,
"--detail", "with-ids",
])
content = ((result.get("data") or {}).get("document") or {}).get("content") or ""
import re
# Find all <h1 ...>...</h1> or <h2 ...>...</h2>
pattern = re.compile(r"<(h\d)\s+id=\"([^\"]+)\"[^>]*>(.*?)</\1>", re.DOTALL)
for m in pattern.finditer(content):
if anchor_text in m.group(3):
return m.group(2)
return None
def find_blocks_after_anchor(doc: str, anchor_block_id: str) -> list[str]:
"""Return block_ids of every block that appears AFTER anchor_block_id, in order.
Used by populate_master_index to delete stale index content before re-inserting.
"""
blocks = fetch_doc_blocks(doc, scope="all")
out: list[str] = []
found = False
for b in blocks:
if found:
out.append(b["block_id"])
elif b["block_id"] == anchor_block_id:
found = True
return out
def find_index_blocks(doc: str, anchor_block_id: str) -> list[str]:
"""Return block_ids of the index content (blocks between anchor h1 and the next style section).
The index lives immediately after the anchor h1 and ends at the first
subsequent style section marker — either an H1 (the style title) or a
bookmark that precedes it. This avoids deleting style sections.
"""
blocks = fetch_doc_blocks(doc, scope="all")
out: list[str] = []
found_anchor = False
for b in blocks:
if not found_anchor:
if b["block_id"] == anchor_block_id:
found_anchor = True
continue
# Stop at the first style-section marker after the anchor.
if b["block_type"] in ("h1", "bookmark"):
break
out.append(b["block_id"])
return out
# ---------------------------------------------------------------------------
# XML builders
# ---------------------------------------------------------------------------
CATEGORY_LABELS = {
"purchase_intent": "购买意向",
"positive_feature": "正向功能反馈",
"negative_feature": "负向功能反馈",
"user_scenario": "用户场景",
"aesthetic": "审美与风格",
"brand_emotion": "品牌情感",
}
def build_basic_info_xml(info: dict[str, Any]) -> str:
title = info.get("title") or "未获取"
platform = info.get("platform") or "unknown"
creator = info.get("creator_name") or "未指定"
style_name = info.get("style_name") or ""
source_url = info.get("source_url") or ""
brand = info.get("brand") or ""
target_label = " / ".join(x for x in (brand, style_name) if x) if (brand or style_name) else "未指定"
lines = [
f"<p><b>标题</b>　{xml_escape(title)}</p>",
f"<p><b>平台</b>　{platform}　·　<b>达人</b>　{xml_escape(creator)}</p>",
f"<p><b>品牌 / 产品</b>　{xml_escape(target_label)}</p>",
]
if source_url:
lines.append(f'<p><b>链接</b>　<a href="{xml_escape(source_url)}">{xml_escape(source_url)}</a></p>')
return "<callout emoji=\"📌\" background-color=\"light-blue\" border-color=\"blue\">" + "".join(lines) + "</callout>"
def build_data_table_xml(
info: dict[str, Any], metrics: dict[str, Any], raw_total: int, filtered_total: int,
) -> str:
"""XML table mirroring analyze_note._data_table_markdown."""
has_view = bool(metrics.get("view_count"))
likes = info.get("like_count")
favorites = info.get("favorite_count")
comments = info.get("comment_count_metric") or info.get("comment_count_total")
shares = info.get("share_count")
valid_ratio = safe_div(filtered_total, raw_total)
rows: list[str] = []
rows.append(_tr(("维度", "指标", "数值"), header=True))
rows.append(_tr(("基础", "点赞", _num(likes))))
rows.append(_tr(("基础", "收藏", _num(favorites))))
rows.append(_tr(("基础", "评论", _num(comments))))
rows.append(_tr(("基础", "分享", _num(shares))))
rows.append(_tr(("传播", "曝光量", _num(metrics.get("view_count")) if has_view else "未采集")))
rows.append(_tr(("互动率", "点赞率", _pct(metrics.get("like_rate", 0)))))
rows.append(_tr(("互动率", "评论率", _pct(metrics.get("comment_rate", 0)))))
rows.append(_tr(("互动率", "收藏率", _pct(metrics.get("favorite_rate", 0)))))
rows.append(_tr(("互动率", "分享率", _pct(metrics.get("share_rate", 0)))))
rows.append(_tr(("互动率", "综合互动率", _pct(metrics.get("engagement_rate", 0)))))
fav_to_cmt = metrics.get("favorite_to_comment", 0) or 0
rows.append(_tr(("互动率", "收藏评论比", f"{fav_to_cmt:.2f}")))
rows.append(_tr(("质量", "有效率", _pct(valid_ratio))))
return (
'<callout emoji="📊" background-color="light-blue" border-color="blue">'
"<table>"
"<colgroup><col span=\"3\" width=\"120\"/></colgroup>"
"<thead>" + rows[0] + "</thead>"
"<tbody>" + "".join(rows[1:]) + "</tbody>"
"</table>"
"</callout>"
)
def _tr(cells: tuple[str, str, str], *, header: bool = False) -> str:
tag = "th" if header else "td"
bg = ' background-color="light-gray"' if header else ""
return (
f"<tr>"
+ "".join(f"<{tag}{bg}>{xml_escape(c)}</{tag}>" for c in cells)
+ "</tr>"
)
def build_keyword_distribution_xml(
distribution: dict[str, Any], total_filtered: int,
) -> str:
overall = distribution.get("overall", {}) if isinstance(distribution, dict) else {}
relevant = distribution.get("relevant", {}) if isinstance(distribution, dict) else {}
relevant_total = distribution.get("relevant_total", 0) if isinstance(distribution, dict) else 0
if not overall:
return "<p>（无关键词分布数据）</p>"
def fmt(d: dict[str, int]) -> str:
return "".join(
f"<li>{xml_escape(CATEGORY_LABELS[k])}：{d.get(k, 0)} 条</li>"
for k in CATEGORY_LABELS
)
overall_block = (
f"<p><b>全评论（{total_filtered} 条）</b></p>"
f"<ul>{fmt(overall)}</ul>"
)
if relevant_total:
relevant_block = (
f"<p><b>我方产品相关（{relevant_total} 条）</b></p>"
f"<ul>{fmt(relevant)}</ul>"
)
else:
relevant_block = "<p><b>我方产品相关（0 条）</b></p><p><i>（无评论明确提及我方产品/品牌）</i></p>"
return (
'<callout emoji="🔑" background-color="light-cyan" border-color="cyan">'
+ overall_block + relevant_block
+ "</callout>"
)
# 模块 emoji 和颜色映射
_MODULE_STYLES = {
"模块1": ("🎯", "light-blue", "blue"), # 笔记表现归因
"模块2": ("📊", "light-green", "green"), # 评论质量分层
"模块3": ("🛡️", "light-red", "red"), # 品牌安全与控评
"模块4": ("🎬", "light-purple", "purple"), # 脚本有效性拆解
"模块5": ("🤝", "light-yellow", "orange"), # 达人匹配度评估
"模块6": ("💡", "light-cyan", "cyan"), # 可沉淀的选题资产
"模块7": ("✅", "light-green", "green"), # 下一篇可执行优化清单
}
def _split_analysis_modules(analysis_text: str) -> list[str]:
"""将 LLM 输出的 7 模块分析拆分为独立 callout 块。
每个模块用不同颜色的 callout 渲染，视觉层次更分明。
"""
if not analysis_text or analysis_text == "（无分析结果）":
return ['<callout emoji="💡" background-color="light-gray" border-color="gray">'
'<p>（无分析结果）</p></callout>']
# 按模块标记拆分
import re as _re
parts = _re.split(r'(模块\d+[｜|])', analysis_text)
# parts: [前缀, "模块1｜", 内容1, "模块2｜", 内容2, ...]
blocks: list[str] = []
# 处理模块前的引言文字（如有）
if parts and parts[0].strip():
intro = parts[0].strip()
if len(intro) > 20:
blocks.append(f'<callout emoji="📋" background-color="light-gray" border-color="gray">'
f'<p>{xml_escape(intro)}</p></callout>')
# 逐模块渲染
i = 1
while i < len(parts) - 1:
module_header = parts[i] # "模块1｜"
module_body = parts[i + 1] # 该模块的内容
i += 2
# 提取模块编号
m = _re.match(r'模块(\d+)', module_header)
if not m:
continue
mod_num = m.group(1)
mod_key = f"模块{mod_num}"
emoji, bg, border = _MODULE_STYLES.get(mod_key, ("📌", "light-gray", "gray"))
# 模块标题行 + 正文
body_text = module_body.strip()
if not body_text:
continue
# 截断过长内容
if len(body_text) > 800:
body_text = body_text[:800] + "\n...（已截断）"
blocks.append(
f'<callout emoji="{emoji}" background-color="{bg}" border-color="{border}">'
f'<p>{xml_escape(body_text)}</p></callout>'
)
# 如果没有拆出模块（LLM 没按格式输出），整体兜底
if not blocks:
text = analysis_text
if len(text) > DOC_ANALYSIS_LIMIT:
text = text[:DOC_ANALYSIS_LIMIT] + "\n...（已截断）"
blocks.append(f'<callout emoji="💡" background-color="light-blue" border-color="blue">'
f'<p>{xml_escape(text)}</p></callout>')
return blocks
def build_note_section_xml(
info: dict[str, Any],
metrics: dict[str, Any],
distribution: dict[str, Any],
raw_comments: list[str],
filtered_comments: list[str],
analysis: str,
*,
bookmark_name: str,
doc_token: str,
) -> str:
title = info.get("title") or info.get("source_url") or "未获取"
platform = info.get("platform") or "unknown"
# 截断标题用于 H3，避免超长
short_title = title[:30] + "…" if len(title) > 30 else title
h_text = f"📝 {short_title}（{platform}）"
analysis_text = analysis or "（无分析结果）"
if len(analysis_text) > DOC_ANALYSIS_LIMIT:
analysis_text = analysis_text[:DOC_ANALYSIS_LIMIT] + "<br/>...（内容过长，已截断）"
# Bookmark requires href attribute; use the doc's own URL (target for incoming #bookmark links).
bookmark_href = f"{FEISHU_BASE_URL}/docx/{doc_token}#{bookmark_name}"
# 将分析文本按模块拆分，逐段渲染
analysis_parts = _split_analysis_modules(analysis_text)
return (
f"<bookmark name=\"{xml_escape(bookmark_name)}\" href=\"{xml_escape(bookmark_href)}\"></bookmark>"
f"<h3>{xml_escape(h_text)}</h3>"
+ build_basic_info_xml(info)
+ build_data_table_xml(info, metrics, len(raw_comments), len(filtered_comments))
+ build_keyword_distribution_xml(distribution, len(filtered_comments))
+ "".join(analysis_parts)
)
def build_master_summary_table_xml(
per_platform: dict[str, dict[str, Any]],
) -> str:
"""Build the 4-row master summary table (平台/笔记数/详情)."""
rows = []
for p, info in per_platform.items():
url = info.get("url") or ""
n = info.get("count", 0)
link_cell = f'<a href="{xml_escape(url)}">打开 →</a>' if url else "—"
rows.append(
f"<tr><td>{xml_escape(p)}</td><td>{n}</td><td>{link_cell}</td></tr>"
)
return (
"<table>"
"<colgroup><col width=\"120\"/><col width=\"80\"/><col width=\"300\"/></colgroup>"
"<thead>" + _tr(("平台", "笔记数", "详情"), header=True) + "</thead>"
"<tbody>" + "".join(rows) + "</tbody>"
"</table>"
)
def build_per_platform_index_table_xml(
platform: str, state: dict[str, Any], platform_doc_url: str,
) -> tuple[str, int]:
"""Build index table XML for one platform section in the master doc.
Returns (xml_string, row_count).
"""
rows: list[str] = []
for nid_str, entry in state.get("written_notes", {}).items():
if entry.get("platform") != platform:
continue
bookmark = entry.get("bookmark_name") or f"note-{nid_str}"
link = f"{platform_doc_url}#{bookmark}"
rows.append(
f'<tr><td>{nid_str}</td>'
f'<td><a href="{xml_escape(link)}">查看 →</a></td></tr>'
)
rows.sort(key=lambda r: int(re.search(r">(\d+)<", r).group(1)) if re.search(r">(\d+)<", r) else 0)
body = "".join(rows)
return (
f"<h2>📑 {platform} 索引（{len(rows)} 条）</h2>"
"<table>"
"<colgroup><col width=\"100\"/><col width=\"180\"/></colgroup>"
"<thead>" + _tr(("note_id", "详情"), header=True) + "</thead>"
"<tbody>" + body + "</tbody>"
"</table>"
), len(rows)
# Local re for sort
import re # noqa: E402
# ---------------------------------------------------------------------------
# Per-style doc builders (used by single-archive layout)
# ---------------------------------------------------------------------------
def _doc_skeleton() -> str:
"""Initial skeleton for the single archive doc.
Title + intro + H1 anchor placeholder for the 款式索引. The actual index
table is populated later via populate_index_table() (replace-on-update).
"""
return (
"<title>👜 笔记分析报告 — 按款式汇总</title>"
"<p>本汇总由 write_notes_to_doc.py 自动生成。点击款式名称进入详情。</p>"
+ "<h1>⚠️ 款式索引（点击跳转）</h1>"
)
def build_style_overview_xml(style_name: str, aggregated: dict[str, Any]) -> str:
"""Callout block with style-level rollup metrics."""
plat = aggregated.get("platform_breakdown") or {}
plat_str = " / ".join(f"{k} {v}" for k, v in plat.items() if v) or "—"
creator = aggregated.get("creator_breakdown") or {}
creator_str = " / ".join(f"{k}×{v}" for k, v in list(creator.items())[:3]) or "—"
lines = [
f"<p><b>款式</b>　{xml_escape(style_name)}</p>",
f"<p><b>笔记总数</b>　{aggregated.get('note_count', 0)} 篇　·　"
f"<b>平台分布</b>　{xml_escape(plat_str)}</p>",
f"<p><b>头部达人</b>　{xml_escape(creator_str)}</p>",
f"<p><b>总互动</b>　"
f"点赞 {_num(aggregated.get('total_likes'))}　·　"
f"收藏 {_num(aggregated.get('total_favorites'))}　·　"
f"评论 {_num(aggregated.get('total_comments'))}　·　"
f"分享 {_num(aggregated.get('total_shares'))}</p>",
f"<p><b>总曝光量</b>　{_num(aggregated.get('total_views')) if aggregated.get('total_views') else '未采集'}　·　"
f"<b>综合互动率</b>　{_pct(aggregated.get('engagement_rate') or 0)}</p>",
]
return (
'<callout emoji="👜" background-color="light-yellow" border-color="orange">'
+ "".join(lines)
+ "</callout>"
)
def build_style_section_xml(
style_id: int,
style_name: str,
aggregated: dict[str, Any],
rollup_text: str,
note_sections_xml: str,
*,
doc_token: str,
) -> str:
"""Assemble one full style section: bookmark + H1 + overview + rollup H2 + per-note sections.
`note_sections_xml` should be the concatenation of `build_note_section_xml(...)`
outputs for each note in this style (each already has its own bookmark + H3).
"""
bookmark_name = f"style-{style_id}"
bookmark_href = f"{FEISHU_BASE_URL}/docx/{doc_token}#{bookmark_name}"
h1_text = f"👜 {style_name}（{aggregated.get('note_count', 0)} 篇笔记）"
note_count = aggregated.get("note_count", 0)
rollup_in_callout = (
'<callout emoji="💼" background-color="light-purple" border-color="purple">'
f'<p>{xml_escape(rollup_text or "（无整体分析结果）")}</p></callout>'
)
parts: list[str] = [
f'<bookmark name="{xml_escape(bookmark_name)}" href="{xml_escape(bookmark_href)}"></bookmark>',
f"<h1>{xml_escape(h1_text)}</h1>",
build_style_overview_xml(style_name, aggregated),
"<h2>💼 款式整体分析</h2>",
rollup_in_callout,
f"<h2>📚 单篇笔记分析（{note_count} 篇）</h2>",
]
if note_sections_xml:
parts.append(note_sections_xml)
return "".join(parts)
def build_style_index_table_xml(
written_styles: dict[str, Any],
*,
archive_url: str,
) -> str:
"""Build the 款式索引 table at the top of the archive doc.
Rows are sorted by note count DESC (driven by caller-supplied dict order,
which write_notes_to_doc.py maintains from db.get_all_styles_with_notes).
Each row: 款式名 | 笔记数 | 跳转链接.
"""
rows: list[str] = []
for sid_str, entry in written_styles.items():
if not entry.get("doc_token"):
continue
# Find note count from entry (carried over by writer)
n = entry.get("note_count", 0)
bookmark = entry.get("bookmark_name") or f"style-{sid_str}"
link = f"{archive_url}#{bookmark}"
rows.append(
f"<tr><td>{xml_escape(str(entry.get('style_name', sid_str)))}</td>"
f"<td>{n}</td>"
f'<td><a href="{xml_escape(link)}">跳转 →</a></td></tr>'
)
if not rows:
rows.append("<tr><td>—</td><td>0</td><td>—</td></tr>")
return (
"<table>"
"<colgroup><col width=\"160\"/><col width=\"80\"/><col width=\"300\"/></colgroup>"
"<thead>" + _tr(("款式", "笔记数", "详情"), header=True) + "</thead>"
"<tbody>" + "".join(rows) + "</tbody>"
"</table>"
)
# ---------------------------------------------------------------------------
# Permission smoke test
# ---------------------------------------------------------------------------
def permission_smoke_test() -> bool:
print("→ Running permission smoke test (--as bot, docs:document:create)...")
try:
run_lark([
"docs", "+create",
"--api-version", "v2",
"--content", "<title>__smoke_test__</title><p>please ignore</p>",
])
except RuntimeError as exc:
print(f" ✗ Smoke test failed: {exc}")
print()
print(" The hermes-analyzer bot identity lacks the required scopes.")
print(" Run: lark-cli auth login --profile hermes-analyzer")
print(" And ensure the app is granted these scopes:")
print(" - docs:document:create")
print(" - docs:document:update")
print(" - docs:document:read")
print(" Then retry.")
return False
print(f" ✓ Smoke test passed. (A throwaway doc was created — delete it manually from Drive.)")
return True
if __name__ == "__main__":
sys.exit(0 if permission_smoke_test() else 1)
src/gyxx_flow/modules/content_marketing/runtime/data/tools/friday_relogin.bat
+39
View File
@@ -0,0 +1,39 @@
@echo off
if not defined GYXX_PYTHON set "GYXX_PYTHON=python"
REM Friday relogin: 4 QR scans — B站, 蒲公英, 星图(+同步到抖音评论), 小红书评论
REM 星图登录完成后自动把 cookie 同步到抖音评论 scraper,不需要单独再扫抖音码
REM Register: schtasks /Create /SC WEEKLY /D FRI /TN YingxiaoYunying_FridayRelogin /TR %PROJECT_DIR%\data\tools\friday_relogin.bat /ST 10:00 /F
setlocal
for %%I in ("%~dp0..\..") do set "PROJECT_DIR=%%~fI"
if not defined GYXX_DATA_ROOT (
if not defined GYXX_PROJECT_ROOT (
echo ERROR: GYXX_DATA_ROOT or GYXX_PROJECT_ROOT must be defined. 1>&2
exit /b 3
)
set "GYXX_DATA_ROOT=%GYXX_PROJECT_ROOT%\var"
)
set LOG_DIR=%GYXX_DATA_ROOT%\logs\content_marketing\scheduled
set PYTHON=%GYXX_PYTHON%
where %PYTHON% >nul 2>&1 || set PYTHON=python
if not exist "%LOG_DIR%" mkdir "%LOG_DIR%"
set TS=%date:~0,4%%date:~5,2%%date:~8,2%_%time:~0,2%%time:~3,2%%time:~6,2%
set TS=%TS: =0%
set LOG_FILE=%LOG_DIR%\friday_relogin_%TS%.log
echo === Friday relogin started at %date% %time% === > "%LOG_FILE%"
echo PROJECT_DIR=%PROJECT_DIR% >> "%LOG_FILE%"
cd /d "%PROJECT_DIR%"
echo --- Launch 5 QR relogins in parallel; send screenshots; retry up to 3 rounds --- >> "%LOG_FILE%"
call %PYTHON% data\tools\friday_relogin_parallel.py --max-attempts 3 --round-timeout 300 --screenshot-delay 60 >> "%LOG_FILE%" 2>&1
set FINAL_RC=%ERRORLEVEL%
echo === Friday relogin finished at %date% %time% (exit=%FINAL_RC%) === >> "%LOG_FILE%"
endlocal & exit /b %FINAL_RC%
src/gyxx_flow/modules/content_marketing/runtime/data/tools/friday_relogin_parallel.py
+408
View File
@@ -0,0 +1,408 @@
"""Launch five QR relogins in parallel, notify via analyzer Lark, retry 3 times."""
from __future__ import annotations
import argparse
import ctypes
from ctypes import wintypes
import datetime as dt
import json
import os
import shutil
import subprocess
import sys
import time
from dataclasses import dataclass
from pathlib import Path
from typing import Callable
import psutil
from gyxx_flow.modules.content_marketing.runtime.runtime_paths import PATHS
PROJECT_DIR = PATHS.module_root
ANALYZER_DIR = PATHS.tmp_root / "relogin"
MARKETING_RECIPIENT = "ou_24cc944d6e43c69c59d6560ad4e2ae6e"
TECHNICAL_RECIPIENT = "ou_7ad5fc8012e2f741afc5346e05ffd447"
@dataclass(frozen=True)
class Platform:
label: str
command: tuple[str, ...]
cookie_file: Path
required_cookie: str
PLATFORMS = {
"douyin": Platform(
"\u6296\u97f3", ("data/tools/relogin_douyin.py",),
PATHS.state_root / "cookies/douyin_cookies.json", "sessionid",
),
"bilibili": Platform(
"B\u7ad9", ("data/tools/relogin_bilibili.py",),
PATHS.state_root / "cookies/bilibili_cookies.json", "SESSDATA",
),
"pgy": Platform(
"\u84b2\u516c\u82f1", ("data/tools/relogin_pgy.py",),
PATHS.state_root / "cookies/pgy_cookies.json", "",
),
"xingtu": Platform(
"\u661f\u56fe", ("data/tools/relogin_xingtu.py", "--force"),
PATHS.state_root / "cookies/xingtu_cookies.json", "sessionid",
),
"xiaohongshu": Platform(
"\u5c0f\u7ea2\u4e66", ("data/tools/relogin_xiaohongshu.py",),
PATHS.state_root / "cookies/xiaohongshu_cookies.json", "web_session",
),
}
DEFAULT_RECIPIENTS = {
"pgy": MARKETING_RECIPIENT,
"xingtu": MARKETING_RECIPIENT,
"douyin": TECHNICAL_RECIPIENT,
"bilibili": TECHNICAL_RECIPIENT,
"xiaohongshu": TECHNICAL_RECIPIENT,
}
def recipient_for_platform(platform_name: str, override: str | None = None) -> str:
"""Return the responsible recipient, unless a manual all-platform override is set."""
if override:
return override
try:
return DEFAULT_RECIPIENTS[platform_name]
except KeyError as exc:
raise ValueError(f"no Lark recipient configured for platform: {platform_name}") from exc
def group_platforms_by_recipient(
platform_names: list[str], override: str | None = None,
) -> dict[str, list[str]]:
grouped: dict[str, list[str]] = {}
for name in platform_names:
grouped.setdefault(recipient_for_platform(name, override), []).append(name)
return grouped
def run_with_retries(
platform_names: list[str],
*,
max_attempts: int,
run_round: Callable[[list[str], int], dict[str, bool]],
) -> dict[str, bool]:
status = {name: False for name in platform_names}
pending = list(platform_names)
for attempt in range(1, max_attempts + 1):
if not pending:
break
round_result = run_round(pending, attempt)
for name in pending:
status[name] = bool(round_result.get(name))
pending = [name for name in pending if not status[name]]
return status
def _valid_cookie(platform: Platform) -> bool:
if not platform.cookie_file.is_file():
return False
if not platform.required_cookie:
return platform.cookie_file.stat().st_size > 10
try:
import json
values = json.loads(platform.cookie_file.read_text(encoding="utf-8"))
return any(
item.get("name") == platform.required_cookie and item.get("value")
for item in values if isinstance(item, dict)
)
except Exception:
return False
def _descendant_pids(pid: int) -> set[int]:
try:
return {child.pid for child in psutil.Process(pid).children(recursive=True)}
except (psutil.Error, OSError):
return set()
def _terminate_process_tree(process: subprocess.Popen) -> None:
"""Terminate the whole Windows subprocess tree, including Chrome/Playwright."""
try:
subprocess.run(
["taskkill", "/PID", str(process.pid), "/T", "/F"],
capture_output=True,
text=True,
check=False,
)
except OSError:
process.terminate()
try:
process.wait(timeout=10)
except subprocess.TimeoutExpired:
process.kill()
def _window_for_process(pid: int) -> int | None:
pids = _descendant_pids(pid) | {pid}
handles: list[int] = []
user32 = ctypes.windll.user32
callback_type = ctypes.WINFUNCTYPE(ctypes.c_bool, ctypes.c_void_p, ctypes.c_void_p)
def callback(hwnd, _lparam):
process_id = ctypes.c_ulong()
user32.GetWindowThreadProcessId(hwnd, ctypes.byref(process_id))
if process_id.value in pids and user32.IsWindowVisible(hwnd):
handles.append(int(hwnd))
return True
user32.EnumWindows(callback_type(callback), 0)
return handles[0] if handles else None
def _platform_from_window_title(title: str) -> str | None:
"""Identify a login window without relying on Chrome's process tree."""
if "\u84b2\u516c\u82f1" in title:
return "pgy"
if "\u6296\u97f3\u6388\u6743" in title or "\u5de8\u91cf\u661f\u56fe" in title:
return "xingtu"
if "\u6296\u97f3" in title:
return "douyin"
if "bilibili" in title.lower() or "\u8d26\u53f7\u767b\u5f55" in title:
return "bilibili"
if "\u5c0f\u7ea2\u4e66" in title:
return "xiaohongshu"
return None
def _login_windows() -> dict[str, int]:
"""Map visible Chrome login windows to platforms by their page titles."""
user32 = ctypes.windll.user32
found: dict[str, int] = {}
callback_type = ctypes.WINFUNCTYPE(ctypes.c_bool, ctypes.c_void_p, ctypes.c_void_p)
def callback(hwnd, _lparam):
if not user32.IsWindowVisible(hwnd):
return True
length = user32.GetWindowTextLengthW(hwnd)
if length <= 0:
return True
buffer = ctypes.create_unicode_buffer(length + 1)
user32.GetWindowTextW(hwnd, buffer, length + 1)
platform = _platform_from_window_title(buffer.value)
if platform:
found[platform] = int(hwnd)
return True
user32.EnumWindows(callback_type(callback), 0)
return found
def _capture_window(hwnd: int, output: Path) -> None:
"""Maximize and render one window directly, even when it is not foreground."""
from PIL import Image
user32 = ctypes.windll.user32
gdi32 = ctypes.windll.gdi32
user32.ShowWindow(hwnd, 3) # SW_MAXIMIZE
time.sleep(1)
rect = wintypes.RECT()
user32.GetWindowRect(hwnd, ctypes.byref(rect))
width, height = rect.right - rect.left, rect.bottom - rect.top
if width <= 0 or height <= 0:
raise RuntimeError(f"invalid window size: {width}x{height}")
window_dc = user32.GetWindowDC(hwnd)
memory_dc = gdi32.CreateCompatibleDC(window_dc)
bitmap = gdi32.CreateCompatibleBitmap(window_dc, width, height)
previous = gdi32.SelectObject(memory_dc, bitmap)
try:
if not user32.PrintWindow(hwnd, memory_dc, 2): # PW_RENDERFULLCONTENT
raise RuntimeError("PrintWindow failed")
class BitmapInfoHeader(ctypes.Structure):
_fields_ = [
("biSize", wintypes.DWORD), ("biWidth", wintypes.LONG),
("biHeight", wintypes.LONG), ("biPlanes", wintypes.WORD),
("biBitCount", wintypes.WORD), ("biCompression", wintypes.DWORD),
("biSizeImage", wintypes.DWORD),
("biXPelsPerMeter", wintypes.LONG),
("biYPelsPerMeter", wintypes.LONG),
("biClrUsed", wintypes.DWORD),
("biClrImportant", wintypes.DWORD),
]
byte_count = width * height * 4
pixels = ctypes.create_string_buffer(byte_count)
header = BitmapInfoHeader(
ctypes.sizeof(BitmapInfoHeader), width, -height, 1, 32, 0,
byte_count, 0, 0, 0, 0,
)
rows = gdi32.GetDIBits(
memory_dc, bitmap, 0, height, pixels, ctypes.byref(header), 0
)
if rows != height:
raise RuntimeError(f"GetDIBits returned {rows}/{height} rows")
image = Image.frombuffer(
"RGB", (width, height), pixels, "raw", "BGRX", 0, 1
).copy()
finally:
gdi32.SelectObject(memory_dc, previous)
gdi32.DeleteObject(bitmap)
gdi32.DeleteDC(memory_dc)
user32.ReleaseDC(hwnd, window_dc)
output.parent.mkdir(parents=True, exist_ok=True)
image.save(output, "PNG")
def _tile_windows(handles: list[int]) -> None:
"""Place four login windows in a 2x2 grid so screenshots cannot overlap."""
user32 = ctypes.windll.user32
screen_w = user32.GetSystemMetrics(0)
screen_h = user32.GetSystemMetrics(1)
cell_w = max(640, screen_w // 2)
cell_h = max(450, screen_h // 2)
for index, hwnd in enumerate(handles):
x = (index % 2) * cell_w
y = (index // 2) * cell_h
user32.ShowWindow(hwnd, 9) # SW_RESTORE
user32.MoveWindow(hwnd, x, y, cell_w, cell_h, True)
def _send_lark(recipient: str, text: str | None = None, image: Path | None = None) -> None:
"""Send a notification through the locally installed Lark CLI."""
lark_env = {
key: value for key, value in os.environ.items()
if not key.upper().startswith("HERMES_")
}
def run_lark_cli(args: list[str], cwd: Path) -> None:
cwd.mkdir(parents=True, exist_ok=True)
command = shutil.which("lark-cli.cmd") or shutil.which("lark-cli") or "lark-cli.cmd"
completed = subprocess.run(
[command, *args],
cwd=cwd,
env=lark_env,
capture_output=True,
text=True,
encoding="utf-8",
errors="replace",
check=False,
)
if completed.returncode:
detail = (completed.stderr or completed.stdout).strip()
raise RuntimeError(f"lark-cli failed ({completed.returncode}): {detail}")
if text:
# ensure_ascii keeps the Windows subprocess argv ASCII-only while the
# JSON parser restores the original Chinese text for Feishu.
content = json.dumps({"text": text}, ensure_ascii=True)
run_lark_cli(
["im", "+messages-send", "--user-id", recipient,
"--as", "bot", "--content", content],
cwd=ANALYZER_DIR,
)
if image:
run_lark_cli(
["im", "+messages-send", "--user-id", recipient,
"--as", "bot", "--image", f"./{image.name}"], cwd=image.parent,
)
def _run_round_factory(args):
python = sys.executable
screenshot_root = PATHS.tmp_root / "qrcode/friday_relogin"
def run_round(names: list[str], attempt: int) -> dict[str, bool]:
started: dict[str, subprocess.Popen] = {}
stamp = dt.datetime.now().strftime("%Y%m%d_%H%M%S")
for name in names:
platform = PLATFORMS[name]
cmd = [python, *platform.command, "--login-timeout", str(args.round_timeout)]
started[name] = subprocess.Popen(cmd, cwd=PROJECT_DIR)
time.sleep(args.screenshot_delay)
title_windows = _login_windows()
windows = {
name: title_windows.get(name) or _window_for_process(process.pid)
for name, process in started.items()
}
_tile_windows([hwnd for hwnd in windows.values() if hwnd])
time.sleep(2)
for name, process in started.items():
platform = PLATFORMS[name]
hwnd = windows[name]
if not hwnd:
print(f"[{name}] no visible browser window for QR screenshot", flush=True)
continue
output = screenshot_root / f"{stamp}_{name}_round{attempt}.png"
try:
_capture_window(hwnd, output)
if not args.no_send:
recipient = recipient_for_platform(name, args.recipient)
_send_lark(
recipient,
text=(
f"{platform.label} \u767b\u5f55\u4e8c\u7ef4\u7801"
f"\uff08\u7b2c {attempt}/3 \u6b21\uff09\uff0c"
"\u8bf7\u5728 3 \u5206\u949f\u5185\u626b\u7801\u3002"
),
)
_send_lark(recipient, image=output)
print(f"[{name}] QR screenshot: {output}", flush=True)
except Exception as exc:
print(f"[{name}] screenshot/send failed: {exc}", flush=True)
# Put the still-open login windows back into a compact layout after
# taking full-size screenshots one at a time.
_tile_windows([hwnd for hwnd in windows.values() if hwnd])
results: dict[str, bool] = {}
for name, process in started.items():
try:
rc = process.wait(timeout=args.round_timeout + 90)
except subprocess.TimeoutExpired:
_terminate_process_tree(process)
rc = 1
results[name] = rc == 0 and _valid_cookie(PLATFORMS[name])
print(f"[{name}] round={attempt} exit={rc} success={results[name]}", flush=True)
return results
return run_round
def main() -> int:
parser = argparse.ArgumentParser(description="Friday five-platform parallel QR relogin")
parser.add_argument(
"--recipient", default=None,
help="Override the configured recipients and send every platform to one open_id",
)
parser.add_argument("--max-attempts", type=int, default=3)
parser.add_argument("--round-timeout", type=int, default=300)
parser.add_argument("--screenshot-delay", type=int, default=60)
parser.add_argument("--no-send", action="store_true", help="Do not send Lark messages")
args = parser.parse_args()
status = run_with_retries(
list(PLATFORMS), max_attempts=args.max_attempts,
run_round=_run_round_factory(args),
)
failed = [name for name, ok in status.items() if not ok]
if failed and not args.no_send:
for recipient, names in group_platforms_by_recipient(failed, args.recipient).items():
labels = "\u3001".join(PLATFORMS[name].label for name in names)
_send_lark(
recipient,
text=(
"\u4ee5\u4e0b\u5e73\u53f0\u8fde\u7eed 3 \u6b21\u672a\u626b\u7801\uff0c"
"\u5df2\u6062\u590d\u5e76\u7ee7\u4f7f\u7528\u65e7 Cookie\uff1a"
f"{labels}"
),
)
print(f"Final status: {status}", flush=True)
return 1 if failed else 0
if __name__ == "__main__":
raise SystemExit(main())
src/gyxx_flow/modules/content_marketing/runtime/data/tools/generate_creator_report.py
+910
View File
@@ -0,0 +1,910 @@
#!/usr/bin/env python
"""
generate_creator_report.py — 达人合作数据筛选与报价分析报告
=====================================================
从 PostgreSQL 读取 cmt_cooperations + cmt_creators + cmt_styles，
按指令要求生成完整的9章分析报告，输出为 Markdown 文件。
用法:
python generate_creator_report.py
python generate_creator_report.py --style 宙斯 # 只分析指定款式
python generate_creator_report.py --output report.md
"""
import argparse
import json
import os
import subprocess
import sys
from collections import defaultdict
from datetime import datetime, timedelta
from pathlib import Path
from gyxx_flow.modules.content_marketing.runtime.runtime_paths import (
PATHS,
resolve_layer_output,
)
import psycopg
from dotenv import load_dotenv
BASE_DIR = PATHS.tools_root
DATA_DIR = PATHS.exports_root
load_dotenv(PATHS.state_root / "config/db.env")
from gyxx_flow.modules.content_marketing.runtime.data.tools.db import get_db_config # noqa: E402
DB_CONFIG = get_db_config()
LARK = os.path.expandvars(r"%APPDATA%\npm\lark-cli.cmd")
if not os.path.exists(LARK):
LARK = "lark-cli.cmd"
PLATFORM_CN = {
"xiaohongshu": "小红书",
"douyin": "抖音",
"bilibili": "B站",
}
def _log(msg: str) -> None:
safe = str(msg).encode("gbk", errors="replace").decode("gbk", errors="replace")
print(safe, flush=True)
def infer_level(follower_num: int | None, platform: str) -> str:
if not follower_num:
return "未知"
if follower_num >= 500000:
return "头部"
elif follower_num >= 100000:
return "腰部"
elif follower_num >= 10000:
return "KOC"
else:
return "素人"
def score_creator(stats: dict) -> str:
if not stats.get("has_data"):
return "C"
avg_eng_rate = stats.get("avg_engagement_rate", 0) or 0
avg_cpe = stats.get("avg_cpe", 0) or 0
avg_roi = stats.get("avg_roi", 0) or 0
exposure = stats.get("total_exposure", 0) or 0
score = 0
if avg_eng_rate >= 5: score += 3
elif avg_eng_rate >= 3: score += 2
elif avg_eng_rate >= 1: score += 1
if 0 < avg_cpe <= 2: score += 3
elif 0 < avg_cpe <= 5: score += 2
elif avg_cpe > 0: score += 1
if avg_roi >= 2: score += 3
elif avg_roi >= 1: score += 2
elif avg_roi > 0: score += 1
if exposure >= 500000: score += 2
elif exposure >= 100000: score += 1
if score >= 8: return "S"
elif score >= 5: return "A"
elif score >= 3: return "B"
else: return "C"
def one_line_profile(stats: dict) -> str:
level = stats.get("level", "未知")
platform = stats.get("platform", "")
coop_count = stats.get("coop_count", 0)
total_exp = stats.get("total_exposure", 0) or 0
account_type = stats.get("account_type", "")
if total_exp >= 500000:
exp_desc = "爆款制造者"
elif total_exp >= 100000:
exp_desc = "稳定产出"
else:
exp_desc = "基础曝光"
type_desc = f"{account_type}类" if account_type else ""
return f"{level}{type_desc}达人，{PLATFORM_CN.get(platform, platform)}{coop_count}次合作，{exp_desc}"
def generate_report(style_filter: str | None = None) -> str:
conn = psycopg.connect(**DB_CONFIG)
cur = conn.cursor()
where = "WHERE c.name IS NOT NULL AND c.name != ''"
if style_filter:
where += f" AND s.name = '{style_filter}'"
cur.execute(f"""
SELECT
c.name AS creator_name,
c.platform,
c.follower_count,
c.follower_count_num,
c.platform_id,
c.account_type,
c.creator_level,
c.homepage_url,
s.name AS style_name,
co.feishu_record_id,
co.cooperation_date,
co.cooperation_cost,
co.ad_spend,
co.content_direction,
co.note_title,
co.exposure_count,
co.engagement_count,
co.engagement_count_num,
co.data_performance,
co.cpm,
co.is_paid,
co.publish_time,
n.like_count AS note_like_count,
n.collect_count AS note_collect_count,
n.comment_count AS note_comment_count,
n.favorite_count AS note_favorite_count
FROM cmt_cooperations co
JOIN cmt_creators c ON c.id = co.creator_id
JOIN cmt_styles s ON s.id = co.style_id
LEFT JOIN cmt_notes n ON n.feishu_record_id = co.feishu_record_id
{where}
ORDER BY s.name, c.name
""")
cols = [desc[0] for desc in cur.description]
rows = [dict(zip(cols, row)) for row in cur.fetchall()]
cur.close()
conn.close()
if not rows:
return "# 达人合作数据筛选与报价分析报告\n\n无数据。"
# ─── Build per-creator stats ───
creator_stats = defaultdict(lambda: {
"coop_count": 0,
"platform": "",
"follower_count_num": None,
"follower_count": "",
"account_type": "",
"creator_level": "",
"styles": [],
"costs": [],
"cost_dates": [], # (date, cost, style, content_direction) per coop
"exposures": [],
"engagements": [],
"engagement_nums": [],
"cpms": [],
"data_perfs": [],
"dates": [],
"has_data": False,
"note_engagements": [],
"note_exposures": [],
"content_directions": [],
})
for r in rows:
key = r["creator_name"]
s = creator_stats[key]
s["coop_count"] += 1
s["platform"] = r["platform"] or s["platform"]
if r["follower_count_num"] and not s["follower_count_num"]:
s["follower_count_num"] = r["follower_count_num"]
if r["follower_count"] and not s["follower_count"]:
s["follower_count"] = r["follower_count"]
if r["account_type"] and not s["account_type"]:
s["account_type"] = r["account_type"]
if r["creator_level"] and not s["creator_level"]:
s["creator_level"] = r["creator_level"]
if r["style_name"]:
s["styles"].append(r["style_name"])
if r["content_direction"]:
s["content_directions"].append(r["content_direction"])
# Per-cooperation detail for pricing analysis
cost = float(r["cooperation_cost"]) if r["cooperation_cost"] is not None else None
s["cost_dates"].append({
"date": str(r["cooperation_date"]) if r["cooperation_date"] else "",
"cost": cost,
"style": r["style_name"] or "",
"content_direction": r["content_direction"] or "",
"exposure": int(r["exposure_count"]) if r["exposure_count"] is not None else None,
})
if cost is not None:
s["costs"].append(cost)
if r["exposure_count"] is not None:
s["exposures"].append(int(r["exposure_count"]))
if r["engagement_count_num"] is not None:
s["engagement_nums"].append(int(r["engagement_count_num"]))
if r["cpm"] is not None:
s["cpms"].append(float(r["cpm"]))
if r["data_performance"]:
s["data_perfs"].append(r["data_performance"])
if r["cooperation_date"]:
s["dates"].append(str(r["cooperation_date"]))
if r["cooperation_cost"] is not None and r["exposure_count"] is not None:
s["has_data"] = True
note_eng = (r.get("note_like_count") or 0) + (r.get("note_collect_count") or 0) + (r.get("note_comment_count") or 0) + (r.get("note_favorite_count") or 0)
if note_eng > 0:
s["note_engagements"].append(note_eng)
note_exp = r["exposure_count"] if r["exposure_count"] else None
if note_exp:
s["note_exposures"].append(note_exp)
# Compute derived metrics
for key, s in creator_stats.items():
if not s["creator_level"] and s["follower_count_num"]:
s["creator_level"] = infer_level(s["follower_count_num"], s["platform"])
elif not s["creator_level"]:
s["creator_level"] = "未知"
s["level"] = s["creator_level"]
if s["note_engagements"] and s["note_exposures"] and len(s["note_engagements"]) == len(s["note_exposures"]):
rates = [e / max(x, 1) * 100 for e, x in zip(s["note_engagements"], s["note_exposures"])]
s["avg_engagement_rate"] = round(sum(rates) / len(rates), 2)
elif s["note_engagements"] and s["exposures"]:
total_eng = sum(s["note_engagements"])
total_exp = sum(s["exposures"])
s["avg_engagement_rate"] = round(total_eng / max(total_exp, 1) * 100, 2)
else:
s["avg_engagement_rate"] = None
if s["costs"] and s["note_engagements"]:
total_cost = sum(s["costs"])
total_eng = sum(s["note_engagements"])
s["avg_cpe"] = round(total_cost / max(total_eng, 1), 2)
elif s["costs"] and s["engagement_nums"]:
total_cost = sum(s["costs"])
total_eng = sum(s["engagement_nums"])
s["avg_cpe"] = round(total_cost / max(total_eng, 1), 2)
else:
s["avg_cpe"] = None
s["total_exposure"] = sum(s["exposures"]) if s["exposures"] else 0
s["avg_roi"] = None
s["score"] = score_creator(s)
# ─── Build per-style stats ───
style_stats = defaultdict(lambda: {"coop_count": 0, "creators": set(), "exposures": [], "costs": [], "top_creators": []})
for r in rows:
sn = r["style_name"]
ss = style_stats[sn]
ss["coop_count"] += 1
ss["creators"].add(r["creator_name"])
if r["exposure_count"] is not None:
ss["exposures"].append((r["creator_name"], int(r["exposure_count"])))
if r["cooperation_cost"] is not None:
ss["costs"].append(float(r["cooperation_cost"]))
# ─── Level average cost (for premium/discount analysis) ───
level_costs = defaultdict(list)
for name, s in creator_stats.items():
if s["costs"] and s["creator_level"] != "未知":
level_costs[s["creator_level"]].append(sum(s["costs"]) / len(s["costs"]))
level_avg_cost = {lv: round(sum(v)/len(v), 0) for lv, v in level_costs.items() if v}
# ─── Generate Report ───
lines = []
L = lines.append
L("# 达人合作数据筛选与报价分析报告（自动生成）")
L(f"\n> 生成时间：{datetime.now().strftime('%Y-%m-%d %H:%M')}")
if style_filter:
L(f"> 筛选款式：{style_filter}")
L("")
# ═══════════════════════════════════════════════════════════
# 1. 数据质量说明
# ═══════════════════════════════════════════════════════════
L("## 1. 数据质量说明")
L(f"- **总记录数**：{len(rows)}")
L(f"- **涉及达人总数**：{len(creator_stats)}")
L(f"- **涉及款式数**：{len(style_stats)}")
total = len(rows)
has_cost = sum(1 for r in rows if r["cooperation_cost"] is not None)
has_exposure = sum(1 for r in rows if r["exposure_count"] is not None)
has_engagement = sum(1 for r in rows if (r.get("note_like_count") or r.get("note_collect_count") or r.get("note_comment_count") or r.get("note_favorite_count")))
has_follower = sum(1 for r in rows if r["follower_count_num"] is not None)
has_platform = sum(1 for r in rows if r["platform"])
has_level = sum(1 for r in rows if r["creator_level"])
has_content_dir = sum(1 for r in rows if r["content_direction"])
L(f"\n| 字段 | 填充数 | 填充率 |")
L(f"|---|---|---|")
L(f"| 平台 | {has_platform} | {has_platform/total*100:.1f}% |")
L(f"| 粉丝数 | {has_follower} | {has_follower/total*100:.1f}% |")
L(f"| 达人层级 | {has_level} | {has_level/total*100:.1f}% |")
L(f"| 合作花费 | {has_cost} | {has_cost/total*100:.1f}% |")
L(f"| 曝光量 | {has_exposure} | {has_exposure/total*100:.1f}% |")
L(f"| 互动量 | {has_engagement} | {has_engagement/total*100:.1f}% |")
L(f"| 内容方向 | {has_content_dir} | {has_content_dir/total*100:.1f}% |")
L(f"| 进店UV | 0 | 0.0% |")
L(f"| 成交金额 | 0 | 0.0% |")
L(f"| ROI | 0 | 0.0% |")
L(f"| 评论品牌提及率 | 0 | 0.0% |")
L(f"| 差评或舆情风险 | 0 | 0.0% |")
L("")
L('> **说明**：电商转化数据（进店UV、成交金额、ROI）和评论分析数据（品牌提及率、舆情风险）暂未采集，对应分析章节将标注"数据不足"。达人层级根据粉丝数自动推断：>=50万=头部、>=10万=腰部、>=1万=KOC、<1万=素人。')
L("")
# ═══════════════════════════════════════════════════════════
# 2. 整体概览
# ═══════════════════════════════════════════════════════════
L("## 2. 整体概览")
plat_dist = defaultdict(int)
for r in rows:
p = r["platform"] or "未知"
plat_dist[p] += 1
L("\n### 各平台达人分布")
L("| 平台 | 合作记录数 | 达人数 |")
L("|---|---|---|")
plat_creators = defaultdict(set)
for r in rows:
plat_creators[r["platform"] or "未知"].add(r["creator_name"])
for p in sorted(plat_dist, key=plat_dist.get, reverse=True):
cn = PLATFORM_CN.get(p, p)
L(f"| {cn} | {plat_dist[p]} | {len(plat_creators[p])} |")
L("")
level_dist = defaultdict(int)
for s in creator_stats.values():
level_dist[s["creator_level"]] += 1
L("### 各层级分布")
L("| 层级 | 达人数 |")
L("|---|---|")
for lv in ["头部", "腰部", "KOC", "素人", "未知"]:
L(f"| {lv} | {level_dist.get(lv, 0)} |")
L("")
all_costs = [float(r["cooperation_cost"]) for r in rows if r["cooperation_cost"] is not None]
all_exposures = [int(r["exposure_count"]) for r in rows if r["exposure_count"] is not None]
all_engagements_note = []
for r in rows:
ne = (r.get("note_like_count") or 0) + (r.get("note_collect_count") or 0) + (r.get("note_comment_count") or 0) + (r.get("note_favorite_count") or 0)
if ne > 0:
all_engagements_note.append(ne)
total_cost = sum(all_costs) if all_costs else 0
total_exp = sum(all_exposures) if all_exposures else 0
avg_cpe = total_cost / sum(all_engagements_note) if all_costs and all_engagements_note else None
L("### 核心指标汇总")
L("| 指标 | 数值 |")
L("|---|---|")
L(f"| 总合作花费 | ¥{total_cost:,.0f} |")
L(f"| 总曝光量 | {total_exp:,} |")
L(f"| 总互动量 | {sum(all_engagements_note):,} |" if all_engagements_note else "| 总互动量 | 数据不足 |")
L(f"| 平均CPE | ¥{avg_cpe:.2f} |" if avg_cpe else "| 平均CPE | 数据不足 |")
L(f"| 平均ROI | 数据不足 |")
L(f"| 总成交金额 | 数据不足 |")
L("")
# ═══════════════════════════════════════════════════════════
# 3. 达人效能画像（按达人分组汇总）
# ═══════════════════════════════════════════════════════════
L("## 3. 达人效能画像（按达人分组汇总）")
L("")
L("> 按 S/A/B/C 层级分表展示，每个子表不超过飞书 2000 格限制。")
L("")
sorted_creators = sorted(creator_stats.items(),
key=lambda x: ({"S":0,"A":1,"B":2,"C":3}.get(x[1]["score"], 4), -(x[1]["total_exposure"] or 0)))
for tier in ["S", "A", "B", "C"]:
tier_rows = [(n, s) for n, s in sorted_creators if s["score"] == tier]
if not tier_rows:
continue
# Split large tiers to stay under Feishu 2000-cell limit
# 10 cols per row → max 200 rows per table
MAX_ROWS = 200
chunks = [tier_rows[i:i+MAX_ROWS] for i in range(0, len(tier_rows), MAX_ROWS)]
for ci, chunk in enumerate(chunks):
chunk_label = f"（{len(tier_rows)}人）" if len(chunks) == 1 else f"（{len(tier_rows)}人，第{ci+1}/{len(chunks)}批）"
L(f"### {tier}级达人{chunk_label}")
L("")
L("| 达人昵称 | 平台 | 粉丝数 | 层级 | 合作次数 | 合作单品 | 每次金额 | 平均CPE | 平均互动率 | 综合评分 |")
L("|---|---|---|---|---|---|---|---|---|---|")
for name, s in chunk:
plat_cn = PLATFORM_CN.get(s["platform"], s["platform"] or "未知")
follower_str = f"{s['follower_count_num']:,}" if s["follower_count_num"] else s.get("follower_count", "未知")
unique_styles = list(dict.fromkeys(s["styles"]))
styles_str = ", ".join(unique_styles[:3])
if len(unique_styles) > 3:
styles_str += f"等{len(unique_styles)}款"
cost_entries = [e for e in s["cost_dates"] if e["cost"] is not None]
cost_entries.sort(key=lambda e: e["date"] or "")
costs_str = "→".join(f"¥{e['cost']:.0f}" for e in cost_entries) if cost_entries else "-"
avg_cpe_str = f"¥{s['avg_cpe']:.2f}" if s.get("avg_cpe") else "-"
avg_er_str = f"{s['avg_engagement_rate']:.1f}%" if s.get("avg_engagement_rate") else "-"
L(f"| {name} | {plat_cn} | {follower_str} | {s['creator_level']} | {s['coop_count']} | {styles_str} | {costs_str} | {avg_cpe_str} | {avg_er_str} | {s['score']} |")
L("")
# ═══════════════════════════════════════════════════════════
# 4. 达人报价差异分析
# ═══════════════════════════════════════════════════════════
L("## 4. 达人报价差异分析")
L("")
# 4a. 个体报价波动分析
L("### 个体报价波动分析")
L("")
L("> 针对同一达人不同次合作花费不同的情况，结合合作单品、内容形式、合作日期分析报价变化原因。")
L("")
multi_cost_creators = []
for name, s in creator_stats.items():
cost_entries = [e for e in s["cost_dates"] if e["cost"] is not None]
if len(cost_entries) >= 2:
costs = [e["cost"] for e in cost_entries]
if max(costs) != min(costs):
multi_cost_creators.append((name, s, cost_entries))
if multi_cost_creators:
L("| 达人昵称 | 层级 | 合作次数 | 报价区间 | 报价变化 | 波动归因分析 |")
L("|---|---|---|---|---|---|")
for name, s, entries in multi_cost_creators[:30]:
entries.sort(key=lambda e: e["date"] or "")
costs = [e["cost"] for e in entries]
cost_range = f"¥{min(costs):.0f}~¥{max(costs):.0f}"
cost_trend = "→".join(f"¥{e['cost']:.0f}" for e in entries)
# Analyze reasons
reasons = []
# Check content_direction variation
dirs = [e["content_direction"] for e in entries if e["content_direction"]]
if len(set(dirs)) > 1:
reasons.append(f"内容形式不同({', '.join(set(dirs))})")
# Check style variation
styles = [e["style"] for e in entries if e["style"]]
if len(set(styles)) > 1:
reasons.append(f"合作不同款式({len(set(styles))}款)")
# Check date proximity to promotion periods
dates = [e["date"] for e in entries if e["date"]]
promo_months = ["06", "11", "12"] # 618, 双11, 双12
if any(d[5:7] in promo_months for d in dates):
reasons.append("含大促节点")
if not reasons:
reasons.append("报价自然波动")
L(f"| {name} | {s['creator_level']} | {len(entries)} | {cost_range} | {cost_trend} | {'；'.join(reasons)} |")
L("")
else:
L("数据不足，无法分析个体报价波动。")
L("")
# 4b. 同层级溢价/折价分析
L("### 同层级溢价/折价分析")
L("")
L("> 对比同层级、同平台达人的平均合作花费，识别溢价（性价比低）和折价（高性价比洼地）达人。")
L("")
if level_avg_cost:
L("| 达人昵称 | 层级 | 平台 | 个人均价 | 层级均价 | 溢折率 | 判定 |")
L("|---|---|---|---|---|---|---|")
premium_list = []
for name, s in creator_stats.items():
if not s["costs"] or s["creator_level"] == "未知":
continue
personal_avg = sum(s["costs"]) / len(s["costs"])
lv_avg = level_avg_cost.get(s["creator_level"])
if not lv_avg or lv_avg == 0:
continue
ratio = (personal_avg - lv_avg) / lv_avg * 100
if ratio > 30:
tag = "溢价"
elif ratio < -30:
tag = "折价洼地"
else:
tag = "合理"
premium_list.append((name, s, personal_avg, lv_avg, ratio, tag))
premium_list.sort(key=lambda x: -x[4])
for name, s, p_avg, l_avg, ratio, tag in premium_list[:20]:
plat_cn = PLATFORM_CN.get(s["platform"], s["platform"])
L(f"| {name} | {s['creator_level']} | {plat_cn} | ¥{p_avg:.0f} | ¥{l_avg:.0f} | {ratio:+.0f}% | {tag} |")
# Also show top discount
discount_list = [x for x in premium_list if x[5] == "折价洼地"]
for name, s, p_avg, l_avg, ratio, tag in discount_list[:10]:
plat_cn = PLATFORM_CN.get(s["platform"], s["platform"])
L(f"| {name} | {s['creator_level']} | {plat_cn} | ¥{p_avg:.0f} | ¥{l_avg:.0f} | {ratio:+.0f}% | {tag} |")
L("")
else:
L("数据不足，无法进行同层级溢价/折价分析。")
L("")
# ═══════════════════════════════════════════════════════════
# 5. 达人分层与分类
# ═══════════════════════════════════════════════════════════
L("## 5. 达人分层与分类")
tiers = {"S": [], "A": [], "B": [], "C": []}
for name, s in creator_stats.items():
tiers[s["score"]].append(name)
L("\n### 分层矩阵")
for tier in ["S", "A", "B", "C"]:
names = tiers[tier]
if names:
names_str = ', '.join(names[:20]) + ('...' if len(names) > 20 else '')
L(f"**{tier}级**（{len(names)}人）：{names_str}")
else:
L(f"**{tier}级**（0人）：无")
L("\n### 角色分类")
voice_type = sorted(creator_stats.items(), key=lambda x: -(x[1]["total_exposure"] or 0))[:10]
voice_names = [n for n, s in voice_type if s["total_exposure"]]
L(f"**声量型**（高曝光）：{', '.join(voice_names)}")
harvest_type = sorted(creator_stats.items(), key=lambda x: x[1].get("avg_cpe", 9999) or 9999)[:10]
L(f"**收割型**（低成本高效）：{', '.join(n for n, s in harvest_type if s.get('avg_cpe') and s['avg_cpe'] > 0)}")
volume_type = [(n, s) for n, s in creator_stats.items() if s["costs"]]
volume_type.sort(key=lambda x: sum(x[1]["costs"]) / len(x[1]["costs"]))
L(f"**铺量型**（低成本）：{', '.join(n for n, _ in volume_type[:10])}")
# 共创型: high engagement rate + content variety
creative_type = [(n, s) for n, s in creator_stats.items() if s.get("avg_engagement_rate") and s["avg_engagement_rate"] > 0 and len(set(s["content_directions"])) >= 2]
creative_type.sort(key=lambda x: -x[1].get("avg_engagement_rate", 0))
L(f"**共创型**（内容创意强，多方向+高互动）：{', '.join(n for n, _ in creative_type[:10])}")
L("")
# ═══════════════════════════════════════════════════════════
# 6. 合作单品交叉分析
# ═══════════════════════════════════════════════════════════
L("## 6. 合作单品交叉分析")
L("")
L("| 款式 | 合作达人数 | 平均曝光 | 平均花费 | Top3达人 | 单品最佳搭档 |")
L("|---|---|---|---|---|---|")
for sn in sorted(style_stats.keys()):
ss = style_stats[sn]
avg_exp = sum(e for _, e in ss["exposures"]) / len(ss["exposures"]) if ss["exposures"] else 0
avg_cost = sum(ss["costs"]) / len(ss["costs"]) if ss["costs"] else 0
top3 = sorted(ss["exposures"], key=lambda x: -x[1])[:3]
top3_str = ", ".join(f"{n}({e:,})" for n, e in top3)
# Best partner: highest exposure creator
best = top3[0][0] if top3 else "-"
L(f"| {sn} | {len(ss['creators'])} | {avg_exp:,.0f} | ¥{avg_cost:,.0f} | {top3_str} | {best} |")
L("")
# ═══════════════════════════════════════════════════════════
# 7. 趋势与风险评估
# ═══════════════════════════════════════════════════════════
L("## 7. 趋势与风险评估")
L("")
multi_coop = {n: s for n, s in creator_stats.items() if s["coop_count"] >= 2 and len(s["exposures"]) >= 2}
rising = []
declining = []
for n, s in multi_coop.items():
if len(s["exposures"]) >= 2:
first_half = s["exposures"][:len(s["exposures"])//2]
second_half = s["exposures"][len(s["exposures"])//2:]
avg_first = sum(first_half) / len(first_half)
avg_second = sum(second_half) / len(second_half)
if avg_second > avg_first * 1.2:
rising.append(n)
elif avg_second < avg_first * 0.8:
declining.append(n)
L(f"### 表现持续上升（{len(rising)}人）")
if rising:
L("建议加大投入。")
for n in rising[:20]:
s = creator_stats[n]
L(f"- **{n}**：曝光量从前期上升至后期，{s['coop_count']}次合作数据持续改善")
else:
L("数据不足以判断趋势")
L("")
L(f"### 表现下滑（{len(declining)}人）")
if declining:
L("建议减少合作或调整Brief。")
for n in declining[:20]:
s = creator_stats[n]
L(f"- **{n}**：曝光量从前期下降至后期，建议关注内容质量或平台算法变化")
else:
L("数据不足以判断趋势")
L("")
L("### 风险达人")
risk_tags = {"曝光限流": [], "效果一般": [], "挂车不出单": [], "内容卡审": []}
for n, s in creator_stats.items():
for dp in s["data_perfs"]:
for tag in dp.split(","):
tag = tag.strip()
if tag in risk_tags:
risk_tags[tag].append(n)
has_risk = False
for tag, names in risk_tags.items():
if names:
has_risk = True
L(f"- **{tag}**：{', '.join(set(names))}")
if not has_risk:
L("- 暂无风险标记")
L("")
# ═══════════════════════════════════════════════════════════
# 8. 筛选推荐（根据目标场景）
# ═══════════════════════════════════════════════════════════
L("## 8. 筛选推荐（根据目标场景）")
L("")
L("### 场景一：新品上市（重点曝光+种草）")
s_a_creators = [(n, s) for n, s in creator_stats.items() if s["score"] in ("S", "A") and s["total_exposure"]]
s_a_creators.sort(key=lambda x: -x[1]["total_exposure"])
if s_a_creators:
L("| 推荐达人 | 平台 | 粉丝数 | 累计曝光 | 综合评分 | 推荐理由 | 预估CPE |")
L("|---|---|---|---|---|---|---|")
for n, s in s_a_creators[:5]:
plat_cn = PLATFORM_CN.get(s["platform"], s["platform"])
follower_str = f"{s['follower_count_num']:,}" if s["follower_count_num"] else "未知"
reason = f"曝光量{s['total_exposure']:,}，{s['coop_count']}次合作经验"
est_cpe = f"¥{s['avg_cpe']:.2f}" if s.get("avg_cpe") else "待估"
L(f"| {n} | {plat_cn} | {follower_str} | {s['total_exposure']:,} | {s['score']} | {reason} | {est_cpe} |")
else:
L("数据不足，无法推荐")
L("")
L("### 场景二：大促收割（重点转化）")
L("> 进店UV、成交金额、ROI 数据暂未采集，以下基于低成本高效（低CPE+高互动率）推荐：")
low_cpe = [(n, s) for n, s in creator_stats.items() if s.get("avg_cpe") and s["avg_cpe"] > 0]
low_cpe.sort(key=lambda x: x[1]["avg_cpe"])
if low_cpe:
L("| 推荐达人 | 平台 | 平均CPE | 平均互动率 | 综合评分 | 推荐理由 |")
L("|---|---|---|---|---|---|")
for n, s in low_cpe[:5]:
plat_cn = PLATFORM_CN.get(s["platform"], s["platform"])
er = f"{s['avg_engagement_rate']:.1f}%" if s.get("avg_engagement_rate") else "-"
L(f"| {n} | {plat_cn} | ¥{s['avg_cpe']:.2f} | {er} | {s['score']} | CPE低，互动率高 |")
else:
L("数据不足，无法推荐")
L("")
L("### 场景三：日常口碑维护")
koc_creators = [(n, s) for n, s in creator_stats.items() if s["creator_level"] in ("KOC", "素人") and s["total_exposure"]]
koc_creators.sort(key=lambda x: -(x[1].get("avg_engagement_rate") or 0))
if koc_creators:
L("| 推荐达人 | 平台 | 粉丝数 | 平均互动率 | 合作次数 | 推荐理由 |")
L("|---|---|---|---|---|---|")
for n, s in koc_creators[:5]:
plat_cn = PLATFORM_CN.get(s["platform"], s["platform"])
follower_str = f"{s['follower_count_num']:,}" if s["follower_count_num"] else "未知"
er = f"{s['avg_engagement_rate']:.1f}%" if s.get("avg_engagement_rate") else "-"
L(f"| {n} | {plat_cn} | {follower_str} | {er} | {s['coop_count']} | 低成本高互动KOC |")
else:
L("数据不足，无法推荐")
L("")
# ═══════════════════════════════════════════════════════════
# 9. 优化建议与下一步行动
# ═══════════════════════════════════════════════════════════
L("## 9. 优化建议与下一步行动")
L("")
for tier, action in [
("S", "建议签长期协议或独家合作，锁定优质达人资源"),
("A", "增加合作频次或尝试新款式，挖掘更大潜力"),
("B", "优化Brief或换内容形式测试，寻找突破口"),
("C", "暂停合作或淘汰，将预算转移至更高层级达人"),
]:
names = tiers[tier]
L(f"- **{tier}级达人**（{len(names)}人）：{action}")
L("")
# 报价优化建议
L("### 报价优化建议")
L("")
if multi_cost_creators:
L("基于第4章报价差异分析，给出以下议价策略：")
L("")
# Find creators with rising costs
for name, s, entries in multi_cost_creators[:10]:
entries.sort(key=lambda e: e["date"] or "")
costs = [e["cost"] for e in entries]
if costs[-1] > costs[0]:
L(f"- **{name}**：报价呈上升趋势（¥{costs[0]:.0f}→¥{costs[-1]:.0f}），建议锁定当前报价或签长期协议避免继续涨价")
elif costs[-1] < costs[0]:
L(f"- **{name}**：报价有下降空间（¥{costs[0]:.0f}→¥{costs[-1]:.0f}），可尝试进一步议价")
if level_avg_cost:
L("")
L("**层级预算分配建议**：")
for lv in ["头部", "腰部", "KOC", "素人"]:
avg = level_avg_cost.get(lv)
if avg:
L(f"- {lv}达人均价¥{avg:.0f}，建议单次合作预算控制在¥{avg*0.8:.0f}~¥{avg*1.2:.0f}")
L("")
L("---")
L(f"*报告由 generate_creator_report.py 自动生成 · {datetime.now().strftime('%Y-%m-%d %H:%M')}*")
return "\n".join(lines), rows, creator_stats, style_stats
def _compute_last_month():
today = datetime.now().date()
first_of_this_month = today.replace(day=1)
month_end = first_of_this_month - timedelta(days=1)
month_start = month_end.replace(day=1)
return month_start, month_end
def create_feishu_doc(report_md: str, month_start: str, month_end: str) -> str | None:
"""创建飞书文档,返回 doc URL;失败返回 None"""
doc_title = f"达人合作数据筛选与报价分析报告（{month_start}~{month_end[5:]}）"
md_file = PATHS.tmp_root / "creator_report_doc.md"
md_file.parent.mkdir(parents=True, exist_ok=True)
md_file.write_text(report_md, encoding="utf-8")
cmd = [LARK, "docs", "+create",
"--as", "user", "--format", "json",
"--title", doc_title,
"--doc-format", "markdown",
"--content", f"@{md_file.name}"]
env = {**os.environ, "LARK_CLI_NO_PROXY": "1"}
proc = subprocess.run(cmd, capture_output=True, env=env,
encoding="utf-8", errors="replace")
text = proc.stdout or proc.stderr
start = text.find("{")
if start < 0:
_log(f" [ERR] 创建飞书文档失败: {text[:300]}")
return None
resp = json.loads(text[start:])
if not resp.get("ok"):
_log(f" [ERR] 创建飞书文档失败: {resp.get('error')}")
return None
url = resp.get("data", {}).get("document", {}).get("url", "")
if resp.get("warnings"):
_log(f" [WARN] {resp['warnings']}")
return url
def save_to_db(report_md: str, rows: list, creator_stats: dict,
style_stats: dict, month_start: str, month_end: str,
doc_url: str | None, status: str) -> bool:
"""把月度达人报告结果写入 cmt_creator_report"""
time_label = f"{month_start}~{month_end[5:]}"
total_cooperations = len(rows)
total_creators = len(creator_stats)
total_styles = len(style_stats)
all_costs = [float(r["cooperation_cost"]) for r in rows if r["cooperation_cost"] is not None]
all_exposures = [int(r["exposure_count"]) for r in rows if r["exposure_count"] is not None]
all_engagements = []
for r in rows:
ne = (r.get("note_like_count") or 0) + (r.get("note_collect_count") or 0) + (r.get("note_comment_count") or 0) + (r.get("note_favorite_count") or 0)
if ne > 0:
all_engagements.append(ne)
total_cost = sum(all_costs) if all_costs else None
total_exposure = sum(all_exposures) if all_exposures else None
total_engagement = sum(all_engagements) if all_engagements else None
avg_cpe = (total_cost / total_engagement) if total_cost and total_engagement else None
# Level distribution
level_dist = defaultdict(int)
for s in creator_stats.values():
level_dist[s["creator_level"]] += 1
# Platform distribution
plat_dist = defaultdict(int)
for r in rows:
p = r["platform"] or "未知"
plat_dist[p] += 1
plat_dist_creators = defaultdict(set)
for r in rows:
plat_dist_creators[r["platform"] or "未知"].add(r["creator_name"])
platform_dist = {p: len(plat_dist_creators[p]) for p in plat_dist_creators}
# Score distribution
score_dist = defaultdict(int)
for s in creator_stats.values():
score_dist[s["score"]] += 1
doc_title = f"达人合作数据筛选与报价分析报告（{time_label}）"
try:
conn = psycopg.connect(**DB_CONFIG)
cur = conn.cursor()
cur.execute("""
INSERT INTO cmt_creator_report
(month_start, month_end, time_label,
total_cooperations, total_creators, total_styles,
total_cost, total_exposure, total_engagement, avg_cpe,
level_distribution, platform_distribution, score_distribution,
doc_url, doc_title, report_chars, status)
VALUES
(%s, %s, %s,
%s, %s, %s,
%s, %s, %s, %s,
%s, %s, %s,
%s, %s, %s, %s)
ON CONFLICT (month_start) DO UPDATE SET
total_cooperations = EXCLUDED.total_cooperations,
total_creators = EXCLUDED.total_creators,
total_styles = EXCLUDED.total_styles,
total_cost = EXCLUDED.total_cost,
total_exposure = EXCLUDED.total_exposure,
total_engagement = EXCLUDED.total_engagement,
avg_cpe = EXCLUDED.avg_cpe,
level_distribution = EXCLUDED.level_distribution,
platform_distribution = EXCLUDED.platform_distribution,
score_distribution = EXCLUDED.score_distribution,
doc_url = EXCLUDED.doc_url,
doc_title = EXCLUDED.doc_title,
report_chars = EXCLUDED.report_chars,
status = EXCLUDED.status,
updated_at = CURRENT_TIMESTAMP
""", (month_start, month_end, time_label,
total_cooperations, total_creators, total_styles,
total_cost, total_exposure, total_engagement, avg_cpe,
json.dumps(dict(level_dist), ensure_ascii=False),
json.dumps(platform_dist, ensure_ascii=False),
json.dumps(dict(score_dist), ensure_ascii=False),
doc_url, doc_title, len(report_md), status))
conn.commit()
conn.close()
_log(f" 已入库 cmt_creator_report (month={month_start})")
return True
except Exception as exc:
_log(f" [DB ERR] 入库失败: {exc}")
return False
def main():
parser = argparse.ArgumentParser(description="生成达人合作数据筛选与报价分析报告")
parser.add_argument("--style", type=str, help="只分析指定款式 (如 '宙斯')")
parser.add_argument("--output", type=str, default="", help="输出文件路径 (默认 data/reports/ 下自动命名)")
parser.add_argument("--dry-run", action="store_true", help="不写飞书/PG")
args = parser.parse_args()
month_start, month_end = _compute_last_month()
month_start_str = month_start.strftime("%Y-%m-%d")
month_end_str = month_end.strftime("%Y-%m-%d")
_log(f"达人月度分析报告: {month_start_str}~{month_end_str}")
# generate_report returns (report_text, rows, creator_stats, style_stats)
report, rows, creator_stats, style_stats = generate_report(style_filter=args.style)
if args.output:
out_path = resolve_layer_output(
args.output, layer_root=PATHS.exports_root, field="--output"
)
else:
out_dir = DATA_DIR / "reports"
out_dir.mkdir(exist_ok=True)
ts = datetime.now().strftime("%Y%m%d_%H%M%S")
style_tag = f"_{args.style}" if args.style else "_all"
out_path = out_dir / f"creator_report{style_tag}_{ts}.md"
out_path.write_text(report, encoding="utf-8")
_log(f"报告已生成: {out_path} ({len(report)} 字符)")
if args.dry_run:
_log("[DRY-RUN] 跳过飞书文档创建和入库")
return
# 创建飞书文档
doc_url = create_feishu_doc(report, month_start_str, month_end_str)
if doc_url:
_log(f"飞书文档: {doc_url}")
else:
_log("飞书文档创建失败")
# 入库
status = "ok" if doc_url else "doc_failed"
save_to_db(report, rows, creator_stats, style_stats,
month_start_str, month_end_str, doc_url, status)
if __name__ == "__main__":
main()
src/gyxx_flow/modules/content_marketing/runtime/data/tools/init_db.py
+29
View File
@@ -0,0 +1,29 @@
"""Initialize gyxx_super_data tables from schema_gyxx_super_data.sql."""
import re
import sys
from pathlib import Path
# Path setup
_TOOLS_DIR = Path(__file__).resolve().parent
from gyxx_flow.modules.content_marketing.runtime.data.tools import db
def main() -> int:
sql_path = _TOOLS_DIR / "schema_gyxx_super_data.sql"
sql = sql_path.read_text(encoding="utf-8")
# Remove psql meta-commands
sql = re.sub(r"^\\c .*?$", "", sql, flags=re.MULTILINE)
sql = re.sub(r"^\\.*$", "", sql, flags=re.MULTILINE)
with db.get_conn() as conn:
with conn.cursor() as cur:
cur.execute(sql)
conn.commit()
print("Schema initialized successfully.")
return 0
if __name__ == "__main__":
raise SystemExit(main())
src/gyxx_flow/modules/content_marketing/runtime/data/tools/kill_project_chrome.ps1
+7
View File
@@ -0,0 +1,7 @@
$moduleRoot = (Resolve-Path (Join-Path $PSScriptRoot '..\..')).Path
$dataRoot = if ($env:GYXX_DATA_ROOT) { $env:GYXX_DATA_ROOT } else { Join-Path $moduleRoot 'var' }
$profileRoot = Join-Path $dataRoot 'state\content_marketing\browser-profiles'
Get-WmiObject Win32_Process | Where-Object { $_.Name -eq 'chrome.exe' -and $_.CommandLine -like "*$profileRoot*" } | ForEach-Object { Write-Host "Killing PID $($_.ProcessId)"; Stop-Process -Id $_.ProcessId -Force -ErrorAction SilentlyContinue }
Start-Sleep -Seconds 3
$procs = Get-WmiObject Win32_Process | Where-Object { $_.Name -eq 'chrome.exe' -and $_.CommandLine -like "*$profileRoot*" }
if ($procs) { Write-Host 'Remaining project chrome processes:'; $procs | Select-Object ProcessId,CommandLine } else { Write-Host 'No project chrome processes remaining' }
src/gyxx_flow/modules/content_marketing/runtime/data/tools/list_project_chrome.ps1
+4
View File
@@ -0,0 +1,4 @@
$moduleRoot = (Resolve-Path (Join-Path $PSScriptRoot '..\..')).Path
$dataRoot = if ($env:GYXX_DATA_ROOT) { $env:GYXX_DATA_ROOT } else { Join-Path $moduleRoot 'var' }
$profileRoot = Join-Path $dataRoot 'state\content_marketing\browser-profiles'
Get-WmiObject Win32_Process | Where-Object { $_.Name -eq 'chrome.exe' -and $_.CommandLine -like "*$profileRoot*" } | Select-Object ProcessId, ParentProcessId, @{Name='CmdStart';Expression={$_.CommandLine.Substring(0, [Math]::Min(200, $_.CommandLine.Length))}}
src/gyxx_flow/modules/content_marketing/runtime/data/tools/migrations/001_creators_and_cooperations.sql
+132
View File
@@ -0,0 +1,132 @@
-- Migration 001: 扩展 cmt_creators 达人属性 + 新建 cmt_cooperations 合作记录表
-- 日期: 2026-07-10
-- 说明: 从飞书款式合作表同步达人信息和合作数据到数据库
-- ============================================================
-- 1. 扩展 cmt_creators：加达人属性字段
-- ============================================================
-- 加平台字段
ALTER TABLE cmt_creators ADD COLUMN IF NOT EXISTS platform VARCHAR(32) DEFAULT '';
COMMENT ON COLUMN cmt_creators.platform IS '主平台: xiaohongshu/douyin/bilibili';
-- 加平台达人ID
ALTER TABLE cmt_creators ADD COLUMN IF NOT EXISTS platform_id VARCHAR(128) DEFAULT '';
COMMENT ON COLUMN cmt_creators.platform_id IS '平台达人ID (如B站UID、抖音sec_uid等)';
-- 加粉丝数 (保留原始文本，因为飞书数据是"3.7万"这种格式)
ALTER TABLE cmt_creators ADD COLUMN IF NOT EXISTS follower_count VARCHAR(64) DEFAULT '';
COMMENT ON COLUMN cmt_creators.follower_count IS '达人粉丝数(原始文本如3.7万/12.4w)';
-- 加粉丝数(数字解析后)
ALTER TABLE cmt_creators ADD COLUMN IF NOT EXISTS follower_count_num BIGINT DEFAULT NULL;
COMMENT ON COLUMN cmt_creators.follower_count_num IS '达人粉丝数(数字解析后，3.7万=37000)';
-- 加达人主页链接
ALTER TABLE cmt_creators ADD COLUMN IF NOT EXISTS homepage_url TEXT DEFAULT '';
COMMENT ON COLUMN cmt_creators.homepage_url IS '达人主页链接';
-- 加微信号
ALTER TABLE cmt_creators ADD COLUMN IF NOT EXISTS wechat VARCHAR(128) DEFAULT '';
COMMENT ON COLUMN cmt_creators.wechat IS '微信号';
-- 加达人类型 (对应飞书"投放账号类型")
ALTER TABLE cmt_creators ADD COLUMN IF NOT EXISTS account_type VARCHAR(64) DEFAULT '';
COMMENT ON COLUMN cmt_creators.account_type IS '达人类型: 数码/vlog/好物/穿搭/摄影等';
-- 加达人层级
ALTER TABLE cmt_creators ADD COLUMN IF NOT EXISTS creator_level VARCHAR(32) DEFAULT '';
COMMENT ON COLUMN cmt_creators.creator_level IS '达人层级: 头部/腰部/KOC/素人';
-- 修改唯一约束: name 单列唯一 → (name, platform) 联合唯一
-- 因为同一达人可能跨平台存在(如BeiBei贝贝同时在小红书和抖音)
-- 先删除旧约束
ALTER TABLE cmt_creators DROP CONSTRAINT IF EXISTS cmt_creators_name_key;
-- 添加新约束
ALTER TABLE cmt_creators ADD CONSTRAINT cmt_creators_name_platform_key UNIQUE (name, platform);
-- 添加索引
CREATE INDEX IF NOT EXISTS idx_cmt_creators_platform ON cmt_creators (platform);
CREATE INDEX IF NOT EXISTS idx_cmt_creators_platform_id ON cmt_creators (platform_id);
-- ============================================================
-- 2. 新建 cmt_cooperations：合作记录表
-- ============================================================
-- 每条记录 = 一个达人在一个款式下的一次合作
-- 飞书款式合作表的每一行 → 此表一条记录
CREATE TABLE IF NOT EXISTS cmt_cooperations (
id BIGSERIAL PRIMARY KEY,
style_id BIGINT NOT NULL REFERENCES cmt_styles(id) ON DELETE CASCADE,
creator_id BIGINT NOT NULL REFERENCES cmt_creators(id) ON DELETE CASCADE,
feishu_record_id VARCHAR(64) DEFAULT '', -- 飞书行 record_id, 用于回写
-- 合作基本信息
cooperation_date DATE DEFAULT NULL, -- 合作日期 (飞书"制单日期")
cooperation_cost NUMERIC(12,2) DEFAULT NULL, -- 合作花费 (飞书"合作费用")
ad_spend NUMERIC(12,2) DEFAULT NULL, -- 投流金额 (飞书"投流金额")
-- 内容信息
content_direction VARCHAR(255) DEFAULT '', -- 内容方向 (飞书"发布笔记内容方向")
content_format VARCHAR(64) DEFAULT '', -- 内容形式: 单品种草/测评合集/场景植入/开箱Vlog/穿搭变装等
note_title VARCHAR(512) DEFAULT '', -- 发布笔记标题
note_url TEXT DEFAULT '', -- 发布笔记链接
publish_time TIMESTAMPTZ DEFAULT NULL, -- 发布时间
-- 数据表现
exposure_count BIGINT DEFAULT NULL, -- 曝光量 (飞书"曝光量(自动抓取)")
engagement_count VARCHAR(64) DEFAULT '', -- 互动赞藏数 (原始文本如"68.5万")
engagement_count_num BIGINT DEFAULT NULL, -- 互动赞藏数(数字解析后)
data_performance VARCHAR(255) DEFAULT '', -- 数据表现 (逗号分隔: 曝光限流,数据良好)
cpm NUMERIC(10,4) DEFAULT NULL, -- CPM (飞书公式字段)
-- 物流/财务
tracking_number VARCHAR(128) DEFAULT '', -- 快递单号
is_paid BOOLEAN DEFAULT FALSE, -- 是否结款
is_new_direction BOOLEAN DEFAULT FALSE, -- 是否新方向验证
-- 电商效果 (飞书暂无,预留)
shop_uv BIGINT DEFAULT NULL, -- 进店UV
transaction_amount NUMERIC(12,2) DEFAULT NULL, -- 成交金额(元)
roi NUMERIC(10,4) DEFAULT NULL, -- ROI
cpe NUMERIC(10,4) DEFAULT NULL, -- CPE (花费/互动量)
engagement_rate NUMERIC(8,4) DEFAULT NULL, -- 互动率 (互动量/曝光量)
-- 评论分析 (飞书暂无,预留)
brand_mention_rate NUMERIC(8,4) DEFAULT NULL, -- 评论品牌提及率
risk_flag VARCHAR(255) DEFAULT '', -- 差评或舆情风险
--
created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
CONSTRAINT uniq_cooperation_feishu_record UNIQUE (feishu_record_id)
);
COMMENT ON TABLE cmt_cooperations IS '合作记录表(每个达人+款式的一次合作)';
COMMENT ON COLUMN cmt_cooperations.cooperation_date IS '合作日期(飞书制单日期)';
COMMENT ON COLUMN cmt_cooperations.cooperation_cost IS '合作花费(元)';
COMMENT ON COLUMN cmt_cooperations.ad_spend IS '投流金额(元)';
COMMENT ON COLUMN cmt_cooperations.content_direction IS '发布笔记内容方向(飞书原始值)';
COMMENT ON COLUMN cmt_cooperations.content_format IS '内容形式: 单品种草/测评合集/场景植入/开箱Vlog等';
COMMENT ON COLUMN cmt_cooperations.exposure_count IS '曝光量(飞书自动抓取)';
COMMENT ON COLUMN cmt_cooperations.engagement_count IS '互动赞藏数(原始文本如68.5万)';
COMMENT ON COLUMN cmt_cooperations.data_performance IS '数据表现(逗号分隔: 曝光限流,数据良好,表现优秀等)';
COMMENT ON COLUMN cmt_cooperations.shop_uv IS '进店UV(预留,飞书暂无)';
COMMENT ON COLUMN cmt_cooperations.transaction_amount IS '成交金额(元,预留)';
COMMENT ON COLUMN cmt_cooperations.roi IS 'ROI=成交金额/花费(预留)';
COMMENT ON COLUMN cmt_cooperations.cpe IS 'CPE=花费/互动量(预留)';
COMMENT ON COLUMN cmt_cooperations.engagement_rate IS '互动率=互动量/曝光量(预留)';
COMMENT ON COLUMN cmt_cooperations.brand_mention_rate IS '评论品牌提及率(预留)';
COMMENT ON COLUMN cmt_cooperations.risk_flag IS '差评或舆情风险(预留)';
CREATE INDEX IF NOT EXISTS idx_cmt_cooperations_style ON cmt_cooperations (style_id);
CREATE INDEX IF NOT EXISTS idx_cmt_cooperations_creator ON cmt_cooperations (creator_id);
CREATE INDEX IF NOT EXISTS idx_cmt_cooperations_date ON cmt_cooperations (cooperation_date);
-- ============================================================
-- 3. 给新表加 updated_at 触发器
-- ============================================================
DO $$
BEGIN
IF NOT EXISTS (
SELECT 1 FROM pg_trigger WHERE tgname = 'trg_cmt_cooperations_updated_at'
) THEN
CREATE TRIGGER trg_cmt_cooperations_updated_at
BEFORE UPDATE ON cmt_cooperations
FOR EACH ROW EXECUTE FUNCTION update_updated_at_column();
END IF;
END;
$$;
src/gyxx_flow/modules/content_marketing/runtime/data/tools/migrations/003_creator_report.sql
+43
View File
@@ -0,0 +1,43 @@
-- 达人月度分析报告表
-- 每月1号自动跑 generate_creator_report.py，结果入库 + 飞书文档
-- UNIQUE(month_start) — 每月一份全量报告
CREATE TABLE IF NOT EXISTS cmt_creator_report (
id BIGSERIAL PRIMARY KEY,
month_start DATE NOT NULL,
month_end DATE NOT NULL,
time_label VARCHAR(32) NOT NULL,
-- 统计快照
total_cooperations INTEGER DEFAULT 0, -- 合作记录总数
total_creators INTEGER DEFAULT 0, -- 达人总数
total_styles INTEGER DEFAULT 0, -- 款式数
total_cost NUMERIC(14,2), -- 总合作花费
total_exposure BIGINT, -- 总曝光量
total_engagement BIGINT, -- 总互动量
avg_cpe NUMERIC(10,2), -- 平均CPE
-- 层级分布快照 (JSON)
level_distribution JSONB DEFAULT '{}', -- {"头部":2,"腰部":36,"KOC":131,"素人":68,"未知":1}
-- 平台分布快照 (JSON)
platform_distribution JSONB DEFAULT '{}', -- {"douyin":196,"xiaohongshu":164,"bilibili":38}
-- 评分分布快照 (JSON)
score_distribution JSONB DEFAULT '{}', -- {"S":2,"A":62,"B":31,"C":143}
-- 飞书文档
doc_url TEXT,
doc_title VARCHAR(200),
report_chars INTEGER, -- 报告字符数
-- 状态
status VARCHAR(32), -- ok/llm_failed/doc_failed/exception
created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
CONSTRAINT uniq_creator_report_month UNIQUE (month_start)
);
CREATE INDEX IF NOT EXISTS idx_cmt_creator_report_month ON cmt_creator_report (month_start);
COMMENT ON TABLE cmt_creator_report IS '达人月度分析报告表(每月1号自动生成,全量达人筛选与报价分析)';
COMMENT ON COLUMN cmt_creator_report.time_label IS '月时间范围标签 如 2026-06-01~06-30';
COMMENT ON COLUMN cmt_creator_report.level_distribution IS '层级分布快照 JSON {"头部":N,"腰部":N,...}';
COMMENT ON COLUMN cmt_creator_report.platform_distribution IS '平台分布快照 JSON {"douyin":N,...}';
COMMENT ON COLUMN cmt_creator_report.score_distribution IS '评分分布快照 JSON {"S":N,"A":N,...}';
COMMENT ON COLUMN cmt_creator_report.doc_url IS '飞书文档链接(达人分析报告)';
COMMENT ON COLUMN cmt_creator_report.status IS 'ok/llm_failed/doc_failed/exception';
src/gyxx_flow/modules/content_marketing/runtime/data/tools/migrations/004_style_category.sql
+4
View File
@@ -0,0 +1,4 @@
ALTER TABLE cmt_styles
ADD COLUMN IF NOT EXISTS style_category VARCHAR(64) DEFAULT '';
COMMENT ON COLUMN cmt_styles.style_category IS '款式风格，来源于飞书全域事项表的风格单选字段';
src/gyxx_flow/modules/content_marketing/runtime/data/tools/migrations/005_style_product_profile.sql
+23
View File
@@ -0,0 +1,23 @@
-- Product profile fields for cmt_styles.
ALTER TABLE cmt_styles
ADD COLUMN IF NOT EXISTS brand VARCHAR(128) DEFAULT '',
ADD COLUMN IF NOT EXISTS erp_style_codes TEXT[] DEFAULT '{}',
ADD COLUMN IF NOT EXISTS platform_product_ids JSONB DEFAULT '{}'::jsonb,
ADD COLUMN IF NOT EXISTS product_type VARCHAR(128) DEFAULT '',
ADD COLUMN IF NOT EXISTS dimensions VARCHAR(255) DEFAULT '',
ADD COLUMN IF NOT EXISTS colors TEXT[] DEFAULT '{}',
ADD COLUMN IF NOT EXISTS material TEXT DEFAULT '',
ADD COLUMN IF NOT EXISTS capacity VARCHAR(128) DEFAULT '',
ADD COLUMN IF NOT EXISTS weight VARCHAR(128) DEFAULT '',
ADD COLUMN IF NOT EXISTS compatible_devices TEXT DEFAULT '',
ADD COLUMN IF NOT EXISTS features TEXT DEFAULT '',
ADD COLUMN IF NOT EXISTS selling_points TEXT DEFAULT '',
ADD COLUMN IF NOT EXISTS product_source_urls TEXT[] DEFAULT '{}',
ADD COLUMN IF NOT EXISTS product_research_status VARCHAR(32) DEFAULT 'pending',
ADD COLUMN IF NOT EXISTS product_researched_at TIMESTAMPTZ;
COMMENT ON COLUMN cmt_styles.platform_product_ids IS 'Product IDs grouped by Feishu platform';
COMMENT ON COLUMN cmt_styles.product_source_urls IS 'Public pages used to verify product facts';
COMMENT ON COLUMN cmt_styles.product_research_status IS 'verified, partial, or pending';
CREATE INDEX IF NOT EXISTS idx_cmt_styles_brand ON cmt_styles (brand);
src/gyxx_flow/modules/content_marketing/runtime/data/tools/monday_self_run.bat
+55
View File
@@ -0,0 +1,55 @@
@echo off
if not defined GYXX_PYTHON set "GYXX_PYTHON=python"
REM Monday 13:00 self-operated pipeline: Bilibili + Chanmama -> sync to cmt_notes (with INSERT)
REM Feishu write-back happens inside each scraper; sync step writes PostgreSQL.
REM Register: schtasks /Create /SC WEEKLY /D MON /TN YingxiaoYunying_MondaySelf /TR %PROJECT_DIR%\data\tools\monday_self_run.bat /ST 13:00 /F
setlocal
for %%I in ("%~dp0..\..") do set "PROJECT_DIR=%%~fI"
if not defined GYXX_DATA_ROOT (
if not defined GYXX_PROJECT_ROOT (
echo ERROR: GYXX_DATA_ROOT or GYXX_PROJECT_ROOT must be defined. 1>&2
exit /b 3
)
set "GYXX_DATA_ROOT=%GYXX_PROJECT_ROOT%\var"
)
set LOG_DIR=%GYXX_DATA_ROOT%\logs\content_marketing\scheduled
set PYTHON=%GYXX_PYTHON%
if not exist "%PYTHON%" set PYTHON=python
if not exist "%LOG_DIR%" mkdir "%LOG_DIR%"
set TS=%date:~0,4%%date:~5,2%%date:~8,2%_%time:~0,2%%time:~3,2%%time:~6,2%
set TS=%TS: =0%
set LOG_FILE=%LOG_DIR%\monday_self_%TS%.log
echo === Monday self-operated run started at %date% %time% === > "%LOG_FILE%"
echo PROJECT_DIR=%PROJECT_DIR% >> "%LOG_FILE%"
cd /d "%PROJECT_DIR%"
echo --- [step 1] self_bilibili_scraper.py --- >> "%LOG_FILE%"
call %PYTHON% self_bilibili_scraper.py >> "%LOG_FILE%" 2>&1
set RC_BILI=%ERRORLEVEL%
echo self_bilibili exit=%RC_BILI% >> "%LOG_FILE%"
echo --- [step 2] chanmama_scraper.py --- >> "%LOG_FILE%"
call %PYTHON% chanmama_scraper.py >> "%LOG_FILE%" 2>&1
set RC_CM=%ERRORLEVEL%
echo chanmama exit=%RC_CM% >> "%LOG_FILE%"
echo --- [step 3] sync_metrics_to_cmt_notes.py (with INSERT) --- >> "%LOG_FILE%"
call %PYTHON% data\tools\sync_metrics_to_cmt_notes.py >> "%LOG_FILE%" 2>&1
set RC_SYNC=%ERRORLEVEL%
echo sync exit=%RC_SYNC% >> "%LOG_FILE%"
set FINAL_RC=0
if not "%RC_BILI%"=="0" set FINAL_RC=1
if not "%RC_CM%"=="0" set FINAL_RC=1
if not "%RC_SYNC%"=="0" set FINAL_RC=1
echo === Monday self-operated run finished at %date% %time% (bili=%RC_BILI%, cm=%RC_CM%, sync=%RC_SYNC%, final=%FINAL_RC%) === >> "%LOG_FILE%"
endlocal & exit /b %FINAL_RC%
src/gyxx_flow/modules/content_marketing/runtime/data/tools/monthly_creator_report.bat
+41
View File
@@ -0,0 +1,41 @@
@echo off
if not defined GYXX_PYTHON set "GYXX_PYTHON=python"
REM Monthly creator report: run on 1st of each month at 08:30
REM 1. Read cmt_cooperations + cmt_creators + cmt_styles from PG
REM 2. Generate 9-section analysis report (Markdown)
REM 3. Create Feishu doc + save to cmt_creator_report
REM Register: schtasks /Create /SC MONTHLY /D 1 /TN YingxiaoYunying_MonthlyCreatorReport /TR %PROJECT_DIR%\data\tools\monthly_creator_report.bat /ST 08:30 /F
setlocal
for %%I in ("%~dp0..\..") do set "PROJECT_DIR=%%~fI"
if not defined GYXX_DATA_ROOT (
if not defined GYXX_PROJECT_ROOT (
echo ERROR: GYXX_DATA_ROOT or GYXX_PROJECT_ROOT must be defined. 1>&2
exit /b 3
)
set "GYXX_DATA_ROOT=%GYXX_PROJECT_ROOT%\var"
)
set LOG_DIR=%GYXX_DATA_ROOT%\logs\content_marketing\scheduled
set PYTHON=%GYXX_PYTHON%
where %PYTHON% >nul 2>&1 || set PYTHON=python
if not exist "%LOG_DIR%" mkdir "%LOG_DIR%"
set TS=%date:~0,4%%date:~5,2%%date:~8,2%_%time:~0,2%%time:~3,2%%time:~6,2%
set TS=%TS: =0%
set LOG_FILE=%LOG_DIR%\monthly_creator_report_%TS%.log
echo === Monthly creator report started at %date% %time% === > "%LOG_FILE%"
echo PROJECT_DIR=%PROJECT_DIR% >> "%LOG_FILE%"
cd /d "%PROJECT_DIR%"
call %PYTHON% -u -X utf8 data/tools/generate_creator_report.py >> "%LOG_FILE%" 2>&1
set RC=%ERRORLEVEL%
echo monthly_creator_report exit=%RC% >> "%LOG_FILE%"
echo === Monthly creator report finished at %date% %time% (rc=%RC%) === >> "%LOG_FILE%"
endlocal
src/gyxx_flow/modules/content_marketing/runtime/data/tools/monthly_summary.bat
+41
View File
@@ -0,0 +1,41 @@
@echo off
if not defined GYXX_PYTHON set "GYXX_PYTHON=python"
REM Monthly summary: run on 1st of each month at 08:00
REM 1. Compute last month date range
REM 2. Query PG for notes per style
REM 3. For each style: note analysis + LLM cross-compare + Feishu doc + write back + save to DB
REM Register: schtasks /Create /SC MONTHLY /D 1 /TN YingxiaoYunying_MonthlySummary /TR %PROJECT_DIR%\data\tools\monthly_summary.bat /ST 08:00 /F
setlocal
for %%I in ("%~dp0..\..") do set "PROJECT_DIR=%%~fI"
if not defined GYXX_DATA_ROOT (
if not defined GYXX_PROJECT_ROOT (
echo ERROR: GYXX_DATA_ROOT or GYXX_PROJECT_ROOT must be defined. 1>&2
exit /b 3
)
set "GYXX_DATA_ROOT=%GYXX_PROJECT_ROOT%\var"
)
set LOG_DIR=%GYXX_DATA_ROOT%\logs\content_marketing\scheduled
set PYTHON=%GYXX_PYTHON%
where %PYTHON% >nul 2>&1 || set PYTHON=python
if not exist "%LOG_DIR%" mkdir "%LOG_DIR%"
set TS=%date:~0,4%%date:~5,2%%date:~8,2%_%time:~0,2%%time:~3,2%%time:~6,2%
set TS=%TS: =0%
set LOG_FILE=%LOG_DIR%\monthly_summary_%TS%.log
echo === Monthly summary started at %date% %time% === > "%LOG_FILE%"
echo PROJECT_DIR=%PROJECT_DIR% >> "%LOG_FILE%"
cd /d "%PROJECT_DIR%"
call %PYTHON% -u -X utf8 monthly_summary_all.py --max-workers 4 >> "%LOG_FILE%" 2>&1
set RC=%ERRORLEVEL%
echo monthly_summary exit=%RC% >> "%LOG_FILE%"
echo === Monthly summary finished at %date% %time% (rc=%RC%) === >> "%LOG_FILE%"
endlocal
src/gyxx_flow/modules/content_marketing/runtime/data/tools/pull_all_tables.py
+163
View File
@@ -0,0 +1,163 @@
"""
拉取 24 张多维表格的全量记录,保存为 JSON。
每张表输出: {base_token, table_id, name, fields, total_records, records}
其中 records 元素为 {record_id, 字段名: 值, ...}
"""
import json
import subprocess
import sys
import time
from pathlib import Path
from gyxx_flow.modules.content_marketing.runtime.runtime_paths import PATHS
TABLES = [
(1, "UkbabpqRYanmD7sk7ksceVuhnwf", "tbl2BdhIaohZNkOD", "盖亚斜挎"),
(2, "OsYvbUe1iaJYeKs5WvLcU4rVnHc", "tblASZDx24q5Bjwy", "极星pro"),
(3, "Qs4mbzyAaaQ8SasnDpkc9lA5npe", "tblMok3p5E6QydPq", "盖亚微单"),
(4, "KnBRbH9iQaH3PAsAUk5cHLxNnMg", "tblcZ8y5126gTHe7", "盖世m1"),
(5, "Ssp8bXmrwahRBbsko2BcuiLanad", "tblqNsVvKj4EPyzA", "星迹2"),
(6, "BJaWbLEOKaHONMshOidcxvsKngh", "tblR0tCfPNk34ULj", "星云2"),
(7, "N43abi7IlaS8XosnUXec6gH2n7g", "tblasQwwbXwpGbpL", "宙斯"),
(8, "W80vbtPDoaaHTOs5Tipc6CTLn6c", "tblUaRl444G5IhVY", "阿波罗2"),
(9, "CtW5bUcc9a5rqRsZkwucUWDVnHb", "tbl8WgoFrOjkqOD5", "阿波罗x1"),
(10, "JsVLbso8Ba6RjbslOl1cJX8lnyb", "tblwqyhbymn9Aour", "波塞冬edge"),
(11, "IQrbbxAHpa9oiWs631GcPkHznob", "tblmWRjVp0wQiVeG", "逐星"),
(12, "MAEZbKrNaabJbDs5ZY7caq1ynrd", "tblcT5gqVRsPBuOE", "逐星GT"),
(13, "RKfkb5JkIa9FttsgHvNc2nJYnbf", "tblu1yXKkQqwje0d", "星云mini"),
(14, "VrLtbkldsaggOms3F4Bcu3DinCg", "tblO5hzLek6Nk43J", "晨星2"),
(15, "GubZbTrMSahLg5sdonIc8kR8nZe", "tbl1tl5CI0LhKeuk", "极星双肩"),
(16, "VwpibmlakaqHuLsGChccMb3BnNd", "tbl7fPd2dMFKQdih", "凌云air"),
(17, "CQcibpRjgaUCg7sUTMwcnrhunMh", "tblFHe5uONCWxbEh", "觅光"),
(18, "XtHjbqpwcab3J4sWP6Fcfgh5nYt", "tblqRgK6MI5C1H4K", "凌云3"),
(19, "GNOkbKNpna8pDJsLMfzcuZt3ncf", "tbljwMiRxb4SlpGr", "瑞白"),
(20, "U2JGbbWcIal6WvsxURRcTMwEnhf", "tblVEdURLsfRZykJ", "黑曜石吐司包"),
(21, "U9S8bLYsNa8Ul8sPffAcrJ02nHI", "tblhpEmlXGyqCGuh", "黑曜石托特包"),
(22, "UPk3bR9MOaTkgrsG1qLc7IUvnQe", "tbllXFWRI59dLCM2", "曜影"),
(23, "PwrGbXcpRaxJnOsd9cacFou8nAf", "tbludhnt5lkmdvSF", "黑曜石电脑包"),
(24, "CNgPbjuXiaPoxPsShROcYbRDnbb", "tblEymEPk7lVaQN1", "拓界2"),
]
OUT_DIR = PATHS.raw_root / "feishu_tables"
OUT_DIR.mkdir(parents=True, exist_ok=True)
import os
LARK_CLI = os.path.expandvars(r"%APPDATA%\npm\lark-cli.cmd")
if not os.path.exists(LARK_CLI):
LARK_CLI = "lark-cli.cmd"
def call_lark(token: str, table_id: str, offset: int, limit: int) -> dict:
"""调用 lark-cli base +record-list,返回 JSON"""
# LARK_CLI_NO_PROXY=1 避免代理警告污染 stderr
cmd = [
LARK_CLI, "base", "+record-list",
"--base-token", token,
"--table-id", table_id,
"--as", "user",
"--format", "json",
"--limit", str(limit),
"--offset", str(offset),
]
env = {**os.environ, "LARK_CLI_NO_PROXY": "1"}
proc = subprocess.run(cmd, capture_output=True, text=True, env=env, encoding="utf-8", shell=False)
if proc.returncode != 0:
raise RuntimeError(f"lark-cli 调用失败(return={proc.returncode}): {proc.stderr}\nstdout: {proc.stdout[:500]}")
# 找到第一个 { 开始
text = proc.stdout
start = text.find("{")
if start < 0:
raise RuntimeError(f"lark-cli 返回无 JSON: {text[:500]}")
# 用 strict=False 容忍控制字符(GBK 字符串残留)
return json.loads(text[start:], strict=False)
def pull_table(idx: int, token: str, table_id: str, name: str) -> dict:
if not table_id:
return {"index": idx, "name": name, "token": token, "table_id": table_id,
"status": "SKIP_NO_TABLE", "total_records": 0}
page_size = 200
offset = 0
fields = None
field_ids = None
records = []
while True:
try:
resp = call_lark(token, table_id, offset, page_size)
except Exception as exc:
return {"index": idx, "name": name, "token": token, "table_id": table_id,
"status": "ERROR", "error": str(exc), "partial_records": len(records)}
if not resp.get("ok"):
return {"index": idx, "name": name, "token": token, "table_id": table_id,
"status": "API_ERROR", "error": resp.get("error", resp),
"partial_records": len(records)}
data = resp["data"]
if fields is None:
fields = data["fields"]
field_ids = data["field_id_list"]
rid_list = data["record_id_list"]
row_data = data["data"]
for i, row in enumerate(row_data):
rec = {"record_id": rid_list[i]}
for j, fid in enumerate(field_ids):
rec[fid] = row[j] if j < len(row) else None
records.append(rec)
offset += len(row_data)
has_more = data.get("has_more", False)
print(f" [{idx:02d}] {name} pulled={len(row_data)} total={len(records)} has_more={has_more}", flush=True)
if not has_more or len(row_data) == 0:
break
time.sleep(0.3)
return {
"index": idx,
"base_token": token,
"table_id": table_id,
"name": name,
"fields": fields,
"field_id_to_name": dict(zip(field_ids, fields)),
"total_records": len(records),
"records": records,
"status": "OK",
}
def main():
summary = []
for idx, token, table_id, name in TABLES:
print(f"[{idx:02d}/24] {name} (table={table_id}) START", flush=True)
result = pull_table(idx, token, table_id, name)
summary.append({
"index": idx, "name": name, "token": token, "table_id": table_id,
"status": result["status"],
"total_records": result.get("total_records") or result.get("partial_records", 0),
})
if result["status"] == "OK":
safe = str(idx) + "-" + name.translate(str.maketrans({
"/": "_", "\\": "_", ":": "_", "*": "_", "?": "_",
"\"": "_", "<": "_", ">": "_", "|": "_", " ": "_"
}))
out_path = OUT_DIR / f"{safe}.json"
with open(out_path, "w", encoding="utf-8") as f:
json.dump(result, f, ensure_ascii=False, indent=2)
print(f" -> saved {out_path} ({result['total_records']} records)", flush=True)
else:
print(f" -> FAILED: {result.get('error', result.get('status'))}", flush=True)
time.sleep(0.5)
with open(OUT_DIR / "_summary.json", "w", encoding="utf-8") as f:
json.dump(summary, f, ensure_ascii=False, indent=2)
print("\n========== 汇总 ==========", flush=True)
for s in summary:
print(f" {s['index']:02d} {s['name']:>16s} {s['status']:>12s} {s['total_records']:>6d}")
if __name__ == "__main__":
main()
src/gyxx_flow/modules/content_marketing/runtime/data/tools/rebuild_mapping.py
+132
View File
@@ -0,0 +1,132 @@
"""
rebuild_mapping.py — 动态重建 款式_多维表格_对照.json
工作原理:调用 feishu_mapping.refresh_mapping() 动态读取「合作达人」地址表,
自动发现所有款式(当前 34 款,新增款式自动往后追加 index)。
用法:
python rebuild_mapping.py # 动态重建 + 写盘 + 打印 diff
python rebuild_mapping.py --diff # 只构建内存 mapping,对比当前 JSON,打印 diff,不写盘
python rebuild_mapping.py --in-place # 等价于默认(直接覆盖 JSON)
python rebuild_mapping.py --print # 打印最新 mapping 到 stdout(不写盘)
依赖: feishu_mapping.py (内部使用), lark-cli (动态读取)
"""
import argparse
import json
import sys
from pathlib import Path
from gyxx_flow.modules.content_marketing.runtime.runtime_paths import PATHS
PROJECT_DIR = PATHS.module_root
from gyxx_flow.modules.content_marketing.runtime import feishu_mapping # noqa: E402
MAPPING_PATH = feishu_mapping.MAPPING_PATH
def diff_entry(new: dict, old: dict | None) -> list[str]:
"""对比单张表的新旧条目"""
if not old:
return [f" + 新增表 [{new['index']:02d}] {new['name']}"]
diffs = []
new_fm = new.get("field_map", {})
old_fm = old.get("field_map", {})
for k, v in new_fm.items():
if k not in old_fm:
diffs.append(f" + [{new['index']:02d}] {new['name']}: 新增字段 {k} = {v['field_name']}({v['field_id']})")
elif old_fm[k].get("field_id") != v["field_id"]:
diffs.append(f" ~ [{new['index']:02d}] {new['name']}: {k} field_id 变更 {old_fm[k]['field_id']} -> {v['field_id']}")
for k in old_fm:
if k not in new_fm:
diffs.append(f" - [{new['index']:02d}] {new['name']}: 字段 {k}({old_fm[k]['field_name']}) 不存在了!")
if new.get("total_records", 0) != old.get("total_records", 0):
diffs.append(f" · [{new['index']:02d}] {new['name']}: total_records {old.get('total_records', 0)} -> {new.get('total_records', 0)}")
return diffs
def diff_mapping(new: dict, old: dict | None) -> list[str]:
"""对比新旧 mapping"""
if old is None:
return [f" + 全新 mapping,共 {len(new.get('tables', []))} 款"]
new_by_name = {t["name"]: t for t in new.get("tables", [])}
old_by_name = {t["name"]: t for t in old.get("tables", [])}
diffs = []
for name in new_by_name:
diffs.extend(diff_entry(new_by_name[name], old_by_name.get(name)))
for name in old_by_name:
if name not in new_by_name:
diffs.append(f" - 表 [{old_by_name[name]['index']:02d}] {name}: 不再出现在地址表!")
return diffs
def print_core_field_check(tables: list[dict]) -> None:
"""打印每张表的核心字段映射情况"""
print(f"\n核心字段映射检查:")
for t in tables:
missing = [k for k in ["creator_name", "note_title", "read_count_7d"] if k not in t["field_map"]]
if missing:
print(f" [WARN] [{t['index']:02d}] {t['name']}: 缺 {missing}")
else:
cn = t['field_map']['creator_name']['field_name']
nt = t['field_map']['note_title']['field_name']
r7 = t['field_map']['read_count_7d']['field_name']
print(f" [OK] [{t['index']:02d}] {t['name']}: creator='{cn}' title='{nt}' read7='{r7}'")
def main():
ap = argparse.ArgumentParser(description="动态重建 款式_多维表格_对照.json")
ap.add_argument("--diff", action="store_true", help="只构建内存 mapping,对比当前 JSON,打印 diff,不写盘")
ap.add_argument("--in-place", action="store_true", help="直接覆盖现有 JSON(等价于默认行为)")
ap.add_argument("--print", action="store_true", help="只打印最新 mapping 到 stdout(不写盘)")
args = ap.parse_args()
# 读旧 JSON
old_mapping = None
if MAPPING_PATH.exists():
try:
old_mapping = json.loads(MAPPING_PATH.read_text(encoding="utf-8"))
except Exception as exc:
print(f"[WARN] 旧 JSON 解析失败: {exc}")
# 决定是否写盘
write = not (args.diff or args.print)
# 动态重建
print(f"动态读取「合作达人」地址表 + 各款式字段定义...\n")
new_mapping = feishu_mapping.refresh_mapping(force=True, write=write)
if new_mapping is None:
print("[FATAL] refresh_mapping 失败")
sys.exit(1)
tables = new_mapping.get("tables", [])
print(f"\n========== 重建结果 ==========")
print(f"表数: {len(tables)}")
if old_mapping:
old_n = len(old_mapping.get("tables", []))
if old_n != len(tables):
print(f" (旧 {old_n} 款 -> 新 {len(tables)} 款)")
# diff
diffs = diff_mapping(new_mapping, old_mapping)
if diffs:
print(f"\n变更 ({len(diffs)} 项):")
for d in diffs:
print(d)
else:
print(f"\n无字段变化")
print_core_field_check(tables)
# 落盘 / 打印
if args.print:
print(f"\n{json.dumps(new_mapping, ensure_ascii=False, indent=2)}")
elif args.diff:
print(f"\n(--diff 模式,未写文件)")
elif write:
print(f"\nsaved (in-place): {MAPPING_PATH}")
else:
print(f"\n(saved): {MAPPING_PATH}")
if __name__ == "__main__":
main()
src/gyxx_flow/modules/content_marketing/runtime/data/tools/relogin_bilibili.py
+87
View File
@@ -0,0 +1,87 @@
"""每周一重登 B 站。
强制清掉 cookie JSON + 浏览器 profile,确保下次启动时必须扫码。
任务计划: 每周一 09:00
跑法:
python data/tools/relogin_bilibili.py # 实际跑(会弹二维码)
python data/tools/relogin_bilibili.py --dry-run # 只看会做什么
"""
import argparse
import datetime as dt
import shutil
import subprocess
import sys
from pathlib import Path
from gyxx_flow.modules.content_marketing.runtime.runtime_paths import PATHS
from gyxx_flow.modules.content_marketing.runtime.data.tools.relogin_transaction import begin, commit, rollback
PROJECT_DIR = PATHS.module_root
COOKIE_FILE = PATHS.state_root / "cookies/bilibili_cookies.json"
STATE_FILE = PATHS.state_root / "cookies/bilibili_storage_state.json"
PROFILE_DIR = PATHS.state_root / "browser-profiles/bilibili"
SCRAPER = PROJECT_DIR / "bilibili_comment_scraper.py"
def reset_state():
"""删 cookie JSON + storage_state + 清浏览器 profile。"""
state = begin((COOKIE_FILE, STATE_FILE), PROFILE_DIR)
print(" [OK] 旧 Cookie/Profile 已暂存,失败时将自动恢复")
return state
def main() -> int:
ap = argparse.ArgumentParser(description="每周一重登 B 站")
ap.add_argument("--login-timeout", type=int, default=300, help="扫码超时(秒)")
ap.add_argument("--dry-run", action="store_true", help="只打印计划,不删不跑")
args = ap.parse_args()
print("== Relogin: B 站 ==")
print(f" 今天: {dt.date.today()} ({dt.date.today().strftime('%A')})")
print(f" cookie: {COOKIE_FILE}")
print(f" profile: {PROFILE_DIR}")
print()
if args.dry_run:
print("[DRY-RUN] 实际跑会: 删 cookie + 清 profile + 弹二维码")
return 0
old_state = reset_state()
cmd = [
sys.executable,
str(SCRAPER),
"login", # dummy url (login-only 模式不需要真实 url)
"--login-only",
"--login-timeout",
str(args.login_timeout),
]
print(f"\n [RUN] {' '.join(cmd)}")
try:
rc = subprocess.call(cmd, cwd=str(PROJECT_DIR))
except BaseException:
rollback(old_state)
raise
print(f"\n exit={rc}")
login_ok = False
if rc == 0:
# 验证 SESSDATA 是否真写入了
try:
import json
cookies = json.loads(COOKIE_FILE.read_text(encoding="utf-8"))
sessdata = any(c.get("name") == "SESSDATA" and c.get("value") for c in cookies)
login_ok = bool(sessdata)
print(f" SESSDATA 写入: {'OK' if sessdata else 'FAIL — 没有看到 SESSDATA,登录可能没成功'}")
except Exception as exc:
print(f" [WARN] 验证 cookie 时出错: {exc}")
if rc != 0 or not login_ok:
rollback(old_state)
print(" [ROLLBACK] 未完成扫码,已恢复旧登录状态")
return rc or 1
commit(old_state)
return 0
if __name__ == "__main__":
sys.exit(main())
src/gyxx_flow/modules/content_marketing/runtime/data/tools/relogin_douyin.py
+93
View File
@@ -0,0 +1,93 @@
"""Force a fresh Douyin login while preserving the previous usable session."""
from __future__ import annotations
import argparse
import datetime as dt
import json
import subprocess
import sys
from pathlib import Path
from gyxx_flow.modules.content_marketing.runtime.runtime_paths import PATHS
try:
from .relogin_transaction import begin, commit, rollback
except ImportError: # Direct script execution from data/tools.
from gyxx_flow.modules.content_marketing.runtime.data.tools.relogin_transaction import begin, commit, rollback
PROJECT_DIR = PATHS.module_root
COOKIE_FILE = PATHS.state_root / "cookies/douyin_cookies.json"
STATE_FILE = PATHS.state_root / "cookies/douyin_storage_state.json"
PROFILE_DIR = PATHS.state_root / "browser-profiles/douyin"
SCRAPER = PROJECT_DIR / "douyin_comment_scraper.py"
LOGIN_COOKIE = "sessionid"
def reset_state():
state = begin((COOKIE_FILE, STATE_FILE), PROFILE_DIR)
print(" [OK] Old Cookie/Profile staged; failures will roll back automatically")
return state
def is_cookie_valid() -> bool:
if not COOKIE_FILE.exists():
return False
try:
cookies = json.loads(COOKIE_FILE.read_text(encoding="utf-8"))
return any(
isinstance(cookie, dict)
and cookie.get("name") == LOGIN_COOKIE
and cookie.get("value")
for cookie in cookies
)
except Exception:
return False
def main() -> int:
parser = argparse.ArgumentParser(description="Relogin Douyin")
parser.add_argument("--login-timeout", type=int, default=300)
parser.add_argument("--dry-run", action="store_true")
args = parser.parse_args()
print("== Relogin: Douyin ==")
print(f" date: {dt.date.today()}")
print(f" cookie: {COOKIE_FILE}")
print(f" profile: {PROFILE_DIR}")
print(f" current cookie: {'present' if is_cookie_valid() else 'missing/invalid'}")
if args.dry_run:
print("[DRY-RUN] Would stage Cookie/Profile and open the QR login window")
return 0
old_state = reset_state()
command = [
sys.executable,
str(SCRAPER),
"login",
"--login-only",
"--login-timeout",
str(args.login_timeout),
]
print(f"\n [RUN] {' '.join(command)}")
try:
return_code = subprocess.call(command, cwd=str(PROJECT_DIR))
except BaseException:
rollback(old_state)
raise
login_ok = return_code == 0 and is_cookie_valid()
print(f"\n exit={return_code}")
print(f" {LOGIN_COOKIE} written: {'OK' if login_ok else 'FAIL'}")
if not login_ok:
rollback(old_state)
print(" [ROLLBACK] Login incomplete; restored previous Cookie/Profile")
return return_code or 1
commit(old_state)
return 0
if __name__ == "__main__":
raise SystemExit(main())
src/gyxx_flow/modules/content_marketing/runtime/data/tools/relogin_pgy.py
+106
View File
@@ -0,0 +1,106 @@
"""每周一重登小红书蒲公英。
强制清掉 cookie JSON + 浏览器 profile,确保下次启动时必须扫码。
任务计划: 每周一 09:00
跑法:
python data/tools/relogin_pgy.py # 实际跑(会弹二维码)
python data/tools/relogin_pgy.py --dry-run # 只看会做什么
"""
import argparse
import datetime as dt
import shutil
import subprocess
import sys
from pathlib import Path
from gyxx_flow.modules.content_marketing.runtime.runtime_paths import PATHS
PROJECT_DIR = PATHS.module_root
COOKIE_FILE = PATHS.state_root / "cookies/pgy_cookies.json"
PROFILE_DIR = PATHS.state_root / "browser-profiles/pgy"
SCRAPER = PROJECT_DIR / "pgy_xhs_scraper_v2.py"
def reset_state() -> tuple[Path | None, Path | None]:
"""Temporarily move the old login state aside so it can be restored."""
ts = dt.datetime.now().strftime("%Y%m%d_%H%M%S_%f")
cookie_backup = None
profile_backup = None
if COOKIE_FILE.exists():
backup = COOKIE_FILE.with_suffix(f".json.bak.{ts}")
COOKIE_FILE.rename(backup)
cookie_backup = backup
print(f" [OK] 备份 cookie -> {backup.name}")
else:
print(f" [INFO] {COOKIE_FILE.name} 不存在,跳过")
if PROFILE_DIR.exists():
profile_backup = PROFILE_DIR.with_name(f"{PROFILE_DIR.name}.bak.{ts}")
PROFILE_DIR.rename(profile_backup)
print(f" [OK] 暂存 {PROFILE_DIR.name}/ -> {profile_backup.name}/")
else:
print(f" [INFO] {PROFILE_DIR.name}/ 不存在,跳过")
return cookie_backup, profile_backup
def restore_previous_state(cookie_backup: Path | None, profile_backup: Path | None) -> None:
"""Discard an incomplete login and restore the last known-good state."""
if COOKIE_FILE.exists():
COOKIE_FILE.unlink()
if cookie_backup and cookie_backup.exists():
cookie_backup.rename(COOKIE_FILE)
print(f" [ROLLBACK] 已恢复旧 cookie: {COOKIE_FILE.name}")
if PROFILE_DIR.exists():
shutil.rmtree(PROFILE_DIR)
if profile_backup and profile_backup.exists():
profile_backup.rename(PROFILE_DIR)
print(f" [ROLLBACK] 已恢复旧浏览器状态: {PROFILE_DIR.name}/")
def finish_successful_relogin(profile_backup: Path | None) -> None:
"""Keep the new state and remove the bulky old profile backup."""
if profile_backup and profile_backup.exists():
shutil.rmtree(profile_backup)
def main() -> int:
ap = argparse.ArgumentParser(description="每周一重登小红书蒲公英")
ap.add_argument("--login-timeout", type=int, default=300, help="扫码超时(秒)")
ap.add_argument("--dry-run", action="store_true", help="只打印计划,不删不跑")
args = ap.parse_args()
print("== Relogin: 小红书蒲公英 ==")
print(f" 今天: {dt.date.today()} ({dt.date.today().strftime('%A')})")
print(f" cookie: {COOKIE_FILE}")
print(f" profile: {PROFILE_DIR}")
print()
if args.dry_run:
print("[DRY-RUN] 实际跑会: 删 cookie + 清 profile + 弹二维码")
return 0
cookie_backup, profile_backup = reset_state()
cmd = [
sys.executable,
str(SCRAPER),
"--login-only",
"--login-timeout",
str(args.login_timeout),
]
print(f"\n [RUN] {' '.join(cmd)}")
try:
rc = subprocess.call(cmd, cwd=str(PROJECT_DIR))
except BaseException:
restore_previous_state(cookie_backup, profile_backup)
raise
print(f"\n exit={rc}")
if rc != 0 or not COOKIE_FILE.exists():
print(" [WARN] 新登录未完成,恢复旧登录状态")
restore_previous_state(cookie_backup, profile_backup)
return rc or 1
finish_successful_relogin(profile_backup)
return rc
if __name__ == "__main__":
sys.exit(main())
src/gyxx_flow/modules/content_marketing/runtime/data/tools/relogin_transaction.py
+47
View File
@@ -0,0 +1,47 @@
"""Transactional backup/rollback helpers for interactive relogin scripts."""
from __future__ import annotations
import datetime as dt
import shutil
from dataclasses import dataclass
from pathlib import Path
@dataclass
class LoginStateBackup:
files: dict[Path, Path]
profile: Path
profile_backup: Path | None
def begin(files: tuple[Path, ...], profile: Path) -> LoginStateBackup:
stamp = dt.datetime.now().strftime("%Y%m%d_%H%M%S_%f")
backups: dict[Path, Path] = {}
for source in files:
if source.exists():
backup = source.with_name(f"{source.name}.bak.{stamp}")
source.rename(backup)
backups[source] = backup
profile_backup = None
if profile.exists():
profile_backup = profile.with_name(f"{profile.name}.bak.{stamp}")
profile.rename(profile_backup)
return LoginStateBackup(backups, profile, profile_backup)
def rollback(state: LoginStateBackup) -> None:
for source, backup in state.files.items():
if source.exists():
source.unlink()
if backup.exists():
backup.rename(source)
if state.profile.exists():
shutil.rmtree(state.profile)
if state.profile_backup and state.profile_backup.exists():
state.profile_backup.rename(state.profile)
def commit(state: LoginStateBackup) -> None:
if state.profile_backup and state.profile_backup.exists():
shutil.rmtree(state.profile_backup)
src/gyxx_flow/modules/content_marketing/runtime/data/tools/relogin_xiaohongshu.py
+90
View File
@@ -0,0 +1,90 @@
"""每周重登小红书。
强制清掉 cookie JSON + 浏览器 profile,确保下次启动时必须扫码。
跑法:
python data/tools/relogin_xiaohongshu.py # 实际跑(会弹二维码)
python data/tools/relogin_xiaohongshu.py --dry-run # 只看会做什么
"""
import argparse
import datetime as dt
import json
import shutil
import subprocess
import sys
from pathlib import Path
from gyxx_flow.modules.content_marketing.runtime.runtime_paths import PATHS
from gyxx_flow.modules.content_marketing.runtime.data.tools.relogin_transaction import begin, commit, rollback
PROJECT_DIR = PATHS.module_root
COOKIE_FILE = PATHS.state_root / "cookies/xiaohongshu_cookies.json"
STATE_FILE = PATHS.state_root / "cookies/xiaohongshu_storage_state.json"
PROFILE_DIR = PATHS.state_root / "browser-profiles/xiaohongshu"
SCRAPER = PROJECT_DIR / "xiaohongshu_comment_scraper.py"
LOGIN_COOKIE = "web_session"
def reset_state():
state = begin((COOKIE_FILE, STATE_FILE), PROFILE_DIR)
print(" [OK] 旧 Cookie/Profile 已暂存,失败时将自动恢复")
return state
def is_cookie_valid() -> bool:
if not COOKIE_FILE.exists():
return False
try:
cookies = json.loads(COOKIE_FILE.read_text(encoding="utf-8"))
return any(c.get("name") == LOGIN_COOKIE and c.get("value") for c in cookies)
except Exception:
return False
def main() -> int:
ap = argparse.ArgumentParser(description="重登小红书")
ap.add_argument("--login-timeout", type=int, default=300, help="扫码超时(秒)")
ap.add_argument("--dry-run", action="store_true", help="只打印计划,不删不跑")
args = ap.parse_args()
print("== Relogin: 小红书 ==")
print(f" 今天: {dt.date.today()}")
print(f" cookie: {COOKIE_FILE}")
print(f" profile: {PROFILE_DIR}")
print(f" 当前 cookie 状态: {'有效' if is_cookie_valid() else '无效/不存在'}")
print()
if args.dry_run:
print("[DRY-RUN] 实际跑会: 删 cookie + 清 profile + 弹二维码")
return 0
old_state = reset_state()
cmd = [
sys.executable,
str(SCRAPER),
"login",
"--login-only",
"--login-timeout",
str(args.login_timeout),
]
print(f"\n [RUN] {' '.join(cmd)}")
try:
rc = subprocess.call(cmd, cwd=str(PROJECT_DIR))
except BaseException:
rollback(old_state)
raise
print(f"\n exit={rc}")
login_ok = rc == 0 and is_cookie_valid()
if rc == 0:
print(f" {LOGIN_COOKIE} 写入: {'OK' if login_ok else 'FAIL — 没有看到 web_session,登录可能没成功'}")
if not login_ok:
rollback(old_state)
print(" [ROLLBACK] 未完成扫码,已恢复旧登录状态")
return rc or 1
commit(old_state)
return 0
if __name__ == "__main__":
sys.exit(main())
src/gyxx_flow/modules/content_marketing/runtime/data/tools/relogin_xingtu.py
+299
View File
@@ -0,0 +1,299 @@
"""每 3 个周一重登一次抖音/星图。
强制清掉 cookie JSON + 浏览器 profile,确保下次启动时必须扫码。
星图登录完成后,自动把登录态 cookie 同步到抖音评论 scraper,避免再扫一次码。
任务计划: 每周一 09:00 (脚本内部判断是否到了该重登的周一,不到就退出)
3 周循环基准: 任意一个周一,后续 (今天 - BASE_DATE) // 7 % 3 == 0 才是星图周。
改 BASE_DATE 就改了重登周期。
跑法:
python data/tools/relogin_xingtu.py # 实际跑
python data/tools/relogin_xingtu.py --dry-run # 看今天是不是星图周
python data/tools/relogin_xingtu.py --force # 忽略 3 周判断,强制重登
"""
import argparse
import datetime as dt
import json
import os
import shutil
import subprocess
import sys
from pathlib import Path
from gyxx_flow.modules.content_marketing.runtime.runtime_paths import PATHS
import psutil
PROJECT_DIR = PATHS.module_root
COOKIE_FILE = PATHS.state_root / "cookies/xingtu_cookies.json"
PROFILE_DIR = PATHS.state_root / "browser-profiles/xingtu"
SCRAPER = PROJECT_DIR / "xingtu_scraper_v2.py"
# 抖音评论 scraper 的 cookie 文件 (登录态共享目标)
DY_COOKIE_FILE = PATHS.state_root / "cookies/douyin_cookies.json"
DY_STATE_FILE = PATHS.state_root / "cookies/douyin_storage_state.json"
DY_PROFILE_DIR = PATHS.state_root / "browser-profiles/douyin"
# 需要同步的抖音登录 cookie 名
DY_LOGIN_COOKIES = {"sessionid", "sessionid_ss", "sid_guard", "sid_tt", "passport_csrf_token"}
# 3 周循环起点(必须是周一)。星图 cookie ~22 天,每 3 周(21 天)重登正好。
BASE_DATE = dt.date(2026, 6, 22)
LOGIN_COOKIE_NAMES = {"sessionid", "sessionid_ss", "sid_tt", "sid_guard"}
def transaction_file() -> Path:
return COOKIE_FILE.with_name(".xingtu_relogin_transaction.json")
def _write_transaction(cookie_backup: Path | None, profile_backup: Path | None) -> None:
marker = transaction_file()
marker.parent.mkdir(parents=True, exist_ok=True)
temp = marker.with_name(f"{marker.name}.tmp")
temp.write_text(json.dumps({
"cookie_backup": str(cookie_backup) if cookie_backup else "",
"profile_backup": str(profile_backup) if profile_backup else "",
}, ensure_ascii=False, indent=2), encoding="utf-8")
os.replace(temp, marker)
def _clear_transaction() -> None:
marker = transaction_file()
if marker.exists():
marker.unlink()
def _remove_path(path: Path) -> None:
if path.is_dir():
shutil.rmtree(path)
elif path.exists():
path.unlink()
def recover_interrupted_state() -> bool:
"""Restore a relogin transaction that was killed after moving old state."""
marker = transaction_file()
if not marker.exists():
return False
try:
state = json.loads(marker.read_text(encoding="utf-8"))
except Exception:
return False
cookie_backup = Path(state["cookie_backup"]) if state.get("cookie_backup") else None
profile_backup = Path(state["profile_backup"]) if state.get("profile_backup") else None
terminate_profile_processes()
if cookie_backup and cookie_backup.exists():
_remove_path(COOKIE_FILE)
cookie_backup.rename(COOKIE_FILE)
if profile_backup and profile_backup.exists():
_remove_path(PROFILE_DIR)
profile_backup.rename(PROFILE_DIR)
_clear_transaction()
print(" [RECOVER] 已恢复上次被中断的星图登录状态")
return True
def has_valid_login_cookie_file(path: Path | None = None) -> bool:
path = path or COOKIE_FILE
try:
cookies = json.loads(path.read_text(encoding="utf-8"))
except Exception:
return False
return any(
isinstance(cookie, dict)
and cookie.get("name") in LOGIN_COOKIE_NAMES
and cookie.get("value")
for cookie in cookies
)
def terminate_profile_processes() -> None:
"""Stop only browser processes using Xingtu's dedicated profile."""
profile_text = str(PROFILE_DIR.resolve()).casefold()
matches = []
for process in psutil.process_iter(["pid", "cmdline"]):
try:
command = " ".join(process.info.get("cmdline") or []).casefold()
if process.pid != os.getpid() and profile_text in command:
matches.append(process)
except (psutil.Error, OSError):
continue
for process in matches:
try:
process.terminate()
except (psutil.Error, OSError):
pass
_gone, alive = psutil.wait_procs(matches, timeout=5)
for process in alive:
try:
process.kill()
except (psutil.Error, OSError):
pass
def is_xingtu_week(today: dt.date, base: dt.date) -> bool:
days = (today - base).days
if days < 0:
return False
return (days // 7) % 3 == 0
def reset_state() -> tuple[Path | None, Path | None]:
"""Temporarily move the old login state aside so it can be restored."""
ts = dt.datetime.now().strftime("%Y%m%d_%H%M%S_%f")
cookie_backup = None
profile_backup = None
if COOKIE_FILE.exists():
cookie_backup = COOKIE_FILE.with_suffix(f".json.bak.{ts}")
if PROFILE_DIR.exists():
profile_backup = PROFILE_DIR.with_name(f"{PROFILE_DIR.name}.bak.{ts}")
_write_transaction(cookie_backup, profile_backup)
if cookie_backup:
COOKIE_FILE.rename(cookie_backup)
print(f" [OK] 备份 cookie -> {cookie_backup.name}")
else:
print(f" [INFO] {COOKIE_FILE.name} 不存在,跳过")
if profile_backup:
PROFILE_DIR.rename(profile_backup)
print(f" [OK] 暂存 {PROFILE_DIR.name}/ -> {profile_backup.name}/")
else:
print(f" [INFO] {PROFILE_DIR.name}/ 不存在,跳过")
return cookie_backup, profile_backup
def restore_previous_state(cookie_backup: Path | None, profile_backup: Path | None) -> None:
"""Discard an incomplete login and restore the last known-good state."""
terminate_profile_processes()
if COOKIE_FILE.exists():
COOKIE_FILE.unlink()
if cookie_backup and cookie_backup.exists():
cookie_backup.rename(COOKIE_FILE)
print(f" [ROLLBACK] 已恢复旧 cookie: {COOKIE_FILE.name}")
if PROFILE_DIR.exists():
shutil.rmtree(PROFILE_DIR)
if profile_backup and profile_backup.exists():
profile_backup.rename(PROFILE_DIR)
print(f" [ROLLBACK] 已恢复旧浏览器状态: {PROFILE_DIR.name}/")
_clear_transaction()
def finish_successful_relogin(profile_backup: Path | None) -> None:
"""Keep the new state and remove the bulky old profile backup."""
if profile_backup and profile_backup.exists():
shutil.rmtree(profile_backup)
_clear_transaction()
def sync_to_douyin_scraper() -> None:
"""把星图的登录态 cookie 同步到抖音评论 scraper,避免再扫一次码。"""
if not COOKIE_FILE.exists():
print(" [SKIP] 星图 cookie 不存在,无法同步")
return
try:
xingtu_cookies = json.loads(COOKIE_FILE.read_text(encoding="utf-8"))
except Exception as exc:
print(f" [SKIP] 读取星图 cookie 失败: {exc}")
return
# 提取登录态 cookie
login_cookies = [c for c in xingtu_cookies if isinstance(c, dict) and c.get("name") in DY_LOGIN_COOKIES]
if not login_cookies:
print(" [SKIP] 星图 cookie 中没有找到登录态,跳过同步")
return
# 清除抖音评论 scraper 的旧 profile (强制刷新)
if DY_PROFILE_DIR.exists():
shutil.rmtree(DY_PROFILE_DIR)
print(f" [OK] 清空 {DY_PROFILE_DIR.name}/")
# 读取或创建抖音评论 cookie 文件
dy_cookies = []
if DY_COOKIE_FILE.exists():
try:
dy_cookies = json.loads(DY_COOKIE_FILE.read_text(encoding="utf-8"))
except Exception:
dy_cookies = []
# 合并: 同名 cookie 覆盖,新 cookie 追加
dy_by_name = {c.get("name"): c for c in dy_cookies if isinstance(c, dict)}
for c in login_cookies:
# 修正 domain: 星图的 cookie domain 可能是 .douyin.com 子域,评论采集需要 .douyin.com
domain = c.get("domain", "")
if "xingtu" in domain or "oceanengine" in domain:
c["domain"] = ".douyin.com"
dy_by_name[c["name"]] = c
DY_COOKIE_FILE.parent.mkdir(parents=True, exist_ok=True)
DY_COOKIE_FILE.write_text(
json.dumps(list(dy_by_name.values()), ensure_ascii=False, indent=2),
encoding="utf-8",
)
print(f" [OK] 同步 {len(login_cookies)} 个登录 cookie 到 {DY_COOKIE_FILE.name}")
def main() -> int:
ap = argparse.ArgumentParser(description="每 3 周周一重登星图")
ap.add_argument("--base-date", default=BASE_DATE.isoformat(),
help="3 周循环起点(必须是周一),格式 YYYY-MM-DD")
ap.add_argument("--login-timeout", type=int, default=300)
ap.add_argument("--force", action="store_true", help="忽略 3 周判断,强制重登")
ap.add_argument("--dry-run", action="store_true", help="只打印计划")
args = ap.parse_args()
today = dt.date.today()
base = dt.date.fromisoformat(args.base_date)
is_xw = is_xingtu_week(today, base)
print("== Relogin: 抖音/星图 ==")
print(f" 今天: {today} ({today.strftime('%A')})")
print(f" 3 周基准: {base}")
print(f" 星图周? {'是' if is_xw else '否'}")
print()
if not is_xw and not args.force:
print("今天不是星图周,跳过 (传 --force 强制重登)")
return 0
if args.dry_run:
print(f"[DRY-RUN] 实际跑会: 删 cookie + 清 profile + 弹二维码")
return 0
recover_interrupted_state()
terminate_profile_processes()
cookie_backup, profile_backup = reset_state()
cmd = [
sys.executable,
str(SCRAPER),
"--login-only",
"--login-timeout",
str(args.login_timeout),
]
print(f"\n [RUN] {' '.join(cmd)}")
try:
rc = subprocess.call(cmd, cwd=str(PROJECT_DIR))
except BaseException:
restore_previous_state(cookie_backup, profile_backup)
raise
finally:
terminate_profile_processes()
print(f"\n exit={rc}")
if rc != 0 or not has_valid_login_cookie_file():
print(" [WARN] 新登录未完成,恢复旧登录状态")
restore_previous_state(cookie_backup, profile_backup)
return rc or 1
finish_successful_relogin(profile_backup)
if rc == 0:
sync_to_douyin_scraper()
return rc
if __name__ == "__main__":
sys.exit(main())
src/gyxx_flow/modules/content_marketing/runtime/data/tools/retry_failed.py
+507
View File
@@ -0,0 +1,507 @@
"""补跑缺失 / 陈旧 / 部分失败的款式(自动检测 + 重试,粒度细到 record_id)。
定义"需要补跑":
MISSING - v2 JSON 文件不存在(进程崩了 / 中途中断) -> 整款重跑
STALE - v2 JSON 文件存在,但 mtime 超过 --stale-hours 小时 -> 整款重跑
ALL_FAILED - v2 JSON 文件新鲜,但 matched=0 且 filled=0 -> 整款重跑
HALF - v2 JSON 文件新鲜,部分 record 失败(matched>0 但有 failed) -> 只补失败的 record_id
HALF 默认开启(因为只补失败的单条,不再昂贵);
可用 --no-include-half 关掉。
用法:
python data/tools/retry_failed.py # 三个平台都补
python data/tools/retry_failed.py --platform xt # 只补星图
python data/tools/retry_failed.py --platform pgy,xt
python data/tools/retry_failed.py --dry-run # 只看不跑
python data/tools/retry_failed.py --style 8 # 强制补指定款式(忽略检测)
python data/tools/retry_failed.py --style 8,15,20
python data/tools/retry_failed.py --max-attempts 3 # 每款式最多重试 3 次
python data/tools/retry_failed.py --stale-hours 4 # 4h 以上的算陈旧
python data/tools/retry_failed.py --no-include-half # 不补 HALF
"""
import argparse
import datetime as dt
import json
import subprocess
import sys
import time
from pathlib import Path
from gyxx_flow.modules.content_marketing.runtime.runtime_paths import PATHS
PROJECT_DIR = PATHS.module_root
V2_DIR = PATHS.normalized_root / "v2_results"
# 复用 run_all.py 的 SCRIPTS 字典,保持单一事实
SCRIPTS = {
"bili": {
"name": "B 站",
"script": "bilibili_scraper.py",
"label": "bili",
},
"pgy": {
"name": "小红书蒲公英",
"script": "pgy_xhs_scraper_v2.py",
"label": "xhs",
},
"xt": {
"name": "星图",
"script": "xingtu_scraper_v2.py",
"label": "xt",
},
}
SELF_SCRIPTS = {
"bili": {**SCRIPTS["bili"], "extra_args": ["--self-operated"]},
"pgy": {**SCRIPTS["pgy"], "extra_args": ["--self-operated"]},
"xt": {
"name": "抖音(自营)",
"script": "self_douyin_scraper.py",
"label": "xt",
"extra_args": [],
},
}
def _mapping_cache_path(self_operated: bool = False) -> Path:
name = "款式_多维表格_对照_自营.json" if self_operated else "款式_多维表格_对照.json"
cache = PATHS.normalized_root / "mappings" / name
return cache if cache.exists() else PATHS.config_root / name
def load_current_styles(self_operated: bool = False) -> list[dict]:
"""只读加载当前款式缓存。
补跑属于结果审计阶段，不能在 import 或 ``--dry-run`` 时触发飞书刷新、
更不能把自营缓存作为副作用写回。映射刷新由 run_all 在采集开始前负责。
"""
path = _mapping_cache_path(self_operated)
try:
data = json.loads(path.read_text(encoding="utf-8"))
except Exception as exc:
log(f"[ERROR] 读取款式缓存失败 {path}: {exc}")
return []
by_index: dict[int, dict] = {}
for raw in data.get("tables", []):
if not isinstance(raw, dict) or not raw.get("index") or not raw.get("name"):
continue
try:
idx = int(raw["index"])
except (TypeError, ValueError):
continue
style = dict(raw)
style["index"] = idx
style["name"] = str(raw["name"])
by_index[idx] = style
return [by_index[idx] for idx in sorted(by_index)]
def canonical_result_filename(style: dict, platform_key: str,
self_operated: bool = False) -> str:
"""返回当前款式唯一允许参与完整度判断的规范结果文件名。"""
if platform_key not in SCRIPTS:
raise ValueError(f"unknown platform: {platform_key}")
idx = int(style["index"])
name = str(style["name"])
self_sfx = "_self" if self_operated else ""
if platform_key == "pgy":
suffix = f"{self_sfx}_v2.json"
elif platform_key == "xt":
suffix = f"{self_sfx}_xingtu_v2.json"
else:
suffix = f"{self_sfx}_bilibili_v2.json"
return f"{idx:02d}-{name}{suffix}"
def log(msg: str) -> None:
ts = dt.datetime.now().strftime("%H:%M:%S")
safe = msg.encode("gbk", errors="replace").decode("gbk", errors="replace")
print(f"[{ts}] {safe}", flush=True)
def _result_rows(data: dict) -> list[dict]:
raw = data.get("results")
if raw is None:
raw = data.get("details")
if not isinstance(raw, list):
return []
return [row for row in raw if isinstance(row, dict)]
def _result_is_unresolved(row: dict) -> bool:
"""兼容新状态契约与旧版 matched/ok/error/reason 字段。"""
status = row.get("status")
if status == "blocked_input":
return False
explicit_failure = bool(
row.get("matched") is False
or row.get("ok") is False
or row.get("write_ok") is False
or row.get("error")
or row.get("reason")
)
if status == "success":
return explicit_failure
if status:
return True
return explicit_failure
def _result_is_success(row: dict) -> bool:
status = row.get("status")
if status is not None:
return status == "success" and not _result_is_unresolved(row)
if _result_is_unresolved(row):
return False
# 旧版 B 站成功行有 ok=True；旧版 pgy/xt 成功行有命中标题或曝光值。
return bool(
row.get("ok") is True
or row.get("matched") is True
or row.get("matched_title")
or row.get("read_count") is not None
or row.get("play_count") is not None
)
def requested_records_succeeded(data: dict,
requested_record_ids: list[str]) -> tuple[bool, list[str]]:
"""核验局部补跑请求的每个 record_id 都恰好有一条 success 终态。"""
rows_by_id: dict[str, list[dict]] = {}
for row in _result_rows(data):
rid = row.get("record_id")
if rid is not None:
rows_by_id.setdefault(str(rid), []).append(row)
failed: list[str] = []
seen: set[str] = set()
for raw_rid in requested_record_ids:
rid = str(raw_rid)
if rid in seen:
continue
seen.add(rid)
candidates = rows_by_id.get(rid, [])
if len(candidates) != 1 or not _result_is_success(candidates[0]):
failed.append(rid)
return not failed, failed
def _summary_completed(data: dict) -> tuple[bool, str]:
"""核验整款结果覆盖、唯一性和终态；blocked_input 是已解释终态。"""
total = data.get("total")
if total is None:
total = data.get("total_b_records")
if total is None:
total = data.get("total_tasks")
try:
total = int(total or 0)
except (TypeError, ValueError):
return False, "invalid total"
if total < 0:
return False, "invalid total"
rows = _result_rows(data)
record_ids = [str(row["record_id"]) for row in rows if row.get("record_id")]
if len(rows) != total:
return False, f"result coverage {len(rows)}/{total}"
if len(record_ids) != len(rows) or len(set(record_ids)) != len(record_ids):
return False, "missing or duplicate record_id"
unresolved = [rid for rid, row in zip(record_ids, rows) if _result_is_unresolved(row)]
if unresolved:
return False, f"unresolved records: {','.join(unresolved[:5])}"
if _summary_declares_incomplete(data):
return False, "summary declares unresolved failures"
return True, ""
def _summary_declares_incomplete(data: dict) -> bool:
if data.get("complete") is False:
return True
for key in (
"unresolved", "retryable_failures", "write_failures",
"missing_results", "duplicate_results",
):
try:
if int(data.get(key) or 0) > 0:
return True
except (TypeError, ValueError):
return True
return False
def extract_failed_records(data: dict, platform_key: str) -> list[str]:
"""从 v2 JSON 内容里提取失败的 record_id 列表。
xingtu/pgy: results 数组,失败标志 matched: false(可能加 reason/error)
bilibili: details 数组,失败标志 ok: false(可能加 action)
"""
rids: list[str] = []
for row in _result_rows(data):
rid = row.get("record_id")
if not rid:
continue
# blocked_input 是可解释但不可重试的终态，不应反复补跑。
if _result_is_unresolved(row):
rids.append(str(rid))
return list(dict.fromkeys(rids))
def find_failed(platform_key: str, stale_hours: float, include_half: bool,
self_operated: bool = False, *,
styles: list[dict] | None = None,
v2_dir: Path | None = None,
now: dt.datetime | None = None) -> dict[int, dict]:
"""扫 v2_results,返回 {idx: {reason, failed_rids}} 字典。
reason 取值: MISSING / STALE / ALL_FAILED / HALF
failed_rids: HALF 时给失败 record_id 列表;其他场景为空(整款重跑)。
self_operated=True 时只扫自营达人结果文件(*_self_*.json)
"""
if platform_key not in SCRIPTS:
raise ValueError(f"unknown platform: {platform_key}")
current_styles = styles if styles is not None else load_current_styles(self_operated)
result_dir = v2_dir or V2_DIR
current_time = now or dt.datetime.now()
out: dict[int, dict] = {}
for style in current_styles:
idx = int(style["index"])
path = result_dir / canonical_result_filename(style, platform_key, self_operated)
if not path.exists():
out[idx] = {"reason": "MISSING", "failed_rids": []}
continue
mtime = dt.datetime.fromtimestamp(path.stat().st_mtime)
age_h = (current_time - mtime).total_seconds() / 3600.0
if age_h > stale_hours:
out[idx] = {"reason": "STALE", "failed_rids": []}
continue
try:
data = json.loads(path.read_text(encoding="utf-8"))
except Exception:
out[idx] = {"reason": "INVALID", "failed_rids": []}
continue
total = data.get("total")
if total is None:
total = data.get("total_b_records")
if total is None:
total = data.get("total_tasks")
try:
total = int(total or 0)
except (TypeError, ValueError):
out[idx] = {"reason": "INVALID", "failed_rids": []}
continue
rows = _result_rows(data)
failed_rids = extract_failed_records(data, platform_key)
resolved = sum(1 for row in rows if not _result_is_unresolved(row))
if total > 0 and resolved == 0:
out[idx] = {"reason": "ALL_FAILED", "failed_rids": []}
continue
record_ids = [str(row["record_id"]) for row in rows if row.get("record_id")]
coverage_invalid = (
len(rows) != total
or len(record_ids) != len(rows)
or len(set(record_ids)) != len(record_ids)
)
if coverage_invalid:
# 缺失记录无法从结果中推导 record_id，只能整款重跑。
out[idx] = {"reason": "INCOMPLETE", "failed_rids": []}
continue
if include_half and failed_rids:
out[idx] = {"reason": "HALF", "failed_rids": failed_rids}
continue
if _summary_declares_incomplete(data):
out[idx] = {"reason": "INCOMPLETE", "failed_rids": []}
return out
def run_one_style(platform_key: str, idx: int, max_attempts: int,
self_operated: bool = False,
record_ids: list[str] | None = None, *,
style: dict | None = None,
v2_dir: Path | None = None) -> bool:
"""跑单款式,失败重试。返回最终是否成功。
record_ids 非空时,只跑这些 record_id(--record 参数,粒度细到单条笔记)。
"""
info = SELF_SCRIPTS[platform_key] if self_operated else SCRIPTS[platform_key]
if style is None:
style = next(
(item for item in load_current_styles(self_operated)
if int(item["index"]) == int(idx)),
None,
)
if style is None:
log(f" [ERROR] 当前款式映射中不存在编号 {idx}")
return False
result_path = (v2_dir or V2_DIR) / canonical_result_filename(
style, platform_key, self_operated,
)
cmd = [sys.executable, str(PROJECT_DIR / info["script"]), "--style", str(idx)]
cmd.extend(info.get("extra_args", []))
for rid in record_ids or []:
cmd += ["--record", rid]
label = f"[{platform_key}{'-self' if self_operated else ''}] 款式 {idx:>2}"
desc = f"整款" if not record_ids else f"{len(record_ids)} 条"
for attempt in range(1, max_attempts + 1):
log(f" {label} 尝试 {attempt}/{max_attempts} ({desc}): {info['script']} --style {idx}"
+ (f" --record x{len(record_ids)}" if record_ids else ""))
before_mtime_ns = result_path.stat().st_mtime_ns if result_path.exists() else None
try:
rc = subprocess.call(cmd, cwd=str(PROJECT_DIR))
except KeyboardInterrupt:
raise
except Exception as exc:
log(f" [ERROR] 进程异常: {exc}")
rc = -1
if rc != 0:
log(f" [FAIL] exit={rc}")
else:
try:
after_mtime_ns = result_path.stat().st_mtime_ns
if before_mtime_ns is not None and after_mtime_ns == before_mtime_ns:
raise RuntimeError("规范结果文件未在本次补跑中更新")
payload = json.loads(result_path.read_text(encoding="utf-8"))
if not isinstance(payload, dict):
raise RuntimeError("规范结果不是 JSON object")
if record_ids:
verified, failed_rids = requested_records_succeeded(payload, record_ids)
reason = f"仍失败/缺失 record: {','.join(failed_rids[:5])}"
else:
verified, reason = _summary_completed(payload)
if verified:
log(" [OK] exit=0 且结果完整度校验通过")
return True
log(f" [INCOMPLETE] exit=0 但{reason}")
except Exception as exc:
log(f" [INCOMPLETE] exit=0 但结果校验失败: {exc}")
if attempt < max_attempts:
time.sleep(2)
return False
def main() -> int:
ap = argparse.ArgumentParser(description="补跑缺失 / 陈旧的款式")
ap.add_argument("--platform", type=str, default="bili,pgy,xt",
help="逗号分隔 bili/pgy/xt,默认全跑")
ap.add_argument("--style", type=str, default="",
help="强制指定要补的款式(逗号分隔),忽略检测")
ap.add_argument("--dry-run", action="store_true", help="只打印计划,不跑")
ap.add_argument("--max-attempts", type=int, default=1,
help="每个款式最多重试次数(默认 1)")
ap.add_argument("--stale-hours", type=float, default=26.0,
help="超过多少小时算陈旧(默认 26h)")
ap.add_argument("--no-include-half", action="store_true",
help="不补 HALF 款式(部分 record 失败的款);默认会补,只跑失败的 record_id")
ap.add_argument("--include-self-operated", action="store_true",
help="同时补自营达人(默认只补合作达人)")
ap.add_argument("--self-operated-only", action="store_true",
help="只补自营达人")
args = ap.parse_args()
include_half = not args.no_include_half
keys = [k.strip() for k in args.platform.split(",") if k.strip()]
for k in keys:
if k not in SCRIPTS:
print(f"[ERROR] 未知平台: {k} 可选: {','.join(SCRIPTS.keys())}")
return 1
forced: list[int] = []
if args.style:
for s in args.style.split(","):
s = s.strip()
if not s:
continue
try:
idx = int(s)
except ValueError:
print(f"[ERROR] --style 解析失败: {s}")
return 1
forced.append(idx)
# 决定要跑哪些轮: 合作达人 + 自营达人
rounds: list[tuple[bool, str]] = [] # (self_operated, label)
if not args.self_operated_only:
rounds.append((False, "合作达人"))
if args.include_self_operated or args.self_operated_only:
rounds.append((True, "自营达人"))
any_failed = False
for self_op, round_label in rounds:
styles = load_current_styles(self_op)
if not styles:
log(f"[ERROR] {round_label} 当前款式缓存为空，无法安全补跑")
any_failed = True
continue
styles_by_index = {int(style["index"]): style for style in styles}
invalid_forced = [idx for idx in forced if idx not in styles_by_index]
if invalid_forced:
log(f"[ERROR] {round_label} 不存在款式编号: {invalid_forced}")
any_failed = True
print(f"{'=' * 78}")
print(f" Retry Failed Styles — {round_label}")
print(f" 平台: {','.join(SCRIPTS[k]['name'] for k in keys)}")
print(f" stale-hours: {args.stale_hours} max-attempts: {args.max_attempts}")
print(f" include-half: {include_half} dry-run: {args.dry_run}")
if forced:
print(f" 强制指定: {forced}")
print(f"{'=' * 78}")
grand_total = grand_ok = grand_fail = 0
grand_failed_list: list[tuple[str, int]] = []
for k in keys:
info = SCRIPTS[k]
if forced:
targets = {idx: {"reason": "强制", "failed_rids": []}
for idx in forced if idx in styles_by_index}
else:
targets = find_failed(
k, args.stale_hours, include_half,
self_operated=self_op, styles=styles,
)
log(f"[{k}{'-self' if self_op else ''}] {info['name']} 待补: {len(targets)} 个 -> {sorted(targets.keys())}")
if not targets:
continue
if args.dry_run:
for idx in sorted(targets.keys()):
info_i = targets[idx]
n = len(info_i["failed_rids"])
detail = f", {n} 条 record" if n else ""
log(f" [DRY-RUN] 款式 {idx:>2} ({info_i['reason']}{detail})")
grand_total += len(targets)
continue
for idx in sorted(targets.keys()):
info_i = targets[idx]
grand_total += 1
ok = run_one_style(k, idx, args.max_attempts,
self_operated=self_op,
record_ids=info_i["failed_rids"] or None,
style=styles_by_index[idx])
if ok:
grand_ok += 1
else:
grand_fail += 1
grand_failed_list.append((k, idx))
any_failed = True
print(f"\n{'=' * 78}")
print(f" {round_label} 补跑汇总: 总 {grand_total} 成功 {grand_ok} 失败 {grand_fail}")
if grand_failed_list:
print(f" 仍失败的:")
for k, idx in grand_failed_list:
print(f" - [{k}] 款式 {idx}")
print(f"{'=' * 78}\n")
return 1 if any_failed else 0
if __name__ == "__main__":
try:
sys.exit(main())
except KeyboardInterrupt:
print("\n[中断]", file=sys.stderr)
sys.exit(130)
src/gyxx_flow/modules/content_marketing/runtime/data/tools/schema_gyxx_super_data.sql
+342
View File
@@ -0,0 +1,342 @@
-- 数据库：gyxx_super_data
-- 表结构：cmt_styles / cmt_creators / cmt_style_creators / cmt_notes / cmt_comments
-- 适用于 PostgreSQL 14+
-- 如果数据库还没创建，取消下面一行的注释并执行：
-- CREATE DATABASE gyxx_super_data WITH ENCODING = 'UTF8' LC_COLLATE = 'zh_CN.UTF-8' LC_CTYPE = 'zh_CN.UTF-8';
\c gyxx_super_data;
-- 款式表
CREATE TABLE IF NOT EXISTS cmt_styles (
id BIGSERIAL PRIMARY KEY,
name VARCHAR(255) NOT NULL UNIQUE,
style_category VARCHAR(64) DEFAULT '',
brand VARCHAR(128) DEFAULT '',
erp_style_codes TEXT[] DEFAULT '{}',
platform_product_ids JSONB DEFAULT '{}'::jsonb,
product_type VARCHAR(128) DEFAULT '',
dimensions VARCHAR(255) DEFAULT '',
colors TEXT[] DEFAULT '{}',
material TEXT DEFAULT '',
capacity VARCHAR(128) DEFAULT '',
weight VARCHAR(128) DEFAULT '',
compatible_devices TEXT DEFAULT '',
features TEXT DEFAULT '',
selling_points TEXT DEFAULT '',
product_source_urls TEXT[] DEFAULT '{}',
product_research_status VARCHAR(32) DEFAULT 'pending',
product_researched_at TIMESTAMPTZ,
platform VARCHAR(64) DEFAULT '',
feishu_base_url TEXT DEFAULT '',
feishu_base_token VARCHAR(64) DEFAULT '',
feishu_table_id VARCHAR(64) DEFAULT '',
feishu_view_id VARCHAR(64) DEFAULT '',
created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP
);
COMMENT ON TABLE cmt_styles IS '款式表';
COMMENT ON COLUMN cmt_styles.name IS '款式名称';
COMMENT ON COLUMN cmt_styles.style_category IS '款式风格';
COMMENT ON COLUMN cmt_styles.platform IS '平台分组，如天猫';
CREATE INDEX IF NOT EXISTS idx_cmt_styles_platform ON cmt_styles (platform);
-- 博主/达人表
CREATE TABLE IF NOT EXISTS cmt_creators (
id BIGSERIAL PRIMARY KEY,
name VARCHAR(255) NOT NULL,
platform VARCHAR(32) DEFAULT '', -- 主平台: xiaohongshu/douyin/bilibili
platform_id VARCHAR(128) DEFAULT '', -- 平台达人ID (B站UID/抖音sec_uid等)
follower_count VARCHAR(64) DEFAULT '', -- 达人粉丝数(原始文本如3.7万/12.4w)
follower_count_num BIGINT DEFAULT NULL, -- 达人粉丝数(数字解析后,3.7万=37000)
homepage_url TEXT DEFAULT '', -- 达人主页链接
wechat VARCHAR(128) DEFAULT '', -- 微信号
account_type VARCHAR(64) DEFAULT '', -- 达人类型: 数码/vlog/好物/穿搭/摄影等
creator_level VARCHAR(32) DEFAULT '', -- 达人层级: 头部/腰部/KOC/素人
created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
CONSTRAINT cmt_creators_name_platform_key UNIQUE (name, platform)
);
COMMENT ON TABLE cmt_creators IS '博主/达人表';
COMMENT ON COLUMN cmt_creators.name IS '博主/达人名称';
COMMENT ON COLUMN cmt_creators.platform IS '主平台: xiaohongshu/douyin/bilibili';
COMMENT ON COLUMN cmt_creators.platform_id IS '平台达人ID (B站UID/抖音sec_uid等)';
COMMENT ON COLUMN cmt_creators.follower_count IS '达人粉丝数(原始文本如3.7万/12.4w)';
COMMENT ON COLUMN cmt_creators.follower_count_num IS '达人粉丝数(数字解析后,3.7万=37000)';
COMMENT ON COLUMN cmt_creators.homepage_url IS '达人主页链接';
COMMENT ON COLUMN cmt_creators.wechat IS '微信号';
COMMENT ON COLUMN cmt_creators.account_type IS '达人类型: 数码/vlog/好物/穿搭/摄影等';
COMMENT ON COLUMN cmt_creators.creator_level IS '达人层级: 头部/腰部/KOC/素人';
CREATE INDEX IF NOT EXISTS idx_cmt_creators_name ON cmt_creators (name);
CREATE INDEX IF NOT EXISTS idx_cmt_creators_platform ON cmt_creators (platform);
CREATE INDEX IF NOT EXISTS idx_cmt_creators_platform_id ON cmt_creators (platform_id);
-- 款式-博主关系表（多对多）
CREATE TABLE IF NOT EXISTS cmt_style_creators (
id BIGSERIAL PRIMARY KEY,
style_id BIGINT NOT NULL,
creator_id BIGINT NOT NULL,
created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
UNIQUE (style_id, creator_id),
CONSTRAINT fk_cmt_sc_style FOREIGN KEY (style_id) REFERENCES cmt_styles (id) ON DELETE CASCADE,
CONSTRAINT fk_cmt_sc_creator FOREIGN KEY (creator_id) REFERENCES cmt_creators (id) ON DELETE CASCADE
);
COMMENT ON TABLE cmt_style_creators IS '款式-博主关系表';
CREATE INDEX IF NOT EXISTS idx_cmt_style_creators_creator ON cmt_style_creators (creator_id);
-- 笔记表
CREATE TABLE IF NOT EXISTS cmt_notes (
id BIGSERIAL PRIMARY KEY,
style_id BIGINT NOT NULL,
creator_id BIGINT NOT NULL,
platform VARCHAR(32) NOT NULL,
title VARCHAR(512) DEFAULT '',
url TEXT NOT NULL UNIQUE,
feishu_record_id VARCHAR(64) DEFAULT '',
view_count BIGINT DEFAULT NULL, -- 曝光量 / 播放量
like_count BIGINT DEFAULT NULL,
collect_count BIGINT DEFAULT NULL, -- 小红书 / 抖音收藏
favorite_count BIGINT DEFAULT NULL, -- B 站收藏
comment_count BIGINT DEFAULT NULL,
share_count BIGINT DEFAULT NULL,
scraped_at TIMESTAMPTZ DEFAULT NULL,
self_operated BOOLEAN DEFAULT FALSE, -- TRUE=自营达人笔记, FALSE=合作达人笔记
publish_time TIMESTAMPTZ DEFAULT NULL, -- 笔记发布时间(从飞书 publish_time 字段同步)
created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
CONSTRAINT fk_cmt_notes_style FOREIGN KEY (style_id) REFERENCES cmt_styles (id) ON DELETE CASCADE,
CONSTRAINT fk_cmt_notes_creator FOREIGN KEY (creator_id) REFERENCES cmt_creators (id) ON DELETE CASCADE
);
COMMENT ON COLUMN cmt_notes.view_count IS '曝光量 / 播放量 (来自 yingxiaoyunying 每日采集)';
COMMENT ON COLUMN cmt_notes.collect_count IS '收藏量: 小红书/抖音 (来自 yingxiaoyunying 每日采集)';
COMMENT ON COLUMN cmt_notes.favorite_count IS '收藏量: B 站 (来自 comment-data-collector)';
COMMENT ON TABLE cmt_notes IS '笔记表';
COMMENT ON COLUMN cmt_notes.platform IS '平台：xiaohongshu / douyin / bilibili';
COMMENT ON COLUMN cmt_notes.url IS '笔记链接';
COMMENT ON COLUMN cmt_notes.scraped_at IS '最近一次抓取时间';
COMMENT ON COLUMN cmt_notes.self_operated IS 'TRUE=自营达人笔记, FALSE=合作达人笔记';
COMMENT ON COLUMN cmt_notes.publish_time IS '笔记发布时间(从飞书 publish_time 字段同步)';
CREATE INDEX IF NOT EXISTS idx_cmt_notes_style ON cmt_notes (style_id);
CREATE INDEX IF NOT EXISTS idx_cmt_notes_creator ON cmt_notes (creator_id);
CREATE INDEX IF NOT EXISTS idx_cmt_notes_platform ON cmt_notes (platform);
CREATE INDEX IF NOT EXISTS idx_cmt_notes_scraped_at ON cmt_notes (scraped_at);
-- 评论表
CREATE TABLE IF NOT EXISTS cmt_comments (
id BIGSERIAL PRIMARY KEY,
note_id BIGINT NOT NULL,
platform VARCHAR(32) NOT NULL,
level VARCHAR(16) NOT NULL,
platform_comment_id VARCHAR(64) DEFAULT '',
parent_comment_id VARCHAR(64) DEFAULT '',
reply_to_comment_id VARCHAR(64) DEFAULT '',
user_id VARCHAR(128) DEFAULT '',
nickname VARCHAR(255) DEFAULT '',
content TEXT DEFAULT NULL,
like_count BIGINT DEFAULT 0,
reply_count BIGINT DEFAULT 0,
ip_location VARCHAR(128) DEFAULT '',
comment_created_at TIMESTAMPTZ DEFAULT NULL,
raw_create_time VARCHAR(64) DEFAULT '',
scraped_at TIMESTAMPTZ DEFAULT NULL,
created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
UNIQUE (note_id, platform_comment_id),
CONSTRAINT fk_cmt_comments_note FOREIGN KEY (note_id) REFERENCES cmt_notes (id) ON DELETE CASCADE
);
COMMENT ON TABLE cmt_comments IS '评论表';
COMMENT ON COLUMN cmt_comments.level IS '层级：comment 一级评论 / reply 回复';
CREATE INDEX IF NOT EXISTS idx_cmt_comments_note_id ON cmt_comments (note_id);
-- ============================================================
-- 合作记录表 (达人 + 款式维度)
-- ============================================================
-- 每条记录 = 一个达人在一个款式下的一次合作
-- 飞书款式合作表的每一行 → 此表一条记录
CREATE TABLE IF NOT EXISTS cmt_cooperations (
id BIGSERIAL PRIMARY KEY,
style_id BIGINT NOT NULL REFERENCES cmt_styles(id) ON DELETE CASCADE,
creator_id BIGINT NOT NULL REFERENCES cmt_creators(id) ON DELETE CASCADE,
feishu_record_id VARCHAR(64) DEFAULT '', -- 飞书行 record_id, 用于回写
-- 合作基本信息
cooperation_date DATE DEFAULT NULL, -- 合作日期 (飞书"制单日期")
cooperation_cost NUMERIC(12,2) DEFAULT NULL, -- 合作花费 (飞书"合作费用")
ad_spend NUMERIC(12,2) DEFAULT NULL, -- 投流金额 (飞书"投流金额")
-- 内容信息
content_direction VARCHAR(255) DEFAULT '', -- 内容方向 (飞书"发布笔记内容方向")
content_format VARCHAR(64) DEFAULT '', -- 内容形式: 单品种草/测评合集/场景植入/开箱Vlog/穿搭变装等
note_title VARCHAR(512) DEFAULT '', -- 发布笔记标题
note_url TEXT DEFAULT '', -- 发布笔记链接
publish_time TIMESTAMPTZ DEFAULT NULL, -- 发布时间
-- 数据表现
exposure_count BIGINT DEFAULT NULL, -- 曝光量 (飞书"曝光量(自动抓取)")
engagement_count VARCHAR(64) DEFAULT '', -- 互动赞藏数 (原始文本如"68.5万")
engagement_count_num BIGINT DEFAULT NULL, -- 互动赞藏数(数字解析后)
data_performance VARCHAR(255) DEFAULT '', -- 数据表现 (逗号分隔: 曝光限流,数据良好)
cpm NUMERIC(10,4) DEFAULT NULL, -- CPM (飞书公式字段)
-- 物流/财务
tracking_number VARCHAR(128) DEFAULT '', -- 快递单号
is_paid BOOLEAN DEFAULT FALSE, -- 是否结款
is_new_direction BOOLEAN DEFAULT FALSE, -- 是否新方向验证
-- 电商效果 (飞书暂无,预留)
shop_uv BIGINT DEFAULT NULL, -- 进店UV
transaction_amount NUMERIC(12,2) DEFAULT NULL, -- 成交金额(元)
roi NUMERIC(10,4) DEFAULT NULL, -- ROI (成交金额/花费)
cpe NUMERIC(10,4) DEFAULT NULL, -- CPE (花费/互动量)
engagement_rate NUMERIC(8,4) DEFAULT NULL, -- 互动率 (互动量/曝光量)
-- 评论分析 (飞书暂无,预留)
brand_mention_rate NUMERIC(8,4) DEFAULT NULL, -- 评论品牌提及率
risk_flag VARCHAR(255) DEFAULT '', -- 差评或舆情风险
--
created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
CONSTRAINT uniq_cooperation_feishu_record UNIQUE (feishu_record_id)
);
COMMENT ON TABLE cmt_cooperations IS '合作记录表(每个达人+款式的一次合作)';
COMMENT ON COLUMN cmt_cooperations.cooperation_date IS '合作日期(飞书制单日期)';
COMMENT ON COLUMN cmt_cooperations.cooperation_cost IS '合作花费(元)';
COMMENT ON COLUMN cmt_cooperations.ad_spend IS '投流金额(元)';
COMMENT ON COLUMN cmt_cooperations.content_direction IS '发布笔记内容方向(飞书原始值)';
COMMENT ON COLUMN cmt_cooperations.content_format IS '内容形式: 单品种草/测评合集/场景植入/开箱Vlog等';
COMMENT ON COLUMN cmt_cooperations.exposure_count IS '曝光量(飞书自动抓取)';
COMMENT ON COLUMN cmt_cooperations.engagement_count IS '互动赞藏数(原始文本如68.5万)';
COMMENT ON COLUMN cmt_cooperations.data_performance IS '数据表现(逗号分隔: 曝光限流,数据良好,表现优秀等)';
COMMENT ON COLUMN cmt_cooperations.shop_uv IS '进店UV(预留,飞书暂无)';
COMMENT ON COLUMN cmt_cooperations.transaction_amount IS '成交金额(元,预留)';
COMMENT ON COLUMN cmt_cooperations.roi IS 'ROI=成交金额/花费(预留)';
COMMENT ON COLUMN cmt_cooperations.cpe IS 'CPE=花费/互动量(预留)';
COMMENT ON COLUMN cmt_cooperations.engagement_rate IS '互动率=互动量/曝光量(预留)';
COMMENT ON COLUMN cmt_cooperations.brand_mention_rate IS '评论品牌提及率(预留)';
COMMENT ON COLUMN cmt_cooperations.risk_flag IS '差评或舆情风险(预留)';
CREATE INDEX IF NOT EXISTS idx_cmt_cooperations_style ON cmt_cooperations (style_id);
CREATE INDEX IF NOT EXISTS idx_cmt_cooperations_creator ON cmt_cooperations (creator_id);
CREATE INDEX IF NOT EXISTS idx_cmt_cooperations_date ON cmt_cooperations (cooperation_date);
-- ============================================================
-- 周笔记汇总表 (款式 + 周维度)
-- ============================================================
CREATE TABLE IF NOT EXISTS cmt_weekly_summary (
id BIGSERIAL PRIMARY KEY,
style_id BIGINT NOT NULL REFERENCES cmt_styles(id) ON DELETE CASCADE,
week_start DATE NOT NULL,
week_end DATE NOT NULL,
time_label VARCHAR(32) NOT NULL,
note_count INTEGER DEFAULT 0,
total_comments INTEGER DEFAULT 0,
total_views BIGINT,
total_engagement BIGINT,
doc_url TEXT,
doc_title VARCHAR(200),
summary_chars INTEGER,
status VARCHAR(32),
created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
CONSTRAINT uniq_weekly_style_week UNIQUE (style_id, week_start)
);
CREATE INDEX IF NOT EXISTS idx_cmt_weekly_summary_week ON cmt_weekly_summary (week_start);
CREATE INDEX IF NOT EXISTS idx_cmt_weekly_summary_style ON cmt_weekly_summary (style_id);
COMMENT ON TABLE cmt_weekly_summary IS '周笔记汇总表(款式+周维度,LLM 分析报告飞书文档链接)';
COMMENT ON COLUMN cmt_weekly_summary.time_label IS '周时间范围标签 如 2026-06-29~07-05';
COMMENT ON COLUMN cmt_weekly_summary.doc_url IS '飞书文档链接(每周笔记汇总报告)';
COMMENT ON COLUMN cmt_weekly_summary.status IS 'ok/no_reports/no_notes/exception';
-- ============================================================
-- 月笔记汇总表 (款式 + 月维度,横向对比分析)
-- ============================================================
CREATE TABLE IF NOT EXISTS cmt_monthly_summary (
id BIGSERIAL PRIMARY KEY,
style_id BIGINT NOT NULL REFERENCES cmt_styles(id) ON DELETE CASCADE,
month_start DATE NOT NULL,
month_end DATE NOT NULL,
time_label VARCHAR(32) NOT NULL,
note_count INTEGER DEFAULT 0,
total_comments INTEGER DEFAULT 0,
total_views BIGINT,
total_engagement BIGINT,
doc_url TEXT,
doc_title VARCHAR(200),
summary_chars INTEGER,
status VARCHAR(32),
created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
CONSTRAINT uniq_monthly_style_month UNIQUE (style_id, month_start)
);
CREATE INDEX IF NOT EXISTS idx_cmt_monthly_summary_month ON cmt_monthly_summary (month_start);
CREATE INDEX IF NOT EXISTS idx_cmt_monthly_summary_style ON cmt_monthly_summary (style_id);
COMMENT ON TABLE cmt_monthly_summary IS '月笔记汇总表(款式+月维度,LLM 横向对比分析报告)';
COMMENT ON COLUMN cmt_monthly_summary.time_label IS '月时间范围标签 如 2026-06-01~06-30';
COMMENT ON COLUMN cmt_monthly_summary.doc_url IS '飞书文档链接(月度汇总报告)';
-- ============================================================
-- 达人月度分析报告表 (每月1号自动生成)
-- ============================================================
CREATE TABLE IF NOT EXISTS cmt_creator_report (
id BIGSERIAL PRIMARY KEY,
month_start DATE NOT NULL,
month_end DATE NOT NULL,
time_label VARCHAR(32) NOT NULL,
total_cooperations INTEGER DEFAULT 0,
total_creators INTEGER DEFAULT 0,
total_styles INTEGER DEFAULT 0,
total_cost NUMERIC(14,2),
total_exposure BIGINT,
total_engagement BIGINT,
avg_cpe NUMERIC(10,2),
level_distribution JSONB DEFAULT '{}',
platform_distribution JSONB DEFAULT '{}',
score_distribution JSONB DEFAULT '{}',
doc_url TEXT,
doc_title VARCHAR(200),
report_chars INTEGER,
status VARCHAR(32),
created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
CONSTRAINT uniq_creator_report_month UNIQUE (month_start)
);
CREATE INDEX IF NOT EXISTS idx_cmt_creator_report_month ON cmt_creator_report (month_start);
COMMENT ON TABLE cmt_creator_report IS '达人月度分析报告表(每月1号自动生成,全量达人筛选与报价分析)';
COMMENT ON COLUMN cmt_creator_report.time_label IS '月时间范围标签 如 2026-06-01~06-30';
COMMENT ON COLUMN cmt_creator_report.level_distribution IS '层级分布快照 JSON {"头部":N,"腰部":N,...}';
COMMENT ON COLUMN cmt_creator_report.platform_distribution IS '平台分布快照 JSON {"douyin":N,...}';
COMMENT ON COLUMN cmt_creator_report.score_distribution IS '评分分布快照 JSON {"S":N,"A":N,...}';
COMMENT ON COLUMN cmt_creator_report.doc_url IS '飞书文档链接(达人分析报告)';
COMMENT ON COLUMN cmt_creator_report.status IS 'ok/doc_failed/exception';
CREATE INDEX IF NOT EXISTS idx_cmt_comments_platform ON cmt_comments (platform);
CREATE INDEX IF NOT EXISTS idx_cmt_comments_level ON cmt_comments (level);
CREATE INDEX IF NOT EXISTS idx_cmt_comments_comment_created_at ON cmt_comments (comment_created_at);
-- 自动更新 updated_at 的触发器函数
CREATE OR REPLACE FUNCTION update_updated_at_column()
RETURNS TRIGGER AS $$
BEGIN
NEW.updated_at = CURRENT_TIMESTAMP;
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
-- 为需要自动更新 updated_at 的表创建触发器
DO $$
DECLARE
tbl TEXT;
BEGIN
FOR tbl IN SELECT unnest(ARRAY['cmt_styles', 'cmt_creators', 'cmt_style_creators', 'cmt_notes', 'cmt_comments', 'cmt_cooperations', 'cmt_weekly_summary', 'cmt_monthly_summary', 'cmt_creator_report'])
LOOP
IF NOT EXISTS (
SELECT 1 FROM pg_trigger WHERE tgname = 'trg_' || tbl || '_updated_at'
) THEN
EXECUTE format('CREATE TRIGGER trg_%I_updated_at BEFORE UPDATE ON %I FOR EACH ROW EXECUTE FUNCTION update_updated_at_column()', tbl, tbl);
END IF;
END LOOP;
END;
$$;
src/gyxx_flow/modules/content_marketing/runtime/data/tools/style_analyzer.py
+362
View File
@@ -0,0 +1,362 @@
"""Style-level rollup analyzer for the Feishu doc backfill.
Aggregates metrics + keyword distributions across all notes of one style,
then calls Hermes for a cross-note synthesis analysis that becomes the
"💼 款式整体分析" section in the doc.
Sibling of analyze_note.py — same import surface (call_hermes_analyzer,
ANALYSIS_SYSTEM_PROMPT, code_filter_comments) but operates at style scope.
"""
import re
import sys
import time
from pathlib import Path
from typing import Any
from gyxx_flow.modules.content_marketing.runtime.runtime_paths import PATHS
# Path setup
_TOOLS_DIR = PATHS.tools_root
_PROJECT_ROOT = PATHS.module_root
from gyxx_flow.modules.content_marketing.runtime.data.tools.analyze_comments import (
ANALYSIS_SYSTEM_PROMPT,
call_hermes_analyzer,
)
CATEGORY_LABELS = {
"purchase_intent": "购买意向",
"positive_feature": "正向功能反馈",
"negative_feature": "负向功能反馈",
"user_scenario": "用户场景",
"aesthetic": "审美与风格",
"brand_emotion": "品牌情感",
}
# 5-section deep prompt at style scope (mirrors SINGLE_NOTE_DEEP_PROMPT_TEMPLATE
# but across all notes of one style).
STYLE_ROLLUP_DEEP_PROMPT = """你是一位品牌内容分析师。本次任务是把同一款式的所有笔记的标题、平台、点赞/收藏/评论/分享数据和有效评论内容，汇总成一份**款式整体分析报告**，用于投放团队对这款产品的内容策略做一次通盘复盘。
输出结构（严格 5 大节，总长 800-1200 字）：
### 一、执行摘要（不超过 150 字）
用一段话点明这款产品的整体定位和投放现状：覆盖多少篇笔记、整体互动表现、主要用户反馈倾向、是否存在明显短板。
### 二、数据表现
- 该款式总笔记数 / 有效评论总数
- 总点赞 / 总收藏 / 总评论 / 总分享（如曝光量汇总可用则一并列出）
- 综合互动率（基于汇总数据）
- 跨笔记表现差异：高/低表现笔记的标题特征（如"通勤""扫街""旅行"等场景标签）
### 三、受众洞察
基于所有有效评论的合并视角：
- 正向反馈汇总（被夸的核心卖点 / 场景 / 审美点）
- 负向问题/风险点（被吐槽的尺寸 / 材质 / 价格 / 功能）
- 购买意向强弱（直接询问价格/链接 vs 围观闲聊）
- 典型使用场景（被反复提及的目标人群和场景）
- 跨平台差异（如果 bilibili / douyin / xiaohongshu 反馈侧重不同，请分别说明）
### 四、内容策略
基于跨笔记样本，提炼以下三类规律：
- 哪种标题/封面/切入点最受欢迎
- 哪种内容结构（测评 / 街拍 / Vlog / 教程）种草效率最高
- 评论区的高频问题/梗，可作为后续内容选题/卖点提炼的素材
### 五、下一步（按 P0/P1/P2 排序，最多 5 条）
- P0（立即）：【问题】→【行动】→【责任方】
- P1（本周）：...
- P2（下月）：...（如无可省略）
## 硬性约束
1. 引用评论必须直接引述原文（用「」包裹），禁止使用"#N""第N条"等编号引用。
2. 禁止"根据数据/数据显示/数据来源/通过 XX 可知"等出处类表述。
3. 若有效评论 < 30 条，必须在执行摘要后注明"样本量较小，结论仅供参考"。
4. P0/P1/P2 必须明确标注【问题】→【行动】→【责任方】三段式。
5. 若款式涉及多品牌联合推广，请仅聚焦我方品牌/产品相关反馈。
6. 不要重复列出指标定义，专注解读与行动。
"""
def _to_number(value: Any) -> float:
try:
if value is None or value == "":
return 0.0
return float(value)
except (TypeError, ValueError):
return 0.0
def safe_div(num: Any, denom: Any) -> float:
try:
n = float(num)
d = float(denom)
if d == 0:
return 0.0
return n / d
except (TypeError, ValueError):
return 0.0
def log(msg: str) -> None:
print(f"[{time.strftime('%H:%M:%S')}] {msg}", flush=True)
# ---------------------------------------------------------------------------
# Aggregation
# ---------------------------------------------------------------------------
def aggregate_style_metrics(notes: list[dict[str, Any]]) -> dict[str, Any]:
"""Sum raw engagement counts across all notes of a style.
Returns:
{
"note_count": int,
"total_likes": int, "total_favorites": int, "total_comments": int,
"total_shares": int, "total_views": int,
"engagement_rate": float, # (likes+comments+favorites+shares) / views
"platform_breakdown": {"bilibili": n, "douyin": n, "xiaohongshu": n},
"creator_breakdown": {creator_name: n}, # top creators by note count
"notes_meta": [{"id", "platform", "title", "like_count", "comment_count"}, ...]
}
"""
totals = {"likes": 0.0, "favorites": 0.0, "comments": 0.0, "shares": 0.0, "views": 0.0}
platforms = {"bilibili": 0, "douyin": 0, "xiaohongshu": 0}
creators: dict[str, int] = {}
notes_meta: list[dict[str, Any]] = []
for n in notes:
likes = _to_number(n.get("like_count"))
favs = _to_number(n.get("favorite_count"))
cmts = _to_number(n.get("comment_count")) or _to_number(n.get("comment_count_total"))
shares = _to_number(n.get("share_count"))
views = _to_number(n.get("view_count"))
totals["likes"] += likes
totals["favorites"] += favs
totals["comments"] += cmts
totals["shares"] += shares
totals["views"] += views
plat = n.get("platform") or "unknown"
platforms[plat] = platforms.get(plat, 0) + 1
creator = n.get("creator_name") or "未指定"
creators[creator] = creators.get(creator, 0) + 1
notes_meta.append({
"id": n.get("id"),
"platform": plat,
"title": (n.get("title") or "")[:60],
"like_count": int(likes) if likes else 0,
"comment_count": int(cmts) if cmts else 0,
"creator_name": creator,
})
engagement = safe_div(
totals["likes"] + totals["comments"] + totals["favorites"] + totals["shares"],
totals["views"],
)
# Top creators by note count
top_creators = dict(sorted(creators.items(), key=lambda kv: -kv[1])[:5])
return {
"note_count": len(notes),
"total_likes": int(totals["likes"]),
"total_favorites": int(totals["favorites"]),
"total_comments": int(totals["comments"]),
"total_shares": int(totals["shares"]),
"total_views": int(totals["views"]),
"engagement_rate": engagement,
"platform_breakdown": platforms,
"creator_breakdown": top_creators,
"notes_meta": notes_meta,
}
def merge_style_distribution(per_note_distributions: list[dict[str, Any]]) -> dict[str, Any]:
"""Element-wise sum of overall + relevant dicts across notes."""
overall: dict[str, int] = {k: 0 for k in CATEGORY_LABELS}
relevant: dict[str, int] = {k: 0 for k in CATEGORY_LABELS}
relevant_total = 0
for d in per_note_distributions:
if not isinstance(d, dict):
continue
for k, v in (d.get("overall") or {}).items():
overall[k] = overall.get(k, 0) + int(v or 0)
for k, v in (d.get("relevant") or {}).items():
relevant[k] = relevant.get(k, 0) + int(v or 0)
relevant_total += int(d.get("relevant_total") or 0)
return {"overall": overall, "relevant": relevant, "relevant_total": relevant_total}
def _pct(v: float) -> str:
return f"{v * 100:.2f}%" if v else "—"
def _num(v: float) -> str:
try:
return f"{int(v):,}"
except (TypeError, ValueError):
return "—"
def _format_distribution_md(d: dict[str, Any], total_filtered: int) -> str:
"""Mirror analyze_note._keyword_distribution_markdown."""
overall = d.get("overall", {})
relevant = d.get("relevant", {})
relevant_total = d.get("relevant_total", 0)
if not overall:
return "（无关键词分布数据）"
def fmt(m: dict[str, int]) -> str:
return "\n".join(f"- {CATEGORY_LABELS[k]}：{m.get(k, 0)} 条" for k in CATEGORY_LABELS)
return "\n".join([
f"### 全评论（{total_filtered} 条）",
fmt(overall),
"",
f"### 我方产品相关（{relevant_total} 条）",
fmt(relevant) if relevant_total else "_（无评论明确提及我方产品/品牌）_",
])
def _format_metrics_md(agg: dict[str, Any]) -> str:
plat = agg["platform_breakdown"]
plat_str = " / ".join(f"{k} {v}" for k, v in plat.items() if v) or "—"
creator_str = " / ".join(f"{k} {v}" for k, v in agg["creator_breakdown"].items()) or "—"
return "\n".join([
"| 维度 | 指标 | 数值 |",
"| --- | --- | --- |",
f"| 规模 | 笔记总数 | {agg['note_count']} |",
f"| 规模 | 平台分布 | {plat_str} |",
f"| 规模 | 主要达人 | {creator_str} |",
f"| 基础 | 总点赞 | {_num(agg['total_likes'])} |",
f"| 基础 | 总收藏 | {_num(agg['total_favorites'])} |",
f"| 基础 | 总评论 | {_num(agg['total_comments'])} |",
f"| 基础 | 总分享 | {_num(agg['total_shares'])} |",
f"| 传播 | 总曝光量 | {_num(agg['total_views']) if agg['total_views'] else '未采集'} |",
f"| 互动率 | 综合互动率 | {_pct(agg['engagement_rate'])} |",
])
# ---------------------------------------------------------------------------
# Prompt building
# ---------------------------------------------------------------------------
def sample_representative_comments(
per_note_comments: list[list[str]],
per_note_meta: list[dict[str, Any]],
*,
max_total: int = 30,
per_note_cap: int = 2,
) -> list[str]:
"""Pick up to `max_total` filtered comments across notes.
Tries to take `per_note_cap` from each note, prioritizing longer comments
(proxy for substance). If still under `max_total`, fills with remaining
longer comments.
"""
all_comments: list[tuple[str, dict[str, Any]]] = []
for comments, meta in zip(per_note_comments, per_note_meta):
# Sort by length desc; take up to per_note_cap
sorted_c = sorted(comments, key=len, reverse=True)[:per_note_cap]
for c in sorted_c:
all_comments.append((c, meta))
# Final sort: longer first, then by note id asc for stability
all_comments.sort(key=lambda cm: (-len(cm[0]), cm[1].get("id") or 0))
return [c for c, _ in all_comments[:max_total]]
def build_style_rollup_prompt(
style_name: str,
brand: str,
aggregated: dict[str, Any],
merged_distribution: dict[str, Any],
sample_comments: list[str],
note_summaries: list[str],
) -> str:
"""Build user_content for the style-level LLM call.
`note_summaries` are 1-2 line "title | platform | likes | comments" snippets
to give the LLM a sense of which notes exist.
"""
target_label = brand or style_name
lines: list[str] = [
f"本款式「{style_name}」共采集 {aggregated['note_count']} 篇笔记，跨 {sum(1 for v in aggregated['platform_breakdown'].values() if v)} 个平台。",
f"以下是基于这 {aggregated['note_count']} 篇笔记的合并数据 + 代表性评论样本，请输出款式整体分析报告。",
"",
"## 款式汇总数据",
_format_metrics_md(aggregated),
"",
"## 关键词分布（合并所有笔记）",
_format_distribution_md(
merged_distribution,
sum((m or {}).get("overall", {}).get(k, 0)
for m in [merged_distribution]
for k in CATEGORY_LABELS),
),
"",
f"## {aggregated['note_count']} 篇笔记概览（标题 | 平台 | 点赞 | 评论）",
]
for meta in aggregated["notes_meta"]:
lines.append(
f"- #{meta['id']} | {meta['platform']} | 点赞 {meta['like_count']} | 评论 {meta['comment_count']} | {meta['title']}"
)
if sample_comments:
lines.extend([
"",
f"## 代表性评论样本（共 {len(sample_comments)} 条，来自多篇笔记）",
])
for c in sample_comments:
lines.append(f"- {c}")
return "\n".join(lines)
# ---------------------------------------------------------------------------
# LLM call
# ---------------------------------------------------------------------------
def analyze_style(
style_name: str,
brand: str,
aggregated: dict[str, Any],
merged_distribution: dict[str, Any],
sample_comments: list[str],
note_summaries: list[str],
*,
doc_text_limit: int = 6000,
) -> str:
"""Call Hermes for the style-level rollup analysis."""
if not aggregated["note_count"]:
return "（该款式暂无笔记可分析）"
log(
f"调用 Hermes 生成款式整体分析（{style_name}，{aggregated['note_count']} 篇笔记，"
f"{aggregated['total_comments']} 条总评论）..."
)
user_content = build_style_rollup_prompt(
style_name, brand, aggregated, merged_distribution, sample_comments, note_summaries,
)
target_term = brand or style_name
brand_block = ""
if target_term:
brand_block = f"""
【实体隔离】
本次分析的目标：品牌「{brand or '未指定'}」、产品「{style_name}」。
本款式可能涉及多品牌联合推广，请严格将分析对象限定为我方品牌/产品：
- 仅将「{target_term}」相关的信息计入种草指标
- 评论中涉及的其他品牌或产品，仅作为场景搭配或联合推广的陪衬
"""
system_prompt = ANALYSIS_SYSTEM_PROMPT + brand_block + "\n\n" + STYLE_ROLLUP_DEEP_PROMPT
result = call_hermes_analyzer(system_prompt, user_content)
if len(result) > doc_text_limit:
result = result[:doc_text_limit] + "\n\n...（内容过长，已截断）"
return result

Some files were not shown because too many files have changed in this diff Show More