feat: collect OneTalk product messages

This commit is contained in:
YBF
2026-09-14 13:04:23 +08:00
parent c9468f91d3
commit 4042c34d4b
12 changed files with 1219 additions and 5 deletions
@@ -373,3 +373,70 @@ normalizeOneTalkMessageId("message-1.PNM"); // "message-1.PNM"
### 2026-09-06 实时消息分类回归
生产 0.8.15 把 SDK history 的 `message.type === 1` 前置校验用于 live。实际 live 的 `lastMessage.message` 没有 type,导致 `invalid_observation` 与发送 `send_state_lost`。外层 `type` 是会话判别,不能复制进 message 冒充消息类别。history 继续严格校验;live 不要求不存在的字段,但已有未知类别、完整身份、内容白名单校验继续生效。内部 `upstreamType: 1` 是通过来源适配与内容解码后的业务分类标记,不证明 live 原文含 type。
## Scenario: OneTalk 商品详情链接的安全归一化
### 1. Scope / Trigger
- Triggerraw 文本或 SDK flat history 的 `msgType=101/subType=1` 可能是已验证的 OneTalk 商品详情链接,且 URL query 含 `chatToken` 或加密跳转信息。
- Scope`@trade-message-center/onetalk-contract``normalizeOneTalkProductUrl()` 是唯一 URL 解析/验证 ownerMAIN world 在跨 bridge 前调用它。页面 bridge、Service Worker、Bright 和 Mind 只处理归一化的 product content。
### 2. Signatures
```ts
normalizeOneTalkProductUrl(value: unknown): {
sourceUrl: string;
productId: string;
} | null;
type OneTalkProductContent = {
version: 1;
kind: "product";
sourceUrl: string;
productId: string;
};
```
### 3. Contracts
- 仅接受未经 trim 的 `https://chinese.alibaba.com/product-detail/<slug>-<nonzero decimal productId>.html``slug` 只含 ASCII 字母、数字和 `-`,路径、host、协议、大小写和 ID 都不能由调用方自行宽松解释。
- raw URL 可携带 query;成功时只返回 `origin + pathname` 的 query-free `sourceUrl`,绝不返回、记录或跨 MAIN world 传递 raw URL、`chatToken`、加密 target 或 SDK `originalData`
- 已归一化 product content 再调用同一函数校验,且其 `sourceUrl``productId` 必须与结果完全相等;因而已持久化/transport 的 `sourceUrl` 不得带 query、fragment、credentials、port 或额外字段。
-`new URL()` 之前必须检验原始 authority 和 pathname。WHATWG 会静默消除默认/空 port、空 fragment 和 dot segment;不得在已规范化的 `url.pathname` 上判定原始输入合法。
### 4. Validation & Error Matrix
| 输入 | 行为 |
| --- | --- |
| 验证 host/path 的 raw URL,带 query | 产生严格的 query-free `product` 内容 |
| 普通文本、错误 host/path/slug、前导零 ID | 继续既有 `text`,不是 anomaly |
| 显式或空 port、userinfo、空/非空 fragment、空白 | 不归类为 product |
| literal/percent-encoded dot segment 或 percent-encoded ID | 不归类为 product;不得利用 URL 自动规范化放行 |
| product `sourceUrl` 含 query、ID 不一致或含 extra key | shared decoder 拒绝;不得写库/ACK/publish |
### 5. Good / Base / Bad Cases
- GoodMAIN decoder 将 `...?chatToken=secret` 转为只含安全 `sourceUrl``productId` 的对象,再走既有 observation 链路。
- Base:不匹配的 HTTPS URL 保持文本内容,图片、文件、名片、询盘和订单的分类不变。
- Bad:在 content decoder、server 或 read projection 各写一份 regex/URL parser,或把 query-stripped 字段错误当成 raw URL 原样转发。
### 6. Tests Required
- contract 覆盖 raw query 清除、exact-shape frame、ID 一致性与错误 host/path/slug/extra keys。
- 回归覆盖默认/空 port、userinfo、空/非空 fragment、literal/encoded dot segment、percent-encoded product ID,防止 WHATWG 自动规范化绕过边界。
- MAIN raw 与 flat history 同时断言输出 product 且序列化结果不含 query credential;普通 URL 继续为 text。
- 服务端读取断言 history 与 `message.created` 投影相同,手工 query-bearing JSONB 在读取边界失败;迁移只放宽 kind CHECK,保留既有 kind 和名片 marker。
### 7. Wrong vs Correct
```ts
// Wrong: URL 已丢失原始 dot segment / default port 语法,无法再安全判断。
const url = new URL(rawUrl);
if (url.hostname === "chinese.alibaba.com") return { sourceUrl: url.href };
// Correct: 先验证 raw authority/path,再解析并只导出安全派生字段。
const product = normalizeOneTalkProductUrl(rawUrl);
if (product !== null) {
return { version: 1, kind: "product", ...product };
}
```
@@ -7,13 +7,13 @@
当前已建立 OneTalk Bright 事实存储 schema,定义位于
[`apps/server/src/database/schema/onetalk.ts`](../../../../apps/server/src/database/schema/onetalk.ts)
- `onetalk_message`:页面事实消息。`channel_account_id + conversation_id + message_id` 复合主键负责幂等;收件和确认发件通过 `direction` 区分。`content` 是唯一内容事实,只承载 shared contract 的 versioned `text | image | file | business_card | inquiry | order` JSON其中 `business_card` 严格只保存 `{ version: 1, kind: "business_card" }` marker,不保存客户资料、包含认证信息的完整 OneTalk envelope、业务卡 raw 正文或 SDK payload。
- `onetalk_message`:页面事实消息。`channel_account_id + conversation_id + message_id` 复合主键负责幂等;收件和确认发件通过 `direction` 区分。`content` 是唯一内容事实,只承载 shared contract 的 versioned `text | image | file | business_card | inquiry | order | product` JSON`product` 仅保存 query-free 的商品详情 `sourceUrl` 与路径提取的 `productId`其中 `business_card` 严格只保存 `{ version: 1, kind: "business_card" }` marker,不保存客户资料、包含认证信息的完整 OneTalk envelope、业务卡 raw 正文或 SDK payload。
- `onetalk_conversation`:插件发现的技术会话和共享同步锚点。`channel_account_id + conversation_id` 复合主键,不按 binding 或设备复制;`conversation_kind` 只接受显式 `direct`,未知历史会话保持 `null``sync_phase``sync_result``latest_message_id``history_complete` 表达同步进度及锚点状态,并允许零消息会话。
- `onetalk_contact_profile`Bright 当前联系人资料事实。`channel_account_id + conversation_id` 复合主键,不建立到技术会话表的外键;资料字段允许显式 `null`,只有严格较新的 `observed_at_ms` 才能覆盖整行。它是名片读取 view 的唯一客户资料来源,不是消息事实的嵌入列。
- `onetalk_message_anomaly`:缺字段、协议和同步异常的独立诊断事实。`fingerprint` 仅用于诊断合并;`payload` 必须由写入边界清洗,不能被消息读取、发送或锚点流程消费。
- `onetalk_buyer_fact`:买家标签、特征与联系细节的当前事实,独立于联系人资料。`channel_account_id + conversation_id` 复合主键;`tags_state`/`features_state`/`contact_details_state` 各来源独立表达 confirmed/failedconfirmed 来源整体替换、failed 只更新自身审计;`fact_fingerprint` 必须由最终合并后的集合计算。
生成的初始迁移为 `apps/server/drizzle/0000_rapid_winter_soldier.sql`,其中显式维护 PostgreSQL 表/字段 `COMMENT ON` 备注(Drizzle 当前版本不会从 TypeScript 注释自动生成数据库备注)。未确认发送不进入任何一张表,普通运行路径不提供物理删除;profile 当前行由读取服务另行受限读取后在内存组合,不复制进 conversation 或 message。媒体切换 migration `0005_young_squadron_supreme` 是一次性开发数据重置:仅 `DELETE` 本仓库拥有的 OneTalk message/anomaly/profile/conversation 事实,再删除 `text/content_type` 并为 `content` 加 v1 kind CHECK;不触及授权、binding 或其它渠道。后续增量迁移:`0006_onetalk_buyer_facts` 新增 `onetalk_buyer_fact` 表;`0007_center_contact_time``onetalk_conversation` 增加规范会话活动时间 `last_contact_time_ms` 与白名单 `message_preview``0008_lucky_captain_universe` 为 buyer fact 增加 contact details 白名单列;`0009_remove_image_dimensions` 移除 image content 的 `width`/`height``0010_wooden_naoko` 将 content v1 CHECK 扩展到六类 kind`0011_mushy_baron_strucker` 在建立名片 exact CHECK 前,将已有带客户资料字段的 `business_card` content 归一化为 marker。
生成的初始迁移为 `apps/server/drizzle/0000_rapid_winter_soldier.sql`,其中显式维护 PostgreSQL 表/字段 `COMMENT ON` 备注(Drizzle 当前版本不会从 TypeScript 注释自动生成数据库备注)。未确认发送不进入任何一张表,普通运行路径不提供物理删除;profile 当前行由读取服务另行受限读取后在内存组合,不复制进 conversation 或 message。媒体切换 migration `0005_young_squadron_supreme` 是一次性开发数据重置:仅 `DELETE` 本仓库拥有的 OneTalk message/anomaly/profile/conversation 事实,再删除 `text/content_type` 并为 `content` 加 v1 kind CHECK;不触及授权、binding 或其它渠道。后续增量迁移:`0006_onetalk_buyer_facts` 新增 `onetalk_buyer_fact` 表;`0007_center_contact_time``onetalk_conversation` 增加规范会话活动时间 `last_contact_time_ms` 与白名单 `message_preview``0008_lucky_captain_universe` 为 buyer fact 增加 contact details 白名单列;`0009_remove_image_dimensions` 移除 image content 的 `width`/`height``0010_wooden_naoko` 将 content v1 CHECK 扩展到六类 kind`0011_mushy_baron_strucker` 在建立名片 exact CHECK 前,将已有带客户资料字段的 `business_card` content 归一化为 marker`0013_nifty_thunderbird` 只重建同一 CHECK 以加入 `product`,不回填、更新或删除历史事实
## Scenario: Schema 注释与 PostgreSQL 备注
@@ -213,7 +213,7 @@ repository.updateSyncState(context, update, conversationId, conversationKind) ->
- 重复消息返回 `duplicate`,不得覆盖首次事实或再次触发外部事件;允许只更新 `last_observed_at`
- 跨 workspace 收到相同 `channel_account_id + conversation_id + message_id` 时,必须沿用同一条已存在事实:返回 `duplicate`,保留首次写入的 `workspace_id``mind_user_id``binding``device_id`,不得因后续 workspace 改写来源上下文。
- 消息读取按 `channel_account_id + conversation_id` 读取共享事实;workspace 隔离由 Mind 授权 scope 负责,不能把 `workspace_id` 加入消息事实主键或作为第二套消息副本维度。
- `content jsonb` 必须是对象,且 `version=1``kind in (text,image,file,business_card,inquiry,order)`;应用边界再用 shared exact decoder 验证完整字段。`kind=business_card` 时 JSON 必须精确等于 `{ "version": 1, "kind": "business_card" }`,不得把 `contactName` 等 view 字段写回数据库。不得保留顶层 `text``content_type`、raw content、`params``sign`、完整 `contact` 或平行投影列。
- `content jsonb` 必须是对象,且 `version=1``kind in (text,image,file,business_card,inquiry,order,product)`;应用边界再用 shared exact decoder 验证完整字段。`kind=business_card` 时 JSON 必须精确等于 `{ "version": 1, "kind": "business_card" }``kind=product` 时仅允许共享合同校验过的 query-free canonical `sourceUrl` 与相等的 `productId`,不得把 `contactName``chatToken`、原始 URL 或商品卡 DTO 写回数据库。不得保留顶层 `text``content_type`、raw content、`params``sign`、完整 `contact` 或平行投影列。
- 名片 view 只能由读取服务先按 `channel_account_id + conversation_id` 读取 `onetalk_contact_profile`,再以内存方式投影四个批准字段;profile 缺失返回 marker,单字段缺失返回 `null`,不得 JOIN 其他账号/会话,也不得回退到登录人资料。
- `content` 或文本相同本身不构成重复;只要 `message_id``conversation_id` 不同,就按新的 OneTalk 事实入库。
- `discoverConversation` 只按账号/会话幂等 upsert,不清空已有消息计数、同步结果或锚点。
@@ -2,6 +2,7 @@
import {
isOneTalkMessageContent,
normalizeOneTalkProductUrl,
ONETALK_CONTENT_VERSION,
ONETALK_MAX_MEDIA_SIZE_BYTES,
type OneTalkMessageContent,
@@ -215,6 +216,17 @@ const optionalUrl = (
const normalizeText = (content: Record<string, unknown>): OneTalkRawContentDecodeResult => {
const text = isObjectRecord(content.text) ? content.text.content : undefined;
const product = normalizeOneTalkProductUrl(text);
if (product !== null) {
const normalized = {
version: ONETALK_CONTENT_VERSION,
kind: "product" as const,
...product,
};
return isOneTalkMessageContent(normalized)
? { status: "decoded", content: normalized }
: { status: "ignored" };
}
const normalized = { version: ONETALK_CONTENT_VERSION, kind: "text" as const, text };
return isOneTalkMessageContent(normalized)
? { status: "decoded", content: normalized }
@@ -198,6 +198,46 @@ test("normalizes exact SDK flat history media through the shared media contract"
);
});
test("normalizes raw and SDK flat product URLs without leaking source queries beyond MAIN world", () => {
const sourceUrl =
"https://chinese.alibaba.com/product-detail/HAGO-Men-s-Breathable-Mid-Rise-1601456609478.html";
const rawUrl = `${sourceUrl}?chatToken=secret-token&targetEncrypt=encrypted-target`;
const expected = {
status: "decoded",
content: {
version: 1,
kind: "product",
sourceUrl,
productId: "1601456609478",
},
};
const raw = decodeOneTalkRawContent({ contentType: 1, text: { content: rawUrl } });
const flat = decodeOneTalkMessageContent({
source: "sdk_flat_history",
msgType: 101,
subType: 1,
originalData: { text: rawUrl },
});
assert.deepEqual(raw, expected);
assert.deepEqual(flat, expected);
assert.equal(JSON.stringify(raw).includes("chatToken"), false);
assert.equal(JSON.stringify(flat).includes("encrypted-target"), false);
for (const text of [
"https://example.test/product-detail/HAGO-1601456609478.html",
"https://chinese.alibaba.com/product-detail/HAGO-01601456609478.html",
"https://chinese.alibaba.com/product-detail/./HAGO-1601456609478.html",
"https://chinese.alibaba.com/product-detail/HAGO/../HAGO-1601456609478.html",
"https://chinese.alibaba.com/product-detail/%2e/HAGO-1601456609478.html",
]) {
assert.deepEqual(decodeOneTalkRawContent({ contentType: 1, text: { content: text } }), {
status: "decoded",
content: { version: 1, kind: "text", text },
});
}
});
test("classifies only complete flat-history business-card tuples without collecting profile data", () => {
const result = decodeOneTalkMessageContent(flatBusinessCard());
assert.deepEqual(result, {
@@ -0,0 +1,2 @@
ALTER TABLE "onetalk_message" DROP CONSTRAINT "onetalk_message_content_v1_chk";--> statement-breakpoint
ALTER TABLE "onetalk_message" ADD CONSTRAINT "onetalk_message_content_v1_chk" CHECK (jsonb_typeof("onetalk_message"."content") = 'object' and ("onetalk_message"."content" ->> 'version') = '1' and ("onetalk_message"."content" ->> 'kind') in ('text', 'image', 'file', 'business_card', 'inquiry', 'order', 'product') and (("onetalk_message"."content" ->> 'kind') <> 'business_card' or "onetalk_message"."content" = '{"version":1,"kind":"business_card"}'::jsonb));
+857
View File
@@ -0,0 +1,857 @@
{
"id": "5bdeaafd-a264-4693-9a5d-72fcc245c5d3",
"prevId": "bf5d1038-fa8c-416d-8142-898614bd4bf3",
"version": "7",
"dialect": "postgresql",
"tables": {
"public.onetalk_buyer_fact": {
"name": "onetalk_buyer_fact",
"schema": "",
"columns": {
"channel_account_id": {
"name": "channel_account_id",
"type": "text",
"primaryKey": false,
"notNull": true
},
"conversation_id": {
"name": "conversation_id",
"type": "text",
"primaryKey": false,
"notNull": true
},
"buyer_tags": {
"name": "buyer_tags",
"type": "text[]",
"primaryKey": false,
"notNull": false
},
"buyer_features": {
"name": "buyer_features",
"type": "text[]",
"primaryKey": false,
"notNull": false
},
"email": {
"name": "email",
"type": "text",
"primaryKey": false,
"notNull": false
},
"registration_date": {
"name": "registration_date",
"type": "text",
"primaryKey": false,
"notNull": false
},
"company_website": {
"name": "company_website",
"type": "text",
"primaryKey": false,
"notNull": false
},
"tags_state": {
"name": "tags_state",
"type": "onetalk_buyer_fact_source_state",
"typeSchema": "public",
"primaryKey": false,
"notNull": true
},
"features_state": {
"name": "features_state",
"type": "onetalk_buyer_fact_source_state",
"typeSchema": "public",
"primaryKey": false,
"notNull": true
},
"contact_details_state": {
"name": "contact_details_state",
"type": "onetalk_buyer_fact_source_state",
"typeSchema": "public",
"primaryKey": false,
"notNull": false
},
"tags_last_attempted_at": {
"name": "tags_last_attempted_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true
},
"features_last_attempted_at": {
"name": "features_last_attempted_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true
},
"contact_details_last_attempted_at": {
"name": "contact_details_last_attempted_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"tags_confirmed_at": {
"name": "tags_confirmed_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"features_confirmed_at": {
"name": "features_confirmed_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"contact_details_confirmed_at": {
"name": "contact_details_confirmed_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"tags_error_code": {
"name": "tags_error_code",
"type": "text",
"primaryKey": false,
"notNull": false
},
"features_error_code": {
"name": "features_error_code",
"type": "text",
"primaryKey": false,
"notNull": false
},
"contact_details_error_code": {
"name": "contact_details_error_code",
"type": "text",
"primaryKey": false,
"notNull": false
},
"fact_fingerprint": {
"name": "fact_fingerprint",
"type": "text",
"primaryKey": false,
"notNull": true
},
"received_at": {
"name": "received_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {},
"foreignKeys": {},
"compositePrimaryKeys": {
"onetalk_buyer_fact_channel_account_id_conversation_id_pk": {
"name": "onetalk_buyer_fact_channel_account_id_conversation_id_pk",
"columns": ["channel_account_id", "conversation_id"]
}
},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.onetalk_contact_profile": {
"name": "onetalk_contact_profile",
"schema": "",
"columns": {
"channel_account_id": {
"name": "channel_account_id",
"type": "text",
"primaryKey": false,
"notNull": true
},
"conversation_id": {
"name": "conversation_id",
"type": "text",
"primaryKey": false,
"notNull": true
},
"ali_id": {
"name": "ali_id",
"type": "text",
"primaryKey": false,
"notNull": true
},
"account_id": {
"name": "account_id",
"type": "text",
"primaryKey": false,
"notNull": false
},
"login_id": {
"name": "login_id",
"type": "text",
"primaryKey": false,
"notNull": false
},
"name": {
"name": "name",
"type": "text",
"primaryKey": false,
"notNull": false
},
"company_name": {
"name": "company_name",
"type": "text",
"primaryKey": false,
"notNull": false
},
"country_code": {
"name": "country_code",
"type": "text",
"primaryKey": false,
"notNull": false
},
"current_time_zone": {
"name": "current_time_zone",
"type": "double precision",
"primaryKey": false,
"notNull": false
},
"service_type": {
"name": "service_type",
"type": "text",
"primaryKey": false,
"notNull": false
},
"avatar_url": {
"name": "avatar_url",
"type": "text",
"primaryKey": false,
"notNull": false
},
"observed_at_ms": {
"name": "observed_at_ms",
"type": "bigint",
"primaryKey": false,
"notNull": true
},
"profile_fingerprint": {
"name": "profile_fingerprint",
"type": "text",
"primaryKey": false,
"notNull": true
},
"observation_status": {
"name": "observation_status",
"type": "onetalk_contact_profile_observation_status",
"typeSchema": "public",
"primaryKey": false,
"notNull": true
},
"received_at": {
"name": "received_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {},
"foreignKeys": {},
"compositePrimaryKeys": {
"onetalk_contact_profile_channel_account_id_conversation_id_pk": {
"name": "onetalk_contact_profile_channel_account_id_conversation_id_pk",
"columns": ["channel_account_id", "conversation_id"]
}
},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.onetalk_conversation": {
"name": "onetalk_conversation",
"schema": "",
"columns": {
"channel_account_id": {
"name": "channel_account_id",
"type": "text",
"primaryKey": false,
"notNull": true
},
"conversation_id": {
"name": "conversation_id",
"type": "text",
"primaryKey": false,
"notNull": true
},
"participant_ids": {
"name": "participant_ids",
"type": "text[]",
"primaryKey": false,
"notNull": false
},
"biz_type": {
"name": "biz_type",
"type": "integer",
"primaryKey": false,
"notNull": false
},
"conversation_type": {
"name": "conversation_type",
"type": "integer",
"primaryKey": false,
"notNull": false
},
"conversation_kind": {
"name": "conversation_kind",
"type": "text",
"primaryKey": false,
"notNull": false
},
"join_time_ms": {
"name": "join_time_ms",
"type": "bigint",
"primaryKey": false,
"notNull": false
},
"modify_time_ms": {
"name": "modify_time_ms",
"type": "bigint",
"primaryKey": false,
"notNull": false
},
"last_message_at_ms": {
"name": "last_message_at_ms",
"type": "bigint",
"primaryKey": false,
"notNull": false
},
"last_contact_time_ms": {
"name": "last_contact_time_ms",
"type": "bigint",
"primaryKey": false,
"notNull": false
},
"message_preview": {
"name": "message_preview",
"type": "text",
"primaryKey": false,
"notNull": false
},
"sync_phase": {
"name": "sync_phase",
"type": "onetalk_sync_phase",
"typeSchema": "public",
"primaryKey": false,
"notNull": true,
"default": "'initial'"
},
"sync_result": {
"name": "sync_result",
"type": "onetalk_sync_result",
"typeSchema": "public",
"primaryKey": false,
"notNull": true,
"default": "'incomplete'"
},
"latest_message_id": {
"name": "latest_message_id",
"type": "text",
"primaryKey": false,
"notNull": false
},
"history_generation": {
"name": "history_generation",
"type": "text",
"primaryKey": false,
"notNull": true,
"default": "'initial'"
},
"history_complete": {
"name": "history_complete",
"type": "boolean",
"primaryKey": false,
"notNull": true,
"default": false
},
"message_count": {
"name": "message_count",
"type": "integer",
"primaryKey": false,
"notNull": true,
"default": 0
},
"first_discovered_at": {
"name": "first_discovered_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"last_observed_at": {
"name": "last_observed_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"anchor_updated_at": {
"name": "anchor_updated_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
}
},
"indexes": {
"onetalk_conversation_last_observed_idx": {
"name": "onetalk_conversation_last_observed_idx",
"columns": [
{
"expression": "channel_account_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "last_observed_at",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
},
"onetalk_conversation_contact_time_idx": {
"name": "onetalk_conversation_contact_time_idx",
"columns": [
{
"expression": "channel_account_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "last_contact_time_ms",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "conversation_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {},
"compositePrimaryKeys": {
"onetalk_conversation_channel_account_id_conversation_id_pk": {
"name": "onetalk_conversation_channel_account_id_conversation_id_pk",
"columns": ["channel_account_id", "conversation_id"]
}
},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.onetalk_message": {
"name": "onetalk_message",
"schema": "",
"columns": {
"channel_account_id": {
"name": "channel_account_id",
"type": "text",
"primaryKey": false,
"notNull": true
},
"conversation_id": {
"name": "conversation_id",
"type": "text",
"primaryKey": false,
"notNull": true
},
"message_id": {
"name": "message_id",
"type": "text",
"primaryKey": false,
"notNull": true
},
"sender_id": {
"name": "sender_id",
"type": "text",
"primaryKey": false,
"notNull": true
},
"binding": {
"name": "binding",
"type": "text",
"primaryKey": false,
"notNull": true
},
"mind_user_id": {
"name": "mind_user_id",
"type": "text",
"primaryKey": false,
"notNull": true
},
"workspace_id": {
"name": "workspace_id",
"type": "text",
"primaryKey": false,
"notNull": true
},
"device_id": {
"name": "device_id",
"type": "text",
"primaryKey": false,
"notNull": true
},
"direction": {
"name": "direction",
"type": "onetalk_message_direction",
"typeSchema": "public",
"primaryKey": false,
"notNull": true
},
"observation_type": {
"name": "observation_type",
"type": "onetalk_observation_type",
"typeSchema": "public",
"primaryKey": false,
"notNull": true
},
"sent_at_ms": {
"name": "sent_at_ms",
"type": "bigint",
"primaryKey": false,
"notNull": true
},
"content": {
"name": "content",
"type": "jsonb",
"primaryKey": false,
"notNull": true
},
"participant_ids": {
"name": "participant_ids",
"type": "text[]",
"primaryKey": false,
"notNull": true
},
"read_status": {
"name": "read_status",
"type": "integer",
"primaryKey": false,
"notNull": true
},
"message_status": {
"name": "message_status",
"type": "integer",
"primaryKey": false,
"notNull": true
},
"unread_count": {
"name": "unread_count",
"type": "integer",
"primaryKey": false,
"notNull": true
},
"first_observed_at": {
"name": "first_observed_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"last_observed_at": {
"name": "last_observed_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {
"onetalk_message_conversation_time_idx": {
"name": "onetalk_message_conversation_time_idx",
"columns": [
{
"expression": "channel_account_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "conversation_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "sent_at_ms",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "message_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
},
"onetalk_message_sender_idx": {
"name": "onetalk_message_sender_idx",
"columns": [
{
"expression": "sender_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {},
"compositePrimaryKeys": {
"onetalk_message_channel_account_id_conversation_id_message_id_pk": {
"name": "onetalk_message_channel_account_id_conversation_id_message_id_pk",
"columns": ["channel_account_id", "conversation_id", "message_id"]
}
},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {
"onetalk_message_content_v1_chk": {
"name": "onetalk_message_content_v1_chk",
"value": "jsonb_typeof(\"onetalk_message\".\"content\") = 'object' and (\"onetalk_message\".\"content\" ->> 'version') = '1' and (\"onetalk_message\".\"content\" ->> 'kind') in ('text', 'image', 'file', 'business_card', 'inquiry', 'order', 'product') and ((\"onetalk_message\".\"content\" ->> 'kind') <> 'business_card' or \"onetalk_message\".\"content\" = '{\"version\":1,\"kind\":\"business_card\"}'::jsonb)"
}
},
"isRLSEnabled": false
},
"public.onetalk_message_anomaly": {
"name": "onetalk_message_anomaly",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true,
"default": "gen_random_uuid()"
},
"fingerprint": {
"name": "fingerprint",
"type": "text",
"primaryKey": false,
"notNull": true
},
"anomaly_type": {
"name": "anomaly_type",
"type": "text",
"primaryKey": false,
"notNull": true
},
"binding": {
"name": "binding",
"type": "text",
"primaryKey": false,
"notNull": false
},
"mind_user_id": {
"name": "mind_user_id",
"type": "text",
"primaryKey": false,
"notNull": false
},
"workspace_id": {
"name": "workspace_id",
"type": "text",
"primaryKey": false,
"notNull": false
},
"channel_account_id": {
"name": "channel_account_id",
"type": "text",
"primaryKey": false,
"notNull": false
},
"device_id": {
"name": "device_id",
"type": "text",
"primaryKey": false,
"notNull": false
},
"conversation_id": {
"name": "conversation_id",
"type": "text",
"primaryKey": false,
"notNull": false
},
"missing_fields": {
"name": "missing_fields",
"type": "text[]",
"primaryKey": false,
"notNull": true
},
"observation_source": {
"name": "observation_source",
"type": "text",
"primaryKey": false,
"notNull": true
},
"payload": {
"name": "payload",
"type": "jsonb",
"primaryKey": false,
"notNull": true
},
"status": {
"name": "status",
"type": "onetalk_anomaly_status",
"typeSchema": "public",
"primaryKey": false,
"notNull": true,
"default": "'open'"
},
"occurrence_count": {
"name": "occurrence_count",
"type": "integer",
"primaryKey": false,
"notNull": true,
"default": 1
},
"first_seen_at": {
"name": "first_seen_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"last_seen_at": {
"name": "last_seen_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"resolved_at": {
"name": "resolved_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
}
},
"indexes": {
"onetalk_message_anomaly_fingerprint_uidx": {
"name": "onetalk_message_anomaly_fingerprint_uidx",
"columns": [
{
"expression": "fingerprint",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"concurrently": false,
"method": "btree",
"with": {}
},
"onetalk_message_anomaly_scope_idx": {
"name": "onetalk_message_anomaly_scope_idx",
"columns": [
{
"expression": "channel_account_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "conversation_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "status",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
}
},
"enums": {
"public.onetalk_anomaly_status": {
"name": "onetalk_anomaly_status",
"schema": "public",
"values": ["open", "resolved", "ignored"]
},
"public.onetalk_buyer_fact_source_state": {
"name": "onetalk_buyer_fact_source_state",
"schema": "public",
"values": ["pending", "confirmed", "failed"]
},
"public.onetalk_contact_profile_observation_status": {
"name": "onetalk_contact_profile_observation_status",
"schema": "public",
"values": ["confirmed", "partial"]
},
"public.onetalk_message_direction": {
"name": "onetalk_message_direction",
"schema": "public",
"values": ["sent", "received"]
},
"public.onetalk_observation_type": {
"name": "onetalk_observation_type",
"schema": "public",
"values": ["new", "history"]
},
"public.onetalk_sync_phase": {
"name": "onetalk_sync_phase",
"schema": "public",
"values": ["initial", "incremental"]
},
"public.onetalk_sync_result": {
"name": "onetalk_sync_result",
"schema": "public",
"values": ["succeeded", "succeeded_with_anomalies", "failed", "incomplete"]
}
},
"schemas": {},
"sequences": {},
"roles": {},
"policies": {},
"views": {},
"_meta": {
"columns": {},
"schemas": {},
"tables": {}
}
}
+7
View File
@@ -92,6 +92,13 @@
"when": 1789190400000,
"tag": "0012_onetalk_history_generation",
"breakpoints": true
},
{
"idx": 13,
"version": "7",
"when": 1789361108983,
"tag": "0013_nifty_thunderbird",
"breakpoints": true
}
]
}
+1 -1
View File
@@ -116,7 +116,7 @@ export const onetalkMessage = pgTable(
index("onetalk_message_sender_idx").on(table.senderId),
check(
"onetalk_message_content_v1_chk",
sql`jsonb_typeof(${table.content}) = 'object' and (${table.content} ->> 'version') = '1' and (${table.content} ->> 'kind') in ('text', 'image', 'file', 'business_card', 'inquiry', 'order') and ((${table.content} ->> 'kind') <> 'business_card' or ${table.content} = '{"version":1,"kind":"business_card"}'::jsonb)`,
sql`jsonb_typeof(${table.content}) = 'object' and (${table.content} ->> 'version') = '1' and (${table.content} ->> 'kind') in ('text', 'image', 'file', 'business_card', 'inquiry', 'order', 'product') and ((${table.content} ->> 'kind') <> 'business_card' or ${table.content} = '{"version":1,"kind":"business_card"}'::jsonb)`,
),
],
);
@@ -0,0 +1,29 @@
// 验证 OneTalk 商品内容只扩展 JSONB kind 约束,不重写既有事实。
import assert from "node:assert/strict";
import { readFile } from "node:fs/promises";
import { resolve } from "node:path";
import test from "node:test";
import { migrationDirectory } from "../src/database/migration-config.ts";
const migrationPath = resolve(process.cwd(), migrationDirectory, "0013_nifty_thunderbird.sql");
test("extends only the content-kind check with product while preserving existing constraints", async () => {
const sql = await readFile(migrationPath, "utf8");
assert.match(sql, /DROP CONSTRAINT\s+"onetalk_message_content_v1_chk"/i);
assert.match(sql, /ADD CONSTRAINT\s+"onetalk_message_content_v1_chk"/i);
assert.match(
sql,
/\('text', 'image', 'file', 'business_card', 'inquiry', 'order', 'product'\)/,
);
assert.match(
sql,
/<> 'business_card' or "onetalk_message"\."content" = '\{"version":1,"kind":"business_card"\}'::jsonb/,
);
assert.equal((sql.match(/\bALTER TABLE\b/gi) ?? []).length, 2);
assert.equal((sql.match(/\bUPDATE\b/gi) ?? []).length, 0);
assert.equal((sql.match(/\bDELETE\b/gi) ?? []).length, 0);
assert.equal(sql.includes("history_generation"), false);
});
@@ -535,6 +535,46 @@ test("projects one persisted media fact identically for history and message.crea
assert.equal("text" in historyMessage, false);
});
test("projects one persisted product fact identically and rejects raw-query JSONB at the read boundary", () => {
const product = {
version: 1 as const,
kind: "product" as const,
sourceUrl:
"https://chinese.alibaba.com/product-detail/HAGO-Men-s-Breathable-Mid-Rise-1601456609478.html",
productId: "1601456609478",
};
const fact: OneTalkMessage = {
messageId: "product-1",
conversationId: "conversation-1",
senderId: "sender-1",
participantIds: ["sender-1", "account-1"],
direction: "received",
sentAtMs: 1_700_000_000_000,
readStatus: 1,
messageStatus: 2,
unreadCount: 0,
content: product,
};
const historyMessage = projectCenterMessage({
channelAccountId: scope.channelAccountId,
...fact,
});
const liveMessage = toOneTalkCenterMessage(fact);
assert.deepEqual(historyMessage, liveMessage);
assert.deepEqual(historyMessage.content, product);
assert.equal(JSON.stringify(historyMessage).includes("chatToken"), false);
const malformed = {
...fact,
content: { ...product, sourceUrl: `${product.sourceUrl}?chatToken=secret` },
} as unknown as OneTalkMessage;
assert.throws(
() => toOneTalkCenterMessage(malformed),
/Invalid persisted OneTalk message content/,
);
});
test("CenterMessage accepts normalized media and structured-card values", () => {
const verifiedImage = {
...messageFieldsForProjectionTest(),
+85 -1
View File
@@ -13,6 +13,7 @@ export const ONETALK_CONTENT_KINDS = [
"business_card",
"inquiry",
"order",
"product",
] as const;
export type OneTalkMessageContentKind = (typeof ONETALK_CONTENT_KINDS)[number];
@@ -92,13 +93,22 @@ export type OneTalkOrderContent = {
actions: OneTalkOrderAction[];
};
/** MAIN world 从商品详情原始 URL 派生的无凭据商品引用。 */
export type OneTalkProductContent = {
version: typeof ONETALK_CONTENT_VERSION;
kind: "product";
sourceUrl: string;
productId: string;
};
export type OneTalkMessageContent =
| OneTalkTextContent
| OneTalkImageContent
| OneTalkFileContent
| OneTalkBusinessCardContent
| OneTalkInquiryContent
| OneTalkOrderContent;
| OneTalkOrderContent
| OneTalkProductContent;
export type OneTalkCenterMessageContent = OneTalkMessageContent | OneTalkBusinessCardViewContent;
@@ -134,6 +144,9 @@ const MAX_EXTENSION_LENGTH = 64;
const MAX_MD5_LENGTH = 128;
const MAX_MEDIA_URL_LENGTH = 8 * 1024;
const MAX_ORDER_ACTIONS = 100;
const PRODUCT_DETAIL_HOST = "chinese.alibaba.com";
const PRODUCT_DETAIL_PATH = /^\/product-detail\/(?:[A-Za-z0-9]+-)+([1-9]\d*)\.html$/u;
const PRODUCT_URL_AUTHORITY = /^https:\/\/([^/?#]*)/u;
const CONTENT_KEYS = {
text: ["version", "kind", "text"],
@@ -180,6 +193,7 @@ const CONTENT_KEYS = {
"statusMessageKey",
"actions",
],
product: ["version", "kind", "sourceUrl", "productId"],
} as const;
const ORDER_ACTION_KEYS = ["name", "messageKey", "payStep"] as const;
@@ -363,6 +377,53 @@ const isOptionalMediaUrl = (
return value === null || isMediaUrl(value, expectedPath, allowedFileActions);
};
/**
* 只在 MAIN world 解析 OneTalk 原始商品详情 URL;query 会在这里被剥离,不能进入跨层合同。
*/
export const normalizeOneTalkProductUrl = (
value: unknown,
): { sourceUrl: string; productId: string } | null => {
const authority =
typeof value === "string" ? PRODUCT_URL_AUTHORITY.exec(value)?.[1] : undefined;
if (
typeof value !== "string" ||
value.length === 0 ||
value.length > MAX_TEXT_LENGTH ||
value.trim() !== value ||
/\s/u.test(value) ||
!value.startsWith("https://") ||
authority !== PRODUCT_DETAIL_HOST ||
value.includes("#")
) {
return null;
}
const rawPath = value.slice(`https://${PRODUCT_DETAIL_HOST}`.length).split("?", 1)[0];
const pathnameMatch = PRODUCT_DETAIL_PATH.exec(rawPath);
if (pathnameMatch === null) return null;
try {
const url = new URL(value);
if (
url.protocol !== "https:" ||
url.hostname !== PRODUCT_DETAIL_HOST ||
url.port.length > 0 ||
url.username.length > 0 ||
url.password.length > 0 ||
url.hash.length > 0 ||
url.pathname !== rawPath
) {
return null;
}
return {
sourceUrl: `${url.origin}${rawPath}`,
productId: pathnameMatch[1],
};
} catch {
return null;
}
};
const isOneTalkTextContent = (value: Record<string, unknown>): value is OneTalkTextContent => {
return (
hasExactKeys(value, CONTENT_KEYS.text) &&
@@ -471,6 +532,26 @@ const isOneTalkOrderContent = (value: Record<string, unknown>): value is OneTalk
);
};
const isOneTalkProductContent = (
value: Record<string, unknown>,
): value is OneTalkProductContent => {
if (
!hasExactKeys(value, CONTENT_KEYS.product) ||
value.version !== ONETALK_CONTENT_VERSION ||
value.kind !== "product" ||
typeof value.sourceUrl !== "string" ||
typeof value.productId !== "string"
) {
return false;
}
const normalized = normalizeOneTalkProductUrl(value.sourceUrl);
return (
normalized !== null &&
normalized.sourceUrl === value.sourceUrl &&
normalized.productId === value.productId
);
};
/** 解码 exact-shape 的已归一化 OneTalk 消息内容。 */
export const decodeOneTalkMessageContent = (value: unknown): OneTalkMessageContentDecodeResult => {
if (!isPlainRecord(value) || value.version !== ONETALK_CONTENT_VERSION) return { ok: false };
@@ -493,6 +574,9 @@ export const decodeOneTalkMessageContent = (value: unknown): OneTalkMessageConte
if (value.kind === "order" && isOneTalkOrderContent(value)) {
return { ok: true, content: value };
}
if (value.kind === "product" && isOneTalkProductContent(value)) {
return { ok: true, content: value };
}
return { ok: false };
};
@@ -52,6 +52,7 @@ import {
isNextOneTalkConversationsDiscoveredFragment,
isOneTalkMessage,
isOneTalkMessageContent,
normalizeOneTalkProductUrl,
type OneTalkMindScope,
type MockAuthorizationRecord,
type OneTalkPluginScope,
@@ -204,6 +205,14 @@ const orderContent = {
actions: [{ name: "pay", messageKey: "order.pay", payStep: "deposit" }],
};
const productContent = {
version: ONETALK_CONTENT_VERSION,
kind: "product" as const,
sourceUrl:
"https://chinese.alibaba.com/product-detail/HAGO-Men-s-Breathable-Mid-Rise-1601456609478.html",
productId: "1601456609478",
};
const observedMessage = {
messageId: "message-1",
conversationId: "conversation-1",
@@ -1312,6 +1321,73 @@ test("stores exact business-card markers and only exposes customer fields in pub
}
});
test("normalizes raw product URLs and only accepts their exact query-free product facts", () => {
const rawUrl = `${productContent.sourceUrl}?chatToken=secret&targetEncrypt=encrypted`;
assert.deepEqual(normalizeOneTalkProductUrl(rawUrl), {
sourceUrl: productContent.sourceUrl,
productId: productContent.productId,
});
assert.equal(isOneTalkMessageContent(productContent), true);
assert.deepEqual(decodeOneTalkMessageContent(productContent), {
ok: true,
content: productContent,
});
assert.equal(isOneTalkMessage({ ...observedMessage, content: productContent }), true);
assert.equal(
decodeOneTalkFrame({
...frameBase,
type: "message.observed",
payload: {
historyGeneration: "generation-1",
observationSource: "history",
message: { ...observedMessage, content: productContent },
},
}).ok,
true,
);
assert.equal(
decodeOneTalkFrame({
...frameBase,
connectionType: "mind_page",
scope: mindScope,
type: "message.created",
payload: { message: { ...centerMessage, content: productContent } },
}).ok,
true,
);
for (const value of [
` ${rawUrl}`,
`${rawUrl} `,
"http://chinese.alibaba.com/product-detail/HAGO-1601456609478.html",
"https://www.alibaba.com/product-detail/HAGO-1601456609478.html",
"https://chinese.alibaba.com:443/product-detail/HAGO-1601456609478.html",
"https://chinese.alibaba.com:/product-detail/HAGO-1601456609478.html",
"https://user:password@chinese.alibaba.com/product-detail/HAGO-1601456609478.html",
"https://chinese.alibaba.com/product-detail/HAGO-1601456609478.html#details",
"https://chinese.alibaba.com/product-detail/HAGO-1601456609478.html#",
"https://chinese.alibaba.com/product-detail/./HAGO-1601456609478.html",
"https://chinese.alibaba.com/product-detail/HAGO/../HAGO-1601456609478.html",
"https://chinese.alibaba.com/product-detail/%2e/HAGO-1601456609478.html",
"https://chinese.alibaba.com/product-detail/HAGO/%2E%2E/HAGO-1601456609478.html",
"https://chinese.alibaba.com/product-detail/HAGO-%31%36%30%31%34%35%36%36%30%39%34%37%38.html",
"https://chinese.alibaba.com/product-detail/HAGO-01601456609478.html",
"https://chinese.alibaba.com/product-detail/HAGO-1601456609478.htm",
"https://chinese.alibaba.com/product-detail/HAGO__invalid-1601456609478.html",
]) {
assert.equal(normalizeOneTalkProductUrl(value), null);
}
for (const content of [
{ ...productContent, sourceUrl: rawUrl },
{ ...productContent, productId: "1601456609479" },
{ ...productContent, rawUrl },
]) {
assert.equal(isOneTalkMessageContent(content), false);
assert.deepEqual(decodeOneTalkMessageContent(content), { ok: false });
}
});
test("rejects legacy image dimensions as exact-shape extras", () => {
for (const [key, value] of [
["width", 1_280],