feat: index OneTalk sync ledger queries

This commit is contained in:
YBF
2026-09-15 02:39:09 +08:00
parent 648b17e728
commit 61df41d2ea
15 changed files with 757 additions and 102 deletions
@@ -84,7 +84,7 @@ OneTalkContactProfileStore.discardPendingProfile(input: {
### Durable lifecycle
- `ONE_TALK_SYNC_DATABASE_VERSION``8`。独立的 `onetalk_contact_profiles` store 以账号/会话对为键;消息、candidate、checkpoint、anomaly 各 store 保持独立。
- `ONE_TALK_SYNC_DATABASE_VERSION``9`。独立的 `onetalk_contact_profiles` store 以账号/会话对为键;消息、candidate、checkpoint、anomaly 各 store 保持独立。v8→v9 只建立查询 index 并保留 ledger record`hasProfileRecord(account)` 走账号 index countpending flush 以 `[channelAccountId, pending.observedAtMs]` 的合法账号前缀范围读取。无 `pending.observedAtMs` 的 uploaded/rejected record 不得出现在该 index 结果中,也不得回退到 object-store 全表扫描。
- durable profile 指纹是上传去重的唯一闸门。每收到一条 profile,先在 `putPendingProfile()` 之前读取它的 `[channelAccountId, conversationId]` 记录。若 `pending.fingerprint``lastUploadedFingerprint` 与来入指纹相同,读后即结束:不更新时间戳/高水位、不写 IndexedDB、不发 `profile_observed` 诊断、不 flush、不发送。指纹不同时走既有的 durable-first pending 写入。
- 同键观察对 `getProfile → putPendingProfile` 转换做串行化。并发的相同指纹可以各自执行读取,但最多一个能写入、诊断或 flush。读取失败对调用方保持可见;tracker 清理不得制造第二个未处理 rejection。durable 读进行期间协调器被 dispose 时,其完成不产生写入、诊断或 flush。
- 只有 readwrite transaction 完成且当前 pending 的 fingerprint 与 `observedAtMs` 仍然匹配时,ACK 才标记 uploaded。更旧、未知、重复或不匹配的 ACK 一律 no-op。future-skew 错误只移除精确匹配的 pending snapshot,并记录其被拒观察水位。
@@ -123,7 +123,7 @@ OneTalkContactProfileStore.discardPendingProfile(input: {
- Contract:精确 profile/frame 字段、协议版本、方向/scope、direct discovery 类型、敏感/未知字段拒绝、空/超限批次和 `256 KiB` 字节上限。
- Observerinitial snapshot、可重复的 `syncData` 发布、精确 snapshot/collect payload 校验、targeted direct 查找、群聊/未知目标排除、登录身份、登出/切账号、CRM 客户匹配和头像 URL 校验。
- Ledgerpre-v7`oldVersion < 7`)升级先清空全部旧 OneTalk store 再重建当前 profile 与消息同步状态,v7→v8 升级保留五个既有 store;账号/会话键、durable-first 顺序、同 pending/已上传指纹一次读零写、同键并发写、读中 dispose、显式读取失败、ACK/CAS、future-skew 丢弃、重连与重启恢复。升级必须保留 configuration/deviceId,且不得 rekey 或重试旧 pending ledger。
- Ledgerpre-v7`oldVersion < 7`)升级先清空全部旧 OneTalk store 再重建当前 profile 与消息同步状态,v7→v8 升级保留五个既有 storev8→v9 逐条保留六个 OneTalk store record 并创建 scope index;账号/会话键、durable-first 顺序、同 pending/已上传指纹一次读零写、同键并发写、读中 dispose、显式读取失败、ACK/CAS、future-skew 丢弃、重连与重启恢复。测试必须区分 profile account/pending index 与 `objectStore.getAll()`,并确认 uploaded record 不会混入 pending flush。升级必须保留 configuration/deviceId,且不得 rekey 或重试旧 pending ledger。
- Service Worker:既有 Bright binding/read/sync 授权、page-first 与 auth-first 两种顺序的首次认证 setup、ledger 读取后授权丢失、实时 sent/received targeted collect、历史排除、50/51 条 profile 批次、ACK 计数、future 错误映射和过期回调/页面身份 fencedirect discovery 始终携带 `conversationType: "direct"`
- 必须执行定向 typecheck、contract/extension focused 与全量测试、format check 和 `git diff --check`。真实 Chromium、Bright PostgreSQL 和生产 Mind 集成是另行执行的外部检查。
@@ -61,10 +61,18 @@ rejected
### Profile ledger (independent state machine)
联系人资料不使用消息 candidate/checkpoint/anomaly store。`ONE_TALK_SYNC_DATABASE_VERSION=8`:从 `oldVersion < 7` 升级时,transaction 必须清空五个 OneTalk store(消息、candidate、checkpoint、anomaly、`onetalk_contact_profiles`),不重键或回补旧 ledger;从 v7 升级到 v8 时只新增 `onetalk_conversation_bootstraps` store,必须保留五个既有 store 的记录。Chrome 配置、deviceId、binding 与其它渠道存储不属于该删除范围;首次升级后必须重新采集 profile 并从 clean state 执行 full syncv7→v8 不得因为新增 bootstrap store 清理既有事实。v7 checkpoint 新增 OneTalk 会话列表来源的 `latestMessageAtMs`,用于同步完成后更新服务端会话活动时间。
联系人资料不使用消息 candidate/checkpoint/anomaly store。`ONE_TALK_SYNC_DATABASE_VERSION=9`:从 `oldVersion < 7` 升级时,transaction 必须清空五个 OneTalk store(消息、candidate、checkpoint、anomaly、`onetalk_contact_profiles`),不重键或回补旧 ledger;从 v7 升级到 v8 时只新增 `onetalk_conversation_bootstraps` store,必须保留五个既有 store 的记录;从 v7/v8 升级到 v9 时只为既有消息、checkpoint、candidate、anomaly、profile records 建立账号、会话与状态查询索引,六个 store 的记录逐条保留。legacy clear 完成后才建 index,避免为必然删除的 records 建索引。Chrome 配置、deviceId、binding 与其它渠道存储不属于该删除范围;首次 pre-v7 升级后必须重新采集 profile 并从 clean state 执行 full syncv7→v8/v9 不得因为 schema 演进清理既有事实。v7 checkpoint 新增 OneTalk 会话列表来源的 `latestMessageAtMs`,用于同步完成后更新服务端会话活动时间。
清空后的 `onetalk_contact_profiles` 业务键为 `channelAccountId + conversationId`;记录包含 key、账号、conversationId、资料字段中的 aliId、lastUploadedFingerprint、uploaded/rejected observed high-water mark、updatedAt、lastUploadedAt 和最新 pending 清洗 profile。资料上传到 Bright persistence,不调用 Mind profile HTTP。
### v9 scope-bounded reads
- `checkpoint` 使用 `by_account`candidate 使用 `by_account_status``by_account_conversation``by_account_conversation_status`message/anomaly 使用 `by_account_conversation`profile 使用 `by_account``by_account_pending_observed_at`。bootstrap 继续只按稳定主键读取,不为无消费者的查询建 index。
- 已知 ACK `candidateKey` 必须直接主键读取,且只接纳仍为 `pending_ack` 的 candidate;没有 request key 时才兼容查找 `(channelAccountId, conversationId, messageId)`。确认后只比较 checkpoint 当前 `latestMessageId` candidate 与本次 confirmed candidate 的 `candidateOrder`,不得重建全会话候选列表。
- profile pending 读取必须使用 `[channelAccountId, pending.observedAtMs]` 的合法账号前缀 `IDBKeyRange`;缺少 `pending.observedAtMs` 的 uploaded/rejected record 没有该 index entry,不得由调用方再过滤整表。
- 会话重建必须在一个 readwrite transaction 中,对目标 scope index 执行 `getAllKeys()` 并删除 message/candidate/anomaly,再直接删除该会话 checkpoint;不得在 transaction 外读后删,也不得使用 object-store `getAll()`
- v9 profile 一旦打开,v8 binary 打开同一数据库会因版本较低失败。发布回滚只能通过 v10 前向迁移,或保留 v9 schema 并仅禁用非 schema 行为;不得重发 v8 或清除 durable ledger。
Observation 先写最新 pending,再由现有 Bright WebSocket 发送 contact.profile.observed。同 fingerprint 且无 pending 时只推进更高 observed high-water mark;任何不高于 uploaded/rejected high-water mark 的不同 fingerprint 也跳过。断线、Service Worker 重启或新页面连接只从 pending 重建发送。收到 contact.profile.ack 后,必须等待 readwrite transaction oncomplete,且只确认仍匹配的 fingerprint 和 observedAtMs;迟到旧 ACK 不得删除新 pending。收到 `ws.error` `profile_observed_at_future` 时只丢弃该 request 的精确 pending,避免无限重试。
消息与 profile 的耐久状态仍然分离:历史/实时消息中的 `business_card` 只持久化 `{ version: 1, kind: "business_card" }` marker。由于 `__conversationListData__` 与消息采集可能异步,消息候选不得等待或读取 profile ledger;当前客户资料只在服务端读取时按 `[channelAccountId, conversationId]` 以内存方式组合到对外 view。没有 profile 时保留 marker,不能 fallback 到登录人资料。
@@ -171,7 +179,7 @@ Service Worker 重启后必须从 IndexedDB 恢复:
- 断线恢复会重新发现会话并恢复未确认事实和 completion 声明。
- \`delivery_unknown\` 不创建发送任务、不自动重发;迟到消息继续进入普通 observation。
- 页面同步路由、Bright ACK 和 IndexedDB 事务顺序在重启/断线下保持一致。
- `oldVersion < 7` 升级到 v8 必须断言五个 OneTalk store 均为空、没有 profile 重键/重试分支,且下一次 bootstrap 重新采集 profile、会话活动时间并执行 full sync;v7→v8 必须断言新增 bootstrap store 且五个既有 store 逐条保留;配置、deviceId 与其它渠道数据在两条路径保持不变。
- `oldVersion < 7` 升级到 v9 必须断言五个 OneTalk store 均为空、没有 profile 重键/重试分支,且下一次 bootstrap 重新采集 profile、会话活动时间并执行 full sync;v7→v8 必须断言新增 bootstrap store 且五个既有 store 逐条保留;v8→v9 必须断言六个 OneTalk store 的 records 逐条保留、目标 index 存在且读取不调用 `objectStore.getAll()`配置、deviceId 与其它渠道数据在所有升级路径保持不变。
## 7. Wrong vs Correct
@@ -17,7 +17,7 @@ OneTalk 插件同时需要以下能力时,遵循本总览和对应子规范:
| --- | --- |
| [OneTalk 页面桥、Port 与命令路由](./page-bridge.md) | MAIN/ISOLATED/SW 页面桥、Port 注册、页面身份和 command 路由 |
| [OneTalk 耐久同步与连接生命周期](./durable-sync.md) | IndexedDB、full/incremental/live、ACK、checkpoint、重启恢复和连接生命周期 |
| [OneTalk 联系人资料 Bright 持久化](./contact-profile-sync.md) | profile 白名单、Bright profile frame、v7 清空后重采集 ledger、ACK/HWM/future-skew 和账号/epoch 隔离 |
| [OneTalk 联系人资料 Bright 持久化](./contact-profile-sync.md) | profile 白名单、Bright profile frame、pre-v7 清空后重采集、v9 scope index、ACK/HWM/future-skew 和账号/epoch 隔离 |
| [OneTalk 扩展安装实例设备身份](./device-identity.md) | deviceId 生成、迁移、独立存储、配置清除和生命周期 |
| [OneTalk Service Worker 状态与诊断](./runtime-diagnostics.md) | getSnapshot、错误投影、敏感信息脱敏和 development 构建 |
| [OneTalk PWA 出站发送 SOP](./send-sop.md) | sendUIMessages 输入、SDK-only 发送和 WebSocket 旁路事实确认 |
@@ -36,7 +36,7 @@ OneTalk MAIN world
-> Bright WebSocket upload
-> per-message ACK
联系人资料事实使用独立路径:OneTalk MAIN snapshot/syncData → safe profile envelope + page identity → Service Worker v7 clean-state profile ledger → existing Bright plugin WebSocket `contact.profile.observed` → Bright guarded profile transaction → `contact.profile.ack`
联系人资料事实使用独立路径:OneTalk MAIN snapshot/syncData → safe profile envelope + page identity → Service Worker pre-v7 clean-state / v9 scope-indexed profile ledger → existing Bright plugin WebSocket `contact.profile.observed` → Bright guarded profile transaction → `contact.profile.ack`
\`\`\`
服务端命令:
@@ -247,7 +247,7 @@ writer.sendSendConfirmation(result);
- `authorization_unavailable` 表示授权依赖暂时不可用,只关闭当前 Bright socket 并沿既有连接退避自动重连;只有凭证、授权版本、binding、scope 或协议版本等确定性错误才阻断自动重连并进入 unauthorized。
- Service Worker 重启从 IndexedDB 恢复 checkpoint、候选和模式,不信任旧内存 cursor。
- live `messageType: "new"`sent 或 received)只触发所属 conversation 的 profile collecthistory 不触发。profile 相同 fingerprint 在 coordinator 的唯一 durable read 后静默结束,不能通过 MAIN `seen`、时间水位或 timer 再建第二去重状态。
- `oldVersion < 7` 的数据库先执行 v7 全量 OneTalk state 清空(v7→v8 只新增 bootstrap store、保留既有事实),再重新采集 profile、会话活动时间并执行 full sync;之后的重启/重连才从当前 ledger 恢复 pending。ACK 只在当前 `[channelAccountId, conversationId, fingerprint, observedAtMs]` 的 IndexedDB transaction commit 后生效;future-skew 整批拒绝并只丢弃匹配 pending。
- `oldVersion < 7` 的数据库先执行全量 OneTalk state 清空(v7→v8 只新增 bootstrap store、保留既有事实v8→v9 只新增 scope index、逐条保留 records),再重新采集 profile、会话活动时间并执行 full sync;之后的重启/重连才从当前 ledger 恢复 pending。v9 profile pending flush 只读取 `[channelAccountId, pending.observedAtMs]` index,不得扫描其它账号或已上传 record。ACK 只在当前 `[channelAccountId, conversationId, fingerprint, observedAtMs]` 的 IndexedDB transaction commit 后生效;future-skew 整批拒绝并只丢弃匹配 pending。
### Send and protocol boundaries
@@ -0,0 +1,3 @@
{"file":".trellis/spec/project/architecture.md","reason":"Review state ownership, transaction boundaries, source layout, and no duplicate derived source."}
{"file":".trellis/spec/chrome-extension/frontend/onetalk/durable-sync.md","reason":"Verify durable ledger, migration, ACK, and recovery behavior against the canonical contract."}
{"file":".trellis/spec/chrome-extension/frontend/onetalk/contact-profile-sync.md","reason":"Verify profile pending and ACK behavior remains independent from message ledger changes."}
@@ -0,0 +1,74 @@
# OneTalk IndexedDB 查询索引设计
## Design decision
采用一次 v9 schema migration、scope/status 索引和精确 ACK 主键读取。保留所有 ledger records 和 JSON 序列化主键;将“整表传输到 JS 后过滤”改为 IndexedDB 内部按索引范围限制。confirmed candidate 不是临时队列,继续作为可恢复、可审计的 durable ledger。
## Storage schema
`ensureSyncStores` 拆成“确保 store 存在”和“确保 index 存在”两部分,只在 `onupgradeneeded` 的 versionchange transaction 执行。v9 的私有 index
| Store | Index | Key path | Consumer |
| --- | --- | --- | --- |
| message | `by_account_conversation` | `[channelAccountId, conversationId]` | rebuild delete keys |
| checkpoint | `by_account` | `channelAccountId` | resume、retry completion、bootstrap snapshot |
| candidate | `by_account_status` | `[channelAccountId, status]` | account pending resume |
| candidate | `by_account_conversation_status` | `[channelAccountId, conversationId, status]` | drain、awaiting-anchor、pending count |
| candidate | `by_account_conversation` | `[channelAccountId, conversationId]` | rebuild delete keys |
| anomaly | `by_account` | `channelAccountId` | account anomaly read |
| anomaly | `by_account_conversation` | `[channelAccountId, conversationId]` | conversation anomaly/read-delete |
| contact profile | `by_account` | `channelAccountId` | record count |
| contact profile | `by_account_pending_observed_at` | `[channelAccountId, pending.observedAtMs]` | pending flush |
bootstrap 继续只有主键,因为只按稳定 `(channelAccountId, migrationId)` 读取。profile pending index 使用已有嵌套标量;没有 pending 的 record 没有 index entryv8 的 pending record 会在创建 index 时自动进入,不增加平行 `hasPending` 字段或 backfill source of truth。
## Migration and compatibility
1. `ONE_TALK_SYNC_DATABASE_VERSION` 从 8 升至 9。
2. 先确保六个 stores 存在;若 `oldVersion < 7`,按既有契约清空五个 legacy stores;最后通过 `indexNames.contains` 幂等建 index。
3. v7/v8 → v9 保留 records,浏览器从现有 records 建 index;不重键、不回放、不清库。
4. 保留 `onblocked` rejection,不增加 silent fallback。
先清 legacy 再建 index,避免为必然删除的数据建索引。v9 数据库不能由 v8 代码打开,因此线上回滚只能是 v10 前向兼容修复,不能直接重发旧扩展。
## Query API and call-site refactor
以带范围的 `readByIndex``readKeysByIndex``countByIndex` 和既有 `readOne` 取代通用 `readAll`。所有生产调用已有账号 scope,收紧 optional account 参数,不保留跨账号全表枚举。
| Current path | Replacement | Result bound |
| --- | --- | --- |
| `listCheckpoints(account)` | checkpoint `by_account.getAll(only(account))` | account checkpoints |
| `listPendingCandidates(account)` | candidate `by_account_status.getAll(only([account, pending_ack]))` | account pending |
| conversation candidate list + filter | `by_account_conversation_status` | conversation and status |
| request key → pending list → find | `getCandidateByKey(requestKey)` + `pending_ack` guard | one candidate |
| latest confirmed from all candidates | checkpoint `latestMessageId` candidate vs just-confirmed candidate by `candidateOrder` | at most two candidates |
| completion pending check | status index `count` | one count |
| `listAnomalies` | account or scope index | account or conversation |
| `listPendingProfiles(account)` | profile pending compound index range | account pending profiles |
| `hasProfileRecord(account)` | profile account-index `count` | one count |
| `clearConversationHistory` | scope-index `getAllKeys`; checkpoint direct delete | target conversation keys only |
`AckCompletionCoordinator` 保持协议语义的具体变更:
- `activateAwaitingCandidates` 精确取 anchor,只列出 `awaiting_anchor`
- `drainConversation` 只列出 `pending_ack`,保留排序、批次和 in-flight guards。
- 有 request key 的 ACK 直接读取 candidate,并要求它仍是 `pending_ack`;没有 key 时保留 `(account, conversation, messageId)` 回退。
- durable status 更新后,以 checkpoint 的 `latestMessageId` 精确读取当前最新 candidate,并与刚确认 candidate 用 `candidateOrder` 比较;因此乱序 ACK 仍得到相同最大 confirmed message,无需物化整会话。
- `tryComplete` 用 pending index count 代替全 candidate `some`。explicit-start gate 改为发送/ACK 调用携带精确的 in-memory 标记,不扫描 durable history 寻找该标记;现有 reset/clear 时同步清理。
不解析或依赖 JSON key 的字符串排序,不改变既有 `getCandidate(account, conversation, messageId)`
## Transaction and failure semantics
会话重建仍使用同一个四 store readwrite transaction:每个 scope index `getAllKeys(range)` 后对返回 key `delete`,最后等待该 transaction `oncomplete`。不在 transaction 外先读后删,不引入异常吞没或 fallback。
ACK 顺序仍为 durable candidate 读取 → candidate 状态写入 → checkpoint 写入(若需要)→ completion 判断。既有 lifecycle、generation、quiesce 和 authorization guards 均不因缩小查询范围而绕开。
## Test design
1. 扩展 storage/profile tests 的 Fake IndexedDB,使其可观测 `objectStore.getAll`,并最小实现 `createIndex``index``getAll``getAllKeys``count`;加入 v8 seed → v9 upgrade。
2. 以两账号、两会话、不同 candidate status、账号级 anomaly、pending/non-pending profile 断言精确结果,并断言目标读取未调用 object-store `getAll`
3. 覆盖会话清理只删除目标会话;覆盖精确 request key、批量/乱序 ACK、pending resume、awaiting-anchor、completion gate 和 profile flush。
4. 保留并重跑 `<7` clear、v7→v8 保留断言;新增 v8→v9 preservation。
功能测试证明查询范围,不将 fake timing 当性能证据。真实 Chromium profile 的 IDB latency、index-build 时间、Worker memory/termination 采样作为后续独立验证。
@@ -0,0 +1,3 @@
{"file":".trellis/spec/project/architecture.md","reason":"Preserve state ownership, async boundaries, and source responsibilities during storage/query refactor."}
{"file":".trellis/spec/chrome-extension/frontend/onetalk/durable-sync.md","reason":"Preserve durable-first, candidate ACK, checkpoint, restart, and migration contracts."}
{"file":".trellis/spec/chrome-extension/frontend/onetalk/contact-profile-sync.md","reason":"Preserve independent profile ledger and ACK semantics while indexing profile reads."}
@@ -0,0 +1,41 @@
# Implementation plan: OneTalk IndexedDB scoped queries
## Scope and ownership
Primary implementation owner is `apps/chrome-extension/src/onetalk/service-worker/storage.ts`. Affected consumers are `sync-engine/ack-completion.ts` and existing test doubles under `apps/chrome-extension/test/`; Bright, Mind, page-bridge and authorization contracts are out of scope.
## Ordered work
1. Before each production-symbol edit, run GitNexus impact analysis and load the target frontend/durable-sync specs.
2. Add v9 index constants/helpers and versionchange migration; preserve old-version clear order and existing open/blocked failure behavior.
3. Replace generic all-store reader with bounded primary/index helpers. Add exact candidate-key, status-scoped candidate, scoped anomaly/profile, and transaction-local `getAllKeys` delete APIs.
4. Refactor `AckCompletionCoordinator` to use direct request-key reads, status-scoped lists, checkpoint-vs-candidate latest comparison, and indexed pending completion checks. Keep queue/generation/ACK order unchanged.
5. Update type-conforming test fakes and focused tests; add v8→v9 compatibility and query-path instrumentation.
6. Run focused tests, typecheck, build, full tests and format check; review for retention, protocol or fallback regressions.
7. Before commit, run GitNexus `detect_changes()` and inspect affected flows. Commit/release remain separately authorized.
## Validation commands
```bash
pnpm --filter @trade-message-center/chrome-extension test -- onetalk-sync-storage.test.js
pnpm --filter @trade-message-center/chrome-extension test -- onetalk-contact-profile-storage.test.js
pnpm --filter @trade-message-center/chrome-extension test -- onetalk-sync-engine.test.js
pnpm typecheck
pnpm build
pnpm test
pnpm format:check
```
Use the repository-supported focused-test argument form confirmed at implementation time. Any unexpectedly affected backend unit test keeps the project 60-second timeout rule.
## Review gates
- Every target query carries an account/conversation/status IndexedDB range; target code has no `objectStore.getAll()`.
- No confirmed candidate deletion, primary-key shape change, raw payload index or Bright frame change.
- v8 preservation and `<7` cleanup are verified independently.
- New reads/writes stay in IndexedDB transactions and `oncomplete` remains the durable boundary.
- Release notes state the no-binary-rollback v9 constraint.
## Rollback
Before release, source rollback is normal Git reversal. After a profile receives v9, do not deploy v8 against that profile; publish a v10 forward migration that restores compatible behavior, or retain v9 schema code and explicitly disable only a non-schema behavior. Never clear durable ledger data as rollback.
@@ -0,0 +1,67 @@
# 修复 OneTalk IndexedDB 查询全表扫描
## Goal
在不改变 OneTalk durable-first、候选 ACK、锚点推进、profile ledger 或历史保留语义的前提下,消除同步账本按整个对象仓库 `getAll()` 后在内存过滤的查询路径。目标是让 ACK、重连恢复、会话重建和 profile flush 的读取范围由账号、会话和状态键限定,而不是由历史总量决定。
## Confirmed facts
- `storage.ts` 当前数据库版本为 8`ensureSyncStores` 仅为六个 OneTalk store 建立 `{ keyPath: "key" }`,没有二级索引(`apps/chrome-extension/src/onetalk/service-worker/storage.ts:16,275-288`)。
- `readAll` 调用 `objectStore.getAll()``listCheckpoints``listCandidates``listPendingCandidates``listAnomalies``listPendingProfiles``hasProfileRecord` 都经由该路径读取整表(`storage.ts:384-390,605-611,644-671,722-730,841-862`)。
- `clearConversationHistory` 对 message、checkpoint、candidate、anomaly 四个 store 各调用一次 `getAll()`,再逐记录删除(`storage.ts:740-765`)。
- 每条有效观察都会持久化 candidate,确认后保留;仅显式会话重建清理候选。因此 candidate 表的规模会随历史同步积累(`storage.ts:514-555`)。
- 已关联的 ACK 请求携带精确 `candidateKey`,但 `handleAcknowledgementResult` 仍先 `listPendingCandidates(account)``.find(key)`;批量 ACK 对每个结果串行执行(`ack-completion.ts:377-405,545-548`)。
- 认证/重连会调用 `resumePendingCandidates``retrySentCompletions``sync-engine.ts:340-350,367-378`)。现有调用点都传入 `channelAccountId`;没有生产调用需要跨账号列出 pending candidate 或 pending profile。
- v7→v8 必须保留既有 OneTalk ledger`oldVersion < 7` 才会清空五个 legacy store。该兼容契约不可倒退(`.trellis/spec/chrome-extension/frontend/onetalk/durable-sync.md`)。
## Requirements
### R1. Schema migration
- 将同步数据库升级到 v9;为既有和新建 store 幂等创建查询索引。
- v8→v9 只建立索引,必须逐条保留 message、checkpoint、candidate、anomaly、profile 和 bootstrap 记录;不得重键、清库或回放观察。
- `oldVersion < 7` 的既有清库规则保持不变;清理完成后再建立索引,避免为必然删除的 legacy 记录建立索引。
- index 名称和 key path 是 `storage.ts` 私有 schema,不能泄露到 Bright、Mind 或页面桥合同。
### R2. Scope-bounded reads
- candidate:支持按精确 key、`[channelAccountId, conversationId]` 和候选状态读取;按账号恢复 pending 时只读取该账号的 `pending_ack`
- checkpoint:按账号列出恢复/完成声明所需 checkpoint;按精确主键删除单会话 checkpoint。
- message、candidate、anomaly:会话重建只读取匹配 `[channelAccountId, conversationId]` 的 keys,再在同一 readwrite transaction 删除;不得物化不相关记录。
- profile`hasProfileRecord` 使用账号索引计数;pending flush 仅从该账号的 pending profile 索引读取。
- anomaly 的现有账号/会话查询保持结果和账号级 anomaly 的排除语义。
### R3. ACK and sync semantics
- ACK 已有 `candidateKey` 时必须以主键直接读取,且只接受 `pending_ack` candidate;保留没有 request key 时按 `(account, conversation, messageId)` 的兼容查找。
- 按会话 drain 只读取 `pending_ack`;激活增量候选只读取 `awaiting_anchor` 及精确 anchor candidate。
- ACK 后最新 confirmed message 的选择继续遵守 `candidateOrder`。不得因索引重构把较旧、乱序 ACK 覆盖成最新锚点。
- `tryComplete` 只查询是否仍有 pending candidate;现有 explicit-start/boundary gate 语义不得变化。
- 不删除 confirmed candidate,不改业务主键、不改 Bright frame、不改变 ACK 先 durable write 后推进的顺序。
### R4. Verification
- 测试必须证明 v8→v9 记录保留和 index 创建;`oldVersion < 7` 的清理行为继续成立。
- 存储层测试必须能区分 `objectStore.getAll()``IDBIndex.getAll()` / `getAllKeys()` / `count()`,并断言目标查询未走全表读取。
- 同步引擎测试必须覆盖精确 request key ACK、批量 ACK、断线恢复、增量 anchor 和 completion 路径,确认 observable 协议行为未变。
## Out of scope
- 不引入 confirmed candidate 过期/压缩策略,不删除历史消息或改变 retention。
- 不变更 IndexedDB database name、JSON 序列化主键、OneTalk/Bright/Mind frame、页面桥或授权逻辑。
- 不承诺本任务已在真实用户 profile 的数万条数据上测得 worker 卡顿或内存下降;该问题需要单独的真实浏览器性能采样。
- 不为只按主键读取的 bootstrap store 增加无消费者的索引。
## Acceptance criteria
- [ ] AC1: v9 upgrade 为当前 records 建立所需索引,v8→v9 六个 OneTalk store 的记录数量与内容保持不变;`oldVersion < 7` 仍按现有契约清空五个 legacy store。
- [ ] AC2: `storage.ts` 不再存在用于业务查询或会话删除的 `objectStore.getAll()`;所有目标读取均由主键或相应 `IDBIndex` 限定。
- [ ] AC3: 正常 ACK 不再调用 `listPendingCandidates` 来按已知 request key 找 candidate;乱序 ACK 下 `latestMessageId` 仍是 `candidateOrder` 最大的 confirmed candidate。
- [ ] AC4: 重连恢复、pending drain、awaiting-anchor 激活、completion、profile flush 和会话重建的结果与现有 focused tests 一致,且不读取其他账号/会话记录。
- [ ] AC5: 目标存储、profile、同步引擎测试以及 typecheck、build、全量 test 和 format check 通过。
## Risks and operational constraints
- 建立新索引需要在首次打开 v9 数据库时遍历现有记录,成本为一次性迁移;升级被阻塞时必须继续显式失败,不能伪造成功或回退到全表扫描。
- IndexedDB 版本不可降级。v9 已写入后,旧的 v8 扩展不能打开同一数据库;发布回滚必须使用前向 v10 修复或等待客户端更新,不能直接回退扩展包。
- 测试中的轻量 Fake IndexedDB 当前没有 `IDBIndex` 行为。测试 harness 需要扩展为最小的可观测 index 模型,或替换为能验证真实 index API 的测试依赖;不得仅靠 mock 返回值宣称查询走了索引。
@@ -0,0 +1,26 @@
{
"id": "onetalk-indexeddb-query-indexes",
"name": "onetalk-indexeddb-query-indexes",
"title": "修复 OneTalk IndexedDB 查询全表扫描",
"description": "为 OneTalk 同步账本设计索引、迁移与查询重构方案,消除 candidate、checkpoint 和 profile 的全表读取。",
"status": "in_progress",
"dev_type": null,
"scope": null,
"package": null,
"priority": "P2",
"creator": "ybf",
"assignee": "ybf",
"createdAt": "2026-09-14",
"completedAt": null,
"branch": "09-14-onetalk-indexeddb-query-indexes",
"base_branch": "main",
"worktree_path": null,
"commit": null,
"pr_url": null,
"subtasks": [],
"children": [],
"parent": null,
"relatedFiles": [],
"notes": "",
"meta": {}
}
@@ -222,11 +222,14 @@ export type OneTalkSyncStore = {
conversationId: string,
messageId: string,
) => Promise<OneTalkSyncCandidate | null>;
getCandidateByKey: (key: string) => Promise<OneTalkSyncCandidate | null>;
listCandidates: (
channelAccountId: string,
conversationId: string,
status?: OneTalkSyncCandidateStatus,
) => Promise<OneTalkSyncCandidate[]>;
listPendingCandidates: (channelAccountId?: string) => Promise<OneTalkSyncCandidate[]>;
listPendingCandidates: (channelAccountId: string) => Promise<OneTalkSyncCandidate[]>;
hasPendingCandidates: (channelAccountId: string, conversationId: string) => Promise<boolean>;
putCandidate: (candidate: OneTalkSyncCandidate) => Promise<void>;
updateCandidate: (
candidate: OneTalkSyncCandidate,
@@ -246,9 +249,7 @@ export type OneTalkContactProfileStore = {
channelAccountId: string,
conversationId: string,
) => Promise<OneTalkContactProfileLedgerRecord | null>;
listPendingProfiles: (
channelAccountId?: string,
) => Promise<OneTalkContactProfileLedgerRecord[]>;
listPendingProfiles: (channelAccountId: string) => Promise<OneTalkContactProfileLedgerRecord[]>;
hasProfileRecord: (channelAccountId: string) => Promise<boolean>;
putPendingProfile: (
channelAccountId: string,
@@ -337,6 +338,21 @@ const toStoredObservation = (
};
};
const ONE_TALK_MESSAGE_BY_ACCOUNT_CONVERSATION_INDEX = "by_account_conversation";
const ONE_TALK_CHECKPOINT_BY_ACCOUNT_INDEX = "by_account";
const ONE_TALK_CANDIDATE_BY_ACCOUNT_STATUS_INDEX = "by_account_status";
const ONE_TALK_CANDIDATE_BY_ACCOUNT_CONVERSATION_STATUS_INDEX = "by_account_conversation_status";
const ONE_TALK_CANDIDATE_BY_ACCOUNT_CONVERSATION_INDEX = "by_account_conversation";
const ONE_TALK_ANOMALY_BY_ACCOUNT_INDEX = "by_account";
const ONE_TALK_ANOMALY_BY_ACCOUNT_CONVERSATION_INDEX = "by_account_conversation";
const ONE_TALK_CONTACT_PROFILE_BY_ACCOUNT_INDEX = "by_account";
const ONE_TALK_CONTACT_PROFILE_BY_ACCOUNT_PENDING_OBSERVED_AT_INDEX =
"by_account_pending_observed_at";
const ensureIndex = (store: IDBObjectStore, name: string, keyPath: string | string[]): void => {
if (!store.indexNames.contains(name)) store.createIndex(name, keyPath);
};
const ensureSyncStores = (database: IDBDatabase): void => {
const stores = [
ONE_TALK_MESSAGE_STORE_NAME,
@@ -354,6 +370,49 @@ const ensureSyncStores = (database: IDBDatabase): void => {
}
};
const ensureSyncIndexes = (transaction: IDBTransaction): void => {
ensureIndex(
transaction.objectStore(ONE_TALK_MESSAGE_STORE_NAME),
ONE_TALK_MESSAGE_BY_ACCOUNT_CONVERSATION_INDEX,
["channelAccountId", "conversationId"],
);
ensureIndex(
transaction.objectStore(ONE_TALK_CHECKPOINT_STORE_NAME),
ONE_TALK_CHECKPOINT_BY_ACCOUNT_INDEX,
"channelAccountId",
);
const candidateStore = transaction.objectStore(ONE_TALK_CANDIDATE_STORE_NAME);
ensureIndex(candidateStore, ONE_TALK_CANDIDATE_BY_ACCOUNT_STATUS_INDEX, [
"channelAccountId",
"status",
]);
ensureIndex(candidateStore, ONE_TALK_CANDIDATE_BY_ACCOUNT_CONVERSATION_STATUS_INDEX, [
"channelAccountId",
"conversationId",
"status",
]);
ensureIndex(candidateStore, ONE_TALK_CANDIDATE_BY_ACCOUNT_CONVERSATION_INDEX, [
"channelAccountId",
"conversationId",
]);
ensureIndex(
transaction.objectStore(ONE_TALK_ANOMALY_STORE_NAME),
ONE_TALK_ANOMALY_BY_ACCOUNT_INDEX,
"channelAccountId",
);
ensureIndex(
transaction.objectStore(ONE_TALK_ANOMALY_STORE_NAME),
ONE_TALK_ANOMALY_BY_ACCOUNT_CONVERSATION_INDEX,
["channelAccountId", "conversationId"],
);
const profileStore = transaction.objectStore(ONE_TALK_CONTACT_PROFILE_STORE_NAME);
ensureIndex(profileStore, ONE_TALK_CONTACT_PROFILE_BY_ACCOUNT_INDEX, "channelAccountId");
ensureIndex(profileStore, ONE_TALK_CONTACT_PROFILE_BY_ACCOUNT_PENDING_OBSERVED_AT_INDEX, [
"channelAccountId",
"pending.observedAtMs",
]);
};
const clearLegacyOneTalkStores = (transaction: IDBTransaction): void => {
for (const storeName of [
ONE_TALK_MESSAGE_STORE_NAME,
@@ -377,6 +436,7 @@ const openSyncDatabase = (factory: IDBFactory): Promise<IDBDatabase> => {
if ((event?.oldVersion ?? 0) < 7 && request.transaction) {
clearLegacyOneTalkStores(request.transaction);
}
if (request.transaction) ensureSyncIndexes(request.transaction);
};
request.onsuccess = () => resolve(request.result);
request.onerror = () => reject(request.error ?? new Error("IndexedDB open failed"));
@@ -651,14 +711,36 @@ const readRequest = <T>(request: IDBRequest<T>): Promise<T> => {
});
};
const readAll = async <T>(database: IDBDatabase, storeName: string): Promise<T[]> => {
const readByIndex = async <T>(
database: IDBDatabase,
storeName: string,
indexName: string,
query: IDBValidKey | IDBKeyRange,
): Promise<T[]> => {
const transaction = database.transaction(storeName, "readonly");
const completion = transactionResult(transaction);
const result = await readRequest(transaction.objectStore(storeName).getAll());
const result = await readRequest(
transaction.objectStore(storeName).index(indexName).getAll(query),
);
await completion;
return result as T[];
};
const countByIndex = async (
database: IDBDatabase,
storeName: string,
indexName: string,
query: IDBValidKey | IDBKeyRange,
): Promise<number> => {
const transaction = database.transaction(storeName, "readonly");
const completion = transactionResult(transaction);
const result = await readRequest(
transaction.objectStore(storeName).index(indexName).count(query),
);
await completion;
return result;
};
const readOne = async <T>(
database: IDBDatabase,
storeName: string,
@@ -874,11 +956,12 @@ export const createOneTalkSyncStore = (
const listCheckpoints = async (channelAccountId: string): Promise<OneTalkSyncCheckpoint[]> => {
const database = await getDatabase();
const checkpoints = await readAll<OneTalkSyncCheckpoint>(
return readByIndex<OneTalkSyncCheckpoint>(
database,
ONE_TALK_CHECKPOINT_STORE_NAME,
ONE_TALK_CHECKPOINT_BY_ACCOUNT_INDEX,
channelAccountId,
);
return checkpoints.filter((checkpoint) => checkpoint.channelAccountId === channelAccountId);
};
const putCheckpoint = async (checkpoint: OneTalkSyncCheckpoint): Promise<void> => {
@@ -911,34 +994,56 @@ export const createOneTalkSyncStore = (
);
};
const getCandidateByKey = async (key: string): Promise<OneTalkSyncCandidate | null> => {
const database = await getDatabase();
return (
(await readOne<OneTalkSyncCandidate>(database, ONE_TALK_CANDIDATE_STORE_NAME, key)) ??
null
);
};
const listCandidates = async (
channelAccountId: string,
conversationId: string,
status?: OneTalkSyncCandidateStatus,
): Promise<OneTalkSyncCandidate[]> => {
const database = await getDatabase();
const candidates = await readAll<OneTalkSyncCandidate>(
return readByIndex<OneTalkSyncCandidate>(
database,
ONE_TALK_CANDIDATE_STORE_NAME,
);
return candidates.filter(
(candidate) =>
candidate.channelAccountId === channelAccountId &&
candidate.conversationId === conversationId,
status === undefined
? ONE_TALK_CANDIDATE_BY_ACCOUNT_CONVERSATION_INDEX
: ONE_TALK_CANDIDATE_BY_ACCOUNT_CONVERSATION_STATUS_INDEX,
status === undefined
? [channelAccountId, conversationId]
: [channelAccountId, conversationId, status],
);
};
const listPendingCandidates = async (
channelAccountId?: string,
channelAccountId: string,
): Promise<OneTalkSyncCandidate[]> => {
const database = await getDatabase();
const candidates = await readAll<OneTalkSyncCandidate>(
return readByIndex<OneTalkSyncCandidate>(
database,
ONE_TALK_CANDIDATE_STORE_NAME,
ONE_TALK_CANDIDATE_BY_ACCOUNT_STATUS_INDEX,
[channelAccountId, "pending_ack"],
);
return candidates.filter(
(candidate) =>
candidate.status === "pending_ack" &&
(channelAccountId === undefined || candidate.channelAccountId === channelAccountId),
};
const hasPendingCandidates = async (
channelAccountId: string,
conversationId: string,
): Promise<boolean> => {
const database = await getDatabase();
return (
(await countByIndex(
database,
ONE_TALK_CANDIDATE_STORE_NAME,
ONE_TALK_CANDIDATE_BY_ACCOUNT_CONVERSATION_STATUS_INDEX,
[channelAccountId, conversationId, "pending_ack"],
)) > 0
);
};
@@ -992,11 +1097,13 @@ export const createOneTalkSyncStore = (
conversationId?: string,
): Promise<OneTalkSyncAnomaly[]> => {
const database = await getDatabase();
const anomalies = await readAll<OneTalkSyncAnomaly>(database, ONE_TALK_ANOMALY_STORE_NAME);
return anomalies.filter(
(anomaly) =>
anomaly.channelAccountId === channelAccountId &&
(conversationId === undefined || anomaly.conversationId === conversationId),
return readByIndex<OneTalkSyncAnomaly>(
database,
ONE_TALK_ANOMALY_STORE_NAME,
conversationId === undefined
? ONE_TALK_ANOMALY_BY_ACCOUNT_INDEX
: ONE_TALK_ANOMALY_BY_ACCOUNT_CONVERSATION_INDEX,
conversationId === undefined ? channelAccountId : [channelAccountId, conversationId],
);
};
@@ -1015,24 +1122,20 @@ export const createOneTalkSyncStore = (
] as const;
const transaction = database.transaction([...storeNames], "readwrite");
const completion = transactionResult(transaction);
for (const storeName of storeNames) {
for (const [storeName, indexName] of [
[ONE_TALK_MESSAGE_STORE_NAME, ONE_TALK_MESSAGE_BY_ACCOUNT_CONVERSATION_INDEX],
[ONE_TALK_CANDIDATE_STORE_NAME, ONE_TALK_CANDIDATE_BY_ACCOUNT_CONVERSATION_INDEX],
[ONE_TALK_ANOMALY_STORE_NAME, ONE_TALK_ANOMALY_BY_ACCOUNT_CONVERSATION_INDEX],
] as const) {
const store = transaction.objectStore(storeName);
const request = store.getAll();
const request = store.index(indexName).getAllKeys([channelAccountId, conversationId]);
request.onsuccess = () => {
for (const record of request.result as Array<{
key: string;
channelAccountId: string;
conversationId?: string;
}>) {
if (
record.channelAccountId === channelAccountId &&
record.conversationId === conversationId
) {
store.delete(record.key);
}
}
for (const key of request.result) store.delete(key);
};
}
transaction
.objectStore(ONE_TALK_CHECKPOINT_STORE_NAME)
.delete(syncKey(channelAccountId, conversationId));
await completion;
};
@@ -1042,8 +1145,10 @@ export const createOneTalkSyncStore = (
putCheckpoint,
persistObservedBatch: persistBatch,
getCandidate,
getCandidateByKey,
listCandidates,
listPendingCandidates,
hasPendingCandidates,
putCandidate,
updateCandidate,
recordAnomaly,
@@ -1110,27 +1215,27 @@ export const createOneTalkContactProfileStore = (
};
const listPendingProfiles = async (
channelAccountId?: string,
channelAccountId: string,
): Promise<OneTalkContactProfileLedgerRecord[]> => {
const database = await getDatabase();
const records = await readAll<OneTalkContactProfileLedgerRecord>(
return readByIndex<OneTalkContactProfileLedgerRecord>(
database,
ONE_TALK_CONTACT_PROFILE_STORE_NAME,
);
return records.filter(
(record) =>
record.pending !== undefined &&
(channelAccountId === undefined || record.channelAccountId === channelAccountId),
ONE_TALK_CONTACT_PROFILE_BY_ACCOUNT_PENDING_OBSERVED_AT_INDEX,
IDBKeyRange.bound([channelAccountId], [channelAccountId, []], false, true),
);
};
const hasProfileRecord = async (channelAccountId: string): Promise<boolean> => {
const database = await getDatabase();
const records = await readAll<OneTalkContactProfileLedgerRecord>(
database,
ONE_TALK_CONTACT_PROFILE_STORE_NAME,
return (
(await countByIndex(
database,
ONE_TALK_CONTACT_PROFILE_STORE_NAME,
ONE_TALK_CONTACT_PROFILE_BY_ACCOUNT_INDEX,
channelAccountId,
)) > 0
);
return records.some((record) => record.channelAccountId === channelAccountId);
};
const putPendingProfile = async (
@@ -18,9 +18,7 @@ import {
candidateOrder,
candidateTimestamp,
isNonEmptyString,
latestConfirmedMessageId,
nowOr,
pendingCandidate,
scopedKey,
} from "./helpers.ts";
import { CheckpointCoordinator } from "./checkpoint-coordinator.ts";
@@ -59,7 +57,7 @@ export class AckCompletionCoordinator {
private readonly checkpoints: CheckpointCoordinator;
private readonly isBootstrapRunning: () => boolean;
private readonly requestCandidates = new Map<string, CandidateRequest>();
private readonly explicitlySentCandidates = new Set<string>();
private readonly explicitlySentConversations = new Set<string>();
private readonly discovered = new Set<string>();
private readonly pendingDiscoveries = new Map<
string,
@@ -89,6 +87,7 @@ export class AckCompletionCoordinator {
public resetConnection(): void {
this.requestCandidates.clear();
this.explicitlySentConversations.clear();
this.discovered.clear();
this.pendingDiscoveries.clear();
this.pendingIncompletes.clear();
@@ -97,15 +96,8 @@ export class AckCompletionCoordinator {
/** 清掉单会话的易失 ACK/discovery 状态,不能影响其它会话。 */
public clearConversation(conversationId: string): void {
const key = scopedKey(this.scope.channelAccountId, conversationId);
const candidatePrefix = `${JSON.stringify([
this.scope.channelAccountId,
conversationId,
]).slice(0, -1)},`;
this.discovered.delete(key);
this.explicitlySentCandidates.forEach((candidateKey) => {
if (candidateKey.startsWith(candidatePrefix))
this.explicitlySentCandidates.delete(candidateKey);
});
this.explicitlySentConversations.delete(key);
for (const [requestId, request] of this.requestCandidates) {
if (request.conversationId === conversationId) this.requestCandidates.delete(requestId);
}
@@ -153,9 +145,13 @@ export class AckCompletionCoordinator {
conversationId: string,
anchorMessageId: string | null,
): Promise<void> {
const candidates = await this.store.listCandidates(channelAccountId, conversationId);
const candidates = await this.store.listCandidates(
channelAccountId,
conversationId,
"awaiting_anchor",
);
const anchorCandidate = anchorMessageId
? candidates.find((candidate) => candidate.messageId === anchorMessageId)
? await this.store.getCandidate(channelAccountId, conversationId, anchorMessageId)
: undefined;
const anchorTime = anchorCandidate ? candidateTimestamp(anchorCandidate) : null;
@@ -295,9 +291,9 @@ export class AckCompletionCoordinator {
});
return;
}
const candidates = (await this.store.listCandidates(channelAccountId, conversationId))
.filter(pendingCandidate)
.sort(candidateOrder);
const candidates = (
await this.store.listCandidates(channelAccountId, conversationId, "pending_ack")
).sort(candidateOrder);
for (const candidate of candidates) {
if (this.lifecycle.isDisposed()) return;
if (!this.bright.isOnline()) return;
@@ -327,17 +323,16 @@ export class AckCompletionCoordinator {
if (!this.lifecycle.getHistoryGeneration(conversationId)) return;
const checkpoint = await this.checkpoints.getCheckpoint(channelAccountId, conversationId);
if (!checkpoint || !checkpoint.historyComplete || checkpoint.phase !== "uploading") return;
const candidates = await this.store.listCandidates(channelAccountId, conversationId);
if (candidates.some(pendingCandidate)) return;
const explicitCompletion = candidates.some((candidate) =>
this.explicitlySentCandidates.has(candidate.key),
if (await this.store.hasPendingCandidates(channelAccountId, conversationId)) return;
const explicitCompletion = this.explicitlySentConversations.has(
scopedKey(channelAccountId, conversationId),
);
if (!this.lifecycle.isBusinessFrameGateOpen() && !explicitStart && !explicitCompletion) {
return;
}
if (!this.bright.isOnline()) return;
if (checkpoint.completionSent && !forceResend) return;
const latestMessageId = latestConfirmedMessageId(candidates) ?? checkpoint.anchorMessageId;
const latestMessageId = checkpoint.latestMessageId ?? checkpoint.anchorMessageId;
const reportAnchorNotFound =
checkpoint.anomalyCode !== undefined &&
!(checkpoint.mode === "full" && checkpoint.anchorState === "not_found");
@@ -401,8 +396,7 @@ export class AckCompletionCoordinator {
): Promise<void> {
let candidate: OneTalkSyncCandidate | null = null;
if (requestKey) {
const all = await this.store.listPendingCandidates(this.scope.channelAccountId);
candidate = all.find((item) => item.key === requestKey) ?? null;
candidate = await this.store.getCandidateByKey(requestKey);
}
if (
!candidate &&
@@ -415,7 +409,7 @@ export class AckCompletionCoordinator {
result.messageId,
);
}
if (!candidate) return;
if (!candidate || candidate.status !== "pending_ack") return;
const nextStatus =
result.status === "accepted" || result.status === "duplicate"
? "confirmed"
@@ -423,18 +417,9 @@ export class AckCompletionCoordinator {
? "anomaly"
: "rejected";
await this.queue.enqueue(candidate.channelAccountId, candidate.conversationId, async () => {
const durableCandidate = await this.store.getCandidate(
candidate.channelAccountId,
candidate.conversationId,
candidate.messageId,
);
if (!durableCandidate) return;
const durableStatus =
durableCandidate.status === "confirmed" ||
durableCandidate.status === "anomaly" ||
durableCandidate.status === "rejected"
? durableCandidate.status
: nextStatus;
const durableCandidate = await this.store.getCandidateByKey(candidate.key);
if (!durableCandidate || durableCandidate.status !== "pending_ack") return;
const durableStatus = nextStatus;
await this.store.updateCandidate(durableCandidate, durableStatus);
traceOneTalkObservedMessages([durableCandidate.message], {
stage: "bright_ack",
@@ -467,13 +452,23 @@ export class AckCompletionCoordinator {
if (checkpointForAck !== checkpoint) {
await this.store.putCheckpoint(checkpointForAck);
}
const candidates = await this.store.listCandidates(
candidate.channelAccountId,
candidate.conversationId,
);
const previousLatest = checkpoint.latestMessageId
? await this.store.getCandidate(
candidate.channelAccountId,
candidate.conversationId,
checkpoint.latestMessageId,
)
: null;
const latestConfirmed = [
previousLatest?.status === "confirmed" ? previousLatest : null,
durableStatus === "confirmed" ? durableCandidate : null,
]
.filter((item): item is OneTalkSyncCandidate => item !== null)
.sort(candidateOrder)
.at(-1);
await this.checkpoints.checkpointLatestMessageId(
checkpointForAck,
latestConfirmedMessageId(candidates),
latestConfirmed?.messageId ?? null,
);
}
await this.tryComplete(candidate.channelAccountId, candidate.conversationId);
@@ -582,7 +577,9 @@ export class AckCompletionCoordinator {
);
if (sent) {
if (explicitStart)
candidates.forEach((candidate) => this.explicitlySentCandidates.add(candidate.key));
this.explicitlySentConversations.add(
scopedKey(first.channelAccountId, first.conversationId),
);
return true;
}
this.lifecycle.emitDiagnostic({
@@ -32,6 +32,10 @@ const profile = (
observationStatus: "confirmed",
});
globalThis.IDBKeyRange = {
bound: (lower, upper) => ({ lower, upper }),
};
class Request {
constructor(result, schedule = true) {
this.result = result;
@@ -47,6 +51,11 @@ class Store {
constructor(name) {
this.name = name;
this.records = new Map();
this.objectStoreGetAllCalls = 0;
this.indexNames = {
values: new Set(),
contains: (indexName) => this.indexNames.values.has(indexName),
};
}
get(key) {
@@ -54,9 +63,40 @@ class Store {
}
getAll() {
this.objectStoreGetAllCalls += 1;
return new Request([...this.records.values()]);
}
createIndex(name, keyPath) {
this.indexNames.values.add(name);
this.indexes ??= new Map();
this.indexes.set(name, keyPath);
}
index(name) {
const keyPath = this.indexes?.get(name);
if (keyPath === undefined) throw new Error(`Missing index: ${name}`);
const keyFor = (record) => {
const valueFor = (path) =>
path.split(".").reduce((value, segment) => value?.[segment], record);
return Array.isArray(keyPath) ? keyPath.map(valueFor) : valueFor(keyPath);
};
const matches = (record, query) => {
const key = keyFor(record);
if (query?.lower !== undefined && Array.isArray(key)) {
return key[0] === query.lower[0] && key.every((value) => value !== undefined);
}
return JSON.stringify(key) === JSON.stringify(query);
};
const matchingEntries = (query) =>
[...this.records.entries()].filter(([, record]) => matches(record, query));
return {
getAll: (query) => new Request(matchingEntries(query).map(([, record]) => record)),
getAllKeys: (query) => new Request(matchingEntries(query).map(([key]) => key)),
count: (query) => new Request(matchingEntries(query).length),
};
}
put(record) {
this.transaction.stage(this.records, record.key, record);
}
@@ -230,11 +270,44 @@ test("resets legacy OneTalk stores before v7 while opening the current database
});
test("reports whether an account has any durable profile ledger record", async () => {
const store = createOneTalkContactProfileStore(new Factory(), () => 100);
const factory = new Factory();
const store = createOneTalkContactProfileStore(factory, () => 100);
assert.equal(await store.hasProfileRecord("account-1"), false);
await store.putPendingProfile("account-1", profile());
assert.equal(await store.hasProfileRecord("account-1"), true);
assert.equal(await store.hasProfileRecord("account-2"), false);
assert.equal(
factory.database.stores.get(ONE_TALK_CONTACT_PROFILE_STORE_NAME).objectStoreGetAllCalls,
0,
);
});
test("lists only an account's pending profiles through the pending-observed index", async () => {
const factory = new Factory();
const store = createOneTalkContactProfileStore(factory, () => 100);
const pending = profile("pending-account-1", null, 100, "conversation-1");
const uploaded = profile("uploaded-account-1", null, 101, "conversation-2");
const otherAccount = profile("pending-account-2", null, 102, "conversation-3");
await store.putPendingProfile("account-1", pending);
await store.putPendingProfile("account-1", uploaded);
await store.markProfileUploaded({
channelAccountId: "account-1",
conversationId: uploaded.conversationId,
fingerprint: uploaded.profileFingerprint,
observedAtMs: uploaded.observedAtMs,
uploadedAt: 200,
});
await store.putPendingProfile("account-2", otherAccount);
assert.deepEqual(
(await store.listPendingProfiles("account-1")).map((record) => record.key),
[contactProfileKey("account-1", pending.conversationId)],
);
assert.equal(
factory.database.stores.get(ONE_TALK_CONTACT_PROFILE_STORE_NAME).objectStoreGetAllCalls,
0,
);
});
test("does not return true before the readwrite transaction commits", async () => {
@@ -35,6 +35,10 @@ class FakeRequest {
class FakeStore {
constructor() {
this.records = new Map();
this.indexNames = {
values: new Set(),
contains: (name) => this.indexNames.values.has(name),
};
}
put(record) {
@@ -49,6 +53,10 @@ class FakeStore {
return new FakeRequest([...this.records.values()].map((record) => structuredClone(record)));
}
createIndex(name) {
this.indexNames.values.add(name);
}
clear() {
this.records.clear();
}
@@ -56,6 +56,7 @@ class MemoryStore {
this.checkpoints = new Map();
this.candidates = new Map();
this.anomalies = new Map();
this.listPendingCandidateCalls = 0;
}
async putBatch() {}
@@ -137,15 +138,21 @@ class MemoryStore {
);
}
async listCandidates(channelAccountId, conversationId) {
async getCandidateByKey(key) {
return this.candidates.get(key) ?? null;
}
async listCandidates(channelAccountId, conversationId, status) {
return [...this.candidates.values()].filter(
(candidate) =>
candidate.channelAccountId === channelAccountId &&
candidate.conversationId === conversationId,
candidate.conversationId === conversationId &&
(status === undefined || candidate.status === status),
);
}
async listPendingCandidates(channelAccountId) {
this.listPendingCandidateCalls += 1;
return [...this.candidates.values()].filter(
(candidate) =>
candidate.status === "pending_ack" &&
@@ -153,6 +160,15 @@ class MemoryStore {
);
}
async hasPendingCandidates(channelAccountId, conversationId) {
return [...this.candidates.values()].some(
(candidate) =>
candidate.channelAccountId === channelAccountId &&
candidate.conversationId === conversationId &&
candidate.status === "pending_ack",
);
}
async putCandidate(candidate) {
const { requestId: _requestId, ...durableCandidate } = candidate;
this.candidates.set(candidate.key, { ...durableCandidate });
@@ -553,6 +569,150 @@ test("ignores an ACK with a stale generation even when its request has no echoed
);
});
test("resolves a request-keyed ACK without enumerating account pending candidates", async () => {
const store = new MemoryStore();
const bright = new FakeBright();
const engine = createOneTalkSyncEngine({
scope,
bright,
store,
pageRuntime: { routePageCommand: async () => ({ status: "completed" }) },
now: () => 500,
createRequestId: (kind) => `${kind}-request-keyed-ack`,
});
authenticate(bright, [{ conversationId: "conversation-1", latestMessageId: null }]);
await engine.ingestObservedBatch({
channelAccountId: scope.channelAccountId,
conversationId: "conversation-1",
observationSource: "live",
messages: [rawMessage("request-keyed-ack", 500)],
});
await waitFor(() => bright.messages.length === 1, "candidate was not sent");
store.listPendingCandidateCalls = 0;
bright.emit({
type: "message.ack",
requestId: bright.messages[0].requestId,
payload: {
status: "accepted",
conversationId: "conversation-1",
messageId: "request-keyed-ack",
},
});
await tick();
await tick();
assert.equal(
(await store.getCandidate(scope.channelAccountId, "conversation-1", "request-keyed-ack"))
.status,
"confirmed",
);
assert.equal(store.listPendingCandidateCalls, 0);
});
test("resolves every result in a batch ACK by its request key", async () => {
const store = new MemoryStore();
const bright = new FakeBright();
const engine = createOneTalkSyncEngine({
scope,
bright,
store,
pageRuntime: { routePageCommand: async () => ({ status: "completed" }) },
now: () => 500,
createRequestId: (kind) => `${kind}-batch-ack`,
});
authenticate(bright, [{ conversationId: "conversation-1", latestMessageId: null }]);
await engine.ingestObservedBatch({
channelAccountId: scope.channelAccountId,
conversationId: "conversation-1",
observationSource: "history",
mode: "full",
messages: [rawMessage("batch-first", 100), rawMessage("batch-second", 200)],
});
await waitFor(() => bright.batches.length === 1, "candidate batch was not sent");
store.listPendingCandidateCalls = 0;
bright.emit({
type: "messages.ack",
requestId: bright.batches[0].requestId,
payload: {
results: [
{
status: "accepted",
conversationId: "conversation-1",
messageId: "batch-first",
},
{
status: "duplicate",
conversationId: "conversation-1",
messageId: "batch-second",
},
],
},
});
await tick();
await tick();
assert.equal(
(await store.getCandidate(scope.channelAccountId, "conversation-1", "batch-first")).status,
"confirmed",
);
assert.equal(
(await store.getCandidate(scope.channelAccountId, "conversation-1", "batch-second")).status,
"confirmed",
);
assert.equal(store.listPendingCandidateCalls, 0);
});
test("keeps the candidate-order maximum checkpoint after out-of-order ACKs", async () => {
const store = new MemoryStore();
const bright = new FakeBright();
const engine = createOneTalkSyncEngine({
scope,
bright,
store,
pageRuntime: { routePageCommand: async () => ({ status: "completed" }) },
now: () => 500,
createRequestId: (kind) => `${kind}-${bright.messages.length}`,
});
authenticate(bright, [{ conversationId: "conversation-1", latestMessageId: null }]);
await engine.ingestObservedBatch({
channelAccountId: scope.channelAccountId,
conversationId: "conversation-1",
observationSource: "live",
messages: [rawMessage("first", 100)],
});
await engine.ingestObservedBatch({
channelAccountId: scope.channelAccountId,
conversationId: "conversation-1",
observationSource: "live",
messages: [rawMessage("second", 200)],
});
await waitFor(() => bright.messages.length === 2, "candidates were not sent");
for (const message of [...bright.messages].reverse()) {
bright.emit({
type: "message.ack",
requestId: message.requestId,
payload: {
status: "accepted",
conversationId: "conversation-1",
messageId: message.message.messageId,
},
});
await tick();
await tick();
}
assert.equal(
(await store.getCheckpoint(scope.channelAccountId, "conversation-1")).latestMessageId,
"second",
);
});
test("does not replace an anchor generation with a late discovery generation", () => {
const bright = new FakeBright();
const lifecycle = new OneTalkSyncLifecycle(
@@ -10,6 +10,7 @@ import {
ONE_TALK_CHECKPOINT_STORE_NAME,
ONE_TALK_CONTACT_PROFILE_STORE_NAME,
ONE_TALK_CONVERSATION_BOOTSTRAP_STORE_NAME,
ONE_TALK_SYNC_DATABASE_VERSION,
ONE_TALK_MESSAGE_STORE_NAME,
ONE_TALK_RENDERED_CARD_LEDGER_STORE_NAME,
createOneTalkRenderedCardLedgerStore,
@@ -43,6 +44,11 @@ class FakeRequest {
class FakeStore {
constructor() {
this.records = new Map();
this.objectStoreGetAllCalls = 0;
this.indexNames = {
values: new Set(),
contains: (name) => this.indexNames.values.has(name),
};
}
put(record) {
@@ -54,9 +60,43 @@ class FakeStore {
}
getAll() {
this.objectStoreGetAllCalls += 1;
return new FakeRequest([...this.records.values()].map((record) => structuredClone(record)));
}
createIndex(name, keyPath) {
this.indexNames.values.add(name);
this.indexes ??= new Map();
this.indexes.set(name, keyPath);
}
index(name) {
const keyPath = this.indexes?.get(name);
if (keyPath === undefined) throw new Error(`Missing index: ${name}`);
const keyFor = (record) => {
const valueFor = (path) =>
path.split(".").reduce((value, segment) => value?.[segment], record);
return Array.isArray(keyPath) ? keyPath.map(valueFor) : valueFor(keyPath);
};
const matches = (record, query) => {
const key = keyFor(record);
if (query?.lower !== undefined && Array.isArray(key)) {
return key[0] === query.lower[0];
}
return JSON.stringify(key) === JSON.stringify(query);
};
const matchingEntries = (query) =>
[...this.records.entries()].filter(([, record]) => matches(record, query));
return {
getAll: (query) =>
new FakeRequest(
matchingEntries(query).map(([, record]) => structuredClone(record)),
),
getAllKeys: (query) => new FakeRequest(matchingEntries(query).map(([key]) => key)),
count: (query) => new FakeRequest(matchingEntries(query).length),
};
}
clear() {
this.records.clear();
}
@@ -140,6 +180,7 @@ class FakeFactory {
ONE_TALK_CHECKPOINT_STORE_NAME,
ONE_TALK_ANOMALY_STORE_NAME,
ONE_TALK_CONTACT_PROFILE_STORE_NAME,
ONE_TALK_CONVERSATION_BOOTSTRAP_STORE_NAME,
]);
request.transaction = transaction;
transaction.oncomplete = () => request.onsuccess?.();
@@ -280,6 +321,46 @@ test("preserves existing stores while adding new stores from v7 and v8", async (
}
});
test("preserves v8 ledger records while creating v9 query indexes", async () => {
const factory = new FakeFactory(8);
for (const storeName of [
ONE_TALK_MESSAGE_STORE_NAME,
ONE_TALK_CANDIDATE_STORE_NAME,
ONE_TALK_CHECKPOINT_STORE_NAME,
ONE_TALK_ANOMALY_STORE_NAME,
ONE_TALK_CONTACT_PROFILE_STORE_NAME,
ONE_TALK_CONVERSATION_BOOTSTRAP_STORE_NAME,
]) {
factory.database.createObjectStore(storeName).put({ key: storeName, v8: true });
}
const store = createOneTalkSyncStore(factory, () => 500);
await store.listCheckpoints("account-1");
assert.equal(factory.database.version, ONE_TALK_SYNC_DATABASE_VERSION);
for (const storeName of factory.database.stores.keys()) {
assert.equal(factory.database.stores.get(storeName).records.get(storeName).v8, true);
}
assert.equal(
factory.database.stores
.get(ONE_TALK_CANDIDATE_STORE_NAME)
.indexNames.contains("by_account_status"),
true,
);
assert.equal(
factory.database.stores
.get(ONE_TALK_CONTACT_PROFILE_STORE_NAME)
.indexNames.contains("by_account_pending_observed_at"),
true,
);
assert.equal(
factory.database.stores
.get(ONE_TALK_ANOMALY_STORE_NAME)
.indexNames.contains("by_account_conversation"),
true,
);
});
test("clears only one conversation history ledger after its transaction commits", async () => {
const factory = new FakeFactory();
const store = createOneTalkSyncStore(factory, () => 500);
@@ -357,7 +438,16 @@ test("clears only one conversation history ledger after its transaction commits"
assert.deepEqual(await store.listAnomalies("account-1", "conversation-1"), []);
assert.notEqual(await store.getCheckpoint("account-1", "conversation-2"), null);
assert.equal((await store.listCandidates("account-1", "conversation-2")).length, 1);
assert.equal((await store.listPendingCandidates("account-1")).length, 1);
assert.equal((await store.listAnomalies("account-1")).length, 1);
for (const storeName of [
ONE_TALK_MESSAGE_STORE_NAME,
ONE_TALK_CHECKPOINT_STORE_NAME,
ONE_TALK_CANDIDATE_STORE_NAME,
ONE_TALK_ANOMALY_STORE_NAME,
]) {
assert.equal(factory.database.stores.get(storeName).objectStoreGetAllCalls, 0);
}
});
test("stores bootstrap markers by account and stable migration identifier", async () => {