DOC3_R11_3_ADDENDA_A_R2_2.md
Current Specs/DOC3/DOC3_R11_3_ADDENDA_A_R2_2.md
# DOC3 R11.3 — Semantic Skill Learning: Demonstration Mode, Graph Integration, and Universal Knowledge Extraction — Addenda A R2.2
**Status:** Consolidated revision incorporating R2 red team adjudication
**Date:** 2026-04-15
**Revision:** R2.1 (incorporates ADJ-01 through ADJ-72 from DOC3_R11_3_ADDENDA_A_R2_ADJUDICATION_CARD_R1)
**Target doc:** DOC3
**Related owners:** DOC72, DOC24, DOC1, DOC8, DOC11, EC Core, Q UI
**Supersedes:** `DOC3_R11_3_SEMANTIC_SKILL_LEARNING_R2.md`
**Adjudication sources:**
- `DOC3_R11_3_RED_TEAM_ADJUDICATION_CARD_V5_FINAL.md` — 133 items, 53 incorporated in R2.
- `DOC3_R11_3_ADDENDA_A_R2_ADJUDICATION_CARD_R1.md` — 72 items from R2 red team (6 reviewers, 2 rounds). 58 DOC3 items + 10 cross-doc items incorporated in this R2.1.
**Shared contract reference:** `DOC24_KNOWLEDGE_DELIVERY_SHARED_CONTRACTS_R2.md` — defines the DOC72-governed payload schemas that this pipeline fills. DOC3 does NOT redefine those schemas; it references them.
**Purpose:** Complete the `demonstrating_skill` entry mode and establish the universal pipeline through which ALL DOC3 learning modes produce graph-backed procedural knowledge. This proposal defines: how external app actions are observed (demonstration mode), how raw observations become semantic intent procedures stored as DOC72-governed shared knowledge contracts, how those procedures compound through the DOC72 entity graph, how they become routable through DOC24's delivery architecture via direct injection (not SKILL.md materialization), how MCP tools and demonstrated procedures compose at execution time, how SKILL.md files relate to graph procedure nodes, and how this pipeline generalizes to coaching, autonomous practice, import, skill mining, and retrospective learning.
---
## Adjudication Traceability
Items marked with **[Source: XX]** reference specific adjudication entries. The full adjudication context is in `DOC3_R11_3_RED_TEAM_ADJUDICATION_CARD_V5_FINAL.md`.
---
## 1. Executive Summary
R11.2 gave us the `LearnSession` lifecycle, the `demonstrating_skill` entry mode, the observation mode taxonomy, the proposal/install pipeline, and the route family. What R11.2 did not specify:
- How external app observation mechanically works (the capture layer)
- What raw observation events look like
- How raw events become semantic intent procedures stored in DOC72-governed shared knowledge contracts
- How demonstration output writes to the DOC72 entity graph through EC's `entity_knowledge_write` command
- How demonstrated procedures become routable through DOC24's direct injection pipeline (not SKILL.md)
- How concurrent user narration enriches interpretation
- How Quick Promote maps onto the existing state machine with proper scope limitation
- How demonstrated knowledge compounds across future sessions
- How the extraction bundle pipeline generalizes to all learning modes (coaching, practice, import, mining)
- How DOC24 tools and MCP tools compose with demonstrated procedures at execution time via the execution-strategy cache
- How SKILL.md files relate to graph procedure nodes under the two-materialization-kind model
- What the three-stage dedup algorithm looks like with hard vetoes and weighted scoring
- How observation fidelity, streaming interpretation, and trust mode improve the UX
- How confidence events, error codes, and the normative end-to-end sequence provide inspectability
This proposal specifies all of that. The user activates demonstration mode, performs a workflow in any external app, optionally narrates what they're doing and why, and the system captures, interprets, and promotes the result into graph-backed procedural knowledge that compounds across skills and contexts. The same pipeline, with different input sources, applies to every DOC3 learning mode.
**Key architectural pivot (from adjudication):** Graph-backed procedures execute through DOC24 direct injection into the LLM's context, not through SKILL.md materialization. The interpretation pipeline fills DOC72-governed shared knowledge contracts. DOC24 renders injection cards from those contracts. SKILL.md is retained only for native/imported skills not yet graph-ingested, and as an optional export format.
---
## 2. Architectural Principles
### 2.1 This is an enhancement, not a new system
R11.3 does not introduce a new runtime object, a new state machine, or a new route family. It specifies the operational behavior of `entry_mode: "demonstrating_skill"` within the existing `LearnSession` lifecycle.
### 2.2 Semantic intent over mechanical replay
The observed UI events are ephemeral evidence. The durable knowledge is semantic intent: "insert a table of contents with automatic heading detection" — never "click References tab, click Table of Contents dropdown, select Automatic Table 1." The LLM bridges intent to current UI at runtime. This is DOC72 §4.2's architectural decision and it is non-negotiable for demonstrated procedures.
### 2.3 Demonstrations produce bundles, not single procedures
A single demonstration may teach multiple procedures (atomic and composite), preferences, constraints, standing procedure candidates, vocabulary mappings, and goal links. The canonical output is a `KnowledgeExtractionBundle` — a structured multi-output artifact.
### 2.4 The DOC72 graph handles all storage
No new node types, no new tables, no new categories. Demonstrated procedures are `procedure` nodes. Demonstrated preferences are `memory_directive` nodes. Goal links are edges. Execution traces are `execution_trace` nodes. The graph's combination of typed nodes, flexible JSON payloads stored as DOC72-governed shared knowledge contracts, and open-ended edge relation types handles the infinite variety of granular procedural knowledge without schema changes.
### 2.5 Observation infrastructure exists
OpenClaw's macOS companion app already holds the TCC permissions for Accessibility, Screen Recording, and input monitoring. PeekabooBridge provides GUI automation and screenshot reading. The MCP Automator provides AppleScript/click integration. The Accessibility bridge can query native UI element trees. The same infrastructure that enables OpenClaw to *control* external apps enables ELNOR to *observe* them. ELNOR adds the interpretation and knowledge-writing layer on top.
### 2.6 No silent durable writes of high-impact knowledge
Quick Promote creates private experimental abilities fast, but:
- All graph writes carry provenance, confidence, and environment scope
- Memory directive candidates remain proposal-governed through DOC1 unless the user explicitly confirms
- Standing procedure candidates require explicit user approval before activation
- Quick Promote installs PROCEDURES ONLY — memory directives, standing procedures, and goals are shunted to Pending Abilities (§8.4)
### 2.7 Shared knowledge contracts are the canonical payload
**[Source: A1, Problem 4, SR-19]**
The interpretation pipeline fills `ProcedureKnowledgeContractSchema` (defined in KDA R2 §2.3, governed by DOC72). The graph stores it as the procedure node's `payload` JSON. DOC24 reads it for rendering. One schema, three consumers. DOC3 does NOT redefine the contract — it instructs the interpretation LLM to produce it and validates the result before graph write.
### 2.8 One context authority
**[Source: SR-36]**
DOC24's knowledge delivery pipeline feeds DOC10/DOC11/OpenClaw's existing prompt assembly chain. DOC24 produces rendered knowledge cards. It does NOT create a parallel prompt assembler that bypasses DOC10/DOC11.
### 2.9 Canonical payload stores semantic knowledge only
**[Source: SR-19]**
Execution bindings (tool directives, invalidation flags, resolution timestamps) live in DOC24's derived execution-strategy cache (KDA R2 §3.5), not in the canonical DOC72 payload. The canonical payload stores durable semantic knowledge only.
### 2.10 Procedures are linear by default
**[Source: SR-17, SR-31]**
The `steps` array is ordered and sequential. Branching logic is modeled as separate atomic procedures linked via `requires_preceding_procedure` or governed by DOC23 Tasks. Forward-compatible `execution_topology` and `decision_points` fields exist but are empty for v1.
---
## 3. User Experience — End State
### 3.1 Primary flow
1. User triggers demonstration mode via Learn page ("Demonstrating Skill" card), contextual shortcut ("Watch My Actions"), or conversational command ("Elnor, watch how I do this").
2. **Capture preflight** runs: system checks adapter availability for the target app, verifies TCC permissions, and reports predicted observation quality. **[Source: Patch 1, H2]**
3. **Screenshot consent flow** — if screenshots will be captured, the user sees: "This demonstration will capture screenshots of [Word]. Screenshots are used for interpretation only and are purged after 24 hours. [Enable screenshots] [Observation only — no screenshots]." **[Source: Grok Stress #6]**
4. `LearnSession` enters `armed` state. Persistent observation banner appears. Menu bar status item shows "Elnor is ready to observe."
5. **App focus detection** — system waits for the user to focus a real application (ignores Finder, System Preferences, etc.). On first real app focus, session transitions to `capturing` and target app is set automatically. **[Source: Problem 1]**
6. Observation layer captures raw UI events through the Accessibility bridge and supplementary sources. UI shows structural labels for **major events only** (significance filtering). **[Source: D1]**
7. User optionally narrates concurrent context via **voice (push-to-talk)** or typed narration in Q. Silent demonstration mode is fully supported — interpretation adapts to narration absence. **[Source: Problem 2, Grok Stress #1]**
8. User stops the demonstration.
9. System runs semantic interpretation (LLM-assisted) on the buffered events + narration. **Streaming partial results** via SSE as procedures are extracted. **[Source: D3]**
10. User reviews the `KnowledgeExtractionBundle` in a **summary/detail panel**: procedures learned, preferences captured, goal links inferred. **[Source: D2]**
11. User corrects, accepts, or rejects individual items using **full bundle editing** (step-level editing, splitting, merging, reclassification). **[Source: D4]**
12. User clicks Quick Promote (fast path — procedures only) or submits to full review pipeline (standard path). **Trust Mode** available for experienced users: auto-accepts procedures ≥ 0.80 confidence. **[Source: Patch 3, Grok Stress #2, SR-02]**
13. Graph writes occur via EC's `entity_knowledge_write` command. Procedures stored as `ProcedureKnowledgeContractSchema` payloads. EC validates payloads against `NodePayloadValidatorRegistry` before write. **[Source: Patch 5, SR-32]**
14. Install saga runs: `graph_written → render_contract_validated → execution_strategy_cached → availability_refreshed → routable`. **[Source: Patch 6, SR-16, SR-25]**
15. Ability availability refreshes. The demonstrated procedure is immediately discoverable through DOC24's routing cascade via direct injection.
### 3.2 Example: Creating a caption page for a brief
User: "Elnor, watch how I make a caption page."
[System: Capture preflight passes for Word. Screenshot consent shown and accepted. Session armed. User focuses Word — target app detected, capturing begins.]
[User works in Word: selects case caption text, sets font to Times New Roman 12pt, applies all-caps, centers court name, left-aligns parties, adjusts margins]
[User narrates via push-to-talk: "I always do Times New Roman for court filings." ... "Never use the automatic page numbering for the appendix, it screws up the exhibit numbering." ... "This is for when I need to make a document filing-ready."]
User: "Stop."
[Streaming interpretation begins — partial results appear within 15 seconds for this simple workflow]
Elnor presents (summary view):
> **Procedures learned:**
> - "Create caption page for brief" (composite — 5 steps) ✓
> 1. Select case caption text on first page
> 2. Set font to Times New Roman
> 3. Set font size to 12pt
> 4. Apply all-caps formatting to case caption
> 5. Set paragraph alignment (center for court name, left for parties)
>
> **Preferences captured:** (→ Pending Abilities)
> - "Use Times New Roman for court filings" (constraint, `assertion_class: "constraint"`)
> - "Do not use automatic page numbering for appendices" (constraint, `assertion_class: "constraint"`)
>
> **Goal link:**
> - All procedures linked to "Make document filing-ready"
>
> **Application:** Microsoft Word, macOS desktop
> **Observation fidelity:** 0.82 (green)
>
> **Quick Promote available** — create as private experimental ability?
> _This installs the procedure now. Preferences, constraints, and standing automations stay in Pending Abilities._
### 3.3 Non-goal
This is not a full no-code workflow builder UI. It is not a replacement for DOC23 task pipelines. The purpose is fast teaching and capture that feeds the intelligence substrate.
---
## 4. Observation Layer — How Capture Works
### 4.1 Observation adapter sources
The observation layer uses OpenClaw's existing macOS infrastructure through a set of adapters. All adapters feed structured events to EC through a unified event pipeline.
```ts
// packages/contracts/src/learning/observation-adapter.ts
// Extension of §0C.22 — adds demonstration-mode operational detail
export const DemonstrationAdapterSourceSchema = z.enum([
"accessibility_bridge", // macOS AX API via PeekabooBridge — primary structured source
"screenshot_vision", // Periodic screenshots interpreted by vision model — fallback
"applescript_state", // On-demand state queries for scriptable apps — supplementary, ADVISORY ONLY
"mcp_receipt", // Structured receipts from MCP tool calls — inside_agent
"browser_cdp", // Chrome DevTools Protocol events — OpenClaw's managed browser
"browser_dom", // [R2.2] Injected content script in Q embedded browser
"browser_extension", // [R2.2] OpenClaw Chrome extension content script
"wrapper_receipt", // Receipts from OpenClaw skill/wrapper executions — inside_agent
]);
```
### 4.2 Accessibility bridge — primary capture mechanism
The macOS Accessibility API (AXUIElement) provides structured, semantic UI events across all native applications. OpenClaw's macOS companion app already holds the Accessibility TCC permission. PeekabooBridge / Clawd Cursor's accessibility bridge already queries UI element trees by role, label, and value.
For demonstration mode, the accessibility bridge operates in **passive observation mode** — listening for AX notifications rather than performing actions.
**AX notifications consumed:**
| AX Notification | What it captures | Example |
|---|---|---|
| `AXMenuItemSelected` | Menu navigation and selection | References → Table of Contents → Automatic Table 1 |
| `AXValueChanged` | Text field, slider, picker changes | Font Size → 12 |
| `AXFocusedUIElementChanged` | Focus transitions between elements | Cursor moved to heading paragraph |
| `AXWindowCreated` / `AXWindowMoved` | Window lifecycle | Dialog "Insert Table of Contents" opened |
| `AXSelectedTextChanged` | Text selection changes | User selected case caption text |
| `AXTitleChanged` | Document/window title changes | "Henderson Brief.docx" |
| `AXSheetCreated` | Modal sheets and dialogs | Print dialog opened |
| `AXApplicationActivated` | App switching | User switched from Word to Excel |
**Adapter contract for accessibility bridge:**
```ts
// packages/contracts/src/learning/ax-bridge-adapter.ts
import { z } from "zod";
export const AXBridgeConfigSchema = z.object({
adapter_id: z.literal("accessibility_bridge"),
target_app_bundle_ids: z.array(z.string().max(120)), // ["com.microsoft.Word"] — plural for multi-app [Source: C4]
capture_notifications: z.array(z.string().max(80)), // Which AX notifications to subscribe to
poll_interval_ms: z.number().int().min(100).max(5000).default(500), // For state polling
ignore_element_roles: z.array(z.string().max(40)).default([
"AXScrollBar", "AXSplitter", "AXGrowArea", // Noise suppression
]),
max_events_per_second: z.number().int().min(1).max(50).default(20), // Rate limiting
schema_version: z.literal(1),
});
```
### 4.3 Screenshot + vision — state-aware fallback
**[Source: C2]**
For applications where AX events are sparse or unhelpful (some Electron apps, canvas-based UIs, heavily custom controls), the observation layer takes periodic screenshots interpreted by a vision-capable model.
**State-aware escalation** replaces the simple "<1 AX event / 5s" threshold:
```ts
// packages/contracts/src/learning/screenshot-adapter.ts
import { z } from "zod";
export const ScreenshotAdapterConfigSchema = z.object({
adapter_id: z.literal("screenshot_vision"),
capture_interval_ms: z.number().int().min(1000).max(10000).default(3000),
target_app_bundle_ids: z.array(z.string().max(120)),
crop_to_app_window: z.boolean().default(true),
vision_model: z.string().max(80).default("gemini-2.5-pro"),
max_screenshots_per_session: z.number().int().min(10).max(500).default(100),
redaction_regions: z.array(z.object({
x: z.number(), y: z.number(), width: z.number(), height: z.number(),
})).default([]),
schema_version: z.literal(1),
});
// State-aware escalation — replaces simple event-count threshold [Source: C2]
export function shouldEscalateToVision(state: CaptureEscalationState): boolean {
// No escalation if app is showing progress indicator or busy state
if (state.app_busy_indicator_visible) return false;
// No escalation if target app is not currently focused
if (!state.target_app_focused) return false;
// No escalation if user has not actively inputted recently (idle)
if (state.ms_since_last_user_input > 10_000) return false;
// Cap consecutive near-identical frames (max 3-5)
if (state.consecutive_similar_frames >= 3) return false;
// Escalate if AX events are sparse while user is actively working
return state.meaningful_ax_events_last_5s < 1;
}
```
### 4.4 AppleScript state queries — advisory only
**[Source: C3]**
For apps with AppleScript/JXA scripting dictionaries (Word, Excel, Outlook, Safari), the observation layer makes on-demand state queries. **AppleScript queries are `advisory_state_probe` only — they must NOT be the sole source of truth for any semantic bundle field.**
```ts
// packages/contracts/src/learning/applescript-adapter.ts
import { z } from "zod";
export const AppleScriptAdapterConfigSchema = z.object({
adapter_id: z.literal("applescript_state"),
target_app_bundle_id: z.string().max(120),
query_on_ax_event: z.boolean().default(true),
query_interval_ms: z.number().int().min(1000).max(10000).default(2000),
queries: z.array(z.object({
query_id: z.string().max(80),
script: z.string().max(2000),
description: z.string().max(200),
stability_class: z.enum(["stable", "fragile"]).default("stable"), // [Source: C3]
})),
schema_version: z.literal(1),
});
// Fallback matrix per state type [Source: C3]
// | State needed | AppleScript available | Fallback |
// |---------------------|-----------------------|-----------------------------------|
// | Current font | Yes (stable) | AX value on font dropdown |
// | Selection content | Yes (fragile) | AX selected text change |
// | Document name | Yes (stable) | AX window title |
// | Paragraph style | Yes (fragile) | Screenshot + vision interpretation |
// | Page count | Yes (stable) | No fallback — omit from bundle |
// Auto-disable consistently failing queries [Source: C3]
// If a query fails >50% of attempts across 5+ invocations, disable it for the session.
```
### 4.5 Per-app adapter profiles
**[Source: C1]**
AX support varies dramatically by app. The adapter manager uses per-app profiles to select the right combination:
| App | AX Quality | Primary Adapter | Secondary | Screenshot Interval | Structural Label Rules |
|---|---|---|---|---|---|
| Word | High | accessibility_bridge | applescript_state | On demand only | Menu paths, font changes, style names |
| Bitwig | Low | screenshot_vision | keyboard_shortcut, midi, accessibility_bridge | 2000ms | Visual diff primary, shortcut secondary |
| Chrome | Medium | browser_cdp (if managed), else accessibility_bridge | screenshot_vision | 3000ms | DOM events (managed), AX (unmanaged) |
| Acrobat | Medium | accessibility_bridge | screenshot_vision | 2500ms | Page navigation, annotation tools |
| Default | Unknown | accessibility_bridge | screenshot_vision | 5000ms | Generic AX labels |
**Low-AX-quality app warning:** When the profile indicates low AX quality, the system shows: "Bitwig has limited observation support. Please describe your actions as you work." **[Source: C1]**
### 4.6 Adapter selection, composition, and multi-app support
**[Source: C4]**
Multiple adapters operate simultaneously during a single demonstration session. Sessions support up to **5 concurrent target apps** for cross-app workflows.
```ts
// apps/ec-service/src/learning/demonstration-adapter-manager.ts
import { z } from "zod";
export const AdapterSelectionResultSchema = z.object({
session_id: z.string().uuid(),
target_app_bundle_ids: z.array(z.string().max(120)).max(5), // Plural — multi-app [Source: C4]
selected_adapters: z.record(
z.string(), // app_bundle_id
z.array(DemonstrationAdapterSourceSchema),
),
adapter_configs: z.record(z.string(), z.unknown()),
primary_adapter_per_app: z.record(z.string(), DemonstrationAdapterSourceSchema),
fallback_strategy: z.enum([
"ax_primary_screenshot_fallback",
"screenshot_primary",
"ax_plus_applescript",
"narration_only",
]),
schema_version: z.literal(1),
});
// Selection logic:
// 1. Check per-app adapter profile (§4.5) — use known-good configuration
// 2. If no profile: check if app has known AppleScript dictionary → add applescript_state
// 3. Always add accessibility_bridge (works for all native Mac apps)
// 4. If app is known to have poor AX support → add screenshot_vision
// 5. If app is OpenClaw's managed browser → use browser_cdp instead of AX
// 6. If no observation adapters available → fall back to narration_only
// Multi-app adapter switching [Source: C4]:
// Adapter manager listens for AXApplicationActivated notifications
// When user switches to a different target app, activate that app's adapters
// Raw events carry app_bundle_id for attribution
```
### 4.7 Raw observation event schema
All adapter sources produce events that conform to a unified raw observation event schema:
```ts
// packages/contracts/src/learning/raw-observation-event.ts
import { z } from "zod";
import { DemonstrationAdapterSourceSchema } from "./observation-adapter";
export const RawObservationEventSchema = z.object({
event_id: z.string().uuid(),
session_id: z.string().uuid(),
adapter_source: DemonstrationAdapterSourceSchema,
timestamp: z.string().datetime(),
// Application context — per-event attribution for multi-app sessions [Source: C4]
app_bundle_id: z.string().max(120).optional(),
app_name: z.string().max(80).optional(),
window_title: z.string().max(200).optional(),
// From accessibility bridge
ax_notification: z.string().max(80).optional(),
element_role: z.string().max(60).optional(),
element_label: z.string().max(240).optional(),
element_value: z.string().max(500).optional(),
element_value_before: z.string().max(500).optional(),
menu_path: z.array(z.string().max(120)).optional(),
parent_element_label: z.string().max(240).optional(),
keyboard_shortcut: z.string().max(40).optional(),
// From screenshot + vision
screenshot_ref: z.string().max(240).optional(),
visual_interpretation: z.string().max(1000).optional(),
visual_diff_from_previous: z.string().max(500).optional(),
// From AppleScript state query
script_query_id: z.string().max(80).optional(),
script_result: z.string().max(1000).optional(),
// From MCP/wrapper receipts
receipt_ref: z.string().max(240).optional(),
// Event significance classification [Source: D1]
significance: z.enum(["major", "minor", "noise"]).default("minor"),
// Undo detection [Source: Problem 3]
is_undo_event: z.boolean().default(false),
undoes_event_id: z.string().uuid().optional(),
// Metadata
caused_state_change: z.boolean().default(false),
structural_label: z.string().max(400),
redaction_mode: z.enum(["none", "mask_text", "hash_only"]).default("mask_text"),
schema_version: z.literal(1),
});
```
### 4.8 Event significance classification
**[Source: D1]**
The raw structural label stream is overwhelming during capture. Events are classified by significance:
- **`major`** — meaningful user actions that change document state: menu selections, value changes, keyboard shortcuts that trigger commands. Shown in the live capture feed by default.
- **`minor`** — focus changes, window moves, scroll events. Hidden by default, shown on toggle.
- **`noise`** — scrollbar movements, splitter drags, grow area interactions. Never shown in UI — only buffered for interpretation context.
Classification is deterministic from the event data (no LLM call):
```ts
// apps/ec-service/src/learning/event-significance.ts
export function classifySignificance(event: RawObservationEvent): "major" | "minor" | "noise" {
if (event.is_undo_event) return "major"; // Undos are always significant [Source: Problem 3]
if (event.menu_path?.length) return "major";
if (event.keyboard_shortcut) return "major";
if (event.caused_state_change) return "major";
if (event.element_value !== undefined && event.element_value_before !== undefined) return "major";
if (event.ax_notification === "AXFocusedUIElementChanged") return "minor";
if (event.ax_notification === "AXWindowMoved") return "noise";
if (["AXScrollBar", "AXSplitter", "AXGrowArea"].includes(event.element_role ?? "")) return "noise";
return "minor";
}
```
### 4.9 Structural label generation
The `structural_label` is computed from the raw event data without any LLM call. Multi-app sessions prefix with app name:
```ts
// apps/ec-service/src/learning/structural-label.ts
export function computeStructuralLabel(event: RawObservationEvent, isMultiApp: boolean): string {
const prefix = isMultiApp && event.app_name ? `[${event.app_name}] ` : ""; // [Source: C4]
if (event.is_undo_event) {
return `${prefix}⎌ Undo`; // [Source: Problem 3]
}
if (event.menu_path?.length) {
return `${prefix}Menu: ${event.menu_path.join(" → ")}`;
}
if (event.keyboard_shortcut) {
return `${prefix}Shortcut: ${event.keyboard_shortcut}${event.element_label ? ` (${event.element_label})` : ""}`;
}
if (event.element_value !== undefined && event.parent_element_label) {
const before = event.element_value_before ? `${event.element_value_before} → ` : "";
return `${prefix}${event.parent_element_label}: ${before}${event.element_value}`;
}
if (event.ax_notification === "AXFocusedUIElementChanged" && event.element_label) {
return `${prefix}Focus: ${event.element_label} (${event.element_role || "unknown"})`;
}
if (event.visual_interpretation) {
return `${prefix}[vision] ${event.visual_interpretation.slice(0, 200)}`;
}
if (event.script_query_id && event.script_result) {
return `${prefix}[state] ${event.script_query_id}: ${event.script_result}`;
}
return `${prefix}${event.adapter_source}: ${event.ax_notification || "event"}`;
}
```
### 4.10 App focus detection after arming
**[Source: Problem 1]**
When a session enters `armed` state, the system watches for `AXApplicationActivated` notifications to detect which app the user switches to. System UI apps (Finder, System Preferences, Dock, Notification Center) are ignored. The first real app focused becomes the session target:
```ts
// apps/ec-service/src/learning/app-focus-detector.ts
const SYSTEM_UI_BUNDLE_IDS = [
"com.apple.finder",
"com.apple.systempreferences",
"com.apple.dock",
"com.apple.notificationcenterui",
"com.apple.controlcenter",
];
export function isTargetableApp(bundleId: string): boolean {
return !SYSTEM_UI_BUNDLE_IDS.includes(bundleId);
}
```
On detection, emit SSE event `demonstrate.target_app_detected` with the app name and bundle ID.
**[ADJ-66]** Fullscreen backup: also listen for `AXFocusedWindowChanged` when `AXApplicationActivated` is not received within 2 seconds of a Mission Control interaction. Some fullscreen macOS apps suppress `AXApplicationActivated`.
### 4.10A App version change during active capture
**[ADJ-42]** If the target app's version changes during an active `capturing` session (detected via `AXTitleChanged`, process version query, or adapter restart), emit `demonstrate.app_version_changed_mid_session` SSE event. Auto-stop capture with recoverable warning: "App updated during observation — please restart demonstration." Session transitions to `stopped` with partial events preserved for interpretation.
### 4.11 Undo detection preprocessing
**[Source: Problem 3]**
Before interpretation, the event buffer is preprocessed to detect undo sequences:
- Detect ⌘Z keystrokes and `Edit → Undo` menu selections
- Mark the undo event with `is_undo_event: true`
- If possible, identify the event being undone via value-before/value-after matching and set `undoes_event_id`
- Add to interpretation prompt: "Events marked as `is_undo_event: true` represent mistakes — do not include the undone action in procedures. The corrected action that follows the undo IS the intended behavior."
### 4.12 Event buffer management
**[Source: H2]**
```ts
const EVENT_BUFFER_POLICY = {
max_events_per_session: 500,
overflow_strategy: "keep_bookends_summarize_middle",
// On overflow: keep first 50 + last 200 events, summarize middle as digest
overflow_summary_model: "gemini-2.5-flash", // Cheap model for summarization
};
```
### 4.13 Privacy and retention rules
All rules from DOC3 §0C.22 apply, plus:
1. Observation mode is opt-in (explicit user action).
2. Persistent UI banner visible while observation is active.
3. Raw observations are proposal inputs, NOT canonical memory.
4. `text_entry` adapter (if used): `mask_text` by default. `none` requires explicit per-session consent.
5. `clipboard` adapter (if used): capture event class only, NOT raw payload.
6. **[ADJ-01]** Raw observation events and screenshots MUST be purged strictly 24 hours after the session reaches a non-blocking terminal state (`installed_private`, `approved`, `rejected`, `cancelled`).
7. User can view and delete captured observations before proposal generation.
8. **[ADJ-65]** Screenshots captured by `screenshot_vision` are cropped to the target app window, stored ephemerally, and MUST be purged within 24 hours of session terminal state (inheriting the raw-event TTL), regardless of session metadata retention.
9. **Per-session screenshot consent flow:** User is prompted before screenshots are captured. Declining limits observation to AX + narration only. **[Source: Grok Stress #6]**
10. **[ADJ-01]** Masked evidence digest MAY be retained for 30 days for provenance queries only. Off-target events (from non-target apps) MUST be dropped immediately and MUST NOT be persisted.
11. **[ADJ-04]** Menu bar status item includes a 'Pause Screenshots' toggle (distinct from Pause All). When screenshots are paused mid-session, the screenshot adapter stops but AX and narration continue. Multi-app observation: when the active window matches a privileged app class (mail inbox, DMS viewer, password manager), screenshot capture MUST auto-disable for that window unless the user performs a one-time elevated consent for that specific app family.
12. **[ADJ-67]** Optional `strict_privacy_mode: boolean` session flag. When enabled: raw events are purged immediately after interpretation completes (not after 24h). Screenshots are held in memory only during capture (never written to disk) and discarded after vision model processing.
---
---
### 4.5A Browser Adapter Profiles (R2.2)
**[Source: R2.2 Proposal §1]**
Browser observation uses DOM events instead of AX accessibility events. Two adapter paths, two observation mechanisms that work together. Observation auto-follows user focus across tabs — no pre-selection needed.
#### 4.5A.1 Path A: Q Embedded Browser (DOC20 §6.19)
When a DOC3 learning session is active and the user interacts with the Q embedded browser:
**Dual observation mechanism** (per DOC20 Q Browser Intake Architecture §5):
1. **CDP (Chrome DevTools Protocol)** — existing `browser_cdp` adapter source (§4.1). Provides page-level events: navigation, DOM mutations, network requests.
2. **Injected content script** — new `browser_dom` adapter source. Q injects a user interaction observation script into the active webview via `webview.executeJavaScript(observationScript)`. Captures user actions (clicks, typing, form submissions) with immediate access to target element semantic labels. Forwarded to EC via Electron IPC channel `learn:browser-event`.
CDP handles page-level events. The injected script handles user interaction events. Both feed into the same `RawObservationEventSchema` pipeline.
**Semantic label resolution** (injected script, per-event): `aria-label` → `placeholder` → visible text content (truncated to 120 chars) → `name` attribute → `title` attribute → `role` + tag name → "element"
**Incognito gate (per-tab).** When the user activates a tab whose webview partition is `q-browser-private` (the non-persistent partition per DOC20 R4.3 §6.19.11B), the observation script MUST NOT inject for that tab. If the user switches into an incognito tab during an active learning session, the live action feed shows "🔒 Observation paused — incognito tab" and resumes when the user switches to a non-incognito tab. The session itself is not terminated. If ALL open tabs are incognito when the user clicks "Teach [agent name]", the Learn button shows disabled state with tooltip: "No observable tabs — all tabs are incognito."
**BrowserPageVisitSchema flag:** When observation activates for a page, set `user_interactions.demonstration_mode_used = true` on the visit record for that page.
**Safety rules:** The observation script MUST NOT: capture `input[type=password]` field values, capture content from cross-origin iframes, modify the page DOM in any way, send data to external servers (IPC to EC only).
#### 4.5A.2 Path B: Standalone Chrome via OpenClaw Extension
When a DOC3 learning session is active and the user interacts with Chrome:
1. EC signals OpenClaw gateway: `learn:activate_browser_observation { session_id }`
2. DOC11 forwards to Chrome extension via extension WebSocket
3. Extension injects content script into the active tab (same event listeners as Path A injected script; CDP is NOT available to extension content scripts)
4. Events flow: content script → extension background → WebSocket → DOC11 gateway → EC
5. EC maps to `RawObservationEventSchema` with `adapter_source: "browser_extension"`
6. On session end: EC sends `learn:deactivate_browser_observation { session_id }`
Error handling: if an observed tab closes during capture, extension sends `learn:tab_closed { tab_id, session_id }` and the live action feed shows "⚠ Observed tab '{title}' was closed." If the extension disconnects, gateway reconnection per §23.4 (ADJ-31: 3 retries, exponential backoff).
#### 4.5A.3 Browser Observation Eligibility
Before injecting the observation script into any tab or window, check eligibility:
| Check | Condition | Action if failed |
|---|---|---|
| Credential page | `input[type=password]` detected on page | Pause observation for this tab. Feed shows "Observation paused — credential page detected." Record tab-switch event only. |
| Extension/system page | URL starts with `chrome-extension://`, `chrome://`, `about:`, `devtools://` | Skip injection entirely. Feed shows "System page — not observed." |
| User-excluded domain | Domain is on user's excluded domain list (stored in settings) | Pause observation. Feed shows "Observation paused — excluded domain." |
| Incognito tab | Webview partition is `q-browser-private` (DOC20 R4.3 §6.19.11B) | Skip injection for this tab. Feed shows "🔒 Observation paused — incognito tab." |
When observation is paused for a tab, the system records only the tab-switch event (no element-level observation). If the user navigates away from the credential/excluded page within the same tab, observation resumes.
#### 4.5A.4 Popup Windows, Iframes, and SPA Navigation
**Popup windows.** Web apps frequently open auth popups (OAuth, "Login with Google") or utility windows via `window.open()`. Q's BrowserWindow registers a `new-window` handler. When a popup is created during a learning session, the popup window receives the same observation script injection as a tab (after eligibility check). The popup is treated as an additional observed surface; closing it fires a "Popup '{title}' closed" feed entry. For Chrome extension (Path B), the extension's `chrome.windows.onCreated` handler detects new windows and injects the content script.
**Same-origin iframes.** The injected observation script walks `window.frames` on injection and on subsequent DOM mutations, injecting into each same-origin iframe via `frame.contentDocument`. Cross-origin iframes are skipped (Same-Origin Policy enforces this; the boundary is documented explicitly).
**SPA navigation.** React/Vue/Angular apps navigate via `history.pushState` / `replaceState`, not full page loads. The injected content script monkey-patches `history.pushState` and `history.replaceState` to emit `BrowserObservationEventSchema { event_type: "spa_route_change", page_url: location.href }` on invocation. It also listens for `popstate` events. CDP `Page.frameNavigated` is consumed in parallel as the authoritative source for Path A. The monkey-patch MUST NOT modify pushState/replaceState semantics — only emit telemetry on invocation.
#### 4.5A.5 Tab Auto-Follow
Observation follows the user's focus automatically. No tab pre-selection or tab picker UI.
1. When recording starts, the **currently active tab** is the initial observation target. The capture overlay confirms: "Observing: {page title} ({domain})"
2. If the user switches to a different tab during capture, the tab switch is captured as a step in the live action feed. The observation script auto-injects into the new tab (after eligibility check, §4.5A.3). Both tabs are now observed.
3. If the user switches back to a previously observed tab, no re-injection needed. The feed shows: "Switch to tab '{title}'"
4. This continues for all tab switches during the session. Every eligible tab the user touches gets observed.
**Soft tab cap:** After 6 observed tabs, the feed shows an informational note: "Many tabs observed — interpretation may be less precise. Consider focusing your demonstration." Observation continues — no hard block.
**Popup windows:** Treated as new tabs for observation purposes. Eligibility check runs before injection.
**Tab close during observation:** Feed shows "⚠ Tab '{title}' closed." Observation continues on remaining tabs.
**Tab navigation:** If an observed tab navigates to a new URL, observation continues. Eligibility check re-runs on the new URL.
#### 4.5A.6 Browser Observation Event Schema
```ts
export const BrowserObservationEventSchema = z.object({
event_id: z.string().uuid(),
session_id: z.string().uuid(),
adapter_source: z.enum(["browser_dom", "browser_extension"]),
event_type: z.enum([
"click", "input", "change", "submit", "keydown",
"navigate", "tab_switch", "scroll", "select_text",
"spa_route_change", "dom_mutation_significant",
]),
target: z.object({
semantic_label: z.string().max(200),
role: z.string().max(40),
tag: z.string().max(20),
element_id: z.string().max(80).optional(),
field_value: z.string().max(500).optional(),
}),
tab_id: z.string().max(80),
tab_title: z.string().max(200),
page_url: z.string().max(500),
page_title: z.string().max(200),
caused_state_change: z.boolean().optional(),
timestamp: z.string().datetime(),
schema_version: z.literal(1),
});
```
## 5. Narration Capture — Concurrent Conversational Input
### 5.1 Why narration matters
The most powerful demonstrations involve the user talking while doing. "I'm doing this because..." / "Never do it that way because..." / "This is for when I need to..." Narration provides the *why* behind the *what* — the same mechanical action means different things in different contexts, and narration disambiguates.
### 5.2 Voice narration — primary input during demonstration
**[Source: Problem 2]**
Voice narration via OpenClaw's push-to-talk (PTT) is the primary narration input during outside-agent demonstrations. This avoids the app-switching problem (user would have to switch from Word to Q to type narration, generating noise AX events). Typed narration in Q remains available as fallback.
When voice narration is active, the adapter manager filters out AX events caused by the narration app switch (Q gaining/losing focus).
### 5.3 Silent demonstration mode
**[Source: Grok Stress #1]**
A busy litigator drafting under deadline pressure will not narrate. The system must produce acceptable results from pure observation.
```ts
// Session config extension
export const DemonstrationSessionConfigSchema = z.object({
// ... existing fields ...
narration_optional: z.boolean().default(true), // [Source: Grok Stress #1]
});
```
When `narration_optional: true` and no narration is provided, the interpretation prompt adapts: "No narration was provided for this demonstration. Rely entirely on the observed events to infer intent. Mark confidence lower on steps where intent is ambiguous."
The observation fidelity score (§6.7) redistributes narration weight to other sources to avoid penalizing silent demonstrations.
### 5.4 Narration capture schema
```ts
// packages/contracts/src/learning/narration-capture.ts
import { z } from "zod";
export const NarrationCaptureSchema = z.object({
narration_id: z.string().uuid(),
session_id: z.string().uuid(),
message: z.string().max(5000),
timestamp: z.string().datetime(),
input_method: z.enum(["voice_ptt", "typed"]).default("typed"), // [Source: Problem 2]
narration_kind: z.enum([
"explanation", // "I'm doing this because..."
"constraint", // "Never do X" / "Always do Y"
"goal", // "This is for when I need to..."
"correction", // "No, not that way — do it this way"
"context", // "This is the Henderson matter" / general context
"step_annotation", // "This step is important because..."
]),
nearest_event_id: z.string().uuid().optional(),
schema_version: z.literal(1),
});
```
**Narration kind classification:** Uses keyword detection, not an LLM call. Misclassification at this stage is harmless — the LLM semantic interpretation step uses the raw narration text regardless of the preliminary classification.
---
## 6. Semantic Interpretation — Raw Events to Knowledge Contracts
### 6.1 When interpretation runs
Semantic interpretation is a **post-capture** LLM-assisted step. It does NOT run on the hot path during capture. It runs when:
- The user clicks Stop (session transitions to `stopped`)
- The user clicks Quick Promote (interpretation is prerequisite)
- The user explicitly requests interpretation before stopping
### 6.2 Interpretation states
**[Source: Patch 2]**
During and after capture, interpretation state is tracked honestly:
```ts
export const LiveInterpretationStatusSchema = z.enum([
"tentative", // Gray badge — provisional structural label only, no semantic interpretation
"stable", // Blue badge — structural label has been consistent for 3+ events
"confirmed", // Green badge — post-stop interpretation complete + user reviewed
]);
```
Green badges appear ONLY after post-stop interpretation and user review. During capture, structural labels display with gray or blue badges. This prevents trust violations from premature "Understood" indicators.
### 6.3 Interpretation input
The LLM receives:
- All buffered `RawObservationEvent` entries for the session (with structural labels)
- All `NarrationCapture` entries
- The target application entity from the DOC72 graph (including `known_capabilities`)
- Existing procedure nodes for this application (to detect overlap and avoid duplication)
- The active work context (matter entity, document context if available)
- The user's known preferences and constraints linked to this app (from DOC72 graph)
### 6.4 Interpretation prompt structure
**[Source: A1, SR-06, SR-08, SR-09, SR-21, SR-22, SR-23, SR-24, SR-31, Problem 6]**
The interpretation prompt instructs the LLM to produce DOC72-governed `ProcedureKnowledgeContractSchema` payloads directly:
```
You are analyzing a demonstration session where the user showed how to perform
a workflow in {app_name} on {platform}.
OBSERVATION EVENTS (chronological):
{raw_events_with_structural_labels}
(Events marked is_undo_event: true represent mistakes — do NOT include the undone
action. The corrected action that follows IS the intended behavior.)
USER NARRATION DURING DEMONSTRATION:
{narration_entries}
(If empty: No narration was provided. Rely entirely on observed events. Mark
confidence lower on steps where intent is ambiguous.)
EXISTING KNOWLEDGE ABOUT THIS APP:
- Known capabilities: {known_capabilities}
- Existing procedures: {existing_procedure_summaries}
- Known user preferences for this app: {existing_preferences}
CURRENT CONTEXT:
- Active matter: {matter_name_if_any}
- Active document: {window_title}
INSTRUCTIONS:
1. SEMANTIC INTENT, NOT MECHANICAL REPLAY. Store what the action MEANS: "Set the
font to Times New Roman" not "Click Home tab, click Font dropdown, scroll to
Times New Roman."
2. OUTPUT FORMAT. Each procedure must conform to ProcedureKnowledgeContractSchema.
Required fields on each step:
- intent (one-line summary)
- detailed_instructions (what the LLM reads to execute)
- parameters (structured array with name, value, context)
- verification (how to check success)
- capability_area (what app feature area this exercises)
- step_nature: "mechanical" | "cognitive" | "hybrid" [SR-09]
3. TRIGGER PHRASES AND USE_CONDITIONS ARE SEPARATE. [SR-21]
- trigger_phrases: 3-5 routing discovery aliases ("create caption page", etc.)
- semantic_lookup_phrases: alternative phrasings for semantic search
- use_conditions: execution-time applicability ("When the user asks to create a
caption page for a court filing"). Rendered into injection cards.
- These are DIFFERENT FIELDS with DIFFERENT PURPOSES. Do NOT merge them.
4. STRUCTURAL METADATA. [SR-22]
When applicable, extract:
- artifact_class: what kind of document artifact ("caption_page", "toc", etc.)
- document_region: where in the document ("first_page_caption_block", etc.)
- derivation_quality: "high" | "medium" | "low" based on observation clarity
5. VARIABLE BINDING. [SR-08]
For parameters that should be context-dependent at runtime, set is_variable: true
and use a placeholder value like "[Client Name]" instead of literal text.
6. FORWARD-COMPATIBLE BRANCHING. [SR-17, SR-31]
Set execution_topology: "linear" and decision_points: [] for all procedures.
If the workflow contains branching logic, model branches as SEPARATE atomic
procedures, not conditional steps within a single procedure.
7. COMPOSITE DETECTION. Group related actions into procedures. If the overall
workflow is a composite of reusable steps, identify both the composite and its
atomic constituents. See §6.6 for pre-segmentation rules.
8. OVERLAP CHECK. Check against existing procedures for this app. If overlap found,
set overlaps_existing_procedure_id. See §7 for dedup rules.
9. MEMORY DIRECTIVES. [ADJ-10] From narration, extract each as a full
MemoryDirectiveKnowledgeContractSchema:
- memory_type: preference | constraint | standing_order | vocabulary_mapping |
style_profile | document_archetype | heuristic | correction
- summary: one-line description
- assertion_class: durable_fact | preference | constraint | standing_order |
vocabulary_rule | heuristic
- applies_when: at least one condition when this applies (REQUIRED)
- does_not_apply_when: conditions when this does NOT apply
- scope: global | app_specific | project_specific | document_type_specific | context_specific
- scope_details: elaboration on the scope
- priority_class: absolute | strong | default | suggestion
- source_certainty: user_explicitly_stated | inferred_from_actions
- injection_format: imperative | conditional | contextual | cautionary
10. STANDING PROCEDURES. If narration suggests a trigger condition ("whenever I
get this kind of email..."), propose a standing procedure candidate.
11. GOAL LINKS. If narration suggests broader objectives, propose goal links.
CRITICAL PRIVACY DIRECTIVE: [SR-06, SR-23]
You MUST generalize all specific text inputs, names, addresses, and privileged
data. Replace them with semantic placeholders (e.g., [Client Name], [Case Number],
[Court Name]) in the detailed_instructions and intent fields. Never write raw
observed text values into the contract. The procedure must work for ANY document,
not just the one being demonstrated. Set literal_retention_class appropriately:
- "placeholder_only" (default) for client names, case numbers, dates, amounts
- "retain_if_user_confirmed" for values that might be universal but need confirmation
- "retain_allowed" for generic values safe to retain (e.g., "Times New Roman")
12. PROCEDURE IDENTITY. [ADJ-09] For each procedure, provide:
- canonical_name: a clear, reusable name ("Create Caption Page for Brief")
- description: a one-paragraph summary of what this procedure does and when
13. CONDITIONS AND CONSTRAINTS. [ADJ-09] For each procedure:
- preconditions: what must be true BEFORE this procedure runs
- postconditions: what should be true AFTER successful completion
- non_use_conditions: when this procedure should NOT be used
- constraints: procedure-scoped rules with:
scope: "this_procedure" | "this_app" | "global"
priority: "absolute" | "strong" | "default" | "suggestion"
source: "user_stated" | "inferred"
14. [ADJ-11] Do NOT populate `resolved_execution` on any step. This field
is populated downstream by the execution-strategy cache (KDA R2 §3.5).
Leave it undefined.
15. [ADJ-12] The LLM SHALL produce `capability_area` per step. EC derives
the procedure-level `capability_tags` array at write time. Do NOT
produce `capability_tags` at the procedure level.
16. [ADJ-48] For memory directives, `memory_type` describes WHAT IT IS and
`assertion_class` describes HOW CERTAIN the claim is. A heuristic
(memory_type) MUST have assertion_class: "heuristic". A preference
(memory_type) MUST have assertion_class: "preference". Do NOT cross-assign.
Return a JSON object matching the KnowledgeExtractionBundle schema.
```
### 6.5 KnowledgeExtractionBundle — canonical interpretation output
**[Source: A1, Grok H28, SR-04, SR-27]**
The bundle is the universal output for ALL DOC3 learning modes. The `ProcedureCandidateSchema` carries the full `ProcedureKnowledgeContractSchema` as its payload, not a simplified subset.
```ts
// packages/contracts/src/learning/knowledge-extraction-bundle.ts
import { z } from "zod";
import { ProcedureKnowledgeContractSchema } from "../knowledge/procedure-knowledge-contract";
import { MemoryDirectiveKnowledgeContractSchema } from "../knowledge/memory-directive-knowledge-contract";
export const ProcedureCandidateSchema = z.object({
candidate_index: z.number().int().min(0),
// The full shared knowledge contract — DOC72-governed, filled by interpretation
contract: ProcedureKnowledgeContractSchema,
// Dedup detection
overlaps_existing_procedure_id: z.string().max(160).optional(),
// Interpretation metadata
interpretation_confidence: z.number().min(0).max(1),
});
export const MemoryDirectiveCandidateSchema = z.object({
candidate_index: z.number().int().min(0),
// The full shared knowledge contract — DOC72-governed
contract: MemoryDirectiveKnowledgeContractSchema,
linked_procedure_indices: z.array(z.number().int()).optional(),
source_narration_id: z.string().uuid().optional(),
interpretation_confidence: z.number().min(0).max(1),
});
export const StandingProcedureCandidateSchema = z.object({
trigger_description: z.string().max(500),
action_summary: z.string().max(500),
safety_class: z.enum(["notify", "confirm_first", "auto_safe"]),
confidence: z.number().min(0).max(1),
linked_procedure_indices: z.array(z.number().int()).optional(),
});
export const GoalCandidateSchema = z.object({
goal_description: z.string().max(500),
linked_procedure_indices: z.array(z.number().int()),
is_existing_goal: z.boolean(),
existing_goal_id: z.string().max(160).optional(),
});
export const KnowledgeExtractionBundleSchema = z.object({
bundle_id: z.string().uuid(),
session_id: z.string().uuid(),
bundle_source: z.enum([
"demonstration",
"coaching",
"autonomous_practice",
"skill_improvement",
"import_decomposition",
"retrospective_learning",
"skill_mining",
"native_skill_ingestion",
"conversational_capture", // [R2.2] From §25.8 conversational procedure capture
]),
procedure_candidates: z.array(ProcedureCandidateSchema).default([]), // [ADJ-15] min(1) removed; empty-bundle is business error, not schema constraint
memory_directive_candidates: z.array(MemoryDirectiveCandidateSchema).default([]),
standing_procedure_candidates: z.array(StandingProcedureCandidateSchema).default([]),
goal_candidates: z.array(GoalCandidateSchema).default([]),
supporting_trace: z.object({
raw_event_count: z.number().int().min(0),
narration_count: z.number().int().min(0),
capture_duration_seconds: z.number().min(0),
adapter_sources_used: z.array(DemonstrationAdapterSourceSchema),
screenshot_count: z.number().int().min(0).default(0),
observation_fidelity_score: z.number().min(0).max(1), // [Source: Grok H28]
}),
interpretation_model: z.string().max(80),
interpretation_confidence: z.number().min(0).max(1),
// [ADJ-14] Renamed from contract_quality to avoid collision with per-procedure field
extraction_quality_summary: z.object({
extraction_model_ref: z.string().max(160),
overall_quality_score: z.number().min(0).max(1),
field_quality: z.record(z.string(), z.number().min(0).max(1)).default({}),
missing_fields: z.array(z.string()).default([]),
}).optional(),
schema_version: z.literal(1),
});
// [ADJ-36] Standing procedure and goal candidates are draft extraction objects.
// Before graph write, EC MUST transform them into StandingProcedureKnowledgeContractSchema /
// GoalKnowledgeContractSchema payloads and validate via NodePayloadValidatorRegistry.
// If these contracts are not yet defined in KDA R2, this is a cross-doc obligation (CD-6).
```
### 6.6 Composite procedure pre-segmentation
**[Source: A2]**
The LLM alone cannot reliably detect composites. A deterministic pre-segmentation step runs BEFORE LLM interpretation:
```ts
// apps/ec-service/src/learning/composite-detection.ts
export function detectCompositeGroups(events: RawObservationEvent[]): CompositeGroup[] {
// Step 1: Cluster events by temporal adjacency + app + artifact class
// Gap > 10 seconds between meaningful events = new cluster
// Different app_bundle_id = new cluster (unless multi-app composite intended)
const clusters = clusterByTemporalAdjacency(events, 10_000);
// Step 2: For each pair of adjacent clusters, apply 4 link tests
// Require 2 of 4 to pass for composite emission:
// 1. Shared artifact — both clusters operate on the same document/file
// 2. Shared goal — narration links them to the same objective
// 3. Ordered dependency — cluster B's preconditions match cluster A's postconditions
// 4. No unrelated branch — no cluster of unrelated actions between them
// Step 3: Minimum composite size = 2 atomic procedures
// Maximum composite depth = 1 level (composites contain atomics, not other composites)
// Step 4: Pass composite groups to LLM for confirmation/rejection
// LLM also applies independence test: "Would the user ever do step 3 WITHOUT steps 1-2?"
// If LLM says yes → keep as separate atomics. If no → confirm composite.
return compositeGroups;
}
// [ADJ-72] CompositeGroup type definition
export const CompositeGroupSchema = z.object({
group_id: z.string().uuid(),
cluster_indices: z.array(z.number().int()).min(2),
link_tests_passed: z.number().int().min(2).max(4),
link_test_results: z.record(z.string(), z.boolean()),
});
// Composite groups are included as `pre_segmented_groups` in the interpretation
// prompt input. The LLM confirms or rejects each group in the bundle output
// via a `composite_groups_confirmed: string[]` field (group_ids).
```
### 6.7 Observation fidelity score
**[Source: Grok H28, SR-04]**
The fidelity score aggregates observation quality into a single number that gates Quick Promote eligibility and informs review rigor:
```ts
// apps/ec-service/src/learning/observation-fidelity.ts
export function computeObservationFidelityScore(
trace: SupportingTrace,
hasNarration: boolean,
): number {
if (hasNarration) {
// Standard weights
return (
(trace.ax_event_rate * 0.4) +
(trace.narration_coverage * 0.3) +
(trace.adapter_health * 0.2) +
(trace.screenshot_coverage * 0.1)
);
} else {
// Silent demonstration — redistribute narration weight [Source: SR-04]
return (
(trace.ax_event_rate * 0.6) +
(trace.adapter_health * 0.3) +
(trace.screenshot_coverage * 0.1)
);
}
}
// [ADJ-46] Thresholds (corrected — no overlapping ranges):
// ≥ 0.70 → Green indicator. Quick Promote enabled.
// 0.60 to < 0.70 → Amber. Quick Promote enabled with warning "Limited observation — review recommended."
// 0.40 to < 0.60 → Amber. Quick Promote DISABLED; full review required.
// < 0.40 → Red. Quick Promote DISABLED; force detailed review mode.
// Banner: "Limited observation — review each step carefully."
// [ADJ-43] Silent mode fidelity redistribution:
// When narration_optional = true, weights are:
// (AX event rate × 0.55) + (adapter health × 0.30) + (screenshot coverage × 0.15)
// The narration component is omitted, not zeroed. This allows silent demos with
// good AX coverage to reach >= 0.60.
```
### 6.8 Streaming interpretation
**[Source: D3]**
Complex demonstrations may take time to interpret. Partial results stream via SSE as procedures are extracted:
```ts
const INTERPRETATION_TIMING = {
simple_target_ms: 15_000, // < 15 events
complex_target_ms: 45_000, // 15-100 events
hard_timeout_ms: 90_000, // Abort after 90s
soft_timeout_ms: 30_000, // Show "Complex workflow — this may take a moment"
// On hard timeout: show partial results, offer to complete in background
};
```
SSE events during interpretation:
- `demonstrate.interpretation_progress` — partial procedure extracted (streamed as available)
- `demonstrate.interpretation_complete` — all procedures extracted
- `demonstrate.interpretation_timeout` — hard timeout reached, partial results available
### 6.9 Unified pipeline reference function
**[Source: Problem 7]**
The complete interpretation pipeline as a single reference:
```ts
// apps/ec-service/src/learning/interpret-demonstration.ts
export async function interpretDemonstration(
session: LearnSession,
rawEvents: RawObservationEvent[],
narrations: NarrationCapture[],
graphContext: GraphContext,
): Promise<KnowledgeExtractionBundle> {
// 1. Preprocess: detect undos, classify significance [Problem 3, D1]
const preprocessed = preprocessEvents(rawEvents);
// 2. Pre-segment: detect composite groups deterministically [A2]
const compositeGroups = detectCompositeGroups(preprocessed);
// 3. Compute observation fidelity score [Grok H28, SR-04]
const fidelityScore = computeObservationFidelityScore(
computeTrace(preprocessed, narrations),
narrations.length > 0,
);
// === R2.2 AMENDMENT: Steps 3A-3B (curation after temporal analysis, before prompt) ===
// 3A. Build curated interpretation items from user-edited display items (§20.3A)
// Steps 1-3 used raw temporal order. Now apply user curation for prompt construction.
type CuratedInterpretationItem =
| { kind: "raw_event"; display_item: RawEventDisplayItem; raw_event: RawObservationEvent }
| { kind: "manual_step"; display_item: RawEventDisplayItem }
| { kind: "narration"; display_item: RawEventDisplayItem }
| { kind: "system"; display_item: RawEventDisplayItem };
const curatedItems: CuratedInterpretationItem[] = displayItems
.filter(item => !item.excluded)
.sort((a, b) => a.display_index - b.display_index)
.map(item => {
if (item.source_kind === "raw_event") {
const rawEvent = preprocessed.find(e => e.event_id === item.raw_event_id);
return { kind: "raw_event" as const, display_item: item, raw_event: rawEvent };
}
return { kind: item.source_kind as "manual_step" | "narration" | "system", display_item: item };
});
// 3B. Scrub annotations and manual steps for PII before prompt construction.
// ADJ-02 scrubber runs on LLM OUTPUT; user-typed INPUT also needs scrubbing.
for (const item of curatedItems) {
if (item.display_item.user_annotation) {
item.display_item.user_annotation = scrubLiteralValues(item.display_item.user_annotation);
}
if (item.kind === "manual_step") {
item.display_item.display_text = scrubLiteralValues(item.display_item.display_text);
}
}
// === END R2.2 AMENDMENT ===
// 4. Build interpretation prompt with privacy directive [SR-06, SR-23]
// [R2.2] Step 4 now receives curatedItems (user logical order) instead of preprocessed (temporal order)
// Manual steps and narrations are first-class items, not prompt annotations.
// If user reordered events, prompt includes 'user reordered the sequence' guidance.
const prompt = buildInterpretationPrompt(
curatedItems ?? preprocessed, narrations, compositeGroups, graphContext,
);
// 5. Run LLM interpretation with streaming [D3]
const bundle = await runInterpretation(prompt, session.config.interpretation_model);
// 6. Post-interpretation scrubber: validate privacy compliance [SR-23]
scrubLiteralValues(bundle);
// 7. [ADJ-12] EC derives capability_tags from step capability_area values
for (const proc of bundle.procedure_candidates) {
proc.contract.capability_tags = [...new Set(
proc.contract.steps.map(s => s.capability_area).filter(Boolean)
)];
}
// 7A. [ADJ-69] Validate environment block completeness
for (const proc of bundle.procedure_candidates) {
if (!proc.contract.environment?.app_name) {
throw new Error("PROCEDURE_ENVIRONMENT_INCOMPLETE");
}
}
// 8. [ADJ-35] Validate ALL contracts against schemas (not just procedures)
for (const proc of bundle.procedure_candidates) {
ProcedureKnowledgeContractSchema.parse(proc.contract);
}
for (const md of bundle.memory_directive_candidates) {
MemoryDirectiveKnowledgeContractSchema.parse(md.contract);
}
// Standing/goal candidates validated at EC transform time (see §13 note)
// 9. Set bundle metadata
bundle.supporting_trace.observation_fidelity_score = fidelityScore;
return bundle;
}
```
### 6.10 Overlap detection and enrichment
When the interpretation detects that a demonstrated procedure overlaps with an existing procedure in the graph, it sets `overlaps_existing_procedure_id`. At write time, this triggers enrichment instead of creation:
- The existing procedure's confidence is incremented (α += 0.75 for a new supporting trace)
- New trace evidence is linked via `validated_by_trace` edge
- Any new steps or refinements from the demonstration are proposed as a procedure update through DOC3's revision pipeline
- Any new constraints or preferences are proposed as new memory directives linked to the existing procedure
**See §7 for the full three-stage dedup algorithm that makes this determination.**
---
## 7. Three-Stage Dedup Algorithm
**[Source: A3, SR-22, SR-11]**
### 7.1 Hard vetoes — deterministic, no LLM
Before any scoring, the following conditions PREVENT a merge. These use FIXED FIELDS on the shared contract (not annotations):
```ts
// apps/ec-service/src/learning/dedup-hard-vetoes.ts
export function hasHardVeto(incoming: ProcedureKnowledgeContract, existing: ProcedureKnowledgeContract): boolean {
// Different app family → never merge
if (incoming.environment.app_entity_id !== existing.environment.app_entity_id) return true;
// Different artifact_class (when both present) → never merge [SR-22]
if (incoming.artifact_class && existing.artifact_class &&
incoming.artifact_class !== existing.artifact_class) return true;
// Different document_region (when both present) → never merge [SR-22]
if (incoming.document_region && existing.document_region &&
incoming.document_region !== existing.document_region) return true;
// Contradictory non_use_conditions → never merge
if (hasContradictoryConditions(incoming.non_use_conditions, existing.use_conditions)) return true;
if (hasContradictoryConditions(existing.non_use_conditions, incoming.use_conditions)) return true;
// Step-count ratio > 3:1 with low step similarity → never merge
const ratio = Math.max(incoming.steps.length, existing.steps.length) /
Math.min(incoming.steps.length, existing.steps.length);
if (ratio > 3) {
const stepSimilarity = computeStepSimilarity(incoming.steps, existing.steps);
if (stepSimilarity < 0.3) return true;
}
// [ADJ-55] Constraint parameter conflict veto
// Fires when: same scope, same parameter name, different parameter values
if (hasConstraintParameterConflict(incoming.constraints, existing.constraints)) return true;
return false;
}
```
### 7.2 Weighted scoring
If no hard veto fires, compute a weighted similarity score:
```ts
// [ADJ-18] Field names aligned exactly to KDA R2 §2.3
const DEDUP_WEIGHTS = {
intent_similarity: 0.35, // Canonical name + description
step_similarity: 0.30, // Semantic step overlap
use_conditions_overlap: 0.15, // KDA R2 field: use_conditions
postconditions_overlap: 0.10, // KDA R2 field: postconditions
capability_tags_overlap: 0.10, // KDA R2 field: capability_tags (plural)
} as const;
// [ADJ-20] Empty-vs-empty scoring rule:
// When computing overlap for postconditions or capability_tags, if BOTH arrays
// are empty, the overlap score for that dimension is 0.0 (no evidence of
// similarity), not 1.0. Only populated arrays contribute positive similarity.
```
### 7.3 Three-stage matching pipeline
```ts
// apps/ec-service/src/learning/dedup-pipeline.ts
export async function findDuplicates(
incoming: ProcedureKnowledgeContract,
existingProcedures: ProcedureNode[],
): Promise<DedupMatch[]> {
const matches: DedupMatch[] = [];
// [ADJ-21] Stage 0.5: Hot Buffer Check
// Before querying the durable graph, check incoming contract against
// an in-memory buffer of procedures written in the last 5 minutes.
// Acquire dedup lock key per (app_entity_id, canonical_name_norm, artifact_class)
// before write. Only one promote flow may pass dedup for a given key at a time.
// The name/alias match (Stage 1) queries SQL directly and is immediately consistent;
// the embedding match (Stage 2) may lag. Stage 0.5 covers the gap.
const hotBufferMatch = checkHotBuffer(incoming, hotBuffer);
if (hotBufferMatch) {
return [{ existing_id: hotBufferMatch.id, score: 1.0, source: "hot_buffer" }];
}
for (const existing of existingProcedures) {
// Stage 0: Hard veto check
if (hasHardVeto(incoming, existing.payload)) continue;
// Stage 1: Name/alias match (fast, deterministic)
const nameScore = computeNameSimilarity(incoming.canonical_name, existing.payload.canonical_name);
// Stage 2: Embedding similarity (cached, <20ms)
const embeddingScore = await computeEmbeddingSimilarity(incoming, existing.payload);
// Stage 3: Weighted composite score
const score = computeWeightedScore(incoming, existing.payload);
if (score >= 0.78) {
matches.push({ existing_id: existing.id, score, nameScore, embeddingScore });
}
}
return matches.sort((a, b) => b.score - a.score);
}
```
### 7.4 Thresholds and actions
**[ADJ-19]** Exhaustive dedup action table:
| Score | Margin | Action |
|---|---|---|
| ≥ 0.90 | ≥ 0.08 | Auto-enrich: increment confidence (α +0.75), link new trace, update steps if improved |
| ≥ 0.90 | < 0.08 | User review (ambiguous top match) |
| 0.78–0.89 | Any | User review: "Merge / Keep Separate / Create Variant" options |
| < 0.78 | Any | New procedure — no duplicate concern |
**Margin requirement:** The gap between the top match and the second match must be ≥ 0.08 for auto-enrich. If margin < 0.08 between ANY of the top 3 candidates (not just top 2), force user review.
**[ADJ-56]** Auto-enrich guard: Auto-enrich requires ALL of: `score >= 0.90`, `margin >= 0.08`, AND `step_similarity >= 0.75`. If step_similarity is below 0.75, force user review even if overall score exceeds 0.90.
### 7.5 Nightly duplicate audit
Procedures in the same app with score ≥ 0.85 but not merged appear in a duplicate-review queue in the Knowledge Manager. This catches duplicates that the intake missed.
**[ADJ-63]** The duplicate-review queue shows the same options as intake dedup review (§7.4): Merge / Keep Separate / Create Variant. Items dismissed with 'Keep Separate' are marked `dedup_reviewed: true` and do not re-surface unless one of the procedures is modified. Unreviewed items re-surface nightly until addressed.
### 7.6 Post-write async verification
After a graph write, an async job checks for duplicates against the full graph (not just the pre-filtered set from interpretation). If found, the duplicate is surfaced as a review item — never auto-merged post-write.
---
### 7.2A Memory Directive Dedup Extension (R2.2)
**[Source: R2.2 Proposal §4]**
Extends the procedure dedup algorithm (§7) to memory directives. When a new memory directive candidate is produced from any intake path, check for existing duplicates.
**Match dimensions:**
```ts
const MEMORY_DIRECTIVE_DEDUP_WEIGHTS = {
summary_similarity: 0.35,
scope_match: 0.25,
applies_when_overlap: 0.20,
assertion_class_match: 0.10,
memory_type_match: 0.10,
} as const;
```
**Action thresholds:** Same as procedure dedup (§7.4):
- ≥ 0.90 with margin ≥ 0.08 → auto-enrich (update confidence, add provenance)
- 0.78–0.89 → user review ("Merge / Keep Separate")
- < 0.78 → new directive, no duplicate concern
**Auto-enrich safety rule:** Auto-enrich may add provenance and update confidence, but it MUST NOT broaden `applies_when` conditions, change `priority_class`, or merge corrections/standing orders without DOC1-governed review. Broadening requires user confirmation.
**Cross-modal awareness:**
```ts
capture_modality: z.enum([
"observed_demonstration",
"coaching",
"conversational_capture",
"retrospective_learning",
"imported_skill",
"skill_mining",
]);
```
If app/environment is explicit or inferable, use normal hard vetoes. If app/environment is missing (common for conversational capture), do not hard-veto; route to review unless name/scope/applies_when are overwhelmingly similar (≥ 0.90).
**Empty-array rule (ADJ-20):** When both `applies_when` arrays are empty, overlap score is 0.0.
---
## 8. LearnSession Integration — No Parallel State Machine
### 8.1 How demonstration sessions use the existing lifecycle
A demonstration session is a `LearnSession` with:
- `entry_mode: "demonstrating_skill"`
- `observation_mode: "outside_agent"` or `"mixed"`
- `target_app_whitelist`: set to the target app(s) or empty for any-app capture
- `narration_optional: boolean` — whether narration is expected **[Source: Grok Stress #1]**
**[ADJ-05]** The session follows the existing state machine from DOC3 §0C.1A. R11.3 adds no new top-level `LearnSession` states. Interpretation runs while `LearnSession.state = "stopped"` and is surfaced through `LearningRuntimeSnapshot.interpretation_phase`.
### 8.2 State machine path — standard demonstration
```
idle → armed → [target_app_detected] → capturing → [paused ↔ capturing] → stopped
[interpretation_phase = "running"] → reviewing → drafting_proposal → testing_proposal →
ready_for_review → installing_private → installed_private → approved
```
**[ADJ-05]** Interpretation runs as a sub-phase of `stopped`, not a separate state.
### 8.3 State machine path — Quick Promote
Quick Promote compresses the pipeline. The session skips `reviewing`, `awaiting_questions`, `testing_proposal`, and `ready_for_review`:
```
idle → armed → [target_app_detected] → capturing → stopped
[interpretation_phase = "running"] → drafting_proposal → installing_private →
installed_private
```
**[ADJ-57]** Quick Promote skips `reviewing`, `testing_proposal`, and `ready_for_review`.
### 8.4 Quick Promote scope limitation
**[Source: Patch 3, SR-03]**
Quick Promote installs PROCEDURES ONLY. All other bundle items are preserved for separate review:
```ts
// packages/contracts/src/learning/quick-promote.ts
import { z } from "zod";
export const QuickPromoteScopeSchema = z.enum([
"procedure_only_minimal", // Default: procedures only
"procedure_plus_existing_goal", // Procedures + link to EXISTING goal (no new goal creation)
]);
// During Quick Promote:
// 1. procedure_candidates → installed as experimental_private abilities
// 2. memory_directive_candidates → shunted to Pending Abilities for separate review
// 3. standing_procedure_candidates → shunted to Pending Abilities
// 4. goal_candidates with is_existing_goal: false → shunted to Pending Abilities
// 5. goal_candidates with is_existing_goal: true → edge created (if scope allows)
//
// SCOPE LEAK PREVENTION [Source: SR-03]:
// Any procedure constraints with scope: "global" or scope: "this_app" **[ADJ-37]**
// are STRIPPED from the procedure payload during Quick Promote, converted to
// standalone memory_directive candidates, and shunted to Pending Abilities.
// Only constraints with scope: "this_procedure" remain embedded.
export const QuickPromoteFooterText =
"This installs the procedure now. Preferences, constraints, and standing automations stay in Pending Abilities.";
```
Quick Promote is a **policy flag on the install request**, not a separate route or state:
```ts
export const InstallProposalRequestSchema = z.object({
install_lane: z.enum(["experimental_private", "approved_workspace"]).default("experimental_private"),
quick_promote: z.boolean().default(false),
quick_promote_scope: QuickPromoteScopeSchema.default("procedure_only_minimal"),
proposed_title: z.string().max(200).optional(),
linked_goal_ids: z.array(z.string().max(160)).optional(),
schema_version: z.literal(1),
});
```
When `quick_promote: true`:
- Proposal must pass minimum lint validation (environment scope present, ≥ 1 trace, semantic steps non-empty, trigger phrases present)
- Validation run is skipped
- Install lane is forced to `experimental_private`
- On the first quick promote for any new app family, a one-line user confirmation is required before install:
> "Create 'Insert TOC in Word' as a private experimental ability?"
### 8.5 Quick Promote eligibility
```ts
export function isQuickPromoteEligible(
session: LearnSession,
bundle: KnowledgeExtractionBundle,
): boolean {
return (
session.entry_mode === "demonstrating_skill" &&
bundle.procedure_candidates.length >= 1 &&
bundle.supporting_trace.raw_event_count >= 1 &&
bundle.supporting_trace.observation_fidelity_score >= 0.6 && // [Source: Grok H28]
bundle.procedure_candidates.every(p =>
p.contract.steps.length >= 1 &&
p.contract.environment.platform !== "unknown" &&
p.contract.trigger_phrases.length >= 3 // [Source: Patch 7 → SR-21]
)
);
}
```
### 8.6 Mid-demo Quick Promote
**[Source: Patch 4]**
The user can Quick Promote during capture without stopping the session:
```ts
export const CaptureSegmentSchema = z.object({
segment_id: z.string().uuid(),
session_id: z.string().uuid(),
start_event_index: z.number().int(),
end_event_index: z.number().int(),
sealed: z.boolean(), // True after Quick Promote — never mutated by later events
installed_ability_id: z.string().max(160).optional(),
});
```
**Rules:**
1. When mid-demo Quick Promote is triggered, seal the current segment at the cut boundary
2. Build Quick Promote from the sealed segment only
3. Continue capturing on a new segment
4. **[ADJ-06]** During mid-demo Quick Promote, `LearnSession.state` MUST remain `capturing`. Install progress of the promoted segment is tracked exclusively via `CaptureSegment.install_saga_id` (a child saga), not the parent session state.
5. **[ADJ-06]** Capture segments do not create separate LearnSession instances. A session with any active segment remains one blocking LearnSession, even if earlier segments were already promoted.
6. **[ADJ-06]** When the session finally reaches `stopped`, the remaining unsealed segment undergoes standard interpretation. The resulting bundle UI MUST visually indicate that it is a continuation of the previously Quick Promoted ability (linking to the installed `ability_id`), and defaults to the standard review path.
### 8.7 Trust Mode
**[Source: Grok Stress #2, SR-02]**
For experienced users who want to reduce review friction:
```ts
export const TrustModeConfigSchema = z.object({
enabled: z.boolean().default(false),
// Auto-accept threshold for procedures
auto_accept_confidence_floor: z.number().min(0.75).max(1.0).default(0.80),
// Require at least one full manual review per app family before enabling
requires_prior_manual_review_per_app: z.boolean().default(true),
});
```
**Trust Mode behavior:**
- **[ADJ-07]** Auto-installs eligible procedures to `install_lane = "experimental_private"` unless the user explicitly invoked a reviewed shared-install path. Trust Mode does not infer or auto-promote to `approved_workspace` or `shared_promoted`
- ALL items ≥ 0.80 → items below 0.80 are shown for manual review
- If ALL items are ≥ 0.80, the entire bundle is promoted in one tap
- `memory_directive`, `standing_procedure`, and `goal` items — even those ≥ 0.80 — are forced to `candidate` state in Pending Abilities
- Trust Mode does NOT override DOC1 Write Gate governance for non-procedure items
- Trust Mode does NOT override Quick Promote's scope limitation (§8.4)
### 8.8 Session concurrency
**[Source: Patch 10]**
Only actively-in-progress states block new sessions:
```ts
export function blocksNewLearnSession(state: LearnSessionState): boolean {
// [ADJ-05] "interpreting" removed — interpretation is a sub-phase of "stopped"
const BLOCKING_STATES: LearnSessionState[] = [
"armed", "capturing", "paused", "stopped",
"reviewing", "drafting_proposal", "testing_proposal", "ready_for_review",
"installing_private",
];
// These do NOT block:
// "installed_private", "approved", "rejected", "cancelled", "quarantined"
return BLOCKING_STATES.includes(state);
}
```
### 8.9 Feature support snapshot
**[Source: Patch 1]**
Before showing the full "teach once, use immediately" flow, Q checks that the critical seams are wired:
```ts
export const LearnFeatureSupportSnapshotSchema = z.object({
demonstration_capture: z.boolean(),
semantic_interpretation: z.boolean(),
graph_write: z.boolean(),
doc24_injection: z.boolean(),
ability_lookup: z.boolean(),
doc11_gateway: z.boolean(),
// [ADJ-39] Blocker-specific degraded states
blockers: z.array(z.enum([
"gateway_observation_unavailable",
"tcc_accessibility_missing",
"tcc_screen_recording_missing",
"interpretation_model_unavailable",
"graph_write_unavailable",
"doc24_rendering_unavailable",
"ability_lookup_unavailable",
])).default([]),
degraded_actions: z.array(z.enum([
"capture_only",
"review_only_no_install",
"no_quick_promote",
"no_later_use_proof",
])).default([]),
});
// [ADJ-39] Q must render blocker-specific degraded states, not a generic "unavailable" badge.
// Route: GET /api/learn/feature-support
// If any critical blocker is false, Q shows degraded UI with specific guidance
// on what's not available and what the user can still do.
```
### 8.10 LearningRuntimeSnapshot extension
Add demonstration-related fields to the existing `LearningRuntimeSnapshotSchema` (DOC3 §0C.7):
```ts
// Extension to LearningRuntimeSnapshotSchema
quick_promoted: z.boolean().default(false),
trust_mode_used: z.boolean().default(false),
demonstration_bundle_id: z.string().uuid().optional(),
observation_fidelity_score: z.number().min(0).max(1).optional(),
materialization_kind: z.enum(["graph_injection", "native_skill_bundle"]).optional(), // [Source: SR-25]
// [ADJ-05] Interpretation sub-phase tracking
interpretation_phase: z.enum(["idle", "running", "completed", "failed"]).default("idle"),
interpretation_started_at: z.string().datetime().optional(),
interpretation_completed_at: z.string().datetime().optional(),
interpretation_error: z.string().max(240).optional(),
```
---
## 9. Bundle Review and Editing
### 9.1 Summary/detail review panel
**[Source: D2]**
After interpretation completes, the bundle review panel shows two views:
- **Summary view (default):** One-line items grouped by type (procedures, memory directives, standing procedures, goals) with inline accept/reject controls. Shows interpretation confidence per item.
- **Detail view:** Full steps with preconditions, constraints, parameters, verification conditions. Toggled per item or globally.
- **[ADJ-03] Privacy Review Banner:** If any candidate contains parameters marked `retain_if_user_confirmed`, the panel MUST surface a distinct warning banner listing the literal values (e.g., 'Retain literal value "Times New Roman"?'). Unconfirmed values MUST automatically default to `placeholder_only` upon graph write.
- **[ADJ-62] Streaming props:** `isInterpreting: boolean` — when true, show spinner + partial results via SSE. `streamingBundle: Partial<KnowledgeExtractionBundle> | null` — the in-progress bundle with candidates growing as the LLM extracts them.
### 9.2 Full bundle editing
**[Source: D4]**
Step-level editing with a discriminated union of edit operations:
```ts
// packages/contracts/src/learning/bundle-edit.ts
import { z } from "zod";
export const BundleEditOperationSchema = z.discriminatedUnion("operation", [
z.object({
operation: z.literal("replace_step"),
procedure_index: z.number().int(),
step_index: z.number().int(),
updated_step: SemanticStepSchema.partial(), // [ADJ-47] Typed instead of z.record(unknown)
}),
z.object({
operation: z.literal("move_step"),
procedure_index: z.number().int(),
from_step_index: z.number().int(),
to_step_index: z.number().int(),
}),
z.object({
operation: z.literal("delete_step"),
procedure_index: z.number().int(),
step_index: z.number().int(),
}),
z.object({
operation: z.literal("insert_step"),
procedure_index: z.number().int(),
after_step_index: z.number().int(),
new_step: SemanticStepSchema.omit({ step_index: true }), // [ADJ-47]
}),
z.object({
operation: z.literal("split_step"),
procedure_index: z.number().int(),
step_index: z.number().int(),
split_into: z.array(SemanticStepSchema.omit({ step_index: true })).min(2), // [ADJ-47]
}),
z.object({
operation: z.literal("reclassify_item"),
source_kind: z.enum(["procedure", "memory_directive", "standing_procedure", "goal"]),
source_index: z.number().int(),
target_kind: z.enum(["procedure", "memory_directive", "standing_procedure", "goal"]),
transformation_payload: z.record(z.string(), z.unknown()), // [ADJ-23] Required: valid target contract
}),
z.object({
operation: z.literal("merge_procedures"),
procedure_indices: z.array(z.number().int()).min(2),
merged_name: z.string().max(200),
}),
z.object({
operation: z.literal("split_procedure"),
procedure_index: z.number().int(),
split_at_step_index: z.number().int(),
new_procedure_name: z.string().max(200),
}),
]);
// Route: POST /api/learn/sessions/:sessionId/bundle-edits
// Error codes: LEARN_SESSION_NOT_FOUND, BUNDLE_VERSION_NOT_FOUND,
// INVALID_BUNDLE_EDIT, BUNDLE_ITEM_LOCKED
```
### 9.3 Provenance severing on user edit
**[Source: SR-05]**
**[ADJ-50]** When a user manually edits a semantic step during bundle review:
1. The edited step MUST clear its `source_event_ids` array (the observation events no longer produced this step — the user did)
2. Append a procedure-level annotation: `{ annotation_type: "step_provenance_override", content: JSON.stringify({ step_index, source: "user_explicitly_stated" }) }`
3. This prevents the system from claiming observation-based confidence for user-authored content
### 9.3A Reclassification Rules
**[ADJ-23]** When `reclassify_item` is invoked, the server validates `transformation_payload` against the target kind's shared knowledge contract schema before applying. Supported directions:
- **Procedure → Memory Directive:** Maps `canonical_name` → `summary`, `use_conditions[0]` → `applies_when[0]`. Requires user input for `assertion_class`, `scope`. Discards `steps`, `trigger_phrases`, `postconditions`. All other target fields must be provided in `transformation_payload`.
- **Memory Directive → Procedure:** Returns `RECLASSIFY_NOT_SUPPORTED`. User must manually create a new procedure candidate.
- **Procedure → Standing Procedure:** Maps `steps` → `action_steps`, `use_conditions` → `trigger_conditions[0].conditions`. Requires manual trigger refinement.
- All other directions: Require `transformation_payload` with all required target fields.
Q frontend MUST pre-populate `transformation_payload` from source fields where mappable, then open an editor for the user to fill remaining required fields.
---
### 9.4 Error recovery options
**[Source: D4]**
When interpretation fails or produces poor results:
- `retry_with_narration` — re-run interpretation with additional narration context
- `manual_edit_steps` — open the full bundle editor for manual correction
- `manual_create` — discard interpretation, let user author steps from scratch
- `discard` — cancel the session
**[ADJ-08]** Empty bundle retry state transition: `stopped [interpretation_phase = "failed"] → stopped [interpretation_phase = "idle"]` when the user clicks "Retry with narration." New narration is accepted in this phase. The user may also provide additional context before re-triggering interpretation.
If interpretation detects missing events (gaps in the observation stream), insert `UNOBSERVED` placeholder steps highlighted in amber for user review.
---
## 10. SSE Event Stream Extensions
### 10.1 New event kinds for demonstration sessions
Add to the existing SSE event stream (`GET /api/learn/sessions/:sessionId/events`):
```ts
// New event kinds added to LearningLifecycleEventSchema (§0C.13)
// Target app detected during armed state [Source: Problem 1]
"demonstrate.target_app_detected"
// Payload: { app_name: string, app_bundle_id: string }
// Raw event captured during capture — structural label, no LLM
"demonstrate.raw_event_captured"
// Payload: { structural_label: string, adapter_source: string, event_id: string, significance: string }
// Narration received
"demonstrate.narration_captured"
// Payload: { narration_id: string, narration_kind: string, message_preview: string }
// Interpretation started
"demonstrate.interpretation_started"
// Payload: { event_count: number, narration_count: number }
// Partial procedure extracted during streaming interpretation [Source: D3]
"demonstrate.interpretation_progress"
// Payload: { procedure_name: string, step_count: number, tentative: boolean }
// Interpretation complete
"demonstrate.interpretation_complete"
// Payload: { bundle_id: string, procedure_count: number, directive_count: number, goal_count: number, fidelity_score: number }
// Interpretation timed out — partial results available [Source: D3]
"demonstrate.interpretation_timeout"
// Payload: { partial_procedure_count: number, remaining_events: number }
// Quick promoted
"demonstrate.quick_promoted"
// Payload: { ability_id: string, procedure_count: number, materialization_kind: string }
// Install saga state change [Source: Patch 6]
"demonstrate.install_saga_progress"
// Payload: { saga_id: string, state: string, procedure_id: string }
```
### 10.2 Event-Consumer Subscription Matrix
**[ADJ-51]**
| Component | Subscribes to |
|---|---|
| `LearnCaptureOverlay` | `target_app_detected`, `raw_event_captured`, `narration_captured`, `app_version_changed_mid_session` |
| `BundleReviewPanel` | `interpretation_progress`, `interpretation_complete`, `interpretation_timeout` |
| `QuickPromoteDrawer` | `quick_promoted`, `install_saga_progress` |
| `FidelityIndicator` | `raw_event_captured` (for live fidelity recalculation) |
---
### 10.3 Display Item Events (R2.2)
**[Source: R2.2 Proposal §8]**
Use `demonstrate.display_item_*` namespace. Payloads use `display_item_id` (not `display_index`, since indices change after reorder).
```ts
"demonstrate.display_item_added"
// Payload: RawEventDisplayItem (§20.3A.1)
"demonstrate.display_item_updated"
// Payload: { display_item_id: string, changes: Partial<RawEventDisplayItem> }
"demonstrate.display_item_excluded"
// Payload: { display_item_id: string }
"demonstrate.display_item_restored"
// Payload: { display_item_id: string }
"demonstrate.display_item_annotated"
// Payload: { display_item_id: string, annotation: string }
"demonstrate.display_item_manual_inserted"
// Payload: RawEventDisplayItem (with source_kind: "manual_step")
"demonstrate.display_items_reordered"
// Payload: { item_order: Array<{ display_item_id: string, new_display_index: number }> }
"demonstrate.display_items_grouped"
// Payload: { group_id: string, group_label: string, display_item_ids: string[] }
"demonstrate.display_items_ungrouped"
// Payload: { group_id: string }
"demonstrate.observation_paused_for_scope"
// Payload: { tab_id: string, reason: string }
"demonstrate.observation_resumed_for_scope"
// Payload: { tab_id: string }
"demonstrate.interpretation_started"
// Payload: { session_id: string, curated_item_count: number }
```
**§10.2 Event-Consumer Subscription Matrix update:**
| Component | Subscribes to |
|---|---|
| `LiveActionFeed` | All `demonstrate.display_item_*` events, `observation_paused/resumed`, `interpretation_started` |
## 11. Route Additions
### 11.1 Routes under existing families
**[Source: Patch 11]**
**[ADJ-58]** All demonstration-specific routes live under `/api/learn/`. The dispatch proof route lives under `/api/dispatches/` (DOC10's route family):
```http
# Feature support snapshot [Source: Patch 1]
GET /api/learn/feature-support
# Capture capability preflight [Source: H2]
GET /api/learn/capture-capability?app_bundle_id=X
# Trigger semantic interpretation
POST /api/learn/sessions/:sessionId/interpret
# Get the knowledge extraction bundle
GET /api/learn/sessions/:sessionId/demonstration-bundle
# Get raw observation events (review/debug)
GET /api/learn/sessions/:sessionId/raw-events
# Get observation evidence digest [Source: Patch 2]
GET /api/learn/sessions/:sessionId/observation-evidence
# [ADJ-24] All bundle mutations use bundle-edits (correct-bundle and edit-semantic-steps DEPRECATED)
POST /api/learn/sessions/:sessionId/bundle-edits
# Quick Promote [Source: Patch 3]
POST /api/learn/sessions/:sessionId/quick-promote
# Undo promote — archives nodes, available within 24h [Source: H2]
POST /api/learn/sessions/:sessionId/undo-promote
# Get adapter status
GET /api/learn/sessions/:sessionId/adapter-status
# [ADJ-52] Trust Mode settings
PATCH /api/learn/settings/trust-mode
# Get install saga status [Source: Patch 6]
GET /api/learn/install-sagas/:sagaId
# Ability proof route [Source: Patch 9]
GET /api/dispatches/:dispatchId/ability-proof
```
### 11.2 Key route contracts
```ts
// POST /api/learn/sessions/:sessionId/interpret
export const InterpretRequestSchema = z.object({
force_reinterpret: z.boolean().default(false),
});
export const InterpretResponseSchema = z.object({
bundle_id: z.string().uuid(),
bundle: KnowledgeExtractionBundleSchema,
interpretation_duration_ms: z.number().int(),
});
// GET /api/learn/capture-capability?app_bundle_id=X [Source: H2]
export const CaptureCapabilityResponseSchema = z.object({
app_bundle_id: z.string().max(120),
predicted_quality: z.enum(["high", "medium", "low", "unsupported"]),
available_adapters: z.array(DemonstrationAdapterSourceSchema),
blockers: z.array(z.object({
kind: z.enum(["tcc_missing", "adapter_unavailable", "app_not_installed"]),
description: z.string().max(200),
})),
recommended_config: AdapterSelectionResultSchema.optional(),
});
// POST /api/learn/sessions/:sessionId/undo-promote [Source: H2]
export const UndoPromoteRequestSchema = z.object({
confirm: z.boolean(), // Must be true
});
export const UndoPromoteResponseSchema = z.object({
archived_node_ids: z.array(z.string().max(160)),
// If nodes have been modified since promote, require manual review
requires_manual_review: z.boolean(),
modified_since_promote: z.array(z.string().max(160)),
});
// Available within 24 hours of promote. Archives nodes (doesn't delete).
// GET /api/dispatches/:dispatchId/ability-proof [Source: Patch 9]
export const AbilityProofResponseSchema = z.object({
dispatch_id: z.string().uuid(),
selection_receipt: z.object({
source: z.literal("doc10_routing"),
matched_procedure_id: z.string().max(160),
match_score: z.number(),
routing_path: z.string().max(500),
}),
rendering_receipt: z.object({
injection_card_rendered: z.boolean(),
rendering_tier: z.enum(["compact", "standard", "full"]),
token_cost: z.number().int(),
materialization_kind: z.enum(["graph_injection", "native_skill_bundle"]),
}),
// [ADJ-29] Execution outcome proof — completes the feedback loop
execution_outcome_receipt: z.object({
outcome: z.enum(["success", "failure", "partial", "aborted"]),
procedure_execution_event_id: z.string().uuid(),
confidence_event_applied: z.boolean(),
confidence_event_kind: z.string().max(80).optional(),
}).optional(),
explain_match: z.string().max(1000),
});
// [ADJ-29] Q MUST NOT render "learned ability used" unless
// execution_outcome_receipt.outcome = "success". If no receipt exists, render "proof pending."
// [ADJ-53] POST /api/learn/sessions/:sessionId/quick-promote
export const QuickPromoteRequestSchema = z.object({
scope: QuickPromoteScopeSchema.default("procedure_only_minimal"),
confirm: z.boolean(), // Required on first-in-app-family
});
export const QuickPromoteResponseSchema = z.object({
installed_ability_ids: z.array(z.string().max(160)),
pending_items_count: z.number().int(),
saga_id: z.string().uuid(),
});
```
---
## 12. Cross-Doc Enum Alignment
**[Source: A1]**
DOC3's interpretation pipeline uses internal vocabulary that must map to DOC72 canonical values. This section is the authoritative mapping:
```ts
// packages/contracts/src/learning/cross-doc-enum-map.ts
export const DemonstrationWriteCanonicalMap = {
// [ADJ-13] Source mapping REMOVED. The shared knowledge contract's `source` enum
// (KDA R2 §2.3) is the canonical vocabulary. DOC72 adopts this natively when it
// adopts the shared contract. No translation is required.
// source: REMOVED — use ProcedureKnowledgeContractSchema.source directly,
// Standing procedure initial lifecycle state
standing_procedure_initial_state: {
"draft": "candidate", // DOC72 uses "candidate" not "draft"
},
// Memory directive: lifecycle vs maturity separation
// DOC3 uses lifecycle_state for session state, DOC72 uses maturity for governance
memory_directive_maturity: {
// Default: "observation" (DOC1 governance)
// Bypass conditions: assertion_class == "durable_fact" AND confidence >= 0.85 AND user confirmed → "active"
default: "observation",
bypass_to: "active",
},
// Procedure lifecycle states during demonstration pipeline
procedure_lifecycle: {
quick_promoted: "validated", // Quick Promote → validated lifecycle
standard_path: "captured", // Standard path → captured lifecycle
approved: "active", // After full review → active lifecycle
},
} as const;
```
---
## 13. DOC72 Graph Writes — Through Existing Commands
### 13.1 Write path
**[Source: Patch 5]**
All demonstration writes go through DOC72's existing `entity_knowledge_write` command. No new command types are needed. **DOC3 must never write directly to DOC72 storage. All graph mutations flow through EC's `entity_knowledge_write` command.**
**[ADJ-17]** All graph writes in §13 MUST consume `BundleResolutionSchema`, not the raw bundle:
```ts
export const BundleResolutionSchema = z.object({
bundle_id: z.string().uuid(),
accepted_procedure_candidate_indices: z.array(z.number().int()).default([]),
accepted_existing_goal_links: z.array(z.object({
procedure_candidate_index: z.number().int(),
existing_goal_id: z.string().max(160),
})).default([]),
accepted_memory_directive_candidate_indices: z.array(z.number().int()).default([]),
accepted_standing_procedure_candidate_indices: z.array(z.number().int()).default([]),
accepted_new_goal_candidate_indices: z.array(z.number().int()).default([]),
schema_version: z.literal(1),
});
```
Under Quick Promote: `accepted_new_goal_candidate_indices`, `accepted_memory_directive_candidate_indices`, and `accepted_standing_procedure_candidate_indices` MUST be empty.
**[ADJ-27]** All writes originating from a single bundle install MUST run under a unified `write_set_id`. If any required write fails before the saga reaches `routable`, EC MUST soft-archive (`is_active = 0`) all nodes written under that ID and mark the saga `failed_partial_rollback`.
```ts
// packages/contracts/src/learning/demonstration-write-context.ts
import { z } from "zod";
export const DemonstrationWriteContextSchema = z.object({
source: z.literal("manual_demonstration"),
learn_session_id: z.string().uuid(),
bundle_id: z.string().uuid(),
supporting_trace_ids: z.array(z.string().uuid()),
quick_promoted: z.boolean(),
trust_mode_used: z.boolean().default(false),
interpretation_model: z.string().max(80),
observation_fidelity_score: z.number().min(0).max(1),
});
```
### 13.2 EC write-time Zod validation
**[Source: SR-32]**
EC's DAO layer MUST validate the JSON payload against the appropriate shared knowledge contract Zod schema before executing the SQLite INSERT/UPDATE:
```ts
// apps/ec-service/src/dao/node-payload-validator.ts
export const NodePayloadValidatorRegistry = {
procedure: ProcedureKnowledgeContractSchema,
memory_directive: MemoryDirectiveKnowledgeContractSchema,
domain_concept: DomainConceptKnowledgeContractSchema,
goal: GoalKnowledgeContractSchema,
obligation: ObligationKnowledgeContractSchema,
standing_procedure: StandingProcedureKnowledgeContractSchema,
} as const;
export function validateNodePayload(
nodeKind: keyof typeof NodePayloadValidatorRegistry,
payload: unknown,
) {
return NodePayloadValidatorRegistry[nodeKind].parse(payload);
}
```
Invalid payloads are rejected at write time. Each payload must carry `schema_version` for forward compatibility. Nightly drift audits scan for payloads that predate the current schema version.
**[ADJ-02]** EC MUST run a deterministic literal scrubber after LLM output and before graph write. If `literal_retention_class = "placeholder_only"` and the value fails the placeholder heuristic (no `[` brackets, not in the known-safe vocabulary list, and matches patterns for emails, phone numbers, case numbers, or exact matches against the active matter's protected entity names), EC MUST reject the payload with `LEARN_PRIVACY_VALIDATION_FAILED` and surface the offending field paths for user review. Quick Promote is blocked until all flagged fields are resolved.
**[ADJ-48]** EC SHALL reject memory_directive writes where `memory_type` and `assertion_class` are semantically incompatible. Valid pairs: heuristic→heuristic, preference→preference, constraint→constraint, standing_order→standing_order, vocabulary_mapping→vocabulary_rule. `style_profile`, `document_archetype`, and `correction` may use any assertion_class with explicit justification.
### 13.3 What gets written — procedure nodes
For each procedure candidate in the bundle:
```ts
// entity_knowledge_write with node_kind = "procedure"
{
command_type: "entity_knowledge_write",
node_kind: "procedure",
canonical_name: candidate.contract.canonical_name,
alpha: resolveInitialAlpha(bundle.bundle_source), // [ADJ-16] Dynamic per §25.2
// resolveInitialAlpha returns 2.0 + mode_alpha_increment (2.95 for demonstration, 2.50 for coaching, etc.)
beta: 2.0,
lifecycle_state: DemonstrationWriteCanonicalMap.procedure_lifecycle[
quick_promoted ? "quick_promoted" : "standard_path"
],
// The full shared knowledge contract IS the payload
payload: candidate.contract, // ProcedureKnowledgeContractSchema — validated by SR-32
edges: [
{ target_id: candidate.contract.environment.app_entity_id, relation_type: "part_of_application" },
// For composites: uses_procedure edges
...(candidate.contract.used_procedure_ids ?? []).map(id => ({
target_id: id, relation_type: "uses_procedure",
})),
// Goal links from bundle
...resolved_goal_ids.map(id => ({ target_id: id, relation_type: "serves_goal" })),
// Trace link
{ target_id: trace_node_id, relation_type: "validated_by_trace" },
// [ADJ-49] For composites spanning multiple apps:
...constituentAppEntityIds
.filter(id => id !== candidate.contract.environment.app_entity_id)
.map(id => ({ target_id: id, relation_type: "spans_application" })),
],
provenance: {
entry_type: "demonstration",
source_description: `Demonstrated by user in ${app_name}, session ${session_id}`,
source_ref: `learn_session:${session_id}`,
confidence_contribution: 0.95,
},
write_context: demonstration_write_context,
idempotency_key: `demo_${session_id}_proc_${candidate.candidate_index}`, // [Source: H2]
}
```
### 13.4 What gets written — memory directive nodes
**[Source: SR-24]**
For each memory directive candidate:
```ts
{
command_type: "entity_knowledge_write",
node_kind: "memory_directive",
canonical_name: candidate.contract.summary,
lifecycle_state: DemonstrationWriteCanonicalMap.memory_directive_maturity.default, // "observation"
// Maturity bypass: assertion_class == "durable_fact" AND confidence >= 0.85 AND user confirmed → "active"
payload: candidate.contract, // MemoryDirectiveKnowledgeContractSchema — includes assertion_class
edges: [
...linked_procedure_ids.map(id => ({
source_id: id, target_id: this_node_id, relation_type: "constrained_by",
})),
],
provenance: {
entry_type: "user_narration_during_demonstration",
source_ref: `learn_session:${session_id}`,
source_description: candidate.source_narration_id
? `User stated during demonstration`
: `Inferred from demonstration actions`,
},
write_context: demonstration_write_context,
idempotency_key: `demo_${session_id}_md_${candidate.candidate_index}`,
}
```
### 13.5 What gets written — execution trace
One `execution_trace` node per demonstration session:
```ts
{
command_type: "entity_knowledge_write",
node_kind: "execution_trace",
canonical_name: `Demonstration trace: ${composite_procedure_name}`,
payload: {
source_session_id: session_id,
raw_event_count: bundle.supporting_trace.raw_event_count,
narration_count: bundle.supporting_trace.narration_count,
capture_duration_seconds: bundle.supporting_trace.capture_duration_seconds,
adapter_sources_used: bundle.supporting_trace.adapter_sources_used,
observation_fidelity_score: bundle.supporting_trace.observation_fidelity_score,
app_bundle_id: target_app_bundle_id,
outcome: "demonstrated",
context_refs: {
matter_id: active_matter_id,
document_title: window_title,
},
},
}
```
### 13.6 What gets written — standing procedure candidates
Standing procedure candidates from demonstrations are written as `standing_procedure` nodes at `candidate` state (per cross-doc enum map §12) and require explicit user approval:
```ts
{
command_type: "entity_knowledge_write",
node_kind: "standing_procedure",
canonical_name: candidate.action_summary,
lifecycle_state: "candidate", // mapped from DOC3 "draft" per §12
payload: {
trigger_description: candidate.trigger_description,
action_summary: candidate.action_summary,
action_safety_class: candidate.safety_class,
source: "demonstration_proposed",
linked_procedure_ids: /* resolved from bundle */,
},
}
```
### 13.7 What gets written — goal nodes and edges
If the bundle contains goal candidates:
- If `is_existing_goal: true` and `existing_goal_id` is set: create `serves_goal` edges from procedures to the existing goal
- **[ADJ-17]** If `is_existing_goal: false`: create a new `goal` node at `suggested` lifecycle state, then create edges. **This branch applies ONLY to the reviewed install path, never to Quick Promote.** Under Quick Promote, `accepted_new_goal_candidate_indices` in `BundleResolutionSchema` MUST be empty.
### 13.8 Procedure versioning
**[Source: H2]**
```ts
// Extension to the procedure payload for versioning
export const ProcedureVersionMetadata = {
version: z.number().int().min(1).default(1),
prior_version_snapshot: z.unknown().optional(), // Full previous payload for rollback
};
```
When a procedure is corrected or updated, the version counter increments and the prior version snapshot is stored. This enables rollback if a correction degrades the procedure.
---
### 13.8 Durable Provenance — RawToSemanticMappingSchema (R2.2)
**[Source: R2.2 Proposal §5]**
Written atomically with the procedure node during the install saga, as part of the `write_set_id` batch (ADJ-27). Linked via `validated_by_trace` edge.
#### 13.8.1 Display Text Masking
Before persisting display text in the durable trace, apply masking to prevent PII/sensitive literal values from surviving the 24h raw event purge:
- **Type events:** "Type '{value}' in {label}" → mask the value using the same `literal_retention_class` logic as the interpretation pipeline. If the value resembles an email, case number, name, or path, mask to "Type '[value]' in {label}"
- **Navigation events:** Strip query parameters from URLs. "Navigate to amazon.com/s?k=whole+foods" → "Navigate to amazon.com/[path]"
- **Narration text:** Preserved as-is (the user chose to narrate; their intent is to provide context)
- **Element labels:** Preserved (these are UI chrome, not user content)
Each display step carries a `retention_class`:
```ts
retention_class: z.enum([
"safe_generic", // Element labels, menu names — safe to persist
"masked_literal", // Literal values replaced with [value] placeholder
"user_provided", // Narration or manual step — user explicitly provided
]);
```
#### 13.8.2 Schema
```ts
export const RawToSemanticMappingSchema = z.object({
mapping_id: z.string().uuid(),
procedure_id: z.string().max(160),
session_id: z.string().uuid(),
capture_modality: z.enum([
"observed_demonstration",
"coaching",
"conversational_capture",
]),
raw_display_steps: z.array(z.object({
display_item_id: z.string().uuid(),
display_index: z.number().int(),
display_text: z.string().max(200),
retention_class: z.enum(["safe_generic", "masked_literal", "user_provided"]),
event_category: z.string().max(40),
source_kind: z.enum(["raw_event", "manual_step", "narration", "system"]),
target_app: z.string().max(80).optional(),
excluded: z.boolean(),
user_annotation: z.string().max(500).optional(),
manually_inserted: z.boolean(),
timestamp: z.string().datetime(),
})),
semantic_steps: z.array(z.object({
step_index: z.number().int(),
intent: z.string().max(200),
source_display_indices: z.array(z.number().int()),
})),
interpretation_model: z.string().max(80),
interpretation_confidence: z.number().min(0).max(1).optional(),
observation_fidelity: z.number().min(0).max(1).optional(),
user_edits_before_interpretation: z.object({
steps_deleted: z.number().int().default(0),
steps_reordered: z.boolean().default(false),
steps_manually_inserted: z.number().int().default(0),
steps_annotated: z.number().int().default(0),
steps_grouped: z.number().int().default(0),
}),
environment_snapshot_at_capture: z.object({
app_bundle_id: z.string().max(120),
app_version_at_capture: z.string().max(40).optional(),
tools_used: z.array(z.string().max(120)).default([]),
required_capabilities: z.array(z.string().max(80)).default([]),
tcc_permissions_needed: z.array(z.string().max(40)).default([]),
platform: z.enum(["darwin", "linux", "win32"]),
tool_catalog_version_at_capture: z.string().max(40).optional(),
app_entity_id: z.string().max(160).optional(),
}).optional(), // Null for conversational capture
created_at: z.string().datetime(),
schema_version: z.literal(1),
});
```
#### 13.8.3 Provenance-Only Clarification
`environment_snapshot_at_capture` is **historical provenance only**. It MUST NOT be used to determine current routability, tool choice, or injection details. Current execution bindings live exclusively in DOC24/KDA execution-strategy cache and may be regenerated or invalidated independently.
#### 13.8.4 Relationship to Execution-Strategy Cache (KDA R2 §3.5)
The environment snapshot and the execution-strategy cache are complementary, not redundant:
- **Cache** (KDA §3.5): derived, rebuildable, runtime-current. Keyed (procedure_id, app_entity_id, tool_catalog_version). Per-step tool bindings. Invalidated when app tool catalog changes.
- **Snapshot** (this section): immutable, historical, written once at install time.
When KDA's invalidation cascade fires for a procedure (SR-39), compare the new tool catalog against the snapshot's `tools_used`. If a tool from the snapshot is no longer in the catalog, surface a "procedure may be stale" warning in the Skills & Connectors detail drawer (DOC20 §6.29.4.3) with the specific tool delta.
#### 13.8.5 Lifecycle
- Written atomically with procedure node during install saga. If saga rolls back (ADJ-27), trace node also rolls back.
- If user later edits the procedure, trace node is NOT updated — preserves original learning provenance.
- Survives 24h raw event purge because it stores masked display-text only.
#### 13.8.6 Conversational Capture Provenance
Conversationally-captured procedures (§25.8) produce a simplified mapping:
- `raw_display_steps` contains the user's message as a single display item with `source_kind: "system"` and `retention_class: "user_provided"`
- `semantic_steps` contains the extracted procedure steps; all `source_display_indices` reference index 0
- `environment_snapshot_at_capture` is null (no app observation occurred)
- `observation_fidelity` is null
- `capture_modality` is `"conversational_capture"`
---
## 14. DOC24 Delivery Integration
### 14.1 The DOC24 injection pivot — no SKILL.md runtime
**[Source: Problem 4]**
Graph-backed procedures execute through DOC24 direct injection, not SKILL.md materialization. The interpretation pipeline fills `ProcedureKnowledgeContractSchema`. DOC24 renders injection cards from those contracts. CIL injects them into the LLM's context. The LLM receives the same quality of instructions — but assembled on demand, at the right level of detail, from the single source of truth.
This changes DOC3 §0C.9A (execution model) and §0C.23 (materialization):
- **§0C.9A amended:** Execution for graph-backed procedures is via DOC24 injection cards, not SKILL.md loading. The LLM reads the injection card and performs the procedure using the tools and UI available.
- **§0C.23 scope reduced:** `materializeSkill()` is retained ONLY for: (a) native/imported skills not yet graph-ingested (Layer 1 truth), (b) optional export format for portability. It is NOT the runtime execution path for graph-backed procedures.
### 14.2 Install saga with two materialization kinds
**[Source: Patch 6, SR-16, SR-25]**
```ts
// packages/contracts/src/learning/install-saga.ts
import { z } from "zod";
export const AbilityMaterializationKindSchema = z.enum([
"graph_injection", // Graph-backed → DOC24 injection cards
"native_skill_bundle", // Native/imported → OpenClaw SKILL.md system
]);
export const InstallSagaStateSchema = z.enum([
// For graph_injection:
"graph_written",
"render_contract_validated",
"execution_strategy_cached",
"availability_refreshed",
"routable",
// For native_skill_bundle:
"bundle_materialized",
"workspace_projected",
"loadability_checked",
// Shared terminal:
"failed",
]);
export const InstallSagaRecordSchema = z.object({
saga_id: z.string().uuid(),
session_id: z.string().uuid(),
materialization_kind: AbilityMaterializationKindSchema,
current_state: InstallSagaStateSchema,
procedure_id: z.string().max(160),
started_at: z.string().datetime(),
completed_at: z.string().datetime().optional(),
failure_reason: z.string().max(500).optional(),
schema_version: z.literal(1),
});
```
**For `graph_injection`:**
1. `graph_written` — procedure payload written to DOC72 via `entity_knowledge_write`
2. `render_contract_validated` — confirm payload is renderable by DOC24 templates
3. `execution_strategy_cached` — resolve each step against app tools, populate execution-strategy cache. **This step is SYNCHRONOUS** — the saga does not advance until resolution completes. If resolution fails, procedure becomes routable but with all steps as `ui_navigation` fallback. **[Source: SR-16]**
4. `availability_refreshed` — ability availability snapshot updated
5. `routable` — trigger phrases indexed, ability discoverable via lookup
**For `native_skill_bundle`:**
1. `bundle_materialized` — SKILL.md generated from skill source
2. `workspace_projected` — SKILL.md projected to `~/.openclaw/workspace/skills/`
3. `loadability_checked` — manifest valid, no tool-name collisions
**[ADJ-25]** Partial failure rules for `graph_injection`:
- If `graph_written` succeeds but `render_contract_validated` fails: node remains `is_active = true`, `ability_usable_now = false`. Emit `LEARN_INSTALL_RENDERABILITY_FAILED`. Routing MUST exclude. Q shows `private_degraded` with repair CTA. EC queues re-render attempt on next DOC24 pipeline restart.
- If `execution_strategy_cached` fails: **[ADJ-59]** write all cache entries with `path: "ui_navigation"` (no tool directives). The canonical procedure payload in DOC72 remains unchanged — only the derived execution-strategy cache reflects the fallback. Node remains routable with `execution_strategy_mode = "semantic_only_fallback"`.
- If `availability_refreshed` fails: saga remains in retryable non-routable state. Retry policy: 3 attempts with exponential backoff (1s, 2s, 4s).
- Every non-terminal failure MUST set `repair_action` and `ability_usable_now = false`. Saga remains queryable from `GET /api/learn/install-sagas/:sagaId`.
**[ADJ-25]** Install saga failure schema:
```ts
export const InstallSagaFailureReasonSchema = z.enum([
"graph_write_failed",
"render_contract_invalid",
"execution_strategy_cache_failed",
"availability_refresh_failed",
]);
```
**Q rendering during saga:**
- In progress: "Installing..."
- `routable`: "Private active"
- `failed` or stalled: "Private degraded" with error detail and repair CTA
**If graph write itself fails:** Soft-delete any orphaned nodes (`is_active = 0`). **[ADJ-27]** If any write fails mid-bundle, all nodes written under the same `write_set_id` are soft-archived.
### 14.3 Ability availability and routing
**[Source: Patch 8, Patch 9]**
When a demonstrated procedure is installed, the existing DOC3 ability availability refresh triggers (DOC3 §0C.11B). The procedure becomes an entry in `AbilityAvailabilitySnapshot` with:
- `install_lane: "experimental_private"` (or `"approved_workspace"` if fully approved)
- `ability_usable_now: true`
- `materialization_kind: "graph_injection"`
**Default lookup includes `experimental_private`:** The default `install_lanes_allowed` in `AbilityLookupQuerySchema` must include `experimental_private`. Without this, Quick Promoted abilities are unfindable. Filter: `experimental_private` matches only show abilities owned by the requester. **[Source: Patch 8]**
**Explain-match response:** When an ability is used, the user can query the proof route (`GET /api/dispatches/:dispatchId/ability-proof`) to see why that ability was selected, at what rendering tier, and from what materialization kind. **[Source: Patch 9]**
### 14.4A Procedure Outcome Evaluation
**[ADJ-28]** After execution, EC SHALL evaluate the agent's final state/output against the procedure's `postconditions` and `verification` criteria:
- All verifiable postconditions satisfied + user accepts result → emit `validated_successful_use` (α +0.75)
- Postconditions partially satisfied → no confidence event (insufficient signal)
- Postconditions clearly failed + user confirms correction → emit `correction_after_runtime_failure` (β +1.0)
- First success after a prior correction → emit `correction_fix_validated` (α +0.25)
- Ambiguous or aborted → no confidence delta
EC is the confidence-event emitter. It consumes runtime execution receipts from DOC11 and evaluates them against stored procedure postconditions. DOC8 Satisfaction Matrix nightly verification MAY supplement with LLM-based deeper outcome analysis.
---
#### 14.4A.2 Conversational Capture Signals (R2.2)
**[Source: R2.2 Proposal §6]**
Add to BDSM §9.1 signal inventory:
```ts
export const ConversationalCaptureSignalSchema = z.object({
signal_type: z.literal("conversational_capture_offered"),
offer_id: z.string().uuid(),
disposition_id: z.string().uuid().optional(),
conversation_id: z.string().uuid(),
tier: z.enum(["tier_2"]),
outcome: z.enum([
"accepted", "declined", "extraction_was_wrong", "revised",
"dismissed_timeout", "dismissed_user_closed",
"captured_no_procedures_extracted",
]),
classifier_confidence: z.number().min(0).max(1),
predicted_node_kind: z.enum([
"invocable_procedure", "memory_directive", "standing_procedure", "hybrid_procedure", "ambiguous",
]),
context_class_key: z.string().max(80),
capture_mode_snapshot: z.enum(["active", "prompted", "passive"]),
target_agent_id: z.string().max(160),
first_offer_for_user: z.boolean(),
dedup_match_existing_id: z.string().max(160).optional(),
dedup_match_score: z.number().min(0).max(1).optional(),
timestamp: z.string().datetime(),
schema_version: z.literal(1),
});
export const PendingCandidateResolvedSignalSchema = z.object({
signal_type: z.literal("conversational_capture_pending_candidate_resolved"),
candidate_id: z.string().uuid(),
outcome: z.enum(["accepted", "declined", "deferred"]),
context_class_key: z.string().max(80),
timestamp: z.string().datetime(),
schema_version: z.literal(1),
});
```
**BDSM Gate 2 Consumption:** Gate 2 (experience-weighted relevance, §7.1) consumes both signals per `context_class_key`. Over time: high acceptance in `litigation` context → more aggressive Tier 2 offers; high decline in `personal` context → raises confidence threshold, fewer pop-ups.
**EC Core Blunt Throttle:** EC Core (not BDSM) tracks the recent decline rate across all Tier 2 offers. If 7 out of the last 10 were declined (`user_declined_capture` or `dismissed_user_closed`): EC shows a one-time settings suggestion: "I've been offering to save procedures and you've mostly declined. Would you like to switch to Prompted mode or Passive mode?" Transparent, user-controlled.
### 14.4 Routing discovery
**[Source: Problem 5, Patch 7 → SR-21]**
When the user later issues a request like "make this filing-ready," DOC24's semantic routing cascade:
1. Resolves "filing-ready" to the goal entity → traverses `serves_goal` edges → discovers the demonstrated composite procedure. **Goal-aware routing** traverses goal edges as a first-class discovery path. **[Source: Problem 5]**
2. The ability lookup contract (`POST /api/abilities/lookup`, DOC3 §0C.11C) finds the procedure via matching against `trigger_phrases` and `semantic_lookup_phrases` (NOT `use_conditions` — these are separate fields per SR-21)
3. DOC24 renders an injection card from the stored contract at the appropriate tier (compact/standard/full per token budget)
4. CIL injects the card into the LLM's context
**Routing decision observability:** Which procedures were considered and rejected during routing, with rejection reasons, is logged and surfaceable via `inspect_knowledge` tool. **[Source: H2]**
---
## 15. DOC1 Memory Governance Integration
### 15.1 Memory directive maturity
**[Source: SR-24]**
Memory directives from demonstrations enter at `observation` maturity by default. This prevents casual demonstration narration from silently becoming permanent behavioral rules.
**Maturity bypass conditions (per DOC72 §10.4):**
- User explicitly confirms the directive during bundle review (clicks "Accept" on the specific directive)
- Confidence ≥ 0.85
- `assertion_class` is `"durable_fact"` (per SR-24 — NOT keyed on `source_certainty`)
When all three conditions are met, the directive enters at `active` maturity.
`assertion_class` and `source_certainty` are DIFFERENT dimensions:
- `source_certainty` = HOW we know something (user stated, inferred, imported)
- `assertion_class` = WHAT KIND of claim (durable_fact, preference, constraint, standing_order, vocabulary_rule, heuristic)
### 15.2 Constraint precedence
Per DOC72 §10.3: if a procedure says "navigate to references" but a demonstrated preference says "use the keyboard shortcut," the preference wins at execution time. CIL injects the preference alongside the procedure, and the LLM adapts.
---
## 16. DOC8 Self-Learning Integration
### 16.1 Feedback signals from demonstrations
Demonstration outcomes feed DOC8 through DOC72 §11.6's existing feedback interface:
**ExtractionOutcomeEvent** for each demonstrated item:
```ts
{
source_surface: "demonstration",
extracted_item_kind: "procedure" | "memory_directive" | "standing_procedure",
outcome: "confirmed" | "used_successfully" | "rejected" | "corrected",
user_visible: true,
}
```
### 16.2 Execution outcome events
**[Source: SR-26]**
After a procedure executes (through any path), emit a `ProcedureExecutionOutcomeEvent`:
```ts
export const ProcedureExecutionOutcomeEventSchema = z.object({
procedure_id: z.string().max(160),
dispatch_id: z.string().uuid(),
execution_path_used: z.enum(["mcp_tool", "applescript", "ui_navigation", "mixed"]),
outcome: z.enum(["success", "partial_success", "failure", "aborted"]),
failed_step_index: z.number().int().optional(),
failure_reason: z.string().max(500).optional(),
duration_ms: z.number().int(),
user_accepted_result: z.boolean().optional(),
timestamp: z.string().datetime(),
});
```
This event feeds: (a) DOC72 confidence updates per SR-33, (b) DOC8 experience records, (c) execution trace node creation, (d) DOC24 injection manifest outcome tracking. Without this event, the confidence model has no way to learn from execution.
### 16.3 Skill mining promotion thresholds
**[Source: G3]**
When DOC8 detects a recurring pattern across multiple execution traces, it proposes a procedure candidate via the skill mining pipeline. Promotion thresholds:
```ts
const SKILL_MINING_THRESHOLDS = {
min_traces: 5, // At least 5 traces showing the pattern
min_success_rate: 0.80, // 80% of traces must be successful
min_confidence: 0.50, // Pattern confidence floor
min_distinct_contexts: 2, // At least 2 different matters/documents
min_calendar_day_span: 2, // Traces must span at least 2 calendar days
doc8_signal: "pattern_stable", // DOC8 must classify the pattern as stable
};
```
Mined candidates enter at `lifecycle_state: "captured"` with α=2.25 (base 2.0 + 0.25 mining), requiring user review before promotion.
### 16.4 What DOC8 learns from demonstrations over time
- **Interpretation quality:** Tracks confirmation/correction rate per app family. If Word demonstrations are frequently corrected, DOC8 flags the interpretation prompt for revision.
- **Promotion success:** Tracks whether quick-promoted skills are later corrected, quarantined, or promoted to approved. If quick-promoted skills are frequently corrected, DOC8 may recommend requiring full validation.
- **Generalization opportunities:** When the same procedure is demonstrated across multiple contexts (Henderson brief, Narayanan motion), DOC8 detects the cross-context pattern and proposes confidence promotion.
- **Standing procedure viability:** Tracks whether demonstration-proposed standing procedures are activated, used, or ignored.
---
## 17. Correction Propagation
### 17.1 When a demonstrated procedure is corrected
Corrections can come from:
- User correction during bundle review (immediate)
- User correction during later execution (via DOC72 §10.5 correction pipeline)
- DOC8 pattern detection (automated proposal)
- CANDOR red-team finding (DOC14)
### 17.2 What correction triggers
```ts
export const ProcedureCorrectionPropagationSchema = z.object({
corrected_procedure_id: z.string().max(160),
correction_kind: z.enum([
"step_updated",
"step_added",
"step_removed",
"precondition_changed",
"environment_changed",
"constraint_added",
"variant_created",
]),
affected_composite_procedure_ids: z.array(z.string().max(160)),
preserve_prior_variant_refs: z.array(z.string().max(160)).optional(),
reason: z.string().max(500),
schema_version: z.literal(1),
});
```
**Propagation actions:**
1. The corrected procedure node is updated in the graph (version incremented per §13.8)
2. The correction is recorded as a provenance entry
3. β is incremented per SR-33 confidence event table
4. All composite procedures using this procedure (via `uses_procedure` edges) are flagged for review
5. If the prior version of the procedure is still valid in other environments, it's preserved as a `specializes` variant
6. DOC24 execution-strategy cache entries for this procedure are invalidated
---
## 18. Consolidated Confidence Event Table
**[Source: SR-33, A5, SR-01]**
This is the single authoritative reference for all confidence events affecting procedure nodes. A coding agent implements this directly:
```ts
export const ProcedureConfidenceEventSchema = z.enum([
"accepted_manual_demonstration",
"validated_successful_use",
"correction_during_review",
"correction_after_runtime_failure",
"correction_fix_validated",
"app_version_changed",
]);
export const PROCEDURE_CONFIDENCE_DELTAS = {
accepted_manual_demonstration: { alpha: 0.95, beta: 0.0 },
validated_successful_use: { alpha: 0.75, beta: 0.0 },
correction_during_review: { alpha: 0.0, beta: 0.35 },
correction_after_runtime_failure: { alpha: 0.0, beta: 1.0 },
correction_fix_validated: { alpha: 0.25, beta: 0.0 },
app_version_changed: { alpha: 0.0, beta: 0.0 },
} as const;
```
**Rules:**
- `correction_fix_validated` is emitted ONLY after a subsequent validated success on the corrected procedure, not at correction time
- `app_version_changed` does NOT modify α/β — it sets `staleness_state = "verification_required"`, applies amber warning badge, and forces `confirm_first` safety class until next validated success **[Source: SR-01]**
- **[ADJ-44]** No `execution_score_cap` (per SR-01). Staleness verification after app version changes is enforced through `confirm_first` safety class and amber warning, not through a numeric cap on routing scores. See DOC24 R2.4 adjudication for rationale.
- After first post-update success: remove warning badge, restore normal safety class
---
## 19. Error Table
**[Source: H1, SR-29]**
| Error Code | User Message | Recovery | Telemetry |
|---|---|---|---|
| `LEARN_INTERPRETATION_LOW_CONFIDENCE` | "I couldn't reliably tell what you were teaching." | Open bundle editor + ask for narration | `learn.interpretation.low_confidence` |
| `PROCEDURE_MATCH_AMBIGUOUS` | "This looks similar to an existing ability." | Open merge review (§7.4) | `learn.dedup.review_required` |
| `ABILITY_AVAILABILITY_STALE` | "I saved the procedure but it's not discoverable yet." | Retry availability refresh | `learn.ability_sync.failed` |
| `LEARN_APP_UNSUPPORTED` | "No observation support for this app. Use voice narration or switch to a supported app." | Offer narration-only capture or app switch | `learn.capture.unsupported_app` |
| `LEARN_INTERPRETATION_EMPTY_BUNDLE` | "I couldn't extract any procedures from the demonstration. Try adding narration or demonstrating in smaller segments." | Open manual semantic authoring drawer | `learn.interpretation.empty_bundle` |
| `LEARN_INSTALL_RENDERABILITY_FAILED` | "Procedure saved but not yet usable. Check Knowledge Manager." | Keep node `is_active = true` but `usable_now = false`; surface as `private_degraded` | `learn.install.renderability_failed` |
| `LEARN_PRIVACY_VALIDATION_FAILED` | **[ADJ-02]** "Some values may contain specific information. Please review before saving." | Surface flagged fields in Privacy Review Banner. Quick Promote blocked until resolved. | `learn.privacy.validation_failed` |
**Gap detection:** If interpretation detects missing events (observation gaps), insert `UNOBSERVED` placeholder steps highlighted in amber for user review. This is a visual indicator, not an error code.
---
## 20. UI Surface Requirements
### 20.1 Entry points
- **Learn page → "Demonstrating Skill" card** — creates a `LearnSession` with `entry_mode: "demonstrating_skill"`
- **Contextual shortcut → "Watch My Actions"** — same, with `observation_mode: "outside_agent"` pre-selected
- **Conversational command** — "Elnor, watch how I do this" / "Learn how I make a caption page"
### 20.2 Menu bar status item and Q sidebar panel
**[Source: D5]**
The observation UI uses two surfaces to avoid interfering with the target app:
**Menu bar status item (macOS):** Always visible during active capture. Shows "Elnor is observing Word" with event counter. Click brings Q to front. When target app is fullscreen, this is the only visible indicator.
**Q sidebar panel (320px):**
- Structural label feed (major events only by default, toggle to show all)
- Adapter status indicator (green/amber/red per adapter)
- Observation fidelity score indicator (green ≥ 0.7, amber 0.4-0.7, red < 0.4)
- Narration input (typed fallback when PTT not available)
- Controls: Pause, Resume, Stop, Quick Promote
- Event counter: "23 events captured, 4 narrations"
- Collapsible. Keyboard shortcut ⌥⌘O to toggle.
- Auto-collapse after 3 seconds of no hover.
**Hotkeys:**
- ⌘⌥L — expand/collapse sidebar panel
- ⌘⌥P — pause/resume capture
- ⌘⌥S — stop capture
**Degraded states:**
- If TCC permission missing: compact amber banner, no fake "recording healthy" badge
- If adapter degraded: amber indicator with specific adapter status
### 20.3 Capture overlay component contract
**[Source: Patch 12]**
```ts
// apps/q-frontend/src/features/learn/LearnCaptureOverlay.tsx
export interface LearnCaptureOverlayProps {
sessionId: string;
state: LearnSessionState;
targetAppName: string;
eventCount: number;
narrationCount: number;
fidelityScore: number;
adapters: AdapterStatus[];
structuralLabels: StructuralLabelEntry[]; // Most recent N labels
significanceFilter: "major" | "all";
interpretationStatus: LiveInterpretationStatus;
onPause: () => void;
onResume: () => void;
onStop: () => void;
onQuickPromote: () => void;
onToggleSignificance: () => void;
installSagaState: InstallSagaState | null, // [ADJ-68] Non-null when segment promoted
onUndoPromote: (() => void) | null, // [ADJ-68] Available for 24h after segment promote
}
// Renders in Q sidebar panel (320px), NOT as floating overlay
```
### 20.3A Live Action Feed (R2.2)
**[Source: R2.2 Proposal §2]**
Replaces the event-count-only display with a numbered step list showing each action as it happens. Deterministic formatting — no LLM call for the live display.
#### 20.3A.1 RawEventDisplayItemSchema
```ts
export const RawEventDisplayItemSchema = z.object({
display_item_id: z.string().uuid(),
display_index: z.number().int(),
display_text: z.string().max(200),
source_kind: z.enum(["raw_event", "manual_step", "narration", "system"]),
raw_event_id: z.string().uuid().optional(), // Null for manual_step, narration, system
event_category: z.enum([
"click", "type", "navigate", "select_menu", "open", "close",
"scroll", "keystroke", "shortcut", "copy_paste", "drag_drop",
"context_menu", "modal_action", "file_picker",
"screenshot", "narration", "app_switch", "tab_switch",
"focus", "resize", "manual_step", "observation_paused", "unknown",
]),
target_label: z.string().max(120).optional(),
target_app: z.string().max(80).optional(),
timestamp: z.string().datetime(),
screenshot_thumbnail_ref: z.string().max(240).optional(),
has_narration: z.boolean().default(false),
narration_text: z.string().max(500).optional(),
excluded: z.boolean().default(false),
user_annotation: z.string().max(500).optional(),
manually_inserted: z.boolean().default(false),
group_id: z.string().uuid().optional(),
group_label: z.string().max(160).optional(),
format_confidence: z.number().min(0).max(1).default(1.0),
schema_version: z.literal(1),
});
```
#### 20.3A.2 formatRawEventForDisplay()
`formatRawEventForDisplay(event: RawObservationEvent): RawEventDisplayItem`
Deterministic mapping. Sets `format_confidence` based on label quality — when AX descriptions are absent or generic, confidence drops and display text shows a generic fallback.
**macOS AX events:**
| AX Event Type | Label? | Display text | format_confidence |
|---|---|---|---|
| AXPress | AXDescription | "Click '{description}'" | 1.0 |
| AXPress | AXRole only | "Click {role} element" | 0.6 |
| AXPress | Neither | "Click in {app_name}" | 0.3 |
| AXValueChanged | Label + short value ≤40 | "Type '{value}' in {label}" | 1.0 |
| AXValueChanged | Label + long value >40 | "Edit {label} field" | 0.8 |
| AXValueChanged | No label | "Edit field in {app_name}" | 0.3 |
| AXMenuItemSelected | Item label | "Select '{item}' from menu" | 1.0 |
| AXMenuOpened | Menu label | "Open '{menu}' menu" | 1.0 |
| AXWindowCreated | Title | "Open '{title}'" | 1.0 |
| AXURLChanged | URL | "Navigate to {domain}{path}" | 1.0 |
| AXFocusedUIElementChanged | Label | "Focus on '{label}'" | 0.7 |
| AXSelectedTextChanged | — | "Select text in {app}" | 0.8 |
| AXScrolled | Significant | "Scroll in {app}" | 0.5 |
**Additional patterns:**
| Pattern | Detection | Display text |
|---|---|---|
| Keyboard shortcut | Modifier+key within 200ms | "Press {mod}+{key}" (e.g., "Press ⌘B") |
| Copy/Paste | ⌘C / ⌘V or AXClipboardChanged | "Copy" / "Paste" |
| Undo/Redo | ⌘Z / ⌘⇧Z | "Undo" / "Redo" |
| Drag and drop | AXDrag sequence | "Drag '{source}' to '{target}'" |
| Context menu | Right-click → AXMenuOpened | "Right-click on '{element}'" |
| Modal dialog | AXWindowCreated with modal role | "Dialog: '{title}'" |
| File picker | AXWindowCreated with file dialog | "Open file picker" |
**Browser DOM events:** See §4.5A.6 mapping table.
**Special events:** screenshot → thumbnail inline; narration → "📎 '{text}'"; app switch → "Switch to {app}"; observation paused → "⏸ Observation paused — {reason}"
**Significance filtering:** Major events as numbered steps. Minor events collapsed as "N minor actions" (expandable). Noise hidden.
**Low-confidence display:** When `format_confidence < 0.5`, display shows generic text with annotation prompt: "What did you do here? [Add note]"
#### 20.3A.3 Per-Step Controls
| Control | When | Interaction | Effect |
|---|---|---|---|
| Delete | During + after | Trash icon on hover | `excluded: true`, dims |
| Undo delete | During + after | Click dimmed step | Restores |
| Annotate | During + after | Click step → text input | `user_annotation`, fed to prompt |
| Insert manual | After only | "+" between steps | `source_kind: "manual_step"` |
| Reorder | After only | Drag handle | Changes `display_index` |
| Copy | After only | Right-click → Copy | Copies `display_text` |
| Group | After only | Multi-select + Group | Collapses under `group_id` |
| Ungroup | After only | Click group → Ungroup | Restores individuals |
#### 20.3A.4 Capture Overlay Layout
Three vertical zones in Q sidebar panel (320px):
**Zone 1 (top, ~48px):** App icon+name, recording timer, pulsing red dot (●), fidelity score, Pause/Resume, Done/Stop.
**Zone 2 (middle, scrollable):** Numbered step list. Auto-scrolls during capture. Each step: number, action text, delete icon. Narration with 📎, screenshots inline, minor groups collapsed. Observation-paused banners inline. After Stop: full controls appear.
**Zone 3 (bottom, ~36px):** Adapter health badges, event count, contextual tips.
#### 20.3A.5 Stop → Interpretation → Review Transition
1. **Freeze.** Zone 2 stops scrolling. All controls available.
2. **Pre-interpretation editing.** User cleans up. "Start Interpretation" button appears. Trust Mode: auto-start after 3s.
3. **Interpretation.** Progress bar: "Interpreting {N} events..."
4. **Complete.** Raw steps fade to "Show {N} raw events" toggle. Semantic steps appear with raw→semantic mapping: "Derived from steps 1-3, 5" (clickable).
#### 20.3A.6 Pre-interpretation → Prompt Integration
User edits modify interpretation input per §6.9 steps 3A-3B:
- Deleted steps stripped. Prompt: "User removed {N} events."
- Annotations injected alongside events.
- Manual steps as first-class `CuratedInterpretationItem` entries.
- Reordered steps in user's `display_index` order. Prompt: "User reordered."
- Grouped steps as composite. Biases LLM toward one semantic step per group.
### 20.4 Bundle review panel
**[Source: D2, Patch 12]**
After interpretation completes:
- **Summary view (default):** One-line items grouped by type with inline accept/reject. Observation fidelity score displayed.
- **Detail view:** Full steps, preconditions, constraints, parameters, verification.
- **Quick Promote button** with eligibility indicator and scope footer text.
- **Trust & Quick Promote button** (when Trust Mode enabled for this app family).
- **Full Review button** — enters standard proposal pipeline.
- **Bundle edit toolbar** — step-level editing, splitting, merging, reclassification.
### 20.5 Quick Promote drawer component contract
**[Source: Patch 12]**
```ts
// apps/q-frontend/src/features/learn/QuickPromoteDrawer.tsx
export interface QuickPromoteDrawerProps {
bundleId: string;
eligibleProcedures: ProcedureCandidate[];
pendingItems: { // Items shunted to Pending Abilities
memoryDirectives: MemoryDirectiveCandidate[];
standingProcedures: StandingProcedureCandidate[];
goals: GoalCandidate[];
};
isFirstInAppFamily: boolean; // If true, show confirmation prompt
trustModeAvailable: boolean;
onConfirmQuickPromote: (scope: QuickPromoteScope) => void;
onSwitchToFullReview: () => void;
onCancel: () => void;
}
// Footer: "This installs the procedure now. Preferences, constraints, and standing automations stay in Pending Abilities."
```
### 20.6 Ability proof drawer
**[Source: Patch 12]**
```ts
// apps/q-frontend/src/features/learn/AbilityProofDrawer.tsx
export interface AbilityProofDrawerProps {
dispatchId: string;
proof: AbilityProofResponse;
onClose: () => void;
}
// Shows: why this ability matched, rendering tier, token cost, materialization kind
```
### 20.7 Required UI states
| State | What the user sees |
|---|---|
| Preflight checking | "Checking observation support for {app}..." |
| Screenshot consent | "This demonstration will capture screenshots..." with Enable/Decline buttons |
| Armed | "Ready to observe. Switch to {app} and start working." |
| Target detected | "{app} detected. Capturing your actions." |
| Capturing | Sidebar panel with structural label feed, event counter, fidelity indicator |
| Paused | "Observation paused. Resume when ready." |
| Stopped / Interpreting | "Interpreting your demonstration..." with progress indicator |
| Interpretation streaming | Partial results appearing as procedures are extracted |
| Interpretation complete | Bundle review panel (summary view) |
| Quick promoted | **[ADJ-38]** "Private experimental ability created: {name}. [Undo] Review recommended." Toast with Undo button active for 24 hours. Also available in Knowledge Manager recent activity. Calls `undo-promote` route. |
| Trust mode promoted | "Trusted: {N} abilities created. {M} items need review." |
| Interpretation failed | "Couldn't interpret the demonstration. Try with more narration or a simpler workflow." |
| App unsupported | "No observation support for {app}. Use voice narration or try a different app." |
| Empty bundle | "Couldn't extract procedures. Try narrating or demonstrating smaller segments." |
| Adapter unavailable | "Observation adapters are not available for {app}. You can describe the workflow instead." |
| Adapter degraded | "Some observation sources are limited. Results may be less detailed." Fidelity score shown. |
| Install saga in progress | "Installing..." with saga state indicator |
| Install complete | "Private active" |
| Install degraded | "Private degraded — not yet discoverable" **[ADJ-54]** UI MUST poll `GET /api/learn/install-sagas/:sagaId` every 1s for 30s, then every 5s up to 5 minutes, then surface retry CTA. On saga transition to `routable`, refresh and show "Private active." |
| Preflight failed | **[ADJ-40]** "Cannot observe {app}: {blocker description}. {remediation}." With specific guidance per blocker type (TCC missing → 'Grant Accessibility access in System Settings', adapter unavailable → 'Use voice narration instead'). |
| Low fidelity (< 0.6) | "Limited observation — review each step carefully." Quick Promote disabled. |
---
## 21. Telemetry and Receipts
### 21.1 Learning receipts additions
Add to existing DOC3 §0C.12 receipt stream:
- `demonstration.session_started` — with target app, adapter configuration, narration_optional flag
- `demonstration.target_app_detected` — with app name, bundle ID
- `demonstration.capture_started` — first meaningful event received
- `demonstration.narration_received` — narration captured (count, not content)
- `demonstration.capture_stopped` — with event count, duration, fidelity score
- `demonstration.interpretation_started` — with model, event count
- `demonstration.interpretation_progress` — streaming partial result
- `demonstration.interpretation_complete` — with bundle summary, quality score
- `demonstration.interpretation_timeout` — hard timeout with partial count
- `demonstration.bundle_corrected` — with correction count, provenance severed count
- `demonstration.bundle_edited` — with edit operation types applied
- `demonstration.quick_promoted` — with ability ID, procedure count, scope
- `demonstration.trust_mode_promoted` — with auto-accepted count, manual review count
- `demonstration.install_saga_progress` — with saga ID, state transition
- `demonstration.graph_writes_complete` — with node IDs created
- `demonstration.adapter_error` — with adapter source, error code
- `demonstration.undo_promoted` — with archived node IDs
- `demonstration.dedup_match_found` — with match score, action taken
- `demonstration.execution_outcome` — from ProcedureExecutionOutcomeEvent
---
## 22. File Layout Additions
### 22.1 Contract files
```text
packages/contracts/src/learning/raw-observation-event.ts
packages/contracts/src/learning/narration-capture.ts
packages/contracts/src/learning/knowledge-extraction-bundle.ts
packages/contracts/src/learning/demonstration-write-context.ts
packages/contracts/src/learning/procedure-correction-propagation.ts
packages/contracts/src/learning/ax-bridge-adapter.ts
packages/contracts/src/learning/screenshot-adapter.ts
packages/contracts/src/learning/applescript-adapter.ts
packages/contracts/src/learning/install-saga.ts
packages/contracts/src/learning/quick-promote.ts
packages/contracts/src/learning/bundle-edit.ts
packages/contracts/src/learning/cross-doc-enum-map.ts
packages/contracts/src/learning/confidence-events.ts
```
### 22.2 EC service files
```text
apps/ec-service/src/learning/demonstration-adapter-manager.ts
apps/ec-service/src/learning/demonstration-capture-buffer.ts
apps/ec-service/src/learning/structural-label.ts
apps/ec-service/src/learning/event-significance.ts
apps/ec-service/src/learning/narration-classifier.ts
apps/ec-service/src/learning/semantic-interpreter.ts
apps/ec-service/src/learning/composite-detection.ts
apps/ec-service/src/learning/observation-fidelity.ts
apps/ec-service/src/learning/dedup-pipeline.ts
apps/ec-service/src/learning/dedup-hard-vetoes.ts
apps/ec-service/src/learning/demonstration-bundle-writer.ts
apps/ec-service/src/learning/quick-promote-eligibility.ts
apps/ec-service/src/learning/correction-propagation.ts
apps/ec-service/src/learning/install-saga-manager.ts
apps/ec-service/src/learning/app-focus-detector.ts
apps/ec-service/src/learning/undo-detector.ts
apps/ec-service/src/dao/node-payload-validator.ts
```
### 22.3 Q frontend files
```text
apps/q-frontend/src/features/learn/LearnCaptureOverlay.tsx
apps/q-frontend/src/features/learn/StructuralLabelFeed.tsx
apps/q-frontend/src/features/learn/BundleReviewPanel.tsx
apps/q-frontend/src/features/learn/BundleEditToolbar.tsx
apps/q-frontend/src/features/learn/AdapterStatusIndicator.tsx
apps/q-frontend/src/features/learn/QuickPromoteDrawer.tsx
apps/q-frontend/src/features/learn/AbilityProofDrawer.tsx
apps/q-frontend/src/features/learn/FidelityIndicator.tsx
apps/q-frontend/src/features/learn/TrustModeToggle.tsx
```
### 22.4 Ephemeral storage paths
```text
ELNOR_MEMORY/system/learning/demonstrations/<session_id>/raw_events.jsonl
ELNOR_MEMORY/system/learning/demonstrations/<session_id>/narrations.jsonl
ELNOR_MEMORY/system/learning/demonstrations/<session_id>/screenshots/
ELNOR_MEMORY/system/learning/demonstrations/<session_id>/bundle.json
ELNOR_MEMORY/system/learning/demonstrations/<session_id>/adapter_config.json
ELNOR_MEMORY/system/learning/demonstrations/<session_id>/segments/
ELNOR_MEMORY/system/learning/demonstrations/<session_id>/evidence_digest.json
```
All ephemeral. Raw events purged 24h after session terminal. Masked evidence digest retained 30 days.
---
## 23. DOC11 Integration — OpenClaw Gateway Seam
### 23.1 Gateway observation mode
DOC11 must expose a mechanism for EC to activate passive observation through the OpenClaw macOS stack:
```ts
type GatewayObservationRequest = {
session_id: string;
mode: "start_observation" | "stop_observation" | "query_status";
target_app_bundle_ids: string[]; // Plural for multi-app [Source: C4]
adapter_sources: DemonstrationAdapterSource[];
adapter_configs: Record<string, unknown>;
};
type GatewayObservationResponse = {
session_id: string;
status: "active" | "stopped" | "error";
active_adapters: Array<{
source: DemonstrationAdapterSource;
healthy: boolean;
events_captured: number;
}>;
event_stream_endpoint: string;
};
```
### 23.2 Event flow
```
User acts in Word
→ macOS Accessibility API fires AX notifications
→ OpenClaw macOS app / PeekabooBridge captures AX event
→ Gateway forwards event to EC via local WebSocket event stream
→ EC classifies significance [D1]
→ EC checks for undo [Problem 3]
→ EC buffers event in demonstration session store
→ EC computes structural label
→ EC emits SSE event to Q frontend
→ Q renders structural label chip in sidebar panel (major events only)
```
### 23.3 Runtime truth boundary
OpenClaw's native runtime owns the actual observation mechanism (what AX events are captured, what screenshots are taken). EC owns the interpretation and knowledge writing. DOC11 is the boundary.
### 23.4 Gateway error and reliability contract
**[ADJ-31]** The DOC11 observation seam requires the following for independent implementability:
**Error codes:**
```ts
export const GatewayObservationErrorSchema = z.enum([
"GATEWAY_UNAVAILABLE",
"TCC_REVOKED",
"APP_TERMINATED",
"ADAPTER_INIT_FAILED",
]);
```
**Event sequencing:** Gateway events MUST carry a monotonic `gateway_seq` integer. EC MUST ACK the highest contiguous seq received. On gap detection (missing seq number), EC MUST request replay from the missing seq.
**Reconnection policy:** If the WebSocket event stream disconnects, EC retries 3 times with exponential backoff (1s, 2s, 4s). If all retries fail, the session transitions to `stopped` with `adapter_error` receipt and partial events are preserved for interpretation.
**Scope:** Gateway requests are localhost-only, no auth required. Rate limit: max 10 requests/second per session.
---
## 24. Normative End-to-End Sequence
**[Source: SR-30]**
This is the normative state-transition sequence with required handoffs and acceptance criteria at each stage:
```
capture → [app focused, adapters healthy, TCC granted]
→ stopped [interpretation_phase = "running"] → [bundle non-empty, fidelity ≥ threshold] [ADJ-05]
→ review → [user accepted items, scope-leak check passed (§8.4), BundleResolution gated (ADJ-17)]
→ promote → [routing hints present (≥ 3 trigger phrases), lint passed, contracts validated]
→ install saga → [graph_written → render_contract_validated → execution_strategy_cached → availability_refreshed] [ADJ-26]
→ routable → [trigger phrases indexed, ability in snapshot, materialization_kind set]
→ discovered → [routing match, relevance scored, explain-match available]
→ rendered → [tier selected, budget checked, stale directives stripped]
→ executed → [outcome event emitted (SR-26), EC evaluates postconditions (ADJ-28)]
→ confidence updated → [per SR-33 event table]
```
**[ADJ-08]** If interpretation fails (`interpretation_phase = "failed"`), retry transition: `stopped [interpretation_phase = "failed"] → stopped [interpretation_phase = "idle"]` when user clicks "Retry with narration."
Each `→` transition has a required precondition in brackets. If the precondition fails, the transition produces a defined error code from §19 and a defined recovery action.
---
## 25. Generalization to All Learning Modes
### 25.1 The bundle pipeline is the universal output
The `KnowledgeExtractionBundle` defined in §6.5 serves as the universal output for ALL DOC3 learning modes, not just `demonstrating_skill`. The observation layer differs per mode, but the interpretation → bundle → graph write path is identical.
### 25.2 Per-mode mapping
| Learning Mode | Raw Input | Observation Mechanism | Confidence Boost (α) | Source (DOC72 canonical) | Notes |
|---|---|---|---|---|---|
| **Demonstrating Skill** | AX events + screenshots + narration | accessibility_bridge, screenshot_vision, applescript_state | +0.95 | `"manual"` | Highest confidence — user explicitly taught |
| **Coaching Agent** | Agent execution traces + MCP receipts + user guidance | mcp_receipt, wrapper_receipt | varies (see §25.3) | `"doc3_learned"` | Per-step confirmation required [Source: G1] |
| **Autonomous Agent Practice** | Agent execution traces only | mcp_receipt, wrapper_receipt | +0.50 | `"doc3_learned"` | Capped at 10/app, 30 total unreviewed [Source: G2] |
| **Improving Existing Skill** | Delta against existing procedure | Same as original mode | +0.25 (incremental) | `"doc3_learned"` | Updates existing node. Creates `specializes` variant if contextual. |
| **Import Skill** | SKILL.md text content | LLM decomposition | +0.50 | `"imported"` | Three-tier quality: full/partial/failed |
| **Learn From This** | Completed execution traces (retrospective) | Existing trace data | +0.75 | `"trace_captured"` | Easiest to implement — traces already exist |
| **Skill Mining** (background) | Patterns across multiple traces | DOC8 pattern detection | +0.25 | `"doc3_learned"` | Requires 5 traces, 0.80 success, 2 contexts [Source: G3] |
| **Native Skill Ingestion** | Existing SKILL.md files | LLM decomposition | +1.00 (α=4.0, β=2.0) | `"system"` | High confidence — native skills are trusted |
### 25.3 Coaching mode — per-step confirmation
**[Source: G1]**
Coaching-mode procedures require per-step user confirmation:
- Procedures start at `lifecycle_state: "captured"` not `"validated"`
- Agent-performed steps are marked in the review panel with per-step confirm/reject
- Unconfirmed agent steps: α +0.50 (lower than confirmed +0.75)
- User-guided steps: auto-confirmed
### 25.4 Autonomous practice safety caps
**[Source: G2]**
```ts
const AUTONOMOUS_PRACTICE_CAPS = {
max_unreviewed_per_app: 10,
max_unreviewed_total: 30,
// When cap reached: pause writing new candidates, surface notification
nightly_cleanup: {
// Candidates at "captured" with confidence < 0.35 and no traces after 30 days → archive
confidence_floor: 0.35,
max_age_days: 30,
},
};
```
### 25.5 Import decomposition detail
When a SKILL.md is imported (or a native skill is ingested), the LLM decomposes its instructions into procedure nodes, application entity links, preconditions/postconditions, and constraint memory directives.
**Quality flag:**
```ts
type ImportDecompositionQuality = "full" | "partial" | "failed";
```
- `full`: Clean semantic steps extracted. Graph nodes created. SKILL.md becomes Layer 2 projection.
- `partial`: Some steps extracted. Both graph nodes and original SKILL.md available. Hybrid runtime: graph for decomposed portions, SKILL.md for un-decomposed.
- `failed`: Decomposition not possible. SKILL.md used as-is through standard OpenClaw ReAct loop. Retry after 14 days.
### 25.6 Native skill ingestion pipeline
```ts
export async function ingestNativeSkill(skillPath: string): Promise<KnowledgeExtractionBundle> {
// 1. Read SKILL.md content
// 2. Send to LLM for decomposition (same prompt structure as §6.4 but with SKILL.md text as input)
// 3. Produce KnowledgeExtractionBundle with bundle_source: "native_skill_ingestion"
// 4. Write graph nodes with source: "system", high confidence (α=4.0, β=2.0)
// 5. Mark the SKILL.md as a projection of the new graph nodes
// Idempotency: if graph nodes already exist (canonical_name + app entity match), enrich don't duplicate
}
```
### 25.7 Implementation priority
1. **Learn From This** — traces already exist, no observation needed. Highest ROI.
2. **Demonstrating Skill** — the core of this proposal. Requires observation layer.
3. **Native Skill Ingestion** — populate the graph from day one.
4. **Coaching Agent** — agent traces feed the same pipeline.
5. **Import Skill** decomposition.
6. **Autonomous Agent Practice** — same pipeline, lower confidence, safety caps.
7. **Improving Existing Skill** — delta generation against existing graph nodes.
8. **Skill Mining** — background pattern detection. Depends on DOC8 integration.
---
### 25.8 Conversational Procedure Capture (R2.2)
**[Source: R2.2 Proposal §3]**
A new learning mode for capturing procedures from normal conversation without entering a formal learning session. All tiers are entry points into the same pipeline — same `KnowledgeExtractionBundleSchema` output (with `bundle_source: "conversational_capture"`), same DOC3 interpretation, same DOC72 write, same dedup check.
#### 25.8.1 Learning Eligibility Gate
Before any capture attempt (Tier 1, 2, or 3), check pre-detection gates **in order**:
1. **Global pause learning** (DOC72 §20.10) active → no detection. No Tier 2 pop-up. No Tier 3 tagging. Master switch.
2. **PropA `effective_collection_mode`** for this conversation is `blocked` → no detection. Per-conversation switch.
3. **Capture mode = Passive** (§25.8.8) → detection MAY run for telemetry but no Tier 2 pop-up. Tier 3 tagging allowed.
4. **Capture mode = Prompted** → detection runs. Subtle indicator (lightbulb) instead of pop-up.
5. **Capture mode = Active** → full Tier 2 pop-up flow.
Tier 1 (Learn button) always works if gates 1 and 2 pass.
#### 25.8.2 Two-Stage Classification
The classifier runs on each user message (Tier 2) or batched transcripts (Tier 3):
```ts
export const ConversationalProcedureClassificationSchema = z.object({
classification: z.enum([
"invocable_procedure", // Multi-step, user-request-triggered
"standing_procedure", // Event-trigger + action
"hybrid_procedure", // Event trigger + sequential steps
"memory_directive", // Preference, constraint, style rule — no steps
"one_time_instruction", // Do this now, not reusable
"ambiguous",
]),
predicted_node_kind: z.enum([
"invocable_procedure", "standing_procedure", "hybrid_procedure",
"memory_directive", "ambiguous",
]),
capture_tier: z.enum(["tier_1", "tier_2", "tier_3", "tier_4"]),
classifier_confidence: z.number().min(0).max(1),
trigger_kind: z.enum([
"explicit_teach", "user_request_trigger", "event_trigger",
"style_preference", "one_time_task", "none",
]),
reason_codes: z.array(z.string().max(80)),
schema_version: z.literal(1),
});
```
**Two stages:**
Stage 1 — Pattern detection: has_temporal_trigger, has_imperative_sequence, has_process_naming, has_standing_order.
Stage 2 — Output-class prediction:
- Sequential steps (≥2 ordered imperatives) + temporal trigger → `hybrid_procedure`
- Sequential steps + no event trigger → `invocable_procedure`
- Event trigger + no sequential steps → `standing_procedure`
- Assertion only (preference/constraint/style) → `memory_directive`
- Insufficient signal → `ambiguous`
**Classifier precedence:** If temporal trigger + preference/constraint but NO ordered steps → `memory_directive`, NOT procedure. "Whenever you write an email, use formal language" → `memory_directive`. "When drafting client emails, first summarize, then provide context, then close with next steps" → `invocable_procedure`.
**Tier thresholds:**
- Tier 2: `classifier_confidence ≥ 0.82`, classification is procedure/standing/hybrid/directive with high confidence
- Tier 3: `0.55–0.81`, possible knowledge but uncertain
- Tier 4: `< 0.55`, or `one_time_instruction`
#### 25.8.3 Tier 1 — Explicit Capture (Learn Button)
Chat panel **Learn** button (icon). Dropdown with two modes:
**"Describe a process"** — one-shot. User types procedure description in next message → DOC3 coaching-mode interpretation (§25.3) synchronously → `KnowledgeExtractionBundle` → procedure available immediately → toast: "Saved as '[name].' View on Skills & Connectors." Mode deactivates after send. NOT latching.
If interpretation produces zero procedures:
```ts
export const ConversationalCaptureResultSchema = z.object({
result: z.enum([
"procedure_extracted",
"memory_directive_extracted",
"standing_procedure_extracted",
"no_extractable_knowledge",
"blocked_by_policy",
]),
candidate_counts: z.object({
procedures: z.number().int(),
memory_directives: z.number().int(),
standing_procedures: z.number().int(),
}),
next_actions: z.array(z.enum([
"revise_description", "save_as_memory_directive", "discard",
])).default([]),
schema_version: z.literal(1),
});
```
Toast: "I couldn't extract a procedure. [Revise description] [Save as preference] [Discard]"
**"Teach [agent name]"** — opens the recorder (§20.3A). Agent name dynamic, never hardcoded. Full demonstration mode.
**Initial confidence:** `mode_alpha_increment: +0.75`.
#### 25.8.4 Tier 2 — High-Confidence Detection (UI Pop-up)
When classifier detects high-confidence knowledge (`classifier_confidence ≥ 0.82`) and eligibility gate passes:
**Pre-pop-up dedup check:** Run §7 dedup (Stage 0.5 + Stage 1 + Stage 2, <50ms) against extracted candidate. If match found:
```
┌────────────────────────────────────────┐
│ Update existing procedure? │
│ "Spec Review Process" │
│ This looks like an update to a │
│ procedure you already have. │
│ [Yes, update] [Save as new] [No] │
└────────────────────────────────────────┘
```
If no match, adaptive pop-up by `predicted_node_kind`:
| predicted_node_kind | Pop-up text |
|---|---|
| `invocable_procedure` | "Save as procedure? '[name]'" |
| `memory_directive` | "Save as preference? '[summary]'" |
| `standing_procedure` | "Save as standing rule? '[summary]'" |
| `hybrid_procedure` | "Save as procedure? '[name]' (includes trigger)" |
| `ambiguous` | "Save as standing rule? '[summary]'" + "[Save as procedure instead]" |
Multi-agent: shows "Save as procedure for {agent name}?" with "▾ Other agents" affordance.
**Buttons:**
- **Yes** — DOC3 pipeline silently. Toast: "Saved."
- **No** — `user_declined_capture`. Tier 3 skips.
- **That's wrong** — `extraction_was_wrong`. Tier 3 still processes.
- **Revise** — activates Tier 1 with extracted context pre-loaded. Same pipeline.
- **Dismissed (timeout, 60s)** — `dismissed_timeout`. Tier 3 safety net.
- **Dismissed (user closed)** — `dismissed_user_closed`. Tier 3 skips.
**Initial confidence:** `mode_alpha_increment: +0.50`.
**Onboarding:** Before `procedure_capture_discovery` milestone, pop-up includes explainer. After milestone, bare pop-up.
**Cold-start throttle:** First 14 days OR until 3 accepted offers: max 2 Tier 2 pop-ups/day. After decline, suppress `context_class_key` for 24 hours.
#### 25.8.5 Tier 3 — Medium-Confidence Detection (Nightly Review)
For `classifier_confidence` 0.55–0.81:
1. EC tags utterance with `intake_tag: "possible_procedure_candidate"`.
2. DOC72 nightly conversation mining processes tagged utterances.
3. PropA `P0_master_extraction` produces lightweight candidates.
4. Candidates appear in **Pending Abilities** next morning.
5. User accepts → DOC3 full interpretation → rich `ProcedureKnowledgeContractSchema`.
6. `mode_alpha_increment: +0.30`.
**DOC72 ASSESS expansion — extraction facets:**
```ts
extraction_facets: z.array(z.enum([
"memory_directive", "standing_procedure", "invocable_procedure",
"hybrid_procedure", "obligation", "entity_fact",
]));
```
Precedence:
- Event trigger + action, no steps → `standing_procedure` → existing DOC72 path
- User-invoked + sequential steps, no event trigger → `invocable_procedure` → DOC3 Tier 3
- Event trigger + sequential steps → `hybrid_procedure` → BOTH a standing_procedure node (trigger) AND an invocable_procedure node (steps), linked via `triggers_procedure` edge
- Assertion only → `memory_directive`
#### 25.8.6 Tier 4 — Not a Procedure
"Call me Will", "I prefer Bluebook citations" — no steps, no trigger, no reuse. Normal DOC72 conversation mining. DOC1 governance. No DOC3 involvement. Tier 4 is a classification result, not a DOC3 entry point.
**DOC1 clarification:** DOC72 §3.5 already unifies DOC1 memories as `memory_directive` nodes in DOC72 SQLite. DOC1 owns governance, not storage. No changes.
#### 25.8.7 Disposition Records
```ts
export const ConversationalCaptureDispositionSchema = z.object({
disposition_id: z.string().uuid(),
offer_id: z.string().uuid(),
utterance_ref: z.string().max(240),
utterance_hash: z.string().max(160),
conversation_id: z.string().uuid(),
tier: z.enum(["tier_1", "tier_2"]),
disposition: z.enum([
"captured",
"captured_no_procedures_extracted",
"user_declined_capture",
"extraction_was_wrong",
"revised_via_tier1",
"dismissed_timeout",
"dismissed_user_closed",
]),
classifier_confidence: z.number().min(0).max(1).optional(),
capture_mode_snapshot: z.enum(["active", "prompted", "passive"]),
procedure_id: z.string().max(160).optional(),
target_agent_id: z.string().max(160),
target_agent_display_name: z.string().max(80),
disposed_at: z.string().datetime(),
schema_version: z.literal(1),
});
```
**Tier 3 coordination:** Check disposition by `utterance_hash`:
- `captured`, `user_declined_capture`, `revised_via_tier1`, `dismissed_user_closed` → skip
- `extraction_was_wrong`, `captured_no_procedures_extracted` → process (second chance)
- `dismissed_timeout` → process (user may not have seen)
- No disposition → process normally
#### 25.8.8 Capture Mode Setting
| Mode | Tier 2 | Tier 3 | Note |
|---|---|---|---|
| **Active** | Auto-detect + pop-up (cold-start throttle) | Normal | Full proactive capture |
| **Prompted** | Subtle lightbulb indicator | Normal | Default during first week |
| **Passive** | No real-time detection | Normal | Manual only via Learn button |
Tier 1 (Learn button) always available regardless.
#### 25.8.9 Scope and Agent Visibility
**Scope inference:** `principal_id`, `scope` (personal/firm_shared/matter_scoped), `scope_inference_basis: "inferred_from_conversation_context"` (add to DOC72 canonical values). User-overridable via Skills & Connectors.
**Agent visibility:**
```ts
agent_visibility: z.object({
default_policy: z.enum([
"creator_agent_only", // Default for personal
"all_principal_agents", // Default for firm_shared
]),
creator_agent_id: z.string().max(160),
approved_agent_ids: z.array(z.string().max(160)).default([]),
denied_agent_ids: z.array(z.string().max(160)).default([]),
});
```
**Procedures do NOT receive PropA sensitivity classification.** Scope controls visibility. Agent visibility controls routing.
---
## 26. SKILL.md ↔ Graph Procedure Relationship
### 26.1 Two materialization kinds — one registry
**[Source: SR-25, Problem 4]**
With the DOC24 injection pivot, the relationship between SKILL.md and graph procedures is formalized:
```ts
export const AbilityMaterializationKindSchema = z.enum([
"graph_injection", // Graph-backed → DOC24 renders injection cards from contract
"native_skill_bundle", // Native/imported → OpenClaw reads SKILL.md at runtime
]);
```
One ability registry, two kinds, one lookup surface. A graph-backed procedure never masquerades as a native skill bundle.
### 26.2 When SKILL.md is truth vs projection
| Scenario | SKILL.md role | Graph role | Runtime path |
|---|---|---|---|
| Native SKILL.md, no graph node | Layer 1 truth | N/A | OpenClaw reads SKILL.md |
| Graph node exists (demonstrated/learned) | Not used at runtime | Layer 1 truth | DOC24 injection card |
| SKILL.md exported from graph | Export artifact (Layer 2) | Layer 1 truth | DOC24 injection (not the export) |
| Imported SKILL.md, fully decomposed | Layer 2 projection | Layer 1 truth | DOC24 injection |
| Imported SKILL.md, partially decomposed | Layer 1 for un-decomposed portions | Layer 1 for decomposed | Hybrid |
| User manually edits projected SKILL.md | Conflict detected | Layer 1 truth | System proposes graph update from diff |
### 26.3 Projection and export
When a graph procedure node is exported to SKILL.md (portability format):
- `materializeSkill()` generates the SKILL.md from the graph contract
- Quality gate: compare against prior version, flag regressions
- Original SKILL.md preserved alongside projection for rollback
This is an **export convenience feature**, not a runtime-critical path.
### 26.4 Native skill migration
**[Source: F2, F3]**
When native SKILL.md files are ingested into the graph:
- `proposeSkillMigration()` generates a diff view with approval flow and backup
- For `full` decomposition with <10% diff: auto-replace eligible
- Everything else: hold for user review
- Original always preserved at `.pre-graph-migration` backup path
---
## 27. DOC24 Tool Representation in DOC72
### 27.1 Why tools are not procedure nodes
DOC24 tools and MCP tools are invocation endpoints, not semantic knowledge. They don't have the six dimensions that make procedure nodes valuable (content, provenance, temporal, confidence, connections, experience). Creating full graph nodes for every tool would bloat the graph with thin, context-free entries.
### 27.2 Where tools live in the graph
Tools are represented as **metadata on application entities**, not as separate nodes. The application entity is the hub that connects procedures (via `part_of_application`), MCP tools (via `mcp_tools` array), AppleScript capabilities (via `applescript_capabilities` array), known capabilities, and constraints.
**DOC72 ApplicationPayload extension:**
```ts
type ApplicationPayload = {
// ... existing fields from DOC72 §4.1 ...
// MCP tool capabilities (populated when MCP server connects)
mcp_tools?: Array<{
tool_name: string;
semantic_capability: string; // Normalized by one-time LLM call [Source: E2]
mcp_server_id: string;
input_schema_ref?: string;
available: boolean;
last_health_check: string;
}>;
// AppleScript capabilities (populated from adapter discovery)
applescript_capabilities?: Array<{
capability_id: string;
semantic_capability: string;
script_template: string;
read_only: boolean;
}>;
// App volatility — informs freshness probe frequency [Source: B1]
app_volatility?: z.enum(["stable_native", "moderate", "volatile_web"]);
};
```
### 27.3 The exception: tool-combination workflows
If a user demonstrates a workflow involving MCP/DOC24 tools in a specific sequence, that demonstration produces procedure nodes. The procedure stores semantic intent; the individual tool calls are execution mechanisms referenced via the execution-strategy cache.
---
## 28. Knowledge Store Integration Map — DOC3 Updates
### 28.1 Updated DOC3 store inventory
| # | Store | Layer | Action | Notes |
|---|---|---|---|---|
| 1 | Availability catalog | SHARE_ID | Add `doc72_procedure_id`, `materialization_kind` | Existing — extended |
| 2 | Skill catalog | SHARE_ID | Add `doc72_procedure_id` | Existing — add graph link |
| 3 | Interpretations | FEED_IN | Unchanged | Existing |
| 4 | Validation runs | FEED_IN | Unchanged | Existing |
| 5 | Validation step results | FEED_IN | Unchanged | Existing |
| 6 | Categories | FEED_IN | Unchanged | Existing |
| 7 | Learning runtime state | OPERATIONAL | Add demonstration fields (§8.10) | Extended |
| 8 | Observation scope | OPERATIONAL | Unchanged | Existing |
| 9 | Checkpoint receipts | OPERATIONAL | Unchanged | Existing |
| 10 | Learning receipts | OPERATIONAL | Add demonstration receipt types (§21) | Extended |
| 11 | Skill config | LEAVE | Unchanged | Existing |
| **12** | **Installed SKILL.md files** | **Layer 2 (projection) or Layer 1 (native)** | **Track materialization_kind** | **Clarified** |
| **13** | **Capability manifests** | **SHARE_ID** | **Map to DOC72 application entity capabilities** | **NEW** |
| **14** | **Demonstration bundles** | **OPERATIONAL** | **Ephemeral during session, graph writes on install** | **NEW** |
| **15** | **Raw observation events** | **OPERATIONAL** | **Ephemeral, purged 24h after session** | **NEW** |
| **16** | **Narration captures** | **OPERATIONAL** | **Ephemeral, purged with session** | **NEW** |
| **17** | **Proposals** | **SHARE_ID** | **Add `doc72_procedure_ids` for graph-linked procedures** | **Extended** |
| **18** | **Install saga records** | **OPERATIONAL** | **Track saga state per ability** | **NEW** |
| **19** | **Evidence digests** | **OPERATIONAL** | **Masked digest, 30 day retention** | **NEW** |
---
## 29. Procedure-Specific Lifecycle Policies — DOC72 Configuration Requirements
### 29.0 Why this section exists
⚠️ **INTEGRATION OBLIGATION — MUST BE ADDRESSED DURING DOC72 CROSS-DOC PASS** ⚠️
Procedures behave fundamentally differently from other knowledge nodes. If DOC72's default policies are applied uniformly, demonstrated skills will silently degrade, lose injection priority, and eventually get archived. DOC72 already has the per-node-kind configuration machinery. What's missing is explicit procedure-specific values.
### 29.1 Per-node-kind lifecycle table
**[Source: B1, B2, B3, B4]**
| Node / subtype | Decay mode | Staleness source | Auto-archive |
|---|---|---|---|
| procedure (validated) | Event-driven, no time decay | App version + capability-tag activity [Source: B2] | Never (dormant after 1095d) [Source: B3] |
| procedure (candidate/mined) | 365d half-life | Own last review/corroboration | Archive after 180d unreviewed |
| standing_procedure | No time decay | Trigger health/auth | Never |
| memory_directive (explicit preference/constraint) | No time decay | Explicit confirmation/contradiction | Never |
| memory_directive (vocabulary_mapping) | 730d review cycle | Use frequency + contradiction | Never |
| memory_directive (heuristic) | 180d half-life | Outcome drift | Archive after 365d stale |
| domain_concept | 365d half-life | Own verification | Yes |
| goal (explicit) | No time decay | Manual review | Never |
| goal (inferred) | 180d half-life | Own review/confirmation | Yes |
| obligation | Deadline-relative: infinite before, 90d after | Deadline + completion | Yes (after completion + 90d) |
**This table belongs in DOC72, not DOC3. DOC3 references it.**
### 29.2 Capability-tag-scoped freshness
**[Source: B2, SR-07]**
A procedure inherits freshness from its parent app ONLY when:
- (a) App version hasn't changed since last validation, AND
- (b) At least one recent validation trace overlaps the procedure's `capability_tags`
Never-used procedures don't get unlimited inherited freshness — if never used after demonstration, transition to `aging` after 180 days.
### 29.3 Dormant review state
**[Source: B3]**
```ts
const PROCEDURE_DORMANT_POLICY = {
trigger: "1095_days_inactive_plus_parent_app_stale",
effect: "dormant",
// Dormant procedures:
// - Excluded from proactive suggestions
// - Eligible on direct request with inline warning
// - "needs_review" badge in Knowledge Manager
warning_text: "This procedure hasn't been validated recently.",
};
```
### 29.4 Procedure injection policy
When a procedure is DIRECTLY RELEVANT to the current task, inject it regardless of staleness state. Staleness affects injection priority ONLY for proactive/anticipatory injection.
### 29.5 Where this must be documented
**Primary location:** DOC72, as a new §4.2A "Procedure and Skill Lifecycle Policy."
**Cross-doc obligation:** DOC72 must add per-node-kind lifecycle policies for all node kinds in the table above. Without these policies, demonstrated and learned skills will silently degrade under DOC72's default time-decay and staleness rules.
---
## 30. DOC24 Additions Required
### 30.1 Execution strategy resolution
DOC24 needs a step after pack mounting: per-step execution path optimization. This resolves each semantic step against available execution mechanisms. The execution-strategy cache (KDA R2 §3.5) stores the results. See KDA R2 for the full specification.
DOC3's responsibility: produce procedures with `capability_area` on each step so the resolver can match against tools. The resolution itself is DOC24-owned.
### 30.2 Ability availability as routing input
DOC24's semantic routing (Layer E) must include DOC3's `AbilityAvailabilitySnapshot`. This is a single new step in the routing cascade, not a redesign.
### 30.3 Routing relevance boost for learned procedures
**[Source: Grok Stress #5]**
When a learned procedure matches AND the current app matches the procedure's `environment` **[ADJ-60]**, apply a +0.15 relevance score boost. This prefers learned procedures over generic alternatives of similar relevance without forcing poorly-matched procedures.
### 30.4 MCP semantic_capability generation
**[Source: E2]**
When MCP server connects, batch LLM call generates `semantic_capability` strings from tool names + descriptions. Cached on app entity. Re-run when tool list changes (hash comparison). Quality score: if <0.7, fall back to UI navigation.
### 30.5 Hybrid execution coherence
**[Source: E3]**
After steps that switch between MCP and UI navigation: capture AX state snapshot, compare against expected postconditions (from step's `verification` field), retry via UI navigation if mismatch, abort with recoverable error if recovery fails.
---
## 31. Cross-Document Integration Obligations
| Target doc | What this proposal requires | Status |
|---|---|---|
| DOC72 | **Requires additions.** (1) Adopt shared knowledge contracts as canonical payload schemas (§2.7). (2) Extend `SemanticStep` with all fields from KDA R2 §2.3. (3) Add per-node-kind lifecycle policies (§29). (4) Add `mcp_tools`, `applescript_capabilities`, `app_volatility` to `ApplicationPayload` (§27.2). (5) Add new edge types: `requires_preceding_procedure`, `conflicts_with`, `platform_variant_of`, `supersedes_procedure`, `spans_application`. (6) Add `NodePayloadValidatorRegistry` for write-time validation (SR-32). **[ADJ-71]** (7) Adopt the per-node-kind lifecycle policies table from §29 as DOC72 §4.2A. **[ADJ-36]** (8) Define `StandingProcedureKnowledgeContractSchema` and `GoalKnowledgeContractSchema` before the full pipeline can be validated by SR-32. | Additions required |
| DOC24 | **Requires adoption of KDA R2.** (1) Execution strategy resolution. (2) Ability availability as routing input. (3) Routing boost for learned procedures. (4) Execution-strategy cache with SR-39 invalidation cascade. **[ADJ-41]** (5) For composite procedures with `spans_application` edges, execution-strategy resolution MUST resolve tools on a per-step basis using each step's owning app entity, not globally defaulting to the composite's primary app. All specified in KDA R2. | Adoption required |
| DOC1 | No changes. Memory directive candidates from demonstrations enter through existing governance. Maturity bypass per §10.4 when `assertion_class: "durable_fact"` AND confidence ≥ 0.85 AND user confirmed. | Compatible |
| DOC8 | No changes needed to DOC8 itself. Demonstration outcomes feed existing ExtractionOutcomeEvent interface. Skill mining thresholds (§16.3) are DOC3-owned configuration. | Compatible |
| DOC11 | **Requires update.** Must expose gateway observation mode for passive AX/screenshot/AppleScript capture (§23). | Update required |
| DOC10 | **[ADJ-30] Requires update.** Must emit `LearnedAbilitySelectionReceiptSchema` for learned-procedure routing decisions. Must register corresponding XDI ledger rows. | Update required |
| DOC15 | **[ADJ-32] Requires update.** Procedure injection cards positioned below authority injections, same level as entity cards. Direct-target procedures elevate to just below authority. **MUST enforce `procedure_sub_budget_pct: 0.25` (25% of effective packet budget, rising to 70% when direct target) per KDA R2 §3.6. Procedure cards MUST NOT consume minimum reserves allocated to authority or entity injections.** | Update required |
| DOC21 (Master UI) | **Requires update.** New: LearnCaptureOverlay (sidebar), BundleReviewPanel, QuickPromoteDrawer, AbilityProofDrawer, FidelityIndicator, TrustModeToggle, menu bar status item. | Update required |
| DOC22 (Page Inventory) | **Requires update.** New components in Learn page, Knowledge Manager additions. | Update required |
| DOC20 §6.18.2 (Content Registry) | **Requires update.** New stored content types: demonstration bundle (ephemeral), raw observation events (ephemeral), evidence digest (30d), install saga records (operational). | Update required |
| Knowledge Store Integration Map | **Requires update.** DOC3 inventory expands from 11 to 19. SKILL.md materialization-kind tracking. | Update after acceptance |
| DOC11/DOC16 | **[ADJ-33] New cross-doc obligation.** Must emit `app_tool_catalog_changed { app_entity_id, new_tool_catalog_version }` when MCP tool catalog changes, to trigger DOC24 execution-strategy cache invalidation cascades per KDA R2 §3.5. | New obligation |
| DOC23 | **[ADJ-64] Future dependency (Phase 3).** When branching procedures are implemented, DOC23 Tasks must support orchestrating procedure branches via `requires_preceding_procedure` edge traversal. Not needed for v1. | Phase 3 |
| DOC3 Companion R9.2 | **[ADJ-34] Requires update.** New routes from §11.1 must be registered: `interpret`, `capture-capability`, `quick-promote`, `undo-promote`, `bundle-edits`, `install-sagas/:sagaId`, `ability-proof`, `feature-support`, `observation-evidence`, `trust-mode`. | Update required |
---
### 31.2 R2.2 Cross-Document Obligations
| Target doc | Obligation | Source |
|---|---|---|
| **EC Core** | Store and command handlers for: display items (CRUD), display-item edit operations (exclude/restore/annotate/reorder/group), capture offers, dispositions, capture-mode settings, browser observation eligibility decisions, EC throttle decline counters. | §20.3A, §25.8, §14.4A |
| **DOC72** | ASSESS stage (§20A.2 Stage 2) expanded with `extraction_facets` model: classify procedure candidates into `{standing_procedure, invocable_procedure, hybrid_procedure}`. New `triggers_procedure` edge type for hybrid linkage. Invocable and hybrid procedure candidates route to DOC3 Tier 3 interpretation pipeline. | §25.8.5 |
| **DOC72** | Write `RawToSemanticMappingSchema` trace node atomically with procedure during install saga. | §13.8 |
| **DOC72** | Add `"inferred_from_conversation_context"` to canonical `scope_inference_basis` values. | §25.8.9 |
| **DOC11** | Chrome extension observation mode: content script, `learn:activate_browser_observation` / `learn:deactivate_browser_observation` lifecycle, `learn:tab_closed` error signal, gateway event routing for `browser_extension` adapter source. | §4.5A.2 |
| **DOC20** | Learn button on chat panel with "Describe a process" + "Teach [agent name]" dropdown. "Teach" disabled with tooltip when Q Browser incognito active. | §25.8.3, §4.5A.1 |
| **DOC20** | Tier 2 UI pop-up: adaptive text by `predicted_node_kind` ("Save as procedure?" / "Save as preference?" / "Save as standing rule?"). Dedup-aware variant ("Update existing?") when match found. Multi-agent variant shows target agent. Cold-start throttling. [Yes] [No] [That's wrong] [Revise] buttons. | §25.8.4 |
| **DOC20** | §6.19.15 embedded browser observation hook: dual CDP + injected script, auto-follow tab switches, observation eligibility gate, set `demonstration_mode_used` on BrowserPageVisitSchema, per-tab incognito gate. | §4.5A.1-5 |
| **DOC20** | Capture mode setting (Active/Prompted/Passive) in Settings surface. | §25.8.8 |
| **BDSM (DOC24 Add. A)** | `conversational_capture_offered` signal in §9.1 signal inventory. `pending_candidate_resolved` signal for Tier 3. Gate 2 consumes both per `context_class_key`. | §14.4A.2 |
| **BDSM (DOC24 Add. A)** | Onboarding milestone: `procedure_capture_discovery` — first Tier 2 pop-up includes explainer. | §25.8.4 |
| **PropA** | `P0_master_extraction` prompt should recognize `invocable_procedure` and `hybrid_procedure` facets in Tier 3 nightly extraction. Collection mode check gates Tier 2 — if `effective_collection_mode` is `blocked`, no pop-up, no tagging. | §25.8.1, §25.8.5 |
| **DOC24/KDA** | Confirm `RawToSemanticMappingSchema` is provenance only and does not alter execution-strategy cache semantics. When invalidation cascade fires, compare new tool catalog against snapshot's `tools_used` for staleness warnings. | §13.8.3-4 |
| **DOC1** | Memory-directive dedup auto-enrich MUST NOT broaden `applies_when` or change `priority_class` without DOC1-governed review. | §7.2A |
### 31.3 R2.2 Route Contracts
```http
# Display item operations (during/after capture)
POST /api/learn/sessions/:sessionId/display-items/:itemId/exclude
POST /api/learn/sessions/:sessionId/display-items/:itemId/restore
POST /api/learn/sessions/:sessionId/display-items/:itemId/annotate
POST /api/learn/sessions/:sessionId/display-items/insert-manual
POST /api/learn/sessions/:sessionId/display-items/reorder
POST /api/learn/sessions/:sessionId/display-items/group
POST /api/learn/sessions/:sessionId/display-items/ungroup
POST /api/learn/sessions/:sessionId/start-interpretation
# Conversational capture
POST /api/learn/conversational-capture/offers/:offerId/accept
POST /api/learn/conversational-capture/offers/:offerId/decline
POST /api/learn/conversational-capture/offers/:offerId/revise
# Settings
PATCH /api/learn/settings/conversational-capture-mode
```
---
## 32. Acceptance Criteria
This proposal is considered implemented when:
1. A `demonstrating_skill` session can be started, and the observation layer captures AX events from an external macOS app.
2. Capture preflight accurately reports adapter availability and predicted quality.
3. App focus detection after arming correctly identifies the target app.
4. Structural labels appear in the Q sidebar panel as major events are captured, with significance filtering.
5. User narration during capture is buffered and classified, via voice PTT or typed input.
6. Silent demonstrations (no narration) produce acceptable bundles with appropriately lower confidence.
7. Undo sequences are detected and excluded from interpretation.
8. Semantic interpretation produces a `KnowledgeExtractionBundle` with procedure candidates stored as `ProcedureKnowledgeContractSchema` payloads, including trigger_phrases, use_conditions (as separate fields), capability_tags, artifact_class, and assertion_class on memory directives.
9. Observation fidelity score is computed and displayed, gating Quick Promote eligibility.
10. Streaming interpretation shows partial results for complex demonstrations.
11. The user can review bundles in summary/detail view and edit at the step level (replace, move, delete, insert, split, reclassify, merge).
12. Quick Promote installs procedures only, shunting memory directives and standing procedures to Pending Abilities. Global/app-scoped constraints are stripped from procedure payload during Quick Promote.
13. Trust Mode auto-accepts procedures ≥ 0.80 and routes everything else to review.
14. EC validates payloads against `NodePayloadValidatorRegistry` before graph write.
15. Cross-doc enum mapping correctly translates DOC3 internal values to DOC72 canonical values.
16. Install saga completes: graph_written → render_validated → strategy_cached → availability_refreshed → routable.
17. Demonstrated procedures are discoverable through DOC24's routing cascade via trigger_phrases and semantic_lookup_phrases.
18. Subsequent demonstrations of the same procedure enrich rather than duplicate (three-stage dedup with hard vetoes).
19. Correction propagation cascades to composite procedures and invalidates execution-strategy cache entries.
20. The system is not Word-specific — the per-app adapter profiles support other apps.
21. The `KnowledgeExtractionBundle` pipeline is usable by at least one other learning mode beyond `demonstrating_skill` (recommended: `Learn From This`).
22. DOC24 execution strategy resolution can select MCP tools over UI navigation for procedure steps where both are available.
23. ProcedureExecutionOutcomeEvent is emitted after execution, feeding confidence updates.
24. Undo-promote route archives nodes within 24 hours.
25. Feature support snapshot accurately reports which critical seams are wired.
---
## 33. Acceptance Tests
**[Source: Patch 13]**
### 33.1 Core pipeline tests
1. **Capture → Interpret → Promote:** Start demonstration in Word, perform 5 menu actions, stop, verify bundle contains ≥ 1 procedure with ≥ 3 steps.
2. **Quick Promote timing:** Simple demonstration (<15 events) produces routable ability in < 30 seconds after stop.
**[ADJ-61]** Stage timing targets for simple demos: interpretation ≤ 15s, saga total ≤ 5s (graph_written ≤ 200ms, render_contract_validated ≤ 200ms, execution_strategy_cached ≤ 3s, availability_refreshed ≤ 500ms, trigger indexing ≤ 500ms). If any stage exceeds 2× its target, mark `private_degraded` and complete remaining stages async.
3. **Dedup detection:** Demonstrate a procedure that overlaps an existing one (same app, >90% step similarity). Verify auto-enrich (confidence increment, not new node).
4. **Privacy scrubbing:** Demonstrate with client name visible. Verify the stored procedure uses `[Client Name]` placeholder, not the literal value.
5. **Multi-app workflow:** Demonstrate a workflow spanning Word and Excel. Verify composite procedure with `spans_application` edge.
6. **Undo handling:** During demonstration, perform an action, undo it, do it differently. Verify the undone action does NOT appear in the procedure.
7. **Silent demonstration:** Demonstrate without narration. Verify bundle is produced with lower fidelity score but acceptable procedures.
8. **Trust Mode:** Enable Trust Mode. Demonstrate with all items ≥ 0.80. Verify one-tap promotion. Verify non-procedure items go to Pending Abilities.
9. **Scope leak prevention:** Quick Promote a procedure that includes a `scope: "global"` constraint. Verify the constraint is stripped to Pending Abilities.
### 33.2 Integration tests
10. **End-to-end discovery:** After promoting a procedure, issue a natural language request matching a trigger phrase. Verify the procedure is discovered and injected.
11. **Install saga completion:** Verify all saga states transition correctly and Q renders appropriate status at each stage.
12. **Execution outcome feedback:** Execute a promoted procedure, verify `ProcedureExecutionOutcomeEvent` is emitted and confidence updates per SR-33.
---
## 34. Future Capabilities
**[Source: H3]**
### Phase 1 (day-one)
- **Golden demonstration benchmark corpus:** 25-50 benchmark demos per app family for regression testing interpretation quality.
- **Capture preflight route** (already specified in §11.1).
- **Native skill auto-ingestion job:** One-time migration of existing SKILL.md files into graph on first boot.
### Phase 2 (post-launch)
- **Replay and Verify mode:** Execute a procedure against the real app and verify each step. Validates procedure correctness.
- **Batch observation / "Watch my morning":** Long-running passive capture with batch interpretation for recurring patterns.
- **Demonstration templates:** Pre-built "what to demonstrate" guides for common apps (Word, Excel, Outlook).
### Phase 3 (mature system)
- **Incremental interpretation during capture:** Lightweight partial interpretation every 10 events, shown as tentative groupings.
- **Watch and Correct mode:** Streaming tentative interpretation with real-time user correction.
- **Cross-procedure co-occurrence detection:** DOC8 tracks "users who do A always do B after."
- **Show Me How / Reverse demonstration:** Elnor demonstrates TO the user, validating the procedure and teaching simultaneously.
- **Procedure template detection:** DOC8 detects 3+ composites with overlapping structure and proposes parameterized templates.
- **User-controlled learning budget:** Per-day learning slot cap + "Focus learning on current matter" toggle.
- **Branching procedures:** Implement `execution_topology: "branching"` with `decision_points` rendering and execution. The forward-compatible schema fields (SR-31) are already in place.
- **Confidence visualization:** Timeline chart in Knowledge Manager showing α, β, confidence over time with events marked.
- **Procedure review queue:** Knowledge Manager filter "Needs review" grouped by app family + capability_tags.