# Docs and search answers (DOC-1): research and design

Status: **approved (owner delegated, my recommendation, 2026-11-13).** Code read on `track/DS-design-system` (e618b7d7), 2026-11-13. Free/premium follows the working rule "needs embeddings = premium" until BM decides.

## Summary

The engine is mostly there: chunks with heading breadcrumbs, a provider-agnostic metadata filter, request-scoped source tracking, page-aware instructions, streaming, a panel mount and a renderables pipeline. Missing are the surfaces (no search hook, no block, no modal), sources as data, section links, taxonomy scoping and copyable code. The track is wiring plus two indexing changes: heading anchors and taxonomy terms in vector metadata.

## What exists today

| Item | Today | Gap |
| --- | --- | --- |
| 1. Search answer box | No search hook anywhere. Premium keyword providers (`premium/app/FunctionCall/KeywordSearch/`: WP core, Relevanssi, SearchWP, ElasticPress) run only inside `kb_vector_search`. `POST kimaai/v1/chatbot` (`app/ChatBot/ChatBot.php`) is session-based: `message`, `postId`, `lang`, `sessionId`, `stream`. | Surface, sessionless endpoint, cache. Free has no retrieval; its only grounding is the current page. |
| 2. Ask-AI modal | Mount modes `floating`, `inline`, `absolute`, `panel` (`src/chatbot/core/config/types.js`); `PanelMountShell.jsx` renders chat without launcher. `src/chatbot/index.js` mounts once on `#kimaai-chatbot`. | No modal, no search tab, one mount point. |
| 3. Citations | Premium. `SourceTracker` collects title, URL, snippets from retrieval (`ToolRouter.php` 527, 547). `AnswerQuality::decorate()` appends "Sources:" and up to 3 markdown links when `citationsEnabled` (default off). Chunks store `heading`, a breadcrumb like `Title > Install > Docker` (`Embedding.php` 504). | `ContentProcessor::walkDomBlocks()` keeps heading text, not `id`, so links open the page top. Sources are text: no cards; when streaming they arrive only with `done`. |
| 4. Scope | Premium. `MetaFilter::translate()` maps `field => value|array` (AND) to Pinecone or Qdrant syntax. `ToolRouter::buildMetaFilter()` sets `lang` and `post_type`, filter `kimaai_kb_search_meta_filter`. Keyword providers know only `post_type` and `lang`. | No taxonomy terms in vector metadata (`post_id`, `post_type`, `slug`, `lang`, `heading` only); `kimaai_embedding_vector_metadata` can add them. |
| 5. Block | No `register_block_type`. `[kimaai_chatbot]` (`app/ChatBot/ChatbotAssets.php`) has no attributes, prints the global widget's `#kimaai-chatbot` (duplicate id when both are on), detected only in singular `post_content`. | Block, attributes, several instances. |
| 6. Page-aware | Free. `app/ChatBot/ChatbotContentAwareness.php` fills `{post_url}`, `{post_title}`, `{post_content}` (used by the default instructions); `awareness` default off; cut to `maxContextLength`. | `postId` is `get_the_ID()`; on search and archive views core sets `$post` to the first result (`wp-includes/class-wp.php` 672), so that hit is sent as "current page" (read in core, not reproduced). Whole page only, no effect on retrieval. |
| 7. Code | `src/global/components/MarkdownMessage.jsx` renders fences as `.codecard pre code`; Geist Mono bundled (DS-5); message-level copy exists (`useCopyToClipboard`). | Fence language discarded, no per-block copy, no highlighting. |

## How others do it

Vendor docs only, read 2026-11-13; nothing tested hands-on.

1. **Answer box.** Jetpack Search AI Answers (paid, launched May 2026) shows a short summary with source links above the results, generated automatically once enabled, with a hand-off to chat [1][2]. TidBITS found it often missed direct answers [3]. Nobody documents streaming or caching for this box [unverified].
2. **Modal.** Cmd/Ctrl+K opens a modal; results appear as you type; an "Ask AI about `<query>`" entry or toggle moves to chat with the query; "Back to results" returns. Inkeep [4], Kapa (search and Cmd+K opt-in; search does not count against the question quota) [5][6], Algolia (hybrid mode hands the query over without sending it twice) [7][8][9], BetterDocs ("Continue Chat") [10].
3. **Citations.** Built from structured data: Inkeep inline pills plus a Sources list [11], Algolia "Related sources" [8]. Anchor deep links: documented by none [unverified].
4. **Scope.** Set by the installer: Kapa source groups, for example one per major version [12]; Inkeep filters from the current page's meta tags, for search and chat [13]; Algolia facet filters [14].
5. **Components.** Inkeep embedded chat, embedded search and custom triggers [15]; Kapa advises several entry points [16].
6. **Page context.** Only Inkeep documents it: current URL sent as a header, read in the prompt [17].
7. **Code.** Inkeep takes a custom Prism highlighter [18]; per-block copy is undocumented everywhere [unverified].

Accessibility: streaming text stays out of the live region; a separate polite status announces "Generating" and "Answer ready" once; focus never moves [19][20]. The modal uses the ARIA combobox pattern with `aria-activedescendant` [21].

For KimaAI: modal and answer box are table stakes. Section links, taxonomy scope and page context are rarely documented and map onto WordPress data (heading ids, taxonomies, the current post), so that is where KimaAI can lead.

## Design

Shared rules:

- **One pipeline.** Every surface calls the existing chat pipeline (`ChatbotRequestPreparer`, runners, SSE) with two new request fields, `surface` and `scope`. Gate, rate limit, guardrails and retention apply unchanged.
- **Sources as data**, persisted with the message through the renderables pipeline (`AssistantRenderables`), so history and API clients see the same thing.
- **No AI call on page load** unless the owner enables auto mode; auto is blocked while `gdprConsent` is required.
- **Accessibility** as above, checked with axe like `tests/e2e/specs/chatbot-a11y.spec.ts`.

**Code blocks (free).** `MarkdownMessage` keeps the fence language as label and `data-lang`, adds a copy button per block (`aria-label` "Copy code", "Copied" announced) and `tabindex="0"` on `pre` for keyboard scrolling. No highlighter now: it would add weight to every page with the widget. Coordinate with UI-3 (copy components).

**Page context (free).** Send `postId` only when `is_singular()`. The frontend sends `section`, the id of the heading nearest above the viewport; that section goes first before the cut, also as `{post_section}`. Awareness on by default for new installs only.

**Structured sources and related cards (premium).** The response, the SSE `done` event and history carry `sources`: title, URL with anchor, breadcrumb, type label, 160-character excerpt. The widget lists cited sources under the answer and up to `relatedPagesCount` (0 to 5, default 3) further hits as cards. Clients that do not send `renders_sources` keep today's markdown. Same-site links open in the same tab.

**Section anchors (premium).** `walkDomBlocks()` reads each heading's `id` (core "HTML anchor", TOC plugins) and `finalizeChunk()` stores `anchor`. Without an id the link uses `#:~:text=<heading>`; unsupported browsers ignore it and open the page top [24][25]. Old vectors keep plain URLs until re-synced; Index Health says so.

**Scope by product or version (premium).** Setting `vectorDbScopeTaxonomies` (for example `product`, `doc_version`). Vectors get `tax_<slug>: [term slugs]`; Pinecone and Qdrant both match a list when any element matches [22][23]. `buildMetaFilter()` takes the scope from the request (block or shortcode attribute), else from the current page's terms when `vectorDbScopeFromPage` is on. Request scope is untrusted: only configured taxonomies and existing terms pass. Keyword providers map `tax_*` to `tax_query`. Term changes queue a re-embed. KimaAI adds no version logic; a "latest" term is the owner's choice.

**Inline chat block (free).** First `index.js` mounts every `[data-kimaai-chatbot]` root with config from data attributes and its own session key; the shortcode gains `mode`, `height`, `scope`, `greeting`. Then block `kimaai/chat`: `block.json` in `src/blocks/chat/` (new webpack entry), dynamic, registered from PHP on `init`, `viewScript` the existing `kimaai-chatbot` handle so assets load only where it renders. Editor: static preview and Inspector controls, no AI calls.

**Ask-AI modal (free; answers improve with premium).** Block `kimaai/ask-ai` plus optional `askAiTakeOverSearch`, which adds an "Ask AI" trigger to theme forms via `get_search_form` and `render_block_core/search`. Search tab: core `GET /wp/v2/search` as a combobox, first option "Ask AI: `<query>`". Chat tab: panel mount started with the query. `Ctrl/Cmd+K` behind `askAiShortcut` (default off). Esc closes and returns focus.

**Search answer box.** Endpoint `POST kimaai/v1/answer` (`query`, `lang`, `scope`, `stream`), `surface: search`, no user tools, no history. Free grounds on the top N posts of the current search (IDs signed into the box); premium uses retrieval and citations. Cache: transient keyed by normalised query, language, scope and an index version bumped on every sync or post update; TTL `searchAnswerCacheTtl` (default 6 h); only grounded answers with sources; a hit does not count against the rate limit. Placement: `loop_start` on the main search query for classic themes, block `kimaai/search-answer` for block themes, auto-inserted via `render_block_core/query` when `searchAnswerAutoInsert` is on. Default trigger is a "Generate AI answer" button (`searchAnswerMode`: `button` or `auto`). Results render first; the box fills client-side, with sources, thumbs and "Continue in chat". New settings section `searchAnswer` in `app/Admin/config/chatbot.php`.

Tests for every item: PHPUnit (`tests/unit`) for PHP, Jest (`tests/js`) for components, Playwright (`tests/e2e`) with axe for each new surface, on a classic and a block theme where themes matter.

## Items for the track

One commit each; quick wins first, then the indexing later items need.

| Item | Title | Acceptance criteria |
| --- | --- | --- |
| DOC-2 | Code blocks with copy | Language label and copy button per block; "Copied" announced; `pre` keyboard-scrollable; Jest and axe green. |
| DOC-3 | Page context: singular only, section first | `postId` 0 on search and archives; visible section first and `{post_section}` works; default on for new installs only; PHPUnit and Jest. |
| DOC-4 | Sources as data | `sources` in JSON, `done` event and history; markdown kept for older clients; PHPUnit. |
| DOC-5 | Source list and related cards | Cited list and `relatedPagesCount` cards under answers; same-site links in same tab; Jest and e2e (fake provider). |
| DOC-6 | Section anchors in the index | Chunks store `anchor`; links use `#id` or a text fragment; old vectors fall back; Index Health flags re-sync; PHPUnit. |
| DOC-7 | Taxonomy terms in vector metadata | `vectorDbScopeTaxonomies`; vectors carry `tax_<slug>`; term changes re-embed; PHPUnit. |
| DOC-8 | Scoped retrieval | `scope` validated; vector and keyword legs filter; scope from page when enabled; PHPUnit for Pinecone and Qdrant output. |
| DOC-9 | Several chat mounts | Each root mounts with own config and session; shortcode attributes; floating widget unchanged (e2e). |
| DOC-10 | `kimaai/chat` block | Registered from `block.json`; assets only where rendered; static editor preview; PHPUnit, Jest, e2e. |
| DOC-11 | Ask-AI modal | Search tab with "Ask AI" first option, chat tab with query, combobox and dialog semantics; Jest and axe. |
| DOC-12 | `kimaai/ask-ai` block and search take-over | Block plus trigger on classic and `core/search` forms; shortcut behind setting; e2e on both theme types. |
| DOC-13 | Answer endpoint and cache | Gate, rate limit, guardrails apply; free and premium grounding; cache key and invalidation tested; PHPUnit. |
| DOC-14 | Search answer box | Classic and block placement; button by default, auto blocked under required consent; results never wait; e2e and axe. |

Free: DOC-2, 3, 9 to 12, and DOC-13/14 with free grounding. Premium: DOC-4 to 8 and retrieval grounding in DOC-13/14.

## Decisions (owner delegated, my recommendation, 2026-11-13)

1. **Search-page answers are stored as sessions** with `surface: search`, so feedback analytics and the knowledge-gap miner see them; the chat history retention setting applies to them like to every session.
2. **Free grounding from the visible results**: the free plugin answers from the top N posts of the current search, premium from retrieval with citations. A free feature that works on its own, as WordPress.org guideline 5 expects.
3. **No syntax highlighting now.** It would add weight to every page with the widget; a lazy-loaded opt-in is listed under "Later" in the roadmap.

## Sources

1. Jetpack, AI Answers launch post: <https://jetpack.com/?p=459657>
2. Jetpack support, AI Answers: <https://jetpack.com/support/search/ai-answers/>
3. TidBITS, testing Jetpack Search AI Answers: <https://tidbits.com/2026/08/17/testing-jetpack-searchs-ai-answers-feature-on-tidbits/>
4. Inkeep search bar: <https://docs.inkeep.com/ui-components/react/search-bar>
5. Kapa search mode: <https://docs.kapa.ai/integrations/website-widget/features/search>
6. Kapa widget behaviour: <https://docs.kapa.ai/integrations/website-widget/configuration/behavior>
7. Algolia, DocSearch reimagined: <https://www.algolia.com/blog/product/docsearch-reimagined>
8. DocSearch v4 API: <https://docsearch.algolia.com/docs/v4/api>
9. DocSearch hybrid mode: <https://docsearch.algolia.com/docs/hybrid-mode>
10. BetterDocs AI-powered smart modal search: <https://betterdocs.co/docs/ai-powered-smart-modal-search/>
11. Inkeep citations: <https://docs.inkeep.com/guides/agents/customer-assistant/citations>
12. Kapa source groups: <https://docs.kapa.ai/knowledge-sources/use-source-groups>
13. Inkeep filters: <https://docs.inkeep.com/ui-components/customization-guides/filters>
14. DocSearch migration to v5: <https://docsearch.algolia.com/docs/migrating-from-v4>
15. Inkeep component index: <https://docs.inkeep.com/llms.txt>
16. Kapa best practices: <https://docs.kapa.ai/integrations/website-widget/guides/best-practices>
17. Inkeep, passing context: <https://docs.inkeep.com/talk-to-your-agents/customization/passing-context>
18. Inkeep UI release notes: <https://docs.inkeep.com/release-notes/ui-components>
19. Accessible streaming chat UI: <https://azukiazusa.dev/en/blog/accessible-streaming-chat-ui>
20. Smashing Magazine, accessible LLM chat: <https://smashingmagazine.com/2026/11/making-immersive-llm-chat-experiences-more-accessible/>
21. MDN, combobox role: <https://developer.mozilla.org/docs/Web/Accessibility/ARIA/Reference/Roles/combobox_role>
22. Pinecone metadata (lists of strings, any-element match): <https://docs.pinecone.io/guides/index-data/indexing-overview>
23. Qdrant filtering (`match.any` on arrays): <https://qdrant.tech/documentation/concepts/filtering/>
24. MDN, text fragments (ignored when unsupported): <https://developer.mozilla.org/en-US/docs/Web/URI/Reference/Fragment/Text_fragments>
25. caniuse, text fragments (Firefox 131+, Safari version disputed between 16.1 and 18.x): <https://caniuse.com/url-scroll-to-text-fragment>
