Structured Content Is AI-Ready Content

Ask an AI assistant built on a folder of PDFs what your refund policy is, and it will answer confidently. It may well quote the version from two years ago. Nobody told it that document had been replaced, because the folder never knew either.

That isn't a model problem. Any model will do the same thing with the same inputs. It's a content problem: the assistant was handed text without knowing what any of it was, whether it was still current, or whether it was ever meant to be public.

If you've modeled your content properly in a headless CMS, you already have what many AI projects spend months building: a source of truth that knows what each piece of content is, which language it's in, when it last changed and whether it's live.

Why structure matters more than the model

Retrieval-augmented generation, where an assistant looks up relevant passages and answers from them, succeeds or fails on five properties of the content it retrieves from. None of them is the language model:

Each row is something you would otherwise rebuild with heuristics. A well-modeled CMS gives you all of them, which is why the content-modeling decisions you made for the website turn out to be AI decisions too.

Index what's published, nothing else

Contentful keeps published and draft content apart. The Delivery API returns what's published; the Preview API returns drafts as well. An indexer should only ever hold a Delivery API token.

That sounds obvious until you look at what lives in drafts: next quarter's pricing, an announcement waiting for legal sign-off, a product page for something that hasn't launched. The website never shows those, because the website reads published content. An assistant indexed from the Preview API will happily quote them to anyone who asks.

Public doesn't automatically mean an assistant should paraphrase it, either. Legal terms, regulated claims and anything with contractual weight are better linked to than summarized. A true/false field on those content types (call it exclude from assistant) lets editors make that call entry by entry, in the same place they make every other publishing decision.

Chunk along the seams your content already has

Most retrieval pipelines split text into fixed-size windows, because a PDF offers nothing better. Structured content does. A Service entry already has a title and a summary written to stand alone, and a body divided by headings its author chose. Those are the natural units of meaning, and they're the units an assistant should retrieve:

• The summary is its own chunk. It's the best short answer to “what is this?” that exists, because someone wrote it for exactly that purpose.
• Rich text splits at headings. A Contentful rich-text field is a JSON document, not HTML, so finding the heading-2 and heading-3 nodes is a loop, not a parsing exercise.
• References become filters, not text. The industries a service links to are metadata, so a question about healthcare searches only healthcare content.
• Every chunk carries its provenance: entry ID, content type, locale, canonical URL and last-updated time.

// chunk.ts: split a Contentful entry into retrieval chunks along its own structure.
import { BLOCKS, type Block, type Document } from '@contentful/rich-text-types';
import { documentToPlainTextString } from '@contentful/rich-text-plain-text-renderer';

export type Chunk = {
  id: string;            // entry + locale + position; re-indexing replaces them all
  entryId: string;
  contentType: string;
  locale: string;
  url: string;           // the canonical page an answer will cite
  section: string;
  updatedAt: string;
  text: string;
};

const SPLIT_AT = new Set<string>([BLOCKS.HEADING_2, BLOCKS.HEADING_3]);

// Sync API entries carry every locale: fields.title['en-US'], fields.title['de-DE'], ...
export function chunkEntry(entry: any, locale: string, url: string): Chunk[] {
  const f = entry.fields;
  const base = {
    entryId: entry.sys.id,
    contentType: entry.sys.contentType.sys.id,
    locale,
    url,
    updatedAt: entry.sys.updatedAt,
  };
  const chunks: Chunk[] = [];
  const push = (section: string, text: string) => {
    if (!text.trim()) return;
    chunks.push({ ...base, id: `${base.entryId}:${locale}:${chunks.length}`, section, text });
  };

  // The summary was written to stand alone, so it is a chunk of its own.
  push('Summary', [f.title?.[locale], f.summary?.[locale]].filter(Boolean).join('\n'));

  // The body becomes one chunk per heading-delimited section.
  const body: Document | undefined = f.body?.[locale];
  let section = 'Introduction';
  let buffer: Block[] = [];
  const flush = () => {
    const doc = { nodeType: BLOCKS.DOCUMENT, data: {}, content: buffer } as Document;
    push(section, documentToPlainTextString(doc, '\n'));
    buffer = [];
  };
  for (const node of (body?.content ?? []) as Block[]) {
    if (SPLIT_AT.has(node.nodeType)) {
      flush();
      section = documentToPlainTextString(node);
    } else {
      buffer.push(node);
    }
  }
  flush();
  return chunks;
}

Keep the index fresh without crawling

The usual approach to freshness is re-crawling on a schedule, which leaves the index somewhere between an hour and a week out of date, with deletions discovered late or not at all. Contentful's Sync API fits better. The first request, with initial=true, returns everything that's published, along with a nextSyncUrl. Calling that URL later returns only what has changed since, including entries that were unpublished or deleted.

// sync.ts: keep the search index in step with what is published.
import { chunkEntry } from './chunk';
import { index } from './search';    // your index client: upsert(chunks), deleteByEntry(id)
import { state } from './state';     // durable storage for the sync URL

const { SPACE_ID, ENVIRONMENT = 'master', CDA_TOKEN } = process.env;   // a Delivery token, never Preview
const INDEXED_TYPES = new Set(['service', 'article', 'caseStudy']);
const LOCALES = ['en-US'];
const PATHS: Record<string, string> = { service: '/services', article: '/blog', caseStudy: '/work' };

const canonicalUrl = (type: string, entry: any, locale: string) =>
  `https://www.example.com${PATHS[type]}/${entry.fields.slug?.[locale]}`;

export async function sync() {
  // No `type` filter: we need DeletedEntry items as well as entries.
  let url: string | null = (await state.get('nextSyncUrl')) ??
    `https://cdn.contentful.com/spaces/${SPACE_ID}/environments/${ENVIRONMENT}/sync?initial=true`;

  while (url) {
    const res = await fetch(url, { headers: { Authorization: `Bearer ${CDA_TOKEN}` } });
    if (!res.ok) throw new Error(`sync failed: HTTP ${res.status}`);
    const page = await res.json();

    for (const item of page.items) {
      const kind = item.sys.type;
      if (kind === 'Asset' || kind === 'DeletedAsset') continue;

      await index.deleteByEntry(item.sys.id);           // replace, never append
      if (kind === 'DeletedEntry') continue;            // unpublished or deleted: gone from answers

      const type = item.sys.contentType.sys.id;
      if (!INDEXED_TYPES.has(type)) continue;
      if (item.fields.excludeFromAssistant?.['en-US']) continue;

      for (const locale of LOCALES) {
        await index.upsert(chunkEntry(item, locale, canonicalUrl(type, item, locale)));
      }
    }

    if (page.nextPageUrl) {
      url = page.nextPageUrl;
    } else {
      await state.set('nextSyncUrl', page.nextSyncUrl);   // saved only after the last page
      url = null;
    }
  }
}

A few details in that loop matter more than they look:

• No type filter. Asking the Sync API for type=Entry would leave out DeletedEntry items, and the index would never hear about removals.
• Deletions are handled by ID first. A deleted entry comes back without a content type to filter on, so the indexer removes its chunks before checking anything else.
• The sync URL is saved only after the last page. A crash halfway through repeats some work instead of silently skipping it. If the saved URL is ever lost, clear the index and run an initial sync again.

Webhooks make it close to real time. Contentful can call a URL whenever an entry is published, unpublished or deleted. It's tempting to update the index straight from the webhook payload. Don't: treat the webhook as a doorbell, not a delivery. Its only job is to trigger the sync, which then fetches the changes properly. If a webhook is missed or arrives out of order, the next sync still gets everything right, because the sync URL is the source of truth, not the webhook.

// app/api/contentful-webhook/route.ts: the webhook only rings the doorbell.
import { queue } from '@/lib/queue';

export async function POST(request: Request) {
  // A shared secret, sent as a custom header configured on the webhook in Contentful.
  if (request.headers.get('x-webhook-secret') !== process.env.WEBHOOK_SECRET) {
    return new Response('forbidden', { status: 403 });
  }
  await queue.enqueue('contentful-sync');   // the sync job fetches the actual changes
  return new Response(null, { status: 202 });
}

Answers that cite their source

Because every chunk carries a canonical URL, every answer can link to the page it came from. Make that a requirement, not a feature:

• Always cite. Treat an answer without a link to its source as a bug. Citations let people check the answer, and they send people to your site instead of ending the conversation in a chat window.
• Say “I don't know” when retrieval comes back weak. If nothing scores above a threshold you set, the right answer is a link to search or to a person, not the model's best guess.
• Show how fresh it is. The last-updated time is already in the metadata. For time-sensitive content like pricing and policies, show it next to the answer.

What to change in your model (not much)

If the model is already shaped by meaning, the changes for AI are small, and most of them improve the website too:

• A real summary field on every content type that has a page: two sentences of plain text, written to stand alone. It doubles as the meta description.
• References for audience and industry, not just mentions in the body, so they can filter retrieval.
• A review-by date on time-sensitive content, so stale pricing and policies get flagged before an assistant quotes them.
• An exclude-from-assistant flag for content that should be linked to rather than paraphrased.
• Locale discipline. Where a reviewed translation exists, index it and filter by the visitor's language, instead of letting the model translate English on the fly.

Where this stops

A CMS knows what you've published. It doesn't know a customer's order status, their contract terms or the state of their support ticket. An assistant that needs those needs governed access to the systems that hold them, with authorization that still holds when the caller is a model rather than a person. That's a different and harder problem than the one in this article.

It's usually the second problem, though. The first is making sure an assistant can answer the questions your website already answers: correctly, from current content, with a link. Most organizations already have that content in a CMS. With a well-shaped model, putting it to work is an integration project, not a new platform.

Questions we get asked