Giving your prompts fresh web context: a practical pattern with an MCP scraping server

Most prompt failures I debug are not wording problems. The model simply doesn't have the facts: the page changed last week, the docs moved, or the answer lives on a site it never saw. Better phrasing won't fix that. Better context will.

The pattern

1. Search for candidate pages relevant to the question.
2. Scrape the top results into clean Markdown (no nav bars, cookie banners or scripts).
3. Trim to the sections that matter and keep the source URL next to each chunk.
4. Prompt with a strict instruction: answer only from the supplied context and cite the URL.

Step 4 is where prompt engineering actually pays off. A template I use:

You are given excerpts from web pages, each starting with SOURCE: <url>.
Answer the question using only these excerpts.
If the excerpts don't contain the answer, say so.
Cite the SOURCE url after each claim.

QUESTION: {question}
EXCERPTS:
{chunks}

Doing steps 1-2 from inside the assistant

Instead of writing a scraper per site, you can connect an MCP server so the assistant fetches pages itself. I've been using XCrawl, which exposes search, scrape, map and crawl as MCP tools (package xcrawl-mcp) and returns pages as Markdown or JSON. It handles JavaScript-rendered pages, proxies and retries on its side, so the assistant gets readable text back instead of raw HTML.

A typical session then looks like: "search for the pricing page of X, scrape it, and summarise the plan limits with citations." The model calls search, then scrape, then answers from the returned Markdown using the template above.

Tips

• Keep chunks small (a few hundred tokens each) and put the source URL in every chunk, not just once.
• Ask for "I don't know" explicitly; models are far more willing to admit gaps when the instruction allows it.
• For multi-page docs, map the site first, pick the relevant URLs, then scrape only those. It's cheaper and the context stays focused.
• Re-fetch rather than cache when freshness matters (pricing, changelogs, status pages).

Fresh, well-labelled context beats clever wording almost every time. Curious how others here structure retrieval prompts.