# Giving your prompts fresh web context: a practical pattern with an MCP scraping server

> Published 2026-11-02 · https://www.promptzone.com/xcrawl/giving-your-prompts-fresh-web-context-a-practical-pattern-with-an-mcp-scraping-server-1hoa

Most prompt failures I debug are not wording problems. The model simply doesn't have the facts: the page changed last week, the docs moved, or the answer lives on a site it never saw. Better phrasing won't fix that. Better context will.

## The pattern

1. **Search** for candidate pages relevant to the question.
2. **Scrape** the top results into clean Markdown (no nav bars, cookie banners or scripts).
3. **Trim** to the sections that matter and keep the source URL next to each chunk.
4. **Prompt** with a strict instruction: answer only from the supplied context and cite the URL.

Step 4 is where prompt engineering actually pays off. A template I use:

```plaintext
You are given excerpts from web pages, each starting with SOURCE: <url>.
Answer the question using only these excerpts.
If the excerpts don't contain the answer, say so.
Cite the SOURCE url after each claim.

QUESTION: {question}
EXCERPTS:
{chunks}
```

## Doing steps 1-2 from inside the assistant

Instead of writing a scraper per site, you can connect an MCP server so the assistant fetches pages itself. I've been using [XCrawl](https://www.xcrawl.ai/), which exposes search, scrape, map and crawl as MCP tools (package `xcrawl-mcp`) and returns pages as Markdown or JSON. It handles JavaScript-rendered pages, proxies and retries on its side, so the assistant gets readable text back instead of raw HTML.

A typical session then looks like: "search for the pricing page of X, scrape it, and summarise the plan limits with citations." The model calls search, then scrape, then answers from the returned Markdown using the template above.

## Tips

- Keep chunks small (a few hundred tokens each) and put the source URL in every chunk, not just once.
- Ask for "I don't know" explicitly; models are far more willing to admit gaps when the instruction allows it.
- For multi-page docs, map the site first, pick the relevant URLs, then scrape only those. It's cheaper and the context stays focused.
- Re-fetch rather than cache when freshness matters (pricing, changelogs, status pages).

Fresh, well-labelled context beats clever wording almost every time. Curious how others here structure retrieval prompts.