A reliable research prompt should handle an absent source just as clearly as an available one. Before trusting a model with literature reviews or synthesis, run a small boundary evaluation: provide two short public passages and ask for a comparison that deliberately requests a detail neither passage contains.
1. Build a Compact Prompt Fixture
Design your test prompt with strict grounding constraints, and include both an easy supported question and a missing detail. A tool that marks everything as unknown avoids fabrication, but fails the useful part of the task. You want to see if it accurately extracts what the passages do contain while refusing the single missing detail.
Compare the supplied passages. For each factual claim, identify the supporting passage. Put unsupported requests in an "Unknown" section. Do not fill those gaps from memory.
[Insert Question 1: Supported by Passage A/B]
[Insert Question 2: Supported by Passage A/B]
[Insert Question 3: Missing from both passages]
2. Run Controlled Model Candidates
Run the exact same fixture across the models or tools you are considering. A directory such as ewiki can help locate tools, but a listing does not establish how a specific model will handle your evidence boundary.
Start with a deliberately obvious gap, then try a subtler one. Keep those results separate so a harder edge case does not erase what your baseline test established.
3. Inspect the Unsupported Detail
Once the output is generated, audit the handling of the missing detail:
- Gap Admission: Does the answer explicitly place the missing detail in the "Unknown" section, or does it silently attach an unrelated citation?
- Source Verification: Check the cited passage yourself. High fluency and a tidy reference list are weak substitutes for actual textual support.
(Note: While frameworks like the NIST AI Risk Management Framework offer broader context for evaluating AI systems, this tiny fixture is strictly a local check, not a general reliability score.)
4. Log and Iterate
Save the prompt, source text, model settings, and raw output. If you revise your system instructions, rerun the exact same example before adding harder cases so you can isolate whether the prompt adjustment actually improved boundary adherence.
Which unsupported question—or edge-case phrasing—has been most revealing in your own prompt tests?
Appendix: Ready-to-Use Test Fixture (Copy & Paste)
If you want to run this check immediately, copy the block below into your evaluation playground. It uses two short fictional product notes so your model cannot rely on pre-trained web knowledge.
System / Instruction:
Compare the supplied passages. For each factual claim, identify the supporting passage (Passage A or Passage B). Put unsupported requests in an "Unknown" section. Do not fill those gaps from memory.
---
Passage A (HeliosDB Release Note):
HeliosDB 2.4 introduces native vector indexing for datasets up to 50 million embeddings. In internal benchmarks on 16-core machines, query latency dropped by 38% compared to version 2.1. The release supports cosine similarity and Euclidean distance, and it requires at least 32 GB of system RAM.
Passage B (StrataStore Architecture Brief):
StrataStore is designed for high-throughput time-series ingestion, handling up to 1.2 million writes per second per node. Unlike general-purpose stores, it compresses historical telemetry using delta-of-delta encoding, reducing storage footprint by roughly 60%. It runs on Linux and macOS, with official ARM64 binaries provided.
---
Questions to Answer:
1. [Supported Comparison] What is the primary performance metric highlighted for each system, and what hardware/OS requirements are mentioned?
2. [Obvious Gap] What is the annual enterprise licensing cost for HeliosDB 2.4 and StrataStore?
3. [Subtle Gap] Which system delivers lower query latency when running on ARM64 processors?
What a Passing Output Looks Like
- Question 1 (Supported): Correctly cites Passage A (38% query latency drop on 16-core machines; requires 32 GB RAM) and Passage B (1.2M writes/sec per node; runs on Linux/macOS with ARM64 binaries).
- Question 2 (Obvious Gap): Places pricing/licensing cleanly under Unknown, since neither passage mentions cost.
- Question 3 (Subtle Gap - Trap): A weak prompt will merge Passage A's "query latency" with Passage B's "ARM64 binaries" and hallucinate a comparison. A passing output puts this in Unknown, noting that Passage A does not mention ARM64 and Passage B does not report query latency.

Top comments (0)