PromptZone - AI Prompts, Guides and Tools for Builders

Cover image for Lakegres: Vector Search on Open Table Formats Without Data Duplication
Maeve Kobayashi
Maeve Kobayashi

Posted on

Lakegres: Vector Search on Open Table Formats Without Data Duplication

Onehouse announced Lakegres this week, an open-source extension that brings native vector search capabilities directly to data lake tables—flagged on a Hacker News thread for its potential to simplify AI data pipelines. It enables semantic search over existing Apache Iceberg, Apache Hudi, and Delta Lake tables without requiring data duplication into a separate vector database.

Project: Lakegres | Type: PostgreSQL extension | License: Apache 2.0
Core Function: Hybrid filtering (vector + SQL) | Tables: Iceberg, Hudi, Delta Lake

How It Works: A Query Engine, Not a Store

Lakegres is a PostgreSQL foreign data wrapper. It doesn't store vectors itself. Instead, it connects your Postgres instance directly to your data lakehouse tables (Iceberg/Hudi/Delta). When you run a vector similarity query, Lakegres pushes the computation down to the data lake, performing the search on the open table formats where your data already resides.

This architecture means your embedding vectors live alongside your traditional structured data in one table. You can run a single query that combines vector similarity search (e.g., ORDER BY embedding <=> '[0.1, 0.2,...]') with standard SQL filters (e.g., WHERE date > '2024-01-01' AND customer_tier = 'premium').

Lakegres: Vector Search on Open Table Formats Without Data Duplication

The Practical Benchmark: Eliminating the ETL Step

The primary benchmark isn't raw QPS, but total system simplicity. Traditional pipelines require a costly extract, transform, and load (ETL) process: new data arrives in the lakehouse, a job computes embeddings, and those embeddings are copied to a dedicated vector store like Pinecone or Weaviate.

With Lakegres, the pipeline collapses. Embeddings are written once to the lakehouse table. The same table is queried for analytics, reporting, and now, vector search. Onehouse's testing shows this removes hours of latency and the operational overhead of running and synchronizing a separate database system.

Bottom line: The performance gain is in pipeline reduction, not just query speed, by removing the entire data synchronization step.

How to Try It: Connect Your Lakehouse

Setup involves deploying the extension to a PostgreSQL-compatible database. The official GitHub repository provides the source and instructions.

"Basic setup steps"
  1. Build or acquire the extension from the Lakegres repo.
  2. Load it into your Postgres instance (e.g., CREATE EXTENSION lakegres;).
  3. Create a foreign server pointing to your object storage (S3, ADLS, GCS) and metastore (Hive, AWS Glue, Nessie).
  4. Import your existing table schema as a foreign table in Postgres.
  5. Query using standard SQL with the vector distance operator (<=>).

Once configured, you can run hybrid queries directly from any PostgreSQL client, including application ORMs, BI tools like Metabase, and notebook environments.

Pros and Cons: Trade-offs for Architecture

Advantages:

  • No Data Duplication: The single biggest benefit. Maintains one copy of truth in the lakehouse.
  • Unified Governance & Freshness: Security, access controls, and data freshness are managed at the lakehouse level. Queries always hit the latest data.
  • Powerful Hybrid Queries: Native combination of semantic vector search with rigorous SQL business logic.
  • Reduced Cost & Complexity: Eliminates the need to provision, pay for, and operate a separate vector database service.

Limitations & Considerations:

  • Query Latency: For ultra-low-latency, high-QPS online serving, a dedicated vector database will likely outperform a query pushed to object storage.
  • New Project: As a nascent open-source tool, it lacks the battle-tested reliability and managed service offerings of established vector DBs.
  • PostgreSQL Dependency: Requires a Postgres instance to act as the query endpoint.

Alternatives and Comparisons: When to Use What

Lakegres fits a specific architectural niche. It's not a direct replacement for all vector database use cases.

Feature Lakegres Pinecone/Weaviate (Managed VDB) pgvector (PostgreSQL)
Data Location Open Table Formats (Iceberg/Hudi/Delta) Proprietary Cloud Storage Inside PostgreSQL
Data Movement None (direct query) Required ETL (copy in) Requires import into PG
Hybrid Query Yes (native SQL + vector) Often limited or via workarounds Yes (native SQL + vector)
Best For Existing lakehouse, batch/analytical workloads High-QPS online serving, standalone apps Apps already on Postgres, simple embeddings
Management Manage lakehouse tables Fully managed service Self-manage PostgreSQL

Who Should Use Lakegres (And Who Shouldn't)

Use Lakegres if:

  • Your organization has already standardized on a modern lakehouse (Iceberg/Hudi/Delta).
  • Your primary use case involves analytical or batch-oriented semantic search (e.g., document retrieval for RAG, customer segmentation analysis) rather than millisecond-latency online inference.
  • You need to perform complex hybrid filtering where semantic search is one clause in a larger analytical SQL query.
  • Your team wants to avoid introducing a new database technology and can support a PostgreSQL extension.

Avoid Lakegres for now if:

  • You require sub-50ms p95 latency for a user-facing application.
  • You need a fully managed, hands-off service.
  • Your data isn't in a supported open table format.
  • Your use case is a simple semantic search with no need for joint SQL predicates.

Lakegres represents a maturation of the data stack, targeting the real pain point of data duplication. Its success will depend on query optimization, but it correctly identifies synchronization complexity as the next barrier for production AI.

Top comments (0)