Why Instacart Ditched Elasticsearch for Postgres
A simple, practical look at how search engines work, why Instacart replaced Elasticsearch with Postgres, and how they actually made it 2x faster.
Why Instacart Ditched Elasticsearch for Postgres
If you spend any time on tech Twitter or Reddit, you have probably seen the advice: "Throw away your Elasticsearch cluster. Postgres can do all your search now!"
It sounds like a dream. Who wouldn't want to shut down a complex, memory-hungry search cluster and run everything inside the database they already own?
The hype exploded when Instacart published an engineering retrospective titled How Instacart Built a Modern Search Infrastructure on Postgres. Developers celebrated it as proof that Elasticsearch was dead and Postgres had won.
Except two critical details were lost in the headlines:
- Instacart didn't do this yesterday. They finished moving text search to Postgres way back in 2021. Their 2025 post was looking back at that journey and explaining how they recently added AI vector search (
pgvector). - Instacart wasn't having slow search queries. Their bottleneck was billions of price and inventory updates hammering their cluster every single day.
To understand why Instacart migrated (and why copying them blindly might backfire), we need to understand how search works under the hood, why Elasticsearch struggled with grocery data, and how Postgres actually gave Instacart 2x faster search performance.
1. How Search Actually Works
Traditional databases are great at finding exact matches, like WHERE id = 42 or WHERE email = 'alex@example.com'.
Search engines are different: they need to find documents containing words like "running" even if the user typed "run", and rank the most relevant items first.
The Inverted Index (The Book Index Analogy)
If you want to find the word "database" in a 500-page book, you don't read page 1 through 500. You flip to the index at the back of the book. You look up "database" and see the page numbers: 12, 45, 89.
That is an Inverted Index:
Word (Term) Matching Documents
─────────────────────────────────
runner → Doc 3, Doc 17, Doc 92
shoe → Doc 3, Doc 17, Doc 41, Doc 92
size → Doc 3, Doc 88To find documents matching "runner" AND "shoe", the search engine simply looks at the two lists and finds the overlap: Doc 3, Doc 17, and Doc 92.
Text Cleaning (Analysis)
Before any text is indexed, the search engine cleans it up:
- Lowercasing: Converts everything to lowercase so
"Shoe"matches"shoe". - Stemming: Trims word endings so
"runners","running", and"runner"all reduce to the base root word"runner". - Stop Words: Throws out common filler words like
"the","a", and"is"that don't help relevance.
Why Search Engines Hate Frequent Updates
Elasticsearch is powered by a library called Apache Lucene. Lucene uses a very specific design rule:
Once a search index file (called a segment) is written to disk, it is NEVER modified.
Making files read-only makes search queries blisteringly fast because hundreds of queries can read the same file without needing locks.
The catch: What happens when you update a product's price?
Because the files cannot be edited, Elasticsearch cannot update data in place. Instead, it marks the old document as deleted in a side file and writes a completely new document to a new file.
This detail is what created Instacart's crisis.
2. The Instacart Nightmare: Write Amplification
To understand Instacart's problem, you have to understand how grocery shopping works.
Instacart lists billions of items across tens of thousands of grocery stores. A gallon of milk at a store in New York has a different price and inventory count than the same milk in San Francisco. Prices change, sales start, and items go out of stock thousands of times per second.
The Denormalization Trap
Because Elasticsearch cannot perform fast SQL-style joins across multiple tables, search data is denormalized, meaning all product details, store IDs, prices, and stock levels are smashed together into a single JSON document:
{
"product_id": 101,
"title": "Organic Whole Milk 1 Gallon",
"brand": "Horizon Organic",
"description": "Grade A pasteurized fresh organic milk...",
"store_id": 42,
"price": 4.99,
"in_stock": true
}Now, what happens when store #42 changes the price from $4.99 to $4.79?
Because Elasticsearch files are read-only, updating the price forces Elasticsearch to delete the entire document and re-index all the unchanged text (title, brand, long description) from scratch!
Instacart was processing billions of price and inventory updates every day. Their search cluster was spending almost all of its CPU re-indexing the word "Organic Whole Milk" over and over again rather than actually searching.
When bad data occurred, re-indexing their catalog took days.
3. How Instacart Got 2x Better Performance on Postgres
Instacart decided to pull text search out of Elasticsearch and bring it into sharded Postgres.
The result?
- Write workload dropped by 10x
- Search response times got ~2x faster
How did moving from a dedicated search engine to a relational database make searches faster?
Here is the secret:
1. Normalizing Data (Separating Static from Volatile)
Instacart split their data into normalized SQL tables:
- Products Table: Holds static data (title, brand, description) and the full-text search index. This table rarely changes.
- Inventory Table: Holds rapidly changing data (price, in_stock, store_id). This table is updated constantly.
When a price changes, Postgres updates a tiny row in the inventory table. The search index on the product table is never touched. This single change eliminated 90% of their write workload overnight!
2. Moving Compute to the Data Layer (No Network Bloat)
In their old Elasticsearch architecture, search queries had to retrieve large product documents over the network into application servers, where application code joined and filtered the items by store and price in memory.
With Postgres, filtering by store, checking inventory, and joining tables happened directly inside the database engine on ultra-fast local NVMe drives.
Instead of moving megabytes of raw JSON across the network to filter it in Python or Ruby, Postgres filtered everything locally and sent only the small, clean final result list back to the user.
The takeaway: Instacart didn't win because Postgres is a faster text search algorithm. They won because a normalized database model eliminated write amplification and pushed filtering directly to fast local storage.
4. How Postgres Does Full-Text Search
So how does Postgres actually search text?
Postgres uses two built-in building blocks: tsvector and the GIN index.
tsvector (The Word List)
A tsvector takes your text, cleans it, stems words, and stores their positions:
SELECT to_tsvector('english', 'The Runners Shoes, Size 10!');
-- Output: '10':5 'runner':2 'shoe':3 'size':4You can create a generated search column on your table that updates automatically:
ALTER TABLE products ADD COLUMN search_vector tsvector
GENERATED ALWAYS AS (
setweight(to_tsvector('english', coalesce(title, '')), 'A') ||
setweight(to_tsvector('english', coalesce(description, '')), 'B')
) STORED;(The 'A' and 'B' tags let you tell Postgres that words in the title are more important than words in the description).
The GIN Index (Generalized Inverted Index)
To make searches fast, you build a GIN index on your tsvector column:
CREATE INDEX idx_products_search ON products USING gin(search_vector);Under the hood, Postgres creates a B-tree of all unique words. Each word points directly to the rows where it appears.
Searching is as simple as:
SELECT id, title
FROM products
WHERE search_vector @@ to_tsquery('english', 'runner & shoe')
LIMIT 10;5. The Real Trade-Offs: Postgres vs Elasticsearch
If Postgres is so great, why does Elasticsearch exist at all? Because plain Postgres search has clear limits.
| Feature | Elasticsearch | Plain Postgres FTS |
|---|---|---|
| Best For | Massive text search & log analytics | Relational apps needing built-in search |
| Data Freshness | ~1 second delay (Near Real-Time) | Instant on transaction commit (ACID) |
| Relevance Quality | Industry-standard BM25 ranking | Basic frequency scoring (ts_rank) |
| Joins Across Tables | Weak / Requires flat denormalized data | Native, ultra-fast relational SQL joins |
| Top-10 Speed on Huge Tables | Skips non-relevant items early | Scores and sorts all matching rows first |
| Operational Overhead | High (JVM tuning, cluster nodes, sync) | Low (It's just your existing database) |
Where Plain Postgres Search Struggles
- Relevance Ranking (No BM25): Elasticsearch uses BM25, an algorithm that factors in word rarity (rare words matter more than common words) and document length. Postgres's
ts_rankdoes not know how rare a word is across your whole catalog, so its ranking can feel crude on large, diverse catalogs. - Top-K Speed on Huge Corpora: In Postgres, if a search matches 500,000 rows, Postgres has to find all 500,000 rows, score all 500,000 of them, sort them, and only then return the top 10.
LIMIT 10does not prevent Postgres from scoring all matches first.
6. The Dual-Write Headache (Why Keeping Two Databases Sucks)
When you run Elasticsearch alongside Postgres, your application has to keep both systems in sync:
When state lives in two independent databases, you inevitably hit three classic failure modes:
- Silent Data Drift: If your app crashes after saving to Postgres (Step 1) but before indexing in Elasticsearch (Step 2), that record is permanently missing from search results.
- Backpressure & Dropouts: During traffic spikes, Elasticsearch's write queue fills up, causing index requests to time out or fail.
- Pipeline Lag: If you use streaming tools like Debezium and Kafka to sync WAL changes, high write volume causes search results to lag behind by minutes or hours.
Consolidating search into Postgres doesn't just fix your sync pipeline; it completely deletes the need for one.
7. The Game Changer: BM25 Inside Postgres
What if you could keep the simplicity of Postgres, but get the advanced relevance scoring of Elasticsearch?
Over the past two years, open-source projects have brought native BM25 search directly into Postgres:
- ParadeDB (
pg_search): Embeds Tantivy (a blazing-fast Rust port of Lucene) directly inside Postgres as a native index. - VectorChord-BM25: A native BM25 index extension for Postgres.
- Tiger Data (
pg_textsearch): Timescale's open-source BM25 extension.
These extensions let you run real BM25 ranked queries directly in SQL:
-- Example with ParadeDB pg_search
CREATE INDEX idx_products_bm25 ON products
USING bm25 (id, title, description)
WITH (key_field='id');
-- Fast, relevance-ranked query with title boosting!
SELECT id, title, score
FROM products
WHERE products @@@ 'title:running^3 OR description:shoes'
ORDER BY score DESC
LIMIT 10;Companies like Modern Treasury (managing 10TB+ of financial data across 500M+ rows) and Bilt Rewards (handling 7,000 writes/second) have deployed Postgres BM25 extensions in production, cutting query timeouts by 95% while avoiding the complexity of an external Elasticsearch cluster.
8. When Should You Keep Elasticsearch?
Elasticsearch is still an unbeatable tool when used for its true strengths:
- Relevance is your core product: You need deep custom synonym dictionaries, phonetic matching (sounds-like search), multi-language analyzers, or machine-learning search ranking.
- Massive Datasets (100M+ to Billions of Rows): Your data spans hundreds of millions or billions of documents that must be sharded across dozens of dedicated search nodes.
- Complex E-Commerce Faceting: Your users rely on instant multi-category filter aggregations ("Shoes (1,240), Size 10 (450), Blue (89)") updated on every keystroke.
- Logs, Metrics, and Observability: Ingesting millions of events per second into time-series indices is the Elastic Stack's bread and butter.
9. Practical Decision Guide
Here is a straightforward way to choose the right search architecture for your next project:
The 3-Step Rule of Thumb:
- Starting out or under 5 million rows? Start with Postgres built-in FTS. It is fast, simple, and already in your database.
- Need better relevance or vector AI search in Postgres? Use ParadeDB (
pg_search) orpgvectorbefore adding a new database to your infrastructure. - Operating at massive scale or building complex log observability? Use Elasticsearch or OpenSearch.
Summary
The real lesson from Instacart is simple:
Instacart didn't leave Elasticsearch because Postgres had a smarter search algorithm. They left because running a second copy of their data in a denormalized search engine created an unsustainable write-amplification nightmare.
By moving search into Postgres, they normalized their data, pushed joins and filtering directly to local NVMe storage, and eliminated 90% of their write overhead, achieving 2x faster search latency in the process.
Every database you add to your stack comes with an operational tax. If you need Elasticsearch's distributed power and advanced relevance, pay that tax gladly.
If you don't, keeping search inside Postgres will keep your architecture simple, reliable, and blisteringly fast.
Further Reading
- How Instacart Built a Modern Search Infrastructure on Postgres (Instacart Engineering)
- Elasticsearch Was Never a Database (ParadeDB Engineering)
- VectorChord-BM25 Architecture (VectorChord Engineering)
- Tiger Data pg_textsearch (Timescale Open Source)
- PostgreSQL Full Text Search Documentation