How we gave a Spark matching pipeline something Spark doesn’t offer out of the box, a real search index, and made the matching step several times faster without changing results.
Indexes: the thing Spark quietly doesn’t give you
If you work in PySpark, you learn early that there’s no such thing as an index. Spark has no persistent index you can build once and reuse; every query and join means reading the data and scanning it again. For many workloads that’s fine; Spark is built to scan huge datasets in parallel. But when you keep coming back to the same large dataset and only need a small slice each time, the lack of an index really hurts.
And it’s a strange thing to miss, because anyone from the database world knows what to reach for when a query gets slow: add an index, and it jumps straight to the rows that matter. That basic tool is gone once you move to Spark.
That was our situation. We had a job that matches a batch of incoming records against a large reference dataset, comparing them field by field, scoring how close they are, and keeping the pairs above a threshold. The reference data barely changes, but every run re-read all of it and scored against everything. As it grew, the job got slower and slower, with no “just add an index” button to reach for.
So we built one.
The achievement isn’t a clever query or a bigger cluster. It’s that we gave PySpark the missing capability: a genuine, reusable search index over a Spark dataset that we store once and then use to pull back only the records that could possibly match, the way a database index skips straight to the rows you need. Results stayed identical; the work Spark had to do shrank dramatically.
The idea: index once, search many times
Instead of reading the full reference dataset every run, we index it once. Then, for each incoming batch, we use that index to pull out only the handful of reference records that could possibly match, and run the expensive scoring on that small set instead of everything.
The tool that made this possible is a split-index search engine: a Rust-based, Quickwit/Tantivy-style search library that plugs into Spark as a data source. It indexes the columns you care about and packs the whole index into a single self-contained file called a split.
The key property of a split is partial reads: you open it, ask “which rows contain this value?”, and it reads only the tiny slice of the file holding that answer, not the whole thing. That’s what makes searching cheap, and it’s the closest thing to a real database index you can get inside Spark.
What a split file actually is
A split bundles everything an index needs into one file, with a small footer at the end recording where each section lives. The sections are:
- The inverted index: “which rows contain this term?” This is what search uses.
- The document store: the original rows, for retrieval.
- The fast fields: columnar values, for quick counts and sums.
- The footer: a table of byte offsets so a reader can jump straight to any section.
Because the footer knows where each section sits, a lookup reads the footer plus one small section and stops; it never has to touch the document store or the fast fields. That “read only what you asked for” behavior is why a split can stand in for a database index.
The single most important lesson
Here’s the thing that took us longest to learn, worth stating plainly: an index speeds up search, not a join. It does not make a scan-everything-and-join operation faster. It only helps if you actually use it to narrow down the candidates before you do the expensive work.
At first we thought: build an index over the reference data, point the join at it, done. That did nothing. The join still needed every row, so the index was pure overhead, a build step for no speedup.
The win only appears when you flip the flow around:
- Look at the incoming batch and collect the values it actually contains.
- Search the index for reference rows that share those values.
- Run the expensive scoring only on that small candidate set.
That’s the whole trick: the index is a filter that runs before the join, not a faster join. It’s exactly how a database uses an index; it just does it invisibly. In Spark, we had to make that flow explicit.
Getting the index types right
By default, a text column is indexed as tokenized full-text, good for “contains this word,” wrong for “equals exactly this value.” For exact-match columns, you have to tell the engine to treat them as keyword values instead. Miss this, and your exact-match lookups behave like fuzzy text search and quietly return the wrong rows. A one-line setting, but easy to overlook.
Why we ran this on a classic cluster, not serverless
This matters if you’re on a managed Spark platform. The split-index engine is a native library: a custom package that does its own low-level reads and writes directly against cloud object storage. Two consequences ruled out serverless:
- It needs a custom library installed on the cluster. Serverless compute is a locked-down runtime, with no installing arbitrary native libraries or controlling the environment. A classic cluster lets you upload the library and attach it.
- It does its own native I/O to storage, so it needs storage credentials handed to it directly, separate from whatever the platform manages for you. On a classic cluster, you set that in the cluster configuration.
So the setup was a classic, single-user cluster: a runtime whose versions match the library exactly (a mismatch throws cryptic errors), the library attached, and the engine’s settings and credentials in the cluster config. We used single-user access mode specifically, since shared mode blocked the low-level operations and some session config the engine needs. The index build and search steps run here; everything downstream, the scoring itself, is ordinary Spark and runs wherever.
Proving we didn’t break anything
A faster pipeline that returns different results is worthless. So before adopting this, we ran the old and new paths side by side on the same data and compared the exact set of matched pairs:
- Old path: read the full reference set, score everything.
- New path: search the index for candidates, materialize them, score the small set.
The test asserted two things: the new path was meaningfully faster, and its set of matches was identical to the old one. We kept it only because both held.
There’s reasoning behind that confidence, not just a lucky run: the narrowing search is built to return a superset of whatever the scoring step could keep. If a reference row can’t clear the search filter, it could never have survived scoring either, so narrowing can’t drop a real match. The parity test is the proof; the superset argument is why it passes.
The payoff
Once narrowing was in place, the shape of the work changed completely. Instead of scoring against the entire reference dataset every batch, we scored against a small, targeted slice, often a tiny fraction of the whole.
In our benchmark on a representative dataset, the matching step went from scanning and scoring the full reference set to touching only the relevant candidates, and the run time dropped several-fold, with an identical set of matches. The bigger the reference data grew, the better the trade looked: the old path scaled with the whole dataset, the new one with how much the incoming batch actually overlaps it. In effect, we gave PySpark the one thing it was missing for this kind of work: a real, reusable index.
What we’d tell someone starting out
- Spark has no built-in index, but you can add one. A split-based search engine gives you a persistent, reusable index over a Spark dataset, a genuine gap-filler for repeat-access workloads.
- An index accelerates search, not joins. If you’re not using it to narrow candidates before the expensive step, it’s just overhead.
- Benchmark before you commit. Run old-vs-new on real data and check both speed and result parity. If the narrowing isn’t selective, the index may not be worth it.
- Materialize before you join. Don’t join the live index; write the narrowed results out and read them back.
- Mind the small stuff: exact-match columns as keyword type, version your index locations to dodge the schema cache, append-not-overwrite on first write, and many small AND-searches over one big OR.
- Know your runtime. A native library needs a cluster you can install it on and hand credentials to, not a locked-down serverless one.
None of the pieces are exotic. The win came from getting the flow right: index once, search to narrow, materialize, then do the expensive work on the small set, and prove the answer never changed.
Want to do the same in your own workflows?
If your Spark pipelines keep re-scanning the same large dataset and you’d like to bring real, reusable indexing to them, or you’re just curious whether it fits your workload, we’d love to hear from you. Reach out and let’s talk it through.