How we gave a Spark matching pipeline something Spark doesn’t offer out of the box, a real search index, and made the matching step several times faster without changing results. Indexes: the thing Spark quietly doesn’t give you If you work in PySpark, you learn early that there’s no such thing as an index. Spark […]
Bringing Index to Spark with Split Indexes Read More »
Spark’s Coalesce vs Repartition vs Repartition-by-Range – My Experience with them If you’ve spent any time tuning Spark jobs, you’ve run into the classic question: do I call `coalesce()`, `repartition()`, or `repartitionByRange()`? All three change how your data is partitioned across the cluster, but they behave very differently under the hood — and choosing
Spark’s Coalesce vs Repartition vs Repartition-by-Range – My Experience with them Read More »
Modern data platforms live or die by how quickly you can ship code to production.For one of our recent projects, that speed was determined by something deceptively simple: adding a custom Python module to our distributed jobs. We started on AWS EMR Serverless and later moved to Azure Databricks for compute jobs.Both platforms can absolutely