How we gave a Spark matching pipeline something Spark doesn’t offer out of the box, a real search index, and made the matching step several times faster without changing results. Indexes: the thing Spark quietly doesn’t give you If you work in PySpark, you learn early that there’s no such thing as an index. Spark
Bringing Index to Spark with Split Indexes Read More »
Spark’s Coalesce vs Repartition vs Repartition-by-Range – My Experience with them If you’ve spent any time tuning Spark jobs, you’ve run into the classic question: do I call `coalesce()`, `repartition()`, or `repartitionByRange()`? All three change how your data is partitioned across the cluster, but they behave very differently under the hood — and choosing
Spark’s Coalesce vs Repartition vs Repartition-by-Range – My Experience with them Read More »
Modern data platforms live or die by how quickly you can ship code to production.For one of our recent projects, that speed was determined by something deceptively simple: adding a custom Python module to our distributed jobs. We started on AWS EMR Serverless and later moved to Azure Databricks for compute jobs.Both platforms can absolutely