InfoQ Homepage Big Data Content on InfoQ
-
/articles/continuous-authorization-cloud/en/smallimage/continuous-authorization-cloud-thumbnail-1781599988842.jpg)
Designing Continuous Authorization for Sensitive Cloud Systems
Most cloud systems make one authorization decision at login. Everything after runs on trust established at authentication time. For systems handling regulated data, that gap is where breaches happen. This article presents a continuous authorization architecture covering risk-tiered evaluation, behavioral baselines, privacy-preserving audit trails, and a phased and incremental rollout.
-
/articles/spark-oom-kubernetes-misconfigurations/en/smallimage/spark-oom-kubernetes-misconfigurations-thumbnail-1780044756757.jpg)
Two Misconfigurations That Caused Spark OOM Failures on Kubernetes
After migrating Spark pipelines to Azure Kubernetes Service, two infrastructure settings interacted destructively: spark.kubernetes.local.dirs.tmpfs=true backed shuffle spill with RAM instead of disk, and a hard podAffinity rule forced all executors onto one node. Together, they caused repeated OOM kills invisible to standard diagnostics.
-
/articles/time-series-storage-design/en/smallimage/Time-Series-Storage-Design-Choices-That-Shape-Cost-and-Performance-thumb-1778155792101.jpg)
Time-Series Storage: Design Choices That Shape Cost and Performance
Every time-series database makes a set of storage design decisions: how to lay out rows, when to compress, what to partition on. These decisions determine cost and query performance more than the choice of database itself. This article works through those fundamentals from first principles, using widely available tools like PostgreSQL and Apache Parquet to make each trade-off measurable.
-
/articles/micro-batch-streaming-lessons-learned/en/smallimage/micro-batch-streaming-lessons-learned-thumbnail-1777381781538.jpg)
From Batch to Micro-Batch Streaming: Lessons Learned the Hard Way in a Delta Index Pipeline
This article describes how a production delta-index pipeline migrated from scheduled batch to micro-batch Spark Structured Streaming. It covers why record-level streaming was rejected, how partition-based watermarks replaced fragile S3 completion markers, overlap-window correctness, and restart-as-design strategies for better predictability in object-store–based ingestion systems.
-
/articles/agent-reinforcement-learning-apache-spark/en/smallimage/thumbnail-1769431334097.jpg)
Autonomous Big Data Optimization: Multi-Agent Reinforcement Learning to Achieve Self-Tuning Apache Spark
This article introduces a reinforcement learning (RL) approach grounded in Apache Spark that enables distributed computing systems to learn optimal configurations autonomously, much like an apprentice engineer who learns by doing. The author also implements a lightweight agent as a driver-side component that uses RL to choose configuration settings before a job runs.
-
/articles/secure-multi-party-computation/en/smallimage/secure-multi-party-computation-thumbnail-1743067702920.jpg)
How to Compute without Looking: a Sneak Peek into Secure Multi-Party Computation
This article shows how you can compute a function across multiple parties that do not trust each other without forcing them to share their individual inputs. This technique can be used to split secrets among parties, perform logical operations, or count votes in a way that ensures data privacy is preserved.
-
/articles/zero-knowledge-proofs-intro/en/smallimage/zero-knowledge-proofs-1710429219949.jpg)
Zero-Knowledge Proofs for the Layman
This article will introduce you to zero-knowledge proofs, a kind of cryptography you can use to provide the proof you know a secret, such as a private key or the solution to a problem, without ever sharing it to an interested party. While many articles exist on the topic, this will not require any high math knowledge.
-
/articles/impact-machine-learning-climate/en/smallimage/minimising-the-impact-of-machine-learning-on-our-climate-small-1685091723283.jpg)
Minimising the Impact of Machine Learning on our Climate
This article introduces the field of green software engineering, showing the Green Software Foundation’s Software Carbon Intensity Specification, which is used to estimate the carbon footprint of software, and discusses ideas on how to make machine learning greener. It aims to give you the tools to take an active part in the climate solution.
-
/articles/data-protection-federal/en/smallimage/logo-1673874259012.jpg)
Data Protection Methods for Federal Organizations and beyond
The Federal Data Strategy describes a plan to “accelerate the use of data to deliver on mission, serve the public, and steward resources while protecting security, privacy, and confidentiality." This article covers what it is and how it can be applied to any organization.
-
/articles/anatomy-code-obfuscation/en/smallimage/logo-1667849003893.jpg)
Who Moved My Code? An Anatomy of Code Obfuscation
In this article, we introduce the topic of code obfuscation, with emphasis on string obfuscation. Obfuscation is an important practice to protect source code by making it unintelligible. Obfuscation is often mistaken with encryption, but they are different concepts. In the article we will present a number of techniques and approaches used to obfuscate data in a program.
-
/articles/transatlantic-data-privacy-2022-virtual-panel/en/smallimage/virtual-panel-the-new-US-EU-data-privacy-framework-small-image-1663768696087.jpg)
Virtual Panel: the New US-EU Data Privacy Framework
Recent rulings by several European courts have set important precedents for restricting personal data transmission from the EU to the US. As a consequence, the US and EU have started working on a new agreement. In this virtual panel, three knowledgeable experts discuss where the existing agreements fall short, and whether a new privacy agreement could improve the current situation.
-
/articles/cloud-native-apache-dolphinscheduler/en/smallimage/logo-1655898721435.jpg)
Embracing Cloud-Native for Apache DolphinScheduler with Kubernetes: a Case Study
This article shares how Apache DolphinScheduler was updated to use a more modern, cloud-native architecture. This includes moving to Kubernetes and integrating with Argo CD and Prometheus. This improves substantially the user experience of deploying, operating, and monitoring DolphinScheduler.