REQUIRED TECHNICAL DEPTH
- Streaming & Messaging: Deep experience with distributed message queues (Kafka, RabbitMQ, AWS SQS, or Google Pub/Sub). You understand consumer groups, partitioning, and offset management.
- Change Data Capture (CDC): Proven experience implementing CDC pipelines (using Debezium or native database tools) to capture and stream transactional changes from legacy/ERP databases without impacting performance.
- Stream Processing: Proficiency with stream processors or declarative data pipelines (Apache Flink, Apache Spark Structured Streaming, or Benthos).
- Schema & Serialization: Strong experience with schema validation and evolution using Avro, Protobuf, or JSON Schema (with Pydantic or similar) to ensure data quality across disparate systems.
- Languages: Proficiency in Python or Java/Scala for building custom pipeline logic and data validation layers.
- Data Governance & Observability: Experience implementing data quality frameworks, lineage tracking, and observability tools (such as DataHub, OpenMetadata, Amundsen, or Great Expectations) to ensure trust and reliability in the data platform.
KEY RESPONSIBILITIES
- Architect and implement event-based ELT/ETL pipelines that capture, validate, and route data from ERP and legacy systems to a cloud data lake or warehouse in real time.
- Design for atomic-level orchestration: ensure that complex, distributed data workflows either complete fully or roll back cleanly, maintaining consistency across legacy sinks and modern targets.
- Manage and optimize message queue infrastructure to guarantee low-latency, fault tolerant event ingestion.
- Evaluate and implement modern data orchestration frameworks (such as Temporal, Apache Airflow, or Dagster) to manage complex workflow dependencies and stateful, long-running transactions.
- Establish data governance practices, including data quality monitoring, schema validation, and metadata management, using tools like DataHub or OpenMetadata to provide full visibility into data lineage and trust.
- Collaborate with the architecture team to integrate with cloud ecosystems (AWS, GCP, or Azure) as we scale our infrastructure and enterprise orchestration capabilities.
WHAT WE ARE LOOKING FOR
- 7-10 years of experience, with a heavy focus on streaming data architectures in the last 3-4 years.
- Deep understanding of distributed systems concepts: idempotency, exactly-once processing, and transactional boundaries across heterogeneous systems.
- A “builder” mentality—comfortable with greenfield development and navigating the constraints of legacy ERP integrations.
- Experience implementing Saga patterns or similar distributed transaction models to ensure atomicity across microservices and data pipelines.
- Strong coding skills in Python or Java, with a focus on data-heavy applications
- Proven experience with data observability and governance tools to ensure data quality and lineage across complex pipeline ecosystems.