Senior Data Pipeline Engineer (Python / Airflow)...

Job Title : Senior Data Pipeline Engineer (Python / Airflow)

Location : Guadalajara, Mexico

Duration : Long Term

Responsibilities

• Design, build, and maintain Airflow DAGs using TaskFlow, dynamic DAGs,
• deferrable operators, providers, and the Secrets backend; manage cross‑DAG dependencies and SLAs.
• Develop Python ETL / ELT code to ingest from APIs, object storage, message buses, and databases; package code as reusable libraries.
• Operate Airflow on managed or self‑hosted platforms (e.g., Azure, Kubernetes deployments); implement blue / green or canary DAG releases.
• Implement data quality and testing with unit tests for operators / hooks, and DAG validation in CI.
• Build event‑driven pipelines for near‑real‑time processing; manage schemas and compatibility.
• Model and manage data stores across SQL and blob storage; design partitioning, clustering, and retention.
• Observability & lineage : instrument metrics / logs, set SLAs / alerts, drive post‑incident reviews and reliability improvements.
• Security & governance : apply least‑privilege IAM, secrets management, PII
• handling, and data contracts; enforce RBAC in Airflow and warehouses.
• CI / CD & IaC : build pipelines to lint / test / deploy DAGs and Python packages;
• provision infra with Terraform / Helm; containerize with Docker.
• Cost & performance : tune task parallelism, autoscaling, storage formats, and compute footprints to optimize cost / perf.
• Collaboration : work closely with Android / backend teams to define interfaces and data contracts; document decisions and operational runbooks.

Skills and Qualifications