Job Title : Senior Data Pipeline Engineer (Python / Airflow)
Location : Guadalajara, Mexico
Duration : Long Term
Responsibilities
- Design, build, and maintain Airflow DAGs using TaskFlow, dynamic DAGs,
- deferrable operators, providers, and the Secrets backend; manage cross‑DAG dependencies and SLAs.
- Develop Python ETL / ELT code to ingest from APIs, object storage, message buses, and databases; package code as reusable libraries.
- Operate Airflow on managed or self‑hosted platforms (e.g., Azure, Kubernetes deployments); implement blue / green or canary DAG releases.
- Implement data quality and testing with unit tests for operators / hooks, and DAG validation in CI.
- Build event‑driven pipelines for near‑real‑time processing; manage schemas and compatibility.
- Model and manage data stores across SQL and blob storage; design partitioning, clustering, and retention.
- Observability & lineage : instrument metrics / logs, set SLAs / alerts, drive post‑incident reviews and reliability improvements.
- Security & governance : apply least‑privilege IAM, secrets management, PII
- handling, and data contracts; enforce RBAC in Airflow and warehouses.
- CI / CD & IaC : build pipelines to lint / test / deploy DAGs and Python packages;
- provision infra with Terraform / Helm; containerize with Docker.
- Cost & performance : tune task parallelism, autoscaling, storage formats, and compute footprints to optimize cost / perf.
- Collaboration : work closely with Android / backend teams to define interfaces and data contracts; document decisions and operational runbooks.
Skills and Qualifications
8+ years in data engineering or backend engineering with strong Python expertise.2+ years Airflow 2.x expertise (operators, hooks, sensors, TaskFlow, schedulertuning).Proven experience designing reliable ETL / ELT at scale (batch and streaming) withrobust testing and monitoring.Strong SQL and data modeling skills; hands‑on with one or more data warehouses(BigQuery, Redshift, Snowflake) and relational systems (PostgreSQL / MySQL).Familiarity with security best practices (RBAC, OAuth2 / OIDC for serviceintegrations), API gateways, and secrets management (Vault / AWS Secrets Manager / GCP Secret Manager).Comfortable operating in production : monitoring, troubleshooting, and performance tuning.Excellent written and verbal communication; clear trade‑off communication andautonomous execution with well‑documented decisions.