This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Senior Data Engineer (AWS to GCP) based in India.
You will join a high-performing data engineering team building and evolving large-scale commercial data products within a dynamic media and streaming environment.
The role sits at the heart of a major cloud transformation, with key systems being migrated from AWS to Google Cloud Platform (GCP).
Once the migration is complete, the focus will shift toward optimisation, performance tuning, and delivering new data capabilities that directly support business decision-making.
You will work on end-to-end data pipelines, modern cloud-native architectures, and large-scale datasets that power critical analytics and reporting.
This is a hands-on engineering role with strong ownership, where your contributions will have visible impact across multiple business domains.
You will collaborate closely with analysts, product owners, and engineers in an Agile, fast-moving, and collaborative environment.
The position is ideal for someone who enjoys solving complex data challenges and improving data reliability at scale.
Accountabilities:
- Design, build, and maintain scalable data pipelines and data products on Google Cloud Platform (GCP), ensuring performance, reliability, and scalability.
- Support the migration of existing AWS-based data workloads to GCP, ensuring smooth transition and minimal disruption.
- Develop and optimise ETL/ELT workflows using Python, SQL, and PySpark for large-scale data processing.
- Build, maintain, and enhance data transformations using BigQuery and Dataproc.
- Create and manage orchestration workflows using Apache Airflow (Cloud Composer).
- Implement and improve data quality frameworks, ensuring consistency, accuracy, and trust in data outputs.
- Collaborate with cross-functional teams (analysts, product, engineering) to translate business needs into robust data solutions.
- Participate in code reviews, technical discussions, and continuous improvement initiatives.
- Contribute to platform reliability, monitoring, documentation, and operational excellence.
Requirements:
- Strong commercial experience as a Data Engineer working with cloud-based data platforms.
- Hands-on expertise with Google Cloud Platform (GCP), especially BigQuery.
- Proven experience building and maintaining Airflow pipelines (Cloud Composer preferred).
- Strong programming skills in Python and solid SQL expertise.
- Experience with PySpark and distributed/large-scale data processing systems.
- Good understanding of data modelling and modern ETL/ELT design patterns.
- Experience working with Git-based workflows in collaborative engineering environments.
- Strong communication skills and ability to work effectively with both technical and non-technical stakeholders.
- Experience working in Agile delivery environments.
- Nice to have: experience with Dataproc, Dataplex, Great Expectations or similar data quality tools, AWS services (S3, MWAA, Databricks), and cloud migration projects.
- Exposure to media, streaming, or large-scale consumer platforms is a plus.
Benefits:
- Competitive compensation aligned with senior-level expertise.
- Flexible working arrangements (remote/hybrid depending on team setup).
- Opportunity to work on a large-scale cloud migration and modern data platform.
- Career growth through exposure to complex, high-impact data products.
- Ongoing training, mentoring, and professional development support.
- Collaborative, Agile working environment that values learning and continuous improvement.
- Opportunity to work on international, large-scale datasets and systems.