Senior Data Engineer

Role Overview We are seeking an experienced Senior Data Engineer to design, build, and maintain scalable data pipelines and cloud-based data solutions. The role covers data ingestion, processing, storage, orchestration, and delivery to Analytics and BI teams within an AWS-based data ecosystem.

Responsibilities

Design and maintain batch and real-time data pipelines. Integrate data from APIs, databases, files, and external sources using Apache Kafka and AWS services. Develop and orchestrate workflows with Apache Airflow, including monitoring, retry mechanisms, and alerting.

Build and optimize distributed data processing solutions using Apache Spark, PySpark, Spark SQL, and AWS EMR. Manage Data Lake and Data Warehouse environments using Amazon S3 and Amazon Redshift. Design analytical data models, including Star and Snowflake schemas.

Prepare and optimize datasets for Analytics and Business Intelligence use cases, particularly for Qlik. Develop high-quality solutions using Python and SQL, following best practices for Git, code reviews, testing, and documentation.

Ensure data quality, security, governance, and compliance, including AWS IAM, encryption, and access control. Monitor production data pipelines and proactively address operational and performance issues. Mandatory Requirements Proven experience as a Senior Data Engineer or in a similar role.

Advanced proficiency in Python and SQL. Hands-on experience with: Apache Spark / PySpark Apache Kafka Apache Airflow Strong AWS experience, particularly with: Amazon S3 Amazon Redshift AWS EMR Experience with Data Lakes, Data Warehouses, ETL/ELT processes, and data modeling.

Strong understanding of batch and streaming architectures. Experience with performance optimization, Git, testing practices, and production support. Technical Stack Python SQL Apache Spark / PySpark Apache Kafka Apache Airflow AWS EMR Amazon S3 Amazon Redshift Git Qlik Originally posted on Himalayas