AWS Data Architect - Python/PySpark

Role Overview: We are seeking an experienced AWS Data Architect to design and implement a scalable data platform requirement. The candidate should have majorly worked on diverse formats, requiring extensive transformation, normalization, and standardization. The architect will lead the creation of a robust, scalable, and governed data platform leveraging AWS services such as Redshift, AWS Glue, and S3, ensuring seamless ingestion, transformation, and analytics readiness. Key Responsibilities: - Design end-to-end data architecture for ingesting data from multiple external systems. - Define data lake and data warehouse architecture using AWS services (S3, Redshift). - Establish robust data ingestion and processing patterns (batch, ETL/ELT). - Ensure scalability, performance, and cost optimization. - Design ingestion pipelines for structured, semi-structured, and unstructured data. - Handle various ingestions mechanisms like SFTP-based/API or event driven with varying file formats (CSV, XML, JSON, etc.). - Implement data validation, error handling, and reconciliation processes. - Lead complex data transformation workflows to unify disparate datasets into a consistent canonical format. - Define and enforce data quality and consistency standards across all incoming datasets. Optimize Redshift workloads: 1. Query tuning 2. Workload management (WLM) 3. Data distribution strategies Define data governance frameworks: 1. Metadata management 2. Data lineage 3. Data quality checks - Should have implemented monitoring, alerting, and audit mechanisms. - Proven track record of designing end-to-end cloud data platforms. - Experience building scalable, resilient architectures on AWS. Technical Skills: - Amazon Redshift - AWS Glue (ETL, PySpark) - Amazon S3 - Experience loading multiple format data ingestion config driven solution - SQL (advanced) - Python / PySpark - Data warehousing concepts - ETL/ELT design patterns