Sr. AWS Data Engineer (ARC/HR/SAWSDE/03/0726)
Full-timeMid · 5+ yearsH1B likely
About this role
Role Overview:
We are seeking a highly skilled Senior Data Engineer with 5+ years of experience to join our team. This role is focused on designing and building robust data assets through high-performance data pipelines. You will be a key player in modernizing our data infrastructure, transitioning legacy codebases into clean, scalable architectures, and ensuring the highest standards of code quality through Test-Driven Development (TDD).
Key Responsibilities:
- Data Pipeline Development: Design, develop, and maintain complex ETL/ELT pipelines to build high-value data assets.
- Legacy Modernization: Lead the code refactorization of legacy codebases, improving readability, maintainability, and performance.
- System Optimization: Perform deep code optimization using Spark SQL and PySpark to handle large-scale datasets efficiently.
- Quality Assurance: Implement a Test-Driven Development (TDD) approach, writing comprehensive unit tests to ensure functionality and catch bugs early in the lifecycle.
- Complex Problem Solving: Isolate and resolve difficult bugs, including those related to performance bottlenecks, concurrency issues, and complex logic flaws.
- Cloud Architecture: Design and deploy solutions utilizing the full AWS stack, explaining the trade-offs and benefits of specific services for various use cases.
Technical Requirements:
Core Programming & Data Engineering
- 5+ years of experience in hands-on programming with Python and PySpark.
- Expertise in Boto3 and various Python frameworks and libraries, adhering strictly to Python best practices (PEP 8).
- Strong experience in Spark SQL and PySpark optimization techniques (e.g., partitioning, caching, broadcast joins).
Cloud & Infrastructure (AWS)
- Deep architectural knowledge of AWS services, including: S3, EC2, Lambda, Redshift, CloudFormation
DevOps & Tools
- Advanced understanding of Git (branching strategies, PR reviews).
- Experience with JFrog Artifactory for dependency management and artifact storage.
- Proficiency in CI/CD pipelines and automated testing frameworks.
Professional Attributes:
- Analytical Mindset: Ability to debug complex, non-obvious issues in distributed systems.
- Clean Coder: Passion for writing "clean code" and mentoring junior engineers on maintainability.
- Architectural Thinking: Ability to explain the "why" behind choosing specific AWS components over others.