Note: The job is a remote job and is open to candidates in USA. Mphasis is a company that applies next-generation technology to help enterprises transform businesses globally. They are seeking a highly skilled Lead Data Engineer to design, implement, and maintain scalable data pipelines and architectures that support data-driven initiatives.
Responsibilities
- Design and develop robust data pipelines using AWS services, ensuring high availability and performance
- Implement data processing solutions using Lambda, Glue, and EMR to handle large datasets efficiently
- Utilize Kafka for real-time data streaming and processing
- Collaborate with data scientists and analysts to understand data requirements and deliver solutions that meet business needs
- Manage and optimize data storage solutions using S3, DynamoDB, MongoDB, Snowflake, and Databricks
- Develop and maintain ETL processes using Airflow and other orchestration tools
- Utilize Terraform for infrastructure as code to manage AWS resources effectively
- Ensure data quality and integrity through rigorous testing and validation processes
- Provide technical leadership and mentorship to junior data engineers
- Stay updated with the latest industry trends and technologies to continuously improve data engineering practices
Skills
- Must have at least 10 years of experience
- Must have telecom experience
- Bachelor's degree in Computer Science, Engineering, or a related field
- Strong analytical and troubleshooting skills
- Excellent communication and collaboration abilities
- Proven ability to work independently and as part of a team
- Strong expertise in AWS Lambda is a deal breaker
- Proficient in Java and Python programming languages
- Experience with PySpark for big data processing
- Knowledge of Kafka for real-time data streaming
- Hands-on experience with AWS EMR and S3 for data storage and processing
- Familiarity with AWS Glue for ETL processes
- Experience with data visualization tools like Looker
- Proficient in using Terraform for infrastructure management
- Knowledge of NoSQL databases such as DynamoDB and MongoDB
- Experience with data warehousing solutions like Snowflake and Databricks
- Familiarity with workflow orchestration tools like Airflow
- Experience with containerization technologies such as Docker
- Knowledge of machine learning frameworks and libraries
- Familiarity with Agile methodologies and DevOps practices
- Strong problem-solving skills and ability to work in a fast-paced environment
Company Overview
Company H1B Sponsorship