We are seeking a skilled Senior Data Engineer to design, develop, and maintain robust data pipelines and infrastructure. Your role will involve collaborating with the data engineering team to implement innovative solutions that meet business requirements. You'll work closely with stakeholders to translate requirements into technical specifications, troubleshoot pipeline issues, and ensure data quality and reliability. Proficiency in Apache Spark, Databricks, Azure Data tools, and big data technologies is required. If you have a strong background in data warehousing, SQL, and cloud platforms like AWS or Azure, and possess excellent problem-solving and communication skills.
Responsibilities:
Design, develop, and maintain robust and scalable data pipelines and infrastructure to support business needs.
Collaborate with the data engineering team to develop and implement innovative data-driven solutions.
Work closely with stakeholders to gather requirements and translate them into technical specifications.
Troubleshoot and debug issues related to data pipelines and infrastructure to ensure data quality and reliability.
Monitor and optimize data pipelines and infrastructure for performance and efficiency.
Stay current with the latest data engineering technologies, tools, and trends, and apply this knowledge to improve existing systems.
Qualifications:
Bachelor's degree in Computer Science, Information Technology, or a related field.
Minimum of 3 years of experience in data engineering.
Proficiency with Apache Spark and Databricks data platform.
Experience with Azure Data toolset and big data technologies such as Hadoop and Kafka.
Strong background in data warehousing and data lakes, including data modeling and data quality management.
Proficiency in SQL and NoSQL databases.
Experience with cloud computing platforms such as AWS, Azure, and GCP.
Strong problem-solving and analytical skills, with a keen attention to detail.
Excellent communication and interpersonal skills, with the ability to work effectively in a collaborative team environment.
Apache Spark, Databricks, Azure Data toolset, Hadoop, Kafka, SQL databases, NoSQL databases, Data warehousing, Data lakes, Cloud computing platforms (AWS, Azure, GCP).