Azure Data Engineer

Contract Period: Six months (Extendable)

Experience: 3+ years of experience

Seniority level: Mid

We Have:

• Mission-led project that will revolutionize our ability to restore coral reefs affected by global warming.
• Team of Developers, Architects, ML Engineers, Bank-End Engineers and Front-End Developers interacting with back-end service architecture deployed on a Cloud Platform• Innovation, team culture is at the heart of what we do
• An international team that promotes a multicultural working environment
• Offering great career growth opportunities working for international top clients within the global technology market
• An open mind for new ideas and methodologies and in fact the person filling the role is encouraged to come up with new data business concepts/products/features etc.
• Adaptability. We share data knowledge and push to develop as innovation to create solutions that enable our clients goals.
• A passion for solving hard problems.

You Have:

• Develop scalable and efficient data pipelines using Azure Data Factory, Databricks, and Synapse Analytics.
• Experience with Programming languages like Python and libraries like PySpark, Spark Streaming.
• Experience with working data frames like Pandas and Spark Data frames.
• Implement data storage solutions such as Azure Data Lake, Azure SQL Database, and Cosmos DB.
• Implement data pipelines to feed real-time and batch data to AI/ML models.
• Support deployment and monitoring of machine learning models using Azure Machine Learning and ML Ops best practices
• Integrate data from various on-premises and cloud sources.
• Create and optimize Extract, Transform, Load (ETL) processes to ensure data quality and consistency.
• Design and implement data models for both operational and analytical systems.
• Ensure alignment with business and technical requirements.
• Monitor and tune data pipelines and databases to optimize performance and reduce costs.
• Implement caching strategies and partitioning to enhance data retrieval times.
• Implement data security measures, including encryption and access controls, following best practices and organizational standards.
• Ensure compliance with relevant regulations such as GDPR and HIPAA.
• Work closely with data analysts, data scientists, and business stakeholders to understand data needs and deliver actionable insights.
• Utilize third-party tools like Tableau, Power BI, or Looker for data visualization and advanced analytics.
• Collaborate with DevOps teams to automate and manage data workflows.
• Set up monitoring for data pipelines to proactively identify and resolve issues.
• Perform root cause analysis for any data discrepancies or failures.
• Design and implement scalable data models tailored for AI/ML and analytical workloads.
• Support feature engineering and data preparation processes for machine learning models.

Good to have