We are seeking a talented Data Engineer to join our team. You will play a pivotal role in understanding business and technical challenges, developing and deploying robust data pipelines, and ensuring adherence to industry and internal best practices. This role requires a combination of technical expertise, communication skills, and a proactive mindset to drive data engineering excellence.
TasksProjects Understanding and Communication Collaborate with users to understand challenges and align solutions with their needs.Ensure clarity in architectural designs provided by the Data Architect.Maintain active communication with peers, Data Architects, and Project Managers regarding progress and solutions.Development Define and communicate interface contracts for new or updated systems.Build data pipelines using Databricks and ensure they adhere to best practices.Deploy necessary infrastructure using Terraform.Conduct peer reviews and seek feedback for code improvements.Testing Define test criteria in collaboration with Project Managers.Perform rigorous testing and document results comprehensively.Deployments Present developed solutions during deployment reviews.Monitor and address issues during post-deployment periods.Ensure compliance with deployment, logging, and monitoring standards.Requirements3+ Years of proficiency in PySpark and Spark SQL for data processing.3+ Years of hands-on experience with Databricks, including Unit Catalog and Delta Live Tables (DLT).Knowledge of Azure Data Lake Storage and data lifecycle management.Familiarity with orchestration tools (e.g., Apache Airflow) for ETL/ELT pipelines.Practical knowledge of data security and privacy in cloud environments.At least 1 year of experience with Terraform, adhering to GitOps best practices.The role is primarily remote, with onboarding and occasional office visits in Leiria (approximately once a month).Preferred Skills Experience with Databricks Asset Bundles.Familiarity with Kubernetes, Apache Kafka, and Vault.Personal Traits Strong adaptability to various contexts and stakeholders.Proactive ownership of deliverables.A continuous improvement mindset.Clear communication and teamwork skills.Analytical problem-solving and stress management abilities.BenefitsOpportunity to work on global projects and grow professionally.Inclusive, collaborative, and supportive work environment.Regular team events, celebrations and team building actions.Follow-ups tailored to your profile and career goals.Flex Benefits - You can choose what benefits are more suitable for you.Support for certifications and advanced learning programs to enhance your expertise.
#J-18808-Ljbffr