We are seeking a highly skilled and experienced Senior Data Engineer to join our client team.
The ideal candidate will play a key role in designing, building, and maintaining scalable and efficient data processing pipelines, as well as supporting the deployment of advanced data solutions.
ResponsibilitiesPipeline Development: Design, develop, and maintain scalable data processing pipelines for handling large volumes of structured and unstructured data in a distributed environment.Microservices Development: Build and maintain production-ready microservices in Python to serve data and features at scale.Tooling and Automation: Create and enhance internal tools to support CI/CD pipelines, experiment tracking, and data versioning workflows.Data Quality Assurance: Ensure data integrity and accuracy by implementing robust data quality processes and monitoring solutions.System Design: Architect and optimize distributed software systems to meet high performance, scalability, and reliability requirements.Collaboration: Work closely with cross-functional teams, including data scientists, software engineers, and product managers, to deliver data solutions that drive business insights.QualificationsExtensive experience in software development with Python in high-performance, large-scale production environments.Hands-on expertise with Apache Spark and PySpark for distributed data processing.Strong knowledge of data modeling and handling both structured and unstructured data.Significant experience working with Apache Kafka for real-time data streaming and messaging.Proven experience with cloud environments (e.g., AWS, GCP, Azure).Expertise in designing and implementing distributed software systems.Strong problem-solving abilities and excellent communication skills.Nice-to-Have SkillsExperience with the Databricks Platform.Familiarity with streaming data and frameworks for real-time processing (e.g., Flink, Spark Streaming).Knowledge of NoSQL databases, such as Redis and Neo4j.Understanding of ML algorithms and the ML lifecycle.Area: Data and BI
Location: Lisboa
Contract Type: FULL_TIME
Specialization: Tecnologias de informação
Industry: TI
Salary: Negotiable
Work Type: Híbrido
Experience Level: Gerente
Job Reference: 2310342/001
Posted Date: 25 de Novembro de 2024
Consultant: Cátia Carlos
#J-18808-Ljbffr