Senior Big Data Python Engineer Let'S Make Things Fly! Lisbon

Let's make things Fly!Your Role & Responsibility: Data Transformation and Processing: Extract, transform, and load (ETL) Flight Data Monitoring data using Spark and PySpark, applying relevant algorithms based on engineering requirements to ensure accuracy and consistency.Algorithm Implementation: Develop, code, and test PySpark algorithms that meet specific definitions and requirements provided by aircraft engineers, addressing use cases such as anomaly detection, event recognition, and trend analysis in flight data.Timeseries Data Management: Structure and optimize large time-series datasets in PostgreSQL Timeseries, with a focus on efficient storage, retrieval, and processing for real-time insights.Platform Management: Maintain data processing pipelines within Azure, ensuring scalability, performance, and data security for big data workloads.API and Data Delivery: Publish processed data via API or other methods to ensure seamless integration with frontend applications, enabling real-time data access.Monitoring and Optimization: Continuously monitor data pipelines and workflows for performance and reliability, optimizing code and infrastructure to meet SLAs and operational goals.Collaboration with Aircraft Engineers: Work directly with aircraft engineers to translate their data analysis needs and specifications into PySpark algorithms, ensuring that data processing outputs align with technical and operational requirements.Documentation and Best Practices: Document data transformation processes, maintain code quality standards, and contribute to best practices in big data engineering.

#J-18808-Ljbffr