Automation of the extensive raw sensory data from the ships of a large shipbuilding company.
Services
-
Engineering
-
Training
-
Management
Our role
Processing the output of 250+ sensors of a single ship demands a solid solution, even more if additional insights for the company and customers need to be presented in a single dashboard.
For a Dutch shipbuilding company with a large fleet of boats, tugs, vessels, cargo ships and ferries we processed the data related to fuel efficiency, emissions and energy consumption for electric vehicles.
The challenge
Over the years, the client has collected the growing wave of ship data with various systems and created many implementations, which differed in form but not in function. On top of that, iterations often led to breakdowns of the code. One of the reasons was that the team was not used to working with cloud products which only led to more inefficiency.
We were asked to align this incoming stream of data and educate the team to work with the new cloud products.
The solution
After a period of intensive developing and validating we were able to implement several products successfully that were verified with intensive test-cases. Last but certainly not least we trained the data team in using Microsoft Azure Synapse and taught them to use tools such as PySpark to write efficient Spark code.
- A generator to scale the deployment of pipelines, triggers and notebooks in Azure Synapse based on a configuration file.
- Runtime has been moved to Azure Synapse and translated to PySpark to utilize the computational power of the Spark Cluster to improve the execution speed of the codebase.
- A data processing pipeline to aggregate and added business logic to the raw sensor data.
- Functionality is packaged into a python package and released to Azure Synapse through automatic release pipelines.
The outcome
We successfully implemented a comprehensive, sustainable, and future-proof data platform solution for data processing. This solution significantly improved code quality and provided a better developer experience for the team. It was not a mere quick fix but a complete overhaul, addressing both technical and human aspects of the implementation.
One noteworthy achievement is the development of an automatic vessel data refresh feature for customer dashboards. This innovation has significantly reduced the maintenance effort required for existing dashboards. As a result, onboarding new customers and deploying existing implementations based on configuration has become much faster, reducing the time to value to just half a day.