2x Faster Deployments with Real-Time ETL Pipeline and Predictive Healthcare Models

Project Type

Advanced Data Engineering
 ETL Process Optimization
Predictive Analytics Development

Overview

We optimised the ETL pipelines for a major healthcare provider, speeding up insights and improving the accuracy of predictive analytics for patient outcomes. By redesigning the data pipeline, we increased dashboard query performance by 7X. This allowed for better health monitoring, leading to improved patient care and operational efficiency. It also saved costs by streamlining data processes, reducing errors, and minimising delays in critical data availability, which are crucial for timely patient treatment decisions.

Business Challenge

The customer is a leading healthcare provider with thousands of medical devices installed across hospitals and clinics. These devices automate essential processes such as patient monitoring and medical testing, generating data on metrics like heart rate and blood pressure fluctuations, which are captured by sensors and collected on a centralised IoT server. The customer utilises an in-house developed dashboard to monitor this data and predict potential equipment malfunctions, patient health deteriorations, and other critical incidents. However, the dashboard is slow, not scalable, and depends on manual data updates every 2-3 days, resulting in significant delays in generating insights and predicting health-related issues.

The client wanted to enable

  • Central monitoring and diagnosis of medical equipment across various healthcare facilities while ensuring the cost of integrating more devices remains low and consistent.
  • A solution to more accurately forecast potential downtimes and malfunctions of medical equipment in a cost-effective manner, potentially saving millions of dollars by preventing medical errors and unnecessary procedures.
  • Near real-time updates to their dashboards for better operational visibility and more responsive patient care.

    • Accropolix’s Solution

    • We built an ETL data pipeline that improved historical insights from IoT-equipped medical devices, used by QA analysts, healthcare administrators, and the procurement team.
    • To enable faster query processing times, we developed a Spark-based data lake along with a high-speed dashboard.
    • We utilised open-source technologies to seamlessly integrate and scale the predictive analytics and forecasting components within the new solution, while also automating the deployment process.
    • A centralised IoT MQTT Broker collected healthcare-related IoT data, such as patient telemetry and fault data from sensors across healthcare facilities.
    • The raw data was enriched, processed, and harmonised before being stored in an S3 data lake, which served as the backbone for monitoring and analytical operations.
    • Data was routed through several Airflow and PySpark-based pipelines, which processed the data to generate feature tables displayed in the dashboard's user interface.
    • These pipelines utilised machine learning scripts to predict and forecast health-related events and equipment performance.
    • Dashboards were updated using SQL-based queries powered by tools such as Trino, Databricks, and Delta Lake.
    • We designed appropriate tables based on data from various pipelines, with the flexibility to modify them to meet evolving healthcare requirements.
    • The data was further processed through a specialised healthcare data platform using tools like Amazon HealthLake, enabling the development of predictive machine learning models.

      • Migration Strategy Plan

      Business Outcomes

      Accropolix developed a highly efficient and almost instant ETL data pipeline for the healthcare provider that speed up query processing times and increased the system's ability to handle more data. This setup improved the way dashboards are monitored, making it quicker to spot and fix potential issues. The entire system uses infrastructure as code, which helps speed up the introduction of new updates and features. Moreover, the approach helps keep costs low when new features are added.

      1. 7x improvement in query performance

      2. 3 days to 1 hour reduction in data update speed

      3. 2x faster deployment time for new features

      About Accropolix

      Accropolix specialises in predictive modelling and ETL pipelines, enhancing healthcare efficiency through optimised data processing and real-time alerts for improved patient care outcomes.

      Begin Your Project!

      Reach out to us at vivek@accropolix.ai to create strong and scalable data pipelines.