7/10/23 CELR Data Lake Migration Meeting notes

Dari Shirazi: --

Megan Light:

Erroll Rosser: X

Vanessa Holley : X

Teresa Jue:

Kristin Peterson : X

Brooke Beaulieu: X

Cheri Gatland-Lightener: X

Tom Russell: X

Mel Kourbage:--

Rachael Aubert

Scott Rank: X

Gretl Glick: X

Dawn Sievert

Laura Carlton: X

Patrick Caneer: X

Sam Abrams: X

Kim Del Guercio: --

Michelle Meigs

Rachael Aubert: X

Emily Augustini: X

Overall Status Updates

• Project Status:

  • CELR DL Migration to CDC Platform

  • 2 main pieces to migration:

    • 1- Capability to process data in Azure at CDC

    • 2-Integration with DEX at CDC Front Door

  • Peraton: Overview of DL Migration Activities:

    • 1-CELR Pipeline-Validation of HL7, CSV for CV19 info into merged Lab Record from both formats

      • 4 major areas: Focus on Pipeline--Key Activities

        • 1-Re-engineer CELR CSV and HL7 Pipelines from Amazon Workspace (AWS) to CDC Azure Cloud.

        • 2-Splitting pieces: Migrate Ingest, Validation, and Redact functionality for COVID-19 Lab Records submitted to CDC from CELR to Data Exchange (DEX).

        • 3-Develop transformation and lab result algorithms for Covid-19 Lab Results submitted to CDC

        • 4-Re-engineer CELR Portal within CDC Azure Cloud

    • 2: DEX Integration:

      • Consensus Technical Solution: Option 1: SDS (Send to DEX Service) / Upload API

      • Connectivity test successfully completed w/ AIMS & DEX Team

      • In Progress:

        • AIMS: Confirm & identify ongoing operations, resources, support/training for AIMS>DEX, Upload API: Tentative: Late August

        • DEX: Continue internal DEX performance/load testing: Target readiness for Mid August 2023

        • AIMS: Send CELR Production data (n = 24 hours~) to DEX STG Environment: Target: ~August 21, 2023 (will confirm dates as we assess readiness)

        • DEX can accommodate routing and validation of HL7; DEX can accommodate routing of CSV files, but cannot currently validate (dev in progress)

        • Dependency: CELR Migration to CDC Platform

DL Migration Updates: CELR

• Status Updates and Timeline:

  • Peraton Update:

    • Key Activities:

      • 1-Re-engineer CELR HL7 pipeline into Azure cloud: Currently in QA Testing

      • 2-Re-engineer CSV pipeline into Azure Cloud: Currently in QA Testing [Separate from DEX integration]

        • Migrate HL7 & CSV pipelines into DEX

          • HL7: Ingest validation, redaction--currently in CELR (Aims)--migrating into DEX platform: Completed

          • CSV: Ingest, validation, redaction: Will be migrated to DEX, dev in progress (can currently accept CSV, validation dev in progress): Timeframe: TBD (DEX evaluation for dev/ start date in progress) [70% of data for CELR is CSV uploads)

      • 3: Re-engineer portal: In QA testing for HL7 & CSV

      • 4: Generating final data set and transmission of data set from AIMS>HHS (Redshift DB): Not yet started (Implementation plan in place): Timeframe: Start July 31, 2023)--ER to provide update on status/timelines ongoing

      • 5: Implement processing pipeline in EZDX Data bricks: Includes processing of data, move into EZDX, integrate with DEX pipeline to accommodate different data formats: Dev in progress to implement change in schema: Started 7/3/23

    • Date to cutover to full production for both HL7 & CSV ON CDC: TBD--Will shift, dependent on CSV development/validation by DEX team: ~6 months/Jan 2024 (will shift dependent on ongoing development)

      • HL7 is ready

      • CSV--unknown: dependent on being able to process CSV through DEX, and process through data bricks

      • Potential to phase in HL7 while continuing to work on CSV development? Risk: 2 different sources (AIMS for CSV and CDC Platform for HL7): Complexities would be high-risk

DL Migration Updates: DAART

• Status Updates and Timeline:

  • Peraton: For DAART/Antimicrobial Reporting:

    • Current Phase: architecture/design, development yet to be started

    • Completed analysis for where/type of architecture to include in EZDX platform

    • Created tech architecture design to support AR Reporting

      • For CELR, would add on additional phase: multi-elr processing for inclusion in data bricks solution

  • CGL: DAART is priority, additional discussions on down-stream processing for readiness; Goal is to create centralized, scalable DL for ELR reporting cross program areas/reporting streams

  • DEX: There would also be a DEX component for DAART for ingest (CSV and HL7 validation)

    • ER: Would also need to discuss onboarding process for DAART

  • Timeline for DAART Migration: TBD

Questions

• CGL: Is the 6 month timeline dependent on DEX support in production? ER: Yes, partially, but also dependent on migrating Redshift data into Azure cloud and reconciliation.

  • Questions outstanding: Are we migrating historic data (potentially 4 billion records) and need to reconcile? Or are we focusing on current data set? Both options would impact timeline

• PC: Have the design phase discussions been internal to Peraton?

  • ER: Yes, internal to Peraton; wanted to ensure integration with CELR/other ELR reporting platforms; Additional discussions to be held

Next Steps & Action Items

• APHL: Provide updates on CELR/DEX integration activities as they progress forward

• Peraton: Provide updates CELR Migration/integration to EZDX/Azure/Databricks

• APHL/Peraton/CDC: Confirm DAART DL migration timeline, identify impacted processes and onboarding/communication activities