3/8/23 CELR Data Lake Migration Meeting notes

Dari Shirazi: X

Megan Light: X

Erroll Rosser: X

Vanessa Holley : X

Teresa Jue: X

Kristin Peterson : X

Brooke Beaulieu: X

Cheri Gatland-Lightener: X

Tom Russell: X

Mel Kourbage:--

Norris Kpamegan : X

Gretl Glick: X

Jonathon Tewodros: X

Welcome & Introduction

Overall Status Updates

• Project Status:

  • CDC: Previously discussed: Few components

  • 2 main pieces to migration:

    • 1- Capability to process data in Azure at CDC

    • 2-Integration with DEX at CDC Front Door

  • Both happening in parallel, but different dev initiatives (may be helpful to bring on DEX team for future meetings)

  • Peraton: :

    • 1-CELR Pipeline-Validation of HL7, CSV for CV19 info into merged Lab Record from both formats

      • 4 major areas: Focus on Pipeline--Key Activities

      • 1-Re-engineer CELR CSV and HL7 Pipelines from Amazon Workspace (AWS) to CDC Azure Cloud.

      • 2-Splitting pieces: Migrate Ingest, Validation, and Redact functionality for COVID-19 Lab Records submitted to CDC from CELR to Data Exchange (DEX).

      • 3-Develop transformation and lab result algorithms for Covid-19 Lab Results submitted to CDC

      • 4-Re-engineer CELR Portal within CDC Azure Cloud

      • CDC EOY: June 30, 2023 Target end of fiscal year

    • Dependencies:

      • DEX--must have ability to ingest data from AIMS, AND storing/receiving in consistent format in how current CV19 Data is ingested

      • Pilot/test timeframes within DEX Team will hopefully be shared (quarterly review week of 3/16)

      • Dex-focus has been on HL7 Pipeline dev, has not yet focused on CSV Dev pipeline (bulk of data for CELR is submitted via CSV currently)

    • 2-Data sent to Redshift, data management processing, de-duping, data mapping, cleaning; data is sent and exported to HHS Protect daily

• 2-Splitting pieces: Migrate Ingest, Validation, and Redact functionality for COVID-19 Lab Records submitted to CDC from CELR to Data Exchange (DEX).

  • Ready for internal testing by June 30th 2023

  • Dependencies long-term:

    • AIMS to DEX connectivity, what data will need to be received in which environment, transport protocols, pilot testing << these should come through to DEX Team

    • Conversations within Peraton and DEX team in progress, need to loop in AIMS/APHL for engagement/identifying technical solutions

    • 1 example for receiving data from AIMS TO DEX for EIP through Azure

• Additional dependencies:

  • Merged lab record sent to RedShift --what does that format look like? What is the process for creating the merged lab record? How will this impact the data processing and migrating it to DEX? If the data is processed and provided to Redshift/timeframe--

  • APHL: Comparing Redshift to Azure will be complicated based by timing of data processing; columns added to data, may not be present in Redshift--due to historic differences. Re-ingesting ALL CSV s and HL7 will be different --will have cleaning needs and will have all the columns in it, comparison of data in Redshift to Azure will be challenging from a historic perspective.

  • Peraton: Testing plans will need to accommodate these historic data changes and prepare for comparison/assessment

CELR Integration

• CELR integration in Azure

Questions

• APHL: Is 6/30 target date the expected date for go-live or piloting/testing?

  • Peraton: 6/30 is the target for when the pipelines are ready for testing (this is not the expected timeline for being live for data cutover)--June 30th is the expected target for INTERNAL Testing of pipelines for HL7 and CSV; different timeline for DEX for integration testing (DEX is key dependency for when the CSV & HL7 pipelines would be ready for integration testing)

    • This would be testing with TEST Data, not production data; test data will need to be developed/re-purposed/created by Peraton--conversations will need to be had with DEX team as to what data will be used

  • APHL: Will need to coordinate with DEX team as to timelines to provide integration testing

  • CDC: To clarify, these are independent dev iniatives/timelines? Peraton: By time DEX is ready, should be ready for E2E Testing

  • CDC: Conversations around CELR, DEX--DEX is also focused on broader lab data flows, for multiple use cases (such as ARLN/DAART)

  • Peraton: Once data from CELR would flow through Dex, that would be 1 use case, AR would be 2nd use case…Would recommend common pipeline for DEX, but being able to identify a common lab schema and working with the labs to send a common lab data format

  • Increase Meeting frequency?

  • AIMS: S3 bucket would be easiest/simplest transport for both teams, should be easiest/lowest LOE

Next Steps & Action Items

• Include DEX Team to discuss statuses, timelines on next monthly meeting

• Interim meeting: Schedule meeting with DEX & AIMS team to discuss provisioning S3 Bucket Gretl Glick to send AIMS contact for meeting to Erroll Rosser

  • March 23 @3pmET tentative meeting timeslot for AIMS & DEX team