6/29/23 CELR Data Lake Migration Meeting notes

Dari Shirazi: X

Megan Light:

Erroll Rosser: X

Vanessa Holley : X

Teresa Jue: --

Kristin Peterson :

Gretl Glick: X

Cheri Gatland-Lightener: X

Tom Russell: --

Geo Miller: X

Norris Kpamegan :

Marcelo Caldas:

Alissa McShane: X

Ryan Harrison: X

Don Lindsay:

Marion Anandappa: X

Leslyn Mcnabb:

Overall Status Updates

• Reviewing AIMS connection to DEX options: Research/Analysis: In Progress

  • AIMS/APHL Updates:

  • 6/29/29: AIMS: Successfully Tested option 1 (send to dex with upload api). Option 4 (receive from aims) would be the nearly same code with 2 changes

    1. Option 1: SDS (Send to DEX Service) / Upload API

       1. Tested connectivity successfully

       2. Prototyped runtime using Docker image & lambda using aims sandbox

       3. Next:

          1. Determine what a load/performance test should look like & perform the test

          2. Determine if we want the runtime to be in Lambda, Mirth, Fargate, or k8s

    2. Option 4: RAS (Receive from AIMS Service)

       1. With a slight modification, the SDS can run as a poller

       2. Have built similar poller services many times, can be done with this for performance testing in a few days if there is interest

       3. Would run on the DEX/Azure side to poll SQS and get data from AIMS S3

  • Next step: AIMS/CDC to confirm decision as to whether to move forward with Option 1 or Option 4

Mtg Notes:

• Notes:

  • AIMS/CDC to confirm decision as to whether to move forward with Option 1 or Option 4

  • Consensus Decision: Option 1: SDS (Send to DEX Service) / Upload API

  • Next steps:

    • AIMS: Discussed Option 1: Ongoing resources, training, monitoring needed for long-term support

      • GM: Tested upload API successfully, documenting requirements, training, prototype of code to integrate with AIMS, polling with SQSQ, and sending objects to S3

      • Code base: Similar for Option 1 and 4

      • AIMS: Received funding for ongoing support for CELR

      • Next steps: Determine operational support, monitoring, security, run service, etc., plan for ongoing operations

    • Timeline: Prod: HL7 sent to DEX: TBD for CELR

      • Dependency: CELR Migration to CDC Platform

      • AIMS: Both CSV and HL7 messages sent to CELR

  • Question:

    • EIP+M blocker/connection with CELR? EIP+M and CELR are completely independent

    • CDC: Can we queue CELR for production cutover in Q4 (October 2023)?

      • AIMS: Dependent on CELR migration to CDC Platform

      • Peraton: Outside of DEX scope, working with Marion to migrate CELR Pipelines to move into CDC Azure --In Progress;

        • Dependency: A- getting prod data from AIMS to DEX, B-Getting data from DEX to EZDX, c-Analytic Pipeline/data in redshift and moving that data into cdc cloud (that is routed to HHS)

        • Focus for Dex is Currently A

        • AIMS: Regardless of whether CELR has migrated, is the request to route all prod data for CELR to DEX (even if DL has not yet migrated to CDC)?

        • RH: Geo had sent test data, so the question is for prod data--when can this data be expected?

        • ER: If prod data is routed to DEX, would also need able to process CELR data; we can confirm connectivity Q3 with DEX, and that data can be used to test processing pipelines for CELR migration, and can also test analytic pipelines

        • RH: Any additional steps needed from AIMS for confirming connectivity? ER: If AIMS has sent data over, we can confirm data can be routed/processed successfully; would like to do e-2-e test

      • GM: E2E test: What is the testing criteria? Has sent CELR Test files, but has not done load testing with associated meta-data for HL7; identify meta-data values for E2E test

    • RH:

      • Q3: Load test, meta-data

        • If CELR Volume is sent to staging environment currently, would likely fail; team is currently conducting internal DEX performance/load testing, would want to then do load testing with CELR volume

          • Validation of meta-data: Not a concern from DEX, but would create new label for AIMS-CELR, and then would route based on label

      • Q4--Oct 2023: Prod data (dependency: CELR DL needs to migrate to CDC

      • Action Item (long-term): Short-term: Confirm Connectivity with AIMS/DEX using API; long-term: Peraton to conduct full E2E with Meta Data and successfully process through all pipelines

  • It would be helpful to know size of avg message/batch. I know HL7 messages are small (KB). What are size of CSV? Does it just stream in or still batch? What size GB?

    • Action Item: Geo to confirm volume for CELR; 1-Max flow during pandemic; 2-current flow --concurrency and file size

      • ER: Volume has drastically reduced, given the end of the emergency, jurisdictions are ending sending covid data

  • GM: Testing: Need representative CELR data (both CSV and HL7) to send to staging:

    • AIMS: May be helpful to send 1 days worth of data to DEX; Is STG environment HIPAA-Compliant, Secure for receiving production data?

      • DEX: Short answer is yes; PRD & STG are both secure environments for PHI data

      • GM: Has STG Environment; will route data from CELR ONB (PRD) to DEX STG

      • DEX: Timeline? Would like DEX to complete internal load testing before CELR load testing is sent to DEX STG: 4-6 weeks to complete internal load testing (mid-August ~): Soft Constraint

      • Action Item: AIMS to discuss internally-- Confirm & identify ongoing operations, resources, support/training: Tentative: Late August

• DEX: JUL-SEP (Current Q): Moving from Alpha to Beta Maturity, scalability, performance, high-availability (closed beta-auto-scaling, but not multi-region)

  • Plan: Announce on OCT 1 that HL7 and validation product are closed-beta; use cases ready for routing (including CELR); will determine titrate/flow of use cases until ready to accept all data/use cases

• ER: How does CSV data transmission fit into DEX?

  • DEX: Short answer is that nothing you can use in production this quarter; CSV can be sent, but there would be no validation; a little bit of observability

    • Future plan: CSV validation will be done in Accenture environment; request has been put forth to transform data being sent in MMG into CSV--this is out of scope for DEX (would need to write a service, Rule-level validator similar to IGAMT; )

    • ER: Validation for CSV is not as critical for HL7 for CELR

• Action items: Ryan to share the CSTE Slidedeck (publicly available)

• Next Meeting: July 27, 2023

DRAFT Data Flow Diagram

Questions

Next Steps & Action Items

• Next steps:

  • @Geoffery Miller @dari.shirazi@aphl.org AIMS: Confirm & identify ongoing operations, resources, support/training for AIMS>DEX, Upload API: Tentative: Late August

  • @Geoffery Miller Confirm volume for CELR; Provide DEX Team overview of (1) Max flow during pandemic; (2) current flow, include concurrency and file size metrics

  • Ryan Harrison: Will share the DEX Slide deck presented at CSTE (publicly available)

  • DEX: Continue internal DEX performance/load testing: Target readiness for Mid August 2023

  • AIMS: Send CELR Production data (n = 24 hours~) to DEX STG Environment: Target: ~August 21, 2023 (will confirm dates as we assess readiness)

  • @Gretl Glick Schedule Checkpoint Mtg: July 27, 2023 from 2-3pm