6/1/23 CELR Data Lake Migration Meeting notes

Dari Shirazi:

Megan Light:

Erroll Rosser:

Vanessa Holley :

Teresa Jue: --

Kristin Peterson :

Brooke Beaulieu:

Cheri Gatland-Lightener:

Tom Russell: --

Mel Kourbage:

Norris Kpamegan :

Marcelo Caldas:

Gretl Glick:

Ryan Harrison:

Don Lindsay:

Geo Miller:

Alissa McShane:

Overall Status Updates

• Reviewing AIMS connection to DEX options: Research/Analysis: In Progress

  • AIMS/APHL Updates:

    1. SDS (Send to DEX Servce) / Upload API

       1. Tested connectivity successfully

       2. Prototyped runtime using Docker image & lambda using aims sandbox

       3. Next:

          1. Determine what a load/performance test should look like & perform the test

          2. Determine if we want the runtime to be in Lambda, Mirth, Fargate, or k8s

    2. AWS DataSync

       1. Prerequisite for transfer test: EC2 instance to run data sync agent - did not complete this

       2. Tested connectivity to S3 and Azure locations

       3. Further research/additional information needed regarding cost / architecture questions for scaling this solution long term

    3. Azure Logic Apps / Connectors

       1. Further research/additional information needed regarding cost implications in azure to run

       2. Need help understanding continued monitoring & maintenance of a logic app

       3. https://learn.microsoft.com/en-us/microsoft-365/community/power-automate-vs-logic-apps

    4. RAS (Receive from AIMS Service)

       1. With a slight modification, the SDS can run as a poller

       2. Have built similar poller services many times, can be done with this for performance testing in a few days if there is interest

       3. Would run on the DEX/Azure side to poll SQS and get data from AIMS S3

Mtg Notes:

DRAFT Data Flow Diagram

Questions

Next Steps & Action Items

• Next steps: