5/11/23 CELR Data Lake Migration Meeting notes

Date

11 May 2023

Attendees:

APHL	CDC	Peraton
Dari Shirazi:	Megan Light:	Erroll Rosser:
Vanessa Holley :	Teresa Jue:	Kristin Peterson :
Brooke Beaulieu:	Cheri Gatland-Lightener:	Tom Russell:
Mel Kourbage:	Norris Kpamegan :	Marcelo Caldas:
Gretl Glick:		Don Lindsay:

Goals

Update on Data Lake Migration Status
DEX Overview and Status Update

Discussion topics

Item	Notes
Welcome & Introduction
Overall Status Updates	DEX Development, Status Updates, and Timeline: Reviewing AIMS connection to DEX options: Research/Analysis: In Progress Upload API - AIMS has reviewed docs, but have not yet requested to be added to the DEX activity. Pro: resumable technology looks like it solves a the problem of supporting large file uploads to web apis. Con: is that it would be a client agent within AIMS that would need continual maintenance and also the ability to track success/failure (queue/deadletter) and team prepared to debug if certificates, credentials, etc change. AWS data sync to azure blob storage - Background: the integration for azure blob storage is notably a “preview” feature on AWS’s side. Pro: job would be configured on each side with no code to maintain. Con: A downside is that this is a job that has to run on some cron frequency - so data flow will be in chunks as opposed to a “realtime” stream. Alternative research: potential other project’s integration into azure using Azure Connectors for SQS and S3 to see if it would apply to this problem. This would match how AIMS already internally handles event driven processing & would inherit what the team already knows how to use. Need to investigate the effort on the Azure side to make configurations and/or services to use these connectors.
Questions
Next Steps & Action Items	Previous Action Items-4/27/23: Dari/aims to research S3 to Azure mechanism, determine options as to what is possible: In progress GM: Validate both Azure and data api upload documentation: In progress ~~RH: Will share Data API documentation: Complete~~ ~~Gretl to schedule meeting for same timeslot, 2 weeks--5/11: Complete~~

Action items

Quick decisions not requiring context or tracking

For quick, smaller decisions that do not require extra context or formal tracking, use the “Add a decision…” function here.

Decisions requiring context or tracking

For decisions that require more context (e.g., documentation of discussion, options considered) and/or tracking, use the decision template to capture more information.