Copy of 6/1/23 CELR Data Lake Migration Meeting notes

Date

01 Jun 2023

Attendees:

APHL	CDC	Peraton
Dari Shirazi:	Megan Light:	Erroll Rosser:
Vanessa Holley :	Teresa Jue: --	Kristin Peterson :
Brooke Beaulieu:	Cheri Gatland-Lightener:	Tom Russell: --
Mel Kourbage:	Norris Kpamegan :	Marcelo Caldas:
Gretl Glick:	Ryan Harrison:	Don Lindsay:
Geo Miller:
Alissa McShane:

Goals

Update on Data Lake Migration Status
DEX Overview and Status Update

Discussion topics

Item	Notes
Overall Status Updates	DEX Development, Status Updates, and Timeline: Upload API: Working ON v1.1-In progress, status of upload V1.1.2-Will work on authenticating via SAMS SSO Milestones on progress Reviewing AIMS connection to DEX options: Research/Analysis: In Progress Upload API - AIMS has reviewed docs, but have not yet requested to be added to the DEX activity. Pro: resumable technology looks like it solves a the problem of supporting large file uploads to web apis. Con: is that it would be a client agent within AIMS that would need continual maintenance and also the ability to track success/failure (queue/dead letter) and team prepared to debug if certificates, credentials, etc change. AWS data sync to azure blob storage - Background: the integration for azure blob storage is notably a “preview” feature on AWS’s side. Pro: job would be configured on each side with no code to maintain. Con: A downside is that this is a job that has to run on some cron frequency - so data flow will be in chunks as opposed to a “real time” stream. Alternative research: potential other project’s integration into azure using Azure Connectors for SQS and S3 to see if it would apply to this problem. This would match how AIMS already internally handles event driven processing & would inherit what the team already knows how to use. Geo is working on proto-typing this option and will have an update next time we meet Need to investigate the effort on the Azure side to make configurations and/or services to use these connectors. Research in progress, AIMS has not requested access Other Options: Will continue to research
Mtg Notes:
Questions
Next Steps & Action Items	Next steps:

Action items

Continue to research options, decide on option, AIMS will create data flow diagram Geoffery Miller
Provide Geo with paid Azure subscription for prototype/testing dari.shirazi@aphl.org or Alissa McShane
Gretl Glick Next Meeting: Schedule: June 1, 2023 at 1pm ET

Quick decisions not requiring context or tracking

For quick, smaller decisions that do not require extra context or formal tracking, use the “Add a decision…” function here.

Decisions requiring context or tracking

For decisions that require more context (e.g., documentation of discussion, options considered) and/or tracking, use the decision template to capture more information.