5/29/24 CELR/DEX Prod Migration Planning Discussion

Geo Miller: X

Erroll Rosser:

Anthony Kimbo:

Gretl Glick: X

Scott Rank

Loretta Foster:

Cosmin:

Chase Farmer: X

Beth Wingerd:

5/29/24

• Weekly recap

  • All systems have been a go, until yesterday

  • Files have been going well--no cause for alarm for past few weeks

  • New developments

    • DEX attempted prod release yesterday, but needed to do rollback due to network issues

    • Will try to do another prod release tomorrow

    • may impact go/no go

    • Ever since Friday, have been seeing interesting counts

    • AIMS has been getting more files than it appears what has been uploaded to DEX

    • Missing files in DEX API; files seem to come in, track through post processing, then they disappear

      • Can see the files arrived at DEX, but

    • AIMS says they sent 100 files, but dex only received 50 files,

    • So dex is dropping files?

    • Info end point is sending success

    • DEX HAS Received file, but file is not triggering post processing--disappears at post processing

    • AIMS can investigate file ids, can re-play if needed for certain time period

    • If we re-play them, then we determine if some weird blip in Azure; Dex can replay them internally

  • AIMS Update:

    • CC to deploy 1.0v

    • Alert monitoring--will change to red if any outbound messages fail and all re-ries fail, DLQ alarms will be triggered

    • With release, also started sending rabies data to DEX STG for evaluation, with understanding priority is Covid ELR data

    • Working on moving development environment to AIMS SB--targeting on June 10th, will be working on EICR

    • Submitted CC request for AIMS to DEX prod, to be implemented on June 3; can delay based on go-no go

Dev Updates:

• Current DEX>STG workstream will be retired, migrated to Meta Data V2 and verify end point 5/22

  • 5/22-5/29: Testing of meta data v2, end point in STG, then

  • Remaining AIMS development: Cloudwatch monitoring and alert creation: Target: 5/24/24

  • 5/28/24: AIMS will be submitting CR/CAB, then migrate CELR Reporting to DEX Prod on 5/29 (or date to be chosen)

  • Would be migrated to PRD by last week of May

  • Could also potentially migrate Rabies to STG for UAT

  • Production Ops Team:

    • If DEX sees data related issues, would open HD ticket, for triaging, can pull into integration teams

    • Send email to support@aimsplatform.org (@Gretl Glick to send this over to Anthony)

      • Geo: For Instance, known issues seen--send email to DEX listserve [dexuploadapi@cdc.gov], but will also open a ticket on AIMS for tracking/time spent

      • Chase: If a file was sent to the wrong location, that might not be an issue

      • General communications/non-incident tickets--can be sent to Geo directly/emailed to team

      • If issue is with DEX, issues will be submitted to dexuploadapi@cdc.gov to communicate with DEX team and associated service desk/ticketing system

Production Readiness Review and Migration Steps

• Consensus Production Cutover Date: 6/3/24!

• REVISION 3TIMELINE

  • - 5/15 - v2 CELR Routing updates to staging

  • - 5/15 - CELR ready for v2 in staging

  • - 5/22 - APHL deploys /info endpoint check and v2 to staging

  • - 5/22 - 5/28 - APHL | DEX | CELR End to End verification in staging

  • - 5/28 - DEX outage comms - w/o - v2.0.X CR Approval

  • - 5/28 - APHL change control request

  • - 5/28 - v2 CELR Routing updates to production

  • - 5/28 - CELR ready for v2 in production

  • - 5/30 - proposed Upload v2.0.x production release

  • - 5/31 - GO/NO GO DECISION

  • - 6/3 - pre-cutover meeting

  • - 6/3 - APHL proposed cutover

  • - Production monitoring - APHL | DEX

• Checkpoints during the production migration

  • Go/no go Meeting : 5/31 [Anthony to schedule]

  • Production Migration Working Session: 6/3 (Geo OOO--wouter can join) [Anthony to schedule]

    • Tuesday, 6/4: 10:30am ET DL O&M Meeting [Gretl will forward meeting invite]

    • Wednesday 6/5: 11am ET CELR>DEX Meeting

    • Thursday 6/6: 8:30am ET DL O&M Meeting [Gretl will forward meeting invite]

    • Friday 6/7: To be Scheduled [Anthony will send meeting invite

Next Steps/Action Items

Schedule Production Migration Meeting Series:

• Checkpoints during the production migration

  • Go/no go Meeting : 5/31 [Anthony to schedule]

  • Production Migration Working Session: 6/3 (Geo OOO--wouter can join) [Anthony to schedule]

    • Tuesday, 6/4: 10:30am ET DL O&M Meeting [Gretl will forward meeting invite]

    • Wednesday 6/5: 11am ET CELR>DEX Meeting

    • Thursday 6/6: 8:30am ET DL O&M Meeting [Gretl will forward meeting invite]

    • Friday 6/7: To be Scheduled [Anthony will send meeting invite