stable_2021-03-11

Generic datamodel/etl changes

• fixed various issues with data quality causing it not to run in each hospital
• ETL should run more efficiently in most places due to additional stastistics optimization
• elasticsearch patient ID fetching phase should go considerably faster with less CPU/memory usage in most cases
• ETL now only runs if all migrations have run
• data quality now only runs if the previous ETL succeeded
• the app<>dwh right to be forgotten integration now uses safer encryption protocols
• hospitals in which the CTcue databases reside on a SQL server 2017+ instance may see a considerably performance change due to usage of the built-in string_agg aggregation function

EHR-specific datamodel/etl changes

hix

• in order to take less recent row changes into consideration, the following categories' extraction strategy has changed:
  • care_activity is now fully loaded during every incremental update
  • form source table VRLIJST_VROPSLG now has an incremental time window of six months (up from one month)
• steps are taken during ETL to remove more fake/test patients during ETL than before
• new forms:
  • Trials: this form allows filtering on patients who are already enrolled in a trial
  • Vrijheidsbeperkende interventies: a mental health care form
• a preliminary PDF conversion module exists. Usage is on request and in small increments to test the limits of the system.

clarity

In order to take less recent row changes into consideration, the following categories/tables are now fully loaded at every incremental update

• report (source tables HNO_INFO, ORDER_IMPRESSION, ORDER_NARRATIVE, ORDER_RES_COMP_CMT)
• observation
• order
• measurement

nexus

Nexus measurement ETL module will run considerably faster

CTcue application integrations

The following table-valued function call returns all possible original pseudo_ids of a specified pseudo_id: select original_pseudo_id from dbo.get_original_pseudo_ids(<pseudo_id>).