Scientific Registry of Transplant Recipients Secure Site

How were organ procurement organization (OPO) performance metrics altered in response to COVID-19 for the reports released in January 2021?

For the reports released in January 2021, all risk-adjusted OPO performance metrics (eligible death donation rate, deceased donor organ yield) did not include donors after March 12, 2020, the day prior to the declaration of a national public health emergency on March 13, 2020.

 

How are the reports changing for the reports to be released in July 2021?

Risk-adjusted transplant program performance metrics will be altered in the following ways:

Waitlist Mortality Rate:

Kidney and Lung Candidates: Candidates on the waitlist 1/1/2019-3/12/2020 will be evaluated.

Liver, Heart, Pancreas, and Intestine Candidates: Candidates on the waitlist 1/1/2019-12/31/2020 will be evaluated

Transplant Rate:

The first quarter following declaration of a national emergency will be carved out of the transplant rate evaluations for all organ types. Candidates on the waitlist 1/1/2019-3/12/2020 and 6/13/2020-12/31/2020 will be evaluated. Waitlist time between March 13, 2020 and June 12, 2020 will not be evaluated.

Offer Acceptance Rate:

These evaluations will return to normal reporting cohorts. Offers received 1/1/2020-12/31/2020 will be evaluated.

Overall Mortality Following Listing:

Patient follow-up will continue to be truncated on 3/12/2020. The evaluation period will be: 1/1/2019-3/12/2020.

1-month and 1-year Posttransplant Graft and Patient Survival:

Evaluations cohorts will continue to exclude transplants and follow-up time beyond March 12, 2020. Transplants performed 1/1/2018-3/12/2020 will be evaluated with follow-up only evaluated through 3/12/2020.

3-year Patient and Graft Survival:

Evaluations cohorts will continue to exclude transplants and follow-up time beyond March 12, 2020. Transplants performed 1/1/2015-12/31/2017 will be evaluated with follow-up only evaluated through 3/12/2020.

Risk-adjusted OPO performance metrics will be altered in the following ways:

Eligible Death Conversion Rate:

Eligible deaths during the first quarter of the national emergency will be excluded from evaluation. Eligible deaths 1/1/2020-3/12/2020 and 6/13/2020-12/31/2020 will be evaluated.

Deceased Donor Organ Yield:

Deceased donors during the first quarter of the national emergency will be excluded from evaluation. Deceased donors 1/1/2019-3/12/2020 and 6/13/2020-12/31/2020 will be evaluated.

 

Who made these decisions regarding changes to the performance metrics for the January and July 2021 reports?

The SRTR Review Committee (SRC) reviewed data about the potential impact of the pandemic on the performance metrics at their meetings on July 7, 2020, September 2, 2020, January 20, 2021, and February 9, 2021, making recommendations to the SRTR and HRSA. HRSA subsequently approved the SRC’s recommendations.

 

What did the SRTR Review Committee (SRC) consider when making these recommendations?

The COVID-19 pandemic impacted the nation’s transplant system and healthcare system more broadly in ways that were unprecedented. The SRTR maintains a public web-based application detailing the impact of the pandemic on various aspects of transplantation. The committee’s primary concern was whether the performance metrics published by the SRTR could be systematically biased as a result of the pandemic, given the pandemic affected portions of the country differently at different times. The committee reviewed trends in performance metrics by month leading up to and following the declaration of a national public health emergency as well as assessments of geographic variability in the effect of the pandemic during the window of time being evaluated by each performance metric. For the January 2021 reports, the committee felt there was not yet enough information to conclude the risk of bias would be minimal, and therefore made the recommendation to censor all performance metrics for transplant programs and OPOs at the start of the national emergency. The committee reviewed updated data in January and February of 2021 and concluded some metrics could resume full reporting, some should eliminate the early period of the pandemic, and some should continue to truncated on March 12, 2020 as detailed above. Recommendations to resume reporting, where made, were made after concluding that the risk of bias due to differential effects of the pandemic based on geography or time were minimal.

 

I see data in the January 2021 SRTR reports after March 12, 2020, but I thought all reporting was stopped on March 12, 2020. Why is that?

Modifications to the reports were only made for risk-adjusted performance metrics. There are many areas of the report that simply report on the state of transplantation with unadjusted data. Because it continues to be valuable to report on the state of transplantation, reporting of unadjusted trends and counts continued. The risk-adjusted performance metrics, because they are designed to compare how a program or OPO is doing relative to expectation, were altered in response to the pandemic.

 

How were transplant program performance metrics altered in response to COVID-19 for the reports released in January 2021?

For the reports released in January 2021, all risk-adjusted transplant performance metrics (transplant rate, waitlist mortality rate, offer acceptance rate, overall mortality following listing, posttransplant outcomes) did not include data after March 12, 2020, the day prior to the declaration of a national public health emergency on March 13, 2020. Follow-up for all risk-adjusted outcomes of waitlist candidates and transplant recipients was stopped (i.e., statistically censored) on March 12, 2020.

 

Will the metrics be altered again in the future?

Yes. The SRTR Review Committee (SRC) will continue to evaluate the data and make recommendations to the SRTR and HRSA about how the performance metrics will be handled beyond the July 2021 reports.

 

Where can I go if I still have questions?

Please email us at SRTR@SRTR.org.

When are CUSUMs updated?

On the first day of each month.

 

Where do I find the CUSUMs?

Go to “Reports” on the nav-bar across the top. On the reports page, a menu will appear on the left. Select “CUSUM,” which takes you to the CUSUM page.

 

What is CUSUM?

CUSUM (short for "cumulative sum") is a quality-control method used in statistical analysis. CUSUM charting techniques are used to detect a change in a process. As used by SRTR, a CUSUM chart is constructed by plotting an OPO’s yield over time. The chart consists of a line with a point plotted for each calendar day. The CUSUM line goes down on days when the OPO recovered a donor whose organs were transplanted. Conversely, the line goes up on days when the OPO recovered a donor whose organs were not transplanted. Finally, the CUSUM line is flat on days when the OPO recovered no donors. If organ yield at the OPO is at a rate lower than expected, the line on the chart goes up over time. Thresholds can be set to indicate when an OPO should react to the data in the chart, possibly by initiating an internal review to determine whether action should be taken. For more information, see these two excellent papers, which are specific to the field of transplantation:

 

• Axelrod DA, Guidinger MA, Metzger RA, et al. Transplant quality assessment using a continuously updatable, risk-adjusted technique (CUSUM). Am J Transplant. 2006;6:313-323. (Link to article.)

 

Axelrod DA, Kalbfleisch JD, Sun RJ, et al. Innovations in the assessment of transplant center performance: implications for quality improvement. Am J Transplant. 2009:9:959-969. (Link to article.)

 

Why is SRTR producing CUSUM charts?

Currently, data on an OPO’s yield as published in its OSR are derived from data on donors at that OPO during a recent 2-year period. A limitation of the OSR methodology is that it presents the “average” yield over that 2-year period. It is impossible to tell whether yield improved or declined within the period. CUSUM charts allow OPOs to see their yield by calendar day, allowing trends in quality to be more apparent.

 

Do the charts indicate when a transplant program should be concerned?

Yes. The one-sided CUSUM chart can be used to determine when performance appears to be sufficiently worrisome to warrant action by the OPO. It attempts to answer the question, “Do the data fit better with a hypothesis that the OPO’s yield is average, or with the hypothesis that the OPO’s yield is 10% lower than expected?” A statistical measure of the difference between the likelihood of those two hypotheses is plotted on the one-sided CUSUM chart. The absolute value and interpretation of the CUSUM value and accompanying Y-axis is less important to interpreting the one-sided CUSUM. Rather, one should use the chart to identify when the data are approaching a predetermined threshold of interest. The threshold line is shown as a red horizontal line near the top of the chart. Its exact location differs for different OPOs and organs. The location is determined based on the number of donors and expected yield, and is designed to reach this level only 5% of the time if the program is truly performing as expected. If a program reaches the threshold line, the CUSUM chart resets to zero, essentially starting the monitoring process over from that point forward. This occurs because the program will theoretically review its processes at or near the time of reaching the threshold line, and may wish to resume the monitoring process from the start given any changes it may have made.

 

Do the CUSUM charts indicate whether an OPO will be flagged by CMS or MPSC?

No. CUSUM charts are meant to be used for quality improvement by allowing OPOs to better track performance in near-real time. In contrast, the data used by CMS and MPSC are summaries of "average" performance over a 3- and 2-year window, respectively. Imagine an OPO that performed very well for the first half of a 3-year window but changed its processes and performed poorly during the second half. Looking at data summarizing the entire 3-year period, one might infer that the OPO’s performance was as expected; however, the CUSUM chart would reveal that a problem began during the second half of the period. Hence, the picture revealed by the CUSUM analysis will not necessarily be the same as the picture revealed in the data reviewed by CMS or MPSC. Generally, if a program’s CUSUM chart does not indicate a concern, it is unlikely that the program will be flagged by CMS or MPSC. If the CUSUM chart does indicate a concern, flagging by CMS or MPSC is more likely.

 

How often are the CUSUM charts updated? 

SRTR releases updated CUSUM charts monthly on the SRTR secure website. There is usually a one-month gap between the release date and the last day data were collected. For instance, a May CUSUM will show data through March 31.

  

What period do the CUSUM charts cover?

Charts cover the most recent 4-year period. They include the most current data available, but OPOs are cautioned that they may be less reliable for the most recent days due to lags in reporting donor recoveries to OPTN. We encourage OPOs to be diligent in their reporting of data to OPTN to optimize the utility of the charts.

 

Here is an article that explains the methodology in greater detail.

https://www.srtr.org/media/1074/snyder_et_al_cusum.pdf

 

Are the CUSUM charts made public?

No. The charts are produced solely for the purpose of quality monitoring by OPOs. They are not provided to other parties or used by CMS or OPTN to identify OPOs for review. That is not to say that CMS or OPTN could never move toward using a CUSUM or CUSUM-like methodology for identifying programs in need of review.

 

What is the meaning of CUSUM Y-Axis increments?

On the one-sided CUSUMs, Y-axes are constrained sums of likelihood ratios, so they don't have meaningful units.

 

The signal threshold in the one-sided CUSUMs is set through a simulation. Five hundred simulated CUSUM curves are generated, and the threshold is set so that 5% of the simulated curves cross the threshold. The threshold can thus move up or down (a little) depending on the randomness of the simulations.

 

For the jump between events, the size of the jump is (# of observed transplants)*LN(0.9) + 0.1 * (# of expected transplants). Since the number of expected transplants varies from day to day, the size of the jump would vary. The number of expected transplant on any one day is zero on days with no recovered donors.

 

The E-O (or two-sided) CUSUMs are the more important charts. The one-sided CUSUM is merely the litmus for what specifically is happening if you see an anomaly on the two-sided chart. The E-O CUSUM charts show the Expected - Observed over time. The X-axis is time (day) and the Y-axis is the Expected - Observed.

 

Why is the timing of the CUSUMs not the same as the OSR cohorts?

Unlike the OSRs, which show the yield within a 2-year period (a cohort), the CUSUMs show the trend in yield over a 4-year period.

 

Why did my OPO’s CUSUM hit the threshold last month, but this month it’s not showing up that way?

The CUSUMs are run on the available data. Any CUSUMs run with the current dataset, and based on those current data, may not show a threshold hit. You will still see that spike in the old CUSUMs, however, because they are run on the data that were available at that time and are static.

 

The signal threshold in the one-sided CUSUMs is set through a simulation. Five hundred simulated CUSUM curves are generated, and the threshold is set so that 5% of the simulated curves cross the threshold. The threshold itself can thus move up or down (a little) depending on the randomness of the simulations.

How often does SRTR rebuild the donor yield models?

The donor yield models are currently set to be completely rebuilt once every 3 years. The OPO model was last rebuilt for the spring 2017 release.

 

How often does SRTR refit the donor yield models? What does refit mean?

SRTR refits the models every time the reports are run, i.e., every 6 months. This means that SRTR uses the most recent national data to produce new model estimates for each existing covariate. Refitting typically results in only minor shifts in the model over time, but allows performance monitoring to keep pace with changing national performance.

 

How often is the Donor Yield Calculator updated?

The calculator is updated each time the donor yield models are refit during each biannual reporting cycle.

 

For one of our donor cases, the OPO Yield Calculator expected organ value for the pancreas was “8E-05.” What does that mean?

This just means that the expected yield is almost 0. It’s a common notation in those cases and nothing to be concerned about.

 

We are not able to upload the files that were formatted to the previous calculator.

The current calculator version is formatted to import the current "Donor-level Data for Import into the Yield Calculator" CSV file. trying to import the current data file to an old calculator version or attempting to import the new data into an old calculator will produce an error. 

 

All donor information is being imported into the “Gender” field in the calculator as male donors. The excel file does have the correct gender listed, but when you import it into the calculator it defaults everyone to male.

The “Gender” field should be in the format of “M” or “F,” so if you enter “Male” or “Female,” the program defaults to M for everyone

 

How do I clear data in calculator?

If you need to clear old data and enter new, find the OPO calculator on your computer in “Documents/OPO Yield Calculator” (by default). From there, you can edit the input.txt file. It is a simple comma-delimited file and can be modified in Excel or any other program.

 

If you simply want to clear the input, move the input.txt file to another folder (i.e., “old yield files”) or just rename it. When you open the Calculator again, it will create a new clean Input file in the same folder.

I need the betas/intercepts/adjustments for my OPO from the donation rate model. Where do I find those?

Email your request to srtr@srtr.org.

 

I heard that there is a report that shows all the centers and how they compare in their acceptance practices; where do I find that?

If you are from an OPO, you can find that report on your “Reports” page under OPO Tools.

 

How frequently is the OPO Offer Acceptance Report updated?

Every 6 months along with the OPO Specific Reports release.

 

What constitutes an offer? Does an offer have to be a primary offer for a recipient at a transplant program, or does an offer count for patients who are further down the match run?

Offers are only counted if they were accepted or were declined before an accepted offer. This means that offers for eventually discarded organs are not included.

 

For “hard-to-place” organs, does “over 50 offers” mean more than 50 offers total, or that the organ was turned down more than 50 times before it was accepted?

The data evaluate offers that were previously turned down 50 times or more. This can include turn-downs at the same program. Note that if an organ is offered more than 50 times before being placed, the first 50 offers are not included in the calculation of the offer acceptance ratio of “hard-to-place” organs.

 

Is an "acceptance" only counted if an organ is transplanted?

Yes, only transplanted organs are counted as accepted.

 

When a patient is status I or has a MELD score of greater than 35, and a liver is accepted for transplant, the program will continue to receive offers for that recipient as the recipient is not removed from the list until after the transplant. These offers have to be declined. Are these also counted as declines?

Yes, we include these offers because programs may accept and transplant these organs (leading to a late decline of the previously accepted liver).

 

Is organ offer acceptance data only for single organs?

Yes, offers to candidates listed for multiple organs at the program are not evaluated.

 

Are multi-organ transplant excluded from the offer acceptance data?

Yes, offers to candidates listed for multiple organs at the program are not evaluated.

How can new users be added?

Only administrators can add new users to the secure site, and the potential new user should contact the administrator of the program. For information on how to administrate your secure site, please see this document: Secure Site User Instructions.

 

How can new administrators be added?

Only an SRTR administrator can add program administrators. The administrator for the site must give permission via email to SRTR to add a new administrator.

If an administrator leaves the program, your program will need to designate a new site administrator.

 

Why am I not getting SRTR notices from the OPTN listserv?

The PSR schedule/information is posted to the Transplant Administrators Listserv and the SRTR public and secure sites approximately two weeks prior to the April and October Data Review Periods. Membership to the listserv is restricted to mainly transplant administrators. If you are not an administrator at your program, you may pass the following link to your administrator and request that he or she join. (Link to OPTN membership.)

SRTR attempts to send notices through additional resources. Depending on the nature of the information, SRTR may post it on the news page of our public site. We frequently post announcements on our twitter feed, @SRTRNews. And finally, our newsletter announces important events such as release dates. To sign up for the newsletter, use this link: www.srtr.org/#newsletter