Statistical quality incident response plan

The Australian Bureau of Statistics (ABS) leads a high quality national statistical service. One of the ways this is advanced is through the development and sharing of quality management initiatives used within the ABS that may be applied by other organisations in their management of data.

The ABS has released a series of information papers relating to the quality management of data. The topics of these papers focus on assessing and declaring the quality of statistical products, and managing the quality of statistical processes.

This paper follows on from the Quality Management of Statistical Risk Using Quality Gates, Dec 2010, (cat no. 1540.0), information paper in that it too examines statistical risk and how to manage it. However, the previous paper discusses a mitigation strategy for reducing statistical risks (quality gates), whereas this paper discusses contingency plans for statistical processes when risks are realised. The ABS calls these contingency plans quality incident response plans (QIRPs).

A quality incident response plan (QIRP) is a systematic plan for resolving serious doubts about key statistical results by examining the processes that contributed to the results. The QIRP approach is designed to provide a quick and rigorous high level response to a serious statistical issue be it realised or potential. A key feature of the approach is the formation of a dedicated ad hoc multi-disciplinary team to investigate and resolve the issue in a pragmatic way.

This paper provides information for conducting and creating a quality incident response plan. This helps manage statistical risks by having documented and accepted guidelines on what to do when a quality incident arises.

A quality incident response plan process has the following four broad stages:

1. Initiate a quality incident response plan
2. Manage the implementation of the quality incident response plan
3. Resolve the quality incident (i.e. confirm or reject causal issues)
4. Implement

This section includes the following subsections:

• What is a quality incident?
• Identify a quality incident
• When should a quality incident response plan (QIRP) be initiated?
   

A quality incident occurs when the quality of the data are called into question. This could occur because an anomaly has been identified with data already released in the public domain; or a potential error that could have a negative impact on the outputs and agency is identified in data that are not yet in the public domain.

Throughout this paper the term quality incident is used to mean that either the quality of the data or the processes that produce the data are being called into question (or both), as well as covering the situation when a definite error is identified in the data.

A quality incident that warrants a QIRP response is one where the consequences of the potential risk being realised are severe for either the organisation or the public. A severe situation signals extreme circumstances and is a signal to 'call a QIRP' immediately, postpone the release of the data if need be, and commence investigations into the quality incident.

In order to initiate a quality incident response plan a quality incident needs to be identified. There could be a variety of triggers that indicate that there is or is about to be a quality incident. Some of the obvious triggers that the Australian Bureau of Statistics (ABS) has noted are:

• Staff or data users identify coherence problems;
  • within the current set of data, or between the latest release and historical data, e.g. an estimate for a data series may show an unexpected growth or decline compared to the previous time it was collected, which is outside the accepted tolerance level;
  • with other sources of similar data, e.g. between the unemployment rate and job vacancies;
  • or with real world events, e.g. if a key statistic does not move in the expected direction given all other data in the economy then there might be an issue with the data.
• The data do not meet expectations of sources both internal and external to the organisation, e.g. economists, experts, senior management within the organisation.
• Other indicators in the statistical process indicate there may be an issue, e.g. a quality measure shows high imputation rates; there has been a delay in the receiving of data at a critical time, which has put pressure on the process.
   

Each of the above factors by themselves may not be enough to call a quality incident, but rather a combination of some of them may lead to a quality incident being identified and a quality incident response plan being invoked.

The ABS Data Quality Framework is a useful tool that may help when trying to identify a quality incident. The ABS uses the seven dimensions of the ABS Data Quality Framework; institutional environment, relevance, timeliness, accuracy, coherence, interpretability, and accessibility; to assess, compare and declare the quality of final statistical outputs. However, the ABS Data Quality Framework can also be used as a checklist and applied to intermediate outputs at earlier stages of the statistical cycle.

When an anomaly with the data is identified, it may be useful to create a list of questions relating to each dimension of the ABS Data Quality Framework to help determine if the anomaly is actually a quality incident. By doing so, a determination can be made as to whether the quality incident is severe enough to call a QIRP. Some questions that might be useful to ask in order to determine if a quality incident has occurred are the following:

• Institutional Environment
  • Has there been a legislative change that may have impacted on the data?
  • Has the mode in which the data are collected changed? E.g. formerly face to face interviews are now being collected via telephone or using the internet.
  • Are senior managers questioning the data?
• Relevance
  • Has there been any change to the population of interest for the collection?
  • Have any key data items or questions changed their definition or construction since last time?
  • Are external users questioning the data?
• Timeliness
  • Were there time pressures during the process due to other delays?
  • What parts of the process ran late during the statistical cycle?
  • Has there been insufficient scrutiny of the quality of the outputs due to processes running late?
• Accuracy
  • Has there been a significant change in response to certain questions? E.g. a particular question has a large non response.
  • Have any other quality assurance tools flagged potential issues?
  • Have there been any changes to the methodology underpinning the collection?
• Coherence
  • Do the data make real world sense?
  • Are the data comparable with other statistical releases?
  • Are the data comparable with themselves over time?
• Interpretability
  • Have there been any ‘real world’ events, which may have impacted on the data? E.g. floods, global financial crisis.
  • Have there been any changes to classifications or standards associated with the data? E.g. changes to industry classifications or country classifications.
• Accessibility
  • Has there been a change in the systems that produce the data?
  • Has the format changed for the release of the data? E.g. the layout of the data in the spread-sheet has changed from the previous release.
     

More information on the dimensions of the ABS Data Quality Framework and its uses can be found in the information paper ABS Data Quality Framework, (cat.no 1520.0) or alternatively in the Data Quality Online tool on the National Statistical Services website (http://www.nss.gov.au/dataquality/).

It is important that a quality incident response plan is only used when it is really required. A key point of a contingency plan is to evoke a powerful response to a problem that needs to be dealt with before proceeding with business as usual.

Using a contingency plan too often, or when it is not necessary, may decrease the effectiveness of this response, and waste time and resources, as well as risk undermining the self-confidence of team members. However, it is critical that quality issues are not overlooked for too long or worse still, missed altogether. Hence, it is important to be able to identify quality incidents that require a quality incident response plan.

An assessment of the potential quality incident should be made to determine its impact, if it is realised, on the reputation of the organisation and also on the quality of the data. A quality incident that has potentially severe consequences to the organisation or to users of data should signal that a Quality Incident Response Plan is required. The initial assessment of the potential quality incident may provide enough information for senior managers to be satisfied that the data are correct and hence, a quality incident response plan process will not be invoked.

At the Australian Bureau of Statistics, any quality incident that could cause a loss of reputation, credibility, or impact negatively on provider or user confidence due to the publication of misleading data, would instigate the initiation of a quality incident response plan to deal with the issue. For example, if the Labour Force estimates prior to release revealed that one or more of the headline indicators behaved in an unexpected way, given the previous months' trend and everything else that is known about the current economic situation, then a quality incident response plan would be invoked as a way of further quality assuring the data. This is because the consequence of the data being incorrect and released to the public would not only have a detrimental effect on the reputation of the ABS but may also impact negatively on the economy. A quality incident that is discovered after the data have been released publicly would warrant an immediate quality incident response plan.

For those quality incidents that are identified prior to the public release of data, a risk assessment of the quality incident could be undertaken to assist in determining whether a quality incident response plan is required. Each organisation has its own risk assessment matrix to assist in decision making. However, a risk assessment matrix that may be of use is the ABS Statistical Risk Management Framework . This can be found in the appendix of the Quality Management of Statistical Risk Using Quality Gates information paper.

Knowing when a potential quality incident is serious enough that it requires a quality incident response plan to be invoked is an important part in managing statistical risk. A quality incident response plan assists in determining what the causes are of the potential quality incident and how to resolve them.

This section contains the following subsections:

• Initial quality incident response plan (QIRP) meeting
• Meeting participants
• Establishing the facts, defining the problem, and establish objectives
• Changing mindset - assume that something is not right
• Establish a range of possible sources of the problem
• Contingency plan and action
   

The QIRP follows a step-by-step process starting from when the quality of the data or process are first called into question, at which point a decision is needed as to whether the quality incident is serious enough to require a contingency plan. At the Australian Bureau of Statistics (ABS), the decision to 'call a QIRP' is often made by the senior manager who is responsible for final approval of the statistics for release to the ABS website. This is done in consultation with the manager of the collection area and any other relevant team members who can provide insight into the situation to help with the decision making. If the decision is made to go ahead with a QIRP, then the initial QIRP meeting needs to be planned and organised. The initial meeting will discuss the issues surrounding the quality incident and establish a plan for investigating possible causes of the quality incident. The initial meeting will also be when the contingency planning begins, including brainstorming the possible consequences of the quality incident and the actions that can be taken to minimise the risks.

A key to the successful resolution of a quality incident is clear documentation of the management of the QIRP. Documentation should clearly define the roles, such as who is organising the meetings, who is following up on the investigations, and who is the main ‘driver’ of the QIRP. Careful consideration also needs to be given to identify the key stakeholders, and whether, how and when the stakeholders should be informed about the quality incident and how it is progressing.

Note that whilst the quality incident response plan involves a number of steps, these steps can be run at the same time. There may be an operational requirement to ensure that the steps occur concurrently in order to adequately manage the quality incident within a short time frame.

Once it has been determined that there is a quality incident and a decision is made to use the quality incident response plan, the initial QIRP meeting needs to be organised (usually at short notice), ensuring that all people who may need to be involved in the process are included in the invite.

The meeting's aims are to identify the actual issue or problem, brainstorm possible solutions, and agree on ways to move forward and resolve the issue.

The potential consequences of the quality incident will influence who needs to be involved in the initial meeting. For example, if the issue has been identified prior to any data being published or released then the issue may be able to be handled at a more local level, with senior managers being kept informed of the issue but not necessarily directly involved in the response. However, if the issue is only identified when the data are already in the public domain, then the QIRP meetings may immediately need to involve the senior management of the organisation, as the potential consequences to the reputation of the organisation and impact of incorrect data in the public domain will need to be managed quickly.

Who to include in the initial meeting will depend on the nature of quality incident. Initially, it is probably best to include more people rather than less to ensure that all the issues and potential solutions are identified. Participants at the initial meeting would include those people who are directly responsible for producing the statistics and those people who support the production of the statistics. For example, at the ABS not only would the collection area people be involved, but people from the methodology area and time series analysis area (if relevant) would be involved in the meeting. Other participants might include areas of the ABS who are responsible for other aspects of the collection cycle that may impact on the statistics and process in question. Once again, participants at the meeting will depend on the nature of the quality incident and where it has been identified in the collection cycle process.

Other stakeholders who may be kept informed of progress but do not attend the meeting include areas of the organisation who are not directly involved in the production of the statistics but their work is dependent upon them. For example, at the ABS the publishing area who manages the ABS website and its releases would be kept informed of any potential delays to statistical releases in order to manage their workload accordingly.

Along with identifying the participants, expectations about resourcing also need to be stated. Participating stakeholders need to be aware that they may be expected to allocate dedicated resources to the QIRP process until the issue is resolved.

It is important that all the key people hear all the facts first hand. Once the QIRP is underway there is no time for misunderstandings about what is needed, why, or what it might mean at a later date. It helps if everyone involved is aware of the entire process and their part in it so that they can adapt and help each other. Having all the stakeholders together initially is a more effective way of managing the QIRP compared to having multiple bilateral meetings. The identifying of all of the issues, understanding the problem, working out potential solutions, identifying implications and dependencies, and agreeing on an action plan in a very short period of time is facilitated through having all key players together at the first QIRP meeting.

To effectively manage the QIRP, roles for addressing the quality incident need to be defined up front to ensure that stakeholders are aware of their responsibilities and time is not wasted due to further confusion. The roles that need to be established in this step are:

• A facilitator: It may be useful to have an independent facilitator for the fact gathering process to allow those involved in the investigation process to participate fully. The tasks of the facilitator in the initial and any subsequent meetings would be to draw out all of the potentially relevant facts, to keep the meetings on track, and to ensure that all participants are engaged and consulted. This ensures all relevant information is considered for resolving the quality incident.
• A driver: A driver is the person responsible for driving and coordinating the process from start to finish. Their role includes progressing issues, being aware of who is doing what and when, and documenting the process and the actions to be carried out with the assistance of a note taker. The driver is also responsible for troubleshooting business as usual obstacles that may arise during the process to ensure that the incident is resolved within a short time frame. As part of their role they are also prepared to drop everything else in order to focus on the QIRP.
• A meeting co-ordinator: A meeting co-ordinator is responsible for determining who should be included in the QIRP meetings, scheduling the meetings and circulating agendas.
• A note taker: The note taker is responsible for recording the key facts and points of discussion and passing these notes on to the meeting co-ordinator for circulation post meeting.
• Quality incident investigators: The initial QIRP meeting will typically establish a range of investigations that may be required to find the source of the quality incident. The people responsible for carrying out these investigations and reporting back to the QIRP driver need to be aware of their role and the responsibilities associated with that role.
   

Note that different people are not necessarily required for each of the roles. For example, the driver and meeting co-ordinator may be the same person. However, it is important to have each of these roles identified and allocated so that the underlying responsibilities are not missed.

The objectives of the first QIRP meeting after the quality incident has been discovered are to outline the key facts of the quality incident so far and come up with a definition of the problem, and establish a plan for investigating the problem. The meeting should include a rigorous discussion of the incident, giving the opportunity for everyone involved to present and discuss the key facts surrounding the incident, so that the real issue(s) can be identified and defined. It is critical that detailed notes are taken from this discussion and circulated afterwards so that everyone involved has a clear understanding of what was discussed and decided.

This meeting is also an opportunity to discuss the QIRP approach so all participants understand the process. A QIRP is about working out what the issues are and why they occurred, in order to implement solutions that fix the problem and enable the process to move forward. Solutions may not only apply directly to the process, they may be more fundamental in terms of overall staff capability building across an organisation. The QIRP approach is not about assigning blame; everyone involved should feel free to contribute openly and honestly without fear of retribution. It is the facilitator's role, if one is being used, to draw out all the potentially relevant facts.

It is important that the meeting triggers a shift in mindset to reflect the potentially serious situation. Once a quality incident is suspected, the aim of the meeting should not be to reassure each other that there is no major problem, but rather agree that there is a potentially serious problem. Potential causes and solutions will be more rigorously considered if there is a working assumption that something has gone wrong. From the outset of the meeting, the ability to recognise the changed situation, its significance, and the need to look at the problem differently may be crucial to resolving the quality incident.

The initial QIRP meeting is the time to establish what the possible sources of the quality problem may be and how these will be investigated. There are some different methods and questions that may help to generate a list of potential sources of the problem. The most common of these is to have a brainstorming session that everyone in the meeting is encouraged to participate in. The brain storming session should come up with a list of any potential sources of the problem, which can then be discussed according to whether each idea is feasible or likely. It may be helpful at this point to rank the ideas from most likely to least likely, as this will indicate where the ensuing investigations should start from. The discussion of each possible source should also identify what further information is needed to confirm or deny the idea as a source of the problem.

If there are other standard quality tools in place, such as quality gates, then it is a good idea to check these as they may help to pin point the cause of the problem. Another approach is to examine each step of the statistical process for any problems, inconsistencies or changes, be they to the process, methodology, collection, processing of data, or key people. For any change it is important to understand what has changed, how it was managed, the expected impacts of the change on the process and statistics, and if there were circumstances surrounding the implementation of the change that may have had an effect. For instance, time and resource constraints may have impacted on the implementation of a change, which could mean that the change was not tested to the usual standard.

It is also important to look at whether there are other issues that may be clouding the process and masking the real problems or distracting team members from the main issues. This could include real world issues that are affecting the statistics (e.g. changes in government policy that may impact such as the baby bonus, changes in school leaving age, federal budget decisions, Christmas and New Year public holidays), or methodological adjustments (e.g. treatment of outliers or the seasonal adjustment process).

This process will likely unveil a number of issues that will need to be investigated further. Some documentation will be helpful to keep a track of the roles and responsibilities for investigating each issue. As the initial generation of potential problems is a brainstorming exercise, all suggested causes should be documented. If suggested causes are not followed up then the reasons why should be documented for future reference. It is also important to make a note of any issues that distract from the serious issue.

A list of the possible sources of the cause of the quality incident should be made along with appropriate information regarding who is responsible for investigating, when the investigation needs to be complete and also the priority of the investigation.

An example of how this may be documented is below:

The brainstorming session and discussion from the QIRP meeting may produce a lot of information about the quality issue and the potential sources of the problem, which will need to be investigated. The final part of the meeting should therefore propose a plan for investigating these leads and subsequently dealing with the problem. If the investigations do show the problem, then what is going to be done about it, what needs to happen next, and what will be done later in the plan?

This part of the meeting is about looking ahead to where the data are going, what they will be used for, and anticipating what consequences the quality issue will have further downstream. The ABS looks at these issues differently depending on whether the quality incident is identified before or after data have been released into the public domain. For pre-release issues, time may be an important consideration. If the data are due to be published, a necessary contingency may be to postpone the publication date until the issue has been thoroughly investigated. If the data are due to be sent to another area of the organisation then the other area needs to be informed of the issue and a possible delay. When the data are released, further explanatory notes may be required so that users can interpret the data in the context of the issue.

For post-release quality issues, the focus is on managing the impact of having incorrect data in the public domain. Senior managers should be involved, as they may need to brief the media and key users, clients, and stakeholders of the incident and how the issue is being resolved. For example, the data may need to be amended or re-released, or have additional explanatory notes added. It is important that this process is handled transparently by the organisation. This is to ensure that the steps being taken to fix the current problem and prevent further incidents of that nature occurring are conveyed to the public to assist in their understanding of the data as well as to maintain the public’s trust in the organisation.

Any potential problems as a result of the quality incident and suggested contingency actions should be documented and monitored. Below is a basic template for these contingency actions.

Template and example for recording contingency actions:

At the end of the QIRP meeting, the meeting coordinator should book the next meeting so that the progress of investigations can be checked and discussed, and potential solutions implemented. A meeting with senior managers may also be needed to present the options and to decide on the appropriate way forward. Given the tight time frames for resolving quality incidents, these meetings should be comprehensive but to the point.

This section contains the following subsections:

• Follow up meeting(s)
• Check progress and confirm cause(s)
• Decide how to proceed
   

The immediate time frame after the quality incident response plan (QIRP) meeting is critical to containing the quality incident. This stage focuses on narrowing down the causes of the incident, checking, confirming, and working out solutions. Communication and timing are paramount particularly in terms of keeping the relevant people informed of the progress of the investigations. This is even more the case if steps in the QIRP process are being run concurrently.

There will be at least one follow up meeting after the initial QIRP meeting. The follow up meeting(s) will review the progress of agreed investigations that have been undertaken (usually urgent and high priorities) as a result of the previous QIRP meeting. The purpose of this meeting and any subsequent ones is to share information amongst all participants simultaneously to allow further brainstorming and problem solving to occur amongst the group with all the facts known to all participants.

Once the first investigations have been carried out, everyone involved in the QIRP will need to reconvene. This meeting usually happens quite quickly following the initial meeting as time is usually quite short for resolving the quality incident.

This meeting is a check of the progress and investigations to date to see if any light can be shed on the issue or whether additional analysis or investigations are warranted. Any new information that has come to light should be incorporated into the analysis. The contingency plans created in the first stage should also be updated to reflect any changes.

If a potential cause of the problem has been found, is there enough evidence at hand to confirm that this is the cause, or are further studies required? Is the potential cause identified the sole cause, or do other potential causes need to be ruled out? QIRP participants need to be alert to the possibility that there may be more than one problem. At this point, the area or person responsible for any further investigations should be nominated.

The results of the investigations and any new investigations that result all need to be documented and attached to the initial table that was created to document the process. It is important that everyone involved has a copy of these documents, so that they are aware of the issues and progress as well as ensuring that efforts are not duplicated on the same issue.

When all of the required investigations have been carried out and a cause(s) of the problem has been confirmed, or no cause was found and the evidence available shows that the data are actually correct, a meeting needs to be held to brief senior managers and other relevant parties. This meeting should briefly outline the steps taken thus far in the QIRP, the investigations, and the evidence that they have produced. From this point, a discussion and decision is needed as to how the issue should be treated.

The response options, including a preferred or recommended option, should be presented, keeping in mind the existing systems and the time available to implement a solution as this will impact on the contingency plan and timing. Once a preferred option emerges, an implementation plan needs to be developed clearly spelling out who is doing what and when.

Discussions should include answering the following questions:

• Who should be informed and how?
• Do the media need to be advised?
• Are key users to be informed?
• Does additional information need to be provided for the publication?
• If the data have already been released, is a re-release of corrected data necessary?
• Are there impacts for other surveys or products?
   

Communication, documentation and a clear understanding of roles and responsibilities and time frames is once again paramount at this stage.

Once the preferred option for resolving the quality incident has been decided upon, the various contingency actions need to be put into place, monitored and evaluated. The final step in the quality incident response plan (QIRP) process is to learn from it so that similar problems can be avoided in future, or at least found earlier in the process.

By now the quality issue should be well understood and an appropriate contingency plan to deal with issues resulting from the quality incident devised. The contingency plan needs to be implemented, including clear documentation, which clearly states the roles and responsibilities, as well as who and how people will be informed. The implementation of the contingency plan needs to be closely monitored to make sure that all of the actions are carried out and to determine whether further actions are needed.

It is important that the process and outcomes from the QIRP are written up and that any necessary follow-up action is undertaken and documented. For example, do any procedures for the next cycle of the collection need to be modified? Are new procedures, or quality measures or gates needed to improve the process?

The occurrence of a quality incident should signal that additional quality measures and monitoring need to be incorporated in the planning and process for the next cycle. In this stage of the QIRP there is more time to think and the objective is to look beyond the immediate causes to uncover the underlying lessons and incorporate them into future planning. The primary aim here is to prevent similar quality incidents from occurring again in the future. Ensuring that all discussions and investigations are maintained whilst the QIRP is underway is essential for incorporating the lessons from the process into future cycles.

The Statistical Quality Incident Response Plan, cat. no. 1542.0, is part of the ABS quality management series of information papers. See also:

ABS 2009, ABS Data Quality Framework, cat. no. 1520.0, ABS, Canberra.

ABS 2010, Information Paper: Quality Management of Statistical Processes Using Quality Gates, cat. no. 1540.0, ABS, Canberra.

ABS 2011, Information Paper: Quality Management of Statistical Outputs Produced from Administrative Data, cat. no. 1522.0, ABS, Canberra.

The Statistical Quality Management webpage also contains information on statistical quality management initiatives in the ABS.

This release previously used catalogue number 1542.0.