The new FDA guidelines on Quality Metrics

The new Food and Drug Administration (FDA) guideline “Submission of quality metrics data – Guidance for industry” was published at the end of November 2016. It represents a proposed, new risk-based tool to help both companies and the FDA to better define the timetable for inspections according to articles 704-706 of the US Food and Drug Administration Safety and Innovation Act (FDASIA). FDA and ISPE International collaborated over the last three years to conjunctively build the new approach to inspections, which will base upon predefined and harmonized quality indicators. No more biennial deadlines for companies to host inspectors: FDA’s target aims to an increased flexibility of the entire process on the basis of the quality indicators (“quality metrics”) the companies themselves will send to the Agency as a result an internal self-evaluation process. Other joint goals of FDA and ISPE include the ability to prevent and mitigate episodes of drug deficiency on the market and to encourage the adoption of innovative and last-generation quality management systems.

The first phase of implementation of the new draft guideline will see the launch by FDA of a voluntary reporting scheme of quality metrics, focused on the three indicators described in the document. The US Agency expects that the initiative will see the initial adhesion of manufactures of “covered drug products” finished pharmaceutical forms or of the corresponding active pharmaceutical ingredients. The voluntary implementation phase will not include manufacturers of biological products, blood derivatives, vaccines, in vitro diagnostics, allergen extracts and products for gene or cell therapy.

ISPE International, the international association of professionals operating in the field of pharmaceutical engineering, started the ambitious project in 2013, acting as the connection link between FDA’s requests and proposals developed by a group of about thirty companies (mostly big pharma), for a total of eighty manufacturing sites involved at the global level. The pilot project aimed to demonstrate the sustainability of the quality metrics approach and to the identification of a preliminary set of proposed standard indicators. Two representatives of ISPE Italy, Francesca Maienza and Boris Gabani participated at the working group and tell NCF Pharma World about the initiative.

The meaning of “quality metrics”

«The term “quality metrics” means all those indicators that give an idea of ​​the collective quality characteristics of a certain production site. The number of rejected batches per year compared to the total number of batches produced, or the number of out-of-specification results found by a QC lab, and which are then not confirmed with respect to the total number of batches tested, are examples of generic indicators already used by most companies and that may provide an idea of ​​the quality management of a site», explains Boris Gabani, QC Raw Materials Lab Manager at Roche, who followed the pilot project for the company’s Italian Segrate production site. FDA’s intention in 2013 was to establish a self-evaluation process to identify useful indicators for a preliminary assessment of the qualitative performance of a certain production site. The process should have produced a sufficient set of data to support the Agency decision to increase or decrease the inspection frequency. «The initiative born to address the significant increase of inspection requests received by the FDA. Its target was the collection of standardized data among companies to be used to calculate the risk for each site upon which take the decision if to increase or decrease the frequency of inspection – tells Gabani -. The translation into reality of this need has been possible only now, with the publication of the guideline. ISPE’s supported this need, in order to ensure that indicators were usable in practice and could be applied in a standardized way».

The pilot project

«ISPE acted as an intermediary, as the FDA could not directly interact with individual companies, and run the project over three years in two successive phases», says Francesca Maienza, QC manager of Farmila-Thea and Operation support director of ISPE Italy. The data referred to individual participating sites were collected with the help of a consulting company, which then analysed them under a confidentiality framework. The final data were shared with the company that generated them. The project’s global data were then transferred to the FDA, which used them to decide what were the most appropriate quality metrics to be included into the guideline under preparation. The first phase, conducted in 2014 on a limited number of companies, focused on quantitative quality metrics and run surveys to help evaluate process capability and quality culture within the various participating sites.

«The second wave, in 2015, involved more companies with the goal of identifying the optimal quality metrics indicators to measure the quality level in a harmonized way among different sites. This was also the most difficult aspect, not yet resolved. The collection of data and numbers needed to understand which were the most significant and useful indicators has been very demanding», says Maienza.

«Quality culture-related indicators can be considered to be quantitative, because the collected data are in numerical form. But they identify a parameter – quality culture – which is difficult to measure», tells Boris Gabani. To solve this problem surveys have been used, and interviews to employees and managers were collected in each company to specifically assess the perceived quality culture. «Collection of this kind of indicators could not use numbers, it has been necessary to use percentages relative to the level of agreement on individual questions», adds Gabani.

Phase 2 included also an evaluation of the effort and logistics needed by a company in order to collect data, as well as the development of appropriate tools for the optimal management of quality culture and process capability.

FDA chose three indicators

The second phase of the project started with the identification of a wide set of indicators (Indicators used during the pilot project) to evaluate the sustainability of the approach across different companies. The final data transmitted to FDA led the Agency to select three of them, which were used as the basis for the new guideline. They are the Low acceptance rate (LAR), the Product quality complaint rate (PQCR), and the Invalidated out-of-specification rate (IOOS) (Indicators included into the FDA guideline). «Many of the indicators that were evaluated were real challenges in order to check the site’s reaction and the kind of results that would have been possible to achieve: for example, if big differences were present between different sites, or if data between biotechnological, chemical, or active substance production sites might have been aggregated. The data obtained showed that this type of approach is often unsustainable in order to obtain data to be sent to the FDA, as it is too complex to attribute a precise meaning to the results obtained», tells Boris Gabani.

Low acceptance rate is a performance indicator of the production process that shows the number of batches released compared to the number of batches produced in a certain time unit. Customer satisfaction is assessed through the product quality compliant rate, an indicator of the number of complaints received compared to the number of total batches released in a defined time unit. «The “customer” is both the patient and the buyer, whether it is a health authority or another companies in the case we are working with third parties», says Roche’s expert. Quality control is monitored by the use of the invalidated out-of-specification rate: in this case, the indicator highlights the number of unspecified out-of-specification data, an important factor to assess the goodness of the operations performed by QC labs. «All out-of-specification which are traced but unconfirmed are usually laboratory errors, which solve upon the identification of an error in the analytical process, not in the batch itself. The lower the number, in a given time frame, the more the laboratory works well», adds Gabani.

The future of the project

The FDA draft guideline represents just a first proposal of indicators that can be used by companies around the world. These are still free to send further proposals of new quality metrics to the Agency. «The project is still open, there are many questions and doubts – tells Francesca Mazienza -. We have focused on three quality indicators because we have come to the conclusion that the simpler they are, the easier they can be standardized between different sites to provide comparability between one site and another. They must be simple numbers to calculate, based on very clear parameters and defined time units».

Each company sets its own quality system and identifies the key performance indicators (KPIs) needed to monitor its activities on the basis of the company’s specific features. This heterogeneity made difficult to identify a set of homogeneous indicators to be reproduced across companies that participated in the pilot project. «The FDA guideline suggests the three main indicators identified by the work done with ISPE, which were found to be the most univocal, simple to interpret and use. The guideline is open to use other indicators, as well as to the proposal of the companies to publish reports to suggest other indicators that are understandable and validated», adds the ISPE expert.

Quality metrics self-monitoring tools should not represents a shock for major pharmaceutical companies, already used to operate according to this type of approach. Many of them, in fact, have taken the opportunity of the pilot project as a specific exercise to better target their KPIs in accordance to the work done with ISPE and finalised to the FDA. «Companies were interested in participating to the pilot as they had the opportunity to play a primary role in defining the new indicators, thus helping themselves to measure and understand how to move», says Maienza. The new guideline could also serve as inspiration for the European regulatory authorities, which, according to the experts, might eventually implement a similar instrument also in Europe based on the US’s results. «FDA proved to have a very practical approach to the project, as it chose only three indicators among the twenty identified during the second pilot phase. FDA’s decision has been based on the feedback received, the three indicators are the ones that need less effort from the companies», tells Boris Gabani. According to the expert, FDA realized that multinational companies would be able to afford the costs for a dedicated person to manage the indicators, but smaller companies may not.

To start implement the entire process, a data sharing platform generated by pharmaceutical companies is by now still missing. Boris Gabani assumes the transmission of data to the FDA could occur through a form sent to an online mailbox, or a web-based system with a password-based access for individual companies. This model would allow the transfer of data directly into the system to be consulted by the Agency on the other side of the interface. «To start the process, it is necessary that the guideline diffusion is not only restricted among companies selling their products in the US, but also in Europe. There is also need for some more practical detail in order to communicate these numbers», adds Gabani.

Culture is a must

«All the companies participating to the pilot project accepted to play the game, as activities could have shown some weakness, or new parameters could have emerged which until now no one had ever thought, or new KPIs or recalculated in a different way», says Francesca Maienza about the  practical experience of the companies. The project also had an immediate impact to implement the needed actions to promote knowledge and dissemination of information across all departments. «We realized that sometimes actions decided by few people sitting around a table are not acquired and known by everyone in the company. A shared knowledge of issues by everyone helps to improve», says Maienza.

Boris Gabani tells about Roche’s experience, where a QC lab resource has been devoted entirely to the improvement activities: «We reviewed instruments’ position, workflows and document management in order to improve the process. Although it was not a direct consequence of the FDA quality metrics project, the execution of these activities allowed us to highlight some areas of improvement to work on. For example, we have identified a new indicator, the number of batch records waiting for approval for more than 30 days. It could happen that a batch record of a product that was not to be shortly shipped remained for a long time waiting for approval, while others were moving forward thus invalidating the average of release». Boris Gabani also tells that the new performance self-evaluation modalities helped towards the regular publishing and discussion of KPIs in each department. «The project has increased the self-awareness of values ​​and the desire to share improvements: a positive outcome of the FDA and ISPE project for all companies» concludes Boris Gabani.

Indicators included into the FDA guideline

• LAR– Lot acceptance rate – indicator for the performance of the manufacturing process;
• PQCR – Product quality compliant rate – indicator of the feedback from customers or patients;
• IOOS – Invalidated out-of-specification rate – indicator describing the operativity of QC labotarories.