PharmaR

{riskassessment} ShinyGatherings x Pharmaverse Workshop

Tue, 11 Feb 2025 00:00:00 +0000

At a recent ShinyGatherings, some {riskassessment} contributors presented a workshop that focused on tailoring the application to org-specific risk requirements to simplify R package validation. Though we discussed the end-user experience, this workshop was really geared towards equipping “app deployers” to get the highest and best use out of the application for their organization’s specific needs. We appreciate Appsilon for the opportunity to speak about the application!

About the Workshop

The workshop’s main topics:

Risk Assessment App v3.1.1

Thu, 08 Aug 2024 00:00:00 +0000

Progress is continuing to be made in {riskassessment}. We wanted to share some of the enhancements and updates included in the most recent release of the application. There have been multiple releases, both minor and major, since our last post, so we have a lot of new content to cover!

But before we get into too much detail, if you are new to what we are doing in {riskassessment}, we would like to encourage you to check out our README. There you can find information regarding what {riskassessment} is seeking to accomplish and how you can install and deploy an instance of it for your personal or organizational use.

Updates to the R Validation Hub Executive Team

Tue, 05 Sep 2023 00:00:00 +0000

As the R Validation Hub closes in on its 5th year of activity I want to take a quick trip down memory lane and reflect on how we got here. Perhaps I should start by reminding everyone how it all started. Back in 2018, as the prospect of using R for any regulated analysis was still a hotly contested question, industry participants brought our donated space at Harvard to capacity as attendees gathered for the first ever R/Pharma conference. Amidst a slew of impressive shiny applications (I vividly remember shiny being the decided tone-setting theme that year), one Andy Nicholls offered up a deceptively simple question: what should validated use of R look like?

Risk Assessment App v2.0.0

Wed, 16 Aug 2023 00:00:00 +0000

Welcome! It’s with great excitement and long-awaited anticipation that I get to share some recent updates that have hit the {riskassessment} app’s GitHub repository earlier this month. If this is the first time you’ve heard or seen the application, I’d recommend starting with our README to gain some familiarity with the project, especially with installation instructions. However, (in a nut-shell) the app is a full-fledged R package that seeks augment the utility of the {riskmetric} package within an organizational context.

{riskassessment} App voted best Shiny app at shinyConf 2023! 🎉

Fri, 02 Jun 2023 00:00:00 +0000

The {riskassessment} app, presented by Aaron Clark from the R Validation Hub Executive Committee, was voted best Shiny app at shinyConf 2023. The 2nd Annual Shiny Conference was held in March 2023. It was all virtual with over 4k global registrants. Congratulations!!

The app provides a shiny front-end to augment the utility of the {riskmetric} package, thus user-friendly and interactive access to risk assessment of R packages. The apps functionalities include:

Summary of 2022 Case Studies

Wed, 15 Mar 2023 00:00:00 +0000

Last year, the R validation hub recently initiated a three-part presentation series on case studies in which eight pharmaceutical companies shared their experiences on building a GxP framework with R. These case studies highlighted both easy and challenging aspects of implementing risk assessment for R packages in a GxP environment. In this blog post we attempt to summarize common themes, difference in approaches, and challenges.

All implementations followed the risk validation process for R packages outlined in the white paper. There was a common theme of categorizing package quality into two or three risk categories. Test coverage was identified as a high-importance assessment metric, and the R Foundation was determined to be a trusted resource. Core R and recommended packages were treated as a collective of “low-risk” packages, with some organizations extending this to the tidyverse.

ASA Biopharmaceutical report, Fall 2022

Mon, 05 Dec 2022 00:00:00 +0000

I’m pleased to share that the R Validation Hub’s efforts have been recognised in the ASA Biopharmaceutical report, Fall 2022. Within the edition you can find our paper, Risk Assessment of R Packages: Learnings and Reflections. This paper reflects on our white paper; provides an overview of our {riskmetric} package and Risk Assessment application; and summarises our 2022 case studies (which you can now find on our Case Studies page).

External R Package Qualification Implementation at Merck

Wed, 21 Sep 2022 00:00:00 +0000

Introduction

There has been a growing interest in pharmaceutical industry to use R for clinical trial data analysis and reporting (A&R). Using R for regulatory submission purposes requires careful qualification of R packages given that the open-source packages differ in their quality of development. Many cross-industry initiatives including R Validation Hub and TransCelerate have published framework for qualifying R packages to be used in a regulatory setting (Nicholls, Bargo, & Sims, 2020) (Amoruccio, Lee, & Woodie, 2021). Our organization has been exploring the use of R in a regulatory setting for the past few years. A framework has been developed internally for qualifying external R packages that incorporates elements from both R Validation Hub and TransCelerate framework. This framework is currently being used to qualify both internally developed and externally sourced R packages for use in clinical trial A&R. In this document, we demonstrate this risk-based package qualification framework using the GGally R package. We provide the workflow as well as relevant details regarding the package qualification process used to qualify GGally as a moderate risk R package. We hope this inspires other organizations to use R in a regulatory setting as well as generate discussion to improve our existing framework.

Status Update May 2022

Fri, 20 May 2022 00:00:00 +0000

It’s time to bring another update to you on the current status of the R validation hub:

The riskmetric R package has been stable on CRAN. Recent work has focused on structuring “cohort metrics” – metrics that are conditioned on the package library or execution environment available to R. In R, package behaviors are often dependent on the rest of the R installation, and this new feature will help to make metrics more inspectible and reproducible, as well as allowing us to ask new questions like, “What would be the effect of installing a new package into an R environment?”. Within riskmetric, cohorts will represent our way of capturing this information to help make the context of each metric more explicit. Eric Milliman has been leading this effort, representing the next major feature enhancement for the riskmetric package.

R Package Risk Assessment at Novartis

Mon, 09 May 2022 00:00:00 +0000

1 INTRODUCTION

Whereas data validation is already a standard precursor to any form of scientific analysis in drug development and the validation of in-house built source code used to generate quantitative deliverables follows standard practices as well, the increasing popularity of open source programming languages like R in this context have created a new type of challenge: the validation of the R packages which are imported and used in the drug submission/ approval projects. Such packages are distributed freely, almost always without any warranties, and may be of varying quality. Therefore, Novartis has been working on defining a package risk-based validation approach qualifying R packages. Its risk assessment was designed based on the two business use cases, which reflect current business activities.

Automated R Package Validation at Roche

Fri, 22 Apr 2022 00:00:00 +0000

This case study walks through the automated R package validation process at Roche that utilizes a human-in-the-middle component to reconcile any gaps that arise in the automated metadata checks. The approach balances automation with risk mitigation and encourages in-house package development and iteration by introducing transparency to the validation process. The result reinforces best practices in R programming and package development while ensuring high package quality for use within a regulatory environment.

Risk Assessment of R Packages at Merck KGaA/EMD Serono

Mon, 14 Feb 2022 00:00:00 +0000

Introduction

Like many other companies, Merck KGaA/EMD Serono has embarked on their journey to enable the use R for regulatory submissions. Following the framework introduced by the R validation hub (Nicholls et al., 2020), we started to develop an algorithm to qualify a CRAN package as a Merck standard package in our GxP environment. In a nutshell: Given the R Foundation’s effort to ensure the validity of base and recommended R packages, these packages are classified as level 1. If an additional R package passes the installation qualification and successfully executes available tests, the package will be made available to the user and (temporarily) classified as level 3 package. Then, an automated risk assessment of R packages is performed based on the test coverage score (more is better) and the riskmetric score generated from the meta-information (smaller is better). If pre-defined thresholds are fulfilled, the package is qualified as Merck standard package (i.e., promoted to level 2), otherwise an explicit (manual) risk assessment is needed. This 3-tier model provides a useful framework for the users to define a risk-based quality control of outputs when using R. In this document, we introduce our pathway to a risk-based assessment of R packages at Merck. We provide relevant details on the statistical analysis which led to the definition of thresholds supporting a robust classification of CRAN packages as Merck standard packages. We want to inspire other companies and seek feedback from the community.

Risk Assessment App Update

Thu, 03 Feb 2022 00:00:00 +0000

This is a re-post. The original post from the R Consortium can be found here

Recent update by Marly Gotti on the Risk Assessment Shiny App. Marly is an executive committee member of the R Validation Hub where she advocates for the use of R within a biopharmaceutical regulatory setting, and Senior Data Scientist at Biogen.

The Risk Assessment App is an interactive web application serving as a front end application for the riskmetric R package. riskmetric is a framework to quantify risk by assessing a number of metrics meant to evaluate development best practices, code documentation, community engagement, and development sustainability. The app and riskmetric aim to provide some context for validation within regulated industries.

Some Considerations on Trusted Resources

Fri, 07 Jan 2022 00:00:00 +0000

There is a large variety of contributed R packages, which can be overwhelming when performing their accuracy assessment. These packages can be developed by anyone and may differ in accuracy. The white paper mentions the possibility to define “trusted resources” to simplify the assessment for some of the R packages.

The idea follows vendor assessments / audits to explore the internal validation practices of the vendor for proprietary software. For open-source software such audits are not logistically feasible. However, based on information available in the open-source domain, it may still be possible to perform a virtual audit of a vendor and their practices. In this context, we encourage the publication of software development life cycles (SDLC) documents, which support the process of risk assessment and provide evidence of software trustworthiness.

Status Update: R Validation Hub at R/Pharma 2021

Tue, 19 Oct 2021 00:00:00 +0000

Our favorite meeting of the year is approaching: R/Pharma 2021 will be held virtually November 2-4th, 2021. Also watch out for workshops the week before. Since the R validation Hub is closely associated with R/Pharma, we would like to highlight some of the presentations inspired by the R validation Hub, partner initiatives, or generally related to the topic of validation.

Tuesday: November 2, 2021

11:00am “R Package Validation at Roche” by Coline Zeballos, Roche
11:20am “Statistical Analysis and Pathway to a Risk-based Assessment of R packages at Merck KGaA/EMD Serono” by Juliane Manitz, EMD Serono
1:50 PM “Panel Discussion – Validation” with representatives of the R validation Hub”

Wednesday: November 3, 2021

Status Update: MSA Framework by TransCelerate

Thu, 20 May 2021 00:00:00 +0000

TransCelerate has published “Modernization of statistical analytics (MSA) Framework”. With goals similar to the R Validation Hub, the TransCelerate MSA framework seeks to demonstrate software reliability by establishing principles of accuracy, traceability, and reproducibility for a modern analytical software environment.

The MSA framework is centered around risk-assessment and mitigation practices to demonstrate reliability of software. This framework suggests assessing the accuracy of a software library via a confidence measure built on risk metrics such as published source code, issue management, usage, maturity, etc. If confidence does not meet the highest standards, additional testing is recommended. In addition, the intended use of a particular software library and the impact to the broader business outcome determines the requirement for additional testing. Altogether, the MSA framework is in line with the suggestions published in the R Validation Hub 2020 white paper for R. While the TransCelerate authors suggest their principles apply to a broad range of software, e.g. SAS, R, Python, Julia, etc, they do not provide specific suggestions for the implementation of their framework. The R Validation Hub can support the implementation of MSA inspired features with the R package riskmetric and the respective shiny app.

Status Update: CRAN Release of `riskmetric`

Fri, 02 Apr 2021 00:00:00 +0000

We have reached a major milestone. The R package riskmetric has been released and is now available on CRAN.

What is riskmetric?

riskmetric is a collection of risk metrics to evaluate the quality of R packages following the framework suggested by the R validation hub (see our white paper for details). Various quality metrics are provided which evaluate best practices of software development, code documentation, community engagement and development sustainability. This package serves as a starting point for exploring the heterogeneity of code quality, and begin a broader conversation about the validation of R packages.

Status Update: Conferences 2021

Fri, 12 Mar 2021 00:00:00 +0000

The new year is starting with some good news for the R validation hub. We will be able to present our work at a number of different meetings in 2021:

National Institute of Statistical Sciences (NISS; April 21, 2021): NISS-Merck Meet-Up on Open Source Software in Pharma
Annual Meeting of Statisticians in the Pharmaceutical Industry (PSI; June 21-23, 2021): Contributed session accepted
Joint Statistical Meetings (JSM; August 7-12, 2021): Topic-contributed session accepted

Status Update: A summary of 2020

Wed, 13 Jan 2021 00:00:00 +0000

2020 was a busy year for the R Validation Hub. We released our white paper describing our current thinking on a risk based approach to using R for regulatory work. We started to support the implementation of the white paper with tools such as riskmetric and our risk assessment application. And we started a new sub-team with the aim of producing a follow-up white paper on testing. Throughout, we have continued to share and gain feedback on our proposed approach, presenting at User!; running a workshop at R/Pharma; and speaking at an EU Programming Heads meeting in June.

Risk Assessment Application

Wed, 05 Aug 2020 00:00:00 +0000

Background

Towards the end of 2019, the R Validation Hub received an additional grant from the R Consortium to progress the next phase of our road map and produce a risk assessment app to complement the riskmetric package. In early 2020, Fission Labs were selected as our partner to build the first iteration of the application.

Fission Labs is a software product development services company delivering product life-cycle management and high-end scalable technology solutions. Our mission is to deliver complex products through simple and efficient engineering.

Status Update July 2020

Mon, 20 Jul 2020 00:00:00 +0000

It’s time to bring some updates to you on the current status of the R validation hub and we are have plenty of great developments.

Fission has finished their work on the R package risk assessment app and made the source code is available on github. It is an interactive web application providing a front end for the collection of metrics for R packages via riskmetric package including visualizations and comparison metrics.

Status Update May 2020

Thu, 07 May 2020 00:00:00 +0000

In these uncertain times, we would like to provide you with some good news and update you on the progress from the R validation hub. It has been a little while since you heard from us, but that doesn’t mean we were less active.

Communication: You may have realized that there are no further re-occurring meetings scheduled for the R validation hub since February. Following previous discussions, we are planning regular update releases via the website and newsletters instead. Meetings for the validation hub will be scheduled less frequently as needed. However, we are dependent on your input and continue to encourage volunteers to collaborate on all the different parts of the project.

A Risk-based Approach for Assessing R package Accuracy within a Validated Infrastructure: White Paper Summary

Thu, 30 Jan 2020 00:00:00 +0000

In this article I summarise some of the key themes from the R Validation Hub’s white paper, A Risk-based Approach for Assessing R package Accuracy within a Validated Infrastructure.

What is the R Validation Hub?

The R Validation Hub is a cross-industry initiative whose mission is to enable the use of R by the Bio-Pharmaceutical Industry in a regulatory setting, where the output may be used in submissions to regulatory agencies. The group was initially formed in 2018 by members of PSI’s AIMS SIG but has now expanded to include just under 100 members from multiple organisations across the pharmaceutical sector.

Mon, 01 Jan 0001 00:00:00 +0000

About Community Meetings

Our community meetings are an initiative we began in 2023 to help foster stronger cross-industry connections for those working in biopharma and champion the use of R in the industry. Each meeting centers around a particular discussion topic, such as approaches for assessing R packages’ risk, updates about our R Validation Hub tools, or implementation of GxP R environments within companies.

When are Community Meetings?

Traditionally, these meetings take place every three months. For updates on when the next community meeting is taking place, join our mailing list or watch out for announcements of the meetings on the R Consortium LinkedIn page. As this initiative gains greater momentum, the goal is to make these meetings more frequent.

Mon, 01 Jan 0001 00:00:00 +0000

Our Structure

The R Validation Hub champions a level structure, comprised of many workstreams that contribute toward their focus areas and activities. Find each workstream’s duties below as well as the members that support them.

If you wish to contribute to any of our efforts, please visit our Contribute page.

Strategic Workstream

The Strategic Workstream has the responsibility of aligning on longer-term strategic goals. The intention is that this workstream is to become a floating set of representatives from each of our workstreams.

Mon, 01 Jan 0001 00:00:00 +0000

The R Validation Hub operates in the public domain and all meeting minutes are made available here. Workstreams meet routinely and compile their minutes at their respective links below.

Mon, 01 Jan 0001 00:00:00 +0000

The R Validation Hub is comprised of participants from across the pharmaceutical industry. Participants contribute to the effort through our regular group meetings, as well as supporting the various workstreams that make up the project. Feel free to explore contribution opportunities and subscribe to our mailing list to stay up-to-date on our progress.

If you are a member and your organization does not appear on this list, please let us know!

Mon, 01 Jan 0001 00:00:00 +0000

Many contributed R packages lack documentation expected in software qualification, which is required within pharma and other regulated industries. For pharma, there are various regulations, which require documentation that demonstrates software is used appropriately and works as expected. Thus, industry needs to establish appropriate requirements for R packages using selected metadata and useful risk metrics.

In context of the R Validation Hub, the R package riskmetric has been developed, which seeks to take the first steps in identifying metrics and best practices to quantify the quality of R packages. It provides a framework for retrieving package metadata, assessing package metrics, and summarizing the risk that the package might not provide accurate results. A corresponding Shiny app, that can be used to generate package reports using riskmetric, is under development.

Mon, 01 Jan 0001 00:00:00 +0000

Background

rOpenSci is an organization devoted to “transforming science through data, software and reproducibility.” One of rOpenSci’s focal activities is peer review of R packages, historically focusing on packages that cover the data management lifecycle. This has historically excluded software implementing statistical methods, for which standards and review require addressing a different set of challenges. This year, we have begun tackling these so as to expand our peer review system to explicitly encompass statistical software, under project funded by the Alfred P. Sloan Foundation.

2022 All-hands: Case Studies

Mon, 01 Jan 0001 00:00:00 +0000

Written Case Studies

Written case studies are being developed and peer reviewed on GitHub. As per the README, “The repository contains case studies contributed from companies and individuals, who are implementing a risk-based approach to validate R packages. Feedback in form of community reviews and comments are encouraged. If you want to contribute your own case study, please create a pull request. New contributions will be highlighted in the R validation hub mailing list.”

A Risk-based Approach for Assessing R package Accuracy within a Validated Infrastructure

Mon, 01 Jan 0001 00:00:00 +0000

Andy Nicholls, Statistics Director, Head of Statistical Data Sciences, GSK

Paulo R. Bargo, Director Scientific Computing, Statistics & Decision Sciences, Janssen R&D

John Sims, Director, Analytical Systems Architect & Data Science - Pfizer Vaccine Research

On behalf of the R Validation Hub, an R Consortium-funded ISC Working Group

January 23, 2020

View and/or download the PDF version of this white paper here.

About Us

Mon, 01 Jan 0001 00:00:00 +0000

Our Mission

R Validation Hub is a collaboration to support the adoption of R within a biopharmaceutical regulatory setting

The R Validation Hub is supported by the R Consortium and works closely with many neighboring efforts in the pharmaceutical space and broader R ecosystem.

Executive Committee

R Validation Hub is governed by an executive committee. Here is the committee, as of August 2023:

Doug Kelkhoff (Chair)
Joe Rickert (R Consortium)
Preetham Palukuru (R Consortium)
Juliane Manitz (Communications Workstream Lead)
Coline Zeballos (Repositories Workstream Lead)
Eric Milliman (riskmetric Workstream Lead)
Aaron Clark (riskassessment App Workstream Lead)

Thank you to previous committee members for helping to drive the R Validation Hub forward:

Accomplishments

Mon, 01 Jan 0001 00:00:00 +0000

Contact Us

Mon, 01 Jan 0001 00:00:00 +0000

Stay up-to-date and join our mailing list

For occasional news updates, you can join our mailing list. This will enable you and your colleagues to receive notifications of key updates/software releases, blog posts and invites to all-hands Hub meetings.

Interested in getting more directly involved?

If you would like to contribute to this collaborative effort please email us directly.

Contribute

Mon, 01 Jan 0001 00:00:00 +0000

Join Our Mailing List

For occasional news updates, you can join our mailing list. This will enable you and your colleagues to receive notifications of key updates/software releases, our latest news and blog posts, and invites to all-hands Hub meetings.

Once you’ve joined, you can subscribe to our calendar to keep tabs on all of the R Validation Hub’s all-hands meetings.

Contribute to a Working Group

If you are willing and able to commit more time to the R Validation Hub’s mission, then there are several options depending on your skills and interests. If you do not feel you have the necessary technical skills to contribute then please don’t let that stop you from getting in touch and sharing ideas to support the implementation of a risk-based R package assessment.

Example Package Assessments

Mon, 01 Jan 0001 00:00:00 +0000

Welcome!

Here, we aim to illustrate examples of different validation framework implementations that are used to interpret metrics considered when assessing R packages. Any examples are intended only as a guide to consider how your organization might adopt a solution to the “validation problem”.

Get Started with risk-based software

Looking for curated examples on how package metrics might be assessed and interpreted using {riskmetric}? We’d suggest reviewing the Get Started vignettes for either {riskmetric} or the {riskassessment} application. Both take the user on a highly informative package assessment journey using their respective software.

General Guidances

Mon, 01 Jan 0001 00:00:00 +0000

The Accuracy of R Packages

When assessing the accuracy of R packages, the R Validation Hub differentiates R packages by the following types (see German et al, 2013¹):

base and recommended (core) packages - developed by the R Foundation and shipped with the basic installation
contributed (open source) packages - developed by anyone, and may differ in popularity and accuracy

{alt=“source: German et al (2013): The Evolution of the R Software Ecosystem”}

Glossary

Mon, 01 Jan 0001 00:00:00 +0000

Term	Definition
Open Source Software	Software or language designed for collaboration that has source code available for community review, usage, and modification
System Qualification	The process including the execution of Installation, Operational and Performance tests on a system that ensures the reproducibility of expected results.
Validation	Per the FDA’s Glossary of Computer System Software Development Terminology -“Establishing documented evidence which provides a high degree of assurance (accuracy) that a specific process consistently (reproducibility) produces a product meeting its predetermined specifications (traceability) and quality attributes.”

Looking Forward

Mon, 01 Jan 0001 00:00:00 +0000

Meeting Minutes

Mon, 01 Jan 0001 00:00:00 +0000

The R Validation Hub operates in the public domain and all meeting minutes are made available here.

2022 All-hands: Case Studies

For the 2022 case studies, please see here

Historical minutes

Our Origins & Mission

Mon, 01 Jan 0001 00:00:00 +0000

Origins

The R Validation Hub was formed in 2018 by the PSI AIMS Special Interest Group and is supported by the R Consortium. We work closely with many neighboring efforts in the pharmaceutical space and broader R ecosystem.

Mission

Our mission is to leverage the open-source and collaborative nature of R while supporting its adoption within the biopharmaceutical setting.

Since our inception, we have worked on developing several resources in hopes of their utilization industry-wide. Our solutions for the validation of R packages assist your work by quantifying the “risk” of R packages with several meaningful metrics (see {riskmetric}) and providing a user-friendly, full-fledged R Shiny app as a central hub to gauge the “risk” of packages for your organization (see {riskassessment}). These solutions are useful for a variety of roles, like individual contributors who are curious about a package’s risk or an organization’s administrator(s) for Posit Package Manager, for example.

Partner Initiatives

Mon, 01 Jan 0001 00:00:00 +0000

We have partner initiatives with three main entities: the R Consortium, PHUSE, and Statisticians in the Pharmaceutical Industry (PSI).

The R Validation Hub is an R Consortium Working Group. The R Consortium supports several related working groups. These working groups have a similar overall objective (to support the use of R within the biopharmaceutical industry) but the working groups are complementary: each has its own distinct deliverables; and many of the R Validation Hub’s members are also members of the other working groups.

Presentations

Mon, 01 Jan 0001 00:00:00 +0000

2024

May
- PharmaSUG: Validating R for Pharma, Anuja Das

2023

October
- R/Pharma: Updates from the R Validation Hub: Toward a Pharma Repository, Juliane Manitz, Coline Zeballos
September
- posit::conf(2023): R Validation Hub Status Report and Workshop, Doug Kelkhoff
June
- AZ R Conference: It’s time to integrate the R into PhaRma, Lyn Taylor
March
- ShinyConf 2023: {riskassessment} for R-Package Validation, Aaron Clark
- R Adoption Series: Learnings and Reflection from RvalHub Case Studies (recording), Juliane Manitz, Doug Kelkhoff, Uday Preetham Palakuru, Eric Milliman

2022

June
- useR! Conference: Implementation of Risk Assessment for R Packages: Learnings and Reflection

2021

November

R

Mon, 01 Jan 0001 00:00:00 +0000

R is a free programming language and software environment that is used for statistical computing. R was first created in 1993, and is now supported by the R Foundation for Statistical Computing.

For more information, see What is R?

Base R

As of May 31, 2019, Base R is comprised of 14 packages, with an additional 15 packages included as “Recommended Packages.” These 29 packages compose the core of R and are included with basic R installations. Functionalities of this basic installation include:

Regulations

Mon, 01 Jan 0001 00:00:00 +0000

Regulations Overview

Key guidance documents are provided by different regulatory entities, which oversee activities within their respective domains:

International Conference on Harmonisation of Technical Requirements for Registration of Pharmaceuticals in Human Use (ICH)
The United States Food and Drug Administration (FDA)
European Medicines Agency (EMA)
Japanese Pharmaceuticals and Medical Devices Agency (PMDA)

ICH E9 Statistical Principles for Clinical Trials

At the international level, ICH provides regulatory guidance for the pharmaceutical and medical devices industry. In the ICH E9 guidance on Statistical Principles for Clinical Trials. Regarding “Integrity of Data and Computer Software Validity”, it states:

Regulatory Metrics Repository

Mon, 01 Jan 0001 00:00:00 +0000

An open solution to regulatory quality

Every regulatory submission requires evidence that the software used meets quality standards appropriate for the type of work. The R Validation Hub proposed standards for this process in our 2020 White Paper.

Risk Assessment

Mon, 01 Jan 0001 00:00:00 +0000

The R Validation Hub has develops & maintains useful tools that make the risk assessment approach proposed in our published white paper much easier to adopt for R packages.

Open-source tools

Though these tools likely won’t encapsulate every aspect of your organization’s end-to-end validation pipeline, we are constantly seeking to fill known gaps in the process. Thanks to partnerships with a sleuth of pharma organizations, these tools were designed to leverage industry consensus and provide flexibility for customization when needed. We’re proud that both {riskmetric} and {riskassessment} have claimed membership in the {pharmaverse} suite of packages.

Roadmap

Mon, 01 Jan 0001 00:00:00 +0000

Process and Communication

Publish website [Complete]
Agree high level process [Complete except for finalising diagram]
Develop white paper [Target: October]
Ensure content reflected accurately in website
Tools at pilot implementation stage [August 20th]

Validation / Qualification Suite

Release metrics package
Release Shiny app and reporting tools for risk assessment [Target: January/Q1]
Share test suite
Build test execution tool
Provide an example deployment
Agree criteria used to determine whether a package is fit for the Pharma Repository

Repository

Build Pharma R Repository

The {riskassessment} App

Mon, 01 Jan 0001 00:00:00 +0000

About the App

The app’s main goal is to help those making “package inclusion” requests for validated GxP environments. So, the highest and best of {riskassessment} revolves around two things:

Empower members of your organization to embrace their responsibility to assess package risk themselves, prior to making uninformed IT requests like: “please add package xyz to our validated environment”.
Establish guide rails that adopt to your organizations validation strategy and use of {riskmetric} which culminates in a report for IT that summarizes each package’s adherence to those inclusion requirements.

The {riskmetric} Package

Mon, 01 Jan 0001 00:00:00 +0000

About the Package

Contributed R packages are developed by anyone & everyone, and may differ in popularity and accuracy. As such, the R Validation Hub developed an R package titled riskmetric whose goal is to assess the risk of contributed R packages.

{riskmetric} has four groups of metric criteria:

Unit testing metrics - includes unit test coverage and composite coverage of dependencies
Documentation metrics - availability of vignettes, news tracking, example(s) and return object description for exported functions
Community engagement - number of downloads, availability of the code in a public repository, formal bug tracking and user interaction
Maintainability and reuse - number of active contributors, author / maintainer contacts, and type of license

Note: Even though the quality of software is sometimes measurable, sometimes it is not. For example, assessing the accuracy of a contributed open-source R package should be done outside of {riskmetric}. The term accuracy refers to the risk of an error in the code that, when used, could lead to an incorrect calculation. This incorrect calculation may lead to an incorrect decision during data analysis. The relative impact of an error should be determined by the individual organisation. Thus, impact is not a part of the risk assessment performed by {riskmetric}.

The {riskscore} Package

Mon, 01 Jan 0001 00:00:00 +0000

A data package for cataloging riskmetric results across public repositories.

WARNING: Right now, the {riskscore} is in a PoC stage that is not fully operational. With that said, there are several use cases that make the concept of {riskscore} valuable, including (but not limited to) the following: it …

Guides more effective discussion around how to summarize risk
Helps communicate changes to {riskmetric}’s summarizing algorithm or interpretations of assessment data
Aids the {riskmetric} dev team in identifying “edge cases” for analysis and code refinement.
Provides a channel to distribute handy tools for building {riskmetric} result data (ie, mimicking how our process for external packages could serve as a useful template for when comparing to internal or private repos).
Allows everyone to report risk scores in terms of a “CRAN percentile” instead of just some arbitrary numeric value.
Establishes a central repository for package scores, which can be used for many applications, like generating badge scores or trending in a package’s score over time to measure performance.

Are you interested in supporting package development?

We could always use extra help/feedback! Please consider one of the following options:

Validation Overview

Mon, 01 Jan 0001 00:00:00 +0000

For statistical analyses, the process of validation ensures quality output.

According to the FDA’s Glossary of Computer System Software Development Terminology:

Validation: Establishing documented evidence which provides a high degree of assurance (accuracy) that a specific process consistently (reproducibility) produces a product meeting its predetermined specifications (traceability) and quality attributes.

Since the FDA does not require use of any specific software for statistical analyses, the programming language R can be used if the R installation incorporates all of the following elements:

PharmaR

{riskassessment} ShinyGatherings x Pharmaverse Workshop

About the Workshop

Risk Assessment App v3.1.1

Updates to the R Validation Hub Executive Team

Risk Assessment App v2.0.0

{riskassessment} App voted best Shiny app at shinyConf 2023! 🎉

Summary of 2022 Case Studies

ASA Biopharmaceutical report, Fall 2022

External R Package Qualification Implementation at Merck

Status Update May 2022

R Package Risk Assessment at Novartis

Automated R Package Validation at Roche

Risk Assessment of R Packages at Merck KGaA/EMD Serono

Risk Assessment App Update

Some Considerations on Trusted Resources

Status Update: R Validation Hub at R/Pharma 2021

Status Update: MSA Framework by TransCelerate

Status Update: CRAN Release of `riskmetric`

What is riskmetric?

Status Update: Conferences 2021

Status Update: A summary of 2020

Risk Assessment Application

Background

Status Update July 2020

Status Update May 2020

A Risk-based Approach for Assessing R package Accuracy within a Validated Infrastructure: White Paper Summary

What is the R Validation Hub?

About Community Meetings

When are Community Meetings?

Our Structure

Strategic Workstream

Executive Meeting Minutes

{riskassessment} Application Workstream Minutes

Communications Workstream Minutes

Background

2022 All-hands: Case Studies

Written Case Studies

A Risk-based Approach for Assessing R package Accuracy within a Validated Infrastructure

About Us

Our Mission

Executive Committee

Accomplishments

Contact Us

Stay up-to-date and join our mailing list

Interested in getting more directly involved?

Contribute

Join Our Mailing List

Contribute to a Working Group

Example Package Assessments

Get Started with risk-based software

General Guidances

The Accuracy of R Packages

Glossary

Looking Forward

Meeting Minutes

2022 All-hands: Case Studies

Historical minutes

Our Origins & Mission

Origins

Mission

Partner Initiatives

Presentations

2024

2023

2022

2021

R

Base R

Regulations

Regulations Overview

ICH E9 Statistical Principles for Clinical Trials

Regulatory Metrics Repository

An open solution to regulatory quality

Risk Assessment

Open-source tools

Roadmap

Process and Communication

Validation / Qualification Suite

Repository

`{riskassessment}` Application Workstream Minutes