At the recent INTERPHEX show in New York, Germany-based software developer eschbach GmbH introduced io.Equipment, a new digital equipment log book platform using blockchain technology designed to address the evolving regulations for tracking and tracing along the entire pharmaceutical drug supply chain.
All equipment operations, such as cleaning, maintenance, breakdown, and manufacturing, can be recorded in a digital log book, ensuring compliance with all related regulatory requirements. To make the data tamper proof, each log book record receives a timestamp by sending a cryptographic hash—or ‘fingerprint’—of the record to a distributed blockchain ledger. Using the hash methodology, all records are kept strictly confidential, while still providing proof that the data actually exists.
Among Big Pharma companies, Sanofi already is using the new io.Equipment audit log book platform.
Following a meeting at INTERPHEX with Veit Hora, head of product management and marketing at eschbach GmbH, Pharmaceutical Processing spoke with the company’s founder and CEO Andreas Eschbach to learn more about the company and its new io.Equipment audit package.
Q: Please tell us about the company, when it was founded, and how it has evolved since its early days.
Andreas Eschbach: Our company was founded in 2005, and we have been specializing in software development for transparent communication and documentation for pharmaceutical, chemical, and food manufacturing ever since. Our roots are in the very southwest of Germany, an area where historically there is a lot of pharmaceutical manufacturing. Today our solutions are in use at major players such as Bayer, Sanofi, and DuPont and we have implementations for more than 30,000 active users around the globe. We also recently opened a U.S. office to address our increasing growth in the North American market.
Q: At the recent INTERPHEX show in New York, eschbach introduced a new io.Equipment digital equipment log book solution using blockchain technology. Please describe it. How is it superior to other audit programs on the market?
Eschbach: io.Equipment is designed specifically for pharmaceutical manufacturing shop floors. All equipment operations—such as cleaning, maintenance, breakdown, or manufacturing use—can be recorded in a digital log book, ensuring compliance with all related regulatory requirements. The solution is browser-based, incredibly easy to use, and can also run on tablet computers in the cleanroom. Additionally, it interfaces with other mission critical systems, such as process historians and Computerized Maintenance Management Systems (CMMS) to automatically create log book entries. Above all, io.Equipment features an optional blockchain-enabled audit trail for each log book entry.
Q: Can you tell us the advantage of blockchain in this application and what benefit it provides specifically for the pharmaceutical community?
Eschbach: Essentially, the blockchain integration gives our customers the freedom to prove the authenticity of their records. Today, when records are stored according to FDA CFR 21 part 11 in any system, this is typically done on a self-regulated basis within the corporation’s local IT infrastructure. While it is extremely expensive to maintain these systems and keep them secure, it often doesn’t take more than a basic administrator user account with access to a certain database to go and manipulate records. Eventually, and especially with distributed ledger technologies like blockchain on the rise, people and authorities will lose trust in data based on self-imposed security measures. At eschbach, we strongly believe that any tamperproof record will sooner or later need some type of certificate from a distributed ledger which is outside the corporation’s circle of influence. Also, with regards to privacy requirements in the pharmaceutical community, it is important to note that our blockchain integration keeps all data locked behind the corporation’s firewall. We are only using a fingerprint, a so-called hash value of the actual data, to create timestamps in the distributed ledger.
Q: What areas of the typical manufacturing operation are prime candidates to potentially benefit from the new log book audit platform?
Eschbach: Replacing old systems for FDA CFR 21 Sec. 211.182 equipment cleaning and use log with io.Equipment is a great first step to benefit from blockchain-secured tamperproof records. The system is easy to set up and can be implemented without much impact on the existing solutions ecosystem and IT infrastructure. In the near future, we expect to see our clients using our solution for storing a number of additional manufacturing-related types of records such as sensor data from connected systems, lab readings, or even supply chain information.
Q: Your Shiftconnector software platform already provides an overview of all events on the shop floor, as well as a digital equipment log to track manufacturing usage and ensure compliance with FDA 21 CFR 11. Does io.Equipment bring a new level of security to the existing Shiftconnector platform?
Eschbach: Our Shiftconnector platform already is known for superior data integrity. However, both io.Equipment and Shiftconnector share the same technological core, which makes the blockchain option also available for existing Shiftconnector users. This means that any log book record in Shiftconnector also can be secured with a timestamped hash in the blockchain. Existing Shiftconnector users will be able to use the blockchain option when they upgrade to a newer version that includes the blockchain feature.
Q: Does it essentially replace manual paper audit trails produced by various staff in different sections of the operation? Does that significantly cut down on the potential for human errors or even data theft from occurring?
Eschbach: Yes, absolutely. We estimate that in the U.S. alone, there are more than half a million paper log books actively used in the pharmaceutical industry. With all of these books manually updated, there is considerable potential for human error. Just imagine a new product campaign being started that requires creation of the same record for a dozen pieces of equipment, log book after log book. With a digital solution, all these entries can be created though one single record. Also, doing research or documentation cross checks becomes much easier with an electronic repository. Ultimately, there no longer is any need to walk to a book shelf, everything can be done from the desk or directly in the cleanroom. The bottom line here is that the maturity of the documentation is taken to a new level.
Q: According to Bela Gipp, Professor at the University of Constance and Scientific Advisor to eschbach, by using blockchain-based timestamping, manipulating data will essentially become impossible. In the event of an error, how are corrections noted?
Eschbach: Users can still change events for corrections if they are authorized. However, all those changes are recorded with the audit trail feature, which stores an original of every version of the record. Every one of those versions has its own hash fingerprint timestamp in the blockchain ledger. This architecture allows for corrections but completely prevents any manipulation of the original version of the record in the audit trail.
Q: Is the new digital equipment log book that uses blockchain compatible with all types of legacy systems, or only with those using eschbach software systems?
Eschbach: Typically, a legacy system would be a bookshelf full of paper log books. Customers can seamlessly transition to our digital solution and retire their existing paper-based solution. We designed io.Equipment as Web-based software built on Microsoft .NET technology. The platform also features a flexible API as well as an extensible data model. Generally speaking, these technologies—as well as the architecture of io.Equipment—make the system very compatible for implementations in various environments.
Q: Once all of the data from the different units are collected, how is it compiled? Is all of the info shared throughout the enterprise, or are there different levels of access depending upon the role of the individual user?
Eschbach: At the core of io.Equipment is a very flexible configuration backend. It lets users not only define different rights of access levels, but also different nodes with specific rules for data aggregation. Different roles and hierarchy levels can have different views of the data, according to their needs. While a QA person, for instance, could be more interested in releasing a batch and analyzing the data by product, a process engineer is more likely to be interested in the breakdowns by equipment.
Q: Data never is stored in blockchain. Why is that? Where then is the data stored? Doesn’t that open a different type of security threat?
Eschbach: It is very important to understand that io.Equipment will never store any sensitive data to the distributed blockchain ledger. We are only sending hash values of each record for identification purposes. Actual log book data is never shared; it stays solely in the io.Equipment database, behind a user’s firewall.
Q: Please explain the ‘hash’ system. What is it and how does it work as part of the overall blockchain strategy?
Eschbach: A cryptographic hash can be compared to a fingerprint which we are using to build the next generation audit trail. We are calculating a so-called SHA-256 hash for each original record in the log book. This hash always has a fixed amount of characters which do not contain any original data and cannot be reversed to the original data. Once this fingerprint is sent to the blockchain, it cannot be altered. However, if someone attempts to manipulate the logbook data and create a hash fingerprint from that data, it will certainly be different from the record which is stored in the blockchain. By comparing the hashes of the locally stored records to the hashes in the blockchain for the same records, we can determine whether a record was manipulated or is still in original state.
Q: io.Equipment is browser based. What are the advantages of that approach?
Eschbach: When we first turned to browser-based solutions more than 10 years ago, this approach was pretty new and often clients would prefer locally installed software over running a tool in the web browser. Today, the advantages of web technology are greatly appreciated everywhere. More importantly, every authorized user can easily access the application from any computer or mobile device connected to the Internet or an intranet. Also, new installations or updates can be deployed with minimum effort and rapid turnaround time to thousands of clients in one swoop. Ultimately, the total life cycle costs are much less when choosing a web-based solution.
Q: Does the new system audit only a company’s internal operations, or can it be used to augment a client’s serialization and track-and-trace efforts throughout the entire pharmaceutical supply chain from drug development through to dispensing?
Eschbach: In general, io.Equipment has an open API to interface with external connected systems. For example, we can link blockchain-secured data of equipment usage with a certain batch ID. From our experience, it can be beneficial for a customer’s serialization strategy to integrate io.Equipment log data not only from their own manufacturing facilities but also from their contract manufacturers.
Q: eschbach lists several large pharmaceutical clients, including Bayer, Roche, and Sanofi. Without necessarily identifying any particular company, have existing clients already expressed interest in adding io.Equipment going forward in their operations?
Eschbach: Yes, we are happy to say the appeal of io.Equipment is resonating strongly with our customers. For example, we are proud to name Sanofi as a user of io.Equipment. Generally speaking, many shop floors still use paper-based solutions today. Changing to a digital system, which can also be used on tablets in the cleanroom, offers very clear advantages. For starters, it lowers documentation efforts significantly while simultaneously increasing the documentation quality. Switching from paper logs to io.Equipment is truly a ‘low hanging fruit’ choice in the field of digitization. We’ve also received great feedback from our clients regarding the scalability of the solution, as the architecture allows for an efficient global rollout.
Q: The primary focus is for the pharmaceutical and chemical industries. Why those particular industries? Are there plans to expand more broadly into other industrial communities?
Eschbach: Our company originally started developing our shift log software for pharmaceutical and chemical manufacturing operations. The very first customer for our software was a vitamin manufacturing plant in Switzerland. We always have been close to our customers on the manufacturing level and we iterated the design of our features working with the actual users of the tool. This approach and legacy makes pharmaceutical and chemical our main focus industries. That being said, we have a good fit for the food, energy, and semiconductor sectors where we also have a vigorously growing client base.
Q: Was this the first year that eschbach exhibited at INTERPHEX? What made you choose this particular forum? Was it successful for you?
Eschbach: Yes, this was our first time as an exhibitor at INTERPHEX. We are constantly scanning the market for events that match our company’s strategic approach to the market. INTERPHEX stood out because some of our clients suggested that we’d be a great fit as an exhibitor at the show. We are delighted with our results from the conversations we had with potential clients as well as the potential partnership opportunities that were discussed during the show.
Q: Can you give us a hint as to what you might be working on, possibly in time for next year’s INTERPHEX?
Eschbach: We are not able to reveal the details of our roadmap at this time, but we can share that eschbach has a history of rapid innovation and customer collaboration. You can be sure that we will further develop features in close cooperation with our clients for our entire product portfolio, including Shiftconnector, io.Equipment, and io.Performance OEE.
Q: Are there any other features of the new audit solution with blockchain that you would like to address before we conclude?
Eschbach: When comparing io.Equipment with the many other visionary blockchain products available, we truly believe we can offer a tangible, relevant, and down-to-earth benefit by using blockchain technology right now. For many customers, we realize it’s difficult to implement new technologies into their environment due to an understandable strategy of being cautious and considering all options before adopting it. By implementing io.Equipment, we are convinced that we can deliver a solid digital equipment use log to existing and potential clients, and that we might very well be the first fully operational blockchain solution on the shop floors.
‘Bitcoin is not blockchain, just like Facebook is not the Internet’
Eliminating Risk
As noted above, Eschbach estimates that there are more than half a million paper log books in use in the U.S. pharmaceutical industry today, an often time-consuming exercise requiring multiple operating and supervisory staff to crosscheck the same documents from separate log books compiled on each individual equipment activity and performance.
Another feature of the new digital equipment log book, according to the company, is the ability to access and input via tablet directly within the cleanroom environment, significantly reducing the time it otherwise would take to exit and reenter the cleanroom and standard shop floor.
Since data is stored in internal company systems—not in blockchain—the risk of information tampering or theft through blockchain is eliminated. That said, blockchain does serve to timestamp and certify that a data “campaign” was initiated at a specific date and time allowing companies to access the complete audit information from within their internal systems, which could prove valuable to prove regulatory compliance and in the event of legal challenges.
While many professionals in pharma are hearing about the auditing and security features afforded via blockchain, others may still confuse blockchain with the controversial and potentially risky bitcoin development in recent years.
Yes, bitcoin uses blockchain, but the ongoing controversy surrounds bitcoin itself, not the blockchain platform.
As Andreas Eschbach pointed out following the interview, “Bitcoin is not blockchain, just like Facebook is not the Internet.”
Along with the INTERPHEX launch of the io.Equipment log books using a blockchain audit trail, eschbach GmbH, headquartered in Bad Saeckingen, Germany, also recently opened a new subsidiary in Princeton, NJ to serve clients in the U.S. and Canada. The company plans to expand the U.S. operation, but a specific timetable has not yet been announced.
_________________________________________________________________
Andreas Eschbach heads eschbach GmbH, a company that develops software for interactive management, transparent communication, and business intelligence.