Integrating your applications creates efficiencies and liquifies your data between them. We’ve gained considerable experience over the last few years in how to do this properly - not all integrations are created equal! In this blog post, we share our three most important insights with you, by way of best practices for integrating your applications and avoiding process breakdowns and other disasters.
A wealth of process improvements, cost savings and elimination of less interesting, manual work, is on offer for those who integrate their applications properly.
The underlying, enabling technology is cloud computing. This has displaced the ‘savvy’ system administrator of the past, who had to invent bespoke interfaces or work-arounds for application integration issues. He was often the only one who fully understood how the various network elements hung together, and if he left… well, we’ve all heard that story a few too many times.
'Software as a Service (SaaS)' has become prevalent, and so have the various standards for data exchange between applications. Open APIs are a common example, which work in sync with secure authentication protocols. This eliminates the need to develop customized middleware to ship data from one application to the next, often adding unnecessary complexities and security risks in the process.
These standards facilitate a sustainable and secure approach to integrating applications. Cloud computing has provided the ability to separate physical infrastructure from everything else. This enables you to integrate applications and automate processes, whilst still using the best of breed solutions from different third parties. This means that you can choose the best IT solution for each specific task, while still ensuring all relevant data is shared and synchronized between those tools. The main advantage is that for these solutions, you are not responsible for maintenance or attaining a certain level of service - this is provided to you, as the customer.
Having said all that, there are still many residual risks - e.g. if cloud integrations aren’t built properly or cease working. In particular, if your key business processes rely on application integrations running successfully, you can find yourself in a world of pain if one of them breaks down.
We felt obliged to share our insights into the main risks and solutions for building durable, secure application integrations, and hope to be of assistance to anyone who is grappling with this. Whether you are developing a custom integration on your own, use another company’s product, or leverage Harmonizer’s service to build the integration, the following best practices are of utmost importance:
Security and data management
With cyber attacks on the rise and an ever increasing compliance burden due to new laws and regulations around data management and privacy, cyber security is front and center for most Boards. Of course, there is no one-size-fits-all solution, and any control measures should be commensurate to the risk they are meant to mitigate. Simply put, there is no point putting a $1000 fence around a $10 horse.
We can share the following insights in this regard:
- Infrastructure: An integration is, in reality, a piece of software designed to exchange data between two APIs. The integration will have always to be housed on a server somewhere, whether that is in the cloud or on a physical server on premise. Due to the integration potentially not being considered as a primary piece of software in terms of your operations, there is a risk that information security around this this server isn’t as strict as it should be. For stability and consistency, it is of utmost importance to bring information security controls in line with that of the integrated applications and servers these applications are housed on. Who has access to the server hosting the integration? Who administers the server? Is the server subject to regular information security reviews? These questions need to be asked, if they aren't already.
- Authentication: There are many different acceptable options for authentication. Security of the authentication mechanism is an obviously important first consideration. (For the tech-savvy audience: A better practice method in terms of security is OAuth 2.0, especially in combination with a very strong private key, for example a JWT token and a refresh token mechanism). Another important consideration is that the integration accesses two applications in order to operate - keep this in mind with your risk analysis in terms of the consequences of data from both applications being exfiltrated. To mitigate these risks, it’s important to use strong encryption methods and to ensure access can be revoked quickly in case something does go wrong, to limit any damage. In addition, if at all possible, it is good practice to limit the access to data needed for the operation of the integration to only those with a strict business need.
- Minimization of data storage: Integrations should pass-through, and possibly transform data along the way, but should not store any data permanently. If data is stored, there is an increased, yet minimisable, risk of data loss. In addition, privacy regulations, such as GDPR, require that organizations limit processing and storage of personally identifiable information to a minimum. Should it be absolutely necessary to store data for operating the integration, ensure the data is encrypted in transit and at rest, and securely deleted as soon as it is no longer required for the operations of the integration.
Often, for locally developed custom made integrations, we find there is no (effective) monitoring and logging system. This can often result in key business processes going off the rails and causing damages to the organization, including un-budgeted expenses to remediate, and a lot of work on top of peoples’ day jobs. Often, disproportionate amounts of funds are spent to get the integration operational again. However, the reasons for an integration failing are usually simple in nature. For example, one of the APIs of the integrated applications changed. Or perhaps the account that the integrations use has expired. To prevent integrations from breaking down in foreseeable scenarios, it’s important to think ahead, design monitoring around these scenarios and allocate roles and responsibilities appropriately.
Monitoring ensures that every error or issue with an integration is reported to a central location. The staff member in charge of monitoring is tasked with instigating a prompt investigation into the causes of the error or issue, and to fix it before serious process disruptions occur.
Monitoring has the added benefit of being able to execute automated checks & balances, and report on whether or not the data passing through the integration meets expectations. For example, the format can be checked for accuracy, and data attributes can be checked for completeness. In case something changes in the application the data originates from, intervention can happen directly rather than after issues are experienced by end-users of the application the integration points to. Another proactive check that can be added to the integration is to have the monitoring function look into whether all data has arrived in the correct format for the receiving application.
Since great-working integrations shouldn’t be ‘visible’, it’s easy to forget they need governance too. Who do you go to when something goes wrong with the integration? Who is responsible for instigating and authorising changes to the integration? Changes could be driven internally, e.g. because processes are updated, or externally, e.g. because APIs are changed. Responsibility needs to be allocated for either scenario. In the case of in-house, custom developed solutions, the engineer has often left the organization, or was a contractor/consultant in the first place - this is a risk. If you choose to develop your integration in-house, please consider the following in advance:
- Assign roles and responsibilities around functional administration of the integration. This doesn’t have to be a purely technical resource, but it could be someone who is closely involved with the integration as well. They would be the primary point of contact for any issues with the integration, and coordinate the solution.
- However, do appoint a responsible resource for maintaining the integration technically. This could be an internal, or external resource.
- Create a Service Level Agreement (SLA). Use this to document how elements of the integration will work in practice, such as availability, frequency of data calls, how change requests are dealt with, time to resolve issues, and the process around dealing with issues.
Integration as a Service
As discussed above - setting up an integration requires more than shipping data from A to B. The right expertise and resources need to be available in the organization, at the right time, to enable the integration to work securely, stably and reliably in the longer term. At Harmonizer, our mission is to make this happen in a worry-free manner, and therefore we’ve invented the concept of 'Integration as a Service'. This enables you to use our best practices without delay, and perhaps most importantly, worry free.