As Drupal continues to mature as a platform and gain adoption in the enterprise space, integration with one or more 3rd party systems is becoming common for medium to large scale projects. Unfortunately, it can be easy to underestimate the time and effort required to make these integrations work seamlessly. Here are lessons we’ve learned…
Mailchimp, Recurly, Mollom, Stripe, and on and on—It’s easy to get spoiled by Drupal’s extensive library of contributed modules that allow for quick, easy, and robust integration with 3rd party systems. Furthermore, if a particular integration does not yet exist, the extendability of Drupal is straightforward enough that it could be done given the usual caveats (i.e. an appropriate amount of time, resources, and effort). However, these caveats should not be taken lightly. Our own experiences have unearthed many of the same pain points again and again, which almost always result in waste. By applying this gained wisdom to all subsequent projects involving integrations, we’ve been much more successful in identifying and addressing these issues head on. We hope that in sharing what we’ve learned that you can avoid some of the more common traps.
API and Integration Gotchas
Vaporware
Shocking as it may seem, there are situations where a client will assume an API exists when there isn’t one to be found. Example: a client may be paying for an expensive enterprise software license that can connect to other programs within the same ecosystem, but there may not be an endpoint that can be accessed by Drupal. They key here is to ensure you have documentation up front along with a working example of a read and/or write operation written in php or through a web services call. Doing this as early as possible within the project will help prevent a nasty surprise when it’s too late to change course or stop the project altogether.
Hidden Add-on Fees
An alternative to the scenario above is when an endpoint can be made available for an additional one-time or recurring fee. This can be quite an expensive surprise. It can also result in a difficult conversation with the client, particularly if it wasn’t factored into the budget and now each side must determine who eats the cost. The key to preventing this is to verify (up front) if the API endpoint is included with the client’s current license(s) or if it will be extra.
Limited Feature Sets
One can never assume that an entire feature set is available. Example: an enterprise resource planning (ERP) software solution may provide a significant amount of data and reporting to its end users, but it may only expose particular records (e.g. users, products, and inventory) to an API. The result: a Drupal site’s scope document might include functionality that simply cannot be provided. To avoid this issue, you’ll want to get your hands on any and all documentation as soon as possible. You’ll also want to create an inventory of every feature that requires a read/write operation so that you can verify the documentation covers each and every item.
Documentation
Transcending the “Drupal learning cliff” was and continues to be a difficult journey for many members of the community despite the abundance of ebooks, videos, and articles on the subject matter. Consider how much more difficult building Drupal sites would be if these resources didn’t exist. Now imagine trying to integrate with a systems you’ve never heard of using a language you’re unfamiliar with and no user guide to point you in the right direction.
Sounds scary, doesn’t it?
Integrating with a 3rd party application without documentation is akin to flying blind. Sure you might eventually get to the appropriate destination, but you will likely spend a significant amount of time using trial and error. Worse yet, you may simply miss certain pieces of functionality altogether.
The key here, as always, is to get documentation as soon as you can. Also, pay attention to certain red flags, such as the client not having the documentation readily available or requiring time for one of their team members to write it up. This is particularly important if the integration is a one-off that is specific to the customer versus an integration with a widely known platform (e.g. Salesforce or PayPal).
Business Logic
One of Drupal’s strengths is the ability for other modules to hook in to common events. For example, a module could extend the functionality of a user saving his or her password to email a notification that a password was changed.When integrating with another system, it’s equally important to understand what events may be triggered as a result of reading or writing a record. Otherwise, you may be in for a surprise to find out the external system was firing off emails or trying to charge credit card payments.
Documentation is invaluable to prevent these types of gaffs. However, in our experience it has been important to have access to a support resource that can provide warnings up front.
Support
What happens when the documentation is wrong or the software doesn’t work? If support regarding the API is slow or non-existant, the project may grind to halt until this block is removed. For enterprise level solutions, there is usually some level of support that can be accessed via phone, forums, or support tickets. However, there can sometimes be a sizable fee for this service and your particular questions might not be in scope with respect to what their service provides. In those instances, it might be helpful to contract with a 3rd party vendor or contractor that has performed a similar integration in the past. This can be costly up front while saving a tremendous of time over the course of the project.
Domain Knowledge
As consultants, one of our primary objectives is to merge our expertise with that of the customer’s domain knowledge in order to best achieve their goals. Therefore, it’s important that we understand why the integration should work the way it does instead of just how we read and write data back and forth. A great example of this involves integrating Drupal Commerce with Quickbooks through the Web Connecter application. It’s imperative to understand how the client’s accounting department configures the Quickbooks application and how it manages the financial records. Otherwise a developer may make an assumption that results in an inefficient or (worse) incorrect functionality.
Similar to having a resource available for support on the API itself, it’s invaluable to have access to team members on the client side that use the software on a daily basis so that nothing is missed.
Stability
Medium to large sized companies are becoming increasingly reliant on their websites to sustain and grow their businesses. Therefore, uptime is critical. And if the site depends on the uptime of a 3rd party integration to function properly, it may be useful to consider some form of redundancy or fallback solution. It is also important to make sure that support tickets can be filed with a maximum response time specified in any service licensing agreement (SLA) with the client.
Communication and Coordination
The rule of thumb here is simple: more moving parts in a project equals more communication time spent keeping everyone in sync. Additionally, it’s usually wise to develop against an API endpoint specifically populated with test data so that it will not impact the client’s production data. At some point, the test data will need to be cleared out and production data will need to be imported. This transition could be as simple as swapping a URL or it could involve significant amount of QA time testing and retesting full imports before making the final switch.
The best way to address these issues is simply to buffer in more communication time into the budget than a normal Drupal project.
SDKs
One gotcha that can be particularly difficult to work around is that an API may require you to use their specific software development kit (SDK) instead of a native PHP library. This may require the server to run a different OS (Windows instead of Linux) and web server (IIs instead of Apache). If you’re not used to developing on these platforms, development time may be slowed down by a significant percentage. For example: a back-end developer may not be able to use the same IDE they are accustomed to (with all of their optimized configurations and memorized shortcuts). This requirement may be unavoidable in some circumstances, so the best way to deal with these situations is a simple percentage on the budgeted hours.
VMs
When possible, it is ideal for developers can work on their own machine locally with a fully replicate instance of the API they are interacting with. Example: Quickbooks connecting through their Web Connector application to read and write records from a Drupal Commerce site. To test this connection, it is extremely helpful to have a local virtual machine (VM) with Windows and Quickbooks, which a developer could then use to trigger the process. If a project involves multiple developers, they could each have their own copy to use a sandbox.
Setting up a local VM definitely adds an upfront cost to create. However, for larger projects this investment can generally be recouped many times over with respect to increased development speed and the ability to start from a consistent target.
Final Advice
By now, we hope that we’ve made the case that it’s important to do your due diligence when taking on project involving integrations. And while this entire list of potential pain points may seem overkill, we’ve personally experienced the effects of every one of them at some point in our company’s history. Ultimately, both you and the client want to avoid the uncomfortable conversation of a project’s timeline slipping and going over budget. Therefore, it’s critical to have address these issues thoroughly and as early in the project as possible. If uncertainty is especially high, it’s usually beneficial to include a line item within the project statement of work to evaluate this piece separately. Finally, if you’re able to effectively negotiate the terms of a contract, the budget for the integration shouldn’t be set until an evaluation (even a partial one) has been completed.
Thoughts? Story to share? We’d love to get your feedback on how to improve upon this article.