Lessons learnt from integrating over 100+ services.

Integrations are at the heart of CluedIn. It is only through these integrations that CluedIn can unlock the true potential of your services. It could be something as simple as seeing Stripe data automatically connected with customer data in your CRM of choice. We have spent the last 2 years building robust and deep integrations into the tools that companies are using on a daily basis. It goes without saying, that not all integrations are made equal. Some are easy, some are complex, some are robust and some just don’t seem to want to work at all.

It is no doubt that more and more companies will need to integrate the plethora of systems they use to solve their day to day tasks. If companies don’t, sooner than later, they will realize the negative impact of information silos. This post is for the engineers, the people that are tasked with building the communication between systems. We have collated 2 years of building integrations into the most popular tools today and have come out the other end, knowing how we will continue to build our integrations in the future.

Lesson 1: Client tools are tempting, but in the end, simple REST is the easiest to build, maintain and test. I love that companies like Evernote, Dropbox, Google Drive have built C#, PHP, Node libraries to make it easy to get up an running quickly with talking to a service, however, time after time, we ran into issues with underlying client oddities where if we simply had an open and very easy to understand REST library like RestSharp, we would be able to test, fix and validate the necessary changes.

Lesson 2: Without tests, we were never aware of API changes. There are services like Hitch that allow you to subscribe to changes in API’s. Although the tool is useful as a guide, they don’t cover all the services we offer and to be completely honest, they miss things. Therefore, the way we solved this was to have integration tests that were periodically checking that we were getting the same packets. Without this, I am sure we would miss a lot of small changes in API’s that don’t necessarily make it to the documents.

Lesson 3: Half the time, the documentation is out of sync with the actual data you get sent. Although this is easy to solve with testing endpoints and creating your models based off the responses, there are many occasions where you cannot guarantee that the endpoints are always returning the same types of models e.g  Zoho CRM seems to have a very weird way of handling many results versus one result.

Lesson 4: Webhook’s are hard to test, especially without HTTPS certificates in all environments. Although for good reasons, many services will have issues talking to servers behind SSL under certain CA’s. The solution is often easy, but can lead to large confusion around whether the integrations has been done properly.

Lesson 5: I really hate to single certain integrations out, but there was one set of services that were so far beyond frustrating to integrate with, that it deserves a special mention. Microsoft services. Don’t get me wrong, I understand. For a company that has been around for so long and has so many customers running on different versions of software, it is very hard to consolidate endpoints. Add to this that Microsoft is buying many tools for its Office 365 stack that were complete products that already had their API’s built and have existing customers. Although humanly possible, the experience of working with API’s like the Office 365, SharePoint and to a lesser extent, OneNote, where just excruciating. The lesson is simply, plan for more time to build integrations against Microsoft tools.

Lesson 6: There are a few simple things that companies can do to make it dead simple for developers and tools to integrate with them:

  • Don’t require OAuth to test the API. Give developers a simple Access Token to test that the endpoints work and they can pull data.
  • Have a developer portal to build applications. It makes engineers feel so much more confident that their App is something that others can consume.
  • Have a sandbox setup for developers to test the API’s without the need of touching a line of code. This really helps us figure out what the data will look like.
  • Support OAuth 2 and don’t make your own version of it. Many integrations implemented OAuth 2, but with some small changes, which made it impossible to re-use the same libraries and code for authentication.
  • Give samples of ALL your endpoints, not just a few. For example, if you allow me to pass through parameters, don’t give me examples of just one parameter and how to use it, give examples for all.
  • If you are supporting Webhooks, give us a way to GET, DELETE, POST and PUT them. The amount of integrations that don’t have this make it very hard for developers to build against your webhooks and run integration tests with them.
  • Please give us a way to filter the data we are getting. We need to page, limit and filter by created dates, modified dates and more. Trust me, you are doing yourself a favour when you offer us this.
  • Please give us a way of quickly knowing how much data is available in your service e.g. Google Drive offers and endpoint to give developers information on the amount of usage.

CluedIn Engineering Team

 

Tim Ward

Founder, Developer and Loudest Employee at CluedIn.