When we kicked off developing the Movio Media platform in late 2014, Movio had been using Scala for a few years. After initial prototyping, the Movio Media squad settled on using a microservices architecture for the backend, based on the Play Framework. In this blog, we’ll describe our technology experience developing this within Movio Media.

About Movio Media

Movio Media is now an industry-leading market research platform. By aggregating data across a region, currently with Movio’s North American exhibition customers, it provides film distributors and studios comprehensive market data on the behavior of moviegoers, crucial audience insights, and innovative campaign solutions. Over 15 million avid moviegoers over the age of 14 are profiled in this database from 33% of North American screens of the Large Cinema Circuit (cinemas over 20 screens).

Breaking the mould

Perhaps we didn’t realise when we first decided to use a microservices architecture just how ideally suited this groundbreaking technology choice was for Movio Media. Here’s a quick summary of some of the benefits of this architectural style:

• easier continuous delivery
• finer grained scalability
• the team maintains an ongoing relationship with the product
• each service can use a technology set that is best suited to its purpose
• makes it easier to incrementally evolve the product

You can read more detail here.

We found that adopting the microservices architectural style is significantly simpler for an analytics-heavy system like Movio Media since we do not have to manage distributed transactions. This decreased the barrier to entry for us, enabling us to deliver value quickly while still mastering the finer points of the architecture.

The following diagram illustrates an architecture similar to ours. Our system consists of about 20 microservices, each running with multiple instances in production. These microservices encapsulate a mix of MySQL, Elasticsearch and Kafka datastores. Movio Media is a web application, and all of these microservices run behind load-balanced routers which dispatch requests to the appropriate docker containers on the host.

The development team consists of 3 backend Scala developers and 1 frontend developer. We also get invaluable help from the operations crew, the design crew, and all of the amazing people at Movio.

When developing a new feature, we decide whether to modify an existing microservice or create a new one. Sometimes it is appropriate to extract related functionality from existing microservices to bundle with the new functionality in a new microservice, though moving functionality between services requires a deprecation plan. Many of the instincts we have developed from working on monolithic codebases have analogues in a microservice architecture—loose coupling, single responsibility and abstraction over internal datastores are all important practices.

One of the architectural guidelines we’ve adopted at Movio is that multiple microservices should not share access to a datastore. Movio Media began as a monolith that was split into microservices early on, and some shared database access remains in our core services. We have been progressively replacing this direct access with service calls; giving us the benefits of API versioning and decoupling these services from the representation of documents in the store.

Our microservices communicate via JSON over HTTP, and we make heavy use of Apidoc to define the operations and data structures exposed over our APIs. Some microservices provide rich JSON query DSLs that are compiled to Elasticsearch queries, letting our frontend dynamically build complex queries that are validated in the backend before being run. Our use of Apidoc is worthy of an entire article to itself at some point in future, but suffice to say that it includes customizable Scala code generation which has become a key stage in our methodology.

Movio Media’s Angular frontend uses a mix of ES6 and ES7, with some legacy CoffeeScript. The backend interface for a feature can change during development, so we rely heavily on Apidoc as the source of truth for our API. It has proven effective as a way to document our API to frontend developers here at Movio.

Data is ingested by systems using a mix of streaming and batch import processes, drawing from relational databases and other sources. We use Kafka as our queuing system of choice, performing joins, processing and enriching our data with intermediate microservices before adding the final documents to Elasticsearch for fast querying. We’ve previously blogged about these in ‘Microservices: The rise of Kafka' and 'Microservices: Judgement day' have a read for more discussion about this architectural pattern.

Benefits

Most benefits of a microservice architecture stem from the decoupling of different components, allowing them to be developed and deployed independently.

We like that our small, focused code bases are much easier to understand. Most of our services are between 200-400 lines of code, plus tests. Builds and deployments are fast, since the codebases are small and independent.

Merge conflicts during development are rare, since functionality is kept to focused services in distinct repositories; you’re less likely to affect code being worked on by other team members. This is particularly important to us, since one of our team members works remotely in a different timezone and coordinating multiple manual merges across time zones is certainly something we’re glad to be rid of!

Using Apidoc, we are able to generate client libraries for use by other microservices. Upgrading to a new version of a dependent microservice is usually just a matter of bumping the client library version and fixing the compiler errors.

Challenges

Communication between microservices was something we avoided as long as possible because it adds pressure to keep their interfaces stable. We are more comfortable with this now that we use Apidoc to specify the interfaces.

We initially preferred to duplicate code between services, but now that our infrastructure is more stable we are more comfortable extracting common data processing out, or adding routes for common queries. We maintain a shared library to handle common concerns such as health endpoints and authentication, but this sometimes forces us to update all our microservices in lockstep, which is painful.

It is important to establish clear interfaces between services, especially to highlight breaking API changes. We use semantic versioning for our libraries and services, and separately version individual endpoints. We will soon be hosting our microservices in Kubernetes, making it easier to run multiple service versions in parallel for backwards compatibility.

Testing the assembled final system is a real challenge. We have a staging environment which virtually matches the production environment, but after testing, have to take care to return the two environments to matching configurations with minimal delay. Since we use docker in production we can also use docker compose to orchestrate running all or some of the microservices for a local development environment. One issue we found is the need to continually update the docker compose configuration to keep it in sync with the production environment.

Tooling support for microservice architectures is a pain point—there are few tools that provide static verification of REST APIs, so all the issues that plague interface stability in dynamic languages reappear at this level. This forces us to rely more heavily on Apidoc’s client code generation to provide a weak proof that components communicate in accordance to the API. On the plus side, microservice codebases are usually small and we notice our IDEs and editor tools are much more responsive.

In summary

Movio Media has benefited tremendously from the decoupling and rapid development gains due to our microservice architecture. Developers who understand the appeal of microservices but find themselves saddled with an unyielding, monolithic codebase should consider an incremental approach. Try peeling away a non-transactional business intelligence component that to get a feel for the work required to support such an architecture.

In the near future, we’ll be using Kubernetes to further streamline the deployment and administration of our services, making it easier to run multiple versions of our services in parallel and making our system’s growth even easier to manage.