Author: Nadav S. Samet

Put your alerts in version control with DogPush

At TrueAccord, we take our service availability very seriously. To ensure our service is always up and running, we are tracking hundreds of system metrics (for example, how much heap is used by each web server), as well as many business metrics (how many payment plans have been charged in the past hour).

We set up monitors for each of these metrics on Datadog, that when triggered, will page an on call engineer. The trigger is usually based on some threshold for that metric.

As our team grew and more alerts were added we noticed three problems with Datadog:

  1. Any member of our team can edit or delete alerts in Datadog’s UI. The changes may be intentional or accidental, though our team prefers to review changes before they hit production. In Datadog, the review stage is missing.
  2. Due to the previous problem, sometimes an engineer would add a new alert with uncalibrated thresholds to datadog to get some initial monitoring for a newly written component. As Murphy’s law would have it, the new alert would fire at 3am waking up the on call engineer, and it may not even indicate a real production issue, but a miscalibrated threshold. A review system could better enforce best practices for new alerts.
  3. Datadog also does not expose a way to indicate that an alert should only be sent during business hours. For example, for some of our batch jobs, it is okay if they fail during the night, but we want an engineer to address it first thing in the morning.

To solve these problems, we made DogPush. It lets you manage your alerts as YAML files that you can check in your source control. So you can use your existing code review system to review them, and once they’re approved they get automatically pushed to DataDog — Voila! In addition, it’s straightforward to setup a cron job (or a Jenkins job) to automatically mute the relevant alerts outside business hours.
DogPush is completely free and open source – check it out here.

ScalaPB: TrueAccord’s Protocol Buffer code generator for Scala

When we started working on TrueAccord, we had a limited understanding of various technical aspects of the problem. Naturally, one of those unclear aspects was the data model: what data entities we will need to track, what will be their relationships (one-to-one, one-to-many, and so on), and how easy it is going to be to change the data model as business requirements become known and our domain expertise grows.

Using Protocol Buffers to model the data your service uses for storage or messaging is great for a fast-changing project:

  1. adding and removing fields is trivial, turning an optional field into a repeated field and so on. If we modeled our data using SQL, we will be constantly migrating our database schema.
  2. the data schema (the proto file) serves as an always up-to-date reference documentation for the service’s data structures and messages. People from different teams can easily generate parsers for almost every programming language, and access the same data.

Continue reading

A Reactive HTTP Reverse Proxy in Play!

At TrueAccord, we use Play to develop our backend. Given our development environment, we need to have part of our URL space routed to a different server written in Python. We initially thought of setting up a lightweight HTTP server like nginx that would act as a reverse proxy for both of our development servers, which is a reasonable solution. However, we also wanted to avoid having yet another moving part in our development environment and were curious if we could write something quick in Scala that could achieve this.

As it turns out, writing this little reverse proxy in Scala/Play is relatively straightforward. It’s also pretty impressive that with so few lines of code we get a reactive proxy server that streams the content continuously to the end client while chunks of it are still arriving from the upstream server. A more traditional (and time-intensive) implementation would have buffered the entire upstream response until it was complete and only then sent it to the client..

So, without further ado, here is the code:

In line 14, proxyRequest.stream returns a Future[(WSResponseHeaders, Enumerator[Array[Byte]])]. This means that at some point in the future, our closure at line 15 will get called and will be supplied two things: the headers returned from the upsteam server (WSResponseHeaders) and an Enumerator[Array[Byte]], which is a producer of arrays of bytes. Each array of byte that it will produce is a part of the response body from the upstream server. Conveniently, Play provides a Result constructor that takes producers like this and turns them into responses that can be served to the end client.

flattenMultiMap is a little helper function that converts the query string parameters from the collection type they are given by Play requests to the format expected by WS.url.

Pretty cool, eh?

Why did we use Scala as our main backend language?

scala-logo

Why did we use Scala as our main backend language?

Given what we needed to accomplish when we founded TrueAccord – and how we intend to grow – Scala quickly emerged as our ideal programming language. From its interoperability with Java  to its emphasis on functional data structures and economy of code, Scala has delivered an incredibly versatile engine to power TrueAccord’s Proactive Loss Management system for recovering debt portfolios worth tens of billions of dollars.

Of course there are solid arguments for a range of programming languages. Java may seem the obvious choice, given its popularity and the breadth of its libraries. But many developers rate Java pretty low on the fun scale. There’s the ever-popular Python and Ruby, which many feel are optimum for the front end, but we wanted to use a single language for the entire stack.

We actually started in Python, because our core team was very familiar with it. Python allows you to get off the ground really fast, but as the code grew and more people joined the team, Python’s lack of type safety started to slow us down. Despite our disciplined unit testing, things got messy very quickly. It took quite a bit of reasoning to be completely sure what each function expected as arguments and what it returned. Ultimately we realized we needed something that would make it easy for us to write robust, maintainable and type-safe code without hindering the team’s productivity. Scala was the answer.

Interoperability with Java

Scala allows us to access Java’s huge ecosystem. Java has been around enterprises for a long time, and there is a wide variety of high-quality libraries for any imaginable functionality. From essential libraries like Joda-Time to the handy utilities in Apache Commons, Java has it covered.

It’s also very common to see third-party service providers like payments processors provide Java libraries that access their APIs, which saves us hours of integration work and makes it easy for us to quickly experiment with many platforms. This is how TrueAccord is able to personalize correspondence with individual debtors and make it easy for them to pay creditors via a number of different methods. On top of that, the JVM provides a robust environment to run our code. There’s a variety of parameters we can tune and many tools available for monitoring, artifact deployment and so on. As Scala runs on top of the JVM it enjoys this rich and mature ecosystem.

Functional data structures

When you are writing non-trivial computations, especially in a multithreaded environment, it’s easy for things to go awry. In languages like C and C++ it’s common to have a state that is shared between different threads. When you go that route, you have to be careful about using locks or mutexes around each access to the state to ensure that no two threads can modify this state at the same time. It’s very easy to make mistakes in this regard. Scala encourages writing pure functions that manipulate immutable data structures. When your data structures are immutable, you can freely share them between different execution contexts, and deadlocks and race conditions become things of the past.

Economy of code

Scala is a very expressive language. And yet it lets you express your ideas with very few lines of code. When you define a case class, you are actually communicating a higher level abstraction that relies on other types.

For example, a Person could be a case class that consists of a name, age and an address. The address could be another case class with fields like street, city and zip code. By defining types in this way, usually in one or two lines of code, Scala provides a functionality that is equivalent to 20 to 30 lines of Java code. The savings here comes from not having to implement constructors, getters, equals() and hash() methods. There are many other examples like this that allow the Scala programmer to write fewer lines of code that say quite a lot. This leads to fewer bugs and less code to read and review.

This is not to say there are no drawbacks. Like any technology, Scala is not perfect. For example, Scala code that uses functional programming idioms can be a little confusing to newcomers, but you get better at reading and writing functional code with practice. It’s the same as lifting weights to build muscle. Another drawback is the compilation time. It can take a little while for the compiler to go through a full build, but on the other hand, having immediate feedback from the compiler on mismatching types and other common human errors helps recover that lost time.At TrueAccord, this level of code correctness saves us invaluable time and money in the long run.

Come Work With Us!

We combine a social mission, huge market, a proven team and early traction + revenue. We're early enough for you to have a huge influence on the company's future (and upside) but late enough so we already reached first funding and product market fit - the perfect timing to join. If you want to learn a lot while your actions have a positive impact on the world - come work with us.

Stay up to date! Subscribe for updates on TrueAccord

SF Scala: Nadav Samet, ScalaPB: Working with Protocol Buffers in Scala