developer blog

How we do JSON at Simacan

Wed | 22 Apr 2020 |

How we do JSON at Simacan

There are lots of JSON libraries for Scala available; which one to choose? At Simacan, we need a JSON library that lets us write concise and readable JSON codecs, while enabling us to make our code forward and backward compatible with multiple versions of serialized objects.

We have a microservices cluster and a lot of our microservices are built with Akka and use Akka HTTP as the REST layer. We’re pretty satisfied with Akka HTTP so far. Spray JSON is the only library that works out of the box with Akka HTTP, making it an attractive choice at first sight. However, having worked with Spray JSON for a while, we were not convinced it was the best option, so we started looking for alternatives.

After some research, we decided to switch to Play JSON. This article describes how we came to that conclusion.

Getting the libs work together

The first question we asked ourselves, was how to get other libraries working with Akka HTTP. This shouldn’t be very complicated, but on the other hand, it’s not something you want to think about too much. Luckily, we found the akka-http-json library, written by Heiko Seeberger. This little library saves us from writing the boilerplate ourselves and has support for the following JSON libs:

Choosing a JSON library

A nice thing about Seeberger’s akka-http-json library is that it also features example apps and tests for each supported JSON library. This gave us the opportunity to evaluate each library easily. We just cloned the akka-http-json GitHub repository and started fiddling around with the test code for each library.

We were very impressed by the streaming features and the performace of the Circe library. And we liked the name a lot. (It is the name of a character from the ancient Greek mythology.) However, we were not (yet) able to write a really short and readable JSON (un)marshaller with Circe. This might very well be caused by our lack of experience with Circe. That’s why we’re planning to invest some more time in getting to know Circe.

The other library we were content with, is Play JSON. Some of our team members already had some experience with Play, which helped us to come up with some concise and readable code for the formatter we need, see the examples below. For now, we chose to use Play JSON as our default JSON library. That’s why the remainder of this article focusses on Play JSON for the examples. However, the principles discussed are valid for every JSON library.

Why not use automagical (un)marshalling

Most JSON frameworks offer some way to automagically (un)marshal case classes to JSON and vice versa. (Note that we stick with the Akka HTTP terminology here, so we talk about (un)marshalling, this process is also often referred to as (de)serializing.) While this can save a lot of code, we think it’s not a good idea to use these features. Let’s see why that is.

Suppose we create an API that accepts an Asdf object:

  1. case class Asdf(one: String, two: Int)
case class Asdf(one: String, two: Int)

With Play JSON, we can automatically create a “format”, which will take care of marshalling and unmarshalling between the case class and the JSON representation:

  1. implicit val asdfFormat: Format[Asdf] = Json.format[Asdf]
implicit val asdfFormat: Format[Asdf] = Json.format[Asdf]

We can now use this in a piece of test code:

  1. "PlayJsonSupport" should {
  2. import system.dispatcher
  4. "handle the marshalling and unmarshalling" in {
  5. val asdf = Asdf("one", 2)
  6. Marshal(asdf)
  7. .to[RequestEntity]
  8. .flatMap(Unmarshal(_).to[Asdf])
  9. .map(_ shouldBe asdf)
  10. }
  11. }
"PlayJsonSupport" should {
  import system.dispatcher

  "handle the marshalling and unmarshalling" in {
    val asdf = Asdf("one", 2)
      .map(_ shouldBe asdf)

This all works well, until we have to create a new version of our API. Suppose we add a field to our case class:

  1. case class Asdf(one: String, two: Int, three: Float = 0.0F)
case class Asdf(one: String, two: Int, three: Float = 0.0F)

Note that we set a default value for our added field, because we expect it to be absent in calls from clients that haven’t updated their code to reflect our change. To test this, we add an extra test:

  1. "handle added fields" in {
  2. val entity =
  3. HttpEntity(MediaTypes.`application/json`, """{ "one": "one", "two": 2}""")
  4. Unmarshal(entity)
  5. .to[Asdf]
  6. .map(_ shouldBe Asdf("one", 2, 1.0F))
  7. }
  "handle added fields" in {
    val entity =
      HttpEntity(MediaTypes.`application/json`, """{ "one": "one", "two": 2}""")
      .map(_ shouldBe Asdf("one", 2, 1.0F))

This code fails because Play JSON can’t unmarshal the JSON. You could argue that the new version of the API should be run at another endpoint or that clients should be updated anyway. However, we want to keep the number of endpoints to a minimum, since there are already a lot of them in a microservices cluster. And we believe it’s a good idea for a microservice to be a tolerant reader.

This disqualifies all forms of automatically generated (un)marshallers. So we set out to look for the shortest, most readable hand-coded (un)marshalling code. In Play JSON, we found this solution to meet our requirements best:

  1. implicit val asdfFormat: Format[Asdf] = (
  2. (__ \ "one").format[String] and
  3. (__ \ "two").format[Int] and
  4. (__ \ "three").formatWithDefault[Float](1.0F)
  5. )(Asdf, unlift(Asdf.unapply))
implicit val asdfFormat: Format[Asdf] = (
  (__ \ "one").format[String] and
  (__ \ "two").format[Int] and
  (__ \ "three").formatWithDefault[Float](1.0F)
)(Asdf, unlift(Asdf.unapply))

With this code in place, our test passes.

Getting the most out of Play JSON

So, we chose Play JSON. Even within Play JSON, there are a lot of ways to write (un)marshalling code. The solution shown above is the one we chose eventually. Here’s how we came to that:

  • In general, we prefer to do things in a functional way in Scala. Play JSON offers a functional-style syntax as well as a more imperative style. While it might seem more straightforward to do things the imperative way, we still think that the functional style yields more readable code.
  • Play JSON lets us choose to create a Format that handles marshalling and unmarshalling, or to create separate Reads and Writes functions. We choose to use a Format, as that gives the same functionality in less lines of code. We don’t see many uses for different Reads and Writes functions at the moment.

While evaluating all options, we discovered that not everybody in the team was aware of the different format/read/write functions. This might be because some features are not documented very well. So let’s solve that by giving a little documentation here…

Special formatters

As shown above, we can use formatWithDefault to define a case class member with a default value. This will result in the case class member to always be set, even if there’s no value in the JSON, for example. This means this is not an optional field, but we are resilient for the field being absent in JSON objects. This can be very useful in API’s especially when a field is added in a newer version.

There’s also the wider known formatNullable, which can be used with an optional case class field. This means the case class field has to be an Option. The option will be None if there’s no value in the JSON. And the other way around: if the value in the case class is None, the field will be omitted from the JSON. Then there’s also formatNullableWithDefault, which combines both. See the table for an overview:

function JSON input unmarshalled case class marshalled JSON
format { "one": 1 } (one: 1) { "one": 1 }
formatWithDefault { "one": 1 } (one: 1, two: 2) { "one": 1, "two": 2 }
formatNullable { "one": 1 } (one: 1, two: None) { "one": 1 }
formatNullableWithDefault { "one": 1 } (one: 1, two: Some(2)) { "one": 1, "two": 2 }

Wrapping up

This article described how and why we came to choose Play JSON as our preferred JSON (un)marshalling library for use with Akka HTTP. We also discussed what is our team’s preferred way to use Play JSON.

Author: Bart Kummel | @bkummel