JSON A Cautionary Tale For Backend Systems
A few years ago, I was working on a rather complicated system to do large scale data cleaning, and data augmentation. When we initially sat down to design, the system we realized that we’d probably have a few separate services, and really just remained more narrowly focused on the intent of the data, we’d be shuffling around between systems.
In the early days of the design process , we decided upon a message format for our data, purely based on ease and familiarity, and what we thought at the time was great flexibility. We choose JSON. Looking back, JSON was a somewhat poor choice. The lack of built in backwards compatibility ( versions) , the fact that this was an internal non web facing system, and JSON’s inability to enforce types outside of some very rudimentary means, and by writing a JSON spec enforcer, meant we were really at more of a disadvantage than what we perceived.
Inevitably consuming and producing code got out of sync, - Bugs in the deserialization step, with type conversions, - Lots of time lost coordinating protocol specs, - Lots of time coordinating sync’d deployments, - Lots of time spent building our own versioning, - Lots of time writing out what a spec should do under the JSON spec notation - Lots of time spent building our own type enforcer/validator - Too many different ideas about what made a good json object
The lesson I learned here, was that if you can afford it and you’re working on a system that has very clean lines, or does not need to be web facing then, you might be better off with [Protobufs] or Thrift or some sort of message passing protocol that will allow you to get types, have built in versioning , and not waste type on unit conversions, and additional code to document the code.