A look into how you can build reliable, distributed workflows using Hasura and Temporal.

I recently wrote about the challenges of trying to build a distributed microservices architecture on top of Hasura. 

When it comes to reading data, Hasura is one of the greatest technologies I've ever had the pleasure of building with. It provides powerful capabilities for federation and data aggregation. If you put their GraphQL API in clients' hands, things are lovely.

When it comes to writing data, specifically distributed cross-domain writes, things can be trickier. There's a snare awaiting those building a distributed system of writes on top of Hasura. Typically, this architecture will involve Remote Schemas and Event Triggers.

For context, remote schemas enable you to stitch other GraphQL APIs (i.e., microservices) into one unified super-graph. Event triggers are a mechanism for Change Data Capture (CDC), allowing you to handle changes to data (for example, syncing data from one datastore to another).

Disclaimer: The traditional example you often find in microservices literature is vacation booking, but it's lunch, and I'm hungry, so I offer a sillier example that's more fitting to this moment.

One of my favorite TV shows from childhood was Spongebob. There's one episode I distinctly remember (entitled "Pickles") where a food critic rolls into town and everyone's attention turns toward the fry cook, Spongebob, who's trying to cook up the perfect hamburger. It's a tense moment. The restaurant's reputation is on the line. He needs to get the order correct, but to everyone's surprise, he gets it wrong: he forgets the pickles!

First bun, then patty, followed by ketchup, mustard, pickles, extra onions, lettuce, cheese, tomatoes, and bun, in that order.

Let's imagine we're building an architecture for this fast-food restaurant and each hamburger ingredient gets its own domain. The recipe (workflow) for making a hamburger is complex: we need to execute everything in order, can't miss any steps, and can't perform any steps more than the required number of times.

Let's imagine we have very demanding customers, and that our distributed workflows must be correct.

The anti-pattern I have in mind, specifically, is that of a bulk mutation. The client wants to write data to multiple domains in bulk, but it wants to do so in one HTTP request. Hasura will proxy each part of the mutation to a different remote schema (microservice) in sequence. Each microservice will execute its data insertions and updates in its own database transaction. Additionally, we have a secondary datastore to which we want to sync write events. Event triggers will get asynchronously executed (out of order), and additional logic will be required to coordinate data syncing.

No ACID guarantees (separate DB transactions). Unordered, tightly coupled Event Triggers. Data syncing is a challenge.

It's possible to design our architecture this way. But it involves lots of struggle, and maybe even anti-patterns when it comes to domain boundaries.

There are a few challenges with this architecture:

We have no ACID guarantees when performing a bulk mutation involving multiple remote schemas. There is no atomicity or transactionality. One remote schema operation can succeed, while others might fail. Our data might be left in an inconsistent (incorrect) state. The onus of retrying and ensuring reliability (correctness) falls to the client rather than the backend system.

We have no order guarantees when it comes to Event Trigger execution. This means they need extra cross-domain logic for sleeping, retrying, and checking each other's state. Code becomes more tightly coupled. Domain boundaries start to bleed a little. The code becomes harder to write, read, and maintain.

Domains are tightly coupled to each other.

Distributed transactions are fraught with issues.

Distributed data syncing logic is hard to write, read, maintain, and reason about.

Microservice (decentralized) choreography is harder than (centralized) orchestration.

Ultimately, this anti-pattern may arise in part due to a misunderstanding of how bulk mutations work. As a GraphQL-to-SQL planner, Hasura is able to reason about bulk mutations and bundle them together into one database transaction.

However, a bulk mutation that relies on several remote schemas behaves differently: there is no bundling! That leaves us open to all sorts of potential data inconsistencies.

mutation OrderHamburger(
  $orderID: uuid!
  $lettuceID: uuid!
  $tomatoID: uuid!
  $onionID: uuid!
  $pickleID: uuid!
) {
  # Request 1
  lettuceService {
    addLettuce({
      orderID: $orderID
      lettuceID: $lettuceID
    }) {
      __typename
    }
  }
  # Request 2
  tomatoService {
    addTomatoes({
      orderID: $orderID
      tomatoID: $tomatoID
    }) {
      __typename
    }
  }
  # Request 3
  onionService {
    addOnions({
      orderID: $orderID
      onionID: $onionID
    }) {
      __typename
    }
  }
  # Request 4
  pickleService {
    addPickles({
      orderID: $orderID
      pickleID: $pickleID
    }) {
      __typename
    }
  }
}

Each part of this bulk mutation executes in sequence: lettuce, then tomatoes, then onions, then pickles. Any step along the way can fail. Hasura won't retry these calls for you. They're not atomic.

The onus of ensuring data consistency has shifted to the clients, who now have to encode complex retry logic and use idempotence keys (e.g., $tomatoID) to ensure no duplicate data gets created — after all, you wouldn't want to receive a hamburger with three times as many tomato slices as expected. The client really should not have to worry about ID generation at all: that should be entirely the concern of the backend.

In the best-case scenario, we have code bloat and unnecessary complexity. In the worst case, we have lurking data consistency problems that can disrupt customers and erode trust in your business.

People want their hamburger orders exactly as specified, and nothing less. The stakes could not be any higher.

What if we could change the data flow? Instead of trying to synchronously perform three database transactions in sequence, what if we spawned a Temporal workflow that would asynchronously take care of all the potential transient failures along the way, and result in us eventually receiving the perfect hamburger?

The new GraphQL mutation would look slightly different:

mutation OrderHamburger(
  $orderID: Int!
) {
  # Request 1
  temporalService {
    createOrder({
      orderID: $orderID
      lettuce: true
      tomatoes: true
      onions: true
      pickles: true
    }) {
      __typename
    }
  }
}

The new architecture involves slightly more infrastructure (unless you're using Temporal Cloud), but it's worth it!

New and improved architecture, powered by Hasura and Temporal

The new design is an improvement for a few reasons:

ACID guarantees. Data will be eventually consistent. You don't ever have to worry about "sending an order back to the kitchen."

High cohesion, low coupling. Services are no longer coupled to each other. There's one centralized, cohesive place that describes how the workflow should execute. This is much lower cognitive load when compared to trying to reason about code across disparate HTTP Event Trigger handlers.

Less complexity. Clients don't have to worry as much about idempotence. Lots of the keys for data models can be generated upstream. The client really just needs to send a workflow ID and only the most important metadata about what it wants to accomplish — nothing extraneous.

Improved performance. If the client doesn't need to block, it can go about doing other things while the workflow finishes. System efficiency increases.

Better reliability. Temporal handles retrying for us. Transient failures, like network errors, are abstracted away. Our system is correct.

Temporal is an engine that solves the problem of workflow correctness. Where achieving correctness in a distributed workflow once required lots of complex logic and bespoke infrastructure, Temporal is able to simplify this process across multiple dimensions: developer experience, cognitive complexity, infrastructure provision, etc. Temporal shines when it comes to workflows that are sufficiently complex and involve lots of writes to different domains.

Hasura is not a durable workflow execution engine — nor does it make any claim to be!

Hasura, on the other hand, is incredibly useful for data federation (think API gateway), data aggregation (think data stitching), and data access (think ORM).

Unlike Temporal, Hasura is not a durable workflow execution engine — nor does it make any claim to be!

Don't be tempted to misuse one like the other. Instead, use them both to power a distributed system that is reliable, easy to reason about, and super fun to build.

Hasura and Temporal: How to Make the Perfect Hamburger

Style is all around us, yet it can be hard to define sometimes!

Being one of the hardest problems in computer science, it's fitting that naming gets its own section.

Names — of variables, functions, structs, fields, etc — are important.

I recently found a comprehensive naming cheatsheet through Jack Herrington.

Is your code/system minimalist? Simple? Intuitive? Extensible? Composable? Predictable? Robust? Reliable? Resilient?

Would another engineer be able to understand it?

Does it solve the problem with good style?

There are lots of good adages in TigerBeetle's style guide.

"Style is evolving. A collective give-and-take at the intersection of engineering and art... First principles and knowledge. Precision and poetry. Just like music. A tight beat. A rare groove... Biodigital jazz. This is what we've learned along the way. The best is yet to come."

"Another word for style is design. Our design goals are safety, performance, and developer experience. In that order. All three are important. Good style advances these goals."

"Style is more than readability, and readability is table stakes, a means to an end rather than an end in itself... style is not something to pursue directly. Style is necessary only where understanding is missing."

"Contrary to popular belief, simplicity is also not the first attempt but the hardest revision. It's easy to say “let's do something simple”, but to do that in practice takes thought, multiple passes... many sketches".

"We spend this mental energy upfront, proactively rather than reactively, because we know that when the thinking is done, what is spent on the  design will be dwarfed by the implementation and testing."

"An hour or day of design is worth weeks or months in production."

"Code, like steel, is less expensive to change while it's hot. A problem solved in production is many times more expensive than a problem solved in implementation, or a problem solved in design."

"We do it right the first time. This is important because the second time may not transpire, and because doing good work, that we can be proud of, builds momentum."

"Always motivate, always say why. Never forget to say why. Because if you explain the rationale for a decision, it not only increases the hearer's understanding, and makes them more likely to adhere or comply, but it also shares criteria with them with which to evaluate the decision and its importance."

"Don't forget to say why. Code alone is not documentation. Use comments to explain why you wrote the code the way you did. Show your workings."

Last year, Dan North published the CUPID framework "for joyful coding" — a long-awaited complement to SOLID programming principles.

Domain-based: the solution domain models the problem domain in language and structure

Dan writes about the difference between code understandability versus code habitability.

Habitability is the characteristic of source code that enables folks to understand its construction and intentions and to change it comfortably and confidently.

Where SOLID principles were presented as cold, unforgiving, and code-centric, CUPID properties invite us into a friendlier spectrum where we are neither entirely in violation nor entirely perfect; where we humans can understand, assess, and strive.

One of the most interesting articulations I've recently is regarding how good code is layered to reveal intentions. Intention-revealing code is like "having a 2-minute tutorial, a 10-minute tutorial, and a deep dive."

Writing idiomatic code is an act of empathy since it reduces cognitive load.

There are lots of great Go style guides out there, such as Uber's or Effective Go, but the one from Mattermost includes a few things worth highlighting.

"Always prefer synchronous functions by default. Async calls are hard to get right. They have no control over goroutine lifetimes and introduce data races. If you think something needs to be asynchronous, measure it and prove it."

I appreciate the call for rigor. I kind of view them as saying concurrency in Go is not without its tricky aspects, despite the language being famous for its concurrency primitives. I tend to try to avoid writing concurrent code as often as I try to avoid using reflection.

This is one I've seen countless times (from former Java developers) and a habit I personally had when I was starting out. The Mattermost guide encourages us to try to avoid mocks and truly test dependencies. When interfaces are necessary, they should be declared on the consuming side, not on the implementation side.

"Often, different packages have non-overlapping sets of functionalities to consume. If you do find several consumers of the package, remember that interfaces can be composed. So define small chunks of functionalities in different interfaces, and let consumers compose them as needed."

Polymorphism reminds me of Randall from "Monsters, Inc." He can change his color and appearance to blend in with his environment.

Similarly, polymorphic code relies on an abstraction that can blend in and be reused across situations.

This post discusses a few strategies for polymorphism, reusability, and DRY when using a few of my favorite technologies: GraphQL (specifically with Hasura) and Golang (specifically with genqlient).

Example 1: Lots of similar-looking tables

Suppose you're working with a denormalized SQL database. You have a few tables that all share an identical set of columns. Say we're building a Dungeons & Dragons game, and for whatever reason, we have separate tables for Humans, Orcs, Dwarves, and Elves. There's a common set of columns on each of these tables: strength, intelligence, luck, and dexterity.

Our GraphQL client library, genqlient, will generate code for each field we're using in our query. We will naturally have some redundancy: a separate struct for a Dwarf that looks very much like the struct that gets generated for an Elf, which looks similar to the one for Orc, for Human, etc.

query GetCharacters {
  dwarf {
    strength
    intelligence
  }
  elf {
    strength
    intelligence
  }
}

We'll have two separate structs that each behave the same way. We certainly don't want to write duplicate functions for dealing with these structs. The solution is to use Golang interfaces.

type Character interface {
  GetStrength() int
  GetIntelligence() int
}

We can write our code once (dealing with Character) — which is much better than writing duplicate functionality for each implementation of Character.

Recently, I ran into an issue particular to genqlient. I wanted to craft a query resolver that would allow clients to retrieve a set of elves in a variety of ways:

When genqlient generates code based off of Hasura's schema, it generates what's called a boolean expression.

type ElfBoolExp struct {
  And          []*CompatCcnetEntityIdsBoolExp   `json:"_and,omitempty"`
  Not          *CompatCcnetEntityIdsBoolExp     `json:"_not,omitempty"`
  Or           []*CompatCcnetEntityIdsBoolExp   `json:"_or,omitempty"`
  Strength     *IntComparisonExp                `json:"strength,omitempty"`
  Intelligence *IntComparisonExp                `json:"strength,omitempty"`
  Dexterity    *IntComparisonExp                `json:"strength,omitempty"`
  Luck         *IntComparisonExp                `json:"strength,omitempty"`
}

The boolean expression lets us query and filter for entities in powerful ways!

The omitEmpty tags on each field are very important for nullability with genqlient. They let you explicitly avoid sending NULL where a zero value was meant to be sent instead.

Ultimately, the problem is not what ElfBoolExp looks like, but what IntComparisonExp looks like.

type IntComparisonExp struct {
  Eq     *int  `json:"_eq"`
  Gt     *int  `json:"_gt"`
  Gte    *int  `json:"_gte"`
  In     []int `json:"_in"`
  IsNull *bool `json:"_isNull"`
  Lt     *int  `json:"_lt"`
  Lte    *int  `json:"_lte"`
  Neq    *int  `json:"_neq"`
  Nin    []int `json:"_nin"`
}

Notice how there are no omitEmpty tags! This is a huge problem, as it means the developer would have to set all nine fields, even if they're only interested in the Eq field.

This presents a major problem for developer experience.

The solution I settled on was to use a GraphQL fragment and write multiple variations of the query, one for each field.

fragment ElfFields on Elf {
  strength
  intelligence
  dexterity
  luck
}

query GetElfByStrength(
  $strength: Int!
) {
  elf(
    where: {
      strength: { _eq: $strength }
    }
  ) {
    ...ElfFields
  }
}

query GetElfByIntelligence(
  $intelligence: Int!
) {
  elf(
    where: {
      intelligence: { _eq: $intelligence }
    }
  ) {
    ...ElfFields
  }
}

GraphQL fragments are reusable sets of fields which can DRY your queries. Fragments enable reusability, but they're not exactly the embodiment of the concept of polymorphism. For that, we'd need to use actual GraphQL interfaces. Unfortunately, Hasura does not enable interfaces, so we're getting as close as we possibly can.

Polymorphism with GraphQL, Hasura, Genqlient, and Golang

Enterprises that offer platforms or marketplaces for other institutions or businesses will inevitably face modeling challenges. How do you bring all of these entities under one tent? Do you create The Perfect Model™, the one-size-fits all approach? Is that even possible?

It's the problem of modeling heterogeneity in the real world.

It's as much of a product / business question as a technological one. Should you do so? Are you capable of doing so?

What do you do when one of your customers wants new data state (or new functionality) that no other customers have asked for?

If you're clever, you can generalize and abstract it. Maybe you can successfully map multiple differing implementations to one standard.

If you're unlucky, you'll opt to model every new ad-hoc piece of custom state and custom functionality, trying to generalize where possible.

You might attempt to model some schema-less state in your database (maybe this means using a NoSQL database to put more control in customers' hands, or maybe it means creating an attributes table in your SQL database).

But that's only half the battle. What about sanitizing and validating the data for those attributes? Are you persisting validation rules as well? Something has to enforce types (e.g., my custom attribute called "number_of_pets" needs to be a number, not a string or a boolean). What about additional validations (e.g., preventing negative numbers)?

Since 2017, I've enjoyed the developer experience of using protocol buffers, and that enjoyment has only multiplied since Buf entered the scene. They recently announced the imminent release of new tool called "protovalidate", which will bring the power of Common Expression Language (CEL) to protobufs. Based on what I've seen from CEL, it seems expressive enough to meet lots of customers' needs.

The challenge is allowing customers to build their own schemas and schema validations.

Anyone can write a protobuf. Not everyone can build a no-code engine that allows customers to define their own attributes and validations, auto-generates the appropriate protobufs at runtime, and (un)marshals them on-demand.

I'm not sure if this could work, but using protobufs doesn't seem like the worst way to go, especially considering how quickly tooling around them has improved in just the last year alone.

If you're interested in building an architecture consisting of microservices, SQL, and GraphQL, chances are you've heard of Hasura.

For small teams that need to ship quickly, it's one of the best choices out there because:

You get a GraphQL API over your data for free.

You get to join your data in a myriad ways (even across remote APIs and remote databases).

You get to federate microservices' GraphQL APIs together into one super-graph.

There are other benefits, such as authorization, but these are the three I noticed first.

Hasura's main core value prop is "instant GraphQL on all your data" in seconds. This means teams can avoid the tedium of building out an API (e.g., deciding on a transport protocol, crafting the API spec, building transport handlers, building a persistence layer, etc). Once you have a fluency with GraphQL, things become really easy.

Writing a GraphQL query to traverse from table to table is much easier and faster to do than using an ORM, even a compile-safe, database-first one like sqlboiler. What's more, ORMs are limited to working with one database; they don't allow you to stitch data together from multiple remote data sources, unlike Hasura's Remote Relationships feature. (If you're building a greenfield project, you probably only have one database, but if you're migrating an existing legacy system, remote relationships are invaluable).

The last Hasura feature to highlight is Remote Schemas, which let you federate GraphQL APIs together as one super-graph. If you're building a microservices architecture, each microservice could provide its own remote schema, which could get subsumed under Hasura's graph. This pattern of letting clients access one central hub/gateway for everything is sometimes called API Composition.

Lots of data. It's all connected. Everyone's happy... Or are they?

Disclaimer: Not everyone will face these challenges; they're particularly to a certain topology and tech stack, and not necessarily faults inherent to Hasura's GraphQL Engine. They might be mitigated by looser coupling, better tooling, and more awareness around best practices and gotchas.

One of the first questions that typically gets asked (and re-asked many times) is: "Do we even need microservices if Hasura provides authorized reads and writes for free?" It's a good question; the answer might be influenced by your organization's communication structures (see Conway's Law), but ultimately it's one you'll have to answer with your team.

"Do we need microservices for writes and reads?"

If you do adopt a service-oriented architecture, you may run into another question: "Can we get away with using remote schemas (services) for our writes, but direct-to-database GraphQL queries for our reads?"

I think one answer is: "It depends on how complex your business logic is. How complex is the query that determines whether a user can perform an action on an object?"

For simple domains, Hasura authorization rules might be simple.

For more complex domains (think infinitely recursive structures, like Organizational Unit hierarchies), those rules might be much harder to write, rougher on the eyes, or maybe impossible. I say "impossible" because in certain cases where the domain is truly a graph, as ironic as it may seem, GraphQL isn't sufficient: it doesn't support arbitrary-length paths like true graph databases.

For cases where auth logic is super complex, you might consider decoupling auth from business logic using a dedicated authorization engine.

This one trips up a lot of newcomers: they see a "not found" error and spend hours puzzled!

From my experience, this is almost always due to a permissions issue (they forgot to grant role permissions on a table column or a field in a remote schema).

More broadly, permissions can present design challenges when an operation deals with more than one domain. Take the classic example of booking a vacation: you need to book a flight, a car, and a hotel. These can be imagined as three separate domains, three separate microservices, and three separate databases.

But what happens if one service has to interact with another? With Hasura, this is often accomplished with a relationship, which lets you traverse from a table in one domain to a table in another. Things get tricky. Do you grant Service A the ability to perform reads or writes on Service B's tables? That's certainly one approach, which can be accomplished by granting your service an inherited role, which lets your service read and write to its domain, as well as read from other domains.

What about cross-domain writes? How are they orchestrated? These days, my preferred method is to coordinate writes through an orchestration engine, such as Temporal. Rather than performing them in one database transaction (which might involve some hemorrhaging of our domain boundaries), they can be performed one by one in a durable workflow.

This is fundamentally a design question: orchestration (centralized) vs choreography (decentralized). It's not a problem inherent to Hasura per se, but it may crop up if you're building a service-oriented architecture on top of Hasura, and it may be mitigated by an engine like Temporal.

Bulk mutations are not always transactional

Unlike bulk mutations over a database, a bulk mutation involving multiple remote schemas is not transactional. This might be obvious to everyday Hasura users, but it has important ramifications for how you design a microservice architecture powered by Hasura.

In a common example, you might have an external client that uses one bulk mutation, where each part of the selection set gets proxied to a different remote schema and results in sequential execution of multiple database transactions. Consequently, the system suffers a loss of ACID guarantees — yet another reason to consider Temporal, or at the very least consider how important idempotence might be to your system.

Getting Event Triggers to work together is hard

Beyond that, if your doing a migration and find yourself needing to do a data sync, you may consider Hasura's Event Triggers mechanism. But you'll run into the same challenge of distributed coordination: several different Event Triggers getting invoked asynchronously, and often not in sequential order. From the docs: "Event webhook notifications will be delivered at least once, and may arrive out of order with respect to the underlying event."

This feels like a missing feature, especially when compared to other Change-Data-Capture (CDC) solutions like Debezium. In the absence of order guarantees, we have to write extra coordination logic to "wait" for other event triggers to execute first. That logic also demands cross-domain permissions (e.g., Service B's event trigger now needs read permissions to see if Service A's event trigger has been executed).

The biggest struggle I've encountered with Hasura and microservices is the tight coupling when you use Hasura's remote schemas (federation) feature and its core features of reading and writing data to a database.

To really get a sense of that tight coupling, we'll rehearse what needs to be done when building a "Hasura-powered microservice." The first step you'll do is to introspect Hasura's GraphQL schema, which produces what's known as an SDL file (schema definition language). The Hasura team actually made a nice tool called graphqurl for introspecting GraphQL schemas.

Once you've got your schema, the next move is to pick a GraphQL client library (preferably a type-safe one like genqlient) and start crafting your queries and mutations.

The GraphQL operations you write are directly influenced by the Hasura schema. Every time you break your database schema (e.g., remove or rename a column), which is bound to happen in greenfield projects, you have to update the SDL file by re-introspecting the schema, and only then can you make modifications to your service's queries and mutations.

It would be nice if the CI (continuous integration) system could take care of this for us, but unfortunately, engineers need the ability to update the SDL file locally since they're testing changes locally.

There are several problems with this SDL file:

An engineer can overwrite it without having the latest state.

If an engineer forgets to update the SDL file, it might not be the end of the world. Perhaps someone added a new table or a new column that your service doesn't particularly care about. In the case of a breaking change, such as a column getting renamed, the service might stop working altogether if it isn't updated as well.

If an engineer overwrites the SDL file without having the latest state, the new schema file may be missing a few important tables and columns, which could lead to compilation errors for other engineers. 

Lastly, the SDL file may be constantly changing, especially in cases where Hasura oversees lots of different business domains that are managed by separate teams. If 20+ engineers are all using the same Hasura metadata, it's going to be very noisy. Even the smallest change, like adding a previously unused field type to the schema, can have a disruptive impact across multiple codebases. The end result is lots of unhappy engineers.

Decouple the code so that every change doesn’t echo throughout the entire code base.

Fortunately, there are strategies to mitigate each of these problems:

Forgetting to update the schema: PR reviewers and engineers should be vigilant about accompanying code changes. If a new column or table is added, the reviewer should ask: "Where are the code changes to go along with this schema change?"

Overwriting the schema with stale state: PR reviewers should carefully check to see if the SDL file underwent a breaking change. Are columns or tables missing? Has anything been renamed? Ideally, the changes should be additions, not deletions. Better yet, breaking change detection should be automated with a tool like graphql-inspector (or, in the future, with Hasura Schema Registry, which was just announced at the time of writing).

The schema undergoes a lot of changes: To reduce this noise, services should introspect the schema with the exact same token they use at runtime. Unless your microservice is actually using the admin secret, it's a mistake to introspect with the admin secret; doing so will leave the door open to runtime issues due to forgetting to add role permissions for certain fields. It's much better to catch these issues at compile-time rather than runtime.

Ultimately, I see minimalism as one of the core tenets of GraphQL. Only select the fields you care about. Only provision the permissions required. The bare necessities. Use only what you need.

It would be nice if there existed tooling to warn on any GraphQL fields you're not using in code, or any fields on which your role has been granted permissions that aren't used in queries or mutations. Perhaps something like remove-unused-fields CLI.

Another quality-of-life improvement would be if there were a Hasura view that could let you manage all of a role's permissions in one central place, rather than having to check each individual table. All this info is available in YAML metadata files, so maybe that's a potential hackathon project: something that crawls those files and aggregates all of the role's table/column permissions.

Each microservice in my imagined architecture also provides a GraphQL API over its domain, which gets federated into Hasura as a remote schema. At the same time, each service relies on Hasura for data retrieval.

Whenever the service's API (remote schema spec) evolves, there are also accompanying changes to Hasura metadata (in remote_schemas.yaml) that need to be subsequently deployed.

On the other hand, whenever the database schema evolves, those migrations need to occur before any changes to consuming services are deployed.

Releases are therefore a multi-step process, and deployments are bidirectionally and tightly coupled — and the answer to "what order do I merge/deploy stuff in?" is "it depends."

Deploying Hasura but forgetting to deploy the microservice will result in inconsistent metadata, meaning the entire remote schema will be unusable.

A couple of improvements could go a long way:

Serve what's not broken: What if Hasura only made the relevant parts of the remote schema unusable? (If there's an inconsistency between what Hasura thinks of one query versus what the actual query spec is, why should that prevent the use of all other queries and all other mutations?)

Serve what's possible / what you were serving before: What should happen if remote schema permissions reference a new field that doesn't exist yet because the remote schema hasn't been deployed? What if Hasura still let all requests through, filtering ones that referenced any fields that don't exist yet? This is similar to the concept of backward compatibility.

Dependency-aware deployments: What if your team had smarter deployment tooling? What if every time remote_schemas.yaml changed, our deployment system figured out which remote schemas also needed to be redeployed? (This is conceivable with something like repository_dispatch in GitHub Actions, dependencies in Buildkite, or directed acyclic graphs (DAGs) in GitLab).

We had an incident that occurred during a data migration, where one of our remote schemas suffered a barrage of invocations from Event Triggers. The service crashed due to excessive load, and suddenly Hasura's metadata was marked as inconsistent because the HTTP connection was broken. Moments later, the service was spun up again (three cheers for self-healing), but the metadata was never marked as consistent.

There's a GitHub issue discussing this and proposing solutions, among them a new component that periodically detects inconsistent metadata and reloads remote schemas.

The remote schema auto-reloader is something I'm considering building out in the open. In its first phase, it will just periodically detect inconsistent metadata (due to broken remote schema HTTP connections) and periodically reload. In its second incarnation, it might be Kubernetes-aware: it will know which Pods map to which Remote Schemas, and will watch for Pod crash and start events to know when exactly to reload remote schemas. It should be a fun project!

Hasura is a great tool. It comes with its difficult moments, but none that can't be surmounted.

Lots of the challenges I highlighted can be fixed by smarter tooling and better practices; for example:

greater decoupling of services, domains, and databases;

adopting an engine to facilitate long-lived transactions across multiple business domains;

having automated breaking change detection;

fostering greater awareness of where tight coupling exists;

building test suites to detect regressions;

making sure to have the latest state before "pulling" the schema;

leveraging a dependency-aware continuous deployment system.

Hasura and Microservices: The Good and The Bad

With the recent malfunction of Twitter Circles (a feature that only shows your tweets to a select set of followers) we were reminded of the importance of data privacy. Breaches are bad! In the best case, users are left unhappy and trust in the platform erodes. In the worst case, you have legal problems on your lap.

When we build APIs, we're providing a gateway into data, allowing clients the ability to read and mutate (change) the data. That gateway must be secured. It's important that clients can only read and mutate data they're allowed to read. Deciding whether a client is allowed to perform some action on some object is known as authorization.

Often, using roles is sufficient for locking down a system. Roles are easy to construct and easy to assign to users. However, roles lack granularity. Role-based access control is like sand: they're both coarse.

Sometimes, the decision to allow a user access needs to consider a great number of variables, which could mean performing an expensive query that traverses lots of different tables and data stores. Imagine a system that supports groups, and sub-groups, and sub-sub-groups, ad infinitum: "a long chain of nested group memberships" — or what you might call "recursive pointer chasing." Permissions in such a system are highly dynamic and can depend on lots of disparate pieces of information, which if computed on-the-fly at runtime could result in service fan-out.

Accounting for all the possible combinations is difficult. It's a real pain point a lot of enterprises face. Trying to do this in code reminds me of the meme about implementing an isEven function with bespoke cases for a countably infinite number of cases. Auth code often gets inadvertently bound to business features, making it hard to deliver new ones or change existing logic.

There are also major observability challenges with authorization. Baked-in auth makes the code hard to test and harder to debug. There isn't an easy place you can go to see who has access to what, whether authorization is working, or why requests are or aren't authorized.

What if instead of making this query on the fly, we could simply perform a lookup? What if we could decouple the logic of the authorization from our business logic? What if inspecting and modifying authorization rules was easy? What if we could easily see who has access to what, and could trace through why a request failed?

Lots of projects have been focusing on these questions, sparked by the release of Google's paper in 2019, titled "Zanzibar: Google’s Consistent, Global Authorization System". Zanzibar is a scalable, reliable, centralized solution for authorization that respects causal ordering (see section 2.2) of user actions. It's widely used across multiple Google apps: Calendar, Cloud, Drive, Maps, Photos, and YouTube. In short, it's fast, correct, and powerful. Since then, several open-source projects have provided "authorization as a service" — AuthZed's SpiceDB, OpenFGA, Ory's Keto, Topaz, just to name a few.

In contrast to role-based access control (RBAC) and attribute-based access control (ABAC), Zanzibar and its ilk use graph-based data structures to centralize data so access control lists (ACLs) are less cumbersome to manage. This is known as ReBAC: relationship-based access control.

In Zanzibar, to answer the decision question, it's as simple as performing a lookup and checking for the presence of a tuple (user, relation, object). You can conceptualize a tuple as a relationship in a directed graph. User groups define a set of rules on how to interpret a collection of tuples (a graph).

In cases where one relation implies another, we call this a concentric relation — for example, "owner" implies "editor", which implies "viewer". We thus have inheritance between relations to the same object. We can also model logical unions (any of the relations), intersections (all these relations), and exclusions (not these relations). We can model downward flowing permissions (e.g., if you can view the parent, you can view the child).

The process of transforming your thinking from RBAC and direct ACLs to  using a graph of tuples will teach you a lot about your business domain  as well.

Zanzibar Paper — with commentary from AuthZed

Permify — Google Zanzibar In A Nutshell

The Evolution of Ory Keto: A Global Scale Authorization System

AuthZ: Carta’s highly scalable permissions system

Himeji: A Scalable Centralized System for Authorization at Airbnb

From the Earth to the edge of the universe

We're able to look up at the night sky and see distant galaxies because outer space is transparent, not opaque, to light. Light is able to pass right through out, with its path and properties largely unchanged by any encountered objects. This wasn't always the case. In the earliest years of the universe, it would have been full of ionized particles which would have scattered all this light.

The word transparent comes from the roots trans ("across, beyond, through") and parere "come in sight, appear").

In systems engineering, we say a change or feature is "transparent" if the system after change adheres to the previous external interface as much as possible while changing its internal behavior. Everyone is shielded; the change causes no disruptions. Any human users, subsystems, and super-systems can continue interacting with the changed system, with their data flow and process fully intact and unperturbed.

A "white box" system. You can view its internals. This is literal transparency, not to be confused with what we're talking about.

Kevin Chen

Hasura and Temporal: How to Make the Perfect Hamburger

In Search of Style

Polymorphism with GraphQL, Hasura, Genqlient, and Golang

Custom Customer

Hasura and Microservices: The Good and The Bad

Fine-grained Authorization

Semantics of Light and Time

Running microservices with Taskfile