Guides

Event Sourcing and Commands

Suggest Edits

In the previous page, we established that manifests and movements are the core building blocks of how Maven models the daily execution of a linehaul network. As a dispatching system, Maven does not create the manifests and movements and expects the fleets TMS to send these objects over as they are created and updated throughout the day.

Traditionally this would mean sending over the manifest object and movement object directly via a CRUD style API. While this can work, there are a few drawbacks to this approach: most notably the difficulty in maintaining a clean history of the actions that took place in executing the operation. Being able to produce accurate history of the linehaul network is critical for operations, customer visibility, reporting, and analysis for process improvement. Maven feels so strongly about this fact that we designed the entire linehaul system around an architectural pattern called event sourcing.

What is Event Sourcing?

In a traditional system, users perform actions called commands. Commands trigger updates to data objects that are then persisted in a database. The interactions with the database can be categorized as either a Create, Read, Update, or Delete (CRUD). A Create operation will add a new record to the database. A Read operation will access data that is already there. But it's the Update and Delete that we are interested in, because both operations are destructive. An Update will overwrite the previous state with something new, essentially forgetting about the past history, while a Delete actively destroys an existing record.

Event Sourcing takes a different approach. When a command from a user is received, the event-sourced system will translate it into an event. The event can be used to update any state held in memory, but only the event is stored in the database. Events are considered to be something that happened in the past. This means, they can be created, or read, but they can’t be updated or deleted. You can’t rewrite history. This prevents any potential for data loss because all events are maintained from the very beginning.

For example, in a banking system a command might be “WithdrawMoney”. The corresponding event could be “MoneyWithdrawn”. Note the difference in naming. Events are always written as past tense to illustrate their historical nature. When a “MoneyWithdrawn” event is applied, the in-memory balance of the bank account can be adjusted by the appropriate amount. However, the balance itself isn’t stored in the database, only the events are.

Maven follows this approach by creating an API that ingests commands that indicate some real world action that was performed by the operation. Maven has designed the catalog / library of commands to most closely match the events that can take place within a linehaul operation. Therefore when viewing the history of a manifest or movement, it should be very intuitive to understand what occurred.

What is a Command?

A command represents an intention to perform an action (i.e. I want to do something). For example ArriveMovement

When processing a command, validation on the command is performed to ensure it is a valid command. (i.e. has the movement departed, does the movement exist in the system?).

If validation fails, the command is rejected.

If validation succeeds, the associated event is created and saved in the event stream (i.e. I did something). For example MovementArrived.

Anatomy of a Command

{
  commandType: "NameOfCommand",
  payload: {
    // JSON Payload
  },
  referenceIds: {
    // Ex. Load ID
  },
  eventSource: {
    userId: "12093", // User ID (if exists)
    displayName: "Kent, Clark", // Can be "SYSTEM" if no userID
    system: "AS400" // Ex. Scheduler / AS400 Dispatch / etc...
  }
}`
  • commandType: This is the name of a command (i.e. ArriveMovement)
  • payload: This is a JSON payload of the information to pass along in a command
  • referenceIds: This is a space to uniquely identify the object in which the command is being performed on. Reference IDs are defined by the integrating system and must be unique and remain constant throughout the lifecycle of the object. For example, a loadId can be passed in as a referenceId when calling the createManifest command which will be stored alongside the created manifest in Maven. To send updates for the same manifest, pass in the loadId referenceId in subsequent commands.
  • eventSource: This is meta data to provide context about who performed the command. Was it system generated? Was it a specific user who performed this action? This information is used for auditing commands/events when viewing history.

Commands API

To send linehaul commands to Maven - use the following API endpoint: (Refer to API documentation here)

POST https://integrations.mavenmachines.com/manifests/commands

This endpoint is used to send both manifest and movement level commands. If you're wondering why there isn't a separate movements/commands endpoint, remember that movements belong to manifests so a movement command really is just a special flavor of a manifest command.

The commands API is designed to ingest batches of commands. This means, for large operations that are generating lots of commands in short periods of time, the endpoint can be called once with a list of commands. For smaller operations, it is also acceptable to call the endpoint individually per command.

In summary, the payload of the command API is simply an array of commands:

{
  commands: [
    { 
      // command 1 payload 
    }, 
    { 
      // command 2 payload 
    },
    ...
  ]
}

Reference IDs

Maven integrates with a variety of different systems that each have their own constraints about what constitutes a unique identifier for a manifest or movement. Some may call it manifestId and others may call it loadIdor tripNumber. Rather than Maven enforcing a rule saying a manifest unique ID must be called manifestId or a movement unique ID must be called movementId, we introduce the concept of reference IDs to allow the integrating system to create their own definition of uniqueness.

The reference IDs field in the command payload is a map of key/value pairs that is used to uniquely identify the manifests and movements sent in the API. Care must be taken to ensure that the values passed in the reference IDs will remain unchanged throughout the lifecycle of the respective object as well as remain unique over time.

Below is an example of creating a manifest that is going from Philadelphia to Charlotte. 

{
  commandType: "CreateManifest", 
  payload: {
    manifestNumber: "PHLCLT001"
  },
  referenceIds: {
    loadNumber: "2024-12-11-9204820"
  }
}

At first, we might think that the manifestNumber can be used as a reference ID but assuming the manifest PHLCLT001 is run every day, this ID would not become unique after it was used the first time. So in this example, the integrating system generates a unique load number that includes the date + random ID. The reference ID is a key value pair with the key being loadNumber and the value being 2024-12-11-9204820.

The fields in the commands payload are not used to identify the object - only the reference IDs. To prove this point, suppose that a few hours after this manifest was made, the destination needs to change from Charlotte to Richmond and we want to update the manifest number accordingly. 

{
  commandType: "UpdateManifestDetails", 
  payload: {
    manifestNumber: "PHLRMD001"
  },
  referenceIds: {
    loadNumber: "2024-12-11-9204820"
  }
}

As long as the same reference ID is used for both commands, Maven will know that this update applies to the manifest that was previously named PHLCLT001.

It is also possible to send multiple reference IDs for a single object / command. For example: 

{
  commandType: "CreateManifest", 
  payload: {
    manifestNumber: "PHLCLT001"
  },
  referenceIds: {
    loadNumber: "2024-12-11-9204820",
    hookNumber: "HOOK-991039001"
  }
}

In the example above, we are associating 2 reference IDs to the PHLCLT001 manifest number. Because these references were sent together in one command, Maven can now ingest future commands for this manifest by only passing one of the reference IDs. For example:

{
  commandType: "UpdateManifestDetails", 
  payload: {
    scheduledCloseTime: "2024-06-04T14:01:34+000"
  },
  referenceIds: {
    loadNumber: "2024-12-11-9204820",
  }
}

{
  commandType: "PlanShipmentsOnManifest", 
  payload: {
    eventTime: "2024-06-04T12:01:00+000",
    proNumbers: ["90239402"]
  },
  referenceIds: {
    hookNumber: "HOOK-991039001"
  }
}

Both of these commands will be processed for the PHLCLT001 manifest number.

Some additional notes:

  • Another way to think of reference IDs is that its the IDs used for the computers to communicate with one another - NOT what the humans see in the UI. The reference IDs are just used for Maven to uniquely look up a manifest or movement. While the example above are using the reference keys of hookNumber and loadNumber, these keys may be different for your system. (Ex. an internal database ID)
  • Reference IDs don't just need to be defined on the initial creation commands. It is possible to introduce additional reference IDs halfway through an object's lifecycle if the integrating system requires it.
  • A manifest and movement will have their own reference IDs. Therefore, the referenceId to pass in for manifest level commands will be different than the referenceIds passed in for movement level commands.

Associated Reference IDs

As mentioned in the manifest and movements article, movements are owned by manifests. This is important to understand conceptually as it relates to how reference IDs are use when defining movements and manifests together.

Consider the scenario of generating a manifest from Philadelphia to Charlotte that relays through Richmond. This is 1 manifest with 2 movements. The CreateManifest command is shown below: 

{
  commandType: "CreateManifest", 
  payload: {
    manifestNumber: "PHLCLT001",
    originLocationCode: "PHL",
    destinationLocationCode: "CLT",
    associatedMovements: [{
      sequenceNumber: 1,
      referenceIds: {
        legId: "2024-12-11-1820301"
      }
    }, {
      sequenceNumber: 2,
      referenceIds: {
        legId: "2024-12-11-1080123"
      }
    }],
  },
  referenceIds: {
    loadNumber: "2024-12-11-9204820",
  },
}

In this example, we are creating a manifest with 2 movements. Although we have not yet sent in the commands to create the movements, we had to pass in the reference IDs of the movements so that we could associate them to the manifest.

Let's now create the movements:

{
  commandType: "CreateMovement", 
  payload: {
    originLocationCode: "PHL",
    destinationLocationCode: "RMD",
  },
  referenceIds: {
    legId: "2024-12-11-1820301",
  },
}

{
  commandType: "CreateMovement", 
  payload: {
    originLocationCode: "RMD",
    destinationLocationCode: "CLT",
  },
  referenceIds: {
    legId: "2024-12-11-1080123",
  },
}

Because we previously associated the movement's reference IDs in the CreateManifest command, Maven knows to associate these movements with the PHLCLT001 manifest.

Batch Processing

The simplest way to call the command API is to send one command per HTTP request. This will work for most use cases and is the preferred method for most fleets. However, if your system is generating a lot of commands in a short time period, it makes the most sense to send multiple commands in one HTTP request. This helps reduce the amount of network calls being made. On Maven's side, sending commands in batches is split up and processed in parallel to ensure higher throughput and performance.

When sending a batch of commands, multiple scenarios can occur that need to be considered for processing. What happens if multiple commands are sent for the same manifest or movements? What happens if validation fails for one of the commands? Understanding how Maven processes multiple commands in one request is important for properly developing and testing your integration.

Consider the diagram below:

When Maven ingests a request to the commands API, the first step is to preform command grouping. This is done so that all commands that relate to each other are processed sequentially in the order they are sent. Command grouping is done by using the reference IDs. That is to say - if 3 commands are sent in for reference ID A - all of those commands will be grouped together and processed 1 by 1 (as shown in the diagram above).

Once the commands are grouped - each group is processed in parallel - with each command in the group being processed sequentially. Processing involves performing validation on the command and then saving the respective event if validation is successful.

Once all events are saved, the responses (i.e. was the command processed successfully) are mapped back to the original request in the respective order in which they were sent.

In practice this looks like the following:

HTTP Request

{
  commands: [{
    commandType: "GenericCommand", 
    payload: {},
    referenceIds: {
      loadNumber: "A"
    }
  }, {
    commandType: "GenericCommand", 
    payload: {},
    referenceIds: {
      loadNumber: "B"
    }
  }, {
    commandType: "GenericCommand", 
    payload: {},
    referenceIds: {
      loadNumber: "A"
    }
  }]
}

HTTP Response

{
  commandReponses: [{
    success: true
  }, {
    success: true
  }, {
    success: true
  }]
}

📘

Processing Parallel Requests

The command processing engine is designed for high throughput by processing command groups in parallel. Within the context of a single API request, Maven guarantees that for a given command group, each command will be processed sequentially.

However, if multiple API requests are made in parallel that each contain commands that refer to the same reference IDs (i.e. sending multiple updates to the same manifest in 2 separate API calls at the same time) - Maven cannot guarantee the order in which those individual commands are processed.

In fact, if enough parallel calls to the same object (i.e. a manifest or a movement) are made at the same time - Maven will return a Too Much Contention error and commands will need to be resent.

Therefore it is recommended that when sending multiple commands for the same entity (manifest and its movements) - ensure those are sent in the same API request or one after another.

Errors

When processing an invdividual API request, there are a few categories of errors that can occur.

Schema Validation Errors

Schema validation errors occur when the request is malformed. For example, a missing required field, using the wrong data type, or sending in an empty payload. When a schema validation error is encountered, a 400 HTTP response is sent back along with details of the schema validation error. Importantly, if a schema validation error is encountered, no commands are processed. Command processing occurs only if there are no schema validation errors found in the request.

Command Validation Errors

Command validation errors occur if the business rules of individual commands are invalid. For example, sending in a driver code that doesn't exist, unloading time < loading time, or arriving a movement that hasn't departed. Within a command group, if command validation fails, the validation error is recorded and all subsequent commands within the group are not processed. Processing of other command groups are unaffected.

A 200 HTTP response is still returned if any individual command fails command validation.

Consider the diagram and HTTP request below:

HTTP Request

{
  commands: [{
    commandType: "CreateManifest", // Command 1
    payload: {
    },
    referenceIds: {
      refId: "A"
    }
  }, {
    commandType: "UpdateManifestDetails",  // Command 2
    payload: {
      destinationLocationCode: "DOES_NOT_EXIST"
    },
    referenceIds: {
      refId: "A"
    }
  }, {
    commandType: "CreateManifest",   // Command 3
    payload: {},
    referenceIds: {
      refId: "B"
    }
  }, {
    commandType: "UpdateManifestDetails",  // Command 4
    payload: {
      manifestNumber: "1234"
    },
    referenceIds: {
      refId: "A"
    }
  }, {
    commandType: "UpdateManifestDetails",  // Command 5
    payload: {
      originionLocationCode: "PHL"
    },
    referenceIds: {
      refId: "B",
      refId: "C"
    }
  }, {
    commandType: "UpdateManifestDetails",  // Command 6
    payload: {
      destinationLocationCode: "RMD"
    },
    referenceIds: {
      refId: "C"
    }
  }]
}

Command 2 will fail command validation because a bad location ID was sent in. This means that command 4 will not be processed but the other command group is unaffected. This plays out as follows:

HTTP Response

{
  commandReponses: [{
    success: true // Command 1
  }, {
    success: false, // Command 2
    errors: ["Command Validation Error: Desination location code does not exist"]
  }, {
    success: true // Command 3
  }, {
    success: false,// Command 4
    errors: ["Command Processing Error: Command not processed because preceding command failed validation or processing"]
  }, {
    success: true // Command 5
  }, {
    success: true // Command 6
  }]
}

Command Processing Errors

Command processing errors occur if a command isn't processed because a preceding command failed validation or if a server error occurred. In this case, an HTTP 200 response will still be sent with details of the command processing error shown in the response.

Event Sources

To provide better traceability and audibility of who performed a command and/or where it came from - you can use the eventSource field to pass in useful context. 

{
  commandType: "GenericCommand",
  payload: {
    // JSON Payload
  },
  referenceIds: {
    // Ex. Load ID
  },
  eventSource: {
    userId: "12093",
    displayName: "Kent, Clark",
    system: "AS400" 
  }
}`

The eventSource field is an optional object that accepts following fields. Note

Field NameField TypeDescription
userIdstringUnique identifier of the user who performed the command (ex. 10293)
displayNamestringHuman readable display name of the user who performed the command (ex. Kent, Clark)
systemstringHuman readable name of the system where the command origination from (i.e. Scheduler, AS400, TMS, etc...)

Note all the source fields are optional. None, 1, 2, or all 3 fields can be passed in to the command.

Updated 6 months ago