Asynchronous Communication with Manhattan Active® solutions via Google Pub/Sub

Introduction

Manhattan Active® Platform is API-first, cloud native, and microservices-based. For enterprise systems to integrate and communicate, Manhattan Active Platform provides both synchronous and asynchronous communication options. All asynchronous communication between Manhattan Active solutions and external systems is done via Google Pub/Sub. This document provides guidelines on how to set up asynchronous communication with Manhattan Active solutions over Google Pub/Sub.

What is Google Pub/Sub?

Google Cloud Pub/Sub is a real-time messaging service that allows applications to publish and subscribe to events. It’s a publish/subscribe-based mechanism for sending and receiving messages between applications and services. You can find more information on Pub/Sub on the Google Cloud website here: https://cloud.google.com/pubsub/docs/overview

Integrating with Google Pub/Sub

Google Pub/Sub provides two connectivity approaches:

1. Google provides an SDK that can be added as a library to your application: https://cloud.google.com/pubsub/docs/reference/libraries
2. REST API which can be invoked from your application: https://cloud.google.com/pubsub/docs/reference/rest

Authenticating to Pub/Sub

The Manhattan Associates project team will provide a JSON Key file to your integration team. The key file can be used to authenticate against Google Pub/Sub for both the integration approaches mentioned earlier - either the Google Pub/Sub SDK or the REST API. Please review Google Pub/Sub documentation for details on how to use a key file for authentication. Some external applications and middleware systems, such as Boomi, require a .p12 certificate instead for Pub/Sub Authentication, which the Manhattan project team can provide on request.

Pub/Sub Endpoints

The Manhattan Associates project team will create Pub/Sub endpoints for each external interface identified in the solution design. The interfaces may either be “Inbound”, where messages are sent to the Manhattan Active solution, or “Outbound” where messages are sent by the Manhattan Active solution out to external systems.

Outbound Interfaces

The Manhattan Active solution will publish outbound messages to a Pub/Sub topic, and the consumer(s) on the external system will have to read those messages from the corresponding Pub/Sub Subscription. The name of the Subscription will be in the following format: projects/{{project-id}}/subscription/{{interface-name}}

Note:

1. The project id will differ by environment – STAGE, VPT, or PRODUCTION. The project team will provide the project id value for each environment.

2. If multiple external systems need to consume a copy of the same message, then the external middleware system can consume the message from a single Pub/Sub subscription and fan out copies of the message to each receiver.

Inbound Interfaces

When external systems have to send messages to the Manhattan Active solution, they must publish those messages to the appropriate Pub/Sub Topic. The name of the Topic will be in the following format: projects/{{project-id}}/topic/{{interface-name}} If multiple systems need to send the same messages, they may send them to the same Topic. For example, if there are multiple Distribution Centers and Stores needing to send Inventory updates to Manhattan Active OMNI, they can all publish to a single Pub/Sub Topic.

The Manhattan project team will create the required Pub/Sub Topics and Subscriptions based on the solution design.

Message Payloads

Google Pub/Sub requires the message payloads to be Base64 encoded. The structure of the payload for each interface is documented in the Interface Mapping Document, created as a part of the Solution Design.

Message Payload Size

Google Pub/Sub limits the maximum size of a message to 10 MB. In case there is a possibility of messages exceeding the 10 MB limit for an interface, Manhattan Active Platform can be configured to send/receive GZIP compressed payloads for that interface. Note that when GZIP compression is enabled for an interface, it will apply to all messages for that interface, irrespective of whether specific messages exceed the 10 MB limit or not.

Message Acknowledgment

Once a consumer reads a message from a Google Pub/Sub Subscription, it must send an acknowledgment must be sent to Pub/Sub. This is used to ensure Guaranteed Delivery of messages. Please refer to Pub/Sub documentation on how to acknowledge receipt of messages.

Messages must be acknowledged even if message processing fails due to any kind of error, including validation errors. If a message is not acknowledged in a timely manner (by default 10 seconds), Google Pub/Sub will not remove the message from the subscription, leading to duplicate reads.

If a message sent from a Manhattan Active solution fails to process on the external system, the external system must not use a NACK response as it can cause infinite looping of that message. Instead, the failed message must be acknowledged and sent to a Dead Letter Queue for offline triage.

Duplicate Messages

To ensure Guaranteed Delivery of messages, Google Pub/Sub depends on message acknowledgments to determine if a consumer has received a message from the subscription. The message acknowledgments must be sent to Google Pub/Sub within a deadline, which by default is 10 seconds. If the acknowledgment threshold is crossed, Google Pub/Sub will assume the message was lost and will re-deliver the message to the consumer, which can lead to message duplication.

Manhattan Active Platform uses a custom Pub/Sub attribute to track inbound messages and identify duplicate messages being re-delivered by Pub/Sub. When sending messages to Manhattan Active solutions via Google Pub/Sub, the sending system must include the MSG_ID_PK attribute populated with a unique value. The unique value for the MSG_ID_PK attribute may be generated from a UUID library. The Manhattan Active solution can be configured to track each MSG_ID_PK value it reads from incoming messages and discard messages that have an MSG_ID_PK value that has been received previously. Since the duplication check works solely on the uniqueness of the MSG_ID_PK value, it is important that if the external system is resending a message, then it must not change the MSG_ID_PK value.

When the Manhattan Active solution sends outbound messages to external systems over Google Pub/Sub, it will include the MSG_ID_PK attribute to help external consumers identify and de-duplicate messages on their side.

Message Attributes

When an external system sends messages to the Manhattan Active solution, it must include the following Pub/Sub Attributes on every message:

Every message sent by a Manhattan Active solution to an external system over Google Pub/Sub will have the following attributes (not an exhaustive list):

Message Sequencing

There is no guarantee of sequencing/ordering of messages sent from the Manhattan Active Solution via Pub/Sub. If the external systems require sequencing of messages being sent by the Manhattan Active solution, it is recommended that the external middleware system stages the incoming messages first locally and then sequences it for its consumption as needed.

Author

• Devraj Roy: Architect, Technical Services, PSO.

References

• Google Pub/Sub Overview
• Google Pub/Sub Client Libraries
• Google Pub/Sub REST API