Configure Data Stream for Google PubSub

Objective

This guide will walk you through the steps for enabling the Data Streaming from Manhattan Active® to Google Cloud Pub/Sub endpoint owned and managed by the customer. The below steps also guides authorization and network access required to post events to this Pub/Sub endpoint.

Gravina is Manhattan Active® Platform’s data replication solution to provide most real-time data streaming service to target systems. It provides set of distributed services that capture row-level changes in databases so that applications can see and respond to those changes. DataStream utilizes JSON for exchanging data between internal & external components.

Google Cloud Pub/Sub provides messaging between applications. Cloud Pub/Sub is designed to provide reliable, many-to-many, asynchronous messaging between applications. Publisher applications can send messages to a “topic” and other applications can subscribe to that topic to receive the messages.

Before You Begin

• Deployment model:

  • The current scope of this document considers the target Google Cloud Pub/Sub, is owned and managed by Customer. To meet customer business and preferences, we continue to extend our ability to support multiple deployment models.

• Target Google Cloud Pub/Sub configuration:

  • Google Project: Customer owned Google project.
  • Region/Location: Google Cloud Pub/Sub is Global Service. but, we prefer to have both producer and subscriber in the same region.
  • Pub/Sub topic name ( Optional ): Any preferred topic name to use. By default, Manhattan Active® uses the default gravina_cdc_stream_<customer

• Network Connectivity & whitelist:

  • Pub/Sub Targets: At present we support Google Cloud Pub/Sub.
  • Network Traffic: Since both Source and targets are managed in GCP, traffic is routed through Google backbone interface.
  • Access whitelist : If Customer Google Cloud Pub/Sub endpoint is allowed to connect only from the trusted IP sources, then Manhattan Active® Platform NAT range needs to whitelist in Customer network.

• Data inclusion list for DataStreaming

  • By default, Gravina ignores all tables for DataStreaming unless we define to include in replicator configuration.
  • Get the list of database schemas and tables required to enable for DataStream to target system.

• Subscription Strategy:

  • By default, gravina creates the default subscription which can be used by Customer Application to process the data stream events from Pub/Sub
  • Customer owns the configuration of target application subscription(s) and attach to Pub/Sub topic and processing of messages.

Steps

• Generate service-account & Key:

  • Login to Google Cloud Console –> Go to Google Cloud Project –> IAM & Admin
    • Service Accounts
    • Create Service Account : Ex: gravina-pubsub-sa
    • Go to “Keys”
    • ADD KEY –> Create New Key –> Select Key Type : JSON –> Create –> The JSON Key file gets downloaded.
    • Provide the JSON Key file to Manhattan CLoud Ops Team in secured way.

• Grant Pub/Sub Editor permissions to service-account:

  • Login to Google Cloud Console –> Go to Google Cloud Project –> IAM & Admin
    • IAM
    • Click on “ADD”
    • Give the gravina-pubsub-sa as principle & Role as “Pub/Sub Editor”
    • Save

• Provide the configuration details to Manhattan Cloud team for replicator configuration.

  • Send the Service account “JSON Key file” which have the following details
    • project_id
    • service-account-id
    • Key & secret
  • Pub/Sub Topic Name ( If any explicitly needs to configure )
  • Inclusion Table List ( If only required specific tables )

• Configure Network Access Whitelist in Customer Google Cloud project:

  • If the Customer Google project is restricted access, then Manhattan team provide the NAT IP range from which the traffic will be initiated to Pub/Sub endpoint.
  • Customer needs to whitelist the NAT IP range in Google Cloud Project to allow the access.

• Target Pub/Sub topic Subscription:

  • The configuration of subscription to topic and processing of messages are completely managed by Customer based on customer business needs.
  • By default, gravina creates the subscription which can be configured and used by Customer Application to process the data stream events from Pub/Sub topic.

• Monitoring & Alerts:

  • Google Cloud Console provides the default monitoring statistics & metrics.
    • Publish message request count.
    • Average message size.
    • Unacked message count.
    • Oldest unacked message age.
  • Customer responsible to configure the required monitoring and Alerting based on the business needs.

Learn More

• Concept Guides
• Quick Starts

Authors

• Srinivasa Rao Jammula: Director, Manhattan Active® Platform, R&D.