Configure Data Stream for Google PubSub

Learn how you can configure Data Stream for your target Google Cloud Pub/Sub endpoint.

Objective

This guide will walk you through the steps for enabling the Data Streaming from Manhattan Active® to Google Cloud Pub/Sub endpoint owned and managed by the customer. The below steps also guides authorization and network access required to post events to this Pub/Sub endpoint.

Gravina is Manhattan Active® Platform’s data replication solution to provide most real-time data streaming service to target systems. It provides set of distributed services that capture row-level changes in databases so that applications can see and respond to those changes. DataStream utilizes JSON for exchanging data between internal & external components.

Google Cloud Pub/Sub provides messaging between applications. Cloud Pub/Sub is designed to provide reliable, many-to-many, asynchronous messaging between applications. Publisher applications can send messages to a “topic” and other applications can subscribe to that topic to receive the messages.

Before You Begin

  • Deployment model:

    • The current scope of this document considers the target Google Cloud Pub/Sub, is owned and managed by Customer. To meet customer business and preferences, we continue to extend our ability to support multiple deployment models.
  • Target Google Cloud Pub/Sub configuration:

    • Google Project: Customer owned Google project.
    • Region/Location: Google Cloud Pub/Sub is Global Service. but, we prefer to have both producer and subscriber in the same region.
    • Pub/Sub topic name ( Optional ): Any preferred topic name to use. By default, Manhattan Active® uses the default gravina_cdc_stream_<customer
  • Network Connectivity & whitelist:

    • Pub/Sub Targets: At present we support Google Cloud Pub/Sub.
    • Network Traffic: Since both Source and targets are managed in GCP, traffic is routed through Google backbone interface.
    • Access whitelist : If Customer Google Cloud Pub/Sub endpoint is allowed to connect only from the trusted IP sources, then Manhattan Active® Platform NAT range needs to whitelist in Customer network.
  • Data inclusion list for DataStreaming

    • By default, Gravina ignores all tables for DataStreaming unless we define to include in replicator configuration.
    • Get the list of database schemas and tables required to enable for DataStream to target system.
  • Subscription Strategy:

    • By default, gravina creates the default subscription which can be used by Customer Application to process the data stream events from Pub/Sub
    • Customer owns the configuration of target application subscription(s) and attach to Pub/Sub topic and processing of messages.

Steps

  • Generate service-account & Key:

    • Login to Google Cloud Console –> Go to Google Cloud Project –> IAM & Admin
      • Service Accounts
      • Create Service Account : Ex: gravina-pubsub-sa
      • Go to “Keys”
      • ADD KEY –> Create New Key –> Select Key Type : JSON –> Create –> The JSON Key file gets downloaded.
      • Provide the JSON Key file to Manhattan CLoud Ops Team in secured way.
  • Grant Pub/Sub Editor permissions to service-account:

    • Login to Google Cloud Console –> Go to Google Cloud Project –> IAM & Admin
      • IAM
      • Click on “ADD”
      • Give the gravina-pubsub-sa as principle & Role as “Pub/Sub Editor”
      • Save
  • Provide the configuration details to Manhattan Cloud team for replicator configuration.

    • Send the Service account “JSON Key file” which have the following details
      • project_id
      • service-account-id
      • Key & secret
    • Pub/Sub Topic Name ( If any explicitly needs to configure )
    • Inclusion Table List ( If only required specific tables )
  • Configure Network Access Whitelist in Customer Google Cloud project:

    • If the Customer Google project is restricted access, then Manhattan team provide the NAT IP range from which the traffic will be initiated to Pub/Sub endpoint.
    • Customer needs to whitelist the NAT IP range in Google Cloud Project to allow the access.
  • Target Pub/Sub topic Subscription:

    • The configuration of subscription to topic and processing of messages are completely managed by Customer based on customer business needs.
    • By default, gravina creates the subscription which can be configured and used by Customer Application to process the data stream events from Pub/Sub topic.
  • Monitoring & Alerts:

    • Google Cloud Console provides the default monitoring statistics & metrics.
      • Publish message request count.
      • Average message size.
      • Unacked message count.
      • Oldest unacked message age.
    • Customer responsible to configure the required monitoring and Alerting based on the business needs.

Learn More

Authors

  • Srinivasa Rao Jammula: Director, Manhattan Active® Platform, R&D.