Let us know how we can help
Sales & Pricing
Speak to a rep about your business needs
Help & Support
See our product support options
General inquiries and locations
Contact us
Control-M + GCP Data Fusion
Your GCP Data Fusion workflow starts before a single pipeline or Workflow Template ever executes — and it can fail when something upstream breaks. A Cloud Storage object arrives late. A BigQuery load job misses its processing window. Control-M orchestrates the full chain: Cloud Storage data arrival, BigQuery transformation, GCP Data Fusion execution, and downstream delivery.
common workflow issues
Does this sound like your week?
These aren’t edge cases. They’re the normal operating conditions for teams running GCP Data Fusion pipelines across multiple tools. Here’s how Control-M handles each one.
CLOUD STORAGE · ETL
Your Cloud Storage data arrived late. The 6:00 AM pipeline already started.
Control-M makes data arrival an upstream dependency instead of relying on a fixed pipeline start time. The GCP Data Fusion job waits for the required condition before execution, preventing incomplete input from cascading into downstream processing.
PIPELINE FAILURE
Data Fusion failed overnight. Downstream BigQuery processing is still queued.
Control-M monitors the GCP Data Fusion job status, applies defined failure handling, and prevents dependent jobs from proceeding after an unsuccessful run. Operations sees the failed step in the wider workflow instead of discovering bad downstream output later.
RUNTIME PARAMETERS
Today’s partition changed. Your Data Fusion pipeline needs the right runtime values.
Control-M passes JSON-based runtime parameters into the GCP Data Fusion job, letting the workflow supply run-specific values without creating separate job definitions. Parameterized execution keeps recurring pipelines reusable while coordinating each run with its upstream context.
SLA RISK
Data Fusion is still running. The 8:00 AM analytics handoff is at risk.
Control-M tracks the GCP Data Fusion job as part of the end-to-end workflow and associates it with SLA management. Teams can identify timing risk in context and act before a delayed ETL run becomes a missed business delivery.
CROSS-TOOL DEPENDENCY
Data Fusion succeeded. Your downstream process still needs a reliable handoff.
Control-M detects completion through job-status monitoring and releases configured downstream dependencies only when the required conditions are met. Data Fusion becomes one governed step in the production flow rather than an isolated pipeline requiring separate scheduling logic.
INTEGRATION FACTS
Control‑M + GCP Data Fusion
|
workload.types |
batch ETL pipelines · single pipeline execution · Workflow Template pipeline execution · parameterized pipelines · pipeline abort · third-party job log retrieval |
|
trigger.type |
time schedule · upstream job completion · file arrival · Control-M dependency · API-driven submission · workflow condition |
|
cross_tool.deps |
Cloud Storage file arrival · BigQuery job · GCP Composer DAG · GCP Dataflow job · REST API call · downstream analytics job · file delivery confirmation |
|
cloud.platforms |
Google Cloud Platform · GCP Data Fusion · Cloud Storage · BigQuery · Cloud Composer · Dataproc |
|
error_handling |
status polling · failure tolerance · pipeline abort · downstream cascade prevention · job log retrieval · SLA management · dependency-based recovery |
|
throughput |
batch pipelines · real-time Data Fusion workloads · 50 simultaneous GCP Data Fusion jobs per Agent · configurable status polling |
|
observability |
pipeline status · pipeline results · pipeline output · third-party job logs · end-to-end dependency visibility · SLA monitoring |
end-to-end orchestration
One production workflow. Every tool in the stack.
Control-M orchestrates workflows across GCP Data Fusion, Cloud Storage, BigQuery, Cloud Composer, Dataflow, file transfers, and cloud services in a single job flow — with dependency tracking, SLA visibility, and automated recovery across all of them.
- Cross-tool dependency: Cloud Storage → GCP Data Fusion pipeline → BigQuery → analytics handoff
- Data-aware triggers: file arrival, API event, upstream job completion, pipeline completion
|
GCP Data Fusion |
pipeline execution · runtime parameters · status monitoring · logs · failure handling |
|
Cloud Storage |
file arrival dependency · upstream data readiness · file-driven workflow initiation |
|
BigQuery |
query execution · data processing · downstream dependency · analytics handoff |
|
GCP Composer |
DAG coordination · upstream/downstream dependencies · cross-workflow orchestration |
|
GCP Dataflow |
batch processing · streaming processing · job coordination · dependency management |
|
Dataproc |
Spark jobs · Hadoop workloads · processing dependencies · scheduled execution |
|
File transfers |
managed transfer · arrival detection · delivery confirmation · downstream release |
airflow coexistance
Control-M doesn’t replace your Airflow DAGs. It runs the layer above them.
The objection is common: “We’re already on Airflow.” The issue isn’t what Airflow does – it’s what happens before and after Airflow runs. That’s where pipelines actually fail.
Airflow manages its DAG. Control-M manages everything surrounding it.
airflow handles
DAG-level orchestration inside the data pipeline
- DAG-level task orchestration within data pipelines
- Python operators, sensors, and task dependencies
- Execution graph for jobs that run inside your pipeline
- Manages retries within a single DAG context
control-m adds
The coordination layer around your DAGs
- Coordination layer around DAGs — triggers Airflow based on upstream conditions: file arrivals, API events, other tool completions
- Tracks each DAG’s SLA contribution across the full end-to-end workflow, not just its own routine
- Manages failure recovery when upstream dependencies fail before Airflow even starts
- Existing DAGs don’t need to be rewritten or migrated
MONITOR PIPELINES
Monitor GCP Data Fusion execution across the workflow
GCP Data Fusion exposes pipeline-level execution information, but production dependencies often extend across services and platforms. Control-M brings Data Fusion status, results, output, and surrounding jobs into one operational view so teams can follow the complete workflow:
-
Pipeline execution status
-
Pipeline results and output
-
Retrieved third-party job logs
-
Upstream and downstream dependencies
-
End-to-end workflow visibility
SLA ASSURANCE
Protect delivery SLAs beyond the Data Fusion pipeline
A successful Data Fusion run does not guarantee that the complete data product arrived on time. Control-M associates GCP Data Fusion jobs with end-to-end SLA management, connecting pipeline execution to the upstream and downstream work that determines delivery:
-
End-to-end SLA tracking
-
Cross-platform dependency visibility
-
Upstream failure containment
-
Downstream execution control
-
Centralized operational monitoring
Documentation
Bring order to complex workflows
Learn how Control-M helps teams orchestrate complex processes with greater visibility, coordination, and control.
