The cloud workflow depends on one Windows gateway nobody can rebuild

The dashboards and approvals are cloud services, but their route to the accounting database is one Windows machine under an administrator’s desk. A routine reboot turns the gateway offline, scheduled refreshes fail, and a replacement installation creates a new empty gateway. The existing connections remain bound to the unavailable cluster object. The repair requires a tested continuity system with independent cluster members, recorded dependencies and a replay plan.

The situation: cloud availability ends at one Windows service

COMPOSITE SCENARIO: documented product behaviour combined with recurring operational failure patterns. No client-specific delivery claim is attached. Consider a Perth wholesaler using Power BI for morning stock and margin reporting and Power Automate to write approved order adjustments into an on-premises SQL system. Both workloads use an on-premises data gateway installed in standard mode on a small Windows VM. The gateway was configured by a contractor; the operational vault contains no recovery key, only one person is a gateway administrator, and no dependency register joins cloud connections to data sources.

Windows maintenance restarts the host overnight. The service remains stopped because the VM has exhausted disk space and a proxy change blocks its outbound relay path. Power BI refreshes fail before management arrives. Power Automate runs remain failed or waiting. An operator installs the gateway on another server but registers a new gateway instead of restoring or joining the existing cluster. The new object has no usable bindings, while staff begin manually replaying flows.

The niche failure is a continuity boundary hidden by cloud branding. The gateway service, host, recovery key, connection credentials, service account, DNS, firewall/proxy route and on-premises data source form one operating chain. If any unique link fails, the cloud workload is unavailable.

Contain failed automation before replay

Name an incident lead, business decision owner, gateway administrator, data-source owner and network owner. Pause or isolate cloud processes whose failed or uncertain requests can create financial, stock, access or customer effects. Keep reporting refreshes distinct from write-capable flows: a missed dashboard refresh and an order update with unknown outcome require different recovery decisions.

Capture the gateway cluster and member IDs, status and version; host/service state; failure timestamps; Power BI refresh or DirectQuery errors; Power Automate run IDs and connector responses; connection/data-source mappings; proxy and network-port test results; Windows event and gateway logs; CPU, memory, free disk and spool state. Do not export credentials or recovery keys into the incident record.

Before resubmitting a write action, check the destination. A timeout or gateway failure can occur after the database accepted the operation but before the cloud service received confirmation. Reconcile by business key and use the controlled idempotency pattern. A failed response can follow a committed target operation.

Map the whole availability chain

Start at the cloud workload and trace through its connection, gateway cluster, member, outbound Azure Relay path, proxy/firewall, DNS, service account, driver/provider, data-source network route, database identity and target operation. Record owner, configuration source, monitoring and recovery method at every boundary.

A second gateway member only removes the gateway-host single point if it is installed on a separate failure domain and can independently reach the same data sources with compatible drivers, service configuration and outbound connectivity. Two services on the same VM, two VMs on one failing host, or two nodes behind the same untested proxy do not provide the claimed resilience.

Gateway high availability covers the gateway process and host. SQL Server, file shares, Active Directory, DNS, firewall and the upstream application remain separate dependencies. State the continuity claim narrowly: the cluster can route a new request through another available member when one gateway member is unavailable. Interrupted operations retain workload-specific replay and transaction semantics.

Capture the gateway host before repair

Get-Service -Name PBIEgwService |
  Select-Object Name,Status,StartType
Get-CimInstance Win32_LogicalDisk -Filter "DeviceID='C:'" |
  Select-Object DeviceID,FreeSpace,Size
Test-NetConnection -ComputerName <approved-relay-fqdn> -Port 443

A stopped service directs the next branch to Windows service and gateway logs. Low disk requires a controlled cleanup and log-retention check. A failed TCP test sends the incident to DNS, proxy and firewall evidence before gateway reinstallation. Use Microsoft's current endpoint list for the installed region; do not substitute a guessed wildcard.

Build the cluster as an operating unit

Install the gateway in standard mode on at least two supported, appropriately sized Windows hosts for business-critical workloads. Place members in separate host and maintenance failure domains while keeping network latency to the data sources acceptable. Join the second installation to the existing cluster using the controlled recovery key; do not create an unrelated gateway object.

Keep members on a supported and consistent gateway version. Gateway updates are released frequently. The continuity schedule must include version review, sequential member updates and post-update workload tests. Patch one member at a time and verify it before moving to the next. Remove or disable persistently offline members through the supported administration surface and prevent routing attempts to them.

Choose load-distribution settings from measured concurrency and resource capacity. Failover and load balancing are related but not identical goals. Monitor CPU and memory thresholds, query concurrency, duration, disk and spool usage on each member. If every node is saturated or shares the same blocked route, clustering cannot manufacture capacity.

Control the recovery key and connection state

The recovery key protects gateway configuration and is required to migrate, restore or take over the existing gateway. Microsoft cannot retrieve it. Store it in an approved secret-management process with access by at least two accountable operators, tested retrieval, audit, break-glass conditions and a rotation decision. Do not place it in general documentation.

Document the exact restore path before an incident: installer source, sign-in authority, cluster selection, recovery-key retrieval, host prerequisites, drivers, proxy, service identity and validation sequence. Deleting a cluster destroys the recovery target and triggers a separate rebuild path.

Changing the recovery key has workload-specific consequences. Current Microsoft guidance states that Power BI connection credentials are re-encrypted automatically, whereas connections used by services such as Power Apps and Power Automate must be edited to trigger encryption with the new key. Test every bound service before retiring the legacy key.

Prove node loss with production-shaped work

Run a normal Power BI import refresh, a DirectQuery interaction if used, and representative Power Automate read and write actions. Record which member processes each request and establish baseline duration and resource usage. Then remove one member from service through an approved maintenance action and repeat every journey using new requests.

Confirm refresh, query and connector operations route through an available member. Restore the node, validate health and repeat with the other member unavailable. Test proxy/network-port failure, credential rejection, unavailable data source, disk pressure and a gateway update on one member. Do not induce uncontrolled failure against financial production writes; use synthetic records or an approved test environment.

For write workflows, include failure after destination commit, retry and manual replay. Acceptance requires one authoritative business outcome. A successful gateway response covers transport and execution only. For reporting, verify freshness timestamps and the user-visible stale-data warning on every tested report.

Monitor before users report stale data

Monitor cluster/member online state, service health, version drift, CPU, memory, disk and spool capacity, outbound network tests, query duration, request volume, refresh failures, Power Automate gateway errors and data-source authentication failures. Gateway performance logs include request and query identifiers, durations and workload type; machine bandwidth and latency require separate host/network monitoring.

Alert as soon as cluster capacity falls below two ready members. Total outage is a separate critical alert. A cluster with one healthy member is already in a degraded state. Track ownerless gateways, overshared gateway/data-source access, expired service credentials, repeated offline members and connection changes. Keep verbose diagnostic logging temporary because it can materially increase log volume and may contain sensitive query information.

Every alert needs an owner, severity, response action and evidence link. Assign the degraded-cluster alert to an on-call role, escalation time and tested notification route.

Make recovery a scheduled proof

Run quarterly member-failover and annually bounded restore/takeover exercises, plus tests after gateway, proxy, firewall, driver, service-account or data-source changes. Patch members sequentially. Review the recovery-key escrow and administrator set without exposing the key. Reconcile the complete connection inventory after key rotation or cluster migration.

Record recovery-time and recovery-point objectives per workload. DirectQuery, scheduled refresh and write-capable automation have different tolerance and integrity needs. Define the manual service state when the gateway is unavailable, including how staff identify stale reports and where new requests wait without duplication.

Fabric and Power Platform community reports are useful signals for interface changes and field symptoms, but failover behavior and supported recovery procedures must resolve to current Microsoft guidance and local test evidence. The system is controlled only when a second operator can lose a member, preserve business integrity and restore service without reconstructing the design from memory.