The production app still follows a hotfix that no longer exists in development

The situation: a green deployment and the wrong runtime

COMPOSITE SCENARIO: documented product behaviour combined with recurring operational failure patterns. No client-specific delivery claim is attached. Consider a Perth field-service operator with a model-driven app for site visits. A business rule makes a safety explanation mandatory when an inspection fails, and a solution-aware cloud flow sends the completed record to an integration API. Development and test behave correctly. Production accepts managed solution version 4.7.3, and the deployment history reports success, but one form still uses the pre-release rule while the flow sends to an older endpoint.

Three months earlier, an urgent production edit changed the managed form directly. That edit created an active unmanaged layer above the managed component. The immediate incident was closed, but the change was never reconstructed in development. Separately, the production environment-variable current value still names the former API, and the flow’s connection reference resolves to a connection owned by a departed maker. The release artifact, target configuration and effective runtime state no longer describe one system.

This is environment drift: a material difference between the intended, tested release and what a target environment actually executes. It can arise from direct edits, solution layering, deployment order, missing components, environment-specific values, connection bindings, security, configuration data or platform-generated behavior. “It worked in test” removes some source defects from the first branch. Production layers, values, connections, security and data still require inspection.

Contain the drift before cleaning it

Name a release lead, product owner, Dataverse administrator, integration owner and test authority. Freeze direct customisation and further imports for the affected components. Pause write-capable flows if they can create unsafe, financial or duplicate effects. Preserve the current managed solutions, versions, import times, deployment run, component layers, dependencies, environment-variable definitions and values, connection references, underlying connections, flow state and representative failed records.

Do not start with “remove all active layers.” An unmanaged layer may contain an authorised emergency change that has not reached source, or it may have been created by platform behavior such as establishing a fallback form during import. Bulk removal without attribution can discard the only working version of a rule. Likewise, force-overwriting unmanaged customisations can replace unknown production state without proving that the managed artifact contains the intended behavior.

State the operational boundary. If the form is wrong but record creation remains safe, preserve read access and isolate the failing transition. If the integration destination is uncertain, queue or stop outbound work and reconcile by business identifier before replay. The first goal is to identify and control the effective production state. Screen-to-screen agreement is a later hygiene task.

Separate artifact, effective layer, target configuration and data

A Dataverse component can have managed layers plus one shared unmanaged layer. Managed layers follow their installation order and component merge behavior; an active unmanaged customisation sits above them and can determine runtime behavior. A displayed solution version therefore proves neither that every relevant component was included nor that the newest managed definition is the effective definition.

Environment variables are solution components used to separate portable application logic from values that differ by environment. Their definitions and current/default values require deliberate release handling. A stale current value can be locally valid yet operationally wrong. Connection references are also solution components, but they point to real connections whose authentication, ownership and sharing are target-environment concerns. A reference with the expected name can still resolve to an expired or unsuitable identity.

Pipelines can deploy solutions and prompt for target configuration including connections, connection references and environment variables. Solutions do not carry ordinary rows stored in Dataverse tables. Reference data, security-role assignments, team membership, feature switches or configuration tables may therefore differ even when the artifact is identical. Keep these four columns distinct in the evidence pack: shipped artifact, effective component layer, target binding/value, and target data/security.

Reconstruct effective production state component by component

Begin with the failing journey and expand only when the component dependency requires it. Record the model-driven app, form, business rule, table columns, flow, connector, environment variables, connection references, plug-ins and configuration records it touches. For every solution-aware component, inspect solution layers and identify the top layer, publisher, solution/version and modification context. Compare the component identifiers and dependencies with test and the source solution.

For each environment variable, capture the definition schema name, data type, default value, target current value, consuming components, owner and whether the value was intentionally supplied during import. For each connection reference, capture its connector, bound connection, connection owner or service principal, sharing, authentication health and the apps/flows that use it. Do not place secret values or tokens in the pack.

Join this to deployment history: source and target environment IDs, pipeline host, solution unique name/version, artifact checksum where available, validation result, deployment notes, operator/service identity and timestamps. A pipeline host may contain only part of the deployment history. Query every configured host and preserve solution, stage, initiating identity and timestamp before declaring a release absent. Add audit and change records where licensing and configuration permit, but distinguish evidence retained by the platform from human recollection.

Classify the drift before selecting a repair

Solution Checker is useful static analysis for supported solution components and Object Checker can help with live metadata/layering diagnostics, but neither turns a successful report into end-to-end acceptance. Microsoft scopes Solution Checker to static analysis of supported components. Target configuration and import acceptance remain separate gates.

Repair through the controlled source, then remove the override

Recreate the accepted business intent in the development source. If the production hotfix is valid, implement it deliberately in the unmanaged development solution, review it, run static checks and regression tests, then build a new versioned managed artifact. If it is invalid, document the decision and still prove that the intended managed component produces the required behavior before touching the override.

Deploy the exact tested artifact through the approved route. Supply environment values and connection bindings as controlled target settings, using purpose-built identities and least required authority where supported. Migrate required reference/configuration data through a separately controlled and reconciled process. Resolve missing dependencies and solution overlap. Additional arbitrary solution membership makes ownership and effective layering harder to explain.

Remove an active unmanaged customisation only after identifying the component, preserving its definition, proving the replacement and approving rollback. Work one component or tightly coupled set at a time. Prefer Microsoft-supported layer inspection and removal. Third-party tools such as XrmToolBox expose common operational patterns and may assist an authorised specialist. Popularity supplies discovery candidates only; inspect layer origin and intended definition before any removal.

Prove both the intended journey and the forbidden alternatives

Repeat the failed Perth inspection journey with synthetic records: pass, fail with explanation, draft, reopen, integration success, integration rejection and deliberate timeout. Verify the expected form and rule, the resolved target endpoint, the executing connection identity, one authoritative downstream effect and an explainable failure state. Test as field worker, supervisor, support operator and a user who must not see or change the record.

Then inspect the component layer again. The intended managed definition should be effective and any remaining active layer must have a named reason. Compare solution/version, environment values, connection references, connection health, flow state, security and controlled configuration data against the acceptance record. Open a fresh browser session and test after publish/cache propagation; an administrator’s existing session is weak evidence.

Run negative tests for the former endpoint, former connection owner, omitted role, duplicate delivery and an unrelated organisation’s record. Exercise rollback in test or a bounded recovery plan. Acceptance requires the intended personas to complete the production-shaped journey, prohibited outcomes to fail, and another operator to repeat the capture.

Prevent drift without pretending it can be designed away

Use unmanaged solutions as development source and managed artifacts for test, UAT and production. Give components a clear publisher and ownership boundary, keep overlapping solution membership deliberate, version every release and retain the exact artifact plus test evidence. Put environment variables and connection references in the same release reasoning as the components that consume them.

After legacy drift has been inventoried and reconciled, consider blocking unmanaged customisations in protected targets. That control prevents new direct edits. Existing layers and the emergency-change path still require remediation. Define a hotfix path that captures evidence, reproduces the accepted change in source, releases it normally and closes the temporary exception.

Scan high-risk component layers and target bindings before and after every release, and review the wider environment quarterly. Compare source, test and production with stable component, environment, connection and solution identifiers. Keep a configuration-data manifest, connection-owner lifecycle check and deployment-host inventory. Every intentional environment difference needs an owner, reason, capture and expiry.