fs.mv() fails with AuthorizationFailure via service principal auth when storage account has network restrictions

[joaofonsecapromptly]

Summary

AzureBlobFileSystem.mv() fails with AuthorizationFailure when using service principal (OAuth) authentication against storage accounts that have network firewall rules or ABAC conditions in place. Individual blob operations (read, write, delete, list) all work correctly — only mv() fails.

Root Cause

adlfs.mv() is implemented as start_copy_from_url() + delete. The Azure Copy service performs a server-side copy, meaning Azure's infrastructure fetches the source blob independently of the client. In environments where the storage account is protected by network firewall rules (e.g. private endpoint-only or IP allowlist), the Azure Copy service path is treated as a separate network origin and gets blocked — even when the calling identity has Storage Blob Data Contributor or equivalent RBAC.

The failure manifests as an AuthorizationFailure response from the Copy API, which is misleading because the SP has full data-plane permissions.

Investigation

We ruled out the following hypotheses:

Hypothesis	Test	Result
Missing auth token on source URL	Added source_authorization: Bearer <token> + requires_sync=True to the copy call	Still AuthorizationFailure
Private endpoint DNS resolution	dig <account>.blob.core.windows.net → resolved to public IPs	❌ ruled out
IP firewall blocking client directly	curl → 409 Conflict (Azure responded, not blocked at network level)	❌ ruled out
Missing RBAC role	SP confirmed to have Storage Blob Data Contributor	❌ ruled out

The exact Azure-side restriction (ABAC attribute condition, conditional access policy, or copy service network rule) could not be determined without portal access. What is confirmed: all individual blob operations succeed; only the server-side copy API path fails.

Workaround

We replaced fs.mv() with a client-side read → write → delete sequence, which works reliably:

data = fs.read_bytes(src)
fs.write_bytes(dst, data)
fs.rm(src)

Questions / Feature Request

1. Is this a known limitation of start_copy_from_url() in restricted Azure environments?
2. Would it be feasible to add a fallback to client-side copy in AzureBlobFileSystem.mv() when server-side copy fails with AuthorizationFailure? Or expose a flag (e.g. server_side_copy=False) to opt out of the copy API entirely?

This behaviour is consistent across different storage accounts using SP auth with network-restricted configurations, so it may affect other users in enterprise/secure environments.

Environment

• adlfs version: latest (pinned via fsspec)
• Azure authentication: Service Principal (client credentials, OAuth token)
• Storage account configuration: network firewall rules / restricted copy service path
• All direct blob operations functional; only start_copy_from_url fails

[anjaliratnam-msft]

Hi! Thanks for opening up this issue! We're looking into it to try to get to the bottom of what would be causing the AuthorizationFailure, but if it does end up being a known limitation we're open to adding a flag and adding the client-side copy fallback.