Build a metadata-driven Azure Data Factory pipeline: a control table in Azure SQL, Lookup and ForEach fan-out, a parameterised Copy activity landing Parquet in ADLS Gen2, and per-entity run logging.
A design guide for Azure Data Factory pipelines that survive production: orchestrator and worker pipelines, parameters, idempotent windows, timeouts and retries, concurrency, secrets, monitoring and CI/CD.