Modern enterprises operate in data environments characterized by exponential growth in volume, velocity, and variety of data assets. Traditional data engineering approaches—largely based on manually coded, pipeline-specific logic—struggle to scale under such conditions. These approaches lead to brittle systems, duplicated logic, delayed onboarding of data sources, and increased operational risk. As regulatory requirements and governance expectations continue to intensify, organizations require a fundamentally different paradigm for building and managing data pipelines. This paper presents a comprehensive architecture-driven framework for metadata-driven data engineering, in which metadata functions not merely as descriptive documentation but as executable intelligence that governs pipeline behavior end-to-end. By elevating metadata to a first-class control plane, data ingestion, validation, governance, and observability processes can be declaratively defined, automated, and consistently enforced across heterogeneous data ecosystems. The proposed framework decouples control logic from execution engines, enabling scalable onboarding of data sources, automated quality enforcement, policy-driven governance, and proactive operational monitoring. The paper systematically examines the role of metadata across technical, operational, and business dimensions; proposes a reference architecture for metadata-driven pipelines; and demonstrates how ingestion, validation, governance, and observability can be automated at scale. Challenges related to metadata accuracy, organizational adoption, and interoperability are critically discussed, and future directions—such as AI-assisted metadata management and self-healing pipelines—are explored. The findings suggest that metadata-driven engineering is a foundational capability for building resilient, compliant, and scalable data platforms.