JointFlow: Optimizing Service Deployment for Large-Scale Model Workflows in JointCloud

LLM-based workflows utilize Large Language Models (LLMs) for dynamic user requests, combining task planning and multiple machine learning (ML) models. Existing ML workflow platforms assume static structure deployment, neglecting dynamic orchestration for comprehensive workflows that fulfill users’ diverse requirements. Model selection in dynamic workflows involves various models and parallelism configurations, each with unique accuracy and efficiency trade-offs.To address these limitations, we introduce JointFlow, a solution offering LLM-based workflows as a service by dynamically constructing workflows across heterogeneous JointCloud infrastructures. It models and optimizes dynamic workflows, focusing on accuracy and efficiency trade-offs in model selection and parallelism configurations. A super-DAG represents the dynamic sub-task workflows from LLMs, profiling configurations across infrastructures. JointFlow also seeks the optimal workflow placement strategy. Experiments show it reduces serving costs while achieving throughput objectives compared to state-of-the-art methods.