Generic Pipeline for AIMLFW
Objective
Develop a Flexible, Modular Framework:
Create a dynamic and composable pipeline that enables seamless construction, configuration, and execution of ML workflows within AIMLFW.Accelerate Model Lifecycle Management:
Facilitate streamlined processes for training, retraining, evaluation, and deployment, allowing users to adapt workflows dynamically to changing requirements.
Scope
Core Component Development:
Design and implement essential modules, including a Pipeline Orchestrator, component Registry.Dynamic Pipeline Configuration:
Support JSON/YAML-based configuration for including or excluding component.Extensibility & Integration:
Allow easy integration of custom component, and ensure compatibility with existing AIMLFW components (e.g., Training Manager, MME, deployment modules).Testing & Documentation:
Establish comprehensive testing (unit, integration, and performance) and provide detailed documentation to support adoption and future enhancements.
Design Proposal
@startuml
actor "Component Builder" as CB
actor "Pipeline Designer" as PD
actor "Training Executor" as TE
rectangle "Dynamic Pipeline System" {
(Build Component) as BC
(Design Pipeline) as DP
(Execute Training Job) as ETJ
}
CB --> BC : Builds
PD --> DP : Design using Components
TE --> ETJ : Executes Training Job
BC --> DP : Used in
DP --> ETJ : Referenced in
' Many-to-One Relationship
note right of DP : Many components in one pipeline
note right of ETJ: Pipelines in one training job
@enduml
Comments
Are there any additional reference documents available to help us better understand the requirements?
Requirements Specification (To be confirmed)
1. Input Format Requirements
A. Parameter Specifications
Required Parameters
Data source configuration
Model output path
Model name and version
Training configuration
Evaluation metrics
Optional Parameters
Deployment strategy
Feature engineering settings
Hyperparameter tuning ranges
Resource allocation preferences
B. Input Methods
Configuration Files
YAML/JSON format for:
Data source and feature definitions
Model specifications
Training parameters
Deployment configurations
GUI-based Forms
Web interface for:
Parameter input
Configuration validation
Pipeline visualization
Training Code Integration
Python script requirements
Code template structure
Integration points with existing systems
2. Output Format Requirements
A. Primary Deliverables
Required Outputs
Kubeflow pipeline YAML
Execution logs
Performance metrics
Optional Outputs
Visualization reports
Model performance analytics
Resource utilization statistics
B. Format Options
Pipeline Definitions
Kubeflow pipeline YAML
Argo workflow definitions
Custom pipeline formats
Executable Code
Python scripts (.py)
Jupyter notebooks (.ipynb)
Shell scripts for deployment
Documentation
API documentation
Usage guidelines
Troubleshooting guides
Proposal of Composable Pipeline
Structure
The template designer uploads a pipeline template, which includes blank code blocks and configurable parameters, to the template store.
The template user uploads the code blocks and configuration parameters to fill into the template and upload to training manager (a.k.a. tm).
Key points
Composable and Modular Design
The pipeline is built around a template that separates the overall structure from user-specific code.
This modularity allows different users to plug in their own implementations into designated blank sections without rewriting the entire pipeline or have knowledge for kubeflow.
Template-Based Approach
A template designer creates a “pipeline template” that includes both blank code blocks and configurable parameters.
Users later “fill in” these blank sections with their own code blocks and parameters when uploading to the training manager (a.k.a. tm).
Parameterization and Flexibility
The pipeline design includes configurable parameters (like epochs, model name/version, feature selection, and a flexible dictionary "config") which can be adjusted without altering the core pipeline code.
Reusability and Scalability
By abstracting common pipeline tasks into a template, the proposal promotes reuse across different projects or teams.
Users can modify only the parts that differ from one project to another, which accelerates development and ensures consistency across deployments.
[Appendix] Template example
Following is an example template based on qoe use case. We split the code by its objectives.
Preprocessing part might be blank for user to fill in.
Other part that needs user to fill will be marked by
=== User-filled part start ===and=== User-filled part end ===
# Single-Component version import kfp import kfp.dsl as dsl from kfp.dsl import InputPath, OutputPath from kfp.dsl import component as component from kfp import kubernetes BASE_IMAGE = "traininghost/pipelineimage:latest" # as new artifact after training will always be 1.0.0 artifactversion="1.0.0" # Adding config dictionary to the argument list to quickly change variable without re-transfer the pipeline. # By here I mean argument config is not the same as template config. # Template config is being processed when user transfer the python file into pipeline. @component(base_image=BASE_IMAGE) def train_export_model(featurepath: str, epochs: str, modelname: str, modelversion:str, config: dict): import tensorflow as tf import os from modelmetricsdk.model_metrics_sdk import ModelMetricsSdk # User doesn't need to know this variable. mm_sdk = ModelMetricsSdk() #################################### Preprocessing #################################### # Require modules: pandas, featurestoresdk, modelmetricsdk # Possible variables to turn into template config variable: ['pdcpBytesDl','pdcpBytesUl'] # === User-filled part start === from numpy import array import numpy as np import requests print("numpy version") print(np.__version__) import pandas as pd from featurestoresdk.feature_store_sdk import FeatureStoreSdk fs_sdk = FeatureStoreSdk() print("featurepath is: ", featurepath) features = fs_sdk.get_features(featurepath, ['pdcpBytesDl','pdcpBytesUl']) print("Dataframe:") print(features) features_cellc2b2 = features print(features_cellc2b2) print('Previous Data Types are --> ', features_cellc2b2.dtypes) features_cellc2b2["pdcpBytesDl"] = pd.to_numeric(features_cellc2b2["pdcpBytesDl"], downcast="float") features_cellc2b2["pdcpBytesUl"] = pd.to_numeric(features_cellc2b2["pdcpBytesUl"], downcast="float") print('New Data Types are --> ', features_cellc2b2.dtypes) features_cellc2b2 = features_cellc2b2[['pdcpBytesDl', 'pdcpBytesUl']] def split_series(series, n_past, n_future): X, y = list(), list() for window_start in range(len(series)): past_end = window_start + n_past future_end = past_end + n_future if future_end > len(series): break # slicing the past and future parts of the window past, future = series[window_start:past_end, :], series[past_end:future_end, :] X.append(past) y.append(future) return np.array(X), np.array(y) X, y = split_series(features_cellc2b2.values,10, 1) X = X.reshape((X.shape[0], X.shape[1],X.shape[2])) y = y.reshape((y.shape[0], y.shape[2])) print(X.shape) print(y.shape) # === User-filled part end === #################################### Training #################################### # This variable is fixed. User have to know to save model to the path of this variable. model_location = "./" # === User-filled part start === from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Dense from tensorflow.keras.layers import Flatten, Dropout, Activation from tensorflow.keras.layers import LSTM model = Sequential() model.add(LSTM(units = 150, activation="tanh" ,return_sequences = True, input_shape = (X.shape[1], X.shape[2]))) model.add(LSTM(units = 150, return_sequences = True,activation="tanh")) model.add(LSTM(units = 150,return_sequences = False,activation="tanh" )) model.add((Dense(units = X.shape[2]))) model.compile(loss='mse', optimizer='adam',metrics=['mse']) model.summary() model.fit(X, y, batch_size=10,epochs=int(epochs), validation_split=0.2) model.save(model_location) # === User-filled part end === mm_sdk.upload_model(model_location, modelname, modelversion, artifactversion) #################################### Register Artifact #################################### metric = {} # === User-filled part start === # Tell user that metric needs to be saved in this form. Variable name should also be "metric" # X = input data # y = target data yhat = model.predict(X, verbose = 0) xx = y yy = yhat metric = {'Accuracy': str(np.mean(np.absolute(np.asarray(xx)-np.asarray(yy))<5))} print(metric) # === User-filled part end === data = {} data['metrics'] = [] data['metrics'].append(metric) url = f"http://modelmgmtservice.traininghost:8082/ai-ml-model-registration/v1/model-registrations/updateArtifact/{modelname}/{modelversion}/{artifactversion}" updated_model_info= requests.post(url).json() print(updated_model_info) mm_sdk.upload_metrics(data, modelname, modelversion, artifactversion) @dsl.pipeline( name="qoe Pipeline", description="qoe", ) def super_model_pipeline( # Adding config argument dictionary here as well featurepath: str, epochs: str, modelname: str, modelversion:str, config: dict): trainop=train_export_model(featurepath=featurepath, epochs=epochs, modelname=modelname, modelversion=modelversion, config=config) trainop.set_caching_options(False) kubernetes.set_image_pull_policy(trainop, "IfNotPresent") pipeline_func = super_model_pipeline file_name = "qoe_model_pipeline" kfp.compiler.Compiler().compile(pipeline_func, '{}.yaml'.format(file_name))Blank template by template designer
This template is being designed by template designer, and being saved to template store.
import kfp import kfp.dsl as dsl from kfp.dsl import InputPath, OutputPath from kfp.dsl import component as component from kfp import kubernetes BASE_IMAGE = "traininghost/pipelineimage:latest" # as new artifact after training will always be 1.0.0 artifactversion="1.0.0" # Adding config dictionary to the argument list to quickly change variable without re-transfer the pipeline. # By here I mean argument config is not the same as template config. # Template config is being processed when user transfer the python file into pipeline. @component(base_image=BASE_IMAGE) def train_export_model(featurepath: str, epochs: str, modelname: str, modelversion:str, config: dict): import tensorflow as tf import requests import os from modelmetricsdk.model_metrics_sdk import ModelMetricsSdk # User doesn't need to know this variable. mm_sdk = ModelMetricsSdk() #################################### Preprocessing #################################### # Require modules: pandas, featurestoresdk, modelmetricsdk # === User-filled part start === {{ PreprocessingCode }} # === User-filled part end === #################################### Training #################################### # This variable is fixed. User have to know to save model to the path of this variable. model_location = "./" # === User-filled part start === {{ TrainingCode }} # === User-filled part end === mm_sdk.upload_model(model_location, modelname, modelversion, artifactversion) #################################### Register Artifact #################################### metric = {} # === User-filled part start === {{ EvaluationCode }} # === User-filled part end === data = {} data['metrics'] = [] data['metrics'].append(metric) url = f"http://modelmgmtservice.traininghost:8082/ai-ml-model-registration/v1/model-registrations/updateArtifact/{modelname}/{modelversion}/{artifactversion}" updated_model_info= requests.post(url).json() print(updated_model_info) mm_sdk.upload_metrics(data, modelname, modelversion, artifactversion) @dsl.pipeline( name="qoe Pipeline", description="qoe", ) def super_model_pipeline( # Adding config argument dictionary here as well featurepath: str, epochs: str, modelname: str, modelversion:str, config: dict): trainop=train_export_model(featurepath=featurepath, epochs=epochs, modelname=modelname, modelversion=modelversion, config=config) trainop.set_caching_options(False) # Example config trainop.set_cpu_limit({{ config.cpu }}) kubernetes.set_image_pull_policy(trainop, "IfNotPresent") pipeline_func = super_model_pipeline file_name = "qoe_model_pipeline" kfp.compiler.Compiler().compile(pipeline_func, '{}.yaml'.format(file_name))Code blocks that user provide
This part shows the code block that user uploads.
Preprocessing code section
from numpy import array import numpy as np import pandas as pd from featurestoresdk.feature_store_sdk import FeatureStoreSdk fs_sdk = FeatureStoreSdk() print("featurepath is: ", featurepath) features = fs_sdk.get_features(featurepath, ['pdcpBytesDl','pdcpBytesUl']) print("Dataframe:") print(features) features_cellc2b2 = features print(features_cellc2b2) print('Previous Data Types are --> ', features_cellc2b2.dtypes) features_cellc2b2["pdcpBytesDl"] = pd.to_numeric(features_cellc2b2["pdcpBytesDl"], downcast="float") features_cellc2b2["pdcpBytesUl"] = pd.to_numeric(features_cellc2b2["pdcpBytesUl"], downcast="float") print('New Data Types are --> ', features_cellc2b2.dtypes) features_cellc2b2 = features_cellc2b2[['pdcpBytesDl', 'pdcpBytesUl']] def split_series(series, n_past, n_future): X, y = list(), list() for window_start in range(len(series)): past_end = window_start + n_past future_end = past_end + n_future if future_end > len(series): break # slicing the past and future parts of the window past, future = series[window_start:past_end, :], series[past_end:future_end, :] X.append(past) y.append(future) return np.array(X), np.array(y) X, y = split_series(features_cellc2b2.values,10, 1) X = X.reshape((X.shape[0], X.shape[1],X.shape[2])) y = y.reshape((y.shape[0], y.shape[2])) print(X.shape) print(y.shape)Training code section
# User needs to know to save model to "model_location" location. # User needs to know input variable name and shape. # X is input data # y is target data from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Dense from tensorflow.keras.layers import Flatten, Dropout, Activation from tensorflow.keras.layers import LSTM model = Sequential() model.add(LSTM(units = 150, activation="tanh" ,return_sequences = True, input_shape = (X.shape[1], X.shape[2]))) model.add(LSTM(units = 150, return_sequences = True,activation="tanh")) model.add(LSTM(units = 150,return_sequences = False,activation="tanh" )) model.add((Dense(units = X.shape[2]))) model.compile(loss='mse', optimizer='adam',metrics=['mse']) model.summary() model.fit(X, y, batch_size=10,epochs=int(epochs), validation_split=0.2) model.save(model_location)Evaluation code section
# Tell user that metric needs to be saved in this form. Variable name should also be "metric" # User don't need to re-import module if they have imported above. # X = input data # y = target data yhat = model.predict(X, verbose = 0) xx = y yy = yhat metric = {'Accuracy': str(np.mean(np.absolute(np.asarray(xx)-np.asarray(yy))<5))} print(metric)
Request payload
This is an example for request.
header={ multipart/form-data } file={ filled_template.py } # See example config json={ config: { cpu: 1 } }
I agree it as first step let’s break the single component to multiple components.
I have improvised the model storage SDK as well : https://lf-o-ran-sc.atlassian.net/jira/software/c/projects/AIMLFW/issues/AIMLFW-185?jql=project%20%3D%20%22AIMLFW%22%20ORDER%20BY%20created%20DESC
It should be generic to abstract the underlying technology for pipeline technology.