We wrote a technical deep dive here:

https://www.zenml.io/blog/scaling-zenml-200x-performance-fastapi-database-v0830

Note: Not all orchestrators supported

Note, this only works when running pipelines on remote artifact storage and a service connector.

from pathlib import Path
from zenml import step, pipeline

@step
def create_directory() -> Path:
    # Create a directory with files
    temp_dir = Path("./my_data_directory")
    temp_dir.mkdir(exist_ok=True)
    (temp_dir / "data.csv").write_text("id,value\n1,100\n2,200")
    return temp_dir

@step
def use_directory(dir_path: Path):
    # Use the directory in another step
    print(f"Contents: {list(dir_path.glob('**/*'))}")
    
@pipeline
def directory_pipeline():
    dir_path = create_directory()
    use_directory(dir_path) 

You can then download the files later from the dashboard!

🌟 Pro Features: Workspace Hierarchy & Projects

- Complete refactoring of tenants into workspaces with project subdivision

- Enhanced RBAC with new default roles for improved permission management

- Supercharged tagging system with multi-tag filtering and exclusive tags

- Completely redesigned dashboard with tab-based navigation and faster load times

🏃 Performance Improvements for All Users

- Significantly faster CLI performance

- Better GitLab repository support with improved URL parsing

- Support for environment variables in KubernetesPodSettings

- Fixed SkyPilot Orchestrator with upgraded dependencies

- Improved Kubernetes orchestrator security with API tokens as secrets

- Enhanced Weights & Biases integration with Weave

- GCP service connector project ID override capability

- Fixed ACR support in Azure service connector

- Persistent resource support for Vertex orchestrator

📚 Brand New Documentation

- Improved navigation with clear site sections

- Dedicated ZenML Pro documentation area

- Enhanced search and AI assistance features

📋 Upgrade Notes

Pro users should explore the new workspace hierarchy and projects feature. Resource organization is now more intuitive with workspace-scoped resources (Stacks, Secrets) and project-scoped resources (pipelines, artifacts, models).

Upgrade with: pip install -U zenml

Full details: https://github.com/zenml-io/zenml/releases

Happy modeling! 🎉

🌟 ZenML 0.75.0 Highlights

• Create and update stack components directly from the dashboard

• Custom authentication for service connectors

• Enhanced model artifact retrieval by creation date

• Additional SageMaker environment settings

• Expanded documentation for LLM messaging and model deployers

🔍 ZenML 0.74.0 Key Features

• API Token support in dashboard

• SageMaker pipeline scheduling capabilities

• Azure Container Registry implicit authentication

• Vertex AI persistent resource support

• Performance improvements with run metadata indices

• Timezone consistency across the platform

• Support for custom log formats

📋 Upgrade Notes

Review documentation for service connectors and cloud integrations before upgrading. Pro users should explore the new dashboard capabilities for stack management.

Full changelog: https://github.com/zenml-io/zenml/releases

Happy modeling! 🎉

🏢 Enterprise-Grade Enhancements

- Secure tenant enrollment for self-hosted ZenML Pro deployments

- Advanced CSRF token protection for enhanced security

- Cross-domain authorization flow for multi-domain installations

- Flexible memory configuration for migration pods in Helm deployments

📊 New Experiment Tracking Features

- Vertex AI experiment tracker integration for seamless cloud operations

- Enhanced experiment comparison tooling for deeper insights

- Interactive visualization capabilities for better analysis

- Check out our quick demo to see it in action!

☁️ Cloud & Infrastructure Updates

- Support for latest Airflow KubernetesPodOperator paths

- Revamped Slack alerter implementation for better notifications

- Improved resource reporting with automatic unit conversion

- Enhanced documentation for Kubeflow Pipelines and LLM deployments

🛠️ Quality-of-Life Improvements

- Non-ASCII character support in JSON dumps via environment variables

- Removed gluon from MLflow log suppression list

- Various dashboard fixes for smoother user experience

📋 Upgrade Notes

Before upgrading to 0.73.0:

- Review the on-premises Pro deployment documentation

- Backup your existing configurations

- Test the new features in a non-production environment first

As always, we recommend checking our full changelog for detailed information about all changes and improvements. Join our community channels to share your feedback and experiences with these new features!

Happy modeling! 🎉

Full release notes: https://github.com/zenml-io/zenml/releases/tag/0.73.0

• 📊 Interactive Table View: Compare metadata and metrics across pipeline runs with automatic change tracking and sorting capabilities

• 📈 Parallel Coordinates Plot: Visualize relationships between different metadata parameters across multiple runs

• 🔄 Compare up to 20 pipeline runs simultaneously and analyze any numerical metadata your pipelines generate

• 🔗 Shareable Views: Configurations are preserved in URLs for easy sharing with team members

This feature is currently in Alpha Preview. Join our Slack to share feedback!

ZenML now integrates with Modal, bringing lightning-fast cloud execution capabilities to your ML pipelines. This new step operator allows you to execute individual pipeline steps on Modal's specialized compute instances, offering notable speed, particularly for Docker image building and hardware provisioning. With simple configuration options, you can precisely specify hardware requirements like GPU type, CPU count, and memory for each step, making it ideal for resource-intensive ML workloads.

Many more bugfixes and improvements, so upgrade your server now!

Release notes: https://github.com/zenml-io/zenml/releases/tag/0.71.0

Key Changes

• Artifact Versioning Improvements: The handling of artifact versions has been improved, including the API improvements like the ability to batch artifact version requests to improve the execution times and more types for the step input/output artifacts, including multiple versions of the same artifact (e.g. model checkpoints), to improve the UX using ZenML UI or while working directly with the API.

• Scalability Enhancements: Various scalability improvements have been made, such as reducing unnecessary server requests and incrementing artifact versions server-side. These enhancements are expected to provide significant speed and scale improvements for ZenML users.

• Metadata management: Now, all the metadata-creating functions are gathered under one method called log_metadata. It is possible to call this method with different inputs to log run metadata for artifact versions, model versions, steps and runs.

• The oneof filtering: This allows to filter entities using a new operator called oneof. You can use this with IDs (UUID type) or tags (or other string-typed attributes) like this PipelineRunFilter(tag='oneof:["cats", "dogs"]').

• Documentation Improvements: The ZenML documentation has been restructured and expanded, including the addition of new sections on finetuning and LLM/ML engineering resources.

• Bug Fixes: This release includes several bug fixes, including issues with in-process main module source loading, and more.

Caution: Make sure to back up your data before upgrading!

While this release brings many valuable improvements, the database schema changes and migrations pose a potential risk to users. It is strongly recommended that users:

• Test the upgrade on a non-production environment: Before upgrading a production system, test the upgrade process in a non-production environment to identify and address any issues.

• Back up your data: Ensure that you have a reliable backup of your ZenML data before attempting the upgrade.

- 📊 Stack Components Dashboard: Visualize and manage your stack components directly from the dashboard interface

- 🧠 Filters: ZenML Pro users are now able to filter and sort all resources in the dashboard!

- ⚡ Client-Side Caching: Dramatically improved performance with client-side computation for cached steps, reducing remote orchestrator spin-up time and costs

- 🔄 Unified Onboarding: Combined starter and production setup into a single, intuitive flow for smoother getting started experience

- 📦 New Artifact Management: Introducing register_artifact function to directly link existing data in the artifact store - perfect for frameworks like PyTorch-Lightning

- 🚢 Updated BentoML Integration: Now supporting version 1.3.5 with improved containerization

Important Changes:

- Python 3.8 support has been discontinued

- Several legacy features including the legacy pipeline and step interface have been removed

- Various CLI commands including zenml stack up/down are now deprecated

📚 Expanded Documentation:

- New guides for Kubernetes per-pod configuration

- Best practices for common stacks

- Azure 1-click dashboard deployment

- Comprehensive ZenML Pro documentation

- Detailed guides for custom Dataset classes and Materializers

Upgrade to ZenML 0.68.x today to access all these improvements! Check out the full release notes on GitHub for more details.

Happy modeling! 🎉

- Enhanced SageMaker Orchestration: Support for TrainingJobs with warm pools, boosting performance and potentially reducing costs.

- Revamped DAG Visualizer: Preview DAGs before pipeline completion and enjoy improved visual clarity.

- Flexible Environment Configuration: Use ${ENV_VARIABLE_NAME} syntax in code and config files for adaptable deployments.

- Improved Cloud Integration: Direct pipeline and log URLs for AWS, Azure, and GCP.

- Kubernetes Support for Skypilot: Run Skypilot orchestrator on Kubernetes clusters.

- Updated Deepchecks Integration: Incorporating the latest Deepchecks features for robust data validation.

- Expanded Documentation: Covering Lightning AI, Kubeflow, Comet, Neptune, Hugging Face deployer, and more.

Upgrade to ZenML 0.67.0 today to supercharge your ML workflows! Check out the full release notes on GitHub for more details.

Happy modeling with ZenML! 🎉

Before:

@pipeline(settings={"orchestrator.sagemaker": SagemakerOrchestratorSettings(...)})

def my_pipeline():

Now:

@pipeline(settings={"orchestrator": SagemakerOrchestratorSettings(...)})

def my_pipeline():

Finally, in order to slim down the ZenML library, we removed the numpy and pandas libraries as dependencies of ZenML. If yourcode uses these libraries, you have to make sure they're installed in your local environment as well as the Docker images that get built to run your pipelines (Use DockerSettings.requirements or DockerSettings.required_integrations).

If you’re a pro user update your tenant today with a single click, and locally run pip install zenml —upgrade to immediately get access to all this and above!

Full release notes here: https://github.com/zenml-io/zenml/releases/0.66.0

We're thrilled to announce the 0.65.0 release of ZenML, packed with new features and improvements to streamline your MLOps workflows:

1. New Quickstart Experience: Seamlessly transition from local to cloud-scale ML operations with our enhanced onboarding process.

2. Run Single Step as ZenML Pipeline: Execute individual steps on your active stack, improving flexibility and debugging capabilities.

3. AzureML Integration Upgrade: Our AzureML Step Operator now supports SDKv2 and leverages Service Connectors for improved cloud integration.

4. Enhanced Logging: Benefit from improved traceability with new timestamp features in log messages.

5. Flexible Model Versioning: Utilize templated names for Model Versions, supporting dynamic {date} and {time} placeholders.

6. Accelerate Decorator: Use the run_with_accelerate step wrapper as a Python Decorator for ZenML steps, simplifying integration with accelerate library.

These updates focus on improving developer experience, expanding cloud capabilities, and enhancing flexibility in ML workflows. Update now to take advantage of these powerful new features!

For a complete list of changes and improvements, check out our full release notes on GitHub.

Happy modeling with ZenML!

1. Remote Notebook Integration: Run notebook-defined steps with remote orchestrators.

2. Optimized Docker Builds: Faster development with code uploads to artifact store.

3. AzureML Orchestrator Support: Expanded cloud platform integration.

4. Terraform Modules: Easily provision MLOps stacks across major cloud providers.

These features bridge the gap between experimentation and production, accelerate development cycles, and simplify cloud infrastructure setup.

Update now to experience streamlined MLOps workflows!

For details, visit the recent blog post and release notes.

Moreover, we implemented a new step operator that allows you to run individual steps of your pipeline in Kubernetes pods.

Lastly, we have simplified our pipeline models by removing their versions. No migration is required but if you were using the API, please take out references to pipeline versions, and reference runs directly from now on.

See full release notes here: https://github.com/zenml-io/zenml/releases/tag/0.63.0

Watch the simple tutorial here: https://youtu.be/diyt-Y7GLwY

P.S. OSS users, this will be available in version 0.63.0 onwards. Watch out for the release and upgrade!

terraform apply

And it will deploy the stack + register it to ZenML! It’s really that simple.

AWS, Azure, and GCP are live now. Try them here.

We also updated our Hugging Face integration to support the automatic display of an embedded datasets preview pane in the ZenML Dashboard whenever you return a Dataset from a step. This was recently released by the Hugging Face datasets team and it allows you to easily visualize and inspect your data from the comfort of the dashboard.

Upgrade your servers to 0.62.0 to get the latest feature!

We hope this makes it super easy for existing and new users to set up the infrastructure required to run ZenML pipelines on the cloud in one click. If you are running pipelines locally, try this!

Note: Only a simple AWS stack using Skypilot supported for now but GCP + Azure to come!

0.60.0 also brings a ton of integration updates and further improvements to the docs. Legacy docs can now be found here, solving the problem of search engines indexing older ZenML versions.

Read more about the release here.

💡 Clear orientation: Always know where you are on the dashboard.

🔙 Quick return: Return to any section with a single click.

🎢 Effortless navigation: Less time spent navigating, more time doing your tasks.

You can configure these settings either directly in your code or through a config.yaml file, offering flexibility and control over how step failures are handled. This enhancement aims to improve the robustness and reliability of your workflow executions.

Read more about this feature here → https://github.com/zenml-io/zenml/releases/tag/0.58.0

A significant change in this release is the migration of the Service (ZenML's technical term for deployment) registration and deployment from local or remote environments to the ZenML server. This change will be reflected in an upcoming tab in the dashboard which will allow users to explore and see the deployed models in the dashboard with their live status and metadata.**

Read more here: https://github.com/zenml-io/zenml/releases/tag/0.58.0

• Migration of the Service (ZenML's technical term for deployment)
  registration and deployment from local or remote environments to the ZenML server

• A new Huggingface Model Deployer has been introduced, allowing you to seamlessly
  deploy your Huggingface models using ZenML

• Faster Integration and Dependency Management ZenML now leverages the uv library,
  significantly improving the speed of integration installations and dependency management,
  resulting in a more streamlined and efficient workflow.

• Enhanced Logging and Status Tracking Logging have been improved, providing better
  visibility into the state of your ZenML services.

• Improved Artifact Store Isolation: ZenML now prevents unsafe operations that access
  data outside the scope of the artifact store, ensuring better isolation and security.

• Adding admin user notion for the user accounts and added protection to certain operations
  performed via the REST interface to ADMIN-allowed only.

• Rate limiting for login API to prevent abuse and protect the server from potential
  security threats.

• The LLM template is now supported in ZenML, allowing you to use the LLM template
  for your pipelines.

We also have a brand new video for cloud users to get started quickly. Watch the new onboarding video here.

Go ahead and starting sharing your models and pipelines with your colleageus!

We know a lot you have been struggling with the old DAG visualizer.. It's based on our old codebase, it's buggy, and glitches out with zooming in and out. Well, you don't have to wait any longer. We took all of your feedback and crafted a new experience for you that is 10x better. 

🤜 See all your pipeline runs and which MLOps stack they ran on
🔎 Gain full observability of all configuration of all configuration, parameters, and environments (docker images etc) used for the run
😎 See all your data, logs, and code in one easy to use view
🌞 Gather all your ML visualizations in one view
🕺 View all related models and their metrics that a pipeline generated

Head over to your tenant settings to create custom roles.

Data sits at the heart of every machine learning workflow. Managing and versioning this data correctly is essential for reproducibility and traceability within your ML pipelines. ZenML takes a proactive approach to data versioning, ensuring that every artifact—be it data, models, or evaluations—is automatically tracked and versioned upon pipeline execution.

Read more on the docs!

A Model is simply an entity that groups pipelines, artifacts, metadata, and other crucial business data into a unified entity. Please note that one of the most common artifacts that is associated with a Model in ZenML is the so-called technical model, which is the actual model file/files that hold the weight and parameters of a machine learning training result. However, this is not the only artifact that is relevant; artifacts such as the training data and the predictions this model produces in production are also linked inside a ZenML Model. In this sense, a ZenML Model is a concept that more broadly encapsulates your ML product's business logic.

The easiest way to use a ZenML model is to pass a Model object as part of a pipeline run.

from zenml import Model
from zenml import pipeline

model = Model(
    # The name uniquely identifies this model
    # It usually represents the business use case
    name="iris_classifier",
    # The version specifies the version
    # If None or an unseen version is specified, it will be created
    # Otherwise, a version will be fetched.
    version=None, 
    # Some other properties may be specified
    license="Apache 2.0",
    description="A classification model for the iris dataset.",
)

# This configures it for all steps within the pipeline
@pipeline(model=model)
def training_pipeline():
    ...

Read more on the docs!