Project Title

dagster — Orchestration platform for data assets development, production, and observation

Overview

Dagster is a cloud-native data pipeline orchestrator designed for the entire development lifecycle, offering integrated lineage and observability, a declarative programming model, and best-in-class testability. It allows developers to declare data assets as Python functions and assists in running these functions at the right time, ensuring assets stay up-to-date.

Key Features

Integrated lineage and observability for data assets
Declarative programming model for data pipeline orchestration
Best-in-class testability for data pipelines
Cloud-native and supports the full data development lifecycle

Use Cases

Data engineers using Dagster to maintain and update data tables, datasets, machine learning models, and reports
Companies leveraging Dagster for local development, unit tests, integration tests, staging environments, and production
Data scientists employing Dagster to manage and track the lineage of machine learning models

Advantages

Supports Python 3.9 through Py, making it accessible to a wide range of developers
Provides a web UI for visualizing and managing data asset graphs
Offers a robust framework for testing and maintaining data pipelines

Limitations / Considerations

As a Python-based tool, it may not be suitable for projects requiring other programming languages
The learning curve might be steep for developers not familiar with Python or data orchestration concepts

Apache Airflow: A platform to programmatically author, schedule, and monitor workflows, differing in that it is not cloud-native and has a different approach to orchestration.
Prefect: A cloud-native workflow management system that shares similarities with Dagster in terms of orchestration but has a distinct architecture and feature set.

Basic Information

GitHub: https://github.com/dagster-io/dagster
Stars: 14,079
License: Unknown
Last Commit: 2025-09-26

📊 Project Information

Project Name: dagster
GitHub URL: https://github.com/dagster-io/dagster
Programming Language: Python
⭐ Stars: 14,079
🍴 Forks: 1,824
📅 Created: 2018-04-30
🔄 Last Updated: 2025-09-26

🏷️ Project Topics

Topics: [, ", a, n, a, l, y, t, i, c, s, ", ,, , ", d, a, g, s, t, e, r, ", ,, , ", d, a, t, a, -, e, n, g, i, n, e, e, r, i, n, g, ", ,, , ", d, a, t, a, -, i, n, t, e, g, r, a, t, i, o, n, ", ,, , ", d, a, t, a, -, o, r, c, h, e, s, t, r, a, t, o, r, ", ,, , ", d, a, t, a, -, p, i, p, e, l, i, n, e, s, ", ,, , ", d, a, t, a, -, s, c, i, e, n, c, e, ", ,, , ", e, t, l, ", ,, , ", m, e, t, a, d, a, t, a, ", ,, , ", m, l, o, p, s, ", ,, , ", o, r, c, h, e, s, t, r, a, t, i, o, n, ", ,, , ", p, y, t, h, o, n, ", ,, , ", s, c, h, e, d, u, l, e, r, ", ,, , ", w, o, r, k, f, l, o, w, ", ,, , ", w, o, r, k, f, l, o, w, -, a, u, t, o, m, a, t, i, o, n, ", ]

📚 Documentation

This article is automatically generated by AI based on GitHub project information and README content analysis

dagster

Project Description