Skip to content

Configuration API Reference

Configuration classes for Ray execution and data processing options.

dagster_ray.configs.Lifecycle pydantic-model

Bases: Config

Show JSON schema: 
{
  "properties": {
    "create": {
      "default": true,
      "description": "Whether to create the resource. If set to `False`, the user can manually call `.create` instead.",
      "title": "Create",
      "type": "boolean"
    },
    "wait": {
      "default": true,
      "description": "Whether to wait for the remote Ray cluster to become ready to accept connections. If set to `False`, the user can manually call `.wait` instead.",
      "title": "Wait",
      "type": "boolean"
    },
    "connect": {
      "default": true,
      "description": "Whether to run `ray.init` against the remote Ray cluster. If set to `False`, the user can manually call `.connect` instead.",
      "title": "Connect",
      "type": "boolean"
    },
    "cleanup": {
      "default": "always",
      "description": "Resource cleanup policy. Determines when the resource should be deleted after Dagster step execution or during interruption.",
      "enum": [
        "never",
        "always",
        "on_exception"
      ],
      "title": "Cleanup",
      "type": "string"
    }
  },
  "title": "Lifecycle",
  "type": "object"
}

Fields:

Attributes

create pydantic-field 
create: bool = True

Whether to create the resource. If set to False, the user can manually call .create instead.

wait pydantic-field 
wait: bool = True

Whether to wait for the remote Ray cluster to become ready to accept connections. If set to False, the user can manually call .wait instead.

connect pydantic-field 
connect: bool = True

Whether to run ray.init against the remote Ray cluster. If set to False, the user can manually call .connect instead.

cleanup pydantic-field 
cleanup: Literal['never', 'always', 'on_exception'] = 'always'

Resource cleanup policy. Determines when the resource should be deleted after Dagster step execution or during interruption.

dagster_ray.configs.RayDataExecutionOptions pydantic-model

Bases: Config

Show JSON schema: 
{
  "$defs": {
    "ExecutionOptionsConfig": {
      "properties": {
        "cpu": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Cpu"
        },
        "gpu": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Gpu"
        },
        "object_store_memory": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Object Store Memory"
        }
      },
      "title": "ExecutionOptionsConfig",
      "type": "object"
    }
  },
  "properties": {
    "execution_options": {
      "$ref": "#/$defs/ExecutionOptionsConfig"
    },
    "cpu_limit": {
      "default": 5000,
      "title": "Cpu Limit",
      "type": "integer"
    },
    "gpu_limit": {
      "default": 0,
      "title": "Gpu Limit",
      "type": "integer"
    },
    "verbose_progress": {
      "default": true,
      "title": "Verbose Progress",
      "type": "boolean"
    },
    "use_polars": {
      "default": true,
      "title": "Use Polars",
      "type": "boolean"
    }
  },
  "title": "RayDataExecutionOptions",
  "type": "object"
}

Fields:

Attributes

execution_options pydantic-field 
execution_options: ExecutionOptionsConfig
cpu_limit pydantic-field 
cpu_limit: int = 5000
gpu_limit pydantic-field 
gpu_limit: int = 0
verbose_progress pydantic-field 
verbose_progress: bool = True
use_polars pydantic-field 
use_polars: bool = True

Functions

apply 
apply()
Source code in src/dagster_ray/configs.py 
78
79
80
81
82
83
84
85
86
87
88
89
90
91
def apply(self):
    import ray
    from ray.data import ExecutionResources

    ctx = ray.data.DatasetContext.get_current()

    ctx.execution_options.resource_limits = ExecutionResources.for_limits(
        cpu=self.execution_options.cpu,
        gpu=self.execution_options.gpu,
        object_store_memory=self.execution_options.object_store_memory,
    )

    ctx.verbose_progress = self.verbose_progress
    ctx.use_polars = self.use_polars
apply_remote 
apply_remote()
Source code in src/dagster_ray/configs.py 
 93
 94
 95
 96
 97
 98
 99
100
def apply_remote(self):
    import ray

    @ray.remote
    def apply():
        self.apply()

    ray.get(apply.remote())

dagster_ray.configs.ExecutionOptionsConfig pydantic-model

Bases: Config

Show JSON schema: 
{
  "properties": {
    "cpu": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Cpu"
    },
    "gpu": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Gpu"
    },
    "object_store_memory": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Object Store Memory"
    }
  },
  "title": "ExecutionOptionsConfig",
  "type": "object"
}

Fields:

Attributes

cpu pydantic-field 
cpu: int | None = None
gpu pydantic-field 
gpu: int | None = None
object_store_memory pydantic-field 
object_store_memory: int | None = None