Resource configuration（资源配置(Resource configuration)）

This page covers the configuration of resources for single-node transforms. Distributed Spark transforms are configured with Spark profiles so that controls can be placed on large resource consumption.

Transform resources can be configured by calling with_resources on the transform decorator.

from transforms.api import transform, Input, Output


@transform.using(
    output=Output("/path/output"),
    input=Input("/path/input"),
).with_resources(
    cpu_cores=1
)
def my_compute_function(output, input):
    ...

The following options can be configured:

• cpu_cores: The number of CPU cores to request for the transform's container. Can be a fraction, defaults to two.
• memory_gb: The amount of memory to request for the container, in GB.
• memory_mb: The amount of memory to request for the container, in MB.
• gpu_type: The type of GPU to allocate for the transform, provided as a string.

:::callout{theme="warning"} Only one of memory_gb and memory_mb can be configured. :::

By default, the maximum allowed values are 8 cores and 64 GBs of memory. To increase these limits, contact Palantir Support.

Below is an example configuration of cores, memory, and GPU type.

from transforms.api import transform, Input, Output


@transform.using(
    output=Output("/path/output"),
    input=Input("/path/input"),
).with_resources(
    cpu_cores=8,
    memory_gb=32,
    gpu_type="NVIDIA_T4"
)
def my_compute_function(output, input):
    ...

GPU provisioning

GPUs can only be used in Python transforms if they are available to the project that the transform is in.

:::callout{theme="neutral"} Not all GPU types are available on all environments. :::

The following are GPU types that can be made available for transforms:

• "NVIDIA_T4"
• "NVIDIA_V100"
• "NVIDIA_A10G"
• "NVIDIA_A100"
• "NVIDIA_A16"
• "NVIDIA_H100"
• "NVIDIA_H200"
• "NVIDIA_L4"
• "NVIDIA_L40S"

---

中文翻译

---

资源配置(Resource configuration)

本页面介绍单节点转换(single-node transforms)的资源配置方法。分布式Spark转换(distributed Spark transforms)通过Spark配置文件(Spark profiles)进行配置，便于对大规模资源消耗进行管控。

转换(Transform)的资源可通过在转换装饰器上调用with_resources方法完成配置：

from transforms.api import transform, Input, Output


@transform.using(
    output=Output("/path/output"),
    input=Input("/path/input"),
).with_resources(
    cpu_cores=1
)
def my_compute_function(output, input):
    ...

可配置的参数选项如下： * cpu_cores: 为转换容器申请的CPU核心数，支持填写小数，默认值为2。 * memory_gb: 为容器申请的内存容量，单位为GB。 * memory_mb: 为容器申请的内存容量，单位为MB。 * gpu_type: 为转换分配的GPU类型，取值为字符串格式。

:::callout{theme="warning"} memory_gb与memory_mb二者仅可配置其一，不可同时使用。 :::

默认允许配置的资源上限为8核CPU、64GB内存。若需提升该限额，请联系Palantir支持团队(Palantir Support)。

以下为同时配置CPU核心数、内存、GPU类型的示例：

from transforms.api import transform, Input, Output


@transform.using(
    output=Output("/path/output"),
    input=Input("/path/input"),
).with_resources(
    cpu_cores=8,
    memory_gb=32,
    gpu_type="NVIDIA_T4"
)
def my_compute_function(output, input):
    ...

GPU供给(GPU provisioning)

只有当转换所属的项目已开通GPU使用权限时，你才可以在Python转换中使用GPU。

:::callout{theme="neutral"} 并非所有GPU类型在全部环境中都可用。 :::

支持为转换分配的GPU类型如下： * "NVIDIA_T4" * "NVIDIA_V100" * "NVIDIA_A10G" * "NVIDIA_A100" * "NVIDIA_A16" * "NVIDIA_H100" * "NVIDIA_H200" * "NVIDIA_L4" * "NVIDIA_L40S"

---