跳转至

Compute usage with Contour（使用 Contour 的计算用量）

Running Contour workflows on top of data in Foundry requires the use of Foundry compute, a resource measured in compute-seconds. This documentation details how Contour uses compute and provides information about investigating and managing compute usage in the product.

Each Contour board displaying data is backed by a query to the Contour backend. Foundry serves Contour queries by executing them on an auto-scaling Spark cluster. Contour query compute-seconds are exclusively measured based on the compute-seconds a query actually uses while it is executing. However, in interactive mode, compute resources are “always on” for all Contour analyses to provide faster responses; this means that each individual interactive query uses compute-seconds at a faster rate than batch compute.

The amount of resources that a query uses depends on many factors, such as the size and layout of the input datasets and the complexity of the query (such as the Contour boards used).

How Compute is measured

Contour uses Foundry’s parallelized compute backend to execute queries on Foundry data. Compute in Contour is therefore measured with the following formula.

:::callout{theme="neutral"} Note that the default interactive_contour_usage_rate is 15. This is the rate at which Contour uses compute based on the always-on parallelized compute framework. If you have an enterprise contract with Palantir, contact your Palantir representative before proceeding with compute usage calculations. :::

total_compute_seconds = contour_usage_rate * (driver_compute_seconds + executor_compute_seconds)

Interactive Contour usage

Contour’s “always ready” execution system is designed to be reactive to ad-hoc queries made by end users. The cost of a Contour query in Foundry compute-seconds is based on the time required for executing the query on the Spark cluster and the number of cores used while the query is executed. The number of cores is derived by looking at the queries’ share of the Spark application where the query was scheduled.

In interactive mode, compute is used each time a Contour board is created or updated. This includes “cascading” compute that affects boards downstream from a given board in a Contour path. For example, the following actions will use compute in a Contour analysis:

  • Creating a new board (may affect all downstream boards)
  • Updating a board (may affect all downstream boards)
  • Clicking Update data at the top of a path

Batch Contour usage

Contour paths can be “saved as” datasets. This saves the underlying code generated by the Contour backend into a dataset job specification. This dataset build can then be run on a schedule or ad hoc. These dataset jobs are measured as batch jobs and do not incur the “always on” overhead that interactive Contour usage does.

Contour batch datasets are measured as generic parallelized batch computation, similar to dataset created and updated via Code Repositories and Code Workbook. These batch computations use compute at the same rate as other batch computation.

Investigating Contour usage

Interactive usage from Contour is always attached to the analysis that it is being triggered from. For a given Contour analysis, all interactive compute is attached to the analysis resource itself.

Individual boards that run compute produce Spark metrics that can be viewed from the Builds application. This can be accessed from the View jobs section on the context menu of each board. Batch compute usage from Contour is attached to the dataset that the compute produces, similar to Code Repositories and Code Workbook. For both interactive and Batch compute, you can view your overall usage in the Resource Management application.

Understanding drivers of usage

Contour runs on the Spark computation framework, and is therefore affected by two main factors: data scale and query complexity (the number and type of operations run on the data). The more operations that must be performed on the data, the more compute the backend must use to calculate the result. However, since some operations are much more expensive than others, it is possible to add operations and reduce overall usage (e.g. by adding a filter before an expensive operation).

The number of operations on data is affected by the number of boards in a Contour path. Boards farther down in a path rely on the output from boards earlier in the path. Adding new boards into a path can change the data being sent through the path, requiring re-computation of boards and more compute.

Having more Contour users may incur more compute due to an increase in queries. However, Contour has two features to reduce compute usage by many users: checkpoints and caching.

  • Contour checkpoints are managed automatically by the Contour backend. Checkpoints allow Contour to quickly serve query results that are farther down in a path. This can reduce compute from larger Contour paths over time, but will not eliminate compute completely.
  • Contour also caches the results of identical queries on identical data. That means that if two users open the same Contour analysis in succession, the computation will only be run once. This allows dashboards to be served much more efficiently and reduces the effect of the number of users on compute usage.
  • Note that changing input data or changing a parameter will not allow the cache to be used, but future computations on that same input data and parameters will use the cache again.

Managing usage with contour

Optimizing the analysis and input datasets can reduce cost and analysis time. Learn more about Contour optimization.

Calculating usage

Contour operates under the same Spark measurement principles as other Spark-based products in the platform, such as Code Repositories. With cost-based routing, Contour will automatically optimize the size of the backend drivers and executors in order to efficiently meet the demands of the data size and query type.

 Example Contour Driver Size:
    num_vcpu: 2
    gib_ram: 11
 Example Contour Executor Size:
    num_vcpu: 2
    gib_ram: 12
    num_executors: 3
 Query Duration: 3 seconds
 Interactive Contour Usage Rate: 15

driver_compute_seconds = interactive_usage_rate * driver_compute_seconds
                       = 15 * max(2vcpu, 11gib / 7.5gib) * 3sec
                       = 15 * 2 * 3 = 90 compute-seconds

executor_compute_seconds = interactive_usage_rate * executor_compute_seconds
                         = 15 * max(2vcpu, 12gib / 7.5gib) * 3sec * 3executors
                         = 15 * 2 * 3 * 3 = 270 compute-seconds


total_compute_seconds = driver_compute_seconds + executor_compute_seconds
                      = 90 compute-seconds + 270 compute-seconds = 360 compute-seconds

中文翻译

使用 Contour 的计算用量

在 Foundry 数据上运行 Contour 工作流需要使用 Foundry 计算资源，该资源以计算秒（compute-seconds）为单位计量。本文档详细说明了 Contour 如何使用计算资源，并提供了在产品中调查和管理计算用量的相关信息。

每个显示数据的 Contour 看板（board）都由对 Contour 后端的查询支持。Foundry 通过在自动扩展的 Spark 集群上执行查询来提供 Contour 查询服务。Contour 查询的计算秒数完全基于查询在执行过程中实际使用的计算秒数进行计量。然而，在交互模式（interactive mode）下，计算资源对所有 Contour 分析保持"始终在线"状态，以提供更快的响应；这意味着每个单独的交互式查询消耗计算秒数的速率高于批处理计算（batch compute）。

查询所使用的资源量取决于多种因素，例如输入数据集的大小和布局，以及查询的复杂度（如所使用的 Contour 看板）。

计算用量如何计量

Contour 使用 Foundry 的并行化计算后端在 Foundry 数据上执行查询。因此，Contour 中的计算用量通过以下公式进行计量。

:::callout{theme="neutral"} 请注意，默认的 interactive_contour_usage_rate 为 15。这是 Contour 基于始终在线的并行化计算框架使用计算资源的速率。如果您与 Palantir 签订了企业合同，请在进行计算用量计算之前联系您的 Palantir 代表。 :::

total_compute_seconds = contour_usage_rate * (driver_compute_seconds + executor_compute_seconds)

交互式 Contour 用量

Contour 的"始终就绪"执行系统旨在对终端用户的临时查询做出快速响应。Contour 查询在 Foundry 计算秒数中的成本取决于在 Spark 集群上执行查询所需的时间以及查询执行期间使用的核心数。核心数是通过查看查询在其被调度的 Spark 应用程序中所占的份额得出的。

在交互模式下，每次创建或更新 Contour 看板时都会消耗计算资源。这包括影响 Contour 路径中给定看板下游看板的"级联"计算。例如，以下操作将在 Contour 分析中消耗计算资源：

  • 创建新看板（可能影响所有下游看板）
  • 更新看板（可能影响所有下游看板）
  • 在路径顶部点击更新数据

批处理 Contour 用量

Contour 路径可以"另存为"数据集。这会将 Contour 后端生成的底层代码保存到数据集作业规范中。然后，该数据集构建可以按计划或临时运行。这些数据集作业按批处理作业计量，不会产生交互式 Contour 用量所产生的"始终在线"开销。

Contour 批处理数据集的计量方式与通用的并行化批处理计算相同，类似于通过代码仓库（Code Repositories）和代码工作台（Code Workbook）创建和更新的数据集。这些批处理计算以与其他批处理计算相同的速率消耗计算资源。

调查 Contour 用量

Contour 的交互式用量始终附加到触发该用量的分析上。对于给定的 Contour 分析，所有交互式计算都附加到分析资源本身。

运行计算的各个看板会产生 Spark 指标，这些指标可以从构建（Builds）应用程序中查看。可以通过每个看板上下文菜单中的查看作业部分访问该应用程序。Contour 的批处理计算用量附加到计算所生成的数据集上，类似于代码仓库和代码工作台。对于交互式和批处理计算，您都可以在资源管理（Resource Management）应用程序中查看总体用量。

理解用量驱动因素

Contour 在 Spark 计算框架上运行，因此受两个主要因素影响：数据规模和查询复杂度（对数据执行的操作数量和类型）。需要对数据执行的操作越多，后端用于计算结果的计算资源就越多。然而，由于某些操作比其他操作昂贵得多，因此有可能通过添加操作来减少总体用量（例如，在昂贵操作之前添加过滤器）。

对数据的操作数量受 Contour 路径中看板数量的影响。路径中较靠后的看板依赖于路径中较早看板的输出。在路径中添加新看板可能会改变通过路径发送的数据，从而需要重新计算看板并消耗更多计算资源。

拥有更多 Contour 用户可能会因查询增加而导致更多计算消耗。但是，Contour 有两个功能可以减少多用户带来的计算用量：检查点（checkpoints） 和 缓存（caching）。

  • Contour 检查点由 Contour 后端自动管理。检查点允许 Contour 快速提供路径中较下游的查询结果。随着时间的推移，这可以减少较大 Contour 路径的计算消耗，但不会完全消除计算消耗。
  • Contour 还会缓存相同数据上相同查询的结果。这意味着如果两个用户连续打开同一个 Contour 分析，计算将只运行一次。这使得仪表板能够更高效地提供服务，并减少了用户数量对计算用量的影响。
  • 请注意，更改输入数据或更改参数将无法使用缓存，但未来对相同输入数据和参数的计算将再次使用缓存。

使用 Contour 管理用量

优化分析和输入数据集可以降低成本和分析时间。了解更多关于 Contour 优化的信息。

计算用量

Contour 遵循与平台中其他基于 Spark 的产品（如代码仓库）相同的 Spark 计量原则。通过基于成本的路由，Contour 将自动优化后端驱动（drivers）和执行器（executors）的大小，以高效满足数据规模和查询类型的需求。

 示例 Contour 驱动大小:
    num_vcpu: 2
    gib_ram: 11
 示例 Contour 执行器大小:
    num_vcpu: 2
    gib_ram: 12
    num_executors: 3
 查询持续时间: 3 秒
 交互式 Contour 使用率: 15

driver_compute_seconds = interactive_usage_rate * driver_compute_seconds
                       = 15 * max(2vcpu, 11gib / 7.5gib) * 3sec
                       = 15 * 2 * 3 = 90 compute-seconds

executor_compute_seconds = interactive_usage_rate * executor_compute_seconds
                         = 15 * max(2vcpu, 12gib / 7.5gib) * 3sec * 3executors
                         = 15 * 2 * 3 * 3 = 270 compute-seconds


total_compute_seconds = driver_compute_seconds + executor_compute_seconds
                      = 90 compute-seconds + 270 compute-seconds = 360 compute-seconds