SparkR reference（SparkR 参考文档）

Introduction to SparkR

Code Workbook allows users to use both Spark R and native R. Spark R provides a distributed data frame implementation that supports operations like selection, filtering, and aggregation on large datasets. While users may be more familiar with native R, it is recommended that users first use SparkR to filter large datasets before using native R.

Common SparkR operations

Read the full API documentation ↗ for SparkR to see all possible operations. Below, we outline syntax for common operations.

Filtering

Filter experessions can be a SQL-like WHERE clause passed as a string.

df_filtered <- SparkR::filter(df, "numeric_col > 10")

You can also use column expressions similar to standard R syntax.

df_filtered <- SparkR::filter(df, df$numeric_col > 10)

You can also use SparkR::where with similar syntax.

df_filtered <- SparkR::where(df, "numeric_col > 10")

Column operations

Subset columns using SparkR::select().

df_subset <- SparkR::select(df, "column1", "column2", "column3")

Rename a column with SparkR::withColumnRenamed().

df <- SparkR::withColumnRenamed(df, "old_column_name", "new_column_name")

Add new columns using SparkR::withColumn().

# Add two columns
df <- SparkR::withColumn(df, 'col1_plus_col2', df$col1 + df$col2)
# Multiply a column by a constant
df <- SparkR::withColumn(df, 'col1_times_60', df$col1 * 60)

Aggregations

Use SparkR::groupBy and SparkR::agg to compute aggregates. Calling SparkR::groupBy will create a group by object. Pass the group by object into SparkR::agg to get an aggregated dataframe.

df_grouped <- SparkR::groupBy(df, "group_col1", "group_col2")
df_agg <- SparkR::agg(df_grouped, average_col1=avg(df$col1), max_col=max(df$col1))

---

中文翻译

SparkR 参考文档

SparkR 简介

Code Workbook 支持用户同时使用 Spark R 和原生 R。Spark R 提供了一种分布式数据框(DataFrame)实现，支持对大型数据集进行选择、过滤和聚合等操作。虽然用户可能更熟悉原生 R，但建议用户先使用 SparkR 过滤大型数据集，然后再使用原生 R。

常用 SparkR 操作

请阅读完整的 SparkR API 文档 ↗ 以了解所有可能的操作。下面我们概述常用操作的语法。

过滤

过滤表达式可以是作为字符串传入的类似 SQL 的 WHERE 子句。

df_filtered <- SparkR::filter(df, "numeric_col > 10")

您也可以使用与标准 R 语法类似的列表达式。

df_filtered <- SparkR::filter(df, df$numeric_col > 10)

您还可以使用 SparkR::where，其语法类似。

df_filtered <- SparkR::where(df, "numeric_col > 10")

列操作

使用 SparkR::select() 选取列的子集。

df_subset <- SparkR::select(df, "column1", "column2", "column3")

使用 SparkR::withColumnRenamed() 重命名列。

df <- SparkR::withColumnRenamed(df, "old_column_name", "new_column_name")

使用 SparkR::withColumn() 添加新列。

# 添加两列
df <- SparkR::withColumn(df, 'col1_plus_col2', df$col1 + df$col2)
# 将列乘以常数
df <- SparkR::withColumn(df, 'col1_times_60', df$col1 * 60)

聚合操作

使用 SparkR::groupBy 和 SparkR::agg 计算聚合值。调用 SparkR::groupBy 会创建一个分组对象。将该分组对象传入 SparkR::agg 即可获得聚合后的数据框。

df_grouped <- SparkR::groupBy(df, "group_col1", "group_col2")
df_agg <- SparkR::agg(df_grouped, average_col1=avg(df$col1), max_col=max(df$col1))