Announcements（公告）¶
REMINDER: You can now sign up for the Foundry Newsletter to receive a summary of new products, features, and improvements across the platform directly to your inbox. For more information on how to subscribe, see the Foundry Newsletter and Product Feedback channels announcement.
Lightweight transforms now supports external transforms, incremental transforms, and unmarking¶
Date published: 2024-03-14
We are continuing to expand the feature set for lightweight transforms and are excited to announce that three new features are now available for improving performance, reducing startup times and overhead, and restricting access to resources based on a user's eligibility.
Incremental transforms compatibility¶
Incremental transforms are now compatible with lightweight transforms. You can use this integration to combine the performance benefits of incremental transforms with the reduced startup times of lightweight transforms. To make your lightweight build incremental, you just need to add an @incremental decorator as shown below:
import pandas
@lightweight()
@incremental(require_incremental=True)
@transform(my_input=Input("my-input"), my_output=Output("my-output"))
def my_incremental_transform(my_input, my_output):
my_output.write_pandas(my_input.pandas(mode="added"))
To start creating incremental lightweight transforms, upgrade your repository to the latest version and install the latest version of foundry-transforms-lib-python. For more details, review the documentation.
Improved external transforms performance with lightweight mode¶
External transforms now support the @lightweight API, resulting in improved performance by reducing startup overhead. This enhancement is ideal for external transforms as they are typically not Spark dependent or CPU-intensive, leading to improved latency and throughput. Below is an example:
@lightweight
@use_external_systems(
egress=EgressPolicy('<policy RID>'),
creds=Credential('<credential RID>')
)
@transform(
output=Output('/path/to/output/dataset')
)
def compute(egress, creds, ...):
Support for unmarking¶
Markings and Organizations restrict access to resources based on a user's eligibility. When restricted content is removed or obfuscated while deriving a dependent resource, you may wish to remove the marking and/or Organization on that derived resource. Removing markings, also called unmarking, is now supported for lightweight transforms.
For more information on removing markings in lightweight transforms, review the documentation.
Additional highlights¶
App Building | Slate¶
Refined Tooltip Customization with HTML | Users can now employ HTML to craft styled and multi-line tooltips within charts, enhancing the ability to tailor and present data more effectively.
Enhanced Script Compliance in Slate | Slate has been enhanced to comply with strict mode ↗ standards, ensuring scripts are parsed and errors are handled with greater precision. The use of reserved keywords as variables is now prohibited, aligning with ES6 module conventions.
Enhanced Asynchronous Capabilities in Slate | Slate has been enhanced to accommodate asynchronous functions, paving the way for more intricate use cases such as timeouts between queries and the integration of libraries that utilize async functions. This advancement introduces a more streamlined, promise-based approach to handling asynchronous operations within Slate, while maintaining synchronous function execution.
App Building | Workshop¶
Introducing Workshop's Loop Layouts | Workshop's capabilities have been expanded with the introduction of Loop layouts, allowing users to iterate over sets of objects and present an embedded module for each one. This enhancement streamlines the handling of data collections and elevates the user's ability to design interactive modules. Further information is available in the loop layouts documentation.
Mobile Workshop Module Integration Enhanced | The mobile version of Workshop now boasts a new feature: the integration of the embedded module widget. This advancement streamlines the incorporation of mobile-compatible modules within other modules, fostering a more interactive and engaging user experience on mobile devices. Embedding is exclusively available for mobile modules, as detailed in the mobile mode limitation section of our documentation.
Upgraded Workshop Notepad Template Button Flexibility | Workshop has been updated to permit the conditional visibility and deactivation of the Notepad template option, based on Boolean variables, mirroring the capabilities of the button group widget. This update also introduces tooltips that utilize string variables, offering improved guidance for users.
Enhanced Audio Preview in Workshop | Workshop now showcases a waveform audio player for media set media references, offering users a more intuitive method to preview audio files.
Enhanced Tab Scrolling in Workshop | The new Enable overflow configuration option allows users to effortlessly navigate through overflowing tabs in Workshop's section headers. When activated, arrows will be displayed on both sides of the tabs, ensuring users can access all tabs even when they exceed the section header's width.
App Building | Ontology SDK¶
Developer Console appearance refresh | Developer Console has been revamped with a new sidebar, application header, and light theme, along with an enhanced home page.
Python Batch Actions: Return Edits Support | Users can now use the optional parameter batch_action_config to set return_edits to ALL and receive the edits in the response when using Python batch apply actions.
Analytics | Notepad¶
Introducing batched functions in Notepad | Functions on Objects sections in Notepad may now be configured with batched functions, which can significantly reduce the time taken to generate a document from a template.
Batched functions are functions that accept an object set as an argument and return results for each object in the object set. Users can configure batched functions for Functions on Objects sections that are nested inside generator sections.
Using batched functions can decrease template generation time as unbatched functions must execute once for each object in an object set, whereas a batched function executes only once for the entire object set.
To start using batched functions, users may insert a Functions on Objects section into a Notepad document, select the Batched option from the configuration panel and select a batched function. Learn how to write or configure batched functions.
Revamped Notepad Entry Screen | Notepad's entry screen has been upgraded with a more intuitive search function, enabling users to swiftly locate notes by title or content. This enhancement introduces a sortable resource list with quick-access categories like Recent, My Creations, and Bookmarked. Additionally, a help icon is now available for immediate reference to Notepad guidelines and AIP assistance, alongside simplified processes for generating new Notepad documents or templates.
Improved Markdown Formatting Capabilities in Notepad | Notepad has been upgraded to better interpret Function on Objects output as Markdown, enhancing the creation of dynamic and visually-appealing text formats in templates.
Analytics | Code Workspaces¶
Introducing Container Transforms in Code Workspaces | It is now possible to publish Jupyter® notebooks and R scripts as data transformation pipelines as container transforms. Container transforms are easily reproducible and can be scheduled to run automatically. This feature gives Code Workspaces first-class integration with Foundry and enables you to easily share the outputs of your data analysis workflows with other users. To learn more about using container transforms, read the public documentation for Jupyter® and RStudio®.
Analytics | Contour¶
Streamlined Analysis Tagging in Contour | Contour now offers a refined way to categorize and manage your analyses with the introduction of analysis tagging. To assign a tag, select the more actions menu, proceed to the tags submenu, and specify your chosen tag. For an organized view, analyses can be sorted by these tags within the analysis sidebar, simplifying navigation and management of intricate analysis projects.
Ontology | Ontology Management¶
Refined Ontology Manager Filtering Options | The Ontology Manager Cleanup Tool now boasts improved filtering features, enabling users to order object types by the quantity of associated flags. This enhancement facilitates a more detailed and efficient approach to organizing Ontology cleanup activities, allowing users to effectively prioritize tasks based on distinct criteria.
Enhanced API Name Creation in Ontology Manager | Ontology Manager now generates API names differently. In the past, when an object type display name included '[' and ']' characters, the auto-generated API name would commence with an underscore. With this improvement, the API name will no longer begin with an underscore, offering a more intuitive and user-friendly naming convention. This alteration only impacts the default generated name, which users are advised to customize further.
Customizable Action Type API Names in Ontology Manager | Directly from the Overview tab, Ontology Manager now supports the customization of API names for action types, offering enhanced flexibility.
Ontology | Vertex¶
Enhanced Annotation Features in Vertex | Vertex has been upgraded to include a new annotation mode, allowing users to mark up and highlight using a variety of custom shapes and colors. This update offers users a dynamic and customizable approach to annotating visual content, enhancing the interactive experience.
Security | Projects¶
Streamlined Content Management and Dataset Integration | Users now have the enhanced ability to integrate with datasets and manage content directly from URLs, streamlining the process even in the absence of home folders. This improvement ensures that operations such as importing a sheet via URL are conducted effortlessly, regardless of home folder availability.
Administration | Control Panel¶
Organization markings can be managed in Control Panel | Control Panel now allows Organization administrators to adjust Organization Marking permissions in a new tab within Organization permissions. This enhancement is a stride towards consolidating permissions handling, offering a more unified and straightforward experience. Following the successful implementation of Phase 2 of the migration, anticipated in Q1 2024 and announced in 2023, Control Panel will become the sole hub for Organization permissions configuration, rendering the previous method in Platform Settings obsolete.
Model Integration | AIP Logic¶
Refined Debugging Interface for AIP Logic | The debugging tool within AIP Logic has been upgraded with new features including toggleable block cards and a streamlined prompt visualization, alongside a dedicated function to reset tool calls. These improvements are designed to enhance the user's ability to efficiently navigate and resolve issues within their logic flows.
Introducing Object Sets Outputs in AIP Logic | The Use LLM block within AIP Logic now supports the return of an object set as an output, enhancing the ease of integration with Workshop. This improvement empowers users to run prompts that yield multiple objects, enhancing the analytical functions available in their workflows. For example, with this feature, prompts like "Are there more F1 drivers from Spain or Italy? Return me those drivers" will return more than one object as a result. Example is notional and uses open source data.
Introducing Single Completion LLM blocks in AIP Logic | Single Completion LLM blocks
You can now run a single completion from an LLM block with no special handling or additional instructions. This feature is useful for improving performance when the additional capabilities of Chain-of-thought (CoT) prompting are not needed by adding the least token overhead among all prompting strategies. Note that when using the single completion LLM block, tools and specify return types are not available.
Introducing the Transform Block in AIP Logic | The Transform block allows you to use expressions within AIP Logic. This is a great way to perform transformations on your data with AIP Logic such as convert a string to a date, parse JSON, perform mathematical expressions and more.
Array Output Capabilities Expanded in AIP Logic | The capability to generate array outputs has been introduced to AIP Logic, facilitating the delivery of multiple suggestions or values through a singular response. This advancement enhances the dynamic nature of interactions with LLM blocks, to address problems that demand a spectrum of possible solutions.
Model Integration | Modeling¶
Refined Media Set Integration for Model Deployments | Model deployment procedures have been upgraded to incorporate media sets as selectable inputs, specifically when the model's API contains media references. This enhancement introduces a media set selector that appears contextually for applicable models, simplifying the configuration of deployments that involve media data.
Data Integration | Pipeline Builder¶
Advanced Property Extraction in Pipeline Builder | Pipeline Builder now delivers an enhanced user experience for extracting properties from geoJSON data. Users have the option to extract all properties as a string or define a set of properties to extract using a struct.
Dataset targets support manual data input | Dataset targets in Pipeline Builder now allow manual data entry, akin to the existing feature for object type targets.
Fallback Branches Support in Pipeline Builder | You can now configure fallback branches in the Manage branches dialog for dataset input resolution and pipeline building.
Direct Access to Ontology Manager from Pipeline Builder | Users can now directly open Ontology type outputs in Ontology Manager from Pipeline Builder. When an Ontology type output is selected, a new action is available to the right of that output, allowing users to view it in Ontology Manager without having to switch applications manually.
Streamlined Column Renaming in Pipeline Builder | The "Edit multiple" functionality within Pipeline Builder has been enhanced to align with the "Normalize columns" transformation board, now using lower snake case for column names and retaining special characters. This refinement promotes a more cohesive and user-friendly renaming process, meeting the normalization expectations of our users.
Data Integration | Data Connection¶
Improved Direct Media Extraction Launch | The Data Connection interface now features an additional button on the media extract overview page, enabling users to directly initiate media extracts. This enhancement simplifies the extraction process by offering a straightforward method to execute media extracts without the need to leave the current page.
Optimized Media Extract Scheduling | Directly within the creation workflow, users now have the capability to set up schedules for media extracts.
Streamlined Media Sync Interface in Data Connection | A newly-introduced media sync overview interface in Data Connection offers users a unified dashboard similar to batch syncs. This new version establishes a foundation for upcoming enhancements, such as an integrated media sync initiation feature and the capability to schedule syncs directly within the platform.
Data Integration | Code Repositories¶
Enhanced Media Embedding in Markdown Files | Markdown files within Code Repositories now support direct embedding of media items using markdown syntax. For example: .
Easy Imports for TypeScript Repositories | Import object types, link types, REST APIs and models in a single, consolidated workflow using the redesigned Resource imports panel. Easily search and preview assets before pulling them into a repository to use in TypeScript functions.
中文翻译¶
公告¶
提醒： 您现在可以订阅Foundry新闻通讯(Newsletter)，直接通过邮箱接收关于平台新产品、功能及改进的摘要信息。有关订阅方式的更多信息，请参阅Foundry新闻通讯与产品反馈渠道公告。
轻量级转换现支持外部转换、增量转换及标记移除¶
发布日期：2024-03-14
我们正在持续扩展轻量级转换的功能集，并很高兴地宣布三项新功能现已可用，旨在提升性能、缩短启动时间和开销，并根据用户资格限制资源访问。
增量转换兼容性¶
增量转换(Incremental transforms)现已兼容轻量级转换。您可以通过此集成将增量转换的性能优势与轻量级转换的快速启动特性相结合。要使轻量级构建支持增量模式，只需添加@incremental装饰器，示例如下：
import pandas
@lightweight()
@incremental(require_incremental=True)
@transform(my_input=Input("my-input"), my_output=Output("my-output"))
def my_incremental_transform(my_input, my_output):
my_output.write_pandas(my_input.pandas(mode="added"))
要开始创建增量轻量级转换，请将您的仓库升级至最新版本，并安装最新版本的foundry-transforms-lib-python。更多详情请查阅文档。
轻量级模式下外部转换性能提升¶
外部转换(External transforms)现已支持@lightweight API，通过减少启动开销来提升性能。此增强功能特别适用于外部转换，因为它们通常不依赖Spark或CPU密集型操作，从而改善延迟和吞吐量。示例如下：
@lightweight
@use_external_systems(
egress=EgressPolicy('<policy RID>'),
creds=Credential('<credential RID>')
)
@transform(
output=Output('/path/to/output/dataset')
)
def compute(egress, creds, ...):
支持标记移除¶
标记(Markings)和组织(Organizations)根据用户资格限制资源访问。当受限内容在派生依赖资源时被移除或模糊化处理，您可能希望移除该派生资源上的标记和/或组织。轻量级转换现已支持移除标记（也称为取消标记）。
有关在轻量级转换中移除标记的更多信息，请查阅文档。
其他亮点¶
应用构建 | Slate¶
使用HTML优化工具提示自定义 | 用户现在可以在图表中使用HTML创建样式化多行工具提示，增强数据呈现和定制能力。
Slate脚本合规性增强 | Slate已增强以符合严格模式 ↗标准，确保脚本解析和错误处理更加精确。现在禁止使用保留关键字作为变量，与ES6模块规范保持一致。
Slate异步能力增强 | Slate已增强以支持异步函数，为更复杂的用例铺平道路，例如查询间超时和集成使用异步函数的库。这一进步在Slate中引入了更简洁的基于Promise的异步操作处理方法，同时保持同步函数执行。
应用构建 | Workshop¶
引入Workshop循环布局 | Workshop的功能已通过引入循环布局(Loop layouts)得到扩展，允许用户遍历对象集并为每个对象呈现嵌入模块。此增强功能简化了数据集合的处理，并提升了用户设计交互式模块的能力。更多信息请参阅循环布局文档。
移动端Workshop模块集成增强 | Workshop移动版现已具备新功能：嵌入模块小部件的集成。这一进步简化了在模块中集成移动兼容模块的过程，在移动设备上营造更具交互性和吸引力的用户体验。嵌入功能仅适用于移动模块，详见文档中的移动模式限制部分。
Workshop记事本模板按钮灵活性升级 | Workshop已更新，允许根据布尔变量条件性地显示和停用记事本模板选项，与按钮组小部件的功能类似。此更新还引入了使用字符串变量的工具提示，为用户提供更好的指导。
Workshop音频预览增强 | Workshop现在为媒体集媒体引用展示波形音频播放器，为用户提供更直观的音频文件预览方式。
Workshop标签滚动增强 | 新的启用溢出(Enable overflow) 配置选项允许用户在Workshop的节标题中轻松浏览溢出的标签。激活后，标签两侧将显示箭头，确保即使标签超出节标题宽度，用户也能访问所有标签。
应用构建 | 本体论SDK¶
开发者控制台外观刷新 | 开发者控制台(Developer Console)已通过新的侧边栏、应用标题栏和浅色主题进行改版，同时优化了主页。
Python批量操作：支持返回编辑 | 用户现在可以使用可选参数batch_action_config将return_edits设置为ALL，并在使用Python批量应用操作时在响应中接收编辑内容。
分析 | Notepad¶
Notepad中引入批处理函数 | Notepad中对象函数部分现在可以配置批处理函数(Batched functions)，这可以显著减少从模板生成文档所需的时间。
批处理函数是接受对象集作为参数并为对象集中每个对象返回结果的函数。用户可以配置嵌套在生成器部分内的对象函数部分的批处理函数。
使用批处理函数可以减少模板生成时间，因为非批处理函数必须为对象集中的每个对象执行一次，而批处理函数仅为整个对象集执行一次。
要开始使用批处理函数，用户可以在Notepad文档中插入对象函数部分，从配置面板中选择批处理(Batched) 选项，然后选择一个批处理函数。了解如何编写或配置批处理函数。
Notepad入口界面改版 | Notepad的入口界面已升级，配备更直观的搜索功能，使用户能够通过标题或内容快速定位笔记。此增强功能引入了可排序的资源列表，包含快速访问类别，如最近、我的创作和已收藏。此外，现在提供帮助图标，可即时参考Notepad指南和AIP帮助，同时简化了生成新Notepad文档或模板的流程。
Notepad Markdown格式化能力改进 | Notepad已升级，能更好地将对象函数输出解释为Markdown，增强在模板中创建动态且视觉吸引人的文本格式的能力。
分析 | 代码工作区¶
代码工作区中引入容器转换 | 现在可以将Jupyter®笔记本和R脚本作为容器转换(Container transforms)发布为数据转换管道。容器转换易于复现，并可安排自动运行。此功能使代码工作区与Foundry实现了一流集成，使您能够轻松与其他用户共享数据分析工作流的输出。要了解有关使用容器转换的更多信息，请阅读Jupyter®和RStudio®的公共文档。
分析 | Contour¶
Contour中简化的分析标记 | Contour现在通过引入分析标记(Analysis tagging)提供了一种精细化的方式来分类和管理您的分析。要分配标签，请选择更多操作菜单，进入标签子菜单，并指定您选择的标签。为了有序查看，可以在分析侧边栏中按这些标签对分析进行排序，简化复杂分析项目的导航和管理。
本体论 | 本体论管理¶
本体论管理器筛选选项优化 | 本体论管理器(Ontology Manager)清理工具现在拥有改进的筛选功能，使用户能够按关联标志的数量对对象类型进行排序。此增强功能有助于以更详细和高效的方式组织本体论清理活动，使用户能够根据不同的标准有效确定任务优先级。
本体论管理器API名称创建增强 | 本体论管理器现在以不同方式生成API名称。过去，当对象类型显示名称包含'['和']'字符时，自动生成的API名称会以下划线开头。通过此改进，API名称将不再以下划线开头，提供更直观和用户友好的命名约定。此更改仅影响默认生成的名称，建议用户进一步自定义。
本体论管理器中可自定义的操作类型API名称 | 直接从概览(Overview) 选项卡，本体论管理器现在支持自定义操作类型的API名称，提供更高的灵活性。
本体论 | Vertex¶
Vertex注释功能增强 | Vertex已升级，包含新的注释模式，允许用户使用各种自定义形状和颜色进行标记和突出显示。此更新为用户提供动态且可自定义的视觉内容注释方法，增强交互体验。
安全 | 项目¶
简化的内容管理和数据集集成 | 用户现在拥有增强的能力，可以直接从URL与数据集集成和管理内容，即使在没有主文件夹的情况下也能简化流程。此改进确保诸如通过URL导入工作表等操作能够轻松执行，无论主文件夹是否可用。
管理 | 控制面板¶
组织标记可在控制面板中管理 | 控制面板(Control Panel)现在允许组织管理员在组织权限的新选项卡中调整组织标记(Organization Marking)权限。此增强功能是整合权限处理的一步，提供更统一和直接的体验。继迁移第二阶段成功实施后（预计于2024年第一季度，并于2023年宣布），控制面板将成为组织权限配置的唯一中心，使平台设置中的先前方法过时。
模型集成 | AIP Logic¶
AIP Logic调试界面优化 | AIP Logic中的调试工具已升级，新增功能包括可切换的块卡片和简化的提示可视化，以及专用的重置工具调用功能。这些改进旨在增强用户在其逻辑流中高效导航和解决问题的能力。
AIP Logic中引入对象集输出 | AIP Logic中的使用LLM块现在支持返回对象集作为输出，增强了与Workshop的集成便利性。此改进使用户能够运行产生多个对象的提示，增强其工作流中可用的分析功能。例如，使用此功能，像"西班牙或意大利的F1车手更多？返回这些车手"这样的提示将返回多个对象作为结果。示例为概念性示例，使用开源数据。
AIP Logic中引入单次完成LLM块 | 单次完成LLM块(Single Completion LLM blocks)
您现在可以从LLM块运行单次完成，无需特殊处理或额外指令。此功能在不需要思维链(CoT)提示的额外功能时，通过在所有提示策略中增加最少的令牌开销来提高性能。请注意，使用单次完成LLM块时，工具和指定返回类型不可用。
AIP Logic中引入转换块 | 转换块(Transform block)允许您在AIP Logic中使用表达式。这是在AIP Logic中对数据执行转换的好方法，例如将字符串转换为日期、解析JSON、执行数学表达式等。
AIP Logic中数组输出能力扩展 | AIP Logic中引入了生成数组输出的能力，便于通过单一响应提供多个建议或值。这一进步增强了与LLM块交互的动态性，以解决需要一系列可能解决方案的问题。
模型集成 | 建模¶
模型部署的媒体集集成优化 | 模型部署流程已升级，将媒体集作为可选输入纳入，特别是当模型的API包含媒体引用时。此增强功能引入了一个媒体集选择器，该选择器会在适用模型的上下文中出现，简化涉及媒体数据的部署配置。
数据集成 | Pipeline Builder¶
Pipeline Builder中的高级属性提取 | Pipeline Builder现在为从geoJSON数据提取属性提供了增强的用户体验。用户可以选择将所有属性提取为字符串，或使用结构体定义一组要提取的属性。
数据集目标支持手动数据输入 | Pipeline Builder中的数据集目标现在允许手动数据输入，类似于对象类型目标的现有功能。
Pipeline Builder中的回退分支支持 | 您现在可以在管理分支(Manage branches) 对话框中为数据集输入解析和管道构建配置回退分支。
从Pipeline Builder直接访问本体论管理器 | 用户现在可以直接从Pipeline Builder在本体论管理器中打开本体论类型输出。当选择本体论类型输出时，该输出右侧会出现一个新操作，允许用户在本体论管理器中查看它，而无需手动切换应用程序。
Pipeline Builder中简化的列重命名 | Pipeline Builder中的"编辑多个(Edit multiple)"功能已增强，与"标准化列(Normalize columns)"转换面板保持一致，现在对列名使用小写蛇形命名法并保留特殊字符。此改进促进了更一致和用户友好的重命名过程，满足用户的标准化期望。
数据集成 | 数据连接¶
改进的直接媒体提取启动 | 数据连接(Data Connection)界面现在在媒体提取概览页面上增加了一个按钮，使用户能够直接启动媒体提取。此增强功能通过提供直接执行媒体提取的方法简化了提取过程，无需离开当前页面。
优化的媒体提取调度 | 用户现在可以直接在创建工作流中设置媒体提取的调度。
数据连接中简化的媒体同步界面 | 数据连接中新引入的媒体同步概览界面为用户提供了类似于批量同步的统一仪表板。此新版本为即将推出的增强功能奠定了基础，例如集成的媒体同步启动功能和直接在平台内调度同步的能力。
数据集成 | 代码仓库¶
Markdown文件中媒体嵌入增强 | 代码仓库中的Markdown文件现在支持使用Markdown语法直接嵌入媒体项。例如：。
TypeScript仓库的简易导入 | 使用重新设计的资源导入(Resource imports) 面板，在单一整合工作流中导入对象类型、链接类型、REST API和模型。在将资源拉入仓库以在TypeScript函数中使用之前，轻松搜索和预览资源。