Remote Persistent File List Filters

入站和流入站远程文件通道适配器(FTPSFTP 和其他技术)在默认情况下配置了 AbstractPersistentFileListFilter 的相应实现,并配置了内存中的 MetadataStore。要在集群中运行,可以使用共享 MetadataStore 的过滤器替换它们(有关详细信息,请参阅 Metadata Store)。这些过滤器用于防止多次获取同一文件(除非其修改时间发生改变)。从 5.2 版开始,文件在获取之前立即添加到过滤器中(如果获取失败,则会反转)。

Inbound and streaming inbound remote file channel adapters (FTP, SFTP, and other technologies) are configured with corresponding implementations of AbstractPersistentFileListFilter by default, configured with an in-memory MetadataStore. To run in a cluster, these can be replaced with filters using a shared MetadataStore (see Metadata Store for more information). These filters are used to prevent fetching the same file multiple times (unless it’s modified time changes). Starting with version 5.2, a file is added to the filter immediately before the file is fetched (and reversed if the fetch fails).

发生灾难性故障(例如停电)的情况下,可能会导致正在获取的文件保留在过滤器中,并在重启应用程序时不会重新获取。在这种情况下,您需要手动从 MetadataStore 中删除该文件。

In the event of a catastrophic failure (such as power loss), it is possible that the file currently being fetched will remain in the filter and won’t be re-fetched when restarting the application. In this case you would need to manually remove this file from the MetadataStore.

在以前版本中,文件在获取任何文件后才进行筛选,这意味着在发生灾难性故障后,几个文件可能会处于此状态。

In previous versions, the files were filtered before any were fetched, meaning that several files could be in this state after a catastrophic failure.

为了促进此新行为,已向 FileListFilter 添加了两种新方法。

In order to facilitate this new behavior, two new methods have been added to FileListFilter.

boolean accept(F file);

boolean supportsSingleFileFiltering();

如果筛选器在 supportsSingleFileFiltering 中返回 true,则它*必须*实现 accept()

If a filter returns true in supportsSingleFileFiltering, it must implement accept().

如果远程筛选器不支持单文件筛选(如 AbstractMarkerFilePresentFileListFilter),适配器将恢复为以前的行为。

If a remote filter does not support single file filtering (such as the AbstractMarkerFilePresentFileListFilter), the adapters revert to the previous behavior.

如果使用了多个筛选器(使用 CompositeFileListFilterChainFileListFilter),则所有委托筛选器*都*必须支持单文件筛选,以便组合筛选器支持它。

If multiple filters are in used (using a CompositeFileListFilter or ChainFileListFilter), then all of the delegate filters must support single file filtering in order for the composite filter to support it.

持久的过滤文件列表现在有一个布尔属性 forRecursion。将此属性设置为 true,还将设置 alwaysAcceptDirectories,这意味着出站网关(lsmget)上的递归操作现在将始终在每次遍历完整目录树。这是为了解决目录树中深处更改未被检测到的问题。此外,forRecursion=true 会导致使用文件的完整路径作为元数据存储键;这解决了在不同目录中多次出现具有相同名称的文件时过滤器无法正常工作的问题。重要提示:这意味着无法在顶级目录下的文件找到持久元数据存储中的现有键。因此,该属性默认为 false;这可能会在未来版本中更改。

The persistent file list filters now have a boolean property forRecursion. Setting this property to true, also sets alwaysAcceptDirectories, which means that the recursive operation on the outbound gateways (ls and mget) will now always traverse the full directory tree each time. This is to solve a problem where changes deep in the directory tree were not detected. In addition, forRecursion=true causes the full path to files to be used as the metadata store keys; this solves a problem where the filter did not work properly if a file with the same name appears multiple times in different directories. IMPORTANT: This means that existing keys in a persistent metadata store will not be found for files beneath the top level directory. For this reason, the property is false by default; this may change in a future release.