asp.net - 如何防止 Hangfire 重复作业在连续执行 30 分钟后重新启动

Question

我正在开发一个 asp.net mvc-5 Web 应用程序，并且在使用 Hangfire 工具运行长时间运行的后台作业时遇到问题。 问题是，如果作业执行超过30分钟，那么hangfire会自动启动另一个作业，所以我最终会同时运行两个类似的作业。

现在我有以下内容:-

1. Asp.net mvc-5
2. IIS-8
3. Hangfire 1.4.6
4. Windows 服务器 2012

现在我已经定义了一个 Hangfire 重复作业在每天 17:00 运行。后台作业主要扫描我们的网络中的服务器和虚拟机并更新数据库，循环作业将在执行完成后发送电子邮件。 过去，当执行时间少于 30 分钟时，循环作业运行良好。但如今，随着我们系统的发展，重复性作业在 40 分钟后即可完成，而不是像以前那样需要 22-25 分钟。我收到了 2 封电子邮件，而不是一封电子邮件(两封电子邮件之间的时间间隔约为 30 分钟)。现在我手动重新运行该作业，我注意到问题如下:-

"when the recurring job reaches 30 minutes of continuous execution, a new instance of the recurring job will start, so I will have two instances instead of one running at the same time, so that why I received 2 emails."

现在，如果重复作业花费的时间少于 30 分钟(例如 29 分钟)，我将不会遇到任何问题，但如果重复作业执行时间超过 30 分钟，则出于某种原因或另一个挂火将启动新作业。 尽管当我在作业执行期间访问hangfire仪表板时，我可以发现只有一个事件作业，但当我监视数据库时，我可以从sql分析器中看到有两个作业正在访问数据库。这种情况发生在重复性作业开始后 30 分钟后(在我们的例子中为 17:30)，这就是为什么我收到了 2 封电子邮件，这意味着有 2 个重复性作业在后台运行，而不是一个。

请问有人可以就此提出建议吗？如果当前重复作业执行超过 30 分钟，我如何避免自动启动新的重复作业而发生挂火？ 谢谢

Answer 1

您是否查看过 Hangfire docs 中的 InvisibilityTimeout 设置？ ？

Default SQL Server job storage implementation uses a regular table as a job queue. To be sure that a job will not be lost in case of unexpected process termination, it is deleted only from a queue only upon a successful completion.

To make it invisible from other workers, the UPDATE statement with OUTPUT clause is used to fetch a queued job and update the FetchedAt value (that signals for other workers that it was fetched) in an atomic way. Other workers see the fetched timestamp and ignore a job. But to handle the process termination, they will ignore a job only during a specified amount of time (defaults to 30 minutes).

Although this mechanism ensures that every job will be processed, sometimes it may cause either long retry latency or lead to multiple job execution. Consider the following scenario:

1. Worker A fetched a job (runs for a hour) and started it at 12:00.
2. Worker B fetched the same job at 12:30, because the default invisibility timeout was expired.
3. Worker C (did not fetch) the same job at 13:00, because (it will be deleted after successful performance.)

If you are using cancellation tokens, it will be set for Worker A at 12:30, and at 13:00 for Worker B. This may lead to the fact that your long-running job will never be executed. If you aren’t using cancellation tokens, it will be concurrently executed by WorkerA and Worker B (since 12:30), but Worker C will not fetch it, because it will be deleted after successful performance.

So, if you have long-running jobs, it is better to configure the invisibility timeout interval:

var options = new SqlServerStorageOptions
{
    InvisibilityTimeout = TimeSpan.FromMinutes(30) // default value
};

GlobalConfiguration.Configuration.UseSqlServerStorage("<name or connection string>", options);

截至Hangfire 1.5 this option is now Obsolete 。其他 worker 看不到正在从事的工作。

Say goodbye to confusing invisibility timeout with unexpected background job retries after 30 minutes (by default) when using SQL Server. New Hangfire.SqlServer implementation uses plain old transactions to fetch background jobs and hide them from other workers.

Even after ungraceful shutdown, the job will be available for other workers instantly, without any delays.