node.js - Sequelize "ignoreDuplicate: true"递增主键？

Question

我在带有 Postgresql 的 Node js 项目中使用 sequelize。
我正在解析一些表并在我的数据库上更新它的副本。想象一下，如果我现在外部表中有 50000 条记录，10 分钟后的差异约为 5%。我不需要删除，只需要创建新的。我用了

Model.bulkCreate(data, { ignoreDuplicates: true });

经过 2 小时的工作解析数据库计数 ~100 000 条记录，但最后 ~1 000 000 的 id。看起来像 ignoreDuplicates: true 自动递增 id。如何在不使用太多 CPU 的情况下避免它？ updateOnDuplicate 做同样的事情吗？
我找到的最快的方法是

const _ = require('lodash');

// some code
let existing = await Model.findAll({ attributes: [uniqColumn] });

let toCreate = _.differenceBy(data, existing, uniqColumn);
await Model.bulkCreate(toCreate);

Answer 1

如果您使用 PostgreSQL sequence 生成唯一列，那么即使由于冲突而未插入行，仍会使用序列值。
见 next_val 。

Important: To avoid blocking concurrent transactions that obtain numbers from the same sequence, a nextval operation is never rolled back; that is, once a value has been fetched it is considered used and will not be returned again. This is true even if the surrounding transaction later aborts, or if the calling query ends up not using the value. For example an INSERT with an ON CONFLICT clause will compute the to-be-inserted tuple, including doing any required nextval calls, before detecting any conflict that would cause it to follow the ON CONFLICT rule instead. Such cases will leave unused "holes" in the sequence of assigned values. Thus, PostgreSQL sequence objects cannot be used to obtain "gapless" sequences.