sql - 在 postgres 中使用 upsert 和 nodejs 中的 json 文件

Question

我有一个很大的 json 数据文件，我想将其放入我的数据库中。有些对象是重复的，所以我想更新数据，以防该行已经在数据库中。这是我的代码:

const FILE_PATH = path.join(__dirname, "../../files/apps.json");

const columns = [
          "name",
          "description",
          "ext_id"
        ];

const myFile = fs.readFileSync(FILE_PATH, { encoding: "utf-8" });
const appData = await models.sequelize.query(
          `
        INSERT INTO data (${columns.join(", ")})
        SELECT ${columns.join(", ")}
        FROM (:path)
        ON CONFLICT (ext_id)
          DO UPDATE SET
            ${columns.map(col => `${col} = EXCLUDED.${col}`).join(", ")}
        RETURNING ext_id;
        `,
          { replacements: { path: FILE_PATH } }
        );

如您所见，我想直接读取文件并将其放入数据库。我使用名为 sequelize 的映射器，但在这种情况下使用原始查询。我的直接问题是我收到此错误:

syntax error at or near "'/home/blub/filePath'"

我真的不知道我应该如何指定路径。我试图直接解析它，但随后程序提示/.这里有什么帮助吗？此外，我也不确定查询在语法上是否正确。

Answer 1

这是使用 postgres 的 CTE 的解决方案。
版本:

"sequelize": "^5.21.3"

postgres:9.6

apps.json :

[
    {
        "name": "app-a",
        "description": "app a desc",
        "ext_id": 1
    },
    {
        "name": "app-b",
        "description": "app b desc",
        "ext_id": 2
    },
    {
        "name": "app-c",
        "description": "app c desc",
        "ext_id": 3
    }
]

index.ts :

import { sequelize } from '../../db';
import { Model, DataTypes, QueryTypes } from 'sequelize';
import fs from 'fs';
import path from 'path';

class Data extends Model {}
Data.init(
  {
    name: DataTypes.STRING,
    description: DataTypes.STRING,
    ext_id: {
      type: DataTypes.INTEGER,
      unique: true,
    },
  },
  { sequelize, tableName: 'data' },
);

(async function test() {
  try {
    await sequelize.sync({ force: true });
    const FILE_PATH = path.join(__dirname, './apps.json');
    const columns = ['name', 'description', 'ext_id'];
    const myFile = fs.readFileSync(FILE_PATH, { encoding: 'utf-8' });
    const appData = await sequelize.query(
      `
        with app_json(doc) as (
            values ('${myFile}'::json)
        )
        insert into data (${columns.join(', ')})
        select ${columns.join(', ')}
        from app_json l
            cross join lateral json_populate_recordset(null::data, doc) as p
        on conflict (ext_id) do update 
            set ${columns.map((col) => `${col} = EXCLUDED.${col}`).join(', ')}
        returning ext_id;
    `,
      { type: QueryTypes.INSERT },
    );
    console.log(appData);
  } catch (error) {
    console.log(error);
  } finally {
    await sequelize.close();
  }
})();

执行结果:

Executing (default): DROP TABLE IF EXISTS "data" CASCADE;
Executing (default): DROP TABLE IF EXISTS "data" CASCADE;
Executing (default): CREATE TABLE IF NOT EXISTS "data" ("id"   SERIAL , "name" VARCHAR(255), "description" VARCHAR(255), "ext_id" INTEGER UNIQUE, PRIMARY KEY ("id"));
Executing (default): SELECT i.relname AS name, ix.indisprimary AS primary, ix.indisunique AS unique, ix.indkey AS indkey, array_agg(a.attnum) as column_indexes, array_agg(a.attname) AS column_names, pg_get_indexdef(ix.indexrelid) AS definition FROM pg_class t, pg_class i, pg_index ix, pg_attribute a WHERE t.oid = ix.indrelid AND i.oid = ix.indexrelid AND a.attrelid = t.oid AND t.relkind = 'r' and t.relname = 'data' GROUP BY i.relname, ix.indexrelid, ix.indisprimary, ix.indisunique, ix.indkey ORDER BY i.relname;
Executing (default): with app_json(doc) as (
            values ('[
    {
        "name": "app-a",
        "description": "app a desc",
        "ext_id": 1
    },
    {
        "name": "app-b",
        "description": "app b desc",
        "ext_id": 2
    },
    {
        "name": "app-c",
        "description": "app c desc",
        "ext_id": 3
    }
]'::json)
        )
        insert into data (name, description, ext_id)
        select name, description, ext_id
        from app_json l
            cross join lateral json_populate_recordset(null::data, doc) as p
        on conflict (ext_id) do update 
            set name = EXCLUDED.name, description = EXCLUDED.description, ext_id = EXCLUDED.ext_id
        returning ext_id;
[ [ { ext_id: 1 }, { ext_id: 2 }, { ext_id: 3 } ], 3 ]

查看数据库中的数据记录:

id  name    description ext_id
1   app-a   app a desc  1
2   app-b   app b desc  2
3   app-c   app c desc  3