我运行选择/插入到 postgresql 数据库的 php 脚本。
问题是当我运行脚本选择超过 (400000) 行时，
脚本重复插入相同的行!
我在 postgresql 日志中发现了这个错误:
2014-12-21 13:14:53 AST LOG: checkpoints are occurring too frequently (7 seconds apart)
2014-12-21 13:14:53 AST HINT: Consider increasing the configuration parameter "checkpoint_segments".
我将配置更改为:
- Checkpoints -
checkpoint_segments = 100 # in logfile segments, min 1, 16MB each
checkpoint_timeout = 1h # range 30s-1h
checkpoint_completion_target = 0.9 # checkpoint target duration, 0.0 - 1.0
checkpoint_warning = 30s # 0 disables
但是问题还是出现了。谁能告诉我如何解决这个问题？
更新
我用一个准备查询开始我的脚本:
<?php
// * Connect to Specified Database
set_time_limit(0);
$dbconn = pg_connect("host=localhost port=5432 dbname=postgis user=postgres password=****");
if (!$dbconn) { die("Error in connection: " . pg_last_error());}
// connected to Database
else{
//We prepare the PostgreSQL next messages query. At this stage it is sent off to the Database server.
$nextMessageQuery = pg_prepare($dbconn, 'selectNextMessage', "
WITH
history AS (select id ,userid,assigning_date from userid_history
where id=$2 and assigning_date <$4),
accounttbl AS(select timestamp,userid,position from account
where timestamp >$1 and position is not null
and timestamp between $3 and $4)
select p.value1 ,value2.value3,p.value4, m.userid
from (
SELECT min(next.timestamp)AS value3,next.userid,next.id from(
select history.id,accounttbl.userid,accounttbl.timestamp,history.assigning_date,accounttbl.position
from history
inner join accounttbl
on(
accounttbl.userid=history.userid and
accounttbl.timestamp > history.assigning_date and
( accounttbl.timestamp <(select min(assigning_date) from history h
where h.id=history.id and
h.assigning_date>history.assigning_date)
or
(select min(assigning_date) from history h
where h.id=history.id and h.assigning_date>history.assigning_date) is null
)
)
where timestamp >$1
)next GROUP BY next.userid,next.id
) value2
JOIN nmea m on m.timestamp=value2.value3 and m.userid=value2.userid ,places p
WHERE m.position is not null and ST_DWithin(m.position,p.position,0.0217130577252428)order by ST_Distance(p.position,m.position)");
?>
这只是为了向您展示我的查询有多么复杂。
然后 选择#rows，然后 基于许多比较语句插入或更新到新表
所有这些都在一个复杂的脚本中。 我必须发布所有内容吗？
最佳答案
谢谢大家，
正如 Craig 提到的，数据本身存在错误，而不是数据加载。
实际上，我们选择的某些记录中存在复制!
所以我用了:
SELECT DISTINCT on (time , id) time , id ,.... FROM table
关于php 脚本在 postgresql 上运行时重复自身，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/27833983/