performance - MongoDB 按 ID 慢查询

标签 performance mongodb mongodb-java spring-data-mongodb

我最近将我的 MongoDB 数据库从 Windows Server 迁移到了 CentOS。 版本是2.4.9。 我注意到我通过 _id 字段检索记录的速度很慢! 我在周末运行了一个修复数据库，但没有解决问题。 我有这种通过 ID 检索记录的方法(使用 IN 运算符)(使用 spring data mongodb 1.4.1.RELEASE):

@Override
public Map<String, Record> findAsMapIds(
    final String[] ids,
    final ComponentType... comps
) {
    if( null == ids || 0 == ids.length ) {
        return null;
    }
    Map<String,Record> result = new HashMap<String,Record>();
    final Criteria cr =  where("_id").in( idArrFunction.apply(ids) );
    final Query qry = new Query( cr );
    setFieldsToReturn( qry, comps );
    long start = System.currentTimeMillis();
    List<Record> ritems = getTemplate().find(qry, Record.class);
    long end = System.currentTimeMillis();
    System.out.println( "findAsMapIds()::" + (end-start) );
    for( Record r: ritems ) {
        result.put( r.getId(), r );
    }
    return result;
}

public MongoOperations getTemplate() {
    return template;
}

函数 idArrFunction 返回一个 ObjectId[] 数组

您注意到我只对数据检索调用计时。 我也不会一次传递超过 10 个 ID。 这是我多次调用此方法时在日志中得到的计时输出:

findAsMapIds()::2092
findAsMapIds()::6666
findAsMapIds()::10007
findAsMapIds()::15132
findAsMapIds()::18341
findAsMapIds()::24709
findAsMapIds()::28731
findAsMapIds()::33888
findAsMapIds()::36862
findAsMapIds()::40206

您注意到我每次调用此方法所需的时间都在增加。

以下是收藏统计数据:

> db.record.stats();
{
    "ns" : "mydb.record",
    "count" : 1491427,
    "size" : 18292198304,
    "avgObjSize" : 12264.896843090544,
    "storageSize" : 18899890096,
    "numExtents" : 30,
    "nindexes" : 13,
    "lastExtentSize" : 2146426864,
    "paddingFactor" : 1,
    "systemFlags" : 1,
    "userFlags" : 0,
    "totalIndexSize" : 734401024,
    "indexSizes" : {
        "_id_" : 43561728,
        "orgId" : 63192304,
        "type" : 51018240,
        "orgHierarchy" : 63274064,
        "orgHierarchy_mrn" : 75603472,
        "dos" : 25525472,
        "org_pdt" : 77189616,
        "orgHierarchy_processedDtTm" : 77287728,
        "processedDtTm_-1" : 37536016,
        "orgId_states" : 68367712,
        "states" : 27553120,
        "states_1_processedDtTm_-1" : 41624016,
        "orgId_1_states_1_processedDtTm_-1" : 82667536
    },
    "ok" : 1
}

如果我从 java 中打印出 Query 对象，我会得到这个:

Query: { "_id" : { "$in" : [ { "$oid" : "51cbe96dcb81c43374ca9516"} , { "$oid" : "51cbe892cb81c43374ca5e48"} , { "$oid" : "51cbe910cb81c43374ca7aba"} , { "$oid" : "51c00739cb81e6b83ad29d14"} , { "$oid" : "51c06ff4cb81e6b83ad4f69a"} , { "$oid" : "51c07010cb81e6b83ad4f74e"} , { "$oid" : "51c06da5cb81e6b83ad4e775"} , { "$oid" : "51c081bbcb81e6b83ad56437"} , { "$oid" : "51c07fe2cb81e6b83ad557d2"} , { "$oid" : "51c081bbcb81e6b83ad56433"}]}}, Fields: { "text" : 1 , "type" : 1 , "header" : 1 , "orgId" : 1 , "vitals" : 1 , "_id" : 1 , "processedDtTm" : 1 , "orgHierarchy" : 1 }, Sort: null

另外，如果我运行解释:

> db.record.find({ "_id" : { "$in" : [ ObjectId("51ba0b43cb81e6b83acd9e6b") , ObjectId("51a8feb5cb8168350e373428") , ObjectId("51a90053cb8168350e373948") , ObjectId("51a901fecb8168350e373e82") , ObjectId("51a8f3e9cb8168350e3711d1") , ObjectId("51a8f3eacb8168350e3711d2") , ObjectId("51a8ff3ecb8168350e3735df") , ObjectId("51a8f59ecb8168350e37172b") , ObjectId("51a8f9aacb8168350e372418") , ObjectId("51a8f786cb8168350e371d42")]}}).explain();
{
    "cursor" : "BtreeCursor _id_ multi",
    "isMultiKey" : false,
    "n" : 10,
    "nscannedObjects" : 10,
    "nscanned" : 18,
    "nscannedObjectsAllPlans" : 10,
    "nscannedAllPlans" : 18,
    "scanAndOrder" : false,
    "indexOnly" : false,
    "nYields" : 0,
    "nChunkSkips" : 0,
    "millis" : 0,
    "indexBounds" : {
        "_id" : [
            [
                ObjectId("51a8f3e9cb8168350e3711d1"),
                ObjectId("51a8f3e9cb8168350e3711d1")
            ],
            [
                ObjectId("51a8f3eacb8168350e3711d2"),
                ObjectId("51a8f3eacb8168350e3711d2")
            ],
            [
                ObjectId("51a8f59ecb8168350e37172b"),
                ObjectId("51a8f59ecb8168350e37172b")
            ],
            [
                ObjectId("51a8f786cb8168350e371d42"),
                ObjectId("51a8f786cb8168350e371d42")
            ],
            [
                ObjectId("51a8f9aacb8168350e372418"),
                ObjectId("51a8f9aacb8168350e372418")
            ],
            [
                ObjectId("51a8feb5cb8168350e373428"),
                ObjectId("51a8feb5cb8168350e373428")
            ],
            [
                ObjectId("51a8ff3ecb8168350e3735df"),
                ObjectId("51a8ff3ecb8168350e3735df")
            ],
            [
                ObjectId("51a90053cb8168350e373948"),
                ObjectId("51a90053cb8168350e373948")
            ],
            [
                ObjectId("51a901fecb8168350e373e82"),
                ObjectId("51a901fecb8168350e373e82")
            ],
            [
                ObjectId("51ba0b43cb81e6b83acd9e6b"),
                ObjectId("51ba0b43cb81e6b83acd9e6b")
            ]
        ]
    },
    "server" : "ip-10-0-0-41:27017"
}

我打开了分析器，这是我得到的:

{
   "op":"query",
   "ns":"sytrue.record",
   "query":{
      "_id":{
     "$in":[
        ObjectId("51b1c98dcb81e4c1566b7436"),
        ObjectId("51b1cb3ccb81e4c1566b79ab"),
        ObjectId("51b1d741cb81e4c1566ba219"),
        ObjectId("51b1caa8cb81e4c1566b77d4"),
        ObjectId("51b1c98ccb81e4c1566b7433"),
        ObjectId("51b1c691cb81e4c1566b6a8c"),
        ObjectId("51b1cc21cb81e4c1566b7c9f"),
        ObjectId("51b1c390cb81e4c1566b60c9"),
        ObjectId("51b1d578cb81e4c1566b9c1d"),
        ObjectId("51b1e13dcb81e4c1566bc30e")
     ]
      }
   },
   "ntoreturn":0,
   "ntoskip":0,
   "nscanned":19,
   "keyUpdates":0,
   "numYield":0,
   "lockStats":{
      "timeLockedMicros":{
     "r":NumberLong(380),
     "w":NumberLong(0)
      },
      "timeAcquiringMicros":{
     "r":NumberLong(2),
     "w":NumberLong(4)
      }
   },
   "nreturned":10,
   "responseLength":36745,
   "millis":0,
   "ts":   ISODate("2014-03-24T09:55:20.988   Z"),
   "client":"127.0.0.1",
   "allUsers":[
      {
     "user":"sytrue-user",
     "userSource":"sytrue"
      }
   ],
   "user":"sytrue-user@sytrue"
}
  • 我还注意到，当我通过 ID 检索单个记录时，它是 也很慢(PagingAndSortingRepository 的 findOne 方法)。
  • 我不得不提一下，我在 Windows 服务器上没有遇到这个问题。
  • 请有人解释发生了什么。

更新:我注意到当返回结果时，CPU 被 java 代码消耗，而不是 mongo 服务器进程。所以问题出在mongodb-java驱动或者spring-data-mongodb

最佳答案

在与 MongoDB 开发者聊天时发现问题: 我运行的是 spring framework 4.0.0，速度慢与这个问题有关:

https://jira.spring.io/browse/SPR-11335

将 Spring 更新到 4.0.1 版本(或更高版本)可以解决问题。

这是我调用问题时的堆栈跟踪:

    at java.lang.Class.getGenericSignature(Native Method)
    at java.lang.Class.getGenericInterfaces(Class.java:813)
    at org.springframework.core.SerializableTypeWrapper$2.getType(SerializableTypeWrapper.java:106)
    at org.springframework.core.SerializableTypeWrapper$TypeProxyInvocationHandler.invoke(SerializableTypeWrapper.java:220)
    at com.sun.proxy.$Proxy5.equals(Unknown Source)
    at org.springframework.util.ObjectUtils.nullSafeEquals(ObjectUtils.java:252)
    at org.springframework.core.ResolvableType.equals(ResolvableType.java:682)
    at org.springframework.util.ObjectUtils.nullSafeEquals(ObjectUtils.java:252)
    at org.springframework.core.ResolvableType.variableResolverSourceEquals(ResolvableType.java:719)
    at org.springframework.core.ResolvableType.equals(ResolvableType.java:683)
    at org.springframework.util.ConcurrentReferenceHashMap$Segment.findInChain(ConcurrentReferenceHashMap.java:571)
    at org.springframework.util.ConcurrentReferenceHashMap$Segment.getReference(ConcurrentReferenceHashMap.java:439)
    at org.springframework.util.ConcurrentReferenceHashMap.getReference(ConcurrentReferenceHashMap.java:238)
    at org.springframework.util.ConcurrentReferenceHashMap.get(ConcurrentReferenceHashMap.java:217)
    at org.springframework.core.ResolvableType.forType(ResolvableType.java:1018)
    at org.springframework.core.ResolvableType.forType(ResolvableType.java:998)
    at org.springframework.core.ResolvableType.getGenerics(ResolvableType.java:489)
    at org.springframework.core.ResolvableType.getGeneric(ResolvableType.java:451)
    at org.springframework.core.convert.TypeDescriptor.getElementTypeDescriptor(TypeDescriptor.java:305)
    at org.springframework.core.convert.TypeDescriptor.equals(TypeDescriptor.java:456)
    at org.springframework.util.ObjectUtils.nullSafeEquals(ObjectUtils.java:252)
    at org.springframework.core.convert.support.GenericConversionService$ConverterCacheKey.equals(GenericConversionService.java:436)
    at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:996)
    at org.springframework.core.convert.support.GenericConversionService.getConverter(GenericConversionService.java:235)
    at org.springframework.core.convert.support.GenericConversionService.convert(GenericConversionService.java:176)
    at org.springframework.core.convert.support.GenericConversionService.convert(GenericConversionService.java:162)
    at org.springframework.data.mongodb.core.convert.MappingMongoConverter.getPotentiallyConvertedSimpleRead(MappingMongoConverter.java:715)
    at org.springframework.data.mongodb.core.convert.MappingMongoConverter.readCollectionOrArray(MappingMongoConverter.java:773)
    at org.springframework.data.mongodb.core.convert.MappingMongoConverter.readValue(MappingMongoConverter.java:1048)
    at org.springframework.data.mongodb.core.convert.MappingMongoConverter.access$100(MappingMongoConverter.java:77)
    at org.springframework.data.mongodb.core.convert.MappingMongoConverter$MongoDbPropertyValueProvider.getPropertyValue(MappingMongoConverter.java:999)
    at org.springframework.data.mongodb.core.convert.MappingMongoConverter.getValueInternal(MappingMongoConverter.java:755)
    at org.springframework.data.mongodb.core.convert.MappingMongoConverter$1.doWithPersistentProperty(MappingMongoConverter.java:256)
    at org.springframework.data.mongodb.core.convert.MappingMongoConverter$1.doWithPersistentProperty(MappingMongoConverter.java:249)
    at org.springframework.data.mapping.model.BasicPersistentEntity.doWithProperties(BasicPersistentEntity.java:261)
    at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:249)
    at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:223)
    at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:187)
    at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:183)
    at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:77)
    at org.springframework.data.mongodb.core.MongoTemplate$ReadDbObjectCallback.doWith(MongoTemplate.java:1966)
    at org.springframework.data.mongodb.core.MongoTemplate.executeFindMultiInternal(MongoTemplate.java:1662)
    at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:1487)
    at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:1471)
    at org.springframework.data.mongodb.core.MongoTemplate.find(MongoTemplate.java:527)
    at org.springframework.data.mongodb.core.MongoTemplate.find(MongoTemplate.java:518)
    at com.sytrue.record.persistence.repo.impl.RecordRepositoryImpl.findAsMapIds(RecordRepositoryImpl.java:139)

关于performance - MongoDB 按 ID 慢查询，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/22603694/

上一篇：ruby-on-rails - 为 mongoid 哈希字段设置默认哈希键

下一篇：javascript - 从 Node 中的 mongodb 查询结果创建另一个集合的问题

相关文章：

ruby - 如何用大正则表达式加快扫描大字符串的速度？

java - 查询用户时 JPA 潜在的性能问题

mongodb - 在 NoSQL (MongoDB) 中存储/同步 Facebook Graph

mongodb - 如何在Mongodb java驱动程序中编写多个group by id字段

performance - SMTP Sampler 在发送消息时执行接收 IOExeception

javascript - 附加太多事件处理程序时的 IE8 内存泄漏/性能问题

java - 如何使用 portainer 名称作为主机名连接到 mongodb？

java - Gradle 依赖项在 IntelliJ 中不起作用

java - 尝试通过 Lambda : No address associated with hostname 连接到 Atlas

java - 使用没有迭代器的 Java 复制集合 Mongodb 以进行数据库管理员登录

©2024 IT工具网  联系我们