php - 缓存一个 302MB 的对象

Question

我有一个 302MB 的对象(实际上是一个对象数组)。当我尝试用 memcached 缓存它时，无论我给 memcached 多少内存，它都不起作用，显然是因为 memcached 对它可以缓存的对象有 1MB 的限制。 (最后一部分我可能是错的。我没能找到很好的文档。)

关于如何缓存这个东西有什么建议吗？我在 Linux 上使用 PHP/symfony。

Answer 1

Quoting

15.5.5.4: What is the max size of an object you can store in memcache and is that configurable?

The default maximum object size is 1MB. In memcached 1.4.2 and later you can change the maximum size of an object using the -I command line option.

For versions before this, to increase this size, you have to re-compile memcached. You can modify the value of the POWER_BLOCK within the slabs.c file within the source.

In memcached 1.4.2 and higher you can configure the maximum supported object size by using the -I command-line option. For example, to increase the maximum object size to 5MB:

 $ memcached -I 5m

然而，即使增加内存，这也不是一个好的选择 IMO。一个更好的主意是将对象分成更小的部分，然后缓存它的各个部分。

引用 Why are items limited to 1 megabyte in size?

Short answer: Because of how the memory allocator's algorithm works.

Long answer: Memcached's memory storage engine (which will be pluggable/adjusted in the future...), uses a slabs approach to memory management. Memory is broken up into slabs chunks of varying sizes, starting at a minimum number and ascending by a factorial up to the largest possible value.

Say the minimum value is 400 bytes, and the maximum value is 1 megabyte, and the factorial is 1.20:

slab 1 - 400 bytes slab 2 - 480 bytes slab 3 - 576 bytes ... etc.

The larger the slab, the more of a gap there is between it and the previous slab. So the larger the maximum value the less efficient the memory storage is. Memcached also has to pre-allocate some memory for every slab that exists, so setting a smaller factorial with a larger max value will require even more overhead.

There're other reason why you wouldn't want to do that... If we're talking about a web page and you're attempting to store/load values that large, you're probably doing something wrong. At that size it'll take a noticeable amount of time to load and unpack the data structure into memory, and your site will likely not perform very well.

If you really do want to store items larger than 1MB, you can recompile memcached with an edited slabs.c:POWER_BLOCK value, or use the inefficient malloc/free backend. Other suggestions include a database, MogileFS, etc.