SnappyUnifiedMemoryManager
Related Docs:
object SnappyUnifiedMemoryManager
| package memory
Acquire N bytes of memory to cache the given block, evicting existing ones if necessary.
Acquire N bytes of memory to cache the given block, evicting existing ones if necessary.
whether all N bytes were successfully granted.
Acquire N bytes of memory to unroll the given block, evicting existing ones if necessary.
Acquire N bytes of memory to unroll the given block, evicting existing ones if necessary.
This extra method allows subclasses to differentiate behavior between acquiring storage memory and acquiring unroll memory. For instance, the memory management model in Spark 1.5 and before places a limit on the amount of space that can be freed from unrolling.
whether all N bytes were successfully granted.
Change the off-heap owner to mark it being used for storage.
Change the off-heap owner to mark it being used for storage. Passing the owner as null allows moving ByteBuffers not allocated by BufferAllocators to be also changed and freshly accounted.
Clears the internal map
Clears the internal map
This method will be called if executor is going to be restarted.
This method will be called if executor is going to be restarted. When executor is coming up all accounting from store will be done in bootMemoryManager. When executor stops we will copy the existing entry in this manager to boot manager. Once executor comes back again we will again copy the boot manager entries to run time manager.
Execution memory currently in use, in bytes.
Execution memory currently in use, in bytes.
Initializes the memoryManager
Initializes the memoryManager
the maximum heap memory that is available for use by MemoryManager; callers should leave out some amount of "reserved memory" for unaccounted object allocations
the maximum heap memory that is available for use by MemoryManager; callers should leave out some amount of "reserved memory" for unaccounted object allocations
Total available off heap memory for storage, in bytes.
Total available off heap memory for storage, in bytes. This amount can vary over time, depending on the MemoryManager implementation.
Total available on heap memory for storage, in bytes.
Total available on heap memory for storage, in bytes. This amount can vary over time, depending on the MemoryManager implementation. In this model, this is equivalent to the amount of memory not occupied by execution.
The default page size, in bytes.
The default page size, in bytes.
If user didn't explicitly set "spark.buffer.pageSize", we figure out the default value by looking at the number of cores available to the process, and the total amount of memory, and then divide it by a factor of safety.
Release all storage memory acquired.
Release all storage memory acquired.
Release numBytes of execution memory belonging to the given task.
Release numBytes of execution memory belonging to the given task.
Release N bytes of storage memory.
Release N bytes of storage memory.
Release N bytes of unroll memory.
Release N bytes of unroll memory.
Set the MemoryStore used by this manager to evict cached blocks.
Set the MemoryStore used by this manager to evict cached blocks. This must be set after construction due to initialization ordering constraints.
Storage memory currently in use, in bytes.
Storage memory currently in use, in bytes.
Tracks whether Tungsten memory will be allocated on the JVM heap or off-heap using sun.misc.Unsafe.
Tracks whether Tungsten memory will be allocated on the JVM heap or off-heap using sun.misc.Unsafe.
When there is request for execution or storage memory, critical up and eviction up events are checked. If they are set, try to free the memory cached by Spark rdds by calling memoryStore.evictBlocksToFreeSpace. If enough memory cannot be freed, return the call and let Spark take a corrective action. In such cases Spark either fails the task or move the current RDDs data to disk. If the critical and eviction events are not set, it asks the UnifiedMemoryManager to allocate the space.