mmap() vs. reading blocks
I heard (read it on the internet somewhere) that
mmap() is faster than sequential IO. Is this correct? If yes then why it is faster?
mmap()is not reading sequentially.
mmap()has to fetch from the disk itself same as
- The mapped area is not sequential – so no DMA (?).
mmap() should actually be slower than
read() from a file? Which of my assumptions above are wrong?
I heard (read it on the internet somewhere) that mmap() is faster than sequential IO. Is this correct? If yes then why it is faster?
It can be – there are pros and cons, listed below. When you really have reason to care, always benchmark both.
Quite apart from the actual IO efficiency, there are implications for the way the application code tracks when it needs to do the I/O, and does data processing/generation, that can sometimes impact performance quite dramatically.
1) mmap() is not reading sequentially.
2) mmap() has to fetch from the disk itself same as read() does
3) The mapped area is not sequential – so no DMA (?).
So mmap() should actually be slower than read() from a file? Which of my assumptions above are wrong?
1) is wrong…
mmap() assigns a region of virtual address space corresponding to file content… whenever a page in that address space is accessed, physical RAM is found to back the virtual addresses and the corresponding disk content is faulted into that RAM. So, the order in which reads are done from the disk matches the order of access. It’s a “lazy” I/O mechanism. If, for example, you needed to index into a huge hash table that was to be read from disk, then
mmaping the file and starting to do access means the disk I/O is not done sequentially and may therefore result in longer elapsed time until the entire file is read into memory, but while that’s happening lookups are succeeding and dependent work can be undertaken, and if parts of the file are never actually needed they’re not read (allow for the granularity of disk and memory pages, and that even when using memory mapping many OSes allow you to specify some performance-enhancing / memory-efficiency tips about your planned access patterns so they can proactively read ahead or release memory more aggressively knowing you’re unlikely to return to it).
2) absolutely true
3) “The mapped area is not sequential” is vague. Memory mapped regions are “contiguous” (sequential) in virtual address space. We’ve discussed disk I/O being sequential above. Or, are you thinking of something else? Anyway, while pages are being faulted in, they may indeed be transferred using DMA.
Further, there are other reasons why memory mapping may outperform usual I/O:
- there’s less copying:
- often OS & library level routines pass data through one or more buffers before it reaches an application-specified buffer, the applicaton then dynamically allocates storage, then copies from the I/O buffer to that storage so the data’s usable after the file reading completes
- memory mapping allows (but doesn’t force) in-place usage (you can just record a pointer and possibly length)
- continuing to access data in-place risks increased swapping later: the file/memory-map could be more verbose than data structures into which it could be parsed, so access patterns on data therein could have more delays to fault in more memory pages
- memory mapping can simplify the application’s parsing job by letting the application treat the entire file content as accessible, rather than worrying about when to read another buffer full
- the application defers more to the OS’s wisdom re number of pages that are in physical RAM at any single point in time, effectively sharing a direct-access disk cache with the application
- as well-wisher comments below, “using memory mapping you typically use less system calls”
- if multiple processes are accessing the same file, they should be able to share the physical backing pages
The are also reasons why
mmap may be slower – do read Linus Torvald’s post here which says of
…page table games along with the fault (and even just TLB miss)
overhead is easily more than the cost of copying a page in a nice
And from another of his posts:
quite noticeable setup and teardown costs. And I mean noticeable. It’s things like following the page tables to unmap everything cleanly. It’s the book-keeping for maintaining a list of all the mappings. It’s The TLB flush needed after unmapping stuff.
page faulting is expensive. That’s how the mapping gets populated, and it’s quite slow.
FWIW, the last time this arose for me at work, memory mapped input was 80% faster than
fread et al for reading binary database records into a proprietary database, on 64 bit Linux with ~170GB files.
mmap()can share between process.
- DMA will be used whenever possible. DMA does not require contiguous memory — many high end cards support scatter-gather DMA.
- The memory area may be shared with kernel block cache if possible. So there is lessor copying.
- Memory for
mmapis allocated by kernel, it is always aligned.
“Faster” in absolute terms doesn’t exist. You’d have to specify constraints and circumstances.
mmap() is not reading sequentially.
what makes you think that? If you really access the mapped memory sequentially, the system will usually fetch the pages in that order.
mmap() has to fetch from the disk itself same as read() does
sure, but the OS determines the time and buffer size
The mapped area is not sequential – so no DMA (?).
mmap helps with is that there is no extra user space buffer involved, the “read” takes place there where the OS kernel sees fit and in chunks that can be optimized. This may be an advantage in speed, but first of all this is just an interface that is easier to use.
If you want to know about speed for a particular setup (hardware, OS, use pattern) you’d have to measure.