Windows memory mapped file io

To get the best possible performance, we are hoping that we can hint to the VM system to fetch the next image line possibly causing a page fault while we are processing data in the current one. This would parallelise our processing and the page faults. There does not seem to be an obvious way to do this on Windows! Is there a way of asynchronously touching a page, triggering a page fault without waiting for the page to become available? The right long-term solution is to store our data a different way e.

We need to keep the memory-mapped approach as well at the moment. I do not think you can hint VM system but you can pre-fetch data just by loading next line data while you process previous. You can make this parallel to processing. You should probably fetch more than 1 ahead since processing is most likely much faster then reading from file.

This actually quite nicely fits into producer-consumer pattern. Make image reader and data processor run on separate threads and use some sort of blocking collection like C BlockingCollection with capacity limit to pass data from reader to processor.

Even though this is somewhat oldish topic already, I give a practical example of one implementation, which should be used as a design reference only if you understand the big picture well enough.

We had a challenge in one "big data" telco -application, where we new better than virtual memory manager what pages should be paged in from sparse "huge files". What we did was that we had a dedicated thread "inmemadvisor" to receive "need" requests, which each had a priority. This thread maintains the prioritized list of requests and submits the memory read "hints" to pool of threads, where largest number of threads handles high priority requests and smallest part of the pool handles lowest priority you probably get the idea.

Then we have parameters to control the number of threads in the pool and some other nifty details. So, in short: This is something that one rather would not do, but on the other hand, these are the things wich make programming a positive challenge ;- Btw: Our implementation beats Linux madvice in performance with our application, but is not as generic as it is. Stack Overflow for Teams — Collaborate and share knowledge with a private group. MMFs get to be useful when you write, it gets eventually flushed to disk without you explicitly having to tell it to do so.

And they are useful to process lots of data, much more than can fit in the 2GB virtual memory space. Hans Passant. Marked as answer by nobugz Friday, October 2, AM. MMF is also convenient for inter process communication scenarios. Using it for file IO is only useful in certain scenarios. For reading a file it is essentially useless because the disk is the big bottleneck. Saturday, September 26, AM. Please remember to mark the replies as answers if they help and unmark them if they provide no help.

If you have any feedback, please tell us. Monday, September 28, AM. For a Kbytes of buffer what can you expect And still, you wont break a leg To get the max throughput of this comparasion: - make the buffer bigger - go with multithreaded comparing. Dev s r'us. Hello vips How are you? Hi Thanks for your example, I've been looking for mmap like capabilities in Windows. I'd think that the point of memory mapping a file would be to access it as memory - ie. My vote of 4 manishtera 5-Aug Nice article Frank B Jun A little polish and it could be excellent.

Thanks for publishing, and look forward to reading more from you. Re: Nice article manjunath. Thank you for your comments. Hi - I have a few queries about some of the types you've used in your code use of which makes it less understandable than it could be, IMO : Why the typedefs for int and unsigned int - where does the 'q' qint, quint come from?

I can understand typedefing bit ints - again, I have trouble understanding where the prefix, 's', comes from. But it's sint although its accessor function returns a value of type suint The sstring typedef You use rstring in the article but it's not in the code?

Corollary: In header files, don't write namespace-level using directives or using declarations; instead, explicitly namespace-qualify all names. The second rule follows from the first, because headers can never know what other header includes might appear after them.

One final thing - you use a named file mapping in your code. I suspect you would be better off with an unnamed mapping or allowing the user to name the mapping , as there is potential for the mapping name to be duplicated by someone else on your system, leading to unexpected errors. Re: A Brief Code Review manjunath. First of all I thank you for the detailed code review. I am sorry to not have followed strict coding convention, I will carefully consider all your inputs in all my future posting and code revisions.

As you have pointed out qint and quint may not be necessary, I was only influenced by the coding convention that we follow here inside the company 2.