VM Regress - A Regression, Test and Benchmark suite
  • Introduction
  • Current State
  • ToDo
  • Manual
  • Download
  • Feedback
  • ChangeLog

    This test suite is now obsolete. While versions exist that build against modern kernels, it is unlikely you still want to use it. Some of the methods VM Regress used have been superseded by other methodologes or infrastrcture that is more suitable for analysing the VM with. I suggest instead that you use MM Tests which still includes some of the more useful bits from VM Regress.

    Introduction
    VM Regress is a regrssion, testing and benchmark tool for the Linux Kernel Virtual Memory (VM) system. The first significant problem with developing with the VM is that testing it is extremly tricky. Simulating scenarios reliably is difficult at best and impossible at worst. This leads to the tendancy to rely on stress testing to reveal flaws or to write quick custom tests that only one developer can avail of.

    The second problem is that emperical performance data is virtually impossible to obtain and doing a direct comparison between two VM's relies on subjective data. This makes is impossible to prove one VM is better over another or that a code change or feature introduction is beneficial.

    VM Regress is a tool that will ultimatly be able to perform reliable, reproducable testing on every aspect of the VM including providing empirical performance data. It relies on a combination of userspace tools and kernel modules to accuratly determine the current state of the kernel and to reliably reproduce tests. Testing with such accuracy from userspace is difficult and obtaining test results is similar to watching shadows on a wall.

    Modules are divided up into 4 section. core modules provide infrastructure for the whole suite. sense reveal information about the current running kernel. test test specific code paths and scenarios. bench provide benchmarks for different subsystems.

    VM Regress provides an interface via the proc system. A directory entry is created in /proc/vmregress and new entries are created there as modules are loaded. Generally a small help will be available by cating the entries but the manual or .c file should be consulted for extra information.

    Current State
    In summary, VM Regress covers the following. The manual should be consulted for more information.
  • Easy /proc interface
  • Print out the size and memory usage of some VM structs
  • Print out information about all memory zones
  • Print out information about the vmalloc address area
  • Print out full map of all vma's in use and what pages are present
  • Test physical page allocation/free interface
  • Test page faulting routines
  • Test for /proc interface
  • Bench read/write tests on anonymous/file backed mmap's
  • Produce maps of the test address space showing present/swapped pages
  • Show what pages were referenced over time during a test
  • Show what pages are present with respect to page age
  • Show what pages are present with respect to page reference count
  • Produce webpages for tests with process maps, vmstat output for any test
  • Provides some perl libraries for writing new customised tests
  • Automatically loads/unloads necessary kernel modules

    Here is the output of some sample tests that were generated with VM Regress. The noticable absentee is a fault validation test but it looks very similar to the allocation test. They are all run against 2.4.19-rmap14a.
  • __alloc_pages validation test
  • Anonymous mmap read benchmark
  • Anonymous mmap write benchmark
  • File backed mmap read benchmark
  • File backed mmap write benchmark

    Manual
    Consult the package documentation for detailed info. Each module's c file has a detailed comment at the top explaning it's function in heavy detail. Below are two links to the current README and a reasonably comprehensive manual. The basic manual will eventually be a whitepaper.

    Package readme file
    Basic manual and intro (pdf)

    Download
    Below is the tars for any release and the date of release. The version numbers are simply incremental with no special meaning attached. The series mark a significant milestone. The first series is BumbleBee. The
    ChangeLog for each release is at the end of the page. VM Regress is still not comprehensive by any description but it is beginning to reach the points where it is becoming useful
    Package Series Release Date
    vmregress-0.37.tar.gz BumbleBee December 1st, 2005
    vmregress-0.18.tar.gz BumbleBee November 15th, 2005
    vmregress-0.17.tar.gz BumbleBee September 20th, 2005
    vmregress-0.16.tar.gz BumbleBee August 8th, 2005
    vmregress-0.15.tar.gz BumbleBee June 2nd, 2005
    vmregress-0.14.tar.gz BumbleBee May 3rd, 2005
    vmregress-0.13.tar.gz BumbleBee March 7th, 2005
    vmregress-0.12.tar.gz BumbleBee February 20th, 2005
    vmregress-0.11.tar.gz BumbleBee January, 2005
    vmregress-0.10a.tar.gz BumbleBee Sunday, Aug 31st 2003
    vmregress-0.10.tar.gz BumbleBee Sunday, Jun 29th 2003
    vmregress-0.9.tar.gz BumbleBee Friday, April 18th 2003
    vmregress-0.8a.tar.gz BumbleBee Thursday, March 13th 2003
    vmregress-0.7.tar.gz BumbleBee Thursday, August 29th 2002
    vmregress-0.6.tar.gz BumbleBee Thursday, August 21st 2002
    vmregress-0.5.tar.gz BumbleBee Thursday, August 15th 2002
    vmregress-0.4.tar.gz BumbleBee Sunday, August 11th 2002


    Feedback
    Email mel@csn.ul.ie with feedback and comments.

    ChangeLog
    This is not particularly detailed unfortunatly.
    Version 0.15
      o Updated to latest available kernel version
      o Large number of changes to bench-mmap by Leonid Ananiev
          - The access time measurements surrounds now page touch module call
            rather than all loop with other pages and files touching and
            functions callings.
          - Address access pattern is generated inside main loop but not read
            from huge external file that creates additional memory load.
          - Lognormal and Zipfian random number generator may be used now to
            create access addresses.
          - More statistic data is collected in smaller array volume since
            average and max access time is collected for each page rank and for each
            second interval only. Access time and vmstat data outputs are synchronized.
          - More detailed virtual memory statistic are collected from /proc/vmstat
            rather than from vmstat command output.
    
    Version 0.13
    ------------
      o Updated for 2.6.12-rc3
      o Number of new benchmarks and modules added, all with docs
    
    Version 0.10a
    -------------
      o Updated to support 2.6.0-test4
    
    Version 0.10
    ------------
      o Supports up to 2.5.73
      o Fixed up usage of wrong type for IRQ flags
      o Added patch and module for counting order of page allocations
    
    Version 0.9
    -----------
      o Moved all module initialise code to a global init.c that is #included
      o Moved all read/write proc code to a global proc.c that is #included
      o Fixed the testproc module so that it wouldn't crash with pages > 10
      o Taught the kvirtual module a lot about what is in the linear address space
      o Removed a lot of common code that was in modules
      o Perl scripts now use kernel modules to collect most data
      o Perl scripts use as much common common code as possible
      o Added perl library for the easier creation of reports
      o Miscalculation in pagetable.o meant that mmap benchmarks were unreliable
    
    Version 0.8a
    ------------
      o Minor bug fixes in the core
      o OSDL based merging
        - Fixed the extract_structs.pl script to ensure its a struct been extracted
        - Move the creation of internal.h from Makefiles to the configure script
        - Use configure script to apply kernel patch if requested
      o Read kernel release version directly from Kernel makefile
      o Automatically generate makefiles depending on kernel version from configure
      o Teach extract_structs.pl to identify a struct that is typedef'd
      o vmr_mmzone.h has been expanded to map between different struct and field
        names between kernel versions. Not many differences thankfully
    
    Version 0.7
    -----------
      o Updated bench_mapanon.pl to perform read/write tests
      o Adapted mapanon.o and changed to mmap.o so it can map file descriptors
      o Adapted bench_mapanon.pl to bench_mmap.pl to be a generate mmap benchmark
      o Told benchmark to preserve sampling data
      o Time.pm exports new timing functions
      o mapanon.o changed to mmap.o, handles files or anonymous memory
      o Added graph to show page age vs page presence
      o Added graph to show reference pattern
      o Added replot.pl for easy replotting of time data
      o Fixed access permissions to alloc and fault tests
      o Removed stupid deadlock with alloc and fault modules
      o Various perl lib updates
      o Will now compile against late 2.5.x kernels (untested)
      o Automatically load and unload kernel modules
    
    Version 0.6
    -----------
      o Allow multiple instances of tests to run. Only one will print to proc
      o pagemap.o module will dump out address space with pages swapped/present
      o mapanon.o benchmark, creates and references mmaped areas so that a script
        can simulate program behaviour and see what the process space looks like
        after
      o Created various benchmark perl scripts.
      o Created various support perl modules for running tests in bin/lib/VMR
      o Print out kernel messages
      o Moved the pagemap decode perl routines to a library
      o Fixed CONFIG_HIGHMEM compile error
      o Fixed spinlock redefine errors
      o Fixed use of KERNEL_VERSION macro
      o Fixed various possible deadlocks
    
    Version 0.5
    -----------
      o Added a module kvirtual for printing out the vmalloc address space
      o Allowed proc buffers to grow rather than disabling them
      o Can now pass a pointer to user data into page table walk functions
      o Proc buffers are now stored in the vmr_desc_t struct
      o Proc buffers can be grown
      o Process maps can be printed out to view present/swapped pages
      o Created plot_map.pl which will plot a page present/swapped graph
      o set plot_map.pl to output html pages upon request
      o Updated kernel patches to 2.4.20pre2 . Should apply to 2.5.x
    
    Version 0.4
    -----------
      o Updated alloc.c to take parameter at module load to use GFP_KERNEL
      o Fixed pagetable walk
      o Provided callback interface for walking pagetables (pagetable.o)
      o Added ability to time tests with some measure of accuracy
      o Add zone module to print out all zone information (zone.o)
    
    Version 0.3
    -----------
      o Added kernel patch
      o Test mmaps and page faults (fault.o)
      o Centeralised pringing macros
      o Makefile workarounds for UML quirks
    
    Version 0.2
    -----------
      o Full physical allocation test (alloc.o)
      o Works with 2.5.27
      o Conditionally uses rmap if available
      o Added configure script
      o Alloc test will schedule if necessary
    
    Version 0.1
    -----------
      o Initial release
      o Proc test               (testproc.o)
      o Display struct sizes    (sizes.o)
      o Basic allocation test   (alloc_fast.o)
      o Initial template module (template.o)
      o Makefile