
                    RAMS
                 Jared Davis 


OVERVIEW

   RAMS is a comprehensive system for managing the build process for ACL2
   projects.  The user invokes RAMS using one of many "make" commands, and our
   intention is for RAMS to be as transparent as possible.

   To use RAMS, an ACL2 project need only create a simple Makefile that
   includes the following lines:

   MAKEDIR = ../../../make            # (where the "make" directory lives)
   NAME = :myproject                  # (what is this project/library named)
   LIBS = :lib1 :lib2 :lib3           # (what libraries are needed)
   BOOKS = book1 book2 book3          # (what books to certify)
   include $(MAKEDIR)/Makefile.top    # (invoke RAMS)

   RAMS then handles the rest, automatically calculating dependencies and
   setting up links to other libraries.  In addition, RAMS provides an
   interface to the powerful Overkill system for performing distributed builds
   of "provisional certificates", which can be useful for high speed regression
   testing.

   In the remainder of this document, we describe the RAMS system in more
   detail, explaining both how to structure your project for successful use
   with RAMS and highlighting the powerful features that RAMS provides.



PROJECT STRUCTURE: BOOKS AND PORTCULLIS FILES

   RAMS views each project as a collection of ACL2 books.  (In particular, the
   books which are enumerated in the Makefile's BOOKS line.)

   For every book foo, we expect exactly one file foo.lisp which contains the
   contents of that book.  We call these files "proper" lisp files.  Of course,
   to certify foo.lisp, we may need portcullis commands to lead us to a proper
   certification world, such as defpkg events.  These commands are stored in 
   "improper" files such as .acl2 or .lsp files.

   We use the same algorithm for certifying books as taken by ACL2's standard
   Makefile-generic.  That is, to certify foo.lisp, we first check to see if
   foo.acl2 exists.  If so, we expect that executing (ld "foo.acl2") will be
   sufficient to certify foo.lisp --- that is, we expect that foo.acl2 contains
   an appropriate (certify-book "foo" ...) command.

   Otherwise, if no foo.acl2 file exists, as a special convenience we check to 
   see if the file cert.acl2 exists.  If so, we expect that executing: 

    (ld "cert.acl2")
    (certify-book "foo" ?) 

   Will be sufficient to generate a certificate for foo.  In this way, many
   books can share the same portcullis commands, e.g., common defpkg events.
   You are encouraged to look at, e.g., the .acl2 files in the :bags and :lists
   libraries to see how we typically structure our projects.

   In general, we prefer to avoid individual .acl2 files when possible, and 
   when they are necessary we typically begin them with, (ld "cert.acl2").



MANAGING LINKS TO EXTERNAL LIBRARIES

   We prefer to use the ":dir" feature of include-book and ld whenever we want
   to access external libraries.  For example:

     (include-book "foo" :dir :lists)  ; <-- we like this
     (include-book "../lists/foo")     ; <-- we dislike this
   
   This system is convenient.  For example, suppose that you would like to try
   out an alternate implementation of lists.  Then, rather than having to change
   all of the "../lists/foo" in your project to "../lists2/foo", you can instead
   simply change what :lists points to once and for all.

   How do we control what :lists points to?  First of all, the "make" directory
   has a file called DIRS which simply lists the "default" locations of all of
   the libraries, relative to the "make" directory.  Individual projects can
   override these defaults by providing their own, local DIRS files.  So, in
   our example above, to switch to "lists2", all one needs to do is create a
   DIRS file with the line:

     :lists = ../lists2

   The DIRS files are processed by the RAMS system when "make" is initialized,
   and two files are kept updated with any changes:
 
    Makefile.acl2 

    This file is a list of (add-include-book-dir ...) commands that enable you
    to use ":dir :lists" and so forth in your include-book and ld commands.  We
    typically "ld" this file into cert.acl2 files so that they are available
    throughout our entire project.

    Makefile.dirs

    This file is a list of "ACL2_FOO_BOOKS = ..." lines, which are included by
    the RAMS Makefiles, so that the Makefiles themselves know where to look for
    other libraries.  All of our dependencies (see the next section) use these
    prefixes so that we can identify exactly which files we depend upon.

   Note that when you add new libraries to the CVS repository, you should also
   add their default location to the DIRS file in the "make" directory.  Then,
   your project will become instantly available to anyone who adds its name to
   their own LIBS line.



HOW DEPENDENCIES ARE HANDLED

   After RAMS has established the links to external libraries, it automatically
   computes the dependencies for each foo.cert in your project.  This is done
   by scanning your files for "ld" and "include-book" commands.

   Note that these are somewhat stupid "grep" based checks, so you should avoid
   using macros that include books and so forth which would confuse them.
   Also, do not write "ld" or "include-book" commands that span multiple lines,
   or leave them inside of multiline comments.  We consider these to be "user
   errors" because it is too hard to have our program figure them out.

   The dependency computation will result in two new files being generated:

    Makefile.deps

    The real meat of the dependency information is stored into Makefile.deps.
    This file is too coplicated to explain in detail, so we refer the curious
    reader to the comments at the top of Makefile-deps.bash in the "make"
    directory, which explain the process in detail.

    Importantly, each dependency line is written roughly as follows:

      $(ACL2_FOO_BOOKS)/foo.cert: $(ACL2_FOO_BOOKS)/foo.lisp ...

    By adding this prefix to foo.cert and foo.lisp, we ensure that all of our
    Makefile.deps files can be included together without name clashes.  As a
    result, our Makefile is "deeply aware" of the dependencies of foo.cert, and
    can even descend into other libraries to build the necessary files.


    Makefile.ldeps

    Unfortunately, "make" is not intelligent enough on its own to realize that
    "foo.cert" and $(ACL2_FOO_BOOKS)/foo.cert are the same file.  To address
    this problem, we also create a simple file called Makefile.ldeps, which
    simply contains the following sort of lines:

      foo.cert : $(ACL2_FOO_BOOKS)/foo.cert

    This allows make to do the right thing when the user types "make foo.cert",
    sparing us from typing "make `pwd`/foo.cert".


   The dependency calculation can take some time, but we prefer to run it at
   the beginning of every invocation of "make" so that you will be spared from
   needing to worry about dependencies.  If it truly bothers you, and you are
   sure that your dependencies are up to date, you can add SKIPDEPS=1 to your
   invocation of make, e.g., "make foo.cert SKIPDEPS=1".  If you do this, all
   bets are off as to the status of your dependencies!


   
MANUALLY ADDING DEPENDENCIES OR OTHER TARGETS

   Some situations are not handled by our automatic dependency computation,
   e.g., if you are generating .lisp or .lsp files on the fly, then you might
   need to explain how those files are to be generated, and so forth.  Our hope
   is that these situations are quite rare, so that the convoluted solution
   below does not pose too great a burden.

   We would like to have these dependencies added to Makefile.deps, so that
   they are "deeply known" throughout our projects.  But of course, the file
   Makefile.deps is automatically generated, so any changes you make to it will
   be clobbered.  Instead, we provide the following mechanism: if the file
   MOREDEPS exists, then its entire contents will be appended to the end of
   Makefile.deps when it is generated.  That way, you can manually add more
   dependencies when necessary.

   Note that these dependencies MUST use the $(ACL2_FOO_BOOKS) in order for
   other projects to properly understand which files you are talking about.  Of
   course, this introduces the same problem we addressed in Makefile.ldeps, so
   you will also want to create a MOREDEPS-LOCAL file, which will itself be
   appended to Makefile.ldeps at generation time, and which simply lists lines
   such as

     myfile: $(ACL2_FOO_BOOKS)/myfile

   Note also that your custom Make commands should never assume what directory
   they will begin in.  Instead, you should use "cd $(ACL2_FOO_BOOKS)" where
   FOO is your own name, before issuing such commands.



SELECTING ACL2 EXECUTABLES AND BOOKS

   The file Makefile.top in the Make directory specifies which ACL2 executable
   should be used to build your source files.  It also has a variable for the
   location of the ACL2 books.

   We prefer that if you are going to use a different ACL2 executable than our
   "current standard", you just change this file locally (and be sure to do a
   full "make clean").  However, it is also possible to control pass ACL2=...
   and ACL2_BOOKS=... into the Makefile as arguments when you call Make, or as
   shell variables.  However, this seems error prone.



PROVISIONAL CERTIFICATES AND THE OVERKILL SYSTEM

   RAMS also includes support for generating ".pcert" files -- provisional
   certificates -- for your books.  These are not full certificates, but they
   do not have any order dependencies so you can "provisionally certify" your
   books in any order, or even in parallel.

   The Overkill system is a simple way to instruct many machines to work on
   generating .pcert files for you in parallel.  You can drastically reduce the
   time it takes to discover errors by using many machines at once to work on
   your certification attempts.

   Type "make help-overkill" to see more detailed documentation about the
   Overkill system.



QUICK REFERENCE GUIDE FOR USING RAMS

   There are several Makefile targets that you might be interested in.  (These
   commands are valid in project directories, not the "make" directory)


     make (or, equivalently, "make all") 
     Build certificates (.cert) for each book in BOOKS.

     make foo.cert [bar.cert ...]
     Build the listed certificates (and their dependencies).


     make pcert 
     Build provisional certificates (.pcert) for each book in BOOKS

     make overkill
     Same as "make pcert", but on many hosts using Overkill.  See "make 
     help-overkill" for details.

     make foo.pcert [bar.pcert ...]
     Build the listed pcerts.


     make status
     Check with CVS to summarize the current status of each file.

     make clean
     Remove all generated files, e.g., .cert, .o, .out, etc.

     make deps
     Build Makefile.(acl2|dirs|deps|ldeps) files.  (usually automatic)


     (buggy!) make compile
     Build object files (.o) for each book in BOOKS

     (buggy!) make foo.o [bar.o ...]
     Build the specified object files.


     make help
     Displays this help message.

     make help-overkill
     Displays the Overkill help message.

