Properly document incremental dumps

doc/snapshot.texi

@@ -5,7 +5,7 @@
 @c published by the Free Software Foundation.
 
   A @dfn{snapshot file} (or @dfn{directory file}) is created during
-incremental backups (@pxref{incremental and listed-incremental}).  It
+incremental backups (@pxref{Incremental Dumps}).  It
 contains the status of the filesystem at the time of the dump and is
 used to determine which files were modified since the last backup.
 

doc/tar.texi

@@ -240,8 +240,7 @@ Coping with Scarce Resources
 Performing Backups and Restoring Files
 
 * Full Dumps::                  Using @command{tar} to Perform Full Dumps
-* Inc Dumps::                   Using @command{tar} to Perform Incremental Dumps
-* incremental and listed-incremental::  The Incremental Options
+* Incremental Dumps::           Using @command{tar} to Perform Incremental Dumps
 * Backup Levels::               Levels of Backups
 * Backup Parameters::           Setting Parameters for Backups and Restoration
 * Scripted Backups::            Using the Backup Scripts
@@ -2203,7 +2202,7 @@ archive, or if they do not already exist in the archive.
 
 Normally when creating an archive, @command{tar} strips an initial
 @samp{/} from member names.  This option disables that behavior.
-@FIXME-xref{}
+@xref{absolute}.
 
 @item --after-date
 
@@ -4592,48 +4591,52 @@ http://www.cs.umd.edu/projects/amanda/amanda.html
 ftp://ftp.cs.umd.edu/pub/amanda
 @end smallexample
 
-@ifclear PUBLISH
+@FIXME{
 
 Here is a possible plan for a future documentation about the backuping
 scripts which are provided within the @GNUTAR{}
 distribution.
 
-@smallexample
-.* dumps
-.  + what are dumps
-
-.  + different levels of dumps
-.  - full dump = dump everything
-.  - level 1, level 2 dumps etc, -
-        A level n dump dumps everything changed since the last level
-        n-1 dump (?)
-
-.  + how to use scripts for dumps  (ie, the concept)
-.  - scripts to run after editing backup specs (details)
-
-.  + Backup Specs, what is it.
-.  - how to customize
-.  - actual text of script  [/sp/dump/backup-specs]
-
-.  + Problems
-.  - rsh doesn't work
-.  - rtape isn't installed
-.  - (others?)
-
-.  + the --incremental option of tar
-
-.  + tapes
-.  - write protection
-.  - types of media
-.   : different sizes and types, useful for different things
-.  - files and tape marks
+@itemize @bullet
+@item dumps
+ @itemize @minus
+ @item what are dumps
+ @item different levels of dumps
+  @itemize +
+  @item full dump = dump everything
+  @item level 1, level 2 dumps etc
+        A level @var{n} dump dumps everything changed since the last level
+        @var{n}-1 dump (?)
+  @end itemize
+ @item how to use scripts for dumps  (ie, the concept)
+  @itemize +
+  @item scripts to run after editing backup specs (details)
+  @end itemize
+ @item Backup Specs, what is it.
+  @itemize +
+  @item how to customize
+  @item actual text of script  [/sp/dump/backup-specs]
+  @end itemize
+ @item Problems
+  @itemize +
+   @item rsh doesn't work
+   @item rtape isn't installed
+   @item (others?)
+  @end itemize
+ @item the @option{--incremental} option of tar
+ @item tapes
+  @itemize +
+  @item write protection
+  @item types of media, different sizes and types, useful for different things
+  @item files and tape marks
      one tape mark between files, two at end.
-.  - positioning the tape
+  @item positioning the tape
      MT writes two at end of write,
-       backspaces over one when writing again.
-@end smallexample
-
-@end ifclear
+     backspaces over one when writing again.
+  @end itemize
+ @end itemize
+@end itemize
+}
 
 This chapter documents both the provided shell scripts and @command{tar}
 options which are more specific to usage as a backup tool.
@@ -4646,8 +4649,7 @@ called @dfn{dumps}.
 
 @menu
 * Full Dumps::                  Using @command{tar} to Perform Full Dumps
-* Inc Dumps::                   Using @command{tar} to Perform Incremental Dumps
-* incremental and listed-incremental::  The Incremental Options
+* Incremental Dumps::           Using @command{tar} to Perform Incremental Dumps
 * Backup Levels::               Levels of Backups
 * Backup Parameters::           Setting Parameters for Backups and Restoration
 * Scripted Backups::            Using the Backup Scripts
@@ -4681,9 +4683,10 @@ If you want to dump each filesystem separately you will need to use
 the @value{op-one-file-system} option to prevent @command{tar} from crossing
 filesystem boundaries when storing (sub)directories.
 
-The @value{op-incremental} option is not needed, since this is a complete
-copy of everything in the filesystem, and a full restore from this
-backup would only be done onto a completely empty disk.
+The @value{op-incremental} (@FIXME-pxref{}) option is not needed,
+since this is a complete copy of everything in the filesystem, and a
+full restore from this backup would only be done onto a completely
+empty disk. 
 
 Unless you are in a hurry, and trust the @command{tar} program (and your
 tapes), it is a good idea to use the @value{op-verify} option, to make
@@ -4692,199 +4695,183 @@ also detect cases where the file was modified while (or just after)
 it was being archived.  Not all media (notably cartridge tapes) are
 capable of being verified, unfortunately.
 
-@value{op-listed-incremental} take a file name argument always.  If the
-file doesn't exist, run a level zero dump, creating the file.  If the
-file exists, uses that file to see what has changed.
-
-@value{op-incremental} @FIXME{look it up}
-
-@value{op-incremental} handle old @acronym{GNU}-format incremental backup.
+@node Incremental Dumps
+@section Using @command{tar} to Perform Incremental Dumps
 
-This option should only be used when creating an incremental backup of
-a filesystem.  When the @value{op-incremental} option is used, @command{tar}
-writes, at the beginning of the archive, an entry for each of the
-directories that will be operated on.  The entry for a directory
-includes a list of all the files in the directory at the time the
-dump was done, and a flag for each file indicating whether the file
-is going to be put in the archive.  This information is used when
-doing a complete incremental restore.
+@dfn{Incremental backup} is a special form of @GNUTAR{} archive that
+stores additional metadata so that exact state of the filesystem
+can be restored when extracting the archive. 
 
-Note that this option causes @command{tar} to create a non-standard
-archive that may not be readable by non-@acronym{GNU} versions of the
-@command{tar} program.
+@GNUTAR{} currently offers two options for handling incremental
+backups: @value{op-listed-incremental} and @value{op-incremental}.  
 
-The @value{op-incremental} option means the archive is an incremental
-backup.  Its meaning depends on the command that it modifies.
+The option @option{--listed-incremental} instructs tar to operate on
+an incremental archive with additional metadata stored in a standalone
+file, called @dfn{snapshot file}.  The purpose of this file is to help
+determine what files have been changed, added or deleted since the
+last backup, so that the next incremental backup will contain only
+modified files.  The name of the snapshot file is given as an argument
+to the option:
 
-If the @value{op-incremental} option is used with @value{op-list},
-@command{tar} will list, for each directory in the archive, the list
-of files in that directory at the time the archive was created.  This
-information is put out in a format that is not easy for humans to
-read, but which is unambiguous for a program: each file name is
-preceded by either a @samp{Y} if the file is present in the archive,
-an @samp{N} if the file is not included in the archive, or a @samp{D}
-if the file is a directory (and is included in the archive).  Each
-file name is terminated by a null character.  The last file is
-followed by an additional null and a newline to indicate the end of
-the data.
-
-If the @value{op-incremental} option is used with @value{op-extract}, then
-when the entry for a directory is found, all files that currently
-exist in that directory but are not listed in the archive @emph{are
-deleted from the directory}.
-
-This behavior is convenient when you are restoring a damaged file
-system from a succession of incremental backups: it restores the
-entire state of the file system to that which obtained when the backup
-was made.  If you don't use @value{op-incremental}, the file system will
-probably fill up with files that shouldn't exist any more.
-
-@value{op-listed-incremental} handle new @acronym{GNU}-format
-incremental backup.  This option handles new @acronym{GNU}-format
-incremental backup.  It has much the same effect as
-@value{op-incremental}, but also the time when the dump is done and
-the list of directories dumped is written to the given
-@var{file}.  When restoring, only files newer than the saved time are
-restored, and the directory list is used to speed up operations.
-
-@value{op-listed-incremental} acts like @value{op-incremental}, but when
-used in conjunction with @value{op-create} will also cause @command{tar} to
-use the file @var{file}, which contains information about the state
-of the filesystem at the time of the last backup, to decide which
-files to include in the archive being created.  That file will then
-be updated by @command{tar}.  If the file @var{file} does not exist when
-this option is specified, @command{tar} will create it, and include all
-appropriate files in the archive.
-
-The file, which is archive independent, contains the date it was last
-modified and a list of devices, inode numbers and directory names.
-@command{tar} will archive files with newer mod dates or inode change
-times, and directories with an unchanged inode number and device but
-a changed directory name.  The file is updated after the files to
-be archived are determined, but before the new archive is actually
-created.
-
-@node Inc Dumps
-@section Using @command{tar} to Perform Incremental Dumps
-@UNREVISED
+@table @option
+@item --listed-incremental=@var{file}
+@itemx -g @var{file}
+  Handle incremental backups with snapshot data in @var{file}.
+@end table
 
-@cindex incremental dumps
-@cindex dumps, incremental
+To create an incremental backup, you would use
+@option{--listed-incremental} together with @option{--create}
+(@pxref{create}).  For example:
 
-Performing incremental dumps is similar to performing full dumps,
-although a few more options will usually be needed.
+@smallexample
+$ @kbd{tar --create \
+           --file=archive.1.tar \
+           --listed-incremental=/var/log/usr.snar \
+           /usr}
+@end smallexample
 
-A standard scheme is to do a @emph{monthly} (full) dump once a month,
-a @emph{weekly} dump once a week of everything since the last monthly
-and a @emph{daily} every day of everything since the last (weekly or
-monthly) dump.
+This will create in @file{archive.1.tar} an incremental backup of
+@file{/usr} filesystem, storing additional metadata in the file
+@file{/var/log/usr.snar}.  If this file does not exist, it will be
+created.  The created archive will then be called @dfn{level 0 backup}
+(see the next section for more info on backup levels).
 
-Here is a sample script to dump the directory hierarchies @samp{/usr}
-and @samp{/var}.
+Otherwise, if the file @file{/var/log/usr.snar} exists, it is used to
+determine the modified files.  In this case only these files will be
+stored in the archive.  Suppose, for example, that after running the
+above command, you delete file @file{/usr/doc/old} and create
+directory @file{/usr/local/db} with the following contents:
 
 @smallexample
-#!  /bin/sh
-tar --create \
- --blocking-factor=126 \
- --file=/dev/rmt/0 \
- --label="`hostname` /usr /var `date +%Y-%m-%d`" \
- --listed-incremental=/var/log/usr-var.snar \
- --verbose \
- /usr /var
+$ @kbd{ls /usr/local/db}
+/usr/local/db/data
+/usr/local/db/index
 @end smallexample
 
-This script uses the file @file{/var/log/usr-var.snar} as a snapshot to
-store information about the previous tar dump.
+Some time later you create another incremental backup. You will
+then see:
 
-The blocking factor 126 is an attempt to make the tape drive stream.
-Some tape devices cannot handle 64 kB blocks or larger, and require the
-block size to be a multiple of 1 kB; for these devices, 126 is the
-largest blocking factor that can be used.
+@smallexample
+$ @kbd{tar --create \
+           --file=archive.2.tar \
+           --listed-incremental=/var/log/usr.snar \
+           /usr}
+tar: usr/local/db: Directory is new
+usr/local/db/
+usr/local/db/data
+usr/local/db/index
+@end smallexample
 
-@node incremental and listed-incremental
-@section The Incremental Options
-@UNREVISED
+@noindent
+The created archive @file{archive.2.tar} will contain only these
+three members.  This archive is called @dfn{level 1 backup}.  Notice,
+that @file{/var/log/usr.snar} will be updated with the new data, so if
+you plan to create more @samp{level 1} backups, it is necessary to
+create a working copy of the snapshot file before running
+@command{tar}.  The above example will then be modified as follows:
 
-@value{op-incremental} is used in conjunction with @value{op-create},
-@value{op-extract} or @value{op-list} when backing up and restoring file
-systems.  An archive cannot be extracted or listed with the
-@value{op-incremental} option specified unless it was created with the
-option specified.  This option should only be used by a script, not by
-the user, and is usually disregarded in favor of
-@value{op-listed-incremental}, which is described below.
-
-@value{op-incremental} in conjunction with @value{op-create} causes
-@command{tar} to write, at the beginning of the archive, an entry for
-each of the directories that will be archived.  The entry for a
-directory includes a list of all the files in the directory at the
-time the archive was created and a flag for each file indicating
-whether or not the file is going to be put in the archive.
-
-Note that this option causes @command{tar} to create a non-standard
-archive that may not be readable by non-@acronym{GNU} versions of the
-@command{tar} program.
-
-@value{op-incremental} in conjunction with @value{op-extract} causes
-@command{tar} to read the lists of directory contents previously stored
-in the archive, @emph{delete} files in the file system that did not
-exist in their directories when the archive was created, and then
-extract the files in the archive.
-
-This behavior is convenient when restoring a damaged file system from
-a succession of incremental backups: it restores the entire state of
-the file system to that which obtained when the backup was made.  If
-@value{op-incremental} isn't specified, the file system will probably
-fill up with files that shouldn't exist any more.
-
-@value{op-incremental} in conjunction with @value{op-list} and two
-@value{op-verbose} options causes @command{tar} to print, for each
-directory in the archive, the list of files in that directory at the
-time the archive was created.  This information is put out in a format
-that is not easy for humans to read, but which is unambiguous for a
-program: each file name is preceded by either a @samp{Y} if the file
-is present in the archive, an @samp{N} if the file is not included in
-the archive, or a @samp{D} if the file is a directory (and is included
-in the archive).  Each file name is terminated by a newline character.
-The last file is followed by an additional newline to
-indicate the end of the data. 
-
-@value{op-listed-incremental} acts like @value{op-incremental}, but when
-used in conjunction with @value{op-create} will also cause @command{tar}
-to use the file @var{snapshot-file}, which contains information about
-the state of the file system at the time of the last backup, to decide
-which files to include in the archive being created.  That file will
-then be updated by @command{tar}.  If the file @var{file} does not exist
-when this option is specified, @command{tar} will create it, and include
-all appropriate files in the archive.
-
-The file @var{file}, which is archive independent, contains the date
-it was last modified and a list of devices, inode numbers and
-directory names.  @command{tar} will archive files with newer mod dates
-or inode change times, and directories with an unchanged inode number
-and device but a changed directory name.  The file is updated after
-the files to be archived are determined, but before the new archive is
-actually created.@FIXME-xref{to the description of the file format}.
+@smallexample
+$ @kbd{cp /var/log/usr.snar /var/log/usr.snar-1}
+$ @kbd{tar --create \
+           --file=archive.2.tar \
+           --listed-incremental=/var/log/usr.snar-1 \
+           /usr}
+@end smallexample
 
 Incremental dumps depend crucially on time stamps, so the results are
 unreliable if you modify a file's time stamps during dumping (e.g.,
 with the @option{--atime-preserve} option), or if you set the clock
 backwards.
 
-Despite it should be obvious that a device has a non-volatile value, NFS
-devices have non-dependable values when an automounter gets in the picture.
-This led to a great deal of spurious redumping in incremental dumps,
-so it is somewhat useless to compare two NFS devices numbers over time.
-So @command{tar} now considers all NFS devices as being equal when it comes
-to comparing directories; this is fairly gross, but there does not seem
+Metadata stored in snapshot files include device numbers, which,
+obviously is supposed to be a non-volatile value.  However, it turns
+out that NFS devices have non-dependable values when an automounter
+gets in the picture.  This can lead to a great deal of spurious
+redumping in incremental dumps, so it is somewhat useless to compare
+two NFS devices numbers over time.  The solution implemented currently
+is to considers all NFS devices as being equal when it comes to
+comparing directories; this is fairly gross, but there does not seem 
 to be a better way to go.
 
-@command{tar} doesn't access @var{snapshot-file} when
-@value{op-extract} or @value{op-list} are specified, but the
-@value{op-listed-incremental} option must still be given.  A
-placeholder @var{snapshot-file} can be specified, e.g.,
-@file{/dev/null}.
+Note that incremental archives use @command{tar} extensions and may
+not be readable by non-@acronym{GNU} versions of the @command{tar} program.
+
+To extract from the incremental dumps, use
+@option{--listed-incremental} together with @option{--extract}
+option (@pxref{extracting files}). In this case, @command{tar} does
+not need to access snapshot file, since all the data necessary for
+extraction are stored in the archive itself. So, when extracting, you
+can give whatever argument to @option{--listed-incremental}, the usual
+practice is to use @option{--listed-incremental=/dev/null}.
+Alternatively, you can use @option{--incremental}, which needs no
+arguments.  In general, @option{--incremental} (@option{-G}) can be
+used as a shortcut for @option{--listed-incremental} when listing or
+extracting incremental backups (for more information, regarding this
+option, @pxref{incremental-op}).
+
+When extracting from the incremental backup @GNUTAR{} attempts to
+restore the exact state the file system had when the archive was
+created.  In particular, it will @emph{delete} those files in the file
+system that did not exist in their directories when the archive was
+created.  If you have created several levels of incremental files,
+then in order to restore the exact contents the file system  had when
+the last level was created, you will need to restore from all backups
+in turn. Continuing our example, to restore the state of @file{/usr}
+file system, one would do@footnote{Notice, that since both archives
+were created withouth @option{-P} option (@pxref{absolute}), these
+commands should be run from the root filesystem.}: 
+
+@smallexample
+$ @kbd{tar --extract \
+           --listed-incremental=/dev/null \
+           --file archive.1.tar}
+$ @kbd{tar --extract \
+           --listed-incremental=/dev/null \
+           --file archive.2.tar}
+@end smallexample
+
+To list the contents of an incremental archive, use @option{--list}
+(@pxref{list}), as usual. To obtain more information about the
+archive, use @option{--listed-incremental} or @option{--incremental}
+combined with two @option{--verbose} options@footnote{Two
+@option{--verbose} options were selected to avoid breaking usual
+verbose listing output (@option{--list --verbose}) when using in
+scripts.
+
+Versions of @GNUTAR{} up to 1.15.1 used to dump verbatim binary
+contents of the DUMPDIR header (with terminating nulls) when
+@option{--incremental} or @option{--listed-incremental} option was
+given, no matter what the verbosity level.  This behavior, and,
+especially, the binary output it produced were considered incovenient
+and were changed in version 1.16}:
+
+@smallexample
+@kbd{tar --list --incremental --verbose --verbose archive.tar}
+@end smallexample
+
+This command will print, for each directory in the archive, the list
+of files in that directory at the time the archive was created.  This
+information is put out in a format which is both human-readable and
+unambiguous for a program: each file name is printed as
+
+@smallexample
+@var{x} @var{file}
+@end smallexample
 
-@FIXME{this section needs to be written}
+@noindent
+where @var{x} is a letter describing the status of the file: @samp{Y}
+if the file  is present in the archive, @samp{N} if the file is not
+included in the archive, or a @samp{D} if the file is a directory (and
+is included in the archive).@FIXME-xref{dumpdir format}.  Each such
+line is terminated by a newline character.  The last line is followed
+by an additional newline to indicate the end of the data.          
+
+@anchor{incremental-op}The option @option{--incremental} (@option{-G})
+gives the same behavior as @option{--listed-incremental} when used
+with @option{--list} and @option{--extract} options.  When used with
+@option{--create} option, it creates an incremental archive without
+creating snapshot file.  Thus, it is impossible to create several
+levels of incremental backups with @option{--incremental} option.
 
 @node Backup Levels
 @section Levels of Backups
@@ -4930,8 +4917,8 @@ their use in detail.
 designed to be used together.  While it is possible to restore files by
 hand from an archive which was created using a backup script, and to create
 an archive by hand which could then be extracted using the restore script,
-it is easier to use the scripts.  @value{xref-incremental}, and
-@value{xref-listed-incremental}, before making such an attempt.
+it is easier to use the scripts.  @value{xref-incremental}, before
+making such an attempt. 
 
 @node Backup Parameters
 @section Setting Parameters for Backups and Restoration
@@ -5442,8 +5429,8 @@ positioning.}
 system if they were not in the file system when the archive was made.
 @end quotation
 
-@value{xref-incremental}, and @value{ref-listed-incremental},
-for an explanation of how the script makes that determination.
+@value{xref-incremental}, for an explanation of how the script makes
+that determination. 
 
 @node Choosing
 @chapter Choosing Files and Names for @command{tar}
@@ -6087,7 +6074,7 @@ arguments.
 @strong{Please Note:} @value{op-after-date} and @value{op-newer-mtime}
 should not be used for incremental backups.  Some files (such as those
 in renamed directories) are not selected properly by these options.
-@xref{incremental and listed-incremental}.
+@xref{Incremental Dumps}.
 @end quotation
 
 @noindent

doc/value.texi

@@ -155,9 +155,9 @@
 @set pxref-ignore-zeros @pxref{Reading}
 
 @set op-incremental @option{--incremental} (@option{-G})
-@set ref-incremental @ref{Inc Dumps}
-@set xref-incremental @xref{Inc Dumps}
-@set pxref-incremental @pxref{Inc Dumps}
+@set ref-incremental @ref{Incremental Dumps}
+@set xref-incremental @xref{Incremental Dumps}
+@set pxref-incremental @pxref{Incremental Dumps}
 
 @set op-info-script @option{--info-script=@var{script-name}} (@option{--new-volume-script=@var{script-name}}, @option{-F @var{script-name}})
 @set ref-info-script @ref{Multi-Volume Archives}
@@ -190,9 +190,7 @@
 @set pxref-list @pxref{list}
 
 @set op-listed-incremental @option{--listed-incremental=@var{snapshot-file}} (@option{-g @var{snapshot-file}})
-@set ref-listed-incremental @ref{Inc Dumps}
-@set xref-listed-incremental @xref{Inc Dumps}
-@set pxref-listed-incremental @pxref{Inc Dumps}
+@c No special refs for this option, use {*ref-incremental} instead.
 
 @set op-mode @option{--mode=@var{permissions}}
 @set ref-mode @ref{Option Summary}