| 1 | .\" Copyright (c) 2003-2009 Tim Kientzle |
| 2 | .\" All rights reserved. |
| 3 | .\" |
| 4 | .\" Redistribution and use in source and binary forms, with or without |
| 5 | .\" modification, are permitted provided that the following conditions |
| 6 | .\" are met: |
| 7 | .\" 1. Redistributions of source code must retain the above copyright |
| 8 | .\" notice, this list of conditions and the following disclaimer. |
| 9 | .\" 2. Redistributions in binary form must reproduce the above copyright |
| 10 | .\" notice, this list of conditions and the following disclaimer in the |
| 11 | .\" documentation and/or other materials provided with the distribution. |
| 12 | .\" |
| 13 | .\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND |
| 14 | .\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE |
| 15 | .\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE |
| 16 | .\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE |
| 17 | .\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL |
| 18 | .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS |
| 19 | .\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) |
| 20 | .\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT |
| 21 | .\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY |
| 22 | .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF |
| 23 | .\" SUCH DAMAGE. |
| 24 | .\" |
| 25 | .\" $FreeBSD: head/lib/libarchive/tar.5 201077 2009-12-28 01:50:23Z kientzle $ |
| 26 | .\" |
| 27 | .Dd December 27, 2009 |
| 28 | .Dt tar 5 |
| 29 | .Os |
| 30 | .Sh NAME |
| 31 | .Nm tar |
| 32 | .Nd format of tape archive files |
| 33 | .Sh DESCRIPTION |
| 34 | The |
| 35 | .Nm |
| 36 | archive format collects any number of files, directories, and other |
| 37 | file system objects (symbolic links, device nodes, etc.) into a single |
| 38 | stream of bytes. |
| 39 | The format was originally designed to be used with |
| 40 | tape drives that operate with fixed-size blocks, but is widely used as |
| 41 | a general packaging mechanism. |
| 42 | .Ss General Format |
| 43 | A |
| 44 | .Nm |
| 45 | archive consists of a series of 512-byte records. |
| 46 | Each file system object requires a header record which stores basic metadata |
| 47 | (pathname, owner, permissions, etc.) and zero or more records containing any |
| 48 | file data. |
| 49 | The end of the archive is indicated by two records consisting |
| 50 | entirely of zero bytes. |
| 51 | .Pp |
| 52 | For compatibility with tape drives that use fixed block sizes, |
| 53 | programs that read or write tar files always read or write a fixed |
| 54 | number of records with each I/O operation. |
| 55 | These |
| 56 | .Dq blocks |
| 57 | are always a multiple of the record size. |
| 58 | The maximum block size supported by early |
| 59 | implementations was 10240 bytes or 20 records. |
| 60 | This is still the default for most implementations |
| 61 | although block sizes of 1MiB (2048 records) or larger are |
| 62 | commonly used with modern high-speed tape drives. |
| 63 | (Note: the terms |
| 64 | .Dq block |
| 65 | and |
| 66 | .Dq record |
| 67 | here are not entirely standard; this document follows the |
| 68 | convention established by John Gilmore in documenting |
| 69 | .Nm pdtar . ) |
| 70 | .Ss Old-Style Archive Format |
| 71 | The original tar archive format has been extended many times to |
| 72 | include additional information that various implementors found |
| 73 | necessary. |
| 74 | This section describes the variant implemented by the tar command |
| 75 | included in |
| 76 | .At v7 , |
| 77 | which seems to be the earliest widely-used version of the tar program. |
| 78 | .Pp |
| 79 | The header record for an old-style |
| 80 | .Nm |
| 81 | archive consists of the following: |
| 82 | .Bd -literal -offset indent |
| 83 | struct header_old_tar { |
| 84 | char name[100]; |
| 85 | char mode[8]; |
| 86 | char uid[8]; |
| 87 | char gid[8]; |
| 88 | char size[12]; |
| 89 | char mtime[12]; |
| 90 | char checksum[8]; |
| 91 | char linkflag[1]; |
| 92 | char linkname[100]; |
| 93 | char pad[255]; |
| 94 | }; |
| 95 | .Ed |
| 96 | All unused bytes in the header record are filled with nulls. |
| 97 | .Bl -tag -width indent |
| 98 | .It Va name |
| 99 | Pathname, stored as a null-terminated string. |
| 100 | Early tar implementations only stored regular files (including |
| 101 | hardlinks to those files). |
| 102 | One common early convention used a trailing "/" character to indicate |
| 103 | a directory name, allowing directory permissions and owner information |
| 104 | to be archived and restored. |
| 105 | .It Va mode |
| 106 | File mode, stored as an octal number in ASCII. |
| 107 | .It Va uid , Va gid |
| 108 | User id and group id of owner, as octal numbers in ASCII. |
| 109 | .It Va size |
| 110 | Size of file, as octal number in ASCII. |
| 111 | For regular files only, this indicates the amount of data |
| 112 | that follows the header. |
| 113 | In particular, this field was ignored by early tar implementations |
| 114 | when extracting hardlinks. |
| 115 | Modern writers should always store a zero length for hardlink entries. |
| 116 | .It Va mtime |
| 117 | Modification time of file, as an octal number in ASCII. |
| 118 | This indicates the number of seconds since the start of the epoch, |
| 119 | 00:00:00 UTC January 1, 1970. |
| 120 | Note that negative values should be avoided |
| 121 | here, as they are handled inconsistently. |
| 122 | .It Va checksum |
| 123 | Header checksum, stored as an octal number in ASCII. |
| 124 | To compute the checksum, set the checksum field to all spaces, |
| 125 | then sum all bytes in the header using unsigned arithmetic. |
| 126 | This field should be stored as six octal digits followed by a null and a space |
| 127 | character. |
| 128 | Note that many early implementations of tar used signed arithmetic |
| 129 | for the checksum field, which can cause interoperability problems |
| 130 | when transferring archives between systems. |
| 131 | Modern robust readers compute the checksum both ways and accept the |
| 132 | header if either computation matches. |
| 133 | .It Va linkflag , Va linkname |
| 134 | In order to preserve hardlinks and conserve tape, a file |
| 135 | with multiple links is only written to the archive the first |
| 136 | time it is encountered. |
| 137 | The next time it is encountered, the |
| 138 | .Va linkflag |
| 139 | is set to an ASCII |
| 140 | .Sq 1 |
| 141 | and the |
| 142 | .Va linkname |
| 143 | field holds the first name under which this file appears. |
| 144 | (Note that regular files have a null value in the |
| 145 | .Va linkflag |
| 146 | field.) |
| 147 | .El |
| 148 | .Pp |
| 149 | Early tar implementations varied in how they terminated these fields. |
| 150 | The tar command in |
| 151 | .At v7 |
| 152 | used the following conventions (this is also documented in early BSD manpages): |
| 153 | the pathname must be null-terminated; |
| 154 | the mode, uid, and gid fields must end in a space and a null byte; |
| 155 | the size and mtime fields must end in a space; |
| 156 | the checksum is terminated by a null and a space. |
| 157 | Early implementations filled the numeric fields with leading spaces. |
| 158 | This seems to have been common practice until the |
| 159 | .St -p1003.1-88 |
| 160 | standard was released. |
| 161 | For best portability, modern implementations should fill the numeric |
| 162 | fields with leading zeros. |
| 163 | .Ss Pre-POSIX Archives |
| 164 | An early draft of |
| 165 | .St -p1003.1-88 |
| 166 | served as the basis for John Gilmore's |
| 167 | .Nm pdtar |
| 168 | program and many system implementations from the late 1980s |
| 169 | and early 1990s. |
| 170 | These archives generally follow the POSIX ustar |
| 171 | format described below with the following variations: |
| 172 | .Bl -bullet -compact -width indent |
| 173 | .It |
| 174 | The magic value is |
| 175 | .Dq ustar\ \& |
| 176 | (note the following space). |
| 177 | The version field contains a space character followed by a null. |
| 178 | .It |
| 179 | The numeric fields are generally filled with leading spaces |
| 180 | (not leading zeros as recommended in the final standard). |
| 181 | .It |
| 182 | The prefix field is often not used, limiting pathnames to |
| 183 | the 100 characters of old-style archives. |
| 184 | .El |
| 185 | .Ss POSIX ustar Archives |
| 186 | .St -p1003.1-88 |
| 187 | defined a standard tar file format to be read and written |
| 188 | by compliant implementations of |
| 189 | .Xr tar 1 . |
| 190 | This format is often called the |
| 191 | .Dq ustar |
| 192 | format, after the magic value used |
| 193 | in the header. |
| 194 | (The name is an acronym for |
| 195 | .Dq Unix Standard TAR . ) |
| 196 | It extends the historic format with new fields: |
| 197 | .Bd -literal -offset indent |
| 198 | struct header_posix_ustar { |
| 199 | char name[100]; |
| 200 | char mode[8]; |
| 201 | char uid[8]; |
| 202 | char gid[8]; |
| 203 | char size[12]; |
| 204 | char mtime[12]; |
| 205 | char checksum[8]; |
| 206 | char typeflag[1]; |
| 207 | char linkname[100]; |
| 208 | char magic[6]; |
| 209 | char version[2]; |
| 210 | char uname[32]; |
| 211 | char gname[32]; |
| 212 | char devmajor[8]; |
| 213 | char devminor[8]; |
| 214 | char prefix[155]; |
| 215 | char pad[12]; |
| 216 | }; |
| 217 | .Ed |
| 218 | .Bl -tag -width indent |
| 219 | .It Va typeflag |
| 220 | Type of entry. |
| 221 | POSIX extended the earlier |
| 222 | .Va linkflag |
| 223 | field with several new type values: |
| 224 | .Bl -tag -width indent -compact |
| 225 | .It Dq 0 |
| 226 | Regular file. |
| 227 | NUL should be treated as a synonym, for compatibility purposes. |
| 228 | .It Dq 1 |
| 229 | Hard link. |
| 230 | .It Dq 2 |
| 231 | Symbolic link. |
| 232 | .It Dq 3 |
| 233 | Character device node. |
| 234 | .It Dq 4 |
| 235 | Block device node. |
| 236 | .It Dq 5 |
| 237 | Directory. |
| 238 | .It Dq 6 |
| 239 | FIFO node. |
| 240 | .It Dq 7 |
| 241 | Reserved. |
| 242 | .It Other |
| 243 | A POSIX-compliant implementation must treat any unrecognized typeflag value |
| 244 | as a regular file. |
| 245 | In particular, writers should ensure that all entries |
| 246 | have a valid filename so that they can be restored by readers that do not |
| 247 | support the corresponding extension. |
| 248 | Uppercase letters "A" through "Z" are reserved for custom extensions. |
| 249 | Note that sockets and whiteout entries are not archivable. |
| 250 | .El |
| 251 | It is worth noting that the |
| 252 | .Va size |
| 253 | field, in particular, has different meanings depending on the type. |
| 254 | For regular files, of course, it indicates the amount of data |
| 255 | following the header. |
| 256 | For directories, it may be used to indicate the total size of all |
| 257 | files in the directory, for use by operating systems that pre-allocate |
| 258 | directory space. |
| 259 | For all other types, it should be set to zero by writers and ignored |
| 260 | by readers. |
| 261 | .It Va magic |
| 262 | Contains the magic value |
| 263 | .Dq ustar |
| 264 | followed by a NUL byte to indicate that this is a POSIX standard archive. |
| 265 | Full compliance requires the uname and gname fields be properly set. |
| 266 | .It Va version |
| 267 | Version. |
| 268 | This should be |
| 269 | .Dq 00 |
| 270 | (two copies of the ASCII digit zero) for POSIX standard archives. |
| 271 | .It Va uname , Va gname |
| 272 | User and group names, as null-terminated ASCII strings. |
| 273 | These should be used in preference to the uid/gid values |
| 274 | when they are set and the corresponding names exist on |
| 275 | the system. |
| 276 | .It Va devmajor , Va devminor |
| 277 | Major and minor numbers for character device or block device entry. |
| 278 | .It Va name , Va prefix |
| 279 | If the pathname is too long to fit in the 100 bytes provided by the standard |
| 280 | format, it can be split at any |
| 281 | .Pa / |
| 282 | character with the first portion going into the prefix field. |
| 283 | If the prefix field is not empty, the reader will prepend |
| 284 | the prefix value and a |
| 285 | .Pa / |
| 286 | character to the regular name field to obtain the full pathname. |
| 287 | The standard does not require a trailing |
| 288 | .Pa / |
| 289 | character on directory names, though most implementations still |
| 290 | include this for compatibility reasons. |
| 291 | .El |
| 292 | .Pp |
| 293 | Note that all unused bytes must be set to |
| 294 | .Dv NUL . |
| 295 | .Pp |
| 296 | Field termination is specified slightly differently by POSIX |
| 297 | than by previous implementations. |
| 298 | The |
| 299 | .Va magic , |
| 300 | .Va uname , |
| 301 | and |
| 302 | .Va gname |
| 303 | fields must have a trailing |
| 304 | .Dv NUL . |
| 305 | The |
| 306 | .Va pathname , |
| 307 | .Va linkname , |
| 308 | and |
| 309 | .Va prefix |
| 310 | fields must have a trailing |
| 311 | .Dv NUL |
| 312 | unless they fill the entire field. |
| 313 | (In particular, it is possible to store a 256-character pathname if it |
| 314 | happens to have a |
| 315 | .Pa / |
| 316 | as the 156th character.) |
| 317 | POSIX requires numeric fields to be zero-padded in the front, and requires |
| 318 | them to be terminated with either space or |
| 319 | .Dv NUL |
| 320 | characters. |
| 321 | .Pp |
| 322 | Currently, most tar implementations comply with the ustar |
| 323 | format, occasionally extending it by adding new fields to the |
| 324 | blank area at the end of the header record. |
| 325 | .Ss Pax Interchange Format |
| 326 | There are many attributes that cannot be portably stored in a |
| 327 | POSIX ustar archive. |
| 328 | .St -p1003.1-2001 |
| 329 | defined a |
| 330 | .Dq pax interchange format |
| 331 | that uses two new types of entries to hold text-formatted |
| 332 | metadata that applies to following entries. |
| 333 | Note that a pax interchange format archive is a ustar archive in every |
| 334 | respect. |
| 335 | The new data is stored in ustar-compatible archive entries that use the |
| 336 | .Dq x |
| 337 | or |
| 338 | .Dq g |
| 339 | typeflag. |
| 340 | In particular, older implementations that do not fully support these |
| 341 | extensions will extract the metadata into regular files, where the |
| 342 | metadata can be examined as necessary. |
| 343 | .Pp |
| 344 | An entry in a pax interchange format archive consists of one or |
| 345 | two standard ustar entries, each with its own header and data. |
| 346 | The first optional entry stores the extended attributes |
| 347 | for the following entry. |
| 348 | This optional first entry has an "x" typeflag and a size field that |
| 349 | indicates the total size of the extended attributes. |
| 350 | The extended attributes themselves are stored as a series of text-format |
| 351 | lines encoded in the portable UTF-8 encoding. |
| 352 | Each line consists of a decimal number, a space, a key string, an equals |
| 353 | sign, a value string, and a new line. |
| 354 | The decimal number indicates the length of the entire line, including the |
| 355 | initial length field and the trailing newline. |
| 356 | An example of such a field is: |
| 357 | .Dl 25 ctime=1084839148.1212\en |
| 358 | Keys in all lowercase are standard keys. |
| 359 | Vendors can add their own keys by prefixing them with an all uppercase |
| 360 | vendor name and a period. |
| 361 | Note that, unlike the historic header, numeric values are stored using |
| 362 | decimal, not octal. |
| 363 | A description of some common keys follows: |
| 364 | .Bl -tag -width indent |
| 365 | .It Cm atime , Cm ctime , Cm mtime |
| 366 | File access, inode change, and modification times. |
| 367 | These fields can be negative or include a decimal point and a fractional value. |
| 368 | .It Cm uname , Cm uid , Cm gname , Cm gid |
| 369 | User name, group name, and numeric UID and GID values. |
| 370 | The user name and group name stored here are encoded in UTF8 |
| 371 | and can thus include non-ASCII characters. |
| 372 | The UID and GID fields can be of arbitrary length. |
| 373 | .It Cm linkpath |
| 374 | The full path of the linked-to file. |
| 375 | Note that this is encoded in UTF8 and can thus include non-ASCII characters. |
| 376 | .It Cm path |
| 377 | The full pathname of the entry. |
| 378 | Note that this is encoded in UTF8 and can thus include non-ASCII characters. |
| 379 | .It Cm realtime.* , Cm security.* |
| 380 | These keys are reserved and may be used for future standardization. |
| 381 | .It Cm size |
| 382 | The size of the file. |
| 383 | Note that there is no length limit on this field, allowing conforming |
| 384 | archives to store files much larger than the historic 8GB limit. |
| 385 | .It Cm SCHILY.* |
| 386 | Vendor-specific attributes used by Joerg Schilling's |
| 387 | .Nm star |
| 388 | implementation. |
| 389 | .It Cm SCHILY.acl.access , Cm SCHILY.acl.default |
| 390 | Stores the access and default ACLs as textual strings in a format |
| 391 | that is an extension of the format specified by POSIX.1e draft 17. |
| 392 | In particular, each user or group access specification can include a fourth |
| 393 | colon-separated field with the numeric UID or GID. |
| 394 | This allows ACLs to be restored on systems that may not have complete |
| 395 | user or group information available (such as when NIS/YP or LDAP services |
| 396 | are temporarily unavailable). |
| 397 | .It Cm SCHILY.devminor , Cm SCHILY.devmajor |
| 398 | The full minor and major numbers for device nodes. |
| 399 | .It Cm SCHILY.fflags |
| 400 | The file flags. |
| 401 | .It Cm SCHILY.realsize |
| 402 | The full size of the file on disk. |
| 403 | XXX explain? XXX |
| 404 | .It Cm SCHILY.dev, Cm SCHILY.ino , Cm SCHILY.nlinks |
| 405 | The device number, inode number, and link count for the entry. |
| 406 | In particular, note that a pax interchange format archive using Joerg |
| 407 | Schilling's |
| 408 | .Cm SCHILY.* |
| 409 | extensions can store all of the data from |
| 410 | .Va struct stat . |
| 411 | .It Cm LIBARCHIVE.xattr. Ns Ar namespace Ns . Ns Ar key |
| 412 | Libarchive stores POSIX.1e-style extended attributes using |
| 413 | keys of this form. |
| 414 | The |
| 415 | .Ar key |
| 416 | value is URL-encoded: |
| 417 | All non-ASCII characters and the two special characters |
| 418 | .Dq = |
| 419 | and |
| 420 | .Dq % |
| 421 | are encoded as |
| 422 | .Dq % |
| 423 | followed by two uppercase hexadecimal digits. |
| 424 | The value of this key is the extended attribute value |
| 425 | encoded in base 64. |
| 426 | XXX Detail the base-64 format here XXX |
| 427 | .It Cm VENDOR.* |
| 428 | XXX document other vendor-specific extensions XXX |
| 429 | .El |
| 430 | .Pp |
| 431 | Any values stored in an extended attribute override the corresponding |
| 432 | values in the regular tar header. |
| 433 | Note that compliant readers should ignore the regular fields when they |
| 434 | are overridden. |
| 435 | This is important, as existing archivers are known to store non-compliant |
| 436 | values in the standard header fields in this situation. |
| 437 | There are no limits on length for any of these fields. |
| 438 | In particular, numeric fields can be arbitrarily large. |
| 439 | All text fields are encoded in UTF8. |
| 440 | Compliant writers should store only portable 7-bit ASCII characters in |
| 441 | the standard ustar header and use extended |
| 442 | attributes whenever a text value contains non-ASCII characters. |
| 443 | .Pp |
| 444 | In addition to the |
| 445 | .Cm x |
| 446 | entry described above, the pax interchange format |
| 447 | also supports a |
| 448 | .Cm g |
| 449 | entry. |
| 450 | The |
| 451 | .Cm g |
| 452 | entry is identical in format, but specifies attributes that serve as |
| 453 | defaults for all subsequent archive entries. |
| 454 | The |
| 455 | .Cm g |
| 456 | entry is not widely used. |
| 457 | .Pp |
| 458 | Besides the new |
| 459 | .Cm x |
| 460 | and |
| 461 | .Cm g |
| 462 | entries, the pax interchange format has a few other minor variations |
| 463 | from the earlier ustar format. |
| 464 | The most troubling one is that hardlinks are permitted to have |
| 465 | data following them. |
| 466 | This allows readers to restore any hardlink to a file without |
| 467 | having to rewind the archive to find an earlier entry. |
| 468 | However, it creates complications for robust readers, as it is no longer |
| 469 | clear whether or not they should ignore the size field for hardlink entries. |
| 470 | .Ss GNU Tar Archives |
| 471 | The GNU tar program started with a pre-POSIX format similar to that |
| 472 | described earlier and has extended it using several different mechanisms: |
| 473 | It added new fields to the empty space in the header (some of which was later |
| 474 | used by POSIX for conflicting purposes); |
| 475 | it allowed the header to be continued over multiple records; |
| 476 | and it defined new entries that modify following entries |
| 477 | (similar in principle to the |
| 478 | .Cm x |
| 479 | entry described above, but each GNU special entry is single-purpose, |
| 480 | unlike the general-purpose |
| 481 | .Cm x |
| 482 | entry). |
| 483 | As a result, GNU tar archives are not POSIX compatible, although |
| 484 | more lenient POSIX-compliant readers can successfully extract most |
| 485 | GNU tar archives. |
| 486 | .Bd -literal -offset indent |
| 487 | struct header_gnu_tar { |
| 488 | char name[100]; |
| 489 | char mode[8]; |
| 490 | char uid[8]; |
| 491 | char gid[8]; |
| 492 | char size[12]; |
| 493 | char mtime[12]; |
| 494 | char checksum[8]; |
| 495 | char typeflag[1]; |
| 496 | char linkname[100]; |
| 497 | char magic[6]; |
| 498 | char version[2]; |
| 499 | char uname[32]; |
| 500 | char gname[32]; |
| 501 | char devmajor[8]; |
| 502 | char devminor[8]; |
| 503 | char atime[12]; |
| 504 | char ctime[12]; |
| 505 | char offset[12]; |
| 506 | char longnames[4]; |
| 507 | char unused[1]; |
| 508 | struct { |
| 509 | char offset[12]; |
| 510 | char numbytes[12]; |
| 511 | } sparse[4]; |
| 512 | char isextended[1]; |
| 513 | char realsize[12]; |
| 514 | char pad[17]; |
| 515 | }; |
| 516 | .Ed |
| 517 | .Bl -tag -width indent |
| 518 | .It Va typeflag |
| 519 | GNU tar uses the following special entry types, in addition to |
| 520 | those defined by POSIX: |
| 521 | .Bl -tag -width indent |
| 522 | .It "7" |
| 523 | GNU tar treats type "7" records identically to type "0" records, |
| 524 | except on one obscure RTOS where they are used to indicate the |
| 525 | pre-allocation of a contiguous file on disk. |
| 526 | .It "D" |
| 527 | This indicates a directory entry. |
| 528 | Unlike the POSIX-standard "5" |
| 529 | typeflag, the header is followed by data records listing the names |
| 530 | of files in this directory. |
| 531 | Each name is preceded by an ASCII "Y" |
| 532 | if the file is stored in this archive or "N" if the file is not |
| 533 | stored in this archive. |
| 534 | Each name is terminated with a null, and |
| 535 | an extra null marks the end of the name list. |
| 536 | The purpose of this |
| 537 | entry is to support incremental backups; a program restoring from |
| 538 | such an archive may wish to delete files on disk that did not exist |
| 539 | in the directory when the archive was made. |
| 540 | .Pp |
| 541 | Note that the "D" typeflag specifically violates POSIX, which requires |
| 542 | that unrecognized typeflags be restored as normal files. |
| 543 | In this case, restoring the "D" entry as a file could interfere |
| 544 | with subsequent creation of the like-named directory. |
| 545 | .It "K" |
| 546 | The data for this entry is a long linkname for the following regular entry. |
| 547 | .It "L" |
| 548 | The data for this entry is a long pathname for the following regular entry. |
| 549 | .It "M" |
| 550 | This is a continuation of the last file on the previous volume. |
| 551 | GNU multi-volume archives guarantee that each volume begins with a valid |
| 552 | entry header. |
| 553 | To ensure this, a file may be split, with part stored at the end of one volume, |
| 554 | and part stored at the beginning of the next volume. |
| 555 | The "M" typeflag indicates that this entry continues an existing file. |
| 556 | Such entries can only occur as the first or second entry |
| 557 | in an archive (the latter only if the first entry is a volume label). |
| 558 | The |
| 559 | .Va size |
| 560 | field specifies the size of this entry. |
| 561 | The |
| 562 | .Va offset |
| 563 | field at bytes 369-380 specifies the offset where this file fragment |
| 564 | begins. |
| 565 | The |
| 566 | .Va realsize |
| 567 | field specifies the total size of the file (which must equal |
| 568 | .Va size |
| 569 | plus |
| 570 | .Va offset ) . |
| 571 | When extracting, GNU tar checks that the header file name is the one it is |
| 572 | expecting, that the header offset is in the correct sequence, and that |
| 573 | the sum of offset and size is equal to realsize. |
| 574 | .It "N" |
| 575 | Type "N" records are no longer generated by GNU tar. |
| 576 | They contained a |
| 577 | list of files to be renamed or symlinked after extraction; this was |
| 578 | originally used to support long names. |
| 579 | The contents of this record |
| 580 | are a text description of the operations to be done, in the form |
| 581 | .Dq Rename %s to %s\en |
| 582 | or |
| 583 | .Dq Symlink %s to %s\en ; |
| 584 | in either case, both |
| 585 | filenames are escaped using K&R C syntax. |
| 586 | Due to security concerns, "N" records are now generally ignored |
| 587 | when reading archives. |
| 588 | .It "S" |
| 589 | This is a |
| 590 | .Dq sparse |
| 591 | regular file. |
| 592 | Sparse files are stored as a series of fragments. |
| 593 | The header contains a list of fragment offset/length pairs. |
| 594 | If more than four such entries are required, the header is |
| 595 | extended as necessary with |
| 596 | .Dq extra |
| 597 | header extensions (an older format that is no longer used), or |
| 598 | .Dq sparse |
| 599 | extensions. |
| 600 | .It "V" |
| 601 | The |
| 602 | .Va name |
| 603 | field should be interpreted as a tape/volume header name. |
| 604 | This entry should generally be ignored on extraction. |
| 605 | .El |
| 606 | .It Va magic |
| 607 | The magic field holds the five characters |
| 608 | .Dq ustar |
| 609 | followed by a space. |
| 610 | Note that POSIX ustar archives have a trailing null. |
| 611 | .It Va version |
| 612 | The version field holds a space character followed by a null. |
| 613 | Note that POSIX ustar archives use two copies of the ASCII digit |
| 614 | .Dq 0 . |
| 615 | .It Va atime , Va ctime |
| 616 | The time the file was last accessed and the time of |
| 617 | last change of file information, stored in octal as with |
| 618 | .Va mtime . |
| 619 | .It Va longnames |
| 620 | This field is apparently no longer used. |
| 621 | .It Sparse Va offset / Va numbytes |
| 622 | Each such structure specifies a single fragment of a sparse |
| 623 | file. |
| 624 | The two fields store values as octal numbers. |
| 625 | The fragments are each padded to a multiple of 512 bytes |
| 626 | in the archive. |
| 627 | On extraction, the list of fragments is collected from the |
| 628 | header (including any extension headers), and the data |
| 629 | is then read and written to the file at appropriate offsets. |
| 630 | .It Va isextended |
| 631 | If this is set to non-zero, the header will be followed by additional |
| 632 | .Dq sparse header |
| 633 | records. |
| 634 | Each such record contains information about as many as 21 additional |
| 635 | sparse blocks as shown here: |
| 636 | .Bd -literal -offset indent |
| 637 | struct gnu_sparse_header { |
| 638 | struct { |
| 639 | char offset[12]; |
| 640 | char numbytes[12]; |
| 641 | } sparse[21]; |
| 642 | char isextended[1]; |
| 643 | char padding[7]; |
| 644 | }; |
| 645 | .Ed |
| 646 | .It Va realsize |
| 647 | A binary representation of the file's complete size, with a much larger range |
| 648 | than the POSIX file size. |
| 649 | In particular, with |
| 650 | .Cm M |
| 651 | type files, the current entry is only a portion of the file. |
| 652 | In that case, the POSIX size field will indicate the size of this |
| 653 | entry; the |
| 654 | .Va realsize |
| 655 | field will indicate the total size of the file. |
| 656 | .El |
| 657 | .Ss GNU tar pax archives |
| 658 | GNU tar 1.14 (XXX check this XXX) and later will write |
| 659 | pax interchange format archives when you specify the |
| 660 | .Fl -posix |
| 661 | flag. |
| 662 | This format uses custom keywords to store sparse file information. |
| 663 | There have been three iterations of this support, referred to |
| 664 | as |
| 665 | .Dq 0.0 , |
| 666 | .Dq 0.1 , |
| 667 | and |
| 668 | .Dq 1.0 . |
| 669 | .Bl -tag -width indent |
| 670 | .It Cm GNU.sparse.numblocks , Cm GNU.sparse.offset , Cm GNU.sparse.numbytes , Cm GNU.sparse.size |
| 671 | The |
| 672 | .Dq 0.0 |
| 673 | format used an initial |
| 674 | .Cm GNU.sparse.numblocks |
| 675 | attribute to indicate the number of blocks in the file, a pair of |
| 676 | .Cm GNU.sparse.offset |
| 677 | and |
| 678 | .Cm GNU.sparse.numbytes |
| 679 | to indicate the offset and size of each block, |
| 680 | and a single |
| 681 | .Cm GNU.sparse.size |
| 682 | to indicate the full size of the file. |
| 683 | This is not the same as the size in the tar header because the |
| 684 | latter value does not include the size of any holes. |
| 685 | This format required that the order of attributes be preserved and |
| 686 | relied on readers accepting multiple appearances of the same attribute |
| 687 | names, which is not officially permitted by the standards. |
| 688 | .It Cm GNU.sparse.map |
| 689 | The |
| 690 | .Dq 0.1 |
| 691 | format used a single attribute that stored a comma-separated |
| 692 | list of decimal numbers. |
| 693 | Each pair of numbers indicated the offset and size, respectively, |
| 694 | of a block of data. |
| 695 | This does not work well if the archive is extracted by an archiver |
| 696 | that does not recognize this extension, since many pax implementations |
| 697 | simply discard unrecognized attributes. |
| 698 | .It Cm GNU.sparse.major , Cm GNU.sparse.minor , Cm GNU.sparse.name , Cm GNU.sparse.realsize |
| 699 | The |
| 700 | .Dq 1.0 |
| 701 | format stores the sparse block map in one or more 512-byte blocks |
| 702 | prepended to the file data in the entry body. |
| 703 | The pax attributes indicate the existence of this map |
| 704 | (via the |
| 705 | .Cm GNU.sparse.major |
| 706 | and |
| 707 | .Cm GNU.sparse.minor |
| 708 | fields) |
| 709 | and the full size of the file. |
| 710 | The |
| 711 | .Cm GNU.sparse.name |
| 712 | holds the true name of the file. |
| 713 | To avoid confusion, the name stored in the regular tar header |
| 714 | is a modified name so that extraction errors will be apparent |
| 715 | to users. |
| 716 | .El |
| 717 | .Ss Solaris Tar |
| 718 | XXX More Details Needed XXX |
| 719 | .Pp |
| 720 | Solaris tar (beginning with SunOS XXX 5.7 ?? XXX) supports an |
| 721 | .Dq extended |
| 722 | format that is fundamentally similar to pax interchange format, |
| 723 | with the following differences: |
| 724 | .Bl -bullet -compact -width indent |
| 725 | .It |
| 726 | Extended attributes are stored in an entry whose type is |
| 727 | .Cm X , |
| 728 | not |
| 729 | .Cm x , |
| 730 | as used by pax interchange format. |
| 731 | The detailed format of this entry appears to be the same |
| 732 | as detailed above for the |
| 733 | .Cm x |
| 734 | entry. |
| 735 | .It |
| 736 | An additional |
| 737 | .Cm A |
| 738 | entry is used to store an ACL for the following regular entry. |
| 739 | The body of this entry contains a seven-digit octal number |
| 740 | followed by a zero byte, followed by the |
| 741 | textual ACL description. |
| 742 | The octal value is the number of ACL entries |
| 743 | plus a constant that indicates the ACL type: 01000000 |
| 744 | for POSIX.1e ACLs and 03000000 for NFSv4 ACLs. |
| 745 | .El |
| 746 | .Ss AIX Tar |
| 747 | XXX More details needed XXX |
| 748 | .Ss Mac OS X Tar |
| 749 | The tar distributed with Apple's Mac OS X stores most regular files |
| 750 | as two separate entries in the tar archive. |
| 751 | The two entries have the same name except that the first |
| 752 | one has |
| 753 | .Dq ._ |
| 754 | added to the beginning of the name. |
| 755 | This first entry stores the |
| 756 | .Dq resource fork |
| 757 | with additional attributes for the file. |
| 758 | The Mac OS X |
| 759 | .Fn CopyFile |
| 760 | API is used to separate a file on disk into separate |
| 761 | resource and data streams and to reassemble those separate |
| 762 | streams when the file is restored to disk. |
| 763 | .Ss Other Extensions |
| 764 | One obvious extension to increase the size of files is to |
| 765 | eliminate the terminating characters from the various |
| 766 | numeric fields. |
| 767 | For example, the standard only allows the size field to contain |
| 768 | 11 octal digits, reserving the twelfth byte for a trailing |
| 769 | NUL character. |
| 770 | Allowing 12 octal digits allows file sizes up to 64 GB. |
| 771 | .Pp |
| 772 | Another extension, utilized by GNU tar, star, and other newer |
| 773 | .Nm |
| 774 | implementations, permits binary numbers in the standard numeric fields. |
| 775 | This is flagged by setting the high bit of the first byte. |
| 776 | This permits 95-bit values for the length and time fields |
| 777 | and 63-bit values for the uid, gid, and device numbers. |
| 778 | GNU tar supports this extension for the |
| 779 | length, mtime, ctime, and atime fields. |
| 780 | Joerg Schilling's star program supports this extension for |
| 781 | all numeric fields. |
| 782 | Note that this extension is largely obsoleted by the extended attribute |
| 783 | record provided by the pax interchange format. |
| 784 | .Pp |
| 785 | Another early GNU extension allowed base-64 values rather than octal. |
| 786 | This extension was short-lived and is no longer supported by any |
| 787 | implementation. |
| 788 | .Sh SEE ALSO |
| 789 | .Xr ar 1 , |
| 790 | .Xr pax 1 , |
| 791 | .Xr tar 1 |
| 792 | .Sh STANDARDS |
| 793 | The |
| 794 | .Nm tar |
| 795 | utility is no longer a part of POSIX or the Single Unix Standard. |
| 796 | It last appeared in |
| 797 | .St -susv2 . |
| 798 | It has been supplanted in subsequent standards by |
| 799 | .Xr pax 1 . |
| 800 | The ustar format is currently part of the specification for the |
| 801 | .Xr pax 1 |
| 802 | utility. |
| 803 | The pax interchange file format is new with |
| 804 | .St -p1003.1-2001 . |
| 805 | .Sh HISTORY |
| 806 | A |
| 807 | .Nm tar |
| 808 | command appeared in Seventh Edition Unix, which was released in January, 1979. |
| 809 | It replaced the |
| 810 | .Nm tp |
| 811 | program from Fourth Edition Unix which in turn replaced the |
| 812 | .Nm tap |
| 813 | program from First Edition Unix. |
| 814 | John Gilmore's |
| 815 | .Nm pdtar |
| 816 | public-domain implementation (circa 1987) was highly influential |
| 817 | and formed the basis of |
| 818 | .Nm GNU tar |
| 819 | (circa 1988). |
| 820 | Joerg Shilling's |
| 821 | .Nm star |
| 822 | archiver is another open-source (GPL) archiver (originally developed |
| 823 | circa 1985) which features complete support for pax interchange |
| 824 | format. |
| 825 | .Pp |
| 826 | This documentation was written as part of the |
| 827 | .Nm libarchive |
| 828 | and |
| 829 | .Nm bsdtar |
| 830 | project by |
| 831 | .An Tim Kientzle Aq kientzle@FreeBSD.org . |