From 293141b7083f343cbec98a068441d2af3a93bb89 Mon Sep 17 00:00:00 2001 From: Matthew Dillon Date: Thu, 3 Dec 2009 12:55:24 -0800 Subject: [PATCH] cpdup - Multiple additions, fixes, and one removal * Make -x/-X (cpignore) work with remote sources. * Add Makefile hack for _ST_FLAGS_PRESENT_ on FreeBSD. * Suppress "Handshaked with " and "Not running as root" messages when the -q option was specified. * Mount points are now skipped completely. (Before this, the directories and .cpignore inside them were still read.) * Extract scanning directories (src and dst) from DoCopy() into a function ScanDir(), and use ScanDir() for both src and dst. * Remove threads support. It never worked very well and is getting in the way of new development. The intention is to develop a more efficient pipelined algorithm in the future which does not require threading the program. * Use getopt(3) instead of non-standard parser, making sure that all options still work exactly as documented in the manual page. * Use fnmatch(3) instead of WildCmp(), adding support for bracket expressions (e.g. [xyz0-9]) and escaping (e.g. foo\*). I also suspect that fnmatch() is more efficient. * Use the same syntax for remote paths as scp(1), so local paths that contain colons can be specified (e.g. "./foo:bar"); extract that code into its own function SplitRemote(). * Add protocol commands hc_geteuid() and hc_getgroups(). Use them to properly check permissions for various operations that would require root privileges on the target machine. * Bugfix: If a symlink already existed on the target but the owner/group didn't match, cpdup missed to update it. Added a call to hc_lchown() to fix that, if privileges allow. * Bugfix: The dispatch[] array wasn't initialized, so the slave crashed upon an unknown command instead of calling rc_badop(). * Bump protocol version to 3 in order to be able to maintain backwards compatibility with older clients. * Bump utility version to 1.16. * Make the source compile with WARNS=6. It still doesn't pass -ansi -pedantic, though. :-) * Document -j0, -k and -K in usage message (misc.c). * Update the manual page, fix mdoc markup and various details. * Add "ssh security tips" section to the BACKUPS text file. Submitted-by: Oliver Fromme --- bin/cpdup/BACKUPS | 64 +++- bin/cpdup/Makefile | 6 +- bin/cpdup/PORTING | 6 +- bin/cpdup/cpdup.1 | 141 +++++--- bin/cpdup/cpdup.c | 838 +++++++++++++++++++++----------------------- bin/cpdup/cpdup.h | 13 +- bin/cpdup/hclink.c | 162 +-------- bin/cpdup/hclink.h | 18 +- bin/cpdup/hcproto.c | 176 ++++++++-- bin/cpdup/hcproto.h | 7 +- bin/cpdup/misc.c | 9 +- 11 files changed, 728 insertions(+), 712 deletions(-) diff --git a/bin/cpdup/BACKUPS b/bin/cpdup/BACKUPS index bd83bb19fc..c891d3099c 100644 --- a/bin/cpdup/BACKUPS +++ b/bin/cpdup/BACKUPS @@ -17,7 +17,8 @@ $DragonFly: src/bin/cpdup/BACKUPS,v 1.4 2007/05/17 08:19:00 swildner Exp $ The easiest way to create a LAN backup box is to NFS mount all your backup clients onto the backup box. It is also possible to use cpdup's remote host feature to access your client boxes but that requires root - access to the client boxes and is not described here. + access to the client boxes and is not described here. (But see the + sections "OFF-SITE BACKUPS" and "SSH SECURITY TIPS" below.) Create a directory on the backup machine called /nfs, a subdirectory foreach remote client, and subdirectories for each partition on each @@ -158,7 +159,7 @@ $DragonFly: src/bin/cpdup/BACKUPS,v 1.4 2007/05/17 08:19:00 swildner Exp $ The remote backup box does not use NFS, so setup is trivial. Just create your super-large /backup partition and mkdir /backup/mirrors. Your LAN backup box will need root access via ssh to your remote backup - box. + box. See the section "SSH SECURITY TIPS" below. You can use the handy softlink to get the latest 'box1.date' mirror directory and since the mirror is all in one partition you can just @@ -261,3 +262,62 @@ $DragonFly: src/bin/cpdup/BACKUPS,v 1.4 2007/05/17 08:19:00 swildner Exp $ should do a tower-of-hanoi removal + SSH SECURITY TIPS + + To allow root access via ssh, add the following line to your sshd + configuration on the client boxes (typically /etc/ssh/sshd_config): + + PermitRootLogin forced-commands-only + + If your OpenSSH version is too old to recognize that setting, you + should update to a more recent version immediately. + + On the backup machine, create a special backup key for root: + + mkdir /root/.ssh # if it doesn't already exist + cd /root/.ssh + ssh-keygen -t dsa -N "" -f backup-key + + You now have a key pair, consisting of a secret key called "backup-key" + and a public key called "backup-key.pub". The secret key must *NEVER* + leave the backup machine nor be disclosed in any way! Note that we + haven't procted the secret key with a passphrase (-N "") because it + will be used by cron jobs where no passphrase can be entered. + + On the client boxes, create a file /root/.ssh/authorized_keys. + It should contain just this line: + + command="/usr/local/bin/cpdup -S",from="",no-pty, + no-port-forwarding,no-X11-forwarding,no-agent-forwarding + + This must be on one long line; it has been broken up here for + readability only. Note that the options must be separated by commas + *ONLY* (no spaces). Replace with the IP address or DNS name + of the backup machine. Replace with the contents of the + file /root/.ssh/backup-key.pub from the backup machine (the public key, + not the secret key!). It typically starts with "ssh-dss" followed by + a long character sequence that looks like line noise, followed by a + comment that typically indicates who created the key. + + The format of the authorized_keys file is documented in the sshd(8) + manual page. Please refer to it for more details. + + If you have done all of the above correctly, then the root user on the + backup machine will be able to log into the client boxes as root and + execute "/usr/local/bin/cpdup -S", but nothing else. + + When using cpdup on the backup machine, make sure that the right key is + used by passing the -i option to the ssh command: + + cpdup -F -i/root/.ssh/backup-key ... + + If one or both of the machines involved has a slow processor, it might + be worthwhile to use a faster encryption algorithm, for example: + + cpdup -F -cblowfish-cbc ... + + If your OpenSSH version has been patched to support unencrypted transfers + *AND* you trust the physical network between the machines involved, you + might want to disable encryption alltogether: + + cpdup -F -cnone ... diff --git a/bin/cpdup/Makefile b/bin/cpdup/Makefile index a43c56b040..9675d30fb2 100644 --- a/bin/cpdup/Makefile +++ b/bin/cpdup/Makefile @@ -4,10 +4,12 @@ PROG= cpdup SRCS= cpdup.c hcproto.c hclink.c misc.c fsmid.c -.if !defined(NOPTHREADS) && !defined(BOOTSTRAPPING) -CFLAGS += -DUSE_PTHREADS=1 -pthread +.if defined(.FreeBSD) +CFLAGS += -D_ST_FLAGS_PRESENT_=1 .endif +WARNS?= 6 + .if !defined(NOMD5) SRCS+= md5.c .endif diff --git a/bin/cpdup/PORTING b/bin/cpdup/PORTING index f49b5e3482..2b3d7a8250 100644 --- a/bin/cpdup/PORTING +++ b/bin/cpdup/PORTING @@ -23,10 +23,8 @@ $DragonFly: src/bin/cpdup/PORTING,v 1.2 2008/04/11 17:33:11 dillon Exp $ cd cpdup rm -f md5.c rm -f *.o - cc -D__unused= -D_GNU_SOURCE -D__USE_FILE_OFFSET64 -DNOMD5 \ - -DUSE_PTHREADS -pthread *.c -c - cc -D__unused= -D_GNU_SOURCE -D__USE_FILE_OFFSET64 -DNOMD5 \ - -DUSE_PTHREADS -pthread *.o -o ~/bin/cpdup + cc -D__unused= -D_GNU_SOURCE -D__USE_FILE_OFFSET64 -DNOMD5 *.c -c + cc -D__unused= -D_GNU_SOURCE -D__USE_FILE_OFFSET64 -DNOMD5 *.o -o ~/bin/cpdup BACKUP SCRIPT MODIFICATIONS - you will almost certainly have to adjust the do_cleanup script to extract the proper field(s) from the df output. diff --git a/bin/cpdup/cpdup.1 b/bin/cpdup/cpdup.1 index 73f7a5054a..d41d4e7ee3 100644 --- a/bin/cpdup/cpdup.1 +++ b/bin/cpdup/cpdup.1 @@ -11,41 +11,29 @@ .Sh SYNOPSIS .Nm .Op Fl C -.Op Fl v[vv..] +.Op Fl v Ns Op Cm v Ns Op Cm v +.Op Fl d .Op Fl u .Op Fl I .Op Fl f -.Op Fl F +.Op Fl F Ar ssh-arg .Op Fl s0 .Op Fl i0 .Op Fl j0 .Op Fl l -.Op Fl p Ar number .Op Fl q .Op Fl o .Op Fl m -.Oo -.Fl H -.Ar path -.Oc -.Oo -.Fl M -.Ar file -.Oc +.Op Fl H Ar path +.Op Fl M Ar file .Op Fl V .Op Fl S .Op Fl k -.Oo -.Fl K -.Ar file -.Oc -.Oo -.Fl X -.Ar file -.Oc +.Op Fl K Ar file +.Op Fl X Ar file .Op Fl x -.Ar [[user@]host:]source_dir -.Ar [[user@]host:]target_dir +.Oo Oo Ar user Ns Li @ Oc Ns Ar host : Oc Ns Ar source_dir +.Oo Oo Ar user Ns Li @ Oc Ns Ar host : Oc Ns Ar target_dir .Sh DESCRIPTION The .Nm @@ -55,7 +43,7 @@ softlinks, devices, permissions, and flags are mirrored. By default, .Nm asks for confirmation if any file or directory needs to be removed from the destination and does not copy files which it believes to have already -been synchronized (by observing that the source and destination file's size +been synchronized (by observing that the source and destination files' sizes and mtimes match). .Nm does not cross mount points in either the source or the destination. @@ -66,10 +54,13 @@ refuses to replace a destination directory with a file. The following options are available: .Bl -tag -width flag .It Fl C -If the source or target is a remote host request that the +If the source or target is a remote host, request that the .Xr ssh 1 session be compressed. -.It Fl v[vv] +This is the same as +.Fl F +.Fl C . +.It Fl v Ns Op Cm v Ns Op Cm v Set verboseness. By default .Nm does not report its progress except when asking for confirmation. A single @@ -81,15 +72,22 @@ modifications made to the destination. .Fl vvv will cause all files and directories to be reported whether or not modifications are made. +.It Fl d +Print directories as they are being traversed. +Useful to watch the progress; +this typically produces much less output than +.Fl vv . .It Fl u -Causes the ouptut generated by -.Fl v[vv] +Causes the output generated by +.Fl v +and +.Fl d to be unbuffered. This can be useful for obtaining prompt progress updates through a pipe. .It Fl I will cause .Nm -to print a summary at the end with performance counter. +to print a summary at the end with performance counters. .It Fl f Forces file updates to occur even if the files appear to be the same. If the @@ -97,8 +95,12 @@ the option is used, this option will force a byte for byte comparison between the original file and the file in the hardlink path, even if all the stat info matches, but will still use a hardlink if they match. -.It Fl F -Pass to ssh. For example '-F-p222'. Note the lack of a space. +.It Fl F Ar ssh-arg +Pass +.Ar ssh-arg +to ssh. For example +.Dq Fl F Fl p222 . +Note the lack of a space. .It Fl s0 Disable the disallow-file-replaces-directory safety feature. This safety feature is enabled by default to prevent user mistakes from blowing @@ -109,27 +111,26 @@ Do not request confirmation when removing something. Do not try to recreate CHR or BLK devices. .It Fl l Line buffer verbose output. -.It Fl p Ar number -Use threaded transactions with up to the specified -.Ar number -of threads. -This typically improves operation when a remote host specification is -given. .It Fl q -Quiet operation +Quiet operation. .It Fl o Do not remove any files, just overwrite/add. .It Fl m -Generate and maintain a MD5 checkfile in each directory on the source +Generate and maintain a MD5 checkfile called +.Pa \&.MD5.CHECKSUMS +in each directory on the source and do an MD5 check on each file of the destination when the destination appears to be the same as the source. If the check fails, -.Nm the source is recopied to the destination. When you specify a destination -directory the MD5 checkfile is only updated as needed and may not be updated +directory, the MD5 checkfile is only updated as needed and may not be updated even if modifications are made to a source file. If you do not specify a destination directory the .Nm command forcefully regenerates the MD5 checkfile for every file in the source. +.It Fl M Ar file +Works the same as +.Fl m +but allows you to specify the name of the MD5 checkfile. .It Fl H Ar path .Nm will create a hardlink from a file found under @@ -139,15 +140,19 @@ via .Ar path is identical to the source. Note that a remote host specification should not be used for this option's -path, but the path will be relative to the target machine. +.Ar path , +but the +.Ar path +will be relative to the target machine. .Pp This allows one to use .Nm -to create incremental backups of a filesystem. Create a direct 'level 0' +to create incremental backups of a filesystem. Create a direct +.Sq level 0 backup, and then specify the level 0 backup path with this option when creating an incremental backup to a different target directory. This method works so long as the filesystem does not hit a hardlink limit. -If the system does hit a hardlink limit +If the system does hit a hardlink limit, .Nm will generate a warning and copy the file instead. Note that @@ -156,7 +161,7 @@ must record file paths for any hardlinked file while operating and therefore uses a great deal more memory when dealing with hardlinks or hardlink-based backups. Example use: .Pp -.Dl cpdup -i0 -s0 -I -H /backup/home.l0 /home /backup/home.l1 +.Dl cpdup \-i0 \-s0 \-I \-H /backup/home.l0 /home /backup/home.l1 .Pp WARNING: If this option is used .Nm @@ -170,10 +175,6 @@ or .Fl f option is also used, otherwise only the stat info is checked to determine whether it matches the source. -.It Fl M Ar file -Works the same as -.Fl m -but allows you to specify the name of the MD5 checkfile. .It Fl V This forces the contents of regular files to be verified, even if the files appear to the be the same. Whereas the @@ -187,8 +188,9 @@ This places into slave mode and is used to initiate the slave protocol on a remote machine. .It Fl k -Generate and maintain a FSMID checkfile called .FSMID.CHECK in each -directory on the target. +Generate and maintain a FSMID checkfile called +.Pa \& .FSMID.CHECK +in each directory on the target. .Nm will check the FSMID for each source file or directory against the checkfile on the target and will not copy the file or recurse through the directory @@ -215,10 +217,14 @@ but allows you to specify the name of the FSMID checkfile. .It Fl x Causes .Nm -to use the exclusion file ".cpignore" in each directory on the source to +to use the exclusion file +.Pa \&.cpignore +in each directory on the source to determine which files to ignore. When this option is used, the exclusion filename itself is automatically excluded from the copy. If this option is -not used then the filename ".cpignore" is not considered special and will +not used then the filename +.Pa \&.cpignore +is not considered special and will be copied along with everything else. .It Fl X Ar file Works the same as @@ -235,6 +241,26 @@ copies. sessions are used and .Nm is run on the remote machine(s) in slave mode. +You can use the +.Fl F +option to pass additional flags to the ssh command if necessary. +.Pp +The syntax of remote path specifications is similar to +.Xr scp 1 . +In particular, that means that a local path containing a colon must +be preceded by a slash to prevent it being considered a remote host: +.Ql foo:bar +causes +.Nm +to look for a directory called +.Ql bar +on host +.Ql foo , +while +.Ql \&./foo:bar +denotes the directory +.Ql foo:bar +on the local machine. .Sh DIAGNOSTICS The .Nm @@ -242,6 +268,8 @@ utility exits 0 if no error occurred and >0 if an error occurred. .Sh SEE ALSO .Xr cp 1 , .Xr cpio 1 , +.Xr ssh 1 , +.Xr scp 1 , .Xr tar 1 .Sh HISTORY The @@ -259,7 +287,16 @@ When using the .Fl H option it may not be possible for .Nm -to maintain these hard links. If this occurs +to maintain these hard links. If this occurs, .Nm will be forced to copy the file instead of link it, and thus not be able to make a perfect copy of the filesystem. +.Pp +Currently the remote protocol uses host byte order. Therefore, +.Nm +cannot talk to machines that use a byte order +different from the local machine. +.Pp +When so-called sparse files (i.e. files with "holes") are copied, +the holes will be filled in the target files, so they occupy +more physical disk space than the source files. diff --git a/bin/cpdup/cpdup.c b/bin/cpdup/cpdup.c index 20fa768401..9e176412c5 100644 --- a/bin/cpdup/cpdup.c +++ b/bin/cpdup/cpdup.c @@ -64,7 +64,6 @@ #define HLSIZE 8192 #define HLMASK (HLSIZE - 1) -#define MAXDEPTH 32 /* max copy depth for thread */ #define GETBUFSIZE 8192 #define GETPATHSIZE 2048 #define GETLINKSIZE 1024 @@ -101,12 +100,6 @@ typedef struct copy_info { char *dpath; dev_t sdevNo; dev_t ddevNo; -#ifdef USE_PTHREADS - struct copy_info *parent; - pthread_cond_t cond; - int children; - int r; -#endif } *copy_info_t; struct hlink *hltable[HLSIZE]; @@ -114,7 +107,17 @@ struct hlink *hltable[HLSIZE]; void RemoveRecur(const char *dpath, dev_t devNo); void InitList(List *list); void ResetList(List *list); +char *IterateList(List *list, Node **nodeptr, int n); int AddList(List *list, const char *name, int n); +static int getbool(const char *str); +static char *SplitRemote(char *path); +static int ChgrpAllowed(gid_t g); +static int OwnerMatch(struct stat *st1, struct stat *st2); +#ifdef _ST_FLAGS_PRESENT_ +static int FlagsMatch(struct stat *st1, struct stat *st2); +#else +#define FlagsMatch(st1, st2) 1 +#endif static struct hlink *hltlookup(struct stat *); static struct hlink *hltadd(struct stat *, const char *); static char *checkHLPath(struct stat *st, const char *spath, const char *dpath); @@ -126,14 +129,16 @@ int YesNo(const char *path); static int xrename(const char *src, const char *dst, u_long flags); static int xlink(const char *src, const char *dst, u_long flags); static int xremove(struct HostConf *host, const char *path); -int WildCmp(const char *s1, const char *s2); static int DoCopy(copy_info_t info, int depth); +static int ScanDir(List *list, struct HostConf *host, const char *path, + int64_t *CountReadBytes, int n); int AskConfirmation = 1; int SafetyOpt = 1; int ForceOpt; int DeviceOpt = 1; int VerboseOpt; +int DirShowOpt; int QuietOpt; int NoRemoveOpt; int UseMD5Opt; @@ -144,13 +149,12 @@ int SlaveOpt; int EnableDirectoryRetries; int DstBaseLen; int ValidateOpt; -int CurParallel; -int MaxParallel = -1; int HardLinkCount; int ssh_argc; const char *ssh_argv[16]; -int RunningAsUser; -int RunningAsRoot; +int DstRootPrivs; +int GroupCount; +gid_t *GroupList; const char *UseCpFile; const char *UseHLPath; const char *MD5CacheFile; @@ -168,14 +172,11 @@ int64_t CountLinkedItems; struct HostConf SrcHost; struct HostConf DstHost; -#if USE_PTHREADS -pthread_mutex_t MasterMutex; -#endif - int main(int ac, char **av) { int i; + int opt; char *src = NULL; char *dst = NULL; char *ptr; @@ -184,130 +185,112 @@ main(int ac, char **av) signal(SIGPIPE, SIG_IGN); - RunningAsUser = (geteuid() != 0); - RunningAsRoot = !RunningAsUser; - -#if USE_PTHREADS - for (i = 0; i < HCTHASH_SIZE; ++i) { - pthread_mutex_init(&SrcHost.hct_mutex[i], NULL); - pthread_mutex_init(&DstHost.hct_mutex[i], NULL); - } - pthread_mutex_init(&MasterMutex, NULL); - pthread_mutex_lock(&MasterMutex); -#endif - gettimeofday(&start, NULL); - for (i = 1; i < ac; ++i) { - int v = 1; - - ptr = av[i]; - if (*ptr != '-') { - if (src == NULL) { - src = ptr; - } else if (dst == NULL) { - dst = ptr; - } else { - fatal("too many arguments"); - /* not reached */ - } - continue; - } - ptr += 2; - - if (*ptr) - v = strtol(ptr, NULL, 0); - - switch(ptr[-1]) { + opterr = 0; + while ((opt = getopt(ac, av, ":CdF:fH:Ii:j:K:klM:mopqSs:uVvX:x")) != -1) { + switch (opt) { + /* TODO: sort the branches */ case 'C': CompressOpt = 1; break; case 'v': - VerboseOpt = 1; - while (*ptr == 'v') { - ++VerboseOpt; - ++ptr; - } - if (*ptr >= '0' && *ptr <= '9') - VerboseOpt = strtol(ptr, NULL, 0); + ++VerboseOpt; + break; + case 'd': + DirShowOpt = 1; break; case 'l': setlinebuf(stdout); setlinebuf(stderr); break; case 'V': - ValidateOpt = v; + ValidateOpt = 1; break; case 'I': - SummaryOpt = v; + SummaryOpt = 1; break; case 'o': - NoRemoveOpt = v; + NoRemoveOpt = 1; break; case 'x': UseCpFile = ".cpignore"; break; case 'X': - UseCpFile = (*ptr) ? ptr : av[++i]; + UseCpFile = optarg; break; case 'H': - UseHLPath = (*ptr) ? ptr : av[++i]; + UseHLPath = optarg; break; case 'F': if (ssh_argc >= 16) fatal("too many -F options"); - ssh_argv[ssh_argc++] = (*ptr) ? ptr : av[++i]; + ssh_argv[ssh_argc++] = optarg; break; case 'S': - SlaveOpt = v; + SlaveOpt = 1; break; case 'f': - ForceOpt = v; + ForceOpt = 1; break; case 'i': - AskConfirmation = v; + AskConfirmation = getbool(optarg); break; case 'j': - DeviceOpt = v; - break; - case 'p': - MaxParallel = v; + DeviceOpt = getbool(optarg); break; case 's': - SafetyOpt = v; + SafetyOpt = getbool(optarg); break; case 'q': - QuietOpt = v; + QuietOpt = 1; break; case 'k': - UseFSMIDOpt = v; + UseFSMIDOpt = 1; FSMIDCacheFile = ".FSMID.CHECK"; break; case 'K': - UseFSMIDOpt = v; - FSMIDCacheFile = av[++i]; + UseFSMIDOpt = 1; + FSMIDCacheFile = optarg; break; case 'M': - UseMD5Opt = v; - MD5CacheFile = av[++i]; + UseMD5Opt = 1; + MD5CacheFile = optarg; break; case 'm': - UseMD5Opt = v; + UseMD5Opt = 1; MD5CacheFile = ".MD5.CHECKSUMS"; break; case 'u': setvbuf(stdout, NULL, _IOLBF, 0); break; + case ':': + fatal("missing argument for option: -%c\n", optopt); + /* not reached */ + break; + case '?': + fatal("illegal option: -%c\n", optopt); + /* not reached */ + break; default: - fatal("illegal option: %s\n", ptr - 2); + fatal(NULL); /* not reached */ break; } } + ac -= optind; + av += optind; + if (ac > 0) + src = av[0]; + if (ac > 1) + dst = av[1]; + if (ac > 2) + fatal("too many arguments"); /* * If we are told to go into slave mode, run the HC protocol */ if (SlaveOpt) { + DstRootPrivs = (geteuid() == 0); hc_slave(0, 1); exit(0); } @@ -316,13 +299,9 @@ main(int ac, char **av) * Extract the source and/or/neither target [user@]host and * make any required connections. */ - if (src && (ptr = strchr(src, ':')) != NULL) { - asprintf(&SrcHost.host, "%*.*s", (int)(ptr - src), (int)(ptr - src), src); - src = ptr + 1; - if (UseCpFile) { - fprintf(stderr, "The cpignore options are not currently supported for remote sources\n"); - exit(1); - } + if (src && (ptr = SplitRemote(src)) != NULL) { + SrcHost.host = src; + src = ptr; if (UseMD5Opt) { fprintf(stderr, "The MD5 options are not currently supported for remote sources\n"); exit(1); @@ -330,9 +309,9 @@ main(int ac, char **av) if (hc_connect(&SrcHost) < 0) exit(1); } - if (dst && (ptr = strchr(dst, ':')) != NULL) { - asprintf(&DstHost.host, "%*.*s", (int)(ptr - dst), (int)(ptr - dst), dst); - dst = ptr + 1; + if (dst && (ptr = SplitRemote(dst)) != NULL) { + DstHost.host = dst; + dst = ptr; if (UseFSMIDOpt) { fprintf(stderr, "The FSMID options are not currently supported for remote targets\n"); exit(1); @@ -349,12 +328,21 @@ main(int ac, char **av) fatal(NULL); /* not reached */ } - bzero(&info, sizeof(info)); -#if USE_PTHREADS - info.r = 0; - info.children = 0; - pthread_cond_init(&info.cond, NULL); + + if (dst) { + DstRootPrivs = (hc_geteuid(&DstHost) == 0); + if (!DstRootPrivs) + GroupCount = hc_getgroups(&DstHost, &GroupList); + } +#if 0 + /* XXXX DEBUG */ + fprintf(stderr, "DstRootPrivs == %s\n", DstRootPrivs ? "true" : "false"); + fprintf(stderr, "GroupCount == %d\n", GroupCount); + for (i = 0; i < GroupCount; i++) + fprintf(stderr, "Group[%d] == %d\n", i, GroupList[i]); #endif + + bzero(&info, sizeof(info)); if (dst) { DstBaseLen = strlen(dst); info.spath = src; @@ -369,9 +357,6 @@ main(int ac, char **av) info.ddevNo = (dev_t)-1; i = DoCopy(&info, -1); } -#if USE_PTHREADS - pthread_cond_destroy(&info.cond); -#endif #ifndef NOMD5 md5_flush(); #endif @@ -416,35 +401,109 @@ main(int ac, char **av) exit((i == 0) ? 0 : 1); } +static int +getbool(const char *str) +{ + if (strcmp(str, "0") == 0) + return (0); + if (strcmp(str, "1") == 0) + return (1); + fatal("option requires boolean argument (0 or 1): -%c\n", optopt); + /* not reached */ + return (0); +} + +/* + * Check if path specifies a remote path, using the same syntax as scp(1), + * i.e. a path is considered remote if the first colon is not preceded by + * a slash, so e.g. "./foo:bar" is considered local. + * If a remote path is detected, the colon is replaced with a null byte, + * and the return value is a pointer to the next character. + * Otherwise NULL is returned. + */ +static char * +SplitRemote(char *path) +{ + int cindex; + + if (path[(cindex = strcspn(path, ":/"))] == ':') { + path[cindex++] = 0; + return (path + cindex); + } + return (NULL); +} + +/* + * Check if group g is in our GroupList. + * + * Typically the number of groups a user belongs to isn't large + * enough to warrant more effort than a simple linear search. + * However, we perform an optimization by moving a group to the + * top of the list when we have a hit. This assumes that there + * isn't much variance in the gids of files that a non-root user + * copies. So most of the time the search will terminate on the + * first element of the list. + */ +static int +ChgrpAllowed(gid_t g) +{ + int i; + + for (i = 0; i < GroupCount; i++) + if (GroupList[i] == g) { + if (i > 0) { + /* Optimize: Move g to the front of the list. */ + for (; i > 0; i--) + GroupList[i] = GroupList[i - 1]; + GroupList[0] = g; + } + return (1); + } + return (0); +} + +/* + * The following two functions return true if the ownership (UID + GID) + * or the flags of two files match, respectively. + * + * Only perform weak checking if we don't have sufficient privileges on + * the target machine, so we don't waste transfers with things that are + * bound to fail anyway. + */ +static int +OwnerMatch(struct stat *st1, struct stat *st2) +{ + if (DstRootPrivs) + /* Both UID and GID must match. */ + return (st1->st_uid == st2->st_uid && st1->st_gid == st2->st_gid); + else + /* Ignore UID, and also ignore GID if we can't chgrp to that group. */ + return (st1->st_gid == st2->st_gid || !ChgrpAllowed(st1->st_gid)); +} + +#ifdef _ST_FLAGS_PRESENT_ +static int +FlagsMatch(struct stat *st1, struct stat *st2) +{ + if (DstRootPrivs) + return (st1->st_flags == st2->st_flags); + else + /* Only consider the user-settable flags. */ + return (((st1->st_flags ^ st2->st_flags) & UF_SETTABLE) == 0); +} +#endif + + static struct hlink * hltlookup(struct stat *stp) { -#if USE_PTHREADS - struct timespec ts = { 0, 100000 }; -#endif struct hlink *hl; int n; n = stp->st_ino & HLMASK; -#if USE_PTHREADS -again: -#endif for (hl = hltable[n]; hl; hl = hl->next) { if (hl->ino == stp->st_ino) { -#if USE_PTHREADS - /* - * If the hl entry is still in the process of being created - * by another thread we have to wait until it has either been - * deleted or completed. - */ - if (hl->refs) { - pthread_mutex_unlock(&MasterMutex); - nanosleep(&ts, NULL); - pthread_mutex_lock(&MasterMutex); - goto again; - } -#endif ++hl->refs; return hl; } @@ -535,8 +594,8 @@ checkHLPath(struct stat *st1, const char *spath, const char *dpath) if (hc_stat(&DstHost, hpath, &sthl) < 0 || st1->st_size != sthl.st_size || st1->st_mtime != sthl.st_mtime || - (RunningAsRoot && (st1->st_uid != sthl.st_uid || - st1->st_gid != sthl.st_gid)) + !OwnerMatch(st1, &sthl) || + !FlagsMatch(st1, &sthl) ) { free(hpath); return(NULL); @@ -597,37 +656,6 @@ validate_check(const char *spath, const char *dpath) hc_close(&DstHost, fd2); return (error); } -#if USE_PTHREADS - -static void * -DoCopyThread(void *arg) -{ - copy_info_t cinfo = arg; - char *spath = cinfo->spath; - char *dpath = cinfo->dpath; - int r; - - r = pthread_detach(pthread_self()); - assert(r == 0); - pthread_cond_init(&cinfo->cond, NULL); - pthread_mutex_lock(&MasterMutex); - cinfo->r += DoCopy(cinfo, 0); - /* cinfo arguments invalid on return */ - --cinfo->parent->children; - --CurParallel; - pthread_cond_signal(&cinfo->parent->cond); - free(spath); - if (dpath) - free(dpath); - pthread_cond_destroy(&cinfo->cond); - free(cinfo); - hcc_free_trans(&SrcHost); - hcc_free_trans(&DstHost); - pthread_mutex_unlock(&MasterMutex); - return(NULL); -} - -#endif int DoCopy(copy_info_t info, int depth) @@ -641,17 +669,15 @@ DoCopy(copy_info_t info, int depth) unsigned long st2_flags; int r, mres, fres, st2Valid; struct hlink *hln; - List *list = malloc(sizeof(List)); u_int64_t size; - InitList(list); r = mres = fres = st2Valid = 0; st2_flags = 0; size = 0; hln = NULL; if (hc_lstat(&SrcHost, spath, &st1) != 0) { - r = 0; + r = 1; goto done; } #ifdef SF_SNAPSHOT @@ -759,9 +785,7 @@ relink: if ( st2Valid && st1.st_mode == st2.st_mode -#ifdef _ST_FLAGS_PRESENT_ - && (RunningAsUser || st1.st_flags == st2.st_flags) -#endif + && FlagsMatch(&st1, &st2) ) { if (S_ISLNK(st1.st_mode) || S_ISDIR(st1.st_mode)) { /* @@ -773,7 +797,7 @@ relink: (UseFSMIDOpt && (fres = fsmid_check(st1.st_fsmid, dpath)) == 0) ) { if (VerboseOpt >= 3) { - if (UseFSMIDOpt) + if (UseFSMIDOpt) /* always true!?! */ logstd("%-32s fsmid-nochange\n", (dpath ? dpath : spath)); else logstd("%-32s nochange\n", (dpath ? dpath : spath)); @@ -786,8 +810,7 @@ relink: if (ForceOpt == 0 && st1.st_size == st2.st_size && st1.st_mtime == st2.st_mtime && - (RunningAsUser || (st1.st_uid == st2.st_uid && - st1.st_gid == st2.st_gid)) + OwnerMatch(&st1, &st2) #ifndef NOMD5 && (UseMD5Opt == 0 || !S_ISREG(st1.st_mode) || (mres = md5_check(spath, dpath)) == 0) @@ -805,24 +828,18 @@ relink: */ int changedown = 0; int changedflags = 0; -#ifdef _ST_FLAGS_PRESENT_ - u_int32_t mask; -#endif if (hln) hltsetdino(hln, st2.st_ino); - if (RunningAsRoot && (st1.st_uid != st2.st_uid || - st1.st_gid != st2.st_gid)) { - hc_chown(&DstHost, dpath, st1.st_uid, st1.st_gid); - changedown = 1; + if (!OwnerMatch(&st1, &st2)) { + hc_chown(&DstHost, dpath, st1.st_uid, st1.st_gid); + changedown = 1; } #ifdef _ST_FLAGS_PRESENT_ - /* XXX also check secure-level and jail status? */ - mask = (RunningAsRoot) ? (u_int32_t)-1 : UF_SETTABLE; - if ((st1.st_flags & mask) != (st2.st_flags & mask)) { - hc_chflags(&DstHost, dpath, st1.st_flags); - changedflags = 1; + if (!FlagsMatch(&st1, &st2)) { + hc_chflags(&DstHost, dpath, st1.st_flags); + changedflags = 1; } #endif if (VerboseOpt >= 3) { @@ -877,170 +894,102 @@ relink: * The various comparisons failed, copy it. */ if (S_ISDIR(st1.st_mode)) { - DIR *dir; + int skipdir = 0; if (fres < 0) logerr("%-32s/ fsmid-CHECK-FAILED\n", (dpath) ? dpath : spath); - if ((dir = hc_opendir(&SrcHost, spath)) != NULL) { - struct dirent *den; - int noLoop = 0; - if (dpath) { - if (S_ISDIR(st2.st_mode) == 0) { + if (dpath) { + if (!st2Valid || S_ISDIR(st2.st_mode) == 0) { + if (st2Valid) xremove(&DstHost, dpath); - if (hc_mkdir(&DstHost, dpath, st1.st_mode | 0700) != 0) { - logerr("%s: mkdir failed: %s\n", - (dpath ? dpath : spath), strerror(errno)); - r = 1; - noLoop = 1; - } - - /* - * Matt: why don't you check error codes here? - * (Because I'm an idiot... checks added!) - */ - if (hc_lstat(&DstHost, dpath, &st2) != 0) { - logerr("%s: lstat of newly made dir failed: %s\n", - (dpath ? dpath : spath), strerror(errno)); - r = 1; - noLoop = 1; - } - if (hc_chown(&DstHost, dpath, st1.st_uid, st1.st_gid) != 0){ + if (hc_mkdir(&DstHost, dpath, st1.st_mode | 0700) != 0) { + logerr("%s: mkdir failed: %s\n", + (dpath ? dpath : spath), strerror(errno)); + r = 1; + skipdir = 1; + } + if (hc_lstat(&DstHost, dpath, &st2) != 0) { + logerr("%s: lstat of newly made dir failed: %s\n", + (dpath ? dpath : spath), strerror(errno)); + st2Valid = 0; + r = 1; + skipdir = 1; + } + else { + st2Valid = 1; + if (!OwnerMatch(&st1, &st2) && + hc_chown(&DstHost, dpath, st1.st_uid, st1.st_gid) != 0 + ) { logerr("%s: chown of newly made dir failed: %s\n", (dpath ? dpath : spath), strerror(errno)); r = 1; - noLoop = 1; + /* Note that we should not set skipdir = 1 here. */ } - CountCopiedItems++; - } else { - /* - * Directory must be scanable by root for cpdup to - * work. We'll fix it later if the directory isn't - * supposed to be readable ( which is why we fixup - * st2.st_mode to match what we did ). - */ - if ((st2.st_mode & 0700) != 0700) { - hc_chmod(&DstHost, dpath, st2.st_mode | 0700); - st2.st_mode |= 0700; - } - if (VerboseOpt >= 2) - logstd("%s\n", dpath ? dpath : spath); } - } - - /* - * When copying a directory, stop if the source crosses a mount - * point. - */ - if (sdevNo != (dev_t)-1 && st1.st_dev != sdevNo) { - noLoop = 1; + CountCopiedItems++; } else { - sdevNo = st1.st_dev; - } - - /* - * When copying a directory, stop if the destination crosses - * a mount point. - * - * The target directory will have been created and stat'd - * for st2 if it did not previously exist. st2Valid is left - * as a flag. If the stat failed st2 will still only have its - * default initialization. - * - * So we simply assume here that the directory is within the - * current target mount if we had to create it (aka st2Valid is 0) - * and we leave ddevNo alone. - */ - if (st2Valid) { - if (ddevNo != (dev_t)-1 && st2.st_dev != ddevNo) { - noLoop = 1; - } else { - ddevNo = st2.st_dev; - } + /* + * Directory must be scanable by root for cpdup to + * work. We'll fix it later if the directory isn't + * supposed to be readable ( which is why we fixup + * st2.st_mode to match what we did ). + */ + if ((st2.st_mode & 0700) != 0700) { + hc_chmod(&DstHost, dpath, st2.st_mode | 0700); + st2.st_mode |= 0700; + } + if (VerboseOpt >= 2) + logstd("%s\n", dpath ? dpath : spath); } + } - /* - * scan .cpignore file for files/directories - * to ignore. - */ + /* + * When copying a directory, stop if the source crosses a mount + * point. + */ + if (sdevNo != (dev_t)-1 && st1.st_dev != sdevNo) + skipdir = 1; + else + sdevNo = st1.st_dev; - if (UseCpFile) { - FILE *fi; - char *buf = malloc(GETBUFSIZE); - char *fpath; + /* + * When copying a directory, stop if the destination crosses + * a mount point. + * + * The target directory will have been created and stat'd + * for st2 if it did not previously exist. st2Valid is left + * as a flag. If the stat failed st2 will still only have its + * default initialization. + * + * So we simply assume here that the directory is within the + * current target mount if we had to create it (aka st2Valid is 0) + * and we leave ddevNo alone. + */ + if (st2Valid) { + if (ddevNo != (dev_t)-1 && st2.st_dev != ddevNo) + skipdir = 1; + else + ddevNo = st2.st_dev; + } - if (UseCpFile[0] == '/') { - fpath = mprintf("%s", UseCpFile); - } else { - fpath = mprintf("%s/%s", spath, UseCpFile); - } - AddList(list, strrchr(fpath, '/') + 1, 1); - if ((fi = fopen(fpath, "r")) != NULL) { - while (fgets(buf, GETBUFSIZE, fi) != NULL) { - int l = strlen(buf); - CountSourceReadBytes += l; - if (l && buf[l-1] == '\n') - buf[--l] = 0; - if (buf[0]) - AddList(list, buf, 1); - } - fclose(fi); - } - free(fpath); - free(buf); - } + if (!skipdir) { + List *list = malloc(sizeof(List)); + Node *node = NULL; + char *name; - /* - * Automatically exclude MD5CacheFile that we create on the - * source from the copy to the destination. - * - * Automatically exclude a FSMIDCacheFile on the source that - * would otherwise overwrite the one we maintain on the target. - */ - if (UseMD5Opt) - AddList(list, MD5CacheFile, 1); - if (UseFSMIDOpt) - AddList(list, FSMIDCacheFile, 1); + if (DirShowOpt) + logstd("Scanning %s ...\n", spath); + InitList(list); + if (ScanDir(list, &SrcHost, spath, &CountSourceReadBytes, 0) == 0) { + while ((name = IterateList(list, &node, 0)) != NULL) { + char *nspath; + char *ndpath = NULL; - while (noLoop == 0 && (den = hc_readdir(&SrcHost, dir)) != NULL) { - /* - * ignore . and .. - */ - char *nspath; - char *ndpath = NULL; + nspath = mprintf("%s/%s", spath, name); + if (dpath) + ndpath = mprintf("%s/%s", dpath, name); - if (strcmp(den->d_name, ".") == 0 || - strcmp(den->d_name, "..") == 0 - ) { - continue; - } - /* - * ignore if on .cpignore list - */ - if (AddList(list, den->d_name, 0) == 1) { - continue; - } - nspath = mprintf("%s/%s", spath, den->d_name); - if (dpath) - ndpath = mprintf("%s/%s", dpath, den->d_name); - -#if USE_PTHREADS - if (CurParallel < MaxParallel || depth > MAXDEPTH) { - copy_info_t cinfo = malloc(sizeof(*cinfo)); - pthread_t dummy_thr; - - bzero(cinfo, sizeof(*cinfo)); - cinfo->spath = nspath; - cinfo->dpath = ndpath; - cinfo->sdevNo = sdevNo; - cinfo->ddevNo = ddevNo; - cinfo->parent = info; - ++CurParallel; - ++info->children; - pthread_create(&dummy_thr, NULL, DoCopyThread, cinfo); - } else -#endif - { info->spath = nspath; info->dpath = ndpath; info->sdevNo = sdevNo; @@ -1055,77 +1004,46 @@ relink: info->spath = NULL; info->dpath = NULL; } - } - - hc_closedir(&SrcHost, dir); -#if USE_PTHREADS - /* - * Wait for our children to finish - */ - while (info->children) { - pthread_cond_wait(&info->cond, &MasterMutex); - } - r += info->r; - info->r = 0; -#endif - - /* - * Remove files/directories from destination that do not appear - * in the source. - */ - if (dpath && (dir = hc_opendir(&DstHost, dpath)) != NULL) { - while (noLoop == 0 && (den = hc_readdir(&DstHost, dir)) != NULL) { - /* - * ignore . or .. - */ - if (strcmp(den->d_name, ".") == 0 || - strcmp(den->d_name, "..") == 0 - ) { - continue; - } - /* - * If object does not exist in source or .cpignore - * then recursively remove it. - */ - if (AddList(list, den->d_name, 3) == 3) { + /* + * Remove files/directories from destination that do not appear + * in the source. + */ + if (dpath && ScanDir(list, &DstHost, dpath, + &CountTargetReadBytes, 3) == 0) { + while ((name = IterateList(list, &node, 3)) != NULL) { + /* + * If object does not exist in source or .cpignore + * then recursively remove it. + */ char *ndpath; - ndpath = mprintf("%s/%s", dpath, den->d_name); + ndpath = mprintf("%s/%s", dpath, name); RemoveRecur(ndpath, ddevNo); free(ndpath); } } - hc_closedir(&DstHost, dir); } + ResetList(list); + free(list); + } - if (dpath) { - struct timeval tv[2]; + if (dpath && st2Valid) { + struct timeval tv[2]; - if (ForceOpt || - st2Valid == 0 || - st1.st_uid != st2.st_uid || - st1.st_gid != st2.st_gid - ) { - hc_chown(&DstHost, dpath, st1.st_uid, st1.st_gid); - } - if (st2Valid == 0 || st1.st_mode != st2.st_mode) { - hc_chmod(&DstHost, dpath, st1.st_mode); - } + if (ForceOpt || !OwnerMatch(&st1, &st2)) + hc_chown(&DstHost, dpath, st1.st_uid, st1.st_gid); + if (st1.st_mode != st2.st_mode) + hc_chmod(&DstHost, dpath, st1.st_mode); #ifdef _ST_FLAGS_PRESENT_ - if (st2Valid == 0 || st1.st_flags != st2.st_flags) { - hc_chflags(&DstHost, dpath, st1.st_flags); - } + if (!FlagsMatch(&st1, &st2)) + hc_chflags(&DstHost, dpath, st1.st_flags); #endif - if (ForceOpt || - st2Valid == 0 || - st1.st_mtime != st2.st_mtime - ) { - bzero(tv, sizeof(tv)); - tv[0].tv_sec = st1.st_mtime; - tv[1].tv_sec = st1.st_mtime; - hc_utimes(&DstHost, dpath, tv); - } + if (ForceOpt || st1.st_mtime != st2.st_mtime) { + bzero(tv, sizeof(tv)); + tv[0].tv_sec = st1.st_mtime; + tv[1].tv_sec = st1.st_mtime; + hc_utimes(&DstHost, dpath, tv); } } } else if (dpath == NULL) { @@ -1174,7 +1092,7 @@ relink: * * If we can hardlink, and the target exists, we have to remove it * first or the hardlink will fail. This can occur in a number of - * situations but must typically when the '-f -H' combination is + * situations but most typically when the '-f -H' combination is * used. */ if (UseHLPath && (hpath = checkHLPath(&st1, spath, dpath)) != NULL) { @@ -1231,7 +1149,8 @@ relink: tv[0].tv_sec = st1.st_mtime; tv[1].tv_sec = st1.st_mtime; - hc_chown(&DstHost, path, st1.st_uid, st1.st_gid); + if (DstRootPrivs || ChgrpAllowed(st1.st_gid)) + hc_chown(&DstHost, path, st1.st_uid, st1.st_gid); hc_chmod(&DstHost, path, st1.st_mode); #ifdef _ST_FLAGS_PRESENT_ if (st1.st_flags & (UF_IMMUTABLE|SF_IMMUTABLE)) @@ -1248,7 +1167,7 @@ relink: if (VerboseOpt) logstd("%-32s copy-ok\n", (dpath ? dpath : spath)); #ifdef _ST_FLAGS_PRESENT_ - if (st1.st_flags) + if (DstRootPrivs ? st1.st_flags : st1.st_flags & UF_SETTABLE) hc_chflags(&DstHost, dpath, st1.st_flags); #endif } @@ -1322,7 +1241,8 @@ skip_copy: ); ++r; } else { - hc_lchown(&DstHost, path, st1.st_uid, st1.st_gid); + if (DstRootPrivs || ChgrpAllowed(st1.st_gid)) + hc_lchown(&DstHost, path, st1.st_uid, st1.st_gid); /* * there is no lchmod() or lchflags(), we * cannot chmod or chflags a softlink. @@ -1340,7 +1260,14 @@ skip_copy: } } else { if (VerboseOpt >= 3) - logstd("%-32s nochange\n", (dpath ? dpath : spath)); + logstd("%-32s nochange", (dpath ? dpath : spath)); + if (!OwnerMatch(&st1, &st2)) { + hc_lchown(&DstHost, dpath, st1.st_uid, st1.st_gid); + if (VerboseOpt >= 3) + logstd(" (uid/gid differ)"); + } + if (VerboseOpt >= 3) + logstd("\n"); } CountSourceBytes += n1; CountSourceReadBytes += n1; @@ -1361,8 +1288,7 @@ skip_copy: st2Valid == 0 || st1.st_mode != st2.st_mode || st1.st_rdev != st2.st_rdev || - st1.st_uid != st2.st_uid || - st1.st_gid != st2.st_gid + !OwnerMatch(&st1, &st2) ) { if (st2Valid) { path = mprintf("%s.tmp%d", dpath, (int)getpid()); @@ -1408,11 +1334,89 @@ done: hltrels(hln); } } - ResetList(list); - free(list); return (r); } +int +ScanDir(List *list, struct HostConf *host, const char *path, + int64_t *CountReadBytes, int n) +{ + DIR *dir; + struct dirent *den; + + if ((dir = hc_opendir(host, path)) == NULL) + return (1); + + if (n == 0) { + /* + * scan .cpignore file for files/directories to ignore + * (only in the source directory, i.e. if n == 0). + */ + if (UseCpFile) { + int fd; + int nread; + int bufused; + char *buf = malloc(GETBUFSIZE); + char *nl, *next; + char *fpath; + + if (UseCpFile[0] == '/') { + fpath = mprintf("%s", UseCpFile); + } else { + fpath = mprintf("%s/%s", path, UseCpFile); + } + AddList(list, strrchr(fpath, '/') + 1, 1); + if ((fd = hc_open(host, fpath, O_RDONLY, 0)) >= 0) { + bufused = 0; + while ((nread = hc_read(host, fd, buf + bufused, + GETIOSIZE - bufused - 1)) > 0) { + *CountReadBytes += nread; + bufused += nread; + buf[bufused] = 0; + for (next = buf; (nl = strchr(next, '\n')); next = nl+1) { + *nl = 0; + AddList(list, next, 1); + } + bufused = strlen(next); + if (bufused) + bcopy(next, buf, bufused); + } + if (bufused) { + /* last line has no trailing newline */ + buf[bufused] = 0; + AddList(list, buf, 1); + } + hc_close(host, fd); + } + free(fpath); + free(buf); + } + + /* + * Automatically exclude MD5CacheFile that we create on the + * source from the copy to the destination. + * + * Automatically exclude a FSMIDCacheFile on the source that + * would otherwise overwrite the one we maintain on the target. + */ + if (UseMD5Opt) + AddList(list, MD5CacheFile, 1); + if (UseFSMIDOpt) + AddList(list, FSMIDCacheFile, 1); + } + + while ((den = hc_readdir(host, dir)) != NULL) { + /* + * ignore . and .. + */ + if (strcmp(den->d_name, ".") != 0 && strcmp(den->d_name, "..") != 0) + AddList(list, den->d_name, n); + } + hc_closedir(host, dir); + + return (0); +} + /* * RemoveRecur() */ @@ -1515,6 +1519,23 @@ ResetList(List *list) InitList(list); } +char * +IterateList(List *list, Node **nodeptr, int n) +{ + Node *node = *nodeptr; + + if (node == NULL) + node = list->li_Node.no_Next; + while (node != &list->li_Node) { + if (node->no_Value == n) { + *nodeptr = node->no_Next; + return (node->no_Name); + } + node = node->no_Next; + } + return (NULL); +} + int AddList(List *list, const char *name, int n) { @@ -1530,7 +1551,8 @@ AddList(List *list, const char *name, int n) for (node = list->li_Hash[0]; node; node = node->no_HNext) { if (strcmp(name, node->no_Name) == 0 || - (n != 1 && node->no_Value == 1 && WildCmp(node->no_Name, name) == 0) + (n != 1 && node->no_Value == 1 && + fnmatch(node->no_Name, name, 0) == 0) ) { return(node->no_Value); } @@ -1584,54 +1606,6 @@ shash(const char *s) return(((hv >> 16) ^ hv) & HMASK); } -/* - * WildCmp() - compare wild string to sane string - * - * Return 0 on success, -1 on failure. - */ - -int -WildCmp(const char *w, const char *s) -{ - /* - * skip fixed portion - */ - - for (;;) { - switch(*w) { - case '*': - if (w[1] == 0) /* optimize wild* case */ - return(0); - { - int i; - int l = strlen(s); - - for (i = 0; i <= l; ++i) { - if (WildCmp(w + 1, s + i) == 0) - return(0); - } - } - return(-1); - case '?': - if (*s == 0) - return(-1); - ++w; - ++s; - break; - default: - if (*w != *s) - return(-1); - if (*w == 0) /* terminator */ - return(0); - ++w; - ++s; - break; - } - } - /* not reached */ - return(-1); -} - int YesNo(const char *path) { diff --git a/bin/cpdup/cpdup.h b/bin/cpdup/cpdup.h index 61ef9b7bf7..30725ca9ba 100644 --- a/bin/cpdup/cpdup.h +++ b/bin/cpdup/cpdup.h @@ -22,13 +22,11 @@ #include #include #include +#include #include #ifndef NOMD5 #include #endif -#if USE_PTHREADS -#include -#endif void logstd(const char *ctl, ...); void logerr(const char *ctl, ...); @@ -46,11 +44,10 @@ void md5_flush(void); extern const char *MD5CacheFile; extern const char *FSMIDCacheFile; +extern int QuietOpt; extern int SummaryOpt; extern int CompressOpt; -extern int CurParallel; -extern int RunningAsUser; -extern int RunningAsRoot; +extern int DstRootPrivs; extern int ssh_argc; extern const char *ssh_argv[]; @@ -63,10 +60,6 @@ extern int64_t CountTargetReadBytes; extern int64_t CountWriteBytes; extern int64_t CountRemovedItems; -#if USE_PTHREADS -extern pthread_mutex_t MasterMutex; -#endif - #ifdef DEBUG_MALLOC void *debug_malloc(size_t bytes, const char *file, int line); void debug_free(void *ptr); diff --git a/bin/cpdup/hclink.c b/bin/cpdup/hclink.c index 45e56e42c9..787f014e9f 100644 --- a/bin/cpdup/hclink.c +++ b/bin/cpdup/hclink.c @@ -10,9 +10,6 @@ #include "hclink.h" #include "hcproto.h" -#if USE_PTHREADS -static void * hcc_reader_thread(void *arg); -#endif static struct HCHead *hcc_read_command(struct HostConf *hc, hctransaction_t trans); static void hcc_start_reply(hctransaction_t trans, struct HCHead *rhead); @@ -71,9 +68,6 @@ hcc_connect(struct HostConf *hc) hc->fdin = fdin[0]; close(fdout[0]); hc->fdout = fdout[1]; -#if USE_PTHREADS - pthread_create(&hc->reader_thread, NULL, hcc_reader_thread, hc); -#endif return(0); } } @@ -99,6 +93,7 @@ hcc_slave(int fdin, int fdout, struct HCDesc *descs, int count) bzero(&hcslave, sizeof(hcslave)); bzero(&trans, sizeof(trans)); + bzero(dispatch, sizeof(dispatch)); for (i = 0; i < count; ++i) { struct HCDesc *desc = &descs[i]; assert(desc->cmd >= 0 && desc->cmd < 256); @@ -112,9 +107,6 @@ hcc_slave(int fdin, int fdout, struct HCDesc *descs, int count) hcslave.fdout = fdout; trans.hc = &hcslave; -#if USE_PTHREADS - pthread_mutex_unlock(&MasterMutex); -#endif /* * Process commands on fdin and write out results on fdout */ @@ -164,47 +156,6 @@ hcc_slave(int fdin, int fdout, struct HCDesc *descs, int count) return(0); } -#if USE_PTHREADS -/* - * This thread collects responses from the link. It is run without - * the MasterMutex. - */ -static void * -hcc_reader_thread(void *arg) -{ - struct HostConf *hc = arg; - struct HCHead *rhead; - hctransaction_t scan; - int i; - - pthread_detach(pthread_self()); - while (hcc_read_command(hc, NULL) != NULL) - ; - hc->reader_thread = NULL; - - /* - * Clean up any threads stuck waiting for a reply. - */ - pthread_mutex_lock(&MasterMutex); - for (i = 0; i < HCTHASH_SIZE; ++i) { - pthread_mutex_lock(&hc->hct_mutex[i]); - for (scan = hc->hct_hash[i]; scan; scan = scan->next) { - if (scan->state == HCT_SENT) { - scan->state = HCT_REPLIED; - rhead = (void *)scan->rbuf; - rhead->error = ENOTCONN; - if (scan->waiting) - pthread_cond_signal(&scan->cond); - } - } - pthread_mutex_unlock(&hc->hct_mutex[i]); - } - pthread_mutex_unlock(&MasterMutex); - return(NULL); -} - -#endif - /* * This reads a command from fdin, fixes up the byte ordering, and returns * a pointer to HCHead. @@ -236,24 +187,9 @@ hcc_read_command(struct HostConf *hc, hctransaction_t trans) if (trans) { fill = trans; } else { -#if USE_PTHREADS - pthread_mutex_lock(&hc->hct_mutex[tmp.id & HCTHASH_MASK]); - for (fill = hc->hct_hash[tmp.id & HCTHASH_MASK]; - fill; - fill = fill->next) - { - if (fill->state == HCT_SENT && fill->id == tmp.id) - break; - } - pthread_mutex_unlock(&hc->hct_mutex[tmp.id & HCTHASH_MASK]); - if (fill == NULL) -#endif - { - fprintf(stderr, - "cpdup hlink protocol error with %s (%04x)\n", - hc->host, tmp.id); - exit(1); - } + fprintf(stderr, "cpdup hlink protocol error with %s (%04x)\n", + hc->host, tmp.id); + exit(1); } bcopy(&tmp, fill->rbuf, n); @@ -267,86 +203,13 @@ hcc_read_command(struct HostConf *hc, hctransaction_t trans) } #ifdef DEBUG hcc_debug_dump(head); -#endif -#if USE_PTHREADS - pthread_mutex_lock(&hc->hct_mutex[fill->id & HCTHASH_MASK]); #endif fill->state = HCT_REPLIED; -#if USE_PTHREADS - if (fill->waiting) - pthread_cond_signal(&fill->cond); - pthread_mutex_unlock(&hc->hct_mutex[fill->id & HCTHASH_MASK]); -#endif return((void *)fill->rbuf); fail: return(NULL); } -#if USE_PTHREADS - -static -hctransaction_t -hcc_get_trans(struct HostConf *hc) -{ - hctransaction_t trans; - hctransaction_t scan; - pthread_t tid = pthread_self(); - int i; - - i = ((intptr_t)tid >> 7) & HCTHASH_MASK; - - pthread_mutex_lock(&hc->hct_mutex[i]); - for (trans = hc->hct_hash[i]; trans; trans = trans->next) { - if (trans->tid == tid) - break; - } - if (trans == NULL) { - trans = malloc(sizeof(*trans)); - bzero(trans, sizeof(*trans)); - trans->tid = tid; - trans->id = i; - pthread_cond_init(&trans->cond, NULL); - do { - for (scan = hc->hct_hash[i]; scan; scan = scan->next) { - if (scan->id == trans->id) { - trans->id += HCTHASH_SIZE; - break; - } - } - } while (scan != NULL); - - trans->next = hc->hct_hash[i]; - hc->hct_hash[i] = trans; - } - pthread_mutex_unlock(&hc->hct_mutex[i]); - return(trans); -} - -void -hcc_free_trans(struct HostConf *hc) -{ - hctransaction_t trans; - hctransaction_t *transp; - pthread_t tid = pthread_self(); - int i; - - i = ((intptr_t)tid >> 7) & HCTHASH_MASK; - - pthread_mutex_lock(&hc->hct_mutex[i]); - for (transp = &hc->hct_hash[i]; *transp; transp = &trans->next) { - trans = *transp; - if (trans->tid == tid) { - *transp = trans->next; - pthread_cond_destroy(&trans->cond); - free(trans); - break; - } - } - pthread_mutex_unlock(&hc->hct_mutex[i]); -} - -#else - static hctransaction_t hcc_get_trans(struct HostConf *hc) @@ -360,8 +223,6 @@ hcc_free_trans(struct HostConf *hc __unused) /* nop */ } -#endif - /* * Initialize for a new command */ @@ -439,22 +300,7 @@ hcc_finish_command(hctransaction_t trans) * whead is invalid when we call hcc_read_command() because * we may switch to another thread. */ -#if USE_PTHREADS - pthread_mutex_unlock(&MasterMutex); - while (trans->state != HCT_REPLIED && hc->reader_thread) { - pthread_mutex_t *mtxp = &hc->hct_mutex[trans->id & HCTHASH_MASK]; - pthread_mutex_lock(mtxp); - trans->waiting = 1; - if (trans->state != HCT_REPLIED && hc->reader_thread) - pthread_cond_wait(&trans->cond, mtxp); - trans->waiting = 0; - pthread_mutex_unlock(mtxp); - } - pthread_mutex_lock(&MasterMutex); - rhead = (void *)trans->rbuf; -#else rhead = hcc_read_command(hc, trans); -#endif if (trans->state != HCT_REPLIED || rhead->id != trans->id) { #ifdef __error *__error = EIO; diff --git a/bin/cpdup/hclink.h b/bin/cpdup/hclink.h index 2fdf2bf0df..0428ee3c5f 100644 --- a/bin/cpdup/hclink.h +++ b/bin/cpdup/hclink.h @@ -22,20 +22,10 @@ typedef struct HCTransaction { u_int16_t id; /* assigned transaction id */ int windex; /* output buffer index */ enum { HCT_IDLE, HCT_SENT, HCT_REPLIED, HCT_DONE } state; -#if USE_PTHREADS - pthread_t tid; - pthread_cond_t cond; - int waiting; -#endif char rbuf[65536]; /* input buffer */ char wbuf[65536]; /* output buffer */ } *hctransaction_t; -#if USE_PTHREADS -#define HCTHASH_SIZE 16 -#define HCTHASH_MASK (HCTHASH_SIZE - 1) -#endif - struct HostConf { char *host; /* [user@]host */ int fdin; /* pipe */ @@ -44,20 +34,14 @@ struct HostConf { pid_t pid; int version; /* cpdup protocol version */ struct HCHostDesc *hostdescs; -#if USE_PTHREADS - pthread_mutex_t hct_mutex[HCTHASH_SIZE]; - hctransaction_t hct_hash[HCTHASH_SIZE]; - pthread_t reader_thread; -#else struct HCTransaction trans; -#endif }; struct HCHead { int32_t magic; /* magic number / byte ordering */ int32_t bytes; /* size of packet */ int16_t cmd; /* command code */ - u_int16_t id; /* transaction id */ + u_int16_t id; /* transaction id */ int32_t error; /* error code (response) */ }; diff --git a/bin/cpdup/hcproto.c b/bin/cpdup/hcproto.c index a9e4207c5d..5acb6ddb00 100644 --- a/bin/cpdup/hcproto.c +++ b/bin/cpdup/hcproto.c @@ -39,6 +39,10 @@ static int rc_umask(hctransaction_t trans, struct HCHead *); static int rc_symlink(hctransaction_t trans, struct HCHead *); static int rc_rename(hctransaction_t trans, struct HCHead *); static int rc_utimes(hctransaction_t trans, struct HCHead *); +static int rc_geteuid(hctransaction_t trans, struct HCHead *); +static int rc_getgroups(hctransaction_t trans, struct HCHead *); + +static int getmygroups(gid_t **gidlist); struct HCDesc HCDispatchTable[] = { { HC_HELLO, rc_hello }, @@ -67,6 +71,8 @@ struct HCDesc HCDispatchTable[] = { { HC_SYMLINK, rc_symlink }, { HC_RENAME, rc_rename }, { HC_UTIMES, rc_utimes }, + { HC_GETEUID, rc_geteuid }, + { HC_GETGROUPS, rc_getgroups }, }; static int chown_warning; @@ -82,9 +88,9 @@ silentwarning(int *didwarn, const char *ctl, ...) { va_list va; - if (RunningAsRoot) + if (DstRootPrivs) return(-1); - if (*didwarn == 0) { + if (*didwarn == 0 && QuietOpt == 0) { *didwarn = 1; fprintf(stderr, "WARNING: Not running as root, "); va_start(va, ctl); @@ -149,7 +155,8 @@ hc_hello(struct HostConf *hc) for (item = hcc_firstitem(head); item; item = hcc_nextitem(head, item)) { switch(item->leafid) { case LC_HELLOSTR: - fprintf(stderr, "Handshaked with %s\n", HCC_STRING(item)); + if (QuietOpt == 0) + fprintf(stderr, "Handshaked with %s\n", HCC_STRING(item)); error = 0; break; case LC_VERSION: @@ -702,17 +709,7 @@ rc_close(hctransaction_t trans, struct HCHead *head) static int getiolimit(void) { -#if USE_PTHREADS - if (CurParallel < 2) - return(32768); - if (CurParallel < 4) - return(16384); - if (CurParallel < 8) - return(8192); - return(4096); -#else return(32768); -#endif } /* @@ -1023,6 +1020,9 @@ hc_chown(struct HostConf *hc, const char *path, uid_t owner, gid_t group) struct HCHead *head; int rc; + if (!DstRootPrivs) + owner = -1; + if (hc == NULL || hc->host == NULL) { rc = chown(path, owner, group); if (rc < 0) @@ -1081,6 +1081,9 @@ hc_lchown(struct HostConf *hc, const char *path, uid_t owner, gid_t group) struct HCHead *head; int rc; + if (!DstRootPrivs) + owner = -1; + if (hc == NULL || hc->host == NULL) { rc = lchown(path, owner, group); if (rc < 0) @@ -1182,6 +1185,12 @@ hc_mknod(struct HostConf *hc, const char *path, mode_t mode, dev_t rdev) hctransaction_t trans; struct HCHead *head; + if (!DstRootPrivs) { + /* mknod() requires root privs, so don't bother. */ + errno = EPERM; + return (-1); + } + if (hc == NULL || hc->host == NULL) return(mknod(path, mode, rdev)); @@ -1277,16 +1286,12 @@ hc_chflags(struct HostConf *hc, const char *path, u_long flags) struct HCHead *head; int rc; + if (!DstRootPrivs) + flags &= UF_SETTABLE; + if (hc == NULL || hc->host == NULL) { - rc = chflags(path, flags); - if (rc < 0) { - if (RunningAsUser) { - flags &= UF_SETTABLE; - rc = chflags(path, flags); - } - if (rc < 0) - rc = silentwarning(&chflags_warning, "file flags may differ\n"); - } + if ((rc = chflags(path, flags)) < 0) + rc = silentwarning(&chflags_warning, "file flags may differ\n"); return (rc); } @@ -1320,15 +1325,8 @@ rc_chflags(hctransaction_t trans __unused, struct HCHead *head) } if (path == NULL) return(-2); - rc = chflags(path, flags); - if (rc < 0) { - if (RunningAsUser) { - flags &= UF_SETTABLE; - rc = chflags(path, flags); - } - if (rc < 0) - rc = silentwarning(&chflags_warning, "file flags may differ\n"); - } + if ((rc = chflags(path, flags)) < 0) + rc = silentwarning(&chflags_warning, "file flags may differ\n"); return(rc); } @@ -1582,3 +1580,119 @@ rc_utimes(hctransaction_t trans __unused, struct HCHead *head) return(-2); return(utimes(path, times)); } + +uid_t +hc_geteuid(struct HostConf *hc) +{ + hctransaction_t trans; + struct HCHead *head; + struct HCLeaf *item; + + if (hc == NULL || hc->host == NULL) + return (geteuid()); + + if (hc->version < 3) { + fprintf(stderr, "WARNING: Remote client uses old protocol version\n"); + /* Return 0 on error, so the caller assumes root privileges. */ + return (0); + } + + trans = hcc_start_command(hc, HC_GETEUID); + if ((head = hcc_finish_command(trans)) == NULL || head->error) + return(0); + for (item = hcc_firstitem(head); item; item = hcc_nextitem(head, item)) { + if (item->leafid == LC_UID) + return (HCC_INT32(item)); + } + return(0); /* shouldn't happen */ +} + +static int +rc_geteuid(hctransaction_t trans, struct HCHead *head __unused) +{ + hcc_leaf_int32(trans, LC_UID, geteuid()); + return (0); +} + +static int +getmygroups(gid_t **gidlist) +{ + int count; + + if ((count = getgroups(0, *gidlist)) > 0) { + if ((*gidlist = malloc(count * sizeof(gid_t))) != NULL) { + if ((count = getgroups(count, *gidlist)) <= 0) + free(*gidlist); + } + else + count = -1; + } + else + *gidlist = NULL; + return (count); +} + +int +hc_getgroups(struct HostConf *hc, gid_t **gidlist) +{ + int count, i; + hctransaction_t trans; + struct HCHead *head; + struct HCLeaf *item; + + if (hc == NULL || hc->host == NULL) + return (getmygroups(gidlist)); + + i = 0; + count = 0; + *gidlist = NULL; + + if (hc->version < 3) { + fprintf(stderr, "WARNING: Remote client uses old protocol version\n"); + return (-1); + } + + trans = hcc_start_command(hc, HC_GETGROUPS); + if ((head = hcc_finish_command(trans)) == NULL || head->error) + return(-1); + for (item = hcc_firstitem(head); item; item = hcc_nextitem(head, item)) { + switch(item->leafid) { + case LC_COUNT: + count = HCC_INT32(item); + if (*gidlist != NULL) { /* protocol error */ + free(*gidlist); + *gidlist = NULL; + return (-1); + } + if ((*gidlist = malloc(count * sizeof(gid_t))) == NULL) + return (-1); + break; + case LC_GID: + if (*gidlist == NULL || i >= count) { /* protocol error */ + if (*gidlist != NULL) + free(*gidlist); + *gidlist = NULL; + return (-1); + } + (*gidlist)[i++] = HCC_INT32(item); + break; + } + } + return (count); +} + +static int +rc_getgroups(hctransaction_t trans, struct HCHead *head __unused) +{ + int count, i; + gid_t *gidlist; + + if ((count = getmygroups(&gidlist)) < 0) + return (-1); + hcc_leaf_int32(trans, LC_COUNT, count); + for (i = 0; i < count; i++) + hcc_leaf_int32(trans, LC_GID, gidlist[i]); + if (gidlist != NULL) + free(gidlist); + return (0); +} diff --git a/bin/cpdup/hcproto.h b/bin/cpdup/hcproto.h index adf3d825e0..6dffcf7567 100644 --- a/bin/cpdup/hcproto.h +++ b/bin/cpdup/hcproto.h @@ -7,7 +7,7 @@ #ifndef _HCPROTO_H_ #define _HCPROTO_H_ -#define HCPROTO_VERSION 2 +#define HCPROTO_VERSION 3 #define HCPROTO_VERSION_COMPAT 2 #define HC_HELLO 0x0001 @@ -35,6 +35,8 @@ #define HC_RENAME 0x0024 #define HC_UTIMES 0x0025 #define HC_MKNOD 0x0026 +#define HC_GETEUID 0x0027 +#define HC_GETGROUPS 0x0028 #define LC_HELLOSTR (0x0001|LCF_STRING) #define LC_PATH1 (0x0010|LCF_STRING) @@ -62,6 +64,7 @@ #define LC_TYPE (0x0026|LCF_INT32) #define LC_BLKSIZE (0x0027|LCF_INT32) #define LC_VERSION (0x0028|LCF_INT32) +#define LC_COUNT (0x0029|LCF_INT32) #define XO_NATIVEMASK 3 /* passed through directly */ #define XO_CREAT 0x00010000 @@ -98,6 +101,8 @@ mode_t hc_umask(struct HostConf *hc, mode_t numask); int hc_symlink(struct HostConf *hc, const char *name1, const char *name2); int hc_rename(struct HostConf *hc, const char *name1, const char *name2); int hc_utimes(struct HostConf *hc, const char *path, const struct timeval *times); +uid_t hc_geteuid(struct HostConf *hc); +int hc_getgroups(struct HostConf *hc, gid_t **gidlist); #endif diff --git a/bin/cpdup/misc.c b/bin/cpdup/misc.c index 098f051379..d3e848671b 100644 --- a/bin/cpdup/misc.c +++ b/bin/cpdup/misc.c @@ -155,19 +155,22 @@ fatal(const char *ctl, ...) puts("cpdup [] src [dest]"); puts(" -C request compressed ssh link if remote operation\n" " -v[vv] verbose level (-vv is typical)\n" + " -d print directories being traversed\n" " -u use unbuffered output for -v[vv]\n" " -I display performance summary\n" " -f force update even if files look the same\n" " -F Add to options passed to ssh\n" " -i0 do NOT confirm when removing something\n" + " -j0 do not try to recreate CHR or BLK devices\n" " -l force line-buffered stdout/stderr\n" - " -pN N parallel transactions for for remote\n" - " source or destination\n" " -s0 disable safeties - allow files to overwrite directories\n" " -q quiet operation\n" " -o do not remove any files, just overwrite/add\n" ); puts( + " -k maintain/generate FSMID checkfile on target,\n" + " and compare source FSMIDs against the checkfiles\n" + " -K file -k+specify FSMID checkfile, else .FSMID.CHECK\n" #ifndef NOMD5 " -m maintain/generate MD5 checkfile on source,\n" " and compare with (optional) destination,\n" @@ -181,7 +184,7 @@ fatal(const char *ctl, ...) #endif " -x use .cpignore as exclusion file\n" " -X file specify exclusion file\n" - " Version 1.15 by Matt Dillon and Dima Ruban\n" + " Version 1.16 by Matt Dillon and Dima Ruban\n" ); exit(0); } else { -- 2.41.0