![[BACK]](/cvsweb/icons/back.gif){kind=icon}
Return to fm.ms CVS log ![[TXT]](/cvsweb/icons/text.gif){kind=icon}
![[DIR]](/cvsweb/icons/dir.gif){kind=icon}
Up to [Research Unix] / researchv10dc / vol2 / fm|
Return to fm.ms CVS log | Up to [Research Unix] / researchv10dc / vol2 / fm |
1.1 root 1: .so ../ADM/mac
2: .XX backup 593 "The File Motel: An Owner's Manual"
3: .nr dP 2
4: .nr dV 3p
5: .TL
6: The File Motel:
7: .br
8: An Owner's Manual
9: .AU
10: Andrew G. Hume
11: .AI
12: .MH
13: .AB
14: .PP
15: The File Motel is an incremental user-level file backup system for
16: .UX
17: systems.
18: The first version of the File Motel has been in successful operation
19: for over two years with three sites supporting about 50 systems.
20: The first version supported only Ninth Edition
21: .UX
22: systems, although with only modest inconvenience
23: files could be saved from Sun 3 clients.
24: The second version of the File Motel is a complete reworking
25: of the original system, emphasizing easy portability to most
26: .UX
27: systems.
28: The files are stored in a machine-independent form;
29: as an example, I have recovered a directory onto a Sun 3
30: from a server on a MIPS 120/5 that had been originally
31: saved from a Cray X/MP\-24.
32: The user and administrative interfaces have been streamlined,
33: based on experience in the field.
34: .PP
35: The system has been restructured to look like a kit.
36: Most of the modules, such as the database, networking and media code,
37: have been isolated via simple interface routines.
38: As with a kit, you may not find exactly what you want,
39: but it should be easy to roll your own.
40: .AE
41: .2C
42: .NH
43: An Overview
44: .PP
45: This is a manual for the File Motel |reference(file motel usenix),
46: a backup system for
47: .UX
48: systems.
49: The File Motel consists of a central server system
50: servicing many client systems.
51: The server system is almost always also a client system.
52: The File Motel saves only the files that change on any given client system,
53: using a database to record what versions
54: have been saved for any particular file.
55: Under normal usage patterns, this is on the order of 1\-5% of the user files
56: on the client.
57: This makes backup practical over slow networks to slow backup media.
58: .PP
59: The daily routine in the File Motel starts around midnight when
60: the clients send copies of any new or recently modified files to the server machine.
61: After receiving the files from all the clients,
62: a separate processing step transforms the received files into
63: backup copies, which are then written to the backup media
64: of your choice (typically, WORM disks).
65: Backup and recovery can be performed by anyone with the appropriate permissions;
66: in general, there is no administrative overhead other than
67: fussing with backup media.
68: .PP
69: This description may be a little clearer with some details.
70: The following description includes
71: some sample numbers from our Center's File Motel;
72: other sites will differ.
73: The first step is a client sending files to the server.
74: A shell script configured for the client generates a list of (say 5000)
75: candidates for backup (say, all the files changed in the last week).
76: This list is sent to the server which returns a list of the files (say 900)
77: that really need to be backed up.
78: Each of these files is then transmitted to the server together with a header
79: (which includes a checksum).
80: (On average, about 15MB is sent taking about 20 minutes.)
81: There is an acknowledgement from the server after every file;
82: this allows graceful termination when the server has problems,
83: such as running out of space.
84: The redundancy in the candidate list allows non-critical clients
85: to cope with transient faults (such as a broken network) without
86: administrative intervention by ignoring the fault
87: and getting the files the following night.
88: The client process is normally initiated either by the server machine
89: or by
90: .I cron (8).
91: Exactly the same mechanism is used for user-initiated backups;
92: the only difference is that the system backup is executed by the super-user
93: (and thus has access to all the files on the client system).
94: .PP
95: The files sent by the clients are kept in receiving areas each of
96: 32 subdirectories, each processed in turn.
97: First, the program
98: .I sweep
99: deletes any unnecessary files, assigns each backup copy a name
100: (which is stored in the file's header) and recalculates the file's checksum.
101: The remaining files are fed to
102: .I dbupdate
103: which deletes any unnecessary files and stores the version information
104: in the database.
105: Finally, the surviving files are moved to a staging area for writing to the
106: backup media.
107: .PP
108: The last step is writing the backup copies to the backup media.
109: The only medium currently supported is a WORM disk.
110: In our environment they are preferred because of their large capacity
111: and because you can get reliable jukeboxes (automatic disk changers).
112: Optical jukeboxes come in all sorts of sizes; our center has a
113: SONY WDA 3000-10 with a total capacity of 164GB (328GB after October 1989).
114: .PP
115: There are many other programs in the File Motel,
116: some intended for the user (for example, recovering files),
117: and others for the administrator (usage statistics, backing up the database).
118: .PP
119: The rest of this manual is intended for the caretaker of the File Motel.
120: Section 2 details some of the
121: peculiar aspects of the File Motel that have caused problems in the past;
122: if you can survive these,
123: then installing and running the File Motel
124: ought to be easy.
125: If there are incompatibilities, then installing the File Motel
126: will require (perhaps substantial) work.
127: The File Motel uses many small single-purpose tools;
128: if you need to figure out what is going on (or wrong, as the case may be),
129: these tools are described in Section 3.
130: Sections 4 and 5 are step by step instructions for installing
131: a client and a server respectively.
132: Finally, Section 6 elaborates on media management.
133: .NH
134: Some Things You Should Know
135: .PP
136: .de BL
137: .IP \ \ \ \ \s+3\(bu\s-3
138: ..
139: This section describes some of the assumptions underlying the
140: construction of the File Motel software.
141: Most of these assumptions have caused problems in porting to
142: systems less hospitable than 10th Edition
143: .UX
144: (or V10 for short).
145: .BL
146: Each server has one global name space for all the files saved from all the clients.
147: The file
148: .I z
149: from machine
150: .I mach
151: is stored under the name
152: .I /n/mach/z .
153: It so happens that this is how V10 networked file systems are normally mounted
154: and in fact, all file references actually go through this network name.
155: For other systems, you should define the
156: .CW -DNO_NETNAME
157: switch as described in section 5.
158: .BL
159: All client\-service communication uses a uniform networking interface.
160: That is, a system invokes a service on the remote machine and gets a
161: pair of file descriptors attached to the input and output of that service.
162: Both Berkeley-style sockets and V10-style IPC are supported.
163: For the case of a single system that is both a server and client
164: and has no networking, you will have to write an execution service that
165: constructs pipes to the desired services.
166: Note that it is possible to provide this interface even if all
167: the networking you have is a user program (such as
168: .I rx
169: or
170: .I rsh )
171: that executes a program on another machine.
172: .BL
173: It must be possible to nominate the user that the remote service runs as.
174: Most run as a regular user, say
175: .I fmdaemon ,
176: but some must run as superuser and on V10 systems, one must run as
177: .I bin .
178: .BL
179: The code is reasonably portable; with the canned configuration files
180: it runs on a Cray X-MP/24 (UNICOS),
181: VAX 11/750, Microvax II, 8550, 8600 and 11/780 (V10, Ultrix, 4.3BSD),
182: Sun 3 (SunOs 4.0),
183: MIPS M120/5, M2000 (UMIPS 3.0, 3.10, RISC/os 4.0).
184: Some of the code, notably Ken Thompson's new version of
185: .CW doprint ,
186: makes assumptions about variable argument lists.
187: So far, the code has continued to work on all the systems we have tried
188: (although we can't optimise on the MIPS)
189: but in this world of perverse hardware and compilers, you may not be so lucky.
190: .BL
191: The code assumes no particular byte-ordering but does assume that there
192: is an integer type of at least 32 bits.
193: By and large, the programs allocate all data areas dynamically;
194: whenever there is choice, programs trade space for smaller runtime,
195: so there must be at least 24 bits of data space.
196: If you have a 32 bit machine but have 16 bit ints,
197: you will have trouble (perhaps
198: .I lint
199: will help).
200: .BL
201: It is assumed that the backup medium can hold at least one backup copy
202: and practically, it should hold at least one volume.
203: This is about 20MB by default; if you have smaller backup media
204: and cannot arrange better, change the volume size \(em it is
205: a constant defined in
206: .CW fm/sweep.c .
207: .BL
208: The File Motel depends on each file having a unique name.
209: This continue to cause problems, particularly in the presence of symbolic links.
210: For example, on a system I use
211: .CW /usr/andrew
212: is a symbolic link
213: .CW /usr2/guest/andrew .
214: The right thing to do is to save files under
215: .CW /usr/andrew
216: (so that you can move them from file system to file system and keep their name).
217: Yet, the user may not be aware of this name; if they do a
218: .I pwd
219: to find out, they will get the wrong answer.
220: .NH
221: A Detailed Description
222: .PP
223: The action in the File Motel can be functionally divided into four areas:
224: client selecting and sending files to the server,
225: the server processing the client files onto backup media,
226: client recovering files back from the server,
227: and an assortment of administrative functions.
228: .PP
229: Programs and scripts used by the File Motel live in three places:
230: .CW /usr/bin/backup
231: is the user interface,
232: .CW /usr/lib/filemotel
233: holds all the programs and scripts used by clients,
234: and
235: .CW /usr/filemotel/bin
236: holds the server-specific programs.
237: These are the conventional names \(em they can be reconfigured to taste.
238: Because of this and their length,
239: these abbreviations will be used in the following text:
240: .TS
241: center;
242: lFCW lFCW.
243: $FM /usr/filemotel
244: $FB /usr/filemotel/bin
245: $FL /usr/lib/filemotel
246: .TE
247: .NH 2
248: Client Sends Files to the Server
249: .PP
250: The controlling script here is
251: .CW $FL/doclient :
252: .P1
253: #!/bin/sh
254: $FL/sel | $FL/act
255: .P2
256: The selection script
257: .I sel
258: has to generate a list of absolute filenames.
259: You can use any tools available to you; the File Motel supplies
260: the program
261: .I fcheck
262: which is rather more efficient than
263: .I find (1)
264: and follows symbolic links that are arguments.
265: This is to help clients save files as
266: .CW /usr/andrew/...
267: rather than the less than informative
268: .CW /usr2/guest/andrew/... .
269: A small
270: .I sel
271: file is shown below.
272: .KF
273: .P1 0
274: /usr/lib/filemotel/fcheck 512 7 /etc /usr/* |
275: sed -e '/\e.o$/d
276: /\e/a\e.out$/d
277: /\e/core$/d
278: /\e/foo$/d
279: /^\e/usr\e/tmp\e//d
280: /^\e/usr\e/spool\e//d'
281: cat <<EOF
282: /unix
283: EOF
284: .P2
285: .KE
286: .PP
287: The script
288: .I act
289: works in a straightforward way.
290: First, the filenames are transformed into the input format for
291: .I missing
292: by the program
293: .I iprint .
294: This prepends
295: .CW /n/\fImachine
296: to the filename (unless this is already there) and appends the
297: inode change time and size.
298: There is a convention that an input filename starting with a
299: .CW //
300: is a symbolic link to be followed (that is, use
301: .I stat (2)
302: rather than
303: .I lstat (2)
304: to get the time and size).
305: The size is carried around so that if you choose a file because it is small
306: and it grows dramatically while you are asking about it, you can reject it
307: later on (although this is not done now because no one cares yet).
308: .I Missing
309: takes these names and ships them to the corresponding server
310: .I missing_
311: on the server machine.
312: (Servers for a service
313: .CW abc
314: are called
315: .CW abc_ ).
316: .I Missing_
317: checks the name,time tuples against the database and sends back
318: the lines that are newer than the entry in the database.
319: Transmissions in both directions are checksummed; any errors
320: are reported to standard error and are also logged in the log file
321: on the server machine.
322: .PP
323: The results from
324: .I missing
325: are stored in
326: .CW $FL/files.\fIday\fP .
327: They are given to
328: .I fmpush
329: which actually pushes them to the backup system.
330: .I Fmpush
331: also takes a system name argument for logging purposes.
332: If there are any errors,
333: .I fmpush
334: reports the error and the number of files transmitted.
335: This allows the push to be restarted efficiently:
336: .P1
337: $ pwd
338: $FL
339: $ fmpush wild < files.Tue
340: EOF after 2713 files sent.
341: $ sed 1,2713d files.Tue | fmpush wild
342: .P2
343: Any diagnostics are mailed to the user
344: .I backup
345: and also kept in
346: .CW $FL/files.\fIday\fP.sho .
347: It is not necessary to keep these files around after they have been used
348: but they are relatively small and often useful;
349: for example, a client who normally saves 100 or so files suddenly sends
350: you 10,000 files \(em you can quickly go to that client and check
351: what the files were.
352: When possible, diagnostics are also logged on the backup system.
353: .PP
354: Only regular files, symbolic links and directories have their contents saved;
355: all other files (such as devices) just have their
356: .I stat (2)
357: buffers saved.
358: To preserve machine independence, the content of a directory is saved
359: as a list of null-terminated element names.
360: This removes the need for the server to be able to guess a
361: client's directory structure, although it does lose a small
362: amount of subtle information contained in the freed slots of the
363: directory.
364: .........
365: .NH 2
366: Server Processes Client's Files
367: .PP
368: Received files are processed by the script
369: .CW $FL/munge .
370: This processing is decoupled from either receiving or restoring
371: client files; for example,
372: it is safe to process files while receiving them.
373: Munging is typically started by
374: .I cron ,
375: but you can also cause
376: .I rcv
377: to invoke
378: .I munge
379: automatically,
380: and you can invoke
381: .I munge
382: manually by executing
383: .CW $FL/callmunge .
384: .PP
385: Regardless of how it is called,
386: .I munge
387: scans the 32 receiving subdirectories in each of the receiving areas in
388: .CW $FM/adm/rcvdirs
389: looking for files to process.
390: If it finds any, it calls a program
391: whose name is supplied as
392: .CW $PROCPERM
393: to copy the final copies
394: to the media of your choice.
395: It repeats this scan until it found nothing to do during the last scan.
396: .PP
397: The action within a subdirectory is simple.
398: .CW $FB/sweep
399: looks for files with mode
400: .CW 0 ,
401: .CW 0400 ,
402: or
403: .CW 0600 .
404: Mode
405: .CW 0
406: files are files that are being received
407: (\fIrcv\fP
408: marks a file as done by changing its mode to
409: .CW 0600 )
410: and are ignored unless it is hasn't been modified within the
411: last 12 hours.
412: In this latter case, it is regarded as stale (almost always a network
413: connection was dropped) and unlinked.
414: Mode
415: .CW 0400
416: files have been already processed by
417: .I sweep
418: but for some reason (most often, running out of space)
419: weren't copied to the backup area.
420: Mode
421: .CW 0600
422: files are assigned a backup copy name and after recalculating the
423: checksum are changed to mode
424: .CW 0400 .
425: .I Sweep
426: emits the names of all the files ready to be copied to the backup area
427: and this is saved in a file.
428: .I Munge
429: then makes any needed directories in the backup area that don't exist.
430: .CW $FB/fmmv
431: then moves all the files to be copied to the backup area.
432: We then update the database with information from the files we just copied.
433: .PP
434: Updating the database is a two part process.
435: Run
436: .CW $FB/updatef
437: on the files (use the program
438: .I updatew
439: for files on the WORM)
440: and then feed the output to
441: .CW $FB/dbupdate .
442: In this way, we guarantee that the database is purely a function of
443: the backed up files
444: (assuming none get lost between the backup area and the backup media).
445: The input to
446: .I dbupdate
447: is (roughly) a sequence of backup file headers and contents of
448: backed up directories.
449: .I Dbupdate
450: updates the various databases (described below) and sometimes tries to unlink
451: the backup copies.
452: (This happens when two copies of the same file are
453: in the same receiving subdirectory.
454: .I Sweep
455: happily copies both to the backup area but when
456: .I dbupdate
457: goes to update the main database for the second copy, it discovers
458: it already has this copy and so unlinks the second copy.
459: It doesn't care if the unlink fails because this is just an attempt
460: to be space efficient and in any case, the unlink can fail only if the file
461: has already been committed to the backup media.)
462: .I Dbupdate
463: also appends accounting statistics records for each file, containing the time
464: the file was saved, the size, the owner and the system name, to the file
465: .CW $FM/stat.log .
466: .PP
467: After
468: .I munge
469: is finished scanning the receive areas,
470: it processes the statistics records generated by
471: .I dbupdate
472: by
473: calling
474: .CW $FB/procstats .
475: This reads (and then truncates)
476: .CW $FM/stat.log
477: and adds new records to the files
478: .CW $FM/stat/\fIsystem\f(CW .
479: These records are in machine independent format and have been collapsed to
480: refer to all the files per user/day combination.
481: Even in this compressed format,
482: the statistics records would grow without bound.
483: Accordingly,
484: .I munge
485: calls
486: .CW "procstats -c"
487: to further collapse together all the records older than 30 days for each user.
488: (The number 30 comes from the only program that looks at these statistics,
489: .CW "backup stats" .)
490: .PP
491: Throughout its work,
492: .I munge
493: checks to see if it should exit by checking the existence of a guard file;
494: this is created by
495: .CW $FB/stopmunge .
496: .NH 2
497: The Databases on the Server
498: .PP
499: There are three databases kept on the server, all conventionally kept in
500: .CW $FM/db .
501: The first,
502: .CW filemap ,
503: is the main and only required database;
504: it contains the mappings from filename to last modify date and
505: from (filename, modify date) tuple to backup copy name.
506: The second,
507: .CW dir ,
508: is optional and maps (directory, modify date) tuples to their contents.
509: It is used to make recovery of file trees go (much) faster.
510: The third,
511: .CW fs ,
512: is optional and maps (filename, modify date) tuples to their
513: .I stat
514: buffers.
515: It is used to implement the backup file system.
516: .PP
517: The default implementation of these databases is Peter Weinberger's
518: compressed B-trees (see
519: .I cbt (1)).
520: (The compression refers to eliding common prefixes
521: of successive keys; it does very well on the pathnames used by the File Motel.)
522: The
523: .I cbt
524: database
525: .I db
526: consists of two files,
527: .I db\f(CW.T\fR
528: (the tree part) and
529: .I db\f(CW.F\fR
530: (the data part).
531: As the
532: .I cbt
533: routines do not reclaim space, the
534: .CW .T
535: file can start growing at a very fast rate when the tree is large
536: (say four levels).
537: This has proved to be a real nuisance so there is considerable support
538: for periodic squashing of the database
539: (which reclaims space by rebuilding the database) and
540: for supporting the
541: .I filemap
542: database as a collection of separate databases.
543: .PP
544: The latter is intended to be used in the following way.
545: The file
546: .CW $FM/db/filemap
547: is always the current
548: .I filemap
549: database.
550: If the file
551: .CW $FM/db/filemaplist
552: exists, it is taken as a list of database names,
553: one per line in oldest to newest order, to be used in addition to
554: .CW $FM/db/filemap .
555: These are searched only, never updated.
556: At our site, we produce one of these databases for about every 15\-16GB
557: of backup files.
558: .NH 2
559: Server Sends Files to the Client
560: .PP
561: All requests for files go through a central server
562: .CW $FB/fetch_ .
563: This program simply farms out work to other programs.
564: .I Fetchf
565: attempts to find files that are still under
566: .CW $FM/v .
567: Systems with plenty of mass storage can leave the backup copies
568: online and things will go quite fast.
569: For the files that
570: .I fetchf
571: can't find,
572: .I fetch_
573: looks in the configuration file
574: (\f(CW$FL/conf\fP)
575: to determine the backup media (say
576: .CW j
577: for jukebox).
578: It then calls
579: .CW $FB/fetchj
580: with the appropriate filenames.
581: If you just have a WORM drive, you should use
582: .CW $FB/fetchw
583: instead.
584: These two programs purport to be generic drivers for jukeboxes
585: and single drives; if you have different media (say an Exabyte tape),
586: you should be able simply to load the drivers with your media library
587: to generate the appropriate fetch program.
588: More details are given below in the section on media management.
589: .PP
590: Users can generally access any files they have read permission for,
591: regardless of what system they are on or the system from which the files were
592: stored from.
593: In addition, we trust our users (or more importantly, our network)
594: and so we do no checking of a user's right to retrieve files.
595: Such checking, such as a password, can easily be added to the startup
596: protocol between the program the user calls (\f(CW$FL/fetch\fP)
597: and the server (\f(CW$FB/fetch_\fP).
598: .NH 2
599: Administrivia
600: .PP
601: This section is a bunch of administrative odds and ends for the way we organise
602: the File Motel in our Center.
603: Your details may be different, and indeed ours change over time,
604: but the examples are probably helpful.
605: .NH 3
606: File Layout
607: .PP
608: We store the File Motel under the directory
609: .CW /usr/backup ,
610: which is a file system large enough to hold comfortably the current
611: databases and a squashed version (more on this later).
612: The receiving area is another smallish file system (about 120MB) mounted on
613: .CW /usr/backup/rcv
614: and the holding area
615: .CW /usr/backup/v
616: is another file system of the same size.
617: This is done to isolate the effects of client excesses;
618: the sending processes all know how to deal with running out of space
619: (we practice often).
620: I regard running out of space once a month as tolerable;
621: once a week is too much.
622: To aid searches for files, we keep a file
623: .CW /usr/backup/filenames ), (
624: which is a sorted list of all filenames.
625: This is maintained by the database squasher.
626: .PP
627: The main drawback to Weinberger's B-tree software is that it does
628: not reclaim space in the tree.
629: Thus, over time the tree file gets huge
630: (the rate grows as the depth of the tree).
631: The fix is to periodically squash the tree.
632: We combine this with dumping the database to WORM disk in the script
633: .CW $FB/backupdb .
634: .NH 3
635: Talking to the Clients
636: .PP
637: We have found it best to call the clients rather than have them call us.
638: The load seems more balanced and things get done sooner.
639: We use
640: .I mk
641: as it handles parallel processing; a typical mkfile is
642: .P1 0
643: CLIENTS=Cwild C3k Ctcp!tempel
644: NPROC=3
645:
646: clients:VQ: $CLIENTS
647: PROCPERM=$FB/toworm $FB/munge
648:
649: C%:VQ:
650: set +e; $FB/callclient $stem; exit 0
651: .P2
652: Understanding this completely requires familiarity with
653: .I mk
654: but the intent is clear.
655: We first get the files from the clients by the
656: .CW C%
657: rule and then process them by
658: .CW munge
659: and then put them out on WORM disk by
660: .CW toworm .
661: As we use Datakit, most clients are called using Datakit but some
662: (like
663: .CW tempel
664: in the mkfile) are called using TCP/IP.
665: This convenient piece of magic works on V9 because of Dave Presotto's
666: clever design of the IPC system; you may have to work harder.
667: The
668: .CW set
669: stuff in the
670: .CW C%
671: rule means to keep on processing even if a client gets an error.
672: The entry in
673: .CW /etc/crontab
674: is more or less
675: (this is one physical line folded at the \*(cr because of the column width)
676: .P1 0
677: eval "cd /usr/backup/adm; mk clients 2>&1" |\*(cr
678: mail backup
679: .P2
680: We use the
681: .CW backup
682: mailbox to redirect mail to someone appropriate.
683: .NH 2
684: Disasters
685: .PP
686: Currently, the only disaster we have had that was not the result of a kernel bug
687: is running out of space;
688: this is either inconvenient or quite bad.
689: Running out of space in the receiving or safe areas is just inconvenient.
690: By default, the client's
691: .I fmpush
692: stops when the receiving area runs out of space after saying how many files got
693: transmitted.
694: This is enough information to resend the rest when convenient.
695: Alternatively, you can change
696: .CW $FL/act
697: to give
698: .I fmpush
699: the
700: .CW -r
701: flag; this means that it will retry sending files every hour or so
702: until it succeeds.
703: Running out of space in the holding area is also not too bad;
704: eventually
705: .I munge
706: will put the holding area onto the backup media and then cycle through
707: the receiving area again.
708: .PP
709: The worst effect of running out of space is ruining your database.
710: (This happens rarely for us as
711: we keep our databases on a file system apart from the receiving/holding areas.)
712: Rebuilding the database is not too hard.
713: First, find out the next backup name to be assigned
714: (by a
715: .CW "sweep -n"
716: or by examining the backup media and holding areas).
717: Then, get the most recent backup copy of your database and install it.
718: Set the next backup name you found in the first step with
719: .CW "sweep -s" .
720: You then need to extract the database information for each file added
721: to the database since the backup copy of the database was made.
722: The starting file name is stored in the
723: .CW .N
724: file by
725: .I backupdb .
726: The program
727: .I updatew
728: will extract this from files on a WORM, and
729: .I updatef
730: from regular disk files.
731: The result is fed to
732: .I dbupdate
733: as done in
734: .I munge .
735: .NH
736: Installing the File Motel on a Client System
737: .PP
738: The following instructions assume you have the source
739: .CW fm.cpio
740: somewhere, say
741: .CW /tmp/fm.cpio .
742: Note also that these instructions will change over time;
743: you must follow the online copy of this document included with the source.
744: .IP [1]
745: You will need version 3 of
746: .I mk
747: (or any version
748: dated later than Mar 11, 1989).
749: .IP [2]
750: Select the root directory for the source, set the
751: environment variable
752: .CW FMSRC
753: to its name
754: and export
755: .CW FMSRC .
756: For example,
757: .P1
758: FMSRC=/usr/filemotel/src
759: export FMSRC
760: .P2
761: .IP [3]
762: Install the source tree by
763: .P1
764: cd $FMSRC/..; cpio -iudc < /tmp/fm.cpio
765: .P2
766: .IP [4]
767: Create a
768: .CW CONF
769: file.
770: This describes your installation environment
771: and is included in lower-level mkfiles.
772: The various switches are described in detail below;
773: however, the easiest way is to start with one of the
774: sample configuration files in the directory
775: .CW $FMSRC/conf .
776: .IP [5]
777: If necessary, create the repository for client files:
778: .P1
779: mkdir /usr/lib/filemotel
780: .P2
781: (this is configurable, see
782: .CW FMLIB
783: below)
784: and if you have not defined
785: .CW NO_NETNAME
786: in
787: .CW CONF ,
788: ensure that
789: .CW /n/clientname
790: is a link to
791: .CW / .
792: .IP [6]
793: Initialise the source tree for compiling by
794: .P1
795: mk depend
796: .P2
797: This only needs to be done once.
798: If you have to repeat, you can undo this by
799: .P1
800: mk undepend
801: .P2
802: .IP [7]
803: Compile and install the client software
804: by
805: .P1
806: mk client
807: .P2
808: This can be repeated as often as you like.
809: Only files in
810: .CW $FMLIB
811: and the file
812: .CW $FMBIN/backup
813: (these are configurable, see
814: .CW FMBIN
815: below)
816: are affected.
817: .IP [8]
818: Setup up the dialstring of the server system by
819: .P1
820: echo server-machine-name > $FMLIB/conf
821: .P2
822: The name should match the type of IPC you selected in
823: .CW CONF .
824: For example,
825: .TS
826: center;
827: c c
828: l lFCW.
829: IPC Example
830: Datakit nj/astro/wild
831: Datakit wild
832: IP wild.astro.nj.att.com
833: .TE
834: .IP [9]
835: In theory, you are now operational.
836: A couple of small tests are described in the file
837: .CW SANITY .
838: Some common problems and their cures are described below.
839: .IP [10]
840: You need to construct the script
841: .CW $FMLIB/sel
842: which prints the names of files that you want backed up.
843: There is a sample script (\f(CWsample.sel\fP) in that directory.
844: Be careful not to backup networked file systems by mistake.
845: .IP [11]
846: If you are initiating backup via
847: .I cron (8),
848: add the following command to
849: .CW crontab :
850: .P1
851: eval "/usr/lib/filemotel/sel | \*(cr
852: /usr/lib/filemotel/act 2>&1" | \*(cr
853: mail backup
854: .P2
855: The exact format varies from system to system;
856: the File Motel administrator should tell you what time to set it off.
857: .IP
858: If your client's backup is initiated from the server system,
859: you will have to add the line for
860: .CW fmclient
861: to your flavour of IPC services file.
862: If you communicate to the server by TCP/IP
863: (that is, your
864: .CW CONF
865: file has
866: .CW IPC=socket ),
867: get the
868: .I fmclient
869: line from the file
870: .CW tcp.inetd
871: and add it to
872: .CW /etc/inetd.conf
873: (some systems use
874: .CW /usr/etc/inetd.conf )
875: and add the
876: .I fmclient
877: line from the file
878: .CW tcp.services
879: and add it to
880: .CW /etc/services .
881: You then need to prod
882: .I inetd
883: to look at the new files (commonly by sending it a hangup signal).
884: On some systems, like SunOS, you may need to prod name servers
885: such as the Yellow Pages as well.
886: .IP
887: If you use V10 IPC,
888: add the corresponding line for
889: .CW fmclient
890: from
891: .CW ipc.V10
892: to
893: .CW /usr/ipc/lib/serv.local .
894: The files
895: .CW tcp.inetd
896: and
897: .CW ipc.V10
898: are made by
899: .P1
900: cd $FMSRC; mk ipc.list
901: .P2
902: .NH 2
903: Some Common Installation Problems
904: .PP
905: As a general rule, keep an eye on the log file (on the server)
906: when setting up the File Motel.
907: The most convenient way is a window with a
908: .P1
909: tail -f $FM/log
910: .P2
911: (\f(CW$FM\fP is the root directory of the File Motel.)
912: .PP
913: The most common problem is that the basic IPC software doesn't work.
914: This affects most programs because they involve calling a service on the
915: backup machine.
916: That is why the first thing you try to get working is
917: .I logger
918: which sends messages to the logger process on the backup machine.
919: The kinds of bugs I have seen here are typically bugs in the networking code,
920: particularly TCP/IP.
921: For example, the
922: .I logprint
923: function expects an acknowledgement from the logger server
924: to indicate that everything went okay.
925: On at least two of the systems I use, this sometimes doesn't happen because
926: the closing of the socket by the logger server after sending the ack
927: seems to speed pass the ack and get to
928: .I logprint
929: first.
930: Naturally,
931: .I logprint
932: complains, as might we all.
933: The best solution is to fix the TCP/IP implementation; failing that,
934: you might try a judicious sleep between the ack and the close in
935: the logger server.
936: This is only one example of a general class of timing problems.
937: .PP
938: Another fertile field of failed implementations have to do
939: with user and system names.
940: The File Motel tries to check the validity of system and user names
941: and denies service if there appears to be something sleazy going on.
942: Regrettably, some otherwise sound TCP/IP implementations resemble sleaze.
943: For example, a user
944: .CW mary
945: on a client may appear on the server as the user
946: .CW bill .
947: Or a system may have a system name that is unrelated to the name
948: the networking code uses.
949: An attempt is made to cope with these cases, but may fail with
950: unexpectedly bizarre implementations.
951: If worst comes to worst, simply turn off all the checking
952: and hope no one does anything naughty.
953: (Even if you do this, think hard about allowing remote users
954: to claim they are
955: .CW root ;
956: they will be able to look at all sorts of things.)
957: Unlike X,
958: I implement function and policy.
959: However, all the checking is done in one place
960: .CW serv_$IPC.c ); (
961: feel free to do whatever you like,
962: it's your Motel now.
963: .de XX
964: .IP \\f(CW\\$1\\fP
965: .br
966: ..
967: .NH 2
968: Configuration and Compiling Options
969: .PP
970: The File Motel software is designed for easy installation in heterogeneous
971: environments.
972: The configuration details described below are stored in the file
973: .CW $FMSRC/CONF .
974: .CW $FMSRC
975: contains a number of files
976: containing settings for various systems; you may want to use
977: one of these as a starting point.
978: (Remember that the following information has a small half-life;
979: the truth should be in the online copy of this manual.)
980: The most obvious aspect of configuring the File Motel means choosing on
981: the three directories where files live.
982: The source directory,
983: .CW FMSRC ,
984: has been described above. The other two are
985: .XX FMLIB=/usr/lib/filemotel
986: Change this to wherever you want to put the subprograms.
987: .XX FMBIN=/usr/bin
988: The directory for the (only) user-called command,
989: .CW backup .
990: .LP
991: Configuring the source to your environment is mostly done with
992: .I mk
993: variables and an interface library in
994: .CW src/sys/\f2system .
995: The
996: .I mk
997: variables are
998: .XX RANLIB=ranlib
999: Some systems, mostly BSD-based, whine incessantly unless archive libraries
1000: are processed with some program typically called
1001: .I ranlib .
1002: In this case, set
1003: .CW RANLIB=ranlib ;
1004: otherwise, say if you are on a System V machine, use a harmless program
1005: such as
1006: .CW RANLIB=: .
1007: .XX IPC=socket
1008: Select your favorite type of IPC.
1009: Different clients can use different types and the client's type
1010: need not match the backup system.
1011: (For example, in our Center,
1012: the Cray talks to us via TCP/IP but we talk to it via Datakit.)
1013: The only choices are
1014: .CW socket
1015: and
1016: .CW v10 .
1017: .XX IPCLIB=
1018: Set this if you need a special library in order to use your flavor of IPC.
1019: For example, on V10 systems set
1020: .CW IPCLIB=-lipc .
1021: .XX LIBTYPE=a
1022: This should be set to
1023: .CW a
1024: unless you are on the Cray (which doesn't have archives yet!)
1025: when it should be
1026: .CW o .
1027: .XX COMPAT=
1028: Set
1029: .CW COMPAT=.compat
1030: if you want to be able to process older File Motel files.
1031: (You may have to work hard to get this to work on some systems;
1032: I gave up on the Cray.)
1033: .XX SECTYPE=
1034: Set
1035: .CW SECTYPE=v9
1036: if you are running a McIlroy-Reeds compatible security kernel.
1037: .XX WORMFACE=uda
1038: If you are running the WORM software, you need to say what kind of interface
1039: the WORM is attached to.
1040: The other option
1041: (and the best if you just want to compile without thinking too hard) is
1042: .CW scsi .
1043: The latter may need customizing at your site.
1044: .LP
1045: Currently the system dependent interface library includes the following routines:
1046: .TS
1047: center;
1048: lFCW l.
1049: dirtoents convert directory to element names
1050: ftw traverse file tree
1051: nofile number of fd's available
1052: sysname system name
1053: username user's login name
1054: rx_$IPC call a remote service
1055: serv_$IPC receive calls
1056: service service/socket mapping details
1057: dateadjust do daylight savings/timezone
1058: .TE
1059: .LP
1060: There are a small number of
1061: .CW #define 's
1062: inside
1063: .CW .c
1064: files.
1065: .XX -DSTRINGH="'<string.h>'"
1066: Define the value to be the string function header file.
1067: .XX -DNO_NETNAME
1068: Define this to disable saving and restoring files through
1069: .CW /n/machine-name
1070: although they will still be stored with that prefix.
1071: .NH
1072: Installing the File Motel on a Server System
1073: .PP
1074: The source comes in both
1075: .I cpio
1076: and
1077: .I tar
1078: formats.
1079: As with the client source installation, note that the following description
1080: is dated and the online copy may be significantly different in detail.
1081: .IP [1]
1082: Follow the client installation process steps 1\-8.
1083: You also need to set the place where the administrative binaries are kept.
1084: I do it this way:
1085: .P1
1086: FMAB=$FM/bin
1087: export FMAB
1088: .P2
1089: .IP [2]
1090: Complete
1091: .CW $FMLIB/conf .
1092: You have to specify the default media type and the root of the administrative
1093: file tree (denoted by
1094: .CW $FM
1095: below).
1096: Details are in
1097: .I backup (5);
1098: my File Motel has this configuration:
1099: .P1
1100: wild
1101: j
1102: /usr/backup
1103: .P2
1104: .IP [3]
1105: Everything that doesn't need to run as
1106: .CW root
1107: should run as an otherwise unused id.
1108: By default, this is
1109: .CW fmdaemon ;
1110: if you don't like this, change the define in
1111: .CW libfm/server.c .
1112: Whatever you choose, set up an account for them;
1113: the File Motel requires nothing but their name/uid
1114: (not even a login directory).
1115: By default, all the shell scripts send mail to the mailbox
1116: .CW backup .
1117: This should be set to an alias for the File Motel caretaker.
1118: .IP [4]
1119: Inform your IPC system of the many services the File Motel offers.
1120: See the notes under step 11 in the client installation above but
1121: install everything, not just
1122: .CW fmclient .
1123: (See step 10 below as well.)
1124: You also need to set up the periodic (normally nightly)
1125: calling of clients and/or
1126: the processing of their files by
1127: .CW $FB/munge .
1128: .I Munge
1129: needs the name of a program to copy the files to your backup media;
1130: set the variable
1131: .CW PROCPERM
1132: to that program's name.
1133: As described previously, you also need to periodically backup the databases
1134: with
1135: .CW backupdb ;
1136: it also needs the name of the program to copy files to your media.
1137: .IP [5]
1138: Initialise the log file:
1139: .P1
1140: > $FM/log; chown bin $FM/log
1141: chmod 644 $FM/log
1142: .P2
1143: .IP [6]
1144: Install the server programs:
1145: .P1
1146: mk server
1147: .P2
1148: .IP [7]
1149: Setup the receiving areas.
1150: List their names in
1151: .CW $FM/adm/rcvdirs
1152: and initialise each are by running
1153: .P1
1154: $FB/rcvdirs
1155: .P2
1156: We use one 120MB file system mounted on
1157: .CW $FM/rcv .
1158: .IP [8]
1159: Allocate the safe area for backup copies.
1160: It must have the name
1161: .CW $FM/v
1162: but may be a symbolic link if there is not enough space in
1163: .CW $FM .
1164: We use an identically sized file system to the receive area mounted on
1165: .CW $FM/v .
1166: .IP [9]
1167: After deciding which databases you want maintained,
1168: initialise the databases with
1169: .P1
1170: src/dbinit.sh
1171: .P2
1172: You may want to start off with all three and remove any you don't want later on
1173: (like when they get to be too big).
1174: .IP [10]
1175: Choose how the receiving process
1176: .I rcv
1177: works.
1178: By default, it simply accepts files.
1179: If it is invoked by the name
1180: .CW mrcv ,
1181: it initiates processing of the received by
1182: .I munge
1183: (or more accurately,
1184: .CW $FL/callmunge )
1185: after the first and last files have been received
1186: (you need both in case any one file took longer to receive than
1187: .I munge 's
1188: cycle time).
1189: The advantage is that you will almost never run out of space, as you will be
1190: processing files at the same time as receiving them.
1191: The disadvantage is that everything will run slower.
1192: I use the default behavior; we rarely run out of space and I like to
1193: investigate why some client is sending much more than normal
1194: before accepting it all.
1195: .IP [11]
1196: Finish the client installation starting with step 10, making sure you
1197: do not backup the receiving areas or
1198: .CW $FM/v .
1199: .IP [12]
1200: Add the command
1201: .P1
1202: $FB/rmlocks
1203: .P2
1204: to
1205: .CW /etc/rc
1206: (or whatever passes for system startup on your system).
1207: This simply removes any lockfiles in
1208: .CW $FM/locks .
1209: .NH
1210: Media Management
1211: .PP
1212: An attempt has been made to provide generic media management programs.
1213: For example, the recovery servers
1214: .I fetchw_
1215: and
1216: .I fetchj_
1217: are instances of a single device server and a jukebox server respectively.
1218: To make this work, a media library is used.
1219: To use a new media, such as Exabyte tapes,
1220: implement the routines in the library, and link the library with
1221: .CW fm/media_.o
1222: or
1223: .CW fm/mmedia_.o .
1224: An informal description of the routines follow.
1225: .LP
1226: .CW "mediainit(char *device, char *vol_id)"
1227: .ti +5n
1228: Initialise the media on the device specified by
1229: .I device .
1230: The latter may a full name or any recognizable abbreviation.
1231: If
1232: .I vol_id
1233: is given, it is checked against the media present.
1234: .LP
1235: .CW "char *mediamount(char *vol_id)"
1236: .ti +5n
1237: Mount the media named
1238: .I vol_id
1239: and return the appropriate device
1240: (suitable for use by
1241: .I mediainit ).
1242: Currently, the values given as
1243: .I vol_id
1244: are those returned by
1245: .I medianame
1246: (below).
1247: .LP
1248: .CW "medianame(char *volume)"
1249: .ti +5n
1250: Return the media name containing
1251: .I volume .
1252: .LP
1253: .CW "mediaopen(char *name, Media *m)"
1254: .ti +5n
1255: Set up
1256: .I m
1257: to point at the backup copy
1258: .I name .
1259: The fields in a
1260: .CW Media
1261: include a file descriptor, preferred read block size, and copy size.
1262: .LP
1263: .CW "void mediafiles(int32 v, int32 n, Media *m, Tb **bp)"
1264: .ti +5n
1265: Return a Media and a list (in
1266: .CW *bp )
1267: of backup copy pointers for all backup copies more recent than
1268: file
1269: .I n
1270: in volume
1271: .I v .
1272: A
1273: .CW Tb
1274: has the creation time and initial (1K) block number for a backup copy.
1275: (It is used by
1276: .I dbupdate ).
1277: The size returned in
1278: .I m
1279: is not actually a size but the number of records in
1280: .CW *bp .
1281: .PP
1282: This is not a complete description;
1283: if you have to write new versions of these routines,
1284: look at the existing implementations (in
1285: .CW $FMSRC/media )
1286: and the programs that use them
1287: (all in
1288: .CW $FMSRC/fm ).
1289: .NH
1290: References
1291: .LP
1292: |reference_placement
This archive runs on limited infrastructure. Preserving old code on modern bandwidth. Automated agents are requested to crawl responsibly.