|
|
1.1 root 1: % run this through LaTeX
2:
3: \input lcustom
4: \input version
5:
6: \documentstyle[12pt,DScustom,sfwmac]{article}
7: \setcounter{page}{0}
8: \pagestyle{empty}
9:
10: \begin{document}
11:
12: \title{Changes to\\ The Rand MH Message Handling System:\\
13: MH \#6.4 for 4.3BSD UNIX}
14: \author{Marshall T.~Rose\\
15: Northrop Research and Technology Center\\
16: One~Research Park\\
17: Palos Verdes Peninsula, CA 90274}
18: \date{\ifdraft \versiondate/\\ Version \versiontag/\else \today\fi}
19: \maketitle
20: \footnotetext[0]{\hskip -\parindent
21: This document (version \versiontag/)
22: was \LaTeX set \today\ with \fmtname\ v\fmtversion.}%
23:
24: \begin{abstract}
25: \noindent This document describes the user-visible change to the
26: UCI version of the Rand \MH/ system that were made from \mh5 to \MH/ \#6.4.
27: It is based on the \mh6 changes document,
28: but has been updated to accurately reflect the \MH/ distributed with
29: 4.3\bsd/~\unix/.
30: This document does not describe bug-fixes, per se,
31: or internal changes,
32: unless these activities resulted in a visible change for the \MH/ user.
33:
34: This document is meant to supplement,
35: not supersede, the standard \MH/ User's manual\cite{MH}.
36:
37: Comments concerning this documentation should be addressed to the Internet
38: mailbox {\sf [email protected]}.
39: \end{abstract}
40:
41: \bop\pagestyle{plain}\pagenumbering{arabic}
42:
43: \section* {Acknowledgements}
44: The \MH/ system described herein is based on the original Rand \MH/ system.
45: It has been extensively developed (perhaps too much so) by Marshall T.~Rose
46: and John L.~Romine at the University of California, Irvine.
47: Einar A.~Stefferud, Jerry N.~Sweet,
48: and Terry P.~Domae provided numerous suggestions
49: to improve the UCI version of \MH/.
50: Of course,
51: a large number of people have helped \MH/ along.
52: The list of ``\MH/~immortals'' is too long to list here.
53: However, Van Jacobson deserves a special acknowledgement for his tireless
54: work in improving the performance of \MH/.
55: Some programs have been speeded-up by a factor of 10 or 20.
56: All of users of \MH/, everywhere, owe a special thanks to Van.
57:
58: \section* {Disclaimer}
59: The Regents of the University of California wish to make it known that:
60: \begin{quote}
61: Although each program has been tested by its contributor,
62: no warranty, express or implied,
63: is made by the contributor or the University of California,
64: as to the accuracy and functioning of the program
65: and related program material,
66: nor shall the fact of distribution constitute any such warranty,
67: and no responsibility is assumed by the contributor
68: or the University of California in connection herewith.
69: \end{quote}
70:
71: \bop
72:
73: \section* {Conventions}
74: In this document,
75: certain \LaTeX -formatting conventions are adhered to:
76: \begin{enumerate}
77: \item The names of \unix/ commands, such as \pgm{comp},
78: are presented in {\it text italics}.
79:
80: \item Arguments to programs, such as \arg{msgs},
81: are presented in {\tt typewriter style} and delimited by single-quotes.
82:
83: \item \unix/ pathnames and envariables,
84: such as $$\file{/usr/uci/}\hbox{\qquad and\qquad}\file{\$SIGNATURE},$$
85: are presented in {\sl slanted roman}.
86:
87: \item Text presenting an example, such as
88: \example comp\ -editor\ zz\endexample
89: is presented in {\tt typewriter style}.
90: \end{enumerate}
91:
92: \bop
93:
94: \section* {General Changes}
95: Unlike the changes between \mh4 and \mh5,
96: a large number of user-visible changes have been made in \mh6.
97: These changes have been in the form of bug fixes and several generalizations.
98: The majority of these will not affect novice users.
99: In addition, \mh6 is a great deal faster than \mh5:
100: all programs have been speeded-up significantly,
101: thanks to work done by Van Jacobson as part of the process of including \mh6
102: in the 4.3\bsd/~\unix/ distribution.
103:
104: This document describes all user-visible changes to \mh5 from it's initial
105: release to the intermediate release of \MH/ \#6.4.
106:
107: \subsection* {System-5 Support}
108: In addition to support for \bsd/~\unix/, V7~\unix/ and \xenix/ variants of
109: \unix/,
110: \MH/ finally has support for the AT\&T variant of \unix/, System~5.
111: Hopefully this will greatly expand the number of system which can run \MH/.
112: Ironically,
113: it appears that five ports of earlier versions of \MH/ (including \mh5)
114: were done,
115: but news of the work was not widespread.%
116: \nfootnote{In fact,
117: three groups in one large organization ported \MH/ independently,
118: each without knowledge of the others' work.}
119:
120: \subsection* {Documentation}
121: Several new documents have been included in the \mh6 distribution:
122: The paper {\em MH.5: How to process 200 messages a day and still get some
123: real work done}
124: was presented at the 1985 Summer Usenix Conference and Exhibition in
125: Portland, Orgeon.
126: Another paper, {\em MH: A Multifarious User Agent},
127: has been accepted for publication by Computer Networks.
128: Both describe \MH/,
129: the former from a more technical and somewhat humorous perspective,
130: the latter from a more serious and research-oriented perspective.
131: In addition,
132: a third paper has been included,
133: {\em Design of the TTI Prototype Trusted Mail Agent},
134: which describes a so-called ``trusted'' mail agent built on top of \MH/.
135: This paper was presented at the Second International Symposium on
136: Computer Message Systems in Washington, D.C.
137: A fourth paper,
138: {\em MZnet: Mail Service for Personal Micro-Computer Systems},
139: is also included.
140: This paper,
141: which was presented at the First International Symposium on Computer Message
142: Systems in Nottingham, U.K.,
143: describes a \cpm/-based version of \MH/.
144:
145: In addition,
146: the \MH/ tutorial, {\em The Rand MH Message Handling System: Tutorial},
147: and,
148: {\em The Rand MH Message Handling System: The UCI BBoards Facility},
149: have both been updated by Jerry N.~Sweet.
150:
151: For \MH/ administrators (PostMasters and the like),
152: there's an entirely new document,
153: {\em The Rand MH Message Handling System: Administrator's Guide}.
154: It explains most of the ``ins and outs'' of maintaining an \MH/ system.
155:
156: Finally, all of the manual entries and the \MH/ manual have had a thorough
157: working over.
158: The documentation is expanded, more accurate, and more detailed.
159:
160: \subsection* {Help Listings}
161: When any \MH/ command is invoked with the \switch{help} switch,
162: in addition to listing the syntax of the command and version information,
163: the \MH/ configuration options will be listed.
164: \MH/ has so many configuration options,
165: that when debugging problems, this information is invaluable.
166:
167: \subsection* {The \MH/ Profile}
168: There are two new profile entries worth noting:
169: \eg{MH-Sequences} tells \MH/ the name of the file to record public
170: sequences in.
171: Users of \pgm{vm}, a proprietary, visual front-end to \MH/,
172: make use of this to disable the public sequences feature of \MH/.
173:
174: The profile entry \eg{Unseen-Sequence} names those sequences which should be
175: defined as those messages recently incorporated by \pgm{inc}.
176: The \pgm{show} program knows to remove messages from this sequence once it
177: thinks they have been seen.
178: If this profile entry is not present, or is empty, then no sequences are
179: defined.
180: Otherwise, for each name given, the sequence is first zero'd and then each
181: message incorporated is added to the sequence.
182: As such, this profile entry is rather analogous to the
183: \eg{Previous-Sequence} entry in the user's \MH/ profile.
184:
185: In addition, the \eg{Alternate-Mailboxes} entry in the profile has been
186: expanded to support simple wild-carding.
187: Also, the default for this profile entry is now the user's mail-id at any host.
188: This change was made since \MH/ can no longer reliably figure out what
189: the user's real outgoing address looks like.
190:
191: Finally,
192: when the \pgm{install-mh} program is automatically invoked by \MH/,
193: it won't prompt the user for information.
194: Instead, it notes that it's setting up the default environment.
195: In addition,
196: the \MH/ administrator may set-up a file called \file{mh.profile} in the \MH/
197: library area which is consulted by \pgm{install-mh} when initializing the
198: user's \profile/.
199:
200: \subsection* {The \MH/ Context}
201: The \pgm{folder}, \pgm{scan}, and \pgm{show} programs have been modified to
202: update the user's \MH/ context prior to writing to the user's terminal.
203: This allows the \MH/ user interrupt output to the terminal and still have the
204: expected context.
205: This is especially useful to interrupt long \pgm{scan} listings.
206: This change also introduces a subtle bug between \pgm{show} and messages
207: denoted by the \eg{Unseen-Sequence}.
208: See \man show(1) for the details.
209:
210: \subsection* {Addresses and 822 support}
211: \MH/ now fully supports the RFC-822 routing syntax for addresses
212: (it used to recognize the syntax, but ignore the information present).
213: In addition,
214: there are three major modes for support of non-822 addressing in \MH/:
215: \begin{itemize}
216: \item BERK\hbreak
217: This is useful on sites running \SendMail/.
218: It doesn't support full 822--style addressing,
219: in favor of recognizing such formats as ACSnet, and so on.
220: For sites that can't run in an 822--compliant environment,
221: this is the option to use
222: (at the price of sacrificing some of the power of 822--style addressing).
223: This also drastically reduces the address formatting facilities described
224: below.
225:
226: \item DUMB\hbreak
227: Although not as liberal as BERK,
228: the DUMB option is useful on sites in which the message transport system
229: conforms to the 822 standard,
230: but wants to do all the defaulting itself.
231:
232: \item BANG\hbreak
233: From out in left field,
234: the BANG option favors \UUCP/-style addressing over 822--style addressing.
235: Hopefully when all the \UUCP/ sites around get around to adopting domain-style
236: addresses, this option won't be needed.
237: \end{itemize}
238:
239: The \pgm{ap} program (mentioned momentarily) and the \pgm{ali} program
240: both support a \switch{normalize} switch indicate if addresses should be
241: resolved to their ``official'' hostnames.
242:
243: \subsection* {New Programs}
244: There are five new programs available:
245: The \pgm{ap} program is the \MH/ stand-alone address parser.
246: It's useful for printing address in various formats
247: (and for debugging address strings).
248: The \pgm{dp} program is similar, but works on dates instead of addresses.
249:
250: The \pgm{msgchk} program checks for new mail,
251: possibly using the Post Office Protocol, POP, described below.
252:
253: A new receive mail hook,
254: the \pgm{rcvstore} program,
255: which was written by Julian L.~Onions is available.
256:
257: Finally, a visual front-end to \pgm{msh} called \pgm{vmh} has been included.
258: (This program is discussed in greater detail later on.)
259:
260: \subsection* {Message Numbering}
261: \MH/ now no longer restricts the number of messages which may reside in a
262: folder
263: (beyond that of system memory constraints).
264: This means that message numbers larger than 2000 are permissible.
265: Hopefully this will make life easier for people reading the network news
266: using \MH/.
267:
268: \section* {The WhatNow Shell}
269: In \mh6,
270: there is now the concept of a unified \whatnow/ processor that
271: the four composition programs, \pgm{comp}, \pgm{dist}, \pgm{forw},
272: and \pgm{repl} all invoke.
273: This permits a greater flexibility in building mail applications with \MH/.
274: As a result, there's a new program, \pgm{whatnow}, which acts as the default
275: \whatnow/ program.
276: Consult the \man whatnow(1) manual entry for all the details.
277: The only important user-visible change is the \eg{headers} option went away,
278: which wasn't used that much anyway.
279:
280:
281: The only other thing worth noting is that unless \MH/ has been compiled with
282: the UCI option,
283: the user's \file{\$HOME/.signature} file is not consulted for the user's
284: personal name.
285:
286: \section* {Format Strings}
287: A general format string facility has been added to allow \MH/ users to tailor
288: the output of certain commands.
289:
290: The \pgm{inc}, \pgm{scan}, \pgm{ap}, and \pgm{dp} programs all consult a
291: file containing format strings.
292: Format strings,
293: which look a lot like \man printf(3) strings,
294: give these \MH/ commands precise instructions on how to format their output.
295:
296: As a result,
297: the \pgm{inc} and \pgm{scan} programs no longer have the
298: \switch{size}, \switch{nosize},
299: \switch{time}, \switch{notime},
300: \switch{numdate}, and \switch{nonumdate}
301: switches.
302: These switches have been replaced with the
303: \switch{form~formatfile} switch and the \switch{format~string} switch.
304: The former directs the program to consult the named file for the format
305: strings.
306: The latter directs the program to use the named string as the format.
307: To get the behavior of the old \switch{time} option,
308: use the \switch{form~scan.time} option.
309: Similarly,
310: to get the effect of \switch{size},
311: use \switch{form~scan.size}.
312:
313: A fun form to use is \switch{form~scan.timely} with \pgm{scan}.
314: Try it sometime.
315:
316:
317: The \pgm{repl} command uses a file containing format files to
318: indicate how the reply draft should be constructed.
319: Note that reply templates prior to \mh6 are incompatible with \mh5.%
320: \nfootnote{In fact, reply templates between \mh6 and \MH/ \#6.4 are
321: imcompatible.}
322: Don't worry though,
323: it's quite easy to convert the templates by hand.
324: (Those clever enough to have written a reply template to begin with won't
325: have {\em any\/} problem.)
326:
327: Similarly, when the \pgm{forw} program is constructing a digest,
328: it uses a file containing format strings to indicate how to build the
329: encapsulating draft.
330:
331: Finally, you can use these facilities in \pgm{mhl} as well.
332:
333: \section* {News}
334: The depreciated \MH/ news system (from \mh1) is now de-supported.
335: Use the ``hoopy'' BBoards facility instead.
336:
337: \section* {BBoards}
338: \MH/ maintainers take note:
339: the default home directory for the bboards login has changed from
340: \file{/usr/bboards/} to \file{/usr/spool/bboards/}.
341: Use the \eg{bbhome} directive in your \MH/ configuration file to set
342: it back to the old value if you wish.
343:
344: In addition, the aliases field for a BBoard in the BBoards file is now
345: deemed useful only for addressing, not for user input to \pgm{bbc}.
346: This means when giving the name of a BBoard to \pgm{bbc},
347: only the official name should be used.
348:
349: A final note for mailsystem maintainers:
350: the \MMDFII/ BBoards channel and the \SendMail/ BBoards mailer have been
351: modified to use the standard message encapsulation format when returning
352: failed messages to the list maintainer.
353: This means that the failure notices that the maintainer receives can
354: simply be \pgm{burst}.
355:
356: \subsection* {New Switches in bbc}
357: The \pgm{bbc} program permits you to specify the \eg{mshproc} to use on the
358: command line by using the \switch{mshproc~program} option.
359: There's also a \switch{rcfile~file} option which does ``the obvious thing''.
360: In addition, options which aren't understood by \pgm{bbc} are passed along to
361: the \eg{mshproc}.
362:
363: In addition, the following commands
364: pass any unrecognized switches on to the program that they invoke:
365: \pgm{bbc}, \pgm{next}, \pgm{show}, \pgm{prev}, and \pgm{vmh}.
366:
367: \subsection* {Distributed BBoards}
368: If both BBoards and POP (see the next section) are enabled,
369: then distributed BBoards can be supported on top of the POP service.
370: This allows the \MH/ user to read BBoards on a server machine
371: instead of the local host
372: (which saves a lot of wasted disk space when the same BBoards are replicated
373: several times at a site with several hosts).
374: See the {\em Administrator's Guide\/} for information on how this can be made
375: completely transparent to the \MH/ user.
376:
377: If you have several machines at your site running 4.2\bsd/~\unix/
378: and connected by an \ethernet/ (or other high-speed LAN),
379: you {\em want\/} this software.
380:
381: \subsection* {Visual Front-End to msh}
382: A simple window management protocol has been implemented for \MH/ programs
383: that might wish to act as a back-end to a sophisticated visual front-end.
384:
385: The first implementation of a server side (front-end) program is \pgm{vmh},
386: which uses \man curses(3) to maintain a split-screen interface.
387: Perhaps look for a \pgm{mhtool} program for the SUN next!
388:
389: The \pgm{msh} program has been modified to speak the client side (back-end)
390: of this protocol, if so directed.
391: At present, \pgm{msh} is the only program in the \MH/ distribution which
392: implements the client side of the window management protocol.
393:
394: \subsection* {Updates in msh}
395: Prior to quitting,
396: the \pgm{msh} command now asks if the \pgm{packf\/}'d file you've been
397: perusing should be updated if you've modified it and the file is writable by
398: you.
399: The file can be modified by using \pgm{burst}, \pgm{rmm}, \pgm{rmm},
400: or \pgm{sortm} commands.
401: The file can also be modified by using the \pgm{refile} command without the
402: \switch{link} option.
403: (Or course,
404: the \switch{link} option doesn't actually link anything to the file.)
405:
406: \section* {Distributed Mail}
407: \MH/ now contains a powerful facility for doing distributed mail
408: (having \MH/ reside on a host different than the message transport agent).
409: For general information,
410: consult either the
411: {\em MH.5: How to process 200 messages a day and still get some real work
412: done} paper,
413: or the {\em MH: A Multifarious User Agent} paper.
414: For specific information,
415: consult the {\em Administrator's Guide}.
416: Here's a brief synopsis:
417:
418: This POP facility in \MH/ is based on a modification of the ARPA Post
419: Office Protocol (POP).
420: A POP {\em subscriber\/} is a remote user,
421: on a POP {\em client host},
422: that wishes to pick-up mail on a POP {\em service host}.
423:
424: There are two ways to administer POP:
425: \begin{itemize}
426: \item Naive Mode\hbreak
427: Each user-id in the \man passwd(5) file is considered a POP subscriber.
428: No changes are required for the mailsystem on the POP service host.
429: However,
430: this method requires that each POP subscriber have an entry in the password
431: file.
432: The POP server will fetch the user's mail from wherever maildrops are kept on
433: the POP service host.
434: This means that if maildrops are kept in the user's home directory,
435: then each POP subscriber must have a home directory.
436:
437: \item Smart Mode\hbreak
438: This is based on the notion that the list of POP subscribers and the list of
439: login users are completely separate name spaces.
440: A separate database (similar to the \man BBoards(5) file)
441: is used to record information about each POP subscriber.
442: Unfortunately,
443: the local mailsystem must be changed to reflect this.
444: This requires two changes (both of which are simple):
445: \begin{enumerate}
446: \item Aliasing\hbreak
447: The aliasing mechanism is augmented so that POP subscriber addresses
448: are diverted to a special delivery mechanism.
449: \MH/ comes with a program, \man popaka(8), which generates the
450: additional information to be put in the mailsystem's alias file.
451: \item Delivery\hbreak
452: A special POP channel (for \MMDFII/) or POP mailer (for \SendMail/)
453: performs the actual delivery (\mh6 supplies both).
454: All it really does is just place the mail in the POP spool area.
455: \end{enumerate}
456: Clever mailsystem people will note that
457: the POP mechanism is really a special case of the more general
458: BBoards mechanism.
459: \end{itemize}
460: These two different philosophies are not compatible on the same POP service
461: host: one or the other, but not both, may be run.
462:
463: In addition, there is one user-visible difference,
464: which the administrator controls the availability of.
465: The difference is whether the POP subscriber must supply a password to the POP
466: server:
467: \begin{itemize}
468: \item ARPA standard method\hbreak
469: This uses the standard ARPA technique of sending a username and a password.
470: The appropriate programs (\pgm{inc}, \pgm{msgchk}, and possibly \pgm{bbc\/})
471: will prompt the user for this information.
472:
473: \item \unix/ remote method\hbreak
474: This uses the Berkeley \unix/ reserved port method for authentication.
475: This requires that the two or three mentioned above programs be {\em setuid\/}
476: to root.
477: (There are no known holes in any of these programs.)
478: \end{itemize}
479: These two different philosophies are compatible on the same POP service host:
480: to selectively disable RPOP for hosts which aren't trusted,
481: either modify the \file{.rhosts} file in the case of POP subscribers being
482: \unix/ logins,
483: or zero the contents of network address field of the \man pop(5) file for the
484: desired POP subscribers.
485:
486: The \pgm{inc} command also has two other switches when \MH/ is enabled for
487: POP:
488: \switch{pack~file} and \switch{nopack}.
489: Normally,
490: \pgm{inc} will use the POP to incorporate mail from a POP service host into
491: an \MH/ folder (\eg{+inbox}).
492: However,
493: there are some misguided individuals who prefer to \pgm{msh} to read their
494: maildrop.
495: By using the \switch{pack~file} option,
496: these individuals can direct \pgm{inc} to fetch their maildrop from the POP
497: service host and store it locally in the named file.
498: As expected, \pgm{inc} will treat the local file as a maildrop,
499: performing the appropriate locking protocols.
500:
501: \section* {Rcvmail hooks}
502: In order to offer users of \MH/ increased rcvmail hook functionality,
503: the \pgm{slocal} program has been upgraded to support the semantics of
504: the \MMDFII/ mail-delivery mechanism.
505: This means that users of \mh6 can maintain identical \file{.maildelivery}
506: files regardless of the underlying transport system.
507: See \man mhook(1) for all the details.
508:
509: \subsection* {Field change in rcvpack}
510: The \pgm{rcvpack} rcvmail hook now adds the field name \eg{Delivery-Date:}
511: instead of \eg{Cron-Date:} to messages it \pgm{pack\/}s.
512:
513: \section* {GNU Emacs Support}
514: James Larus' \pgm{mh-e} macro package for GNU Emacs (version~17) is included
515: in the distribution.
516: When loaded in Emacs, this provides a handy front-end.
517:
518: \section* {Other Changes}
519: Here's the miscellany:
520:
521: \subsection* {Continuation Lines}
522: Alias files used by \MH/,
523: display templates used by \pgm{mhl},
524: and format files used by \pgm{forw}, \pgm{repl}, and \pgm{scan} all support
525: a standard continuation line syntax.
526: To continue a line in one of these files,
527: simply end the line with the backslash character (`$\backslash$').
528: All the other files used by \MH/ are in 822--format,
529: so the 822--continuation mechanism is used.%
530: \nfootnote{Looking back,
531: it would have been best had all files in \MH/ used the 822--format.}
532:
533: \subsection* {New switch in ali}
534: The \pgm{ali} command now has a \switch{noalias} switch to prevent
535: system-wide aliases from being interpreted.
536:
537: \subsection* {Modifications to show}
538: The \switch{format}, \switch{noformat}, \switch{pr}, and \switch{nopr}
539: options to \pgm{show} have gone away in favor of a more general mechanism.
540: The \switch{showproc~program} option tells \pgm{show}
541: (or \pgm{next} or \pgm{prev\/}) to use the named program as the \eg{showproc}.
542: The \switch{noshowproc} option tells \pgm{show}, et. al.,
543: to use the \man cat(1) program instead of a \eg{showproc}.
544: As a result, the profile entry \eg{prproc} is no longer used.
545:
546: \subsection* {Front-End to mhl}
547: When outputting to a terminal,
548: the \pgm{mhl} program now runs the program denoted by the profile entry
549: \eg{moreproc}.
550: If this entry is not present,
551: the default is the UCB \pgm{more} program.
552: If the entry is non-empty,
553: then that program is spliced between \pgm{mhl} and the user's terminal.
554: The author uses the \pgm{less} program as his \eg{moreproc}.
555:
556: Of course,
557: if \pgm{mhl} isn't outputting to a terminal,
558: then \eg{moreproc} is not invoked.
559:
560: \subsection* {Switch change in inc}
561: The \switch{ms~ms-file} switch in \pgm{inc} has been changed to
562: \switch{file~name} to be more consistent.
563:
564: \subsection* {Complex Expressions in pick}
565: The \pgm{pick} command now handles complex boolean expressions.
566:
567: \subsection* {Defaults change in prompter and burst}
568: The \switch{prepend} option is now the default in \pgm{prompter}.
569: The \switch{noinplace} option is now the default in \pgm{burst}.
570:
571: \subsection* {Interactive option in rmf}
572: The \pgm{rmf} program has been changed to support an \switch{interactive}
573: switch.
574: If given,
575: then the user is prompted regarding whether the folder should be deleted.
576: If the folder to be removed is not given by the user,
577: this switch is defaulted to on.
578:
579: \subsection* {Trusted Mail Interface}
580: \MH/ now has an interface for so-called ``trusted mail'' applications.
581: Although the modifications to \MH/ to support this are in the public domain,
582: the actual library that \MH/ uses is not.
583: Contact Professor David J.~Farber ({\sf Farber@UDel\/}) for more information.
584:
585: \bibliography{mh6}
586:
587: \showsummary
588:
589: \end{document}
This archive runs on limited infrastructure. Preserving old code on modern bandwidth. Automated agents are requested to crawl responsibly.