|
|
1.1 root 1: .OH 'RPC Protocol Spec''Page \\\\n(PN'
2: .EH 'Page \\\\n(PN''RPC Protocol Spec'
3: .OF 'Sun Microsystems''Release 2.0'
4: .EF 'Release 2.0''Sun Microsystems'
5: .RP
6: .rm DY
7: .TL
8: .ps 20
9: Remote Procedure Call
10: .sp.5
11: Protocol Specification
12: .
13: .H 1 "Introduction"
14: .LP
15: This document specifies a message protocol used in implementing
16: Sun's Remote Procedure Call (RPC) package.
17: The protocol is specified with the
18: eXternal Data Representation (XDR) language.
19: .LP
20: This document assumes that the reader is familiar with both RPC and XDR.
21: It does not attempt to justify RPC or its uses.
22: Also, the casual user of RPC does not need to be
23: familiar with the information in this document.
24: .
25: .H 2 "Terminology"
26: .LP
27: The document discusses servers, services,
28: programs, procedures, clients and versions.
29: A server is a machine where some number
30: of network services are implemented.
31: A service is a collection of one or more remote programs.
32: A remote program implements one or more remote procedures;
33: the procedures, their parameters and results are documented
34: in the specific program's protocol specification
35: (see Appendix 3 for an example).
36: Network clients are pieces of software that initiate
37: remote procedure calls to services.
38: A server may support more than one version of a remote program
39: in order to be forward compatible with changing protocols.
40: .LP
41: For example, a network file service may be composed of two programs.
42: One program may deal with high level applications
43: such as file system access control and locking.
44: The other may deal with low-level file I/O,
45: and have procedures like ``read'' and ``write''.
46: A client machine of the network file service would call
47: the procedures associated with the two programs of the service
48: on behalf of some user on the client machine.
49: .H 2 "The RPC Model"
50: .LP
51: The remote procedure call model is similar to
52: the local procedure call model.
53: In the local case, the caller places arguments to a procedure
54: in some well-specified location (such as a result register).
55: It then transfers control to the procedure,
56: and eventually gains back control.
57: At that point, the results of the procedure
58: are extracted from the well-specified location,
59: and the caller continues execution.
60: .LP
61: The remote procedure call is similar,
62: except that one thread of control winds through two processes \(em
63: one is the caller's process,
64: the other is a server's process.
65: That is, the caller process sends a call message
66: to the server process and waits (blocks) for a reply message.
67: The call message contains the procedure's parameters,
68: among other things.
69: The reply message contains the procedure's results,
70: among other things.
71: Once the reply message is received,
72: the results of the procedure are extracted,
73: and caller's execution is resumed.
74: .LP
75: On the server side,
76: a process is dormant awaiting the arrival of a call message.
77: When one arrives the server process extracts the procedure's parameters,
78: computes the results, sends a reply message,
79: and then awaits the next call message.
80: Note that in this model,
81: only one of the two processes is active at any given time.
82: That is, the RPC protocol does not explicitly support
83: multi-threading of caller or server processes.
84: .
85: .H 2 "Transports and Semantics"
86: .LP
87: The RPC protocol is independent of transport protocols.
88: That is, RPC does not care how a message is passed from one process to another.
89: The protocol only deals with the specification and interpretation of
90: messages.
91: .LP
92: Because of transport independence,
93: the RPC protocol does not attach specific semantics to the
94: remote procedures or their execution. Some semantics can be inferred from (but
95: should be explicitly specified by) the underlying transport protocol.
96: For example, RPC message passing using UDP/IP is unreliable.
97: Thus, if the caller retransmits call messages after short time-outs, the only thing he
98: can infer from no reply message is that the remote procedure was executed
99: zero or more times (and from a reply message, one or more times).
100: On the other hand, RPC message passing using TCP/IP is reliable.
101: No reply message means that the remote procedure was executed at most once,
102: whereas a reply message means that the remote procedure was exactly once.
103: (Note:
104: At Sun, RPC is currently implemented
105: on top of TCP/IP and UDP/IP transports.)
106: .
107: .H 2 "Binding and Rendezvous Independence"
108: .LP
109: The act of binding a client to a service is NOT part of
110: the remote procedure call specification.
111: This important and necessary function
112: is left up to some higher level software.
113: (The software may use RPC itself; see Appendix 3.)
114: .LP
115: Implementors should think of the RPC protocol as the
116: jump-subroutine instruction (``JSR'') of a network;
117: the loader (binder) makes JSR useful,
118: and the loader itself uses JSR to accomplish its task.
119: Likewise, the network makes RPC useful,
120: using RPC to accomplish this task.
121: .
122: .H 2 "Message Authentication"
123: .LP
124: The RPC protocol provides the fields necessary for a client to
125: identify himself to a service and vice versa.
126: Security and access control mechanisms
127: can be built on top of the message authentication.
128: .bp
129: .
130: .H 1 "Requirements"
131: .LP
132: The RPC protocol must provide for the following:
133: .DS
134: 1. Unique specification of a procedure to be called.
135: 2. Provisions for matching response messages to request messages.
136: 3. Provisions for authenticating the caller to service and vice versa.
137: .DE
138: Besides these requirements,
139: features that detect the following are worth supporting
140: because of protocol roll-over errors, implementation bugs,
141: user error, and network administration:
142: .DS
143: 1. RPC protocol mismatches.
144: 2. Remote program protocol version mismatches.
145: 3. Protocol errors (like mis-specification of a procedure's parameters).
146: 4. Reasons why remote authentication failed.
147: 5. Any other reasons why the desired procedure was not called.
148: .DE
149: .
150: .H 2 "Remote Programs and Procedures"
151: .LP
152: The RPC call message has three unsigned fields:
153: remote program number,
154: remote program version number,
155: and remote procedure number.
156: The three fields uniquely identify the procedure to be called.
157: Program numbers are administered by some central authority (like Sun).
158: Once an implementor has a program
159: number, he can implement his remote program;
160: the first implementation would
161: most likely have the version number of 1.
162: Because most new protocols evolve into better,
163: stable and mature protocols,
164: a version field of the call message identifies which version
165: of the protocol the caller is using.
166: Version numbers make speaking old and new protocols
167: through the same server process possible.
168: .LP
169: The procedure number identifies the procedure to be called.
170: These numbers are documented in
171: the specific program's protocol specification.
172: For example, a file service's protocol specification
173: may state that its procedure number 5 is
174: .L read
175: and procedure number 12 is
176: .L write .
177: .LP
178: Just as remote program protocols may change over several versions,
179: the actual RPC message protocol could also change.
180: Therefore, the call message also has the RPC version number in it;
181: this field must be two (2).
182: .LP
183: The reply message to a request message has enough information
184: to distinguish the following error conditions:
185: .IP 1)
186: The remote implementation of RPC does speak protocol version 2. The
187: lowest and highest supported RPC version numbers are returned.
188: .IP 2)
189: The remote program is not available on the remote system.
190: .IP 3)
191: The remote program does not support the requested version number.
192: The lowest and highest supported
193: remote program version numbers are returned.
194: .IP 4)
195: The requested procedure number does not exist
196: (this is usually a caller side protocol or programming error).
197: .IP 5)
198: The parameters to the remote procedure
199: appear to be garbage from the server's point of view.
200: (Again, this is caused by a disagreement about the
201: protocol between client and service.)
202: .
203: .H 2 "Authentication"
204: .LP
205: Provisions for authentication of caller to service and vice versa are
206: provided as a wart on the side of the RPC protocol. The call message
207: has two authentication fields, the credentials and verifier.
208: The reply message has one authentication field,
209: the response verifier.
210: The RPC protocol specification defines all three fields
211: to be the following opaque type:
212: .LS
213: enum auth_flavor {
214: AUTH_NULL = 0,
215: AUTH_UNIX = 1,
216: AUTH_SHORT = 2
217: /* and more to be defined */
218: };
219: .sp.5
220: struct opaque_auth {
221: union switch (enum auth_flavor) {
222: default: string auth_body<400>;
223: };
224: };
225: .LE
226: In simple English, any
227: .L opaque_auth
228: structure is an
229: .L auth_flavor
230: enumeration followed by a counted string,
231: whose bytes are opaque to the RPC protocol implementation.
232: .LP
233: The interpretation and semantics of the data contained
234: within the authentication fields is specified by individual,
235: independent authentication protocol specifications.
236: Appendix 1 defines three authentication protocols.
237: .LP
238: If authentication parameters were rejected,
239: the response message contains information stating
240: why they were rejected.
241: .
242: .H 2 "Program Number Assignment"
243: .LP
244: Program numbers are given out in groups of 0x20000000 (536870912)
245: according to the following chart:
246: .LS
247: 0 - 1fffffff defined by Sun
248: 20000000 - 3fffffff defined by user
249: 40000000 - 5fffffff transient
250: 60000000 - 7fffffff reserved
251: 80000000 - 9fffffff reserved
252: a0000000 - bfffffff reserved
253: c0000000 - dfffffff reserved
254: e0000000 - ffffffff reserved
255: .LE
256: The first group is a range of numbers administered by Sun Microsystems,
257: and should be identical for all Sun customers. The second range
258: is for applications peculiar to a particular customer.
259: This range is intended primarily for debugging new programs.
260: When a customer develops an application that might be of general
261: interest, that application should be given an assigned number
262: in the first range. The third group is for applications that
263: generate program numbers dynamically. The final groups
264: are reservered for future use, and should not be used.
265: .LP
266: The exact registration process for Sun defined numbers is yet
267: to be established.
268: .bp
269: .
270: .H 1 "Other Uses and Abuses of the RPC Protocol"
271: .LP
272: The intended use of this protocol is for calling remote procedures.
273: That is, each call message is matched with a response message.
274: However, the protocol itself is a message passing protocol
275: with which other (non-RPC) protocols can be implemented.
276: Sun currently uses (abuses) the RPC message protocol for the
277: following two (non-RPC) protocols:
278: batching (or pipelining) and broadcast RPC.
279: These two protocols are discussed (but not defined) below.
280: .
281: .H 2 "Batching"
282: .LP
283: Batching allows a client to send an arbitrarily large sequence
284: of call messages to a server;
285: batching uses reliable bytes stream protocols (like TCP/IP)
286: for their transport.
287: In the case of batching, the client never waits for a reply
288: from the server and the server does not send replies to batch requests.
289: A sequence of batch calls is usually terminated by a legitimate
290: RPC in order to flush the pipeline (with positive acknowledgement).
291: .
292: .H 2 "Broadcast RPC"
293: .LP
294: In broadcast RPC based protocols,
295: the client sends an a broadcast packet to
296: the network and waits for numerous replies.
297: Broadcast RPC uses unreliable, packet based protocols (like UDP/IP)
298: as their transports.
299: Servers that support broadcast protocols only respond
300: when the request is successfully processed,
301: and are silent in the face of errors.
302: .
303: .H 1 "The RPC Message Protocol"
304: .LP
305: This section defines the RPC message protocol in the XDR data description
306: language. The message is defined in a top down style.
307: Note: This is an XDR specification, not C code.
308: .LP
309: .LS 0
310: enum msg_type {
311: CALL = 0,
312: REPLY = 1
313: };
314: .LE
315: .LS 0
316: /*
317: * A reply to a call message can take on two forms:
318: * the message was either accepted or rejected.
319: */
320: enum reply_stat {
321: MSG_ACCEPTED = 0,
322: MSG_DENIED = 1
323: };
324: .LE
325: .LS 0
326: /*
327: * Given that a call message was accepted, the following is the status of
328: * an attempt to call a remote procedure.
329: */
330: enum accept_stat {
331: SUCCESS = 0, /* remote procedure was successfully executed */
332: PROG_UNAVAIL = 1, /* remote machine exports the program number */
333: PROG_MISMATCH = 2, /* remote machine can't support version number */
334: PROC_UNAVAIL = 3, /* remote program doesn't know about procedure */
335: GARBAGE_ARGS = 4 /* remote procedure can't figure out parameters */
336: };
337: .LE
338: .LS 0
339: /*
340: * Reasons why a call message was rejected:
341: */
342: enum reject_stat {
343: RPC_MISMATCH = 0, /* RPC version number was not two (2) */
344: AUTH_ERROR = 1 /* caller not authenticated on remote machine */
345: };
346: .LE
347: .LS 0
348: /*
349: * Why authentication failed:
350: */
351: enum auth_stat {
352: AUTH_BADCRED = 1, /* bogus credentials (seal broken) */
353: AUTH_REJECTEDCRED = 2, /* client should begin new session */
354: AUTH_BADVERF = 3, /* bogus verifier (seal broken) */
355: AUTH_REJECTEDVERF = 4, /* verifier expired or was replayed */
356: AUTH_TOOWEAK = 5, /* rejected due to security reasons */
357: };
358: .LE
359: .LS 0
360: /*
361: * The RPC message:
362: * All messages start with a transaction identifier, xid, followed by
363: * a two-armed discriminated union. The union's discriminant is a msg_type
364: * which switches to one of the two types of the message. The xid of a
365: * REPLY message always matches that of the initiating CALL message.
366: * NB: The xid field is only used for clients matching reply messages with
367: * call messages; the service side cannot treat this id as any type of
368: * sequence number.
369: */
370: struct rpc_msg {
371: unsigned xid;
372: union switch (enum msg_type) {
373: CALL: struct call_body;
374: REPLY: struct reply_body;
375: };
376: };
377: .LE
378: .LS 0
379: /*
380: * Body of an RPC request call:
381: * In version 2 of the RPC protocol specification, rpcvers must be equal to 2.
382: * The fields prog, vers, and proc specify the remote program, its version,
383: * and the procedure within the remote program to be called. These fields are
384: * followed by two authentication parameters, cred (authentication credentials)
385: * and verf (authentication verifier). The authentication parameters are
386: * followed * by the parameters to the remote procedure; these parameters are
387: * specified by the specific program protocol.
388: */
389: struct call_body {
390: unsigned rpcvers; /* must be equal to two (2) */
391: unsigned prog;
392: unsigned vers;
393: unsigned proc;
394: struct opaque_auth cred;
395: struct opaque_auth verf;
396: /* procedure specific parameters start here */
397: };
398: .LE
399: .LS 0
400: /*
401: * Body of a reply to an RPC request.
402: * The call message was either accepted or rejected.
403: */
404: struct reply_body {
405: union switch (enum reply_stat) {
406: MSG_ACCEPTED: struct accepted_reply;
407: MSG_DENIED: struct rejected_reply;
408: };
409: };
410: .LE
411: .LS 0
412: /*
413: * Reply to an RPC request that was accepted by the server.
414: * Note: there could be an error even though the request was accepted.
415: * The first field is an authentication verifier which the server generates
416: * in order to validate itself to the caller. It is followed by a union
417: * whose discriminant is an enum accept_stat. The SUCCESS arm of the union is
418: * protocol specific. The PROG_UNAVAIL, PROC_UNAVAIL, and GARBAGE_ARGS arms
419: * of the union are void. The PROG_MISMATCH arm specifies the lowest and
420: * highest version numbers of the remote program that are supported by the
421: * server.
422: */
423: struct accepted_reply {
424: struct opaque_auth verf;
425: union switch (enum accept_stat) {
426: SUCCESS: struct {
427: /*
428: * procedure-specific results start here
429: */
430: };
431: PROG_MISMATCH: struct {
432: unsigned low;
433: unsigned high;
434: };
435: default: struct {
436: /*
437: * void. Cases include PROG_UNAVAIL,
438: * PROC_UNAVAIL, and GARBAGE_ARGS.
439: */
440: };
441: };
442: };
443: .LE
444: .LS 0
445: /*
446: * Reply to an RPC request that was rejected by the server.
447: * The request can be rejected because of two reasons - either the server is
448: * not running a compatible version of the RPC protocol (RPC_MISMATCH), or
449: * the server refused to authenticate the caller (AUTH_ERROR). In the case of
450: * an RPC version mismatch, the server returns the lowest and highest supported
451: * RPC version numbers. In the case of refused authentication, the failure
452: * status is returned.
453: */
454: struct rejected_reply {
455: union switch (enum reject_stat) {
456: RPC_MISMATCH: struct {
457: unsigned low;
458: unsigned high;
459: };
460: AUTH_ERROR: enum auth_stat;
461: };
462: };
463: .LE
464: .bp
465: .
466: .H 1 "Appendix 1: Authentication Parameters Specification"
467: .LP
468: As previously stated, authentication parameters are opaque,
469: but open-ended to the rest of the
470: RPC protocol. This section defines some ``flavors'' of authentication
471: which have been implemented at (and supported by) Sun.
472: .
473: .H 2 "Null Authentication"
474: .LP
475: Often calls must be made where the caller does not know who he is and
476: the server does not care who the caller is. In this case, the auth_flavor value
477: (the discriminant of the opaque_auth's union) of the
478: RPC message's credentials, verifier, and response verifier
479: is AUTH_NULL (0).
480: The bytes of the auth_body string are undefined.
481: It is recommended that the string length be zero.
482: .
483: .H 2 "UNIX Authentication"
484: .LP
485: The caller of a remote procedure may wish to identify himself as he is
486: identified on a
487: .UX
488: system.
489: The value of the
490: .I credential's
491: discriminant of
492: an RPC call message is AUTH_UNIX (1). The bytes of the
493: .I credential's
494: string
495: encode the the following (XDR) structure:
496: .LS
497: struct auth_unix {
498: unsigned stamp;
499: string machinename<255>;
500: unsigned uid;
501: unsigned gid;
502: unsigned gids<10>;
503: };
504: .LE
505: The
506: .L stamp
507: is an arbitrary id which the caller machine may generate.
508: The
509: .L machinename
510: is the name of the caller's machine (like ``krypton'').
511: The
512: .L uid
513: is the caller's effective user id.
514: The
515: .L gid
516: is the callers effective group id.
517: The
518: .L gids
519: is a counted array of groups
520: which contain the caller as a member.
521: The
522: .I verifier
523: accompanying the credentials should be of AUTH_NULL (defined above).
524: .LP
525: The value of the discriminate of the
526: .I "response verifier"
527: received in the reply message from the server may be
528: AUTH_NULL or AUTH_SHORT (2).
529: In the case of AUTH_SHORT, the bytes of the
530: .I "response verifier" 's
531: string encode an
532: .L auth_opaque
533: structure.
534: This new
535: .L auth_opaque
536: structure may now be passed to the server
537: instead of the original AUTH_UNIX flavor credentials.
538: The server keeps a cache which maps short hand
539: .I auth_opaque
540: structures (passed back via a AUTH_SHORT style
541: .L "response verifier" )
542: to the original credentials of the caller.
543: The caller can save network bandwidth and server cpu cycles
544: by using the new credentials.
545: .LP
546: The server may flush the short hand
547: .I auth_opaque
548: structure at any time.
549: If this happens, the remote procedure call message will be rejected due to an
550: authentication error. The reason for the failure will be AUTH_REJECTEDCRED.
551: At this point, the caller may wish to try the original AUTH_UNIX style of
552: credentials.
553: .
554: .H 1 "Appendix 2: Record Marking Standard (RM)"
555: .LP
556: When RPC messages are passed on top of a byte stream protocol
557: (like TCP/IP), it is necessary, or at least desirable,
558: to delimit one message from another in order to detect
559: and possibly recover from user protocol errors.
560: Sun uses this RM/TCP/IP transport for passing
561: RPC messages on TCP streams.
562: One RPC message fits into one RM record.
563: .LP
564: A record is composed of one or more record fragments.
565: A record fragment is a four-byte header followed by
566: .I 0
567: to
568: .I "2\s-2\u31\d\s+2\-1"
569: bytes of fragment data.
570: The bytes encode an unsigned binary number;
571: as with XDR integers, the byte order is from highest to lowest.
572: The number encodes two values \(em
573: a boolean which indicates whether the fragment is the last fragment
574: of the record (bit value 1 implies the fragment is the last fragment)
575: and a 31-bit unsigned binary value which is the length in bytes of the
576: fragment's data.
577: The boolean value is the highest-order bit of the header;
578: the length is the 31 low-order bits.
579: (Note that this record specification is
580: .I not
581: in XDR standard form!)
582: .
583: .H 1 "Appendix 3: Port Mapper Program Protocol"
584: .LP
585: The port mapper program maps RPC program and version numbers
586: to UDP/IP or TCP/IP port numbers.
587: This program makes dynamic binding of remote programs possible.
588: .LP
589: This is desirable because the range of reserved port numbers is very small
590: and the number of potential remote programs is very large. By running only
591: the port mapper on a reserved port, the port numbers of other remote programs
592: can be ascertained by querying the port mapper.
593: .
594: .H 2 "The Port Mapper RPC Protocol"
595: .LP
596: The protocol is specified by the XDR description language.
597: .LP
598: .LS 0
599: Port Mapper RPC Program Number: 100000
600: Version Number: 1
601: Supported Transports:
602: UDP/IP on port 111
603: RM/TCP/IP on port 111
604: .LE
605: .LS 0
606: /*
607: * Handy transport protocol numbers
608: */
609: #define IPPROTO_TCP 6 /* protocol number used for rpc/rm/tcp/ip */
610: #define IPPROTO_UDP 17 /* protocol number used for rpc/udp/ip */
611: .LE
612: .LS 0
613: /* Procedures */
614:
615: /*
616: * Convention: procedure zero of any protocol takes no parameters
617: * and returns no results.
618: */
619: 0. PMAPPROC_NULL () returns ()
620: .LE
621: .LS 0
622: /*
623: * Procedure 1, setting a mapping:
624: * When a program first becomes available on a
625: * machine, it registers itself with the port mapper program on the
626: * same machine. The program passes its program number (prog),
627: * version number (vers), transport protocol number (prot),
628: * and the port (port) on which it awaits service request. The
629: * procedure returns success whose value is TRUE if the procedure
630: * successfully established the mapping and FALSE otherwise. The
631: * procedure will refuse to establish a mapping if one already exists
632: * for the tuple [prog, vers, prot].
633: */
634: 1. PMAPPROC_SET (prog, vers, prot, port) returns (success)
635: unsigned prog;
636: unsigned vers;
637: unsigned prot;
638: unsigned port;
639: boolean success;
640: .LE
641: .LS 0
642: /*
643: * Procedure 2, Unsetting a mapping:
644: * When a program becomes unavailable, it should unregister itself
645: * with the port mapper program on the same machine. The parameters
646: * and results have meanings identical to those of PMAPPROC_SET.
647: */
648: 2. PMAPPROC_UNSET (prog, vers, dummy1, dummy2) returns (success)
649: unsigned prog;
650: unsigned vers;
651: unsigned dummy1; /* this value is always ignored */
652: unsigned dummy2; /* this value is always ignored */
653: boolean success;
654: .LE
655: .LS 0
656: /*
657: * Procedure 3, looking-up a mapping:
658: * Given a program number (prog), version number (vers) and
659: * transport protocol number (prot), this procedure returns the port
660: * number on which the program is awaiting call requests. A port
661: * value of zeros means that the program has not been registered.
662: */
663: 3. PMAPPROC_GETPORT (prog, vers, prot, dummy) returns (port)
664: unsigned prog;
665: unsigned vers;
666: unsigned prot;
667: unsigned dummy; /* this value is always ignored */
668: unsigned port; /* zero means the program is not registered */
669: .LE
670: .LS 0
671: /*
672: * Procedure 4, dumping the mappings:
673: * This procedure enumerates all entries in the port mapper's database.
674: * The procedure takes no parameters and returns a ``list'' of
675: * [program, version, prot, port] values.
676: */
677: 4. PMAPPROC_DUMP () returns (maplist)
678: struct maplist {
679: union switch (boolean) {
680: FALSE: struct { /* void, end of list */ };
681: TRUE: struct {
682: unsigned prog;
683: unsigned vers;
684: unsigned prot;
685: unsigned port;
686: struct maplist the_rest;
687: };
688: };
689: } maplist;
690: .LE
691: .LS 0
692: /*
693: * Procedure 5, indirect call routine:
694: * The procedures allows a caller to call another remote procedure
695: * on the same machine without knowing the remote procedure's port
696: * number. Its intended use is for supporting broadcasts to arbitrary
697: * remote programs via the well-known port mapper's port. The parameters
698: * prog, vers, proc, and the bytes of args are the program number,
699: * version number, procedure number, and parameters the the remote
700: * procedure.
701: *
702: * NB:
703: * 1. This procedure only sends a response if the procedure was
704: * successfully executed and is silent (No response) otherwise.
705: * 2. The port mapper communicates with the remote program via
706: * UDP/IP only.
707: *
708: * The procedure returns the port number of the remote program and
709: * the bytes of results are the results of the remote procedure.
710: */
711: 5. PMAPPROC_CALLIT (prog, vers, proc, args) returns (port, results)
712: unsigned prog;
713: unsigned vers;
714: unsigned proc;
715: string args<>;
716: unsigned port;
717: string results<>;
718: .LE
This archive runs on limited infrastructure. Preserving old code on modern bandwidth. Automated agents are requested to crawl responsibly.