Annotation of 43BSD/etc/named/doc/rfc973.lpr, revision 1.1

1.1     ! root        1: 
        !             2: 
        !             3: Network Working Group                                   Paul Mockapetris
        !             4: Request for Comments: 973                                            ISI
        !             5:                                                             January 1986
        !             6: 
        !             7:                  Domain System Changes and Observations
        !             8: 
        !             9: 
        !            10: STATUS OF THIS MEMO
        !            11: 
        !            12:    This RFC documents updates to Domain Name System specifications
        !            13:    RFC-882 [1] and RFC-883 [2], suggests some operational guidelines,
        !            14:    and discusses some experiences and problem areas in the present
        !            15:    system.  Distribution of this memo is unlimited.
        !            16: 
        !            17:    This document includes all changes to the Domain System through
        !            18:    January, 1986.  Change notices and additional discussion are
        !            19:    available online in file [USC-ISIB.ARPA]<DOMAIN>DOMAIN.CHANGES.
        !            20: 
        !            21: OVERVIEW
        !            22: 
        !            23:    This memo is divided into four major sections:
        !            24: 
        !            25:       "UPDATES" which discusses changes to the domain specification
        !            26:       which are in widespread use and should be regarded as being part
        !            27:       of the specification.
        !            28: 
        !            29:       "OPERATION GUIDELINES" which suggests rules-of-thumb for using the
        !            30:       domain system and configuring your database which are appropriate
        !            31:       in most cases, but which may have rare exceptions.
        !            32: 
        !            33:       "EXPERIENCES" which discusses some unusual situations and common
        !            34:       bugs which are encountered in the present system, and should be
        !            35:       helpful in problem determination and tuning.
        !            36: 
        !            37:       "PROBLEM AREAS" which discusses some shortcomings in the present
        !            38:       system which may be addressed in future versions.
        !            39: 
        !            40: UPDATES
        !            41: 
        !            42:    This section discusses changes to the specification which are final,
        !            43:    and should be incorporated in all domain system software.
        !            44: 
        !            45:    TTL timeouts too small
        !            46: 
        !            47:       The 16 bit TTL field in RRs could not represent a large enough
        !            48:       time interval.  The 16 bit field, using seconds for units, has a
        !            49:       maximum period of approximately 18 hours.
        !            50: 
        !            51:       All time values, including all TTLs and the MINIMUM field of the
        !            52:       SOA RR, are expanded to 32 bits.
        !            53: 
        !            54: 
        !            55: 
        !            56: Mockapetris                                                     [Page 1]
        !            57: 
        !            58: 
        !            59: 
        !            60: RFC 973                                                     January 1986
        !            61: Domain System Changes and Observations
        !            62: 
        !            63: 
        !            64:    CLASS changes
        !            65: 
        !            66:       Class 2, originally reserved for CSNET, is obsolete.  Class 3 has
        !            67:       been assigned for use by CHAOS.
        !            68: 
        !            69:    CNAME usage
        !            70: 
        !            71:       The specification allows CNAME RRs to exist with other RRs at the
        !            72:       same node.  This creates difficulties since the other RRs stored
        !            73:       with the CNAME at the alias might not agree with the RRs stored at
        !            74:       the primary name.
        !            75: 
        !            76:       If a node has a CNAME RR, it should have no other RRs.
        !            77: 
        !            78:    * semantics
        !            79: 
        !            80:       The use of * to represent a single label wildcard, along with the
        !            81:       possibility of multiple * labels, led to difficult server
        !            82:       implementations and complicated search algorithms.  There were
        !            83:       also questions regarding whether a * based specification could
        !            84:       refer to names that were not contained in the zone which had the *
        !            85:       specification.
        !            86: 
        !            87:       While we might want the "inheritability" for some cases, it leads
        !            88:       to implementation difficulties.  The first of these is that
        !            89:       whenever we can't find a RR in a particular zone, we have to
        !            90:       search all parent zones to look for a suitable * result.
        !            91:       (Alternatively we could develop some automatic method for insuring
        !            92:       consistency or insist on careful duplication of inherited data.)
        !            93:       We also must deal with conflicts, i.e. what if a subdomain doesn't
        !            94:       want to inherit defaults.
        !            95: 
        !            96:       Given these difficulties, the solution is to insist that
        !            97:       delegation of authority cancels the * defaults.  This is quite
        !            98:       simple to implement; all you need to do is to check for delegation
        !            99:       before looking for * RRs.
        !           100: 
        !           101:       A second difficulty is the restriction that * match a single
        !           102:       label.  Thus if a name server is looking for RRs for the name
        !           103:       A.B.C.D.E.F, it must check for *.B.C.D.E.F, *.*.C.D.E.F,
        !           104:       *.*.*.D.E.F, etc.  This check must also be careful of zone
        !           105:       boundaries and multiplies the effort to handle a query.
        !           106: 
        !           107:       The solution adopted is to allow a single * label in the leftmost
        !           108:       part of a name stored in a zone, and to allow this label to match
        !           109: 
        !           110: 
        !           111: 
        !           112: 
        !           113: Mockapetris                                                     [Page 2]
        !           114: 
        !           115: 
        !           116: 
        !           117: RFC 973                                                     January 1986
        !           118: Domain System Changes and Observations
        !           119: 
        !           120: 
        !           121:       any number of unknown labels or a single known label in the query
        !           122:       name.  However, the * match is only taken for parts of the tree
        !           123:       which are neither delegated or explicitly represented.
        !           124: 
        !           125:       The algorithm for performing the search in a tree structured
        !           126:       database has the following steps:
        !           127: 
        !           128:       1) Descend in the tree matching labels from right to left.  If a
        !           129:       delegation is found return that;  if the specified node is found
        !           130:       go to step 2, if the tree ends go to step 3.
        !           131: 
        !           132:       2) Look for RRs that answer the query.  If any are found, return
        !           133:       them as the answer.  If none are found, look for answers in a *
        !           134:       node which has the same name as the query name except for the
        !           135:       rightmost label.  (e.g. if you can't find an answer at F.ISI.ARPA,
        !           136:       look for a RR at *.ISI.ARPA)
        !           137: 
        !           138:       3) The search for a desired name has failed; look for a node whose
        !           139:       name is * plus however much matched.  Look for answers there.
        !           140:       (e.g. If you are looking for X.Y.ISI.ARPA and the tree ends at
        !           141:       ISI.ARPA, look at *.ISI.ARPA.  The same thing holds for
        !           142:       Y.ISI.ARPA, or any name of the form <anything>.Z.ISI.ARPA, where Z
        !           143:       is a label that doesn't exist under ISI.ARPA)
        !           144: 
        !           145:       Note that this interpretation means that * matches names that are
        !           146:       not in the tree, no matter how much of the tree is missing, and
        !           147:       also matches one level's worth of known tree.
        !           148: 
        !           149:    AA semantics
        !           150: 
        !           151:       When a name server is responding to a query for a particular name
        !           152:       and finds a CNAME, it may optionally restart the search at the
        !           153:       canonical name.  If the server uses the restart feature, the
        !           154:       answer section of the returned query contains one (or more)
        !           155:       CNAMEs, possibly followed by answers for the primary name.  The
        !           156:       canonical name will usually be in the same zone as the alias, but
        !           157:       this need not be the case.  If the server is authoritative for one
        !           158:       of the names but not both, it is not clear whether the AA bit
        !           159:       should be set.
        !           160: 
        !           161:       The solution adopted is to make the AA refer to the original query
        !           162:       name.
        !           163: 
        !           164: 
        !           165: 
        !           166: 
        !           167: 
        !           168: 
        !           169: 
        !           170: Mockapetris                                                     [Page 3]
        !           171: 
        !           172: 
        !           173: 
        !           174: RFC 973                                                     January 1986
        !           175: Domain System Changes and Observations
        !           176: 
        !           177: 
        !           178:    Master file format
        !           179: 
        !           180:       The present specification uses a somewhat awkward method for
        !           181:       representing domain names in master files.
        !           182: 
        !           183:       The change adopted is that all domain names in this file will be
        !           184:       represented as either absolute or relative.  An absolute domain
        !           185:       name ends with a ".".  A free standing "." is assumed to refer to
        !           186:       the root.  A relative domain name doesn't end with a dot, and is
        !           187:       assumed to be relative to the current origin.
        !           188: 
        !           189:    SERIAL number size
        !           190: 
        !           191:       If the master file changes rapidly, an infrequently updated copy
        !           192:       may miss the wrapping of the sequence number in the SERIAL field
        !           193:       of the SOA, or misinterpret the number of updates that have taken
        !           194:       place.
        !           195: 
        !           196:       The SERIAL field is increased to 32 bits.
        !           197: 
        !           198:    MD and MF replaced by MX
        !           199: 
        !           200:       The original specification uses MD and MF RRs for mail agent
        !           201:       binding.  The problem is that a mailer making a MAILA query, which
        !           202:       asks for both types, can't use the cache since the cache might
        !           203:       have the results for a MD or MF query.  That is, the presence of
        !           204:       one of these types of information in the cache doesn't imply
        !           205:       anything about the other type.  The result was that either mailers
        !           206:       would have to always consult authoritative servers or try to use
        !           207:       partial information; neither of these is really acceptable.
        !           208: 
        !           209:       The change is to replace MD and MF with a new type of RR called MX
        !           210:       which conveys similar information in a single RR type.  MX has
        !           211:       been assigned a type code of 15 decimal.  The format of the MX RR
        !           212:       is a 16 bit preference value followed by a domain name.  A node
        !           213:       may have multiple MX RRs, and multiple MX RRs with the same
        !           214:       preference value are allowed at a given node.
        !           215: 
        !           216: 
        !           217: 
        !           218: 
        !           219: 
        !           220: 
        !           221: 
        !           222: 
        !           223: 
        !           224: 
        !           225: 
        !           226: 
        !           227: Mockapetris                                                     [Page 4]
        !           228: 
        !           229: 
        !           230: 
        !           231: RFC 973                                                     January 1986
        !           232: Domain System Changes and Observations
        !           233: 
        !           234: 
        !           235:       The preference values denote the relative preference that the mail
        !           236:       destination places on the mail agents, with lower values being
        !           237:       "better".  A mailer is expected to at least try the mail agent(s)
        !           238:       with the lowest preference value.  The significance of particular
        !           239:       preference values, the units of preference, and the linearity of
        !           240:       preference values are not defined but left open; preference values
        !           241:       should only be used to establish relative rankings.
        !           242: 
        !           243:       For example, the current RRs:
        !           244: 
        !           245:                        MAIL-ORG   MD    HOST1    
        !           246:                                   MD    HOST2    
        !           247:                                   MF    HOST3    
        !           248: 
        !           249:       might be replaced by:
        !           250: 
        !           251:                        MAIL-ORG   MX    10 HOST1 
        !           252:                                   MX    10 HOST2 
        !           253:                                   MX    20 HOST3 
        !           254: 
        !           255:       The values 10 and 20 have no significance other than 10<20.  A
        !           256:       detailed discussion of the use of MX is the subject of [3].
        !           257: 
        !           258:    Zone transfer
        !           259: 
        !           260:       The original specification states that zone transfers take place
        !           261:       in breadth first order.  The intent was to make the transfer
        !           262:       easier for the accepting name server to handle.  This now doesn't
        !           263:       work out to be very helpful, and is a severe pain for implementers
        !           264:       using various hashing algorithms.  The new rule is that you can
        !           265:       transmit the records in any order you choose, so long as the SOA
        !           266:       node of the zone is transmitted first and last, and no other
        !           267:       duplication occurs.
        !           268: 
        !           269:    IN-ADDR domain renamed
        !           270: 
        !           271:       The name of the IN-ADDR domain is now IN-ADDR.ARPA.  This change
        !           272:       was made because many felt that the use of a top-level name was
        !           273:       inappropriate to network-specific information.
        !           274: 
        !           275: 
        !           276: 
        !           277: 
        !           278: 
        !           279: 
        !           280: 
        !           281: 
        !           282: 
        !           283: 
        !           284: Mockapetris                                                     [Page 5]
        !           285: 
        !           286: 
        !           287: 
        !           288: RFC 973                                                     January 1986
        !           289: Domain System Changes and Observations
        !           290: 
        !           291: 
        !           292: OPERATIONAL GUIDELINES
        !           293: 
        !           294:    This section suggests rules-of-thumb for using the domain system and
        !           295:    configuring your database which are appropriate in most cases, but
        !           296:    which may have rare exceptions.
        !           297: 
        !           298:    Zone delegation
        !           299: 
        !           300:       When a domain wishes to become independent from its parent, the
        !           301:       RRs which mark the delegation in the parent and child zones should
        !           302:       be carefully synchronized to minimize the possibility that
        !           303:       resolvers become confused.
        !           304: 
        !           305:       For example, suppose that we wish to create a new zone called
        !           306:       ISI.EDU under an existing EDU zone, and that the servers for the
        !           307:       child zone are X.ISI.EDU and Y.GOV.
        !           308: 
        !           309:       We might add the following to the parent zone:
        !           310: 
        !           311:        ISI.EDU.      10000 NS  X.ISI.EDU.              
        !           312:                      10000 NS  Y.GOV.                  
        !           313:        X.ISI.EDU.    10000 A   <address of X.ISI.EDU.> 
        !           314:        Y.GOV.        10000 A   <address of Y.GOV.>     
        !           315: 
        !           316:       and the following to the child zone:
        !           317: 
        !           318:        ISI.EDU.      10000 NS  X.ISI.EDU.              
        !           319:                      10000 NS  Y.GOV.                  
        !           320:                      50000 SOA <SOA information>       
        !           321:        X.ISI.EDU.    10000 A   <address of X.ISI.EDU.> 
        !           322:        Y.GOV.        10000 A   <address of Y.GOV.>     
        !           323: 
        !           324:       Note the following:
        !           325: 
        !           326:          In both cases, the A RR for Y.GOV is included, even though
        !           327:          Y.GOV isn't in the EDU or ISI.EDU domains.  This RR isn't
        !           328:          authoritative, but is included to guarantee that the address of
        !           329:          Y.GOV is passed with delegations to it.  Strictly speaking this
        !           330:          RR need not be in either zone, but its presence is recommended.
        !           331:          The X.ISI.EDU A RR is absolutely essential.  The only time that
        !           332:          a server should use the glue RRs is when it is returning the NS
        !           333:          RRs and doesn't otherwise have the address of the server.  For
        !           334:          example, if the parent server also was authoritative for GOV,
        !           335:          the glue RR would typically not be consulted.  However, it is
        !           336:          still a good idea for it to be present, so that the zone is
        !           337:          self-sufficient.
        !           338: 
        !           339: 
        !           340: 
        !           341: Mockapetris                                                     [Page 6]
        !           342: 
        !           343: 
        !           344: 
        !           345: RFC 973                                                     January 1986
        !           346: Domain System Changes and Observations
        !           347: 
        !           348: 
        !           349:          The child zone and the parent zone have identical NS RRs for
        !           350:          the ISI.EDU domain.  This guarantees that no matter which
        !           351:          server is asked about the ISI.EDU domain, the same set of name
        !           352:          servers is returned.
        !           353: 
        !           354:          The child zone and the parent zone have A RRs for the name
        !           355:          servers in the NS RRs that delegate the ISI.EDU domain.  This
        !           356:          guarantees that in addition to knowing the name servers for the
        !           357:          ISI.EDU domain, the addresses of the servers are known as well.
        !           358: 
        !           359:          The TTLs for the NS RRs that delegate the ISI.EDU domain and
        !           360:          the A RRs that represent the addresses of the name servers are
        !           361:          all the same.  This guarantees that all of these RRs will
        !           362:          timeout simultaneously.  In this example, the value 10000 has
        !           363:          no special significance, but the coincidence of the TTLs is
        !           364:          significant.
        !           365: 
        !           366:       These guidelines haven't changed any of the flexibility of the
        !           367:       system; the name of a name server and the domains it serves are
        !           368:       still independent.
        !           369: 
        !           370:       It might also be the case that the organization called ISI wanted
        !           371:       to take over management of the IN-ADDR domain for an internal
        !           372:       network, say 128.99.0.0.  In this case, we would have additions to
        !           373:       the parent zone, say IN-ADDR.ARPA.
        !           374: 
        !           375:       We might add the following to the parent zone:
        !           376: 
        !           377:        99.128.IN-ADDR.ARPA. 2000 NS  Q.ISI.EDU.               
        !           378:                             2000 NS  XX.MIT.EDU.              
        !           379:        Q.ISI.EDU.           2000 A   <address of Q.ISI.EDU.>  
        !           380:        XX.MIT.EDU.          2000 A   <address of XX.MIT.EDU.> 
        !           381: 
        !           382:       and the following to the child zone:
        !           383: 
        !           384:        99.128.IN-ADDR.ARPA. 2000 NS  Q.ISI.EDU.               
        !           385:                             2000 NS  XX.MIT.EDU.              
        !           386:                             5000 SOA <SOA information>        
        !           387:        Q.ISI.EDU.           2000 A   <address of Q.ISI.EDU.>  
        !           388:        XX.MIT.EDU.          2000 A   <address of XX.MIT.EDU.> 
        !           389: 
        !           390:    SOA serials
        !           391: 
        !           392:       The serial field of the SOA RR for a domain is supposed to be a
        !           393:       continuously increasing (mod 2**32) value which denotes the
        !           394: 
        !           395: 
        !           396: 
        !           397: 
        !           398: Mockapetris                                                     [Page 7]
        !           399: 
        !           400: 
        !           401: 
        !           402: RFC 973                                                     January 1986
        !           403: Domain System Changes and Observations
        !           404: 
        !           405: 
        !           406:       version of the database.  The idea is that you can tell that a
        !           407:       zone has changed by comparing serial numbers.  When you change a
        !           408:       zone, you should increment the serial field of the SOA.
        !           409: 
        !           410:    All RRs with the same name, class, and type should have the same TTL.
        !           411: 
        !           412:       The logic here is that all of them will timeout simultaneously if
        !           413:       cached and hence the cache can be reliably used.
        !           414: 
        !           415:    Case consistency
        !           416: 
        !           417:       The domain system is supposed to preserve case, but be case
        !           418:       insensitive.  However, it does nobody any good to put both RRs for
        !           419:       domain name xxx and XXX in the data base - It merely makes caching
        !           420:       ambiguous and decreases the efficiency of compression.  This
        !           421:       consistency should also exist in the duplicate RRs that mark
        !           422:       delegation in the delegator and delegatee.  For example, if you
        !           423:       ask the NIC to delegate UZOO.EDU to you, your database shouldn't
        !           424:       say uzoo.edu.
        !           425: 
        !           426:    Inappropriate use of aliases
        !           427: 
        !           428:       Canonical names are preferred to aliases in all RRs.  One reason
        !           429:       is that the canonical names are closer to the information
        !           430:       associated with a name.  A second is that canonical names are
        !           431:       unique, and aliases are not, and hence comparisons will work.
        !           432: 
        !           433:       In particular, the use of aliases in PTR RRs of the IN-ADDR domain
        !           434:       or in NS RRs that mark delegation is discouraged.
        !           435: 
        !           436: EXPERIENCES
        !           437: 
        !           438:    This section discusses some unusual situations and common bugs which
        !           439:    are encountered in the present system, and should be helpful in
        !           440:    problem determination and tuning.  Put differently, you should try to
        !           441:    make your code defend against these attacks, and you should expect to
        !           442:    be the object of complaint if you make these attacks.
        !           443: 
        !           444:    UDP addresses
        !           445: 
        !           446:       When you send a query to a host with multiple addresses, you might
        !           447:       expect the response to be from the address to which you sent the
        !           448:       query.  This isn't the case with almost all UNIX implementations.
        !           449: 
        !           450: 
        !           451: 
        !           452: 
        !           453: 
        !           454: 
        !           455: Mockapetris                                                     [Page 8]
        !           456: 
        !           457: 
        !           458: 
        !           459: RFC 973                                                     January 1986
        !           460: Domain System Changes and Observations
        !           461: 
        !           462: 
        !           463:    UDP checksums
        !           464: 
        !           465:       Many versions of UNIX generate incorrect UDP checksums, and most
        !           466:       ignore the checksum of incoming UDP datagrams.  The typical
        !           467:       symptom is that your UNIX domain code works fine with other
        !           468:       UNIXes, but won't communicate with TOPS-20 or other systems.
        !           469:       (JEEVES, the TOPS-20 server used for 3 of the 4 root servers,
        !           470:       ignores datagrams with bad UDP checksums.)
        !           471: 
        !           472:    Making up data
        !           473: 
        !           474:       There are lots of name servers which return RRs for the root
        !           475:       servers with 99999999 or similar large values in the TTL.  For
        !           476:       example, some return RRs that suggest that ISIF is a root server.
        !           477:       (It was months ago, but is no longer.)
        !           478: 
        !           479:       One of the main ideas of the domain system is that everybody can
        !           480:       get a chunk of the name space to manage as they choose.  However,
        !           481:       you aren't supposed to lie about other parts of the name space.
        !           482:       Its OK to remember about other parts of the name space for caching
        !           483:       or other purposes, but you are supposed to follow the TTL rules.
        !           484: 
        !           485:       Now it may be that you put such records in your server or whatever
        !           486:       to ensure a server of last resort.  That's fine.  But if you
        !           487:       export these in answers to queries, you should be shot.  These
        !           488:       entries get put in caches and never die.
        !           489: 
        !           490:       Suggested domain meta-rule:
        !           491: 
        !           492:          If you must lie, lie only to yourself.
        !           493: 
        !           494: PROBLEM AREAS
        !           495: 
        !           496:    This section discusses some shortcomings in the present system which
        !           497:    may be addressed in future versions.
        !           498: 
        !           499:    Compression and types
        !           500: 
        !           501:       The present specification attempts to allow name servers and
        !           502:       resolvers to cache RRs for classes they don't "understand" as well
        !           503:       as to allow compression of domain names to minimize the size of
        !           504:       UDP datagrams.  These two goals conflict in the present scheme
        !           505:       since the only way to expand a compressed name is to know that a
        !           506:       name is expected in that position.
        !           507: 
        !           508:       One technique for addressing this problem would be to preface
        !           509:       binary data (i.e. anything but a domain name) with a length octet.
        !           510: 
        !           511: 
        !           512: Mockapetris                                                     [Page 9]
        !           513: 
        !           514: 
        !           515: 
        !           516: RFC 973                                                     January 1986
        !           517: Domain System Changes and Observations
        !           518: 
        !           519: 
        !           520:       The high order two bits of the length octet could contain either
        !           521:       01 or 10, which are illegal for domain names.  To compensate for
        !           522:       the additional bytes of data, we could omit the RDATA length field
        !           523:       and terminate each RR with a binary length field of zero.
        !           524: 
        !           525:    Caching non-existent names
        !           526: 
        !           527:       In the present system, a resolver has no standard method for
        !           528:       caching the result that a name does not exist, which seems to make
        !           529:       up a larger than expected percentage of queries.  Some resolvers
        !           530:       create "does not exist" RRs with TTLs to guarantee against
        !           531:       repetitive queries for a non-existent name.
        !           532: 
        !           533:       A standard technique might be to return the SOA RR for the zone
        !           534:       (note that only authoritative servers can say name does not exist)
        !           535:       in the reply, and define the semantics to be that the requester is
        !           536:       free to assume that the name does not exist for a period equal to
        !           537:       the MINIMUM field of the SOA.
        !           538: 
        !           539:    Cache conflicts
        !           540: 
        !           541:       When a resolver is processing a reply, it may well decide to cache
        !           542:       all RRs found in sections of the reply.  The problem is that the
        !           543:       resolver's cache may already contain a subset of these RRs,
        !           544:       probably with different TTLs.
        !           545: 
        !           546:       If the RRs are from authoritative data in the answer section, then
        !           547:       the cache RRs should be replaced.  In other cases, the correct
        !           548:       strategy isn't completely clear.  Note that if the authoritative
        !           549:       data's TTL has changed, then the resolver doesn't have enough
        !           550:       information to make the correct decision in all cases.
        !           551: 
        !           552:       This issue is tricky, and deserves thought.
        !           553: 
        !           554: REFERENCES
        !           555: 
        !           556:    [1]  Mockapetris, P., "Domain Names - Concepts and Facilities",
        !           557:         RFC-882, USC Information Sciences Institute, November 1983.
        !           558: 
        !           559:    [2]  Mockapetris, P., "Domain Names - Implementation and
        !           560:         Specification", RFC-883, USC Information Sciences Institute,
        !           561:         November 1983.
        !           562: 
        !           563:    [3]  Partridge, C., "Mail Routing and the Domain System", RFC-974,
        !           564:         CSNET-CIC, BBN Laboratories, January 1986.
        !           565: 
        !           566: 
        !           567: 
        !           568: 
        !           569: Mockapetris                                                    [Page 10]
        !           570: 

unix.superglobalmegacorp.com

This archive runs on limited infrastructure. Preserving old code on modern bandwidth. Automated agents are requested to crawl responsibly.