--- pgp/doc/pgformat.doc 2018/04/24 16:40:35 1.1.1.2 +++ pgp/doc/pgformat.doc 2018/04/24 16:42:46 1.1.1.4 @@ -1,5 +1,8 @@ -Internal Data Structures Used by PGP 2.3 (14 Jun 93) -========================================================== +File Formats Used by PGP 2.6 (22 May 94) +======================================== + +***Note: packets that contain a version byte of 2 will contain a version + byte of 3 when using versions of PGP >= 2.6 after 9/1/94. This appendix describes the file formats used externally by Pretty Good Privacy (PGP), the RSA public key cryptography application. The @@ -7,20 +10,19 @@ intended audience includes software engi other hardware environments or trying to implement other PGP- compatible cryptography products, or anyone else who is curious. -[To be included: a description of ASCII armor. And ASCII armored +[To be included: a description of ASCII armor. An ASCII armored file is just like a binary file described here, but with an extra -layer of encoding added, framing lines, and a checksum at the end.] +layer of encoding added, framing lines, and a 24-bit CRC at the end.] Byte Order ---------- -All integer data used by PGP is externally stored most significant -byte (MSB) first, regardless of the byte order used internally by the -host CPU architecture. This is for cross-compatibility of messages -and keys between hosts. This covers multiprecision RSA integers, bit -count prefix fields, byte count prefix fields, checksums, key IDs, and -timestamps. +All integer data used by PGP is externally stored most significant byte +(MSB) first, regardless of the byte order used internally by the host +CPU architecture. This is for portability of messages and keys between +hosts. This covers multiprecision RSA integers, bit count prefix +fields, byte count prefix fields, checksums, key IDs, and timestamps. The MSB-first byte order for external packet representation was chosen only because many other crypto standards use it. @@ -131,24 +133,21 @@ Offset Length Meaning 0 1 CTB for secret-key-encrypted (signed) packet 1 2 16-bit (or maybe 8-bit) length of packet 3 1 Version byte (=2). May affect rest of fields that follow. - + Version byte (=3) for >= PGP2.6 after 9/1/94 4 1 Length of following material that is implicitly included - in MD calculation. + in MD calculation (=5). 5 1 Signature classification field (see below). Implicitly append this to message for MD calculation. 6 4 32-bit timestamp of when signature was made. Implicitly append this to message for MD calculation. -10 2 Validity period, in number of DAYS (0 means forever) - Implicitly append this to message for MD calculation. - -12 8 64-bit Key ID -20 1 Algorithm byte for public key scheme (RSA=0x01). +10 8 64-bit Key ID +18 1 Algorithm byte for public key scheme (RSA=0x01). --Algorithm byte affects field definitions that follow. -21 1 Algorithm byte for message digest (MD5=0x01). -22 2 First 2 bytes of the Message Digest inside the +19 1 Algorithm byte for message digest (MD5=0x01). +20 2 First 2 bytes of the Message Digest inside the RSA-encrypted integer, to help us figure out if we used the right RSA key to check the signature. -24 ? RSA-encrypted integer, encrypted message digest +22 ? RSA-encrypted integer, encrypted message digest (MPI with bitcount prefix). If the plaintext that was signed is included in the same file as the @@ -156,27 +155,34 @@ signature packet, it begins right after packet that contains the message digest. The plaintext has a "literal" CTB prefix. -The validity period field is generally only used for certifying keys. -It should be set to 0 otherwise, for regular message signatures. It -may be useful for PEM-like capabilities in future versions of PGP. -PGP 2.3 will always just set it to 0, and will ignore it. - -There is a length field that specifies how many bytes of material is -implicitly included in the MD calculation. If this length field is -5, it means the following 1-byte classification field and the 4-byte -timestamp are included in the signature packet. If the length byte -is 7, it means the 2-byte validity period is also included. In PGP -2.3, we are using a length field of 5 for the material to be included -in the MD calculation, so the validity period is unused and -unincluded, and is assumed to be zeroed. This makes the whole -signature certificate shorter. +The original idea had a variable length field following the length +of following material byte, before the Key ID. In particular, the +possibility of a 2-byte validity period was defined, although no +previous version of PGP ever generated those bytes. + +Owing to the way the MD5 is computed for the signature, if that field +is variable length, it is possible to generate two different messages +with the same MD5 hash. One would be a file of length N, with a 7-byte +following section consisting of a signature type byte, 4 bytes of +timestamp, and 2 of validity period, while the other would be a file of +length N+2, whose last two bytes would be the siganture type byte and +the first byte of timestamp, and the last three bytes of timestamp and +the validity period would instead be interpreted as a signature type +byte and a timestmap. + +It should be emphasized that the messages are barely different and +special circumstances must arise for this to be possible, so it is +extremely unlilely that this would be exploitable, but it is a +potential weakness. It has been plugged by allowing only the currently +implemented 5-byte option. Validity periods will be added later with +a different format. The signature classification field describes what kind of signature certificate this is. There are various hex values: 00 - Signature of a message or document, binary image. 01 - Signature of a message or document, canonical text. 10 - Key certification, generic. Only version of key - certification supported by PGP 2.0. + certification supported by PGP 2.5. Material signed is public key pkt and User ID pkt. 11 - Key certification, persona. No attempt made at all to identify the user with a real name. @@ -199,6 +205,7 @@ signature certificate this is. There ar 40 - Timestamping a signature certificate made by someone else. Can be used to apply trusted timestamp, and log it in notary's log. Signature of a signature. + (Planned, not implemented.) When a signature is made to certify a key/UserID pair, it is computed across two packets-- the public key packet, and the separate User ID @@ -237,7 +244,7 @@ If a key has been revoked, PGP will not messages, and if an incoming signature uses it, PGP will display a stern warning that this key has been revoked. -NOTE: Key/userid revocation certificates WILL NOT BE SUPPORTED in +NOTE: Key/userid revocation certificates ARE NOT SUPPORTED in this version of PGP. But if we ever get around to supporting them, here are some ideas on how they should work... @@ -277,7 +284,8 @@ PGP versions 2.3 and later use a new for digest into the MPI in the signature packet, a format which is compatible with RFC1425 (formerly RFC1115). This format is accepted but not written by version 2.2. The older format used by versions 2.2 -and earlier is also accepted by version 2.3. +is acepted by versions up to 2.4, but the RSAREF code in 2.5 is +not capable of parsing it. PGP versions 2.2 and earlier encode the MD into the MPI as follows: @@ -305,9 +313,8 @@ whole string equal to the number of byte All this mainly affects the rsa_private_encrypt() and rsa_public_decrypt() functions in rsaglue.c. -There is no checksum included. We do include a copy of 2 bytes of the -MD in the outer packet to help determine if we used the correct RSA -key. +There is no checksum included. The padding serves to verify that the +correct RSA key was used. Conventional Data Encryption Key (DEK) "packet" @@ -322,7 +329,8 @@ digest into the MPI in the signature pac presently based on any RFCs due to the use of the IDEA encryption system.) This format is accepted but not written by version 2.2. The older format used by versions 2.2 and earlier is also accepted by -version 2.3. +versions up to 2.4, but the RSAREF code in 2.5 is unable to cope +with it. PGP versions 2.2 and earlier encode the MD into the MPI as follows: @@ -370,10 +378,10 @@ The decrypted ciphertext may contain a c literal plaintext packet. After decrypting the conventionally-encrypted data, a special 8-byte -random prefix and 2 "key check" bytes are revealed. The random -prefix and key check prefix are inserted before encryption and -discarded after decryption. This prefix group prefix is only visible -only after decrypting the ciphertext in the packet. +random prefix and 2 "key check" bytes are revealed. The random prefix +and key check prefix are inserted before encryption and discarded after +decryption. This prefix group is visible after decrypting the +ciphertext in the packet. The random prefix serves to start off the cipher feedback chaining process with 64 bits of random material. It may be discarded after @@ -387,6 +395,12 @@ to see if they match the 7th and 8th byt key-check bytes meet this criterion, then the conventional key is assumed to be correct. +One unusual point about the way encryption is done. Using the IDEA +cipher in CFB mode, the first 10 bytes are decrypted normally, +but bytes 10 to 17, the first 8 bytes of the data proper, are +encrypted using bytes 2 to 9 (the last 8 bytes of the key check +prefix) as the IV. This is essentially using CFB-16 for one +part of the encryption, while CFB-64 is used elsewhere. Compressed data packet @@ -394,14 +408,16 @@ Compressed data packet Offset Length Meaning 0 1 CTB for Compressed data packet -1 4 32-bit (or maybe 16-bit) length of packet -5 1 Compression algorithm selector byte (1=ZIP) -6 ? compressed data +1 1 Compression algorithm selector byte (1=ZIP) +2 ? compressed data The compressed data begins right after the algorithm selector byte. The compressed data may decompress into a raw literal plaintext data -packet with its own CTB. - +packet with its own CTB. Currently, compressed data packets +are always the last ones in their enclosing object, and the decompressor +knows when to stop, so the length field is omitted. The low two bits +of the CTB are set to 11. This is the only case in PGP where this +is currently done. Literal data packet, with filename and mode @@ -443,6 +459,8 @@ Offset Length Meaning 1 1 8-bit length of packet 2 ? ASCII comment, size is as in preceding length byte +Comment packets are currently not generated by PGP. + Secret key certificate @@ -452,6 +470,7 @@ Offset Length Meaning 0 1 CTB for secret key certificate 1 2 16-bit (or maybe 8-bit) length of packet 3 1 Version byte (=2). May affect rest of fields that follow. + Version byte (=3) for >= PGP2.6 after 9/1/94 4 4 Timestamp 8 2 Validity period, in number of DAYS (0 means forever) 10 1 Algorithm byte for RSA (=1 for RSA). @@ -490,6 +509,7 @@ Offset Length Meaning 0 1 CTB for public key certificate 1 2 16-bit (or maybe 8-bit) length of packet 3 1 Version byte (=2). May affect rest of fields that follow. + Version byte (=3) for >= PGP2.6 after 9/1/94 4 4 Timestamp of key creation 8 2 Validity period, in number of DAYS (0 means forever) 10 1 Algorithm byte for RSA (=1 for RSA). @@ -503,6 +523,8 @@ User ID is enclosed in a separate packet somewhere after the public key packet on a keyring or in any other context. +The validity period is currently always set to 0. + User ID packet @@ -552,7 +574,7 @@ Offset Length Meaning For trust bytes that apply to the preceding key packet, the following bit definitions apply: - Bits 0-2 - OWNERTRUST bits- Trust bits for this key owner. Values are: + Bits 0-2 - OWNERTRUST bits - Trust bits for this key owner. Values are: 000 - undefined, or uninitialized trust. 001 - unknown, we don't know the owner of this key. 010 - We usually do not trust this key owner to sign other keys. @@ -561,9 +583,11 @@ bit definitions apply: 101 - We usually do trust this key owner to sign other keys. 110 - We always trust this key owner to sign other keys. 111 - This key is also present in the secret keyring. - Bits 3-5 - Reserved. - Bit 6 - VISITED bit- only used internally by the maintenance pass. - Bit 7 - BUCKSTOP bit- Means this key also appears in secret key ring. + Bits 3-4 - Reserved. + Bit 5 - DISABLED bit - Means that this key is disabled, and + should not be used. + Bit 6 - Reserved + Bit 7 - BUCKSTOP bit - Means this key also appears in secret key ring. Signifies the ultimately-trusted "keyring owner". "The buck stops here". This bit computed from looking at secret key ring. If this bit is set, then all the @@ -573,7 +597,7 @@ bit definitions apply: For trust bytes that apply to the preceding user ID packet, the following bit definitions apply: - Bit 0-1 - KEYLEGIT bits- Validity bits for this key. + Bit 0-1 - KEYLEGIT bits - Validity bits for this key. Set if we believe the preceding key is legitimately owned by who it appears to belong to, specified by the preceding user ID. Computed from various signature trust packets that @@ -597,7 +621,7 @@ following bit definitions apply: (SIGTRUST=101) COMPLETES_NEEDED and MARGINALS_NEEDED are configurable constants. - Bit 7 - WARNONLY bit- If the user wants to use a not fully validated + Bit 7 - WARNONLY bit - If the user wants to use a not fully validated key for encryption, he is asked if he really wants to use this key. If the user answers 'yes', the WARNONLY bit gets set, and the next time he uses this key, only a warning will be @@ -606,7 +630,7 @@ following bit definitions apply: For a trust byte that applies to the preceding signature, the following bit definitions apply: - Bits 0-2 - SIGTRUST bits- Trust bits for this signature. Value is + Bits 0-2 - SIGTRUST bits - Trust bits for this signature. Value is copied directly from OWNERTRUST bits of signer: 000 - undefined, or uninitialized trust. 001 - unknown @@ -617,38 +641,60 @@ following bit definitions apply: 110 - We completely trust this signature. 111 - ultimately trusted signature (from the owner of the ring) Bits 3-6 - Reserved. - Bit 7 - CONTIG bit- Means this signature leads up a contiguous trusted + Bit 6 - CHECKED bit - This means that the key checking pass (pgp -kc, + also invoked automatically whenever keys are added to the + keyring) has tested this signature and found it good. If + this bit is not set, the maintenance pass considers this + signature untrustworthy. + Bit 7 - CONTIG bit - Means this signature leads up a contiguous trusted certification path all the way back to the ultimately- - trusted keyring owner, where the buck stops. This bit derived - from other trust packets. + trusted keyring owner, where the buck stops. This bit is derived + from other trust packets. It is currently not used for anything + in PGP. + +The OWNERTRUST bits are set by the user. PGP does not modify them. +PGP computes the BUCKSTOP bit by checking to see if the key is on the +secret key ring. If it is, it was created by this user, and thus +controlled by him. + +All other trust is derived from the BUCKSTOP keys in a special +maintenance pass over the keyring. Any good signature made by a given +key has its SIGTRUST equal to the key's OWNERTRUST. Based on +COMPLETES_NEEDED and MARGINALS_NEEDED, if enough trusted signatures are +on a key/userID pair, the key/userid association is considered +legitimate. + +To be precise, an ultimately trusted key has weight 1, a completely +trusted key has weight 1/COMPLETES_NEEDED (or 0 if COMPLETES_NEEDED +is 0), and a marginally trsuted key has weight 1/MARGINALS_NEEDED. +Other trust values have weight 0. If the total weight of the signatures +on a key/userid pair is 1 or more, the userid is considered legitimate. + +When a key has a legitimate userid, the user is asked to set the +OWNERTRUST for the corresponding key. Ths idea is that the userid +identifies someone the user knows, at least by reputation, so once it +has been established who holds the secret key, that person's +trustworthiness as an introducer can be established and assigned to the +key. -Note that the other kinds of trust bytes are mainly derived from the -OWNERTRUST bits. They are also derived from the BUCKSTOP bit (which -will be set after creating a key, or after setting the owner trust to -ultimate), and from the SIGTRUST bits, which is itself derived from a -combination of OWNERTRUST bits and possibly the user's ratification. - -When testing a key's integrity, we follow a trusted contiguous -certification path back up to the owner of the key ring by following -keyring trust bytes (for signatures) that have the CONTIG bits and -SIGTRUST bits set, until we hit a keyring trust byte (for a key) that -has BUCKSTOP bit set. Then we know we've reached the top of the -trust pyramid, the keyring owner. Prior to this operation, we set -all the CONTIG bits by navigating the pyramid from the top down, by -testing the SIGTRUST bits that are "trustwise contiguous" with the -top of the pyramid, in a special keyring maintenance pass. - -The key legitimacy is ultimately determined by a probablistic -fault-tolerant method, as follows. We also set KEYLEGIT if BUCKSTOP is -set, which means that this is our own key. The OWNERTRUST bits can only -become defined (nonzero) if KEYLEGIT is fully set already. At the -moment KEYLEGIT becomes fully set (and not before), we ask the user to -define the OWNERTRUST bits. - -This probablistic fault-tolerant method of determining public key -legitimacy is one of the principle strengths of PGP's key management -architecture, as compared with PEM, for decentralized social -environments. +Once that is done, the key's signatures then have weight establishing +other key/userid associations. + +There is a limit to the depth to which this can go. Keys on the secret +keyring are at depth 0. Keys signed by those keys are at depth 1. +Keys which are fully certified using only signatures from keys at depth +1 or less are at depth 2. Keys which are fully certified using only +signatures from keys at depth 2 or less are at depth 3, and so on. + +If you know all of your trusted introducers personally, and have signed +their keys, then you will never have a key at a depth of greater than 2. +The maximum depth is limited my MAX_CERT_DPETH. It never gets very large +in a well-connected "web of trust". + +This redundant and decentralized method of determining public key +legitimacy is one of the principal strengths of PGP's key management +architecture, as compared with PEM, when used in social structures +that are not miltiary-style rigid hierarchies. The trust of a key owner (OWNERTRUST) does not just reflect our estimation of their personal integrity, it also reflects how competent @@ -661,7 +707,7 @@ To define the OWNERTRUST bits for a key to certify other public keys? (1=Yes, 2=No, 3=Usually, 4=I don't know) ? _ -If a key is added to the key ring the trust bytes are initialized +When a key is added to the key ring the trust bytes are initialized to zero (undefined). @@ -676,21 +722,21 @@ like so: whom you usually trust to sign keys. Do you trust "Oliver North" to certify the key for "Daniel Ellsberg"? - (1=Yes, 2=No, 3=I don't know) ? _ + (1=Yes, 2=No, 3=Somewhat, 4=I don't know) ? _ Or: This key is signed by "Oliver North", whom you usually do not trust to sign keys. Do you trust "Oliver North" to certify the key for "Daniel Ellsberg"? - (1=Yes, 2=No, 3=I don't know) ? _ + (1=Yes, 2=No, 3=Somewhat, 4=I don't know) ? _ An "I don't know" response to this question would have the same effect as a response of "no". -If we had no information about the trustworthyness of the signer (the +If we had no information about the trustworthiness of the signer (the OWNERTRUST field was uninitialized), we would leave the advisory note -off. +off. Certifying a public key is a serious matter, essentially promising to @@ -704,19 +750,28 @@ during a maintenance pass. Later, if a KEYLEGIT bit set that would not have been otherwise computed as set by the maintenance pass logic, it alerts me and asks me to confirm that I really want it set. + [--end of not implemented section] During routine use of the public keyring, we don't actually check the associated signatures certifying a public key. Rather, we always rely on trust bytes to tell us whether to trust the key in question. -We depend on a separate maintenance pass to actually check the key +We depend on a separate checking pass (pgp -kc) to actually check the key signature certificates against the associated keys, and to set the -trust bytes accordingly. - +trust bytes accordingly. This pass checks signatures, and if a +signature fails to verify, obnoxiously alerts the user and drops it from +the key ring. Then it tuns a maintenance pass to calculate the +ring-wide effects of this. + +A failed signature should be exceedingly rare, and it may not even +result in a KEYLEGIT field being downgraded. Having several signatures +certifying each key should prevent damage from spreading too far from a +failed certificate. But if dominoes do keep falling from this, it may +indicate the discovery of an important elaborate attack. -The maintenance pass operates in a top-of-pyramid-down manner as -follows. +The maintenance pass is run every time the keyring changes, and +operates in a top-of-pyramid-down manner as follows. If at any time during any of these steps the KEYLEGIT field goes from not fully set to fully set, and the OWNERTRUST bits are still undefined, @@ -730,16 +785,8 @@ BUCKSTOP keys, find all userid/key pairs the trust value of these signatures by copying the OWNERTRUST of the signer to the SIGTRUST of the signature. If this makes a key fully validated, start looking for signatures made by this key, and update -the trust value for them. +the trust value for them. Repeat until everything has settled down. -If a signature fails to verify, obnoxiously alert the user, drop it from -the key ring, and then do the maintenance pass to calculate all the -ring-wide cascaded effects from this, if any. A failed signature should -be exceedingly rare, and it may not even result in a KEYLEGIT field -being downgraded. Having several signatures certifying each key should -prevent damage from spreading too far from a failed certificate. But if -dominoes do keep falling from this, it may indicate the discovery of an -important elaborate attack. @@ -754,35 +801,33 @@ Here is an example of an ordered collect -------------------------------------------------------------------- Public key packet Keyring trust packet for preceding key - User ID packet for preceding key - Keyring trust packet for preceding user ID/key association - Comment packet - Signature certificate to bind preceding User ID and key pkt + User ID packet for preceding key + Keyring trust packet for preceding user ID/key association + Signature certificate to bind preceding User ID and key pkt Keyring trust packet for preceding signature certificate - Signature certificate to bind preceding User ID and key pkt + Signature certificate to bind preceding User ID and key pkt Keyring trust packet for preceding signature certificate - Signature certificate to bind preceding User ID and key pkt + Signature certificate to bind preceding User ID and key pkt Keyring trust packet for preceding signature certificate Public key packet Keyring trust packet for preceding key - User ID packet for preceding key - Keyring trust packet for preceding user ID/key association - Signature certificate to bind preceding User ID and key pkt + User ID packet for preceding key + Keyring trust packet for preceding user ID/key association + Signature certificate to bind preceding User ID and key pkt Keyring trust packet for preceding signature certificate - User ID packet for preceding key - Keyring trust packet for preceding user ID/key association - Comment packet - Signature certificate to bind preceding User ID and key pkt + User ID packet for preceding key + Keyring trust packet for preceding user ID/key association + Signature certificate to bind preceding User ID and key pkt Keyring trust packet for preceding signature certificate - Signature certificate to bind preceding User ID and key pkt + Signature certificate to bind preceding User ID and key pkt Keyring trust packet for preceding signature certificate Public key packet Keyring trust packet for preceding key - Compromise certificate for preceding key - User ID packet for preceding key - Keyring trust packet for preceding user ID/key association - Signature certificate to bind preceding User ID and key pkt + Compromise certificate for preceding key + User ID packet for preceding key + Keyring trust packet for preceding user ID/key association + Signature certificate to bind preceding User ID and key pkt Keyring trust packet for preceding signature certificate --------------------------------------------------------------------