18 months ago, we published an article (which you can find here) explaining the changes relating to the algorithm of keys used for signing the zones of the top level domains (TLDs) operated by Afnic. In the months that followed, these saw the RSA-SHA256 keys (algorithm number 8, simply referred to as RSA in the rest of this post) replaced by ECDSAP256SHA256 type keys (algorithm number 13, simply called ECDSA below).
All except the .fr TLD.
And yet these migrations went through without incident, and the software and hardware developments to carry out these operations have shown that the adopted solutions were sound and were in line with expectations. But although operationally everything went well, there was one problem – degraded performances of the string providing the signature compared to the one previously used for RSA keys.
These poor performances are relative, as during zone file updates (run every 10 minutes), only modified data is processed and requires signature operations. In this situation, this type of update only takes a few seconds, and it would have acceptable for it to take 10 times longer as it would not have been perceptible. On the other hand, we also have a procedure, though not often used, required if there is a problem when publishing zone files and which will enable a generation from scratch with a possible recreation of all the signatures. For .fr, given its volumes and the validation tests carried out, around thirty minutes were needed for RSA type keys, but it took almost 6 hours for ECDSA type keys, which is unacceptable. Which is why we took the decision to push back the migration of the .fr TLD to give us time to find a reasonable solution.
We are now in a position to say that this solution has been found, hence we are able to change the algorithm for the keys of the .fr zone.
Some technical information
Until recently, Afnic used the BIND 9.11 branch for encoding operations on the zone files. This branch provided 2 techniques for interacting with the HSMs (interior secure boxes containing the private elements of the keys and which carry out all the encoding operations).
The first, historic, used a modified version of OpenSSL and provided very good performances. Unfortunately, it was no longer supported and recent algorithms were no longer taken into account (which is the case for ECDSA, for example). This was the method we continued to use for the .fr zone, and which was dropped in the most recent versions of BIND.
The second, “native”, method, enables BIND to interact directly with the HSMs’ PKCS#11 drivers, without going through an intermediary (such as OpenSSL). Although this proposes the use of the latest algorithms, unfortunately it does not correctly manage multi-threading with the type of HSM we use (AEP KeyperPlus models). This results in particularly poor performances. This is the method we used to use until a few months ago for zones signed with ECDSA type keys. Because of the size of these zones and the limited data to sign, these poor performances did not hinder the implementation of this technology.
More recently, the ISC introduced a new method for the interaction between HSM and BIND 9 (for those who wish to use it, details can be found on this blog BIND 9 PKCS11 · Wiki · ISC Open Source Projects / BIND · GitLab). The idea is to propose a plug and play method which, unlike the 2 methods described above, does not need BIND to be recompiled to communicate with the HSMs. This method uses a standard OpenSSL, but it interfaces with the HSMs using a PKCS#11 driver developed as part of the OpenSC project. It could become the only method proposed for future versions of BIND. Although it is designed to be more universal, it suffers from the same performance problems as the native method.
We therefore put this problem to the ISC (which produces BIND) and our HSM supplier so that they could investigate this case and advise us whether there was a possibility of achieving performances that met our expectations. After a fair amount of dialogue, which enabled us to better qualify the issue and to identify points for improvement, the ISC was able to commit resources to work on this part of the code and to propose several developments of their product to the OpenSC/libp11 project to provide better multi-threading management to optimise interactions with the HSMs. Processing speeds was incredibly faster, with a factor of 3 to 5 compared to the version prior to modification. At the same time, new drivers were available for our HSMs. As for ourselves, we also improved our processes by working on the pre-processing mechanisms, optimising the parallelisation of certain stages, etc. The work on these various issues resulted in a return to expected levels for performances.
Over the last few months, we have therefore been upgrading all our systems (including those of the .fr domain). One of the objectives was to use a much more recent version of BIND (in the 9.16 branch which became the new ESV version in July) and this method via OpenSC/libp11 to sign our zones.
Over several years, we have noticed a stagnation in the number of domains signed in the .fr zone. However, since the start of the year, there has been a marked increase, as shown in the figure below.
Since the start of 2021, 135,000 additional domains were signed, bringing those integrating DS type registrations to almost 15% of the total number of .fr domains (approximately 560,000).
This growth is not without consequence on our ranking amongst the TLDs for country codes (ccTLD) that implement DNSSEC. The .fr TLD has moved to 5th place in terms of the number of DS type registrations. We should note that of these 5 TLDs, 2 already use ECDSA type keys.
|TLD||# of DS in the zone||Algorithm, size of KSK/ZSK keys|
|fr||539786||RSA-SHA256 2048/2048 → ECDSA 512/512|
Put simply, in real terms, for each DS type registration in the zone file, 1 NSEC3 type registration and 2 RRSIG type registrations are created. Out of 12 million registrations contained in the .fr zone file (see table below), over 2 million concern DNSSEC, of which over 1 million are for signatures alone.
|#||Type of registration|
This sudden upsurge in data to be signed and the impact on the size of the zone file are also factors that reinforced our decision to migrate to ECDSA without delay now that performances were once again acceptable. Indeed, during the rollover phase, the zone must be signed twice, by an RSA key and by an ECDSA key, thus doubling the number of signatures.
Regarding the TLDs present in the root, the situation has also changed slightly since our 2020 announcement, given that 41 domains are signed with the ECDSA algorithm and 2 are currently being replaced (almost 50% of these domains are operated by Afnic).
As part of DNSSEC, at Afnic we use OpenDNSSEC to manage the lifecycles of keys. Although there are now alternatives for completing this task, for a long time OpenDNSSEC has been the only user-friendly solution for this type of processing. Thanks to the experience built up in this software solution over time, we have no concerns regarding the running of these operations. Changing the algorithm is not the easiest operation and even the official documents (RFC 6781, for example) are open to interpretation for this type of process. In the case we are interested in, 2 methods are proposed, one “liberal”, the other “conservative”.
The difference between the approaches lies in the start and end of the process. The “conservative” approach stipulates that the new signatures are published before the new keys and that the old signatures are deleted after the old keys have been deleted. This was at one time the only approach supported by some old implementations of validating resolvers. However, clarifications provided by RFC 6840 indicate that these resolvers should ignore the signatures produced by keys not be present in the zone.
OpenDNSSEC uses the “liberal” approach (to my knowledge, only the fantastic Knot-DNS tool proposes the “conservative” method). Below are the details, stage by stage, illustrated using the DNSViz tool.
Before the introduction of new keys
- Date: 13/07/2021
- Zone series number: 2227585342
- The KSK with the tag 35095 is used to sign the keys used in the zone
- The ZSK with the tag 23721 is used to sign the zone.
- The 2 keys are RSA type and have a size of 2,048 bits
- Size of the zone file: 1.1 Gb
- Number of signatures: 1116938
- Size of a signed DNS response for retrieving all the keys: 901 bytes
- Size of a signed NXDOMAIN type DNS response: 1,561 bytes
Introduction of 2 new ECDSA type keys and associated signatures
- Date 14/07/2021
- Introduction of the KSK with the tag 51508 which will sign the keys used in the zone in the same way as key 35095
- Introduction of the ZSK with the tag 30633 which will sign the zone in the same way as key 23721
- The 2 keys are ECDSA type and have a size of 512 bits
- Size of the zone file: 1.3 Gb
- Number of signatures: 2233810
- Size of a signed DNS response for retrieving all the keys: 1,159 bytes
- Size of a signed NXDOMAIN type DNS response: 1,949 bytes
- To launch the next stage, you must wait for these new keys and signatures to be retrieved by the validating resolvers that need them
Modifications at root level
- Date: 19/07/2021
- Removal of the DS corresponding to the KSK with the tag 35095 (RSA)
- Appearance of the DS corresponding to the KSK with the tag 51508 (ECDSA)
- The series number of the root zone corresponding to this change is 2021071901.
During the transition
(some root servers are not yet updated)
Once the transition at root server level has been completed
Removal of the 2 old RSA keys and associated signatures
- Date: 22/07/2021
- Zone series number: 2227666100
- Size of the zone file: < 800 Mb (-25% compared to the data of 13/07/2021)
- Number of signatures: 1124675
- Size of a signed DNS response for retrieving all the keys: 317 bytes (-65% compared to the data of 13/07/2021)
- Size of a signed NXDOMAIN type DNS response: 789 bytes (-50% compared to the data of 13/07/2021)