RIPE Atlas probes, the world's largest Internet monitoring system

The principle

Atlas probes are tiny devices, formerly manufactured specially to RIPE specifications¹ but now commoditised², with dedicated software. Volunteers install them worldwide, connected by a wide variety of networks. The probes run active measurements, for example sending messages to other machines and awaiting their response. There are currently around 8,300 active probes, making them without doubt the largest Internet measurement network³.

One of the original features of this service is that the probes perform these measurements on request. Anyone (you too) can ask “I want to ping 2001:67c:2218:751::105 from thirty probes located in Brazil” and obtain a result.

This allows us to check that a service is working properly from several locations. Very often, with partial outages, a negligent network administrator will simply say “seems OK to me” without thinking that other networks might not be so fortunate. Testing the operation of a service from a single measurement point (vantage point) is not enough.

Some people lease remote machines with various operators around the world to conduct these tests, but this quickly becomes expensive and provides only a very small number of vantage points. Others use a VPN with various different exit points, but this does not allow testing for all possible problems and only allows for a small number of vantage points compared to what Atlas provides.

Afnic, which has the public service concession for the management of the .fr TLD, uses Atlas. Afnic’s agreement with the French State obliges it, among other things, to send a monthly report to the government on service performance and reliability. These reports are largely based on measurements made by Atlas probes. Afnic also uses Atlas to debug problems detected, especially those flagged as originating with a particular ISP⁴.

Atlas probes allow measurements to be performed using the following protocols:

• ICMP (Internet Control Message Protocol), specifically its Echo Request/Echo Reply service as used by the ping command, to monitor the reachability of a machine⁵,
• traceroute (not a protocol, but a way of using Internet protocols to determine the route followed by IP packets to a target),
• DNS (Domain Name System), which is obviously of particular interest to a domain name registry like Afnic,
• TLS (Transport Layer Security), but in a limited version allowing only the server certificate to be recovered,
• NTP (Network Time Protocol),
• HTTP (Hypertext Transfer Protocol) but in a limited version, allowing only communication with Atlas anchors⁶.

An example of measurement using Atlas

Let’s look at a specific example. Each measurement made using Atlas is stored in the RIPE servers and can subsequently be posted. Measurement 52277291 consisted in asking for a DNS resolution of the name www.impots.gouv.fr, at a time when it was being subjected to a denial of service attack, in the spring of 2023⁷. The results can be viewed in several different ways. Here, we use Atlas’ Internet interface to see the return code (rcode) of the DNS resolvers used by the probes (each one uses the resolver configured for the network to which it is connected). We see that more than one third of the probes are given the return code SERVFAIL (Server Failure), indicating that the attack has been a partial success.

Figure 1: One of the possible visualisations of measurement 52277291, an orange dot being an Atlas probe that was unable to obtain resolution of the name requested.

Triggering measurements

The measurements made by the Atlas probes can be triggered in various ways. You can use the online interface to launch measurements and analyse them⁸, but you can also do so through the API⁹, Atlas’ programming interface, for which there are several ready-made programs, but which also allows you to develop your own. Let’s look at the Internet interface first. We’re going to create an ICMP measurement (“ping”) to 192.134.1.25:

Figure 2: Initial definition of a measurement, here with ICMP on IPv4

Then select the probes. Here, ten probes taken from the AS¹⁰ 3215, that of Orange:

Figure 3: Choice of probes for a measurement. Atlas has several criteria for their selection.

Then create the measurement and await the result¹¹:

Figure 4: Results of a measurement. All the dots are green, meaning there were no issues and the outcome is “good”.

Now let’s look at the use of programs, here with the “official” tool ripe-atlas-tools, a command-line program for launching measurements and viewing their results. Once the tool has been installed¹² and configured¹³, we can launch a DNS measurement for example:

% ripe-atlas measure  dns --probes 3  internetsociety.org.lr

Looking good! Measurement 73436126 was created …

…

Probe #1007557

===============================================================================

  ; <<>> RIPE Atlas Tools <<>> internetsociety.org.lr.

  ;; global options: +cmd

  ;; Got answer:

  ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 47937

  ;; flags: qr ra rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

  ;; QUESTION SECTION:

  ;internetsociety.org.lr.         IN     A   

  ;; ANSWER SECTION:

  internetsociety.org.lr.  1200     IN     A      104.219.248.49

  ;; Query time: 2594.44 msec

  ;; SERVER: 127.0.0.11#53(127.0.0.11)

  ;; WHEN: Sat Jun 15 10:25:23 CEST 2024

  ;; MSG SIZE  rcvd: 56

We see that the result is posted so as to resemble the test and debugging commands of conventional networks. Here, the output is similar to that of dig.

Similarly, we can also request a traceroute:

% ripe-atlas measure  traceroute --probes 3 --traceroute-show-asns \

        2001:67c:217c:4::2

Looking good! Measurement 73436086 was created …

Probe #1001423

Sat Jun 15 10:23:53 CEST 2024

  1 2001:4bd8:52:1::81:67        AS199284    2,776 ms     0,943 ms     0,944 ms

  2 2a01:75c0:1000::6a           AS199284    1,164 ms      1.02 ms     3,718 ms

  3 2001:920:0:7::72               AS8220    6,492 ms     6,479 ms      6.43 ms

  4 2001:920:c000:0:212:74:90:32   AS8220   29.847 ms    31.732 ms    32.567 ms

  5 2001:7f8:43::2486:1                     30.298 ms    30.296 ms    30.237 ms

  6 2001:67c:217c:4::2             AS2486    30.32 ms    30.287 ms    30.313 ms

…

For all protocols, the program provides a wide range of options for testing all possible cases. Don’t hesitate to check out the documentation.

Another program using the API is Blaeu, developed within Afnic. Its main use lies in obtaining aggregated statistics. For example, the “Brazilian” measurement referred to at the beginning was created in this way:

% blaeu-reach --requested 30 --country BR 2001:67c:2218:751::105

27 probes reported

Test #73433215 done at 2024-06-15T07:31:02Z

Tests: 78 successful tests (96.3 %), 0 errors (0.0 %), 3 timeouts (3.7 %), average RTT: 202 ms

A traceroute can also be requested from the same country:

% blaeu-traceroute --format --requested 3 --country BR 2001:67c:217c:4::2

…

Test #73612634 done at 2024-06-17T13:51:32Z

…

From:  2804:b48:1975:be00:da58:d7ff:fe03:539    262732    PROSERVER TELECOMUNICACOES S.A., BR

Source address:  2804:b48:1975:be00:da58:d7ff:fe03:539

Probe ID:  61742

1    2804:b48:1975:be00::1    262732    PROSERVER TELECOMUNICACOES S.A., BR    [1.298, 1.178, 0.505]

2    2804:b48:19ff:ffff:10:254:253:1    262732    PROSERVER TELECOMUNICACOES S.A., BR    [4.984, 6.004, 6.56]

3    2804:b48:19ff:ffff:a::a    262732    PROSERVER TELECOMUNICACOES S.A., BR    [6.782, 5.728, 5.852]

4    2804:30c:4:80::10:1    28343    UNIFIQUE TELECOMUNICACOES SA, BR    [15.708, 15.976, 15.698]

5    fd00::10:85:161:26    NA    NA    [16.374, 16.005, 16.461]

6    2804:4c0:ffff:ff0d::1    3549    LVLT-3549, US    [20.371, 19.822, 20.259]

7    ['*', '*', '*']

8    2001:1900:5:2:2::d32    3356    LEVEL3, US    [216.74, 216.324, 217.144]

9    2001:67c:217c:4::2    2486    NIC-FR-DNS-UNICAST-PARIS2 AFNIC Association Francaise pour le Nommage Internet en Cooperation, FR    [218.231, 217.825, 217.329]

You can also view a more attractive visualisation of the result of this traceroute online:

Figure 5: One of the possible visualisations of a traceroute on the RIPE Atlas website

We see that the route from Atlas probe 61742 passed via the operator Lumen (ex-Level 3), probably in the United States. This detour is explained by the lack of an undersea cable between Brazil and Europe.

You can, of course, write your own program using the API (which is how Blaeu was developed). Consult the documentation¹⁴.

Credits to use Atlas

To launch these measurements, you’re going to need credits. No, this is not a new blockchain-based cryptocurrency. Credits are the “currency” issued by RIPE to “pay” for the use of Atlas. Each measurement performed requires the use of credits. They are obtained by hosting an Atlas probe, an anchor, by being an LIR¹⁵, meaning a member of RIPE, or simply by asking nicely on Atlas’ distribution list. The aim of the credits is simply to avoid excessive use of the network of probes, not to prevent students and inquiring minds from using Atlas. We often see, for example, on Atlas mailing list a two-paragraph request from an IT network student, explaining his or her project and asking for millions of credits, which are always granted very quickly¹⁶.

More complex: location of Anycast DNS servers

Anycast is a technique allowing tens or even hundreds of servers around the world to share a single IP address. This technique is much used by managers of DNS servers to spread loads and counter denial of service attacks. Ideally, the physical sites of the Anycast services are situated so as to minimise latency, the time taken to reach them, the best results being obtained when each resolver always directed to the “nearest” physical site¹⁷. But perfection is difficult to attain in this field, in particular because of the limitations of the BGP routing protocol¹⁸. Atlas probes can help with optimal placement by asking the servers it queries for their NSID (Name Server Identifier)¹⁹.

So in this case a hundred Atlas probes in France are asked to query d.nic.fr and to post its NSID:

% blaeu-resolve --requested 100 --country FR --nsid \

              --nameserver d.nic.fr --type SOA fr

Nameserver d.nic.fr

[NSID: dns.th2.nic.fr; a.nic.fr. dnsmaster.afnic.fr. 2238829988 3600 1800 1209600 600] : 88 occurrences

[NSID: dns.ams.nic.fr; a.nic.fr. dnsmaster.afnic.fr. 2238829988 3600 1800 1209600 600] : 3 occurrences

[NSID: dns.lyn.nic.fr; a.nic.fr. dnsmaster.afnic.fr. 2238829988 3600 1800 1209600 600] : 2 occurrences

[NSID: dns.lon.nic.fr; a.nic.fr. dnsmaster.afnic.fr. 2238829988 3600 1800 1209600 600] : 2 occurrences

[NSID: dns.fra.nic.fr; a.nic.fr. dnsmaster.afnic.fr. 2238829988 3600 1800 1209600 600] : 3 occurrences

[TIMEOUT]: 2 occurrences

Test #73613034 done at 2024-06-17T14:02:59Z

Figure 6: Posting of the identity of each Anycast instance of d.nic.fr

(“th2” is in Paris, “lyn” in Lyon, “fra” in Frankfurt, etc.) But the Internet interface result is easier on the eye.

The colours are not ideal but we can see that the Atlas probes in France were nearly all directed to a DNS server in France (the three directed to Frankfurt are located in the east of France, which makes sense).

Conclusion: the power of the cooperative system

As well as being practical, an essential factor for a network and Internet services manager, RIPE Atlas probes are also a perfect example of the power of the cooperative model: the world’s largest Internet measurement network is not a commercial offering but rather an effort driven by copious volunteers, coordinated by a non-profit organisation managed by its members.

1 – Réseaux IP Européens (RIPE, French for “European IP Networks”), the regional registry for IP addresses for Europe and the Middle East.

2 – TP-Link, then Raspberry Pi. There are now also purely app-based probes.

3 – Among the other analogous networks, we would mention the Ring, which has fewer devices but with greater possibilities.

4 – There is no end to the uses to which Atlas probes can be put. For example, in the appeal of the Quadrature du Net (a French digital rights advocacy group) against the government’s decision to block TikTok in New Caledonia in May 2024, among the documents produced in support shows the results of measurements run by probes located in New Caledonia, a rare instance of the legal use of these probes.

5 – The Internet is complicated, and this test is of course not sufficient to determine reachability; Atlas probes are only tools, and the interpretation of their results still requires Internet skills.

6 – Anchors are special probes with extra capabilities that can also serve as targets for measurements, for example for testing connectivity. Afnic hosts one.

7 – Attack forming part of the series of attacks on the domain names of several French public services, claimed by “Anonymous Sudan”, who is probably about as Sudanese as I am Japanese.

8 – The new Internet interface launched in the spring of 2024 substantially improves its analytical possibilities. If you tried the old one and were discouraged, take a look at the new one.

9 – Application Programming Interface.

10 – Autonomous System. An AS is identified by a number, and corresponds broadly speaking to an Internet operator. In practice, problems are generally linked to AS, not to cities or regions and therefore, when reporting an outage, it is often more useful to indicate the AS than the city.

11 – Times are variable and depend on the load of the server controlling the measurements and on any problems that might arise (as a reminder, this is a free service, paid for by RIPE and therefore its members).

12 – pip install ripe-atlas-tools

13 – ripe-atlas configure –set authorisation.create=MY_API_KEY. The necessary API keys are created via the Internet interface, and you’ll need a RIPE account (obtainable automatically and free of charge) and some credits (see hereunder).

14 – As Ripe-atlas-tools and Blaeu are free software, you can also read their source code for inspiration.

15 – Local Internet Registry

16 – This is one of the many advantages of Atlas: it allows IT network researchers and “mere” students to run worldwide measurements. GEODE group researchers, for example, use it regularly.

17 – Nearest in terms of latency, not necessarily in miles.

18 – Border Gateway Protocol. An example of such adjustment is presented in this article.

19 – A mechanism standardised in RFC 5001.