As events unfold in Ukraine and the world is focused on following what is happening, it’s important to remember that critical Internet infrastructure plays a key role in keeping people connected, providing access to all kinds of crucial information and support and helping people stay in contact with loved ones across borders.
At the Internet Society, we have outlined how policy can help us keep the Internet open and available to everyone, everywhere:
- Why the World Must Resist Calls to Undermine the Internet
- Internet Impact Brief: Impact of Ukraine’s Requests to Block Russia’s Access to the Internet
- Internet Impact Brief: How Refusing Russian Networks Will Impact the Internet
Alongside this, the Pulse team has also been looking at what is happening to critical Internet infrastructure in both Russia and Ukraine from various different angles.
This rolling post will provide a series of updates, resources, analysis and data to help everyone understand what’s happening at a technical level.
Check back often for updates!
- 6 April: M-Lab Speedtest: What’s Going on in Ukraine?
- 1 April: Investigating Internal Hijacks for Content Providers on Russian Networks
- 30 March: Top Categories for New Russian Internet blocks (OONI Anomalies)
- 29 March: A Small Hijack of Twitter’s Space and the Role of RPKI
- 28 March: Where are Russian Networks Being Disconnected?
- 26 March: QUIC vs TLS on Russian Networks
- 25 March: OONI Data: Looking For Anomalies and Blocks
M-Lab Speedtest: What’s Going on in Ukraine?
6 April, 2022
Amreesh Phokeer, Internet Measurement and Data Expert, Internet Society
As we enter the 7th week of conflict between Ukraine and Russia, we take a look at the Measurement-Lab (M-Lab) Speedtest dataset. M-Lab provides crowdsourced speedtest data including download/upload throughput and latency measurements from different locations around the world. M-Lab makes use of the NDT (Network Diagnostics Tool) client, which is accessible via the browser. For example when a user visits https://speed.measurementlab.net, the NDT client on the browser initiates a connection to the geographically closest M-Lab server and runs a performance test (bandwidth and latency). The data is then aggregated by M-Lab and is made available via BigQuery. M-Lab maintains a diverse set of dedicated test servers also known as “targets” around the world.
Since the measurements are run by end-users, the M-Lab dataset can provide some useful insights on the situation on the ground, namely by analyzing the changes in the metrics collected. For example, a decrease in the number of tests collected per day can be indicative of possible network disruptions, as is currently the case in Mariupol, which has been cut off from the Internet. After 1 March, 2022, no M-Lab tests were recorded in Mariupol.
Looking at the overall trends, the number of tests recorded in Ukraine stayed (more or less) the same as compared to before the start of the conflict. However, there is a noticeable decrease in the number of tests recorded between 23 February and 2 March, 2022, probably due to the movement of people.
Most of the tests recorded so far are from the cities of Kyiv (22.4%) and Dnipro (7.3%) and the networks recording the most number of tests are Kyivstar (15%) and Vodafone (5.8%), which are both mobile network operators. Mobile Internet remains an essential tool for millions of displaced Ukrainians to stay in touch with friends and families and to get crucial news and information.
Try Our Dashboard
Using the M-Lab dataset, the Internet Society built a dashboard for both Ukraine and Russia so you can navigate the dataset as from 1 February, 2022 and can filter on “client city”, the “client network” and/or the “target country”. You can take a look at the dashboard here.
Investigating Internal Hijacks for Content Providers on Russian Networks
1 April, 2022
Aftab Siddiqui, Senior Manager, Internet Technology – Asia-Pacific, Internet Society
Massimiliano Stucchi, Regional Technical Advisor – Europe, Internet Society
On 29 March we showed how an accidental hijack happened, originating from a network in Russia. At the end of the post we briefly discussed an additional discovery we had made while investigating the incident, and today we would like to explore a little bit more about it.
It seems that a good percentage of networks in Russia are generating false BGP announcements inside their network to redirect traffic destined to Twitter. Let’s see what this means.
Twitter hosts its services on a small number of IP Addresses. These are all part of the network 104.244.42.0/24. This network is announced, or better “originated”, by Twitter’s Autonomous System Number (AS13414).
This BGP announcement is propagated by all the networks Twitter is peering with – i.e. it is in direct contact with – so it can reach the rest of the Internet in order to let people make use of any service Twitter is offering.
Once one of these networks propagating Twitter’s announcement receives this update, it is installed in their local routing table, a sort of telephone directory, after comparing it with any other similar announcement to identify which one is better. ‘Better’ means that the announcement received might have a shorter AS-Path, it might have a smaller number of networks through which data has to flow before reaching Twitter, or this announcement could be “more specific” than another one already in the routing table and so it gets preferred. Being more specific means that the network being announced is “smaller” than another one. A /25 (128 IP addresses) is smaller, and therefore more specificm, than a /24 (256 IP addresses). A /25 announcement would be preferred over a /24 announcement for the same IP address range.
With these small notions in mind, let’s get back to what we have noticed. By means of tools called a “looking glass”, network engineers debug issues related to BGP Network announcements. Network operators voluntarily set up looking glasses so that engineers from other nework operators are able to see what is happening inside their network. This helps in terms of understanding the effects of certain BGP configurations and how counterparts receive and eventually install announcements.
We have used looking glasses from a number of network operators to investigate how many of them are taking special actions to block Twitter.
Out of 26 of them, we could see that six networks had some form of internal hijack for one of Twitter’s prefixes and in some cases even for specific IP addresses, in this case, represented as a /32.
The origin ASN was in most cases a private ASN. These are Autonomous System Numbers that are to be used internally in a network, and, similar to private IP addresses, should not be seen on the wider internet. In some cases, though, the originating ASN was the one from the network operator themselves. \
We believe that this is to generate a shorter AS-Path and make the artificial announcement preferred, as this seems to be the case for larger network operators who peer at Internet Exchanges (IXPs) where they can have a direct session with Twitter.
Top Categories for New Russian Internet blocks (OONI Anomalies)
30 March, 2022
Jim Cowie, Resident Advisor
As the first month of Russia’s invasion of Ukraine came to a close, ISOC Pulse observed significant increases in the rates of anomalous results in OONI’s Russian Web Connectivity tests, compared to the week prior to invasion.
Nearly all categories of web content saw at least some increase in failures to load, including the category of “Control content” (websites unlikely to attract the attention of any censor), suggesting an increase in the background of network congestion.
However, some categories were impaired much more significantly than the average, including social networking (increase in anomalous tests from 4% to 26%), media sharing (from 8% to 20%), and foreign news media overall (from 8% to 20%). These measurements are consistent with Russia’s imposition of Internet content blocks to restrict the domestic availability of information about the Ukrainian war.
A Small Hijack of Twitter’s Space and the Role of RPKI
29 March, 2022
Massimiliano Stucchi, Regional Technical Advisor – Europe, Internet Society
Yesterday, at around 12 UTC, BGPStream, a service that continuously monitors changes in the routing table across the World, notified its followers of some suspicious activity involving a network that is held by Twitter. We can see the notification here in this tweet:
The speculation around this incident is that the operator in question was trying to set up an internal Border Gateway Protocol (BGP) announcement to redirect all the traffic intended for Twitter. But this announcement – only meant to remain on the internal networks – leaked and was propagated to the rest of the Internet.
This is the most plausible explanation, because Russian ISPs, in the last few weeks, have started restricting access to Twitter after Russia’s invasion of Ukraine led to many Twitter posts criticizing the act. Redirecting Twitter traffic is one way to enact these blocks.
There have been many cases in recent years of similar incidents, the most famous being the one involving Youtube and Pakistan Telecom in 2008, about which RIPE NCC produced a detailed video. Incidents of this type can be explained by human error or mistakes in the configuration of routers and filters, while in some other cases they can be attributed to a malfunctioning device.
This particular incident involves an autonomous system, AS8342 – JSC RTComm.RU, which appears to have started announcing a network from Twitter’s.
A Border Gateway Protocol (BGP) announcement has the goal of telling the rest of the Internet World “if you want to reach these IP addresses, you can send traffic to me, to this autonomous system”. BGP, in this regard, is a very simple protocol and relies on external controls to verify the validity of these announcements.
The effects on the routing table have been somewhat limited this time. Differently from the 2008 Youtube – Pakistan Telecom incident, there was additional “protection” put in place by Twitter, in the form of an RPKI Route Origin Authorization (ROA).
As I mentioned previously, BGP relies on external controls to verify the validity of announcements. Two of these are called Internet Routing Registry (IRR) and Routing Public Key Infrastructure (RPKI). The goal of both systems is to answer the question “Who is authorized to announce a specific network/prefix?”. The Internet Routing Registry does this by means of databases run by either the Regional Internet Registries (RIRs), such as the RIPE NCC or ARIN, or by other organizations, such as RADB, with different levels of trust. RPKI is similar, but it is only run by the RIRs, and adds encryption.
In this case, Twitter had created Route Origin Authorisation (ROA) objects in RPKI. These objects, which are created on one of the RIR repositories, are meant to “tell” other Internet operators that Twitter – AS13414 – is the only entity authorized to announce the network 104.244.42.0/24.
This crucial piece of information has helped other operators who implement Route Origin Validation (ROV) to verify that the announcement coming from RTComm was not legitimate, and it helped in avoiding its spread. Not all network operators implement ROV, but whenever an illegitimate announcement such as this one reaches a network that does, it gets blocked, and its propagation becomes more limited.
There is more that could be said about how Twitter is being blocked by network operators in Russia. One example is MTS (Mobile Telesystems PJSC, AS8359), which seems to also have internal BGP announcements redirecting its users who attempt to reach Twitter.
If you are interested in learning more about this, and in a more technical analysis of this incident, you can read this post on the MANRS Blog by Aftab Siddiqui.
Where are Russian Networks Being Disconnected?
28 March, 2022
Amreesh Phokeer, Internet Measurement and Data Expert, Internet Society
Internet Exchange Points (IXPs) are physical locations where network operators, such as Internet Service Providers (ISPs), Content Delivery Networks (CDNs) or edge networks, connect to exchange traffic between one another. Peering at an IXP provides more direct connection to other peers, which would otherwise be reachable through longer transit connections. A shorter connection through an IXP can help optimize the routing of traffic and can help ISPs reduce their costs. It can also help reduce the load time of websites and apps for end users.
On 11 March, 2022, the London Internet Exchange (LINX) announced that they had ceased providing services to two Russian Autonomous System Numbers (ASNs) namely Megafon (AS31133) and Rostelecom (AS12389). Here we look into how IXPs around the world reacted to the EU sanctions on Russia, using data from Packet Clearing House (PCH).
The PCH dataset shows that there are 200 Russian ASNs that are peering at 14 locations where PCH has a route collector. A route collector gathers information about each network connected to an IXP. The locations are: AMS-IX (Amsterdam), DE-CIX (Frankfurt), LINX (London), HKIX (Hong Kong), France-IX (Paris), Equinix SG (Singapore), Equinix-NY (New York), DE-CIX (Madrid), Equinix PL (Warsaw), InterLAN-IX (Bucharest), Any2 (Los Angeles), Giganet-IX (Kyiv), BIX (Sofia) and LIXP (Vilnius).
For each of these locations, we retrieved the daily route collector snapshots and extracted the number of prefixes advertised by Russian peers. Each prefix represents a network that is reachable via the peer. The change in the number of prefixes advertised potentially provides an indication on the impact of ongoing sanctions.
We noticed some interesting patterns at AMS-IX (Amsterdam), DE-CIX (Frankfurt), GIGANET-IX (Kyiv), LINX (London) and Equinix-NY (New York). The heatmaps below show the number of prefixes for each ASN (Y-Axis) over multiple days (X-Axis). Where the boxes are gray, no prefixes were advertised by this ASN on this specific date.
AMS-IX
For the Amsterdam Internet Exchange (AMS-IX), the number of prefixes seen by Russian ASNs seems to have remained stable except for AS3216 (Vimpelcom) where we saw a sharp decrease – 40% – in the number of prefixes.
DE-CIX
The reduction in the number of routes advertised by Vimpelcom was also noticed at DE-CIX a bit later on 3 March, 2022.
LINX
The most noticeable decrease was seen at LINX and the immediate effect of the sanctions were observed on AS12389 (Rostelecom) and AS31133 (Megafon). At least 50% of the routes by Rostelecom were withdrawn on 10 March and Megafon was simply de-peered (no prefixes found) as from 11 March. This means other networks peering on LINX now have less routes to send traffic to Rostelecom networks. They will still be able to send traffic to Rostelecom but will probably need to do so through longer paths, using common paid transit providers such as Lumen or Teliasonera rather than directly through the IXP. Similarly, Megafon will not be reachable through the LINX peering LAN and traffic to Megafon will be sent through alternative routes, through transit providers. These actions may have an increase on latency between LINX members’ networks and Rostelecom/Megafon networks.
Equinix-NY
While most of the sanctions were EU-based, we observed some movements at Equinix-NY, where AS8359 (MTS PJSC) one of the main Russian ISPs, removed some 350 prefixes between 25 February and 8 March, which shows some instability in the number of routes served by MTS.
Giganet-IX (Kiev)
Finally, a handful of Russian ASNs peer at the Giganet Internet Exchange in Kiev. We observed the following ASNs being disconnected from the IX:
- 28 February: AS57629 (LLC IVI.RU), AS42861 (Foton Telecom CJSC)
- 1 March: AS47626 (Timer, LLC)
- 8 March: AS57363 (CDNvideo LLC)
- 11 March: AS8641 (LLC Nauka-Svyaz), AS35598 (Inetcom LLC)
- 15 March: AS60764 (TK Telecom), AS60388 (Limited Liability Company Transneft Telecom)
As we can see, many Russian networks stopped propagating their routes through the Giganet Exchange in Kiev, therefore breaking the direct connections between Ukrainian networks. It might be that these networks deliberately decided to shut down their BGP peering sessions based on instructions from the Russian authorities.
IXPs play an important role in shortening paths between networks. They help keep local traffic local and reduce latency between networks. Removing peers from an IXP reduces the number of direct routes that can be used to send traffic to other peers and this can have an impact on latency as well as on the actual “cost” of sending traffic.
QUIC vs TLS on Russian Networks
25 March, 2022
Max Stucchi, Regional Technical Advisor – Europe, Internet Society
Contributor: Robin Wilton, Director Internet Trust, Internet Society
Today our mission was to explore the impact on IPv6 traffic of all the changes that have affected networks in Russia and Ukraine in the last month. As part of this, we checked some new data available in Cloudflare Radar, which, since yesterday, provides per-Autonomous System Number (ASN) statistics. ASNs identify separate networks and generally represent a single ISP, hosting provider or corporate network. Our goal was to check if Cloudflare’s data matched what we had already seen for some of the networks we were monitoring. However, we soon noticed a strange pattern that was totally unrelated to IPv6.
If you look at these graphs, you will see that there has been a similar shift in the use of either TLS1.3 or QUIC on different networks in Russia:
AS8402 – VIMPELCOM
AS12389 – ROSTELECOM
AS8359 – MTS
As can be seen, each of these three networks show very similar data and a shift that happened at around the same time.
TLS and QUIC
TLS and QUIC are two protocols that enable the encryption of Internet traffic. TLS1.3 represents the latest version of the TLS standard, while QUIC is a new set of protocols created by Google and adopted by Chrome in the last few years.
On the Pulse Enabling Technologies page, you can see that the average usage of TLS 1.3 and QUIC in the world matches the levels seen from the networks we are analyzing until the sudden change around the end of February – that is, around 64% for TLS 1.3 and 18% for HTTP/3-QUIC.
After some research, we found an issue post confirming what we see from this data. The reason for this sudden change appears to be a block specifically targeting QUIC on the port that it usually operates on (port 443), and the blocking of even the initial packet being exchanged between the client and the server.
Blocking and Filtering
The issue post also mentions Open Observatory of Network Interference (OONI) and details a way to verify if QUIC is being blocked by analyzing OONI tests. This is a different type of measurement than the one we used as the basis for our analysis on the blocking of websites in Russia, which we published yesterday. Unfortunately, there are no test reports for that specific type of test run on Russian networks in the last month, so it is not possible to perform any type of analysis.
At this point, we can speculate that these blocks could be part of the system that is helping the Russian government apply selective filters to content that is deemed inappropriate or, as mentioned in yesterday’s post, produced or served by what is considered as an “extremist organization”. It seems unusual that a block would target one specific protocol, but it might also be done this way in order to facilitate filtering with hardware that is dedicated to work with TLS rather than with QUIC.
It is also interesting to note that it seems that not all of the Russian ASNs or networks are experiencing this blocking. For example, we can see that AS39709 – FotonTelecom has data that is more in line with the global averages for TLS 1.3 and QUIC:
AS39709 – FOTONTELECOM
OONI Data: Looking For Anomalies and Blocks
24 March, 2022
Hanna Kreitem, Technical Expert, Middle East, Internet Society
Max Stucchi, Regional Technical Advisor – Europe, Internet Society
The Open Observatory of Network Interference (OONI) Explorer is the world’s largest open dataset on Internet censorship, consisting of millions of measurements collected from more than 200 countries since 2012. Internet users install the OONI Probe on their device, which then conducts various measurements on their network. We use OONI data in the Pulse Internet Shutdowns focus area and have shown, on multiple occasions on the shutdowns tracker, how this data can provide important insights into a country’s Internet usage.
Blocked Websites or Apps
On 26 February Russia started blocking a set of social networks, including Facebook and Instagram. OONI reported this, and we also covered it on the Pulse shutdowns tracker. We started to look at the same measurements from a different angle, examining the number of anomalies recorded by the OONI Web Connectivity tests over the course of the last two weeks, prior to the beginning of the conflict:
This graph above shows the total number of OONI Web Connectivity tests that were run each day in Russia, and shows the number of anomalies detected. Anomalies flag signs of potential network interference, such as the blocking of a website or app. As you can see, there has been a small but continuous increase in these anomalies, meaning that more and more websites are being blocked or cannot be accessed.
A measurement is “confirmed blocked” only when there is absolute certainty that the tested ‘resource’, for example Instagram, is blocked. This applies to websites when an Internet Service Provider (ISP) serves a ‘block page’, which notifies the user that the website is intentionally blocked. Thus we can say that the increase in the percentage of anomalies of all tests reflects that more people are not able to access the website or service and are running tests to see what is happening.
To confirm this, we can take a look at some more specific data provided by OONI. This time we’re looking at data about anomalies for social networking websites, as categorized in OONI’s test list:
We do see a slight increase in the number of anomalies reported, and a large increase in the number of tests run specifically on the 13th of March. This is the day before the Russian government announced that Instagram was going to be blocked, and we can infer that users used OONI to understand if the website was already blocked or not.
Different Types of Networks
To understand how different networks reacted to orders of shutdowns and restrictions, we looked at the distribution of tests and results between different Autonomous Systems (AS). ASs identify separate networks and generally represent a single ISP, hosting provider or corporate network.
There are around 4,600 ASs in Russia and over four million measurements were collected from 390 networks over the past month through OONI Probes. But only 28 networks had measurements covering the whole month and these are the networks displayed in the animated graph below.
Different networks have different purposes. Some networks provide Internet access to users, while others are corporate networks or datacenter, hosting or content provider networks. By looking at specific, per-ASN data we were expecting to see a sharper increase in the number of tests from residential ISPs, because end users are usually the ones running OONI measurements to check whether the websites they visit frequently have been blocked. And, we would expect corporate networks or datacenter networks not to show many signs of OONI tests being run and even less anomalies being returned. Instead, the data shown above tells us that the distribution is similar for every ASN on which OONI measurements have been run.
Increased Anomalies Expected
A closer look at which network had the largest percentage of anomalies showed that AS8359 (Mobile TeleSystems PJSC), a DSL provider from Moscow, moved from less than 7% on 22 February to 27% on 17 March, and then to 21% on 23 March. Another network, AS50544 (a branch of ER-Telecom located in Krasnoyarsk, Russia), a communications provider, started the testing period with 17.4% and ended with 27% of all web connectivity tests returning anomalous results. Parts of the Rostelecom network (AS42610) moved from 3.4% to a 25% anomaly rate. Most networks included in the analysis started the period with an anomaly rate of under 10%, and ended with most converging around 20%.
It has also recently emerged that a court in Russia has determined that Meta, the newly formed parent company of Facebook, is engaging in “extremist” activities and the blocking of Facebook and Instagram continues. This will likely have an impact on the OONI data and the number of anomalies that we are seeing on networks. This should be visible in a few days. We will provide more information as we are able to collect it on this.
Photo: The Digital Artist on Pixabay