The Internet Society Pulse platform provides curated insights into the overall health, availability, and evolution of the Internet. This quarter, we’re excited to announce that we have three new Data Partners – Cloudflare, OONI and W3Techs.
Cloudflare: Bolstering Internet Shutdowns Data
To help everyone gain deeper, data-driven insight into the Internet, we need to present a broad range of data from multiple trusted sources on the Internet Society Pulse platform. As there’s no single organization that can measure all facets of the Internet, collaboration between the Internet measurement community and other organizations collecting data is vital for getting deeper insight into how the Internet is functioning and to help people understand their Internet. Internet Society Pulse curates information on Internet shutdowns around the world, and looks at the economic and human impact of these actions.
Accurate and Timely Reports
“At Cloudflare, we’ve spent a lot of time talking to advocacy groups who report, document and push back on governments that shut down the Internet,” says Alissa Starzak, Head of Public Policy at Cloudflare. “These shutdowns intend to stifle dissent, particularly around elections, and stop the flow of information. Many of the organizations that encourage fair, democratic elections around the world document these shutdowns and often have the expertise and the context to understand what’s happening behind the scenes when they occur. In order to confirm these events, organizations need accurate and timely reports of Internet disruptions to act immediately to intervene.”
Network Overview
Cloudflare runs one of the world’s largest networks, with data centers in more than 100 countries worldwide. This global network gives exceptional visibility into Internet traffic patterns. Cloudflare’s Radar platform offers an alerting and monitoring service that presents insights, threats and trends based on the aggregated data that it collects, including data on Internet traffic patterns, domain names, DDoS attacks, IP addresses and technology trends.
“To help provide insights to these Internet trends, Cloudflare’s Radar platform helps anyone see how the Internet is being used around the globe including when there are significant drops in traffic typically associated with an Internet shutdown,” continues Starzak. “For organizations that track and report on Internet shutdowns, these insights can be made available as instant alerts. Over the past few years, we’ve seen an increasing use of Internet shutdowns and cyberattacks that restrict the availability of information in communities around the world. By providing a system that alerts organizations when Cloudflare has detected significant drops in traffic, we hope these tools will be useful for those that document and track shutdowns to ultimately hold institutions accountable.”
Radar’s instant alerting system helps the Pulse team to pinpoint and investigate possible shutdowns as soon as they are detected, enabling us to quickly update our shutdowns tracker once we have established that an outage is a deliberate disruption.
OONI: Building a Bigger Picture Through Collaboration
The Open Observatory of Network Interference (OONI) provides measurement data relating to Internet censorship. We also use this data on the Pulse Internet Shutdowns page alongside multiple other sources to ensure we are presenting an accurate, global picture of Internet disruptions.
Local Vantage Point
OONI data offers rich network measurement data on the blocking of websites, instant messaging apps (WhatsApp, Telegram, Facebook Messenger, Signal), and circumvention tools (Tor, Psiphon, RiseupVPN). It also provides data on network speed and video-streaming performance. This data is collected by volunteers who run software – the OONI Probe – on their local networks and contribute test results which are openly published in near real-time. As the probe tests are run on local networks, OONI is able to capture a picture of what Internet censorship might look like from the user’s local vantage point
Open Data
“Internet measurement provides insight into what is happening on a network and this can be useful for gathering data that can potentially serve as evidence of Internet censorship,” explains Maria Xynou, Research and Partnerships Director at OONI. “However, confirming that something is really Internet censorship can be tricky, particularly since there are several reasons why an Internet service may appear to be blocked by an Internet Service Provider (ISP) but actually isn’t – transient network failures, DNS misconfiguration, server-side blocking, for example. False positives are common in the field of network measurement. So, it’s necessary to not only be able to examine the raw network measurement data in question, but to also be able to cross-reference multiple different relevant datasets in order to examine whether they all present the same signals of censorship. Therefore, it’s important for Internet measurement projects to provide open data and to collaborate with each other.”
Largest Open Dataset
OONI Probe users have contributed more than 466 million measurements from nearly 23 thousand networks in 240 countries and territories since 2012, and new measurements from around the world are openly published every minute. OONI data is likely the largest open dataset on Internet censorship to date. It is also the only open dataset of this scale where measurements are contributed by volunteers around the world.
“Without collaboration in the Internet measurement field, it’s like having only one piece of a huge puzzle,” adds Xynou. “To investigate and understand Internet censorship events, multiple datasets and perspectives are necessary. This is why we’re excited to collaborate with the Internet Society as a Pulse data partner.”
If you’d like to contribute to OONI’s measurement data, please install and run the OONI Probe.
W3Techs: Measuring Internet Centralization
The Internet owes its strength, resilience, and success in part to its open architecture and its principle of decentralized management. However, as the Internet evolves, we are increasingly seeing the accumulation of core Internet service provision among a relatively small number of actors. Concentrating these core functions might increase the fragility of the Internet and can produce centralized points of failure. W3Techs conducts vendor-independent market research on 26 categories of web technologies, building detailed overviews and providing specific insights in various markets. We use this data in our analysis of Internet Centralization, a new focus area which is due to launch on the Internet Society Pulse platform in Q4.
Independent Market Research
“Tracking and monitoring Internet centralization and the consolidation of key services and infrastructure is really important,” says Matthias Gelbman, Managing Director at W3Techs. “The Internet tends to create quasi-monopolies for a variety of reasons and often tends towards a ‘winner-takes-it-all’ market. Monopolies can be harmful for customers and for the underlying Internet infrastructure. W3Tech’s market research helps to give visibility on the current state of the markets, where we are today and where we are heading in the future.”
W3Techs has monitored web technology usage and collected unbiased market data since 2009, which helps paint an accurate overall picture of which technologies are in use where. For example, this data can help us see which web servers are available over IPv6, which hosting providers favor which operating systems, and which traffic monitoring systems are in use in different countries.
“W3Techs is proud to be a Pulse data partner and to collaborate in this initiative,” adds Gelbman. “Different teams bring different views and use different methods to contribute to a more complete picture, helping everyone to get a better overall picture of the state of the Internet today and in the future.”