1. Home
  2. >
  3. Blog
Categories
Blog Deflect Press Release

Introducing Deflect-next

After several years of laborious effort, we are proud to announce the public release of the new Deflect software ecosystem. In our Github organization https://github.com/deflect-ca you will find all official Deflect related open source repositories that allow you to stand up an entire website protection infrastructure or its individual components. Deflect offers a high-performance content caching and delivery network, cyber attack mitigation tools powered by Banjax and the Baskerville machine learning clearinghouse, user dashboards, APIs and much else. You are reading this post on a rebuilt and refactored Deflect infrastructure and we’re very proud of that!

Below is the story of the how the new Deflect came to be and rationale for making various software choices.

Deflect-next

Created in 2011 Deflect was a relatively simple solution to the many-against-one problem of distributed denial of service (DDoS) attacks targeting civil society’s web servers. By running Deflect’s caching and mitigation software on constantly rotating edge servers strategically located in some of the world’s biggest data centers, we offered a many-against-many scenario, utilizing the widespread nature of the Internet in a similar manner to those organizing attacks against our clients – leveling the playing field and bringing a little more ‘equalitie’ for our clients’ rights to freedom of expression and association with our audience. Deflect edges memorized previous requests for website data (by virtue of reverse-proxy caching techniques) and removed the load from our clients’ web servers. To block the onslaught of bot-driven attacks, we developed rule based attack mitigation software – Banjax – identifying algorithmic behavior that we considered malicious and banning the IPs that exhibited them. 

As we scaled our infrastructure and hundreds of independent media, human rights groups and other non-profits joined the service, millions of daily requests were received on the Deflect network from around the world. This growth was matched by a higher frequency and sophistication of attacks. The Deflect infrastructure was continuously and laboriously patched, improved and upgraded multiple times. As often the case, the software code underpinning the service became cumbersome and replete with complicated configurations and fixes. Moreover, we moved further and further away from our secondary project ambition – to see other technology groups run their own instances of Deflect using our code base. The software stack required a lot of manual configuration and ‘insider knoweledge’. Beginning in 2019 we began to architect and develop a new version of Deflect, using an entirely new method of provisioning, managing and configuring network components, maintaining our agility and adding reproduce-ability as a primary design philosophy.

From ATS to Nginx

The primary tool for serving Deflect clients’ websites is the caching software installed on every network edge. It does all the heavy lifting in our design – fielding requests from the Internet on behalf of our clients’ websites in a reverse proxy fashion. Initially we had opted for the Apache Traffic Server (ATS) built by Yahoo and released open source in early 2000s. The choice was made primarily for its performance levels under stress tests. The software itself was not yet widespread and a little more difficult to set-up and maintain, with configuration for single actions often spread across multiple files. It required every Deflect network operator to dig deep into documentation and source code to figure out what will happen with every change.

Another caching and proxy solution – Nginx – was a more attractive choice for our new network design, with expressive configuration formats and a much larger technical userbase.

From Ansible and Bash to Python and Docker

Deflect clients customize their caching and attack mitigation settings via the Deflect Dashboard, which sends snapshots of these settings to the network controller. Previously, the configuration engine was a mix of Ansible and Bash, and it had overgrown in logic and depth more than what was destined to be managed in these languages. We have now rebuilt the configuration module in Python, supporting better scaling in the future and API communications.

We rebuilt the orchestration module of Deflect – for installing packages, starting and stopping processes, and sending new configuration to network edges, using Docker.

The benefits of containerization are well-known, but in short the advantages to us were: decoupling applications from the host OS (upgrading between Debian versions has historically been a pain point), making our various development and staging environments more reproducible, and letting us create and destroy multiple copies of a container on the same server easily. Once our applications were containerized, we needed a way to start and stop these containers, deciding to write it imperatively in our go-to general-purpose language, Python. There’s a library, docker-py, which connects to the Docker daemon on remote hosts over SSH and provides an API for building images, creating volumes, starting/stopping containers, and everything else we needed. The result is not as simple as Docker Compose or Swarm, but not as complicated as Kubernetes, and is written in a language everyone on our dev team already knows.

From Banjax to Banjax-go

Our primary method of mitigation – Banjax – was previously an ATS plugin, and as such was tightly coupled to the internal details of ATS’s request processing state machine and event loop. Written as C++ code it was tightly coupled with ATS internals and often limited in functionality by the caching server itself. Answering a simple question like “does a request count against a rate limit even if the result has previously been cached, or only if the request goes through to the origin?” required a close reading of Banjax and ATS source code. To port our attack mitigation logic to another server like Nginx would require a similarly detailed understanding of its internals. In addition, we wanted to share Banjax tooling with others – but could not decouple it from ATS – no one from our partners and peers were running this caching software.

We explored an alternate architecture: where attack mitigation logic lives in a separate process (developed in any language, and decoupled from the internals of any specific server) and talks to the server over some inter-process communication channel. We explored using an Nginx plugin which was specialized for this purpose (and developed a proof-of-concept ATS plugin in Lua which did the same thing) but found that a combination between the very widely used `proxy_pass` directive and `X-Accel-Redirect` header was more flexible (authentication responses can redirect the client to arbitrary locations) and probably more portable across servers. As for the choice of language for the authentication service, Python and the Flask framework would have been nice because we use it elsewhere in our stack, but some benchmarks showed Go and Gin being a lot faster (we were aiming for a worst-case overhead of about 1ms on top of requests with a 50th percentile response time of 50ms, and Go/Gin achieves this).

The interface between Nginx and Banjax-go is a good demonstration of Nginx’s expressive configuration file. This first code block says to match on every incoming request (`/`) and proxy the request to `http://banjax-go/auth-request`.

location / {
    proxy_pass http://banjax-go/auth_request
}

Banjax-go then checks the client IP and site name against its lists of IPs to block or challenge. It responds to Nginx with a header that looks like `X-Accel-Redirect: @access_granted` or `X-Accel-Redirect: @access_denied`. These are names of other location blocks in the Nginx configuration, and Nginx performs an internal redirect to one of them.

location @access_granted {
    proxy_pass https://origin-server
}
location @access_denied {
    return 403 "access denied";
}

This is already a lot easier to understand than a plugin which hooks into the internals of Nginx’s or ATS’s request processing logic (reading and writing a configuration file is easier than reading and writing code). Furthermore it composes nicely with the other concepts that can be expressed with Nginx’s scoped configuration: you can control the logging, caching, error-handling and more in each location block and its clear whether it applies to the request to banjax-go, the request to the origin server, or the static 403 access denied message.

Here’s a diagram that shows the above proxy_pass + X-Accel-Redirect flow (follow the red numbers 1, 2, and 3) along with the other interfaces Banjax-go has: the log tailing and the Kafka connection. The log tailing enforces the same regex + rate limit rules that the ATS plugin did, but asynchronously (outside of the client’s request and response) rather than synchronously. The Kafka channel is for receiving decisions from Baskerville (“challenge this IP”) and for reporting
 whether a challenged IP then passed or failed the challenge.

Baskerville client

The machine lead anomaly prediction clearinghouse – Baskerville – is an innovative infrastructure that has been working in production on Deflect for over a year. It is a complicated set-up reliant on edge servers reporting logs to the clearinghouse, where the pre-processing for feature extraction (looking for anomalous behavior in web logs) creates vectors which are then run through the learning model. An anomaly prediction is generated and communicated back to the network edge. The clearinghouse runs on a Kubernetes cluster and requires a large amount of resources for processing.

Recently, we have split the software base into two components – the clearinghouse and the client software (operating on any Linux+nginx web server). The idea was to allow third-party clients, not using Deflect, to benefit from the clearginhouse’s predictions and the Banjax mitigation tool. In this new model, the Baskerville client is installed independently of Deflect and performs:

  • Processes nginx web server logs and calculates statistical features.
  • Sends features to a clearing house instance of Baskerville.
  • Receives predictions from a clearing house for every IP.
  • Issues challenge commands for every malicious IP in a separate Kafka topic.
  • Monitors attacks in Grafana dashboards.
Anyone can benefit from Baskerville’s anomaly predictions and Banjax’s mitigation tools

Deflect-next open source components

  • Deflect – all necessary components to set up your network controller and edge servers – essentially acting as a reverse proxy to you or your clients’ origin web servers.
  • Deflect-API – an interface to Deflect components
  • Edgemanage – a tool for managing the HTTP availability of a cluster of web servers via DNS. If a machine is found to be under-performing, it is replaced by a new host to ensure maximum network availability.
  • Banjax – basic rate-limiting on incoming requests according to a configurable set of regex patterns.
  • Baskerville – an analytics engine that leverages machine learning to distinguish between normal and abnormal web traffic behavior. Used in concert with Banjax for challenging and banning IPs that breach an operator defined threshold.
  • Baskerville client – edge software for pre-processing behaivoural features from web logs and communicating with the Baskerville clearinghouse for anomaly predictions.
  • Baskerville dashboard – A dashboard for users running the Baskerville Client software offering setup, labeling behavior and communicating feedback to the clearinghouse

Happy coding everyone!

  1. Home
  2. >
  3. Blog
Categories
Blog Deflect

Updates from Deflect – 2 – 2022

Since the beginning of this year, we have served over 1.5 billion website requests to approximately 13.5 million unique readers the world over! We mitigated over 17 distinct and significant attacks and kept our clients online 100% of the time! Our combined bot banning technology (machine lead predictions from Baskerville and confirmed anomalies by Banjax) blocked 5,794,533 malicious hits originating from 1,668,388 zombie bots. That’s quite a lot for this early in the season 🙂

Some of the biggest attacks were directed at a Colombian independent journalist website Los Danieles, a Filipino news media Verafiles, a Latin American information agency and an Indian feminist rights portal.

Attacks during January and February 2022. Colours represent different Deflect clients.

In what is a rather unusual occurence, the deflect.ca website itself was attacked on February 07th. Around 11:00-11:03 GMT+0 about 10,000 unique IPs were sending GET / requests to deflect.ca None of these requests were banned as the attack window was too small. Baskerville worked well in classifying about 5,700 of them as malicious. Some requests returned 502 codes but these were generally malicious requests. eQPress behaved well serving up to 2,620 requests per second on nginx with 5,000RPS to the database and 100 mbps of outgoing traffic. No collateral damage was detected to other eQPress clients. We investigated and improved some of our caching logic as a result of this incident.

  1. Home
  2. >
  3. Blog
Categories
Blog DDoS Deflect

Deflect – a year in summary

Once again, the Deflect network grew in size and audience in 2021. Apart from the continuously stellar work of our clients, what stood out the most for the Deflect team tasked with network monitoring and adversary mitigation – was the increasing sophistication and ‘precision’ of Baskerville, outperforming human rule sets written for the Banjax mitigation toolkit request rate-limiting library. Yes, the machine is outperforming humans on Deflect. We won’t get into the philosophical nature of this reality, rather share some statistics and interesting attacks we witnessed this year, with you.

Year in stats

Legitimate no. of requests served10,152,911,060
Legitimate no. of unique readers (IP)77,011,728
Total requests banned – Banjax3,326,915
Total requests challenged by Baskerville2,606,927
% of Deflect clients also using eQpress hosting34 %
Total amount of complete Deflect outage0
Lowest up-time for any Deflect client99.8%
% of clients increase year-on-year21.62%
Largest botnet, by number of bots 19,333
Number of significant DDoS events103

Deflected attacks

What an attack looks like

On November 04, 2021 – a DDoS attack on a Vietnamese media (also hosted on EQPress) began around 16:50 UTC. Between 2,000-2,500 unique IPs where blocked, as originating from United States, Canada, Germany, France and other countries. These bots issued about 825 thousand GET / and GET https://website.com// requests during this attack. Most IPs involved were detected as proxies, and many of them revealed an IPs in X-Forwarded-For header. The underlying WordPress instance received up to 5,000 requests per second, forcing the EQPress server to send up to 30 megabits per second of html responses. Thanks to the FasctCGI cache and overall configuration hardening, the hosting network cluster had enough resources to serve requests until all bots were blocked without any significant issues for the website itself or its neighbors.

Baskerville detected this attack and issued challenge commands to 2,200+ IPs.

Baskerville’s traffic light classification system

This attack targeted an independent investigative journalism website from the Philippines. The attack began on November 15th and continued throughout the next two weeks. Large portions of attack traffic were not accessible to Deflect, targeting the hosting data center with L3/L4 floods.

Almost 4,000 unique IP addresses issued more than 70 millions “GET /” and “GET /?&148294400498e131004165713TT117859756720Q106417752262N” requests against the website, using `cache busting` techniques with random query_string parameters. Attackers also reverted to using forged User-Agents in request strings. Obviously this attack was adapted against Deflect caching defenses. Many of the participating IPa were proxies possible revealing the original sender with X-Forwarded-For header.

Unfortunately, this attack was not fully mitigated in a quick way and caused several hours of downtime for real users. After manually enabling Deflect’s advanced protection mechanisms and adjusting the origin’s configuration, the website became stable again.

A Zambian democratic watchdog organization was attacked twice between August 08-09 and 11-12. It seems that when the attackers came back a second time round, they hadn’t learned their lessons and tried a similar technique and an almost identical botnet.

Servers from different countries (mostly Unites States, Germany, Russia, France) were sending more than 16 millions of GET / and /s=87675957 requests (with random numbers to bypass caching) during the first round of attacks. During the following incident over 137 million malicious requests were recorded and blocked.

Most of these IPs are known as compromised servers that could be used as proxies and MikroTik routers. 383 unique User-Agent headers were used, all of them were Google Chrome variations. We can also see about 400 TOR exit nodes which were used for this attack.

Millions of hits per minute

The first attack was not completely mitigated due to its profile and some traffic was able to hit the origin server, resulting in several hours of partial downtime for real visitors during different phases of this attack. The second attack was completely mitigated as we had already updated our mitigation profiles.

  1. Home
  2. >
  3. Blog
Categories
Blog Deflect Labs News from Deflect Labs

Baskerville – dynamic model updates

The challenge: Design and implement a system to receive and process feedback from clients, to augment and improve the machine learning model. Produce a model that has the flexibility to adapt to clients’ feedback and the changing patterns in request signatures, whilst also allowing for dynamic model deployment, without breaking existing integration.

In other words: we created the Baskerville botnet mitigation system to be able to react to new and constantly changing attack patterns on the Deflect network. Training the machine on past attacks – we have reached a point where Baskerville can identify more malicious actors than what was captured by our static rules. Now, we need to grow this functionality to accept feedback from our clients on prediction accuracy and to be able to regularly deploy new models without any interruption of service.

Model design

There are several approaches for live model updating. You can use simple files, or a cache and an Rest API call, or a pub-sub mechanism, you can use serialized (pickled) models, models stored in a database and many more mechanisms and formats. But the main concept is the same, either check for a new model every X time unit or have a stand by service which is notified whenever a change occurs and takes care of the model reloading – on demand. We are combining approaches for our case.

The model needs to be re-trained regularly in order to follow the constantly changing patterns of traffic. The general design idea is to decouple the feature generation pipeline from the prediction pipeline. As a result, the feature generation pipeline calculates a super-set of features and the prediction pipeline allows different model versions to use any subset of features. In addition, the model supports backward compatibility and uses the default values in case of an outdated feature generation pipeline.

As soon as a new model is available, the prediction pipeline detects this and starts using the new model without any interruption of service. When features need to be changed, the model is deployed the same way but the User Module will need to be updated and re-deployed as well. Clients will be updating this module from our git repository. It’s very important to mention that during this period required for the User Module to be updated, the new model will be able to communicate with the outdated User Model and deliver the predictions in the usual way. The lack of new or modified features in the model’s input will not break the compatibility since the defaults will be used for missing values.

Based on the assumption that it makes sense that all the requests within a time-window should be processed by the same model, the model change should happen either at the end or at the start of the processing period. For the sake of performance, we decided to put the model updating process at the end of the PredictionPipeline, after the predictions were sent to the client via Kafka, so that we can increase the time it takes for the client to get receive predictions. The following figure explains what happens when a new model is stored in the database after a time-window has been processed (during the idle time waiting for a new batch) and during a time-window processing. In the first case, the next time-window will be processed with the old model and at the end the new one will be loaded. In the second case, since the processing of the current time window has not completed yet, we will load the new model at the end of it and the next time-window will have the fresh model to work with. The asynchronous nature of the training and the predicting is the reason behind the design of the reloading. We ran several test runs to make sure the reloading did not affect the performance of the pipeline.

Feedback Dashboard

In order to receive curated feedback from clients (e.g. the prediction was incorrect) we developed and designed a graphical dashboard consisting of two main components: the back-end REST API built using Python Flask with web-socket support via Flask-SocketIO; and the front-end Angular project, relying on node and npm. The feedback process consists of three steps:

  1. Feedback Context: provide some details about the feedback, like reason, period and an optional notes field. The reason can be one of the following: attack, false positive, false negative, true positive, true negative or other. We provide a short description for each reason option.
  2. Filter out the request sets relevant to the feedback using the search filters. The user can also provide a csv files with IPs to use as a filter.
  3. The last step is for the user to submit the feedback results to Baskerville (Clearinghouse). Because labelling and providing feedback is a painstaking process, we designed the process in a way that the user can omit the last step (submit) if they are not ready yet, and can choose to submit later on. The Clearinghouse will receive the feedback at some point (configurable time window of feedback pipeline) and once the feedback is processed, the pipeline will reply to the user feedback reply topic – which by convention is “{organization_uuid}.feedback”.

We also created a Retrain Pipeline as well as a Retrain dashboard page and functionality to make it easier for us to do periodic model updates. This functionality is only available within the Clearinghouse where the model resides.

This work is a result of months of painstaking development, testing and iteration. If you are interested in trying out Baskerville on your own web platforms, please get in touch. Our work is available under an open source licence and is developed with privacy-by-design principles. We encourage third-party adoptions of our tooling, outside of the Deflect ecosystem and will be publishing another blog post in the very near future outlining the launch of the Deflect Labs Clearinghouse. Watch this space!

  • Baskerville: https://github.com/equalitie/baskerville
  • Baskerville User Client: https://github.com/equalitie/baskerville_client
  • Baskerville Dashboard: https://github.com/equalitie/baskerville_dashboard
  • Baskerville Docker components: https://github.com/equalitie/deflect-analytics-ecosystem
  • Pyspark IForest fork: https://github.com/equalitie/spark-iforest
  1. Home
  2. >
  3. Blog
Categories
Blog Deflect Press Release

Deflect partners with technology and media groups

June 01, 2021 – Deflect partners with technology and media groups

Since 2010, Deflect has specialized in protecting online platforms from cyber attacks. Today, our mission and time-tested tooling reaches further and wider than ever before! We are honoured to announce strategic partnerships with well-known Internet Service Providers and digital media entrepreneurs in the Americas and Europe. Our combined service offering includes all manner of web hosting and online collaboration platforms, technical consultancy and web security services. With over a hundred years of collective technology expertise and a dozen common languages between us, this is a partnership that will serve a global clientele and meet the challenges of shrinking online spaces for expression and self-determination.

Our mission is strengthened through this mutually beneficial partnership. We stand together, stronger and ever more resilient, to protect our clients’ platforms with ethical technology solutions, multilingual human resources and a common belief in principles before profits.

Dmitri Vitaliev, Founder deflect.ca

Find out more about our partners’ individual services and mission from the list below. Check out Deflect’s partnership opportunities and write to us!

@colnodo

Colnodo is a non for profit organization working since 1994 providing Internet infrastructure services to activists and civil society organizations.  Colnodo’s main objective is the access, use and appropriation of information and communication technologies (ICT) for social development, human development and the improvement of people’s living conditions through the strengthening of capacities and competencies, education for work, information and knowledge exchange, increased citizen participation, sustainable development and innovation.

@greenhost

Greenhost (Netherlands) is an established infrastructure provider focusing on digital human rights and sustainability. By providing (infrastructure) services to a wide range of organisations supporting human rights, free press and/or censorship circumvention while preserving privacy guarantees. Greenhost makes sure to keep the internet an open and innovative space.

@greennetisp

GreenNet (UK) have been networking people and activist groups for peace, the environment, equality and human rights since 1986 – providing internet services, web design and hosting. Our hardware and software choices are based on expert technical judgment, our ecological sustainability and ethical business values.

@cloud68hq

(Tirana, Tallinn, Worldwide) Cloud68.co provides reliable open source digital infrastructure to for-purpose small & medium teams, organizations and individuals with responsive and friendly support. As a team of long time contributors to digital privacy and open knowledge projects we are committed to help you migrate from big tech as easy as possible.

@sembramedia

SembraMedia is a nonprofit dedicated to empowering diverse voices in Spanish media to publish news and information with independence, journalistic integrity, and a positive impact on the communities they serve. They conduct research, provide training, consulting, and financial support to help media leaders develop more sustainable business models in Latin America, Spain, and the U.S. Hispanic market.

At MainMicro, our goal is to ensure customer satisfaction by providing ongoing support and cost effective solutions for our partners. We take great pride in having a customer retention rate that is among the highest in the industry. For us, when you become a customer you also become a friend, and we become the one-stop shop for all of your IT related needs.

At Black Crow Labs we construct your brand’s ecosystem and tell your story.  By engaging with prospective customers on targeted platforms we integrate your brand into their lives and conversations.

  1. Home
  2. >
  3. Blog
Categories
Blog Deflect

Updates from Deflect – 2 – 2021

Traffic & Attacks

Since the beginning of this year, we have served over 2 billion website hits to approximately 18 million unique readers the world over! We mitigated over 30 distinct attacks and kept our clients online 100% of the time! The Banjax bot banning technology blocked 291,898 malicious hits originating from 58,181 zombie bots. Our machine lead anomaly prediction system Baskerville was further able to identify and challenge suspiciously behaving IPs 1,182,084 times out of which only 16,755 proved to be legitimate readers and were allowed to access the requested website. This equates to 98.58% precision – which is pretty good for a machine!

Most popular countries reading Deflect protected websites

These attacks have helped us confirm that our prior implementation of the Shapley value estimation in Baskerville had lead to positive results. This is a general way to explain the output from the machine learning model by feature importance ranking – to help us decide which feature works best. We used this algorithm to compare an older machine model with a model that uses only the features Shapley values say are important, on a data set that contained the latest attacks. The model with only the most important features outperformed the older model.

Deflect referral program

Financial survival and independence on today’s Internet is tough. Big Tech permeates and controls virtually every aspect of our digital experience. When it comes to Internet infrastructure and network services, corporate giants such as Akamai, AWS and Cloudflare dominate the space. These handful of companies have managed to create an ecosystem where they profit from virtually every transaction or advertising campaign. While we choose our destiny as consumers, the growing problem is a lack of choices. One way or another, we are being pushed towards a handful of companies.

We want to do things differently. Our goal is to succeed in lockstep with our clients, not simply profit from them. The Deflect referral program creates a mutually beneficial commercial opportunity – by registering for this program and installing a ‘Protected by Deflect’ badge on your website with a unique hyperlink, you will receive 50% of the first full month’s fees charged to every new client that subscribed from this link. Write to partner@deflect.network if you want to participate in this program or read more about this and other collaborative opportunities on the Partner Programs page.

New Website

You are reading this update on our freshly minted website – powered by WordPress and hosted on the secure eQpress platform. We decided to build it using the default 2020 theme. This code is supported by the WordPress team, built according to best practices. That’s important when it comes to running the popular (but often compromised) WordPress platform – the ease of installation for new themes and plugins lowers the barrier for entry and makes it highly functional and customizable. At the same time, custom code developments become outdated, insecure and often lead to website hacking and unintended DDoS attacks. Our set-up configuration comes with the following:

  • Protection from DDoS attacks and password brute-force
  • Daily snapshots and differential backup
  • Long term theme support from WordPress
  • SEO management, chat support, Matomo Analytics, Polylang translations

Over 25% of Deflect clients also host their website on eQpress. The service is detailed on the eQpress page and you can request it from the Dashboard, or contact us with questions. 

  1. Home
  2. >
  3. Blog
Categories
Blog

Kandinsky WordPress theme

A new website creation framework ‘Kandinsky‘ has just been released as a WordPress theme by our friends from «Теплица социальных технологий» (Greenhouse for Social Technologies) in response to needs expressed by civil society groups. The free and open source release on Github offers three templates for installation and guides website creators with helpful tips and check lists. We spoke with Alexey, director of the Greenhouse for Social Technologies about their new release:

eQ: What can you tell us about this new theme?

Alexey: Any organization can use it. It’s free and open source. Having said this, it was created with NGOs and public initiatives in mind. It’s not just a customizable WordPress theme. It also contains pre-made content created to address typical challenges that NGOs face on a regular basis. In particular, NGOs, at least in Russia, usually have difficulties with composing content. Therefore, Kandinsky not only fulfills the role of a theme but also of a checklist of what content should be on an NGO website. There are three templates for different use cases (soon, however, we plan to use only one but greatly improve its ability to be customized). Each template contains test content (news, reports, teams, photos from events, description of activities). Based on NGOs’ best practices, we’ve invented three non-existent organizations in order to fill these templates with content that makes sense in the NGO context.

eQ: What motivated you to create it? Was there a real-life use case that drove this initiative?

Alexey: The need came from our own experience – as an organization we help NGO’s with digital communication and we needed some reliable and modern framework that would be fast and easy to install (now, after 8 minutes, the Installation Wizard has a website up and running). We just wanted to help the organizations have something decent on WordPress and not pay anyone or worry constantly that someone might commercialize the theme. Also, we needed it ourselves for 2 kinds of tasks: side-projects and events. Sometimes we launch a hackathon or a series of online webinars and we wanted a dedicated site – we wanted something that could be set up easily, customizable, and had everything we might need. And so we did it. We adhere to the principle ‘eat your own dog food’ and any event web page or side event we build now is based on Kandinsky which saves us lots of time and effort. We stopped considering website creation a budget line.

eQ: What are these new features?

Alexey: Originally, key values of Kandinsky were simplicity, ease of installation, and responsiveness to NGO’s needs. As we’ve learned, we realized that aside from being able to install the theme in a few seconds and have plug-ins ready as well as pre-made content, customization is of key importance. People are annoyed having the same Ford Model “T” in any color given it’s black, to quote famous inventor Henry Ford. They need to set up a website and then make it unlike any other website in a few minutes.

Therefore, freedom of customization is the most important development line now. In the last year, we’ve added:

  • over 100 Google Fonts to choose from
  • customizable header with the ability to turn on/off different elements inside (logo, phone number, links to social networks, etc.), which creates multitude of different options
  • customizable footer

Our next milestone is turning the main page of a website to a Gutenberg block editor (now the main page can be freely edited with the help of a less versatile tool) and then – to the new WordPress system called Full Site Editing.

Deflect clients can create a new eQpress site with the Kandinsky theme by going to the ‘Hosting tab’ in the Deflect Dashboard and selecting to pre-load the installation with this theme.

  1. Home
  2. >
  3. Blog
Categories
Blog Deflect News from Deflect Labs

Updates from Deflect – January 2021

Giant leaps in our machine-lead mitigation tooling have removed some of the heavy load in mitigating attacks from our support team this month. We’re very pleased with the machine’s performance but it won’t replace the humans! Below, we share some traffic highlights, Deflect relevant events and stories from our clients.

January Traffic

Throughout the month of January, Deflect served over 884 million requests to more than 9 million unique readers around the world. Much of the traffic was bound for Los Danieles – a new and very popular independent media publication in Colombia. Every Sunday, their website attracts between 5 and 9 million legitimate hits! Thankfully, Deflect was able to serve over 94% of these requests directly from its edge cache.

Content served from Deflect cache

Another notable traffic event this month coincided with the release of an online report, investigating torture of thousands of Belarus protesters at the hands of the incumbent government forces. Published by the renowned Committee Against Torture, this is a visual investigation, suitable for mature audiences only.

Notable Attacks in January

Sixteen distinct attacks were recorded against Deflect protected websites this month. Of these, five were notable for their strength and consistency, with two attacks continuing over a four-day period. The largest attack, with over 5000 bots participating, was targeting the Vietnamese independent new site Tiếng Dân. This is not the first time their website has been targeted. Approximately half of the attacking bots were discovered and challenged by Baskerville, whilst the other half were blocked by our manual rule sets. Overall, Deflect maintained 100% network up-time in January.

Kandinsky theme options for eQpress

Deflect clients who run or would like to migrate to our secure WordPress hosting platform can now request the installation of a new theme called Kandinsky. Developed by our friends at «Теплица социальных технологий» (Greenhouse for Social Technology) in response to needs expressed by civil society groups, wishing to have an effective and well designed online presence Kandinsky offers three templates and guides website creators with helpful tips and check lists. You can read our full interview with Kandinsky here.

Deflect and the World Social Forum

On January 29th, Deflect staff participated in a live panel during the World Social Forum together with our partners Colnodo and the Foundation for Freedom of the Press (FLIP). Julian Casasbuenas from Colnodo presented the use case of a Colombian independent media site losdanieles.com as an example of Deflect protection and its importance to the development of free journalistic expression in Colombia. The losdanieles.com project was launched by a group of highly reputable journalists who had been implicated in the Colombian parapolitics scandal.

To finish this newsletter, we wanted to share a lovely thank-you video sent to us by LosDanieles.com columnist Daniel Samper Ospina.

Find the Daniels on Twitter @DanielSamperO, @DanielsamperP and @DCoronell

  1. Home
  2. >
  3. Blog
Categories
Advocacy Blog DDoS Deflect General

Distributed Deflect – project review

This is the fifth year of Deflect operations and an opportune time to draw some conclusions from the past and provide a round of feedback to our many users and peers. We fought and won several hundred battles with various distributed denial of service and social engineering attacks against us and our clients, expanding the Deflect offerings of open source mitigation solutions to also include website hosting and attack analytics. However, several important missteps were taken to arrive here and this post will concentrate on lessons learned and the way forward in our battle to reduce to prevalence of DDOS as an all too common technique to silence online voices.

Our reflections and this post were motivated by an external evaluation report of the Distributed Deflect service, which you can read in this PDF. The project itself was a technical long shot and an ambitious community building exercise. Lessons learned from this endeavor are summarized within. Its about a 10 minute read 🙂

During peak times on Deflect throughout 2012-2016 we were serving an average of 3 million unique daily readers and battling with simultaneous DDoS attacks against several clients. The network served websites continuously for the entire 3 1/4 years of project duration, recording less than 30 minutes of down time in total. The project had direct impact on over four hundred independent media, human rights and democracy building organizations.

What we did

eQualit.ie released 10 open source libraries, toolkits and frameworks including tools for network management and DDoS mitigation; a WordPress managed hosting framework; classification and analysis of malicious network behaviour; the Bundler library for website encryption and delivery across an untrusted network, which was also reused in the Censorship.NO project for circumventing Internet filtering infrastructure.

Over three hundred and fifty websites passed through the Deflect protection service. These websites ranged in size and popularity, receiving anything between a dozen daily readers to over a million. Our open door policy meant that websites who had changed their mind about Deflect protection were free to leave and unhindered in any way from doing so. Over the course of the project, we have mitigated over four hundred DDoS attacks and served approximately 1% of Internet users each calendar year (according to our records correlated against Internet World Statistics). Our work also appeared in topical and mainstream media.

Aside from the DDoS protection service, we trained numerous website administrators in web security principles, worked with several small and medium ISPs to set up their own Deflect infrastructure and enabled Internet presence for key organizations and movements involved in national and international events, including the ’13 election in Iran, ’14 elections in Ukraine, Iguala mass kidnapping, Panama papers, and Black Lives Matter among others.

Distributed Deflect

As attacks grew in size, we debated the long-term existence of the project, deciding to prototype an in-kind DDoS mitigation service, whereby websites receiving free protection and any volunteers could join and expand the mitigation network’s size and scope. We wanted to create a service run by the people it protected. The hypothesis envisioned the world’s first participatory botnet infrastructure, whereby the network would be sustained with around a hundred servers run by the Deflect project and several thousand volunteer nodes. Our past experience showed that the best way to mitigate a botnet attack was with a distributed solution, utilizing the design of the Internet to nullify an attack that any single end point/s could not handle by itself. Distributed Deflect brought together people of various background and competencies, blending software development and technical service provision, customer support and outreach, documentation and communications. We designed, prototyped and brought into production core components of a distributed volunteer infrastructure, only to realize that the hypothesis behind our proposal could not scale if we were to maintain the privacy and security of all participants in our network.

An infrastructure that would accept voluntary (untrusted) network resources had to introduce checks for content accuracy and confidentiality, otherwise a malicious node could not only see who was doing what on the Deflect network but delete or change content as it passed through their machine. Our solution was to encrypt web pages as they left the origin server and deliver them to readers as an encrypted bundle, with an additional authentication snippet being sent by another node for verification. Volunteer nodes would only be caching encrypted information and would not be able to replace it with alternative content.

All necessary infrastructure design and software tools to implement this model were built to specification. However, once ready for production and undergoing testing, we realized the error in hypothesis made at the onset. Encrypted bundles grew in size, as all page fonts and various third-party libraries – that make up the majority of web pages today and are usually stored in the browser’s cache – had to be included in each bundle.

This increased network latency and could not scale during a DDoS attack. We were worsening the performance of our infrastructure instead of improving it. Another important factor driving our deliberation was the low cost of server infrastructure. By renting our machines with commercial providers, and using their competitive pricing to our advantage, we have managed to maintain infrastructure costs below 5% of our overall monthly expenditure. Monetary support for a worldwide infrastructure of Deflect servers was not significant when compared with the resources required to service the network. By concentrating development efforts on encrypting and delivering website content from our distributed cache and performance load balancing on a voluntary node infrastructure, we held back work on improving network management and task automation. This meant that the level of entry to providing technical support for the network was set quite high and excluded the participation of technically minded volunteers protected by Deflect.

After several months of further testing, deliberation and consultation with our funders, we decided to abandon the initiative to include voluntary network resources, in favour of continuing the existing mitigation platform and improving its services for clients. As attack mitigation became routine and Deflect successfully defended its clients from relentless DDoS offensives, the team began to look at the impunity currently enjoyed by those launching the attacks. Beginning with a case of a Vietnamese independent media website targeted by bots originating from a state-regulated and controlled Vietnamese ISP, we understood that a story could be extracted from the forensic trail of an attack, that may contain evidence of motivation, method and provenance. If this story could be told, it would give huge advocacy power to the target and begin to peel away at the anonymity enjoyed by its organizers. The cost for attacking Deflectees would raise as exposure and media attention around the event upended the attackers’ goals.

We began to develop an infrastructure that would capture a statistically relevant segment of an attack. Data analysis was achieved through machine-led technology for profiling and classifying malicious actors on our network, visualization tools for human-led investigation and cooperation with peer organizations for tracing activity in our respective networks. This effort became Deflect Labs and in its first twelve months we published three detailed reports covering a series of incidents targeting websites protected by Deflect, exposing their methodology and profiling their networks. Doing some open source intelligence and in collaboration with website staff, we identified a story in each attack exposing possible motivations and identity of the attackers. Following publication and media attention created by these reports, attacks against one of the websites reduced significantly and ceased altogether for the other one.

Bot behavior follows a certain pattern inside the seven dimensional space create by Bothound analytics

Challenges

Many difficulties and problems could be expected with running a high-impact, 24/7 security service for several million daily readers. Fatigue, lack of time for developing new features, round-the-clock emergency coverage and numerous instances of high-stress situations led to burnout and staff turnover. The resources invested in the Distributed Deflect model set back development considerably for other project ambitions.
At around the same time as Deflect was gaining popularity, free mitigation offerings from Cloudflare and Google were introduced in tandem with outreach campaigns targeting independent media and human rights organizations. This led to more options for civil society organizations seeking website protection but made it harder for us to attract the expected number of websites. We started a campaign to define differences in our distinctive approaches to client eligibility, respect for their privacy and clear terms of service, trying a variety of communications and outreach strategies. We were disappointed nonetheless to not have received more support from within our community of peers, as open source solutions and data ownership did not figure highly as criteria for NGOs and media when selecting mitigation options.

… we carry on

Deflect continues to operate and innovate, gradually growing and solidifying. Our ongoing ambitions include offering our clients broader hosting options and coming up with standards and systems for responsible data sharing among like-minded ISPs and mitigation providers. Look out for pleasant graphic user interfaces in our control panels and documentation platforms. We are also prototyping several different approaches to generating revenue in order to sustain the project for the foreseeable future. The goal is to get better without losing track of what we came here to do in the first place. As always, we are here to support our clients’ mission and their right to free expression. We are heartened by their feedback and testimonials.

  1. Home
  2. >
  3. Blog
Categories
Advocacy Blog DDoS

Deflecting cyber attacks against the Black Lives Matter website

Last week and throughout the weekend, Deflect helped mitigate several DDoS attack bursts against the official Black Lives Matter website. At current estimates over 12,000 bots pounded the website just over 35 million times in 24 hours. An unusual trait of this attack was the prevalence of  malicious connections originating from the US. An in-depth analytic report will follow this prima facie bulletin.

 

[one_third]

hits_BLM
Hits against the BLM site

[/one_third][one_third]

unique_ip_country
All unique visitors (IP) by country

[/one_third][one_third_last]

unique_bots_by_country
Unique bots (IP) by country

[/one_third_last]

The Black Lives Matter website had already been attacked in May using a similar method of a WordPress Pingback reflective attack and similarly an unusually high percentage of bots from the US.

unique_ip_banned_ddosrule
Deflect banning rules triggered by the attacks

Despite its intensity, the attack has been successfully contained by Deflect, and the Black Lives Matter website is functional and accessible throughout much of the weekend. Black Lives Matter has released an official statement on this incident together with eQualit.ie, Design Action Collective and May First/People Link:

Keeping a website available when attackers are seeking to take it off-line is essential for many reasons. The most obvious is the importance of protecting the fundamental right to human communication. But the specific targeting that characterizes recent DDOS attacks (on networks supporting reproductive rights, Palestinian rights and the rights of people of color) highlights this type of on-line attack as part of the arsenal being used to quash response and social change movements.

DDOS attacks will increase as our protests and organizing increases and so must our movements’ ability to resist them and stay on-line. The collaborative work that spawned the response to this attack is both an example of this protective effort and yet another step in improving it and making it stronger.

Our organizations work in different areas with different programs but we are united in our  commitment to vigorously preserving our movements’ right to communicate and defeating all attempts to curtail that right. Without the ability to communicate freely, we can’t organize and, if we can’t organize, our world can never be truly free.

Read the Statement on the Recent Attacks on Black Lives Matter’s Website.

We are in the process of studying and classifying these attack using Deflect Labs technology and aim to publish the results in our next Deflect Labs report.