1. Home
  2. >
  3. Press Release
Categories
Blog Deflect Press Release

Introducing Deflect-next

After several years of laborious effort, we are proud to announce the public release of the new Deflect software ecosystem. In our Github organization https://github.com/deflect-ca you will find all official Deflect related open source repositories that allow you to stand up an entire website protection infrastructure or its individual components. Deflect offers a high-performance content caching and delivery network, cyber attack mitigation tools powered by Banjax and the Baskerville machine learning clearinghouse, user dashboards, APIs and much else. You are reading this post on a rebuilt and refactored Deflect infrastructure and we’re very proud of that!

Below is the story of the how the new Deflect came to be and rationale for making various software choices.

Deflect-next

Created in 2011 Deflect was a relatively simple solution to the many-against-one problem of distributed denial of service (DDoS) attacks targeting civil society’s web servers. By running Deflect’s caching and mitigation software on constantly rotating edge servers strategically located in some of the world’s biggest data centers, we offered a many-against-many scenario, utilizing the widespread nature of the Internet in a similar manner to those organizing attacks against our clients – leveling the playing field and bringing a little more ‘equalitie’ for our clients’ rights to freedom of expression and association with our audience. Deflect edges memorized previous requests for website data (by virtue of reverse-proxy caching techniques) and removed the load from our clients’ web servers. To block the onslaught of bot-driven attacks, we developed rule based attack mitigation software – Banjax – identifying algorithmic behavior that we considered malicious and banning the IPs that exhibited them. 

As we scaled our infrastructure and hundreds of independent media, human rights groups and other non-profits joined the service, millions of daily requests were received on the Deflect network from around the world. This growth was matched by a higher frequency and sophistication of attacks. The Deflect infrastructure was continuously and laboriously patched, improved and upgraded multiple times. As often the case, the software code underpinning the service became cumbersome and replete with complicated configurations and fixes. Moreover, we moved further and further away from our secondary project ambition – to see other technology groups run their own instances of Deflect using our code base. The software stack required a lot of manual configuration and ‘insider knoweledge’. Beginning in 2019 we began to architect and develop a new version of Deflect, using an entirely new method of provisioning, managing and configuring network components, maintaining our agility and adding reproduce-ability as a primary design philosophy.

From ATS to Nginx

The primary tool for serving Deflect clients’ websites is the caching software installed on every network edge. It does all the heavy lifting in our design – fielding requests from the Internet on behalf of our clients’ websites in a reverse proxy fashion. Initially we had opted for the Apache Traffic Server (ATS) built by Yahoo and released open source in early 2000s. The choice was made primarily for its performance levels under stress tests. The software itself was not yet widespread and a little more difficult to set-up and maintain, with configuration for single actions often spread across multiple files. It required every Deflect network operator to dig deep into documentation and source code to figure out what will happen with every change.

Another caching and proxy solution – Nginx – was a more attractive choice for our new network design, with expressive configuration formats and a much larger technical userbase.

From Ansible and Bash to Python and Docker

Deflect clients customize their caching and attack mitigation settings via the Deflect Dashboard, which sends snapshots of these settings to the network controller. Previously, the configuration engine was a mix of Ansible and Bash, and it had overgrown in logic and depth more than what was destined to be managed in these languages. We have now rebuilt the configuration module in Python, supporting better scaling in the future and API communications.

We rebuilt the orchestration module of Deflect – for installing packages, starting and stopping processes, and sending new configuration to network edges, using Docker.

The benefits of containerization are well-known, but in short the advantages to us were: decoupling applications from the host OS (upgrading between Debian versions has historically been a pain point), making our various development and staging environments more reproducible, and letting us create and destroy multiple copies of a container on the same server easily. Once our applications were containerized, we needed a way to start and stop these containers, deciding to write it imperatively in our go-to general-purpose language, Python. There’s a library, docker-py, which connects to the Docker daemon on remote hosts over SSH and provides an API for building images, creating volumes, starting/stopping containers, and everything else we needed. The result is not as simple as Docker Compose or Swarm, but not as complicated as Kubernetes, and is written in a language everyone on our dev team already knows.

From Banjax to Banjax-go

Our primary method of mitigation – Banjax – was previously an ATS plugin, and as such was tightly coupled to the internal details of ATS’s request processing state machine and event loop. Written as C++ code it was tightly coupled with ATS internals and often limited in functionality by the caching server itself. Answering a simple question like “does a request count against a rate limit even if the result has previously been cached, or only if the request goes through to the origin?” required a close reading of Banjax and ATS source code. To port our attack mitigation logic to another server like Nginx would require a similarly detailed understanding of its internals. In addition, we wanted to share Banjax tooling with others – but could not decouple it from ATS – no one from our partners and peers were running this caching software.

We explored an alternate architecture: where attack mitigation logic lives in a separate process (developed in any language, and decoupled from the internals of any specific server) and talks to the server over some inter-process communication channel. We explored using an Nginx plugin which was specialized for this purpose (and developed a proof-of-concept ATS plugin in Lua which did the same thing) but found that a combination between the very widely used `proxy_pass` directive and `X-Accel-Redirect` header was more flexible (authentication responses can redirect the client to arbitrary locations) and probably more portable across servers. As for the choice of language for the authentication service, Python and the Flask framework would have been nice because we use it elsewhere in our stack, but some benchmarks showed Go and Gin being a lot faster (we were aiming for a worst-case overhead of about 1ms on top of requests with a 50th percentile response time of 50ms, and Go/Gin achieves this).

The interface between Nginx and Banjax-go is a good demonstration of Nginx’s expressive configuration file. This first code block says to match on every incoming request (`/`) and proxy the request to `http://banjax-go/auth-request`.

location / {
    proxy_pass http://banjax-go/auth_request
}

Banjax-go then checks the client IP and site name against its lists of IPs to block or challenge. It responds to Nginx with a header that looks like `X-Accel-Redirect: @access_granted` or `X-Accel-Redirect: @access_denied`. These are names of other location blocks in the Nginx configuration, and Nginx performs an internal redirect to one of them.

location @access_granted {
    proxy_pass https://origin-server
}
location @access_denied {
    return 403 "access denied";
}

This is already a lot easier to understand than a plugin which hooks into the internals of Nginx’s or ATS’s request processing logic (reading and writing a configuration file is easier than reading and writing code). Furthermore it composes nicely with the other concepts that can be expressed with Nginx’s scoped configuration: you can control the logging, caching, error-handling and more in each location block and its clear whether it applies to the request to banjax-go, the request to the origin server, or the static 403 access denied message.

Here’s a diagram that shows the above proxy_pass + X-Accel-Redirect flow (follow the red numbers 1, 2, and 3) along with the other interfaces Banjax-go has: the log tailing and the Kafka connection. The log tailing enforces the same regex + rate limit rules that the ATS plugin did, but asynchronously (outside of the client’s request and response) rather than synchronously. The Kafka channel is for receiving decisions from Baskerville (“challenge this IP”) and for reporting
 whether a challenged IP then passed or failed the challenge.

Baskerville client

The machine lead anomaly prediction clearinghouse – Baskerville – is an innovative infrastructure that has been working in production on Deflect for over a year. It is a complicated set-up reliant on edge servers reporting logs to the clearinghouse, where the pre-processing for feature extraction (looking for anomalous behavior in web logs) creates vectors which are then run through the learning model. An anomaly prediction is generated and communicated back to the network edge. The clearinghouse runs on a Kubernetes cluster and requires a large amount of resources for processing.

Recently, we have split the software base into two components – the clearinghouse and the client software (operating on any Linux+nginx web server). The idea was to allow third-party clients, not using Deflect, to benefit from the clearginhouse’s predictions and the Banjax mitigation tool. In this new model, the Baskerville client is installed independently of Deflect and performs:

  • Processes nginx web server logs and calculates statistical features.
  • Sends features to a clearing house instance of Baskerville.
  • Receives predictions from a clearing house for every IP.
  • Issues challenge commands for every malicious IP in a separate Kafka topic.
  • Monitors attacks in Grafana dashboards.
Anyone can benefit from Baskerville’s anomaly predictions and Banjax’s mitigation tools

Deflect-next open source components

  • Deflect – all necessary components to set up your network controller and edge servers – essentially acting as a reverse proxy to you or your clients’ origin web servers.
  • Deflect-API – an interface to Deflect components
  • Edgemanage – a tool for managing the HTTP availability of a cluster of web servers via DNS. If a machine is found to be under-performing, it is replaced by a new host to ensure maximum network availability.
  • Banjax – basic rate-limiting on incoming requests according to a configurable set of regex patterns.
  • Baskerville – an analytics engine that leverages machine learning to distinguish between normal and abnormal web traffic behavior. Used in concert with Banjax for challenging and banning IPs that breach an operator defined threshold.
  • Baskerville client – edge software for pre-processing behaivoural features from web logs and communicating with the Baskerville clearinghouse for anomaly predictions.
  • Baskerville dashboard – A dashboard for users running the Baskerville Client software offering setup, labeling behavior and communicating feedback to the clearinghouse

Happy coding everyone!

  1. Home
  2. >
  3. Press Release
Categories
Blog Deflect Press Release

Deflect partners with technology and media groups

June 01, 2021 – Deflect partners with technology and media groups

Since 2010, Deflect has specialized in protecting online platforms from cyber attacks. Today, our mission and time-tested tooling reaches further and wider than ever before! We are honoured to announce strategic partnerships with well-known Internet Service Providers and digital media entrepreneurs in the Americas and Europe. Our combined service offering includes all manner of web hosting and online collaboration platforms, technical consultancy and web security services. With over a hundred years of collective technology expertise and a dozen common languages between us, this is a partnership that will serve a global clientele and meet the challenges of shrinking online spaces for expression and self-determination.

Our mission is strengthened through this mutually beneficial partnership. We stand together, stronger and ever more resilient, to protect our clients’ platforms with ethical technology solutions, multilingual human resources and a common belief in principles before profits.

Dmitri Vitaliev, Founder deflect.ca

Find out more about our partners’ individual services and mission from the list below. Check out Deflect’s partnership opportunities and write to us!

@colnodo

Colnodo is a non for profit organization working since 1994 providing Internet infrastructure services to activists and civil society organizations.  Colnodo’s main objective is the access, use and appropriation of information and communication technologies (ICT) for social development, human development and the improvement of people’s living conditions through the strengthening of capacities and competencies, education for work, information and knowledge exchange, increased citizen participation, sustainable development and innovation.

@greenhost

Greenhost (Netherlands) is an established infrastructure provider focusing on digital human rights and sustainability. By providing (infrastructure) services to a wide range of organisations supporting human rights, free press and/or censorship circumvention while preserving privacy guarantees. Greenhost makes sure to keep the internet an open and innovative space.

@greennetisp

GreenNet (UK) have been networking people and activist groups for peace, the environment, equality and human rights since 1986 – providing internet services, web design and hosting. Our hardware and software choices are based on expert technical judgment, our ecological sustainability and ethical business values.

@cloud68hq

(Tirana, Tallinn, Worldwide) Cloud68.co provides reliable open source digital infrastructure to for-purpose small & medium teams, organizations and individuals with responsive and friendly support. As a team of long time contributors to digital privacy and open knowledge projects we are committed to help you migrate from big tech as easy as possible.

@sembramedia

SembraMedia is a nonprofit dedicated to empowering diverse voices in Spanish media to publish news and information with independence, journalistic integrity, and a positive impact on the communities they serve. They conduct research, provide training, consulting, and financial support to help media leaders develop more sustainable business models in Latin America, Spain, and the U.S. Hispanic market.

At MainMicro, our goal is to ensure customer satisfaction by providing ongoing support and cost effective solutions for our partners. We take great pride in having a customer retention rate that is among the highest in the industry. For us, when you become a customer you also become a friend, and we become the one-stop shop for all of your IT related needs.

At Black Crow Labs we construct your brand’s ecosystem and tell your story.  By engaging with prospective customers on targeted platforms we integrate your brand into their lives and conversations.

  1. Home
  2. >
  3. Press Release
Categories
Advocacy DDoS Press Release

Deflect website security services for free in response to COVID-19

In response and solidarity with numerous efforts that have sprung up to help with communications, coordination and outreach during the COVID-19 epidemic, eQualitie is offering Deflect website security and content delivery services for free until the end of 2020 to organizations and individuals working to help others during this difficult time. This includes:

  • Availability: as demand for your content grows, our worldwide infrastructure will ensure that your website remains accessible and fast
  • Security: protecting your website from malicious bots and hackers
  • Hosting: for existing or new WordPress sites
  • Aanalytics: view real-time statistics in the Deflect dashboard

Deflect is always offered free of charge to not-for-profit entities that meet our eligibility requirements. This offer extends our free services to any business or individual that is responding to societal needs during the pandemic, including media organizations, government, online retail and hospitality services, etc. We will review all applications to make sure they align with Deflect’s Terms of Use.

It takes 15 minutes to set up and we’ll have you protected on the same day. Our support team can help you in English, French, Chinese, Spanish and Russian. If you have any questions please contact us.