# Purge
##### _The modular external cache invalidation framework._

The Purge module for Drupal 8 and Drupal 9 enables invalidation of content from
external caches, reverse proxies and CDN platforms. The technology-agnostic
plugin architecture allows for different server configurations and use
cases. Last but not least, it enforces a separation of concerns and should be
seen as a **middleware** solution.

##### Drush commands
The ``purge_drush`` module adds the following commands for Drush administration:

| Command                         | Alias    | Description                                                  |
|---------------------------------|----------|--------------------------------------------------------------|
| **``cache:rebuild-external``**  | ``cre``  | Invalidate 'everything' using the Purge framework.           |
| **``p:debug-dis``**             | ``pddis``| Disable debugging for all of Purge's log channels.           |
| **``p:debug-en``**              | ``pden`` | Enable debugging for all of Purge's log channels.            |
| **``p:diagnostics``**           | ``pdia`` | Generate a diagnostic self-service report.                   |
| **``p:invalidate``**            | ``pinv`` | Directly invalidate an item without going through the queue. |
| **``p:processor-add``**         | ``pradd``| Add a new processor.                                         |
| **``p:processor-ls``**          | ``prls`` | List all enabled processors.                                 |
| **``p:processor-lsa``**         | ``prlsa``| List available processor plugin IDs that can be added.       |
| **``p:processor-rm``**          | ``prrm`` | Remove a processor.                                          |
| **``p:purger-add``**            | ``ppadd``| Create a new purger instance.                                |
| **``p:purger-ls``**             | ``ppls`` | List all configured purgers in order of execution.           |
| **``p:purger-lsa``**            | ``pplsa``| List available plugin IDs for which purgers can be added.    |
| **``p:purger-mvd``**            | ``ppmvd``| Move the given purger DOWN in the execution order.           |
| **``p:purger-mvu``**            | ``ppmvu``| Move the given purger UP in the execution order.             |
| **``p:purger-rm``**             | ``pprm`` | Remove a purger instance.                                    |
| **``p:queue-add``**             | ``pqa``  | Add one or more items to the queue for later processing.     |
| **``p:queue-browse``**          | ``pqb``  | Inspect what is in the queue by paging through it.           |
| **``p:queue-empty``**           | ``pqe``  | Empty the entire queue.                                      |
| **``p:queue-stats``**           | ``pqs``  | View the queue statistics.                                   |
| **``p:queue-volume``**          | ``pqv``  | Count how many items are currently in the queue.             |
| **``p:queue-work``**            | ``pqw``  | Process one or more chunks of items from the queue.          |
| **``p:queuer-add``**            | ``puadd``| Add a new queuer.                                            |
| **``p:queuer-ls``**             | ``puls`` | List all enabled queuers.                                    |
| **``p:queuer-lsa``**            | ``pulsa``| List available queuer plugin IDs that can be added.          |
| **``p:queuer-rm``**             | ``purm`` | Remove a queuer.                                             |
| **``p:types``**                 | ``ptyp`` | List all supported cache invalidation types.                 |

Several commands understand the ``--format`` parameter allowing you to integrate
the commands in external scripts with JSON or YAML output. See the respective
``drush help <command>`` information for more command detail.

The framework explained
------------------------------------------------------------------------------
Purge isn't just a single API but made up of several API pillars all driven by
plugins, allowing very flexible end-user setups. All of them are clearly
defined to enforce a sustainable and maintainable framework over the longer
term. This also allows everyone to build, improve and fix bugs in only the
plugins they provide and therefore allows everyone to 'scale up' solving
external cache invalidation in the best way possible.

#### Queuer
With Purge, end users can manually invalidate a page with a Drush command or,
theoretically, via a "clear this page" button in the GUI. Caches
are however meant to be transparent to end users and to only be invalidated
when something actually changed - and thus requires external caches to also be
transparent.

When editing content of any kind, Drupal will transparently and efficiently
invalidate cached pages in Drupal's own **anonymous page cache**. When Drupal
renders a page, it can lists all the rendered items on the page in a special
HTTP response header named ``X-Drupal-Cache-Tags``. For example, this allows all
cached pages with the ``node:1`` Cache-Tag in their headers to be invalidated,
when that particular node (node/1) is changed.

Purge ships with the **Core tags queuer**, which replicates everything Drupal
core invalidated onto Purge's queue. So, when Drupal clears rendered items from
its own page cache, Purge will add a _invalidation_ object to its queue so that
it gets cleared remotely as well.

#### Queue
Queueing is an inevitable and important part of Purge as it makes cache
invalidation resilient, stable and accurate. Certain reverse cache systems can
clear thousands of items under a second, yet others - for instance CDNs - can
demand multi-step purges that can easily take up 30 minutes. Although the
queue can technically be left out of the process entirely, it will be required
in the majority of use cases.

###### Statistics tracker
The statistics tracker keeps track of queue activity by actively counting how
many items the queue currently holds and how many have been deleted or released
back to it. This data can be used to report progress on the queue and is easily
retrieved, the data resets when the queue is emptied.

#### Invalidations
Invalidations are small value objects that **describe and track invalidations**
on one or more external caching systems within the Purge pipeline. These
objects float freely between **queue** and **purgers** but can also be created
on the fly and in third-party code.

##### Invalidation types
Purge has to be crystal clear about what needs invalidation towards its purgers,
and therefore has the concept of invalidation types. Individual purgers declare
which types they support and can even declare their own types when that makes
sense. Since Drupal invalidates its own caches using cache tags, the ``tag``
type is the most important one to support in your architecture.

* **``domain``** Invalidates an entire domain name.
* **``everything``** Invalidates everything.
* **``path``** Invalidates by path, e.g. ``news/article-1``.
* **``regex``** Invalidates by reg. expression, e.g.: ``\.(jpg|jpeg|css|js)$``.
* **``tag``** Invalidates by Drupal cache tag, e.g.: ``menu:footer``.
* **``url``** Invalidates by URL, e.g. ``http://site.com/node/1``.
* **``wildcardpath``** Invalidates by path, e.g. ``news/*``.
* **``wildcardurl``** Invalidates by URL, e.g. ``http://site.com/node/*``.

#### Purgers
Purgers do all the hard work of telling external systems what to invalidate
and do this in the technically required way, for instance with external API
calls, through telnet commands or with specially crafted HTTP requests.

Purge **doesn't ship any purger**, as this is context specific. You could for
instance have multiple purgers enabled to both clean a local proxy and a CDN
at the same time.

###### Capacity tracker
The capacity tracker is the central orchestrator between limited system
resources and a never-ending queue of cache invalidation items.

The tracker actively tracks how much items are invalidated during Drupal's
request lifetime and how much PHP execution time has been spent. With this
information it can predict how much processing can happen during the rest of
request lifetime. It is able to predict this since the capacity tracker also
collects timing estimates from the actual purgers. The intelligence it has
is used by the queue service and exceeding the limit isn't possible as the
purgers service refuses to operate when the limits are near zero.

**Runtime measurement**

Purgers are required to provide timing estimates for a single invalidation,
the capacity tracker operates based on this information. Runtime measurement is
a feature available to purgers (most use it) which performs live time tracking
of invalidation processing, and reports gathered measurements back to the
capacity tracker. When a single invalidation was exceptionally slow - let's say
a server was under load - the capacity for this purger drastically drops, but
every faster measure collected after that will result in slow 10% upwards
adjustments. Combined with the capacity tracker, this provides the best balance
between performance and safety.

#### Diagnostic checks
External cache invalidation usually depends on many parameters, for instance
configuration settings such as hostname or CDN API keys. In order to prevent
hard crashes during runtime that affect end-user workflow, Purge allows plugins
to write preventive diagnostic checks that can check their configurations and
anything else that affects runtime execution. These checks can block all purging
but also raise warnings and other diagnostic information. End-users can rely on
Drupal's status report page where these checks also bubble up.

#### Processors
With queuers adding ``tag`` invalidation objects to the queue, this still leaves
the processing of it open. Since different use cases are possible, it is up to
you to configure a stable processing policy that's suitable for your use case.

Possibilities:

* **``cron``** claims items from the queue & purges during cron.
* **``ajaxui``** AJAX-based progress bar working the queue after a piece of
content has been updated.
* **``lateruntime``** purges items from the queue on every request (**SLOW**).

#### Tags Headers
By default, no HTTP response headers with cache tags are added when you install
just ``purge``. Since there is no RFC coverage for this relatively new way of
cache invalidation, every module providing a **purger** is expected to define
its own header and _most importantly_: unset that header too. This means that
if your CDN supports it, its expected that the CDN doesn't render the tags
header to end-users since you likely don't want to leak it. These plugins are
very simple and relies basically only on annotation. If you need to support a
reverse caching layer that isn't supported yet, the ``purge_purger_http``
project provides you with a ``Purge-Cache-Tags`` header.

API examples
------------------------------------------------------------------------------

#### Queueing
Adding invalidations to the queue is the simplest use case and requires a queuer
object so that the queue knows who is adding the given items.

```
$purgeInvalidationFactory = \Drupal::service('purge.invalidation.factory');
$purgeQueuers = \Drupal::service('purge.queuers');
$purgeQueue = \Drupal::service('purge.queue');

$queuer = $purgeQueuers->get('myqueuer');
$invalidations = [
  $purgeInvalidationFactory->get('tag', 'node:1'),
  $purgeInvalidationFactory->get('tag', 'node:2'),
  $purgeInvalidationFactory->get('path', 'contact'),
  $purgeInvalidationFactory->get('wildcardpath', 'news/*'),
];

$purgeQueue->add($queuer, $invalidations);
```

What happens now depends on the **processors you configured**, as some might
purge very quickly after adding items to the queue whereas others might need
a time-based delay before this occurs. Items enter the queue in state ``FRESH``
and normally leave the processor in the states ``SUCCEEDED``, ``FAILED``,
``PROCESSING`` or when no single plugins supported it: ``NOT_SUPPORTED``. Items
that don't succeed, cycle back to the queue until it gets manually cleared.

#### Invalidation without queue
Processing invalidations without going through the queue is possible, but not
the recommended workflow when your invalidations cannot fail. All it takes is to
instantiate invalidation objects and to feed them to the purgers service.

```
use Drupal\purge\Plugin\Purge\Purger\Exception\CapacityException;
use Drupal\purge\Plugin\Purge\Purger\Exception\DiagnosticsException;
use Drupal\purge\Plugin\Purge\Purger\Exception\LockException;
$purgeInvalidationFactory = \Drupal::service('purge.invalidation.factory');
$purgeProcessors = \Drupal::service('purge.processors');
$purgePurgers = \Drupal::service('purge.purgers');

$processor = $purgeProcessors->get('myprocessor');
$invalidations = [
  $purgeInvalidationFactory->get('tag', 'node:1'),
  $purgeInvalidationFactory->get('tag', 'node:2'),
  $purgeInvalidationFactory->get('path', 'contact'),
  $purgeInvalidationFactory->get('wildcardpath', 'news/*'),
];

try {
  $purgePurgers->invalidate($processor, $invalidations);
}
catch (DiagnosticsException $e) {
  // Diagnostic exceptions happen when the system cannot purge.
}
catch (CapacityException $e) {
  // Capacity exceptions happen when too much was purged during this request.
}
catch (LockException $e) {
  // Lock exceptions happen when another code path is currently processing.
}
```
When this code finished successfully, the ``$invalidations`` array holds the
objects it had before, but now each object has changed its state. You can now
verify this by iterating over the objects and by calling ``getState()`` or
``getStateString()`` on them (the latter is only intended for UI presentation):

```
foreach ($invalidations as $invalidation) {
  var_dump($invalidation->getStateString());
}
```

Which could then look like this:

```
string(6) "FAILED"
string(6) "FAILED"
string(9) "SUCCEEDED"
string(10) "PROCESSING"
```

The results reveal why you should **normally not invalidate without going
through the queue**, because items can fail or need to run again later to finish
entirely. The most common use case for direct invalidation is manual UI purging.

#### Queue processing
Processing items from the queue is handled by processors, which users can add
and configure according to their configuration. In essence, processors invoke
the following code to retrieve a dynamically calculated chunk of items from the
queue and feed those to the purgers service:

```
use Drupal\purge\Plugin\Purge\Purger\Exception\CapacityException;
use Drupal\purge\Plugin\Purge\Purger\Exception\DiagnosticsException;
use Drupal\purge\Plugin\Purge\Purger\Exception\LockException;
$purgePurgers = \Drupal::service('purge.purgers');
$purgeProcessors = \Drupal::service('purge.processors');
$purgeQueue = \Drupal::service('purge.queue');

$claims = $purgeQueue->claim();
$processor = $purgeProcessors->get('myprocessor');
try {
  $purgePurgers->invalidate($processor, $claims);
}
catch (DiagnosticsException $e) {
  // Diagnostic exceptions happen when the system cannot purge.
}
catch (CapacityException $e) {
  // Capacity exceptions happen when too much was purged during this request.
}
catch (LockException $e) {
  // Lock exceptions happen when another code path is currently processing.
}
finally {
  $purgeQueue->handleResults($claims);
}
```