Panels is a pluggable rendering engine. Apart from its datastructure, almost nothing in Panels is hardcoded; every level of its rendering process is controlled by plugins (of the CTools variety), and therefore can be overriden. Some of these plugins' behaviors are obvious from the UI, whereas others are much more under-the-hood.

Most site builders and module authors will find that plugins are the only way they need to interact with Panels to do what they want. However, if you are building whole new UIs/applications that are to be driven by the Panels engine, you'll also need the more traditional API that manages Panels editing and CRUD. Panels Node and Mini Panels are examples of 'applications' in this sense, as is OG Panels.

This page is divided into two sections, the first dealing with Panels' rendering system & the plugins the drive it, and the second with the requirements for creating a Panels-driven application/UI. The former is a good starting place and should be of interest to anyone wanting to begin working with the Panels API.

The Panels Rendering System

Panels' rendering system takes a fully-loaded panels_display object and turns it into HTML. It doesn't care how that object was built or where it comes from - it's just a rendering engine. So long as the calling code provides a well-formed panels_display object, the engine will spit out HTML. Exactly how that process works is determined entirely by plugins.

Panels utilizes eight different types of plugins within its rendering system, four owned by Panels and four by CTools. Let's run through that list.

Layout plugins
Owned by Panels.
Layout plugins provide the HTML output skeleton for any panel: they define a set of regions and (optionally) accompanying CSS/JS, very analogous to how themes define block regions. While layout plugins with complex logic are possible (the Flexible layout, for example), most layouts are nothing more than a region list, a tpl.php skeleton, and some CSS.
Style plugins
Owned by Panels.
Style plugins control the markup that wraps both individual panes and whole panel regions. A basic style plugin need not provide more than some basic CSS and a theme function or tpl.php file.
Style Bases
Owned by CTools.
Panels implements the 'style_bases' plugin type in order to interact with the Stylizer system, which technically lives in CTools, though true 'ownership' is a little blurred. Regardless, Panels uses this plugin to interact with Stylaizer in order to allow the creation of style plugins in the UI.
This plugin is probably of interest only to module developers; site builders can safely skip it.
Context plugins
Owned by CTools.
Context plugins are abstracted wrappers around bits of data - a node, user, or taxonomy term are easy examples. Panels uses context plugins so that it can work with all these different bits of data without caring about the details of the datastructure itself. Context can be a confusing concept, and you can learn more in the CTools help.
Note that argument and relationship plugins are technically also used by Panels, but because they're really means for creating more context, they don't get separate entries in this list.
Content Type plugins
Owned by CTools: can utilize context.
At the heart of the Panels rendering process, content type plugins define the rendering logic for every pane. Offering Panels a renderable chunk from your module means defining one of these. For those accustomed to the block system, they can be thought of as a more verbose block definition, contained neatly inside a single plugin.
Content type plugins rely on context to provide them with source data, when necessary; this makes content plugins nice and portable.
Access plugins
Owned by CTools: can utilize context.
Access plugins are essentially portable access checking logic, responsible for answering "Yes" or "No" given some contextual data. Panels uses these to define 'visibility rules' on individual panes. The other place these are commonly seen is in defining 'Selection rules' on Page Manager variants.
If your module introduces new permissioning concepts or access logic (NOT a basic hook_perm() implementation), you may want to consider also encapsulating that logic in one or more of these plugins. For example, Organic Groups might implement an access plugin to allow/deny based on group membership, or the Date module might implement allow/deny logic based on request time.
Cache plugins
Owned by Panels; should utilize context.
Panels cache plugins define caching strategies that can be applied to individual panes, or to whole displays. Caching plugins must deal with two basic issues: the cache backend to use for reading/writing, and TTL-managing strategies.
TTL strategies can (and should) integrate heavily with the context, as that is where the real differential benefit of Panels-based caching is vs. standard core caching.
Display Renderers
Owned by Panels: utilizes everything else.
In the simplest terms, display renderers take a fully-loaded panels_display object and render it. They are very complex, powerful plugins - one could use them to cause Panels to bypass every single other plugin in this list. Fortunately, there is virtually no circumstance under which one would need to implement these plugins in the process of building a single site; only contrib developers working on highly complex use cases are likely to ever need them.
In short, each display renderer plugin is a self-contained Panels rendering engine. If you want to grok the Panels rendering system, study these.

Panels-based Applications

At its core, Panels is a rendering engine with no UI (though it does provide common elements for reuse in admin UIs). So if you're working with Panels via the web UI, it means you're working with a Panels application. The three that ship with this version of Panels are Mini Panels, Panel Nodes, and the integration with CTools' Page Manager.

Panels also implements a task_handler plugin, which is owned by Page Manager in CTools. Task handlers aren't strictly a part of Panels' rendering system itself though (strictly speaking, they wrap it), so we won't cover them here.