Thousands of web sites, especially news sites and weblogs, syndicate their most recent site content for others to display. The syndicated content always includes titles, also known as headlines, for the newest published stories. Each headline acts as a direct link to the stories on the remote site. Along with the headline, most sites typically provide either the first few paragraphs of the story or a short summary. Many individuals use client-based news aggregators on their personal computer to aggregate content, such as Amphetadesk.
Drupal also has a news aggregator built in as a standard feature. With it, you can subscribe to feeds from other sites and display their content for your site users. Simply enable the aggregator module in site administration and enter the feeds that you choose.
What do I need to subscribe to a feed?
The standard method of syndication is using the XML-based Rich Site Summary (RSS). To syndicate a site's content, obtain the full URL of the RSS page providing syndication. Common file tags for RSS pages are .rss, .xml and .rdf. Example: Slashdot RSS.
Most weblog sites that offer syndication will have an obvious link on the main page. Often you need only look for an XML syndication button, such as the one Drupal uses for site syndication.
But some sites do not make their RSS feeds as easy to find. Or maybe you want to find a number of feeds on a given topic, without extensively searching the web. In that case, try an RSS syndication directory such as Syndic8.
NOTE: Enable your site's XML syndication button by turning on the Syndicate block in block management.
Configuring news feeds
To subscribe to an RSS feed on another site, use the RSS/RDF shortcut at the top of the news aggregation page. The link leads directly to the news aggregation configuration section of Drupal site administration.
Once there, select new feed from the left hand menu. Drupal will then ask for the following:
Title -- The text entered here will be used in your news aggregator, within the administration configuration section, and as title for the news feed block. As a general rule, use the web site name from which the feed originates.
URL -- Here you'll enter the fully-qualified URL for the feed for the site you want to subscribe to.
Attributes -- Attributes are keywords which can be used to collect feeds into bundles (see below). Think of these as the means of classifying your feeds. Separate multiple attributes with commas. If you do not plan on using the specific feed in a bundle, this input field can be left blank.
Update interval -- The update interval is how often Drupal will automatically access the RSS URL for the site for fresh content. The 1 hour default is typically the minimum you will want to use. Accessing another site's RSS page more frequently can be considered impolite. After all, it does require the other site's server handle your requests. To use this feature cron.php must be called regularly, otherwise, you'll have to manually update feeds one at a time within the news aggregation administration section by using update items.
Once you submit your new feed, check to see if it is working properly. Select update items on the RSS/RDF page. If you do not see any items listed for that feed, edit the feed and make sure that the URL was entered correctly.
Adding bundles
You may want to follow some feeds more closely than others. Or perhaps you'd like to display a select list of the titles for some feeds as a block for users. Bundles are a way of grouping your feeds into categories. Bundles look for feeds that contain at least one of the keywords, or attributes, associated with the bundle and display those feeds together.
When adding a bundle, Drupal will ask for:
Title -- The title will be used in the news by topics listing in your news aggregator and with the customized block created for the bundle.
Attributes -- Enter one or more of the attributes used to categorize the news feeds already created. Separate multiple attributes with commas. Be careful to use the same spelling. Don't have any feeds with attributes for the bundle? After creating the bundle, edit existing feeds or create new ones and tag them with the attribute.
Using the news aggregator
The news aggregator has a number of ways that it displays your subscribed content:
Latest news -- Displays all incoming content in the order received with:
The title of the original post.
The name of the source, which acts as a link to an individual feed page, listing information about that feed and incoming content for that feed only.
A description, the first few paragraphs or summary of the originating post (if any).
A blog it link. Users can select this link to have Drupal automatically prepare a blog post for the specific item.
A feed link, which acts as a link to an individual feed page, listing information about that feed and incoming content for that feed only.
News by source -- Organizes incoming content by feed, displaying titles which link to the originating post. Also has an icon which acts as blog it link.
News by topic -- Organizes incoming content by bundles, displaying titles which link to the originating post. Also has an icon which acts as blog it link.
News sources -- Displays an alphabetical listing of all subscribed feeds and a description. The title acts as a link to an individual feed page, listing information about that feed and incoming content for that feed only.
RSS feed blocks
In addition to providing subscribed content through the news aggregator, Drupal automatically creates a block for each subscribed feed and every bundle created. Beside each headline in each block, Drupal includes an icon which acts a blog it link. Enable any or all of the blocks using block management.
Subscription list
Drupal automatically generates an OPML feed file that is available by selecting the XML icon on the News Sources page.
Technical details
When fetching feeds Drupal supports conditional GETs, this reduces the bandwidth usage for feeds that have not been updated since the last check.
If a feed is permanently moved to a new location Drupal will automatically update the feed URL to the new address.
", array("%amphetadesk" => "http://www.disobey.com/amphetadesk/", "%rss" => "http://blogs.law.harvard.edu/tech/rss", "%slashdot-rss" => "http://slashdot.org/slashdot.rdf", "%syndic8" => "http://www.syndic8.com/", "%rss-what" => "http://www.xml.com/pub/a/2002/12/18/dive-into-xml.html", "%rss-evolution" => "http://www.webreference.com/authoring/languages/xml/rss/1/", "%admin-news" => url("admin/syndication/news"), "%new-feed" => url("admin/syndication/news/add/feed"), "%update-items" => url("admin/syndication/news")));
case 'admin/system/modules#description':
return t("Used to aggregate syndicated content (RSS and RDF).");
case 'admin/system/modules/aggregator':
return t("Drupal's news aggregator controls how many RSS/RDF items from a single source are displayed in a \"Block\", and on the page that goes with that block.");
case 'admin/syndication/news':
return t("Several web sites, especially news related sites, syndicate parts of their site's content for other web sites to display. Usually, the syndicated content includes the latest headlines with a direct link to that story on the remote site. Some syndicated content also includes a description of the headline. The standard method of syndication is using the XML based Rich Site Summary (RSS) or Resource Description Framework (RDF). To get a feed to work you must run \"cron.php\". To display the feed in a block you must turn on the %block.", array("%block" => l(t("feed's block"), "admin/system/block")));
case 'admin/syndication/news/add/feed':
return t("Add a site that has an RSS/RDF feed. The URL is the full path to the RSS feed file. For the feed to update automatically you must run \"cron.php\". The \"Attributes\" are used to bundle this feed with other feeds (See %bundle), and to tag articles from this feed. Note: If you already have a feed with the URL you are planning to use, the system will not accept another feed with the same URL.", array("%bundle" => l(t("add new bundle"), "admin/syndication/news/add/bundle")));
case 'admin/syndication/news/add/bundle':
return t("Bundles provide a generalized way of creating composite feeds. They allow you, for example, to combine various sport-related feeds into one bundle called Sport. If an article from a feed has been \"tag\"-ged (See %tag too look at and change tags.) with a matching \"Attribute\" then it will be added to the bundle.", array("%tag" => l(t("tag news item"), "admin/syndication/news/tag")));
case 'admin/syndication/news/tag':
return t("This allows you to see and change an news item's \"tag\". All articles are originally tagged with the \"Attributes\" of their feed.");
}
}
function aggregator_help_page() {
print theme('page', aggregator_help('admin/help#aggregator'));
}
function aggregator_settings() {
$number = drupal_map_assoc(array(5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100));
$output = form_select(t('Items per block'), 'aggregator_block_limit', variable_get('aggregator_block_limit', 15), $number, t('The maximum number of news items displayed in one block.'));
$output .= form_select(t('Items per page'), 'aggregator_page_limit', variable_get('aggregator_page_limit', 75), $number, t('The maximum number of news items displayed on one page.'));
return $output;
}
function aggregator_perm() {
return array('administer news feeds', 'access news feeds');
}
function aggregator_link($type) {
if ($type == 'page' && user_access('access news feeds')) {
return array(l(t('news feeds'), 'aggregator', array('title' => t('Read the latest news from syndicated web sites.'))));
}
if ($type == 'system') {
if (user_access('administer news feeds')) {
menu('admin/syndication', t('syndication'), 'aggregator_help_page', 5);
menu('admin/syndication/news', t('RSS/RDF'), 'aggregator_admin');
menu('admin/syndication/news/add/feed', t('new feed'), 'aggregator_admin', 2);
menu('admin/syndication/news/add/bundle', t('new bundle'), 'aggregator_admin', 3);
menu('admin/syndication/news/tag', t('tag items'), 'aggregator_admin', 4);
menu('admin/syndication/news/help', t('help'), 'aggregator_help_page', 9);
}
if (user_access('access news feeds')) {
menu('aggregator', t('news aggregator'), 'aggregator_page', 5);
menu('aggregator/feeds', t('news by source'), 'aggregator_page');
menu('aggregator/bundles', t('news by topic'), 'aggregator_page');
menu('aggregator/sources', t('news sources'), 'aggregator_page');
}
}
}
function aggregator_cron() {
$result = db_query("SELECT * FROM {feed} WHERE checked + refresh < %d", time());
while ($feed = db_fetch_array($result)) {
aggregator_refresh($feed);
}
}
function aggregator_update() {
$result = db_query("SELECT * FROM {feed}");
while ($feed = db_fetch_array($result)) {
aggregator_refresh($feed);
}
}
function aggregator_bundle_block($attributes) {
if ($attributes) {
$keys = explode(",", $attributes);
foreach ($keys as $key) {
$where[] = "attributes LIKE '%". check_query(trim($key)) ."%'";
}
$result = db_query_range("SELECT * FROM {item} WHERE ". implode(" OR ", $where) ." ORDER BY timestamp DESC, iid DESC", 0, variable_get("aggregator_block_limit", 15));
}
$items = array();
while ($item = db_fetch_object($result)) {
$items[] = theme("aggregator_block_item", $item);
}
return theme("item_list", $items);
}
function aggregator_feed_block($feed) {
$result = db_query_range("SELECT * FROM {item} WHERE fid = %d ORDER BY timestamp DESC, iid DESC", $feed->fid, 0, variable_get("aggregator_block_limit", 15));
$items = array();
while ($item = db_fetch_object($result)) {
$items[] = theme("aggregator_block_item", $item);
}
return theme("item_list", $items);
}
function aggregator_block($op, $delta) {
if (user_access('access news feeds')) {
if ($op == "list") {
$result = db_query("SELECT * FROM {bundle} ORDER BY title");
while ($bundle = db_fetch_object($result)) {
$block["bundle:$bundle->bid"]["info"] = "$bundle->title bundle";
}
$result = db_query("SELECT * FROM {feed} ORDER BY fid");
while ($feed = db_fetch_object($result)) {
$block["feed:$feed->fid"]["info"] = "$feed->title feed";
}
return $block;
}
else {
list($type, $id) = split(":", $delta);
switch ($type) {
case "feed":
$feed = db_fetch_object(db_query("SELECT * FROM {feed} WHERE fid = %d", $id));
$block["subject"] = $feed->title;
$block["content"] .= aggregator_feed_block($feed) ."
";
$block["feed:$feed->fid"]["info"] = "$feed->title feed";
}
return $block;
}
function aggregator_remove($feed) {
db_query("DELETE FROM {item} WHERE fid = %d", $feed["fid"]);
db_query("UPDATE {feed} SET checked = 0, etag = '', modified = 0 WHERE fid = %d", $feed['fid']);
drupal_set_message(t("removed news items from '%site'.", array("%site" => $feed["title"])));
}
// Call-back function used by XML parser:
function aggregator_element_start($parser, $name, $attributes) {
global $item, $element, $tag;
switch ($name) {
case "IMAGE":
case "TEXTINPUT":
$element = $name;
break;
case "ITEM":
$element = $name;
$item += 1;
}
$tag = $name;
}
// Call-back function used by XML parser:
function aggregator_element_end($parser, $name) {
global $element;
switch ($name) {
case "IMAGE":
case "TEXTINPUT":
case "ITEM":
$element = "";
}
}
// Call-back function used by XML parser:
function aggregator_element_data($parser, $data) {
global $channel, $element, $items, $item, $image, $tag;
switch ($element) {
case "ITEM":
$items[$item][$tag] .= $data;
break;
case "IMAGE":
$image[$tag] .= $data;
break;
case "TEXTINPUT":
/*
** The sub-element is not supported but we have recognize
** it or its content will end up in the items-array.
*/
break;
default:
$channel[$tag] .= $data;
}
}
function aggregator_refresh($feed) {
global $channel, $image;
// Generate conditional GET headers.
$headers = array();
if ($feed['etag']) {
$headers['If-None-Match'] = $feed['etag'];
}
if ($feed['modified']) {
$headers['If-Modified-Since'] = gmdate("D, d M Y H:i:s", $feed['modified']) ." GMT";
}
// Request feed.
$result = drupal_http_request($feed['url'], $headers);
// Process HTTP reponse code.
switch ($result->code) {
case 304:
db_query("UPDATE {feed} SET checked = %d WHERE fid = %d", time(), $feed['fid']);
drupal_set_message(t("no new syndicated content from '%site'.", array('%site' => $feed['title'])));
break;
case 301:
$feed['url'] = $result->redirect_url;
watchdog('special', "aggregator: updated URL for feed '$feed[title]' to $feed[url]");
case 200:
case 302:
case 307:
// Filter the input data:
if (!valid_input_data($result->data)) {
drupal_set_message(t("failed to parse RSS feed '%site': suspicious input data.", array("%site" => $feed["title"])), 'error');
}
else if (aggregator_parse_feed($result->data, $feed)) {
if ($result->headers['Last-Modified']) {
$modified = strtotime($result->headers['Last-Modified']);
}
/*
** Prepare the image data (if any):
*/
foreach ($image as $key => $value) {
$image[$key] = trim($value);
}
if ($image['LINK'] && $image['URL'] && $image['TITLE']) {
$image = "";
}
else {
$image = NULL;
}
/*
** Update the feed data:
*/
db_query("UPDATE {feed} SET url = '%s', checked = %d, link = '%s', description = '%s', image = '%s', etag = '%s', modified = %d WHERE fid = %d", $feed['url'], time(), $channel['LINK'], $channel['DESCRIPTION'], $image, $result->headers['ETag'], $modified, $feed['fid']);
/*
** Clear the cache:
*/
cache_clear_all();
$message = t("syndicated content from '%site'.", array("%site" => $feed["title"]));
watchdog('regular', "aggregator: $message");
drupal_set_message($message);
}
break;
default:
$message = t("failed to parse RSS feed '%site': %error.", array('%site' => $feed['title'], '%error' => $result->code .' '. $result->error));
watchdog('error', "aggregator: $message");
drupal_set_message($message);
}
}
function aggregator_parse_feed(&$data, $feed) {
global $items, $image, $channel;
// Unset the global variables before we use them:
unset($GLOBALS["element"], $GLOBALS["item"], $GLOBALS["tag"]);
$items = array();
$image = array();
$channel = array();
// parse the data:
$xml_parser = drupal_xml_parser_create($data);
xml_set_element_handler($xml_parser, "aggregator_element_start", "aggregator_element_end");
xml_set_character_data_handler($xml_parser, "aggregator_element_data");
if (!xml_parse($xml_parser, $data, 1)) {
$message = t("failed to parse RSS feed '%site': %error at line %line.", array("%site" => $feed["title"], "%error" => xml_error_string(xml_get_error_code($xml_parser)), "%line" => xml_get_current_line_number($xml_parser)));
watchdog('error', "aggregator: $message");
drupal_set_message($message, 'error');
return 0;
}
xml_parser_free($xml_parser);
// initialize the translation table:
$tt = array_flip(get_html_translation_table(HTML_SPECIALCHARS));
$tt["'"] = "'";
/*
** We reverse the array such that we store the first item last,
** and the last item first. In the database, the newest item
** should be at the top.
*/
$items = array_reverse($items);
foreach ($items as $item) {
unset($title, $link, $author, $description);
// Prepare the item:
foreach ($item as $key => $value) {
$item[$key] = filter_default(strtr(trim($value), $tt));
}
/*
** Resolve the item's title. If no title is found, we use
** up to 40 characters of the description ending at a word
** boundary but not splitting potential entities.
*/
if ($item["TITLE"]) {
$title = $item["TITLE"];
}
else {
$title = preg_replace('/^(.*)[^\w;&].*?$/', "\\1", truncate_utf8($item["DESCRIPTION"], 40));
}
/*
** Resolve the items link.
*/
if ($item["LINK"]) {
$link = $item["LINK"];
}
elseif ($item["GUID"] && (strncmp($item["GUID"], "http://", 7) == 0)) {
$link = $item["GUID"];
}
else {
$link = $feed["link"];
}
/*
** Try to resolve and parse the item's publication date. If no
** date is found, we use the current date instead.
*/
if ($item['PUBDATE']) $date = $item['PUBDATE']; // RSS 2.0
else if ($item['DC:DATE']) $date = $item['DC:DATE']; // Dublin core
else if ($item['DCTERMS:ISSUED']) $date = $item['DCTERMS:ISSUED']; // Dublin core
else if ($item['DCTERMS:CREATED']) $date = $item['DCTERMS:CREATED']; // Dublin core
else if ($item['DCTERMS:MODIFIED']) $date = $item['DCTERMS:MODIFIED']; // Dublin core
else $date = 'now';
$timestamp = strtotime($date); // strtotime() returns -1 on failure
if ($timestamp < 0) {
/*
** The Dublin core's default data format uses ISO 8601 which can't
** be parsed directly using PHP's strtotime(). It is not the only
** valid format so this might fail nonetheless ...
*/
list($year, $month, $day, $hour, $minute, $second) = sscanf($date, "%4d-%2d-%2dT%2d:%2d:%2d");
$timestamp = strtotime("$year-$month-$day $hour:$minute:$second");
}
if ($timestamp < 0) {
$timestamp = time();
}
/*
** Save this item. Try to avoid duplicate entries as much as
** possible. If we find a duplicate entry, we resolve it and
** pass along it's ID such that we can update it if needed.
*/
if ($link && $link != $feed["link"] && $link != $feed["url"]) {
$entry = db_fetch_object(db_query("SELECT iid FROM {item} WHERE fid = %d AND link = '%s'", $feed["fid"], $link));
}
else {
$entry = db_fetch_object(db_query("SELECT iid FROM {item} WHERE fid = %d AND title = '%s'", $feed["fid"], $title));
}
aggregator_save_item(array('iid' => $entry->iid, 'fid' => $feed["fid"], 'timestamp' => $timestamp, 'title' => $title, 'link' => $link, 'author' => $item["AUTHOR"], 'description' => $item["DESCRIPTION"], 'attributes' => $feed["attributes"]));
}
/*
** Remove all the old, expired items:
*/
unset($items);
$result = db_query("SELECT iid FROM {item} WHERE fid = %d ORDER BY timestamp", $feed["fid"]);
while ($item = db_fetch_object($result)) {
$items[] = "iid = '$item->iid'";
}
if (sizeof($items) > 50) {
db_query("DELETE FROM {item} WHERE ". implode(" OR ", array_slice($items, 0, - 50)));
}
return 1;
}
function aggregator_save_item($edit) {
if ($edit["iid"] && $edit["title"]) {
db_query("UPDATE {item} SET title = '%s', link = '%s', author = '%s', description = '%s', attributes = '%s' WHERE iid = %d", $edit["title"], $edit["link"], $edit["author"], $edit["description"], $edit["attributes"], $edit["iid"]);
}
else if ($edit["iid"]) {
db_query("DELETE FROM {item} WHERE iid = %d", $edit["iid"]);
}
else if ($edit["title"] && $edit["link"]) {
db_query("INSERT INTO {item} (fid, title, link, author, description, attributes, timestamp) VALUES (%d, '%s', '%s', '%s', '%s', '%s', %d)", $edit["fid"], $edit["title"], $edit["link"], $edit["author"], $edit["description"], $edit["attributes"], $edit["timestamp"]);
}
}
function aggregator_form_bundle($edit = array()) {
$form = form_textfield(t("Title"), "title", $edit["title"], 50, 64, t("The name of the bundle."));
$form .= form_textfield(t("Attributes"), "attributes", $edit["attributes"], 50, 128, t("A comma-separated list of keywords describing the bundle."));
$form .= form_submit(t("Submit"));
if ($edit["bid"]) {
$form .= form_submit(t("Delete"));
$form .= form_hidden("bid", $edit["bid"]);
}
return form($form);
}
function aggregator_save_bundle($edit) {
if ($edit["bid"] && $edit["title"]) {
db_query("UPDATE {bundle} SET title = '%s', attributes = '%s' WHERE bid = %d", $edit["title"], $edit["attributes"], $edit["bid"]);
}
else if ($edit["bid"]) {
db_query("DELETE FROM {bundle} WHERE bid = %d", $edit["bid"]);
}
else if ($edit["title"]) {
// a single unique id for bundles and feeds, to use in blocks
$next_id = db_next_id("{bundle}_bid");
db_query("INSERT INTO {bundle} (bid, title, attributes) VALUES (%d, '%s', '%s')", $next_id, $edit["title"], $edit["attributes"]);
}
}
function aggregator_form_feed($edit = array()) {
$period = drupal_map_assoc(array(900, 1800, 3600, 7200, 10800, 21600, 32400, 43200, 64800, 86400, 172800, 259200, 604800, 1209600, 2419200), "format_interval");
if ($edit["refresh"] == "") {
$edit["refresh"] = 3600;
}
$form .= form_textfield(t("Title"), "title", $edit["title"], 50, 64, t("The name of the feed; typically the name of the web site you syndicate content from."));
$form .= form_textfield(t("URL"), "url", $edit["url"], 50, 128, t("The fully-qualified URL of the feed."));
$form .= form_textfield(t("Attributes"), "attributes", $edit["attributes"], 50, 128, t("A comma-separated list of keywords describing the feed."));
$form .= form_select(t("Update interval"), "refresh", $edit["refresh"], $period, t("The refresh interval indicating how often you want to update this feed. Requires crontab."));
$form .= form_submit(t("Submit"));
if ($edit["fid"]) {
$form .= form_submit(t("Delete"));
$form .= form_hidden("fid", $edit["fid"]);
}
return form($form);
}
function aggregator_save_feed($edit) {
if ($edit["fid"] && $edit["title"]) {
db_query("UPDATE {feed} SET title = '%s', url = '%s', attributes = '%s', refresh = %d WHERE fid = %d", $edit["title"], $edit["url"], $edit["attributes"], $edit["refresh"], $edit["fid"]);
db_query("DELETE FROM {item} WHERE fid = %d", $edit["fid"]);
}
else if ($edit["fid"]) {
db_query("DELETE FROM {feed} WHERE fid = %d", $edit["fid"]);
db_query("DELETE FROM {item} WHERE fid = %d", $edit["fid"]);
}
else if ($edit["title"]) {
// a single unique id for bundles and feeds, to use in blocks
$next_id = db_next_id("{feed}_fid");
db_query("INSERT INTO {feed} (fid, title, url, attributes, refresh) VALUES (%d, '%s', '%s', '%s', %d)", $next_id, $edit["title"], $edit["url"], $edit["attributes"], $edit["refresh"]);
}
}
function aggregator_save_attributes($edit) {
foreach ($edit as $iid => $value) {
db_query("UPDATE {item} SET attributes = '%s' WHERE iid = %d", $value, $iid);
}
drupal_set_message(t('the attributes has been saved'));
}
function aggregator_get_feed($fid) {
return db_fetch_array(db_query("SELECT * FROM {feed} WHERE fid = %d", $fid));
}
function aggregator_get_bundle($bid) {
return db_fetch_array(db_query("SELECT * FROM {bundle} WHERE bid = %d", $bid));
}
function aggregator_view() {
$result = db_query("SELECT f.*, COUNT(i.iid) AS items FROM {feed} f LEFT JOIN {item} i ON f.fid = i.fid GROUP BY f.fid, f.title, f.url, f.refresh, f.checked, f.attributes, f.link, f.description, f.etag, f.modified, f.image ORDER BY f.title");
$output .= "