Issue #3389611 by pfrenssen, cilefen, xjm, poker10: Update robots.txt with all README files core ships with

Issue #3271222 by Rajab Natshah, yogeshmpawar, BramDriesen: Include Disallow Oembed media links in the robots.txt file for better Drupal SEO

Issue #3123285 by Neslee Canil Pinto, amjad1233, shaktik, Chi, xjm, bkosborne, quietone: Actually exclude user register, login, logout, and password pages from search results in robots.txt (current rules are broken)

Issue #1137848 by Tor Arne Thune, mgifford, GaëlG, ericjenkins: /filter/tips page is listed by search engines

Issue #2364343 by damien_vancouver, criz, ksenzee, joegraduate, Neograph734, droplet, pounard, jp.stacey, ciss: Fix robots.txt to allow Google access to CSS, JavaScript and image files

Issue #2364343 by ksenzee | corbacho: Fixed robots.txt to allow Google access to CSS and JavaScript files.

- Patch #1183208 by effulgentsia, Niklas Fiekas, katbailey, no_commit_credit, catch: remove variable_get()('clean_url()') and switch to index.php/path pattern for dirty URL support.

Nate Lampton authored Oct 30, 2011 and catch committed Nov 01, 2011

Issue #22336 by quicksketch, scor, boombatower, and rfay. Move all core Drupal files under a core subdirectory.

Issue #1137848 by BrockBoland, tim.plunkett, jensimmons: Disallow /filter/tips page from search engines.

- Patch #337820 by Dave Reid, Damien Tournoud, et al: rename menu path 'logout' to 'user/logout'.  For real now.

- commiting patch to cloud.module from Remco.

Again, a large batch of updates - I'm twisting things around here:

 1. improved .htaccess to be more "secure": to keep prying
    eyes out

 2. rewrote the administration section from scratch using a
    modular approach

 3. improved the information gathered by error.php - we can
    now (hopefully) track what bots are crawling us.

 4. fixed a bug in submit.php, fixed a bug in theme zaphod,
    fixed a bug in theme marvin.

 5. rewrote cron from scratch - it now interfaces with
    modules as it should have been from the beginning.
    Very cool if you ask me - it can use UNIX/Linux
    crontabs.

 6. updated widget.inc to be module aware - needs more
    work though - maybe this afternoon?

 7. updated most modules: small bugfixes, improvements, and
    even the documentation

 8. removed diary.php and made it a module - you can now
    run a drop.org site without a diary system if someone
    would prefer so

 9. updated all themes to use the new modules where
    appropriate

 10. added a robots.txt because the error message in the
     watchdog become annoying.

 11. added the new configuration system (mutliple vhosts
     on the same source tree) - use hostname.conf instead
     of config.inc

 12. removed calendar.inc and made it a module

 13. added format_interval() to functions.inc (UnConeD)

 14. whatever I forgot ...