django

check if your site validates using your google sitemap

2006-11-26T15:27:42-06:00

i was playing with the google sitemap (means i was activating the google sitemap support in django :)

and then i had an idea… this sitemap basically describes all the urls of a site. with this, i could simply check all my urls on the w3 validator. automatically. every night, for example.

but this requires to be able to get some kind of machine-readable output from the w3 validator.

as it turns out, there are 2 ways:

the normal validating url looks like : http://validator.w3.org/check?uri=[escaped url] . if you add “&output=soap12” to it, then it returns an xml file, which describes the validation results
regardless of the output-method (human-readable html, or machine-readable xml), the validator adds several custom headers to the http response. one of them is “x-w3c-validator-errors”, which returns the number of validation errors. if there are no errors, it’s zero.

i decided to use the second approach, mostly because it’s simpler.

so i wrote a simple python script, that:

fetches the sitemap file
submits every url in it to the validator, and extracts the validation-error-count.

the script turned out to be very simple:

from urllib2 import urlopen
from httplib import HTTPConnection
from urllib import quote
from elementtree import ElementTree
from time import sleep
 
SITEMAP_URL = 'http://nekomancer.net/sitemap.xml'
VALIDATOR_URL = 'validator.w3.org'
 
LOC_LOCATION = '{http://www.google.com/schemas/sitemap/0.84}url/{http://www.google.com/schemas/sitemap/0.84}loc'
 
def get_urls(sitemap_url):
    c = urlopen(sitemap_url)
    data = c.read()
    c.close()
    tree = ElementTree.fromstring(data)
    return [loc.text.strip() for loc in tree.findall(LOC_LOCATION)]
 
def get_error_count(url):
    c = HTTPConnection(VALIDATOR_URL)
    c.request('HEAD','/check?uri=%s' % quote(url))
    r = c.getresponse()
    c.close()
    return int(r.getheader('x-w3c-validator-errors'))
 
urls = get_urls(SITEMAP_URL)
 
for url in urls:
    print get_error_count(url),url
    sleep(2)

to parse the sitemap.xml i chose to use ElementTree, but i could have used also regular expressions, to match the URL parts

feeds and time

2006-11-01T17:50:08-06:00

i use a lot of rss/atom feeds. by use i mean that i am reading many news-sites, blogs, etc. using their feeds.

for some reason those feeds always contain the last n entries, where n is a fixed number. so a feed contains let’s say the last 10 entries.

this is fine as long as the page (which is represented in the feed) does not get updated too often.

otherwise the following scenario might happen:

let’s say the feed contains the last 10 entries
and you fetch the feed once a day using your feed-reader program
now, for some reason, one day there will be 11 new entries
when you fetch the feed next day, only the last 10 entries will be in the feed
so you missed one article on that website

the problem is, that these feeds do not provide any guarantees about their “completeness”.

(this is not an “in theory only” problem. once i had this exact problem.a website’s feed only contained the last x (50, iirc) entries, and much more arrived daily. so i kept missing them, and if i wanted to be sure that i saw all of them, i had to manually go through the entries on the site. kind of defeats the purpose of the feed. nowadays i’m using bloglines, which is fetching the feeds a lot more often than i did, so i do not have the problem with that site anymore)

what i would like to see, is feeds that contain all the entries for a specified timeframe.for example for the last 48 hours. they can contain of course more entries, but they would have to guarantee, that they contain at least all the entries for the last 48 hours. which would mean, that if you fetch the feed daily, you will not miss any entries.

for example, let’s do it in django:

django contains a complete feed-framework, with documentation, so first go and read the documentation

as you see, the whole issue of “showing the last n entries” is being handled in the items() method:

(the example from the documentation)

def items(self):
    return NewsItem.objects.order_by('-pub_date')[:5]

we want to return at least 10 entries, and we want to make sure that we return all the entries for the last 48 hours.

so we could use something like this:

from datetime import datetime, timedelta
 
def items(self):
    entries = NewsItem.objects.order_by('-pub_date')
    two_days_ago = datetime.now() - timedelta(days=2)
    last_entries = entries.filter(pub_date__gte = two_days_ago)
    if last_entries.count() &gt;= 10:
        return last_entries
    else:
        return entries[:10]

and we’re done.

now if only i could find out how to do it in wordpress (this blog is running on it)

P.S: the code i showed here was not tested extensively, so it might contain bugs. but the basic idea should work.

Manipulators and raw_id_admin

2006-09-03T17:38:33-05:00

for all Django users…

please, please, pretty please use raw_id_admin (django model docs).

because, when you have a model that contains a ForeignKey (and people usually have ForeignKeys), then when you use an automatic manipulator for the given model, it will load in ALL THE DATA FROM ALL THE RELATED MODELS. for example, if you have 20.000 entries in the related table, then it will load in all those 20.000 entries.

except, if you use raw_id_admin. it’s an attribute of the ForeignKey, and contrary to the documentation, it’s effect is not restricted to the admin-framework.

Joseph Heck made some speed-tests regarding this issue