Ross’ PhD Blog Memex

• Currently Listening to: Phoenix — Too Young

I attended the workshop on Software Engineering Challenges for Ubiquitous Computing (SEUC 2006) in Lancaster, presided over by Gerd Kortuem.

After a somewhat hurried paper submission about using AOP in automotive software, I decided to change tack, so my presentation was about what kind of problems software engineers in the automotive space are facing. Admittedly I wasn’t presenting any answers, but my presentation went well and, being the only presenter discussing automotive systems and autonomics explicitly, I got a number of interesting questions which created a good discussion.

Software Considerations for Automotive Pervasive Systems Talk given at SEUC2006, June 1–2, Lancaster UK.

Here’s an excerpt from my talk:

The modern car is a highly sensorial, complex pervasive system, with thousands of sensors and actuators and hundreds of microcontrollers controlling almost all aspects of the car’s operation; from the multimedia & entertainment systems (radio, DVD players) and navigation/mapping software, to communication both to the outside world and also on a more limited scale to other cars nearby on the road.

Finally, and most importantly, are the car’s safety systems. Most of the impetus for adding so much software to cars is the supposed benefits to driver safety. And when it works, this is great, but we must also recognise that the stakes are higher. There are dangers involved that most pervasive systems don’t have to be concerned with.

System Personalisation

Much of the talk and discussion involved the implications of personalisation in automotive systems. In the future and to an extent even now, you can choose which features you want your car to come shipped with. This is likely to increase in scale over time, so that a car’s base configuration can be permuted in thousands of ways for each buyer. Modules need to be unobtrusively integrated and interoperable.

Layered on top of this is the possibility of a car being modified, upgraded or damaged over time. Cars will have to be able to adapt to whatever components they have installed, and thus, there is a lot of autonomic computing involved.

Questions & Answers

A few brief (paraphrased) questions and answers that I remembered to write down (not guaranteed to be correct!):

Will hardware and software become increasingly decoupled in automotive systems?

This doesn’t seem to be the way it’s going. As far as I can see (and this was backed up at the workshop), the hardware and software systems seem to be getting more tightly coupled if anything.

What is the development process at the car OEMs?

I couldn’t answer this, but someone else stepped in and suggested that a lot of OEM’s in-house teams are actually graviating towards being software-only development houses, with hardware being contracted out to other companies.

Does the drive-by-wire filtering of a user’s input spoil the love of driving?

Not really a research question, but interesting nonetheless. I do wonder how many drivers could honestly say they’d prefer the primal thrill of risky, unrestricted driving over the increased safety and stability benefits of these modern cars.

• Currently Listening to: The Strokes — What Ever Happened?

After an interesting meeting today, we’ve each chosen a website to extract data from, to be fed into construct as RDF. The idea of standardising on Python for all of the newly created sensors was brought up, which is good as I’ve already started working on my Python scraper.

Hidden Data

I didn’t mention this in the meeting, but some very useful data, like currency conversion rates, are generally not shown on public-facing websites. To get at them requires a form submission, and then scraping the resulting HTML page. Things I learned during my fourth year project may be able to help here, since one of the sites I tested on was this currency conversion page.

Access to a feed of realtime data costs $540 a year. With my project, for the cost of a HTTP GET request, you could have up to the minute data on any currency available in their system. This was made possible by a very useful Perl module called HTML::Form, which allowed me to simulate form submits, and thus retrieve the HTTP response page. Something similar is bound to exist for Python.

Working with Trees

There are two main approaches to screen-scraping: using heavy, regular expression-laden parsing for certain patterns of text in a string, or constructing a treelike representation of a page in memory, and then traversing this tree looking for certain elements. My favoured method is the latter, since it is generally more robust to small cosmetic changes to the underlying HTML page. Scraper rewrites are still required for when a page is reorganised, but this happens less frequently than a site having a few colours changed around.

Beautiful Soup is a very useful package for Python, which will robustly convert even an invalid HTML page into a tree, and then provides you the methods required to traverse the tree. This way, scrapers can be bashed out pretty quickly. Here’s some code to set it up; after this’ll come the page-specific code that extracts the relevant table rows or whatever is required.

import urllib, sys, re, BeautifulSoup

def get_page(url):
    """Fetches an arbitrary page from the web and prints it."""
    try:
        location = urllib.urlopen(url)
    except IOError, (errno, strerror):
        sys.exit("I/O error(%s): %s" % (errno, strerror))
    content = location.read()

    # Clear out all troublesome whitespace
    content = content.replace("\n", "")
    content = content.replace("\r", "")
    content = content.replace("\t", "")
    content = content.replace("> ", ">")
    content = content.replace("  ", " ")

    location.close()
    return content

def generate_tree(page):
    """Converts a string of HTML into a document tree."""
    return BeautifulSoup.BeautifulSoup(page)

Once you have this set up, fetching a certain element on a page becomes as easy as writing:

print generate_tree(get_page('http://www.imdb.com/')).first('table')

Polling Period

We discussed how often the sensors/scrapers should fetch their target webpage to re-parse it. Polling a page too often is likely to get your IP address blocked. Personally I don’t think this is as big a problem as was made out. Most RSS readers are designed to poll a feed once every 30 minutes to an hour. This is a reasonable period. Bar a few examples (stock quotes specifically), very few sites that we’re monitoring will be updating more frequently than that. In fact, the period could likely be increased. It would be relatively simple to set up a cron job to run each of the sensors in order every 30 minutes.

This approach could then be extended. RSS readers are/should be designed to honour various HTTP headers so that they don’t continually re-fetch the same feed over and over again if it’s not changing. All HTML files are sent with those same headers, so we could have conditions set up that the sensors will first do a HEAD request, and if we get a 304 response or if the Last Modified headers are within the last update cycle, we defer the update until the next cycle.

Ideally, the polling would be adaptive, so we have a single script that takes as input the derived update frequency of each page, and writes a new cron file with modified periodicity for each site. Thus, pages like the Dublin Bus timetables, which I’m working on, will be re-parsed very infrequently, since the site is rarely updated. Conversely, sites that serve constantly-updated information, like stock quotes and currency conversion rates, will be fetched much more often (but never more than a lower bound, like every 10 minutes).

I’ve begun learning some Python, primarily because Mark found an open source 3D graphics package called Blender, which uses packages written in Python.

So far, it looks like it’s similar in many ways to Perl, which is good because I already have plenty of experience with Perl, having used it for my final year project. Also, Lorcan is talking about doing some screen-scraping on major websites to glean data like movie showtimes and current stock prices, to be fed into Construct as contextual data.

I’ve done some screen-scraping in Perl before, but I’m guessing most of the others won’t want to code their screen-scrapers in Perl too. This will lead to serious code maintainability problems, which will invariably happen since every time the source website is updated you may have to recode some or all of your corresponding screen-scraper. Such is the life we’ve chosen. It would be best if we didn’t have to have designated caretakers for each module, so standardising on one language for them all would be nice. And I know which way the tide is turning (Joe characterised Perl with “I don’t like any language where my cat can walk across the keyboard and it will still compile” — touché).

Update: I’ve done some work on a scraper in Python.