Hello, all!
I’ve been missing out on climbing because I’m bad at remembering to check the MIT climbing wall site to see if it’s open. Instead, I decided to learn a little bit about Python’s web scraping, how cron works, and how to set up a scripts page through MIT. All code can be found here.
Python Scraping and Emailing
To actually figure out how to scrape the page to see if the wall is open, we first need to realize which packages we should be using. The answer to that is urllib2 and BeautifulSoup. urllib2 allows us to take a URL and get the HTML from it. We can then use beautiful soup to find contents in the HTML text.
First, we do simple import statements and assign initial variables for later use:
from urllib2 import urlopen
from bs4 import BeautifulSoup
wallPage = "http://scripts.mit.edu/~mitoc/wall/"
openWallPage = urlopen(wallPage).read()
html = BeautifulSoup(openWallPage, "html.parser")
I looked at the actual HTML for the page, and was able to identify the lines that determined the status and changed.
<span class='entry open'><span class='name '>open</span></span>
<span class='entry closed'><span class='name dim'>closed</span></span>
Are the ones which actually change. The current status is given by “class=’name ‘.” So what we really need to do is find out if the contents of <span class=’entry open’> and <span class=’name ‘> are the same. I did this by a simple string check, with some BeautifulSoup parsing thrown in.
openEntry = str(html.find_all(class_="entry open"))[1:-1]
currentStatus = str(html.find_all(class_="name "))[1:-1]
return currentStatus in openEntry
And now we have some a simple script which tells us if the wall is open! There’s probably some magical way to do this better with BeautifulSoup. The string method is a little messy. Note that we have to take off the first and last character in the string, because this method returns something of the form [string], and having the brackets around a string (while looking for a substring) would cause the conditional to always be false.
The next part of this is to send an email whenever this returns true. For this, we need to use the libraries stmplib, and email. The import statements are:
import smtplib
from email.MIMEMultipart import MIMEMultipart
From there, it’s just a little bit of following a pattern. Here you can see what all I did to make Python send an email.
fromAddress = "wall-is-open@mit.edu"
toAddress = "wall-is-open@mit.edu"
message = MIMEMultipart()
message['From'] = fromAddress
message['To'] = toAddress
message['Subject'] = "The wall is open! eom"
text = message.as_string()
server = smtplib.SMTP('outgoing.mit.edu',587)
server.starttls()
server.ehlo()
server.login("MIT_username", "password")
server.sendmail(fromAddress, toAddress, text)
The email address I want to send and receive mail to is wall-is-open@mit.edu, which I can easily make a mailing list (something awesome about being an MIT student). From there, it’s a game of adding the correct SMTP settings. Here I’m using outgoing.mit.edu, since I’ll be sending and receiving from an MIT server. This requires an MIT username and password, which I have omitted here.
Great. Now we have an email protocol and it can send an email if the wall is open, right? Nope. I thought this would work, but it turns out that getting an email every 5 minutes becomes annoying relatively quickly. Surprise there. What this code needs is a way to keep a state. All this means having a file which keeps track of the state by reading only “open” or “closed” depending on the previous state of the wall. This means we need to incorporate the reading and writing of the file.
with open("statusFile.txt", "r") as statusFile:
previousEntry = statusFile.readlines()[0]
if currentStatus in openEntry:
if previousStatus == "closed\n":
#block of the info to send emails, and then...
with open("statusFile.txt", "w") as statusFile:
statusFile.write("open\n")
else:
with open("statusFile.txt", "w") as statusFile:
statusFile.write("closed\n")
And there we go! A lot of the challenge of this came from two sources: 1, trying to do file permissions (since I was SSHing into an Athena computer, and running the code virtually), and 2, getting the correct file path (because this was done using Linux; if you’re also using Linux, you need the complete file path for statusFile.txt).
CRON Scripts
This turned out to be easier than I thought it would. MIT has a different way of using scripts to write cron commands, but it turned out to not be much of an issue. There was actually only one line to write (which I probably wrote inefficiently)
0,5,10,15,20,25,30,35,40,45,50,55 * * * * python ~/Programming/climbing_script.py
And there we go! Now the python script runs once every five minutes, and updates you if the wall is open (and it was previously closed). If you’re on MIT, add yourself to wall-is-on@mit.edu :D
A security researcher by day, a self-studying armchair psychologist by night
Coping With Computers 
Leave a comment