We made a couple of maintenance updates to Tempo last night, and added in a couple of features that numerous users have been asking us for:
Recently I met Alexis Rondeau, one of the two clever fellows that created the site Semapedia.org. The site allows you to create 2D barcodes, called Semapedia Tags, that link to information on Wikipedia. The idea is that you print a tag that links to information about a place or thing, then you stick that tag on the place or thing. Anyone with a 2D scanner program on their phone can then lookup the information at the site when they see the tag. Pretty cool! (I’m still trying to get the program for my Treo installed correctly.)
Anyway, I saw this on their blog and I had to share:
so you are on a bus stop and there is a barcode to download the bus schedule. Great Idea, not. The poster with the barcode takes a whole side of the bus stop, why not just print the time table, how often does that change? Why pay for anything like that. If the service behind the barcode would tell you exactly in realtime where the bus currently is located or tell you if any of your friends are on that bus, then we have something a printed time table cannot provide and is clearly more attractive. Haven’t seen any of the other ideas, but for starters, detect needs, find out what current medias don’t provide and so on.
Indeed! Always go for the simplest solution.
Warren Ellis wonders aloud why “videomail, in these broadband days of ours, has never made a bigger dent. Why I don’t get videomail in my inbox along with email.”
I’d say it’s because most of us see a camera pointed at us and we feel the need to act. It’s rather difficult for a lot of people to “be natural” when they are being filmed.
When most folks seem to be looking for ever faster, ever more seamless (and often literally asynchronous) communication, the last thing they want to do is “have to act” for a two minute videomail.
We all love mash-ups, right? Especially us developers, builders of fine web tools. When we build useful web applications I think we all tend to want to provide integration hooks to other services because our users will get more functionality (in many cases they get more bang for their buck, so to speak) and because it’s kinda cool! Nothing wrong with that, gives you something to get in touch with your users about, sometimes gets you a bit of a press, too.
But mash-ups aren’t all fun and games, they require some careful planning and hard work, even if your current system is well designed with low-coupling and a good MVC model. I saw this post by Hampton over at Unspace and got to thinking that I ought to do a little musing on coding for failure and discuss some of the techniques we’ve used in our services.
When you run a reminder service like PingMe, where your users trust you to deliver their messages without fail and on time, you have to step up your game when it comes to implementing a robust system. When you then integrate your app with an external service like Twitter to provide your users with a useful and cheap SMS/text messaging interface, you have to consider the reliability of that external service and code for failure.
Now, on some level there’s only so much failure you can prevent. Mail systems and domains can go dark, e-mail to sms gateways can blink out, there’s not much you can do about it beyond picking a good MTA and spending a solid amount of time configuring it properly. (We highly recommend Exim, which is the most flexible one out there with great documentation and a strong user/development community.)
The great thing about serious business mail servers like Exim is that they have been very good at handling failure, retrying, and eventually giving up for a very long time, and negotiate this process with other mail servers over a long-established protocols. So if we send a message to your_phone_number@vtext.com (Verizon Wireless’s email-to-sms gateway), and the vtext.com MTA is temporarily unavailable, Exim will try again. And again. And again. And then give up. And our PingMe messaging dispatchers never have to worry about this. The E-mail and SMS handlers simply turn the messages over to Exim on time and wash their hands of the matter.
While most of PingMe’s outbound messages are delivered via e-mail, a large portion go out over Twitter. Without beating a dead horse, and while acknowledging that their reliability has improved quite a bit, Twitter is not like our local MTA, it’s just not as reliable and as a remote HTTP service, not nearly as fast. On the other hand, once in a while our MTA might be down (perhaps I bork the config file and it doesn’t come back from a restart). More importantly, there is no mechanism in place for handling failure. When you send a message to the Twitter API, it either works out or you get a failure. And if you don’t handle that failure, you fail, too!
We handled this by implementing a retry system for our dispatchers. We caused a number of exceptions to bubble up in our test environment, everything from inability to connect to twitter to no network at all, and began catching the exceptions and wrapping them as DeliveryExceptions. If Twitter (or our MTA) is down, the message instance is delayed by a few minutes and marked for retry. We’ll retry numerous times before giving up (there comes a point at which a time-based message loses its relevance…).
Just a little peaking into our messaging code:
rescue DeliveryException => e
@log.error "Caught delivery exception, marking event for retry."
retry_event(event)
...
def retry_event(event)
event.status = Event::STATUS_RETRY
event.retry_count += 1 # up the retry count
event.retry_at = event.dt_when + (5.minutes * event.retry_count)
...
def lock_a_block(type_name)
before = (Time.now.utc).to_s(:db)
ActiveRecord::Base.connection.execute(
<<-END_OF_SQL
UPDATE events SET dispatcher = '#{@name}'
WHERE id IN (
SELECT e.id FROM
(( events e INNER JOIN targets t ON e.target_id = t.id )
INNER JOIN pings p ON e.ping_id = p.id)
INNER JOIN target_types tt ON t.target_type_id = tt.id
WHERE
tt.const = '#{type_name}'
AND
(
(e.dt_when < '#{before}' AND e.status = '#{Event::STATUS_PENDING}')
OR
(e.retry_at < '#{before}' AND e.status = '#{Event::STATUS_RETRY}')
)
...
The code actually gets quite a bit more complicated than that, and I don’t really want to go fully dissecting the polymorphic message handlers we’ve written, but it shows you how handling failure isn’t really an outlier problem, it becomes core to your system. It’s just as important as returning those nice model validation errors that Rails makes so convenient for you.
Another technique we use in PingMe is pipeline prevention. Well, that’s what I call it. But basically you can’t have one Twitter-bound ping holding up every other outbound ping at 5pm EST! We spent a lot of time implementing a system that allows for many concurrent dispatcher daemons, and all Twitter-bound pings go through only two of them, preventing the others from being affected by the high latency when connecting to Twitter. We ended up using the mutex pattern with Postgres:
def acquire_mutex
ActiveRecord::Base.connection.execute(
<<-END_OF_SQL
LOCK mutex IN ACCESS EXCLUSIVE MODE;
END_OF_SQL
)
end
In our time-tracking app Tempo, we allow users to send time entries and start timers by sending messages to our Twitter account (twitter.com/keeptempo), and we have a daemon checking the API for new direct messages every couple of minutes.
Two things have to happen for that to work over direct messaging – both accounts have to be “following” each other. So the user follows us on Twitter, then enters their Twitter ID on their Tempo profile. Tempo does a quick check to make sure you’re following ‘keeptempo’, and then attempts to follow you. Either of those connections to the Twitter API can and often do fail.
So what do we do? We put together a rake task that generates a list of twitter ids on our user’s profiles that we aren’t following, and sends a follow request for each of them. We run it as a periodically and it catches quite a few. Not perfect, but just about the best we can do. It’s better than letting users walk away thinking that it doesn’t work at all! In that case you just look bad, and it’s not even your fault!
But it is your fault, actually, because you have to code for failure, or you look pretty bad when the exceptions bubble up to the surface, literally. Or, worse, you present the user with inaccurate information based on an exception state you didn’t plan for, which can really put you in a bad light.
I stay positive, but I code for failure ;-)
Listen, we need to talk. It’s about your Earthlink spam blocker. The one that does this:
I apologize for this automatic reply to your email.
To control spam, I now allow incoming messages only from senders I have approved beforehand.
If you would like to be added to my list of approved senders, please fill out the short request form (see link below). Once I approve you, I will receive your original message in my inbox. You do not need to resend your message. I apologize for this one-time inconvenience.
Click the link below to fill out the request:
I realize that this has its uses and I’m sure it cuts down on the amount of spam you get.
But, there are better ways. Much, much better ways. And if you’ve been wondering why you never get your activation e-mails for things that you sign up for, that’s pretty much why. You wouldn’t believe how many of these responses we get every day for PingMe and Tempo. We always try to do you the favor and click the links when we have time, but lately there are so many (now requiring us to fill out a form), that we can’t really keep up.
Do everyone a favor, turn that thing off. Cut down on the amount of mail flying around the internet.
Giles is up there doing the Archeopteryx Ruby MIDI generator talk. Best quote so far: “most people don’t use the freedom they have.”
I can’t describe to you how rapid, funny, inspiring and informative his talk is. Maybe I can: we’re all starving for dim sum lunch and are willingly sticking it out to the end of his talk. It’s that awesome.
Hampton Catlin’s on, he’s talking about replacing Javascript, basically. Well, there’s more to it than that. But he says, “I find when programmers think an idea is a really bad idea and they can’t give you a good ****ing reason for it, then it’s a good idea!”
Guilty as charged.
Zed Shaw literally just rocked the house by playing into his own sequencing software and winging some lyrics, guitar and harmonica.
Zed Shaw is up next. He is pacing. He told me he’s especially looking forward to angering the musicians in the audience. And he has a harmonica. Lord help us.
I’m going to break up the Rubyfringe blogging for a minute to stump on a topic that’s close to our hearts here at Zetetic.
When it comes to building a new web application our designing decisions related to user accounts should be driven by a common paradigm: a user object should represent a real life human being. That user object can then be associated with any number of other entities. Fun! Flexible!
You can see this at play in our time-tracking app Tempo: every user has only one account, and can be associated with any number of other people’s projects. It’s a simple many-to-many relationship, a social network relationship really, and it seems like an obvious and straight-forward design decision — but that’s not how a lot of web applications work.
Many really good systems out there have made the decision to build their web applications around companies or organization and not people. From there, people are users in the company (as in “assets” or “resources”), and are walled off from other people in other companies using the same web application. In some cases they even log into different sites! This can require a real flesh-and-blood human being to have multiple accounts, one for each company they work with that uses the system.
This paradigm can provide some conveniences (although none of them unique to it other than the feature of company-as-corral), but it can cause issues in the real world when flexibility is severely limited by the “company” wall. If Tempo followed this paradigm, our users, many of whom work with many different companies, client, and subcontractors, would be at a real disadvantage having to maintain numerous accounts and log into different sites just to summarize their full monthly billing!
Many folks who are independent subcontractors or consultants love to use Tempo because it allows them to work with many customers and partners without having separate logins. They can log in and report on all their time from one place.
Social networking applications, however cliché it is to laud their innovation nowadays, are really paving the way when it comes to recognizing that organizations and companies are just groups of people. They are defined by relationships between people, not a fixed hierarchical structure where the organization is the center of the universe and people are bolted on ad hoc (or grown like leaves on the company tree)!
Consider how useful it would be to add groupings or private channels to an asynchronous messaging system like Twitter. Everyone on Twitter has their own account. Allowing people to join groups would be easily implemented (scalability issues aside). Now imagine Twitter being based on organizations primarily, and having to have one account for each organization. Ugly, right? And unnecessary.
So far the overwhelming blowout talk here was Nick Sieger’s of Sun Microsystems. He’s got the whole thing over here on his blog, with a complete discography of the great music he played.
While listening to this great primer on the history of Jazz and discussion of core concepts, some ideas collided in the back of my brain. I’m pretty familiar with a lot of it, but it’s been a while and I’ve been stuck in some of the same genre/idea ruts. I started thinking about some things I’ve been working on for my band, things I want to try, some of the concepts Nick was discussing and illustrating by playing us music and I suddenly had an explosion in my head and a near-complete song idea! I actually drew out a few bass clef staves and wrote some sheet music notes real quick to try when I get back home (or in a music shop on Queen St. on the way to the after party.).
Nick, in half an hour you just made up every single JavaOne conference I attended ;-)
A major theme, concept, or understanding here at “Ruby Fringe”http://www.rubyfringe.com amongst folks seems to be owning what you do. Not that it’s expressed this way – but I think a lot of what we’re discussing here comes down to owning your work, owning how you spend your time, owning your mistakes and growing from there. From Fail Camp to discussion on sales and business relationships to networking and how you spend your time, to the slogan on the back of the conference tags, “Ask me about my startup!”, this is about owning what you do and consciously running with it.
Quite frankly, it’s exhilarating. This is not a normal programming conference.
First two talks were great (Jay Philips on Adhearsion, Dan Grigsby on hacking the market, basically). While on different topics altogether, both were plugging a common theme here: keep coming up with new ideas, set out on your own, and if you look for the niches and cracks you can fill in the market, you can exploit/hack those efficiencies to your advantage and really make it as an indie with a few friends and not work for the man, so to speak.
Which we already knew (or at least hope) is the case, but it’s awesome having these folks sharing their experiences, ideas, and methods of doing it themselves. It’s basically a lot of people who have that kind of “hack everything” mentality that is the focus of the HOPE/2600 folks and apply it to the question, “how can I set out for myself as a hacker and make a living?”
Also: the memcached talk was a totally excellent geek-out on caching.
Next talk is called “Living on the Edge,” and it’s about what’s going on in Merb edge development, but you see how it fits the broader theme by title.
There’s a great community here in Toronto, looking forward to meeting some New Yorkers and seeing if we have this kind of community back in NYC and Jerz.
More to come.
Stephen and I are in Toronto for the RubyFringe conference! The opening party at Amsterdam brewery was a really good time, but things really started with Fail Camp, hosted by Joey Devilla over at the Rhino. It was awesome. The level of friendliness and communication and idea brewing is very high. It’s like the humid, thick air is caffeinated.
More coherent thoughts and notes to come as the day goes on and the talks on the track progress! Stay tuned.
Update: You can find me on Facebook here, and Stephen here, any of you looking for us.
Web application frameworks like RubyOnRails and Oracle’s Application Express (APEX) are generally built as full suites providing the most common functions of web frameworks in a way that’s extensible so that you don’t have to roll your own. They are incredible products, both great for what they do. But in the real world, many businesses have widely diverse data sources in their environments, and sometimes you’ll have to mix and match.
Case in point: in one of our clients’ environments there’s an instance of APEX providing data reporting and there’s a separate web app for manipulating directory services; both are protected by a single-sign-on solution. If you log into one, you have the cookie that allows you into the other without having to log in again. A recent change requested of the web-app was that a particular text field be given type-ahead suggest / auto-complete functionality for a particular piece of data stored in the Oracle DB schema atop which the APEX app sits.
At first I looked up a number of the various posted ways on how to do an auto-complete in APEX alone, just to get the ball rolling and see what functionality was there. There’s a lot of good information out there in the Application Express section or the OTN forums. I figured at first that I ought to be able to code up the server-side processing in APEX using an On Demand process (basically a bit of PL/SQL) to respond to the very simple, one-parameter request of a type ahead. I figured on the client side I’d just use Ajax.Autocompleter from the script.aculo.us library to keep things simple for the web-app, too.
This ended up tanking due to how APEX handles requests through mod_plsql. To run an On-Demand application process the calling request has to contain a valid session ID number in the query string. Not to mention I’d have to hack up Autocompleter a bit to supply the param name and value in APEX’s tortured query string syntax (/pls/f?p:page_id:session_id::::param,param:value,value:etcetcetc).
But I didn’t shy away from the task, I set about trying to figure out how I could first grab a session ID, import it into the client side javascript and use that to dynamically construct the URL and params. It was ugly, and I never did get that session ID cleanly.
But as you can see in that thread, Louis-Guillaume suggested a much simpler solution: since we’re using mod_plsql anyway (APEX runs inside it), why not just go old school and create a stored procedure to respond to the Autocompleter call?
It’s basically the same code as the On-Demand process. You create your stored procedure, set up a DAD connection string for mod_plsql (this is done in marvel.conf or dads.conf in Oracle HTTP Server (Apache!)), and call it like thus:
http://example.com/
The parameters should literally match what’s defined for the parameters on the PL/SQL procedure. Leaving some off is fine (as long as you set defaults and account for the possibility), but adding extra undeclared parameters causes a problem, as we’ll see in a moment.
So here’s the stored procedure:
CREATE OR REPLACE PROCEDURE autocomplete_for_empname( p_search_text IN context.val%TYPE DEFAULT NULL )
AS
v_employee_row employees%ROWTYPE;
v_count NUMBER := 0;
BEGIN
OWA_UTIL.mime_header ('text/html', FALSE);
HTP.p ('Cache-Control: no-cache');
HTP.p ('Pragma: no-cache');
OWA_UTIL.http_header_close;
IF p_search_text IS NOT NULL THEN
HTP.prn ('<ul>');
FOR v_context_row IN
(SELECT e.* FROM employees e WHERE LOWER(e.name) LIKE '%' || LOWER( p_search_text ) || '%' AND ROWNUM < 11 ORDER BY c.val)
LOOP
v_count := v_count + 1;
HTP.prn ('<li>' || v_employee_row.val || '</li>');
END LOOP;
END IF;
IF v_count = 0 THEN
HTP.prn ( '<li>No results found</li>' );
END IF;
HTP.prn ('</ul>');
EXCEPTION WHEN OTHERS THEN
HTP.p('Error: ' || SQLERRM);
END autocomplete_for_empname;
What this code does is provide the unordered list expected by Ajax.Autocompleter. We can call the stored proc like this to test the results:
http://example.com/
And we’ll get results like:
<ul> <li>Jonathan Myers</li> <li>Jones Johnson</li> <li>Jon From Garfield</li> ... &c. ... </ul>
The constructor for the Autocompleter was an easy setup:
<input type="text" id="autocomplete" name="p_search_text" size="50" />
<div id="autocomplete_choices" class="autocomplete"></div>
<script type="text/javascript">
new Ajax.Autocompleter("autocomplete", "autocomplete_choices", "http://example.com/our_dad_name/autocomplete_for_empname", {
paramName: "p_search_text",
minChars: 3,
method: 'get'
});
</script>
There was one final hang-up. Even though I was getting the correct responses by hitting autocomplete_for_empname directly when testing from my browser, Ajax.Autocompleter was getting a 404 response. I took a look at the request being sent in Firebug, and it turns out that with every Ajax request, Prototype, the supporting library, adds an extra ‘_’ parameter with no value.
I had to open up the prototype.js library and comment out this line in the Ajax.request method:
request: function(url) {
var parameters = this.options.parameters || '';
// if (parameters.length > 0) parameters += '&_=';
Then, all was right with the world. Turns out the extra param is included to get around an old bug in Safari 2.x#.
So there you have it. If you want to do an external autocomplete against data in your APEX schema, use Ajax.Autocompleter and a stored procedure over mod_plsql.
An article on how to create full web apps using OWA and mod_plsql on Oracle from 11 years ago. Brutal!
You kids don’t know how good you’ve got it.
My friend Matt Jones is one of the design leads on Dopplr, “an online tool for business travelers.” It’s a rather slick utility for anybody who does a lot of traveling and wants to share that information with others, connect up in various cities, track their travels, etc. That’s not all it does, but those are the very basics of it.
Anyway, Matt has an excited post out on their blog about some new features they just added – mainly the ability for you to not only send in your trip information via E-mail, SMS, or Twitter, but you can literally forward eTickets and various other itinerary confirmations from airlines and bookers straight to Dopplr, and have it enter all the trip info for you. Check out the slick video demonstration!
This is a huge ease-of-use improvement to their already great service, cutting down on double entry and making manual entry much easier. If you’re a reader of our site, or you use our tools, you’ll know we’re total geeks for streaming information into tools and using command-line interfaces. Nice job, fellas!
