mikhail panchenko / blog

WOW. Thanks CNN! We couldn’t have guessed it from the ELECTION we just had. Is an election not just a massive poll? Am I crazy? Or does some asswipe at CNN need a new job?

This is almost as good as that time they ran a story that basically said “According to the poll, how Americans feel on X issue depends on who you ask.”

No, seriously, code-related posts are going to make a comeback in a big way.

Ask for better reporting in the new system, then complain that the reports in the new system aren’t identical to the reports in the old system. That will just about do it.

The 2008 election has come to a conclusion, and we know that Barack Obama will be the next President. I don’t want to write too much about it, as this isn’t a political blog (let’s face the facts: I can’t even vote yet…) However, what is a blog for, if not to write something pompous on historic occasions.

I’d like to congratulate Barack Obama on getting what he deserves. He has run a classy, warm, inspiring campaign, and he has achieved something incredibly special. His victory is a symbol of so many things that I believe to be American. The amount of blue I see on the electoral map as more and more states come through for Obama reassures me that the values, the strength, and the prosperity that I saw diminish over the 10 years I have spent in this country have not gone away and have returned with renewed strength.

I’d like to commend John McCain on making a classy concession speech. I hope that his supporters honor his request and come together behind our new leader.

I’d like to express how optimistic and proud I feel tonight, and to remind everyone that the nature of a democratic government is that one side always has to lose. Being bitter about not having your candidate win an election is equivalent to being bitter about core principles of democracy.

I truly hope that Barack Obama can accomplish even a fraction of what he has outlined during his campaign.

Finally, I am just excited to have an articulate President. Finally, some real presidential addresses.

It’s been so long since I’ve made a website “for myself” that actually creating the content has become a huge chore. I’m so used to working on what Crowley so aptly calls the “plumbing” that creating a simple layout for a joke website and filling in some bullshit content is the thing I’m putting off - even though the “hard parts” have been done for weeks!

I had an argument with a co-worker the other day, in which his point was that “APC isn’t the solution to everything.” No shit. Not sure what the point of it all was, but what started it was my desire to CACHE MORE.

While caching seems to have become widely accepted by the Web 2.0 “I read Cal Henerson’s book, so I know how to scale” crowd (because Cal says to use it), the rest of the world has decided that it’s just not needed outside of that milieu.

I work in operations. My audience is limited by my company’s payroll, which is somewhere in the neighborhood of 15,000, of which only a fraction actually uses the shit I write (surprise! the HR department does not give a fuck about SLA misses). I don’t work (directly) on a social network. I’ve never built anything that generated crazy traffic. But I know that even our internal tools have gotten to the point where we have to load balance read queries to multiple slaves and our customers complain about load time.

Caching is a good idea

Let’s review what caching is: storing a chunk of already-processed data in shared memory, where it can be accessed via a simple key lookup. No processing, no querying, no magic sauce - very straight forward. What part of accessing data without ever hitting the persistent storage layer, never mind any sort of database, sounds like a bad fucking idea? I don’t care if your app never has more than 5 simultaneous users, it’s fucking common sense. I’ll also talk about how caching can drastically improve the user experience, even in a low traffic environment.

Caching in PHP - something about candy and babies

If, like me, you work on PHP, your life is sweet: you get to use memcached (what everybody else is using) AND a nifty little PECL extension called APC - Alternative PHP Cache. I won’t cover the specifics of how to use APC or memcache (there are search engines for that), but I will hold your hand like a small child and explain why, indeed, APC isn’t the solution to everything, and why most applications can benefit from using both.

APC is mostly known as an opcode cache - it caches the opcodes for your php files, so that your poor webserver doesn’t have to keep re-parsing your shitty code. That’s super nice, but that’s only half of APC. The other half is just a legit, in-memory key-value store with all the functions you’d expect: apc_store and apc_fetch. Can you guess how they work? (http://www.php.net/apc if you can’t…)

Using The Right Hammer

memcached and APC perform, in essence, the same function (there are obviously vast implementation differences, but we don’t care right now). The major difference is that APC lives inside PHP, hanging off the underside of apache (or lighty or nginx or what the fuck ever you hip kids use these days), while memcached is its own server. That difference is what determines what each one of these tools is best suited for.

Since APC runs as part of your webserver, things you store in it will only be accessible by other scripts running on that same physical server. The flipside is that you don’t have to waste a network roundtrip. Using APC for the things it works for to avoid the trip to the memcache server is one of those “premature optimizations” that isn’t “evil” because it’s so fucking easy.

As my co-worker so aptly pointed out, APC isn’t the answer to everything, and neither is memcached. You might have already figured out that each one has an appropriate use: APC is perfect for storing things that the webserver doesn’t need to share with anything else. memcache should be used for things that can be manipulated by multiple webservers. Here are some examples:

APC:

• configuration
• database-backed items used in generating frequently accessed parts of the site
• lists that get hammered in bursts (i.e. autocomplete lookups)

memcached:

• data structures containing data modified by our business logic (i.e., user data in the case of social networks or host data in the case of some sort of ops-y app like the shit I write)
• state/session info, so that your app knows what your user is up to no matter which app server they get bounced to

Remember that your cache shouldn’t contain any information you’re hoping to hold on to longterm. It’s just a proxy to help reduce load on your persistent storage and improve load time. The second case can be the exception, but there you have to make sure your cache is redundant so that a failing memcache instance doesn’t interrupt your user’s workflow.

Naturally, at the outset, when your “startup” is in its infancy and you have just the one EC2 “instance” to work with until that “Series A” hits, it’s fine to dump everything into APC. That’s why it’s important to write or adopt a cache abstraction early on, and create two “identical” APC adapters - one for storing things that REALLY belong in APC, and one for things you will eventually want to move to memcached. The very early, untested stages of just such an abstraction can be found here.

Examples from the land of Operations

Like I said, I’m not about to tell you how to use caching to fix twitter - I haven’t tried it. But I’ll tell you how I use caching in our internal apps to remove bottlenecks. I have to be pretty generic, but I’ll try to make the examples clear anyway. All my examples involve APC because the app I’m working on fits into the above category: it’s an internal app, so we get an app server, a db server and a “fuck you” for a memcached server.

Lists

One of the forms we have contains two drop downs. The first represents a set of categories, while the second represents their sub-categories. Naturally, the second list depends on the first, and needs to be regenerated every time the first one changes. Initially, I just generated the first list in raw markup on the PHP end, then used some XHR sauce to regenerate the second list asyncronously. Using ajax makes my e-manhood feel heavier.

Not that I shouldn’t have been caching this already (these categories and subcategories change once in a blue moon), but I was given an additional reminder: the data for these drop downs came from an API of another internal application, which, I shit you not, takes as long as 5 seconds to process a simple request. All of a sudden, my page was choking hard.

Enter APC. Every time a user hits the form, it checks APC first and loads the first list from there. If not, it actually queries the ass-slow API and then stores it for the next user. I then have a proxy function that queries the API for the second list. This proxy also does the APC dance, so only the first time a certain sub-list is loaded do we have time to make tea and bake a delicious pastry.

To further optimize page load, I ended up making the first list load asyncronously too; now even the user that arrives to a clean cache will see the whole page right away with a “loading” graphic in place of the drop down, while the external API eats its way out of its own shit.

Naturally, since I’m relying on an external system, I could get myself in minor trouble if the list gets updated, and my cache doesn’t know. To avoid this, I only cache for 5 minutes.

Autocomplete

In another form, we have an autocomplete field (thanks, YUI) which, once again, queries an internal API. This API isn’t as slow, but it’s slow enough to ruin the autocomplete “update as you type” experience. Once again, APC to the rescue. This is one of those cases where it actually ended up easier to do the “processing” application side, leaving the database (in this case, the other app’s database) out in the cold alltogether. Luckily, the external application had a method for fetching ALL of the potential items for this autocomplete. What we do now is just cache that full list on first “lookup” and parse out the relevant autocomplete results using PHP in our own code. It is FAST (granted the list is only a few hundred entries). It is definitely faster than waiting for a full API call at every keystroke.

Once again, since the data is external, I’m risking staleness. Just like above, I only cache it for a few minutes. Here in particular, we mostly care about not hitting the persistent storage at every keystroke. As long as it’s cached while the user is typing, we’re ok. The important thing to remember is that this isn’t just useful for slow external API’s. This should be done for your own calls as well - if you’re querying your database every time a user hits a key, you’re doing it wrong. If I have time during the QA phase (as if), I might actually add an XHR call to the page that pokes the API in the background and caches the results for a few minutes. This way the cache is ripe when the typing starts. With data from your own application, you can avoid staleness by adding some sauce to your data logic to invalidate the cache whenever that data is updated.

Hopefully I’ve convinced you to stop being such a pansy and learn how to cache properly.

Like I said before, I’m redoing the look. I’ve decided to start with the wordpress theme, as I’ve discovered it’s more of a bitch to go from a working design to making the wordpress markup play nice than the other way around. I’ll update the rest of the site when I have time/give a shit.

I was serioulsy fucking dismayed when I found out that http://www.ups.com/track is a busted ass looking 404 page. How the fuck, in 2008, do you not have a rewrite for that? FedEx, on the other hand, takes you straight to the appropriate page. Kudos to FedEx for actually paying attention.

I just have to wonder how many times a day that 404 is logged by UPS’s web server without anyone taking notice.