One approach is a technique used by the VILO Malware Defense Suite project. Started by a few researchers at the University of Louisiana, the Center for Advanced Computer Studies project has analyzed thousands of server-side polymorphic malware programs. They take each one-off and normalize the coding. They take what is essentially pseudo-random subroutines and names and pull out the patterns. Because most server-side polymorphic code is based on a very small subset of malware families, the researchers can essentially convert the random-looking code back into the original family member.
On the VILO Web site, a user can submit a malware sample to be analyzed. The incoming sample is compared to a large database of previously collected malware samples that have been normalized and analyzed. The commonalities and differences are noted and a match is made. The VILO Web site will tell you how likely the submitted sample is to previously analyzed and defined malware. Submitters can even compare assembly language binaries against each other, instruction to instruction, to see the comparison themselves.
All of today's popular anti-virus vendors are using similar techniques to recognize server-side polymorphic code, among their quiver of technologies. I like the VILO site because I can submit my own found samples that aren't readily identified by anti-virus software programs, and then compare the coding myself.
If you're into learning more about malware and anti-malware defenses in detail, I highly recommend the Virus Bulletin newsletter. Although a subscription can be expensive (starting at US$175 per year), it's well worth the money. It's probably one of the most respected industry journals, and each month its 20 to 30 pages are chock full of useful information. Each issue contains two or three major articles, an opinion piece, good technical reviews of malware and anti-malware programs, and a solid anti-spam supplement. Most articles are written by highly skilled (and highly opinionated) anti-malware defenders. It's been around for many years, and I look forward to every issue.