In an attempt to stay a step ahead of the bad guys, researchers from North Carolina State University and IBM have addressed a still relatively rare kind of malware threat that, if allowed to take hold, could affect many thousands of cloud users in a single attack. They have developed a program called HyperSentry, which can monitor the security of the hypervisor processes that control the virtual operating systems images in which user applications run, and do it in a way that can avoid detection and evasion, even by extremely sophisticated malware.

With the growing popularity of cloud computing, security concerns have grown along with it. But, to date, most documented security breaches and malware attacks in the cloud have been of the same types found in conventional stand-alone server contexts, such as Distributed Denial of Service (DDoS) on web sites and Trojan Horse viruses in applications. A discussion of such cloud security threats can be found in the recent article, SMB Cloud Is A Hacker’s Paradise.

However, so far, real-world attacks of a cloud-specific nature have been comparatively rare, owing in large part to the nature and role of the hypervisor.

Hypervisors, like Xen and KVM, are relatively simple programs that make cloud computing possible by managing the operation of many individual instances of various operating systems in virtual machines on a single physical server computer. Each virtual machine simultaneously supports many different applications, each of which potentially can support thousands of end-users. The hypervisor runs as a process under the physical server’s “host” operating system, and it in turn controls the multiple “guest” operating system instances.

The hypervisor is thus operating at the highest level of privilege and protection on the physical server, making it harder for a hacker to get at it than the many other less-secure programs in the system. It may be harder, but it is not impossible. And, it is easy to see how an attack on a single hypervisor can threaten the security of a large number of cloud users.

“The concern is that an attacker could compromise a hypervisor, giving them control of the cloud,” says Dr. Peng Ning, professor of computer science at NC State and co-author of a paper describing the research. If a hypervisor is compromised, the attacker could do almost anything: access users’ sensitive information; use the cloud’s computing resources to attack other Internet entities; spread malware; etc.

“HyperSentry solves two problems,” Ning said. “It measures hypervisor integrity in a stealthy way, and it does so in the context of the hypervisor.” The context is the hypervisor’s program memory registers inside the CPU which can be manipulated by malware to prevent detection by conventional security software; by operating directly within the hypervisor’s context, HyperSentry can immediately detect such manipulation. Also, if a compromised hypervisor is aware that it is being watched, the malware can restore the hypervisor to its normal state until the security check has ended, and then resume its wicked ways; by operating “out of band” from the hypervisor, HyperSentry is stealthy and can catch the malware unaware. Once HyperSentry detects a compromised hypervisor, it will notify a cloud administrator, who can take action to respond to the compromise and limit its impact on the cloud.

HyperSentry is currently in the form of a demonstration prototype that its creators expect to release as open source software in the near future. It is also being looked at by Red Hat and IBM for inclusion in future system management facilities. Such capabilities are especially important to IBM now, with the recent release of its “system of systems”, discussed at length in the article Here It Is, Your Moment of zEnterprise.

You can read more about HyperSentry and the research behind it in the paper that will be presented Oct. 5 at the 17th ACM Conference on Computer and Communications Security in Chicago, Ill.