At FOSSCamp in October, skilled eye-candy expert Mirco Müller (also known as MacSlow) hosted a session about using OpenGL in GTK to bring richer user interfaces to desktop Linux applications. Building on the technologies that he presented at FOSSCamp, Müller recently published a blog entry that demonstrates his latest impressive experiments with OpenGL, GTK, and offscreen rendering.
Müller is currently developing the GDM Face Browser, a new login manager for Ubuntu that will include enhanced visual effects and smoothly animated transitions. To implement the face browser, he will need to be able to seamlessly combine OpenGL and conventional GTK widgets. Existing canvas libraries like Pigment and Clutter are certainly robust options for OpenGL development, but they do not offer support for the kind of integration that he envisions.
The solution, says Müller, is to use offscreen rendering and the texture_from_pixmap OpenGL extension. In his experimental demo programs, he loads GTK widgets from Glade user interface files, renders them into offscreen pixmaps, and then uses texture_from_pixmap to display the rendered widgets in a GTK/GLExt drawing area, where they can be transformed and manipulated with OpenGL. Müller has created several demo videos that show this technique can be used to apply animated transitions and add reflections to widgets. The visual effects implemented by Müller with GTK and OpenGL do not require the presence of a compositing window manager.
We talked to to Müller to get some additional insight into the challenges and benefits of incorporating OpenGL into desktop applications. One of the major deficiencies of the current approach, he says, is that users will not be able to interact with GTK widgets while they are being animated with OpenGL—a limitation that stems from lack of support for input redirection at the level of the toolkit.
"Interaction will only be possible at the final/original position of the widget," Müller told us, "since gtk+ has no knowledge of the animation/transformation taking place. I consider it to be much work to get clean input-redirection working in gtk+. There might be some ways to achieve it using hacks or work-arounds, but that should be avoided."
Eye candy or UI improvement?
Although some critics might be inclined to prematurely deride Müller's work as indulgent eye-candy, he primarily envisions developers adopting OpenGL integration to tangibly improve the user experience by increasing usability. "I would like to see [OpenGL] being used in applications for transition effects," he says. "We can right now improve the visual clues for users. By that I mean the UI could better inform them what's going on. Widgets [shouldn't] just pop up or vanish in an instant, but gradually slide or fade in. These transitions don't have to take a lot of time. As a rule of thumb half a second would be sufficient."
In particular, Müller would like to experiment with adding animated transition effects to the GTK notebook and expander widgets. He also has some creative ideas for applying animations to the widget focus rectangle in order to make its movement more visible to the user. Müller also discusses some applications that would benefit from OpenGL-based transitions. In programs like the Totem video player, he says, the playback controls in fullscreen mode could slide in and out rather than just appearing and disappearing. Alluding to the webcam demo that he made for FOSSCamp, he also points out the potential for incorporating OpenGL effects into video chat programs like Ekiga. Müller has also long advocated using visual effects to create richer and more intuitive user interfaces for file and photo management software—ideas that he has put into practice with his brilliant LowFat image viewer.
"The kind of effects possible if you can render everything into a texture, map it to a rectangle or mesh and then do further manipulations with GL (or shaders) are next to limitless," says Müller. "Just look at what Compiz allows you to do to windows now. Imagine that on the widget-level."
We also asked Müller to explain the performance implications of using OpenGL in GTK applications. "The memory-overhead is directly linked to the size of the window, since all the rendering has to happen in an OpenGL-context filling the whole window. The bigger the window, the bigger the needed amount of video memory," Müller explains. "The load caused on the system is directly linked to the animation-refresh one chooses. 60Hz would be super smooth and very slick. But that's a bit of an overkill in most cases. One still gets good results from only 20Hz."
There are still some performance issues with the texture_from_pixmap that are actively being resolved. "Due to some issues in Xorg (or rather DRI) there are still too many copy-operations going on behind the scenes for GLX_EXT_texture_from_pixmap," says Müller. "There are also locking issues and a couple of other things. At the moment I cannot name them all with exact details. But more importantly is the fact that there's currently work being done—in the form of DRI2—by Kristian Hoegsberg (Red Hat) to fix these very issues on Xorg. I cannot stress enough how important this DRI2 work is!"
Although using OpenGL incurs additional memory consumption and system load, Müller says that the impact is small enough to make his current approach viable for projects like the Ubuntu Face Browser.
Although individual developers can use Müller's boiler-plate code to incorporate OpenGL integration into their own GTK programs, Müller suspects that support for this technique will not be included directly in GTK at any time in the near future, which will make it harder for developers to adopt. "Right now it is all happening in application-level code and not inside gtk+," Müller explains. "Since I use GLX_EXT_texture_from_pixmap to achieve efficient texture-mapping out of the wigets' pixmap it is currently a X11-only solution. Therefore I imagine they might want to see a more platform-independent solution to this first." The GTK offscreen rendering feature that Müller uses also currently lacks cross-platform support.
Despite the challenges and limitations, Müller's creative work opens the door for some profoundly intriguing user interface enhancements in GTK and GNOME applications. "There are more things possible than you think," says Müller in some concluding remarks for our readers. "Don't have doubts, embrace what's available. X11 and gtk+ are pretty flexible. People who just don't seem motivated to explore ideas and test their limits (or the limits of the framework), should remember that this is Open Source. It lives and thrives only when people step up and get involved. Just f*****g do it!"