 |
| Photo of eight-foot tall prints, with detail. Click for larger view. |
We were recently given the task of producing a series of animations and high resolution still images based on very dense 3D scan data from an archaeological dig site. The site included four graves and their remains along with an artifact. The data from the site came to us in several pieces of varying and overlapping resolutions and source image quality. Surface textures were based on photogrammetry while the geometry was produced using multiple forms of 3D scanning.
The biggest challenge to this project was the efficient management of the large amount of data involved. By the time this project was done, we had mostly filled a dedicated terabyte drive with its resources. The geometry consisted of eight data sets, most of them containing more than ten million polygons each. The majority of texture maps were sixteen thousand pixels square. Even with a very fast local network, plenty of Solid State Drive space, 64gigs of RAM and dual pro video cards in our graphics workstations, careful scene management was necessary.
The source data consisted of a digital elevation map and aerial photography of the surrounding countryside, a point-cloud scan covering a few acres, 3D scans converted to polygonal geometry for the dig site along with separate, higher resolution scans of the interiors of each grave as well as scans of a reliquary found in one of the graves, both before and after restoration. Each piece of geometry had an associated high resolution surface texture generated through photogrammetry which were then manually cleaned and optimized. The fully assembled scene totaled over nine gigabytes of geometry and texture data.
This may seem excessive, but we were dealing with scientifically accurate and historically significant data. The client wanted the resulting imagery to be as detailed and accurate as possible. The animations would go from aerial photography a thousand feet up, all the way down to within inches of human remains and artifacts. Still renders of entire individual graves would be printed life sized and would be scrutinized from inches away. We wanted to keep as much detail as possible, showing grains of sand, twigs, tiny cracks, along with conveying an understanding of layout of the site as a whole.
We used Autodesk Maya as our primary animation tool for this job, along with Cinema 4D for the aerial photography element and DEM manipulation. Rendering was done with a mix of Autodesk’s Mental Ray and Chaos Group’s V-Ray on our render farm. The farm is a hybrid of in-house OS X machines along with a dynamically scaling cloud based system that runs a combination of Linux and Windows, depending on the required tool. The cloud farm is very small when idle, but can quickly scale up on demand. Our animation team is fortunate in having another team in our company that just happens to specialize in custom cloud computing solutions, networking and vast data storage and manipulation. They made this setup possible. Thanks guys!
One of the problems we had not anticipated was how quickly even our fast network would become saturated when many render machines come on line, simultaneously requesting such large amounts of data. This seriously affects render efficiency when the bulk of the farm is figuratively twiddling it’s thumbs, waiting for the data it needs to get started. The dynamic scaling of our cloud farm is based on how busy the processors of the machines are over a given time period. This lead to a yo-yo effect where machines would be added to the farm as the base machines got busy, then pruned out of the farm because they weren’t actually doing anything as they waited for all the data to come in, then added back when the base machines showed they needed help, etc.
One of the things we did to improve this situation was build an auto synchronizing mirror of our local render server in the same cloud space where the render machines reside. This did introduce a bit of a delay as local data synced to the cloud render server before a render could begin, but it eliminated a significant part of the bottleneck. As segments of a render are completed they are written first to the cloud server, then mirrored back to our local network. Our render farm management tool includes options to throttle the number of machines that can request data at once, but that reduces the speed and efficiency to some extent. A new, higher speed cloud storage system is in testing now, and should completely eliminate the problem once it is more widely available.
 |
| Highest vs. Lowest resolution geometry. Click for larger view. |
On the actual animation and render side of things, we built multiple resolution sets of data, from full resolution down to about 1/10 resolution, along with intermediate versions as necessary. The scene was built with simple reference objects that could be easily swapped in and out at different resolutions as required. This made the interactive process of lighting and animation go smoothly, using only low resolution geometry and textures for initial setup. For the most part, the highest resolutions were not necessary for the animation sequences, except for when the camera got very close to the surfaces. This also made it possible to send jobs to the render farm in smaller parts, reducing the amount of data sent back and forth and increasing efficiency. For animation, it’s a fairly straight forward process to send sets of individual frames to each machine on the farm for rendering. For the very large still images, each image was broken into 256 tiles and those tiles were individually sent to render on different machines on the farm. After all the tiles are rendered, a separate process assembles them into the final composite image.
The extremely large size of some of the print renders pushed the limits of commercial digital image creation and editing tools. Usually when printing a wall sized image it is intended to be viewed from at least several feet away, so the amount of information per square inch can be relatively low. In this case, however, we wanted viewers to be able to walk right up to the prints, look closely from inches away and see all the details as if they were looking at the actual object. Standard file formats like Photoshop’s .psd and the tried and true .tif were not capable of holding all the information in these images. Fortunately Adobe has a less common format, the .psb or Large Document Format that can hold many times the information as the standard Photoshop document. In addition to Adobe’s format, we used the open source OpenEXR file type for animation output. This is a format designed for digital effects production by Industrial Light and Magic and updated in 2013 in conjunction with WETA Digital. It can handle multilayered, high dynamic range images with and without compression and with practically limitless resolution, along with lots of other useful image information. There are plugins available that allow most professional editing tools to read and write OpenEXR.
Lastly, there’s the physical side of all of this. In our current, relatively high-speed internet connected world, we have gotten used to sending information nearly instantly with the click of a mouse, even for large images, videos and animations. That was not possible for this project. Test images and sample renders could be sent for review electronically, but for final delivery the only practical solution was to physically hand deliver a terabyte hard drive.
Ultimately, the client was delighted with the final imagery and animation. Their research announcement got wide distribution and high visibility around the world through many media outlets.
The client attributed at least some of that success to having compelling imagery to go along with the story. This project pushed our technical abilities to new levels, with very satisfying results.