VFX pipeline notes

I was responsible for a pipeline research back at my previous workplace and here are some of the general insights I had from my research. There are some good resources out there on web when it comes to pipeline and there is this really great blog called 'The Art of CG Supervision', by Isa Alsup. Especially check out this collection of articles on studio pipeline on his blog, Pipeline Articles.

In the preliminary research itself I realized that everybody tends to think they have a pipeline but in reality they mix up their workflow with their pipeline. It is very important to note that pipeline is structural process of the whole project while workflow is the step by step process an artist undertakes to complete a particular task and pipeline often divides workflow into more meaningful sections using dataflows.

The term 'pipeline' is a computing term and the term was invented by Douglas McIlroy, one of the designers of the first Unix shells. According to Wikipedia, 'a pipeline is a set of data processing elements connected in series, so that the output of one element is the input of the next one.' This concept transcends well into the whole spectrum of visual effects production where there is always one shot completed by more than one artist or department.

A visual effects pipeline usually comprise of three elements and they are chiefly, Production, Data and Approval. One could also look at these three elements from a managerial perspective and then it could be divided into Technical Management, Information Management and Creative Management. The pipeline can be consists of diverse components and according Alsup, it mainly comprise of people, technology, methods and leadership. It is crucial to note that a digital asset management system is not a pipeline but it is only a dataflow centric element of an effective pipeline.

A effective pipeline is derived by nature of work, resources and the company ideology. The nature of the pipeline can be data centric or artist centric or even technique centric but according to Mayur Patel, author of the book 'The Digital Visual Effects Studio', an artist centric pipeline would be more effective because it gives more importance to the work artists perform rather than the data they generate and this sort of pipeline also tries to provides the artists optimum conidtions to unleash their creativity.

I think it could be questionable that why one should complicate the concept of an artist workflow and asset management system into such a management jargon. Moreover adopting such a system would add more managerial costs to the organisation. But an effective studio pipeline is targeted at long term goals such as streamlining diverse projects, quality control and interactivity at different stages, estimating project overhead, facilitating employment of certain technology or methods and embracing company culture in the process.

I think if you consider the above factors then you can look at the pipeline as a production facilitating element which directly relates to the profit motives of the business and you use it with control and style.

Following are list of resources on pipeline

www.animationpipeline.com
www.3dresearch.blogspot.com

Anatomy of Production Pipelines, CGtantra interview

Zoic Studios’ ZEUS: A VFX Pipeline for the 21st Century

Digital Visual Effects Pipeline and Techniques - Presentation on pipeline

The Production Pipeline, Webinar by Mayur Patel

VFX management, Scott Squries's blog post

Audio interview about PDI DreamWorks' Animation Pipeline

Andrew Whitehurst's article on VFX Pipeline
Pipelines: Plumbing for the next web

Render farm notes

This post is mainly for folks who wonder about how a basic render farm setup works in a animation or visual effects production pipeline. This is nowhere near a technically accurate post since most of it is took from my own personal research on setting up a render farm back at my previous workplace.

Most of render farm uses parallel rendering or distributed rendering for complex or huge amount rendering in the production. The distributed rendering basically breaks up the image sequence into chunks of individual frames and spreads it across the render machines for faster processing.

A very common way to go about distributed rendering is by server clustering, it is way of combining multiple computers or computer servers to perform a common task. There are different ways of clustering meant for different job types. Since our requirement is parallel rendering so the most effective way of clustering would be a parallel gird based cluster, in which dedicated master node (server) is the only computer which artist interact with to submit jobs. This machine then acts as the file server, render manager. This provides a single-system image to the user, who launches the jobs from the master node without ever logging into any worker nodes (other clustered computer under server).

There are quite a lot of render manager software available in the industry today which allows you to effectively set up a parallel cluster based render farm and following are to name a few, DrQueue, Qube, Royal Render, Deadline, Smedge, Muster etc.

Here is a basic flowchart depicting the setup I described above.

Also note that this setup is also commonly known as HPC (High performance computing) depending upon the scale of your farm. It is ironic the main advantage of this setup is not mainly speed, though it is considerably a large factor. But don't expect your frame to be rendered within a split second after your mouse click. It is rather an obnoxious process which will really require some good patience and technical expertise. Other main advantage would be the ability to queue the render jobs and this opens up lot of possibilities like for example, queuing multiple tests version for a shot for a overnight rendering thus increasing your productivity.

Cloud computing and GPU render farm bandwagon.

If you been researching on render farms lately then these are two terms which will hit you again and again and it is important not to get mixed up with both of these technologies since they are not really corresponding to each other.

Cloud computing is more a like remote service thorough web or virtual private networks (VPLAN) where you can use the service provider's hardware for your rendering purpose. And this it makes perfect sense to visual effects production houses who might need to increase the render farm capacity on a particular show's requirement. But acquiring , lodging and maintaining even a one single worker node in a render farm involves lot of costs even the after project is over so cloud render farm suits the industry needs.

Though it sounds like a cost effective scalable solution but comes with its on issues like security, availability of custom scripts, plugins or assets at service provider's site and also remote management of render farm may prone to more network issues.

To understand better the scope of cloud computing, take a look this presentation

GPU rendering on the other hand is going to bring a paradigm shift in the industry where it may to replace most of current CPU based render farms. This doesn't mean the end of CPU based render farms since there still areas (simulation, dynamics and AI) where CPU will perform better than GPU.

Here is snippet took from article called 'Are you ready for GPU revolution?" by Joe at renderstream

To help you understand how GPU acceleration could speed up rendering, lets think of it in terms of bucket rendering. (Please keep in mind this analogy isn’t technically accurate) Most of you are familiar with bucket rendering since modern renderers use that method. As a renderer calculates and ultimately draws pixels, it does so in small portions, or buckets, of a predetermined size. For every number of cores you have in your machine, you will have an equal amount of buckets at render time. For example, a common workstation today will have 4 cores (also known as a quad core) thus you will see 4 buckets at render time. If you have a dual quad core machine you will see 8 buckets and so on…
Today’s GPUs have 240 cores and the next generation will have up to 512 cores. By the time GPU acceleration is available for rendering, there could be even more cores available on the GPU. So, you can start to see how a GPU can have a tremendous impact on rendering. With CPUs we see a bucket work on a small portion of the rendered frame and then move on to another region. With a GPU, the available buckets would essentially fill the entire rendered frame. All portions of the frame would be “worked on” at once allowing for near real time rendering.

It also worth check this a great article on how GPU render helped spped up Avator production. As they described in the article GPU rendering will definitely bring a huge difference in productivity of the artists.

So as you can see I think in the future GPUs will be effectively integrated into current CPU based render farms and it will have huge impact on the render times and the way artists works.

Till then, happy rendering ;)

Porn drives technology and its goes stereoscopic!

They say porn industry drives technology and it didn't took much time for them to join the new millennium's stereoscopic 3D bandwagon. According to a guardin article, in nutshell here is what porn has done for media industry so far.

· Camcorder and VHS video machines were pioneered by porn barons anxious to find a cheap way to mass market blue movies. Take-up of DVD players was driven by pornographers and their customers because the technology enabled users to skip to and from their favourite scenes.

· Pay-per-view cable or satellite TV movies entered the market only after porn firms introduced 'premium' services in hotels and on digital networks. Interactive television, common on digital sport channels, was developed by pornographers to allow users to focus on favourite actors and actresses.

· Internet use and e-business have been driven by smut. There are 80,000 adult websites, which generate annual profits of more than £1 billion - more than any other e-commerce sector.

Compositing showreel 2010

You may also download shot breakdown and I have also designed a single webpage placeholder for my reel, resume and shot breakdown, check it out!

I managed to finish this cut quite quickly in couple of days time, so I am pretty happy about how it turned out. I hope my next show reel will have more high end feature film compositing and matte painting shots.

Most of the work in this show reel was done during my time at apostrophe films. So I thank everyone over there especially my senior artist Faisal and other colleagues in VFX dept.

Houseboat Matte painting

I think I am almost done with this one, I was trying to create a surreal and simple composition which looks convincing.

References Images

WIP1

WIP2

Comparison to original plate

Final matte painting

Nic Marks: The Happy Planet Index

I have already posted two videos (Link1, Link2) on Happiness culture and here is another one from TED which is worth bookmarking here.

Hope these ideas will really influence our work place and society in future.

Stay Happy :)

Pre Viz to Post Viz

Check out these cool videos from Previsualization Society and they plan to do more on videos on the same so keep an eye on their updates on vimeo. Thanks to my friend Zameer for sharing this stuff on FB.

Previs Documentary - Part 1 from Previsualization Society on Vimeo.

Heathentown Pitch Reel from Previsualization Society on Vimeo.

Heathentown Making Of reel from Previsualization Society on Vimeo.

There is cool article on which some light on difference between Previz and Postviz at digitalcontentproducer.com, do check it out. And here is a great fxguide article on Previz studio Third floor which was born of Lucasfilm Skywalker Ranch and following is a profile video about them by Autodesk.

I think coolest thing is that Virtual Cinematography and GPU rendering is closely linked to Previz to Postviz . Both of these stuff are current hot trends which is driving this dynamic industry and soon it will change the way we work.

Here is amazing talk by Nvidia on GPU technology, do check it out, It's really awesome!!!

On the following video you could really see what they mean by real time rendering.