Thursday, December 2, 2010

VFX pipeline notes


I was responsible for a pipeline research back at my previous workplace and here are some of the general insights I had from my research. There are some good resources out there on web when it comes to pipeline and there is this really great blog called 'The Art of CG Supervision', by Isa Alsup. Especially check out this collection of articles on studio pipeline on his blog, Pipeline Articles.

In the preliminary research itself I realized that everybody tends to think they have a pipeline but in reality they mix up their workflow with their pipeline. It is very important to note that pipeline is structural process of the whole project while workflow is the step by step process an artist undertakes to complete a particular task and pipeline often divides workflow into more meaningful sections using dataflows.

The term 'pipeline' is a computing term and the term was invented by Douglas McIlroy, one of the designers of the first Unix shells. According to Wikipedia, 'a pipeline is a set of data processing elements connected in series, so that the output of one element is the input of the next one.' This concept transcends well into the whole spectrum of visual effects production where there is always one shot completed by more than one artist or department.

A visual effects pipeline usually comprise of three elements and they are chiefly, Production, Data and Approval. One could also look at these three elements from a managerial perspective and then it could be divided into Technical Management, Information Management and Creative Management. The pipeline can be consists of diverse components and according Alsup, it mainly comprise of people, technology, methods and leadership. It is crucial to note that a digital asset management system is not a pipeline but it is only a dataflow centric element of an effective pipeline.

A effective pipeline is derived by nature of work, resources and the company ideology. The nature of the pipeline can be data centric or artist centric or even technique centric but according to Mayur Patel, author of the book 'The Digital Visual Effects Studio', an artist centric pipeline would be more effective because it gives more importance to the work artists perform rather than the data they generate and this sort of pipeline also tries to provides the artists optimum conidtions to unleash their creativity.

I think it could be questionable that why one should complicate the concept of an artist workflow and asset management system into such a management jargon. Moreover adopting such a system would add more managerial costs to the organisation. But an effective studio pipeline is targeted at long term goals such as streamlining diverse projects, quality control and interactivity at different stages, estimating project overhead, facilitating employment of certain technology or methods and embracing company culture in the process.

I think if you consider the above factors then you can look at the pipeline as a production facilitating element which directly relates to the profit motives of the business and you use it with control and style.

Following are list of resources on pipeline

Sunday, October 31, 2010

Render farm notes

This post is mainly for folks who wonder about how a basic render farm setup works in a animation or visual effects production pipeline. This is nowhere near a technically accurate post since most of it is took from my own personal research on setting up a render farm back at my previous workplace.

Most of render farm uses parallel rendering or distributed rendering for complex or huge amount rendering in the production. The distributed rendering basically breaks up the image sequence into chunks of individual frames and spreads it across the render machines for faster processing.

A very common way to go about distributed rendering is by server clustering, it is way of combining multiple computers or computer servers to perform a common task. There are different ways of clustering meant for different job types. Since our requirement is parallel rendering so the most effective way of clustering would be a parallel gird based cluster, in which dedicated master node (server) is the only computer which artist interact with to submit jobs. This machine then acts as the file server, render manager. This provides a single-system image to the user, who launches the jobs from the master node without ever logging into any worker nodes (other clustered computer under server).

There are quite a lot of render manager software available in the industry today which allows you to effectively set up a parallel cluster based render farm and following are to name a few, DrQueue, Qube, Royal Render, Deadline, Smedge, Muster etc.

Here is a basic flowchart depicting the setup I described above.
Also note that this setup is also commonly known as HPC (High performance computing) depending upon the scale of your farm. It is ironic the main advantage of this setup is not mainly speed, though it is considerably a large factor. But don't expect your frame to be rendered within a split second after your mouse click. It is rather an obnoxious process which will really require some good patience and technical expertise. Other main advantage would be the ability to queue the render jobs and this opens up lot of possibilities like for example, queuing multiple tests version for a shot for a overnight rendering thus increasing your productivity.

Cloud computing and GPU render farm bandwagon.

If you been researching on render farms lately then these are two terms which will hit you again and again and it is important not to get mixed up with both of these technologies since they are not really corresponding to each other.

Cloud computing is more a like remote service thorough web or virtual private networks (VPLAN) where you can use the service provider's hardware for your rendering purpose. And this it makes perfect sense to visual effects production houses who might need to increase the render farm capacity on a particular show's requirement. But acquiring , lodging and maintaining even a one single worker node in a render farm involves lot of costs even the after project is over so cloud render farm suits the industry needs.

Though it sounds like a cost effective scalable solution but comes with its on issues like security, availability of custom scripts, plugins or assets at service provider's site and also remote management of render farm may prone to more network issues.

To understand better the scope of cloud computing, take a look this presentation

GPU rendering on the other hand is going to bring a paradigm shift in the industry where it may to replace most of current CPU based render farms. This doesn't mean the end of CPU based render farms since there still areas (simulation, dynamics and AI) where CPU will perform better than GPU.



Here is snippet took from article called 'Are you ready for GPU revolution?" by Joe at renderstream
To help you understand how GPU acceleration could speed up rendering, lets think of it in terms of bucket rendering. (Please keep in mind this analogy isn’t technically accurate) Most of you are familiar with bucket rendering since modern renderers use that method. As a renderer calculates and ultimately draws pixels, it does so in small portions, or buckets, of a predetermined size. For every number of cores you have in your machine, you will have an equal amount of buckets at render time. For example, a common workstation today will have 4 cores (also known as a quad core) thus you will see 4 buckets at render time. If you have a dual quad core machine you will see 8 buckets and so on…
Today’s GPUs have 240 cores and the next generation will have up to 512 cores. By the time GPU acceleration is available for rendering, there could be even more cores available on the GPU. So, you can start to see how a GPU can have a tremendous impact on rendering. With CPUs we see a bucket work on a small portion of the rendered frame and then move on to another region. With a GPU, the available buckets would essentially fill the entire rendered frame. All portions of the frame would be “worked on” at once allowing for near real time rendering.
It also worth check this a great article on how GPU render helped spped up Avator production. As they described in the article GPU rendering will definitely bring a huge difference in productivity of the artists.

So as you can see I think in the future GPUs will be effectively integrated into current CPU based render farms and it will have huge impact on the render times and the way artists works.

Till then, happy rendering ;)


Monday, October 25, 2010

Porn drives technology and its goes stereoscopic!

They say porn industry drives technology and it didn't took much time for them to join the new millennium's stereoscopic 3D bandwagon. According to a guardin article, in nutshell here is what porn has done for media industry so far.

· Camcorder and VHS video machines were pioneered by porn barons anxious to find a cheap way to mass market blue movies. Take-up of DVD players was driven by pornographers and their customers because the technology enabled users to skip to and from their favourite scenes.

· Pay-per-view cable or satellite TV movies entered the market only after porn firms introduced 'premium' services in hotels and on digital networks. Interactive television, common on digital sport channels, was developed by pornographers to allow users to focus on favourite actors and actresses.

· Internet use and e-business have been driven by smut. There are 80,000 adult websites, which generate annual profits of more than £1 billion - more than any other e-commerce sector.



Thursday, October 21, 2010

Compositing showreel 2010

You may also download shot breakdown and I have also designed a single webpage placeholder for my reel, resume and shot breakdown, check it out!

I managed to finish this cut quite quickly in couple of days time, so I am pretty happy about how it turned out. I hope my next show reel will have more high end feature film compositing and matte painting shots.

Most of the work in this show reel was done during my time at apostrophe films. So I thank everyone over there especially my senior artist Faisal and other colleagues in VFX dept.

Friday, October 1, 2010

Houseboat Matte painting




I think I am almost done with this one, I was trying to create a surreal and simple composition which looks convincing.

References Images

WIP1
WIP2

Comparison to original plate

Final matte painting


Tuesday, September 21, 2010

Nic Marks: The Happy Planet Index

I have already posted two videos (Link1, Link2) on Happiness culture and here is another one from TED which is worth bookmarking here.

Hope these ideas will really influence our work place and society in future.

Stay Happy :)

Wednesday, August 11, 2010

Pre Viz to Post Viz

Check out these cool videos from Previsualization Society and they plan to do more on videos on the same so keep an eye on their updates on vimeo. Thanks to my friend Zameer for sharing this stuff on FB.





There is cool article on which some light on difference between Previz and Postviz at digitalcontentproducer.com, do check it out. And here is a great fxguide article on Previz studio Third floor which was born of Lucasfilm Skywalker Ranch and following is a profile video about them by Autodesk.



I think coolest thing is that Virtual Cinematography and GPU rendering is closely linked to Previz to Postviz . Both of these stuff are current hot trends which is driving this dynamic industry and soon it will change the way we work.

Here is amazing talk by Nvidia on GPU technology, do check it out, It's really awesome!!!


On the following video you could really see what they mean by real time rendering.

Friday, July 23, 2010

VFX Outsourcing Scare

I been participating and following in lot of discussions (link1, link2, link3) on this matter mainly in VFXtalk forum. The above first link is dated back to 2007 and rest two are recent links. There are quite a lot of discussions happening within the industry and including recent VFX TownHall meeting which grabbed lot of attention in general media.

The reason for this post is that I have an alternative thought on this matter and I think death of outsourcing to India and China can only do good for them in long term. As of now these two big countries are madly serving west, not only in VFX but also in other areas like IT and BPO.

If for some reason one day, this whole outsourcing business comes to an end and all these Indian and Chinese businesses started to look into their own potential and start to create market for themselves then I think the possibilities are limitless for them . We should never forget how much the population of India and China constitutes the entire world population and also both these economies are growly at very high rates with no signs of slowing down.

I feel both India and China are too comfortable with serving western markets that they don't realize the potential of their own markets. Bottom line is that this outsourcing business is not only doing bad for west but also for the east.

Anyway I feel this whole outsourcing business is a cyclical phenomena. By that, I mean soon Indian or Chinese companies would be outsourcing to Philippines, Vietnam or buying out US companies. We shouldn't forget what happened to American brands like Jaguar , Land Rover and Hummer which were bought by Indian and Chinese companies. Same could happen with entertainment business as well. Already Indian companies like Prime Focus and Reliance have strong presence in Hollywood

The actual reason for this scare is overlooked most of the times. If you really study the trends then its obvious that it is the state of the industry not the outsourcing which creates all this emotional responses. This has been addressed by industry pioneer, Scott Ross in two fxguide podcasts. He not only mentions issues but also the solutions in the latest podcast. The first podcast is definitely worth your time and it is one of the best podcast I ever heard at fxguide.com

As a final note, I think we all like to live in our own mind bubble and we would like to ignore the changes happening around us but the time will make us adapt to the reality somehow.

A few alternative perspectives on this subject



VFX Solider, an entire blog dedicated to this issue

Wednesday, July 7, 2010

TED talk on Fractals

Seriously I am not sure why I constantly stumbled on things that I have posted here before. Anyway I am glad it happens and I always come across information that particularly interests me.

I had posted about Loren Carpenter's fractal algorithm last year and today I came across this wonderful talk by a mathematics legend Benoit Mandelbrot, who talks about fascinating fractals forms which can be found in plants, clouds, human body, painting, architecture, and even in finical statistics.

It is amazing to see forms that can be created in Adobe Photoshop or After Effects with few clicks have so much complexity yet simple and related to our universe in various ways.



Monday, July 5, 2010

Classic Matte Painting resources

I remember I had posted few resources on the Albert Withlock back in last year. I just came across these videos about the craft another great master matte painter Peter Ellenshaw. Thanks to one of the member called Thomas on mattepaiting.org for posting these highly valuable videos.







It is amazing to see that how matte painting techniques overlapped so much with overall compositing process of the film. It never stop to amazes me that basic fundamentals of the craft always remains the same while just the tools and techniques advances.



Somehow I also stumbled on a really great blog by Peter Cook with quite lot of articles on classic matte painting. This is a must bookmark for people who are fan of all classic matte painting masters.


I had mentioned about some great resources hosted by vfx artist, Domingo Lizcano in my last post but now I saw his blog which also have few interesting and informative articles on class matte painting.


Sunday, June 27, 2010

Another video on Happiness Culture

Well, I just came across this interesting talk by hotelier Chip Conley on happiness based business model. After my recent rambling on the same, I was very intrigued by this talk as well so as usualI am just bookmarking it here in my blog.

Sunday, June 20, 2010

Cool youtube vids on color space

Check out these couple of videos color space. The first one explains three dimensional visualization of color in XYZ color space. In the second video John Paul Caponigro explains Color Spaces in a very interesting way balloons.

It is important to note three dimensional representation of color because most of advanced color operations works on this principal and it is very much crucial when it comes to color management as well.

Enjoy the videos :)



Wednesday, June 16, 2010

Friday, June 11, 2010

Another cool Pixar article

Its seems like I can't get enough of it... I just came across another cool article on Wired about the Pixar process and if you think I'm just obsessed with their success stories then take a look at this quote for instance.

“It’s important that nobody gets mad at you for screwing up,” says Lee Unkrich, director of Toy Story 3. “We know screwups are an essential part of making something good. That’s why our goal is to screw up as fast as possible.”

Wednesday, June 9, 2010

The Happiness Culture

I saw this cool article called "The Happiness Culture: Zappos Isn't a Company -- It's a Mission" on fastcompany.com. It is a very interesting article about the business culture at an e-commerce company called Zappos and it discusses ideas of CEO, Tony Hsieh who have wrote book on the same topic.

According to the article, here are there main core values of the company.
1. Deliver Wow Through Service
2. Embrace and Drive Change
3. Create Fun and a Little Weirdness
4. Be Adventurous, Creative and Open-Minded
5. Pursue Growth and Learning
6. Build Open and Honest Relationships with Communication
7. Build a Positive Team and Family Spirit
8. Do More with Less
9. Be Passionate and Determined
10. Be Humble

What really made me intriguing is that it's seems a lot similar to Pixar's culture which I discussed in one of my last posts. Apart from the above points they even have a similar training program for all staffs irrespective of the position when they join the company.

Here are few interesting quotes on Hsieh which I found personally interesting

"I think we're just at the beginning where companies are becoming more and more transparent whether they like it or not. People are becoming...just because the information is everywhere and it's pretty hard to control now. So I think moving forward it's going to be only the authentic companies or people can win because everyone else will eventually be outed."

"In the end, the culture is about more than money. "It's not me saying to our employees, this is where our culture is. It's more about giving employees permission and encouraging them to just be themselves."


"There's three types of happiness and really happiness is about being able to combine pleasure, passion, and purpose in one's personal life. I think it's helpful and useful to actually think about all three in terms of how you can make customers happier, employees happier, and ultimately, investors happier."

Take a look at the interview which this article was based on.





Sunday, May 16, 2010

Notes on Stereoscopic 3D

I been following this technology closely for more than a year now and I was convinced that this technology is going to be a game changer when I first heard that Avatar is going to be on 3D. A lot has been happening since Avatar came out and I think one of the crucial outcome was incursion of stereoscopic 3D into main stream broadcast media.

Singapore, as a highly techno friendly country, will be the first among the nations to do a nationwide 3D trial on the broadcast possibly on coming August. I think as for the HD wave it would be interesting to see when viewers will fully adopt this technology to broadcaster's standards. On the other side Samsung is widely promoting it's line of 3D capable TVs and aims to sell more than 2 million 3-D TVs this year and Panasonic has already came up with their affordable 3D productions gears for broadcasters. With all these vibes there is strong urge in the industry to produce more stereoscopic content for this new market that is ranging from music videos to military simulations.

Apart from all these general happenings it was very intriguing to dig deep into the technology itself and to see how it has transformed in the past few years. Stereoscopy is relatively a very old technology and people are being lured with this illusion since 1800s. I think most popular technology which we are all familiar with is Anaglyph Image where two images (left and right) are made up two colors (red and cyan) and superimposed to create illusion of stereoscopy when watched with same color coded glasses. Fortunately we have already come a long way from Anaglyph Images which caused motion sickness to us most of times.

According to Wikipedia, there are two technology for viewing stereoscopy images, mainly active and passive glasses. Active glasses technology needs an expensive wireless electronic glasses which is synced with the display screen and passive glasses uses polarizing filters on the projectors to project two images superimposed on to the screen. So for obvious reasons such as cost and practicality, Active glasses technology is moving more towards broadcast and PC gaming markets and passive glasses towards cinemas.

In cinemas, the most widely used technologies are by RealD and Dolby 3D. Both of them use slightingly different technology that falls under passive glasses technology. According to Wikipedia, The RealD Cinema system uses an electronically driven circular polarizer, mounted in front of the projector and alternating between left- and right- handedness, in sync with the left or right image being displayed by the (digital) movie projector.

Dolby 3D uses Infitec which stands for interference filter technology that splits the color spectrum into six narrow bands which then are divided into 2 in three primary color channels. This technology is a much superior form of Anaglyph technology.

There is also IMAX3D which use polarization technology similar to that of RealD.

As for a standardization, at the moment RealD is the clear winner in the race with the huge success of Avatar. According to Zdnet.com, the company has net loss of 16.3 million from 23.4 million revenue in the year 2008 and for the year end of 2009, the company lost 20.3 million from 95.9 million revenue. Here is an image comparing both Anaglyph and polarizertechnology.

On the other side, broadcast and PC Industry are widely embracing Active glasses or Liquid Shutter Glass technology. Like I mentioned before in this technique liquid crystals are triggered in synchronization with the refresh rate of the screen allowing the glasses to alternately darken over one eye, and then the other. Screens also display different perspective for each eye using a technique called Alternate frame sequencing.

Nvidia is in forefront of this technology along with Samsung in PC gaming market while Samsung plans to dominate broadcast market and others like Panasonic and Sony are joining the trend. All of them uses the same technique ofalternative frame sequencing or time sequencing as mentioned above. It is important note that most of today's screens or monitor's support maximum of 80 hertz refresh rate but in order to support this high speed syncing with the active shutter glasses the screen should have a minimum refresh rate of 120 hertz.

Following is a Panasonic promotional video showing previous technology and advantages of current technology

Panasonic has also offers full 3D solutions from production to delivery, take a look at below video below and here is the link to their full production flow chart.


There are also other technologies in which you don't have to wear any glasses. These technologies comes under the category of Autostereoscopy. Uptil now most of these technologies are expensive and screen based so it may not ne easily adoptable for cinemas or main stream broadcast. One such technique is Pixel barrier where two images (left and right) are interlaced into the screen simultaneously.

Another similar technology is Alioscopy, where you have pixel barrier or filter to make the coded sub pixels a stereoscopic 3D experience. The real beauty of this technology is that it has 8 different perspectives so viewer could enjoy the true 3D experience from different angles. I have personally experienced this technology and found it truly an amazing experiencing but at the same time rendering 8 different views in full HD resolution would be extremely time consuming process.

With all these happenings it is so obvious that a good part of futur entertainment is going to be on 3D. I feel it is crucial to note that even though Stereoscopic 3D is not a new technology but only until Avatar that James Cameron showed us that this technology could be effectively used as a visual language rather than just a gimmick to attract audience.

Unfortunately in case of the films, a lot of production houses are looking at it as marketing tool to attract audience rather than taking it as a story telling medium and I think in broadcast media, stereoscopy is pushed in order to resurrect the penetration of HD into the main stream market considering only two broadcasters worldwide are fully HD now. According to Bill Schultz, co-CEO of Moonsoop Entertainment, L.A., "We have been doing HD for six or seven years, and there are still broadcasters who do not ask for an HD delivery,". So if Stereoscopic 3D becomes successful at least it will make sure that everybody goes HD.

I could be completely wrong here but personally, I don't feel like looking at stereoscopy as a future but instead I would like to see it as matured form of story telling medium because lot of people associate it with black & white to color or silence to sound transition in films. But from what I have seen and experienced, I think this technology has it's limitations compared to conventional 2D films and there are times where 2D might win over 3D films. For example, fast cuts of shots with different scale and depth might work pretty well with the conventional 2D film. But I think doing that sort of cuts in stereoscopic film in which the perception of deptsh changes very quickly between scenes may not be pleasing to your brain no matter how good the edit looks.

Since doing a Stereoscopic film would be technically superior I think a good example to relate what I meant would be, painting with water color and oil paint. You could pain beautiful paintings with both mediums but oil painting add whole new demension as well as technical limiations compared to water color. So it just the choice of the filmmaker that would call for Stereoscopy.

Micheal Bay is one of the director is who has been outspoken about his opinion on 3D as well as 2D to Stereoscopic 3D conversion. Micheal Bay says “The way I shoot is too aggressive for 3D cameras. It’s a time consuming thing,” said Bay. “Who knows… It might be a fad. I’m kinda old school. I’m old school because I like to shoot on film. I like anamorphic lenses, and that is old school.” I agree with him on the above statement and I think it relates to my previous example as well.

For the past few weeks I been testing stereoscopy at my workplace, so I will try to post more on this subject relating to workflow, aesthetic and story tellings aspects as I discover them.

Sunday, May 9, 2010

Ed Catmull on rendering successful business


I stumbled upon this great talk by president of Pixar, Ed Catmull on maintaining successful creative environment. As an artist, personally I feel creating and sustaining a creative environment on workplace is the most crucial part of any art production. Art is all about expressing our emotions or ideas in such way that the audience feel and embrace it with their own collective experience. So if artists are working in an environment where they are intimidated by the managerial forces then it is merely impossible for them express their emotions or ideas.

I found issues raised by Catmull very intriguing and it made me dig more deep into more articles on Pixar's business model.

Later, I came across couple of other good articles and a great video interview of Catmull on The Economist. All of the articles I came across discussed the same issues he addressed in the first talk which is derived from his own tremendous experience in the computer graphics and animation industry.

One of the main point raised by Catmull was that lot of companies confuse the organisational structure with the communication structure which will hinder the creativity of the team. According to him flow of information between all the departments irrespective of structural management is must to facilitate a creativity. This is in very much contrast to the general idea of management where you report only to your manager or supervisor. But according to Catmull, the efficient way to manage people is to trust them and make them responsible for the work they do instead of getting permission from manager for each task.

He gives a good example on above issue in Harvard Business Review as follows, "Within Pixar, members of any department can approach anyone in another department to solve problems, without having to go through 'proper' channels. Managers understand they don't always have to be the first o know about something going on in their realm, and that it's okay to walk into a meeting and be surprised."

The above management philosophy goes together with the peer culture in Pixar which relays lot on trust and honesty. Since everyone have a shared responsibility and trust then it is easy to recover from a failure and Catmull stress that good management is not about preventing the errors but it is about recovering from the failures.

In order to facilitate communication, trust, high morale and spirit practically into work environment of Pixar, they have formulated Pixar University which trains employees at any level on general fine arts education. The classes are mainly on pushing unconventional thinking, creating peer culture and teamwork among the all levels of the company. According to Pixar University Dean Randy Nelson, "If you don't create an atmosphere in which risk can be easily taken, in which weird ideas can be floated, then it's likely you're going to be producing work that will look derivative in the marketplace."

It is very important to note that at Pixar they put more importance on people and workplace than the ideas they create. As Catmull says in the talk, if a good idea is given to a bad team they won't come up with anything but at the same time if the team was good then they would have either fixed it and or would have come up with a idea that works.

So it is very much crucial to put together teams of people to function well because because if they work well together then the ideas they created would sell anyway. The significance of people working together is so much important for Pixar because a movie was never about an idea but it was all about collective ideas from people working effectively in different disciplines and levels. So it makes perfect sense why they put so much money and effort into making the creative space for their artists.

Finally it is interesting to note that great art always comes from unstable environment and like Catmull points out in the above talk that good teams are inherently unstable. So much of what I discussed above goes to the fact that it is essential to have working environment which not only creates great ideas but keep people together and I hope people get that right and not just they get notion of it.

'The prhase is important in the community but it just doesn't have any effect on the behavior"
Ed Catmull

Sunday, March 21, 2010

Matte Painting Breakdown

I been out of this blog for months now, I been busy On and Off these days anyway I feel good that I had completed this personal work. Though I am not very proud of the results but I am very happy that it really helped to get back in touch with my matte painting skills after a long break.

So here goes a short break down of the whole process.

Inspiration and references.
Basically I was inspired by the above shot in Star Trek to start with this matte painting. So I started looking for inspiration in similar matte paintings done by other artists and collected few good reference images to start with my concept.

clean plate
As you can see this image gave me a lot of open space for my concept and it already had some incredible mood and atmospheric perspective to start with.

reference pictures
Above were the most of the reference images I have used early in the process. There might be other images I have used apart from these ones but for now these are the main ones I could relate to in my library.

concept photo collage, version 1
In the above image you can see I have used basic solid shapes and few images to build up the basic composition. I have used the airplane engine exhaust and bullet shell images mainly to get the shape of the main structure.

version 2
After getting the basic composition and direction. I tried to establish the scale and added more structures overall. The sky was basically composite of clouds from different photographs.

version 3
At this stage I continued to add more structures such as cranes, poles and pipes . Most importantly I tried to crush the blacks to match the distance. Getting the atmospheric perspective (level of fog) was the most tricky part and I achieved it by painting blindly with high level of opacity at first and then erasing it with a low opacity large size air brush.

version 4
After getting a right sense of atmosphere I started to paint light and shadows after considering the direction of the light source. I also added more structures and removed some structures from both sides of the composition. At this stage I also started to get some really good critics from my coworkers which really guided me in the right direction.

Version 5
Getting the right combination of haze and light was really tricky part and I also felt it was a very subjective matter. At this point I also started to add more details such as building lights, hard and soft highlights and a glance of sunlight from right to create center of attention. I also adjusted the scale of elements to make it more convincing with the lighting.

version 6

I was happy with the feeling I had with the previous version but I really pushed it a little bit further with help of some really good criticism from my friends and good folks at mattepainting.org

With this last version most of changes were on the lighting and foreground. I made lighting more evenly throughout the scene and the foreground was changed to match the lighting. Also I added more details such as more building lights and finally I changed the sky to match the overall lighting as well.

That's the end of this breakdown and I am really happy to share this process with you all so if you have any queries please do let me know.