Can You Fix It in Post?

Can you fix it in post?

The simple answer is: Yes.

Consider Avatar.  Not only was an entire world created and populated in computers, but even a human actor’s legs were atrophied in post.  So, yes, anything can be fixed in post — given enough time and money.  In the worst case, artists would simply “paint” photorealistic images, pixel by pixel and frame by frame.

It wasn’t always so, especially in electronic imagery.  Before “paint” systems, post was extremely limited.  Cuts, dissolves, wipes, and keys were possible, but, in the days of analog recorders, even those often degraded images.  ”We’ll fix it in post” became a laughter-inducing cliché.

Today, not even counting “painting,” there are many real-time processes that can replace production activities.  Rather than having an image specialist controlling camera parameters during shooting, the raw signals from the sensors can be recorded, with a post-production colorist dealing with them.  Instead of optical filters in front of or behind the lens, post-production filters can achieve much the same effects.  Instead of worrying about large, stable camera mounts or optical image stabilizers, producers can turn to post-production image stabilization.

And then there’s 3D.

The images above are taken from the brochure for Sony’s MPE200 stereoscopic image processor.  They show some of the post-shooting corrections the system can accomplish.  At top left there is correction of inter-camera image center as a lens zooms, at top right correction of inter-camera rotation, and, at bottom, from left to right, correction of interaxial spacing, inter-camera elevation, and even inter-camera distance from the scene.

Let’s start with the interaxial-spacing adjustment.  It can move homologous points in the two eye views closer together or farther apart.  Unfortunately, that’s not the only difference between the two camera views.

The image at top above is what a single camera might see when shooting an edge of a cube or building.  Below it are the views of separated eyes.  The left eye (or left camera) sees more of the left side of the object; the right sees more of the right side.  Depending on the exact positioning, shooting distance, and object, one camera might even see things that the other doesn’t.  There’s no way that Sony’s MPE200 — or anyone else’s post-production processor — can know how to put things into the picture that weren’t there in the original.

Now consider some of the other corrections, like that of inter-camera rotation.  An HDTV frame is a 16:9 rectangle.  If one camera’s rectangle is rotated with respect to another’s, as shown above in the blue and red rectangles, the only way to get them to line up is to trim the content of each, as shown in the green rectangle.  That changes the original framing.

It’s not just a 3D problem.  With a stable mount or optical image stabilization, what the shooter sees is (not counting overscan or intentional changes in post) what the viewer sees.  With post-production image stabilization, it can be very different.

Have a look at the second (Mounts — the problem), third (Mounts — fixed in post), and fourth (Mounts — not exactly fixed) files available on this download page: http://schubincafe.com/blog/2010/06/things-you-can-or-can’t-fix-in-post-video-acquisition/.  They were provided by Aseem Agarwala of Adobe Systems, and they demonstrate the tremendous power of post-production image stabilization.

The first clip is an example of a horribly unstable image, shot with a handheld camera.  The second clip shows the post-processed result — so smooth that it appears to have been shot by an experienced crew with a camera mounted on a dolly on track.

The third clip, however, shows the original and the stabilized versions together.  There’s no question that the image has been marvelously stabilized, but the framing is so different that the second story of the building in the background disappears completely in the corrected version.

It’s not just framing.  If there’s any process that can be perfectly duplicated in post, it’s the adjustment of the color parameters of the signal produced by a camera’s image sensors.  As long as all of the information is recorded, it makes no difference from a technical standpoint whether the adjustments are made at the camera or in a colorist’s suite.  Unfortunately, there are standpoints other than technical.

The two pictures shown above are taken from the book Goodman’s Guide to the Panasonic Varicam by Robert Goodman, AMGMedia Publishers, 2004, http://www.goodmansguide.com/theseries.html (and here’s a link to Goodman’s own site: http://www.histories.com/hjemmeside.html).  The upper picture has the master gamma set to .35; in the lower picture, it’s .75.

Neither is necessarily better, and neither is necessarily “right.”  They are simply different.

Above is another pair of images from the same book.  The upper one has dynamic level set to 500; the lower is at 200.  Notice that all of the detail of the collar is easily seen in the 500.  On the other hand, the face seems desaturated.  Again, neither is necessarily good or right.  But there are major differences between these four pictures (there are even more image pairs in the book, demonstrating other parameters).

If the adjustments were made in production, the director might have liked some characteristics of the image (say, the collar detail) but not others (say, the desaturated face) and changed things to compensate (different lighting, makeup, or clothing, for example).  In post, the video parameters can be changed at will, but the lighting, makeup, and clothing remain the same, unless, of course, pixel-by-pixel and frame-by-frame an artist (or, more likely, a team of artists) repaints the images as they might have been captured in the first place.

If you have enough money and time, you can do anything in post.  For the rest of us, it’s a good idea to try to achieve desired looks in production.

3DTV Today: One Step Back

The Society of Motion-Picture and Television Engineers (SMPTE) just completed an International Conference on Stereoscopic 3D for Media and Entertainment, what it calls “the only scientific gathering focused exclusively on 3D.”  Like many SMPTE conferences, it gazed into the future, with one presentation introducing “the need for additional image processing to get pixel-level geometry matching,” another discussing “Spatial Phase Imaging technology” that “can be made to work with any existing sensor-optics combination, making it amenable to a plethora of applications,” and yet another was titled “What Is Holographic Television, and Will It Ever Be in My Living Room?”

By the accounts of those who attended the event, it was jam-packed with terrific information.  This post, however, is not about the future of 3DTV but about its present.  And it won’t even delve into issues of 3D vision.

The image above is a portion of a television image of the face of Sir James Wilson Vincent Savile, better known as the entertainer Jimmy Savile.  It is clearly a black-&-white image, and it’s from this fascinating web site about experiments to allow color to be recovered from black-&-white video recordings: http://colour-recovery.wikispaces.com/Richard+Russell’s+experiments

The reason the color can be recovered is the same reason the image looks so bad.  When “compatible” color was created, it added a supposedly invisible color subcarrier to the video signal.  But it wasn’t invisible.  People who had been watching pristine images on their black-&-white TVs suddenly got extra dot and line patterns in their pictures.  Those with color TVs got less detail.

When sound was added to movies, something similar happened.  At the left is the four-units-wide by three-units-high (4:3 or 1.33:1 aspect ratio) frame of silent movies.  When a soundtrack was added, it impinged on the frame, giving the picture shape a much squarer 1.16:1 aspect ratio.  The so-called Academy aperture shrunk the picture (losing resolution) to return to a somewhat wider shape (1.375:1).

Now, in this age of digital (no more dot pattern) high-definition (widescreen with no resolution loss) TV, 3DTV sets are being sold.  They can deliver an added sensation of depth (assuming all goes well).  Do they cause anything to be lost as a result?

The picture above is a portion of one eye’s view of a 3D pair of photos shot by Pete Fasciano during a presentation he taught on July 21 about stereoscopic 3D.  I’ve trimmed it to an HDTV-like 16:9 aspect ratio.  Had it been a TV show shot in HDTV, it might have looked like the above.

Someday, we might have a form of 3DTV that will deliver left-eye and right-eye images separately, with full spatial and temporal resolution.  Today, what we have is the HDMI 1.4a standard.  It calls for the above for 1080i HDTV 3D.  The left- and right-eye views are placed side by side and squeezed into the HDTV frame.  Instead of 1080i HDTV’s 1920 pixels across, there are just 960 for each eye’s view.

For today’s 3DTVs that use active-shutter glasses, that’s the only resolution loss.  But some 3DTVs use passive glasses, with different eye views on different scanning lines.  If that’s the case, the side-by-side configuration drops the horizontal resolution from 1920 to 960, and the passive-glasses system drops the vertical from 1080 to 540.

HDMI 1.4a calls for the above for 720p HDTV 3D.  In this case the left-eye image is placed above the right-eye image.  Of course, that drops the vertical resolution from 720 lines to 360.  Now let’s add closed captioning.

Above is a simulated HDTV image with a closed caption.  Were this post about stereoscopic 3D and human vision, I might have pointed out that the caption is in the screen plane, so there will be a visual conflict between it and anything it is occluding that is set to come forward from the screen plane.  But this post is not about that.

Above is the same closed caption, generated, perhaps, by a cable, satellite, or telephone-company set-top box.  The caption appears where the box, not knowing about 3D, thinks it should appear.

Finally, when the captioned 1080i HDTV 3D signal from the set-top box enters the 3DTV, this is the result.  The caption is split in two, is elongated, and appears only half the time.  I’ve illustrated that last by making the caption appear transparent, but it might be a little worse than that.  The portion on the right will appear when the left-eye shutter of active glasses is open; the portion on the left will appear when the right-eye shutter is open.

From a technology standpoint, it’s relatively easy to design a system in which none of these problems will exist.  The presenters and attendees of the SMPTE International Conference on Stereoscopic 3D for Media and Entertainment might be doing that right now.  Unfortunately, 3DTVs are being sold today.

Pico the Hits

Roger Payano has a BS in mechanical engineering and an MS in industrial engineering and has worked in the defense industry, but when he was applauded recently at the Kennedy Center in Washington, D.C. it had nothing to do with his engineering prowess.  He played the title role in the Synetic Theater production of Othello, and everyone in the audience knew what his character was thinking, despite the fact that he did not utter a single word during the course of the play.  Neither did any of his fellow actors.

Synetic Theater won two Helen Hayes awards this year for earlier productions.  Prior to Othello this season they presented critically acclaimed versions of such other plays as Antony and Cleopatra and A Midsummer Night’s Dream, also without any spoken words.  But Othello was different.  Audience members could literally see what Othello was thinking, thanks to a video-based technological breakthrough.

Theater has long used technology.  More than two thousand years ago, the ancient Roman poet we call Horace warned writers not to use a “deus ex machina” (a god from a machine) to resolve plots.  He was referring to a practice in ancient Greek theater, at least 500 years older still, of having a crane drop an actor playing god onto the stage to use supernatural powers to take care of problems.

The exact dates when the spotlight and the lighting dimmer were invented might never be known, but in 1638 Nicola Sabbatini’s Pratica di fabricar scene e macchine ne‘ teatri, a theatrical-technology instruction book, offered plans for both, not to mention designs for set-changing, flying, and storm- and flame-simulating machinery.  Sweden’s Drottningholms Slottsteater, opened in 1766, still uses the original 18th-century, human-powered stage machinery (and lighting control), which can effect a complete set change — wings, flies, etc. — in a matter of seconds.  You can see it in operation here: http://dtm.se/visningar/bakom_kulisserna.asp

The use of motion pictures on a screen in theatrical presentations is much older than Sabbatini’s book.  Documentation exists that an 11-piece shadow puppet was used to entertain Emperor Wu of China more than 2000 years ago.  Johann Zahn’s Oculus Artificialis Teledioptricus Sive Telescopium, first published in 1685, showed how to project moving images using mechanical slides in what are today called “magic lantern” or “stereopticon” projectors.  Below is a moving image from one form of motion slide, as shown on the Dutch magic-lantern site de Luikerwaal: http://www.luikerwaal.com/newframe_uk.htm?/boeken_uk.htm

Below this paragraph is another motion picture, said to be the oldest existing long sequence intended to be projected in a theater (older sequences shot by Eadweard Muybridge exist, and some of those were intended to be projected, but they comprise only a few frames each, arranged around a disk).  The motion sequence below was shot by Louis Le Prince in Leeds, England, in 1888.

Was Le Prince trying to invent what we today call “movies”?  There’s no question that he wanted to be able to capture and project live-action sequences, and some say he succeeded before Edison.  But it’s not clear that he intended audiences to enter an auditorium simply to watch his movies.  At least one contemporary document suggests that his invention was intended to provide motion-picture backdrops for live performances.  In 1896, when the Rosabel Morrison Company performed the opera Carmen at the Lyceum Theater in Elizabeth, New Jersey in front of a projected Eidoloscope movie sequence of a bullfight, Le Prince’s goal might have been achieved.

Moving-image projection in the service of theatrical drama has certainly advanced over the past century or so.  When a new production of La Damnation de Faust opened at the Metropolitan Opera in 2008, it utilized multiple high-definition projectors, fed computer graphics generated live based on input from motion sensing cameras, to provide images that interact with the human performances on the stage.  Thus, a character on stage could pole a gondola, the water rippling in its wake — except for the fact that there was no water, let alone wake and ripples; it was all projected computer graphics.

There are advanced projection systems today that can track moving screens across a stage and not only project on them but also pre-distort the images according to a varying screen angle.  Some can even project on non-flat surfaces.  Here’s one example: http://il.youtube.com/watch?v=LHLqATsdWQo&feature=related

Unfortunately, the more advanced the systems, the less they seem to be appropriate in live drama.  As Avatar showed, it’s possible to do anything in a movie, from atrophying an actor’s legs to creating an entire non-terrestrial civilization.  And movie tickets, despite recent price hikes, usually cost less than those of live theater.  Might computer-graphics-based images, like amplified voices, make live theater seem more like movies, and, if so, would it be worth paying more than a movie-ticket price to see it?

The wordless Synetic Theater performances don’t use amplified voices, of course.  And the company’s foray into video projection, though it involved such advanced concepts as tracking moving screens and presenting images on irregular surfaces, was remarkably live and human.

There were no whirring projector fans.  There was no projection booth.  No illuminated dust motes made the projection beams visible.  At times the images seemed to come directly from Othello’s mind.

In fact, the images came from palm-sized, battery-operated, handheld 3M MPro150 pico-projectors, small enough to be hidden in costumes when not used and barely visible even when they were.  One is shown below on a small tripod.

Introduced for the business-presentation market, pico projectors might not be ideal for that purpose.  With just 15-lumen output, the MPro150 would provide 8 nits of luminance (think brightness) on a plain white screen just 2 feet high.  For comparison, a Panasonic TH-50VX100U 50-inch plasma display, with roughly the same size picture, would offer 1200 nits, 150 times more.  A smaller projected picture would be brighter, but then the business presentation might as well be shown on a laptop screen, also brighter.

On a dark stage, however, the images from the pico projectors in Synetic Theater’s production of Othello seemed perfect.  Tracking moving screens was no problem; the actors using the projectors just turned their wrists.  And the human foibles of such tracking seemed to keep the images human, too.  Similarly, projecting hands on the irregular surface of the waist of a character’s dress required only that the actor doing the projecting aim that way.

Perhaps Othello was the first example of what will be an age of on-stage pico projection.  Either way, it was a superb production.