What’s Next?

These were some of the things that could be seen at Canon Expo at New York’s Javits Convention Center last week: ice skaters pirouetting without ice, people viewing someone dressed as the Statue of Liberty from the moving deck of a fake boat, a machine that can squirt out a printed-and-bound book on demand, and a hand-holdable x-ray system.  Those weren’t directly related to the future of our business.  But what about image sensors with 120 million pixels, others (sensor chips) larger than paperback books, and yet others with more colors than merely red, green, and blue?

[The photo above, by the way, like the others in this post from Canon Expo, was shot by Mark Forman <http://screeningroom.com/> and is used here with his permission (all other rights reserved).]

We can extrapolate from the past to make certain predictions.  It’s extremely likely, for example, that the sun will rise tomorrow (or, for those of a less-poetic bent, that the rotation of the Earth will cause…).  Otherwise, we can’t predict the future, but we’re often put in a position of having to do so:  Will this stock go up?  Will it rain on during an outdoor wedding ceremony?  Will there be a better, less-expensive camera/computer/etc. after a purchase?

That last is usually as assured as a daily sunrise, but how quickly and how great the improvement are hard to know.  For help, there are blogs like this, publications, conferences, and trade shows.

The Internationale Funkausstellung  (IFA) in Berlin is an example of one of the latter.  It’s an international consumer electronics show.

At the latest IFA, among other stereoscopic 3D offerings (including 58-inch, CinemaScope-shaped, 21:9 glasses-based 3D), Philips spinoff Dimenco showed an auto-stereoscopic (no-glasses) 3D display.  Here’s a portion of a photo of it that appeared on TechRadar’s site here: http://www.techradar.com/news/television/hdtv/philips-to-launch-glasses-free-3d-tv-in-2013-713951

This is by no means the first time Philips has ventured into no-glasses 3D, but this one is different.  Autostereoscopic displays usually involve a number of views, and the display resolution gets divided by them.  The more views, the larger the viewing sweet spot and the better the 3D but the lower the resolution.  The new display has five views horizontally and three vertically, but it starts with twice as much resolution as “full 1080-line HD” both horizontally and vertically, so the 3D images end up with a respectable 768 x 720 for each of 15 views.

Perhaps such glasses-free 3D leads to a greater sensation of immersion, but there are other ways to create (or increase) an immersive sensation.  Consider, for example, the CAVE (Cave Automatic Virtual Environment), a room with stereoscopic projections on at least three walls and the floor (sometimes all surfaces).  The photo here is of a CAVE at the University of Illinois in 2001 (it was developed there roughly 10 years earlier).  SGI brought a CAVE to the National Association of Broadcasters convention shortly after it was developed.

Visitors who wore ordinary 3D glasses saw ordinary 3D — boring.  Visitors who got to wear a special pair of 3D glasses that could track their head movements, however, even though they saw exactly the same 3D as the others, were transported into a virtual world responsive to their every movement.  Unfortunately, only one viewer at a time could get the immersive experience.

At Canon Expo, however, there was “mixed reality.”  It’s based on head-mounted displays using two prisms per eye.  One, a special “free-form prism,” delivers images from a small display to the eye.  The other passes “real-world” images from in front of the viewer to both the eye and a video camera that can tell what the viewer is looking at.

The result is definitely mixed reality, a combination of stereoscopic imagery with unprocessed vision, with the 3D virtual images conforming to objects and views in the “real world.”  Virtual images can even be mapped onto real-world surfaces, with the cameras in the headgear telling the processors how to warp the virtual images appropriately.  This photo shows a complex version of the headgear; other mixed-reality viewers at Canon Expo looked little different from some 3D glasses.  Canon’s “interactive mixed reality” brochure showed people wearing the headgear walking around and collaboratively discussing an object that doesn’t exist.

Another form of immersion involves capturing 360-degree images.  At left is the Immersive Media Dodeca® 2360 camera system, combining the images from 11 different cameras and lenses into a seamless panorama.  At Canon Expo, a 360-degree view was achieved with a single lens, a single imaging chip (8984 x 5792, with 3.2 μm pixel pitch) and a mirror shaped like a cross between a donut and a cone that is, in the words of one high-ranking Canon employee, “the single most-precise optical component the company makes.”  The whole package forms a roughly fist-sized bump.

Of course, immersiveness is only one visual sensation.  There are also sharpness and color.

If you work out the math on that  Canon 360-degree image sensor, it comes to about 50 million pixels, which is considerably more than even NHK’s Super Hi-Vision (also known as ultra high-definition television, with four times the detail of 1920 x 1080 HDTV in both the horizontal and vertical directions).  Across the room from Canon’s 360-degree system, however, was their version of ultra-high resolution, with roughly eight times the detail of 1080-line HDTV in both directions.

Four Super Hi-Vision pictures could fit into one from this hyper-resolution sensor.  Canon says its resolution is comparable to the number of human optic nerves.

The full detail of the chip can only (currently) be captured at only about 1.4 frames per second, but while it is shooting hyper-detailed stills, it can (if I interpreted the information provided correctly) simultaneously capture two full-motion full-detail HDTV streams within the image.  The system uses a one-of-a-kind lens, and it’s a work in progress.

The hyper-resolution image sensor had a roughly full-frame 35mm format (comparable to that in the Canon EOS 5D Mark II DSLR), already roughly four-and-a-half times taller than a 2/3–inch format image sensor.  A few feet away was another new sensor that was larger — much larger.  It was made from a semiconductor wafer the size of a dinner plate, and the sensor itself was the size of an old 8-inch-square floppy disk — huge!

What do you get from such a huge sensor?  Extraordinary sensitivity and dynamic range.  One scene (said to have been shot at 60 frames per second with an aperture of f/6.8) showed stars in the sky as seen through a forest canopy — and it was easy to see that the leaves and needles of the trees were green.  In another scene, a woman walks in front of a table lamp, so she is back lit, but every detail and shade of gray in of her front was clearly visible.

Canon Expo demonstrated advances in both immersiveness (aside from the 360-degree and mixed-reality systems, there was also the 9-meter dome projection shown at right) and in spatial sharpness (the hyper-resolution and giant image sensors, the latter because it can deliver more contrast ratio, which affects sharpness).  There are also temporal sharpness (high frame rate) and spatio-temporal sharpness, both of which affect our perceptions of sharpness.  I found no demonstrations of increased temporal or dynamic resolution at Canon Expo, but that doesn’t mean they’re not being developed.

The images at left are portions taken from BBC R&D White Paper number 169 on “High Frame-Rate Television” published in September 2008.  It’s available here: http://www.bbc.co.uk/rd/pubs/whp/whp169.shtml The upper picture shows a toy train shot at the equivalent of 50 frames per second; the lower picture shows the same train at 300-fps.  Note that the stationary tracks and ties are equally sharp in both pictures, but the higher frame rate makes the moving train sharper in the lower picture.

As this post shows, there is immersiveness, and there is sharpness (both spatial and temporal).  Is there anything else that future imaging might bring?  How about advances in color?

Ever since its earliest days, color video has been based on three color primaries.  As this chromaticity diagram shows, however, human vision encompasses a curved space of colors, whereas any three primaries within that space define a triangle, excluding many colors.

At Canon Expo, one portion of the new-technologies section was devoted to hand-held displays that could be tilted back and forth to show the iridescence of butterfly wings and other natural phenomena.  The demonstration wasn’t to highlight the displays but a multi-band camera that captures six color ranges instead of three.

Then there was the Tsuzuri Project exhibit at Canon Expo (http://www.canon.com/tsuzuri/index.html).  It was a gorgeous reproduction of an ancient Japanese screen.  Advanced digital technology was used to capture and reproduce the detail of the original, but then a master gold-leaf artist used his talents to complete the copy.

I look forward to future tools based on what I saw at Canon Expo as well as the BBC’s high frame-rate viewing, Immersive Media’s camera system, and even the Philips autostereoscopic display.  And I’m glad that human artists are still needed to use them.

Tags: 3d, autostereoscopic, BBC R&D, Canon, Canon Expo, Dimenco, Dodeca, immersive media, mixed reality, Philips, Tsuzuri Project, ultra-resolution

The Elephant in the Room: 3D at NAB 2010

As I roamed the exhibits at the NAB show this month, I kept wondering what other year it seemed most like.  And I was not alone.

There were plenty of important issues covered at the show, from citizen journalism to internet-connected TV.  And then there was the elephant in the room.

It would be a lie to say that 3D technologies could be found at every booth on the show floor.  But it was probably the case that there was 3D in at least every aisle.  There was so much 3D that it tended to diminish all other news.

In acquisition technology, for example, LED lighting was near ubiquitous, with focusable instruments, such as the Litepanels Sola, sometimes painfully bright.  Panasonic and Sony both showed models of future inexpensive video cameras with large-format imagers, and Aaton joined the range of those offering “digital magazines” for film cameras.  In small formats, GoPro’s Hero is a complete HD camcorder weighing just three ounces.

In storage technology, Cache-A, For-A, IBM, and Sony all showed in new offerings that tape is not dead.  Meanwhile, iVDR removable-hard-drive storage could be seen in several new products, and Canon introduced new camcorders based on Compact Flash cards.

Cinedeck looks like a viewfinder but includes built-in storage and editing capability. NextoDI’s NVS 2525 can copy either P2 or SxS cards.

In processing, Dan Carew’s Indie 2.0 blog said of Blackmagic Design’s DaVinci Resolve 7.0, “this best-in-class color correction software was formerly US$250,000 (for software and hardware) and is now available in a Mac software only verions for US$995.” http://indie2zero.com/2010/04/16/what-i-liked-and-saw-at-nab-2010/ Immersive Media’s 11-camera spherical views can now be stitched and streamed live.  NewTek’s TriCaster TCXD850 can deal with 22 inputs and virtual sets.  And, though you might not yet be able to figure out why you’d want this capability, Snell’s Kahuna 360 production switcher can deal with up to 16 shows at once.

In wireless distribution, there was VµbIQ’s 60 GHz uncompressed transmitter on a chip and Streambox’s Avenir for bonding up to four cellular modems to create a 20 Mbps channel.  In wired, there was Pleora’s EtherCast palm-sized bidirectional ASI-IP gateways.  And, in technologies that could be applied to either, there were Fraunhofer’s codec with a latency of just one macroblock line and a Harris-LG/Zenith proposal for expanding ATSC mobile transmission to full-channel use.

In presentation, there was a reference picture monitor from Dolby (seen in almost its final form at the HPA Tech Retreat).  Several booths had OLED monitors, from 7-inch at Sony to 15-inch at TVLogic.  Wohler’s Presto router has an LCD video display on each button.  And Ostendo’s CDM43 is a curved monitor with a 30:9 aspect ratio.

That barely scratches the surface of the non-3D news from NAB.  And then there was 3D.

Even All-Mobile Video’s Epic 3D production truck, parked in Sony’s exhibit, wore 3D glasses.  But it was the glasses on visitors to the truck that proved more instructive.

Sony provided RealD circularly polarized glasses to visitors for looking at everything from relatively small monitors to a giant outdoor-type LED display.  As soon as those visitors entered the control room of AMV’s Epic 3D truck and donned their glasses, however, they saw ghosting — crosstalk between the two eye views.  AMV staff were prepared for the shocked looks.  ”Sit down,” they said.  ”There’s a narrow vertical angle, and you have to be head-on to the monitors.”  Sure enough, that solved the problem — at least for those who could sit.

Another potential 3D problem was mentioned in the two-day 3D Digital Cinema Summit before the show opened.  If 3D is shot for a small screen and blown up to cinema size, it can cause eye divergence.  3ality’s camera rigs indicate when this might happen, but it happened anyway on at least one cinema-sized screen at NAB, leading to some audience queasiness.

Buzz Hays of the Sony 3D Technology Center says making 3D is easy, but making good 3D is hard.  There was a lot of 3D at NAB, including both easy and hard, good and bad.

It was hard to count the number of side-by-side and beam-splitter dual-camera rigs at the show, but, in addition to those, there were integrated (one-piece) 3D cameras and camcorders, in various stages of readiness, from 17 different brands, both on and off the show floor.  It seems that all of them were said to be “the first.”

Much could be learned about 3D at the two-day Digital Cinema Summit before the show opened.  It began with Sony’s Pete Lude showing that an ordinary 2D picture can seem 3D when viewed with just one eye, leading a later speaker (me) to quip that watching with an eye patch, therefore, is an inexpensive way to get 3DTV.

3ality’s Steve Schklair followed Lude with an on-screen, live demonstration-tutorial on the effects of different 3D rig settings: height, rotation, lens interaxial, convergence, etc.  He was followed by directors, stereographers, and trainers of 3D-convergence operators, among others.

Although 3D would seem to require more equipment (two cameras and lenses plus a stereo rig at each location) and more personnel (a convergence operator per camera in addition to a stereographer), there is seemingly one saving grace.  According to Schklair and others, 3D can get away with fewer cameras and less cutting than 2D.

The same thing was said of HD, however, in its early days.  Sure enough, when I worked on one show in 1989, we used just four HD cameras feeding the HD truck and twice as many non-HD cameras feeding the non-HD truck.  In the early days, it was common practice to do separate HD and SD productions.  Today, of course, one HD production feeds all, and it typically uses as many cameras and as rapid cutting as an SD show.

Atop a tower of Fujinon’s NAB booth, Pace showed something that recognizes the current economics of 3D.  With virtually no 3DTV audience, it’s hard to justify separate 3D productions, but, with such major players as ESPN, DirecTV, Discovery, and Sky involved in 3D, the elephant cannot be ignored, either.  So the Pace Shadow system places a 3D rig atop the long lens of a typical 2D sports camera.  Furthermore, it interconnects the controls (in a variety of selectable ways) so that the operator of the 2D camera need not be concerned about shooting 3D: one camera position, one operator, different 2D and 3D outputs.

Screen Subtitling came up with similarly clever solutions to the problem of 3D graphics.  Unless text is closer to the viewer (in 3D depth) than the portion of the image that it is obscuring, it can be uncomfortable to read.

Traditionally, subtitles are at the bottom of a screen, where 3D objects are closest to the viewer.  Raise the graphics to the top, and they might work in the screen plane.

Then there’s the issue of putting the graphics on the screen.  With left- and right-eye views, it might seem that two keying systems are required.  But with much 3D being distributed in a side-by-side format, a single keyer can place 3D graphics directly into the side-by-side feed.

There was much more 3D at the show, in every field of video technology (and, perhaps even audio).  In acquisition, for example, aside from integrated cameras, 3D mounts, and even individual cameras designed specifically for 3D (like Sony’s HDC-P1), there were also 3D lens adaptors, precision-matched lenses, precision lens controls, and even relay optics intended to allow wider cameras to be placed closer together, as in this picture shot by Eric Cheng of WetPixel.com: http://wetpixel.com/i.php/full/2010-nab-show-report-las-vegas/

At the other end of the 3D chain, there were both plasma and LCD autostereoscopic (no-glasses) displays using both lenticular and parallax-barrier technology, small OLED displays with active-shutter glasses and giant LED screens with passive circularly polarized glasses.  There were LCD and plasma screens (up to 152-inch at Panasonic) and DLP rear-projectors using active-shutter glasses, and both LCD and laser projection using passive polarized glasses.

There were dual-panel displays with beam splitters, and displays intended to be viewed through long strips of fixed polarized materials (to accommodate all viewers’ heights).  There were many anaglyph displays in the three-different primary-and-complement color combinations.  There were 3D viewfinders using glasses and others with displays for each eye.

Japan’s Burton showed a laser-plasma display that creates 3D images in mid-air.  Normally, they’ve viewed through laser-protection goggles, as in the image at the right at the top of this post.  But as a safety measure, they showed them instead inside an amber tube at NAB.

In storage, it seems that everyone who had anything that could record images had a version that could do so in 3D.  Even Convergent Design’s tiny Nano was available in a 3D version.  The Abekas Mira is an eight-channel digital production server — or it’s a four-channel 3D digital production server.  Want an uncompressed 3D field recorder?  Keisoku Giken’s UDR-D100 was just one such product at the show.

In processing, just about every form of editing and processing had a 3D version.  Monogram showed a touch-screen 3D “truck-in-a-box” production system.  Belgium’s Imec research lab even showed licensable technology for stereoscopic virtual cameras.

There was a range of equipment and services for converting 2D to 3D either in real time or not, automatically and with human assistance.  And there was a large range of processing equipment designed to fix 3D problems, such as camera rotation and height variation.

Sony’s MPE200 is one such device, with a U.S. list price of $38,000.  The MPES3D01/01 software to run it, however, is another $22,500.  With the least-expensive 3D camera at the show (Minoru 3D) retailing for under $60 at amazon.com, it might be said that 3D is cheap, but good 3D costs.

There was 3D test equipment from many manufacturers.  There was high-speed 3D (Antelope/Vision Research). There was 3D coax (Belden 1694D, complete with anaglyph color coding).  Ryerson University is doing eye-tracking research on what viewers look at in 3D and whether it’s different from HD and 4K.

So why was I wondering what year it was?  At NAB shows there have been many technologies shown that never went anywhere.  We still await voice-recognition production switchers, for example, and also voice-recognition captioning.  But those have generally been shown by only one company or a small number of exhibitors.

Digital video effects were among the fastest technologies to penetrate the industry.  First shown at NAB in 1973, they were commonly seen in homes by the end of the decade.

Then there was HDTV.  Its penetration after NAB introduction took much longer, even if dated only from 1989, when an entire exhibition hall was devoted to the subject (there were many earlier NAB displays).  Estimates vary, but U.S. household penetration of HDTV 21 years later seems to be in the vicinity of half.

At least HDTV did eventually penetrate U.S. households.  Visitors to NAB conventions in the early 1980s could see aisle after aisle of exhibits claiming compatibility with one or both competing standards for teletext.  One standard was being broadcast on CBS and NBC; the other on TBS.  There were professional and consumer equipment manufacturers and services offering support.  Based on the quantity and diversity of promotion at NAB, it was hard to imagine that teletext would not take off in the U.S.

So, will 3DTV emulate digital effects, HDTV, U.S. teletext, or none of the above?  Time will tell.

Tags: 3ality, 3d, 3dtv, Aaton, Abekas, All-Mobile Video, Burton, Buzz Hays, Cache-A, Canon, Cinedeck, Convergent Design, Digital Cinema Summit, For-A, Fujinon, GoPro Hero, immersive media, iVDR, Keisoku Giken, Litepanels Sola, Minoru 3D, Monogram, NAB 2010, NewTek, NextoDI, Ostendo, Pace, Panasonic, Pete Lude, Screen Subtitling, Snell, Sony, Steve Schklair, VµbIQ

This Thing Called 3D

It has been a heck of a month for 3D announcements.  Comcast carried The Final Destination in 3D on the day of its DVD release. The Consumer Electronics Show (CES) seemed all about 3D.  The International Telecommunications Union (ITU) issued a report on 3D TV.  The program recently posted for next month’s Hollywood Post Alliance (HPA) Tech Retreat includes not only a 3D-in-the-Home “supersession” but also other presentations on such issues as 3D gaming, 3D projection, 3D vision, and, from Adobe, 3D video stabilization.  Electronic Engineering Times (EET) ran a story on January 21 about an agreement between France’s CEA-Leti and U.S. firm R3Logic “to develop 3D design methodologies for consumer and wireless applications.”  And Computerworld on January 27 talked about 3D video graphics chips moving from games to medical imaging.

What does it all mean?  That’s the sort of question one might ask after reading the front page of a newspaper, one carrying perhaps a dozen stories on different topics, because the 3D discussed in the above paragraph also covers multiple topics, not all of them associated with depth perception. More »

Tags: 3d, 3d glasses, 3D TV, 3dtv, active-shutter, Adobe, American Paper Optics, AnaChrome, anaglyph, Business Week, CES, ChromaDepth, chromostereopsis, chromostereoscopic, ColorCode, Comcast, Communications Research Centre, Computerworld, Curtin University, Depth Enhanced Video, Digital Optical Television Systems, Discovery HD, EE Times, Final Destination, flicker vision, Fox, Fraunhofer Institut, GEForce, Hollywood Post Alliance, holography, HPA, immersive media, integral television, ITU, KMQ, LeaVision, lenticular, live holography, multi-view, NG3D, NHK, NICT, NVidia, over-under, Panoramic stereoscopic, polarized, Prisma-Chrome, prismatic glasses, Pulfrich, RabbitHoles, Rolling Stones, side-by-side, SMPTE, Steel Wheels, Stereographics, stereoscopic, Thomas Edwards, Tournament of Roses, TrioScopics, ultra-high-definition, VH1, Vision III, wiggle vision, wobble stereoscopy