Integration of Fogo Player and SAGE (Scalable Adaptive Graphics Environment) for 8K UHD Video Exhibition

After 4K, 8K UHD resolution can be considered the new challenge for film, visualization and tele-vision. Due to the dimension of 8K image, the remote micro and macro images are considered appropriate for visualization with scientific and educational purposes. This article describes the implementation of the 8K video plug-in to play UHD videos into the SAGE, a collaborative environment for the advancement of scientific research and education developed by EVL (Electronic Visualization Laboratory), upgrading its current resolution for video playing. It also presents the challenges of producing 8K content, the development of the code and introduces the features to view 8K ultra high definition videos in multiple tiles through the use of low cost computer clusters. The SAGE 8K video plug-in implementation was made possible because of the previous development of the Fogo Player, a system to run 4K and 8K content. The results were successfully presented at the CineGrid International Workshop 2013 in San Diego.


Introduction: 8K and SAGE Films
The 8K films, also called Ultra High Definition movies with over 33 million pixels per frame, are considered films whose resolution is one that will replace the 4K or even immediately replace the Full-HD or 2K.In 2002, the DCI (Digital Cinema Initiatives) standardized digital cinema considered the 4K resolution as equivalent to 35 mm film, which gives rise to speculation about the equivalence of an 8K film about the film in 70 mm.The 8K "pixel race" is present in technology exhibitions such as the NAB (National Association of Broadcasters), and in 2013 the company Astrodesign presented the AH-4800 camera along with the a whole 8K system [1] and, Intopix Technologies launched encoders and decoders for 8K "as the only hardware JPEG 2000 codec that allows the processing of Ultra HD images (8K: 7680 × 4320 or 8192 × 4320 pixels)" [2].A 7680 × 4320 resolution is four times greater than 4K definition and has been under development since early 2003 by NHK (Nippon Hoso Kyokai-Japan Broadcasting Corporation) using an array of 16 full-recorders and cameras with two HDDs CCD (Charge-Coupled Device), doubling the 3840 × 2048 resolution.In 2005 UHDTV programs were demonstrated at the 2005 World Exposition, Aichi, Japan [3].Recently NHK announced the development of an 8K sensor that can shoot video at 120 fps and demonstrated 8K broadcasting during the 2012 Olympics in London [4].Technology companies such as Zaxel Systems has announced the "first world's first 8K server that drives the JVC's wobbling 8K projector DLA-VS4800" [5].The Figure 1 illustrates the advancement of the image size for scientific visualization.In the example, the image starts with the 2K (Full HD) and ends in the 8K definition.
Today the 4K cinema is a popular technology; however there still are difficulties in editing, post-production and exhibition.Not many companies are able to complete films in 4K, following the entire workflow process, and not many movie theaters can display them.Evidently, in the case of 8K content, this situation is even more defiant since it has been hard to find content with this resolution due to the embryonic stage of the process.The pixels race raises the question of the boundaries of the image by asking whether the image boundaries refer to the human eye or the computational limit, while keeping in mind the infinity of the economic race.The Japanese technology analyst Damian Thong, before the demonstration of the first streaming content 8K undertaken by NHK Olympics in London, says that "for most typical television viewing distances, normal humans may not be able to discern the difference between 4K and 8K.The more compelling argument for migration may be in other aspects of picture quality that will also improve with the migrations to 4K and beyond, including frame rate, color, and dynamic range" [6].
However digital technologies of super-high definition have enabled the creation of increasingly complex and advanced visualizations.Besides a series of 70 mm films for large events, the production of ultra-sharp images in supercomputing laboratories for research centers, laboratories, space and planetary visualization research centers certainly justify such a wide image.By visualizing phenomena in both macro and micro scales, such laboratories are unequivocal spaces for the production of 8K UHD movies.
The opportunity to combine the 8K technology format to an environment for scientific visualization, education, and research is why LAVID (Laboratory of Digital Video Applications from the Federal University of Paraíba), along with LabCine (Laboratory of Cinematic Arts and Visualization at Mackenzie University), and with the support of RNP (The National Education and Research Network), decided to produce an integration of "Fogo", a player developed by LAVID from the Working Group of Advanced Visualization, with the SAGE system (Scalable Adaptive Graphics Environment), developed by the Electronic Visualization Laboratory at the University of Illinois in Chicago.
According to Renambot and Plepys, "SAGE is cross-platform, open-source middleware that enables users to have a common operating environment, or framework, to access, display and share a variety of content-whether digital cinema animations, high-resolution images, high-definition video-teleconferencing, presentation slides, documents, spreadsheets or laptop screens-in a variety of resolutions and formats, from multiple sources, to one or more tiled display walls" [7].The purposes of the SAGE are to "support the sharing of high-resolution media and data between visualization environments or among groups of collaborators at local or remote sites.Its network-centric architecture allows collaborators to share visualizations (such as remote desktop, video streams, 2D maps, and 3D renderings) on local or remote systems" [8].
In 2012 RNP contemplated various Brazilian educational and research institutions with OptiPortals consisting of a small video-wall with 4 monitors 47" full HD, allowing the visualization of films next to 4K resolution.They are coupled to videoconference equipment, connected to the high speed network, and controlled by SAGE middleware The LabCine OptiPortal running SAGE.In Figure 2 the OptiPortal is exhibiting a 4K animation coming from the Federal University of Paraíba (UFPB) in real time.LabCine and LAVID are connected and equipped with these portals and have done collaborative work in which questions about the need for super-high-definition  images in research are discussed, as in, for example, the images of Mars collected by NASA, telehealth and telemedicine applications.

Fogo Player
The Ultra High Definition (UHD) player called Fogo (Portuguese for fire), developed by LAVID, was the result of the Working Group, supported by RNP and divided into three sectors working simultaneously: LabCine producing content-a movie 4K 3D called StereoEssays [9], LAVID developing computational technology, and in this case a player using the challenges of 3D images that doubled the number of data content and LARC (Laboratory of Computer Architecture and Networks, University of São Paulo) carrying out a systematization of display systems and proposing affordable solutions for viewing 4K movies.
The "Fogo Player" was first demonstrated at the CineGrid@Rio 2011 in September 2011 in Rio de Janeiro, Brazil where the film StereoEssays was released.The second public demonstration was at the 6th Annual International Workshop CineGrid in December 2011 held at UCSD (University of California, San Diego), USA.The Fogo Player 4K solution requires only COTS (commercial off-the-shelf) hardware, and relies on specialized software components for encoding, transmission, decoding and playback.
Usually when working with video compression to transmit the data over a very high demanding network speed, it is necessary to adjust the number of data packages to the video structure itself.In the "Fogo Player" system, in order to fix the problems with the network speed that could create some specific limitations and to and reduce the delay among sending, receiving and exhibiting the UHD videos, we decided to send the video data sliced in small parts.The design of the "Fogo Player" is a set of distributed players that work in a system compared to a cloud.In order to provide a better performance using this design, we developed a player that features different video formats and combine them in only one output stream displaying the partitions of the video on the projector or on a UHD video screen.The "Fogo Player" architecture is designed with four partitions: a) distribution; b) preprocessing; c) streamer and d) player.The main system features also a sub partition that coordinates the video format drives and adjusts them to the output system.The "Fogo" system also features a set of elements that adjust the subsets of the system in order to adapt the player of the UHD videos to create a scale and to fit the projection on the device where the videos will be projected.The design and development of the "Fogo Player" is described in details in the article published by Aquino Junior, Gomes, Silva Neto, M. Duarte Costa and Souza Filho [10].
As explained in the article mentioned above, the "Fogo Player" system was developed to work also in conjunction with the SAGE system and is capable of displaying many subsets of a Ultra High Definition video.The system is also capable to stream and to project 3D (stereoscopic) videos, synchronizing both right and left "eyes" videos outputs.The "Fogo Player" is made and features a system of partitioned components that contains a video and audio "player" software that controls the entire environment of the input and output video streamer.
The system works with a set of four softwares called a) "Brick Player", which is described as a encoder and decoder that splits the UHD videos into quadrants and stream them in the same way, exhibiting the synchronized result as only one video; b) "Controller" software that ensures that the video partitions will be featured all together at the output.The controller is also the software system that clear the buffer in order to prevent, automatically changing the video rate, an overload in the video stream system; c) "Audio Player" that syncs the audio file with the content exhibiting both in the same rate and the d) "Fogo Streamer", which is the software that streams the partitions of the videos and that defines the frame rate, the compression and the encoding and decoding process.
The "Fogo Player" is a PC computer that runs a Linux server and that contains a Intel ® Xeon ® 3.2 GHz processor, 64 GB of RAM memory and 2 ATI FirePro™ graphics cards.Both ATI FirePro™ graphic cards use a software protocol to synchronize the video partitions and to control the video sync at the output.
When running with multiple instances of Brick Players, Fogo Player requires a synchronization mechanism to coordinate the decoding and displaying processes to maintain the coherence of the content being displayed.The Precision Time Protocol (PTP) is used to synchronize the clocks on the different machines.This protocol guarantees a maximum time drift of 106 milliseconds among a group of machines connected in a local network.Clock synchronization is not an issue when all brick players are executed on a single machine.
To investigate the performance improvement obtained by parallelization, we ran a performance evaluation of the decoding process in each configuration (one 4K quadrant, four 2K quadrants, and 16 1K quadrants) using the five samples we described in the last section.For each sample, we performed five replications in each configuration and used a confidence level of 95 percent.
To perform the tests, we used a commodity type machine with one six-core processor and 24 Gbytes of RAM memory.Because the reduction in the compression rate is low, the overall performance is improved when considering a 4K video as a composition of a set of minor quadrants.Specifically, for four 2K quadrants, the reduction in the compression rate is minimal and the decoding delay is almost equal to that in the configuration with 16 1K quadrants.Therefore, we configured the Fogo Player to handle the videos as a set of 2K quadrants.

Integrating Fogo Player and SAGE
A software component was created in order to integrate Fogo Player with SAGE.The integration component, called SAGE plug-in, acts as an adapter and converter, translating instructions that are then passed to a SAGE API.The SAGE plug-in makes use of the SAGE Application Interface Library (SAIL).The SAIL handles all communication with the Free Space Manager, which provides information to direct the flow containing blocks of pixels to the SAGE receivers.
Figure 3 depicts the communication between the Brick Player components with the player controller, each one responsible for processing a different video quadrant.Each instance of Fogo's Brick Player drives an instance of the plug-in developed for SAGE.In Figure 4, the instances of Brick Players would be represented by App1, App2, and App3 blocks.
For the drawing of multiple partitions of a video on tiled display to be synchronized, it is necessary to inform the  Free Space Manager that multiple instances of SAIL comprise a single application.One of the Brick Players is then defined as the Master, and the remaining set is defined as Slave.The IP address of the master Brick Player must be supplied when configuring Slave Brick Players.Each Brick Player instance must receive an object derived from the Graphic Engine class before the start of video reproduction.Objects of this class expose methods to create windows and to draw into window surfaces.If the Fogo Player is configured to delegate the drawing operations to the SAGE, an instance of the SAGE plug-in (which is derived from class Graphic Engine) will be responsible for the drawing operations.
The drawing operation consists of two steps: "blit" and "flip".During the first step, a decoded image is copied to the drawing buffer contained in the Surface object, which is obtained from the object of type Graphic Engine.In the second step, the drawing buffers are swapped, resulting in exhibition of the image on the window.When the SAGE plug-in is used, an object of the SAIL type divides the image into pixel blocks to be sent to SAGE Receivers successively, each of the receivers being responsible for a different region of the Tiled Display that corresponds with the pixel blocks.Before sending the pixel blocks, each instance of SAIL keeps the "flip" execution blocked until all other instances of SAGE plug-in are ready to send these blocks accordingly.

8K Content
At the end of 2013, due to the difficulties mentioned in section 1 regarding the 8K film content, and also due to the high cost of production of such pieces, the team that developed the integration of Fogo Player with SAGE had to create some 8K video samples based on two different low cost approaches.
The first approach was to capture a large set of pictures using the 41 Megapixel camera from a Nokia Lumia 1020 phone, in order to produce a time-lapse 8K video.The second approach was to render some Adobe After Effects vector animations with 8K resolution.The production of the 8K videos using both approaches was based, then, in a set of JPEG 8K frames, which were sliced in 16 parts with 1920 × 1080 pixels (HD).
Each part (a different quadrant) was used to compose a different video using FFmpeg [11].Different frame rates were tested (from 24 to 30 fps) and a set of 8K videos with durations varying from 20 to 60s were produced.Those videos are now being used for 8K exhibition tests and demonstrations.The produced videos will soon be provided as royalty free pieces, in order to foster the spread of UHD content and the development of related tools.

Conclusions and Future Work
The applications described in this article can improve SAGE interactive experiences for educational and scientific research purposes by enhancing the quality of the digital media videos.The integration of SAGE with Fogo was successfully demonstrated at the 8th Annual International CineGrid Workshop and the 8K videos presented at the venue were previously spatially sliced into pieces of 1920 × 1080 and then encoded using FFmpeg and the H.264 codec, and finally packaged using MPEG-2 TS, resulting in 16 files with ".ts" extension.
The Fogo Player team plans to develop a video integration of the Fogo Player with SAGE to run 8K 3D videos, a feature that is still under development for future implementations.