When Tele-Vision Becomes Inter-Vision: Smart TV Test Report

No More Couch Potato

Now we can say that the future Internet technology and application will move toward the development of mobile entertainment devices and home entertainment devices. In “Mobile” oriented, mobile entertainment devices such as smartphone and tablet PC have significantly and successfully caused the big bang explosion in market; in “Home” oriented, lean-back media such as smart TV, connected TV and set-top box invade our living room gradually, and then users just sit down and veg out watching TV and browsing Internet. That is, the user scenario of TV is no longer passively receiving information, but users are able to take the initiate to select, browse and operate various TV applications and services.

The development of TV is perhaps not rapid as mobile devices; however, whether global Internet/system vendors or information/content suppliers are constantly exploring the potential business model. At CES 2011, 2012 and even 2013, all major tech vendors are talking about innovative features for their smart TVs in an attempt to differentiate them. Predictably, with the growth of smart TV technology and usability, the smoothness of human-computer interactions with menus and applications vitally contributes to the better user experience (UX). At this critical moment, Allion Labs, Inc. (Allion) particularly purchases some international brand smart TVs for comprehensive competitive analysis of their technology, interface, performance and usability. It can help users get their ideal smart TV with the most consideration and dominance; after that, there are no more couch potatoes left in the living room.

Smart TV Competitive Analysis

We select two Japanese-related & Korean-related smart TV models as the test targets (Brand A and Brand B), and then implement functionality test and usability evaluation according to the following test scope.

I. Functional Signal Test of Basic I/O Interface

II. Power Consumption Test

III. Ecosystem Test

IV. User Experience

     i. Graphic User Interface (GUI) Analysis

    ii. Usability Test

   iii. Smart Function and Brower Performance Test

Initially, we compare the announced product feature of both smart TVs (Table I).

In the traditional TV functions, we find that Brand A is better than Brand B with years of advantages. But in the smart functions, Brand B has more innovative highlights and breakthroughs, including face recognition, voice activated, motion control and mobile phone synchronization.

I. Functional Signal Test of Basic I/O Interface

In this part, we implement functional signal tests of basic I/O interfaces, including Tuner, USB, HDMI, DLNA and so on. Then, we find that Brand B will appear the condition of “Channel Lost”, which can be improved by adjusting the antenna manually. For common end-users; however, they may not know how to rule out the similar problem. At HDMI test, “Vertical Line Noise” could be seen on Brand B. At DLNA test, Brand A cannot discover the DLNA server from wireless network after executing the Wi-Fi Direct function; also, it cannot playback the JPG file (Photoshop) because of its lake of decoding picture file properly.

II. Power Consumption Test

Based on the default settings of both TVs, there are significant differences in their “Brightness Power Ratios”. Particularly, Brand B has twice the brightness than Brand A (Brand A: 202 nits/ Brand B: 497 nits), so it will indirectly affect the higher power consumption of Brand B. However, when we do different brightness adjustment and observe the power consumption, we find the brightness power ratio of Brand B is better overall than Brand A (Table II).

III. Ecosystem Test

The function of traditional TV is only broadcasting, and the peripheral devices are mostly CD or DVD players. However, because smart TV has more interfaces which can connect to multiple peripheral devices, we choose different types of peripherals such as Blu-ray player, USB drive, USB keyboard and mouse to connect with both TVs for ecosystem test (Table III).

There are more compatible issues on Brand A. First, some Blu-ray players reject Brand A standby CEC command; next, Brand A is unable to read USB storage with NTFS file system, that means Brand A can’t playback the file size which is larger than 4GB (HD movies are averagely more than 4GB). Besides, Brand A is unable to recognize generic USB keyboard and mouse which can increase the usability and satisfaction for operating smart TV. It is one critical user experience issue due to a large number of interactive user scenarios with smart TV.

IV. User Experience

i. Graphic User Interface (GUI) Analysis

We all know that user experience (UX) is not totally new for high-tech industry, but how to optimize the interaction and usability of smart TV will be the key success factors. Like the development of mobile devices, smart TVs seem to integrate all features through Internet, smartphone, tablet PC and TV; meanwhile, the content and function layouts on large size screen becomes very important. Learning the successful interface model of mobile device or developing the brand-new smart TV interface is a question for all vendors, but it’s no doubt that the true feelings and feedbacks of real users are able to strengthen the usability and satisfaction of visual, audio or even emotional reflection, and then to seize the initiative in smart TV market. According to the following Figure I, we can find that Brand A and Brand B have extremely different UI and GUI.

Brand A UI is basically a link to its own cloud entertainment network “SEN”, and is somewhat similar to Windows 8 Modern UI style with box icons. The left side shows TV signal program, the central and right sides are the blocks of application icons, so users can organize their favorite applications. In fact, UI and GUI of Brand A is kind of “Simple Style”, no extra option or icon to interfere with user’s visual cognitive process.

Otherwise, Brand B UI is similar to today’s Android smartphone interface, especially the layout of applications, and GUI is more diverse and rich. The upper-left side is TV signal program, the central block is for Brand B’s customized applications, including learning, sports and etc., and the below side is the arrangement for other applications. Although the overall interface seems fancy, but for users who have been accustomed to the smartphone interface, it should not cause too big burden on the switch of mental model. For users who never used smartphone; nevertheless, the designers have to implement the usability test and evaluation to understand the UI and GUI are easy to use/learn or not.

ii. Usability Test

From these two different UI and GUI designs on smart TVs, we can propose one critical question, “the interface design of smart TV should be Simple or Diverse?” It is worth noting that the TV screen is not only larger than samrtphone or computer, but also its user scenario is unlike other electronic devices which are operated by users in front of the screen. In fact, smart TV is such a lean-back medium which has an image of user relaxing on a couch with veging out browsing and playing applications, so the design of UI and GUI should be considered the interactive process between user scenario and visual cognition. For this issue, we implement the usability test for Brand A and Brand B to observe the effect of both interface designs with remote control on user experience and human-computer interaction.

With the overall interface usability test and satisfaction questionnaire results, we find that Brand A’s simple interface design emphasizes clear and visible style, allowing users to understand the locations of all function intuitively, so we can find that the speed and quality of task completion are rather better. Conversely, Brand B’s diverse interface design is similar to Android system, and the GUI is more vivid. Although it will increase users’ visual loading at the first sight, most of them gradually are able to get started with the adaptation after one task learning. On the one hand, Brand A’s simple interface design also means too boring for users to catch their eyes, and most users feel its interface is not innovative and smart enough. On the other hand, Brand B’s diverse interface design gets better users’ satisfaction and feedback, which indirectly means that users’ objective task performances are not equal to the subjective emotional feelings. Such the result is able to reflect our previous question, “Simple” or “Diverse” interface design indeed have different effect for users.

Based on the contradictory results, vendors, especially in the design of smart TV interface, must do more researches and evaluations on such trade-offs. What smart TV interface can make users feel easy-to-use and emotional satisfaction to achieve the golden cross? Think outside-the-box with unusual design concept should be focused in the future development. In addition, one psychologist said, “The positive emotion is the key for learning, curiosity and creative thinking; that is, happiness can extend the thinking to be improved more creative.” If the interface indeed attracts users initially, the human-computer interaction process will be regarded as a positive motivation for learning even though it’s kind of difficult to use.

iii. Smart Function, Application and Brower Performance Test

In order to understand the performance of both smart TVs, we make some basic check of their smart functions (1^st initiate time required and follow-up initiate time required), applications (YouTube and Skype) and browser access abilities (Table V).

According to Table V, there is no significant difference for both smart TVs on 1^st initiate time required of smart function, but at the follow-up initiate time required, Brand A is almost 1.1 seconds slower than Brand B because it should re-link to SEN every time while initiating. In fact, when users operate smart TV, the user scenario is possibly between watching TV program and using smart function; thus, the interface switch of TV should be more smooth and faster, or it will result in users wait too long to lose patience. And for the usage of applications such as YouTube and Skype, there are also no significant differences.

Next, we particularly use PC, mobile device, Brand A and Brand B to access to some world-renowned websites, and check the correctness of the opened page (Figure II). In this test, we discover two critical issues on Brand A’s browser. One is that Brand A’s networking is faster than Brand B’s while accessing to Amazon and Amazon.cn, but we observe some website functions or advertising icons cannot be fully displayed on Brand A’s browser. After comparing with the webpage display on PC and Brand B, we can confirm that there may be some shortages of Brand A’s HTML or webpage syntax errors.

The other issue is that Brand A’s opened page would automatically switch to the mobile browser mode while accessing to ASOS and Brand B’s official website. According to Figure III, we observe that Brand A’s opened page is significantly different from PC’s, Brand B’s and even mobile device’s view; besides, the whole opened page has some blocks shift happened. All above mean that Brand A’s networking capability still needs to be adjusted to allow users have the similar and familiar browsing experience and operational feedback as PC.

Artificial Intelligence or Artificial Smart

The development of artificial intelligence creates a breakthrough of contemporary high-technology, and more and more electronic devices are attached the so-called “Smart” to strengthen their own capabilities, such as smartphones, smart TV and even smart homes. Nevertheless, all those high-tech devices may bring various kinds of conveniences for human life, but if users have never been able to operate them correctly and effortlessly, or even been dominated by them, to be Smart or not to be Smart, that is the question. So far, the representative of Smart cannot depart from Internet, and it seems like electronic devices with networking features can be unlimitedly regarded as Smart devices. Actually, the electronic devices with Internet have broken through the limitations of space; through critical years of development, they have also created numbers of applications to meet users’ requirements.

As mentioned previously, the successful development of smartphone has brought the revolution of high-tech device usage, and has enhanced better and better user experience and interface. If we want this successful model to be transferred to smart TV, we must deeply research TV’s user scenario and current technological development. After all, the traditional remote control for users is still for watching TV programs, but it is not the suitable interface for networking or browsing. Therefore, many vendors have developed lots of operational modes, including voice activated, touch screen, mouse/keyboard connected and motion control. All above would like to provide users with more operational modes, so that the processes of watching TV and using smart TV can be more flexible to increase uses’ sensational pleasure.

However, only hardware upgraded is not enough, how to make the seamless connection to other devices has also become an innovative solution. Especially, there are thousands of electronic devices in our daily life, so the convergence of user experience will be able to make users feel one device in hand and ubiquitous world around. Take smartphone as an example, we can use it to remote control or share files on TV. Due to the OS of smartphone and smart TV has become similar, and the innovation of applications and wireless network technologies such as Wi-Fi and DLNA , smartphone replaces remote control is expectable, and makes up for the lack of the usability of remote control. In addition, the development of smart TV cannot just focus on the technology, user experience, such as easy-to-use and easy-to-learn, and diversified content are rather essential as well. If the technological development seems to be smarter without user’s real feedback, the device is only “Artificial Smart”. Therefore, through all aspects of user experience researches are able to solve the issues of unfriendly and unfamiliar user interface, so that “Artificial Intelligence” is not a slogan, but an ideal human technology.

Form the point of view, the development of smart TV involves technology, interface and content. As an expert in the field of smart TV validation, Allion combines our testing experience with user experience research to provide this report. We can not only ensure the best technical quality, but also propose the most considerable suggestions greatly contributed to the best user experience. The report, which takes two vendors’ smart TV as examples, wants to provide users with the deeply thinking on smart TV competitive analysis. What to expect when we are expecting.