首页 » 说说 » Google的去码技术【RAISR】-有转载论文

最后更新于 2018年04月11日

谷歌最近发布了神奇的RAISR技术论文,利用网络资源和电脑算法进行图像重绘,这种还是很玄学的,现在这个技术只是发布了论文,正真的程序本身暂时还没有提供下载,所以大家就不要太过期待了,正所谓“解码还需打码人”啊!年轻人要节制啊,要是真的这样了,说不定以后会发布新的打码技术呢!保护双眼从我做起!


中文版论文(来自雷锋网)

  每天都有数以百万计的图片在网络上被分享、储存,用户借此探索世界,研究感兴趣的话题,或者与朋友家人分享假期照片。问题是,大量的图片要嘛被照相设备的像素所限制,要嘛在手机、平板或网络限制下被人为压缩,降低了画质。

  如今高分辨率显示屏幕正在家庭和移动设备上普及,因此,把低分辨率图片转化为高清晰版本,并可在多种设备上查看和分享,正在成为一项巨大的需求。日前,Google 推出了一项新技术 RAISR,其全称是“Rapid and Accurate Image Super-Resolution”,意为“快速、精确的超分辨率技术”。

  RAISR 这项技术能利用机器学习,把低分辨率图片转为高分辨率图片。它的效果能达到甚至超过现在的超分辨率解决方案,同时速度提升大约 10 至 100 倍,且能够在普通的移动设备上运行。而且,Google 的技术可以避免产生混叠效应(aliasing artifacts)。

  之前已经具有透过升采样方式,把低分辨率图片重建为尺寸更大、像素更多、更高画质图片的技术。最广为人知的升采样方式是线性方法,即透过把已知的像素值进行简单、固定的组合,以添加新的像素值。因为使用固定的线性过滤器(一个恒定卷积核对整个图片的无差别处理),该方法速度很快。但是它对于重建高清作品里生动的细节有些力不从心。正如下面这张图片,升采样的图片看起来很模糊,很难称得上画质提升。

  RAISR

  ▲ 左为原始图片;右为升采样处理后图片。

  对于 RAISR,Google 另辟蹊径得采用机器学习,用一对低分辨率、高分辨率图片训练该程序,以找出能选择性应用于低分辨率图片中每个像素的过滤器,这样能生成媲美原始图片的细节。目前有两种训练 RAISR 的方法:

第一种是“直接”方式,过滤器在成对高、低分辨率图片中直接学习。
第二种方法需要先对低分辨率图片应用低功耗的的升采样,然后在升采样图片和高分辨率图片的组合中学习过滤器。
“直接”方式处理起来更快,但第二种方法照顾到了非整数范围的因素,并且更好地利用硬件性能。

  无论是哪种方式,RAISR 的过滤器都是根据图像的边缘特征训练的:亮度和色彩梯度、平实和纹理区域等。这又受到方向(direction,边缘角度)、强度(strength,更锐利的边缘强度更高)和黏性(coherence,一项量化边缘方向性的指标)的影响。以下是一组 RAISR 过滤器,从一万对高、低分辨率图片中学习得到(低分辨率图片经过升采样)。该训练过程耗费约 1 小时。

  RAISR

  注:3 倍超分辨率学习,获得的 11×11 过滤器集合。过滤器可以从多种超分辨率因素中学习获得,包括部分超分辨率。注意当图中边缘角度变化时,过滤器角度也跟着旋转。相似的,当强度提高时,过滤器的锐利度也跟着提高;黏性提高时,过滤器的非均相性(anisotropy)也提高。

  从左至右,学习得到的过滤器与处理后的边缘方向有选择性的呼应。举例来说,最底一行中间的过滤器最适合强水平边缘(90 度梯度角),并具有高黏性(直线的而非弯曲的边缘)。如果这个水平边缘是低对比度的,那么如同图中最上一行,另一个过滤器就被选择。

  实际使用中,RAISR 会在已经学习到的过滤器列表中选择最合适的过滤器, 应用于低分辨率图片的每一个像素周围。当这些过滤器被应用于更低画质的图像时,它们会重建出相当于原始分辨率的细节,这大幅优于线性、双三(bicubic)、兰索斯(Lancos)解析方式。

  RAISR

  ▲ RAISR 演算法运行图式下:原始图像(左),2 倍双三解析(中),RAISR 效果(右)。

  一些运用 RAISR 进行图片增强的范例:

  RAISR

  ▲ 上:原始图片,下:RAISR 2 倍超分辨率效果。

  RAISR

  ▲ 左:原始图片,右:RAISR 3 倍超分辨率效果。

  超分辨率技术更复杂的地方在于如何避免混叠效应,例如龟纹(Moire patterns)和高频率内容在低分辨率下渲染产生的锯齿(对图像人为降级的情形)。这些混叠效应的产物会因对应部分的形状不同而变化,并且很难消除。

  RAISR

  ▲ 左:正常图像;右:右下角有龟纹(混叠效应)的图像。

  线性方法很难恢复图像结构,但是 RAISR 可以。下面是一个例子,左边是低分辨率的原始图片,左 3 和左 5 有很明显的空间频率混淆(aliased spatial frequencies),而右侧的 RAISR 图像恢复了其原始结构。RAISR 的过滤器学习方法还有一项重要的优点:用户可以把消除噪音以及各类压缩演算法的产物做为训练的一部分。当 RAISR 被提供相应的范例后, 它可以在图片锐化之外学会消除这些效果,并把这些功能加入过滤器。

  

  ▲ 左:有强混叠效应的原始图片;右:RAISR 处理后效果。

  超分辨率技术利用不同的方法已经有了不少进展。如今,透过把机器学习与多年来不断发展的成像技术相结合,图像处理技术有了长足的进步,并带来许多好处。举例来说,除了放大手机上的图片,用户还可以在低分辨率和超高清下捕捉、储存、传输图像,使用更少的移动网络数据和储存空间,而且不会产生肉眼能观察到的画质降低。

  小结:自从乔布斯 2010 年在 iPhone 4s 上推出“视网膜屏幕”概念之后,数码产品市场开启了一场超高清显示革命。如今,家用显示器逐步走向 4K,各大手机厂商也竞相推出 2K 旗舰机。但 2K、4K 内容的缺乏一直是困扰行业发展的痛点。之前的超分辨率技术受成本、硬件限制,主要应用于专业领域,未能大范围普及。

  此次 Google RAISR 大幅降低了图像增强的时间成本和硬件要求,有望实现超分辨率技术在消费领域的应用,把充斥网络的低画质图片转化为高清图片,大幅提高视觉效果和用户体验。雷锋网十分期待将来 RAISR 在移动设备的应用,例如把消费者手机拍摄的照片转化为媲美单反画质的高清美图。

RAISR: Rapid and Accurate Image Super Resolution
Enhance! RAISR Sharp Images with Machine Learning

英文原文

Enhance! RAISR Sharp Images with Machine Learning
Monday, November 14, 2016
Posted by Peyman Milanfar, Research Scientist Everyday the web is used to share and store millions of pictures, enabling one to explore the world, research new topics of interest, or even share a vacation with friends and family. However, many of these images are either limited by the resolution of the device used to take the picture, or purposely degraded in order to accommodate the constraints of cell phones, tablets, or the networks to which they are connected. With the ubiquity of high-resolution displays for home and mobile devices, the demand for high-quality versions of low-resolution images, quickly viewable and shareable from a wide variety of devices, has never been greater. With “RAISR: Rapid and Accurate Image Super-Resolution”, we introduce a technique that incorporates machine learning in order to produce high-quality versions of low-resolution images. RAISR produces results that are comparable to or better than the currently available super-resolution methods, and does so roughly 10 to 100 times faster, allowing it to be run on a typical mobile device in real-time. Furthermore, our technique is able to avoid recreating the aliasing artifacts that may exist in the lower resolution image. Upsampling, the process of producing an image of larger size with significantly more pixels and higher image quality from a low quality image, has been around for quite a while. Well-known approaches to upsampling are linear methods which fill in new pixel values using simple, and fixed, combinations of the nearby existing pixel values. These methods are fast because they are fixed linear filters (a constant convolution kernel applied uniformly across the image). But what makes these upsampling methods fast, also makes them ineffective in bringing out vivid details in the higher resolution results. As you can see in the example below, the upsampled image looks blurry – one would hesitate to call it enhanced.
请输入图片描述
Left: Low-res original, Right: simple (bicubic) upsampled version (2x). Image Credit: Masa Ushioda/Seapics/Solent News
With RAISR, we instead use machine learning and train on pairs of images, one low quality, one high, to find filters that, when applied to selectively to each pixel of the low-res image, will recreate details that are of comparable quality to the original. RAISR can be trained in two ways. The first is the "direct" method, where filters are learned directly from low and high-resolution image pairs. The other method involves first applying a computationally cheap upsampler to the low resolution image (as in the figure above) and then learning the filters from the upsampled and high resolution image pairs. While the direct method is computationally faster, the 2nd method allows for non-integer scale factors and better leveraging of hardware-based upsampling. For either method, RAISR filters are trained according to edge features found in small patches of images, - brightness/color gradients, flat/textured regions, etc. - characterized by direction (the angle of an edge), strength (sharp edges have a greater strength) and coherence (a measure of how directional the edge is). Below is a set of RAISR filters, learned from a database of 10,000 high and low resolution image pairs (where the low-res images were first upsampled). The training process takes about an hour.
请输入图片描述
Collection of learned 11x11 filters for 3x super-resolution. Filters can be learned for a range of super-resolution factors, including fractional ones. Note that as the angle of the edge changes, we see the angle of the filter rotate as well. Similarly, as the strength increases, the sharpness of the filters increases, and the anisotropy of the filter increases with rising coherence.
From left to right, we see that the learned filters correspond selectively to the direction of the underlying edge that is being reconstructed. For example, the filter in the middle of the bottom row is most appropriate for a strong horizontal edge (gradient angle of 90 degrees) with a high degree of coherence (a straight, rather than a curved, edge). If this same horizontal edge is low-contrast, then a different filter is selected such one in the top row. In practice, at run-time RAISR selects and applies the most relevant filter from the list of learned filters to each pixel neighborhood in the low-resolution image. When these filters are applied to the lower quality image, they recreate details that are of comparable quality to the original high resolution, and offer a significant improvement to linear, bicubic, or Lanczos interpolation methods.
请输入图片描述
Top: RAISR algorithm at run-time, applied to a cheap upscaler’s output. Bottom: Low-res original (left), bicubic upsampler 2x (middle), RAISR output (right)
Some examples of RAISR in action can be seen below:
请输入图片描述
Top: Original, Bottom: RAISR super-resolved 2x. Original image from Andrzej Dragan
请输入图片描述
Left: Original, Right: RAISR super-resolved 3x. Image courtesy of Marc Levoy
One of the more complex aspects of super-resolution is getting rid of aliasing artifacts such as Moire patterns and jaggies that arise when high frequency content is rendered in lower resolution (as is the case when images are purposefully degraded). Depending on the shape of the underlying features, these artifacts can be varied and hard to undo.
请输入图片描述
Example of aliasing artifacts seen on the lower right (Image source)
Linear methods simply can not recover the underlying structure, but RAISR can. Below is an example where the aliased spatial frequencies are apparent under the numbers 3 and 5 in the low-resolution original on the left, while the RAISR image on the right recovered the original structure. Another important advantage of the filter learning approach used by RAISR is that we can specialize it to remove noise, or compression artifacts unique to individual compression algorithms (such as JPEG) as part of the training process. By providing it with examples of such artifacts, RAISR can learn to undo other effects besides resolution enhancement, having them “baked” inside the resulting filters.
请输入图片描述
Left: Low res original, with strong aliasing. Right: RAISR output, removing aliasing.
Super-resolution technology, using one or many frames, has come a long way. Today, the use of machine learning, in tandem with decades of advances in imaging technology, has enabled progress in image processing that yields many potential benefits. For example, in addition to improving digital “pinch to zoom” on your phone, one could capture, save, or transmit images at lower resolution and super-resolve on demand without any visible degradation in quality, all while utilizing less of mobile data and storage plans. To learn more about the details of our research and a comparison to other current architectures, check out our paper, which will appear soon in the IEEE Transactions on Computational Imaging.

原文链接!

  • weixiao kaixin tushetou jingkong deyi fanu liezui liuhan daku ganga bishi nanguo lihai qian yiwen numu tu yi haixiu se fadai minyan hehe henkaixin huaji biyiyan kuanghan maimeng shui xiaku penqi zhangzui pen aini ye niu laji ok chigua renshi kongbu shuai xiaoxiese touxiao huaixiao jingnu chihuai kaisang xiaoku koubi zhuangbi lianhong kanbujian shafa zhijing xiangjiao dabian yaowan redjing lazhu rizhi duocang chixigua hejiu xixi xiaopen goukun xiaobuchu shenme wusuowei guancha lajing chouyan xiaochi bie zhadanzui zhadanxiao