CMU OpenFace

关于OpenFace的各种依赖库dependencies 的安装配置部分，可以参见之前的文章dlib, OpenFace and face_recognition。
下面纪录一下测试OpenFace本身的各种应用时遇到的问题。

Demo 4: Real-Time Face Embedding Visualization

Running Command
python demos/sphere.py --networkModel nn4.small2.3d.v1.t7

1) NameError: name ‘xrange’ is not defined

File "demos/sphere.py", line 244/259, in <module>
    for i in xrange(len(trackers) - 1, -1, -1):
NameError: name 'xrange' is not defined

NameError: global name ‘xrange’ is not defined in Python 3

xrange() was renamed to range() in Python 3.

2)AttributeError: module ‘cv2’ has no attribute ‘cv’

File "demos/sphere.py", line 279, in <module>
    (0, 0, 0), 1, cv2.cv.CV_AA)
AttributeError: module 'cv2' has no attribute 'cv'
Cleaned up camera.

What is the cv2.cv replacement in OpenCV3?

From OpenCV 2.X OpenCV 3.0 a few things changed.
Specifically:

• cv2.cv doesn’t exists in OpenCV 3.0. Use simply cv2.
• some defines changed, e.g. CV_BGR2HSV is now COLOR_BGR2HSV.

Videofacerec Error #38
FROM cv2.cv.CV_AA or cv2.CV_AA TO cv2.LINE_AA

OpenCV 2–> 3 Changes overview

3) TypeError: a bytes-like object is required, not ‘str’

File "demos/sphere.py", line 231, in <module>
    rep = net.forward(alignedFace)
  File "~/anaconda/lib/python3.5/site-packages/openface/torch_neural_net.py", line 205, in forward
    rep = self.forwardPath(t)
  File "~/anaconda/lib/python3.5/site-packages/openface/torch_neural_net.py", line 164, in forwardPath
    self.p.stdin.write(imgPath + "\n")
TypeError: a bytes-like object is required, not 'str'

Python 3 TypeError: must be str, not bytes with sys.stdout.write()

Python 3 handles strings a bit different. Originally there was just one type for strings: str. When unicode gained traction in the ’90s the new unicode type was added to handle Unicode without breaking pre-existing code1. This is effectively the same as str but with multibyte support.

In Python 3 there are two different types:

• The bytes type. This is just a sequence of bytes, Python doesn’t know anything about how to interpret this as characters.
• The str type. This is also a sequence of bytes, but Python knows how to interpret those bytes as characters.
• The separate unicode type was dropped. str now supports unicode.

This change is incompatible with Python 2 as many return values have changed, leading to subtle problems like this one; it’s probably the main reason why Python 3 adoption has been so slow. Since Python doesn’t have static typing[2] it’s impossible to change this automatically with a script (such as the bundled 2to3).

• You can convert str to bytes with bytes(‘h€llo’, ‘utf-8’); this should produce b’H\xe2\x82\xacllo’. Note how one character was converted to three bytes.
• You can convert bytes to str with b’H\xe2\x82\xacllo’.decode(‘utf-8’)

In your specific piece of code, nextline is of type bytes, not str, reading stdout and stdin from subprocess changed in Python 3 from str to bytes. This is because Python can’t be sure which encoding this uses. It probably uses the same as sys.stdin.encoding (the encoding of your system), but it can’t be sure.

4) Final fix
"~/anaconda/lib/python3.5/site-packages/openface/torch_neural_net.py",

line 206,
rep = self.forwardPath(bytes(str(t+"\n"), sys.stdin.encoding))

line 164,
output1 = self.p.stdout.readline()
output = output1.decode(sys.stdout.encoding)

5) NOTE
line 231, rep = net.forward(alignedFace)

net = openface.TorchNeuralNet(args.networkModel,
        imgDim=args.imgDim,  cuda=args.cuda)
[torch_neural_net.py]

alignedFace = align.align(96, frameSmall, bb,landmarkIndices=openface.AlignDlib.INNER_EYES_AND_BOTTOM_LIP)

return: The aligned RGB image. Shape: (imgDim, imgDim, 3)

align = openface.AlignDlib(args.dlibFacePredictor)[align_dlib.py]

Demo 1: Real-Time Web Demo (1)

Running Command
./demos/web/start-servers.sh

1) ImportError: No module named ‘txaio’

File "./demos/web/websocket-server.py", line 22, in <module>
    import txaio
ImportError: No module named 'txaio'

what is txaio

txaio is a helper library for writing code that runs unmodified on both Twisted and asyncio / Trollius.
The API is identical whether you’re using Twisted or asyncio under the hood.
Until you explicitly select a framework, all txaio API methods just throw a usage error. So, you must call .use_twisted() or .use_asyncio() as appropriate.

import txaio
txaio.use_twisted()
txaio.use_asyncio()

Supposely should work with pip install txaio, have not tried…

Find port on https://anaconda.org/
conda install -c conda-forge txaio
takes almost no time to install

The following NEW packages will be INSTALLED:

    txaio:     2.8.2-py_0    conda-forge

The following packages will be UPDATED:

    conda:     4.3.21-py35_0             --> 4.3.29-py35_0 conda-forge

The following packages will be SUPERSEDED by a higher-priority channel:

    conda-env: 2.6.0-0                   --> 2.6.0-0       conda-forge

Proceed ([y]/n)? 

2) ImportError: No module named ‘twisted’
conda install twisted

The following NEW packages will be INSTALLED:

    appdirs:          1.4.3-py35_0                     
    asn1crypto:       0.22.0-py35ha010153_1            
    attrs:            17.2.0-py35h4a2dd8a_0            
    automat:          0.6.0-py35h93f71ca_0             
    constantly:       15.1.0-py35_0                    
    hyperlink:        17.1.1-py35_0                    
    hypothesis:       3.23.0-py35hcda8997_0            
    incremental:      17.5.0-py35h71fb944_0            
    packaging:        16.8-py35h2e4dc9b_0              
    pyasn1-modules:   0.0.8-py35_0                     
    pympler:          0.5-py35hf41ee38_0               
    service_identity: 17.0.0-py35_0                    
    twisted:          17.9.0-py35h5aaa809_0            
    zope:             1.0-py35_0                       
    zope.interface:   4.4.3-py35hdbc5dd8_0             

The following packages will be UPDATED:

    conda:            4.3.29-py35_0         conda-forge --> 4.3.30-py35hd530ce9_0
    cryptography:     1.4-py35_0                        --> 1.8.1-py35_0         

The following packages will be SUPERSEDED by a higher-priority channel:

    conda-env:        2.6.0-0               conda-forge --> 2.6.0-h36134e3_0     

3) ImportError: No module named ‘autobahn’

File "./demos/web/websocket-server.py", line 25, in <module>
    from autobahn.twisted.websocket import WebSocketServerProtocol, \
ImportError: No module named 'autobahn'

conda install -c conda-forge autobahn

The following NEW packages will be INSTALLED:

    autobahn:  17.10.1-py_0          conda-forge

The following packages will be SUPERSEDED by a higher-priority channel:

    conda:     4.3.30-py35hd530ce9_0             --> 4.3.29-py35_0 conda-forge
    conda-env: 2.6.0-h36134e3_0                  --> 2.6.0-0       conda-forge

4) ImportError: No module named ‘imagehash’
conda install -c conda-forge imagehash

The following NEW packages will be INSTALLED:

    imagehash:  3.1-py35_0   conda-forge
    pywavelets: 0.5.2-py35_1 conda-forge

5) ImportError: No module named ‘StringIO’

  File "./demos/web/websocket-server.py", line 37, in <module>
    import StringIO
ImportError: No module named 'StringIO'

dooh, NOT supported for Python 3 !!!
Python 3 support #8

Currently we only support Python 2. In future releases we want to support both Python 2 and 3. This howto outlines the main steps towards Python 3 support.

• pip install StringIO
  Collecting StringIO
  Could not find a version that satisfies the requirement StringIO (from versions: )
  No matching distribution found for StringIO

Comment out import StringIO SEEMS to run, but…

6) ImportError: No module named ‘urllib3’
conda install urllib3

The following NEW packages will be INSTALLED:

    certifi:   2017.7.27.1-py35h0fdde5e_0            
    pysocks:   1.6.7-py35h3cfcbe1_1                  
    urllib3:   1.22-py35he002d57_0                   

The following packages will be UPDATED:

    conda:     4.3.29-py35_0              conda-forge --> 4.3.30-py35hd530ce9_0

The following packages will be SUPERSEDED by a higher-priority channel:

    conda-env: 2.6.0-0                    conda-forge --> 2.6.0-h36134e3_0  

6a) builtins.NameError: name ‘urllib’ is not defined

File "./demos/web/websocket-server.py", line 127, in onMessage
        self.processFrame(msg['dataURL'], msg['identity'])
      File "./demos/web/websocket-server.py", line 361, in processFrame
        urllib.quote(base64.b64encode(imgdata.buf))
    builtins.NameError: name 'urllib' is not define

Python: Importing urllib.quote

In Python 3.x, you need to import urllib.parse.quote:

import urllib.parse
urllib.parse.quote(“châteu”, safe=”)
‘ch%C3%A2teu’

According to Python 2.x urllib module documentation:

The urllib module has been split into parts and renamed in Python 3 to urllib.request, urllib.parse, and urllib.error.

If you need to handle both Python 2.x and 3.x you can catch the exception and load the alternative.

try:
from urllib import quote # Python 2.X
except ImportError:
from urllib.parse import quote # Python 3+

OR,

from six.moves.urllib.parse import quote

NameError: name ‘urllib’ is not defined “

urllib is part of the standard library. You don’t need to install urllib3 which is a different library. Also, urlopen returns a file-like object, not the html.

7) ImportError: No module named ‘base64enc’
conda install r-base64enc

The following NEW packages will be INSTALLED:

    gsl:         2.2.1-hbdf3209_2    
    icu:         58.2-hea21ae5_0     
    libcxx:      4.0.1-h579ed51_0    
    libcxxabi:   4.0.1-hebd6815_0    
    libgcc:      4.8.5-hdbeacc1_10   
    libgfortran: 3.0.1-h93005f0_2    
    ncurses:     6.0-hd04f020_2      
    pcre:        8.41-h29eefc5_0     
    r-base:      3.3.1-0             
    r-base64enc: 0.1_3-r342h0880998_4

Demo 1: Real-Time Web Demo (2)

Now it runs, but crashes on Annotation tab.
1) object has no attribute ‘is_closed’

txaio.resolve(self.is_closed, self)
    builtins.AttributeError: 'OpenFaceServerProtocol' object has no attribute 'is_closed'

Exception is raised when Websocket server disconnected #454

Are you maybe overriding init but not calling the parent? self.is_closed is created in the constructor…

How to invoke the super constructor?

One way is to call A’s constructor and pass self as an argument, like so:

class B(A):
    def __init__(self):
        A.__init__(self)
        print "hello"

Another way is to use super(), as others have shown. For single-inheritance, it does basically the same thing as letting you call the parent’s constructor.

• However, super() is quite a bit more complicated under-the-hood and can sometimes be counter-intuitive in multiple inheritance situations. On the plus side, super() can be used to handle diamond-shaped inheritance. If you want to know the nitty-gritty of what super() does, the best explanation I’ve found for how super() works is here (though I’m not necessarily endorsing that article’s opinions).

In line with the other answers, there are multiple ways to call super class methods (including the constructor), however in Python-3.x the process has been simplified:

Python-2.x

class A(object):
 def __init__(self):
   print "world"

class B(A):
 def __init__(self):
   print "hello"
   super(B, self).__init__()

Python-3.x

class A(object):
 def __init__(self):
   print("world")

class B(A):
 def __init__(self):
   print("hello")
   super().__init__()

super() is now equivalent to super(, self) as per the docs.

2) assert(type(payload) == bytes)
io — Text, Binary, and Raw Stream I/O Tools

StringIO provides a convenient means of working with text in memory using the file API (read(), write(), etc.). Using StringIO to build large strings can offer performance savings over some other string concatenation techniques in some cases. In-memory stream buffers are also useful for testing, where writing to a real file on disk may slow down the test suite.

3) How to use SimpleHTTPServer

The SimpleHTTPServer module that comes with Python is a simple HTTP server that provides standard GET and HEAD request handlers.

• for python 3 the command is: python -m http.server
  SO, this means each time I run start-servers.sh from outside Docker container, is actually trying to connect docker’s HTTP server!!

3a) WebSocket, Autobahn, WAMP, Twisted/asyncio (see above txaio)
Web Application Messaging Protocol (WAMP)

WAMP is an open standard WebSocket subprotocol that provides two application messaging patterns in one unified protocol:
Remote Procedure Calls + Publish & Subscribe.

Autobahn Libraries

The Autobahn project is maintained by the same people who created the open source Crossbar.io WAMP router.

It provides open-source implementations of the Web Application Messaging Protocol (WAMP) for a range of languages (as well as the industry-standard WebSocket test suite).

WAMP connects components in distributed applications using Publish and Subscribe (PubSub) and routed Remote Procedure Calls (rRPC). It is ideal for distributed, multi-client and server applications such as IoT applications or multi-user database-driven business applications.

Autobahn|Python
WebSocket & WAMP for Python on Twisted and asyncio.

4) How to check opened/closed port on my computer?
How to check opened/closed port on my computer?

• -l or –listening shows only the sockets currently listening for incoming connection.
• -a or –all shows all sockets currently in use.
• -t or –tcp shows the tcp sockets.
• -u or –udp shows the udp sockets.

• -n or –numeric shows the hosts and ports as numbers, instead of resolving in dns and looking in /etc/services.

  netstat -atn # For tcp
  netstat -aun # For udp
  netstat -atun # For both

• netstat -tulnp | grep “port no”

How do I close an open port from the terminal on the Mac?

List open files on a port and copy PID

lsof -i :
Kill it with fire :)
kill -9 “PID”

killport

Simple bash script to kill a process which uses the specified port.

5) Final Fix
Basically the change between string/text type of data to bytes/binary type of data, due to type change from Python 2 to Python 3.

e.g. line 125,128, 349, 367from

self.sendMessage('{"type": "NULL"}')
self.sendMessage('{"type": "PROCESSED"}')
self.sendMessage(json.dumps(msg))

to

self.sendMessage(bytes('{"type": "NULL"}', 'utf-8'))
self.sendMessage(bytes('{"type": "PROCESSED"}', 'utf-8'))
self.sendMessage(bytes(json.dumps(msg), 'utf-8'))

AND, StringIO.StringIO()–> BytesIO()

6) WHY
SO, the logic behind all these conversions are as follows:
on receiving end,

• Image Module in Pillow can open image file, or a file object.

  • Parameters:
    fp – A filename (string), pathlib.Path object or a file object. The file object must implement read(), seek(), and tell() methods, and be opened in binary mode.

• Create file-like object using BytesIO in Python 3

  • 16.2. io — Core tools for working with streams
  • The io module provides Python’s main facilities for dealing with various types of I/O. There are three main types of I/O: text I/O, binary I/O and raw I/O. These are generic categories, and various backing stores can be used for each of them. A concrete object belonging to any of these categories is called a file object. Other common terms are stream and file-like object.

• decode received data from WebSocket and then copy/write to BytesIO object

E.g.

imgdata = base64.b64decode(dataURL[len(head):])
imgF = BytesIO()
imgF.write(imgdata)
imgF.seek(0)
img = Image.open(imgF)

a text stream

f = open("myfile.txt", "r", encoding="utf-8")

In-memory text streams are also available as StringIO objects:

f = io.StringIO("some initial text data")

a binary stream

f = open("myfile.jpg", "rb")

In-memory binary streams

f = io.BytesIO(b"some initial binary data: \x00\x01")

a raw stream by opening a file in binary mode with buffering disabled:

f = open("myfile.jpg", "rb", buffering=0)

Similarly, on sending end,

• create BytesIO object

• save figure to in-memory file

  • matplotlib.pyplot.savefig
  • fname:
    A string containing a path to a filename, or a Python file-like object, or possibly some backend-dependent object such as PdfPages.

• encoded and send

  • content = 'data:image/png;base64,' + \
urllib.parse.quote(base64.b64encode(imgdata.getvalue()))

how to save a pylab figure into in-memory file which can be read into PIL image?

Remember to call buf.seek(0) so Image.open(buf) starts reading from the beginning of the buf:

import io
from PIL import Image
import matplotlib.pyplot as plt

plt.figure()
plt.plot([1, 2])
plt.title("test")
buf = io.BytesIO()
plt.savefig(buf, format='png')
buf.seek(0)
im = Image.open(buf)
im.show()
buf.close()

matplotlib savefig() plots different from show()

You render your matplotlib plots to different devices (e.g., on-screen via Quartz versus to to-file via pdf using different functions (plot versus savefig) whose parameters are nearly the same, yet the default values for those parameters are not the same for both functions.

Put another way, the savefig default parameters are different from the default display parameters.

Aligning them is simple if you do it in the matplotlib config file. The template file is included with the source package, and named matplotlibrc.template. If you did not create one when you installed matplotlib, you can get this template from the matplotlib source, or from the matplotlib website.

Once you have customized this file the way you want, rename it to matplotlibrc (no extension) and save it to the directory .matplotlib (note the leading ‘.’) which should be in your home directory.

The config parameters for saving figures begins at about line 314 in the supplied matplotlibrc.template (first line before this section is: ### SAVING FIGURES).