Unity: CEF Proxy

I used a proxy with a custom build of [CEF] to send speech data to the [Word Detection Unity Plugin].

Copy a recent TestApp build from `Branch 2704` (Windows 64-bit) from [Chromium Embedded Framework 3 Builds] into the `CefGlue.Demo.WinFormsProxy` output folder to get all the dependent libraries.

Or download a build of the [SpeechProxy].

Quick Start Guide:

1. Command-line arguments are used to enable speech detection

CefGlue.Demo.WinFormsProxy.exe --enable-speech-input

2. Use the help menu to [acquire keys] for the Speech API

3. Enable the `Speech API Private API`

4. The Speech API has a `queries per day` quota.

5. Assign the Speech API keys

6. Open the [Chrome Speech Demo] to verify that the Speech API is working

7. The [Google Developers Console] shows traffic hitting the Speech API.

8. The Unity Proxy Speech Detection Plugin communicates with the Speech Proxy (email support@theylovegames.com to get access to the beta)

9. Use a proxy port that matches the port in Unity

10. The proxy URL should be pointed at the HTTPS proxy script https://theylovegames.com/speech_proxy/ when connecting to Unity.

Chromium Embedded Framework

The [Chromium Embedded Framework (CEF)] is a simple framework for embedding Chromium-based browsers in other applications.

[cef overview]

[Xilium.CefGlue] is a .NET/Mono binding for The Chromium Embedded Framework (CEF).

[project website]

[Chromium Embedded Framework 3 Builds]

The CEF [Add support for the Web Speech API] feature requires that [api keys] are generated to work and are capped at [50 requests].

[Google Cloud Speech Samples]

[Cloud Speech Discussion Group]

[SimpleUnityBrowser]

Forums: [UnityCEF]

Forums: [CEF Speech API]:

It is NOT possible to get additional quota for Chrome’s Speech API. Look at the [Cloud Speech API] instead.

Do NOT post to any Chromium groups/mailing lists for questions about the Speech API.

[Google Developers Console]

[Chrome Speech Demo]

Source: [fork]

[Communication between C# and JavaScript with CefGlue]

[JS Bindings]

Raspberry PI 2 with Leap

The Leap Motion Controller requires ARM-9 or better and to make work with the Raspberry PI 2 I used a proxy HTTP server to work with the Raspberry PI 2. The Raspberry PI 2 controls servos based on the data from the leap. Each finger on the leap controls a different servo.

The project in action:

Details about the code:

The proxy sends JSON data for the finger rotations (in degrees).

{
  "thumb": 27.815885630721692,
  "index": 8.8111549114070726,
  "middle": 16.216426372741033,
  "ring": 29.267951404867844,
  "pinky": 86.043786182477533
}

The script: `LeapServos.py`

#!/usr/bin/env python

import RPi.GPIO as GPIO
import datetime
import time
import urllib2
import threading
import json

servo_pin2 = 18
servo_pin3 = 22

# 60 degrees / 0.1seconds 
servo_speed = 0.1

GPIO.setwarnings(False)
GPIO.setmode(GPIO.BOARD)

GPIO.setup(servo_pin2, GPIO.OUT)
GPIO.setup(servo_pin3, GPIO.OUT)

last_time = datetime.datetime.now() 
current_time = datetime.datetime.now() 

accuracy = 0.01
rotation1 = 90
rotation2 = 90
rotation3 = 90
rotation4 = 90
rotation5 = 90
timeRotation1 = 0
timeRotation2 = 0
timeRotation3 = 0
timeRotation4 = 0
timeRotation5 = 0
targetRotation1 = 90
targetRotation2 = 90
targetRotation3 = 90
targetRotation4 = 90
targetRotation5 = 90

pulse2 = GPIO.PWM(servo_pin2, 50)
pulse3 = GPIO.PWM(servo_pin3, 50)

logTime = datetime.datetime.now()

stayAlive = True

def getRotation(rotation):
  if (rotation > 90.0):
   return 180
  if (rotation > 45.0):
   return 90
  else:
   return 0

def newTarget():
 global targetRotation1
 global targetRotation2
 global targetRotation3
 global targetRotation4
 global targetRotation5
 while (stayAlive):
  content = urllib2.urlopen("http://192.168.2.97").read()
  #print (content)
  jsonObject = json.loads(content) 
  #print (jsonObject)
  #for key in jsonObject:
  # print (key)
  #print ("index: " + str(jsonObject["index"]))
  targetRotation1 = getRotation(jsonObject["thumb"])
  targetRotation2 = getRotation(jsonObject["index"])
  targetRotation3 = getRotation(jsonObject["middle"])
  targetRotation4 = getRotation(jsonObject["ring"])
  targetRotation5 = getRotation(jsonObject["pinky"])
  time.sleep(0)
 print ("Thread complete.")
 return

webThread = threading.Thread(target=newTarget, args=(), kwargs={})
webThread.start()

def log():
 global logTime
 global timeRotation2
 global timeRotation3
 global targetRotation2
 global targetRotation3
 if (logTime < datetime.datetime.now()):
  logTime = datetime.datetime.now() + datetime.timedelta(0, 0.5)
  #print "****"
  #print ("2: TargetRotation: " + str(targetRotation2) + " Time: "+str(timeRotation2))
  #print ("3: TargetRotation: " + str(targetRotation3) + " Time: "+str(timeRotation3))
 return

def reset(pulse):
 pulse.start(7.5);
 pulse.ChangeDutyCycle(7.5)
 time.sleep(0.5)
 pulse.ChangeDutyCycle(0)
 return

def compare(timeRotation, rotation, targetRotation):
 global deltaTime
 if (timeRotation >= 100):
  return 0.25
 elif (rotation == targetRotation):
  if (timeRotation >= 0.0):
   return timeRotation - deltaTime.total_seconds()
  else:
   return timeRotation;
 else:
  print "targetRotation changed."
  return 100

def update(pulse, timeRotation, targetRotation):
 cycle = 0
 if (timeRotation >= 100):
  cycle = 0
  if (targetRotation == 90):
   cycle = 7.5
  elif (targetRotation == 0):
   cycle = 2.5
  else:
   cycle = 12.5
  print ("Cycle: "+str(cycle))
  pulse.ChangeDutyCycle(cycle)
 elif (timeRotation >= 0.0 and ((timeRotation - deltaTime.total_seconds()) <= 0.0)):
  pulse.ChangeDutyCycle(0)
  print ("Cycle: "+str(cycle))
 return targetRotation

try:
 reset(pulse2)
 reset(pulse3)
 time.sleep(1)
 print "setup complete"

 while True:
  
  last_time = current_time 
  current_time = datetime.datetime.now() 
  deltaTime = current_time - last_time;

  log()
  timeRotation2 = compare(timeRotation2, rotation2, targetRotation2)
  timeRotation3 = compare(timeRotation3, rotation3, targetRotation3)
  rotation2 = update(pulse2, timeRotation2, targetRotation2);
  rotation3 = update(pulse3, timeRotation3, targetRotation3);

  time.sleep(0);
  
except KeyboardInterrupt:

 print '\r\nProgram shutdown.'
 stayAlive = False
 time.sleep(1)
 GPIO.cleanup();
 print '\r\nProgam complete.'

[LeapServer]

Unreal: Speech Plugins

I created repositories to hold some new Unreal projects.

Documentation: [UnrealHTML5SpeechDetection] [private repo]

[Demo 01 – Speech Dictation]

Research:

Default Local Build: [localhost:8000]

[Unreal: HTML5 – Getting Started]

[How to reduce HTML5 package size?]

Files Required for Final Deployment
-----------------------------------

*.js.gz   - compressed JavaScript files.
*.data    - compressed game content.
*.mem     - compressed memory initialization file.
*.html    - uncompressed landing page.
*.symbols - uncompressed symbols, if necessary.

[HTML5: Call C++ UFUNCTION from Webpage via JavaScript]

[HTML5Platform.Automation.cs] includes [GameX.html.template]

[Connecting C++ and HTML5] [info]

Video: [Getting started with Emscripten – Transpiling C / C++ to JavaScript / HTML5]

[Unreal.js]

[Unreal.js-core]

Issues:

[UE4 Improvements]

[AnswerHub]

Pull Request: [Add support for including project and plugin JS]

Steam: SteamVR Plugin for Razer Hydra

[SteamVR Plugin for Razer Hydra]

** Caution: Be sure to close any Game, UE4, Unreal Launcher, SteamVR, and Unity processes that are running before installing.

Source: [steamvr_driver_hydra]

Install "SteamVR" - steam://install/250820

Install "Sixense SDK for the Razer Hydra" - steam://install/42300

Install "SteamVR Driver for Razer Hydra" - steam://install/491380

[issue]