Connected Personal Objects: Getting Intimate with the Internet of Things

Photo

 

Last week, Guthrie Dolin and I presented at the Planning-ness 2012 conference. The conference was great, and I love it's theme, "Get Excited and Make Things." 

The standard format for a session is present on a topic and then run a workshop in which conference participants make something to present back to the group.

In our session we provide an overview of the Internet of Things before going into detail about the potential of connected objects to encourage specific behaviors. We break down the components of an IoT ecosystem and discuss how they can be used to collect data and drive insights and change.

Following the presentation, the assignment was to use the framework and design a physical product and service around a personal object or situation. We provided a few catalysts and blank worksheets to help get people going. The results were great and we've posted each group's worksheets on slideshare as well.

All-in-all, a great expreience and a ton of fun. Thanks to everyone who turned out to learn about the Internet of Things.

 

Timberland in-store experience on Vimeo

Nice video showcasing in-store work done for Timberland. Among other things, it talks about the value of using a web-based approach for distributing content and gathering data from the customer.

A web based approach like this could also open an experience like this up to include the mobile web. QR codes or other prompts could get customers pulling stories up on their devices and start sharing them outside the store.

eggs

Img_2370

This year's eggs were designed using Nodebox and Inkscape.

Charlie worked mostly in Nodebox. It's patch-based programming was right at his level and copy, rotate and wiggle all got a lot of action.

Wyatt found inspiration in Minecraft and we worked together to write a letter writing script in Python to print out an wiki entry on egg. Then he created a vector version of the pixelated egg in Inkscape.

The video below shows some highlights from production.

Testing OpenCV and Face.com api

Face.com offers a robust API for facial recognition. In short, you send it an image and it will process that image finding faces within it and return information about what it finds. That information can be used to locate features of the face as well as other characteristics of the face. These other characteristics include the age and gender as well as the presence of glasses or a smile. 

As a test, I built a standalone app that sends faces appearing in an attached web cam to the API for analysis. 

First, I built a basic Flash AIR application to upload images to the face.com API. For this I used Jean Nawratil's Flash ActionScript 3.0 Client Library and sent images each time a key was pressed on the keyboard. The only modification I made to Jean's library was to add a parameter to the POST made to the API. The parameter (attributes=all) tells the API to return additional characteristics.

With this in place I integrated OpenCV to locally detect when a face is present in the frame. This way, I could automatically send images to the API, but only those with something for face.com to find -- reducing the load on their servers (and my rate limit).

I've used an AS3 implementation of OpenCV in the past, but the performance was terrible. For this project, I used Wouter Verweirder's AIR Native Extension which offloads the work to native code. It is very responsive.

Using a standard face cascade, OpenCV recognizes faces and indicates approximately where they are in the frame. Based on this detection, my app sends a copy of the image to face.com for further analysis. The round trip to the face.com API is quick and generally takes less than a second, two at the most. I added a timer to the app to further limit how often I send off an image to the API.

Below are some captures from this basic proof of concept. Below each are the characteristics detected by the API.

Davidbliss
Davidbliss-1
Davidbliss-2_copy

In addition to basic face detection, the face.com API allows you to identify and tag people based on images you use to train the algorithm and/or images in a Facebook account. For more information about the API and what it can do, be sure to check out their online demo.

Anatomy of a mobile web experience: google.com | DeviceAtlas

The first point to note here is that Google serves an entirely different HTML document to each device that I emulated—the superficially similar pages that we see on google.com are in fact radically altered depending on the device you're holding. Google changes essentially every aspect of the HTML and its delivery to ensure a good experience. At a very basic level, the pages are cosmetically altered (logo resizing etc.) to fit the device perfectly but in fact the adaptation goes much deeper than that: Google change the very document type delivered, whether or not compression is utilized in its delivery, the extent to which CSS and JavaScript are utilized, CSS vendor prefixes and so on. The range of adaption is impressive in its scope—Google go far beyond merely fitting everything onto the screen.

This is interesting analysis. I would like to see a study like this done for a broad range of popular site and brands.

Fully optimizing the mobile experience on the full range of mobile devices can't be done with responsive design alone. Investing in additional HTML formats based on device detection makes sense for Google, Amazon, Facebook and others who can justify it with a tangible ROI.

This does not take any of the value away from a responsive design approach. Responsive Design makes a great deal of sense for businesses who can't justify the effort required to build and maintain multiple sites or experiences.

What's more, even if you have custom experiences for less capable devices, you should definitely consider making the rich experience responsive for a range of devices capable of delivering this higher level of experience.

Working with Kinect for Windows and Flash

The Kinect for Windows SDK supports development in C++, C# and Visual Basic. None of these options are great for rapid prototyping. Fortunately, AIRKinect from the AS3nui team allows Flash applications to access the Kinect's information using the Microsoft SDK.

AIRKinect is an Adobe Native Extension for Flash (ANE) and provides direct access to the Kinect from within an AIR 3.0 application. Previous options for using Kinect within Flash required a proxy server to get data from the Kinect. In comparison, the ANE approach is more responsive and easier to work with.

New post over on the Odopod blog.