By Adil Shafi, President, ADVENOVATION, Inc.
Mathematics are well and good but nature keeps dragging us around by the nose. (Albert Einstein)
The Pervasive Relevance of Bin Picking
“Wind and Wuthering” by the music band Genesis.
When considering random picking, it is appropriate to begin with two facets of the natural world and our human life. Doing this gives us a basic understanding of the process and helps us draw relevance in the manufacturing setting.
First, the world is filled with entropy, or disordered order, or alternately, ordered disorder. As an example, no two trees are identical yet trees of the same kind look similar. In the autumn, leaves do not fall in a rectangular grid pattern; they look similar but they are uniquely different and when they fall, they fall in a random order driven by the forces of wind and gravity. In a similar manner, it is natural to have parts made in a stamping machine or a die cast to fall in a heap in a bin in random order. By virtue of gravity, they do not fall in an ordered geometric pattern. Thus, entropy and randomness are integral in our natural world.
We as human beings find it naturally easy to see objects in an unordered environment and then to touch or pick them. We do it many times each day; often without even thinking about it.
Humans deal with this disorder of picking leaves off the ground, or separating clothes in a heap by two of the most used of our five
“Bin Picking Shape Variant Food Bags” Courtesy of Adept Technology.
senses: sight and touch. We are able to i) see things, ii) figure out where they are, and then iii) tell our arms, hands and fingers how to acquire them and perform an action. This, too, has a mechanized analogy. The science and art of seeing objects in disarray with machine vision cameras and to figure out where they are in physical space, and then to command robots to acquire them and perform a subsequent action is called “bin picking.”
In the manufacturing business, presentation of parts to robots for handling or assembly creates cost, tooling and maintenance. It is ideal to be able to acquire parts, resting naturally in varied forms of random order, from bins, trays, totes, racks, etc. thereby saving multiple costs.
The Difficulty
Ever since robots became practical to use in factories, more than three decades ago, people have seen numerous bins in factories and hoped and wished and worked to make robots see these random objects and pick them in an automatic manner. Theoretical books have been written, some of our best universities have made serious efforts, companies have attempted, and called this goal the most difficult yet most rewarding of all robot challenges in factories and termed it “The Holy Grail.”
A Brief History
“A bin filled with automotive piston heads” Courtesy of ADVENOVATION, Inc.
For 25 years, technology and the complexity of the task in geometry, lighting, shadows and occlusion have evaded success and frustrated a lot of attempts. In 1986, Professor Berthold Klaus Paul Horn at MIT wrote in his famous book Robot Vision1 “One of the remaining obstacles to the widespread introduction of industrial robots is their inability to deal with parts that are not precisely positioned,” and he went on to offer several computational and practical ideas to make bin picking possible.
Many who worked on bin picking in the 1970’s and 1980’s turned their attention to applications for the Internet and security in the 1990’s and 2000’s. But, now in 2011, there is newfound success for bin picking in the manufacturing industry.
Enabling Breakthroughs
Mechanistic, electrical or computational breakthroughs that make bin picking possible or easier are called “bin picking enablers.” Especially significant are advancements in lighting, sight, computation and adoption of robotics. Today, lighting is more flexible. Cameras have higher resolution. Computational processing power is staggering, and we have new mathematical algorithms for three dimensional (3D) bin picking object location. Other important enablers relate to part handling techniques including part separation, and robot grasping techniques.
Today, lights are more powerful, and we have seen a steady progress in resolution towards “megapixel” cameras. The almost universal
“Axle Shafts in Powertrain” Courtesy of Motoman Robotics.
use of PCs for business and personal use has spurred computational processing power. Part handling techniques such as magnets, air, vibration, gravity and multistage control for precise part control have been extensively used. Robot grippers have improved in sensitivity, versatility, speed and compliance.
The most challenging bin picking enablers have resided in the computational sector. In bin picking, it is easy to calculate 1D linear segments, 2D features and point clouds of 3D scenes. But to make sense from this mountain of data often requires significant mathematical capability such as perspective calibration, triangulation, epipolar geometry, Kalman filtering, stereo matching and situational heuristics to identify possible solutions and then to generate from them viable part pickup candidates. Many excellent papers and books have been written to assist bin picking in this regard. One book in particular that bridges theory and practice is Artificial Vision for Mobile Robots by Nicholas Ayache.2
How to Implement
Those who want to stay current on the technology should go to RIA’s International Conference on Vision Guided Robotics (ICVGR), October 25-27, 2011, Plymouth, Michigan. Sessions and a tabletop trade fair cover advancements and technology for automated handling issues including bin picking. For more information and registration, visit http://www.robotics.org/events/event.cfm?id=108
For a guide on “How to Implement Bin Picking in Your Manufacturing Operation4,” visit http://www.robotics.org/content-detail.cfm/Industrial-Robotics-News/How-to-Implement-Bin-Picking-in-your-Manufacturing-Operation/content_id/1787
Technical Trends in 2011: Various Kinds of Bin Picking
Bin picking is challenging because there are 1) many types of geometries in the parts that are attempted to be bin picked, and 2) they occur in bins with various degrees of structured, semi-structured or random order.
Unfortunately, the term “bin picking” has been used in a generic manner to encompass many different kinds of bin picking; from the extremely easy to the extremely difficult, and from simple installations to intractable problems. It is therefore important to recognize various kinds of bin picking and to recognize, define and work with specific, quantifiable terms:
- 2D bin picking with x, y variation and a roll angle
- 2.5D bin picking with x, y variation, a roll angle and vertical z variation
- 3D bin picking with random x, y, z variation
- 3D bin picking with random x, y, z, yaw, pitch, roll variation
- Bin picking with distinctly separated parts
- Bin picking with touching or overlapping parts
- Bin picking with focus, scale or random shape variation in addition to positional variation
- Bin picking with distinct imaging features
- Bin picking with weak, inconsistent, or no imaging features
These constitute much of the activity in innovation in bin picking in 2011.
Market Progression in 2011: Various Stages of Innovation, and Acceptance
There are, typically, four stages of manufacturing innovation and acceptance: 1) initial discovery or solution in a fragile lab
“Bin Picking Unordered Objects” Courtesy of Scape Technologies.
environment with no operator interface and with questionable reliability; also called “bleeding edge,” 2) adoption of new technology in a handful of installations and managed by highly trained technical personnel for limited durations; also called “leading edge,” 3) adoption by several factories and reliable use for less than 5 years; also called “emerging,” and 4) adoption by most factories and reliable use for more than 5 years; also called “mature.”
There has been no “magic bullet” or technique to solve all geometries for bin picking. Rather a collection of algorithms has emerged that have progressively solved an increasing number of classes of geometries, arranged in a progressively increasing level of randomness. This trend is expected to continue with additional bin picking Enablers in the future e.g., the design of custom VLSI chips to perform current software techniques more quickly in hardware. Color and light handling techniques are expected to provide more viable candidates amongst heaps of random parts for bin picking. Material handling techniques derived especially from space, surgical and rehabilitation handling experience are expected to make bin picking more flexible, versatile and fault tolerant.
In 2011, structured or semi-structured bin picking (in which parts are staggered or skewed but not completely random) is currently in the third “emerging” stage. Random bin picking (in which parts are completely randomly placed) is currently in the second “leading edge” stage. Both areas are destined to improve in the future.
In 2011, to learn more about how the various kinds of bin picking (described above) correspond with the various stages of innovation, implementation and acceptance (described above), visit the ICVGR conference presentations and panel discussion at http://www.robotics.org/events/event.cfm?id=121
Proven Solutions
The following applications have now become feasible for reliable bin picking:
1. Automotive
- Powertrain (engines, cylinder heads, axle shafts, differential carriers, pinions, round parts with stems, connector rods, piston heads, brake rotors and stacks of gears).
- Stamping (flat or bent metal plates with multiple holes, stacked stampings with a progressive skew).
- Final Assembly Products in Boxes in car assembly (trim chassis final) pick operations for placement into cars on moving lines.
2. Packaging
- Strips of medical tablets, flat but randomly arranged in boxes.
- Bags of products e.g., chips, salsa, cheese, cement, etc.
- Lateral or upright layers of tubes (copper, plastic, PVC).
- Layers of products e.g., wooden planks, plastic sheets.
Quotes from Industry Veterans: The Confidence of Established Market Players
In the last five years, a growing set of successful installations and subsequent citation in industry press and demonstrations at trade shows has reinforced the appeal and growing confidence in robotic bin picking.
“Bin Picking Hub Caps” Courtesy of Fanuc Robotics.
According to Edward Roney, Account Manager for North American Distribution Sales at Fanuc Robotics, “End Users are increasingly requesting solutions to work directly with their shipping and receiving containers. This saves End Users time and money in their operations. Fanuc Robotics is developing tools and techniques to fulfill these requests and they include bin picking techniques with robots, vision and end of arm tooling.”
According to Tim DeRosett, Director of Marketing at Yaskawa America, Inc., Motoman Robotics Division, “The ability to locate parts without fixturing and hard tooling is something everyone in manufacturing wants to do. Robots are flexible to program, but without vision, part presentation represents a major tooling and cost impediment to manufacturers. Robots, vision, flexible feeders, and bin picking represent a huge opportunity for advanced manufacturing.”
Greg Garmann, Software & Controls Technology Leader at Motoman Robotics adds: “We see an increasing capability to locate and find parts in containers and bins. This is made possible by advances in machine vision, time of flight and structured light techniques. It is now possible to consider automation solutions for a wider range of assembly, material handling and flexible kitting applications with one or more bins.”
According to François Bertrand, Matrox Imaging V.P. Sales & Marketing, “The continued adoption of flexible automation to improve manufacturing efficiency, throughput and competiveness is spurring the combined use of robotics and vision technology from 2D to 3D guidance and inspection.”
Mark Pilbeam, Account Manager for (Matrox’s Representative) North Coast Technical Sales in Michigan, N.W. Ohio and N. Indiana adds, “If companies are going to be competitive in a world economy, they must turn to Vision Guided Robot Technology and adapt these systems in their factories.”
Expectations and a Prediction
More innovation is on the way, spurred by Internet connectivity, more powerful computation libraries in open source or from proprietary suppliers, and of course, the motivation of financial benefit derived through overcoming entropy in our natural world. This points the way to solving the bin picking problem, for which there is no known theoretical computational impediment, completely by 2020, a number synonymous with perfect vision, if not before. This is my conviction and prediction.
References
- Robot Vision by Berthold Klaus Paul Horn, © 1986 by the Massachusetts Institute of Technology.
- Artificial Vision for Mobile Robots Stereo Vision and Multisensory Perception by Nicholas Ayache, © 1991 by the Massachusetts Institute of Technology.
- Automation World, http://www.automationworld.com/feature-1878 in February 2006 and entitled “Vision Guided Robotics: In search of the Holy Grail.”
- Robotics Online, http://www.robotics.org/content-detail.cfm/Industrial-Robotics-News/How-to-Implement-Bin-Picking-in-your-Manufacturing-Operation/content_id/1787 in April 2007 by Adil Shafi.
Contact Information
Adil Shafi is President of ADVENOVATION, Inc., specializing in software solutions and innovation in the field of Vision Guided Robotics (contact adil@advenovation.com or visit www.advenovation.com).
If you are not an expert on robotics, this webinar allows you to start at the beginning and gain a basic understanding of the technology. As you consider robotics for food and beverage applications, it pays to look at vision systems for inspection and traceability. Like peanut butter and jelly, robots and machine vision often go together.
Posted by Brian Huse 
Robots in America 