Making self-driving industrial machines
A project to help autonomous vehicles navigate better in adverse weather conditions is introduced by GIM Ltd Robotics Engineer and Team Leader, Tommi Tikkanen.
Autonomous vehicles are growing in popularity and acceptance in areas ranging from factory floors to mine sites and personal transportation. Methods for perceiving the environment either between vehicles, or between the vehicle and a remote operator, can vary. Cameras provide visibility, while sensors and satellite systems can offer applications in other areas.
Particularly for outdoor purposes, snow, sleet and other environmental factors can obstruct signals. By developing code and maximising the potential of 3D Lidar, a new programme called GIMNavi, developed by robotics and navigation company GIM Ltd, Finland, is being developed to help overcome such challenges.
The new programme is modular and could be applied to a variety of areas, e.g. autonomous driving on roads, or moving a trolley between stations in a smart factory.
The rise of 3D sensors
Making functional autonomous robots was difficult over 15 years ago – they had to be built from the ground up, reliable open source data libraries about environments were scarce, software that supported the algorithms, such as communication protocols, were usually self-made, and processing power was limited. On top of this, sensor technology was minimal, and laser scanners, or Lidar, were usually limited to two dimensions. This was fine in indoor applications, where a 2D representation of the environment was sufficient, but for outdoors there were geographical elements to consider, such as slopes, trees and special shapes that are hard to model as 2D.
One effort of the company was to build software that enabled efficient inter-process communication, so different components of a mobile robot could be integrated seamlessly. The software was called GIMnet – a custom middleware that could build large software systems and collaborate on developing the same robot.
At the same time, Robot Operating System (ROS), a middleware, was developed by researchers at Stanford University, USA. ROS started gaining popularity during 2011 and became a highly sought after middleware for researchers globally.
Advances in sensor technology helped with gathering data and Lidar supported modelling, but data processing in real-time was hard. A distribution model of the data, called a normal distributions transform (NDT) map, was a solution to this problem.
From this, it was possible to efficiently process groups of tens or hundreds of points by presenting them as a set of 3D Gaussian distributions. A typical NDT map is 10 times faster to process and 10 times smaller than a raw point cloud.
Company co-founder, Jari Saarinen, and fellow researchers developed 3D Lidar-based algorithms and a 3D NDT-based solution for positioning and mapping. Saarinen’s team contributed to NDT localisation using a particle filter. The group showed that NDT representation is an efficient and accurate method for positioning. It functioned in large-scale urban areas and factories with a 2-5cm accuracy, and no artificial infrastructure was needed.
Safety through sensor fusion
The navigation system is designed to support autonomous vehicle movement in large areas for indoors and outdoors. It is modular and can be used for applications including mining and smart factories.
To provide a navigation solution that copes with large spaces, the system uses a sensor fusion software package comprised of modules to assist with positioning and mapping, or obstacle detection and tracking. If required, it can also include navigation algorithms like optimising the trajectory or the route of the robot. The technologies used to enable these functions include 3D Lidar, satellite navigation, and sometimes, a camera.
Traditionally, factories and warehouses have relied on positioning beacons or reflectors to navigate accurately within the plant. A barrier for adoption is the cost and effort required to cover such large areas with these products. In outdoor applications such as construction, farming or mining locations, many companies rely on satellite navigation (GNSS), supported by Real-Time Kinematic (RTK) that normally requires one base station to cover distances up to several kilometres.
RTK is highly accurate in good conditions, but the signal is dependent on line of sight. When obstacles block the view to the base station, the accuracy is not good enough for autonomous navigation. This is where a fusion of technologies becomes effective.
To get around these challenges, the new system uses the whole area from which to draw data, for example the position of buildings, trees and poles, instead of static and selective beacons or reflectors.
3D Lidar technology is a core part of the process. The surveying method measures distance by illuminating targets with laser light and measuring the reflected light with a sensor, which can develop a 3D model of an area. Some 3D systems have 16 scanning Lidar that provide 300,000 points per second, giving a 360-degree view of the environment in real-time. GIM uses Lidar technology over camera-based solutions as they provide more accurate 3D measurements, and function reliably in most weather and lighting conditions. Lidar alone provides great results when there are enough landmarks in the environment. Testing to date has shown an accuracy of about 5cm, which is expected in urban areas or industrial sites. Satellite navigation is still required on open fields and rural areas. Finally, an inertial measurement unit (IMU) is needed to react to rapid turns and accelerations.
In robotics research, cameras provide a final method for positioning and mapping. Visual odometry can be performed from a single camera. The advantage is that even high-quality cameras are an order of magnitude and less expensive compared with Lidar. But cameras are generally avoided as their accuracy decreases with distance and weather conditions, which can compromise vision. Cameras and automotive radars are used as an additional sensor to detect and classify moving obstacles, like other vehicles and pedestrians.
As it is very typical in mobile robotics, the main principle in the system is that most of the sensors and computing is performed on-board the robot to avoid any connectivity problems, which cannot be achieved through cloud computing or external sensors.
The system works by having multiple 3D Lidar sensors on the automated vehicle, which use three axis points data to identify which path to take, and where it is along that route. Items including snow can be picked up through Lidar as it will detect some noise, suggesting there is a minor issue.
The system copes with this information by its positioning being based on normal distributions, meaning no decisions are made based on single points of data that need to be corrected or intervened. Those point measurements are accumulated into a 3D normal distribution rather than a 2D point-to-point, so solid objects can be better identified. This is why noise from rain or snow does not affect the navigation, but heavy and thick snow or fog will have an effect, because the technology is based on light.
You can teach a machine to take the path you want it to take, for example you could teach a shuttle bus to go from factory building A to B. When a vehicle is being taught, the system will record the 3D map of the area as the vehicle goes around. When the path is being repeated, it can be done so without using beacons to determine where the vehicle is. While mapping is taking place, it is ideal to use one beacon, in particular RTK GNSS, so an accurate 3D map can be built. Once this stage is complete, the beacon can be removed as the machine can now use the map to position itself accurately on the path.
The automated vehicle makes the 3D map or collects the map data itself, which can later be used for the robot to know where it is going. The map can also be uploaded to an external server and shared with other robots if desired.
An example of using the system in real-world testing is in retail, where the technology is being used for moving pallets and inventory. The technology can monitor which shelves have stock and which have run out of products.
It lends itself well to uses like supermarkets where the environment is relatively static and can more readily identify changes in stock, and in turn, use that data to either feed back to a distribution centre, or collect additional stock, returning to the same point and refill the shelf.
GIM is also developing an autonomous navigation solution for mining machines. The system has been included by Sensible 4, a sister company of GIM, in its collaboration project Gacha, a self-driving bus designed in collaboration with Muji, Japan. Sensible 4’s speciality is developing systems for detecting and following instructions from items including traffic rules or symbols.
The bus is designed to function in all weather conditions. With no defined front or back, it will serve regions that have suffered population decline and older citizens who can no longer safely drive. It is undergoing test drive programmes in three Finnish cities, with a plan for the vehicle to be rolled out in 2020.
The bus uses a digital map and sensing technology to drive itself in most weather conditions. It can drive on a fixed route, respond to a user’s request and find the most optimal route for a destination.
Gacha won the transportation prize in the Beazley Designs of the Year 2019 competition, which is being exhibited at the Design Museum, London UK, until 9 February 2020.
Prototyping and mass production
The positioning and mapping aspect of the technology is production ready. GIM is prototyping and testing obstacle detection and tracking, which is using a camera as well.
Rise of embedded GPU’s like NVIDIA Jetson product family will drive the cost of computing hardware down. While the best machine-learning models currently need a €1,000 GPU that consumes 200W, very soon it is expected that the price point will reduce to €100 and for energy consumption drop to 10W. It is expected that both open source and proprietary frameworks such as GIMNavi will help accelerate adoption of mobile robotics in real-world industrial applications.
Traditionally, robotics has been hard and unattractive for machine makers to develop because the systems are very complex and the payback period is long. But if we have building blocks, such as centimetre-level positioning in all conditions that are easy to use and widely available, that would greatly accelerate the rise of robots in all industries.