Overview

This project presents deep learning based approaches for stair detection, statistical filtering on images for the estimation of stair alignment, and novel mechanical design for an autonomous stair climbing robot. The primary objective is to solve the problem of indoor locomotion over staircases with the proposed implementation.
This project was partially sponsored by the IEEE VNIT Student Branch and made possible through the active efforts of the Branch Counselor, Dr. Pradyumn Chaturvedi.
Novel Contributions
  • Novel mechanical design for an Autonomous Stair Climbing Robot.
  • Two Deep Learning based approaches for Stair Detection.
  • Statistical filtering on images for estimation of stair alignment.

Hardware Specifications

Electrical Aspects
  • Motor Driver - 18 V, 20 A
  • Microcontroller - Arduino Nano
  • Bluetooth Module - HC-05
  • Battery - 11.1 V, 3S LiPo
  • Motors - 150 RPM, Johnson motors (average torque of 5-10 kg-cm)
Mechanical Aspects
  • Dimensions
    • Height - 20 cm
    • Breadth - 30 cm
    • Length - 80 cm
    • Weight - 5.4 kg
  • Materials
    • 2 mm thick Aluminum sheet
    • 3 mm thick acrylic sheet
    • Plastic wheels
    • Rubber gripped conveyor belts
  • Other components
    • 2 RC suspensions, butt hinge, springs, nut and stud combination, nut and bolt.
Design Aspects
  • Tracked and segmented design.
  • Custom designed 3D printed wheels to avoid slippage.
  • Use of hinges and springs for smooth motion.
  • Supporting wheels and plates for efficient power transmission.
Left : Hardware Version 1
Right : Hardware Version 2
Cyber-Physical Architecture

Stair Detection using YOLOv3

  • Data Collection and Augmentation
    • Data was manually collected. Most of it is from the VNIT campus and some from the Internet.
    • 848 images were collected and annotated. Horizontal flipping was done for data augmentation (total - 1696).
    • Dataset was split into 1488 images for training and 208 images for testing.
    • Left : Sample images and annotations (in yellow).
  • Transfer Learning was done for tiny-YOLOv3 using the ImageNet trained weights as pretraining.
  • The model was deployed on an Nvidia Jetson TX1 module.
Results
  • Over 100 frames per second (fps) on Nvidia GTX 1080Ti using tiny-YOLOv3.
  • Around 20-25 fps on an Nvidia Jetson TX1 using tiny-YOLOv3.
  • Right : Results on testing dataset
Details on the other approach, Stair Segmentation, will be added soon.

Stair Alignment

  • To align the tracked robot w.r.t. the stairs, it is essential to estimate the heading of stairs w.r.t current heading and minimize it w.r.t time.
  • Steps involved :
    • Fitting of lines on the edges of stairs detected inside the ROI.
    • Determination of the midpoint of every stair.
    • Line-fitting on the midpoints of stairs.
    • Computation of desired heading.
Top Image :
  • (a) ROI obtained after Stair Detection
  • (b) Gradient image computed from Scharr operator
  • (c) Binarization of image based on gradient magnitude
  • (d) Binarization of image based on gradient direction
  • (e) Hough line fitting without prevention of cluttering
  • (f) Lines obtained after prevention of cluttering

Right Image :
  • Blue lines represent stair edges.
  • White line represents the current heading.
  • Black line represents desired heading.
  • Yellow points represent the midpoints of the lines.
  • The angle between white line and black line is known as offset angle.

Control System for Alignment

  • PID controller is used to minimize offset angle with time.
  • Velocities of right and left wheel are derived using differential drive model.
  • IMU sensors are used to compute the current state action of robot. IMU readings indicate whether robot is climbing stairs or traversing plane ground.

Image Gallery

Hardware Demonstration

Citation

If you found our work useful, please cite either of the following publications :
  • U. Patil et al., "Deep Learning Based Stair Detection and Statistical Image Filtering for Autonomous Stair Climbing," 2019 Third IEEE International Conference on Robotic Computing (IRC), Naples, Italy, 2019, pp. 159-166. doi: 10.1109/IRC.2019.00031 [IEEEXplore] [BibTex]
  • N. Panchi, K. Agrawal, U. Patil, A. Gujarathi, A. Jain, H. Namdeo, and S. S. Chiddarwar, "Deep learning-based stair segmentation and behavioral cloning for autonomous stair climbing," International Journal of Semantic Computing, vol. 13, no. 04, pp. 497–512, 2019. [Journal Paper Link] [BibTex]

Team

Unmesh Patil
Aniket Gujarathi
Akshay Kulkarni
Aman Jain
Navid Panchi
Khush Agrawal
Kartik Paigwar (Student Mentor)
Lokesh Malke
Radhika Tekade

Faculty Mentors :
Dr. Shital S. Chiddarwar and Dr. Pradyumn Chaturvedi
Thanks for visiting. You were visitor number
counter
  •