CMSC498F, CMSC828K (Spring 2016): Robotics and Perception

Time/Location: Tu/Th 9:30-10:45am,   CSI 2120

Instructor: Cornelia Fermüller
Office hours:  Tu/Th: 11:00-12:00, AVW Bldg. 4459

TA: Aleksandrs Ecins :

Office hours:  Mo/Wed: 11:00-12:00, AVW Bldg. 4468



Link to Piazza Website:

Link to Canvas Website 498F and Canvas Website 828K (Use this for project submission)


This course offers an introduction to the design and programming of robotics systems. The course covers topics in the area of navigation using vision and 3D depth sensors, localization and map making, basic image processing for visual navigation and recognition, and vision and depth based grasping and manipulation. You will be developing algorithms, and learn how to use current state-of the art vision and software tools, such as OpenCV, MoveIt and the Point Cloud Library. The software components will be developed under the Robotic Operating System (ROS).


The course will be organized around a few projects, starting with navigation in a map, then localization using the map, then finding objects, and finally a project of object manipulation. The software will first be developed in simulation, before testing it on the platform, where students will work in groups of three to four.


Programming environment and languages:

As robot platform we will use the Turtlebot, the Phantom Pincher Robot Arm Kit, and the Baxter Robot. Software development will be under ROS/Linux, and programming in C++ and/or Python and/or Matlab.



Students taking the class should be comfortable with linear algebra and calculus.


Workload for 498 and 828

The lectures are designed for both undergraduates and graduates. Graduate students will be assigned additional readings, and they will be required to present one or two papers each during the semester. The graduate students will have different homework, different project assignments, and different exams from the undergraduates.



Homeworks and Projects: 65%

Exam: 35 %

Midterm: Thursday, March 10



R. Siegwart I. Nourbakhsh, and D. Scaramuzza: Autonomous Mobile Robots, Second Edition, MIT Press, 2011, First Edition:  pdf 

Peter Corke: Robotics, Vision and Control, Fundamental Algorithms in Matlab:


Recommended books:

S. Thrun, W. Burghart, D. Fox: Probabilistic Robotics, 


Online Resources :

Robot Systems Programming Course at JHU

Introduction to Autonomous Robotics by N. Correll

Planning Algorithms by S. LaValle


Course Outline:



Assignments/Due dates

week 1



Overview of ROS and available packages

Linear algebra tutorial (from PennState) 

week 2

(2/2, 2/4)


guest lecture: image formation

Hardware and Locomotion Hardware and Locomotion

Read chapter 2 of Correll

week 3

(2/9, 2/11)

An Introduction to ROS (presented by Alex)

Coordinate System Transformations : Representing Position and Orientation

First Project (getting started with ROS) (due 2/19)

week 4


Mobile Robot Kinematics


Project on Kinematics (due 2/25) Code for the project

Read chapter 3.1 and 3.2 in Correll

week 5

(2/23, 2/25)

Basic Control and Mobile Robot Control

Path Planning 



week 6

(3/1, 3/3)

ROS lecture (presented by Alex)

Vision: filters


Project on Control and Path Planning using Potential fields

code video

week 7

(3/8, 3/10)


Vision: edge detection, corner detection, Hough transforms SIFT edge and corner detection, Hough transforms



week 8

(3/22, 3/24)

SIFT histograms

image motion


week 9

(3/29, 3/31)

line fitting

Sensors: depth sensors and inertia based sensors

Project on mapping code (due 3/7/2016)

week 10

(4/5, 4/6)

Probabilistic Robotics (Statistics Prerequisites )

Uncertainty and Error Propagation

Localization with sensors (Markov localization)


week 11

(4/12, 4/14)

Localization with particle filters

Localization tips

Kalman-filtering, Kalman example

Project on localization and mapping code

week 12

(4/19, 4/21)

Multiple View Geometry and Stereo


week 13

(4/26, 4/28)

Multiple View geometry continued

Recognizing specific objects

Project on visual object recognition and navigation, Code

week 14

(5/3, 5/5)

Object Recognition



week 15



Visual Servoing

Discussion of Projects