By ShilpBot Team · 8 min read · Updated
A camera gives a robot millions of numbers per frame, but numbers are not understanding. Computer vision is the set of techniques that turns raw pixels into information a robot can act on: what objects are present, where they are, and how far away.
The vision tasks robots use
| Task | Question it answers | Robot example |
|---|---|---|
| Image classification | What is in this image? | Sorting recyclables into types |
| Object detection | What objects are there and where are their bounding boxes? | A picking robot locating parcels |
| Semantic segmentation | Which class does each pixel belong to? | A delivery robot finding walkable ground |
| Instance segmentation | Which pixels belong to each separate object? | Separating touching items in a bin |
| Pose estimation | What is the position and orientation of an object? | Aligning a gripper to grasp a part |
| Depth estimation | How far is each point? | Avoiding obstacles |
From camera to decision
A typical perception pipeline looks like this:
- Capture an image from a camera.
- Pre-process it: resize, normalise brightness and correct lens distortion.
- Run a model, often a convolutional neural network, to detect or segment objects.
- Convert to world coordinates using camera calibration so pixel positions become real-world positions.
- Pass results to planning, which decides how to move.
Getting depth
A single ordinary camera does not directly measure distance. Robots obtain depth in several ways:
- Stereo cameras use two cameras and measure the shift of a point between the images, similar to human eyes.
- Depth cameras and LiDAR measure distance directly using projected light patterns or laser pulses.
- Learned monocular depth estimates depth from a single image using a neural network. It is convenient, but less reliable than direct measurement.
Why vision in the real world is hard
Models trained on tidy datasets often struggle outside the lab.
- Lighting changes: shadows, glare and low light alter appearance dramatically.
- Motion blur: a moving robot or object smears the image.
- Unfamiliar objects: a detector only recognises classes it was trained on.
- Latency: a model that takes half a second per frame may be too slow for a fast-moving robot.
Practical ways to improve reliability
- Train on images captured by the robot's own camera and in its real environment.
- Use data augmentation such as brightness changes and small rotations.
- Combine vision with other sensors, a technique called sensor fusion.
- Use smaller, optimised models on edge hardware when speed matters more than the last bit of accuracy.
Responsible use of cameras
Robots that capture images of people or private spaces raise privacy concerns. Collect only the data you need, tell people when cameras are operating, and store data securely.
What to learn next
Seeing the world is only half the problem. The robot also needs to know where it is and how to get somewhere safely, covered in Path Planning and SLAM.
Frequently asked questions
What is the difference between object detection and image classification?
Classification assigns one label to a whole image. Detection finds multiple objects in the image and gives each a label and a bounding box showing where it is.
Do robots need a depth camera?
Not always. Depth cameras or LiDAR make obstacle avoidance and grasping much easier, but some robots use stereo cameras or a single camera with learned depth, accepting lower reliability.
Can I run vision models on a small board like a Raspberry Pi?
Yes, with lightweight models and sensible image sizes, though frame rates are modest. For faster inference, people often add an accelerator or use a board designed for machine learning workloads.
