ADAS and Autonomous Driving: Sensing Fundamentals

Gains:

• Ability to explain the roles of camera, radar, LiDAR and sensor fusion in the detection chain and SAE automation levels
• Ability to create data labeling, scenario definition and evaluation metrics for object detection, classification and tracking
• Ability to critically evaluate edge cases, false positive/negative balance and margin of safety in the detection model output

Most vehicles today have features like lane departure warning, automatic emergency braking or adaptive cruise control. These are all under the umbrella of ADAS (Advanced Driver Assistance Systems). At the heart of these systems and further autonomous driving is perception: the way the vehicle "sees" the world around it with sensors and "understands" it with artificial intelligence. In this unit, we will cover the detection chain, sensors, automation levels and the most critical issue, security boundaries.

A warning from the beginning: This area is security-critical. If the detection model does not detect a pedestrian, the result can be fatal. That's why every AI output here must be surrounded by security engineering processes (ISO 26262 and SOTIF in the next units).

Sensors: eyes, ears and radar

An autonomous driving stack combines different sensors because each has strengths and weaknesses:

• Camera: The camera "sees" color, text (traffic sign), lane line and object type best. But it is poor at night, fog, strong sun glare and distance estimation.
• Radar (Radio Detection and Ranging): Measures distance and speed with radio waves. It is less affected by rain/fog and measures speed directly (with the Doppler effect); but its resolution is low, making it difficult to distinguish "is this a pedestrian or a box?"
• LiDAR (Light Detection and Ranging): Shoots a laser and produces a precise three-dimensional "point cloud" of the surroundings; It gives distance and shape very well. But it is expensive and can be affected by heavy fog/rain.
• Ultrasonic: Short range, like parking sensors.

Since neither is sufficient on its own, sensor fusion is performed: by combining the data of different sensors, the weakness of each is compensated for by the strength of the other. For example, the camera says "pedestrian", the radar says "at 12 meters, approaching 5 km/h"; Fusion combines the two and creates a reliable track.

Tip: When evaluating a detection claim, "which sensor can see it and how well?" ask. If the camera's distance estimate at night is poor, a distance decision based on a single camera is questionable.

Detection chain

The typical chain from raw sensor data to decision is as follows:

1. Detection: Finding objects in the image/point cloud ("there's a tool here").
2. Classification: Determining the type of object (pedestrian, bicycle, vehicle, sign).
3. Tracking: Following the object between frames, estimating its speed and direction.
4. Fusion: Combining sensors to create a consistent world model.
5. Prediction: "Will this pedestrian cross the street?"
6. Planning and control: Decision and steering/brake/accelerator command.

Artificial intelligence (especially deep learning) 1-3. very strong in steps; 5-6. The steps are increasingly surrounded by engineering rules and security logic.

SAE automation levels

Autonomy is not a "yes/no" thing, it's gradual. The SAE J3016 standard defines six levels:

Level

Name

Who is driving?

example

0

No automation

all drive

Warning only (blind spot warning)

1

Driver support

driver + single support

Adaptive cruise OR lane keeping

2

Partial automation

Driver (constant supervision)

Speed + lane together, driver responsible

3

Conditional automation

System (under certain condition)

Under certain condition the system drives, the driver is ready to take over

4

high automation

System (in specific area)

Driverless in defined area

5

Full automation

System (in all conditions)

Driverless everywhere (not yet)

Caution: The most dangerous misunderstanding is Level 2. In Level 2, the vehicle supports steering and speed, but the driver retains constant supervision and responsibility. Mistaking the system as "autopilot" and not paying attention has led to fatal accidents. Level 2 does not mean "driverless".

Evaluation metrics: false positive and false negative

Two types of errors are critical when evaluating a detection model:

• False positive: Thinking that there is something that does not exist. For example, thinking that a bridge shadow is an "obstacle" and braking unnecessarily. Disturbing and sometimes dangerous (rear-end collision).
• False negative (false negative / miss): Not being able to see something that exists. For example, not seeing a real pedestrian. This is the deadliest mistake in autonomous driving.

Metrics that summarize these:

• Precision: How much of what you marked is actually correct.
• Recall (sensitivity): How much of what actually exists have you captured.
• mAP (mean Average Precision): Common composite metric for object detection.

Recall (not missing any pedestrian) is vital in matters such as pedestrian detection in autonomous driving; Increasing this sometimes increases the false positive. This trade-off is a security engineering decision, not pure model accuracy.

Edge cases and the long tail

The most difficult part of autonomous driving is not the common situations, but the rare and strange ones: the sign half visible under the snow, the upside-down truck, a costumed pedestrian, furniture fallen on the road. These rare but dangerous situations are called long tail. While a model may be perfect for millions of "normal" miles, it may fail in an extreme situation it has never seen before. That's why scenario-based testing, simulation and real-world validation are endless.

Mini case studies

Case 1 - The value of fusion. While testing automatic emergency braking, an ADAS team found that it was slow to spot a dark-colored vehicle in front of it in bright sun with just the camera. When radar data is added (radar is not affected by glare) the system detects the vehicle 0.4 seconds earlier; This means braking distance of approximately 10 meters at 90 km/h. Result: Fusion closed the single sensor blind spot.

Case 2 - Recall/precision balance. When the threshold of the pedestrian detection model is lowered, the recall increases from 96% to 99%, but false positives (shadow, bushes) increase, which leads to unnecessary braking. The security team makes the decision: avoiding pedestrians is a priority, but comfort is maintained by eliminating false positives with fusion and tracking logic. Conclusion: Metric selection is an engineering/security decision, there is no single "correct number".

Case 3 - Edge case surprise. While a model was 99.8% successful in its validation set, it mistook a photo of a giant pedestrian behind a truck carrying a snow tire advertisement in the field for a "real pedestrian" and braked suddenly. The team adds this edge case to the scenario library and retrains the model. Conclusion: The long tail never "ends"; Constant scenario collection is required.

prompt templates

Template 1 - Generating a scenario list:

Role: You are the ADAS test case engineer. Task: Produce challenging (edge case) test scenarios for automatic emergency braking. Context: Urban, 30-50 km/h, mixed pedestrian and cyclist. Constraint: Prioritize rare/dangerous ones, not frequent scenarios; write the expected correct behavior for each scenario.Output: Scenario | condition | expected behavior | risk level.

Template 2 - Labeling guide outline:

Role: Data labeling quality lead.Task: Write a draft consistency rules for pedestrian labeling.Context: There are pedestrians partially visible, seated, in wheelchairs, and in groups.Output: Rule | example | limit state grade table.

Template 3 - Metric interpretation:

Role: Computer vision evaluation expert. Task: Interpret my pedestrian detection results. Context: precision 0.93, recall 0.90, most misses at night or in the rain. Constraint: Emphasize the importance of recall for security; Tell me under which conditions additional verification is required. Output: Comment + priority areas for improvement.

Template 4 - Fusion logic outline:

Role: You are a sensor fusion engineer. Task: Suggest a draft decision logic when camera and radar conflict. Context: Radar sees moving small object at 15 m while camera says 'no pedestrian'. Constraint: Apply the assumption principle in favor of safety (safe side). Output: Decision table + justification.

Weak prompt / Strong prompt

Weak prompt:

Is my autonomous vehicle model safe?

Without context; "safe" is undefined; AI only speaks in general terms and can give misleading confidence.

Powerful prompt:

Role: You are the ADAS security assessment consultant. Task: Create a checklist for me to question the vulnerabilities of my pedestrian detection model. Context: Urban Level 2 system; Recall falls at night and in the rain; There is only camera + radar, no LiDAR. Constraint: Do not declare 'safe'; specify under what conditions the system is at its limit and what additional testing is required. Output: Risk area | why | recommended test | importance table.

Common mistakes

• Mistaking Level 2 for "driverless". The responsibility lies with the driver; This misconception is fatal.
• Relying on a single sensor. Every sensor has a blind spot; fusion is essential.
• Just looking at average accuracy. 99% success, 1% missed pedestrian could be fatal; recall and edge cases are critical.
• Ignoring extreme cases. The long tail never ends; Scenario collection is continuous.
• Leaving the security decision to the model. Threshold and balance decisions are the responsibility of security engineering.

In summary

• At the heart of ADAS and autonomous driving sensing is camera, radar, LiDAR, and sensor fusion; Each sensor has different strengths/weaknesses.
• The detection chain consists of detection, classification, monitoring, fusion, prediction, and control; AI is strong in the first steps.
• SAE levels range from 0 to 5; In Level 2, the responsibility lies with the driver.
• False negative (pedestrian abduction) is the most fatal error; recall and edge cases take priority.
• Security decisions are engineering responsibility; Model accuracy is not the only criterion.

Application task

Select an ADAS function (e.g. lane keeping assist). (1) Write in a table which sensors support this function and how well. (2) Generate 5 edge case scenarios with Template 1 and specify the expected safe behavior for each. (3) Compare the results of false positive and false negative for this function. (4) List what independent testing is required before claiming the model is "safe".

checklist

• [ ] I evaluated the strengths/weaknesses of the sensors according to function.
• [ ] I have also listed the end case (long tail) scenarios.
• [ ] I compared false positive/negative results for safety.
• [ ] I took into account the security-critical importance of Recall.
• [ ] I clarified the SAE level and who is responsible.
• [ ] I prepared independent testing plan before claiming "safe".