Advanced methods for safe visualization on automotive displays

Camera Monitor Systems (CMSs), for example, for backup cameras or mirror replacements, become increasingly important and already cover safety aspects such as guaranteed latency and no frame freeze. Today's approaches deal only with supervision of the digital interface, LCD backlight, and power supply. This paper introduces methods for advanced safety monitoring of panel electronics and optical display output that aim to enable future CMS based automotive use cases. Our methods are based on correlation of physical measurements with predicted values derived from a corresponding display model. This model was made via calibration measurements and many test patterns. Correlation of the monitoring results with predicted values corresponds to the probability that the RGB data are shown as intended. This implies that an overlying system, an Automotive Safety Integrity Level (ASIL) Prepared Video Safety System (APVSS), ensures that only safety verified RGB data are provided to the panel electronics. In case of failures, our methods enable a safe system state, for example, by deactivating the panel. An additional challenge is to allow graceful degradations, a safe but slightly degraded image may provide a better customer experience compared with no information. We successfully verified our approach by a fully functional prototype and extensive evaluation towards “light‐to‐light” (camera to display output) supervision.


| INTRODUCTION
Safety is essential for vehicles: In modern cars, more and more Camera Monitor Systems (CMSs) are integrated, for example, a backup camera is required by law in the United States. 1 These systems provide obvious benefits in terms of traffic safety. Figure 1 (bottom) shows a typical rear-view CMS. The camera is connected via high-speed video data interface to a head unit, which sends the video and operational data (e.g., climate control) to the central information display. The augmentation of, for example, trajectories as overlay is performed in the camera as head units lack automotive safety standards like ISO 26262. 2 Further trends indicate the replacement of side view mirror with CMS. An example of a CMS for trucks, 3 which entered mass production in 2019, is visualized in Figure 2.
Autonomous cars without steering wheel and pedals like WAYMO cars 4 may populate our streets in future. If those cars fail, they will stop and become obstacles to traffic flow. A remote operator ( Figure 1, top right) can then login to the car system and perform remote control driving (like unmanned aerial vehicles under remote control by a human operator). Such connectivity, which is based on consumer electronics and systems like 5G networks, is far away from fulfilling Automotive Safety Integrity Level (ASIL) requirements 5 and other standards like ISO 16505. 6 So both visualizations (in-car and remote operator) must be of the same quality (image perception, safety like real time, etc.).
As per ISO 26262, requirements related to functional safety arises, if driving relevant information like camera image from a rear-view (backup) camera is shown on display. If the video is routed via a component that does not follow a safety-relevant implementation, the safetyrelevant video streams cannot be "trusted" and require further measures. A possible solution for CMS was presented by one of the authors (Bauer 7,8 ). In this solution, as shown in Figure 3, a watermark is weaved into the video content within the camera serializer (SER) and detected in the display deserializer (DES). Due to its dynamic and unambiguous design, the watermark allows detection and alive verification of the safety-relevant video stream. In combination to the watermark, the video transmission uses protection mechanisms like Cyclic Redundancy Check (CRC), 9 frame and bit error counter, or line-fault detection. Within the display, the digital video timings are checked if they are in compliance with the specification limits. If any of the above methods result in an abnormal state, the display is turned off to avoid an unsafe state, for example, frozen or disrupted image. However, only digital data are supervised as "data to data" and a few electrical parameters including LCD backlight.
Although this configuration works perfectly fine for the given use case of a rear-view camera, we want to aim F I G U R E 1 System overview and challenges of Camera Monitor Systems (CMSs) for in-car rear view and wireless real-time transmission to a remote operator F I G U R E 2 Example of a Camera Monitor Systems (CMSs) as side cameras and for trucks (already in mass production), source: MERCEDES for more flexible and general video transmission safeguarding possibilities. This means one needs to provide solutions for a complete end-to-end safeguard of the video information. It ranges from the transformation of photons into pixels within the camera, through the video system, to the transformation of pixels into photons within the display. For this challenge, the "traditional" solution in Figure 3 needs to be enhanced in terms of capabilities to safeguard the optical-electro or electrooptical conversions in camera and display, respectively. In addition, a perceptual image quality assessment (PIQA) method to assess changes to the video stream is required. These methods are developed within the framework of the ASIL Prepared Video Safety System (APVSS). "Traditional" and new methods will communicate with the APVSS. The APVSS generates a system level decision based on all results of the failure detection methods. A brief overview of the APVSS with the new methods is shown in Figure 4, which targets towards "light-to-light" supervision from light (input) captured by the camera to the light (output) of the display. In this paper, we focus on optical and advanced electrical supervision of the display. It is obvious that advanced supervision towards higher ASIL levels raise the cost of automotive CMS. Cost estimations are provided for selected subassemblies and methods in the corresponding sections.
In this paper, we present the new safeguards of the video signal transformation from pixel into photons by advanced methods for enhanced safety levels applicable to a display by using the following steps: • Methods and measurements of the optical output and electrical power characteristics of the display depending on the RGB pixel data (Sections 2.1 to 2.3). • Development of a display model for the prediction of the optical output and electrical power characteristics based on the RGB pixel data (Sections 2.1 and 2.2). • Building a fully functional demonstrator for validation (Section 3). • Evaluation of the correlation mechanisms of electrooptical and current measurements with predicted values, including dynamic judgement of deviations and validation (Section 4).

| ADVANCED METHODS FOR SUPERVISING A DISPLAY
Since a fail-free CMS system is not achievable, the goal of ASIL is to reduce the risk to a tolerable level, classified as safe, by enhancing today's methods with new ones. Therefore, we want to safeguard the transformation of pixels into visual information within the panel. In other words, we want to determine if a designated RGB pixel information is really shown on the panel. A possible approach is to observe the display by a camera (see Section 2.3, "light-to-light") and determine to which degree the intended image including operational data such as speedometer is reproduced on the display. Such a camera approach is the preferable method for remote operators (see Figures 1 and 4) and could be used in cars as well. However, the latter has some disadvantages in terms of use case coverage (e.g., image occultation during touch screen operation), integration, and cost. Consequently, we checked other methods for measuring the optical output of the display depending on the RGB gray level data and extended today's supervision methods (which ends at the data interface, Figure 3) towards the electro-optical conversion. It is obvious that the time-resolved supply current or, as ultimate method the current of every column power line of an OLED depends on the image to be shown. We tested this approach for LCDs and were able to extract the current of the source drivers, which correlates with the reproduced RGB gray levels. The correlation of the incoming RGB gray level data with those generated by an optical (see Section 2.1) and electrical model (see Section 2.2) permits evaluation of the image reproduction (see Section 4). The methods presented in this paper provide an approximation towards the camera solution, by optical measurements at the panel surface and electrical measurements within panel; hence, overcome the disadvantages of the camera approach and provide a set of methods from which one or more can be selected to achieve a required ASIL level for the panel. As being standard for LCDs, power supply and backlight supervision is not described here.
For supervising a display, its optical output and/or power consumption must be compared with the output of models, which are able to convert digital input gray level data to a target value for the measured output data. Such a software model of the display must be developed and parametrized for the intended panel specifically. This is done in an iterative manner via test images and typical content until measured and target values fit at high correlation numbers. Figure 5 visualizes this approach.
The setup of the model consists of calibration measurements with "simple" test patterns (see Figure 5, top left), which leads to corresponding parameters of the model. After that, the model is evaluated with both test patterns and typical automotive human-machine interfaces (HMIs). A simple example will describe the principle of the fundamental approach for the optical model: We assume a single photodiode located closely in front of the display. This photodiode is subsequentially "exposed" to test patterns of increasing complexity. Each pixel is described by the RGB gray level value (RGB GL, here normalized to 1 to be independent of the gray scale resolution) and gamma value 10 : Intensity is used here instead of luminance as a typical photodiode is not corrected to vision (V(λ) curve). To keep the example simple, it is assumed that the "pixel output" is the sum of R, G, and B subpixel. The gamma value is determined by measurements 10 for RGB; the light output (intensity) is also measured for RGB for, for example, the maximum gray level.
As a typical photodiode has a nonlinear acceptance angle, the measured optical output depends on the pixel F I G U R E 5 Fundamental flowchart of supervising a display by comparison of digital input data and measured output data via prediction of the output by a display model location (x,y; horizontal, vertical) and is integrated as sum in the photodiode: where WF(x,y) is the "weighting" factor of each pixel at the location x,y. This is acquired by subsequently activating every single pixel (or a small cluster to save time and effort) reproducing white. This results in a "heat map" of the display under test which is labelled as model. We can now predict the measured intensity of the photodiode for any random image to be displayed and reproduced based on its input RGB gray level data and their locations.
In the next step, simple test patterns are used for a first validation of the model. If necessary, the model is then fine-tuned by iterative approach. The final step is the evaluation with typical automotive HMI content and video images used in our project for validation. As a result, we are now able to predict the measured value as "target" and compare (correlation in more complex cases) this with the actual measurement. If measurement and target value are identical (within defined limits, e.g., ±2%), the reproduction is considered "good." The larger the deviation, the lower the correlation factor gets (e.g., "−2%" deviation leads to 98%). This deviation factor is judged in our APVSS, which performs the corresponding actions like switching off the display or force a reference image in case of display reproduction failures.
The following sections describe methods for optical (Section 2.1) and electrical (Section 2.2) display measurements. These approaches for multiple photodiodes and current measurements follow the previously described method of setup and evaluate the corresponding display model.

| Optical measurements
The main issue of today's methods is that the actual optical output of the display is not supervised (just digital interface data are controlled). Therefore, for achieving highest safety levels, the result of the electro-optical conversion of the display must be monitored by optical measurement. This can be done using different approaches, which are listed in Table 1. It shows the principles and methods of optical measurements including a rough description of the individual implementation and their accuracy in terms of data acquisition. In addition, the table groups the methods according to the necessity of modifications required to be done to the display and the achievable relative safety level, which usually corresponds to cost as well. "No or minor modification" means that the method could be implemented on a display without affecting its electronics.
It is obvious that modifications of displays being produced in relative low volume add significant cost. Adding a photo sensor to every subpixel might at least double the cost of the thin film transistor (TFT) backplane. A further increase in price results from the acquisition electronics for a million (or more) photodiodes. Opposite to that, photodiodes are relatively cheap, and the cover glass can be used as waveguide. A camera-based supervision in a car is basically easy to implement as a camera for driver supervision is on the horizon, and therefore, cost will be low due to mass production. The data analysis of the optical supervision can be performed by a 32-bit microprocessor, which is automotive standard and therefore with relative low cost.
The most accurate method is to integrate a photo sensor 12 for every subpixel in the TFT backplane, which is costly and lacks potentially of ambient light influence. The camera-based optical supervision is the most Optical measurement principles and methods for advanced display safety and supervision effective one for the use case "remote operator" (see Figures 1 and 4) and is described more in detail in Section 2.3. If a transparent OLED is used, the camera can also be mounted behind the display. Wedge light guides 11 base as well on a camera however collecting the light output of the entire display. Because such a light guide has to be placed in front of a LCD, this method is not applicable as the image quality is significantly disturbed by this optical element. Therefore, we focused on the waveguide principle using eight photodiodes to monitor the electro-optical conversion of the display. Three photodiodes are assembled on top and bottom side (vertical) of the display and one photodiode on left and right side. Left side in Figure 6 shows the principle of the optical measurement with photodiodes: Each photodiode is mounted lateral of the display panel, under a shield, with a diffuse reflector, at a 45 angle and captures light reflected by the reflector. However, the photodiode measures not only the actual value of the light output of the display (I Display ) for its specific field of view but also the incident ambient light (I Ambient ). In order to model the intensity of the display, the influence of the ambient light must be determined. This can be done by measuring this intensity (I Ambient ) during the OFF time of pulse width modulation (PWM) backlight driving scheme (or black frame insertions for OLED); see Figure 6 (right); details of LED backlights can be found in, for example, Weindorf and Lee. 13 The measured intensity of the photodiodes during PWM OFF time is subtracted from I Photodiode of the PWM ON level to calculate the actual intensity value corresponding to the light output I Display of the display.
The method of measuring the optical output of a display by photodiodes as shown in Figure 6 (left) for a LCD is as well applicable for OLEDs. However, the acquisition of the ambient light intensity during OFF period of the backlight is not achievable, as OLEDs have no backlight. A work around could be black frame insertion ("displaying" black for all lines for one frame), which is hardly noticeable for higher frame rates when, for example, repeated once per second.
The following dependencies are required for the calculation of the target intensity for correlation (see Section 2): • relative intensity of the location of the pixel, • gray level of the pixels or subpixels, and • spectral dependency of the photodiode.
In the first step, the relative intensity of the location of the pixel has to be calculated. It is obvious that the gray scale-dependent luminance of pixels near the photodiode has a higher impact than that of distant pixels. Therefore, we calibrated the eight photodiodes by using the following procedure: Small boxes of white pixels are shifted horizontally and vertically on the display (Figure 7, left). To obtain a measurement result of sufficient signal-to-noise ratio, the cluster size has to increase with the distance between white box and photodiode. The cluster sizes and their position are stored in a lookup table to be used by the model. This results in an intensity map (Figure 7, right) illustrating the relative intensities of the pixels with regard to the position of the photodiodes.
In the second step, the gray level of the pixels or subpixels and the spectral dependency of the photodiode are required for the calculation of target intensity for correlation. To determine the gamma curve, as a next calibration step, the photodiode intensity is measured for each RGB primary; Figure 8 (left) shows the part of the intensity distribution for the green color. This behavior corresponds to the "theoretical" gamma curve. As it turns out that some displays have different gray scale characteristics for the primaries and cross RGB color effects can happen, we measured all available color combinations of the individual subpixels, in total 16.7 million; this was performed computer controlled. With this method, the mutual dependencies of the subpixels are measured. An example is plotted in Figure 8 (right) where the intensity dependency between of red and the blue subpixels are F I G U R E 6 Left: Measurement for optical output via light guide principle for LCD (as shown) and OLEDs. Right: Photo diode measurement of a snapshot of the backlight output: The measured ambient light intensity during OFF duration is subtracted from the measured value of the photodiode during ON period visualized. It is easy to notice that saturation happens for these two primaries starting at about gray level 200 (8-bit). Furthermore, the maximum intensity for gray level 255 of the two colors is nearly identical. This effect results from the spectral dependency of the photodiode since its maximum intensity is at 700 nm (red is then stronger weighted than blue). Otherwise, one would expect a much higher intensity for the blue color. Those measured values enable the calculation of the required color combination matrix, in which a value for each color combination is stored. Such a precision is required for the model display data to achieve high correlation values.
To calculate the target intensity (voltage) of every photodiode for random images, a combination of the intensity map of each photodiode and the abovedescribed color combination matrix of the subpixels is taken into account. First, the subpixel combination of each pixel is replaced by the corresponding intensity due to the spectral dependency. Second, the image is divided up into the same cluster sizes as for the calibration. For each of these clusters, the average is calculated and then multiplied by the respective intensity of the map illustrated in Figure 7.
Finally, the evaluated target voltage (intensity) is correlated with the actual measured voltage of each photodiode. Since the values to be compared are both voltages, the correlation is calculated from the difference between the two voltages in relation to the maximum voltage of the photodiode. If the calculated correlation of at least one photodiode is under a predefined threshold value (e.g., 98%), the algorithm passes an error message to the APVSS in the demonstrator setup (see Section 3).

| Electrical measurement
The easiest method to supervise a display electrically is measuring the supply voltage and the current. This allows checking the basic function of the display; however, no correlation to the data to be displayed. For OLEDs (and LEDs), it is obvious that a current is drawn by a subpixel, which depends on its gray level (luminance); see, for example, Ryu et al. 14 We have measured several LCDs where the current of the column (row, line, and data) drivers is also related to the gray level. This is described more in detail below.
Different current measurement principles are listed in Table 2 and sorted by the required effort that provides as well a relative estimation for additional cost. Further columns refer to the achievable ASIL level and the complexity of modifications of the display. "No or minor modification" means in-line measurement of the total power consumption of the source driver (column driver), which sets the gray levels of the subpixels. When measuring with high temporal resolution, the current of each line of the display can be captured. The ultimate measurement is to measure the output of every column driver in a customized display. For OLEDs, this might be realized for high-quality OLEDs as this current measurement can be also used to compensate for aging (burn-in). However, the current method is a significant step beyond the pure digital interface data supervision; its relevance is below an optical supervision. Our intention is to combine low-resolution optical measurements (see Section 2.1) with current measurements for a higher correlation (and therefore ASIL performance) for any input image compared with each method alone.
Measuring and supervising currents is generally less expensive than optical methods, which, however, offer true "light-to-light supervision." Acquiring the current of integrated circuits (ICs) like row and column drivers is made within the IC and therefore cheap in mass production. The data analysis of the column driver supervision (here) can be performed by a 32-bit microprocessor, which is automotive standard and therefore with relative low cost. Implementing current measurements of each column driver output (LCD) or power line (OLED) results in typical 1920 × 3 (RGB) sensors and ADC (analog to digital) converters, which provide better supervision at higher cost. Multiplexing of those thousands of currents might be a suitable compromise to reduce cost. When OLED panels use measurements of currents for other methods like prevention of burn-in, the additional costs for safety supervision are significantly lower.
Column (source) drivers mainly dominate the modulation of the panel electronics' power consumption over the frame time. We measured the current of all subpixels for each given row (line), which depends on all RGB gray levels of this whole row. A vertical resolution of 720 pixels means a total of 720 current levels per frame, integrating all subpixel currents of a single line (row). This principle is shown in Figure 9 on the left and an example of a test pattern on the right. Monochrome and color test patterns from full-screen images over checkerboards to modulation transfer functions (MTF) and automotive HMIs are measured (see Figure 5). This allows creating a model for the target current over frame time for any displayed image (RGB gray level subpixel data) and comparing it with the actual measurement value for correlation.
The model with reference parameters (see Section 2 and Figure 5) is used to predict the current levels of any image (gray level RGB data) row by row. To do so, the mean current level of a row is determined by calculating the arithmetic mean of all subpixels determined by using the matrix. Now, it is possible to correlate the actual measured current I with the predicted target current Î of the model. As basis of the correlation, the line-wise meansquare error (MSE) is calculated using the formula: To increase the chances of detecting a block of slightly erroneous displayed lines, the deviation is weighted depending on the number of consecutive lines deviating more than 1% from the target current. As a result, short deviations, for example, noise, are suppressed, and deviations over multiple lines, for example, a defective line driver or a wrong gamma voltage, affect the correlation much more. Summarizing, the measurements of the cumulated current per row is relatively easy to implement in existing panels and provides significant judgement on display reproduction.

| Camera-based supervision
Supervision by camera of the monitor of a remote operator or an in-car display (which might be difficult to integrate) can also be performed. We investigated the remote operator use case ( Figure 1) using a commercial monitor and a PC (typical for this application). The captured monitor "picture" is analyzed with image processing methods to judge on image quality, reproduction, and operational data using methods presented in Figure 4 towards ASIL. "Traditional" CMS methods of in-car systems (Figure 4, top) are not applicable for remote supervision. Special challenges are image compression for efficient transmission and consumer electronics networks including latency. Figure 10 illustrates our test setup using a standard consumer electronics monitor and a camera (FHD resolution), which is clipped on to the monitor. Video signal processing is performed based on standard algorithms (see, e.g., Solomon and Breckon 16 and below) in the corresponding PC, in which the APVSS is implemented as software. This software compares the incoming metadata and operational data with extracted data from the camera.
The video content presented on the monitor of the remote controller is composed of two planes, as it can be seen in Figure 11: First, the video data, for example, from a rear-view camera, and second, the augmentation data (trajectory, etc.) as an overlay. The processing unit in the car, which merges video and augmentation data, extracts characteristic features from both planes. The video feature data include time stamp, the previously introduced metadata and other control data. The overlay feature data consist of the location and the type (e.g., parameters of lines and curves) including RGB gray levels of augmented tell tales. All feature and operational data (such as speed) is transmitted to the APVSS.
When setting up such a system, an initial calibration is performed for geometric distortions due to the mounting position of the camera and distortions by the camera lens (e.g., barrel). Furthermore, gray scale and color reproduction and their acquisition are not necessarily linear. The initial calibration is performed by test patterns such as full-screen white (Moiré), grids (distortion), and boxes of equidistant gray levels (gamma). The captured video data during operation are processed as shown in Figure 12 after the power on test (Moiré, distortion, gray scale, and color). As Moiré patterns can appear, they are removed by slight defocusing and/or low pass filtering if necessary.
In order to control permanently the gray level reproduction quality of the monitor, nine boxes of equidistant F I G U R E 1 0 Camera-based supervision of a remote operator's monitor for controlling for example autonomous cars without steering wheel F I G U R E 1 1 Block diagram of proceeding towards camera-based supervision. The ASIL Prepared Video Safety System (APVSS) correlates feature and operational data of the supervision camera with the car video source including augmentation and operational data sent by the car gray levels from 0 to 255 are displayed close to the camera. This enables the calculation and improvement of the actual gamma value (including the influence of ambient light 17 ), and the operator can as well visually check the performance. In the case of deviations beyond defined limits, the APVSS sends an alarm and notifies the operator accordingly.
The next steps are the feature extraction (metadata; see Figure 4) using the same algorithm as performed by the car camera(s) and taking the augmentation into account. Furthermore, operational data that are transmitted with the video metadata are compared with visualized data extracted by symbol recognition (see, e.g., EPSON 18 ); an example is the speed "90" (Figure 9, bottom right, "90"). Finally, the APVSS compares the transmitted and reproduced data and outputs the status (e.g., OK) or a warning.
As pointed out above, using supervision by camera, operational data of a remote operator's monitor can be secured without the necessity of the monitor being electrically modified or additional photodetectors built in. The supervision camera can be clipped on to the monitor. Due to the calibration procedure explained above, the camera position can vary within a certain range to ensure reproducible accuracy and reliability. For the remote operator use case, there is hardly any other method than using a camera. The costs for hardware are estimated as being about 100 USD as commercial of the shelf (COTS) cameras can be used. The major cost driver is software development and qualification. This must be judged however in terms of the benefits provided by remote operators.
A camera-based monitoring system for in-car automotive displays involves some challenges, such as integrating the camera in front of the display, difficulties to realize the monitoring of touch displays. The costs for such an in-car camera system are roughly the same as for camera-based supervision of the driver but the cost for other optical and electrical supervision inside the display is saved.

| DEMONSTRATOR FOR POC OF ADVANCED ASIL-ENABLED DISPLAYS
The proof of concept (POC) demonstrator shown in Figure 13 was built within the APVSS project to develop, test, and evaluate advanced methods for CMS from the camera over video interfaces (link) and modifier (e.g., head unit) to the display, including signal processing and display measurements.
A series production APVSS can base on a 32-bit microprocessor, which is automotive standard and therefore with relative low cost. The feature extraction (not in focus of this paper) can be implemented at relative low cost in hardware.
The APVSS demonstrator is using all new mechanisms described in Figure 4 and Section 2. With all those mechanisms, it is ensured that the display only receives valid video data. The demonstrator mock-up ( Figure 13) in detail: • The camera delivers a video stream with 1920 × 720 pixels with 60 Hz frame frequency. • In the next step, the "feature extraction" captures the metadata of the video images at the input system; this mechanism is performed in a very fast computing unit (field programmable gate array [FPGA]). Additionally, in this FPGA, image manipulations ("Modifier") can be made in order to test the safety system APVSS. "Feature extraction" algorithms are beyond the scope of this paper. • The original video signal is transmitted via an automotive SER to a DES representing a typical automotive in car transmission as point-to-point connection. • These video data are processed in a second FPGA for integrity of video data by metadata comparison. • The video output of the second FPGA is the input for the display that receives only valid video data; however, slightly degradations ("graceful degradations") are allowed. • The display is supervised by measuring voltages, currents, optical output, and backlight as described in previous chapters. Faulty images can be passed to the display for simulating display degradations and defects. • A Nvidia JetsonNano system connects prototypically all components of the video path and the display supervision system via UART to the APVSS.
The block diagram in Figure 14 illustrates the display subsystem more in detail. The optical measurement is done by photodiodes, which monitor optical output, and the electrical safeguarding by measuring the current through the column (source) drivers. "Traditional" methods like voltage control and backlight supervision are implemented as well.

| EVALUATION OF PROTOTYPE METHODS FOR IDENTIFICATION OF DISPLAYS' TYPICAL FAILURES
It is important to verify the implemented methods in order to know how well the methods work. Figure 15 (left) shows evaluation results of the methods of the optical and current data acquisition as well as the graphic user interface (GUI) for the prototype evaluation. The actual measured (red waveform) and the target (blue waveform) current, this is displayed in an oscilloscopelike manner. In the given example, it is visible that the measured current correlates with the waveform generated by the model.
The correlation for the photodiodes is indicated in a traffic light manner (red: not safe, yellow: some F I G U R E 1 3 Image of the hardware prototype mock-up: Video path from camera to display including involved components regarding of monitoring of the Camera Monitor System (CMS) overall system status monitoring F I G U R E 1 4 Block diagram of the advanced display supervision by optical and current measurements deviations, and green: no or minor deviations). All correlations and other measurement values such as supply voltage and backlight performance are visualized in numerical format and traffic lights for quick overview. We used an automotive 12.3 00 LCD with 1920 × 720 pixels, 1000 cd/m 2 , and LVDS interface.
We performed evaluations with several different kinds of images (but mostly automotive HMIs) to test and improve our new methods. Figures 16 and 17 show display failure examples that are not or hardly noticeable by a driver. A failure of a row driver ( Figure 16) suppresses warnings (red) at the bottom and distortions in gray level reproduction (here, too bright; Figure 17). This failure results in the loss of significant and safety-relevant details of the rear-view camera image. The deviations are noticeable for the optical as well as the electrical supervision. This is visualized by red "lamps" in the GUI and highlighted by magenta ellipses and arrow, respectively.
In this section, the optical method is examined in more detail and performance tests are performed. Differences in gray level are simulated in a forced false reproduction area for many different area sizes. This block is located directly at the photodiode and is erroneously inserted on the display (Figure 18, left). The output image content for the target intensity calculation the background (BG) gray level was set from 0 to 255 in 17 steps. In addition, for each output image brightness, the erroneously displayed area (foreground [FG]) is displayed with different brightness levels (the same steps as BG), so that the correlation of all combinations is calculated. Figure 18 (right) shows the result of the evaluation measurement for an area size of 75 × 150 pixels. For a low BG gray level (below 75), the faulty reproduction is reliably detected with a FG to BG difference of 90 gray levels. With a BG gray level over 75, a gray scale F I G U R E 1 5 Example, method, and graphic user interface (GUI) for displays' advanced safety of in-car Camera Monitor System (CMS) F I G U R E 1 6 Examples of display failures and successful detection by correlation of optical and current measurements: Row driver failure that is not noticeable for the driver; warning data (red) are not shown on the display (top right) difference of 45 is sufficient to detect the faulty image. The horizontal size of the forced false reproduction area is proportional to the error recognition, so that the simulated display errors are better detected as the size of the incorrectly displayed pixel area increases.
The chance of successfully detecting a fault increases with its severity. A single pixel makes no difference big enough to stand out of the noise. A malfunction of all row drivers would be caught by other methods too, so this one would not be necessary. As shown in Figure 19, with our method a row driver failure is detected if it affects more than 20 consecutive lines of a 720-line high image, which is less than 3% of the entire image. A faulty gamma reproduction (see Figure 20) is detected when the difference results least four gray levels (of 255), which is about 1.5% of the entire gray level range. The method reliably detects a display error as soon as the correlation falls below 98%. However, with the measurement of column driver output currents, this threshold could be raised to 99%.
As conclusion, the demonstrator is suitable as POC of enhanced display supervision. With the implemented electrical and optical monitoring, safety-relevant defects, that the driver might not be able to see directly, can safely be detected. These methods generate a real added value to the already existing state-of-the-art methods. Although the electrical method works more accurately, the optical monitoring with photodiodes must be implemented as it monitors the actual light output of the display. The combination of eight photodiodes and rowbased current measurement results in very safe detection of typical display failures.

| SUMMARY
We have successfully developed, verified, and evaluated new methods for increasing the safety (safeguarding) of the image reproduction of displays in automotive CMSs by: • Optical and electronic comparison of target values (calculated via the actual image RGB data based on test patterns and models) and actual measurements for displays integrated in vehicles.
• Monitoring of displays by a camera and correlation of target and reproduced image as the "ultimate" method. This can be used for remote operator monitors of selfdriving cars without steering wheel in case of emergency or can be integrated into vehicles pinpointing the CMS display.
We have proven our concepts and methods in a fully functional demonstrator with a prototype APVSS. This enables these methods for future deployment in automotive video architectures.
Our evaluations show that our methods are able to detect deviations between the RGB data, send to the display panel, and the actual visible data on the screen. Despite the sensitive detection of deviations, our methods also allow small, adjustable deviations to enable a tradeoff between safety strictness and best user experience depending on the ASIL safety goal.
Our findings help to implement advanced approaches in displays (cars and remote operator) in terms of ASIL safety, efforts, and costs. This enables as well a "safety design kit," where several methods can be chosen from, to achieve a desired safety level. The ultimate goal is "light-tolight" supervision from camera to optical display output.

APVSS
"ASIL Prepared Video Safety System," a system which is introduced in this paper for supervision of displays in terms of safety ASIL Automotive Safety Integrity Level, see, for example, ISO 26262 2 ; risk classification scheme that helps to define safety requirements BG Background of, for example, a character (foreground, FG) Frank Langner is working as a safety specialist at R&D of Mercedes-Benz AG, Stuttgart, Germany, and heavily involved in the APV project towards implementation in series production. With more than 10 years of Safety Experience, he worked as FuSa Engineer and FuSa Manager in several projects in OEM and in Tier1 projects, partly in instrument cluster projects. Frank graduated in electrical and information technology at the University in Darmstadt and started his career in the Test and Systems Engineering.
Karlheinz Blankenbach graduated in physics from the University of Ulm, Germany, where he also received his PhD in 1988. Until 1995, he was with AEG-MIS, Germany, a subsidiary of DAIMLER. In 1995, Karlheinz was appointed as full professor at Pforzheim University, Germany, launching the university's display lab. His main R&D activities are on display systems, display hardware, and software, as well as display metrology. This resulted in many lectures, papers, and projects (governmental and industrial funded). He is member of the SID subcommittee "automotive/vehicular displays and HMI technologies" (AVH) and SID's "International Committee for Display Metrology" (ICDM).
Matthaeus Vogelmann received a bachelor's degree in Electrical Engineering and Information Technology in 2018 from Pforzheim University, Germany. A highlight was a term at Chico State University, California. Since 2018, he has been a research assistant at Display Lab of Pforzheim University. His R&D activities focus on display safety, image processing, and display metrology. He is highly fascinated by displays since he attended several conferences including SID's Eurodisplay 2019.
Adrian von Gaisberg received his bachelor's degree from the University of Pforzheim, Germany, in Electrical Engineering in February 2020. He has been active in the ASIL Prepared Video Project since May 2019. His research interests range from innovative induction hobs and metamaterial to display safety. The topic of his bachelor's thesis was advanced methods of display monitoring.
Lukas Strobel finished his Bachelor in Computer Engineering at the University of Pforzheim, Germany, in February 2020. Lukas' thesis was embedded in the ASIL Prepared Video research project dealing with the optical monitoring using photodiodes and correlation optimization. He has further extended knowledge in automotive bus systems.