Color management has been well established for still images and professional videos but not for consumer videos. It is necessary to develop color management that is suitable for consumer videos. A color management system (CMS) renders reproduced color or preferred color on a display through video signal processing. A complete color management workflow includes color processing from a camera to a display. Recent explosive advances of electronic imaging technologies complicate color management in that the chromaticity characteristics of modern displays and cameras depend on their new implementation technologies. Thus, color management is one of the key issues for modern imaging devices. This paper first reviews issues of video color management in the state of the art. Then, a color management workflow is proposed for an example to solve the difficulties of current video color management. The proposed workflow also can be applied to the improvement of the color management for professional videos. In this workflow, a video frame of the scene captured by a camera is output referred to a virtual display, in which its white point depends on the captured scene to account for changing adapted white. The chromaticity parameters of the virtual display are embedded in the ancillary data of a video frame. The chromaticity parameters of the destination display to show the image are embedded in hardware. A color management module in the destination display retrieves these chromaticity parameters for color gamut mapping which maps colors of the virtual display to colors of the destination display. The chromaticity characterization and calibration of cameras and displays are reviewed and discussed. The color gamut mapping algorithms of several color rendering intents are shown for examples. A method is proposed for calculating the color gamut boundary of a device in hardware so that color gamut mapping is able to operate in time for video applications.