Content-aware image retargeting has been investigated since the last decade as a paradigm of image modification for proper display on the different screen sizes. Modifications, such as seam carving or seam insertion, have been introduced to achieve aforesaid image retargeting. The changes in an image are not easily recognizable by human eyes. Inspired by the blocking artifact characteristics matrix (BACM), a method to detect tampers caused by seam modification on JPEG retargeted images without knowledge of the original image is proposed in this paper. In a BACM block matrix, we found that the original JPEG image demonstrates a regular symmetrical data, whereas the symmetrical data in a block reconstructed by seam modification is destroyed. Twenty-two features are proposed to train the data using a support vector machine classification method. The experimental results clearly demonstrate that the proposed method provides a very high recognition rate for those JPEG retargeted images. The source codes and the complete experimental data can be accessed at http://video.minelab.tw/DETS/.