Self-aligned triple patterning (SATP) lithography is one of the most promising technologies for next-generation semiconductor manufacturing process. Self-aligned patterning attracts much interest because of its significant advantage over the litho-etch-litho-etch patterning in reducing the overlay problem in lithography. However, pattern decomposition in SATP is challenging due to its counterintuitive mask synthesis. It remains relatively unstudied and its practical solutions remain to be proposed. This paper proposes an effective algorithm for SATP layout decomposition without grid-based quantization and thus substantially reduces the number of variables and constraints in solution search. Boolean satisfiability (SAT) and integer linear programming (ILP) are exploited for efficient computation. In addition to deriving high-quality layout decomposition solutions with overlay minimization, our method also allows nondecomposable spot identification to facilitate layout rectification. Experimental results demonstrate the superiority of our method compared to prior work and show the relative advantages of SAT and ILP formulations.