We have investigated the initiation and evolution of phase separation in heteroepitaxial InAlAs films. In misfit-free InAlAs layers, cross-sectional scanning tunneling microscopy (XSTM) reveals the presence of isotropic nanometer-sized clusters. For lattice-mismatched InAlAs layers with 1.2% misfit, quasiperiodic contrast modulations perpendicular to the growth direction are apparent. Interestingly, these lateral modulations are apparently initiated within the first few bilayers of film growth, and both the amplitude and wavelength of the modulations increase with film thickness. The saturation value of the modulation wavelength determined from XSTM coincides with the lateral superlattice period determined from (002) x-ray reciprocal space maps, suggesting that the lateral modulation wavelength represents a periodic composition variation. Together, these results suggest that phase separation in the heteroepitaxial InAlAs thin-film system is a misfit-driven kinetic process initiated by random compositional nonuniformities, which later develop into coupled compositional and surface morphological variations.