Temporary fastening technology has become an essential process in the aviation assembly process due to its functions such as eliminating the initial gap, suppressing the hole-making gap, and increasing the positioning accuracy. However, with the increasing proportion of carbon fiber reinforced polymer used in aviation, new problems have emerged in the installation of temporary fasteners: The contact area between the clamping feet and the composite hole is small. Excessive clamping force will cause damage to the periphery of the composite hole, which brings severe hidden dangers to the bearing capacity and durability of the aircraft composite structure. The deviation of the aperture at the assembly site will obviously aggravate the severity of this situation. Therefore, the size of the temporary fastener installation aperture needs to be carefully studied. In this paper, an FEA model is established in the Abaqus to investigate the effect of aperture deviation on temporary fastening damage. The model includes the reverse model of temporary fasteners and the composite laminates with holes. The 3D Hashin criterion is used to describe the failure of composite materials, the user material subroutine (UMAT) is written to predict the temporary fastening damage process of composite members, and the validity of the model is verified by comparison with experimental results, and the simulation of complex failure damage around the composite hole is realized. The established model is used to explore the influence of the installation aperture’s deviation on the temporary fastening damage of the composite component. The effect of the temporary fastener clamping feet’ installation direction and the degree of the temporary fastener’s off-axis is analyzed in detail. The results show that the deviation of the hole diameter will significantly impact the installation damage of the temporary fasteners of the composite member. As the installation aperture increases, the clamping force that can be applied by the temporary fastener gradually decreases. The installation direction of the clamping feet has little effect on the damage around the composite hole. The off-axis of the temporary fastener will cause serious damage to the composite hole. This paper has also obtained the mapping relationship between the maximum load-bearing clamping force around the composite hole and the hole diameter under the two conditions of no damage and limited damage. This paper’s research results will provide guidance for the tolerance design of the mounting holes on the assembly site and the installation of temporary fasteners to avoid unexpected installation damage to the high-value composite components caused by the temporary fastening process.