Foundation models in the vision domain are large machine learning models pre-trained on vast amounts of natural images to extract relevant features from their input data. In the PAFMIM project, we aim at adapting existing foundation models to sensitive medical images, such as CT and MRI data. The challenges include ensuring a good performance of the models on medical images and guaranteeing privacy for the medical images. Our project addresses the challenges in a joint approach combining competencies from the perspective of medical image computing and privacy-preserving machine learning.