You may need to utilize the gpu_memory_limit and/or lora_on_cpu config alternatives in order to avoid functioning away from memory. If you continue to run outside of CUDA memory, you can try to merge in system RAM https://www.asgdfx.com/