ShadowPEFT – Centralized and Detachable Parameter-Efficient Fine-Tuning
ShadowPEFT – Centralized and Detachable Parameter-Efficient Fine-Tuning
Unlike LoRA and its variants, which inject trainable parameters directly into the weights of the Transformer, requiring tight coupling with the backbone. ShadowPEFT instead enhances the frozen large base model by adding a lightweight, centralized, pretrainable, and detachable Shadow network. This shadow network operates in parallel with the base model, delivering learned corrections to each decoder layer. Because the shadow module is architecturally decoupled from the backbone, it can be independently trained, stored, and deployed, benefiting edge computing scenarios and edge-cloud collaboration computing.
Share cardActual performance
Launch Intel predictions
Analyze your own launch →Correct prediction on native model
Similar products
Fine tuning and RLHF mistralai 7B using DeepSpeed
Number of layers for efficient fine-tuning. Experiments
Terracotta – Platform for fine-tuning and evaluating LLMs
LLM reinforcement fine-tuning platform to improve LLM output
Open Source Reinforcement Fine-Tuning for Your Agents
Quaterion – x100 faster fine-tuning of similarity learning models
Control every aspect of model training and fine-tuning
Automate Emails by Fine-Tuning Our AI To Your Service
Gradient – a web API for fine-tuning and deploying Llama2
Open-source fine-tuning in a Colab notebook