Implement techniques to improve performance without significant parameter increase

[vdyma]

Some of the ideas used in LLaMA-2 are already implemented, such as RoPE, SwiGLU and RMSNorm.
With recent paper from @facebookresearch on MobileLLM, it makes sense to implement some more improvements such as GQA (#7), embedding share, make the network deep and thin and also layer share.
Below is the picture that shows overview of the ablation study of MobileLLM:

Status:

• Implement KV-cache #7 GQA
• SwiGLU
• Deep and thin
• Embedding share
• Layer share