MPT

Description of MPT

MPT (Mosaic Pretrained Transformer) is a family of open models from MosaicML with 7B and 30B parameters, originally designed for efficient work with long context. MPT-7B was trained on ~1T tokens of text and code, while MPT-30B is a larger version with improved reasoning and programming quality. The base context window is 8K tokens, while specialized models such as MPT-7B-StoryWriter support over 65K tokens of context, making MPT especially suitable for long documents and complex dialogues. The MPT architecture is a decoder-only Transformer with ALiBi for context scaling and FlashAttention for faster training and inference, available under a commercially friendly license. MPT can be used to develop documentation and contract assistants, analytical RAG platforms, chatbots with “long memory,” auxiliary AI tools for developers and consultants, as well as embedded modules for SaaS products. The FreeBlock team will select the right MPT configuration, fine-tune it on your data, build a long-context architecture, and integrate the solution into the company’s infrastructure. If you want to read and understand your big data with AI, order project development based on MPT from FreeBlock.

All Technologies