Article
Tensor Parallelism
By
•
•
14Aniket Maurya
aniketmaurya
AI & ML interests
Computer vision, NLP, MLOps
Recent Activity
upvoted
an
article
16 days ago
Tensor Parallelism
updated
a model
26 days ago
aniketmaurya/receipt-model-2025
liked
a model
26 days ago
aniketmaurya/receipt-model-2025
Organizations
aniketmaurya's activity
upvoted
an
article
16 days ago
updated
a
model
26 days ago
liked
a
model
26 days ago
published
a
model
26 days ago
replied to
singhsidhukuldeep's
post
26 days ago
woohoo thanks for checking LitServe @singhsidhukuldeep ! LitServe now has OpenAI API-compatible endpoint and you can also serve a LLM using vLLM engine with LitServe so you get both speed + flexibility.
reacted to
singhsidhukuldeep's
post with 🚀
26 days ago
Post
894
Just tried LitServe from the good folks at
@LightningAI
!
Between llama.cpp and vLLM, there is a small gap where a few large models are not deployable!
That's where LitServe comes in!
LitServe is a high-throughput serving engine for AI models built on FastAPI.
Yes, built on FastAPI. That's where the advantage and the issue lie.
It's extremely flexible and supports multi-modality and a variety of models out of the box.
But in my testing, it lags far behind in speed compared to vLLM.
Also, no OpenAI API-compatible endpoint is available as of now.
But as we move to multi-modal models and agents, this serves as a good starting point. However, it’s got to become faster...
GitHub: https://github.com/Lightning-AI/LitServe
Between llama.cpp and vLLM, there is a small gap where a few large models are not deployable!
That's where LitServe comes in!
LitServe is a high-throughput serving engine for AI models built on FastAPI.
Yes, built on FastAPI. That's where the advantage and the issue lie.
It's extremely flexible and supports multi-modality and a variety of models out of the box.
But in my testing, it lags far behind in speed compared to vLLM.
Also, no OpenAI API-compatible endpoint is available as of now.
But as we move to multi-modal models and agents, this serves as a good starting point. However, it’s got to become faster...
GitHub: https://github.com/Lightning-AI/LitServe
-
- 1 reply
liked
a
model
9 months ago
upvoted
an
article
9 months ago
Article
PaliGemma – Google's Cutting-Edge Open Vision Language Model
•
240
liked
a
model
11 months ago
liked
2
models
about 1 year ago
liked
2
models
over 1 year ago
liked
a
dataset
almost 2 years ago
liked
a
Space
over 2 years ago