1. We focus on real-time monitoring. This is where we see the biggest pain with our customers, so we spent a lot of time researching and building the right metrics that can run at scale, fast and at low cost (and you can try them all in our platform).
2. OpenTelemetry - we think this is the best way to observe LLM app. It gives you a better understanding of how other parts of the system are interacting with your LLM. Say you're calling a vector DB, or making an HTTP call - you get them all on the same trace. It's also better for the customers - they're not vendor locked to us and can easily switch to another platform (or even use them in parallel).
not to mention langsmith? braintrust? humanloop? does that count? not sure what else - lets crowdsource a list here so that people can find them in future
This is a great list, I'm planning on writing some sample apps and blogs about OpenTelemetry for LLM's and this will be helpful. Which are the most popular open source ones amongst these?
But to the point of comparison between these and tools like Traceloop - it's interesting to see this space and how each platform takes it's own path and finds its own use cases.
LangSmith works well within the LangChain ecosystem together with LangGraph, LangServe. But if you're using LlamaIndex, or even just vanilla OpenAI you'll be spending hours to set up your observability systems.
Braintrust and Humanloop (and to some extend other tools I saw in this area) take the path of "full development platform for LLMs".
We try to look at it as developers look at tools like Sentry. Continue working in your own IDE with your own tools (wanna manage your prompts in a DB or in git? Wanna use LLMs your own way with no frameworks? no problem). We install in your app, with one line and we work around your existing code base and make monitoring, evaluation and tracing work.
I started an open list (on github) of awesome open source repos for AI Engineers. It covers repos that help with building RAG apps, Agents, Dataset preparation, Fine tuning, Evaluation, Observability etc. Good to crowdsource these repos and products. https://github.com/sydverma123/awesome-ai-repositories
This is a crowded market, and there are many tools doing the same thing.
How are you differentiating yourself from other tools like:
Langfuse Portkey Keywords ai Promptfoo