Karan Nisar

Small language models are quietly becoming an important part of real AI systems. While building agentic workflows, I’ve found that many steps don’t need a frontier model at all. Smaller models are often faster, cheaper, and easier to deploy. I wrote a short overview of the current SLM landscape. Part 1 of a three-part series. #AI #LLM #SLM https://lnkd.in/g3Vqqd2T

14

1 Comment

Most teams running LLM inference are over-provisioning GPUs “just in case.��� We benchmarked what happens when you combine NVIDIA NIM with NVIDIA Run:ai with the same inference workloads on dramatically fewer GPUs — with minimal throughput loss. How? ✅ GPU Fractions + Bin Packing: Run multiple NIM models on shared GPUs with true memory isolation — consolidate your deployments, increase density, and reduce your compute footprint. ✅ Dynamic GPU Fractions: Let models burst into unused headroom during traffic spikes — no extra GPUs required. ✅ GPU Memory Swap: Keep infrequently used models warm in CPU memory to slash first-request latency. ✅ Inference Priority Scheduling: Prioritize user-facing inference while training flexes around it. The takeaway: You may not need more GPUs — just smarter orchestration of the ones you have. 🔗 Full benchmarks and methodology in the tech blog: https://bit.ly/407QnUs

3

Design Patterns for Better AI Systems: Pattern 2 — Parallel Sequential execution is a great starting point, but it’s not the only option. When tasks don’t depend on each other, agents can run in parallel, significantly reducing overall latency. Example Continuing with the customer support scenario: While one agent analyzes customer intent, another can simultaneously retrieve account details and recent logs. Once both are done, a resolution agent combines the results and determines the next step. Multiple agents working at the same time → faster responses. The Tradeoff Parallel execution improves speed, but it introduces coordination challenges:   • You need to synchronize results   • Handle partial failures   • Decide what happens if one agent finishes late or fails entirely Parallelism is powerful, but it requires clear boundaries and a well-defined merge step. Use it when speed matters and tasks are truly independent.

11

1 Comment

Amazon L5: Applied Scientist vs SDE — why is the comp gap so huge? I came across two offer data points at the same level (Amazon L5), but the compensation difference is huge: Offer 1: AI Applied Scientist (1 YOE, PhD) Base: $210K RSU: $750K over 4 years (vest schedule: 5%, 15%, 40%, 40%) Signing bonus: $190K (1st year) Signing bonus: $115K (2nd year) First Year Total Comp: $437,500 ---- Offer 2: Software Engineer (6 YOE, Master) Base: $200K RSU: $200K over 4 years (vest schedule: 5%, 15%, 40%, 40%) First Year Total Comp: $210,000 ---- Is general SDE demand really declining, or is this just the premium for AI roles right now? Curious to hear what others are seeing 👇

1

If you’re an AI engineer or an AI leader looking to level up your skills in LangChain and LangGraph, this one is for you. I’m teaming up with Arvind Narayanamurthy for a beginner–intermediate, 1.5 hour hands-on workshop where you’ll learn how to move from simple prompts to real AI applications and agentic workflows. Why this matters - Most engineers know how to prompt. Few know how to orchestrate. - LangChain and LangGraph are becoming the backbone of production AI systems. - If you want to build or lead AI engineering teams, you need these mental models. What you’ll learn - Foundations of LLM orchestration and where LangChain fits - Essential building blocks: chat models, prompting patterns, structured outputs, LCEL - Tool calling with MCP adapters - RAG basics end to end - LangGraph workflows and agentic architectures - Two live labs: a micro-app and a simple agent - Quick dive into LangSmith for debugging The workshop is recorded, and everyone gets the replay. If you want practical, real-world skills to build modern AI systems, this is the one to sign up for. Grab your seats: https://lnkd.in/eW2WZdcf

164

27 Comments

Document-centric tasks sit at the core of many enterprise, business, and government workflows. Search is important, but the real challenge is going beyond retrieval, enabling systems to reason across documents, verify facts, and handle multi-step information tasks. Great to see a new model from Databricks moving in this direction. I expect we’ll see many more models, including open-weight ones, designed specifically for document-driven workflows, an area with huge potential across enterprise and government use cases.

23

1 Comment

Einstein didn’t use '8 years' of 'test-time compute.' This is a category error. While the test-time overhang is real, the scaling at inference has brutal diminishing returns (more so than pretraining). Nobody's torching 8 years of compute praying for Physics 2.0. This is science fiction.

203

91 Comments

🧠 TOON: The Smarter, Lighter Alternative to JSON for LLMs Today I came across something that genuinely changes how we think about feeding structured data into Large Language Models: TOON (Token-Oriented Object Notation). If you’ve ever tried putting large JSON blocks into an LLM prompt, you already know the pain: 🔻 too many tokens 🔻 too much duplication 🔻 too many brackets 🔻 models getting confused by repeated keys JSON is great for machines, but it’s terrible for token efficiency. Every {, }, :, and repeated key costs extra tokens… and when you’re working with agents, multi-step reasoning, or large datasets inside a single prompt, that overhead matters. 🔹 What TOON Solves TOON restructures data in a compact, human-readable, LLM-friendly way that uses dramatically fewer tokens than JSON or YAML. Less noise leading to more reasoning space to get better model accuracy. It was introduced publicly on Nov 2, 2025, credited to Johann Schopplich and contributors. Built specifically for LLMs (not web APIs) and it optimizes structure so models interpret data more reliably. 🚀 Why This Matters In one of my own projects, the entire dataset fit into the context window, so we didn’t even need a RAG system. But the JSON made the prompt bloated and the model started: • misreading fields • losing relationships • confusing repeated keys The moment I saw TOON, it clicked that this might be the kind of efficiency we needed all along. 🔍 LLM Inputs = A Data Efficiency Problem The future of AI agents isn’t just about better models; it’s about how efficiently we feed them data. Formats like TOON open the door to lighter prompts, lower costs, and more reliable reasoning. Excited to experiment with this! If you’re working on LLMs, agents, or embedded AI systems, this is worth looking into. #AI #LLM #DataEfficiency #TOON #MachineLearning #ArtificialIntelligence #GenAI #PromptEngineering #AIInnovation #FutureOfAI #TechTrends #LearningEveryDay #ContinuousLearning #CuriosityDriven #GrowthMindset #AIDevelopment #LLMOptimization #TechCommunity #NewTools #ExploreAndLearn

11

Microsoft Just Made 100B Parameter LLMs Run on Your Laptop. Who Governs Them? BitNet breaks the GPU barrier. 1.58 bit quantization. 100 billion parameters on a single CPU. No cloud. No API costs. Local inference at human reading speed. This is a genuine breakthrough for accessibility. But accessibility without governance is liability. Running locally solves the infrastructure problem. It does not solve the governance problem. A 100B parameter model on your laptop can still hallucinate with confidence. It can still produce biased outputs. It can still make decisions with no audit trail. It can still violate constitutional bounds you never defined. Privacy from cloud providers is not the same as privacy from the model itself. Local execution means your data stays on your device. It does not mean the model operates within governance constraints. The question is not where the model runs. The question is whether the model can prove it operated within bounds. YIN-GATEWAY solves this. YIN-GATEWAY is a cryptographic control plane that sits between any LLM and its users. Cloud model, local model, quantized model, full precision model. The deployment changes. The governance requirement does not. Layer 2 Admissibility Gates reject non-compliant queries before execution. The model never sees requests that violate policy. Layer 3 Constitutional Bounds verify every output before it reaches users. If the output would violate governance constraints, it does not proceed. Layer 14 SENTINEL Verification runs independent binary checkpoint validation. Pass or fail. No partial compliance. Layer 15 Human Override requires cryptographic signatures before high-risk outputs proceed. Not a checkbox. A mathematical proof that a human authorized this specific action. BitNet gives you a 100B parameter model on your CPU. YIN-GATEWAY gives you cryptographic proof that the model operated within constitutional bounds, regardless of where it runs. The hardware revolution is here. The governance layer was already built. https://lnkd.in/eb2EdEwR 32 USPTO applications. 3,057 claims. Any model. One governance layer. Enforced behavior. #BitNet #YINMazariArchitecture #YINGATEWAY #AIGovernance #CryptographicCompliance #Microsoft #LocalAI #LLM #PrivacyPreserving #ResponsibleAI #EdgeAI #CPUInference #OpenSource

4

1 Comment

How do you distinguish a reasoning-native agent platform from a glorified LLM router? One signal: its “context supply chain” is reverse-engineered backward from the agent. Good agent platforms turn context engineering from a templating-level hack into a platform-level primitive. This primitive, the context supply chain, lets agents perform their OWN context engineering at reasoning time -- reliably and at scale: A context supply chain governs HOW data reaches the context window with programmatic guarantees (like defensive typing & permissions), and how those guarantees ADAPT when the model router’s LLM or orchestrator’s task changes (like retrievable sub-prompts & cache invalidation). Reasoning-native platforms design this deliberately. They map context window specs under varied agent types & system states, then BACK-SOLVE into a core architecture that delivers high-signal input tokens on demand: 1. The MINIMUM data for grounding reasoning (w/ provenance) 💾 2. OPTIMIZED intermediate representation(s) 🧮 3. User state BEYOND hand-authored user prompts (selections/scopes/filters/etc) 👤 4. The relevant SUBPROMPTS & POLICIES (often RAG’d) to steer trajectories + goal-conditioning 🧭 5. DEFENSIVE type contracts, permission checks, etc to limit failures 🚧 6. APPROPRIATE caching & idempotency keys for consistency 🗝️ 7. The right BALANCE of tools vs. sub-agents 🛠️ 8. The LLM’s “preferred” sequencing/chunking/etc for the task 🤖 Just like context engineering shapes an agent, context supply chains shape an agent platform’s *core* architecture. Context supply chains should feel like “ROOMS”: ⛓️ Resilient – Handle variability, missing inputs, and exceptions gracefully ⚖️ Optimized – Minimize waste: only deliver what’s needed 🔬 Observable – Traceable from raw data → tools, RAG, etc. → #LLM 🧩 Modular – Swapping, re-routing, and re-combining should be easy (even if not automatic) 🔐 Secure – Role-scoped retrieval, PII redaction, audit logs Technical notes for #AI builders: 1. Freight Consolidation 📦 Treat KV/prefix caching as an SLO. Keep system/preamble byte-stable (e.g. don’t prefix per-req timestamps), make context append-only, and ensure deterministic serialization (stable JSON key order). Reuse sessions/threads to max hit-rate & improve TTFT 2. Supply Chain Quality 🧾 Not just #evals (e.g. LangSmith, DeepEval.Metrics, etc) w/ fault injection + agent-as-judge, but also strong typing (e.g. Pydantic), integration tests (e.g. Pytest & GX), + a flexible but deliberate versioning strategy 3. Overflow Warehouses 🏭 Externalize memory to store bulky intermediate generations in obj storage/FS; keep handles + hashes + distilled summaries in-context so compression is always reversible (w/ a tool call!), never lossy 4. Lane Optimization 🚃 Avoid adding/ablating tools mid-run, to constrain the action space w/o mutating it. Leverage OpenAI’s tool_choice, Anthropic’s response prefill, etc + tool prefixes to gate subsets by state / keep caches warm

48

7 Comments