Field Notes

Unlock long videos automatically, transcribe audio cheaper, build sturdier agents

Today is about less manual work in video and audio. And about agents that slip up less in practice.

Key AI news

Pegasus 1.5 analyzes full videos and returns timestamps as JSON

TwelveLabs Pegasus 1.5 is now available in Amazon Bedrock. The model chops long videos into segments and returns structured hits with timestamps as JSON. New: video segmentation, multimodal prompting with image references, and processing without preindexing directly from S3 or as Base64 up to 25 MB. One request can cover up to two hours of video. Two analysis modes are available: general Q&A or time-based metadata. Provisioning runs on demand in us-west-2 via InvokeModel. The data stays within AWS borders. If you have many videos in S3, you can automate search work without your own index.1

Deepgram: 11 to 70 percent more audio per dollar on AWS Trainium2

Deepgram has ported its Nova-3 speech-to-text model for offline batch transcription to AWS Trainium2. Eight replicas run per trn2.3xlarge instance. In the measured batch workloads they processed 11 to 70 percent more audio per dollar than the previous options. The calculations use the instance prices at the time and come from AWS and Deepgram. Customers who run Deepgram in their own VPC on Amazon SageMaker AI can benefit as well. For large archive or contact center analyses you cut unit costs.2

Meta brings semantic tokenization to ads recommendation

Meta describes GenRecs Tokenization as a new data representation for generative recommendations. Instead of single IDs the system uses a stable semantic vocabulary mapping that models content and user preferences together. Internal measurement: plus 0.35 percent conversions in February 2026 in combination with GEM and Adaptive Ranking Model. That helps with cold start for new ads and with transfer between campaigns. Note: the numbers come from the vendor. You benefit when your creatives clearly describe text and images, because the system uses both. (Semantic tokenization: items are represented by meaning-bearing tokens instead of rigid IDs.)3

EBU and NVIDIA extend “sovereign” media standards, Adobe speeds up video search

The European Broadcasting Union is working with NVIDIA on referenceable architectures for newsrooms and production. The goal is to use agents with control over data, infrastructure, and security. Microsoft and Nebius bring options for operation in sovereign clouds. NVIDIA cites Nemotron-Open models as a basis for customizable solutions. Adobe in parallel highlights GPU-accelerated search in Frame.io: about 5x faster embedding, up to 4x throughput, and up to 5x lower latency, plus proxy at 2 fps, roughly 80 percent smaller. UFA uses semantic search to find moments by concept, action, or setting. Sovereign cloud means: workloads and data remain under local jurisdiction.4

Highlights for your workday

  • You have long videos in the S3 archive. With Pegasus 1.5 you define the scenes you want. The model returns segments with start and end times as JSON. This lets you build keyword search without preindexing. Requirement: Bedrock access and us-west-2.1

  • Your batch transcriptions are expensive. Check with your team whether Trainium2 fits for offline STT. Either as self-hosting of Deepgram in SageMaker AI or as an offer from your provider. Goal: more audio per budget.2

  • You build agents for internal workflows. Separate router and domain logic. The orchestrator controls order and checks. The domain agents make the domain decisions. This lowers hallucinations in operations. (Multimodal prompting: combine text inputs with images.)5

Tools and updates

  • TorchNeuron simplifies running Hugging Face models on AWS Trainium. Useful if you want to cut inference costs and already use Trainium.6

  • OpenAI describes improvements for ChatGPT in low-resource languages in programming. Includes evaluations for software engineering, SQLite, and COBOL migration. Relevant for legacy modernization.7

  • Vercel Shop for Shopify is generally available. The Next.js template turns agents into “first-class shoppers” and offers MCP interfaces for product search and carts. Good for teams testing shopping automation.8

  • NVIDIA NeMo Gym: guide to improve agent reliability and efficient tool use in chip design workflows. Transferable to other multistep engineering processes.9

Try it in five minutes

A small exercise for the thin orchestrator pattern with your existing AI chat.

1) Pick a recurring process with multiple steps. Example: onboarding a new data column into report, data pipeline, and dashboard.

2) Write a short router plan with fixed steps. Check whether the column exists. Collect required fields. Check warnings. Delegate to “Schema Agent”, then to “Validation Agent”, then to “Population Agent”. No domain logic in the router.

3) Write the instruction for one domain agent. Example for the Schema Agent: “Work only with schema files in folder X. Deliver changes as a patch. No code generation. List open questions.” Test it with a fictitious input.

Expected result: the domain agent delivers a focused, repeatable step without mixing in other subtasks.

Check criterion: count the needed corrections versus a single-agent attempt with the same task. Fewer corrections suggest fewer context interferences.5

Sources

  1. Pegasus 1.5 from TwelveLabs is now available in Amazon Bedrock (aws.amazon.com)
  2. How Deepgram optimized voice AI at scale with AWS Trainium (aws.amazon.com)
  3. GenRecs Tokenization: Transforming Ads Recommendation at Meta with Stable Model Vocabulary (engineering.fb.com)
  4. NVIDIA Advances Agentic Security for Media, From European Public Service Broadcasting to Global Content Production (blogs.nvidia.com)
  5. The Thin Orchestrator: How We Made Our AI Agents More Reliable by Giving the Router Less to Do (engineering.fb.com)
  6. TorchNeuron: running HF models on AWS Trainium is easier than ever (huggingface.co)
  7. Advancing Coding in Low-Resource Programming Languages (openai.com)
  8. @vercel: Vercel and @Shopify are making agents first-class shoppers. Vercel Shop, our Next.js storefront template for Shopify, is GA.

• Built on Shopify's rebuilt Hydrogen SDK • Agents can (x.com) 9. Optimizing Agent Reliability and Tool Calling Efficiency for Chip Design Using NVIDIA NeMo Gym (developer.nvidia.com)