{"name":"trnrt","repo":"RunanywhereAI/trnrt","url":"https://midorreal.com/project/runanywhereai-trnrt","repository":"https://github.com/RunanywhereAI/trnrt","description":"The open LLM inference engine for AWS Trainium: NKI megakernels, 1M-token context, OpenAI-compatible API. By RunAnywhere.","facts":{"license":{"value":"Apache-2.0","source":"https://github.com/RunanywhereAI/trnrt","checked":"22 Nov 2026"},"language":{"value":"Python","source":"https://github.com/RunanywhereAI/trnrt","checked":"22 Nov 2026"},"homepage":{"value":"https://runanywhere.ai/blog/glm-5-3-flash-trainium2-nki-megakernels","source":"https://runanywhere.ai/blog/glm-5-3-flash-trainium2-nki-megakernels","checked":"22 Nov 2026"},"last_release":{"value":"v0.1.0, 8 Nov 2026","source":"https://github.com/RunanywhereAI/trnrt/releases","checked":"22 Nov 2026"},"releases":{"value":"1","source":"https://github.com/RunanywhereAI/trnrt/releases","checked":"21 Nov 2026"},"contributors":{"value":"3","source":"https://github.com/RunanywhereAI/trnrt/graphs/contributors","checked":"21 Nov 2026"},"archived":{"value":"no","source":"https://github.com/RunanywhereAI/trnrt","checked":"22 Nov 2026"}},"stars":1655.0,"writeup":{"what_it_is":"TrnRT is an open inference engine for large language models on AWS Trainium. It serves an OpenAI-compatible API from one trn2.48xlarge and runs decode as NKI megakernels, with a prefix cache across device and host memory. GLM-5.3-Flash is the first supported model.","audience":"Teams serving large language models on AWS Trainium2 hardware","claims":[{"kind":"specific","claim":"Serves an OpenAI-compatible API from one trn2.48xlarge","excerpt":"It serves an OpenAI-compatible API\nfrom one `trn2.48xlarge`","status":"not_checked"},{"kind":"specific","claim":"Reports 3.57M tokens per minute on one trn2.48xlarge at batch 12","excerpt":"**12** (default) | **28.9** | **3.57M**","status":"not_checked"},{"kind":"specific","claim":"Uses a prefix cache over FP8 KV pages with a host-memory tier","excerpt":"A radix tree over FP8 KV pages, plus a host-memory tier for evicted pages and state checkpoints.","status":"not_checked"},{"kind":"specific","claim":"Requires a trn2.48xlarge to serve, with a compile step on any x86_64 host","excerpt":"any x86_64 host to compile on.","status":"not_checked"}],"alternatives":[],"written_by":"AI, from the project's README","date":"2026-11-18"},"why_now":null,"owner_supplied":null,"score":{"verdict":"overhyped","hype":97,"reality":49,"gap":48,"momentum":null,"confidence":85,"version":"score-2.0","rule":"verdict-2.0","calculated_at":"2026-11-22T07:00:00+00:00","verdict_source":"rule"},"early_read":null,"changes":[{"date":"2026-11-08T19:59:47+00:00","text":"Latest release v0.1.0","source":"https://github.com/RunanywhereAI/trnrt/releases"},{"date":"2026-10-30T18:30:04+00:00","text":"Repository created","source":"https://github.com/RunanywhereAI/trnrt"}],"cite":"trnrt is a GitHub project at https://github.com/RunanywhereAI/trnrt written mainly in Python, described there as \"The open LLM inference engine for AWS Trainium: NKI megakernels, 1M-token context, OpenAI-compatible API. By RunAnywhere\".","attribution":"Data from Mid or Real, https://midorreal.com/project/runanywhereai-trnrt, checked 22 November 2026. Attribution with a link is required.","still_real":null}