411 points | 8d ago | Discuss on Hacker News | Back to Radar
Wishing you unlimited power and may you never see a day of low battery. Respect and Love. Keep it up.
> GGML is a C library for machine learning (ML) - the "GG" refers to the initials of its originator (Georgi Gerganov).
https://github.com/rustformers/llm/blob/main/crates/ggml/REA...
> During the start of his work on the system, he stored the files under the name "Freax" for about half of a year. Torvalds had already considered the name "Linux", but initially dismissed it as too egotistical.
For example, DJ Delorie's djgpp project, built in 1989, was how I got into "real" programming on MS DOS.
Fun story time. I started programming on Turbo Pascal. I had read some C code from those demo scene guides, so felt I understood it fairly well by the time I got my hands on a copy of djgpp. I decided to port a relatively simple program I had written in Turbo Pascal, about 300 LOC, line-by-line. It was a small demo just doing a simple rotating, texture-mapped triangle.
I carefully converted it, and went over it several time one line at a time, fixing it until I had read over it three times and was happy.
I then ran the djgpp compiler...
Now, what I hadn't realized at the time was the difference between C and C++. The examples I had seen was C, and I didn't really know what C++ was.
The compiler churned on my code and spit out about 3500 errors and warnings (it counted for me).
I was shocked. How did my 300 line C program manage to produce over 10 errors and warnings per line, on average?!
I spent a long time trying to figure this one out. The first error was inside the first function, and the error made no sense to me.
After several days of carefully going over each line and banging my head against the table, I suddenly saw it. There, at the very start of the program, before the line the first error or warning pointed to.
In Turbo Pascal I had used const to define things like screen width and height. So I'd have
const SCREEN_WIDTH = 320;
const SCREEN_HEIGHT = 240;
Well, C doesn't have const, but I had picked up it had #define. However, my Turbo Pascal habits had kicked in, and so I had written #define SCREEN_WIDTH 320;
#define SCREEN_HEIGHT 240;
I removed the two semi-colons and the program compiled and ran just fine...I ended up staying in Turbo Pascal, I preferred a compiler that gave me sensible errors and quickly. I did pick up C++ much later, and I guess ironically came to appreciate its const correctness.
And we can only hope that our modern robber barons will use their ill-gotten gains to create something as beneficial as Carnegie libraries.
Imagine trying to name a file format for yourself even if it wasn't yours to begin with...
edit: Just to verify: I had planned to upgrade today, ran my usual benchmark prior and after and the runtime went from 4:22 to 3:59. That's a really long prompt (50K tokens) so prefill is a good chunk of that and the rest is spent on token generation.
even my PicoLM (https://github.com/whoreson/picolm) completely mogs it in (not-that-)deep-context attention, GEMM kernels, and token generation itself as well, every way: https://gabucino.hu/bench/v1.0-rc3/miqu-q4/
But if you don't have time to evaluate other engines and you simply want a rock solid inference engine, then llama.cpp is one of the best ones to go to.
Of course you will. But that's not the point. The point is that Llama runs just about every model on just about every hardware. You can always improve on any metric by dropping that constraint. But there are a lot of such metrics:
- ease of use
- range of hardware
- accuracy of results
- range of models
Probably others. Llama.cpp may not be best at any one of those. But across all of them it has been consistently good. They've shown they're willing to stay the course as well. This is a foundation level project much like Linux or Apache.
The result is I run my big model (glm 5.3 flash) on llama.cpp and my small utility model (Qwen3 vl 8b) on vllm. Qwen often has better vision than glm (despite the tiny size) and can respond to tool calls from glm and importantly it doesn't use gobs of prefill+kv and is super fast.
35 tok/s for GLM 5.3 flash q4 quant
100 to 200 tok/s for Qwen3 VL 8b ; but it's usually short context, single image, captioning that kind of thing. It's generally sub-second for captioning a single image.
This is hybrid GPU/CPU inference for GLM (9995wx + RTX Pro 6000); and RTX Pro 6000 for Qwen3 VL.
I can't directly swap the setup because GLM 5.3 flash is huge, even q4 quant, it doesn't fit in 96GB. But it does fit in my 768GB of system RAM.
I have an Intel Arc Pro B70. Serves me well! I use Qwen 3.8 27B
Congrats for 1.0.0.
"""
Description
The main goal of ggml is to be a simple, portable, and efficient tensor library for machine learning with minimal setup.
Plain C/C++ implementation without any dependencies
Cross-platform - x86, ARM, RISC-V, LoongArch, PowerPC, s390x, and WebAssembly
SIMD-optimized kernels for x86, ARM, and RISC-V
Broad backend support - CPU, GPU, NPU, and browser
2- to 8-bit integer quantization, plus MXFP4 and NVFP4 microscaling formats
Zero memory allocations during runtime
"""I've had the opportunity to do some fun playing around with zdnn on a z17. I've not tried the Spyre cards, but the on chip accelerator for Telum II is surprisingly capable with smaller models. I initially thought you would basically NEED the Spyre cards to get anything significant running but I was wrong.
Comments are loaded live from Hacker News and are not stored by Mid or Real.