Use a proxy cache to run the benchs.

[joanhey]

I know that the bench disabled the docker cache, because the docker cache is terrible because use a lot of HD space.

But lastly docker hub is limiting the downloads. https://docs.docker.com/docker-hub/usage/

With a proxy cache, we can have all these limits a problem of the past, with docker hub.
And we'll have faster runs. Without the HD space used with docker.

We can also think the proxy cache with normal packages systems: npm, packagist, ...
But we need to clean the proxy cache before any run.

I can't comment in #10127 because @KostyaTretyak blocked me, some years ago, without understand it for my part.

The important think is that we can bypass the limits and have faster runs. We use almost the same docker images, and from the community, it'll come one that show exactly all the docker images that use this benchmark. And we'll use less HD space than with Docker cache.

[volyrique]

I am not convinced that Docker is the major problem - certainly my own data based on PR #10125 clearly demonstrates that apt install is by far the slowest step of the image build process, and there are plenty of other implementations that are based on an Ubuntu image and that install extra packages with apt.

[p8]

I'm seeing some of the php builds taking a long time.
For example:
https://tfb-status.techempower.com/unzip/results.2026-01-18-12-50-08-571.zip/results/20260110164656/swoole/build/swoole-sync-mysql.log
I wonder if these would benefit from a docker image with PHP already installed, instead of starting with an Ubuntu image and installing PHP from ppa:ondrej/php.
Maybe that repository is throttled as well?

[p8]

Maybe it would also make sense to show timestamps for each build step, so we can see what are the slow steps?
https://support.circleci.com/hc/en-us/articles/360051997052-Debugging-Docker-Build-Step-Timings

[p8]

Cutelyst builds take more than 40 minutes. It installs a lot package on a plain Ubuntu image:
https://tfb-status.techempower.com/unzip/results.2026-01-18-12-50-08-571.zip/results/20260110164656/cutelyst/build/cutelyst.log

[joanhey]

Yes @p8
Almost all PHP fw use the ubuntu image with ppa:ondrej/php, because was faster always.
The new PHP 8.5 (some weeks ago) use OPCache installed by default !!!
Still using the PHP image, we need to install PHP packages to run the fws.

Some years ago, that was not a problem, but now is another history.
Still we need to compare the PHP image (not official) with a Ubuntu image with installed PHP, and so we can check any problem with the PHP image.

[joanhey]

Also the Swoole and other runtimes, they need to build or use their image.
I try to use the image directly, to don't need to compile it.
And like I said before, it's good to have the 2 versions to find problems.

But, we need that other fws also use an image not specially for this bench, only a general image.

EDIT: some years ago, I changed the database builds in the toolset, to use the official database images !!

[volyrique]

To give you an idea of the extent of the issue that I mentioned, during the last complete run, a12d8f43-58bc-4ad1-9277-aa5ab294c10b, h2o spent 21 minutes building its Docker image. Out of these 17:20 minutes were used to run apt install, spread over 2 invocations (13:45 and 3:35 minutes respectively), and 3:37 minutes (3:17 + 0:14 + 0:06) - compiling and linking code. The rest, including downloading images from Dockerhub, is essentially a measurement error (3 seconds to be precise!). In fact, I have already implemented optimizations such as PR #10173 that are trading off APT traffic for downloads from Dockerhub and that have lead to significant reductions in the corresponding build time.

In case someone is skeptical, there have been 12 complete runs as of this writing since PR #10125 was merged, so you could check any of them. Spoiler alert: They all behaved in pretty much the same way, with one exception. The only anomalous run was 3be5b7f5-ca41-451e-af3a-a0e7213e24cb, during which the h2o image build process completed in 3:16 minutes. However, that run started after a week of downtime for the continuous benchmarking environment - the aftermath of power issues, if I remember correctly, and the only significant pause in a year's time! My theory is that whatever was throttling the APT downloads reset its state during the downtime, so they were executing at their normal pace immediately after. However, things were back to (their miserable) normal during the next run. So, this run could probably be used as a recent reference to determine which test implementations are affected significantly by throttling.

[joanhey]

We need to first fix the DockerHub limit.
That we can do we the same Docker registry mirror.

But later we still can accelerate: apt, apk, npm, cpnm, composer, .... installs. It's the same is it only small seconds benefit in each. We need later to multiply by all the frameworks permutations, and the gain will be notable.

Docker, in last versions, also have to cache package managers using --mount=type=cache but we need to change all dockerfiles, a very big job.

It's for that than I said use a Proxy Cache for all, easier to manage, but it's still only 1 idea.

At the beginning was no problem, because the number of frameworks was small. Later with the change to Docker, we tried to maintain clean dockerfiles to reuse the own docker cache. But now it's out of control with the quantity of frameworks.
As the docker cache for layers use a lot of HD space, not is not used or not completely, because the server end without HD space.

[joanhey]

About what said @p8, the problem is not about apt with Swoole or Cutelist. The time is spend in the build stage, that happen with more frameworks, than compile in each run, even without any change in months.

I'll test to use the Swoole image like I did with Symfony Swoole. But this will only fix a small portion of this problem.

[joanhey]

We need to be creative, as we use a lot of frameworks and more than come.

I propose another idea: Run the frameworks benchs, not alphabetically but per language.

So we can use the Dockers layer cache efficiently (the fastest), and also it's easier to find small fixes in the dockerfiles. As any language use almost the same images, apt and package managers.
After finish the language bench, we can clean the cache, and start with another language.

EDIT: I created a PR to update the Perl frameworks to 5.42. It's only 5 frameworks (with 12 variants in total), but they used 3 different docker images before: perl:5.40, perl:5.40-slim and perl:latest (not permitted latest). Now all use perl:5.42 so we can use the Docker layer cache. And we can use 2 images for different versions, but not 3 for the same version.

EDIT 2: but we can't decide about it, only it's @msmith-techempower than decide, but he need to decide or delegate !!!

[msmith-techempower]

I like this idea, but from a cursory glance it seems like MANY framework test implementations are split on which base images to use (looking through Java is a bit of a mess).

I worry that there might be legacy implementations that still work, but haven't been update to run on a new major/minor version, but maybe we should just try something and see how it works.

Maybe the way to do this is to have a Dockerfile in each language directory which simply renames the base image to use - we build techempower/perl:latest but that is just perl:5.42 under the hood in the Dockerfile. Then, a full run would iterate over the languages, build the base image, then run the first test implementation (Dancer, say) and it's first line would be FROM techempower/perl:latest.

This would also make it so that knowing any language base image means you know them all, and an update to the base would cascade to test implementations and allow a single file to be changed to update all test implementations. Downside is that it puts the onus on whoever is updating the language base image to update all the child test implementations... hmm.

Thoughts?

[joanhey]

It was only an idea, but this bench is bigger with the time, and we need to be creative.

I think than it's easier to work the dockerfiles by language, but when we try, perhaps it's not the best. But we need to move to try solutions, to a difficult problem, than only exist in this big benchmark.

[p8]

A lot of frameworks also split the database and non-database test into separate dockerfiles, which results in builds running twice. I can imagine this is done to not see the database name/adapter in the results. Maybe we should have separate fields for the database display name and the non database display name? This would allow us to merge some of the tests to shorten the total test time.

[joanhey]

@p8 you are correct, also I closed a lot of tests than bench the Json and plaintext with each database bench 😲 . That 2 tests have the same results with any database test.

But like I said, this benchmark grow a lot very fast, and it's a work from all try than is fast and fair play.