mirror of
https://github.com/NixOS/nixpkgs.git
synced 2026-07-06 17:13:24 -05:00
09ddbaf6f8
On an AMD Ryzen 9 5950X I got:
VLM inference took 29.9s
Smaller LLMs are faster, but the current (Gemma4 E2B Q4)
is more reliable in its outputs and can correctly describe
what's on screen.
Allowing more than `--threads 1` makes it much faster,
but at the potential cost of nondeterministic
output of `llama-cli`.
The VLM screenshot analysis running in CI makes it slower,
but spending < 1 minute single-core seems worth it
given that it quite robustly removes the need for a human
to check if the GUI renders correctly.
Potential future improvement:
A bit benefit would be content-addressing of the screenshot:
Most nixpkgs update would likely not change the screenshot
pixels at all, reproducing it bit-identically.
In that case, there's no need to re-run the VLM at all.
Doing the VLM analysis in a separate derivation that takes
the screenshot as only input would make it ossible
to cache these results, while still being less maintenance
than having a "golden" screenshot that needs to be updated
by a human when screenshots actually change.
53 lines
1.4 KiB
Nix
53 lines
1.4 KiB
Nix
{ pkgs, runTest }:
|
|
let
|
|
testFile = pkgs.fetchurl {
|
|
url = "https://raw.githubusercontent.com/asmaloney/libE57Format-test-data/bbcacec05d60f923869545c5eab33d94c390d50e/self/ColouredCubeFloat.e57";
|
|
hash = "sha256-bb95crNYvX3Qhkx4k6Sqe2GjOf1u4nxxswMfdjyXfTM=";
|
|
};
|
|
|
|
vmTest = runTest {
|
|
name = "e57inspector-vm";
|
|
meta.maintainers = with pkgs.lib.maintainers; [
|
|
nh2
|
|
chpatrick
|
|
];
|
|
|
|
nodes.machine =
|
|
{ ... }:
|
|
{
|
|
imports = [
|
|
./common/x11.nix
|
|
];
|
|
|
|
services.xserver.enable = true;
|
|
environment.systemPackages = [
|
|
pkgs.e57inspector
|
|
];
|
|
};
|
|
enableOCR = true;
|
|
|
|
testScript = ''
|
|
start_all()
|
|
machine.wait_for_x()
|
|
|
|
machine.execute("e57inspector ${testFile} >&2 &")
|
|
machine.wait_for_text("File") # menu visible
|
|
|
|
machine.screenshot("screen.png")
|
|
'';
|
|
};
|
|
in
|
|
{
|
|
vm = vmTest;
|
|
|
|
screenshot-analysis = pkgs.callPackage ./vlm-screenshot-question.nix {
|
|
name = "e57inspector-screenshot-analysis";
|
|
screenshot = "${vmTest}/screen.png";
|
|
question = ''
|
|
Look at this screenshot of a desktop application.
|
|
Answer: Does the application show that a file was loaded into it successfully?
|
|
For this, only scans matter, as there are no images in the file.
|
|
The inspector on the left should show child elements below 'Data 3D'.
|
|
'';
|
|
};
|
|
}
|