Back

For me, back in the days, PHP didn't have composer, the way to install module is system wide.

Imagine PHP without composer.

@ikskuh

@dozymoe i'm a bit confused. Do you mean PHP or Python?
I'm non-experienced in PHP, so i can't really know what you mean with composer here.

But pip installs packages globally by default as well, unless you use a virtual env, which i probably should use in my projects, both to archive and reference the correct versions of my deps

composer is per-project package manager, before that it's apt or pecl, I think there is another one, system wide module install that somehow affect the http server too.

The contributed module ecosystem feels inaccessible to me.

PHP was my first introduction to web development, after that it's python.

I have also dab with spring and rails a bit.

@ikskuh

@slott56 Well, one of my use cases of python was a part in my build system where a small task needed to be done.

Problem is that python has a slow interpreter startup on windows, so a simple python -c "" already takes considerable amount of time in that case. Removing that python invocation speed up my build by 2 minutes (out of 18 minutes).

It really depends on what you're building in the end, but afaik the best python speed comes from using FFI libs

@ikskuh what many of us do is discard the shell and rewrite the build system in python. Python does all the work. Instead of being started and stopped for one small thing.

@slott56 Our build system is CMake, which then invokes python, gcc, and a whole bunch of other stuff. Rewriting CMake in Python would be mad ^^

@ikskuh consider the long-running steps started by make. The steps with a small python bit in it. Anything shell-like that starts a small python step can inverted to be in python having python execute the shell steps.

@slott56 The thing i did was having python set two environment variables, then run the compiler as a child process. but together with ccache, the invocation time of python is way higher than the invocation time of ccache itself.

The solution was to yeet that python part, and fork+fix ccache for our usecase.

This way, we removed the whole python dep for that code path.

And no, the inversion wouldn't work for the rest of the stuff

@ikskuh fascinating. Haven’t observed this kind of thing. But. I avoid windows, so my experiences are biased.

@slott56 Windows with Defender is notoriously slow on starting processes. Best would be a single fat build application that wouldn't have to start several thousand executables at all

@ikskuh that’s single app is a very common python use case. Rewriting CMake in general has already been done (https://github.com/tqdm/py-make). Also. https://scons.org/doc/4.1.0/HTML/scons-design/ch03s03.html rethinks Make.

The invoke project can simplify a Makefile. https://www.pyinvoke.org

Your specific makefile can be converted to a python app. Tools like https://pydoit.org make this straightforward.

@slott56 Yeah, but the CMake project i'm talking about is over 40kLOC in size.

You don't just *port* that to something else without the dire need to do so. Otherwise i would've ported it to "zig build" already which suits the task better anyways than sconcs or py-make.

Also we don't use Make, just CMake+Ninja and CMake+MSBuild

@ikskuh Since there's no easy way to unit test that kind of thing, I personally find it worrying when it's that big and untestable.

@slott56 how the heck would you unit-test a software build? Now i'm curious how that would be tackled

@ikskuh for a shell script implementation unit testing is very complicated. (And rarely done.) Testing a makefile and scripts might be doable by tweaking the PATH to provide mock tools. Not sure. That’s why I avoid the shell.

Unit testing a big-old app that does all the build tasks is ordinary python unit testing with ordinary mocks and monkey patches.

An integration or acceptance test might involve a “hello world” fixture that can the built.

@slott56 As said, we don't use shell scripts or Makefiles. We use CMake, which is a completly different thing and not trivial to port to something like Python manually, as we'd have to reimplement CMake from scratch with alls its features on compiler knowledge, dependency tracking, ... and i don't think that's worth the hassle and will probably make the build slower than it is right now

Even then, i don't see how i can unit-test a buildscript. It works if it can fully build the application

@ikskuh unit testing separates the components. It would test the buildscript using mocks instead of actual tools and a testing directory location that’s destroyed and rebuilt each time. (The tools, presumably have their own unit tests, no need to reinvent those.)

@slott56 I mean that part is clear, but still not sure how to do that with mocks.

Because i kinda only have to determine which files are included in the final artifact structure, but that's basically not unit-testable, but only integration-testable.

The whole buildsystem is a monolithic thing which has no "units" in the classical sense.

There is not much *code* to test, 95% of it is declarative anyways. And testing if declarations are correct is definitly not unit testing

@ikskuh disagree. Testing the declarations would be a unit test. A declarative language can still be decomposed into isolated units. It’s often easier because state doesn’t matter.

I often ask how problems are diagnosed. If some small foreach isn’t right, how will it be found? How tested after it’s fixed? Do that up front and the result is unit tests, a sensible approach to trouble-shooting, and often fixes for latent bugs.

@slott56 could you make an example for how a unit test for a CMake script would look like? I still have no idea how that should work in isolation and how it could be tested

@ikskuh cmake is hard. Like all shell-scripty stuff, you're stuck using a mock directory structure. (It's not *technically* scripting, but it's also not an interpreted language that's amenable to mocking.)

The "isolation" means a non-production directory. Maybe another account, if necessary.

Also, replacing gcc and what-not often requires putting a mock gcc script first on the PATH. The mock gcc script could be as simple as `echo $*` so the unit test can see it in the log.

@ikskuh I think find_program() lets you replace real tools with mocks.

@f_dion @slott56 Also good to know! Thanks for the pointers!