Topic: some speedup for ports in FreeBSD

Stephen Montgomery-Smith posted some patches for bsd.port.mk to the ports mailinglist to speed up the package dependency list creation. He did cut down the time from about 2min30sec (package dependency list of gnome2, tested on my system) to about 15-18sec. I enhanced this and now the time is down to about 12sec and a lot less programs to execute in the call (may be important on slow systems).

The patch for bsd.port.mk in my ports-patches directory contains more than only those improvements, the other part is not subject to submission yet.

If nobody finds some problems with the patch I will send it to GNATS and assign it to portmgr for inclusion into one of the next experimental ports builds.

Alexander Leidinger

Nice to see some optimizing in ports smile

F!XMBR

Freedom is the sure possession of those alone who have the courage to defend it. --Pericles

Re: some speedup for ports in FreeBSD

Yeah, nice work.

I have also read that one of the FreeBSD Summer of Code projects is Multithreaded Ports Tree. That would be nice.

"religions, worst damnation of mankind"
[color=Blue]Linux is not UNIX! Face it! It is not an insult. It is fact: GNU is a recursive acronym for

Re: some speedup for ports in FreeBSD

Since you're interested in ports speedup, I'll shamelessly plug my own work here:

http://marc.info/?l=freebsd-ports&m … 38&w=2

The patch there lets you build ports with several make jobs at once. It's great for multi-core machines smile

Re: some speedup for ports in FreeBSD

That would be indeed really nice smile

F!XMBR

Freedom is the sure possession of those alone who have the courage to defend it. --Pericles

Re: some speedup for ports in FreeBSD

vermaden wrote:

Yeah, nice work.

I have also read that one of the FreeBSD Summer of Code projects is Multithreaded Ports Tree. That would be nice.

"Multithreaded" in that you can use multiple make processes (-j option) to compile and install a single port.

Not "multithreaded" in that you can install multiple ports at once.  It's still only a single port build at a time.  But there is progress on getting the tree into shape such that one can install multiple ports at once (in the correct order).  But that will come at a later date.

Re: some speedup for ports in FreeBSD

phoenix wrote:

"Multithreaded" in that you can use multiple make processes (-j option) to compile and install a single port.

Not "multithreaded" in that you can install multiple ports at once.  It's still only a single port build at a time.  But there is progress on getting the tree into shape such that one can install multiple ports at once (in the correct order).  But that will come at a later date.

Yes, several make processes (-j option).

"religions, worst damnation of mankind"
[color=Blue]Linux is not UNIX! Face it! It is not an insult. It is fact: GNU is a recursive acronym for

Re: some speedup for ports in FreeBSD

phoenix wrote:
vermaden wrote:

Yeah, nice work.

I have also read that one of the FreeBSD Summer of Code projects is Multithreaded Ports Tree. That would be nice.

"Multithreaded" in that you can use multiple make processes (-j option) to compile and install a single port.

Not "multithreaded" in that you can install multiple ports at once.  It's still only a single port build at a time.  But there is progress on getting the tree into shape such that one can install multiple ports at once (in the correct order).  But that will come at a later date.

No, actually, the Summer of Code project is exactly about that, allowing several port builds to run in parallel, not make -j. The latter is pretty trivial and doesn't need weeks of attention smile

Re: some speedup for ports in FreeBSD

Actually, it's both, although the SoC summary is very unclear about it.  The initial info that hit the ports mailing list was to enable "make -jX" support for compiling a port using multiple make processes.  Which is what is shown in the [url=http://code.google.com/soc/freebsd/appinfo.html?csaid=9E93133179E28737]summary:

The goal of this project is to mend the current port collection to allow for simultaneousness builds of software contained therein. This will be accomplished by allowing more than one process to build an application concurrently, i.e. through the use of the -jX option, where X is the number of concurrent build processes.

However, the next line in the summary makes it appear that the SoC project will also enable building multiple ports simultaneously:

When several processes are executing at the sametime we will need to avoid the problems of duplicate dependency generation and deadlocks prevention or avoidance.

So, it's not really clear what the SoC will produce.  smile

However, there are patches on the ports mailing list that enable "make -jX" support to the ports tree.

Re: some speedup for ports in FreeBSD

The patch is ready and the commit done in current.

http://www.freebsd.org/cgi/query-pr.cgi?pr=112765

Last edited by Oliver (2007-06-17 11:48:36)

F!XMBR

Freedom is the sure possession of those alone who have the courage to defend it. --Pericles

Re: some speedup for ports in FreeBSD

Perhaps FreeBSD should investigate building ports in "fake" directories and creating packages.  Ports tend to work much better as they are in a "clean" state.  We've got a guy working on this with MidnightBSD.