In python3, os.read() automatically retries after EINTR, which breaks
our ability to interrupt on SIGALRM.
Instead, throw an exception from the SIGALRM handler, which should work
on both python2 and python3.
This fixes a rare deadlock during parallel builds on python3.
For background:
https://www.python.org/dev/peps/pep-0475/#backward-compatibility
"Applications relying on the fact that system calls are interrupted
with InterruptedError will hang. The authors of this PEP don't think
that such applications exist [...]"
Well, apparently they were mistaken :)
Traceback (most recent call last):
File "/nix/store/i0835myyhrfr13lh4y26r58406kk90xj-redo-apenwarr-0.42a/bin/../lib/redo/cmd_ifchange.py", line 54, in main
jobserver.force_return_tokens()
File "/nix/store/i0835myyhrfr13lh4y26r58406kk90xj-redo-apenwarr-0.42a/bin/../lib/redo/jobserver.py", line 482, in force_return_tokens
os.write(_cheatfds[1], 't' * _cheats)
TypeError: a bytes-like object is required, not 'str'
Unfortunately I wasn't able to replicate it, but this is obviously the right fix.
Merge branch 'py6' of https://github.com/mlell/redo
* 'py6' of https://github.com/mlell/redo:
Remove python<3.0 restriction in setup.py
Make compatible to BeautifulSoup4
Accept octal representations of Python 2 (0nnn) and Python 3 (0onnn)
Prevent iterator being changed while iterating
Python 2/3 compatible treatment of max(n, None)
Prevent "Exception ... ignored" in `redo-log ... | head`
Distinguish byte (python2 str type) and unicode strings (python 3 str type)
Set file descriptor as inheritable for all pythons >=3.4
Unify print function usage for Python 2 and 3 via __future__ import
Run 2to3 utility
Remove python interpreter selection
In Python 3, `zip()` returns an iterator that in turn contains references to
`tparts` and `bparts`. Because those two variables are modified within the
loop, the iterator does not advance properly.
Fix this by explicitly obtaining a list.
Python 3 does no longer allow comparisons of different types. Therefore,
explicitly convert f.checked_runid to a number that's always smaller than
`f.changed_runid`
When STDOUT is piped to another program and that program closes the pipe,
this program is supposed to terminate. However, Python 3 prints an error
message because prior to exiting, STDOUT is automatically flushed which
does not work due to it being closed by the other side.
This commit includes code taken from the Python documentation for this case.
The output stream is redirected to /dev/null
Apparently in zsh (when in sh compatibility mode), IFS=/ will split
"/a/b/c/" into 5 parts ("", "a", "b", "c", ""). Other shells all seem
to agree that it's 4 parts ("", "a", "b", "c"). zsh seems maybe more
correct to me, but the majority rules, so we'll warn on it.
Meanwhile, we'll also fix the one place in minimal/do that failed due
to this oddity, since it's relatively easy to avoid.
Reported-by: shamrin@gmail.com
As named, the file would be extracted by git on macOS, then
(un)helpfully normalized by the macOS filesystem. After that,
"git clean -fdx" would delete the file, since it no longer
had the expected name, so git thought it wasn't part of its repo.
I considered pre-normalizing the filename, but a) that would break
on any future OS that normalizes differently; and b) that means we
won't test denormalized filenames. Instead, we'll remove the directory
from git, and create it from sh instead, then figure out what name
it got really created as, and then pass the "real" name to redo.
It's not reliable on all versions of docker, and we haven't had time to
fix it yet, so just turn it off. It's not essential to redo.
Reported-by: fcsmith@gmail.com
Open files in 'rb' (read binary) mode to prevent useless default utf8
encoding in python3, without breaking python2 compatibility.
Reported-by: Tharre <tharre3@gmail.com>
Some people have /usr/bin/python as a link to python3. The script is
designed to work in either python2 or python3, but python3's subprocess
model defaults to close_fds=True, which closes the jobserver fds and
causes an error. Explicitly force close_fds=False to get identical
behaviour on python2 and python3.
Reported-by: Tharre <tharre3@gmail.com>
If we print a line indicating that we've started building a subprogram,
then that's an "interruption" of the detailed output of the current
program; we need to "resume" the current program's logs before printing
any new detailed information that comes through.
Apparently on some (probably buggy) platforms, when CDPATH is set, the
'cd' command will always print text to stdout. This caused fail 122
and 123 in shelltest.od.
CDPATH is an interactive-mode option, so let's clear it when running
scripts.
We're going to rename() it from the temp name to the final name, which
doesn't work across filesystems, so the safest option is to keep it in
the same directory.
Reported-by: spacefrogg@meterriblecrew.net
This is a little simpler than the docs/cookbook/c project, which
doesn't actually have a doc yet because there was too much to explain.
I think I might make that a follow-on cookbook chapter, for people who
have read this simple one.
I think this doc is maybe a little too long; I intended it to be
"here's what you do to get started" but it turned into "here's what you
do to get started, and why it works, in excruciating detail." Not quite
sure how to fix.
(Also updated some other parts of the docs to refer to redoconf as a
real thing now instead of a "maybe someone should write this" thing.)
Apparently some newer docker versions crash if there is no "repository"
file in the resulting image. Its syntax seems simple enough, described
in https://github.com/moby/moby/blob/master/image/spec/v1.md
Reported-by: Matthew Singletary <matt.singletary@gmail.com>
I must have changed this at the last minute when adding it, but I don't
know why. We were redefining f4() inside a subshell, so it never
applied to the parent shell, so it was always considered a failure.
But that's not what we were supposed to be testing.
This is just supposed to be a test of the really rare syntax of
defining a function without enclosing braces, which only fails on posh,
as far as I've seen.
If A calls B, and B produces stderr output, and then A wants to produce
output, the resulting log would be confusing: we'd see 'redo A' and
then 'redo B' and then B's output, but no indicator that B has ended
and we're back in A. Now we show 'redo A (resumed)' before A's output.
If B didn't produce any output, or A doesn't produce any output, we
don't bother with the (resumed) line. This seems nice, as it doesn't
clutter the log when there is no ambiguity anyway.
Because redo targets are nicely isolated (unlike make targets), you
usually only want to debug one of them at a time. Using -x could be
confusing, because you might end up with a dump of output from a
dependency you're not interested in.
Now, by default we'll disable -x when recursing into sub-targets, so
you only see the trace from the targets you are actually trying to
debug. To get recursive behaviour, specify -x twice, eg. -xx.
Same idea with -v.
This accidentally made output look different with --no-log vs with
normal redo-log output, because redo-log has a -u option while plain
redo does not.
(This is on purpose. When running redo, you only want to see the things
that actually happened, so it never passes -u to the auto-launched
redo-log instance. But when reviewing logs later, you might want to
look at the past logs from building a component that was unchanged in
the most recent run.)
* redoconf:
redoconf: a stub rc_include() now sources ./redoconf.rc automatically.
redoconf: assorted minor fixes.
redoconf: move -Wl,-rpath flags to shlib.rc and add -Wl,-z,origin.
mkdocs: don't bother to include cookbook/c/out.*/ dirs.
docs/cookbook/c/allconfig.do: avoid need for '&' backgrounding.
Precompiled headers: supply "-x c-header" or "-x c++-header" if available.
minimal/do: remove dependency on 'seq' command.
Fix some build problems on MacOS X.
redoconf: posix.rc: fix abort when timespec is not available.
redoconf: better handling of required vs optional detectors.
redoconf: clock_gettime() detection needs to depend on -lrt.
redoconf: avoid sed -E in _objlist().
Experimental new redoconf C/C++ build/autoconfiguration system.
This lets us remove the awkward ". ./redoconf.rc" line from zillions
of .od scripts, without paying the price of *always* including that
whole file every time default.do.sh is used.
- libqt4.rc: add detection of the 'moc' command in $MOC.
- libssl.rc: add detection script for openssl.
- default.do.sh: provide a die() function to all .od files. It's just
too useful to not have it (I haven't converted everything to it yet).
- When building out/x/y.z, match against $S/default.z.od and
$S/default.od if they exist.
- *.list files (lists of source/object files to include in a binary)
can now contain *.o files in addition to *.c, *.cc, etc.
- rename compile() to _compile() to avoid polluting the namespace for
scripts called from default.do.sh.
- When building a .so file in the top level directory, no need to
make a symlink of it into the top level directory.
- link.od and link-shlib.od forgot to make use of the $xLIBS variable.
* master:
builder.py: atomically replace the log for a given target.
redo-ifchange regression: if REDO_LOG is not set, assume it's 1.
Explicitly reject target/source filenames with newlines in them.
If redo searched all the way up to /default.do, it would run ./default.do instead.
Overridden files were accidentally getting reclassified as static.
Certain redo post-build failures would still mark a target as built.
minimal/do: remove dependency on 'seq' command.
Previously we were truncating the log if it existed. This would cause
redo-log to produce invalid output if you had the following (admittedly
rare) sequence in a single session:
- start building X
- redo-log starts showing the log for X
- finish building X
- redo-log has not finished showing the log for X yet
- start building X again for some reason
- redo-log sees a truncated logfile.
Now, redo-log can finish reading the original file (which no longer has
a filename since it was overwritten) while the new file is being
created.
At some point this got broken during a refactoring. The result was
that redo-ifchange, run from the command line (as opposed to inside a
.do script) would fail to start the log prettifier.
This only happened if the containing project was buggy, ie. you tried
to build a target that has no .do file available anywhere. However, it
resulted in a confusing outcome for that case, where we'd run the wrong
default.do file with the wrong parameters.
Extended an existing test to catch this mistake.
This is relatively harmless, since we treat them *almost* identically,
except that we print a warning for overridden files to remind you that
something fishy is going on.
Add a test for the actual warning message to ensure it is printed. (I
don't like tests for specific warning messages, but it was necessary in
this case.)
Ankur Kothari
Raphael Das Gupta
Aidan Holm
Avery Pennarun
Avery Pennarun
Johnny Lind
Moritz Lell