mikeash.com pyblog/friday-qa-2011-04-01-signal-handling.html comments

Amitai Hoze - 2016-03-03 17:13:42

Thu, 03 Mar 2016 17:13:42 GMT

Hi, can I use kqueue or GCD to perform non "async-signal-safe" code when my iOS app crashes (e.g. on SIGILL, or SIGTRAP)? Namely, I want to open a dialer using [[UIApplication sharedApplication] openURL:[NSURL URLWithString:@"tel://1111111"]];, is that at all possible?

vb - 2015-01-06 21:52:17

Tue, 06 Jan 2015 21:52:17 GMT

Mike, sorry for commenting on old post.

Is there any way to obtain siginfo_t when handling signal with kqueue?
With epoll on signalfd (Linux) its just as simple as read signalfd_siginfo from fd on which signal is received.

Michael Hecht - 2014-05-20 17:59:43

Tue, 20 May 2014 17:59:43 GMT

Akos, we do something similar. But instead of running leaks at app termination, we run it in response to a specific scripting command in our app's scripting language. This allows us to (optionally) run leaks after each test in a test stream. If a leak is found, we terminate our app which causes the testing driver to capture the leak report and restart on the next test.

I do it using system(). Here's the code:



#define LEAKS_LOG    "~/Library/Logs/DiagnosticReports/JMP-LeaksReport.txt"

void hostDebugLeaks()

{

    /*

     *    This function is called from JSL via the

     *        Debug( Leaks );

     *    command.

     *

     *    It works best if you have launched JMP with the symbol MallocStackLogging set to 1,

     *    like so:

     *        MallocStackLogging=1 ./JMP.app/Contents/MacOS/JMP

     *

     *    If leaks are found, this function writes a leaks report to ~/Library/Logs/

     *    DiagnosticReports/JMP-leaks.log. JMP is then exited.

     *

     *    This can be used with the TestBot in leaks detection mode. In this mode, TestBot will

     *    launch JMP with MallocStackLogging enabled, then run the UT Framework with

     *    _utDailyBuild=1 and _utLeaks=1. This causes UT Framework to check for leaks after

     *    running each unit test. If a leak is found, JMP exits and TestBot captures the leaks

     *    report for the leaking test. Then JMP is restarted and the test stream resumes on the

     *    following test.

     */



#if 0

    // Debugging - force a leak

    int * leaky = new int[42];

    leaky = 0;

#endif



    // Run the 'leaks' command on our process; capture output; look for magic "no leaks" string

    JString leaksCmd( "leaks ^PID 2>&1 | tee ^LOG | grep \": 0 leaks for 0 total leaked bytes.\"" );

    leaksCmd.replace( "^PID", getpid()).replace( "^LOG", LEAKS_LOG );

    int status = system( leaksCmd.value());



    // If status is 0, it means the magic "no leaks" string was found, so we don't have any leaks.

    // Delete the log file and return.

    if( status == 0 ) {

        JString rmLogCmd( "rm -f ^LOG" );

        rmLogCmd.replace( "^LOG", LEAKS_LOG );

        system( rmLogCmd.value());

        return;

    }



    // We have a leak; leave the log file intact and force JMP to exit

    NSLog( @"Leaks detected; exiting" );

    exit( EXIT_FAILURE );

}

I'm sure you can ignore/interpret the parts that are specific to our internal libraries.

Prior to Mavericks, the system() call would hang for us occasionally, because of a crash in leaks as you describe. I had the opportunity to discuss it with an Apple engineer at WWDC a few years back and he thought he knew what the problem was. I submitted a radar and it is apparently now fixed.

Rather than having our app clean up the call stack in the captured leak report as your code does, we have the testing driver ("TestBot" in the above comment) do it. It's a perl script and the cleanup code looks like this:



sub process_leaks_report {

    my( $file ) = @_;

    return unless -e $file;



    my $modified = 0;

    my $contents = '';

    open( LOG_FILE_IN, '<', $file ) || die "Cannot open $file: $!";

    

    while( <LOG_FILE_IN> ) {

        chomp;

        if( s/^(\s*Call stack: .*?:) \| // ) {

            $contents .= "$1\n\t\t";

            $contents .= join( "\n\t\t", reverse split( / \| /, $_ ));

            $contents .= "\n";

            ++$modified;

            next;

        }



        $contents .= "$_\n";

    }

    

    close( LOG_FILE_IN );



    if( $modified ) {

        open( LOG_FILE_OUT, '>', $file ) || die "Cannot create $file: $!";

        print LOG_FILE_OUT $contents;

        close( LOG_FILE_OUT );

    }

}

What interests me is your exclusion of setWindowsMenu:. I see that leak too. Have you reported it? Have you done any further investigation on it?

Akos - 2014-02-12 08:21:00

Wed, 12 Feb 2014 08:21:00 GMT

Thanks, Mike. We are already sending the output to a temporary file. Unfortunately that approach still hangs in pclose when 'leaks' crashes:

#0    0x00007fff83fb96ac in wait4 ()
#1    0x00007fff86afd894 in pclose ()
#2    0x00000001149a409d in DetectLeaks at /Users/asomorjai/Work/DevMain.8/Sources/GSRoot/GSRootDLL/LeakDetectorMac.mm:1442
#3    0x000000011476a25f in GSTermImage at /Users/asomorjai/Work/DevMain.8/Sources/GSRoot/GSRootDLL/GSRootMain.cpp:161
#4    0x00007fff86af1525 in __cxa_finalize ()
#5    0x00007fff86af368b in exit ()
#6    0x000000010000403b in start ()

Here's the actual code; it runs on the main thread:

extern "C" void DetectLeaks (void)
{

    if (!NeedsLeaks ())
        return;

     signal (SIGCHLD, SIG_DFL);

    pid_t    myPid = getpid ();
    char    s[2048], fn[128];

    sprintf (fn, "/tmp/leaks_%d.txt", myPid);
    sprintf (s, "leaks -exclude \"-[NSApplication(NSWindowsMenu) setWindowsMenu:]\" %d > %s\n", myPid, fn);
    printf ("%s\n", s);

    FILE *fp = popen (s, "r");
    if (fp != nullptr) {
        pclose (fp);

        fp = fopen (fn, "r");
        if (fp != nullptr) {
            SInt32 bytesRead = 1;
            while (bytesRead > 0) {
                char buffer [1024];
                bytesRead = fread (buffer, sizeof (char), sizeof (buffer) - 1, fp);
                buffer[bytesRead] = '\0';
                for (char *p = buffer; *p != '\0'; p++) {
                    if (*p == '|')
                        *p = '\n';
                }
                printf ("%s", buffer);
            }
            fclose (fp);
        }
    }
}

How can I detect that 'leaks' has finished/crashed, if —for some reason— I don't get the SIGCHLD signal? Or how can I detect where the SIGCHLD is delivered?

Thanks, Akos

mikeash - 2014-02-12 03:52:06

Wed, 12 Feb 2014 03:52:06 GMT

This is a guess, so please don't take it as basic truth.

I think that leaks will pause your process, at least occasionally, while doing its analysis in order to capture a consistent snapshot of it.

If that's the case, then you end up with a deadlock. Leaks is writing into a pipe that's drained by your process. But your process is paused by leaks, so it can't drain the pipe. If the pipe fills up, leaks will block waiting for you to drain it, but if your process is still paused, you'll never drain it.

A simple workaround would be to have leaks write its data into a file instead of a pipe. Your app can then read out of the file when it's done. You can do this with popen by just tacking on > somefile to the command, or with NSTask by setting its stdout to an NSFileHandle.

Akos - 2014-02-11 17:51:02

Tue, 11 Feb 2014 17:51:02 GMT

May I ask a question?

We are trying to detect memory leaks when quitting our OS X application by calling 'leaks' on our own process. In some cases 'leaks' crashes or hangs, which stalls our app (in a freed/wait4). For some reason I can't catch the SIGCHLD signal in our process.

The leak detection process is called in the termination function, registered with atexit(). The leak detection in its simplest form uses popen/fread/pclose, but I have tried also other approaches, from kevent, GCD, pthreads, and NSPipe. None of them seems to work, I always hang somewhere (wait4, kevent, sigsuspend_nocancel).

I'd appreciate any pointers where I might go wrong; I'm sure it's my inability to capture some important point.

Thanks a lot,

Akos (asomorjai at graphisoft.com)

mikeash - 2014-01-24 15:34:19

Fri, 24 Jan 2014 15:34:19 GMT

I imagine you're right, but note that memcpy is not nearly as simple as it might appear. On OS X, for example, if you're copying enough memory (greater than 40kB or so, last I checked) and the arguments are nicely aligned, it will actually skip the copy altogether and instead use mach virtual memory calls to "copy" the memory without doing any actual copies.

Yuhong Bao - 2014-01-10 07:53:40

Fri, 10 Jan 2014 07:53:40 GMT

"Finally, the list ends with this amusing note: "...and perhaps some others." "Perhaps" is not a nice word to run into in this sort of documentation."
I think this is referring to memcpy(), memset(), and similar functions that are simple enough that they are pretty much always async signal safe.

arwyn - 2011-04-03 20:25:31

Sun, 03 Apr 2011 20:25:31 GMT

Mike, you missed pointing out the ensuing hilarity of reentrancy and recursive locks when used with signal handlers.

At best they provide absolutely no protection what-so-ever because they are run on the same thread and just work. Which is typically what the misguided programmer thinks they wanted.

At worst they hang in the spin lock portion of the locking/unlocking primitive, unless it's entirely atomic implemented via a single CAS (most aren't). Which is just like with a regular lock, but with a such a small deadlock window that most programmer's don't catch them for years.

arwyn - 2011-04-03 20:15:12

Sun, 03 Apr 2011 20:15:12 GMT

Dave -- a race condition in "set a bit" style signal handling would be a bug, not an inherent problem with the method. It's pretty easy to do it safely and race condition free, like so:

signal handler:
do {
    old_bits = gSignalBits;
    new_bits = old_bits | (1 << signum);
    if (old_bits == new_bits)
        return;
} while(!CAS(new_bits,old_bits,&gSignalBits));
nonblocking_write(fd,1);

threaded signal dispatcher:
while(1)
{
    do {
        bits = gSignalBits;
    while(!CAS(bits,0,&gSignalBits));

    if (bits == 0)
        blocking_read(fd,1);
    else
    {
        if ((bits & DO_SOMETHING_BIT) != 0)
            do_something();
    }
}

mikeash - 2011-04-02 14:53:06

Sat, 02 Apr 2011 14:53:06 GMT

If you really want to write a signal handler, take advantage of the fact that write() is a safe call to make from one. Create a pipe, stick the read end into your event system, and write a byte to the write end to signal. Make sure the pipe is nonblocking, though, otherwise you could be in serious trouble.

Of course there's no real reason to do that rather than using the built-in facilities which take care of the difficult parts for you.

Dave Zarzycki - 2011-04-02 14:10:13

Sat, 02 Apr 2011 14:10:13 GMT

Allen -- there are two problems with the "set a bit" style of signal handling.

1) Race conditions. For example:

if (bit) do_something();
// signal fires
r = select();
// select doesn't return -1 with errno == EINTR in this case like one would expect. Therefore: the bit isn't noticed and acted upon until the next FD becomes readable/writable, which may be a long time

This is why pselect() was later invented, so that one might control when the signals fire. The availability of pselect() doesn't help a developer though if they're using a system provided event loop technology rather than rolling their own. This is one of may reasons why facilities like GCD exist.

2) The vast majority of app and library code doesn't check for errno being equal to EINTR after an error and retry. That is why when we were designing GCD, we blocked all of the maskable signals from being delivered on GCD threads.

Allen Brunson - 2011-04-02 13:57:08

Sat, 02 Apr 2011 13:57:08 GMT

Given all the constraints, I've always written raw signal handlers to do nothing but set a flag, which is then noticed in some other part of the program, which takes action there. Can you think of any cases where that won't work?

mikeash - 2011-04-01 21:50:44

Fri, 01 Apr 2011 21:50:44 GMT

That's a good question. The short version is, "reentrant" is a general concept, and "async-signal-safe" applies it to the specific case of signal handlers.

Reentrancy doesn't necessarily have to apply to interrupts. For example, you'll find a note in the Cocoa documentation that NSNotificationCenter is reentrant. That most certainly does not mean that you can call it from a signal handler! Instead, what this means is that you can safely reenter NSNotificationCenter by calling into it from code which is in turn being called by NSNotificationCenter because of a posted notification.

That sort of reentrancy is much more useful (it's a good idea for almost any code with callbacks) and much easier to achieve (just make sure you're in a clean state and not holding any locks when you call the callback).

In the context of signal handling they're really the same, but in general not entirely.

Gwynne Raskind - 2011-04-01 21:24:15

Fri, 01 Apr 2011 21:24:15 GMT

Is there a difference, practically or conceptually, between "reentrant" and the even bigger mouthful "async-signal-safe"?

Dave Zarzycki - 2011-04-01 20:51:30

Fri, 01 Apr 2011 20:51:30 GMT

One more thing:

4) Keep in mind that setting SIGCHLD to SIG_IGN has standards defined side effects. Namely, one cannot call the wait*() family of APIs against child processes when SIGCHLD is ignored.

Dave Zarzycki - 2011-04-01 20:40:58

Fri, 01 Apr 2011 20:40:58 GMT

1) App developers should call signal(SIGPIPE, SIG_IGN) at the top of main() and library developers should defensively/politely set SO_NOSIGPIPE via setsockopt(). Why? Because network connectivity problems can cause SIGPIPE to be sent to your process instead of the more reasonable "read() returns -1 and errno equal EPIPE" error.

2) Always remember to backup and restore the previous signal mask if one uses pthread_sigmask() or sigprocmask(). As a general rule, one cannot assume that one's caller hasn't also fiddled with the mask.

3) Library writers should never install real signals handlers via signal() or sigaction(). The kernel only supports one handler and therefore that is the right of the app, not libraries. Libraries should use technologies like GCD or kqueues directly (if one must).

Kentzo - 2011-04-01 18:41:41

Fri, 01 Apr 2011 18:41:41 GMT

It's important to say that there are two signals which cannot be handled: SIGKILL and SIGSTOP.

mikeash - 2011-04-01 16:43:00

Fri, 01 Apr 2011 16:43:00 GMT

That's a good one indeed. I prefer not having a dedicated thread whenever possible, but that does look like a decent way to go.

Jens Ayton - 2011-04-01 16:34:34

Fri, 01 Apr 2011 16:34:34 GMT

There’s another option: sigwait() on a dedicated thread. It’s as simple as anything involving “dedicated thread”, and cross-platform.