I’m working with apple dispatch queue in C with the following design: multiple dispatch queues enqueue tasks into a shared context, and a dedicated dispatch queue (let’s call it dispatch queue A) processes these tasks. However, it seems this design has a memory visibility issue. Here’s a simplified version of my setup: I have a shared_context struct that holds: task_lis: a list that stores tasks to be prioritized and run — this list is only modified/processed by dispatch queue A (a serially dispatch queue), so I don't lock around it. cross_thread_tasks: a list that other queues push tasks into, protected by a lock. Other dispatch queues call a function schedule_task that locks and appends a new task to cross_thread_tasks call dispatch_after_f() to schedule a process_task() on dispatch queue A process_task() that processes the task_list and is repeatedly scheduled on dispatch queue A : Swaps cross_thread_tasks into a local list (with locking). Pushes the tasks into task_list. Runs tasks from task_list. Reschedules itself via dispatch_after_f(). Problem: Sometimes the tasks pushed from other threads don’t seem to show up in task_list when process_task() runs. The task_list appears to be missing them, as if the cross-thread tasks aren’t visible. However, if the process_task() is dispatched from the same thread the tasks originate, everything works fine. It seems to be a memory visibility or synchronization issue. Since I only lock around cross_thread_tasks, could it be that changes to task_list (even though modified on dispatch queue A only) are not being properly synchronized or visible across threads? My questions What’s the best practice to ensure shared context is consistently visible across threads when using dispatch queues? Is it mandatory to lock around all tasks? I would love to minimize/avoid lock if possible. Any guidance, debugging tips, or architectural suggestions would be appreciated! =============================== And here is pseudocode of my setup if it helps: struct shared_data { struct linked_list* task_list; } struct shared_context { struct shared_data *data; struct linked_list* cross_thread_tasks; struct thread_mutex* lock; // lock is used to protect cross_thread_tasks } static void s_process_task(void* shared_context){ struct linked_list* local_tasks; // lock and swap the cross_thread_tasks into a local linked list lock(shared_context->lock) swap(shared_context->cross_thread_tasks, local_tasks) unlock(shared_context->lock) // I didnt use lock to protect `shared_context->data` as they are only touched within dispatch queue A in this function. for (task : local_tasks) { linked_list_push(shared_context->data->task_list) } // If the `process_task()` block is dispatched from `schedule_task()` where the task is created, the `shared_context` will be able to access the task properly otherwise not. for (task : shared_context->data->task_list) { run_task_if_timestamp_is_now(task) } timestamp = get_next_timestamp(shared_context->data->task_list) dispatch_after_f(timestamp, dispatch_queueA, shared_context, process_task); } // On dispatch queue B static void schedule_task(struct task* task, void* shared_context) { lock(shared_context->lock) push(shared_context->cross_thread_tasks, task) unlock(shared_context->lock) timestamp = get_timestamp(task) // we only dispatch the task if the timestamp < 1 second. We did this to avoid the dispatch queue schedule the task too far ahead and prevent the shutdown process. Therefore, not all task will be dispatched from the thread it created. if(timestamp < 1 second) dispatch_after_f(timestamp, dispatch_queueA, shared_context, process_task); }

Hi there, I have some thread related questions regards to network framework completion callbacks. In short, how should I process cross thread data in the completion callbacks? Here are more details. I have a background serial dispatch queue (call it dispatch queue A) to sequentially process the nw_connection and any network io events. Meanwhile, user inputs are handled by serial dispatch queue ( dispatch queue B). How should I handle the cross thread user data in this case? (I write some simplified sample code below) struct { int client_status; char* message_to_sent; }user_data; nw_connection_t nw_connection; dispatch_queue_t dispatch_queue_A static void send_message(){ dispatch_data_t data = dispatch_data_create(message, len(message), dispath_event_loop->dispatch_queue, DISPATCH_DATA_DESTRUCTOR_DEFAULT); nw_connection_send( nw_connection, data, NW_CONNECTION_DEFAULT_MESSAGE_CONTEXT, false, ^(nw_error_t error) { user_data.client_status = SENT; mem_release(user_data.message_to_sent); }); }); } static void setup_connection(){ dispatch_queue_A= dispatch_queue_create("unique_id_a", DISPATCH_QUEUE_SERIAL); nw_connection = nw_connection_create(endpoint, params); nw_connection_set_state_changed_handler(){ if (state == nw_connection_state_ready) { user_data.client_status = CONNECTED } // ... other operations ... } nw_connection_start(nw_connection); nw_retain(nw_connection); } static void user_main(){ setup_connection() user_data.client_status = INIT; dispatch_queue_t dispatch_queue_B = dispatch_queue_create("unique_id_b", DISPATCH_QUEUE_SERIAL); // write socket dispatch_async(dispatch_queue_B, ^(){ if (user_data.client_status != CONNECTED ) return; user_data.message_to_sent = malloc(XX,XXX) // I would like to have all io events processed on dispatch queue A so that the io events would not interacted with the user events dispatch_async_f(dispatch_queue_A, send_message); // Disconnect block dispatch_async(dispatch_queue_B, ^(){ dispatch_async_f(dispatch_queue_A, ^(){ nw_connection_cancel(nw_connection) }); user_data.client_status = DISCONNECTING; }); // clean up connection and so on... } To be more specific, my questions would be: As I was using serial dispatch queue, I didn't protect the user_data here. However, which thread would the send_completion_handler get called? Would it be a data race condition where the Disconnect block and send_completion_handler both access user_data? If I protect the user_data with lock, it might block the thread. How does the dispatch queue make sure it would NOT put a related execution block onto the "blocked thread"?

Hi all, I'm developing an TCP socket SDK in C. The SDK is using Apple Network Framework and encountered some wired bad access issue occasionally on function nw_connection_send. Looking into the trace stack, it was bad access issue in nw_write_request_create, when it is trying to release a reference. However, I could not found more doc/source code details about nw_write_request_create. // on socket destroy, we will release the related nw_connection. increase_ref_count(socket) nw_connection_t nw_connection = socket->nw_connection; dispatch_data_t data = dispatch_data_create(message_ptr->ptr, message_ptr->len, dispath_event_loop, DISPATCH_DATA_DESTRUCTOR_FREE); // > Bad Access here < // While I check `nw_connection` and `data`, both seems available while the function get called. I tried to call dispatch_retain on `data`, but it was not helpful. nw_connection_send( nw_connection, data, NW_CONNECTION_DEFAULT_MESSAGE_CONTEXT, false, ^(nw_error_t error) { // process the message, we will release message_buf in this function. completed_fn(message_buf); reduce_ref_count(socket) } While I check nw_connection and data, both seems available while the function get called. I tried to call dispatch_retain on data, but it was not helpful. Is there any way to narrow down which object is releasing? As the issue happened occasionally (9 failure out of 10 attempts when I run multiple unit tests at the same time, and I rarely see it when I ran a single unit test). I would assume it was actually a race condition here. Is there a way to track down which object is released? I do understand it would be hard to track without knowing more design details of my SDK, but any related suggestions or ideas would be appreciated. Thanks in advance. More related source code: struct nw_socket{ nw_connection_t nw_connection; nw_parameters_t socket_options_to_params; dispatch_queue_t event_loop; // ... bunch of other parameters... struct ref_count ref_count; } static int s_socket_connect_fn( const struct socket_endpoint *remote_endpoint, struct dispatch_queue_t event_loop) { nw_socket = /*new socket memory allocation, increasing ref count*/ nw_endpoint_t endpoint = nw_endpoint_create_address(/* process remote_endpoint */); nw_socket->nw_connection = nw_connection_create(endpoint, nw_socket >socket_options_to_params); nw_release(endpoint); nw_socket->nw_connection->set_queue(nw_socket->nw_connection, event_loop); nw_socket->event_loop = event_loop; nw_connection_set_state_changed_handler(nw_socket->nw_connection, ^(nw_connection_state_t state, nw_error_t error) { // setup connection handler } nw_connection_start(nw_socket->nw_connection); nw_retain(nw_socket->nw_connection); } // nw_socket is ref counted, call the destroy function on ref_count reduced to 0 static void s_socket_impl_destroy(void *sock_ptr) { struct nw_socket *nw_socket = sock_ptr; /* Network Framework cleanup */ if (nw_socket->socket_options_to_params) { nw_release(nw_socket->socket_options_to_params); nw_socket->socket_options_to_params = NULL; } if (nw_socket->nw_connection) { nw_release(nw_socket->nw_connection); // Print here, to make sure the nw_connection was not released before nw_connection_send call. nw_socket->nw_connection = NULL; } // releasing memory and other parameters } static int s_socket_write_fn( struct nw_socket *socket, const struct bytePtr* message_ptr, // message_ptr is a pointer to allocated message_buf socket_on_write_completed_fn *completed_fn, void *message_buf) { // Ideally nw_connection would not be released, as socket ref_count is retained here. increase_ref_count(socket->ref_count); nw_connection_t nw_connection = socket->nw_connection; struct dispatch_queue_t dispatch_event_loop = socket->event_loop; dispatch_data_t data = dispatch_data_create(message_ptr->ptr, message_ptr->len, dispath_event_loop, DISPATCH_DATA_DESTRUCTOR_FREE); // > Bad Access here < // While I check `nw_connection` and `data`, both seems available while the function get called. I tried to call dispatch_retain on `data`, but it is not helpful. nw_connection_send( nw_connection, data, NW_CONNECTION_DEFAULT_MESSAGE_CONTEXT, false, ^(nw_error_t error) { // process the message, we will release message_buf in this function. completed_fn(message_buf); reduce_ref_count(socket) } }

Hi team, I'm working on an MQTT client for Apple platforms (macOS, iOS, and possibly tvOS and watchOS). I would like the client to listen to messages even when the application is in the background. I would appreciate any suggestions on the best approach to achieve this. Based on iOS Background Execution Limits, it seems that my best bet is to use a long-running background process with BGProcessingTaskRequest while setting up the connection. Does that sound like the right approach? Is there any limits for the bg tasks? I currently have a working BSD socket. I'm not sure if it is necessary to switch to the Network Framework to have the background task working, but I'm open to switching if it's necessary. If the approach works, does that mean I could built a http client to process large upload/download tasks without using NSURLSession? As I'm working on a cross platform project, it would be benefit if I dont need a separate http client implementation for Apple. Any insights on this topic would be greatly appreciated. Additionally, it's off topic, but the link to "WWDC 2020 Session 10063 Background Execution Demystified" (https://developer.apple.com/videos/play/wwdc2020/10063/) is broken. Is there a way to access the content there? Thanks in advance for your help and insights!

I'm having issue with keychain access for my SWIFT project. The keychain operations succeed while I run the test with Xcode.app (GUI), but failed when I run the test through command line tool xcodebuild. I assume I did something wrong with the environment. Is there any suggestion or instruction about how should I setup for the xcodebuild command line tool? Here is my unit test. static func run_shell(_ command: String) -> String { let task = Process() let pipe = Pipe() task.standardOutput = pipe task.standardError = pipe task.arguments = ["-c", command] task.launchPath = "/bin/zsh" task.standardInput = nil task.launch() let data = pipe.fileHandleForReading.readDataToEndOfFile() let output = String(data: data, encoding: .utf8)! return output } func testSecurityDefaultKeychain() throws { print(TLSContextTests.run_shell("security default-keychain")); } Other things I have tried: I got the same result if I use SecKeychainCopyDefault instead of the security command. If I directly run security command in my terminal, it worked fine. > security default-keychain "/Users/runner/Library/Keychains/login.keychain-db" I also tried with sudo xcodebuild & chmod a+x xcodebuild to make sure the tool has permission to access keychain, but it was not helpful. I had a post about the same issue a month ago. At that time I thought it was an issue for CI environment only. However, it turns out it was the xcodebuild. https://forums.developer.apple.com/forums/thread/747794

I'm setting up unit tests for my application using Xcode. When I tried to access default keychain on MacOS using SecKeychainCopyDefault, I got error OSStatus -25307, which means "A default keychain could not be found." The tests worked locally, and the issue only happened with Github Actions. Would anyone have any insight on this issue, or point me to some reading I can refer to? Thanks in advance! Here is some more tests I've done here. I tried to run "security default-keychain" on GithubAction, and I got &gt; security default-keychain "/Users/runner/Library/Keychains/login.keychain-db" However, when I tried to start a shell to run the same security command in my unit test, I got SecKeychainCopyDefault: A default keychain could not be found. Here is my test calling security command from shell: static func run_shell(_ command: String) -&gt; String { let task = Process() let pipe = Pipe() task.standardOutput = pipe task.standardError = pipe task.arguments = ["-c", command] task.launchPath = "/bin/zsh" task.standardInput = nil task.launch() let data = pipe.fileHandleForReading.readDataToEndOfFile() let output = String(data: data, encoding: .utf8)! return output } func testSecurityDefaultKeychain() throws { print(TLSContextTests.run_shell("security default-keychain")); }

Hi all, I'm working on a macOS project with C interface. And I'm trying to import my private key to a SecKeychainRef, but I always got error code -50. Would you have any advice or suggestion in it? Thanks in advance. Here is my code: // Get default keychain SecKeychainRef import_keychain = NULL; OSStatus keychain_status = SecKeychainCopyDefault(&import_keychain); // Create key from ECC key data in X963 format CFMutableDictionaryRef parameters = CFDictionaryCreateMutable(default_alloc, 0, NULL, NULL); CFDictionarySetValue(parameters, kSecAttrKeyType, kSecAttrKeyTypeECSECPrimeRandom); CFDictionarySetValue(parameters, kSecAttrKeyClass, kSecAttrKeyClassPrivate); CFDictionarySetValue(parameters, kSecAttrApplicationLabel, cfLabel); SecKeyRef private_key= SecKeyCreateWithData(hard_code_key_ref, parameters, &key_error); // Add seckey to key chain CFMutableDictionaryRef secItemParams = CFDictionaryCreateMutable(default_alloc, 0, NULL, NULL); CFDictionarySetValue(secItemParams, kSecClass, kSecClassKey); CFDictionarySetValue(secItemParams, kSecValueRef, privateKey); CFDictionarySetValue(secItemParams, kSecUseKeychain, import_keychain); OSStatus key_status = SecItemAdd(secItemParams, NULL); I also tried to test "SecItemAdd" with password value, but it also failed with -25308. I'm not sure if it is related or not. Here is the test code:             CFStringRef server = CFStringCreateWithCString(default_alloc, "example.com", kCFStringEncodingUTF8);             CFStringRef username = CFStringCreateWithCString(default_alloc, "username", kCFStringEncodingUTF8);             CFStringRef password = CFStringCreateWithCString(default_alloc, "password", kCFStringEncodingUTF8);             CFMutableDictionaryRef secItemParams = CFDictionaryCreateMutable(default_alloc, 0, NULL, NULL);             CFDictionarySetValue(secItemParams, kSecClass, kSecClassInternetPassword);             CFDictionarySetValue(secItemParams, kSecValueData, password);             CFDictionarySetValue(secItemParams, kSecAttrAccount, username);             CFDictionarySetValue(secItemParams, kSecAttrServer, server);                         CFDictionarySetValue(secItemParams, kSecUseKeychain, import_keychain);             CFDictionarySetValue(secItemParams, kSecAttrAccessible, kSecAttrAccessibleAlways); OSStatus key_status = SecItemAdd(secItemParams, NULL); The above code failed with "OSStatus -25308 : User interaction is not allowed." Any advice is welcomed. Thank you!