I have a very similar challenge. I am working on extending a legacy FTP BSD/POSIX/UNIX sockets client code base (not libcurl) with RFC4217 "Secure FTP with TLS" support using Network.framework. The first step in that effort was abstracting away BSD/POSIX/UNIX sockets into a FTP channel abstraction. I created a TCP channel type to do so and have BSD/POSIX/UNIX sockets and Network.framework implementations working equally well with "clear" operations. I then moved onto implementing the easier / simpler "implicit" TLS support for Network.framework with a new TLS channel abstraction. That was reasonably straightforward and works with some FTPS server implementations; however, the majority return "425: Cannot secure data connection - TLS session resumption required." when trying to secure the data channel. I then moved onto implementing the more difficult "explicit" TLS support for Network.framework, based on the nw_framer_t pass-through approach intimated at this forum post. After some refinement for generalization, that worked well enough, at least for the "AUTH TLS" FTP command on the FTP control channel. However, once again, when attempting to secure the data channel with the TCP w/ Opportunistic TLS channel type, many servers respond with same "425: Cannot secure data connection - TLS session resumption required." witnessed before with the implicit TLS channel type. This forum thread seems the "freshest" on this topic and after surveying the macOS 14.7 "Sonoma" Network.framework documentation, I do not see any clear way to get the TLS session context or identifier from the secure FTP control channel to then pass and reuse for the secure FTP data channel. Any insights or examples would be greatly appreciated.

DTS case identifier 11028404.

For the particular server I've been testing against, since it does not do a typical server announce on connection, I am unclear of its implementation. However, for vsftpd (a popular FTP server), there is this blog post https://scarybeastsecurity.blogspot.com/2009/02/vsftpd-210-released.html that talks about the following setting added in 2.1.0: require_ssl_reuse If set to yes, all SSL data connections are required to exhibit SSL session reuse (which proves that they know the same master secret as the control channel). Although this is a secure default, it may break many FTP clients, so you may want to disable it. For a discussion of the consequences, see http://scarybeastsecurity.blogspot.com/2009/02/vsftpd-210-released.html (Added in v2.1.0). Default: YES

Made feature request 11147232.

For what it is worth, I was able to recode the explicit FTPS with TCP with opportunistic TLS using the deprecated Secure Transport interfaces, with the SSLSetPeerID being the most important one absent from Network.framework, and was able to successfully do RFC4217 Secure FTP with TLS on both secure control-only channel as well as control+data channels as follows: Uname: Darwin|aesop.local|23.6.0|Darwin Kernel Version 23.6.0: Fri Nov 15 15:13:28 PST 2024; root:xnu-10063.141.1.702.7~1/RELEASE_X86_64|x86_64 Sysinfo: Mac OS X 14.7.2 (Build 23H311) Channel model: Network.framework Channel security: Secure Transport "control" channel (0x60d000000c70) state did change: "<unknown>" (0) -> "initialized" (1) "control" channel (0x60d000000c70) state did change: "initialized" (1) -> "opened" (2) "control" channel (0x60d000000c70) state did change: "opened" (2) -> "connecting" (10) "control" channel (0x60d000000c70) state did change: "connecting" (10) -> "connected" (11) 220: Welcome to test.rebex.net! See https://test.rebex.net/ for more information and terms of use. If you don't have an account, log in as 'anonymous' or 'ftp'. Connected to test.rebex.net. Cmd: FEAT 211: Supported extensions: AUTH TLS;SSL; CDUP CLNT CSID EPSV EPRT MDTM MFMT MLST Type*;Modify*;Create*;Win32.ea;Size*; PASV PBSZ PORT PROT C;P; REST STREAM SIZE TVFS UTF8 End. Cmd: HELP SITE 214: 'SITE' - Custom commands. Cmd: AUTH TLS 234: Switching to TLS. TLS 1.2 connection using cipher ECDHE-RSA-AES128-GCM-SHA256 SSL state is connected Cmd: USER anonymous 331: Anonymous login OK, send your complete email address as your password. Cmd: PASS user@ 230: User 'anonymous' logged in. Logged in to test.rebex.net as anonymous. Cmd: FEAT 211: Supported extensions: AUTH TLS;SSL; CDUP CLNT CSID EPSV EPRT MDTM MFMT MLST Type*;Modify*;Create*;Win32.ea;Size*; PASV PBSZ PORT PROT C;P; REST STREAM SIZE TVFS UTF8 End. Cmd: HELP SITE 214: 'SITE' - Custom commands. Cmd: PBSZ 0 200: 'PBSZ' OK. Cmd: PROT P 200: 'PROT' OK. Cmd: PWD 257: "/" is current directory. Cmd: TYPE I 200: TYPE set to I. "data" channel (0x60c000017c80) state did change: "<unknown>" (0) -> "initialized" (1) "data" channel (0x60c000017c80) state did change: "initialized" (1) -> "opened" (2) Cmd: PASV 227: Entering Passive Mode (194,108,117,16,4,19) "data" channel (0x60c000017c80) state did change: "opened" (2) -> "bound" (8) "data" channel (0x60c000017c80) state did change: "bound" (8) -> "connecting" (10) "data" channel (0x60c000017c80) state did change: "connecting" (10) -> "connected" (11) TLS 1.2 connection using cipher ECDHE-RSA-AES128-GCM-SHA256 SSL state is connected Connected to 194.108.117.16:1043 for PASV. Cmd: RETR readme.txt 150: Opening 'BINARY' data connection. Welcome to test.rebex.net! You are connected to an FTP or SFTP server used for testing purposes by Rebex FTP/SSL or Rebex SFTP sample code. Only read access is allowed. For information about Rebex FTP/SSL, Rebex SFTP and other Rebex libraries for .NET, please visit our website at https://www.rebex.net/ For feedback and support, contact support@rebex.net Thanks! "data" channel (0x60c000017c80) state did change: "connected" (11) -> "closing" (6) "data" channel (0x60c000017c80) state did change: "closing" (6) -> "closed" (7) 226: Transfer complete. Cmd: QUIT 221: Closing session. "control" channel (0x60d000000c70) state did change: "connected" (11) -> "closing" (6) "control" channel (0x60d000000c70) state did change: "closing" (6) -> "closed" (7)

Why? I had an app years ago that supported FTP. One of the first bug reports I had was from someone connecting to a server on VMS. That's a question best answered by my client. Apparently there is interest and demand. That part is easy enough to solve. Just download Apple's curl source and look. As noted above, I've scrubbed through it in detail and, as far as I can see, my implementation and its are nearly identical and experiments exploring the minor differences have not yielded any impact on the interoperability issue. Behaviorally, what curl is doing on macOS 14.7.2 seems at odds with the open source. This is what has led to my question.

You’re testing the curl binary, right? Thus far, binary-only, correct. If you build curl from source, what behaviour do you see? That's an experiment I have not yet embarked on but will do so.

Oh, one other thing. The Secure Transport TLS implementation is itself part of Darwin. Start here but also see here. You might be able to glean some hints from that. Thanks. I'd definitely been plumbing the depths of the former; however, I had not yet cloned and dove into the latter.

Building curl from what's at github.com/apple-oss-distributions/curl fails out of the chute with: unable to find sdk 'macosx.internal' in Xcode. One thing I note in the --verbose output of curl when interacting with SSL/ TLS endpoints is that the output has no equivalent in the sectransp.c libcurl VTLS implementation. The output could and would only be generated from the openssl.c implementation. The contents of xcconfigs/common.xcconfig include: CURL_CONFIGURE_OPTIONS[sdk=macosx*] = --enable-threaded-resolver --with-gssapi --with-ssl=/usr/local/libressl --with-secure-transport which seems to corroborate some of what I am seeing. xcconfigs/libcurl.xcconfig also contains: LIBRARY_SEARCH_PATHS[sdk=macosx*] = $(inherited) /usr/local/libressl/lib HEADER_SEARCH_PATHS[sdk=macosx*] = $(inherited) /usr/local/libressl/include which would seem to indicate there is more than just a pure Secure Transport implementation. However, /usr/local/libressl surely does not exist on a “just installed it” macOS distribution much less on a “just installed Xcode” distribution.

Regrettably, on my macOS 14.7.2 system, /usr/lib/libcurl.4.dylib does not exist (though I am sure it is squirreled away in some alternate path, somewhere). In a different note, trolling through curl/lib/vtls/vtls.c, I found: env = env_tmp = curl_getenv("CURL_SSL_BACKEND"); #ifdef __APPLE__ #if TARGET_OS_OSX if (!env) env = have_openssl() ? "openssl" : "secure-transport"; #endif #endif and CURL_DEFAULT_SSL_BACKEND where CURL_DEFAULT_SSL_BACKEND is configured for openssl which corroborates my observation that the --verbose output seen for curl could only be possible if the implementation were truly OpenSSL. So, setting: setenv CURL_SSL_BACKEND secure-transport Yields output from curl -4 --verbose --ftp-pasv --ftp-ssl-reqd ftp://ftp.sjtu.edu.cn:21/pub/README.NetInstall -o README.NetInstall that is consistent with the sectransp.c implementation: % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Host ftp.sjtu.edu.cn:21 was resolved. * IPv6: (none) * IPv4: 202.38.97.230 * Trying 202.38.97.230:21... * Connected to ftp.sjtu.edu.cn (202.38.97.230) port 21 < 220 (vsFTPd 3.0.2) > AUTH SSL 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0< 234 Proceed with negotiation. * TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 * Server certificate: ftp.sjtu.edu.cn * Server certificate: ZeroSSL RSA Domain Secure Site CA * Server certificate: USERTrust RSA Certification Authority > USER anonymous < 331 Please specify the password. > PASS ftp@example.com 0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0< 230 Login successful. > PBSZ 0 < 200 PBSZ set to 0. > PROT P < 200 PROT now Private. > PWD < 257 "/" * Entry path is '/' > CWD pub * ftp_perform ends with SECONDARY: 0 < 250 Directory successfully changed. > EPSV * Connect data stream passively 0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0< 229 Entering Extended Passive Mode (|||14172|). * Connecting to 202.38.97.230 (202.38.97.230) port 14172 * Trying 202.38.97.230:14172... * Connected 2nd connection to 202.38.97.230 port 14172 * SSL reusing session ID > TYPE I < 200 Switching to Binary mode. > SIZE README.NetInstall < 213 427 > RETR README.NetInstall 0 427 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0< 150 Opening BINARY mode data connection for README.NetInstall (427 bytes). * Maxdownload = -1 * Getting file with size: 427 * TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 * Server certificate: ftp.sjtu.edu.cn * Server certificate: ZeroSSL RSA Domain Secure Site CA * Server certificate: USERTrust RSA Certification Authority { [427 bytes data] * Remembering we are in dir "pub/" < 226 Transfer complete. 100 427 100 427 0 0 85 0 0:00:05 0:00:04 0:00:01 85 * Connection #0 to host ftp.sjtu.edu.cn left intact with that, not only does it still work but I get Wireshark data channel ClientHello that is more consistent with what I see in my libncftp Secure Transport implementation, including the three status_request, signed_certificate_timestamp, and extended_master_secret extensions extensions as well as a set of cipher suites that are more consistent. Also consistent with the openssl backend, Wireshark—for whatever reason—does not pick up the initial control channel ClientHello. I see 234 Proceed with negotiation.\r\n and then a series of what looks to be opaque data, possibly with some certificate chain validation with various certificate data. In short, curl on macOS 14.7.2 with either openssl or secure-transport SSL backends (continues to) work.

To keep the comparison apples-to-apples, I also experimented with the curl --disable-epsv option, with which macOS curl still succeeds: % curl -4 --verbose --disable-epsv --ftp-pasv --ftp-ssl-reqd ftp://ftp.sjtu.edu.cn:21/pub/README.NetInstall -o README.NetInstall && rm -f README.NetInstall % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Host ftp.sjtu.edu.cn:21 was resolved. * IPv6: (none) * IPv4: 202.38.97.230 * Trying 202.38.97.230:21... * Connected to ftp.sjtu.edu.cn (202.38.97.230) port 21 < 220 (vsFTPd 3.0.2) > AUTH SSL < 234 Proceed with negotiation. * TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 * Server certificate: ftp.sjtu.edu.cn * Server certificate: ZeroSSL RSA Domain Secure Site CA * Server certificate: USERTrust RSA Certification Authority > USER anonymous 0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0< 331 Please specify the password. > PASS ftp@example.com < 230 Login successful. > PBSZ 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0< 200 PBSZ set to 0. > PROT P < 200 PROT now Private. > PWD < 257 "/" * Entry path is '/' > CWD pub * ftp_perform ends with SECONDARY: 0 < 250 Directory successfully changed. > PASV * Connect data stream passively 0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0< 227 Entering Passive Mode (202,38,97,230,62,39). * Skip 202.38.97.230 for data connection, reuse ftp.sjtu.edu.cn instead * Connecting to 202.38.97.230 (202.38.97.230) port 15911 * Trying 202.38.97.230:15911... * Connected 2nd connection to 202.38.97.230 port 15911 * SSL reusing session ID > TYPE I < 200 Switching to Binary mode. > SIZE README.NetInstall < 213 427 > RETR README.NetInstall 0 427 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 * Server certificate: ftp.sjtu.edu.cn * Server certificate: ZeroSSL RSA Domain Secure Site CA * Server certificate: USERTrust RSA Certification Authority < 150 Opening BINARY mode data connection for README.NetInstall (427 bytes). * Maxdownload = -1 * Getting file with size: 427 { [427 bytes data] * Remembering we are in dir "pub/" < 226 Transfer complete. 100 427 100 427 0 0 77 0 0:00:05 0:00:05 --:--:-- 85 * Connection #0 to host ftp.sjtu.edu.cn left intact I'll play with Wireshark; however, it would seem there is something in that initial control channel ClientHello (or whatever is happening there) that unlocks the difference in behavior.

FYI, parallel DTS case 11579452.

I've brought up the vsftpd-3.0.5 source code; however, whatever is failing is failing within OpenSSL rather than vsftpd, so I'll need to go a level deeper in pursuing that route. Using: % setenv CURL_SSL_BACKEND secure-transport % curl -4 --verbose --disable-epsv --ftp-pasv --ftp-ssl-reqd ftp://ftp.sjtu.edu.cn:21/pub/README.NetInstall -o README.NetInstall and enabling this block in our libncftp implementation in FTPChannelTLS_SecureTransportCreateSecureContext: result = SetDefaultCipherSuites(ssl_context); if (result != kNoErr) goto done; which establishes a controlled collection of Secure Transport cipher suites that are strong and safe (similar to what curl does), disabling those that Secure Transport supports but are regarded as weak and unsafe due to either a weak key exchange or bulk encryption algorithm and then running: % ./samples/misc/ncftpgetbytes -d stderr --secure --explicit --secure-both ftp://ftp.sjtu.edu.cn:21/pub/README.NetInstall I am able to get identical control and data channel TLS v1.2 ClientHello between the two implementations. In the intervening time, I have also pressed Fetch 5.8 into play, since it appears to support RFC4217 Secure FTP with TLS using Secure Transport. With that, I have three implementations with nearly identical control and data TLS v1.2 ClientHello interactions: macOS 14.7.2 "Sonoma" curl 8.7.1 with the Secure Transport SSL backend. control ClientHello data ClientHello GET Fetch 5.8 control ClientHello data ClientHello LIST data ClientHello GET libncftp control ClientHello data ClientHello GET It feels like this is a one- or two-liner away from working.

Diving into github.com/apple-oss-distributions/Security with OSX/libsecurity_ssl/lib/sslRecord.c and OSX/libsecurity_ssl/lib/sslTransport.c, it looks like the implementation can return errSSLClosedAbort from the following: OSX/libsecurity_ssl/lib/sslRecord.c: errorTranslate when recordErr is errSSLRecordClosedAbort errorTranslate is called from: SSLWriteRecord SSLReadRecord SSLServiceWriteQueue OSX/libsecurity_ssl/lib/sslTransport.c: SSLHandshake when ctx->state is SSL_HdskStateErrorClose. SSLRead when ctx->state is SSL_HdskStateErrorClose or SSL_HdskStateOutOfBandError. SSLReHandshake when ctx->state is SSL_HdskStateErrorClose or SSL_HdskStateOutOfBandError. SSLWrite when ctx->state is SSL_HdskStateErrorClose or SSL_HdskStateOutOfBandError. For some reason, my implementation is triggering one of these cases whereas curl and Fetch 5.8 are not.

But, just to confirm, that means that can reproduce this with a server under your control, right? If so, that’s definitely a progress. Confirmed. A correlating factor seems to be those servers written against OpenSSL, possibly running on Linux. See below. OK. And cases 1 and 2 (curl with Secure Transport, and Fetch 5.8) work whereas case 3 (your libncftp code) fails, right? Confirmed. FTPS Server Secure Control-only Secure Control+Data test.rebex.net Custom .NET Implementation ✅ ✅ Microsoft FTP Service ✅ ❌ proftpd ✅ ❌ pureftpd ✅ ❌ vsftpd ✅ ❌ Until I added code to utilize SSLSetPeerID, Secure Control+Data was ❌ up and down that column. Adding code to work with SSLSetPeerID allowed the test.rebex.net Custom .NET Implementation to work on stop responding with 425: Cannot secure data connection - TLS session resumption required. on attempting to establish the secure data channel session.