New iPhones! New Apple Watch! Even a new entry level iPad! The Apple iPhone event delivered mostly within expectations. I am not going to link all the detailed posts, you can go to Apple’s Newsroom for the official details.
We got release dates for iOS 13 (Sep 19), iOS 13.1 (Sep 30), iPadOS (Sep 30, unclear if this is iPadOS 13.0 or 13.1, but likely the latter) and macOS 10.15 Catalina (a vague ‘October’). (Check this MacStories post with all the dates, including hardware and services.)
So, there are a few extra weeks this year to test and prepare for the next macOS upgrade. It’ll be a busy fall either way. Make the most of it: Test, prepare, file bugs and feedback!
Eric Holtam: “Notice: Do not try and restore BridgeOS on a machine running a beta OS with Configurator 2. Configurator doesn’t allow backward updates to previous BridgeOS versions and ”bricks“ the machine. ‘AMRestoreErrorDomain error 79 – backwards update not allowed: 17P50541b -> 16PXXX)’ ”
William Smith: “UltraThin Updates take 2: For tomorrow’s Microsoft Office for Mac update, Microsoft AutoUpdate will use UltraThin updates to move 16.28 > 16.29. Should be around 176 MB to update all of Office.”
Notarizing Mac Application bundles has its pitfalls, but is overall fairly well documented. However, I have been working on some command line tools written in Swift 5 and figured out how to get those properly signed and notarized.
You cannot get the required certificates with a free Apple Developer account, unless you are member of a team that provides access.
You can download Xcode from the Mac App Store or the developer download page. When you launch Xcode for the first time, it will prompt for some extra installations. Those are necessary for everything to in the article to work.
Developer ID Certificates
There are multiple certificates you can get from the Developer Program. By default you get a ‘Mac Developer’ certificate, which you can use for building and testing your own app locally.
To distribute binaries (apps and command line tools) outside of the App Store, you need a ‘Developer ID Application’ certificate. To sign installer packages for distribution outside of the Mac App Store, you need a ‘Developer ID Installer’ certificate.
We will need both types of Developer ID certificates, the first to sign the command line tool and the second to sign and notarize the installer package.
If you have not created these yet, you can do so in Xcode or in the Developer Portal. If you already have the certificates but on a different Mac, you need to export them and re-import them on the new Mac. Creating new certificates might invalidate the existing certificates! So beware.
Once you have created or imported the certificates on your work machine, you can verify their presence in the Terminal with:
% security find-identity -p basic -v
This command will list all available certificates on this Mac. Check that you can see the ‘Developer ID Application’ and ‘Developer ID Installer’ certificates. If you are a member of multiple teams, you may see multiple certificates for each team.
You can later identify the certificates (or ‘identities’) by the long hex number or by the descriptive name, e.g. "Developer ID Installer: Armin Briegel (ABCD123456)"
The ten character code at the end of the name is your Developer Team ID. Make a note of it. If you are a member of multiple developer teams, you can have multiple Developer ID certificates and the team ID will help you distinguish them.
Application Specific Password for your Developer Account
Apple requires Developer Accounts to be protected with two-factor authentication. To allow automated workflows which require authentication, you can create application specific passwords.
Create a new application specific password in Apple ID portal for your developer account.
You will only be shown the password when you create it. Immediately create a ‘New Password Item’ in your Keychain with the following fields:
Keychain Item Name: Developer-altool
Account Name: your developer account email
Password: the application-specific password you just created
This will create a developer specific password item that we can access safely from the tools.
If you want, you can also store the app specific password in a different password manager, but the Xcode tools have a special option to use Keychain.
A Command Line Tool Project
You may already have a project to create a command line in Xcode. If you don’t have one, or just want a new one to experiment, you can just create a new project in Xcode and choose the ‘Command Line Tool’ template from ‘macOS’ section in the picker. The template creates a simple “Hello, world” tool, which you can use to test the notarization process.
My sample project for this article will be named “hello.”
Preparing the Xcode Project
The default settings in the ‘Command Line Tool’ project are suitable for building and testing the tool on your Mac, but need some changes to create a distributable tool.
Choosing the proper signing certificates
Before you can notarize the command line tool, it needs to be signed with the correct certificates.
in Xcode, select the blue project icon in the left sidebar
select the black “terminal” icon with your project’s name under the “Targets” list entry
make sure the ‘General’ tab is selected
under ‘Signing’ disable ‘Automatically manage signing’
under ‘Signing (Debug)’ choose your Team and choose ‘Developer ID Application’ as the certificate
under ‘Signing (Release)’ choose your Team and choose ‘Developer ID Application’ as the certificate
Enable Hardened Runtime
Enabling the ‘Hardened Runtime’ will compile the binary in a way that makes it harder for external process to inject code. This will be requirement for successful notarization starting January 2020.
from the view where you changed the signing options, click on ‘Build Settings’ in the upper tab row
click on ‘All’ to show all available settings
enter ‘enable hardened’ in the search field, this will show the ‘Enable Hardened Runtime’ setting
set the value in the project column (blue icon) to YES
Change the Install Build Location
If we want to automate the packaging and notarization, we need to know where Xcode builds the binary. The default location is in some /tmp subdirectory and not very convenient. We will change the location for the final binary (the ‘product’) to the build subdirectory in the project folder:
in the same view as above, enter ‘Installation Build’ in the search field, this will show the ‘Installation Build Products Location’ setting
double click on the value in the Project column (blue icon), this will open a popup window
change the value to $ SRCROOT/build/pkgroot
If you manage your code in git or another VCS, you want to add the build subdirectory to the ignored locations (.gitignore)
Build the Binary
You can use Xcode to write, test, and command line tool debug your. When you are ready to build and notarize a pkg installer, do the following:
open Terminal and change directory to the project folder
% xcodebuild clean install
This will spew a lot of information out to the command line. You will see a build subdirectory appear in the project folder, which will be filled with some directories with intermediate data.
After a successful build you should see a pkgroot directory in the build folder, which contains your binary in the usr/local/bin sub-path.
/usr/local/bin is the default location for command line tools in the Command Line Tool project template. It suits me fine most of the time, but you can change it by modifying the ‘Installation Directory’ build setting in Xcode and re-building from the command line.
Build the pkg
Command Line Tools can be signed, but not directly notarized. You can however notarize a zip, dmg, or pkg file containing a Command Line Tool. Also, it is much easier for users and administrators to install your tool when it comes in a proper installation package.
We can use the pkgroot directory as our payload to build the installer package:
I have broken the command into multiple lines for clarity, you can enter the command in one line without the end-of-line backslashes \. You want to replace the values for the identifier, version and signing certificate with your data.
This will build an installer package which would install your binary on the target system. You should inspect the pkg file with Pacifist or Suspicious Package and do a test install on a test system to verify everything works.
If you want to learn more about installer packages and pkgbuild read my book “Packaging for Apple Administrators.”
Notarizing the Installer Package
Xcode has a command line tool altool which you can use to upload your tool for notarization:
The asc-provider is your ten digit Team ID. If you are only a member in a single team you do not need to provide this.
The password uses a special @keychain: keyword that tells altool to get the app-specific password out of a keychain item named Developer-altool. (Remember we created that earlier?)
This will take a while. When the command has successfully uploaded the pkg to Apple’s Notarization Servers, it will return a RequestUUID. Your notarization request will be queued and eventually processed. You can check the status of your request with:
Apple will also send an email to your developer account when the process is complete. I my experience this rarely takes more than a minute or two. (Being in Central EU time zone might be an advantage there). When the process is complete, you can run the above notarization-info command to get some details. The info will include a link that contains even more information, which can be useful when your request is rejected.
Note that the info links expire after 24 hours or so. You should copy down any information you want to keep longer.
Completing the Process
You will not receive anything back from Apple other than the confirmation or rejection of your request. When a Mac downloads your installer package and verifies its notarization status it will reach out to Apple’s Notarization servers and they will confirm or reject the status.
If the Mac is offline at this time, or behind a proxy or firewall that blocks access to the Apple Servers, then it cannot verify whether your pkg file is notarized.
You can, however, ‘staple’ the notarization ticket to the pkg file, so the clients do not need to connect to the servers:
% xcrun stapler staple build/hello-1.0.pkg
You can also use stapler to verify the process went well:
% xcrun stapler validate build/hello-1.0.pkg
But since stapler depends on the developer tools to be installed, you should generally prefer spctl to check notarization:
Obviously, I built a script to automate all this. Put the following script in the root of the project folder, modify the variables at the start of the script (lines 20–38) with your information, and run it.
The script will build the tool, create a signed pkg, upload it for notarization, wait for the result, and then staple the pkg.
You can use this script as an external build tool target in Xcode. There are other ways to integrate scripts for automation in Xcode, but all of this is a new area for me and I am unsure which option is the best, and which I should recommend.
Links and Videos
These links and videos, especially Howard Oakley’s post and Tom Bridge’s PSU Presentation have been hugely helpful. Also thanks to co-worker Arnold for showing me this was even possible.
Notarization is a key part of Apple’s security strategy going in macOS.
As MacAdmins we will usually deploy software through management systems, where the Gatekeeper mechanisms which evaluate notarization are bypassed. There are, however, already special cases (Kernel Extensions) where notarization is mandatory. It is likely that Apple will continue to tighten these requirements in the future. The macOS Mojave 10.14.5 update has shown that Apple may not even wait for major releases to increase the requirements.
If you are building your own tools and software for macOS and plan to distribute the software to other computers, you should start signing and notarizing.
On the other hand, I find the introduction of Notarization to macOS encouraging. If Apple wanted to turn macOS into a “App Store only system” like iOS, they would not have needed to build the notarization process and infrastructure. Instead, Apple seems to have embraced third-party-software from outside the App Store.
Notarization allows Apple to provide a security mechanism for software distributed through other means. It cannot be 100% effective, but when used correctly by Apple and the software developers it will provide a level of validation and trust for software downloaded from the internet.
You can tell from the various articles and support posts that release day is definitely getting closer.
iOS 10.13.1 beta 2 was release and it is questionable if we will see another beta of either iOS, iPadOS or macOS before release. Apple has loosened notarization requirements for macOS Catalina temporarily (until Jan 2020) to help developer handle the transition.
Mike Boylan: “This is true. Age != skill (in both directions). Am fortunate that most people have treated me with respect.” (Thread)
Jason Fried: “When Google puts 4 paid ads ahead of the first organic result for your own brand name, you’re forced to pay up if you want to be found. It’s a shakedown. It’s ransom. But at least we can have fun with it. Search for Basecamp and you may see this attached ad.” (Image)
There are many solutions to get the current logged in user in macOS to use in a shell script. However, the semi-official, “sanctioned” method has always involved a rather elaborate python one-liner, originally published by Ben Toms:
While this has worked wonderfully and more reliably than other solutions, it always looked inelegant, and added a dependency on the python binary. This used to be fine, as the python binary shipped with macOS. But in the macOS Catalina release notes, we have learned that some, yet undefined, future version of macOS will not include the Python, Ruby, and Perl interpreters any more.
With prescient timing, Erik Berglund figured out a different method, which still accesses the same system framework, but through the scutil command rather than the PyObjC bridge. Over time, different contributors in the MacAdmins Slack have optimized this command:
While zsh is a great replacement for bash for interactive terminal use and scripting on the ‘full’ macOS system, there are environments (Recovery) where zsh is not present.
The only shell Mac Admins can rely on in all contexts is now /bin/sh.
Because of this I recommend using /bin/sh for installer scripts (pre- and postinstall scripts in pkgs).
The Posix sh standard does not include the ‘here string’ <<< used in Erik’s command. Nevertheless, when you use the above construct in sh on macOS it will still work fine. This is because sh on macOS is actually handled by bash in sh compatibility mode.
And while bash in sh compatibility mode will recreate the odd syntax and quirks of sh for compatibility, it will also happily run ‘bashims’ that don’t even exist in sh. E.g. double brackets[[...]] or here strings <<<.
As long as bash is taking care of sh scripts, you will be fine. Myself, I would not have noticed this if I had not ran a sh script through shellcheck. Shellcheck told me that Posix sh should not understand the here string:
But we’re fine on macOS, because sh is really bash right? No need to worry…
Predicting the demise of bash
We don’t yet know when, but the /bin/bash binary will eventually be removed from macOS. The version included with macOS has not been updated since 2014. It is unlikely that the twelve year old bash v3.2 will receive patches for security vulnerabilities in the future.
With the switch to zsh as the default shell in Catalina Apple is laying the groundwork to remove bash v3.2 from the system in a future release. cron and login hooks were infamously deprecated in macOS years ago and are still around, so we could be looking at years. But my suspicion is that bash is just one security vulnerability away from being removed.
With the High Sierra and Mojave updates, Apple has shown that they do not need to wait for major releases to remove or add functionality or restrictions to the system.
I believe /bin/bash has more than a few months still. However, Apple’s messaging on this switch is uncharacteristically strong. It would not be terribly surprising if the Catalina Spring Update in March 2020 removed bash.
The amount of old installer packages which would break on such a change is terrifying. Apple might be willing to pay this price, to avoid a publicized security vulnerability. A severe bash vulnerability would definitely get a lot of publicity. Remember Shellshock? That was the last time bash v3.2 was patched.
Instead of removing /bin/bash Apple might be able to replace it with zsh in bash emulation mode. So far, I have found no signs for this. While it wouldn’t be perfect, it would be a safer move.
Python 2.7 will still get updates until the end of 2019. There are also some commands and tools in the system written in python 2.7. It will be harder for Apple to migrate these to other languages, so I think Python has a longer countdown clock. But it cannot hurt to start preparing now.
To test script compatibility with Bourne-compatible shells in macOS Catalina, you can change /var/select/sh to /bin/bash, /bin/dash, or /bin/zsh. If you change /var/select/sh to a shell other than bash, be aware that scripts that make use of bashisms may not work properly.
zsh can be made to emulate sh by executing the command zsh --emulate sh.
We can actually see that Apple has two options here. sh processes could be run by zsh in emulation mode or by dash.
dash is the Debian Almquist shell, which is a minimal implementation of the Posix sh standard. dash is a newcomer to macOS on Catalina. New arrivals on a release where Apple seems intent on cleaning out ballast (32-bit, Kernel Extensions, bash v3, Python, Perl, and Ruby) are immediately of interest.
When you check on Catalina Recovery system, you can see that /bin/dash has been added, but /bin/zsh is absent. (/bin/bash is still present on Catalina Recovery and the sh binary is also still bash.)
Putting all of this together, I would hazard that Apple is planning to use dash to take over from ‘bash as sh.’ This will also bring macOS in line with most other Unix and Linux distributions where dash commonly handles sh processes.
With the symbolic link mechanism described in the support article, Apple could switch sh to dash in an update while bash is still present. This could allow Developers or Mac Admins to switch their managed systems back to bash when they need to mitigate problems.
When Apple switches to dash, sh scripts that use bashisms will break.
You can start testing this in Catalina by changing the symbolic link at /var/select/sh:
% sudo ln -sf /var/select/sh /bin/dash
(Change /bin/dash to /bin/dash to revert to default.)
When I test the above command with dash, it returns:
Syntax error: redirection unexpected
(You can also change the shebang of a script you want to test to #!/bin/dash then you do not have to change the system sh.)
Getting the Current User in sh, Future Proof (I hope)
The solution is easy enough. Replace the here string with a pipe. For sh scripts, change the syntax to:
This will make Posix sh, dash, and shellcheck happy and will still work in bash and zsh.
This syntax (echoing into a pipe) is considered inelegant and inefficient, because it requires an additional process (echo) and an additional pipe. Here docs and here strings were introduced exactly to avoid this. Obviously, if you prefer, you can continue to use the here string form of the command for bash and zsh.
There is no need panic and search and replace the python command in all your scripts immediately. I would recommend replacing this line when you update a script. Keep all of this in mind when working on your administration, installation and workflow scripts going forward.
You may also find scripts in third party documentation and installers that need updating, and you should let the author or vendor know.
/bin/bash (v3.2) is deprecated and has not received updates since 2014
/usr/bin/python (v2.7) is deprecated and will not receive new updates starting in Jan 2020
both will eventually be removed from macOS
we don’t know when exactly, could be years, could be months
even when you do not use Python for scripting, you may be using Python ‘one-liners’ in shell scripts
bash scripts should be moved to zsh (general use) or sh (general and installation scripts)
sh processes in macOS will likely be handled by dash in the future
use shellcheck to find and correct bashisms in sh scripts
Things are heating up for September release season. We got new betas for iOS and iPadOS 13.0 and macOS Catalina beta7. Surprisingly, there also is a beta for iOS 13.1, which includes some features shown at WWDC, which were removed from the 13.0 beta. While I generally approve of this flexibility, it makes testing confusing and more complex.
We also got a second supplemental update for 10.14.6, fixing some more bugs and security holes. The newest 10.14.6 has the build number 18G95. The silver lining in all this version chaos is, that we will likely not end Mojave on a forked hardware build.
Apple sent out the invitations to the iPhone event on September 10, 10am PDT (19:00 CEST). If previous years can be used as a guide, this could mean the iOS 13.0 release will be on September 13 or 20 and the macOS 10.15.0 Catalina release on September 20 or 27.