Using ScreenCaptureKit on macOS

しむどん ： 2024-10-25

Why am I addressing this issue?

OBS is a well-designed software that captures, composites, and distributes game screens. However, on a PC with low specs, running OBS while playing a game can result in a choppy screen. There are many ways to quickly remedy this, such as buying a new PC or installing a better GPU. But these things require money. And I am poor.

One way is to review the OBS settings. While the settings should obviously be reviewed, it may not be effective on a PC with too low specs. It may also be possible to transfer data to another machine, encode them, and distribute them, but OBS is an integrated software, not designed to work in concert with other systems, using individual functions. In other words, it does not follow the idea of "doing one thing well" in the UNIX philosophy.

However, the function needed in this case is a screen capture. The actual function needed should be simpler and smaller. We wanted to grasp the essential aspects of this process.

The idea of "doing one thing well" in UNIX philosophy is still important, and we believe that by practicing it, we can build better software. By grasping the essentials, we can also have the flexibility to respond quickly to changes in the surrounding environment.

When writing software, I often think about the importance of "don't have what you don't need. Having things you don't need adds unnecessary complexity, unnecessarily reduces maintainability, unnecessarily makes things unnecessarily difficult to coordinate, and unnecessarily makes things unnecessarily vulnerable in terms of security. But the point of "need" varies from person to person and requirement to requirement. That is why it is necessary to understand what is necessary. In this case, it would be to capture the screen.

In addition, it is a good opportunity to learn macOS, Swift, and Objective-C as a side benefit. By learning, you can expand your skill set and broaden the scope of your work. This also goes along with one of my life's themes, "Increase what I can do.

So this time, I decided to tackle this.

Survey

First, make something that works at all costs.

Display window

In programming, it is an ironclad rule to start with Hello world. This is not to jump into complicated things at once and get confused, but to start with simple things, and by establishing a firm foothold step by step, you can expand the level of complexity you can tolerate.

The author had just done "Hello world!" on the console in swift the other day, and it is still fresh in my memory. So I'll skip that part and start with displaying the "Hello world!

import SwiftUI

@main
struct MyApp: App {
    var body: some Scene {
        WindowGroup {
            ContentView()
        }
    }
}

struct ContentView: View {
    var body: some View {
        Text("Hello, World!").padding()
    }
}

Let's build it.

swiftc -o window -parse-as-library  \
		-Xlinker -framework -Xlinker SwiftUI -Xlinker -framework -Xlinker Foundation \
		window.swift

Each option you specify has the following meaning.

1. **`-o window`**:
   - Specify the name of the output file. Here, we name the compiled executable `window`.

2. **`-parse-as-library`**:
   - This option indicates that the specified source file is to be compiled as a library. This option is used for files that do not contain entry points (e.g. `main` functions). It builds the library for the purpose of being used primarily as a module.

3. **`-Xlinker`**:
   - This is a prefix to specify options at link time. The following arguments are passed to `linker` at link time. This option provides information about linking that is needed in the process of generating an executable file after Swift code has been compiled.

4. **`-framework`**:
   - This option directs that the specified frameworks be linked. In this case, the `SwiftUI` and `Foundation` frameworks will be linked.

5. **`SwiftUI`**:
   - It is Apple's framework for building user interfaces and is used to declaratively build UIs for modern applications.

6. **`Foundation`**:
   - It is the framework that provides Apple's basic data structures and functionality, supporting classes and functions such as date, array, dictionary, string, and error handling.

7. **`window.swift`**:
   - The name of the Swift source file to be compiled and linked. This file contains Swift code to generate an executable file called `window`.
Explanation by ChatGPT

Let's execute it.

./window

./window-2024-10-25-12-19-03.png

画面をキャプチャする

Let's capture the screen using ScreenCaptureKit. We will create a window to display the results. This code is almost verbatim from https://qiita.com/fuziki/items/ad7d21d4aa01bb0357d3.

import SwiftUI
import ScreenCaptureKit

@main
struct MyApp: App {
    var body: some Scene {
        WindowGroup {
            ContentView()
        }
    }
}

struct ContentView: View {
    @ObservedObject var viewModel = ContentViewModel()
    var stream: SCStream!

    var body: some View {
        if let image = viewModel.image {
            Image(nsImage: image)
                .resizable()
                .scaledToFit()
        } else {
            Text("Hello, world!")
                .padding(120)
        }
    }
}


class ContentViewModel: NSObject, ObservableObject {
    @Published var image: NSImage?
    var stream: SCStream!
    override init () {
        super.init()
        Task {
            do {
                try await setup()
            } catch let error {
                print("error: \(error)")
            }
        }
    }

    @MainActor func setup () async throws {
        let availableContent = try await SCShareableContent.excludingDesktopWindows(
          false, onScreenWindowsOnly: false)

        guard let window = availableContent.windows.first(
                where: {$0.owningApplication?.applicationName == "Emacs"})
        else {
            print("No window")
            return
        }
        let filter = SCContentFilter(desktopIndependentWindow: window)

        let configuration = SCStreamConfiguration()
        configuration.width = Int(window.frame.width)
        configuration.height = Int(window.frame.height)
        configuration.showsCursor = true

        stream = SCStream(filter: filter, configuration: configuration, delegate: nil)
        try stream.addStreamOutput(self, type: .screen, sampleHandlerQueue: DispatchQueue.main)
        print("Start capture")
        try await stream.startCapture()
    }
}

extension ContentViewModel: SCStreamOutput {
    func stream(_ stream: SCStream,
                didOutputSampleBuffer sampleBuffer: CMSampleBuffer,
                of type: SCStreamOutputType
    ) {
        guard let pixelBuffer = sampleBuffer.imageBuffer else { return }
        let size = CGSize(width: CVPixelBufferGetWidth(pixelBuffer),
                          height: CVPixelBufferGetHeight(pixelBuffer))
        let ci = CIImage(cvPixelBuffer: pixelBuffer)
        let cg = CIContext().createCGImage(ci, from: .init(origin: .zero, size: size))!
        self.image = NSImage(cgImage: cg, size: size)
    }
}

In this code, there is a place to select which window to target. For convenience, I have used Emacs, but this should be changed to the window of the application you wish to target.

guard let window = availableContent.windows.first(
        where: {$0.owningApplication?.applicationName == "Emacs"})
else {
    print("No window")
    return
}

window is an instance of SCWindow and the attributes are described in documentation. It is recommended to use it to specify the target well.

Now let's build with switfc.

swiftc -o capture -parse-as-library capture.swift

Let's execute it.

./capture

References