Adobe’s new AI tool turns silly noises into realistic audio effects

Adobe is launching new generative AI filmmaking tools that provide fun ways to create sound effects and control generated video outputs. Alongside the familiar text prompts that typically allow you to describe what Adobe’s Firefly AI models should make or edit, users can now use onomatopoeia-like voice recordings to generate custom sounds, and use reference footage to guide the movements in Firefly-generated videos.

The Generate Sound Effects tool that’s launching in beta on the Firefly app can be used with recorded and generated footage, and provides greater control over audio generation than Google’s Veo 3 video tool. The interface resembles a video editing timeline and allows users to match the effects they create in time with uploaded footage. For example, users can play a video of a horse walking along a road and simultaneously record “clip clop” noises in time with its hoof steps, alongside a text description that says “hooves on concrete.” The tool will then generate four sound effect options to choose from.

This builds on the Project Super Sonic experiment that Adobe showed off at its Max event in October. It doesn’t work for speech, but does support the creation of impact sounds like twigs snapping, footsteps, zipper effects, and more, as well as atmospheric noises like nature sounds and city ambience.

New advanced controls are also coming to the Firefly Text-to-Video generator. Composition Reference allows users to upload a video alongside their text prompt to mirror the composition of that footage in the generated video, which should make it easier to achieve specific results, compared to repeatedly inputting text descriptions alone. Keyframe cropping will let users crop and upload images of the first and last frames that Firefly can use to generate video between, and new style presets provide a selection of visual styles that users can quickly select, including anime, vector art, claymation, and more.

These style presets are only available to use with Adobe’s own Firefly video AI model. The results leave something to be desired if the live demo I saw was any indication — the “claymation” option just looked like early 2000s 3D animation. But Adobe is continuing to add support for rival AI models within its own tools, and Adobe’s Generative AI lead Alexandru Costin told The Verge that similar controls and presets may be available to use with third-party AI models in the future. That suggests that Adobe is vying to keep its place at the top of the creative software foodchain as AI tools grow in popularity, even if it lags behind the likes of OpenAI and Google in the generative models themselves.