Computer-Assisted Automatic Harmonization Processes: A Synchronic Analysis of the Main Musical Tools ()
1. Introduction
1.1. Contextualization
In synthesis, a composition is a process of musical creation that can be classified, according to the sequential method of working, into three distinct instances of occurrence: 1) a melody is created according to a certain harmonic progression, i.e. the melodic line is conceived from notes directly linked to the established chords; 2) creation of a melody (monodic line) combined with harmonization (or vice versa), where both are interwoven simultaneously, in a creative process that is variably reciprocal and connected. This way of working is very common in popular music, adding lyrics (poetry) as a third constituent of this structure; and 3) the generation of a harmonization based on a predetermined melody (Gang, Lehmann, & Wagner, 1998), taking into account some variables, such as “harmonic time”, “tonality”, “harmonic fields” derived from the respective key, as well as Gregorian modes, among others. This third instance is the focus of this work.
Thus, each musical creator depends on their skills as an instrumentalist. This stems from their training, as well as being based on their own repertoire, whether popular or classical (Tayyebi, Demir, Nemutlu, & Karadoğ, 2019). In this context, there are also a number of factors associated with their musical education in childhood (Román & Caudeli, 2019). Because of these aspects and others, each composer ends up developing a different method of carrying out the work in harmonization. This becomes necessary because the determination of the chords establishes a new level in the elaboration and conduction of an arrangement, which in turn is made on top of the music conceived/created. The notation of the chords will reinforce vertically what the melody wanted to say horizontally, because the treatment of the chords can establish different meanings, effects and perceptions for the listener, either specific or general, both in the individual bars and in the set of bars that follow each other, closing with the various cadences (Green, 1979).
In addition, nowadays, with chord notation initially made up with the help of automated technological tools, it is possible to manually interfere, a posteriori, in the harmonization results obtained. This allows for humanized reharmonization processes that enrich and expand the creator’s sound intentions, giving them more productivity, precision and optimization. In this way, it is possible to infer (by addition, substitution or enharmony), for example, with the addition of borrowed chords, diminished tetrads, secondary dominants, SubV chords, chromatic chords, suspended chords, inversions, etc. (Almada, 2012).
Thus, the scope of this article is to present, from a quantitative-qualitative exploratory study in tonal music, an overview of the current state of the art on tools that treat a melody as a database from which, depending on the basic programming language, it is possible to obtain automatic harmonizations. These algorithmic technologies can be defined here as the resource used for this purpose, categorized into two lines of configuration: program and plug-in. A computer program or software can be conceptualized as a set of instructions, installed independently, that describe a priority task to be carried out by means of the melody (input), obtaining a harmonization (output). The plug-in has the same functional role when it comes to data input and output, but it is installed as a resource in a particular piece of software, where it is assigned an additional (optional) function. In other words, the plug-in is built into the software, as illustrated in Figure 1 below.
![]()
Figure 1. A software can be divided into two types: plug-in or program.
This work also aims to point out the advantages, limitations and peculiarities of some of these technological tools, as well as discuss the most viable alternatives and minimum characteristics. This research represents a vital source of information for composers and arrangers who have had, in their preparatory musical life, pedagogical training based exclusively on monophonic wind instruments. This includes clarinetists, saxophonists or trumpet players, for example, who have no skills whatsoever on polyphonic instruments with keys and strings: guitar, ukulele, cavaquinho, keyboard or piano.
1.2. Literature Review
The automated harmonization of a monophonic line is something that has been proposed and studied for some time in Western tonalism. This is possible by using basic rules that range from traditional harmony to functional harmony, as there is a probability of certain diatonic chords occurring according to certain predictability criteria (Kostka & Santa, 2018), as illustrated in Figure 2.
Figure 2. Chord progression in major mode based on estimates (Kostka & Santa, 2018).
According to this linear criterion of expectation or obviousness, it is possible that, from the ordered model proposed above, a sequence of chords can occur based on a melody proposed in advance. This can happen, for example, in the following way: degree I can migrate to any degree of the harmonic scale. If it migrates to degree iii, it can make the sequence IV, V, vi, viio and I, as well as other similar sequences that may go in a different direction at some point. This is just one possibility among the infinite alternatives, given a few preliminary parameters.
One of the first theoreticians to propose this alternative used linguistic concepts for a computational basis (Winograd, 1968), and this application concept was later expanded (Rothgeb, 1979). A heuristic data approach was also suggested (Ebcioglu, 1986) and finally, a model based on Bach’s chorales (Steels, 1986), as illustrated in Figure 3. In this case, the harmonic interval rules established by this composer serve as a rule for voice leading.
Figure 3. Johann Sebastian Bach’s first chorale, published in 1748 (notebook N˚ 04).
Subsequently, different methods have emerged and approaches have been developed, as well as other commercial tools launched on the computer market. Some of this software has fallen into disuse or become obsolete, as is the case with the i-Ring application (Lee & Jang, 2004). This was probably due to more elaborate competitors that were able to perform the same tasks with a higher level of efficiency, considering the results obtained by the first software.
MySong is a system developed by researchers at the University of Washington, in partnership with Microsoft, which uses the voice as a melodic basis from which a harmonic accompaniment is generated (Simon, Morris, & Basu, 2008). The system was tested on 13 musically untrained participants who recorded their singing voice and then received a piano chord sequence response from the software (Morris, Simon, & Basu, 2008).
A study for automatic harmonization and interaction using Pure-Data (PD) was described in an interpretive study for flute (Figueiró, 2008). In this work, the notes played by the flute were recognized, mapped (pitches determined) and sequenced. A score is then generated consisting of two systems. The first system is the flute melody and the second system is the computer itself, which exposes the overlapping of notes at specific points (forte tempos). This is how the harmony is generated.
There are also alternative solutions on the market for harmonizing arpeggiated notes. The Casio digital keyboard is an automatic harmonizer with a rotary pitch bend (model CT-X800C2-BR) and an internal plug-in.
In general terms, there are currently a series of four approaches to the subject of automatic harmonization (Santibáñez & Biscainho, 2017). In the field of Symbolic Artificial Intelligence, the first approach involves grammar-based methods. The idea is based on the fact that language is structurally and syntactically similar to a melodic line. To this end, one of the coding systems used was developed by Lindenmayer, also known as L-systems (Lindenmayer, 1968).
The second branch of Symbolic Artificial Intelligence involves Constraint Satisfaction Problems (CSPs). This latter approach is based on a search for possibilities circumscribed by constraints, which facilitates the appropriate delimitation of possible/involved results.
Optimization methods are the second line on which the subject of automatic harmonization resides. Within this field, sophisticated algorithms use the strategy of evolutionary harmonization based exclusively on physiological factors (Moroni, Manzolli, Zuben, & Gudwin, 2002). But there is also the line of knowledge-based evolutionary algorithms (McIntyre, 1994), where a generic instruction line encodes a set of harmonic rules to be applied.
The third approach involves the well-known Neural Networks (Shibata, 1991) and (Costa, 2019), in which an associative memory is proposed in the form of a module. The system thus generates a series of associative tables of chord transitions, which can be generated by the Effective Boltzmann Machine (EBM) (Bellgard & Tsang, 1993).
The system solves the problem of harmonization as a non-deterministic and non-sequential process, in the sense that to harmonize a melodic event, it uses information from future events, which is why the practical implementation has a high computational requirement, but some possible improvements are proposed such as the use of deterministic Boltzmann machines (Santibáñez & Biscainho, 2017).
Finally, the fourth approach involves Bayesian Networks, which are graphical models of associations of a set of random data to represent factorizations of particular distributions. Mixed methods are also considered here, as they involve Markov models (Eddy, 1996) and other models of knowledge and rules. The system is capable of generating harmonic sequences from neo-Riemannian transformations (Cohn, 1998). In this context, hidden Markov models are yet another subclass of simple Bayesian networks used in time series modeling. Also included in this fourth approach are Probabilistic Finite-State Machines (PFSMs).
In this way, it is understood that all the tools mentioned here have equivalent purposes, but can generate different results because they derive from computational logics with numerous particularities and varied algorithms. Thus, whether plug-in or app, each of them can generate feedback that varies according to the expectations, requirements, possibilities and demands of a given user.
2. State of the Art of the Most Popular Tools
2.1. DAW-Type Computer Stations
A DAW (Digital Audio Workstation) is a digital audio workstation on which you can treat a particular track, which in turn will become a finished phonogram. With it you can record, edit, mix, import sounds, use effects and, finally, export audio in various types of formats, depending on the use to be made of it. The first DAW used as a computer workstation was the Fairlight CMI (Computer Musical Instrument), commercially launched in 1979. In the years since, several DAWs have been released and others have fallen into disuse. The most widely used DAWs on the market today include: Tracktion, Synclavier, Studio One, Soundtrap, SawStudio, Samplitude, Renoise, Reason, Reaper, Pro Tools, Podium, Nuendo, Music Maker, MuLab, Mixcraft, Mixbus, MetaSynth, Logic Pro, GarageBand, FL Studio, Digital Performer, Cubase, Cakewalk by BandLab, Bitwig Studio, Audiotool, AudioFrame, Adobe Audition, Acid Pro and Ableton Live.
DAWs serve as a reference because small programs called plug-ins are installed in them. In this context, plug-ins perform supporting functions that are limited to, for example, harmonizing a melody in the general MID format. The MID file (mid or midi extension) is a hexadecimal standard created in 1982 to standardize the digital sounds of musical instruments. This standard was given the name Standard MIDI Files (SMFs). It can be recorded in two basic types: 0 (in this format, all the tracks are grouped into a single track) and 1 (in this format, the tracks are separated into separate tracks).
2.2. MIDI Standard (Musical Instrument Digital Interface) and
Plug-Ins
In 1991, a standard was adopted by the MIDI Manufactures Association (MMA) and Japan MIDI Standards. It was called General MIDI System Level 1 or simply General MIDI (GM). This was necessary in order to establish the same timbres for correlated numbers from 1 to 128, or alternatively, from 0 to 127. This standard has been accepted worldwide to this day. That’s why the midi format can be imported into any DAW environment and even harmonized automatically by a specific plug-in.
Most harmonization plug-ins are built into software or DAWs. They work as a small set of instructions with a specific function. Some other plug-ins, depending on the level of development or employment, can be sold separately, as small applications capable of assigning additional functions that make them independent. These are generally more expensive and are purchased on single sites. Here are some of these applications, listed below in Table 1.
Table 1. List of the 10 most renowned plug-ins (sold separately).
1 |
Captain Chords Epic |
$99.00 |
2 |
Cthulhu |
$39.00 |
3 |
Scaler 2 |
$59.00 |
4 |
INTUITIVE-AUDIO.COM |
£40.00 |
5 |
InstaChord W. A. Production |
$69.00 |
6 |
ChordPotion |
$49.00 |
7 |
AutoTheory |
$99.00 |
8 |
Liquid Notes |
$149.00 |
9 |
Midi Madness 3 |
£59.00 |
10 |
Hookpad |
$149.00 |
2.3. Dedicated Software
In this study, we also looked at the most robust tools that can be used exclusively for the automatic harmonization of monophonic lines. These are generally more professional, for exclusive use only, and require a good level of technical knowledge in the field of computer science from the user. In other, more extreme cases, they require the user to have knowledge of programming and data compilation. This differs greatly from plug-ins. But in general, what is presented here is the current state of the art of software (programs as such) that are in sync with the real demands of Western tonal music.
To begin the usual account of these programs, mention should be made of a harmonizer based on “musical experience” that was developed under the name Harmonic Analyzer (Temperley, 2012). This is an example of a program designed exclusively for automatic harmonization. In this specific case, melodies are harmonized using an interpolated probabilistic method. Using its own algorithm, the distribution of chords is done on a two-dimensional checkerboard graph, with monochrome graduations indicating the calculated/suggested priority chords (Raczyński, Fukayama, & Vincent, 2013). A compiler is needed to run the specific database, according to the language adopted in the original project.
More up-to-date and robust tools are being applied for more accurate processing of Western tonal music. HarmTrace is designed in the advanced programming language Haskel and is compiled for topical application (De Haas, Magalhães, Wiering, & Veltkamp, 2013). It was initially presented as a harmonization solution for jazz, but it can also be used normally in the context of classical music.
One of the most widely used powerful tools is one that uses the hidden Semi-Markov model (Groves, 2013), known in the literature as the Hidden Semi-Markov Model (HSMM). The differentiation of this method is that a particular style of music such as Rock’n Roll can be established. The HSMM method also has the advantage of establishing harmonic limits and considering an even more complex variable: note duration. According to Figure 4, the final result provided by the system is a score with the degrees of traditional harmony (with or without inversions).
The Hidden Markov Model (HMM) is widely used to determine a chord sequence (Raczyński, Fukayama, & Vincent, 2013). This method uses an interpolated probabilistic model capable of predicting: tonality, most likely chords, progressions resulting from the cycle of just fifths, descending progressions of thirds and harmonic compatibility, among other variables. Figure 5 below shows the final result of the HMM-derived process, based on the main melody of Radiohed’s Karma Police, by composers Colin Greenwood, Jonny Greenwood, Ed O’Brien, Thom Yorke and Philip Selway.
Figure 4. Harmonization using the HSMM method (Groves, 2013).
Figure 5. Harmonization using the HMM method (Raczyński, Fukayama, & Vincent, 2013).
Some more recent studies have used the same method (HMM) in the world of American jazz (Kaliakatsos-Papakostas, Velenis, Pasias, Alexandraki, & Cambouropoulos, 2023). In this particular work, the harmonic progression pattern of an already known song was used and, on top of the other unharmonized song, a sequence was created from a cross-harmonization reference. This uses the hidden Markov model and the Viterbi algorithm (Viterbi, 1967).
Figure 6. Example of automatic harmonization in four voices (Zhu et al., 2021): BacHMMachine harmonization excerpt on the melody of J.S. Bach’s Choral Ach Gott und Herr, BWV 255.
The automatic determination of chords in a monodic line can also be approached with the help of an interpretable and scalable model for algorithmic harmonization, known as BacHMMaquine (Zhu, Hahn, Mak, Jiang, & Rudin, 2021). This robust tool employs a theory-driven structure together with a data-driven model. The reference is the lead voice: soprano. The advantage of this model is that it can be used in various classical music genres, as well as popular rhythms such as rock. The final chord is drawn from the three lower voices, which have been generated in block form, as shown in Figure 6. There are also contrapuntal variations.
Due to the diversity of tools used to automate tonal harmonies, there have been recent studies comparing different methods. In one of these iconic studies (Yeh et al., 2021), four different techniques were compared for a given monodic line (melody shown in Figure 7): model matching, hidden Markov model, generic algorithm (Ponce de León, Iñesta, Calvo-Zaragoza, & Rizo, 2016) and deep learning. Considering 48 different triads, the evaluation was conducted taking into account 9226 pairs of chords. In addition, the subjective participation of 202 (two hundred and two) participants was taken into account as evaluators of the results obtained.
Figure 7. Hey Jude melody—The Beatles, 1968 (Yeh et al., 2021).
For practical purposes, this work used an authorial melody in D major, as shown in Figure 8. Three methods were used in this monodic line: harmonization based on practical experience (empiricism as a natural process), the use of the “Add Simple-Harmony” plug-in for automatic harmonization in the Sibelius First (music notation) software and, finally, the use of the portable Band-in-a-box V2021 for harmonic automation of the given melody. The result is shown below, indicating a variation of chords that are largely derived from the harmonic field of which they are part: D (Dmaj7), Em (Em7), F#m (F#m7), G (Gmaj7), A (A7), Bm (Bm7) and C#o (C#Ø7).
Figure 8. Introduction to the waltz “Simone Ribeiro Leite” (authored by: Roniere Leite Soares).
Between triads and tetrads of the Ionian mode, in just eight introductory bars, seven chords (D, Dmaj7, Em, F#m7, G, A and A7) from this harmonic field were contemplated, out of a possible fourteen. In other words, half of them are present in the three pentagrams presented.
Figure 9. Screenshot of the import of the *.mid file into the internal Band-in-a-box environment.
The harmonization suggested by Sibelius was the simplest, while the first, done by hand (empirically), was considered intermediate. However, of all the tools tested in the introduction of Simone Ribeiro Leite’s Waltz, the one that made harmonization most sophisticated was the function built into Band-in-a-box® for Windows (version 2021), as shown in Figure 9. Its versatility is due to the fact that harmonization can be suggested according to the musical genre selected in advance. It is also possible to choose the rhythmic style to be incorporated into the version to be harmonized. The result was given according to the pre-chosen jazz style. Chords with sixths, minor seconds, altered fifths and major sevenths were suggested, adding tension, interchangeable and substitute notes.
Experimenting with any tools, from the simplest (plug-ins) to the most complex (lines of instruction contained in programs), can bring more comfortable ways for the composer to deal with harmonizing a melody.
3. Results and Conclusion
In the realm of Western tonality, any instrumental music composer who has limitations in defining the harmony of their creations can make use of the technological tools that are now available (free or paid) on the web. This can help the music creator to determine an automatic harmonization that consequently makes them develop an arrangement with more fluidity, precision and productivity. This does not prevent composers who have mastered the process of harmonization and reharmonization from making use of tools that complement and expand the possibilities of intervention in the process of arrangement and orchestration, for example. As a natural consequence, one way or another, precious gains are made in the production line of the work. The methods discussed here and the tools listed and commented on here are alternatives that give interested composers new potential. Thus, it is concluded that:
In the search for the best alternative, you can use free plug-ins, such as those tried on Sibelius and Band-in-a-box, or you can purchase more powerful paid plug-ins, such as the ten listed in Table 1 (in dollars or pounds).
For a more systematic and methodical application, as discussed in this research, other approaches that have more robust applications should be considered, such as Haskel advanced programming language, Viterbi algorithm, Markov Models, L-Systems (by Lindenmayer) or Boltzmann’s Effective Machine.
It’s also possible to try out other software, such as i-Ring or Microsoft’s MySong, even though these are relatively outdated. But other powerful tools are also available, such as Pure-Data (PD), Harmonic Analyzer, HarmTrace, HSMM, HMM and BacHMMaquine. All of these have been covered in this work.
In the end, we have to address composers who have not yet embraced the new computer technologies. In today’s world, it is inevitable that they will be immune to tools that do not influence their work routine. At the latest, with the recent advent of Artificial Intelligence (AI) products and machine learning, creative music will be produced on an inseparable two-way street: the human way and the data science way.