[{"content":"Lejun Min, Junyan Jiang, Jingwei Zhao, Ziyu Wang, Yixiao Zhang, and Gus Xia · Music X Lab\nA practical guide from Music X Lab. Adapted from the original collaborative manuscript; examples and recommendations retain the authors’ original wording.\n(You usually need to use “anonymous” on submission if it is required by the conference. If it’s the first time for you to write a paper, share with your group a google doc template of the conference. If you are experienced writer, feel free to directly work on overleaf.)\nAbstract\rThis is a guideline on how to write a Music AI paper in Music X Lab. Please read it carefully, especially if it’s the first time for you to write a computer-music paper for ISMIR, NIME, ICLR, NeuriPS, or a related conference. In this guideline, we will show you the key points to cover in each section of the paper, as well as some common mistakes you should avoid. Hope you find this guidance helpful. In general, the abstract should be a short summary of the intro, the intro is a summary of the whole paper, and your title should be a one-sentence take-away message of your abstract. 1 Introduction\rIntroduction is a short “story telling” about your research. A clear and concise logic flow is the key to a successful introduction — you should make the length of the “chain of logic” as short as possible, and try to introduce fewer concepts. As lengthy logic flow with many new concepts are hard to understand. When introducing new concepts, keywords or even the key statement shall be in italics when (and only when) it appears the first time. Sometimes, you can re-emphasize them using italics when it appears again in a new section (say, Methodology).\nIn general, a chain of logic contains the following parts: 1) a general background of the field and its current focus, 2) the main problem of current approaches, 3) your novel solution, and 4) main results. The third part is a summary of the methodology section, and here you only need to highlight the high-level ideas. The fourth part is optional, but it’s usually included in the intro. At the very end of the intro section, we usually re-state our contributions as bullet points. (It’d be helpful to read at least 1-2 previous papers, especially the ones sharing a similar topic, to get some concrete ideas.)\nFor an excellent paper, its introduction is by itself enough to win an acceptance. The reviewer is already convinced by the story, and s/he only needs to double-check the experiment to make sure there are no major flaws. Please start to draft the intro at least 3 weeks before the paper deadline. (if you are an experienced writer, maybe you can do it 10 days before ISMIR ddl and 2 weeks before ICLR). Seek Gus for help/edit on this part as early as possible, since the structure as well as the terminologies used in the intro will be unfolded and reused in the rest of the paper.\n2 Related work\rRelated work is usually arranged after introduction. In special cases (when it’s a super innovative study and there are very few related papers), you can put related work before conclusion. Also, if the paper is short, say a 4 paper ICASSP paper, you can merge related work with the intro. In this section, we will first talk about the structure of related and then discuss some common issues. 2.1 Structure of related work\rWe need a very brief paragraph to summarize the content of the whole section between 2 and 2.1. The previous short paragraph (related work is usually … common issue) is an example. The same idea also applied to every following sections. After this structural summarization, you dive into each topic of related works.\n2.2 Content of related work\rIn the related work section, you usually summarize the works in concise words. You don’t have to mention each aspect of the papers but need to highlight the points that are related to your study. When you cite a work, don’t use “[ref No.]” to begin a sentence, instead try to use the system/model name (e.g. Transformer) or authors names (Chin et al.).\n2.3 The Essence of “Related”\rThe essence of related work is not only to show or list the existing works, but more importantly, to highlight how your study is different. The differences should be conveyed in a very concise way.\nMethodology\rThere is actually no clear Cliché (or 套路) for a good methodology. It is the actual content of your research and of course each study is so different. In many cases, you first talk about data representation and then talk about the model architecture (but things can be highly customized).\nModel architecture\rThe model graph is usually the most important part (at least the most important picture) of your paper. Many people in your field are going to directly jump into the model graph after reading the abstract. So, the model graph should be a clear “graphical abstract” of your method, and the caption itself should be enough to explain the figure.\nHow to draw nice graphs\ryixiao.zhang@qmul.ac.uk\nWhy do we need these figures? Because Figures can attract human’s eyes, and understand the paper easier. Research has shown that the level of difficulty in understanding different elements follows this order: images \u0026lt; tables \u0026lt; text \u0026lt; pseudo code. Try to use a teaser figure on the first page! Then people will look at it immediately after reading the title. It can be very simple, just showing your workflow, or a concept graph, or example. You will need an architecture diagram. It should be understandable, even can let people skip the methodology section at the first glance.\rHere is the correct order that help you make a figure easier.\nDecide this is your workflow or your architecture. Mixing them may be confusing. Decide it is a single column graph or a double column graph. It decides the direction of your figure. Prepare a piece of paper. Draw a rectangle. Decide your core component in small rectangles. Place small rectangles into large rectangle. Modularity is always good. Put your hierarchy mind here. Fill the main details. Always draw the most important elements first. It is okay not put everything insides. Put those details into your captions. Open your preferred software. Diagram: draw.io Color: coloors.co (do not use default colors) Icons: thenounproject.com\rFor each small rectangle, keep them in a harmonic color scheme. It is okay to use different style across small rectangles. Use shallow colors as background. Use dark colors as text and components. Never use big red, big green, big blue, etc. Keep the color not that sticking. Check it again. Keep the structure balanced. Align elements. The design should not be too narrow. Make it relaxing. Keep the text bigger if possible. Write your captions. It should be detailed enough, including a brief introduction of every small rectangle.\rPlease refer 刘洋 学术论文.pdf for a more detailed guidance.\nGenerated example\rIf your work is about music generation, especially in the symbolic domain, it’s better to show generated high-quality examples in an individual section. Examples are poly-dis, audio-to-symbolic, and whole-song-gen.\nThe score should be well-formatted. First, pay attention to the general formatting issues. Here is a non-exhaustive list of things you should pay attention to.\nClefs (are they missing in screen shot) Time/key signatures (Are they correct? Are they missing?) Barlines (are they missing? Make sure the last bar in your screen shot should be ended with a barline)\rThen, take care of the note-spelling, note durations, and voices. Even though we usually generate MIDIs or so-called symbolic scores, where such information is usually missing, we have to infer it, because otherwise the score is not readable. We can use a footnote to mention what information is inferred. See the sheet music below, it’s not readable at all, worse than a piano-roll. (Also, do you notice the background color of this figure is not white? It should be avoided as well.) Figure 1. A poor sheet music illustration.\rFinally, try to make some analysis directly on the sheet music if necessary. In my experience, this is the most efficient method. If you try to write analysis elsewhere, you will use additional space. Remember the sheet music already costs you a lot of space.If you are not certain about the wording, consult your colleagues.\nIn conclusion, using sheet music in symbolic music generation is an effective way to demonstrate your sample quality. Although whether conventional music notation is suitable for all music types is debatable, this is one of the few methods that musicians can read. Other music formats are just unreadable. It also demonstrates you are a well-trained computer music scientist and you care about music.\nThe softwares to use are: Musescore/Finale/Sibelius for score editing and Microsoft Powerpoint for post-editing.\nExperiments\rDataset\rThe dataset section usually contains one subsection with the title “dataset” or “datasets,” or more subsections in case of complicated routines either in data collection or preprocessing. In any cases, the dataset subsection(s) should contain the following content:\nIntroduction: start with a brief description of the dataset you use. I.e., We use a Chinese pop music dataset POP909 [citation] to train and evaluate the model. Most relevant information includes the format (e.g., audio recordings, or multi-track performance piano MIDI), size (number of songs and the total length) and genre distribution of the dataset. Introduce each dataset in one or two sentences. Visualization: if text-based introduction is hard to make sense, provide a visualization of one or two samples in your dataset. If space is limited, consider merging it with a plot in “case study.” Source: properly reference the source of self-collected datasets. Describe them in detail if they have never appeared in previous literature. If you feel it necessary, provide a simple reason why existing datasets fail to meet our requirements and we have to create our own. Avoid listing too many reasons, since the readers might think you are not confident enough. Crawled datasets: clearly state the website/CDs/books you collect the data from and how you collect them (e.g., batch-downloaded using a specific genre filter). Provide a footnote to a project repo which details your crawled dataset. If the crawled dataset itself cannot be made publicly available due to copyright issues, in your project repo, provide a URL list to the songs you collected (e.g., if you got songs from Youtube) or a rough stepwise guidance to crawl the datasets. Self-created (recorded or annotated) datasets: clearly state the recording (annotation) environment (i.e., audio recorded on an Roland-XXX player piano in a large auditorium, audio synthesized with the FluidR3 soundfont [cite], labels annotated with Sonic Visualizer [cite]), how many people participated in the recording (annotation) process, the musical background of the players/annotators (i.e., amateur, intermediate or expert), the recorded (annotated) formats, and other important setup information. Always keep in mind that the reproduction of an identical recording environment or materials is often very hard for other laboratories. It is very useful to write down important setups in the paper, and make your recorded (annotated) datasets publicly available after your paper is accepted.\rData pre-processing: Describe all data pre-processing techniques used. If necessary, you may split it into one or more subsections. Dataset clean-up: If necessary, briefly describe how you filtered out illegal or dirty samples in the dataset. Feature representation: Describe in detail how you transform raw data (audio, MIDI or score) into the features (spectrogram, piano roll). Write all hyperparameters you used associated with the transformation. Example list of hyperparameters:\rSTFT Spectrogram: Sample rate, hop length, window shape (only if you are not using hanning) and window size. CQT Spectrogram: hop length, quality factor, base frequency, number of bins, number of bins per octave Piano roll: time resolution, how you represent onsets/sustain/rests Piano tree/REMI/Compound word etc: citation to the original paper, changes you made compared to the original paper\nNotice: DO NOT WRITE THIS SUBSECTION IN YOUR METHODOLOGY SECTION. The best practice is always to write your methodology as general as possible, and your experiment as specific as possible. Data augmentation: Describe in detail how you augment the data (along with the hyperparameters). E.g., Pitch shifting: e.g., from -6 semitones to +5 semitones Time stretching/resampling: e.g., from x0.8 to x1.25 Random truncating: length in seconds or length in beats, number of samples per song (in each training epoch) Noise addition: what kind of noise, what SNR Random note addition/deletion: you may omit some details (be sure to open-source your code) and just say randomly delete/add x% of notes SpecAugment, remixing etc: cite the original paper that proposes this augmentation method and write important hyperparameters you overrode\rDataset split: Write how you split datasets into the training/validation/test set (e.g., 80% for training, 10% for validation and 10% for testing). It is not recommended to go without the validation set. If multiple samples may come from the same song, mention whether you randomly split the dataset on a song level (correct) or on a sample level (incorrect). If you do cross validation, write down how many folds (usually 5 folds to 10 folds) you used.\rIf available space is extremely limited, consider retaining the most important descriptions of the dataset only (e.g., format, genre distribution, and a simple citation to the source) to make the following parts of experiments make sense to readers. Expand your dataset description in your supplement materials and/or your repository.\nExperiments (methods, objective metrics \u0026amp; writing)\raik2mlj@gmail.com\nBoth objective and subjective evaluations are to prove that our model does excel at its selling point. You cannot simply brag without providing scientific statistics. First, we should find out what baseline models should we compare to, and what metrics suit our demonstration best. This applies to both objective and subjective evaluation.\nAccording to the selling point of your study, find relevant baseline models that achieve similar goals. E.g., for symbolic music generation study, you can find Music Transformer, REMI, MuseCoco, MMT, MMM, FIGARO, PopMag, Polyffusion, etc., each with varied focuses (controls, multi-track, performance, arrangement, etc.). If your selling point is quality and controllability, FIGARO, REMI, and Polyffusion might be a good start. You can borrow the metrics from the papers of those baseline models! E.g., for symbolic music generation study, FIGARO provides comprehensive controllability metrics, PopMag has some good musicality metrics. Your model design may already provide several internal/specialized/common metrics to use! E.g., if you are using fixed-length models like Transformer, Perplexity (PPL) is commonly used to measure fluency. If your model contains several internal interpretable layers, you can prove their efficacy accordingly. In case you need to design a new objective metric, try to find similar metrics that deal with different objects, but whose calculative methods can be borrowed. E.g., many objective metrics can be regarded as the similarity / distance between two objectives (cosine similarity, distribution similarity, l2 distance…).\rNow we have a list of baseline models and metrics. Let’s do the writing!\nEvaluation Intro. It’s recommended to write an intro paragraph that summarizes the logic of the evaluation part. It should be concise and clear, stating the tasks (objective and/or subjective experiments) and the reason of such experimental design. If it’s a hassle to explain, you can also design a table that lists all the attributes (tasks, metrics, baselines) like this one in Polyffusion: Figure 2. Evaluation tasks, metrics, and baselines in Polyffusion.\rEvaluation Metrics. Next, elaborate all the subjective and/or objective metrics in each task. Give them good names (perhaps abbreviated) or symbols. When writing detailed implementation, refer to another paper if it is borrowed, or put down with words/equations if it is original or modified. If it takes too much space to explain, you can put extra details in the appendix.\nBaseline Models. Then, elaborate the baseline models we compare to in each task. Briefly summarize their model design and what they achieve. Usually, citing the original paper with a bit of explanation will be enough, but do mention the implementation difference if you have tweaked something (e.g. model trained on a custom dataset, parameters adjusted, etc.)\n(Optional) Ablation Study. Ablation studies are the experiments where you remove some designs/tricks from the proposed model and test its performance. In short, you may have some special baselines that are the reduced variations of your own model! This is especially important if you have a complicated pipeline, or if you want to emphasize the necessity of these designs/tricks.\nDesign a Good Table. Now you have got a bunch of data from the objective experiments, let’s put them into a nice table.\nRead the LaTeX guideline of your target conference first. They provide detailed styling for tables/charts that authors must adhere to. E.g., IJCAI suggests authors to use \\toprule, \\midrule, and \\bottomrule for better border styling in tables. Also notice where the table caption should be. (usually on top of the table, while for figures the captions should be at the bottom.) There are some awesome online tools for creating LaTeX table! I use this one to speed up the process, where you can paste multiple columns and rows from Excel and easily adjust the styling. (Just my suggestion) Avoid drawing unnecessary borders. Usually, horizontal split lines are clear enough for dividing the content. Use vertical split lines when there are multiple tasks or groups. Use visual hints to make it easier to comprehend. \\uparrow and \\downarrow after the metric to indicate “the larger/smaller, the better”, respectively. As a convention, make the best statistics bold. (Just my suggestion) Have a slightly different background color for Ground Truth using \\rowcolor from the xcolor package.\rAbbreviate the terminologies/model names if they are too long to fit the page width.\rHere is an example from Whole-Song Generation: Figure 3. An example results table from Whole-Song Generation.\rComparative Results. It’s the time to draw a conclusion: your model does beat the baseline models! If not all baselines on all metrics, at least in some ways.\nLet the data speaks. Grab the most essential message in the data and convey the positive conclusion in an objective tone. E.g., your model outperforms the baselines in several metrics, which indicates something does work better. Try to give a plausible explanation for something counterintuitive/worse than the baselines.\rSubjective Evaluation and User Study\rIf your work is about music generation, you probably need to do a subjective evaluation. Music X Lab has its established way of doing it. You can refer to EC2-VAE as a simple example. Later work may get more complicated but the methods keep more or less the same. In this section, we will show you how to design the survey, collect \u0026amp; analyze the data, and plot proper graphs to show your results. Python scripts mentioned in this section can be found here and here.\nWhat a Survey Should Look Like\rFor subjective evaluation, we typically want to compare mm models (including ours, baseline(s), and optionally ground truth) based on kk metrics. Suppose we have generated ll samples for each model, and now we will need to create a survey for evaluators to listen, compare, and rate.\nIn a nutshell, the survey should have a total of ll pages and each page should have mm questions for respective models. Within each question, evaluators should be able to listen to the sample and give kk ratings for respective metrics on a 5-point Likert scale. Note that if your task is conditional generation, then the mm samples in one page should be based on a common conditional input. For example, Figure 4 shows one partial survey page for Polyffusion on the task of accompaniment generation. There are m=3m=3 accompaniment samples/models based on the same conditional input (a lead melody in this case). They will be evaluated based on k=4k=4 metrics: Creativity, Naturalness, Musicality, and Coherency. The melody condition is also provided as context together with the instructions at the beginning of this page. Figure 4. A partial accompaniment-generation survey page for Polyffusion.\rSurvey Creation\rIn this section, we will talk about hands-on practice for creating a survey. The premise for survey creation is that you have rendered generation samples in a playable media format and hosted them on a media-sharing platform. With samples ready, we will use SurveyMonkey to design and distribute surveys under a subscribed account by Gus. In the following, we will go through the complete survey creation pipeline from sample rendition to survey design.\nSample Anonymization: It is very helpful to use an intuitive file naming scheme for generated samples, such as sample01_ours.mp3, sample03_baseline.mp3, sample10_groundtruth.mp3, etc. However, it will be very awkward if we leak this prior message to the evaluator. What we should do here is to 1) make a copy of each sample, 2) rename the copy based on a 4-digit random number, and 3) save a check table for corresponding anonymized and original sample pairs (important!). We will provide one Python script for this anonymization. To put samples onto the survey pages, you will consult the original names to make sure of the sample order, but the exact file to put on should be the anonymized version.\nSample Rendering and Hosting: If your generated samples are MIDI, then typically you will need to synthesize them as mp3 audio. Such a synthesis can be done in bulk using midi2audio but the sound quality may be limited. It is recommended to use BandLab (a free web client, if you do not have a DAW set up), where better sound fonts will be provided. However, you’ll have to do it one by one manually. After rendering, it is recommended to upload the anonymized mp3 files to SoundCloud, an audio hosting/sharing platform. From there you can generate a sharing link for each file that can be further embedded to the survey pages. Refer to the python script to automate your GUI to batch copy the embed codes from Soundcloud.\nSurvey Creation Quick Start: On the home page of SurveyMonkey after login, you will see a listing of a few previous surveys of Music X Lab. To quickly start your own survey, you can select the template survey, click the Options button (or ⋯\\cdots on the right), and click Make a Copy. This will make a copy of an already established survey and you just need to make modifications to that. On the 1st page, you can set up the survey title, lab logo(s), and welcome messages. The 2nd page typically aims to collect relevant information from evaluators; you don’t need to modify anything. Edit Question Pages: The 3rd page and after that will be question pages. Each question page starts with an instruction block followed by a few question blocks (one question for one model). You can edit each block by clicking on it. A more specific guidance is as follows:\nEdit Instruction Block: The instruction block contains instruction texts telling evaluators what to do and briefly defining the evaluation metrics. You may need to discuss with Gus about what metrics to apply and how to define them for specific tasks. For conditional generation tasks, the conditional input should also be included as an embedded audio. Edit Question Blocks: For each question block, you should 1) update the embedded sample audio; and 2) update the evaluation metrics. If you need more questions on one page, you can copy existing question blocks and then update the audio and metrics in that copy. You are not encouraged to make other modifications or create new questions from scratch just in case certain unwanted features are triggered. Maintain a Consistent Question Order: If the question order on your 1st question page is, say, ours, baseline_1, and baseline_2 (note that questions correspond to models), then this order should be exactly maintained for the rest l-1l-1 question pages. The reason is to facilitate post-survey data analysis. Note that the audio files to be embedded are anonymized and you should check the original names to make sure of the order. Question Randomization: SurveyMonkey offers a useful feature to randomize the question order on the evaluator’s side when they fill out the survey. This would help enhance objectivity. Yet on our side, the questions should be arranged in a consistent order as mentioned before. To enable question randomization, on the upper right of each question page, click Page Logic -\u0026gt; Question randomization. Make sure that all the question blocks are selected and that the instruction block is not selected. Make Use of Page Template: After setting up the 1st question page, you can copy it and use it as a template for the rest pages. In this template, we have already set up all necessary features (e.g., number of questions, metrics, question randomization, etc), and you will only need to update the embedded sample audio.\rEarly Finish with Concluding Page: The best suitable number of question pages in a survey would be 6 ~ 10 pages if not more. That could be a lot and hence we do not expect evaluators to finish all of them. Considering a feasible workload (say, 10 ~ 20 minutes), we can let evaluators conclude the survey early by inserting a concluding page. For example, we can let evaluators just fill the first 3 question pages by inserting a concluding page after that. We then enable page randomization so that all 10 question pages will have a chance to be filled.\nPage Randomization: SurveyMonkey offers a useful feature to randomize pages on the evaluator’s side. This feature is helpful when we only require evaluators to fill, say, the first 3 out of 10 question pages. With page randomization enabled, each page will have a chance to appear earlier on the evaluator’s side and hence be filled, regardless of its absolute page index on our side. To enable page randomization, on the upper right of any question page, click Page Logic -\u0026gt; Page randomization. Make sure that all question pages are selected and that the welcome page, the information page, and the concluding page are not selected.\nSurvey Preview and Distribution\rNote that there is a taskbar, as shown in Figure 5, on top of the survey design panel. During survey creation, DESIGN SURVEY is selected. After survey creation, you should click PREVIEW \u0026amp; SCORE to preview and try out your survey. After preview, you should click COLLECT RESPONSES to distribute your survey. Figure 5. The survey design, preview, and response collection toolbar.\rSurvey Preview: It is always good to preview and try out your survey before distributing it to the public. During the preview, you should go through the complete survey to make sure that the following features are well-functioning:\nNo typo that may cause distraction or confusion; Audio files are correctly embedded and are playable; All questions are compulsory questions: make sure that the evaluator cannot go to the next page if any metric in any question is not rated on the current page; One answer is allowed per row; Page randomization and Question randomization are correctly set (note that during preview, you will view the survey on the evaluator’s side). Make sure that the survey ends at the concluding page.\rSurvey Distribution: Click COLLECT RESPONSES in the taskbar, and then click Send surveys your way. You will see a range of options to distribute your survey, the best of which is probably to Share a survey link. You can send this link to quite a few targets such as:\nWeChat Research: NYU风格伴奏编配 全球音乐技术学生交流群 (Ask Yixiao or Ziyue if you are not in the group) Your WeChat Moment\rMailing list Music X Lab mailing list [Ask Gus to Join] ISMIR mailing list [Click to Join] NIME mailing list [Ask to Join]\rA few things to notice:\nIf you share your survey with a WeChat group, you may consider sending a red packet as an incentive to attract more people to help with the survey. If you decide to share with the ISMIR/NIME mailing list, you should be aware that these are formal channels. Make sure that your survey is well prepared.\rAnalysis with ANOVA\rRecall that our survey has a few pages and each page has mm questions regarding respective models in a consistent order. Each question has kk metrics and each metric will receive numeric ratings from 1 (very low) to 5 (very high) from evaluators. Now suppose our survey receives a total of nn ratings for each metric of each model. This results in a tensor of shape k×m×nk\\times m\\times n. That is, for each metric, we have a rating matrix of shape m×nm\\times n. After calculating the mean rating for each model, we need to further calculate the deviation (i.e., error bar) and assess the statistical significance. Here the deviation comes from two sources: Various standards from different evaluators (inter-subject component); Intrinsic randomness (within-subject component).\rSpecifically, the inter-subject component is an irrelevant term which tends to be very impactful (resulting in a large error bar), so we wish to detach it from the within-subject component. To realize that, we resort to within-subject ANOVA.\nIn this subsection, we discuss hands-on practice in the analysis of the exported survey results. We will not mention much of the mathematical foundations of ANOVA but we refer readers to a few reference textbooks to explore more insights and reasoning behind this art.\nExport Survey Result: Recall the taskbar on top of the survey creation panel. Click ANALYZE RESULTS -\u0026gt; Save As -\u0026gt; Export File -\u0026gt; All responses data. Then you will see a pop-up window asking you to configure the export format. In this window, set FILE FORMAT to CSV (you will need to post-process the exported spreadsheet). Other options should be left default. Then click EXPORT -\u0026gt; Download.\nExported Spreadsheet: The exported result is a spreadsheet with condensed response information. The first 13 columns (A to M) record IP, timestamp, and other information of evaluators (one evaluator per row), which usually can be safely discarded. After column M will be the ratings. Each column records the ratings for one metric of one model on one page. This spreadsheet traverses each page, model, and metric in a depth-first manner. Since evaluators may not complete all pages, some cells may be empty.\nPost-processing: Suppose we have kk metrics and mm models, and our survey receives a total of nn ratings for each metric of each model (note that nn is not the number of evaluators. One evaluator can give several ratings to one model on different survey pages). For post-processing, we need to identify the non-zero entries in the spreadsheet and arrange them into kk tables of shape n×mn\\times m. Specifically, we have one table per metric. This can be done using Python scripts with Pandas. In each table, the row corresponds to rating instances (i.e., nn rows) and the column to models (mm columns). You should know which column corresponds to which model. There should be no empty cell in the processed table. Meanwhile, the string-based ratings [‘very low’, ‘low’, ‘neutral’, ‘high’, ‘very high’] should be converted to integers [1, 2, 3, 4, 5]. We will provide one Python script for this post-processing spreadsheet conversion.\nMean Rating: You can save the kk tables as kk tabs of an Excel spreadsheet. Then the mean rating (for each model) is simply the average of each column.\nWithin-Subject Anova: In each Excel tab, click Data in the taskbar, and then click Data Analysis. Select Anova: Two factors Without Replication and then click OK. A configuration panel will pop up. For input range, you should select the n×mn\\times m table with integer ratings. Set Alpha to 0.05 (by default). In the Output options, select Output Range and then click anywhere in the tab to print out the analysis result. Then click OK. The ANOVA result is a table as shown in Figure 6. Figure 6. An example ANOVA output table.\rInterpreting Anova Results: Statistical Significance: If the (Columns, P-value) cell has a value smaller than 0.05, it means there exists a significant difference among different columns. That is, at least one model has a significantly different rating from the others. However, it does not tell us which exact model it is. We cannot yet say our model is significantly better. Rather, we have to check if the lower error bar of our model is sufficiently higher than the upper error bar of all the baselines. If so, then empiracally we can claim that our proposed model gains a significantly better performance than the baselines. If the p-value is larger than 0.05, or if the error bars have a large intersection, then we can only claim a marginally better performance. Alternatively, in a more rigorous manner, you can conduct ANOVA on binary pairs between Ours and each baseline model. You can claim significantly better performance if p \u0026lt; 0.05/[k×(k-1)/2]0.05\\ /\\ [k\\times (k-1)\\ /\\ 2] for each pair (using the Bonferroni correction when multiple tests are performed). Error Bar: We compute the standard error of mean (SEMSEM) as follows:\rSEM=MSEnSEM\\ =\\ \\sqrt{MSE/n},\nwhere MSEMSE is the value of the (Error, MS) cell in the ANOVA table. nn is the number of ratings. Note that SEMSEM is not the same thing as standard deviation (SD=MSESD=\\sqrt{MSE}). In general, SEM\u0026lt;SDSEM\\ \u0026lt;\\ SD. The SEMSEM indicates how much the mean ratings would likely vary if we were to repeat the experiment. The lower SEMSEM, the more significance there likely to be (note that it is only a correlation, but we cannot simply tell the statistical significance from SEMSEM). By using the MSEMSE computed by ANOVA, SEMSEM is shared by the mean ratings of each model under the same tab (metric).\nReferences on within-subject ANOVA\nRepeated Measures ANOVA. [link] Two Factor ANOVA without Replication. [link] H. Sahai and M. I. Ageel. The analysis of variance: fixed, random and mixed models. Chapter 3, pp. 125-168. Springer Science \u0026amp; Business Media, 2012. [link]\rPlotting Results\rFor each of the kk metrics, we will get mm mean ratings with a shared error bar via within-subject ANOVA. Now we will showcase this result using a bar plot. An example is shown in Figure 7. Here, we have k=3k=3 metrics (Creativity, Naturalness, and Musicality) and m=4m=4 models (GT, Trf., Polyf, and Samp.). The y-axis represents the range of the Likert scale (1 to 5). The x-axis lists the mm models under each of the kk metrics so that we have a straightforward comparison among the models regarding each metric. Figure 7. Mean ratings and error bars for three evaluation metrics and four models.\rTo create such a plot, typically we can apply the Matplotlib package in Python. Specifically:\nimport matplotlib.pyplot as plt Plot mean ratings using plt.bar(⋯\\cdots) Plot error bars using plt.errorbar(⋯\\cdots) Add a legend by plt.legend(⋯\\cdots)\rTo make a visually more elegant plot, you can use Seaborn together with Matplotlib in a combo. You are encouraged to explore the documents and other resources (e.g., Stack Overflow) of both packages to write your own plotting script. We will also provide one Python script.\nConclusion\rConclusion seems to be a re-statement of abstract, but a good conclusion is much more than that. You can incorporate a broader context of research in the writing, linking the study to some other important studies that seems far away but in essence very related from your own unique perspective. You can also point out the limitations and future work here, if there is no individual session for that as per required by the conference.\nAppendix\rThe policy on code, demo, and presentation maintenance\rDemos:\n1. Put only paper-mentioned demos on OVERLEAF E.g.: if you show the score of a generated piece in your paper, you should put the corresponding MIDI files on overleaf under the ./demo folder.\n2. Put all demos on GOOGLE DRIVE under AccArrangement/presentation_demos_upload.\n3. Timeline: Demos are encouraged to be uploaded shortly after paper submission and no later than one week after the camera ready.\nPresentations:\n1. Put all presentations (slides, poster, and video recordings) on GOOGLE DRIVE, one paper per folder. Please include both pdf and source files (pptx, keynote) for slides. Slides 里最好直接嵌入demo里的音频, 尽量不要用record+定时播放slides功能。\n2. Timeline: no later than the formal presentation.\nCodes:\n1. By submission: You should have your own git repository.\n2. By camera ready: The repository should be clean enough, shared to the public, and include a link to the demo page or google drive demo. Try your best to release the trained model and dataset, unless the dataset has a copyright issue. Be sure to include the repository link in the camera ready paper!\n3. IMPORTANT: By formal presentation, the Music X Lab git should link to the project repo\nHow to make a good demo page\rThis might not be a comprehensive guide, but rather several recommendations that I find handy to use.\nGitHub Page. This is probably the most convenient way to deploy a lightweight static website. To deploy a GitHub Page for a paper submitted to a double-blind conference, the following steps are necessary:\nYou should register a new GitHub account with a good account name (e.g., “polyffusion” for the polyffusion paper), and USE A EMAIL ADDRESS THAT PEOPLE CANNOT EASILY GUESS THE OWNER (e.g., a newly registered QQ email is just some random numbers, which is good for this purpose). You probably need to add an ssh key to the new anonymous account, unless you choose to upload/change website code directly in the browser. Create a repository named “{accountname}.github.io”. This is a special repository that enables Page deployment. Clone this repo to your machine. In your cloned repo, you should configure your local git information to match this anonymous GitHub account. This is the key for GitHub to recognize you as the new anonymous account instead of your personal account. Execute these commands in the shell (git shell, cmd, bash, etc.): git config user.name {accountname} git config user.email {acountemail}\rAfter you push your code to the GitHub repo, take a look at the repo panel to make sure your personal account does NOT appear in the “Contributors” list.\rFigure 8. A contributors list that reveals an author’s identity.\rThis is bad! You expose yourself! After the paper is deployed by GitHub, take a look at your webpage at https://{accountname}.github.io.\rFigure 9. The check mark indicating that a GitHub Pages deployment is ready.\rThis tick means the page is ready!\nTemplates for GitHub Pages. Jekyll (which is the static website generator GitHub uses as default) has a good community that has made many templates for different uses. Here are some templates I found suitable for academic purpose. Of course, for a one-page demo website, you can always write HTML/CSS from scratch without any theming. This is up to your liking!\nMinima: an official Jekyll theme that provides a one-page minimal writing environment. (used by Polyffusion and Whole-Song Generation) Al-folio: a popular academic blog theme used by many CMU courses and conferences. Bear: a minimal, text focused Jekyll theme. Browse all the templates if you haven’t found your favorite.\rGoodies for Musicians. Some useful tools for a music demo page!\nhtml_midi_player: 🎹 Play and display MIDI files on the web. They now even have a python launcher! For some advanced styling (centering, bar lines, phrase divisions with background colors), you can take a look at the discussions and the Whole-Song Generation repo.\r","permalink":"https://blog.musicxlab.com/2026/09/16/how-to-write-a-music-ai-paper/","summary":"A practical Music X Lab guide to writing a Music AI paper, from the introduction and model diagrams to experiments, user studies, and demo pages.","title":"How to Write a Music AI Paper"},{"content":" Gus G. Xia · Music X Lab, Machine Learning Department · Mohamed bin Zayed University of Artificial Intelligence\nFull-text web edition of arXiv:2503.21106v4 (14 April 2025). Read the PDF. Reproduced under CC BY 4.0; adapted for blog layout, with the paper’s wording, figures, equations, and references retained. Third-party artwork retains its original attribution.\n“The created universe carries Yin at the back and Yang in front, and through the union of the pervading principle, it reaches harmony.”\n—Laozi, Dao De Jing, Chapter 42\nHuman perception operates across multiple levels of abstraction simultaneously. For example, when we listen to music, we perceive raw acoustic signals at the most basic level, interpret musical scores at a higher level, and recognize even more abstract structures such as chords and forms. Representations at different levels function like distinct languages, each with its own semantics. Yet, these levels of representation influence one another, and we rely on such interactions to better understand and predict the world. Consider a music example again: a trained musician with theoretical knowledge can better anticipate upcoming low-level acoustic events, while an improviser attuned to the nuances of low-level music flow can make more informed decisions about which note to play next.\nTo model such entangled dynamics of hierarchical representations during perception, I propose function alignment as a new theory of mind and intelligence in this position paper. Note that this is not a technical paper in the engineering sense, but a methodological proposal. The theory is implementation-agnostic: in principle, it could be expressed through linear, nonlinear, probabilistic, or energy-based models—though from a modern AI perspective, a neural network implementation of function alignment is likely to offer the most natural blueprint for the human mind.\nAs shown in the graphical model in Figure 1, the 𝒚\\boldsymbol{y}-sequence represents the true dynamics of physical reality, the 𝒙\\boldsymbol{x}-sequence captures the low-level representation in the human mind, and the 𝒛\\boldsymbol{z}-sequence corresponds to a higher-level, more abstract representation. While additional layers may exist above 𝒛\\boldsymbol{z}, we limit our illustration to two levels for clarity. In this framework, the dynamics of 𝒙\\boldsymbol{x} and 𝒛\\boldsymbol{z} are defined as function aligned, characterized by the following three key properties:\nFigure 1: An illustration of function alignment: 𝒙\\boldsymbol{x}-𝒛\\boldsymbol{z} is function aligned, while 𝒙\\boldsymbol{x}-𝒚\\boldsymbol{y} is not. 1. Both x\\boldsymbol{x} and z\\boldsymbol{z} serve as functional descriptions of y\\boldsymbol{y}—they encode different levels of abstraction of the same underlying reality.\n2. Both x\\boldsymbol{x} and z\\boldsymbol{z} are auto-regressive processes, dynamically influencing each other’s predictions. Unlike typical hierarchical models where higher layers passively summarize lower ones, here 𝒙\\boldsymbol{x} and 𝒛\\boldsymbol{z} actively “listen” to each other, forming a bidirectional alignment.\n3. 𝒙\\boldsymbol{x} and z\\boldsymbol{z} are aligned in time, where “time” refers not strictly to physical time but to a generalized logical sequence that governs inference and decision-making.\nThe second property, represented by the diagonal cross-connections, is the most critical distinction between function alignment and conventional hierarchical time-series models. These cross-level, cross-time influences enable mutual adaptation rather than one-way abstraction.\nNotably, 𝒙\\boldsymbol{x} and 𝒚\\boldsymbol{y} are not function-aligned, as indicated by the absence of diagonal connections—𝒙\\boldsymbol{x} merely passively models the underlying reality 𝒚\\boldsymbol{y} without influencing it. For instance, Newtonian mechanics provides a macro-scale description of physical reality based on underlying microscopic particle interactions, but the law F=m​aF=ma does not affect subatomic physics. Another example of a non-function-aligned hierarchical structure can be seen in programming: a Python program is interpreted into lower-level C-executable behavior, but the relationship is unidirectional. Such “concealed” hierarchical structures arise from a lack of function alignment. In contrast, the dynamics of the 𝒙\\boldsymbol{x}-sequence and the 𝒛\\boldsymbol{z}-sequence are deeply entangled and aligned—they are both perceptual representations, interconnected through neural mechanisms, and capable of shaping each other’s evolution.\nNeural Hierarchical Representation and Symbolic Thinking Figure 2: Liberation by Escher. A plausible way to implement the interactions in Figure 1 is through neural networks. As previously mentioned, we illustrate only two levels of representation, but in reality, the hierarchy could extend far beyond 𝒛\\boldsymbol{z}. The vertical bidirectional arrows between different levels of representation (e.g., 𝒙\\boldsymbol{x} and 𝒛\\boldsymbol{z}) can be interpreted as an encoding-decoding mechanism, where higher-level representations are more abstract.\nThe encoding process inevitably leads to a loss of detail. At some sufficiently abstract level, representations reach what Hofstadter, in Gödel, Escher, Bach (Hofstadter, 1999), calls the “crystallization point”, where they transition into symbolic forms and their dynamics become rule-based. This hierarchical transition from abstraction to symbolism is conceptually similar to Escher’s artwork, where homogeneous, meaningless sub-symbolic forms at the bottom gradually evolve into distinct symbols with precise meanings at the top.\nThe focus of this section is not on how symbolic, rule-based processes emerge from sub-symbolic dynamics (this will be explored in the future part of this position paper). Instead, we examine how these two seemingly distinct processes—symbolic reasoning and neural computation—can function together as an organic whole, a relationship we refer to as function alignment.\nAs pointed out in the paper Thinking Like Transformers (Weiss et al., 2021), rule-based programs can be “compiled” into a transformer. In other words, at the “hardware” level, the dynamics of some abstract representation still run on a neural system, while at the “software” level, the internal semantics can be a symbolic, rule-based program.\nNow, consider function alignment among the following three hierarchical levels of representation:\n1. The 𝒙\\boldsymbol{x}-sequence, representing subsymbolic dynamics.\n2. The 𝒛\\boldsymbol{z}-sequence, representing a neural-symbolic system, such as a Transformer that processes embeddings of symbols.\n3. The 𝒛′\\boldsymbol{z}^{\\prime}-sequence, another neural-symbolic system, which shares the same symbol vocabulary as 𝒛\\boldsymbol{z} but whose internal dynamics follow a compiled rule-based program.\nThis framework provides a natural explanation for the unity of intuition and rationality—how humans can both follow rules and break them while simultaneously having an intuitive grasp of the underlying rationales. This flexibility lies in the ability to utilize different inference pathways at various levels of representation.\nFrom Meaning to Explanation: The Bounds of Interpretation The function alignment framework certainly provides more insights—we can now offer precise and rigorous definitions for some deeply abstract and meta-level concepts central to mind and intelligence, including meaning, interpretability, analogy, and rationality.\nMeaning: What does a particular 𝒚\\boldsymbol{y}-sequence (some physical events) mean to us? Its meaning is simply the totality of hierarchical representations it triggers in the mind. For instance, what might a short musical phrase mean to a trained musician? It could evoke subsymbolic-level emotional flow, symbolic-level representations such as melody, harmony, and form, and even highly abstract semantic layers, such as “a lullaby my mother used to sing when I was a child,” which can be verbalized in natural language.\nDepending on context, different levels of representation may carry different weights of significance. For example, in the handwritten message, “Top secret: the password to that old key safe is 9527,” the core meaning clearly lies in the symbolic content—those four digits. But if this same sentence were found in a centuries-old letter, its value might shift toward its stylistic aspects, such as the aesthetic quality of its calligraphy—a subsymbolic representation.\nIdeally, we would communicate by transferring the entirety of meaning—every level of representation that constitutes our internal feelings and thoughts—from one mind to another. But in practice, we cannot (at least not yet). Instead, we encode only selected layers into communicable media (such as text, speech, or music), and it falls to the receiver to interpret the intended meaning based on their own internal function alignments.\nInterpretability, then, is the ability to express one representational layer (including 𝒚\\boldsymbol{y}, the lowest physical layer) in terms of another—whether from subsymbolic to symbolic, symbolic to subsymbolic, or from one symbolic layer to a more abstract one. For example, a parent might interpret their baby’s cry as “I need food” or “I am cold”; a singer might interpret a symbolic score into a rich subsymbolic vocal performance; and in Eastern cultures, the phrase “today’s moon is so beautiful” might be interpreted as the abstract symbolic expression “I love you.” In the narrowest sense, interpretability is the ability to describe something using natural language—our shared symbolic representation layer 𝒛\\boldsymbol{z}.\nFunction alignment suggests that interpretability is intrinsically bounded. Even when layers are aligned, they are not identical, and any interpretation inevitably sacrifices the unique dynamics of the original layer. For instance, interpreting a vocal performance through text will necessarily omit nuanced vocal expressions. Conversely, interpreting a written score through singing may miss explicit structural information such as harmonic progression.\nMuch has been debated (Fodor and Pylyshyn, 1988; Saba, 2022) about whether artificial neural networks or human minds are “interpretable.” Within the function alignment framework, “bounded interpretability” clarifies that we can indeed interpret low-level neural outputs 𝒙\\boldsymbol{x} using high-level symbolic representations 𝒛\\boldsymbol{z}, especially when agents share a common symbolic vocabulary. However, this power is limited: certain dynamics at the 𝒙\\boldsymbol{x}-level cannot be expressed within 𝒛\\boldsymbol{z}, regardless of alignment.\nAnalogy-making, from this perspective, is a form of indirect interpretation. Instead of interpreting a target representation at one layer directly using a source representation from another layer, an analogy uses an alternative target that shares internal representations with the original. For example1:\n• “Life is an adventure” draws structure from purposeful, unpredictable journeys;\n• “Argument is war” relies on shared representations of attack, defense, and victory;\n• “Your love is like moonlight” evokes both an emotional tone and an elegant feeling.\nIn other words, analogy-making is interpretation via style transfer: when two targets are juxtaposed, the human mind instinctively extracts their shared source representation structure. Sometimes, only the second (metaphorical) target appears, and the mind reconstructs the original target through context—such as in:\n• “He is the father of modern physics”—where “founder” is omitted, yet readers infer not only his foundational role but also an emotional framing of responsibility, guidance, and care;\n• “I cannot swallow that idea”—where “comprehend” is replaced by a physical metaphor that conveys not only cognitive resistance but also a visceral sense of stress or discomfort—linking symbolic misunderstanding to subsymbolic embodiment.\nThis “shared source” view of analogy aligns closely with the notion of “conceptual skeletons” proposed in Gödel, Escher, Bach (Hofstadter, 1999) and in Surfaces and Essences (Hofstadter and Sander, 2013). And the function alignment framework extends this view further: the shared representation need not reside solely at an abstract symbolic level. It may span multiple representational layers—embodied, subsymbolic, symbolic—and the more layers involved in alignment, the greater the analogy’s explanatory and expressive power.\nMany Zen koans and classical poetry draw on natural phenomena not to illustrate abstract notions but to evoke subsymbolic felt states across layers. For example, “the body is like the Bodhi tree; the mind is like a bright mirror” encodes not just visual and symbolic meaning, but also an introspective experiential resonance. So, too do the aforementioned analogies of “moonlight” and “swallow.” In these cases, analogies become a conduit for multi-layered profound experience.\nAnalogies are not only powerful but also ubiquitous and sometimes hard to be noticed. Consider the simple act of pointing to an object and saying its name—“this is a chair”—is actually a form of analogy. The visual form of the chair is 𝒚\\boldsymbol{y}, the acoustic form of the chair is 𝒚′\\boldsymbol{y^{\\prime}}, and through function alignment, both 𝒚\\boldsymbol{y} and 𝒚′\\boldsymbol{y^{\\prime}} will trigger the same symbolic-level representation 𝒛=“chair”\\boldsymbol{z}=\\text{``chair''}. Despite its effectiveness, analogy is bounded in the same way as interpretability: we rely on partially aligned internal representations to make sense of each other’s outputs. When alignment breaks down, interpretation collapses into nonsense.\nExplanation: We are now ready to define explanation within the function alignment framework. To explain any target is to reveal the causal representations—those pointing to the target—within the function alignment process, through interpretation and analogy. This aligns with the tradition of causal modeling, where an explanation identifies structural dependencies, often formalized as directed acyclic graphs (DAGs). Indeed, the function alignment framework itself can be viewed as a dynamic causal model, where vertical bidirectional arrows function as real-time feedback, without violating local acyclicity.\nHowever, our view departs from traditional causal modeling in a subtle but crucial way: we regard explanation not just as “passive description”, but as “active prescription.” Most conventional explanations—whether graphs or language—are z-level communicable representations, which are “symbolic shadows” of an underlying structure. To truly understand an explanation is not just to parse its symbolic content, but to align it with subsymbolic flows. E.g., one may study tonal music theory through graphs or language, but its real explanatory power emerges only when it is felt and aligned with auditory and embodied perceptions.\nIn traditional causal theory, a flawed model can simply be updated. However, under function alignment, a misaligned mental model may lead to explanatory closure and even interfere with the system it claims to describe. Consider a musician clinging to diatonic theory while listening to a microtonal piece: the sounds are heard, but perception collapses. Worse, if the musician is performing and imposes this frame on others, the shared experience becomes forcibly re-centered around the wrong model.\nIn this sense, explanation becomes a double-edged tool: when aligned, it reveals; when misaligned, it blinds. The key question is no longer “Is this explanation true?” but rather: “Is the explanation well-aligned? Is it timely, adaptive, and capable of shaping the very process it seeks to explain?” Truth becomes not a static match, but a dynamic resonance—between model and flow, between language and life.\nRationality, therefore, is the capacity to explain using logical or symbolic language. As we have argued, such interpretability is bounded—symbolic reasoning typically occurs at higher-level representational layers, while actual behavior and perception are rooted in lower layers. This view provides a structural and representational grounding for Herbert Simon’s theory of bounded rationality (Simon, 1990). Simon proposed that humans are rational within limits—capable of reasoning, but constrained by cognitive resources. We now have a structural mechanism underlying this description: the limited cognitive capacity for rationality arises from the bounded interpretability inherent in function alignment. When an agent explains a satisficing action using a logic-based framework, it performs an interpretive operation that is, by definition, partial and bounded. People do, in fact, optimize across all levels of representation—including embodied feelings and subsymbolic dynamics—which, in Simon’s terms, is to “satisfice.” However, such optimization cannot be fully explained at the rational, symbolic level.\nIn contrast, complete explanation is possible only within a pure formal system. In such systems, all reasoning and representation occur within a single symbolic layer. There is no information loss about alignment, because no cross-level interpretation is required; explanation collapses into logical operations, and all causes and effects are expressed in the same formal vocabulary. Thus, the ideal of perfect rationality coincides with perfect alignment—achievable only in formal systems.\nAgent-Based Intelligence and Isomorphic Alignment We now formalize function alignment within a mathematical framework. Again, though many layers of representation may exist, here we only formulate two layers for simplicity. Without losing generality, the function alignment between 𝒙\\boldsymbol{x} and 𝒛\\boldsymbol{z} can be modeled as a coupled system:\nxt\\displaystyle x_{t} =f⁡(𝒙\u0026lt;t,𝒛\u0026lt;t,zt)\\displaystyle=f(\\boldsymbol{x}_{\u0026lt;t},\\boldsymbol{z}_{\u0026lt;t},z_{t}) (1) zt\\displaystyle z_{t} =g⁡(𝒛\u0026lt;t,𝒙\u0026lt;t,xt)\\displaystyle=g(\\boldsymbol{z}_{\u0026lt;t},\\boldsymbol{x}_{\u0026lt;t},x_{t}) (2) Here, ff and gg can potentially be any form of function representing the forward dynamics. The entanglement arises because each update depends on the other’s past and current state. Specifically, if we want to distinguish between the cross-time mapping and the coupling effect at each time point, the system can be alternatively formulated as:\nxt\\displaystyle x_{t} =f⁡(𝒙\u0026lt;t,𝒛\u0026lt;t)\\displaystyle=f(\\boldsymbol{x}_{\u0026lt;t},\\boldsymbol{z}_{\u0026lt;t}) (3) zt\\displaystyle z_{t} =g⁡(𝒛\u0026lt;t,𝒙\u0026lt;t)\\displaystyle=g(\\boldsymbol{z}_{\u0026lt;t},\\boldsymbol{x}_{\u0026lt;t}) (4) subject toE⁡(xt,zt)\u0026lt;δ\\displaystyle\\text{subject to}\\quad E(x_{t},z_{t})\u0026lt;\\delta (5) where EE can be regarded as a soft energy function enforcing structural compatibility and semantic isomorphism, rather than perfect reconstruction. In practice, abstract representations zz almost always lose some information of xx, so the alignment is inherently asymmetric. That is, while xx may fully determine zz, the reverse is not generally true. Nonetheless, by maintaining E⁡(xt,zt)E(x_{t},z_{t}) within a bounded tolerance δ\\delta, we preserve semantic coherence across layers. We can therefore define a loss term:\nℒFA=α1⋅dist​(xt,f⁡(𝒙\u0026lt;t,𝒛\u0026lt;t))+α2⋅dist​(zt,g⁡(𝒛\u0026lt;t,𝒙\u0026lt;t))+β⋅E⁡(xt,zt)\\mathcal{L}_{\\text{FA}}=\\alpha_{1}\\cdot\\text{dist}(x_{t},f(\\boldsymbol{x}_{\u0026lt;t},\\boldsymbol{z}_{\u0026lt;t}))+\\alpha_{2}\\cdot\\text{dist}(z_{t},g(\\boldsymbol{z}_{\u0026lt;t},\\boldsymbol{x}_{\u0026lt;t}))+\\beta\\cdot E(x_{t},z_{t}) (6) Isomorphic alignment among agents: We now extend the function alignment framework to multi-agent scenarios. Consider two individual autoregressive processes (agents), each evolving with the same functional form:\n𝒙t\\displaystyle\\boldsymbol{x}_{t} =hθx​(𝒙\u0026lt;t)\\displaystyle=h_{\\theta_{x}}(\\boldsymbol{x}_{\u0026lt;t}) (7) 𝒛t\\displaystyle\\boldsymbol{z}_{t} =hθz​(𝒛\u0026lt;t)\\displaystyle=h_{\\theta_{z}}(\\boldsymbol{z}_{\u0026lt;t}) (8) Here, hθxh_{\\theta_{x}} and hθzh_{\\theta_{z}} represent the internal dynamics of each agent. If these processes describe different aspects of a shared underlying phenomenon—such as performance-level and harmony-level dynamics in music—we may apply a post-hoc function alignment to couple them into a unified system. This converts the initially independent agents into a jointly evolving structure, as formalized earlier in Equations (1–5).\nThe notion of isomorphic alignment refers to the condition where the aligned system can be written as:\n𝒘t\\displaystyle\\boldsymbol{w}_{t} =[𝒙t,𝒛t]\\displaystyle=[\\boldsymbol{x}_{t},\\boldsymbol{z}_{t}] (9) 𝒘t\\displaystyle\\boldsymbol{w}_{t} =hθw​(𝒘\u0026lt;t)\\displaystyle=h_{\\theta_{w}}(\\boldsymbol{w}_{\u0026lt;t}) (10) where [𝒙t,𝒛t][\\boldsymbol{x}_{t},\\boldsymbol{z}_{t}] denotes the concatenation of the two states at time tt, and θw\\theta_{w} represents the unified dynamics after alignment.\nIsomorphic alignment occurs when the unified dynamics hθwh_{\\theta_{w}} preserves the same functional form as the original agents hθxh_{\\theta_{x}} and hθzh_{\\theta_{z}}—differing only in parameterization. In this sense, the aligned system is structurally isomorphic to its components. This mirrors the idea of conjugacy in Bayesian inference, where the prior and posterior belong to the same distributional family. However, in this case, the isomorphism results not from within-model temporal updating, but from structural alignment across multiple generative processes.\nLinear case: To better illustrate the idea of isomorphic alignment, let us consider a simple linear case. If we regard Figure 1 as a first-order linear dynamical system, horizontal and diagonal arrows across time steps represent transition matrices, while vertical arrows within a single time step reflect instantaneous coupling, captured via covariance matrices. The aligned systems can be formulated as:\nxt\\displaystyle x_{t} =A​xt−1+C​zt\\displaystyle=Ax_{t-1}+Cz_{t} (11) zt\\displaystyle z_{t} =B​zt−1+D​xt\\displaystyle=Bz_{t-1}+Dx_{t} (12) We use degrees of freedom (DoF) to characterize the structure. Suppose that before alignment, 𝒙={x1,x2,…},xi∈ℝn\\boldsymbol{x}=\\{{x}_{1},{x}_{2},\\dots\\},{x}_{i}\\in\\mathbb{R}^{n} and 𝒛={z1,z2,…},zi∈ℝm\\boldsymbol{z}=\\{{z}_{1},{z}_{2},\\dots\\},{z}_{i}\\in\\mathbb{R}^{m} are two independent linear dynamical processes, each governed by:\nxt\\displaystyle x_{t} =A′​xt−1\\displaystyle=A^{\\prime}x_{t-1} (13) zt\\displaystyle z_{t} =B′​zt−1\\displaystyle=B^{\\prime}z_{t-1} (14) Their degrees of freedom are:\n• DOF𝒙\\texttt{DOF}_{\\boldsymbol{x}}: n2n^{2} for transition + n⁡(n+1)/2n(n+1)/2 for initial covariance\n• DOF𝒛\\texttt{DOF}_{\\boldsymbol{z}}: m2m^{2} for transition + m⁡(m+1)/2m(m+1)/2 for initial covariance\nNow consider a combined process wt=[xt,zt]∈ℝn+m{w}_{t}=[{x}_{t},{z}_{t}]\\in\\mathbb{R}^{n+m} modeled as a first-order linear system, whose degrees of freedom are:\nDOF𝒘=(n+m)2+(n+m)​(n+m+1)2\\texttt{DOF}_{\\boldsymbol{w}}=(n+m)^{2}+\\frac{(n+m)(n+m+1)}{2} Function alignment imposes three types of structural connections: (1) horizontal autoregression, (2) vertical intra-step coupling, and (3) diagonal cross-step feedback.\n• Horizontal transitions: 𝒙t→𝒙t+1\\boldsymbol{x}_{t}\\rightarrow\\boldsymbol{x}_{t+1} and 𝒛t→𝒛t+1\\boldsymbol{z}_{t}\\rightarrow\\boldsymbol{z}_{t+1}, contributing DOF𝒙+DOF𝒛\\texttt{DOF}_{\\boldsymbol{x}}+\\texttt{DOF}_{\\boldsymbol{z}}\n• Vertical coupling (within-time): n​mnm DoF\n• Diagonal alignment (cross-time interaction): additional 2​n​m2nm DoF\nPutting all these together, we observe that the function-aligned system recovers the full degrees of freedom:\nTotal function-aligned DoF=DOF𝒙+DOF𝒛+3​n​m=DOF𝒘\\texttt{Total function-aligned DoF}=\\texttt{DOF}_{\\boldsymbol{x}}+\\texttt{DOF}_{\\boldsymbol{z}}+3nm=\\texttt{DOF}_{\\boldsymbol{w}} This matches exactly the DoF of the unified system 𝒘\\boldsymbol{w}, showing that full function alignment renders two interacting representational processes structurally isomorphic to a single unified linear dynamical agent. If any of the alignment arrows—horizontal, vertical, or diagonal—is missing, the system loses structural completeness and can no longer be regarded as fully integrated. This confirms that full function alignment achieves structural isomorphism. Q.E.D.\nNonlinear and long-dependency generalization: In nonlinear or higher-order systems, degrees of freedom may no longer be analytically tractable. In such cases, we may resort to alternative structure criteria, such as symmetry, gradient continuity, or variational closure.\nMoreover, when alignment extends across longer time horizons or spans multiple representational agents, strict isomorphic alignment—i.e., full connectivity where everything connects to everything—may not be necessary and achievable. For instance, in the brain, inter-hemispheric links are often far sparser than intra-regional connections. What matters is not universal connectivity, but the presence of functional bridges across abstraction layers, enabled directly or indirectly via the “diagonal arrows” in our framework. We refer to such generalized cases as relaxed isomorphic alignment.\nUnder this broader view, the principle remains: alignment must preserve the system’s capacity for dynamic co-adaptation across time and abstraction. When cross-layer pathways, especially diagonal arrows, are broken, the system fragments, introducing adaptation bottlenecks or isolating representational islands. Alignment integrity thus becomes a necessary condition for unified learning and intelligent behavior.\nImplication: This structure demonstrates that function alignment is not only a perceptual architecture but also a structural condition for agent unification. Different components of the mind—or the brain—may specialize in distinct representational dynamics, yet through function alignment, they can operate as a unified cognitive agent. More generally, simple intelligent agents at the micro-level can be recursively aligned to form macro-level agents, while preserving relaxed isomorphic structure. This recursive principle offers a scalable blueprint for constructing life-like intelligence, in both artificial and biological systems.\nBeyond Modeling: Insights for Psychology, Philosophy, and Zen Function alignment offers not only a computational theory of mind but also a lens through which we can deepen our understanding of ourselves. Across psychology, philosophy, and contemplative traditions like Zen, we encounter a core duality: the mind has two systems—one intuitive, one analytical. The challenge has always been not to choose one over the other, but to integrate them in a way that leads to wisdom and harmony.\nIn cognitive science, the duality is known as System 1 and System 2 (Kahneman, 2011); in AI, Mode 1 and Mode 2 (e.g., Hierarchical JEPA (LeCun, 2022)); in traditional philosophies, Yin (or feminine) and Yang (or masculine); in Zen and the Art of Motorcycle Maintenance (Pirsig, 1974), romantic and classical understanding. A harmonic integration of these two modes is not merely an intellectual task, but a living art and experience.\nSplit brains and minds: Psychological studies, especially on split-brain patients, offer striking evidence of function alignment and agent-based intelligence, even at the physical brain level. In one well-known setup (Gazzaniga, 2012): when light is shown to the left visual field, the right hemisphere perceives it, but the left hemisphere (responsible for language) cannot report it via natural language at the 𝒛\\boldsymbol{z} level. Still, the subject, at the behavior 𝒙\\boldsymbol{x}-level can press the button to indicate “seeing the light.” This suggests that each hemisphere can function as an agent, but only the left brain has access to symbolic language expression. Without aligned input from perception, reasoning is blind.\nKoan of “mirror becomes the mask”: If split-brain patients show what happens when symbolic processing is disconnected, our daily struggle is often the opposite: an over-identification with the symbolic layer, just as a performer who overemphasizes music theory may lose touch with the underlying flow of music itself. All minds are function-aligned but to varying degrees. A sharp rational mind is smart, but only a deeply aligned mind is wise. My favorite story about this second kind of misalignment comes from Sadhguru:\nA man, having promised to quit drinking, again returned home late and drunk. On the way, he scratched his face on a branch. Not wanting his wife to know, he quietly applied bandages in the bathroom and sneaked into bed without a sound. The next morning, his wife slapped him: “You drank again!” He was shocked: “How did you know?” She pointed at the mirror. “The bandages were all over it!”\nThis is indeed a great metaphor of misalignment: 𝒛\\boldsymbol{z} is the mirror of 𝒙\\boldsymbol{x}, and when aligned, 𝒛\\boldsymbol{z} should reflect and serve 𝒙\\boldsymbol{x}. But when we are not conscious enough, misalignment arises, and 𝒛\\boldsymbol{z} becomes ego and a distortion we confuse with truth. Despite smartness, we may act for value detached from experience, argue logic without feeling, and become minds that speak without seeing.\nTowards integration: The practice of Zen is, in essence, training in experiencing the truth and a deep function alignment. Enlightenment is not knowing more intellectually but reconnecting to the raw flow of perception x\\boldsymbol{x} that is as close to 𝒚\\boldsymbol{y} as possible, followed by a non-egoic reintroduction of symbolic framing and deep function alignment. The Zen master’s sudden shout, the nonsensical koan, and the silent meditation practice of “just sit” are designed not to teach knowledge about truth, but to interrupt overactive 𝒛\\boldsymbol{z}-level symbolic thinking and restore access to the unfiltered experience.\nAs a well-known Zen analogy goes:\nFirst, mountains are mountains. Then, mountains are not mountains. Finally, mountains are once again mountains.\nThese three stages beautifully trace the arc of function alignment:\n• The first stage reflects a symbolic-dominant misalignment—where socially conditioned values obscure unfiltered experience.\n• The second stage marks disorientation from symbols, allowing one to contact direct perceptual truth.\n• The third stage represents deep re-integration, where symbols are rebuilt—fresh, aligned, and transparent.\nIn brief, this is not a rejection of rationality but a transcendental path where symbols are no longer a substitute for truth but its humble servant. It is the arc of function alignment in full: from confusion, to liberation, to a deep ease of being—a mind no longer at war with itself but aligned in harmony.\nConclusion and Outlook In conclusion, this paper proposes function alignment as a theory of mind that is not only intuitively compelling, but structurally grounded. Unlike many existing accounts of cognition that rely on pre-theoretical concepts or loosely specified metaphors, this framework makes explicit how meaning, interpretation, and analogy emerge from concrete relationships among representational layers. Each concept introduced, whether symbolic reasoning or feeling-level resonance, corresponds to a definable pattern of interaction within the alignment model. In this sense, function alignment forms a coherent representational language, capable not only of modeling minds, but of serving as a blueprint for building them.\nMoreover, this theory does something unique: it explains explanation, and it gives meaning to the very concept of meaning. It shows why interpretation is inherently bounded—meaning is layered, and it must be aligned to be understood. This perspective offers a unified theoretical grounding for many fragments of mind science, such as bounded rationality, symbol grounding, and analogy-making. Once treated as isolated phenomena and concepts, they now emerge as structural consequences of representational dynamics.\nFurthermore, function alignment bridges domains too often kept apart. It is not built upon any philosophy or belief system. Rather, philosophies, psychologies, and even contemplative systems like Zen may find themselves reconstructible within it. If symbolic thought is to serve experience rather than obscure it, then we need not only more knowledge, but better alignment. Function alignment offers a shared foundation where logic and perception, explanation and intuition, can meet—not in conflict, but in coherence and harmony.\nFinally, this first part has focused on the foundational aspects of the function alignment framework. It leaves several critical aspects for future development: the nature of action, interaction with environments and other agents, the emergence of symbolic language, and the question of how such alignment mechanisms might be realized via AI systems. As a foundational entry, this work sets the stage for these exciting developments to come in a larger program.\nAcknowledgments and Disclosure of FundingI would like to thank Roger Dannenberg, Yann LeCun, He He, and Maigo Wang for their insightful discussions on hierarchical modeling. Also, I would like to thank Chao Shi and Rongfeng Li for the discussion on the mathematical formulation of function alignment. I am grateful to Liwei Lin, Junyan Jiang, Yuxuan Wu, Ziyu Wang, and Daniel Chin for their contributions to the initial development of function alignment, as well as their support with pilot studies, experiments, and paper formatting. References Fodor and Pylyshyn (1988) Jerry A Fodor and Zenon W Pylyshyn. Connectionism and cognitive architecture: A critical analysis. Cognition, 28(1-2):3–71, 1988. Gazzaniga (2012) Michael Gazzaniga. Who’s in Charge?: Free Will and the Science of the Brain. Hachette UK, 2012. Hofstadter (1999) Douglas R Hofstadter. Gödel, Escher, Bach: an eternal golden braid. Basic books, 1999. Hofstadter and Sander (2013) Douglas R Hofstadter and Emmanuel Sander. Surfaces and essences: Analogy as the fuel and fire of thinking. Basic books, 2013. Kahneman (2011) Daniel Kahneman. Thinking, fast and slow. macmillan, 2011. LeCun (2022) Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review, 62(1):1–62, 2022. Pirsig (1974) Robert M. Pirsig. Zen and the Art of Motorcycle Maintenance. William Morrow \u0026amp; Company, New York, 1974. Saba (2022) Walid S. Saba. New research vindicates fodor and pylyshyn: No explainable ai without structured semantics, 2022. URL https://cacm.acm.org/blogcacm/new-research-vindicates-fodor-and-pylyshyn-no-explainable-ai-without-structured-semantics/. Simon (1990) Herbert A Simon. Bounded rationality. Utility and probability, pages 15–18, 1990. Weiss et al. (2021) Gail Weiss, Yoav Goldberg, and Eran Yahav. Thinking like transformers. In International Conference on Machine Learning, pages 11080–11090. PMLR, 2021. 1. Several examples here draw from the book Metaphors We Live By (Lakoff \u0026amp; Johnson, 2008) ↩\n","permalink":"https://blog.musicxlab.com/2026/09/16/function-alignment/","summary":"Gus G. Xia introduces function alignment, a theory of mind and intelligence connecting hierarchical representations, meaning, interpretation, and analogy.","title":"Function Alignment: A New Theory of Mind and Intelligence, Part I: Foundations"},{"content":"This blog post introduces the “hidden methodology” behind many core works on deep music generation in Music X Lab. We hope it can help you better distill the main idea from technical details. – Gus, Dec 2023\nBackground of automated music generation\rFor a long time, music generation has been modeled as a “sequence prediction” problem. As in estimating the stock price, the task of music generation is to predict a music sequence based on some contexts, say, to predict the melody given the underlying chords, or to predict the upcoming notes based on existing ones. However, we observed an intrinsic defect of such generation via prediction approach – in any given contexts, there are many ways/directions to develop the music, so which one shall the model follow?\nUnfortunately, most data-driven approaches would not choose any particular direction, but somewhat an averaged version of all possibilities. This often leads to a mediocre music without a clear structure, and that is why most AI-made music still lacks the exploration and dynamic in genuinely creative works. Even worse, there is little room to interact with the black-box models except for sampling from the learned distribution repeatedly. From a musician’s perspective, composing a piece is certainly very different from predicting a stock’s price. What we need is not a “correct” estimate, but a creative choice. Moreover, we wish to interpret and control such choices as a way to extend our own musical expression.\nA different philosophy – generation via analogy-making\rA solution to the problems above lies in analogy-making — or, in a modern terms, music style transfer1. The underlying idea is that most creation is not entirely new but a recombination of existing features (representations). A simple example (in the visual domain) is a “red rabbit” – a rare thing in nature but almost everyone can create an image in their mind by applying “red” (a concept) to “rabbit” (another concept). Similarly, if a model can learn useful music concepts, we can generate new music by making an analogy, e.g., what if piece A was re-composed/re-arranged using a different feature (e.g. chord, texture, form, etc.).\nAssuming we have M music features and N pieces, each piece with a unique value for each feature. Through analogy-making, we can in theory create (NM - N) new pieces by recombining the M features of the N pieces. That is a huge gain! From the lens of causality, such “generation via analogy-making” belongs to “counterfactual reasoning”, i.e., to imagine something non-existent based on observations.\nComing back to deep learning\rThe good news is that deep representation learning models are good feature learners, and we already see pioneer works in neural image style transfers. Now, it is our mission to work out solutions to: 1) learn abstract representations of music, 2) disentangle the representation into human-interpretable parts (concepts) using various inductive biases, and 3) control the generation by manipulating (sampling, interpolating, recombining, etc.) different music concepts.\nFigure 1. The generation via analogy-making methodology: first do interpretable concepts learning via representation disentanglement, and then do controllable music generation.\rThe graph above is referred to as the “trinity of interpretable representation learning” in Music X Lab. Many of our core works follow such methodology — disentangling “melody contour” and “rhythmic pattern” of monophonic melody using EC2VAE2, disentangling “chords” and “texture” of piano score using Poly-dis3, learning “piano texture” from both score and audio in A2S4, learning the “orchestration function” of multi-track polyphonic pieces in Q\u0026amp;A5, and disentangling “pitch” and “timbre” using a unified model for zero-shot source separation, transcription and synthesis6, etc. Several more recent works, e.g. whole-song generation7 and AccoMontage 38, even applied interpretable representations in a hierarchical setup.\nOn unifying sequence generation and representation disentanglement\rAn even better news is that “generation via prediction” and “generation via analogy-making” do not necessarily conflict; rather, we can unify these two methodologies — a straightforward approach we often use is to throw the learned disentangled representations into whatever generative model (autoregressive, diffusion, masked language model, etc.). In other words, let the learned disentangled representations be the “language” of the generative model. The underlying idea is that interpretable concepts shall be useful features for (downstream) generation tasks.\nWe can either just use these representations as “controls”, or even better, also ask the models to predict these disentangled representations. Such ”representation-enhanced generation” has two major benefits. First, the (entire) generation process becomes more interpretable and controllable. Second, the generation results are usually much more coherent, as the model is now producing music feature-by-feature, measure-by-measure, and sometimes even phrase-by-phrase rather than naive note-by-note or midi-event by midi-event. Here are some examples of controllable generation using disentangled representation: 1) piano arrangement by predicting texture based on a lead sheet using polydis3, 2) a more flexible control of chord and texture using Polyfussion9, 3) flexible music inpainting using long-term (4 bar to 8 bar) melody and rhythm representations10 11, and 4) automatic orchestration generation based on “orchestration function representation” and chords using AccoMontage 38, and 5) whole-song generation7, a model that applies different feature controls on different levels of compositional hierarchy.\nIn the end, feature extraction and sequence modeling should go hand in hand. Concretely, we recently showed that predictive modeling can \"return the favor\" and help representation disentanglement — to use sequence prediction as an inductive bias for more interpretable disentanglement. The underlying idea is that a good representation should help us better predict the future. We will leave it as a rough idea in this blog, but you can check out SPS12 and also a line of research on self-supervised learning in the last suggested follow-up reading. Suggested follow-up reading\rSymbolic Music Representation learning and disentanglement: from monophonic to polyphonic (coming soon)\rMultimodal Music representation learning：on connecting MIR with Music Generation (coming soon)\rHierarchical disentanglement: towards effective and efficient sequence modeling (coming soon)\rWhat is proper inductive bias for self-supervised learning? (coming soon)\rReference\rG. Xia, S.Dai. \"Music Style Transfer: A Position Paper,\" 6th International Workshop on Musical Metacreation, Spain, June 2018. ↩R. Yang, D. Wang, Z. Wang, T. Chen, J. Jiang and G. Xia. \"Deep Music Analogy Via Latent Representation Disentanglement,\" in Proc. 20th International Society for Music Information Retrieval Conference, Delft, Nov 2019. ↩Z. Wang, D. Wang, Y. Zhang, G. Xia, \"Learning Interpretable Representation for Controllable Polyphonic Music Generation,\" in Proc. 21st International Society for Music Information Retrieval Conference, Montréal, Oct 2020. ↩Z. Wang, D. Xu, G. Xia, Y. Shan, \"Audio-to-symbolic Arrangement via Cross-modal Music Representation Learning,\" in Proc. 47th International Conference on Acoustics, Speech and Signal Processing, Singapore \u0026amp; Online, May 2022. ↩J. Zhao, G. Xia, Y. Wang. \"Q\u0026amp;A: Query-Based Representation Learning for Multi-Track Symbolic Music re-Arrangement,\" in Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, Macao, August 2023. ↩L. Lin, Q. Kong, J. Jiang, G. Xia. \"A unified model for zero-shot music source separation, transcription and synthesis,\" in Proc. 22nd International Society for Music Information Retrieval Conference, Online, Nov 2021. ↩Under review. ↩J. Zhao, G. Xia, Y. Wang. \"AccoMontage-3: Full-Band Accompaniment Arrangement via Sequential Style Transfer and Multi-Track Function Prior.\" arXiv preprint arXiv:2310.16334, 2023. ↩L. Min, J. Jiang, G. Xia, J. Zhao. \"Polyffusion: A Diffusion Model for Polyphonic Score Generation With Internal and External Controls,\"in Proc. 24th International Society for Music Information Retrieval Conference, Italy, Nov 2023. ↩S. Wei, G. Xia, W. Gao, L. Lin, Y. Zhang, \"Music Phrase Inpainting Using Long-term Representation and Contrastive Loss,\" in Proc. 47th International Conference on Acoustics, Speech and Signal Processing, Singapore \u0026amp; Online, May 2022. ↩S. Wei, Z. Wang, W. Gao, G. Xia, \"Controllable Music Inpainting With Mixed-level and Disentangled Representation,\" in Proc. 48th International Conference on Acoustics, Speech and Signal Processing, Rhodes Island, Greece, June 2023. ↩X. Liu, D. Chin, Y. Huang, G. Xia, \"Learning Interpretable Low-dimensional Representation via Physical Symmetry,\" in Advances in Neural Information Processing Systems, New Orleans, US, Dec 2023. ↩\r","permalink":"https://blog.musicxlab.com/2024/01/04/analogy-making-vs-prediction-a-debate-on-the-philosophy-of-automated-music-generation/","summary":"A perspective on interpretable music representations, analogy-making, and controllable music generation.","title":"Analogy-making vs. prediction: a debate on the philosophy of automated music generation"},{"content":"“Music is the one incorporeal entrance into the higher world of knowledge, which comprehends mankind but which mankind cannot comprehend.” — Ludwig van Beethoven\nOur Vision\rMusic manifests the most complex and subtle forms of creation of human minds. The composition process of music is almost free from the limitations of the physical world, fully leveraging imagination and creative intelligence. For this reason, Beethoven refers to music as “incorporeal” and refers to creative intelligence, the magnificent faculty accessible to humans, as the “higher world of knowledge.” From an AI perspective, the best way to uncover the mystery of creative intelligence is to realize it via computational efforts — to conceive being from void, to develop many from one, to construct whole from parts, to make analogy among seemingly distant scenarios, and music is a perfect subject of this endeavor.\nOn the other hand, the appreciation of music involves profound subjective experiences, especially aesthetic perception, which goes beyond utilities and cost functions that can be easily measured by static equations. The inner feelings, the dynamic notion of beauty, taste, good, and the self “I”, are what make ourselves “mankind” and what machines are yet to encompass. Hence, teaching machines to perceive structures, expressions and representations of music and to appreciate music with a taste is essentially to incorporate humanity into intelligent agents.\nOur Teams and Projects\rOn the one hand, we are musicians, and we are curious about how indeed gifted musicians understand and create music. On the other hand, we are computer scientists and we believe that the best way to uncover the mystery of musicianship is to re-create it via computational efforts in a human-centered way. That is why we have been developing various intelligent systems that can help people better create, perform, and learn music.\nThree of our most representative projects are: 1) deep music representation learning and style transfer, 2) human-computer interactive performance, and 3) computer-aided multimodal music tutoring. The first one is a new field (as well as a hot topic since 2018) that lies in the core of deep learning, relating to many other domains such as NLP and CV, and we were lucky to be one of the teams who laid the groundwork. The other two projects both have great practical value, and at the same time call for truly interdisciplinary efforts (music practice, educational psychology, hardware \u0026amp; interface design, real-time systems, etc.). We were proud to help promote them as the host of NIME2021 via the conference theme “learning to play, playing to learn.”\nIn a big picture, these three projects aim to seamlessly “close the loop” for the next-generation AI-extended music experience, in which I envision a workflow as follows: i) a user first sketches a short melody segment or a motif, ii) a music-generation agent extends it to a full song with accompaniment, while the user is free to transfer the style of any part of the piece, iii) a tutoring agent helps the user to learn to play the piece via interactive visual and haptic feedback, and finally, iv) the user and an accompaniment agent perform the performance on (a virtual) stage.\nSuggested follow-up reading\rAnalogy-making vs. prediction: a debate on the philosophy of automated music generation\n","permalink":"https://blog.musicxlab.com/2022/08/18/the-big-picture-of-music-x-labs-research/","summary":"Gus Xia on creative intelligence, aesthetic perception, and the research vision of Music X Lab.","title":"The big picture of Music X Lab’s research"}]