AI for spritesheet generation?
Friday, August 16, 2024 - 07:56
I tried several AI text-to-image tools, but as of now none is yet able to cope with spritesheets, they create random, inconsistent frames.
Does it exist any specific AI tool trained on spritesheets?
Hi Jumpjack, not that I know and those inconsistencies are still inherent to a.i., your only chance is to process every single sprite one by one in my opinion, otherwise you will get different faces and everything will be inconsistent.
Even if you try one by one, if you want consistent faces you have more chances if the base image is well defined and your prompts are consistent. In other words: I never managed to get any consistent characters /faces with low resolution pixelated faces / sprites (I'm not saying is not possible, just I never managed to do it).
So that I explain myself better: to achieve consistency you don't let the A.I. draw the sprites, you draw them yourself and then change the style or improve it with the help of A.I., sometimes you add a human process at the end to correct imperfections. A.I. is in the middle of the process, not the only process and A.I. improves your work, it does not replace it.
The tool and model you are looking for that process hundreds of sprites with perfect consistency with a single prompt does not exist, maybe in a few years, but not yet.
I hope this help.
This may help people modulate their expectations in regards to A.I.
A.I. ≠ Replace your whole workflow with a single prompt
yah if you are comfortable with the ethical implications of AI, then it can be a useful tool for concept art. as far as generating actual pixel art, you just aren't going to get anything useful out of the box. inconsistent generations are the norm.
but if you are looking for a starting point, to refine, improve, and animate yourself, it can be done.
if you are looking for something genuinely unique, ai can't do that. ai algorithms are trained on existing art (99% of which is stolen by web-scraping, but different topic) and can really only imitate what is in the dataset combined with language parsing in the prompt. having a start image and an understanding of prompt engineering is helpful, and understanding that you are going to be doing dozens if not hundreds of generations and iterations to get what you want.
glitchart's statement of " A.I. ≠ Replace your whole workfload with a single prompt " is 100% on point. 150%. 200%.
real art techniques will always be supreme, and ai generations are only truly useful as part of a workflow for human artists.
In the meantime I found this amazing AI experiment by META, called "Animated drawings"! https://sketch.metademolab.com/canvas
Unfortunately it currently can only export in an useless MP4 format and cannot import custom BVH motion capture files (there are hundreds of free BVH files here: https://sites.google.com/a/cgspeed.com/cgspeed/motion-capture), but maybe the python version can? I don't know how to use it.
I also think that "Universal LPC Spritesheet Generator" associated to an AI could give amazing results... but unfortunately I can't do also this.
But I'll give a try to use ChatGPT to:
- Convert the MP4 fron Animated Drawings into a spritesheet
- Modify Universal LPC Spritesheet Generator to allow uploading just the 4 needed faces, which must be overlayed properly to all characters.
Ok ChatGPT-4o just abandoned me due to overwork ;-) , but if anybody wants to continue...
https://github.com/jumpjack/MotionCapSpritesheet/blob/main/mp4-to-sprite...
I'm using creart and wombo dream, if I make a lot of requests I can get something like that.
Two times I got something that could be used as a sprite-sheet, each of these 2 times with 2 images. 2 images is the minimum to get something that could be animated (animated-gif often had only 2 images). This is not a professionnal quality, but we can be happy with something like that for a free and/or open-source game or demo.
Currently im training lora to create tileset and so far its going good, I was using this as ispiration. But decided to dich entire tileset cuz it was more 3D-ish than flat 2D.
I did draw my own inital tiles and used tilesetter to create all variations of tiles, i decided to make first minimalistic tilest as v1. In version v2 i took all tiles that are repeating and set em as single instance of it, you could rotate those and saving some space for future.
For V3 what i wanna create is possible 1 click entire tileset with repaeting tiles that has variations like: Cracks, flowers, snow, dirt, vines etc, depending on what type of tiles it is. Currently building "blueprint"
So based on what i did learn is that you can create animation generator training your own lora/finetuned model, but you need to create same animation in same format with same heights and width, otherwise lora/model won't know what is where, it will get confused and starts to throw up and make something that is NOT a spritesheet.
To A.I. to learn something you feed it with a objects of diffrent rotations, colors, like apple, but is it fully red?, is it rotten?, has holes?, etc. Each of states has to be seperate as dataset (text file that has words like "apple worm, apple, red, organic" etc. So whatever you want to make him learn you can, it requres some nice GPU and more time than anything.
If you want to generate animations with AI you can. You would probably spend 90% of time just in setup dataset before training and about 10% try and error. But there is always smart way.
I use trigger words and lora (smaller in size), and if i can train my own model hopefully one day.
For example, to create data set you need to first figure out what you want, Enitre set? What comes in that tileset, walking? Running? Jumping? Slashing? Dying? Revivng? Casting Spells,....list can be pretty big. If you want to make such spritesheet it would be really big and kinda not possible as of right now.
You can do spritesheet of 1536x1536 as highest resolution as base. Im using 1024px SDXL model as base for lora.
So now you have canvas to work with, pick up to some space that your PC can generate and stick with that for beggining. To Setup animation, lets use walking as example. In 1024 x 1024 px we can put somewhat 128 x 256, Now remove 16 for spacing, and you have 112x 496 pixels to work for 1 frame. Since we have limited space you can have 64 frames but all frames dose not have to be used. Or you can generate 8 animations for each row and 8 collums for each animation or 4 animations and more frames. You get point.
Now lets say you created your first "blueprint". There is now more ways to create rest of variations. There is hard and time consuming, and dirty and easy but maybe less quality.
Hard way
First one is create all variations of animations as blueprints like : walking, dying, springing, jumping...whatever you wish (it takes time). And use that as transparent to draw other characters in similar animations, so that you can have 10-15 datasets of diffrenct characters in those animations.
Easy way
Use created bluepring and use A.I. to generate similar animations with diffrent characters, and use those genearted as dataset. Use controlnet for pose, canny for cohirance. *Canny will try to draw similar guy*, recommend to look at google that you out of controlnets. Depth can be good to see what is in front and what is in back...usefull for legs/arms.
Note
That once creating "blueprint" you can create other effects, like creating naked dude/girl (some cloth) and draw over with animation as overlay in back a sword swing. Or Diffrent shoes, armor etc, that can match with that animations. This way you have by having "blueprint" you quickly add/remove things you need.
Creating other assets with A.I. is not using prompts only but using your imagation. I'm using Krita with AI, i do have prompt but hes imaging that i want i just draw shape of it, again depending on "noise" it can generate existing or something compleatly new. Mostly you want to be really going into detalies about your characters. Where is legs, arms, what is where, is hair growing out of hat? Are ears on sides out hat or they are merged. A.I. as of right now won't create even with perfect dataset out of box correctt sprite or picture, you have to correct it, eather drawing yourself or prompting inpainting.......
You will notice that Creating AI is more about creating dataset that training any lora/model. Training is more like test to see if it works. To see if what you created pleases you, is it flawless? That is debetable.